What Alopex Chirps Is — A Library for Forming Clusters, and the Consensus It Adopted

Building a distributed database means building the same things again before getting to the actual work: a way to find nodes, a way to judge whether a peer is alive, a way to carry messages between nodes, and a way to settle what was decided. The same four appear, in one form or another, in distributed runtimes, service meshes, and edge clusters.

Alopex Chirps is a library that takes on exactly those four. It is published on crates.io under Apache-2.0 OR MIT.

This blog previously ran a four-part series, Consensus Algorithms, Underground, organizing consensus algorithms by failure model and implementation structure. This article applies that perspective to Chirps: what it takes on, and which consensus it adopted. Links to all four parts are collected at the end.

What It Handles and What It Does Not

What Chirps provides is a control plane. It does not handle data itself.

Its scope is the following.

• Node identity (a persistent node_id) and cluster join via a static seed list
• Membership propagation through SWIM-style gossip
• Messaging over QUIC/TLS (send_to / broadcast / subscribe)
• Ordering and commitment through Raft
• Event notification for join, leave, and status change

Outside its scope are the storage engine, query processing, scheduling, and the contents of the state machine. What gets applied, and how, is decided by the caller.

It can be placed underneath a distributed database, or used as the control plane of a distributed runtime, a service mesh, or an edge/IoT cluster.

Three Layers, Each Answering a Different Question

Chirps is internally divided into three layers, and that separation is the center of the design.

Consensus Algorithms, Underground Part 1 established that leader election, ordering, and commitment are separate problems. In Chirps that separation is the crate boundary itself: chirps-transport-quic, chirps-gossip-swim, and chirps-raft-storage with the raft module correspond to the three rows above.

The important part is the difference in the third column. What SWIM returns is an estimate; what Raft returns is a decision. Confusing the two turns a gossip misjudgment directly into a forked history.

Why QUIC

QUIC is a transport built on UDP. It combines the retransmission and ordering that TCP provided, encryption via TLS, and multiplexing of several streams. Chirps uses the Rust implementation quinn, opening one UDP endpoint per node for communication with all peers.

There are two reasons for the choice.

The first is that streams are independent of one another. A QUIC connection carries multiple streams, and head-of-line blocking does not occur between them. In TCP, when a single packet is lost on a connection, everything queued behind it waits. Chirps places consensus messages, gossip, snapshot transfer, and application messages on the same connection, so this property is a prerequisite. A single packet lost in the middle of a large snapshot transfer must not stall Raft heartbeats. The implementation opens a unidirectional stream per message and assigns a priority by stream kind.

The second is that encryption is a given. QUIC embeds TLS; a plaintext connection is not an option. Chirps uses rustls and identifies connections with the ALPN protocol alopex. There is no need for a design that layers TLS onto node-to-node communication afterwards.

Consensus Uses openraft

The choice was openraft v0.9.17. Cargo.toml pins it with an equals sign: openraft = { version = "=0.9.17", features = ["serde"] }.

As Consensus Algorithms, Underground Part 2 established, most of a production Raft implementation lives outside the papers: persistence ordering, batching, backpressure, snapshots, membership changes, linearizable reads. What Chirps implements itself is storage and transport, not consensus.

The role of Chirps is therefore to form a cluster and give it ordering and commitment. Connect the nodes, track liveness, and do not lose what was decided. It is a dependency to add once those four are needed, not something to read in order to choose an algorithm.

Gossip Estimates Do Not Enter Consensus

Chirps has two notions of membership, and not conflating them is the practical reason the layers are separated.

The first is SWIM gossip membership. It holds alive / suspect / dead status plus an incarnation number, transitioning via ping, indirect ping, and timeouts.

PING B
  ↓ no response
PING-REQ B (via C, D)
  ↓ no response
mark B suspect
  ↓ suspect_to_dead_timeout
mark B dead

The second is Raft membership, that is, the voter set. It is changed through change_membership, which goes through openraft’s joint consensus.

leader.add_learner(4, BasicNode::default()).await?;
leader.change_membership(BTreeSet::from([1, 2, 4])).await?;

As Consensus Algorithms, Underground Part 1 established, crashes and delays cannot be distinguished. A gossip verdict of dead therefore does not mean the node may be removed from the voter set. SWIM status changes are delivered to the caller as events, and changes to the voter set are made explicitly through a separate path.

Failure detection and membership dissemination are not consensus. The distinction laid out in Consensus Algorithms, Underground Part 4 is implemented in Chirps as a layer boundary.

The Vessel Is Provided; the Meaning Is the Caller’s

Chirps holds no state machine. The caller implements StateMachine and passes it in.

#[async_trait]
impl StateMachine for KvStateMachine {
    type Command = Vec<u8>;
    type Response = Vec<u8>;

    async fn apply(&mut self, log_id: LogId<u64>, command: Self::Command)
        -> StateMachineResult<Self::Response> { /* update state here */ }

    async fn snapshot(&self) -> StateMachineResult<Box<dyn AsyncSnapshotData>> { /* ... */ }
    async fn restore(&mut self, snapshot: Box<dyn AsyncSnapshotData>) -> StateMachineResult<()> { /* ... */ }
}

Both command and response are Vec<u8>. Chirps guarantees only the ordering and commitment of byte sequences and does not interpret their contents. Whether a KV store, a job queue, or configuration distribution runs on top is the caller’s concern.

Where the log lives is Chirps’ concern. WalRaftStorage implements openraft’s storage abstraction, backed by the WAL primitives of alopex-core 0.3. Five kinds of record are written.

pub enum RaftWalRecord {
    AppendLog(Entry<ChirpsTypeConfig>),
    Vote(Vote<ChirpsNodeId>),
    SnapshotApplied(SnapshotMeta<ChirpsNodeId, BasicNode>),
    Truncate(LogId<ChirpsNodeId>),
    Purge(LogId<ChirpsNodeId>),
}

The distinction organized in Consensus Algorithms, Underground Part 1 — that appending, committing, and applying are separate events — appears here as record kinds. The file header carries the magic "RWAL", a format version, group_id, and node_id. Snapshots carry the magic "SNAP" and a CRC32 per chunk; the unit of verification is the chunk, not the file.

Vote persistence has no configuration knob. save_vote performs a synchronous write without exception.

self.write_frame(&WalFrame::Record(RaftWalRecord::Vote(*vote)), true)

Log appends, by contrast, are tunable through WalStorageConfig::fsync_interval. The default of 0 means an fsync per entry; a value greater than 0 batches the fsync every that many entries. Votes are always synchronous and not configurable, log appends are configurable, and the default is the safe side.

Four Kinds of Traffic Share One Connection

Consolidating the control plane into a single library means traffic of different natures flows over the same QUIC connection: consensus messages, gossip, snapshot transfer, and the caller’s application messages.

Priorities are therefore assigned per stream kind.

Even among Raft-derived traffic, Raft is High while RaftSnapshot is Normal. The I/O saturation from snapshot transfer noted in Consensus Algorithms, Underground Part 2 manifests as a new node’s catch-up pushing heartbeats aside and inducing an unrelated leader election.

Profiles are also enforced at runtime.

if is_raft_frame(frame) && matches!(requested, MessageProfile::Ephemeral) {
    warn!("Ephemeral requested for Raft traffic; overriding to Control");
    return Ok(MessageProfile::Control);
}

A request to send a Raft frame over the best-effort path without retransmission is warned about and returned to the reliable path. The enforcement is necessary precisely because the caller’s application messages share the same API.

Using It

The transport is reused as-is. ChirpsRaftTransport implements openraft’s RaftNetwork and RaftNetworkFactory and delegates to Chirps’ MessageBackend, so the consensus layer never holds a socket of its own.

leader.initialize(BTreeSet::from([1, 2, 3])).await?;
let response = leader.propose(b"hello".to_vec()).await?;
leader.trigger_snapshot().await?;

propose does not forward automatically when the node is not the leader; it returns NotLeader along with the leader ID. The caller performs the redirect.

The default timings assume a LAN: 150 ms election timeout, 50 ms heartbeat, a maximum AppendEntries batch of 1,000, and a snapshot threshold of 10,000 entries. As Consensus Algorithms, Underground Part 1 noted, these numbers are the partial synchrony assumption itself and must be raised in environments with larger RTTs.

For operations, Prometheus metrics (raft_state, raft_term, raft_commit_index, raft_applied_index, raft_leader_id, among others) and structured log events under target="raft" are emitted. Commit and apply, separated in Consensus Algorithms, Underground Part 1, are separate series in the metrics as well; a widening gap between them indicates a stall on the state machine side.

What Is Available Now, and What Comes Next

The read path is consolidated into propose. is_leader() merely consults current_leader in the metrics, so it cannot be used to identify a stale leader after a partition. Linearizable reads corresponding to the ReadIndex and leader lease approaches noted in Consensus Algorithms, Underground Part 2 are future work.

RaftNode handles a single group. GroupId is used in the WAL path and in the validation of incoming frames, and messages whose group ID does not match are rejected. Since the separation of identity and routing is already in place, Multi-Raft — running a Raft group per range, as covered in Consensus Algorithms, Underground Part 2 — is the next implementation to build on top. For use underneath a distributed database, that is the one that matters.

MessageProfile supports Control and Ephemeral. Durable is a stub that warns and returns an error when requested; it is to be implemented as delivery backed by a durable queue.

Summary

• Chirps takes on only the control plane of a distributed system. It handles node discovery, membership, messaging, and consensus, and does not handle storage or the contents of the state machine.
• Internally it separates into QUIC (whom to connect to), SWIM (who appears to be alive), and Raft (which history is official). The layers are separated because an estimate and a decision are different kinds of answer.
• Consensus uses openraft v0.9.17; what is implemented in-house is storage and transport.
• A gossip verdict of dead does not change the voter set. The voter set is changed explicitly through change_membership.
• The state machine is implemented by the caller. Chirps guarantees ordering and commitment of byte sequences and does not interpret them.
• Because consensus, gossip, snapshots, and application traffic share one connection, the paths are separated by stream priority and profile enforcement.
• Linearizable reads, Multi-Raft, and the Durable profile are future work.

The Series: Consensus Algorithms, Underground

The four-part series referenced throughout this article:

• Consensus Algorithms, Underground (Part 1) — Crashes and Delays Are Indistinguishable
• Consensus Algorithms, Underground (Part 2) — The Crash Failure Family: Paxos and Raft
• Consensus Algorithms, Underground (Part 3) — Lying Nodes, from PBFT to DAG-BFT
• Consensus Algorithms, Underground (Part 4) — A World Where Identities Cannot Be Counted: PoW and PoS