Simple PBFT demo

A four-node demo of the PBFT consensus algorithm over QUIC, written to learn how it works.

Overview

A small network of four replicas and a client that agree on every request with Practical Byzantine Fault Tolerance. The replicas run a tiny key-value store, so a request is either PUT:key:value or GET:key.

Problem

PBFT lets a group of servers agree even when some of them lie, but reading about its message rounds did not make it click for me.

I wanted to watch real processes go through each round, sign their messages, and recover when the leader stops responding.

Requirements

Agree on every request with one faulty node out of four
Sign every message and check it before use
Run requests in order and only once
Replace a primary that stops making progress

Architecture

The client sends a signed request to the primary. The primary assigns a sequence number and sends a pre-prepare to the other replicas. Each replica sends a prepare; with 2f matching prepares it sends a commit, and with 2f + 1 matching commits it runs the request and replies. With four nodes, f is 1. If progress stalls for a second, the replicas start a view change and the next node becomes primary.

client      primary (0)   replica 1    replica 2    replica 3
  | request     |             |            |            |
  |------------>|             |            |            |
  |             | pre-prepare |            |            |
  |             |------------>|----------->|----------->|
  |             |<-- prepare, all to all -------------->|
  |             |<-- commit, all to all --------------->|
  |<-- reply from each replica -------------------------|

Technology choices

Rust and Tokio: Each node handles many connections and timers at once without threads per peer.
QUIC with quinn and rustls: Encrypted, multiplexed links between nodes with generated certificates.
Ed25519 and SHA-256 with ring: Fast signatures on every message and digests that tie each round to one request.
postcard: Compact binary messages that are cheap to sign.

Implementation

Each replica keeps a log per sequence number with the request, the pre-prepare, and the prepares and commits it has seen. Every phase checks that the digest matches the pre-prepare before it counts a vote.

Requests run strictly in sequence order, and a repeated request is dropped by its timestamp.
A view change carries proofs of prepared requests, so the new primary proposes them again.
A keygen binary creates the four key pairs, and the nodes retry connections while their peers start.

Major challenges

Starting four nodes that must all reach each other before any request
Counting only votes whose digest matches the request being agreed on
Carrying prepared requests safely through a view change

Testing

There are no automated tests yet. I checked the protocol by running the four nodes and the client, sending PUT and GET requests, and following each round in the logs that every node prints.

Results

Agreement holds with one faulty node out of four
Every message is signed with Ed25519 and verified before use
A stalled primary is replaced through a view change

What I'd improve

Add automated tests that run a faulty replica on purpose
Add checkpoints so the message logs can be trimmed

Code

GitHub