mqtt-security-test/DEMO.md

The enrollment demo: what it is, what it does, what it checks

A throwaway prototype that proves the board enrollment design works across three languages and a real broker. It covers Stage 0 and Stage 1 only (a raw board becomes an FN board and uses its FN topics). Activation (FN to SN) is described in docs/sn-as-username-process.md but is not built here.

Everything is in crypto-interop-spike/.

1. The idea in one paragraph

A board with no credentials connects anonymously and sends a public key. The backend answers with its own public key and a signature. Each side combines the two public keys into the same secret and turns it into the broker password, so the password is never sent. The backend creates the broker account with that password, the board logs in with it, and from then on the board can only use its own FN topics.

2. What runs

flowchart LR
    subgraph Docker network
      V[viewer<br/>web page :8099]
      M[broker<br/>Mosquitto 2.1 + Dynamic Security]
      G[backend<br/>Go]
      C[board-c<br/>C + libsodium]
      P[board-cpp<br/>C++ + OpenSSL]
    end
    C -- MQTT --> M
    P -- MQTT --> M
    G -- MQTT --> M
    G -. log file .-> V
    C -. log file .-> V
    P -. log file .-> V

Service	Language and libraries	What it does
broker	eclipse-mosquitto:2.1-alpine	MQTT broker with the Dynamic Security plugin; starts with only an admin client.
backend	Go (standard library crypto, paho MQTT)	Configures the broker, answers enrollment requests, creates FN accounts, acknowledges hello.
board-c	C, libsodium 1.0.20 or newer, libmosquitto	Simulated board.
board-cpp	C++17, OpenSSL 3, libmosquitto	A second simulated board using different libraries, to prove they interoperate.
viewer	Go standard library	Web page with a live view of all log lines.

The two boards are intentionally written with different crypto libraries: if the backend (Go) and both boards derive the same password, the algorithms and byte formats really are compatible.

3. How to run it

cd /home/ryan/Media/Code/Code/robot-market/crypto-interop-spike/docker
./run-docker.sh        # builds, starts, streams logs, prints the result

• Open http://localhost:8099 for the live view.
• docker compose logs -f shows the same lines in a terminal.
• Stop everything with docker compose down (run it in the docker directory, or pass -f .../docker/compose.yaml).
• Exit code 0 from board-c and board-cpp means all their checks passed.

Timing (set in compose.yaml): the backend waits 15 s, board-c 20 s, board-cpp 35 s before starting, and every step pauses 2 s (STEP_DELAY). These delays exist only so a person can follow along.

4. What happens, step by step

Backend (once at start):

1. Connects to the broker as the Dynamic Security admin (backend).
2. Provisions the broker: role backend-service with its own client backend-svc; role enroll for anonymous clients; role board-bootstrap for FN boards; default ACL access set to deny for everything else.
3. Connects the service client and subscribes to enroll/request/+ and board/+/hello.

Each board (ten steps):

Step	Board action
1	Generate a random Client ID, an X25519 key pair and a nonce from the CSPRNG.
2	Connect anonymously using that Client ID.
3	Subscribe to its own enroll/response/<clientid>; also try another board's topic and a wildcard.
4	Publish the request on enroll/request/<clientid> with its public key, nonce and MAC label.
5	Wait for the answer (re-send the same request if it is lost).
6	Verify the backend's Ed25519 signature with the pinned public key, plus tamper tests.
7	Derive the shared secret (X25519) and the broker password (HKDF-SHA256).
8	Log in as the FN with the derived password; also try wrong credentials.
9	Use the FN topics: subscribe, publish hello, wait for ack-hello; also try forbidden topics.
10	Print the result and exit with code 0 (all passed) or 1.

Backend per request (when it sees a board's request): reserve the Client ID, assign FN-00000N, create its own temporary key, derive the password, log only its fingerprint, create the broker client (username = FN, Client ID = FN, role board-bootstrap), sign the answer, publish it.

5. Message formats and cryptography

• Request (enroll/request/<clientid>): lines B=<64 hex>, n1=<32 hex>, mac=<text>.
• Response (enroll/response/<clientid>): lines fn=<FN>, S=<64 hex>, sig=<128 hex>. No password, no secret.
• Shared secret: X25519 between the board's private key and the backend's temporary public key S (and the reverse on the backend).
• Password: hex of HKDF-SHA256(shared, salt = n1, info = "robot-market/mqtt-pw/v1|" + FN, 32 bytes).
• Signature: Ed25519 over S || B || n1 || clientid || FN using the backend's key (seed in compose.yaml); the board holds the public half (BACKEND_PUBKEY_HEX), like firmware would.
• Logs: passwords are never logged; a short SHA-256 fingerprint (16 hex characters) is logged by both sides so they can be compared.

6. The 19 checks each board performs

#	Check	What it proves
1	Anonymous CONNECT accepted	The enrollment listener allows credential-less boards.
2	Subscribe to own enroll/response/<clientid> granted	The %c ACL gives each board its own mailbox.
3	Subscribe to another board's response topic denied (SUBACK 128)	Boards cannot read each other's replies.
4	Wildcard enroll/response/# denied	A board cannot read every reply at once.
5	Response received	The request/response round trip works through the broker.
6	Response contains fn, S, sig	The message format parses.
7	S and signature decode to 32 and 64 bytes	Field sizes are correct.
8	Signature over `S
9	Tampered signature (1 bit) rejected	Forged signatures are caught.
10	Reply bound to a different nonce rejected	Replayed or mismatched replies are caught.
11	Reply with a swapped FN rejected	The FN is part of what is signed.
12	X25519(b, S) succeeds (non all-zero)	Key agreement works and degenerate results are rejected.
13	Login with a wrong password refused	The broker really checks the derived password.
14	Right password but a different Client ID refused	Client ID binding is enforced.
15	Login as FN with derived password and bound Client ID accepted	The board derived exactly the password the backend set.
16	Subscribe to own board/<FN>/cmd granted	%u substitution works for the FN role.
17	Subscribe to another FN's cmd denied	An FN board is limited to its own topics.
18	enroll/response/# denied for the FN board	The FN board has left the enrollment permissions behind.
19	Backend's ack-hello received	Publishing hello was permitted and the backend's reply reached only this board.

Visible on the web page in addition: the two fingerprints (backend and board) must match, and the backend logs one FN per board with no duplicates.

7. Reading the web page

• Three lanes, one per program (board C, backend, board C++), in time order.
• Cards at the top: progress bar, checks passed and failed, final result, and a green chip when the board's password fingerprint equals the backend's.
• "Now" box: a plain-language explanation of the latest step.
• Options: follow live, only checks and steps, hide countdown lines. Click a long hex value to expand it.
• A refresh replays the whole run, because the server keeps every line.
• Run again button (shown only when the stack is started with docker/compose.restart.yaml, as run-docker.sh does; the server deployment does not use it; there the button opens the Gitea "Restart demo" workflow page instead): restarts the backend, both boards and the broker (clean Dynamic Security state), clears the logs and replays the whole run. It needs the Docker socket mounted into the viewer container (/var/run/docker.sock), which gives that container control of Docker on the host, so the page is bound to 127.0.0.1 and the restart endpoint requires a custom header that other web pages cannot send. Demo use only.

8. File map

Path	Contents
docker/compose.yaml	The stack, timings, secrets (demo values), shared log volume.
docker/broker/	mosquitto.conf, entrypoint.sh (creates the admin client on every start).
docker/backend/	Go backend (main.go), vendored dependencies, Dockerfile.
docker/board-c/	C board (main.c), Dockerfile.
docker/board-cpp/	C++ board (main.cpp), Dockerfile.
docker/viewer/	Log viewer and restart button (main.go, index.html), Dockerfile.
docker/run-docker.sh	Build, run, stream logs, show results.
go/, c/, c-libsodium/, cpp/, run.sh, run-libsodium.sh	The earlier fixed-key programs that check the crypto libraries against the RFC test vectors and against each other (no networking).

9. What it proved, and what it did not

Proved

• Go (standard library), libsodium and OpenSSL produce identical X25519, HKDF and Ed25519 results, including RFC 7748, RFC 5869 and RFC 8032 test vectors.
• The whole enrollment works over a real broker with Dynamic Security, with the password never transmitted.
• The ACLs behave as designed: own mailbox only, no wildcards, FN topics only, Client ID binding.
• Retries are safe: a repeated request does not create a second FN (the first version of the backend did; fixed with an in-flight reservation).

Findings from building it

• %c/%u substitution in dynsec ACLs does not work in Mosquitto 2.0.22; it works in 2.1.
• Changing a connected client's roles disconnects it, so the backend uses a separate admin client and service client.
• mosquitto_ctrl addRoleACL needs an explicit allow or deny.
• Go dependencies are vendored because the Docker build could not reach the Go module proxy in this environment.

Not covered

• Activation (FN to SN), operator approval, SAS, replacement and retirement.
• TLS, real secret storage (demo secrets are in compose.yaml), rate limiting and expiry of unclaimed FNs.
• Real hardware: nothing ran on an ESP32 or Orange Pi; the libsodium board was built against Alpine's libsodium, not the ESP-IDF component.
• Domain-separated signatures and zeroizing of secrets (recommended in the design doc, not in the prototype).
• Load, failure injection (broker restart, lost responses beyond the retry), and concurrency beyond two boards.