mqtt-5-test/docs/VERIFICATION.md

Sprint 5 verification

Checked 2026-10-07. Scope: isolated test-mqtt-5 only. Go race tests on local Linux; Docker Go observer at http://127.0.0.1:8080; remote Mosquitto test broker 109.122.252.129, TLS 8884 and plaintext 1884. Independent Python socket clients use the ESP32 test account; Go tests use private backend/simulated-device handoffs. These checks do not prove physical ESP32 behavior.

Status: implemented with full sprint completion BLOCKED by broker restart verification. Receipt ACK functionality is implemented and verified live.

Repeatable checks

Run from the test project. rtk is a local command convention; use ordinary Linux commands on the server.

rtk env GOCACHE=/tmp/mqtt5-go-cache go test -race ./...
rtk env GOCACHE=/tmp/mqtt5-go-cache go vet ./...
rtk env GOCACHE=/tmp/mqtt5-go-cache MQTT5_INTEGRATION_CREDENTIAL_DIR=/home/ryan/mqtt-test-credentials go test -race -v ./internal/integration
rtk docker compose up -d --build observer
rtk python3 scripts/check_receipts.py --credentials esp32-test-kit/connection.txt --ca esp32-test-kit/ca.crt
rtk docker compose ps -a

The normal unit suite skips remote tests without the opt-in credential directory; that skip is not remote evidence. The separately executed opt-in suite passed all remote tests. Tests create temporary simulated clients and clean them up; the Docker simulator stays stopped.

Results and limits

Feature	Outcome	Evidence and limit
Unit/race regression and vet	PASS	All packages pass; includes ACK routing, arbitrary/empty payloads, binary correlation copying, duplicate receipt delivery, bounded queue overflow, publish failure, worker shutdown, correlation ordering and viewer tests.
Device receipt ACKs	PASS	Independent Python subscribers/publishers: MQTT 3.1.1 and 5 × plaintext and TLS; text, arbitrary JSON, empty payloads. Byte count and SHA-256 match original bytes; optional message ID echoed. Incoming message and outgoing receipt confirmed through viewer snapshot.
Native receipt routing	PASS	Actual MQTT 5 Response Topic and binary Correlation Data echoed exactly.
Loop prevention	PASS	ACK/command/receipt topics and receipt payloads suppressed; retained replay suppressed. Unit tests and bounded live no-extra-reply check.
Session resume/loss/expiry	PASS	Real TLS connection interrupted through loopback relay; session-present flag checked; subscriptions restored and fresh commands complete. Pending execution remains unknown after connection loss.
Broker-queued expiry	PASS	Persistent subscriber offline; Message Expiry=1 second with still-valid 15-second application deadline; expired packet absent, fresh control delivered after session resume.
Device application deadlines	PASS	Existing simulator unit tests prevent expired command execution; remote missing/late/wrong-correlation checks produce timeout/late/unknown rather than success.
Harmless commands and duplicates	PASS	Native correlated ping/version over TLS/plaintext; cached application duplicate executes once; conflicting ID rejected. This applies to in-memory simulator command execution, not ESP32 persistence.
Wire QoS 1 retransmission	PASS	Independent client sends QoS 1, deliberately does not read PUBACK, drops socket, resumes persistent session, resends same packet ID/payload with DUP=1. Subscriber gets two receipts; this is controlled wire retransmission, not proof of automatic hardware reconnect behavior.
Abrupt Will	PASS	Real abrupt TLS loss emits nonretained offline Will; creation timestamp retained, no invented disconnect timestamp.
Graceful offline/status and stale display	PASS / limited	Existing simulator graceful status behavior and unit tests; raw viewer labels observed time and does not derive current health from retained payloads. No general device liveness algorithm is claimed.
Scenario results	PASS	Opt-in tests publish feature/environment/endpoint/version/status/observation/time; observer logs and viewer expose reported evidence separately from untested ESP32 compatibility. Results are bounded and transient.
Authentication/TLS negatives	PASS	Independent external checks reject anonymous CONNECT, invalid password, untrusted CA and wrong hostname; successful TLS/plain sessions provide positive controls.
Unrestricted test topic access	PASS	ESP32 account publishes generated device topics and subscribes to their ACKs. ACL enforcement intentionally removed at user's request.
Graceful broker restart and persistence	BLOCKED	Server docker compose exec -T broker true fails with OCI runtime exec failed ... procReady not received. Server python3 scripts/verify.py --insecure --restart fails on its initial container health exchange, before restart. Existing broker accepts external MQTT traffic. Resolve server Docker runtime/container-exec issue, then rerun verifier; do not treat prior Sprint 2 evidence as a fresh Sprint 5 pass.
Physical ESP32 receipt consumption	Not tested	Developer must subscribe before publish, receive receipt_ack and confirm in firmware logs. User-reported inbound hardware messages are not independent proof of return-path consumption.

Final browser refresh verified 21 receipt ACKs, zero reply failures/drops, and the reported Will scenario marked passed while ESP32 native compatibility remained not_tested. Final Docker status: observer running, simulator exited. The regenerated ESP32 ZIP includes the updated wildcard subscription and receipt instructions.

Acceptance gate

• Bounded isolated scenarios and cleanup: PASS for executed scenarios.
• Resume/loss subscription recovery: PASS.
• Expired queued commands not delivered; simulator repeated valid ID executes once: PASS, bounded in-memory behavior only.
• Evidence/environment attached and no simulator results presented as hardware: PASS.
• Logs/viewer expose receipt failures, queue drops, timeout/unknown/late events and evidence limits: PASS.
• Required graceful broker restart/persistence check: BLOCKED. Sprint 5 remains unclosed until verified.

Manual device check

1. Connect using the ESP32 handoff. Subscribe to robot-market/test/v1/esp32-lab-001/acks (or the handoff wildcard) and wait for subscription success.
2. Publish any nonretained payload to robot-market/test/v1/esp32-lab-001/telemetry.
3. Confirm inbound payload, outbound receipt_ack_sent, and ACK counters in the browser. Confirm the device receives JSON with type=receipt_ack, ok=true, and the original source topic.
4. A receipt means the backend observed bytes. Broker PUBACK only means broker transport acceptance; neither proves device command execution or durable storage. Publishing on /commands or /acks does not create a receipt ACK.
5. Repeat with MQTT 3.1.1 and MQTT 5 if the firmware library supports protocol selection; record the actual library/build and CONNACK behavior. For MQTT 5, test native response/correlation properties explicitly.