mqtt-5-test/docs/sprints/05-reliability.md

Sprint 5 — Reliability scenarios and compatibility evidence

Status: implemented, with live ACK and reliability checks PASS; full completion BLOCKED by the server container-exec failure preventing the broker restart verifier. See verification evidence. Read the plan, sprint index and shared rules, and README.

Goal and scope

Turn the working local environment into a repeatable test harness with explicit, evidence-backed feature results.

Deliverables and likely files

• tests/integration/ or equivalent opt-in Go integration suite; minimal scenario helpers.
• Simulator controls for delayed/missing ACKs and controlled harmless duplicate tests.
• docs/VERIFICATION.md scenario checklist and known limitations; event/result output consumable by logs and current-process viewer.

Tasks

1. Implement explicit scenario result records with feature, environment, endpoint/version, status, observation and time; unknown remains not_tested.
2. Exercise broker graceful restart, network interruption, session resume/loss/expiry and subscription restoration.
3. Test queued command expiry separately from device-side expiry after delivery; synchronize clocks and use bounded test deadlines.
4. Test duplicate requests, conflicting IDs, wrong/late/reordered/missing ACKs, invalid payloads and overloaded buffers.
5. Verify Will after abrupt client stop versus graceful offline status, and stale-status handling.
6. Test auth/TLS failures and unrestricted cross-device topic access (the user explicitly disabled ACLs for this test broker). Test and native Response Topic/Correlation Data echo. Attempt actual QoS 1 retransmission with controlled interruption; label unreproduced behavior honestly.
7. Surface structured scenario verdicts in logs and viewer without adding persistent accounting. Document why repeated application publication is different from wire-level DUP retransmission.

Added scope: backend receipt ACKs (2026-10-07)

Implement device publication → backend receipt → device application ACK, separate from broker PUBACK and device replies to commands. Raw observation continues to accept arbitrary payloads without schema, timestamp or device-ID checks.

• Enable with receipts.enabled=true in raw observe mode; Docker enables it.
• Subscribe on the device to <namespace>/<topic-device>/acks before publishing. Route by the incoming topic, not JSON or username. A valid in-namespace MQTT 5 Response Topic overrides the default, with exact Correlation Data echoed.
• Reply at QoS 1, retain false: type=receipt_ack, ok=true, original source_topic, observed time, payload byte length and SHA-256; echo optional message_id/request_id without requiring them.
• Never reply to commands, ACK/receipt topics, receipt_ack payloads or retained replays. Repeated live publications may each get a receipt ACK; receipt confirmation does not execute a device action.
• Keep reply queue bounded and log receipt sent/failed/dropped. Broker acknowledgement of the reply does not prove the device consumed it. Do not claim persistence or exactly-once handling.
• Test raw text, empty/invalid JSON, optional IDs, mismatched JSON identity, unchanged timestamp placeholder, native reply routing/correlation, duplicate delivery, reply failure, queue overload, cancellation and ACK-loop prevention. Run live independent subscriber/publisher checks over MQTT 3.1.1 and MQTT 5 with both plaintext/TLS, then inspect UI inbound/outbound events.

Dependencies and required inputs

Sprints 1–4; local broker control and a safe way to interrupt only test connections/containers. Do not change host-wide firewall or unrelated networking.

Acceptance criteria

• Scenarios are bounded, repeatable, isolated, and leave no unrelated processes/network changes.
• Resume and session-loss paths both recover subscriptions and readiness correctly.
• Expired commands are not executed; repeated valid IDs execute once within the documented in-memory retention window.
• Every compatibility verdict contains evidence/environment; simulator results are not marked physical-device results.
• Logs/browser explain failures, gaps, timeouts, late responses and limits without secrets.

Automated checks

Unit/race suite and opt-in real-broker scenarios with explicit pass/fail/skip outcomes and cleanup. Avoid timing-only assertions where an observable state can synchronize the test.

Manual verification

Follow the checklist with the viewer open; observe reconnect, expiry, Will and late ACK events. Review the resulting feature table and distinguish unsupported, blocked and failed.

Risks and unresolved decisions

QoS 1 wire retransmission may be nondeterministic; report not reproduced rather than substituting an application duplicate. Broker graceful persistence is not crash-durability proof.

Mandatory end-of-sprint testing and completion gate

After implementing this sprint, run its automated checks and manual verification above. Inspect the final changes for correctness, scope, secret exposure, and compatibility with completed sprints. Fix failures and rerun affected checks before declaring completion.

Record the exact commands/checks, actual outcomes, verification environment, and each acceptance criterion as PASS, FAIL, or BLOCKED. Include applicable regression checks for previously completed behavior. A skipped/unavailable required check is BLOCKED, never PASS; do not claim full sprint completion while required checks fail or remain blocked. Do not run unrelated or production-project tests.

Report the results and remaining limitations to the user. Wait for authorization before the next sprint unless continued execution was explicitly authorized. Even with permission to continue, do not bypass this verification gate.

Copy-ready implementation prompt

Implement Sprint 5 only in /home/ryan/Media/Code/Code/robot-market/test-mqtt-5 after authorization. Read applicable AGENTS.md, /home/ryan/.codex/RTK.md, README.md, docs/PLAN.md and docs/sprints/05-reliability.md. Use rtk. Never alter production brokers, siblings, or host-wide networking.

Build a focused opt-in real-broker reliability/compatibility harness around the existing Go backend, simulator and local viewer. Cover graceful broker restart, isolated network interruption, session resume versus loss/expiry, subscription restoration, broker-queued Message Expiry, device application deadlines, duplicates/conflicting request IDs, wrong/late/reordered/missing ACKs, Will versus graceful offline state, validation/overload and TLS/auth failures and unrestricted authenticated topic access. Prove native Response Topic/Correlation Data behavior with actual properties. Attempt controlled QoS 1 retransmission separately from deterministic repeated application requests and report unreproduced cases honestly.

Emit explicit feature/environment/evidence/status results to logs and the existing viewer, leaving untested features not_tested. Keep all state and test operations bounded, provide cleanup, and avoid a database or monitoring stack. Run meaningful unit/race and isolated integration checks. Update docs/VERIFICATION.md and known limitations; distinguish local simulator, remote and hardware evidence. Report results and wait.

ESP32 reference: read docs/PLAN.md section 6.1 for the supplied Esp32MqttConfig. It is connection configuration, not a payload schema or a blocker for Go implementation. Preserve the proposed Go test contracts. Device-specific MQTT 5 selection depends on the actual C library/build/init; inspect TLS/CA handling, role-specific credentials and single publish_topic limitations during device integration. Do not modify existing C projects. Separate MQTT 3.1.1 payload interoperability from native MQTT 5 feature evidence.

Mandatory completion gate: after implementing this sprint, run its listed automated and manual checks plus relevant regression checks, inspect the final changes for correctness/scope/secrets/compatibility, fix failures and rerun affected checks. Report exact checks, real outcomes, environment, and every acceptance criterion as PASS/FAIL/BLOCKED. Missing required checks are BLOCKED, never a pass. Do not mark the sprint complete with unresolved failures/blockers or bypass verification when continuing to another sprint.