Deploy MQTT test backend with private viewer and Gitea CI/CD

docs/BACKEND-SERVER.md

@@ -0,0 +1,45 @@
+# Backend deployment and Gitea Actions
+
+Repository: `ssh://git@git.my-rm.com:2222/Robot-Market/mqtt-5-test.git`. Deployment branch: `main`. Target: existing test server `109.122.252.129`, separate Compose project `mqtt5-test-backend`, runtime `/opt/test-mqtt-5/backend`. Broker remains in `/opt/test-mqtt-5/deploy`.
+
+The backend uses TLS to the existing broker, accepts raw test messages and sends receipt ACKs. No simulator is deployed. Private credentials and CA are provisioned on the server; never include ESP32 ZIP, private JSON, local .env or certificates/keys in Git. Event history is in memory and resets when the backend restarts.
+
+## Access the UI
+
+Run on your local computer and keep it running:
+
+```sh
+ssh -N -L 8080:127.0.0.1:18080 root@109.122.252.129
+```
+
+Open http://127.0.0.1:8080. If local port 8080 is occupied, use `-L 8081:127.0.0.1:18080` and open http://127.0.0.1:8081. The server listens only on 127.0.0.1:18080. No public firewall port is required. Stop a local receipt observer while using this server backend to avoid two independent receipt ACK publishers.
+
+## CI/CD
+
+`.gitea/workflows/backend.yml` runs on main pushes, main pull requests and manual dispatch. The test job builds the Docker test target (unit tests plus vet); remote integration tests are opt-in and skipped without private handoffs. The deploy job runs only on main, never for a PR. It builds an immutable UTC timestamp plus commit tag, then runs `server/deploy.sh`. No registry or SSH deployment secret is needed: the existing ubuntu-latest runner uses this server's Docker socket.
+
+The runner configuration allows `/opt/test-mqtt-5/backend` in `container.valid_volumes`; only the deploy job requests that mount as `/runtime`. The prior runner configuration is backed up before this addition. Existing frontend job mounts remain unchanged. This workflow assumes that label continues to use the same server Docker daemon; moving the runner requires updating the deployment arrangement.
+
+`deploy.sh` locks the runtime directory, validates Compose, preserves the previous image-tag file and Compose, deploys only backend, and checks container health for up to 60 seconds. Health requires the UI snapshot to report MQTT connected. Failed updates restore the previous release; a failed first deployment stops the backend. Images are retained for rollback. No host-wide image prune is run. Remote device return-path checks should still be performed after significant MQTT changes.
+
+## Server operations
+
+Standard Linux commands on the server:
+
+```sh
+cd /opt/test-mqtt-5/backend
+docker compose --env-file .env -f compose.yaml ps
+docker compose --env-file .env -f compose.yaml logs -f backend
+docker compose --env-file .env -f compose.yaml restart backend
+```
+
+Manual rollback to the preserved release:
+
+```sh
+cd /opt/test-mqtt-5/backend
+cp .env.previous .env
+cp compose.previous.yaml compose.yaml
+docker compose --env-file .env -f compose.yaml up -d --force-recreate backend
+```
+
+During deployments do not run a simultaneous manual rollback. Before changing broker credentials, update the server's private `secrets/backend.json` (root:10001, mode 0640, parent directory 0750) and restart backend. Do not rotate the broker's existing credentials as part of ordinary backend deployments.

docs/CONFIGURATION.md

@@ -0,0 +1,73 @@
+# Configuration and offline CLI
+
+Requires Go 1.27.1. The CLI supports offline `check-config`, MQTT `observe`, and `command ping|get_version`. The device simulator is available separately; see [RUNNING.md](RUNNING.md) for TLS/plaintext commands and visible message flow. An optional read-only HTTP/SSE viewer is available in observe mode; see [RUNNING.md](RUNNING.md).
+
+```sh
+rtk go run ./cmd/test-mqtt5 --help
+rtk go run ./cmd/test-mqtt5 --config config.example.toml check-config
+rtk go run ./cmd/test-mqtt5 --config config.example.toml --device-id lab-001 --log-level debug check-config
+rtk go build -buildvcs=false -o bin/test-mqtt5 ./cmd/test-mqtt5
+```
+
+Flags precede the subcommand. `-buildvcs=false` avoids stamping errors when this folder sits inside an unavailable/incomplete parent Git checkout. It does not alter Git state.
+
+## Precedence and environment mapping
+
+Defaults < optional TOML file < environment < explicitly supplied flags. No file is loaded automatically. An explicit missing/malformed file fails. Unknown keys and invalid types fail, including in empty TOML tables. Relative file paths resolve against the process working directory.
+
+Every key in [config.example.toml](../config.example.toml) has an environment equivalent: prepend `TEST_MQTT5_`, uppercase, and replace the dot between section/key with `__`. Preserve single underscores in field names.
+
+| TOML key | Environment name | Example |
+| --- | --- | --- |
+| mqtt.url | TEST_MQTT5_MQTT__URL | tls://localhost:8883 |
+| mqtt.client_id | TEST_MQTT5_MQTT__CLIENT_ID | rm5-lab-observer-01 |
+| mqtt.password | TEST_MQTT5_MQTT__PASSWORD | Provide a test-only secret outside tracked files |
+| mqtt.password_file | TEST_MQTT5_MQTT__PASSWORD_FILE | secrets/backend-password |
+| mqtt.session_expiry | TEST_MQTT5_MQTT__SESSION_EXPIRY | 5m |
+| tls.ca_file | TEST_MQTT5_TLS__CA_FILE | certs/ca.pem |
+| topics.device_id | TEST_MQTT5_TOPICS__DEVICE_ID | sim-001 |
+| logging.format | TEST_MQTT5_LOGGING__FORMAT | json |
+| simulator.ack_delay | TEST_MQTT5_SIMULATOR__ACK_DELAY | 0s |
+| web.enabled | TEST_MQTT5_WEB__ENABLED | false |
+
+All fields and defaults are listed in the example TOML. Durations must be strings with units (`0s`, `5s`, `10m`); bare numbers and overflowing values fail. Booleans accept true/false. Integers reject fractional values. Empty environment values override previous values, including clearing a password; nonempty password and password_file cannot be combined.
+
+## Validation limits
+
+- Broker URL defaults to verified `tls://host:port`. Test-only `mqtt://host:port` requires `mqtt.allow_plaintext=true` (or --allow-plaintext). Port is 1–65535; embedded credentials/path/query/fragment are prohibited.
+- Device/client IDs and namespace segments use safe identifiers; namespace remains under robot-market/test/v1.
+- QoS is 0 or 1. Session expiry is whole seconds, 0–4294967295; keepalive is whole seconds, 1–65535.
+- Positive operation/connection/shutdown intervals are bounded; reconnect_max is at most 30s and >= reconnect_min.
+- Command lifetime is whole seconds, 1–60; dedup retention must exceed it.
+- Payload limit is 256–16384 bytes; preview is 64–4096 and no larger than payload limit.
+- Pending/queue/history/dedup capacities are 1–10000; viewer count is 1–8.
+- Viewer address is 127.0.0.1 with a valid port. The viewer binds only to 127.0.0.1; max viewers is 1–8 and each viewer queue is 1–256 events.
+- Certificate/key must be paired. Configured secret/certificate paths must reference readable, nonempty regular files up to 1 MiB.
+
+## Output and limitations
+
+Success uses configured text or JSON slog output, with `event=config_valid`, `network_checked=false` and `credentials_checked=false`. Success remains visible at the configured level, including warn/error. Invalid configuration/CLI errors go to stderr with exit 1/2 and omit supplied values. Help exits 0. Cancellation exits 1.
+
+Empty test passwords are allowed for offline validation. This check does not load/validate password-file contents, parse certificates, establish trust, verify the broker hostname, authenticate, or test MQTT compatibility. Runtime modes parse certificates, authenticate, and check MQTT transport acknowledgements. CA paths are optional; TLS uses system trust plus a configured test CA, with hostname verification always enabled.
+
+Keep real credentials in ignored config.local.toml, environment or secrets files. Never log a complete Config. Default previews of MQTT payloads are allowlisted as described in CONTRACTS.md.
+
+## Checks
+
+```sh
+rtk gofmt -l cmd internal
+rtk go vet ./...
+rtk go test ./...
+rtk go test -race ./...
+```
+
+During Sprint 1 verification, module/build caches used `/tmp/test-mqtt5-mod` and `/tmp/test-mqtt5-build`; these are temporary local caches, not required for ordinary usage.
+
+Additional test observation settings:
+
+- `topics.observe_devices`: comma-separated additional device IDs, maximum 16; observe mode only. Each device is subscribed using exact category topics. `topics.device_id` remains the target for single-device command mode.
+- `logging.raw_payloads`: false by default; enables bounded raw MQTT payload display in logs and browser for test diagnostics. Docker observer explicitly enables it. Invalid contracts still produce validation errors and never imply successful compatibility.
+
+`topics.observe_all=true` is observe-mode-only: subscribe to `<namespace>/#`, record raw payloads and skip all JSON contract validation. Compose enables this setting for functional testing.
+
+`receipts.enabled=true` enables asynchronous backend receipt ACK publication in raw observe mode only. The reply queue uses limits.inbound_queue; publication uses deadlines.operation. It does not introduce JSON admission checks.

docs/CONTRACTS.md

@@ -0,0 +1,76 @@
+# Test message contracts
+
+Sprint 1 implements structural validation and safe inspection previews. Sprint 3 implements network exchange, native MQTT properties, request tracking and duplicate execution handling. These contracts are independent of the ESP32 connection configuration.
+
+## Topics and common fields
+
+Topics: `robot-market/test/v1/{device_id}/{telemetry|status|commands|acks}`. The namespace may have additional safe lab segments under `robot-market/test/v1`.
+
+Every payload is a JSON object with `schema_version: 1` and a `device_id` matching the topic. IDs contain 1–64 letters/digits/underscores/hyphens, beginning with a letter/digit. Timestamps use RFC3339 UTC ending in `Z`. Maximum payload is 16 KiB. Unknown fields, duplicate keys (including case aliases), nulls, arrays, invalid UTF-8, trailing JSON and excessive nesting are rejected. Validation errors exclude raw values.
+
+| Category | Additional required fields | Rules |
+| --- | --- | --- |
+| telemetry | message_id, boot_id, sequence, timestamp, readings | Sequence is an unsigned integer, including zero. Readings has 1–16 safe-name finite numeric values. |
+| status | boot_id, state and one time field | State is online/offline. Use timestamp for explicit publication, or will_configured_at for a preconfigured offline Will. Never both. |
+| commands | request_id, issued_at, expires_at, command | Only ping/get_version. Lifetime is 1–60 seconds. |
+| acks | request_id, timestamp, stage, ok | ok must be an explicit boolean. Stage rules below. |
+
+Status optionally reports implementation and reported_protocol (`5` or `3.1.1`). Reported protocol is self-reported, not compatibility proof.
+
+## Examples
+
+Telemetry:
+
+```json
+{"schema_version":1,"device_id":"sim-001","message_id":"evt-001","boot_id":"boot-001","sequence":0,"timestamp":"2026-10-06T10:00:00Z","readings":{"uptime_s":12}}
+```
+
+Status:
+
+```json
+{"schema_version":1,"device_id":"sim-001","boot_id":"boot-001","state":"online","timestamp":"2026-10-06T10:00:00Z","implementation":"go-simulator","reported_protocol":"5"}
+```
+
+Command:
+
+```json
+{"schema_version":1,"device_id":"sim-001","request_id":"req-001","issued_at":"2026-10-06T10:00:00Z","expires_at":"2026-10-06T10:00:15Z","command":"ping"}
+```
+
+Acceptance:
+
+```json
+{"schema_version":1,"device_id":"sim-001","request_id":"req-001","timestamp":"2026-10-06T10:00:01Z","stage":"accepted","ok":true}
+```
+
+Completion:
+
+```json
+{"schema_version":1,"device_id":"sim-001","request_id":"req-001","timestamp":"2026-10-06T10:00:01Z","stage":"completed","ok":true,"result":{"pong":true}}
+```
+
+Failure:
+
+```json
+{"schema_version":1,"device_id":"sim-001","request_id":"req-001","timestamp":"2026-10-06T10:00:15Z","stage":"rejected","ok":false,"error":{"code":"expired","message":"command deadline passed"}}
+```
+
+Accepted requires ok=true with no result/error. Completed requires ok=true and exactly one result: pong=true or a bounded version identifier. Rejected/failed requires ok=false, a safe error code and a 1–256-byte error message, without result. Error messages are always redacted in inspection previews because they can contain arbitrary data.
+
+## Deadline and acknowledgement semantics
+
+`Command.Validate` checks structure and lifetime. `CheckDeadline(now)` additionally rejects commands before issued_at and at/after expires_at. Historical examples remain structurally valid even after their deadline. Device clocks must be synchronized for execution checks.
+
+Transport acknowledgement, application acceptance and device completion are separate. The structural decoder validates ACK shape. The Sprint 3 backend additionally matches active own requests, command-specific results and native correlation; simulator deduplication and transport properties are covered by local/remote tests. Broader expiry/fault and physical-device verification remain later sprints. JSON request_id is not a native MQTT property.
+
+## Safe inspection
+
+`events.SafePreview` decodes and validates before rendering known fields. Unknown/sensitive fields, invalid JSON and binary payloads cause complete preview omission; payload length remains visible. Free-form ACK error messages become `[redacted]`. Preview output is at most the requested limit, capped at 4 KiB, and has an explicit truncation flag. Truncated text is display text, not necessarily valid JSON.
+
+The event model provides direction, topic, identities, QoS, retain/DUP, properties, lifecycle kind, request ID and attributable latency. The implemented transport bounds and sanitizes property metadata before creating events; do not attach raw broker user properties or credentials.
+
+## Native MQTT 5 transport (Sprint 3)
+
+Commands use non-retained QoS 1 with Response Topic equal to the exact device ACK topic, bounded Correlation Data and MQTT Message Expiry alongside the JSON application deadline. Backend command mode uses the request ID bytes as native correlation. The simulator accepts opaque nonempty correlation values up to 64 bytes and echoes them unchanged on every ACK. ACK publications are non-retained QoS 1 and have no Response Topic. Pending completion requires matching device, topic, JSON request ID, native correlation and command-specific result; retained ACKs are rejected. Completed may arrive before accepted without regressing state.
+
+Dedup identity includes canonical command JSON and native correlation. In-progress duplicates replay acceptance, completed duplicates replay the cached terminal ACK, and conflicting reuse is rejected without execution. Cache retention is bounded and memory-only. MQTT transport acknowledgement is separate from application admission and execution; dropped input, missing completion and connection loss cannot be reported as execution success.

docs/DEPLOYMENT.md

@@ -0,0 +1,162 @@
+# Deploy the isolated test broker on your server
+
+This package is ready for operator deployment. Compose, setup generation and Mosquitto configuration were checked locally. The user deployed the broker; server authentication/ACL/graceful-persistence checks and external TLS/plaintext MQTT exchanges passed on 2026-10-06.
+
+The user requested server deployment artifacts and an optional plaintext connection alongside TLS. Both listeners require test credentials. Topic ACLs are disabled for functional testing: every authenticated account can publish and subscribe to any topic. Plaintext sends credentials/messages without encryption; expose that test port only to intended test clients.
+
+## Files to copy
+
+Copy the project's `deploy/` directory to a separate server directory, for example `/opt/robot-market-mqtt5-test`. Do not copy existing production credentials/configuration. The scripts require Linux, Docker Engine, Docker Compose, Python 3 and OpenSSL. All deployment commands use standard Linux tools.
+
+The official image is pinned to:
+
+```text
+eclipse-mosquitto:2.1.2-alpine@sha256:38c0da4f2ef84284d47b3b3eeea1cb3bdeabe81ee10caf0cd5c5ff61ee3ea408
+```
+
+This immutable multi-platform index includes Linux amd64, ARM64, ARMv6, 386, ppc64le and s390x. Confirm your server matches a supported platform. The originally proposed `2.1.2` tag is absent; `2.1.2-alpine` is the verified official tag. [Official image metadata](https://github.com/docker-library/official-images/blob/master/library/eclipse-mosquitto)
+
+Commands below run from the copied deployment directory, not the Go project root:
+
+```sh
+cd /opt/robot-market-mqtt5-test
+docker compose version
+docker compose pull
+sudo python3 scripts/prepare.py --host mqtt-test.example.com
+```
+
+Replace `mqtt-test.example.com` with the exact server DNS name **or public IP** your clients will use. Do not include a scheme or port. The helper makes a dedicated private test CA and a 90-day server certificate with that name/IP plus localhost/127.0.0.1 SANs. A client connecting under a different name must fail hostname verification.
+
+Preparation generates random test passwords, hashes them using the pinned image's `mosquitto_passwd`, and never prints them or passes them in process arguments. It runs a network-disabled disposable hashing container; it does not start the broker. It refuses to overwrite `.env`, `certs/`, `secrets/` or `data/`. If it fails after partial generation, inspect privately and use a fresh dedicated directory rather than overwriting unknown state.
+
+Default runtime owner is numeric UID/GID 1883:1883. The helper sets broker-readable files to owner/group-restricted modes and gives the data directory to that UID. CA signing keys remain outside the broker mounts. Running preparation with sudo is needed to assign that ownership on a normal server. Rootless Docker/user-namespace mappings require deliberate permission adaptation; they have not been verified here.
+
+## Select ports and start
+
+Edit the generated `.env` for available server ports. Default host ports intentionally avoid common production ports:
+
+| Transport | Default server port | Exposure |
+| --- | --- | --- |
+| MQTT over TLS | 8884 -> container 8883 | Base Compose file |
+| Plaintext MQTT | 1884 -> container 1883 | Only with insecure override |
+
+The base file publishes TLS on IPv4 `0.0.0.0`. To limit exposure, set TLS_BIND_IP to the appropriate server interface. The broker's internal plaintext listener is available to containers on its Docker network; the base file does not publish it on the server host.
+
+TLS only:
+
+```sh
+docker compose config --quiet
+docker compose run --rm --no-deps broker --test-config -c /mosquitto/config/mosquitto.conf
+docker compose up -d
+```
+
+TLS plus plaintext, as requested:
+
+```sh
+docker compose -f compose.yaml -f compose.insecure.yaml config --quiet
+docker compose -f compose.yaml -f compose.insecure.yaml run --rm --no-deps broker --test-config -c /mosquitto/config/mosquitto.conf
+docker compose -f compose.yaml -f compose.insecure.yaml up -d
+```
+
+Use the same file combination for subsequent operations. `config --quiet` avoids printing resolved configuration. Passwords are file-mounted, not Compose environment variables. All application payloads still need validation regardless of transport.
+
+Allow the chosen TLS port through the provider firewall/security group and server firewall. If using plaintext, allow 1884 only from intended test client IPs. Docker-published ports may bypass ordinary UFW rules; use provider rules or Docker-aware filtering (DOCKER-USER/nftables appropriate to your Docker setup) and verify from an external client. Do not edit existing broker listeners or expose 1883/8883 merely because they are common defaults.
+
+## Credentials and topics
+
+Private generated files (read locally on the server, do not commit):
+
+| File | Purpose |
+| --- | --- |
+| secrets/clients/test-backend.json | Dedicated Go backend account |
+| secrets/clients/sim-001.json | Simulator/device test identity sim-001 |
+| secrets/clients/esp32-lab-001.json | Physical test identity esp32-lab-001 |
+| secrets/runtime/passwords | Hashed broker account database |
+| secrets/runtime/health.conf | Health-only account in protected CLI config |
+| certs/server/ca.crt | Public test CA to copy to clients |
+| certs/server/server.key | Broker private key; stays on server |
+| certs/authority/ca.key | Private CA signing key; not mounted into broker |
+
+The Go backend username is `test-backend`; its password is in its JSON handoff file. Device usernames match their test device IDs. **Functional test mode: any authenticated account can publish and subscribe to any topic.** Username and topic device IDs do not need to match. The Mosquitto ACL plugin is not loaded. No password/certificate regeneration is needed when switching an existing test deployment to this mode.
+
+Both MQTT 3.1.1 and MQTT 5 clients can connect. The browser observer subscribes to `robot-market/test/v1/#` and shows raw payloads without requiring a JSON schema, matching device ID or valid timestamp.
+
+Copy `ca.crt` and the backend credential handoff through your normal private file-transfer mechanism. When ready, provide the broker hostname/IP, selected TLS/plaintext ports and a path to the private backend credential file plus the public CA on your development computer. There is no need to paste passwords into chat.
+
+If you changed ports in `.env`, update the port metadata in the client JSON files or explicitly use the correct ports. For the existing C config, mqtt_server/port/user/pass/ca_cert come from these test settings. Wi-Fi remains device-specific. Select the actual C MQTT protocol/library separately.
+
+The Go backend/simulator support verified TLS and explicit test-only plaintext transport (mqtt.allow_plaintext=true or --allow-plaintext with a mqtt:// URL). See [RUNNING.md](RUNNING.md). TLS hostname/CA verification is never disabled.
+
+## Verify after deployment
+
+Before attaching real devices, run:
+
+```sh
+sudo python3 scripts/verify.py --restart
+```
+
+For the plaintext override:
+
+```sh
+sudo python3 scripts/verify.py --insecure --restart
+```
+
+The verifier requires a running dedicated test broker. It uses temporary protected client config files and pinned-image CLI containers in that broker's network namespace, with no new host ports. It tests TLS health exchange, authorized device/backend flow, anonymous/bad-password/bad-CA/wrong-host rejection, unrestricted cross-device publication and reception. Optional flags check plaintext and graceful restart retained/session recovery. Commands do not expose real credentials as arguments/output.
+
+Run it on a quiet test instance: it temporarily sets and clears the retained status for esp32-lab-001 and, with --restart, restarts this broker. It never publishes retained commands. The current functional verifier expects cross-device traffic to succeed. Graceful restart is not crash-durability evidence. A tool failure is not an MQTT denial. The verifier checks MQTT 5 PUBACK reason codes for publishes: Mosquitto 2.1.2 can exit zero after reporting a rejected publish, so an exit code alone is insufficient. Successful publications must receive a non-error PUBACK; missing acknowledgements are inconclusive. The user ran the corrected verifier on 2026-10-06 against the deployed server; all server-container checks passed. External TLS/plaintext authenticated exchanges also passed from the development computer on 2026-10-06.
+
+The container health check independently publishes and receives a unique token over authenticated TLS and clears its retained health probe. It verifies listener/auth/ACL/message flow, not just process liveness.
+
+Inspect status and logs (add the insecure file when applicable):
+
+```sh
+docker compose ps
+docker compose logs --tail 50 broker
+docker compose exec broker /bin/sh /usr/local/bin/mqtt-healthcheck
+docker compose port broker 8883
+```
+
+Verify externally from the development computer as well: DNS/IP, firewall reachability, certificate chain and SAN, authentication, and actual MQTT exchange. The server-container verifier does not prove external routing. Record separate local-artifact, server-runtime and physical-hardware results.
+
+## Stop, update, rollback and persistence
+
+```sh
+docker compose stop broker
+docker compose start broker
+docker compose restart broker
+docker compose down
+```
+
+The broker saves session/retained data under `data/`. Graceful shutdown has 30 seconds; autosave runs every 30 seconds. Queue sizes, max clients, packet sizes and container resources are bounded. Docker stdout logs rotate at 5 MiB × 3 files. In-memory/broker counters are not per-device durable accounting.
+
+For a consistent backup, stop this test broker first, then archive `.env`, tracked configuration, `certs/`, `secrets/` and `data/` into a private backup location. Restart after backup. For example, from the deployment directory:
+
+```sh
+docker compose stop broker
+sudo tar -czf /root/mqtt5-test-backup.tar.gz .env compose.yaml compose.insecure.yaml mosquitto.conf acl scripts certs secrets data
+sudo chmod 600 /root/mqtt5-test-backup.tar.gz
+docker compose start broker
+```
+
+Choose a unique backup name each time. Before an update, preserve the old Compose image reference and a consistent data snapshot. Change the pinned tag/digest intentionally, pull, run --test-config, then recreate only this broker and rerun verification. For rollback, stop the broker, restore the prior configuration/image and matching data snapshot in a dedicated directory, fix UID/GID modes, and start it. Do not assume newer persistence data can be opened by an older image.
+
+`down` keeps bind-mounted data. Deleting `data/` is an explicit reset and loses sessions/retained messages; do not delete production paths or add automatic destructive cleanup.
+
+## Certificate renewal
+
+This package uses a private test CA, not automatic public ACME issuance. Store the signing key privately and track CA/server expiration. Renew before the 90-day server certificate expires. Do not rerun preparation over existing state; it deliberately refuses to rotate credentials/certificates implicitly.
+
+To renew a server certificate, generate a new key/CSR in a private staging directory, sign with the existing test CA using the SANs in server.ext, and verify chain plus hostname/IP. Keep old cert/key for rollback. Stop only this broker, install the new matching pair in certs/server with owner/group and 0640 permissions, start it, rerun TLS health and external client verification. Removing/replacing the whole directory while the container runs can leave old bind-mount references; stop/recreate when replacing directories. When CA trust changes, distribute the new public CA to clients before switching server trust.
+
+A public certificate can be integrated later with a deliberate health-check hostname/trust adjustment; public CAs do not issue localhost SANs, so simply replacing this private certificate can break the local TLS health check. No remote DNS, ACME method or renewal timer is assumed.
+
+## Validation completed during preparation
+
+- Registry manifest/digest verified for the actual 2.1.2-alpine tag.
+- Base and TLS+plaintext Compose configurations: PASS (`config --quiet`).
+- Shell/Python syntax: PASS.
+- Setup helper with throwaway credentials/certificates in /tmp: PASS, hashed passwords produced without secret output.
+- Mosquitto 2.1.2 `--test-config` in network-disabled disposable container: PASS; plugins/configuration accepted, no broker started.
+- Server-container runtime TLS/plaintext/auth/ACL/graceful-restart checks: PASS, based on user-provided verifier output on 2026-10-06; external TLS/plaintext authenticated exchanges also passed.
+- External client connectivity: PASS, authenticated MQTT 5 device telemetry publish/backend receive on TLS 8884 (CA and hostname verification enabled) and plaintext 1884. Non-retained unique probes only.
+- Physical hardware: NOT VERIFIED.

docs/DEVICE-MESSAGE-DIAGNOSTICS.md

@@ -0,0 +1,45 @@
+# Current functional test mode — 2026-10-07
+
+The user requested unrestricted functional testing. This supersedes the earlier diagnosis and restrictions described below.
+
+- The live remote broker's ACL plugin was removed over SSH and the broker restarted. An original configuration backup is on the server. Password authentication remains; any authenticated account can publish/subscribe to any topic.
+- The Docker observer was rebuilt and restarted with `topics.observe_all=true`. It subscribes to `robot-market/test/v1/#`, displays raw payloads and performs no JSON-schema, device identity or timestamp validation.
+- Independent socket-client checks passed for MQTT 3.1.1 and MQTT 5, over plaintext 1884 and TLS 8884. Each used the **esp32-lab-001** account for BOTH publication and subscription on **sim-001/telemetry**, sending the user's exact JSON with a literal `{{$isoTimestamp}}`. The broker delivered it and the browser snapshot recorded `message_received`. Arbitrary device/category topic plus plain text also passed in all four combinations.
+- Go tests passed with the race detector. These checks exercise the deployed broker and viewer without using the simulator; the Postman application itself still needs a successful connection.
+
+Live broker logs also show repeated clients attempting **TLS against the plaintext listener** (host port 1884 mapped to container 1883), resulting in protocol errors before MQTT authentication. A client doing this cannot publish anything, regardless of payload or topic permissions. In Postman create a new MQTT request and use `mqtt://109.122.252.129:1884` for plaintext, or `mqtts://109.122.252.129:8884` for TLS with the supplied CA. SSL certificate verification settings control certificate checks, not the choice between plaintext and TLS.
+
+## Earlier diagnosis (superseded)
+
+
+
+## Findings and corrections
+
+The deployed observer was connected to the remote broker but subscribed only to `sim-001`. The supplied physical-device kit publishes under `esp32-lab-001`, so those packets never reached the observer. Compose now configures both identities using exact ACL-authorized topics: telemetry, status, commands and acks. Message device identity is derived from the topic before contract validation and display.
+
+Incoming payloads previously had to match the simulator JSON contract before their contents could be displayed. Arbitrary Postman/device payloads produced a generic validation event with omitted contents. The Docker test observer now explicitly enables bounded raw diagnostics. Failed contracts show their received contents and a validation reason, while successful compatibility evidence still requires a valid contract. Default non-Docker configuration keeps raw diagnostics disabled.
+
+Device accounts can publish their own telemetry/status/acks and read commands. A Postman subscriber must use the test-backend account to receive telemetry. Publishing credentials and topic identity must match. No broker ACL change was needed.
+
+## Verification
+
+The observer Docker image was rebuilt and the observer container recreated. The simulator remains stopped. Go tests passed with the race detector. Existing opt-in Go remote tests were skipped in that run; the following independent live check was run separately.
+
+A standalone Python socket client, independent of the Go simulator and Paho transport, used the ESP32 kit credentials to publish QoS 1 non-retained telemetry. A separate test-backend client subscribed to the same exact topic. Each check verified successful CONNECT, SUBACK, PUBACK, exact payload delivery to the independent subscriber, and matching events in `/api/snapshot`.
+
+| Transport | Protocol | Valid telemetry | Plain-text diagnostic |
+| --- | --- | --- | --- |
+| TCP 1884 | MQTT 3.1.1 | PASS | PASS |
+| TCP 1884 | MQTT 5 | PASS | PASS |
+| TLS 8884 with trusted CA | MQTT 3.1.1 | PASS | PASS |
+| TLS 8884 with trusted CA | MQTT 5 | PASS | PASS |
+
+Valid payloads appeared as `message_received` for `esp32-lab-001`. Plain-text payloads appeared as `validation_failed`, with the exact bounded payload visible. This proves the deployed broker/client/viewer path; it does not claim that physical ESP32 firmware or the Postman UI was exercised.
+
+## Quick physical-device or Postman test
+
+Connect to `mqtt://109.122.252.129:1884`, using the ESP32 kit username/password and a distinct client ID. Publish `hello from esp32` to `robot-market/test/v1/esp32-lab-001/telemetry`, QoS 1, retain false. It should appear in the browser at http://127.0.0.1:8080 as a contract error with its text. Then publish the kit JSON to get a valid contract event.
+
+To receive that publication in Postman, use the local test-backend credential handoff and subscribe to `robot-market/test/v1/esp32-lab-001/telemetry`. A device-account telemetry subscription is not authorized. To receive commands as a device, subscribe to `robot-market/test/v1/esp32-lab-001/commands`.
+
+Refresh the browser after rebuilding and clear topic/device filters. Subscriptions cover only the configured devices and four categories, not arbitrary broker topics. Payloads are limited to 16 KiB, previews to 4 KiB. The observer cannot determine a publisher's negotiated protocol from a normal forwarded payload; hardware MQTT 5 proof still requires broker session evidence or native-property command replies.

docs/PLAN.md

@@ -0,0 +1,288 @@
+# MQTT 5 test backend plan
+
+Planning date: 2026-10-06. Status: Sprints 1–4 implemented and verified, including the local browser viewer; later sprints await authorization.
+
+## 1. Objective and authorization
+
+Build a small Go test backend that receives telemetry/status, sends harmless commands, correlates application acknowledgements, and makes every relevant input/output and compatibility result understandable through logs and an optional local browser viewer.
+
+The user authorized Sprints 1–4, then directed Sprint 2 toward a server Compose package they will deploy themselves, including optional plaintext MQTT alongside TLS. Sprint 1 is complete; Sprint 2 artifacts are validated locally, the user deployed the server broker, and server runtime plus external connection checks passed. Existing projects remain untouched. Sprint 3 MQTT 5 transport/backend/simulator, logs, deduplication, deadlines and graceful shutdown are implemented and verified locally and against the dedicated remote broker. Later sprints still require authorization.
+
+Project root: `/home/ryan/Media/Code/Code/robot-market/test-mqtt-5`.
+
+Follow applicable AGENTS.md instructions and `/home/ryan/.codex/RTK.md`; prefix shell commands with `rtk`. The existing server repository's build/test restrictions apply when working there; do not build or test that reference project as part of this work.
+
+## 2. Findings and evidence
+
+Reference root: `../robot-market-vending-backend/server_backend` (relative to the new project root).
+
+| Evidence | Finding | Limit |
+| --- | --- | --- |
+| `go-back/go.mod` | Go 1.25.0, `github.com/eclipse/paho.mqtt.golang v1.5.0` | This is the MQTT 3 client, not the MQTT 5 implementation |
+| `go-back/internal/mqtt/client.go` | TLS >=1.2, optional CA/client certificates, 30-second keepalive, auto reconnect, role-based client IDs | Does not establish the remote broker's active configuration |
+| `go-back/internal/config/config.go` | Default QoS 1 and CleanSession=false | Defaults are not evidence of deployed values |
+| `go-back/cmd/ingest/main.go` and `internal/ingest/subscriber.go` | Startup subscription registration; connection callback primarily logs | No explicit subscription rebuilding on lost broker session in this inspected path |
+| `docs/contracts/handoffs/server-mqtt-correlation-ack-handoff-fa.md` | JSON `requestId` or `message_id`/`sync_job_id`; explicit `ok:true`; exact response-topic matching | These are application fields, not native MQTT 5 properties |
+| `docs/phases/phase2/12-device-apt-repo-foundation.md` | Bookworm ARM64 packages for `app-core`, `io`, `mqtt-bridge`, `storage`, `payment` | Evidence of a Linux device service; its source/library/version was not available |
+| `docs/architecture/runtime-and-deploy.md` | Mosquitto deployment references external `vending/docker` | Actual deployment artifacts, OS, CPU, listeners and certificates unavailable |
+| Repository instructions | FN identifies the board; SN identifies the machine | This project must not allocate or mutate those identities |
+
+Existing topics include `devices/...`, `device/{plate}/...`, and `vending/{serial}/...`. Some synchronization responses use `device/{plate}/sync/resp/{message_id}`. The new namespace intentionally isolates test traffic.
+
+No ESP32 firmware, PlatformIO project, ESP-IDF configuration, or device MQTT implementation was found in the supplied parent project tree during discovery. “Device backend” appears to include a Linux service; the relationship to ESP32 remains unknown. Do not infer direct ESP32 MQTT connectivity or firmware capabilities from these documents.
+
+Do not read/copy production secrets. No existing code changes are planned.
+
+## 3. Architecture
+
+```mermaid
+flowchart LR
+    Browser[Local browser] -->|Loopback HTTP + SSE| Backend[Go test backend]
+    Backend -->|Outbound MQTT 5 / TLS| Broker[Isolated Mosquitto]
+    Simulator[Local Go simulator] -->|Outbound MQTT 5 / TLS| Broker
+    Device[Test hardware or device service] -->|Outbound TLS / verified protocol| Broker
+```
+
+The diagram arrows show connection initiation. MQTT messages travel in both directions over those connections. No public inbound access to the development computer is required.
+
+The current broker deliverable is a server deployment package under deploy/. The user will deploy a separate broker instance using available TLS and optional plaintext test ports; existing production listeners stay untouched. No permanent local broker is required. If local isolation is later needed, bind host ports to loopback.
+
+### Go structure and runtime
+
+Planned entry points: `cmd/test-mqtt5` and `cmd/device-sim`. The backend has `observe` and `command` CLI modes; `command` supports `ping` and `get_version`. The viewer belongs to long-running observation mode. An observer can display command/ack traffic from a separate command-mode client but must not claim that traffic as its own measured pending-request latency.
+
+Use focused `internal/config`, `internal/contracts`, `internal/mqtt`, `internal/backend`, `internal/simulator`, and `internal/events` packages; add `internal/web` only with the viewer sprint. Package names may be simplified if responsibilities remain clear. Use standard `flag`, `net/http`, `embed`, `encoding/json`, and `log/slog` rather than frameworks.
+
+Construct dependencies explicitly. Use contexts and operation deadlines, contextual wrapped errors, fixed worker counts, bounded channels, and clear mutex/event-loop ownership. Interfaces only support actual testing or transport boundaries. Callbacks copy necessary data and enqueue quickly; they do not do HTTP, disk work, slow logging, or application execution. Connection hooks signal a supervised worker rather than starting unbounded goroutines.
+
+On SIGINT/SIGTERM: stop admitting commands, stop HTTP/SSE, resolve pending requests as interrupted/unknown, publish graceful offline status where applicable within a bounded deadline, disconnect MQTT, cancel and join workers. Return a meaningful exit status.
+
+### Resource defaults
+
+Initial lab defaults: one simulator device, telemetry every 5 seconds, 16 KiB maximum application payload, 128 pending commands, 256 queued inbound events, 1,000 displayed events, 4 KiB payload preview, eight SSE viewers, and a 64-event queue per viewer. Make necessary limits configurable and validate them; do not make a flag for every setting.
+
+On telemetry/event overflow: count and expose a gap/drop. On command-processing overload: never execute work that was not admitted; emit a rejection when feasible, otherwise allow a visible application timeout. MQTT receipt/transport ACK is not proof of admission. Event-stream loss must not affect command tracking. Clamp negotiated MQTT receive/packet limits to supported values and test the actual pinned client's behavior.
+
+## 4. Dependencies and version policy
+
+These are proposal pins established during discovery, not a tested lockfile. Before implementation, confirm tag existence, Go requirements, security advisories, and image platform/digest availability. Report any required version change instead of silently changing the approved design.
+
+| Dependency | Proposed version | Source |
+| --- | --- | --- |
+| Go toolchain | 1.27.1 | https://go.dev/dl/ |
+| `github.com/eclipse/paho.golang` including `autopaho` | v0.23.0 (requires Go >=1.24) | https://github.com/eclipse-paho/paho.golang/releases/tag/v0.23.0 |
+| `github.com/knadh/koanf/v2` | v2.3.7 (requires Go >=1.23) | https://github.com/knadh/koanf/releases/tag/v2.3.7 |
+| `github.com/knadh/koanf/parsers/toml/v2` | v2.2.2 | https://pkg.go.dev/github.com/knadh/koanf/parsers/toml/v2@v2.2.2 |
+| `github.com/knadh/koanf/providers/file` | v1.2.1 | https://pkg.go.dev/github.com/knadh/koanf/providers/file@v1.2.1 |
+| `github.com/knadh/koanf/providers/env` | v1.1.0 | https://pkg.go.dev/github.com/knadh/koanf/providers/env@v1.1.0 |
+| Mosquitto | Verified `eclipse-mosquitto:2.1.2-alpine@sha256:38c0da4f2ef84284d47b3b3eeea1cb3bdeabe81ee10caf0cd5c5ff61ee3ea408` | https://mosquitto.org/download/ |
+| ESP32 SDK/client | Not selected until device source/board is known | https://github.com/espressif/esp-mqtt |
+
+Paho is released but still pre-1.0: API stability must not be assumed. Commit exact module versions and go.sum when implementation begins. Do not use `paho.mqtt.golang` in the new MQTT 5 implementation. No Node/npm dependency is needed for the viewer.
+
+Additional official references:
+
+- Autopaho controls and lifecycle: https://raw.githubusercontent.com/eclipse-paho/paho.golang/v0.23.0/autopaho/auto.go
+- koanf loading/providers: https://github.com/knadh/koanf
+- MQTT semantics: https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html
+- Mosquitto settings: https://mosquitto.org/man/mosquitto-conf-5.html
+- Mosquitto 2.1 migration notes: https://mosquitto.org/blog/2026/01/version-2-1-0-released/
+- ESP-MQTT MQTT 5 example: https://components.espressif.com/components/espressif/mqtt/versions/1.0.0/examples/mqtt5?language=en
+
+ESP-MQTT can support MQTT 5, but build configuration and runtime protocol selection must be checked. Do not choose or migrate an ESP32 framework without inspecting the actual device project.
+
+## 5. Configuration
+
+Precedence: defaults < TOML file < environment < explicitly supplied CLI overrides. Use typed structs and load once at startup. A supplied missing/malformed file is an error. Only flags actually visited override merged values.
+
+Prefix: `TEST_MQTT5_`. Strip that prefix, lowercase, and map `__` to nested `.` while preserving single underscores: `TEST_MQTT5_MQTT__CLIENT_ID` -> `mqtt.client_id`. Reject unknown keys under this prefix. Treat empty secrets deliberately; never silently fall back to a different credential.
+
+Sections cover broker URL, username/password or password-file, TLS CA/client certificate paths, client ID, topic namespace/device ID, QoS, initial Clean Start, Session Expiry, keepalive/backoff, operation/shutdown deadlines, logging, event limits, simulator behavior, and optional web viewer. Mutually exclusive secret sources receive actionable validation errors.
+
+Use duration strings such as `5s` and `5m`, explicit decode hooks/conversion, and overflow/range checks before converting MQTT seconds to uint32. Reject bare numeric durations, invalid QoS, unsupported URL schemes, partial certificate pairs, wildcard/path injection in IDs, and deadlines inconsistent with deduplication retention. Unknown TOML keys should fail startup.
+
+Flags: `--config`, `--device-id`, `--log-level`, and optionally `--web` are sufficient. Command arguments select the harmless operation and request ID. Never provide secrets through flags or log complete configuration objects.
+
+TLS remains the default for broker connections. Per user request, the server package also offers an explicit Compose override publishing authenticated plaintext MQTT. Sprint 3 adds an explicit test-only mqtt.allow_plaintext setting (default false) to permit a mqtt:// URL when selected; implemented Sprint 3 validation supports plaintext only with the explicit opt-in. TLS connections always verify hostnames/SANs, using system roots and optionally a test CA. Client certificate/key is optional. Never set InsecureSkipVerify. Ignore local TOML, generated certificates, private keys, broker data, and credential files in Git. Examples contain placeholders only.
+
+## 6. Topics and payloads
+
+### 6.1 ESP32 connection configuration (supplied by user)
+
+The user supplied this C connection configuration and clarified that it is not the application payload contract:
+
+```c
+typedef struct
+{
+    const char* wifi_ssid;
+    const char* wifi_password;
+    const char* mqtt_server;
+    int         mqtt_port;
+    const char* mqtt_user;
+    const char* mqtt_pass;
+    const char* subscribe_topic;
+    const char* publish_topic;
+    const char* ca_cert;
+} Esp32MqttConfig;
+```
+
+This struct does not constrain the Go application's message structs or establish payload encoding. Go backend/simulator implementation is not blocked on a C serializer. The topic/payload examples in section 6.2 remain the proposed test contract; actual device payload compatibility is verified when the device code is available.
+
+| C setting | Meaning for this environment | Go/backend impact |
+| --- | --- | --- |
+| wifi_ssid / wifi_password | ESP32 network connection | Device-only; never copy Wi-Fi credentials into Go config or logs |
+| mqtt_server / mqtt_port | Address of the isolated test broker and TLS listener | Go uses its own TLS URL for the same reachable broker; confirm whether C expects hostname or URI |
+| mqtt_user / mqtt_pass | Dedicated device credentials | Go uses separate backend credentials; ACLs reflect each role |
+| subscribe_topic | Device's test command subscription | Set to `robot-market/test/v1/{device_id}/commands` for the proposed contract |
+| publish_topic | A configured device publication topic | One string alone does not establish support for telemetry, status and ACK topics; inspect publish call sites before choosing a device adaptation |
+| ca_cert | Device trust material | Inspect whether this is PEM text or a path; Go uses system roots/its own CA path. Never copy C pointers as configuration values |
+
+The existing fields can still describe MQTT 5 broker connectivity. No struct change is required solely to move application payloads to MQTT 5. This struct has no explicit protocol-version field, so it cannot establish that the device uses MQTT 3.1.1 or MQTT 5: inspect the actual library/version, build options and client initialization. Port and certificate settings alone also do not prove TLS is enabled or hostname verification is enforced.
+
+If the library supports MQTT 5, select it in client initialization/build configuration. Only add a protocol-version field to Esp32MqttConfig if runtime selection between 3.1.1 and 5 is actually needed. Do not prescribe enum values or exact library calls before identifying the C library. MQTT 5 Session Expiry, Clean Start, Will, Response Topic, Correlation Data and Message Expiry are configured through that library's supported APIs and need not all become fields in this small struct. Preserve JSON/application correlation for any 3.1.1 compatibility tests; never label it native MQTT 5 metadata. Reference: [OASIS MQTT 5 specification](https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html).
+
+For the proposed four-topic test contract, inspect whether existing publish calls accept a topic per message. If yes, derive telemetry/status/ACK topics from a test namespace and device ID. If publication is fixed to publish_topic, propose the smallest device-only extension (for example namespace/device ID or explicit additional topic settings) before implementing it. Do not silently change the Go contract to one multiplexed topic, or subscribe to production topics. No C changes are authorized by documenting these alternatives.
+
+Verification for device integration: isolated broker host/port, CA representation, TLS enablement and hostname verification, device clock, separate credential/ACL behavior, exact subscription/publication topics, library protocol selection and actual message/property exchange. For a 3.1.1 endpoint, ordinary payload interoperability with the Go MQTT 5 backend is testable through Mosquitto, but native property handling remains unsupported unless that endpoint is upgraded. The Go backend and simulator continue to use Paho MQTT 5 as planned.
+
+### 6.2 Proposed test topics and payloads
+
+Namespace: `robot-market/test/v1/{device_id}`. Test IDs use a restricted safe alphabet, such as `sim-001` or `esp32-lab-001`. Optional FN/SN metadata is informational only.
+
+| Suffix | Direction | Default QoS | Retain |
+| --- | --- | --- | --- |
+| telemetry | Device -> backend | 1 | false |
+| status | Device -> backend | 1 | true |
+| commands | Backend -> device | 1 | false |
+| acks | Device -> backend | 1 | false |
+
+The proposed test messages have `schema_version:1` and `device_id`; match identity against the topic. Esp32MqttConfig does not define these payloads. Real-device payload support must be checked separately. Reject oversized input before decoding. Validate JSON objects, command allowlists, bounded strings, required identifiers, RFC3339 UTC timestamps, supported stages, and reasonable deadline ranges. MQTT payload-format/content-type properties can label JSON, but payload validation remains necessary.
+
+Telemetry example:
+
+```json
+{"schema_version":1,"device_id":"sim-001","message_id":"evt-001","boot_id":"boot-001","sequence":1,"timestamp":"2026-10-06T10:00:00Z","readings":{"uptime_s":12}}
+```
+
+Online status example:
+
+```json
+{"schema_version":1,"device_id":"sim-001","boot_id":"boot-001","state":"online","timestamp":"2026-10-06T10:00:00Z","implementation":"go-simulator","reported_protocol":"5"}
+```
+
+Reported protocol is self-reported metadata, not independent proof. An offline Will carries `state:"offline"`, boot ID and `will_configured_at`; it must not label a preconstructed timestamp as the disconnect time. The observer adds its own reception time.
+
+Command and completion examples:
+
+```json
+{"schema_version":1,"device_id":"sim-001","request_id":"req-001","issued_at":"2026-10-06T10:00:00Z","expires_at":"2026-10-06T10:00:15Z","command":"ping"}
+```
+
+```json
+{"schema_version":1,"device_id":"sim-001","request_id":"req-001","timestamp":"2026-10-06T10:00:01Z","stage":"completed","ok":true,"result":{"pong":true}}
+```
+
+Acceptance is a separate ACK with `stage:"accepted"`. Terminal stages are `completed`, `rejected`, and `failed`; rejection/failure contains a bounded error code/message. Out-of-order or repeated ACKs must not regress terminal state. A completion received before acceptance still resolves the request correctly.
+
+MQTT 5 command properties: Response Topic equals the exact device `acks` topic; Correlation Data is bounded opaque bytes and is echoed unchanged. Compare native bytes and JSON request ID when native correlation is enabled. Reject response topics outside the exact allowed topic, including wildcards. A documented JSON-only compatibility mode is separately selected and cannot count as a native-property test.
+
+These contracts are test-only and are not derived from Esp32MqttConfig. Existing `requestId`, `message_id`, `/resp` and sync behavior is not silently reinterpreted; any real-device payload adaptation requires an explicit mapping, and no production namespace bridge is included.
+
+## 7. Reliability and session policy
+
+- Client IDs include lab/role/instance, e.g. `rm5-lab-observer-01`, `rm5-lab-command-01`, `rm5-lab-device-sim-001`. Concurrent processes need distinct configured instance IDs. Never reuse a production ID.
+- Initial Clean Start=true and Session Expiry=300 seconds. Within a running process, reconnect with Clean Start=false. Client-process restart deliberately starts fresh because client protocol/pending/dedup state is initially in memory. Cross-process durable client recovery is outside this baseline.
+- On every CONNACK inspect Session Present. When false, restore required subscriptions and check SUBACK reasons before readiness. Callback handlers must already exist when a resumed session can deliver queued messages. If session exists, retain its subscriptions rather than blindly replaying them.
+- Reconnect backoff uses jitter and a bounded 1–30-second range; context cancellation always stops it. Initial connection has a deadline. Denials are observable and do not produce tight retry loops.
+- Publish retained online status only after command subscription readiness. Configure retained QoS 1 offline Will with explicit zero Will Delay. Graceful shutdown attempts offline publication before normal disconnect. Network failure detection still depends on keepalive; absence/staleness is unknown, not proof of current online state.
+- Commands default to a 15-second application lifetime and MQTT Message Expiry consistent with remaining lifetime. Expired messages are never newly published or executed. A delayed delivery must be checked against expires_at even if broker expiry was set.
+- No unbounded offline application publish queue. If disconnected, reject new CLI commands visibly. Already submitted QoS traffic may still complete or be retransmitted after the caller times out.
+- Distinguish publish attempt, successful transport acknowledgement, application acceptance, and completion. Inspect MQTT reason codes; a returned publish result is not automatically success.
+- Register correlation before publishing and after ACK subscription readiness. Local timeout means unknown delivery/execution outcome; late ACKs are visible and classified late.
+- Intentional retry reuses request ID, operation, and original deadline. The simulator deduplicates by device/request ID, remembers in-progress and final results, and rejects same-ID/different-content conflicts. Do not rely on the MQTT DUP bit as the application deduplication key.
+- Dedup cache is bounded (initially 1,024 requests, ten-minute retention; command lifetime capped below retention). Do not evict active/unexpired protected entries to accept more work: reject overload. Cache loss on restart is documented; no exactly-once side-effect guarantee is claimed.
+- Message Expiry limits broker delivery; it cannot cancel already delivered work. The device checks application deadlines before work and applies its own cancellation policy. Device time must be synchronized before TLS/deadline hardware tests.
+
+## 8. Logs and input/output inspection
+
+Use one sanitized event model for logs and browser display. Event fields include event ID, observed UTC timestamp, direction (`in`, `out`, `internal`), local client/device identity, topic, QoS, retain/DUP flags when available, payload byte length, bounded preview, MQTT properties, request ID, outcome/reason, and measured latency where locally attributable.
+
+Differentiate events such as `publish_attempt`, `transport_ack`, `message_received`, `validation_failed`, `command_accepted`, `command_completed`, `command_timeout`, `late_ack`, `duplicate_request`, `connection_up`, `connection_down`, and `event_gap`. Never display a publish attempt as proven device receipt.
+
+Default safe previews show the explicitly allowed harmless test fields. Redact secret/token/password fields before logging; do not dump arbitrary invalid JSON or binary payloads. Invalid payload events show metadata and validation errors without echoing raw input. Diagnostic preview opt-in remains bounded and sanitized. Never log credentials, TLS private material, config dumps, or URL userinfo.
+
+Provide text logs for humans and JSON logs for scripts. Basic counters: attempts/connections/reconnects, publish attempts/transport successes/failures, received messages, validation failures, ACK stages, own-request acceptance/completion latency, timeouts, duplicate requests/ACKs, and dropped inspection events. Periodic and shutdown summaries are enough; no metrics server is required.
+
+Usage labels are explicit: application payload bytes exclude MQTT/TCP/TLS overhead; subscriber counts describe its deliveries, not total broker traffic; $SYS generally measures the broker; in-memory counters are not durable accounting. Telemetry history means only the bounded current-process event buffer unless separately approved.
+
+## 9. Optional browser viewer (Sprint 4 complete)
+
+Use Go net/http + embedded HTML/CSS/vanilla JavaScript. Default disabled; `--web` enables `127.0.0.1:8080`. No CDN, frontend build tool, or separate service. Changing to a public bind is outside this baseline.
+
+Read-only pages show connection/readiness, counters, an event timeline with direction/topic/device/request filters, payload/property details, command lifecycle, and a compatibility table. Show empty, disconnected, truncated, and event-gap states. Pause freezes display, not backend processing.
+
+Use GET endpoints for initial sanitized event snapshot and Server-Sent Events for live updates. Event IDs allow replay from the bounded ring on reconnect; if history is gone, emit an explicit gap/reset. Slow browser clients use bounded queues and are disconnected or notified of gaps without blocking MQTT. Clean up subscriptions on close. No durable browser history or command-publishing HTTP endpoint.
+
+Render text via textContent, never payload-supplied HTML. Add a restrictive CSP, same-origin behavior, expected Host/Origin checks, no permissive CORS, and no remote script loading. This is a local developer tool, not a public authenticated dashboard. SSE reconnects must not duplicate event counters or multiply handlers.
+
+## 10. Compatibility reporting
+
+Track each feature per endpoint with `not_tested`, `passed`, `failed`, `unsupported`, or `blocked`, plus evidence, timestamp, endpoint identity, client version when known, and environment (`local_simulator`, `remote_simulator`, `physical_device`). A protocol claim alone cannot mark a feature passed.
+
+| Feature | Go simulator | Existing Linux device service / ESP32 |
+| --- | --- | --- |
+| TLS, topic/payload contract, QoS 1 | Planned automated proof | Unknown until actual client/source and configuration inspected |
+| JSON command/ACK correlation | Planned | Requires test contract support; old payloads are not automatically compatible |
+| Native Response Topic / Correlation Data | Planned end-to-end echo verification | Requires MQTT 5 client and property handling |
+| Session expiry and subscription restoration | Planned controlled outage scenarios | Requires configured client behavior and observed results |
+| Broker Message Expiry | Planned queued-delivery scenario | Broker-side behavior plus device deadline enforcement tested separately |
+| Will/retained online state | Planned crash and graceful-stop scenarios | Requires test configuration and freshness interpretation |
+| Duplicate request handling | Planned deterministic repeat plus optional wire retransmission | Requires actual application idempotency behavior |
+
+A backend cannot infer another subscriber's MQTT version merely from a delivered message. Use device client source/configuration plus test-broker connection evidence and successful native-property behavior. Cross-version delivery alone proves neither MQTT 5 support nor property preservation.
+
+Initially, live events and scenario results are separate concepts: merely seeing traffic does not auto-pass every compatibility row. Scenario tests emit explicit verdicts. The browser can show supported live evidence and loaded/available current-process results; untested rows remain not_tested. No upload/import framework or persistent report database is planned.
+
+## 11. Broker deployment and operations
+
+Prefer Docker Compose on a supported server. If Docker is unavailable, propose an exact native package/systemd alternative before implementation. Confirm OS/CPU, permissions, existing port use, DNS/SANs, certificate provisioning, and firewall first.
+
+Use dedicated test data/configuration and a verified image digest. Enable authenticated TLS and disable anonymous access. The base Compose file publishes only TLS (default server port 8884); the user-requested insecure override additionally publishes plaintext MQTT (default 1884), with the same credentials/ACLs and explicit firewall scoping. Plaintext is not a certificate-verification bypass. Mosquitto 2.1 has new password/ACL plugin guidance; validate the exact pinned release syntax and plugin paths in the container rather than copying a 2.0 recipe.
+
+Backend can publish only test commands for allowed devices and read their telemetry/status/ACKs. Each device can read only its commands and publish only its own telemetry/status/ACKs. A health-check account can only access a separate health topic. Deny production topics, cross-device writes, and unnecessary $SYS access. MQTT authentication does not remove the need for topic/payload identity validation.
+
+Persist broker sessions/retained messages for restart tests; configure queue, packet and client limits. Restrict config/certificate/password-file ownership and modes to the actual image UID/GID. Mount secrets read-only; avoid exposing them in command arguments or Compose output. Bound container logs, document persistence checkpoint/crash-loss limits, and avoid promising crash durability from a graceful-restart test.
+
+Health check performs authenticated TLS subscribe/publish/receive with a unique ID and timeout on the health topic. It should detect broken listener/auth/ACL behavior, not merely a live process. Clients must still perform independent remote hostname verification.
+
+Firewall allows only the selected TLS port for intended clients where feasible. Public-CA certificate issuance/renewal or private test CA is chosen after DNS/network details are known. Renewal copies readable cert material safely, reloads/restarts the isolated broker as required, and verifies the served certificate. Do not weaken hostname checks for IP-only access; use a matching IP SAN or proper DNS.
+
+Runbook includes start, stop, logs, health, config validation, backup, update, and rollback. Back up configuration and a consistent broker data snapshot before update. Rollback restores the prior image and compatible data snapshot; newer persistence data may not be backward-compatible. No production broker changes are authorized.
+
+## 12. Verification and implementation sequence
+
+Sequence: config/contracts -> server Compose/setup package and operator deployment -> backend/simulator using the provided test-broker endpoint -> browser viewer -> reliability/compatibility scenarios -> remote operational review -> conditional physical-device integration. Detailed sprints are separate files under `docs/sprints/`, linked from [SPRINTS.md](SPRINTS.md). After each sprint, execute its automated and manual checks, review changes, fix failures and rerun affected verification. Report every acceptance criterion with evidence as PASS/FAIL/BLOCKED. Required checks that cannot run block full completion; never count a skip as a pass.
+
+Unit checks cover precedence/typed decoding, validation, correlation including wrong device/topic/properties, terminal-state ordering, timeout and late ACKs, deadline checks, duplicate/conflicting IDs, limits, redaction, cancellation, and shutdown. Use gofmt, go vet, focused tests and race detection in the new project only.
+
+Real-broker scenarios cover initial exchange, bad credentials, forbidden topics, invalid CA/hostname, broker restart, network interruption, session resume versus loss/expiry, subscription recovery, queued command expiry, missing/delayed ACK, Will, graceful shutdown, and bounded resource behavior. Deterministic repeated application messages test idempotency; actual QoS 1 retransmission needs controlled transport interruption and may not reproduce on every run. Record that distinction.
+
+Browser verification covers live traffic, escaped hostile strings, redaction/truncation, filters, reconnection/replay/gap, slow clients, and shutdown. Compare rendered events to logs and avoid double-counting observer traffic as sender-owned latency.
+
+Every report separates verified locally, verified against remote broker, verified on physical hardware, and not verified. Sprint 1 local configuration, payload validation, safe previews and offline CLI behavior are verified. Sprint 2 Compose/setup/certificate/configuration checks are verified locally. Server-container MQTT security and graceful persistence checks passed in user-provided output. External MQTT 5 authenticated publish/receive passed from this development computer over verified TLS and plaintext on 2026-10-06. Sprint 3 unit/race checks, real CLI log/signal checks, TLS/plaintext command/telemetry exchanges, native correlation, duplicate/conflict handling, timeout/late ACK/unknown outcomes and reconnect with retained/lost sessions passed. Physical hardware remains unverified.
+
+## 13. Dependencies, risks and open inputs
+
+| Input/risk | Effect | Response |
+| --- | --- | --- |
+| ESP32 MQTT initialization/library unknown | Blocks exact device-side MQTT 5 selection and TLS/topic verification, not Go implementation | Inspect the C client setup before the device sprint; use supplied Esp32MqttConfig as connectivity reference |
+| Missing device source/board | Blocks real-device client/version/contract decisions | Continue with simulator; obtain paths and board details before hardware sprint |
+| Missing remote OS/CPU/access/DNS | Blocks exact remote deployment/cert choices | Validate local artifacts first; collect details without requesting secrets in chat |
+| Local container tooling unavailable | Blocks real-broker tests | Report prerequisite; propose native isolated broker if appropriate |
+| Client-ID collision | Disconnects another lab client | Configured role/instance IDs; separate observer and command instances |
+| Device clock drift | TLS or deadline failures | Synchronize clock and report clock-dependent tests honestly |
+| In-memory state loss | Lost history/dedup/pending requests on restart | Explicit limits; fresh client session on process restart; harmless operations only |
+| Browser overload or hostile payload | Inspection instability or injection | Bounded fan-out and text-only sanitized rendering |
+| Dependency/image availability | Proposed pins may need revision | Recheck official metadata and report material changes before adopting them |
+
+User inputs needed before their dependent work: device service/firmware paths and available boards; remote OS/CPU/tooling/free port/access and certificate situation; whether usage beyond local counters and bounded recent history is required. None blocks writing these documents or reviewing the local-first sprints.

docs/RUNNING.md

@@ -0,0 +1,193 @@
+# Run the MQTT 5 backend and simulator
+
+Sprints 3–4 use the dedicated deployed broker. All commands below are standard Linux commands, run on your development computer. The existing production project is not involved.
+
+## Run with Docker Compose
+
+`Dockerfile` builds both Go programs into one image. The root `compose.yaml` starts the observer and browser viewer; its optional `simulator` profile starts the Go test device. This connects outbound to the already deployed remote broker. The broker itself remains a separate Compose deployment under `deploy/`.
+
+The Compose setup uses Linux host networking so the Go viewer can keep its enforced `127.0.0.1:8080` bind. It does not publish the viewer on an external interface. Run this on a Linux Docker Engine host. The container runs as your host UID/GID so it can read the private, read-only credential bind mounts without changing their permissions.
+
+Copy `.env.docker.example` to `.env` and set the paths for the CA and private JSON handoffs on this computer. `.env` is ignored by Git; it contains paths only, not credential contents. For the current setup, the example paths point to `$HOME/mqtt-test-credentials`. Set `LOCAL_UID` and `LOCAL_GID` to `id -u` and `id -g` if they differ from the example.
+
+Start the observer and wait for its `subscriptions_ready` log:
+
+```bash
+docker compose up --build -d observer
+docker compose logs -f observer
+```
+
+Then open <http://127.0.0.1:8080>. In another terminal, start the simulator profile:
+
+```bash
+docker compose --profile simulator up -d simulator
+```
+
+Follow both event logs with `docker compose logs -f observer simulator`. Stop the simulator and observer with `docker compose --profile simulator down`. For a harmless ping, run the one-shot command container:
+
+```bash
+docker compose run --rm --no-deps \
+  -e TEST_MQTT5_WEB__ENABLED=false \
+  -e TEST_MQTT5_TOPICS__OBSERVE_DEVICES= \
+  -e TEST_MQTT5_TOPICS__OBSERVE_ALL=false \
+  -e TEST_MQTT5_RECEIPTS__ENABLED=false observer \
+  --broker tls://109.122.252.129:8884 --ca /run/test-mqtt/ca.crt \
+  --credentials /run/test-mqtt/backend.json \
+  --client-id rm5-docker-command-01 command ping
+```
+
+The command container inherits the CA and backend credential mounts from the observer service. The `-e` override disables the viewer for this short-lived `command` mode.
+
+## Build
+
+```bash
+cd /home/ryan/Media/Code/Code/robot-market/test-mqtt-5
+go build -buildvcs=false -o bin/test-mqtt5 ./cmd/test-mqtt5
+go build -buildvcs=false -o bin/device-sim ./cmd/device-sim
+```
+
+Client credential handoffs are private JSON files copied from the server. `--credentials` reads only username/password; it never logs them. Select the broker URL and CA explicitly. Flags precede subcommands. Every running process must use a distinct client ID: sharing one disconnects the previous client. Simulator and command modes have separate defaults; command mode generates a new client ID unless configuration overrides it.
+
+## TLS: open three terminals
+
+Run the simulator in the first terminal:
+
+```bash
+./bin/device-sim \
+  --broker tls://109.122.252.129:8884 \
+  --ca "$HOME/mqtt-test-credentials/ca.crt" \
+  --credentials "$HOME/mqtt-test-credentials/sim-001.json" \
+  --client-id rm5-lab-simulator-01
+```
+
+Run the observer in the second:
+
+```bash
+./bin/test-mqtt5 \
+  --broker tls://109.122.252.129:8884 \
+  --ca "$HOME/mqtt-test-credentials/ca.crt" \
+  --credentials "$HOME/mqtt-test-credentials/test-backend.json" \
+  --client-id rm5-lab-observer-01 observe
+```
+
+Send harmless commands in the third:
+
+```bash
+./bin/test-mqtt5 \
+  --broker tls://109.122.252.129:8884 \
+  --ca "$HOME/mqtt-test-credentials/ca.crt" \
+  --credentials "$HOME/mqtt-test-credentials/test-backend.json" command ping
+
+./bin/test-mqtt5 \
+  --broker tls://109.122.252.129:8884 \
+  --ca "$HOME/mqtt-test-credentials/ca.crt" \
+  --credentials "$HOME/mqtt-test-credentials/test-backend.json" command get_version
+```
+
+You should see `connection_up`, `subscriptions_ready`, simulator online status and periodic telemetry. Follow a command's request ID through `publish_attempt`, `transport_ack`, `command_accepted`, and `command_completed`. A completed ping contains `pong:true`; version returns `test-sim-1.0.0`. Transport ACK confirms broker acceptance, not execution. Fast application ACKs can be logged before the transport publish call returns.
+
+The observer sees other clients' command/ACK traffic. It cannot measure their own-request latency: it labels ACKs without an active own request as `late_ack`, with outcome `no active own request; latency not measured`. Command mode measures its own latency and prints final counters.
+
+Press Ctrl+C to stop the observer or simulator. The simulator attempts retained offline status before disconnecting. Its configured Will also reports offline on an ungraceful connection loss, with `will_configured_at` rather than an invented disconnect timestamp. A Will is an availability indication, not proof of why a device stopped.
+
+## Browser viewer
+
+The viewer is optional, starts only with the long-running `observe` process, and listens on `127.0.0.1:8080`. Create an ignored local config file:
+
+```bash
+cat > viewer.local.toml <<'EOF'
+[web]
+enabled = true
+EOF
+```
+
+Add `--config viewer.local.toml` to the observer command in the TLS section, before `observe`, then open <http://127.0.0.1:8080>. Keep the observer running. Start the simulator in its own terminal to generate telemetry, and use a third terminal for `command ping` or `command get_version`. The page refreshes event counters and streams sanitized events live. Its device JSON-contract row means the payload matched the Go contract; it does not identify hardware or infer the device MQTT version. It offers direction, topic, device and request filters, pause/resume, and clear. The viewer has no publish endpoint; commands remain CLI-only.
+
+The timeline keeps at most 1,000 displayed events; the server snapshot is limited by configured event history and SSE replay to 500 events. A gap notice means the page must resynchronize after events were evicted or its bounded queue overflowed. Compatibility evidence distinguishes the Go observer's MQTT 5 session from device telemetry and native MQTT 5 request/correlation support; the latter is not inferred from a payload.
+
+## Plaintext testing
+
+Use the same commands with `--broker mqtt://109.122.252.129:1884 --allow-plaintext` instead of the TLS URL/CA flags. The opt-in is required in each process, and credentials/ACLs still apply. Plaintext carries credentials and payloads without encryption. TLS certificate verification is never disabled.
+
+For example:
+
+```bash
+./bin/device-sim --broker mqtt://109.122.252.129:1884 --allow-plaintext \
+  --credentials "$HOME/mqtt-test-credentials/sim-001.json"
+
+./bin/test-mqtt5 --broker mqtt://109.122.252.129:1884 --allow-plaintext \
+  --credentials "$HOME/mqtt-test-credentials/test-backend.json" command ping
+```
+
+Run only one simulator for sim-001 at a time; the dedicated device identity and topics represent one simulated device.
+
+## JSON logs and configuration
+
+Create an ignored `logging.local.toml` to inspect structured JSON events:
+
+```toml
+[logging]
+format = "json"
+level = "info"
+```
+
+Add `--config logging.local.toml` before the subcommand. Native correlation bytes appear base64-encoded in JSON. Payload previews contain only validated contract fields and are bounded; malformed inputs are omitted and free-form ACK error messages are redacted. MQTT user properties and arbitrary response topics are not logged. Log levels warn/error suppress normal info events; use info for input/output inspection.
+
+Defaults < TOML < environment < explicitly supplied flags. The private handoff supplied with `--credentials` acts as an explicit username/password override. TOML/client-ID settings can override role defaults; use `--client-id` to prevent collisions when sharing a full config file. `check-config` remains offline even with a credential handoff.
+
+Runtime requires `mqtt.clean_start=true` for the initial process connection. Default session expiry is 300 seconds; reconnects inside that process use Clean Start=false. When the broker session is missing, subscriptions are rebuilt and SUBACK reason codes checked before readiness. Reconnect backoff uses bounded jitter (default 1–30 seconds). Initial readiness, publish and shutdown operations are bounded.
+
+Defaults are 256 inbound messages, 1,000 retained inspection events, 128 pending commands, 1,024 simulator dedup entries retained for 10 minutes, 16 KiB payloads and 4 KiB previews. The simulator processes commands in one worker. Full dedup capacity rejects new requests rather than evicting an in-progress entry. Incoming overflow counts `event_gap`; a MQTT receipt ACK does not prove simulator admission. Dedup retention/cache and client protocol state are memory-only; restarting a client does not promise durable deduplication or recovery.
+
+Commands are non-retained QoS 1, have a default 15-second deadline plus MQTT Message Expiry, and use exact native Response Topic/Correlation Data. Backend native correlation is the request ID's bytes; the simulator supports any nonempty correlation value up to 64 bytes and echoes it unchanged. It never executes beyond the application deadline. Completed duplicates replay a cached ACK; in-progress duplicates replay acceptance. Conflicting command/correlation reuse is rejected. Timeouts and connection loss report execution unknown, without an application outbox or automatic command retry. MQTT QoS 1 itself may retransmit an in-flight packet during reconnect, which is why deduplication is required.
+
+## Verification
+
+Offline checks:
+
+```bash
+gofmt -l cmd internal
+go vet ./...
+go test ./...
+go test -race ./...
+go mod verify
+```
+
+Real-broker tests are explicitly opt-in and use only the dedicated test accounts/topics. They run simulators, publish non-retained test commands/telemetry, update sim-001 retained status, and briefly drop their own local test connection to verify reconnection. Do not run them while another sim-001 simulator is active. They do not restart or reconfigure the broker.
+
+```bash
+MQTT5_INTEGRATION_CREDENTIAL_DIR="$HOME/mqtt-test-credentials" \
+  go test -race -v -count=1 -timeout=90s ./internal/integration
+```
+
+Without that variable, integration tests skip network access; those skips are not evidence of remote success. The recorded Sprint 3 remote run used the copied credential files against 109.122.252.129 and passed on both transports. Physical ESP32 interoperability and the larger reliability matrix remain later sprints. Browser viewer checks are recorded in [Sprint 4](sprints/04-browser-viewer.md).
+
+## Postman and physical-device functional testing
+
+The Docker observer subscribes to `robot-market/test/v1/#`. It displays every received message under that namespace as `message_received`, including arbitrary topics, plain text, any JSON shape, empty messages and literal timestamp placeholders. There is no observer payload schema, timestamp or device identity validation. `topics.observe_all=true` enables this mode. Outside functional observation, command/simulator contracts remain available for their structured round-trip tests.
+
+The test broker has password authentication but no topic ACL. Any existing test account, including `esp32-lab-001`, may publish and subscribe to `robot-market/test/v1/sim-001/telemetry` or another test topic. Plaintext uses `mqtt://109.122.252.129:1884`; TLS uses `mqtts://109.122.252.129:8884` in Postman. Select a distinct client ID, MQTT 5.0, QoS 1 and click Connect before Send.
+
+Send `hello from postman` to `robot-market/test/v1/sim-001/telemetry`. The literal payload must appear in the browser at http://127.0.0.1:8080. If it does not, inspect Postman's connection result and publish acknowledgement; the server must receive the publication before the observer can show it. Refresh the browser and clear filters. The simulator can stay stopped.
+
+Traffic display is separate from protocol-feature evidence: raw receipt demonstrates delivery, not a native MQTT 5 correlated command completion. Resource limits remain: 16 KiB message payload, 4 KiB display preview and bounded event queues/history. Binary payloads display as base64.
+
+## Device receipt ACKs
+
+Docker enables `TEST_MQTT5_RECEIPTS__ENABLED=true`. Subscribe the device to `<namespace>/<topic-device>/acks` before sending telemetry/live status or a custom message. The backend sends `type=receipt_ack`, `ok=true`, source topic, timestamp, byte length and SHA-256, echoing optional JSON message_id/request_id. No JSON or identity validation is required. Native MQTT 5 Response Topic and Correlation Data are supported. Commands, ACK/receipt topics, receipt_ack payloads and retained replays are excluded to prevent feedback loops.
+
+Replies have QoS 1, retain=false. The bounded reply queue reports receipt_ack_sent, receipt_ack_failed or receipt_ack_dropped in logs/UI. A duplicate live delivery can receive another receipt; no exactly-once, persistence or reconnect outbox is promised. Subscribe first: a non-retained reply sent before a subscription is established may be missed.
+
+Run the independent-client verification with the simulator stopped:
+
+```bash
+python3 scripts/check_receipts.py --credentials esp32-test-kit/connection.txt --ca "$HOME/mqtt-test-credentials/ca.crt"
+```
+
+This checks plaintext/TLS, MQTT 3.1.1/5, text/arbitrary JSON/empty payloads, no ACK loop, and native MQTT 5 response/correlation echo using unique test topic paths.
+
+### Reliability evidence in the viewer
+
+Run the opt-in suite with `MQTT5_INTEGRATION_CREDENTIAL_DIR` pointing at the private client handoff directory (backend/device JSON and CA). Each named remote test writes a structured `scenario_result` to verbose test output and publishes it to `robot-market/test/v1/scenarios/results`. The raw observer exposes these as reported compatibility rows with environment, endpoint, protocol, observation and UTC time. These are test-client reports, not independently verified ESP32 claims. Results are bounded by viewer history and disappear on observer restart. Missing rows remain untested; test publication failure fails the test.
+
+The raw viewer shows observation time and payload previews; it does not interpret an old retained `online` payload as current device health. A Will's `will_configured_at` is when the Will was created, not the instant the connection broke. See [VERIFICATION.md](VERIFICATION.md) for outcomes and limits.

docs/SPRINTS.md

@@ -0,0 +1,42 @@
+# Ordered sprints and implementation prompts
+
+Status: **Sprints 1–3 complete with local/remote MQTT 5 checks; Sprint 4 complete with a live browser check; Sprint 5 implemented with restart verification blocked; Sprints 6–7 awaiting authorization**.
+
+Read [PLAN.md](PLAN.md) for the design and [../README.md](../README.md) for boundaries. Sprints 1–5 are implemented; Sprint 5 has an unresolved verification gate. Each remaining prompt is a future handoff and requires authorization for that sprint.
+
+## Execution rules
+
+- Work only in `/home/ryan/Media/Code/Code/robot-market/test-mqtt-5`. Existing projects are read-only references.
+- Read applicable AGENTS.md files and `/home/ryan/.codex/RTK.md`; prefix shell commands with `rtk`.
+- Implement only the authorized sprint. Report acceptance criteria, actual checks/results, limitations, and remaining work, then wait unless continuing across sprints was explicitly authorized.
+- Do not modify production brokers, credentials, listeners, device repositories, or data. No dangerous device commands.
+- Record verification environment accurately. A skipped check is not a pass. Local simulator evidence is not hardware evidence.
+- Keep documentation practical: update the existing plan/README and targeted contract/runbook/checklist files; do not add repetitive status reports.
+- Proposed file paths are guidance. Keep packages small; do not create abstractions merely to match a directory list.
+- Recheck dependency pins before first installation. Explain material deviations. Do not download SDKs or install tools unnecessarily.
+
+## Sprint files
+
+Each sprint is stored in its own file with tasks, deliverables, dependencies, acceptance criteria, automated checks, manual verification, risks, a copy-ready prompt, and a mandatory end-of-sprint test/check gate.
+
+| Sprint | Document | Status |
+| --- | --- | --- |
+| 1 | [Go foundation, configuration, contracts and event model](sprints/01-foundation.md) | Complete; local checks PASS |
+| 2 | [Server broker deployment package](sprints/02-local-broker.md) | Complete; artifacts, server runtime and external connectivity verified |
+| 3 | [MQTT 5 backend, simulator and visible message flow](sprints/03-backend-simulator.md) | Complete; unit/race, real CLI and remote TLS/plaintext checks PASS |
+| 4 | [Small Go-served browser viewer](sprints/04-browser-viewer.md) | Complete; local/race checks and live browser verification PASS; ESP32 hardware remains untested |
+| 5 | [Reliability scenarios and compatibility evidence](sprints/05-reliability.md) | Implemented; ACK/reliability checks PASS, restart verification BLOCKED |
+| 6 | [Remote test-broker deployment and connectivity](sprints/06-remote-broker.md) | Not started |
+| 7 | [Conditional real-device compatibility](sprints/07-device-compatibility.md) | Not started |
+
+## Mandatory completion policy
+
+After every sprint, execute its tests and manual checks, review final changes, fix failures, rerun affected checks, and report acceptance criteria with actual evidence. Missing required verification is BLOCKED, not passed. Do not mark an unverified sprint complete or bypass this gate when authorized to continue across sprints.
+
+## ESP32 connection reference
+
+The supplied `Esp32MqttConfig` configures connectivity, not payloads. Go backend/simulator work can proceed without a C serializer. Device integration must check the C library/version, TLS/protocol selection and whether its single publish_topic setting can support the proposed telemetry/status/ACK topics. See [PLAN.md section 6.1](PLAN.md#61-esp32-connection-configuration-supplied-by-user).
+
+## Approval checkpoint
+
+Current order: Sprint 2 server package, user deployment and broker details, Sprint 3 local backend/simulator against that broker, Sprint 4 viewer, Sprint 5 reliability, Sprint 6 operational review and conditional Sprint 7 hardware. The user may authorize a subset. Documentation edits do not authorize application implementation, deployment, or hardware changes.

docs/VERIFICATION.md

@@ -0,0 +1,60 @@
+# Sprint 5 verification
+
+Checked 2026-10-07. Scope: isolated `test-mqtt-5` only. Go race tests on local Linux; Docker Go observer at http://127.0.0.1:8080; remote Mosquitto test broker 109.122.252.129, TLS 8884 and plaintext 1884. Independent Python socket clients use the ESP32 test account; Go tests use private backend/simulated-device handoffs. These checks do not prove physical ESP32 behavior.
+
+Status: implemented with full sprint completion **BLOCKED** by broker restart verification. Receipt ACK functionality is implemented and verified live.
+
+## Repeatable checks
+
+Run from the test project. `rtk` is a local command convention; use ordinary Linux commands on the server.
+
+```sh
+rtk env GOCACHE=/tmp/mqtt5-go-cache go test -race ./...
+rtk env GOCACHE=/tmp/mqtt5-go-cache go vet ./...
+rtk env GOCACHE=/tmp/mqtt5-go-cache MQTT5_INTEGRATION_CREDENTIAL_DIR=/home/ryan/mqtt-test-credentials go test -race -v ./internal/integration
+rtk docker compose up -d --build observer
+rtk python3 scripts/check_receipts.py --credentials esp32-test-kit/connection.txt --ca esp32-test-kit/ca.crt
+rtk docker compose ps -a
+```
+
+The normal unit suite skips remote tests without the opt-in credential directory; that skip is not remote evidence. The separately executed opt-in suite passed all remote tests. Tests create temporary simulated clients and clean them up; the Docker simulator stays stopped.
+
+## Results and limits
+
+| Feature | Outcome | Evidence and limit |
+| --- | --- | --- |
+| Unit/race regression and vet | PASS | All packages pass; includes ACK routing, arbitrary/empty payloads, binary correlation copying, duplicate receipt delivery, bounded queue overflow, publish failure, worker shutdown, correlation ordering and viewer tests. |
+| Device receipt ACKs | PASS | Independent Python subscribers/publishers: MQTT 3.1.1 and 5 × plaintext and TLS; text, arbitrary JSON, empty payloads. Byte count and SHA-256 match original bytes; optional message ID echoed. Incoming message and outgoing receipt confirmed through viewer snapshot. |
+| Native receipt routing | PASS | Actual MQTT 5 Response Topic and binary Correlation Data echoed exactly. |
+| Loop prevention | PASS | ACK/command/receipt topics and receipt payloads suppressed; retained replay suppressed. Unit tests and bounded live no-extra-reply check. |
+| Session resume/loss/expiry | PASS | Real TLS connection interrupted through loopback relay; session-present flag checked; subscriptions restored and fresh commands complete. Pending execution remains unknown after connection loss. |
+| Broker-queued expiry | PASS | Persistent subscriber offline; Message Expiry=1 second with still-valid 15-second application deadline; expired packet absent, fresh control delivered after session resume. |
+| Device application deadlines | PASS | Existing simulator unit tests prevent expired command execution; remote missing/late/wrong-correlation checks produce timeout/late/unknown rather than success. |
+| Harmless commands and duplicates | PASS | Native correlated ping/version over TLS/plaintext; cached application duplicate executes once; conflicting ID rejected. This applies to in-memory simulator command execution, not ESP32 persistence. |
+| Wire QoS 1 retransmission | PASS | Independent client sends QoS 1, deliberately does not read PUBACK, drops socket, resumes persistent session, resends same packet ID/payload with DUP=1. Subscriber gets two receipts; this is controlled wire retransmission, not proof of automatic hardware reconnect behavior. |
+| Abrupt Will | PASS | Real abrupt TLS loss emits nonretained offline Will; creation timestamp retained, no invented disconnect timestamp. |
+| Graceful offline/status and stale display | PASS / limited | Existing simulator graceful status behavior and unit tests; raw viewer labels observed time and does not derive current health from retained payloads. No general device liveness algorithm is claimed. |
+| Scenario results | PASS | Opt-in tests publish feature/environment/endpoint/version/status/observation/time; observer logs and viewer expose reported evidence separately from untested ESP32 compatibility. Results are bounded and transient. |
+| Authentication/TLS negatives | PASS | Independent external checks reject anonymous CONNECT, invalid password, untrusted CA and wrong hostname; successful TLS/plain sessions provide positive controls. |
+| Unrestricted test topic access | PASS | ESP32 account publishes generated device topics and subscribes to their ACKs. ACL enforcement intentionally removed at user's request. |
+| Graceful broker restart and persistence | BLOCKED | Server `docker compose exec -T broker true` fails with `OCI runtime exec failed ... procReady not received`. Server `python3 scripts/verify.py --insecure --restart` fails on its initial container health exchange, before restart. Existing broker accepts external MQTT traffic. Resolve server Docker runtime/container-exec issue, then rerun verifier; do not treat prior Sprint 2 evidence as a fresh Sprint 5 pass. |
+| Physical ESP32 receipt consumption | Not tested | Developer must subscribe before publish, receive receipt_ack and confirm in firmware logs. User-reported inbound hardware messages are not independent proof of return-path consumption. |
+
+Final browser refresh verified 21 receipt ACKs, zero reply failures/drops, and the reported Will scenario marked passed while ESP32 native compatibility remained not_tested. Final Docker status: observer running, simulator exited. The regenerated ESP32 ZIP includes the updated wildcard subscription and receipt instructions.
+
+## Acceptance gate
+
+- Bounded isolated scenarios and cleanup: PASS for executed scenarios.
+- Resume/loss subscription recovery: PASS.
+- Expired queued commands not delivered; simulator repeated valid ID executes once: PASS, bounded in-memory behavior only.
+- Evidence/environment attached and no simulator results presented as hardware: PASS.
+- Logs/viewer expose receipt failures, queue drops, timeout/unknown/late events and evidence limits: PASS.
+- Required graceful broker restart/persistence check: BLOCKED. Sprint 5 remains unclosed until verified.
+
+## Manual device check
+
+1. Connect using the ESP32 handoff. Subscribe to `robot-market/test/v1/esp32-lab-001/acks` (or the handoff wildcard) and wait for subscription success.
+2. Publish any nonretained payload to `robot-market/test/v1/esp32-lab-001/telemetry`.
+3. Confirm inbound payload, outbound `receipt_ack_sent`, and ACK counters in the browser. Confirm the device receives JSON with `type=receipt_ack`, `ok=true`, and the original source topic.
+4. A receipt means the backend observed bytes. Broker PUBACK only means broker transport acceptance; neither proves device command execution or durable storage. Publishing on /commands or /acks does not create a receipt ACK.
+5. Repeat with MQTT 3.1.1 and MQTT 5 if the firmware library supports protocol selection; record the actual library/build and CONNACK behavior. For MQTT 5, test native response/correlation properties explicitly.

docs/sprints/01-foundation.md

@@ -0,0 +1,104 @@
+# Sprint 1 — Go foundation, configuration, contracts and event model
+
+Status: complete; verified locally on 2026-10-06. Read [the plan](../PLAN.md), [sprint index and shared rules](../SPRINTS.md), and [README](../../README.md).
+
+
+### Goal and scope
+
+Produce a runnable configuration-checking CLI and validated message/event contracts without connecting to a broker.
+
+### Deliverables and likely files
+
+- `go.mod`, `go.sum`, `.gitignore`.
+- `cmd/test-mqtt5/main.go` with configuration validation and help.
+- `internal/config/`, `internal/contracts/`, `internal/events/` and focused tests.
+- `config.example.toml`, README startup/configuration instructions, `docs/CONTRACTS.md`.
+
+### Tasks
+
+1. Verify the proposed Go/koanf/parser/provider pins against official metadata and establish the module. Keep MQTT dependency installation until needed if that avoids unused modules.
+2. Implement defaults < TOML < environment < explicitly visited flags. Map `TEST_MQTT5_` with double-underscore nesting; support `--config`, `--device-id`, `--log-level`.
+3. Decode typed durations/numbers/bools, reject unknown settings, and validate TLS URL, credentials source, namespace/IDs, QoS, deadlines, certificate pair, limits, simulator and optional web settings.
+4. Define telemetry/status/command/ACK structs and validation; allow only ping/get_version. Define explicit ACK stages and application deadlines.
+5. Define sanitized event fields, bounded previews, secret redaction, and direction/outcome vocabulary. Do not echo raw malformed or binary payloads.
+6. Add signal-aware command structure, contextual errors, and text/JSON slog output. Document examples without secrets and ignore local config/secret/data artifacts.
+
+### Dependencies and required inputs
+
+Sprint authorization; an available compatible Go toolchain and dependency access. No server/hardware input is required.
+
+### Acceptance criteria
+
+- CLI validates example configuration and exits successfully without MQTT or HTTP connections.
+- Explicit CLI overrides win; unspecified flags do not override TOML/environment. Invalid configuration fails with key-specific errors and no secrets.
+- Payload examples validate; wrong identity, stage, command, timestamp, size, or missing ID fails.
+- Safe event previews preserve useful ping/version input/output while redacting sensitive fields.
+- No files in existing projects are changed.
+
+### Automated checks
+
+gofmt, go vet, focused config/contracts/events tests; precedence table including empty values and invalid durations; redaction and truncation tests. Run race detection where concurrent code exists.
+
+### Manual verification
+
+Run help and configuration validation with defaults/example/local overrides. Inspect text and JSON error output for clarity and secret absence.
+
+### Risks and unresolved decisions
+
+Exact toolchain availability and dependency resolution; generated module metadata may reveal incompatible pins. Report and resolve before claiming compatibility.
+
+### Mandatory end-of-sprint testing and completion gate
+
+After implementing this sprint, run its automated checks and manual verification above. Inspect the final changes for correctness, scope, secret exposure, and compatibility with completed sprints. Fix failures and rerun affected checks before declaring completion.
+
+Record the exact commands/checks, actual outcomes, verification environment, and each acceptance criterion as PASS, FAIL, or BLOCKED. Include applicable regression checks for previously completed behavior. A skipped/unavailable required check is BLOCKED, never PASS; do not claim full sprint completion while required checks fail or remain blocked. Do not run unrelated or production-project tests.
+
+Report the results and remaining limitations to the user. Wait for authorization before the next sprint unless continued execution was explicitly authorized. Even with permission to continue, do not bypass this verification gate.
+
+### Copy-ready implementation prompt
+
+```text
+Implement Sprint 1 only in /home/ryan/Media/Code/Code/robot-market/test-mqtt-5 after the user authorizes implementation. Read applicable AGENTS.md, /home/ryan/.codex/RTK.md, README.md, docs/PLAN.md and docs/sprints/01-foundation.md. Prefix shell commands with rtk. Existing sibling projects are read-only; do not deploy or contact a production broker.
+
+Build a small Go CLI that validates configuration without networking. Verify proposed pins (Go 1.27.1; koanf/v2 v2.3.7; TOML v2 parser v2.2.2; file provider v1.2.1; env provider v1.1.0) and report required deviations. Implement typed configuration with defaults < TOML < TEST_MQTT5_ environment < explicitly supplied flags; use __ for nesting and --config. Validate durations, TLS, identities, QoS, session/deadline settings and bounded resource limits. Add versioned telemetry/status/ping/get_version/ACK contracts and validation. Define shared sanitized input/output events for logs and a future browser viewer, including redaction and bounded previews. Add example TOML, secret/data ignore rules and focused contract/configuration docs.
+
+Use idiomatic Go, slog, contextual errors and signal cancellation. Run formatting, vet and meaningful config/contract/redaction tests. Demonstrate successful config validation and actionable invalid-config errors without leaking secrets. Do not implement MQTT, the viewer, remote deployment or device integration. Report actual checks, limitations and acceptance results, then wait for the next sprint authorization.
+
+ESP32 reference: read docs/PLAN.md section 6.1 for the supplied Esp32MqttConfig. It is connection configuration, not a payload schema or a blocker for Go implementation. Preserve the proposed Go test contracts. Device-specific MQTT 5 selection depends on the actual C library/build/init; inspect TLS/CA handling, role-specific credentials and single publish_topic limitations during device integration. Do not modify existing C projects. Separate MQTT 3.1.1 payload interoperability from native MQTT 5 feature evidence.
+
+Mandatory completion gate: after implementing this sprint, run its listed automated and manual checks plus relevant regression checks, inspect the final changes for correctness/scope/secrets/compatibility, fix failures and rerun affected checks. Report exact checks, real outcomes, environment, and every acceptance criterion as PASS/FAIL/BLOCKED. Missing required checks are BLOCKED, never a pass. Do not mark the sprint complete with unresolved failures/blockers or bypass verification when continuing to another sprint.
+```
+
+## Completion evidence
+
+Environment: local Linux/amd64, Go 1.27.1. All authored files are confined to test-mqtt-5. No existing server/device project was built, tested or modified. No remote broker or physical device was contacted.
+
+Pinned Go configuration modules resolved without version deviations; go.mod/go.sum record the resolved versions in this project (no Git commit was created). Paho remains deferred to the MQTT sprint.
+
+| Acceptance criterion | Result | Evidence |
+| --- | --- | --- |
+| Example configuration validates without MQTT/HTTP connections | PASS | Built CLI and go run check-config both succeed; output says network_checked=false |
+| Explicit flag precedence; unspecified flags preserve file/env | PASS | Configuration precedence tests and CLI override smoke check |
+| Invalid config produces key-specific errors without secrets | PASS | Validation table, malformed TOML, unknown keys, invalid identity and secret sentinel smoke checks |
+| Valid/invalid payload contracts behave as specified | PASS | Telemetry/status/command/ACK tests; duplicate keys, nulls, deadline boundaries, identity/stage/version/size checks |
+| Safe previews preserve useful fields and redact sensitive input | PASS | Preview tests for ping, free-form error redaction, malformed/binary/unknown fields and bounds |
+| Existing projects remain unchanged by this work | PASS | All edit commands target the new project; caches and test binary use /tmp |
+
+Actual checks:
+
+- `rtk proxy gofmt -l cmd internal`: PASS, no unformatted files.
+- `rtk proxy env GOMODCACHE=/tmp/test-mqtt5-mod GOCACHE=/tmp/test-mqtt5-build go vet ./...`: PASS.
+- Same cache settings with `go test ./...`: PASS, all four packages.
+- Same cache settings with `go test -race ./...`: PASS, all four packages.
+- Same cache settings with `go mod verify`: PASS.
+- Same cache settings with `go build -buildvcs=false -o /tmp/test-mqtt5-cli ./cmd/test-mqtt5`: PASS.
+- Same cache settings with `go run ./cmd/test-mqtt5 --config config.example.toml check-config`: PASS.
+- Built CLI help/default/example/explicit flag checks: PASS.
+- Built CLI JSON format with secret-bearing environment, invalid identity and malformed secret-bearing TOML: PASS; expected exit codes, values absent from output.
+- Canceled-context CLI test: PASS.
+
+The initial standalone build could not obtain VCS metadata from the surrounding checkout; disabling VCS stamping fixed the build without changing Git state. Initial compilation caught a koanf API mismatch; it was corrected before passing all checks.
+
+Limits: file validation checks existence/readability/size, not certificate contents or credential correctness. No network/session/ACK correlation/idempotent execution/hardware verification is claimed. Future event producers must sanitize and bound transport property metadata. Remote and physical-device checks: NOT VERIFIED.
+
+Next: Sprint 2 isolated local TLS broker, after authorization.

docs/sprints/02-local-broker.md

@@ -0,0 +1,96 @@
+# Sprint 2 — Server broker deployment package
+
+Status: COMPLETE. Artifacts, server-container runtime and external TLS/plaintext connection checks passed on 2026-10-06. Read [the plan](../PLAN.md), [sprint index and shared rules](../SPRINTS.md), and [README](../../README.md).
+
+
+### Goal and scope
+
+Prepare an independent server-deployable Mosquitto Compose package for the user to deploy. Include TLS and an optional authenticated plaintext listener per the user's updated request. Validate artifacts locally; defer runtime checks to the deployed server.
+
+### Deliverables and likely files
+
+- `deploy/compose.yaml`, broker configuration/ACL templates and health-check helper.
+- Local-only certificate/credential provisioning helper; generated secrets ignored.
+- `docs/DEPLOYMENT.md` server setup and verification instructions.
+
+### Tasks
+
+1. Verify Mosquitto 2.1.2-alpine image/platform availability and pin its digest; validate version-specific password/ACL plugin configuration.
+2. Publish TLS in the base server Compose file and plaintext only with an explicit override; both remain authenticated with identical ACLs. Enable persistence, bounded queues/packets/logs, and correct container ownership/modes.
+3. Generate a dedicated local test CA and server certificate with matching server hostname/IP and localhost SANs. Keep keys and generated credentials outside tracked files.
+4. Provision separate backend, device and health users. Restrict each device to its own topics; deny production/cross-device access.
+5. Add authenticated TLS subscribe/publish/receive health checking with a deadline and dedicated topic. Keep secrets out of arguments/output.
+6. Document start/stop/inspect, certificate trust, persistence cleanup, backup, update and rollback. Destructive reset remains explicit and test-instance-only.
+
+### Dependencies and required inputs
+
+Sprint 1; available Docker/Compose and registry access. If unavailable, report the blocker and propose an isolated native alternative. Remote server access is not needed.
+
+### Acceptance criteria
+
+- Deployed server broker starts and passes an actual authenticated MQTT health exchange.
+- Anonymous access, bad credentials, bad CA/hostname and forbidden topics fail as expected.
+- Per-device ACLs prevent cross-device publishing/subscription. Commands cannot reach production namespaces.
+- Graceful restart preserves selected retained/session state; evidence identifies graceful restart rather than crash durability.
+- Configuration and examples contain no real credentials; plaintext host access is enabled only through the explicit override and remains authenticated.
+
+### Automated checks
+
+Compose validation, broker config/startup validation using the pinned image, health exchange, positive/negative TLS/auth/ACL smoke checks. Do not print resolved secret-bearing Compose configuration.
+
+### Manual verification
+
+Inspect port binding, mounts, ownership, bounded logs and restart behavior. Verify server hostname trust and localhost health checking with the generated CA.
+
+### Risks and unresolved decisions
+
+Host container permissions, plugin paths in the exact image, architecture support, port conflicts. Remote OS/DNS/certificate policy remains deferred.
+
+### Mandatory end-of-sprint testing and completion gate
+
+After implementing this sprint, run its automated checks and manual verification above. Inspect the final changes for correctness, scope, secret exposure, and compatibility with completed sprints. Fix failures and rerun affected checks before declaring completion.
+
+Record the exact commands/checks, actual outcomes, verification environment, and each acceptance criterion as PASS, FAIL, or BLOCKED. Include applicable regression checks for previously completed behavior. A skipped/unavailable required check is BLOCKED, never PASS; do not claim full sprint completion while required checks fail or remain blocked. Do not run unrelated or production-project tests.
+
+Report the results and remaining limitations to the user. Wait for authorization before the next sprint unless continued execution was explicitly authorized. Even with permission to continue, do not bypass this verification gate.
+
+### Copy-ready implementation prompt
+
+```text
+Implement Sprint 2 only in /home/ryan/Media/Code/Code/robot-market/test-mqtt-5 after authorization. Read applicable AGENTS.md, /home/ryan/.codex/RTK.md, README.md, docs/PLAN.md and docs/sprints/02-local-broker.md. Prefix shell commands with rtk. Existing projects and production brokers are read-only. Do not deploy remotely.
+
+Build/maintain the separate server Mosquitto deployment package, using verified 2.1.2-alpine and its platform-compatible immutable digest. The user will deploy it and provide endpoint/credential-file details. Check exact 2.1 password/ACL plugin syntax and image paths. Publish TLS in the base Compose file and optional authenticated plaintext in a separate override; do not weaken TLS certificate verification. Disable anonymous access. Add dedicated local test certificate generation, ignored secrets, separate backend/device/health identities, least-privilege per-device ACLs, persistence, bounded broker queues/logs and correct file ownership. Implement an authenticated TLS MQTT subscribe/publish/receive health check with safe secret handling. Document server start/stop/inspect, backup/update/rollback and remote prerequisites.
+
+Verify real health exchange, invalid credentials/CA/hostname, forbidden/cross-device topics and graceful broker restart persistence. If container tooling is unavailable, report it and propose a native isolated alternative rather than claiming tests passed. Do not implement the Go MQTT application or viewer. Report acceptance results and remaining inputs, then wait.
+
+ESP32 reference: read docs/PLAN.md section 6.1 for the supplied Esp32MqttConfig. It is connection configuration, not a payload schema or a blocker for Go implementation. Preserve the proposed Go test contracts. Device-specific MQTT 5 selection depends on the actual C library/build/init; inspect TLS/CA handling, role-specific credentials and single publish_topic limitations during device integration. Do not modify existing C projects. Separate MQTT 3.1.1 payload interoperability from native MQTT 5 feature evidence.
+
+Mandatory completion gate: after implementing this sprint, run its listed automated and manual checks plus relevant regression checks, inspect the final changes for correctness/scope/secrets/compatibility, fix failures and rerun affected checks. Report exact checks, real outcomes, environment, and every acceptance criterion as PASS/FAIL/BLOCKED. Missing required checks are BLOCKED, never a pass. Do not mark the sprint complete with unresolved failures/blockers or bypass verification when continuing to another sprint.
+```
+
+## Completion evidence and remaining server checks
+
+User steering: deploy the broker on the server; provide a Compose package for operator deployment; support plaintext alongside TLS for testing. No server changes or permanent local broker were performed.
+
+| Acceptance criterion | Status | Evidence / remaining action |
+| --- | --- | --- |
+| Deployable pinned broker config | PASS for artifacts | Verified official multi-platform digest; Mosquitto 2.1.2 --test-config exits 0 in a network-disabled disposable container |
+| Health MQTT exchange | PASS | User ran the verifier on the deployed server: authenticated TLS exchange passed |
+| Bad credentials / CA / hostname rejected | PASS | Deployed verifier rejected anonymous access, invalid credentials, untrusted CA and wrong hostname |
+| Per-device ACL isolation / production topic denial | PASS | Deployed verifier confirmed cross-device publish/read blocking and production topic denial |
+| Graceful retained/session persistence | PASS | Deployed verifier confirmed retained status and offline session recovery after graceful restart; crash durability not tested |
+| No tracked credentials; optional plaintext explicit | PASS | Generated secrets confined to /tmp for validation; ignored server paths; merged Compose ports/auth configuration checked |
+
+Checks actually run locally:
+
+- docker buildx imagetools inspect: 2.1.2-alpine digest verified; bare 2.1.2 tag was absent and corrected.
+- docker compose config --quiet for TLS-only and TLS+plaintext: PASS.
+- Compose JSON assertions: ports, non-root runtime, read-only mounts/filesystem, immutable image and log limits PASS; no credential environment values.
+- Python AST/helper --help and sh -n healthcheck: PASS.
+- prepare.py with throwaway /tmp output and owner 1000:1000: PASS, real OpenSSL generation and container-based password hashing.
+- Generated hashed passwords/secret modes/CA-key isolation: PASS without displaying values.
+- OpenSSL chain, configured hostname, localhost and 127.0.0.1 checks: PASS; wrong hostname rejected.
+- Repeat setup refuses overwrite; invalid hostname inputs rejected: PASS.
+- Exact pinned image --test-config with temporary mounts and --network none: PASS. No listening broker started.
+
+On 2026-10-06 the user ran the corrected verify.py --insecure --restart on 109.122.252.129. All server-container checks passed, including optional authenticated plaintext reception. The initial cross-device failure was a verifier exit-code bug; the corrected verifier requires MQTT PUBACK Not authorized (135). Server runtime acceptance is complete. External authenticated MQTT 5 device telemetry publication/backend reception passed from the development computer on TLS 8884 with CA/hostname verification and plaintext 1884, using unique non-retained probes. Physical hardware remains NOT VERIFIED. See DEPLOYMENT.md for operator commands.

docs/sprints/03-backend-simulator.md

@@ -0,0 +1,99 @@
+# Sprint 3 — MQTT 5 backend, simulator and visible message flow
+
+Status: COMPLETE on 2026-10-06; local unit/race, CLI and dedicated remote-broker checks passed. Read [the plan](../PLAN.md), [sprint index and shared rules](../SPRINTS.md), and [README](../../README.md).
+
+
+### Goal and scope
+
+Run the complete harmless local command/telemetry loop with structured input/output logs and counters.
+
+### Deliverables and likely files
+
+- `cmd/device-sim/main.go`; `cmd/test-mqtt5` observe/command modes.
+- `internal/mqtt/`, `internal/backend/`, `internal/simulator/`; event buffer and counters.
+- Focused unit/integration tests, README quick start and contract updates.
+
+### Tasks
+
+- Support the user-requested plaintext test connection through explicit mqtt.allow_plaintext=false-by-default and a mqtt:// URL only when enabled. Keep TLS verification intact, preserve test namespace/ACLs, and update configuration tests/example/docs.
+
+1. Integrate `github.com/eclipse/paho.golang v0.23.0`/autopaho with verified TLS, deadlines, bounded jittered backoff and configured role/instance IDs.
+2. Implement explicit initial Clean Start/session expiry and subscription readiness/restoration. Copy callback input into bounded workers; define visible overflow behavior.
+3. Simulator publishes telemetry, retained online status and an offline Will; handles ping/get_version and emits accepted/completed ACKs.
+4. Register pending correlation before publish; use exact Response Topic and native Correlation Data; validate device/topic/request ID/properties and transport reason codes.
+5. Enforce application deadline plus MQTT Message Expiry; implement bounded deduplication with in-progress entries, cached ACK replay and conflicting-ID rejection.
+6. Produce readable text/JSON events for inbound/outbound payload previews/properties and lifecycle outcomes. Separate attempted publication from transport ACK and device completion.
+7. Add own-request latency/counters, late ACK classification, bounded buffers and graceful shutdown. No offline command outbox or automatic ambiguous retries.
+
+### Dependencies and required inputs
+
+Sprint 1, Sprint 2 server package deployed and verified, broker hostname/ports, private test backend/device credential files and public CA. No physical device is required.
+
+### Acceptance criteria
+
+- Observer receives valid telemetry/status; command mode receives correlated ping/version results from the simulator.
+- Logs show safe payload preview, topic, direction, QoS, request ID and native properties; no credentials leak.
+- Wrong-topic/device/correlation ACKs cannot complete a pending request; late/out-of-order ACKs do not corrupt state.
+- An intentional repeated request ID does not repeat simulated execution; conflicting reuse is rejected.
+- Missing ACK and disconnect produce truthful timeout/unknown outcomes. SIGINT/SIGTERM exits within the configured bound.
+- Session settings, queue/dedup limits and restart limitations are documented.
+
+### Automated checks
+
+gofmt, go vet, unit tests and race detector. Real-broker integration for initial exchange, correlation, timeout, duplicate/conflicting request and shutdown. Scope full fault matrix to Sprint 5.
+
+### Manual verification
+
+Run observer and simulator in separate terminals, then run command mode with a distinct client ID. Follow one request through publish_attempt, transport_ack, accepted and completed. Compare safe previews to actual test payloads.
+
+### Risks and unresolved decisions
+
+Autopaho callback/ACK timing and concurrent state ownership; inspect pinned behavior instead of assuming backpressure guarantees. Transport success is never execution proof.
+
+### Mandatory end-of-sprint testing and completion gate
+
+After implementing this sprint, run its automated checks and manual verification above. Inspect the final changes for correctness, scope, secret exposure, and compatibility with completed sprints. Fix failures and rerun affected checks before declaring completion.
+
+Record the exact commands/checks, actual outcomes, verification environment, and each acceptance criterion as PASS, FAIL, or BLOCKED. Include applicable regression checks for previously completed behavior. A skipped/unavailable required check is BLOCKED, never PASS; do not claim full sprint completion while required checks fail or remain blocked. Do not run unrelated or production-project tests.
+
+Report the results and remaining limitations to the user. Wait for authorization before the next sprint unless continued execution was explicitly authorized. Even with permission to continue, do not bypass this verification gate.
+
+### Copy-ready implementation prompt
+
+```text
+Implement Sprint 3 only in /home/ryan/Media/Code/Code/robot-market/test-mqtt-5 after authorization. Read applicable AGENTS.md, /home/ryan/.codex/RTK.md, README.md, docs/PLAN.md and docs/sprints/03-backend-simulator.md. Use rtk for shell commands. Do not edit siblings, production infrastructure, or physical firmware.
+
+Using the existing configuration/contracts and isolated deployed test broker, implement a Go MQTT 5 observer/command CLI and device simulator with github.com/eclipse/paho.golang v0.23.0/autopaho, never paho.mqtt.golang. Support only ping/get_version plus telemetry/status. Use distinct predictable client IDs, TLS hostname verification, bounded workers/state, initial Clean Start=true, 300-second session expiry, subscription readiness/restoration, bounded jittered reconnect, retained online/offline status and explicit Will semantics. Commands are non-retained QoS 1 with a 15-second default deadline and MQTT Message Expiry. Correlate exact device/topic/request ID and native Response Topic/Correlation Data; inspect MQTT reason codes. Implement bounded request deduplication, cached ACK replay, conflict rejection, late ACKs, and graceful signal shutdown.
+
+Make inputs/outputs visible in text/JSON slog events with sanitized bounded payload previews and MQTT properties. Distinguish publication attempt, transport acknowledgement, application acceptance and completion; expose counters and own-request latency. Do not introduce a database, offline outbox, HTTP viewer or dangerous commands. Run meaningful unit/race and real-broker happy-path/correlation/timeout/dedup/shutdown tests. Update quick start and contracts, report verified environment and limitations, then wait.
+
+ESP32 reference: read docs/PLAN.md section 6.1 for the supplied Esp32MqttConfig. It is connection configuration, not a payload schema or a blocker for Go implementation. Preserve the proposed Go test contracts. Device-specific MQTT 5 selection depends on the actual C library/build/init; inspect TLS/CA handling, role-specific credentials and single publish_topic limitations during device integration. Do not modify existing C projects. Separate MQTT 3.1.1 payload interoperability from native MQTT 5 feature evidence.
+
+Mandatory completion gate: after implementing this sprint, run its listed automated and manual checks plus relevant regression checks, inspect the final changes for correctness/scope/secrets/compatibility, fix failures and rerun affected checks. Report exact checks, real outcomes, environment, and every acceptance criterion as PASS/FAIL/BLOCKED. Missing required checks are BLOCKED, never a pass. Do not mark the sprint complete with unresolved failures/blockers or bypass verification when continuing to another sprint.
+```
+
+## Completion evidence
+
+Implemented only in the isolated test project; existing projects and server configuration were not changed. Added pinned Paho v0.23.0/autopaho, shared bounded transport, observer and safe command CLI, simulator, event history/counters and [RUNNING.md](../RUNNING.md). Plaintext requires explicit opt-in; TLS verifies CA and hostname. No browser, database, offline application outbox or physical-device implementation was added.
+
+| Acceptance criterion | Status | Evidence |
+| --- | --- | --- |
+| Observer telemetry/status and correlated ping/version | PASS | Remote race tests on TLS 8884 and plaintext 1884; real CLI observer/simulator/command check |
+| Safe input/output logs, properties, lifecycle and counters | PASS | CLI JSON logs contain previews, native correlation and publish/transport/accepted/completed stages; copied credential passwords absent; bounded history tests |
+| Wrong topic/device/correlation, late/out-of-order ACK handling | PASS | Unit tests reject wrong identity/topic/correlation/retention/response/result/deadline; completion before acceptance remains terminal; remote wrong-correlation and late-ACK tests |
+| Repeated request ID and conflicting reuse | PASS | Unit and remote tests replay cached completion with one execution; changed command with same ID is rejected |
+| Timeout, disconnect unknown outcome and bounded shutdown | PASS | Remote deadline/late-ACK tests; abrupt relay connection loss returns unknown; CLI SIGTERM and simulator cancellation complete within configured 5-second bound |
+| Session/queue/dedup/restart settings documented | PASS | RUNNING.md records limits, overflow, in-memory state and no automatic application retry; real reconnect tests cover Session Present=true and false with subscription restoration |
+
+Checks run from the new project with Go 1.27.1 and temporary module/build caches:
+
+- `gofmt -l cmd internal`: no unformatted files.
+- `go vet ./...`: PASS.
+- `go test -race ./...`: PASS; default remote tests explicitly skip without the opt-in variable, separately verified below.
+- `go mod verify`: all modules verified.
+- `go build -buildvcs=false` for both CLI binaries: PASS.
+- `MQTT5_INTEGRATION_CREDENTIAL_DIR=/home/ryan/mqtt-test-credentials go test -race -v -count=1 -timeout=90s ./internal/integration`: PASS against 109.122.252.129. Covers TLS/plaintext happy paths, native properties, dedup/conflict, deadline/late ACK, local disconnect and actual TLS connection loss/reconnect with retained and lost broker sessions. The relay touches only its own loopback client connection; no broker restart/configuration change.
+- Real built CLI processes: observer/simulator/command ping; JSON previews/native properties/lifecycle; no credential output; successful bounded SIGTERM shutdown: PASS. Temporary check script/output lived under /tmp; the reproducible manual commands are in RUNNING.md.
+- Structural source review with graphify AST extraction: 20 Go files, 204 nodes and 660 extracted relationships; source and concurrency review completed without adding graph artifacts to the project.
+
+Verified environment: Go MQTT 5 backend and local simulator against the dedicated remote Mosquitto 2.1.2 broker, both TLS and plaintext. Larger fault/compatibility matrix, crash durability, durable client/dedup recovery, browser viewing and physical ESP32 interoperability remain unverified or scheduled for later sprints. All Sprint 3 acceptance criteria passed; Sprint 4 requires authorization.

docs/sprints/04-browser-viewer.md

@@ -0,0 +1,81 @@
+# Sprint 4 — Small Go-served browser viewer
+
+Status: COMPLETE for local implementation and verification; ESP32 hardware evidence remains outstanding. Read [the plan](../PLAN.md), [sprint index and shared rules](../SPRINTS.md), and [README](../../README.md).
+
+
+### Goal and scope
+
+Inspect the same safe message stream through a browser without adding another service or frontend toolchain.
+
+### Deliverables and likely files
+
+- `internal/web/` server, embedded HTML/CSS/vanilla JavaScript and tests.
+- `internal/events/` bounded snapshot/SSE fan-out if not already present.
+- Viewer configuration and README browser instructions.
+
+### Tasks
+
+1. Add opt-in viewer on `127.0.0.1:8080` in observation mode; default disabled.
+2. Provide read-only snapshot and SSE endpoints using the sanitized event model. Add event IDs, bounded replay, Last-Event-ID handling, gap/reset indicators, heartbeat and cleanup.
+3. Show connection/readiness, counters, filterable direction/topic/device/request timeline, bounded payload/property details and command-stage observations.
+4. Show compatibility rows with explicit not_tested/pass/fail/unsupported/blocked evidence. Do not infer native MQTT 5 support from a message's mere arrival.
+5. Render payloads as text, apply restrictive CSP/Host/Origin handling, use no CORS wildcard/CDN, and enforce viewer/queue limits.
+6. Ensure slow/disconnected viewers cannot block MQTT or grow memory; integrate bounded HTTP shutdown. No HTTP command-publishing endpoint.
+
+### Dependencies and required inputs
+
+Sprint 3 running locally; browser availability. No design assets, JavaScript package installation, server access or hardware required.
+
+### Acceptance criteria
+
+- Opening the loopback page shows live simulator input/output and state changes matching the logs.
+- Filters, pause/resume, reconnect, expired replay buffer and empty/disconnected states behave clearly.
+- Hostile payload strings render literally; secrets remain redacted; large previews show truncation.
+- Multiple/slow viewers obey bounds and cannot affect command completion.
+- Compatibility rows distinguish evidence from unknown status; observed ACK timing is not mislabeled as sender-owned latency.
+
+### Automated checks
+
+Go HTTP tests for snapshot/SSE/replay/gap, redaction, origin/host validation, cleanup and shutdown; race detector for fan-out. Use browser automation if available; no heavyweight frontend test stack solely for this page.
+
+### Manual verification
+
+Open the viewer while simulator traffic flows; run ping from CLI; inspect payload/properties and lifecycle. Disconnect/reopen the page, exercise filters and pause, send a harmless HTML-like test string, and verify literal rendering.
+
+### Completion evidence
+
+- `gofmt -w internal/web/server.go internal/web/server_test.go internal/events/store.go internal/events/store_test.go internal/cli/cli.go internal/config/config.go`: PASS.
+- `node --check internal/web/assets/app.js`: PASS after fixing the page script parse error discovered during browser review.
+- `go test ./internal/web ./internal/events`: PASS, including snapshot escaping, host/origin policy, SSE replay gaps, slow queue behavior and viewer cleanup/limit. Tests use handler concurrency because this sandbox denies test-process localhost listeners.
+- Regression checks: `go test ./...`, `go test -race ./...`, `go vet ./...`, `go mod verify`, and `gofmt -l cmd internal`: PASS.
+- Local browser verification against the deployed isolated broker: PASS. Observer connected with MQTT 5; simulator retained status and periodic telemetry appeared; the compatibility panel showed backend MQTT 5 passed, JSON contract passed, and native ESP32 round trip not tested. A harmless ping displayed command payload, response topic, correlation data, accepted/completed ACKs, and telemetry. The command CLI reported `pong:true`. Topic filter, pause/resume and clear filters worked.
+- One initial browser pass exposed a JavaScript parse error; it was fixed and the page was reloaded successfully. The simulator was stopped after verification.
+- Security boundary: listener binds only to validated `127.0.0.1`; snapshots/SSE are read-only, browser strings render through `textContent`, with same-host/origin checks, CSP and bounded event/viewer queues. There is no HTTP publish route.
+- Acceptance outcomes: live simulator input/state/counters PASS; filters/pause/resume and browser reload/reconnect PASS; bounded replay/gap and viewer limits PASS by automated tests; hostile string safety PASS by JSON escaping and text-only-sink regression test (an arbitrary HTML-like MQTT message was not sent because the contract rejects free-form fields and the viewer intentionally has no publish endpoint); command stages/observer latency labeling and compatibility evidence PASS.
+- Not claimed: ESP32 hardware/MQTT-version compatibility, native MQTT 5 property support on the ESP32, or long-running browser stress testing. Continue in Sprint 5/7 as planned.
+
+### Risks and unresolved decisions
+
+Long-lived SSE connections and event gaps; handle them explicitly. Public access/authentication and browser command submission remain outside scope.
+
+### Mandatory end-of-sprint testing and completion gate
+
+After implementing this sprint, run its automated checks and manual verification above. Inspect the final changes for correctness, scope, secret exposure, and compatibility with completed sprints. Fix failures and rerun affected checks before declaring completion.
+
+Record the exact commands/checks, actual outcomes, verification environment, and each acceptance criterion as PASS, FAIL, or BLOCKED. Include applicable regression checks for previously completed behavior. A skipped/unavailable required check is BLOCKED, never PASS; do not claim full sprint completion while required checks fail or remain blocked. Do not run unrelated or production-project tests.
+
+Report the results and remaining limitations to the user. Wait for authorization before the next sprint unless continued execution was explicitly authorized. Even with permission to continue, do not bypass this verification gate.
+
+### Copy-ready implementation prompt
+
+```text
+Implement Sprint 4 only in /home/ryan/Media/Code/Code/robot-market/test-mqtt-5 after authorization. Read applicable AGENTS.md, /home/ryan/.codex/RTK.md, README.md, docs/PLAN.md and docs/sprints/04-browser-viewer.md. Prefix shell commands with rtk. Existing projects are read-only; do not deploy remotely.
+
+Add a minimal optional browser viewer to the working Go backend using net/http, embed, HTML/CSS and vanilla JavaScript. No Node/npm, CDN, framework or separate service. Bind only 127.0.0.1:8080, default disabled. Display the same sanitized input/output events as logs, including topic/direction/device/request filters, bounded payload and MQTT property details, connections/readiness, counters and observed command stages. Provide read-only snapshot plus SSE with event IDs, bounded replay, explicit gaps, viewer limits and slow-client cleanup. Display compatibility statuses and evidence without assuming native support from ordinary traffic. Keep locally measured own-request latency distinct from observer timing.
+
+Use textContent for untrusted payloads, restrictive CSP and same-origin/Host checks. No HTTP publishing endpoint or public exposure. Verify real simulator traffic in the browser and meaningful Go HTTP/race tests for event replay, cleanup, redaction and shutdown. Update README, report actual browser verification and limitations, then wait.
+
+ESP32 reference: read docs/PLAN.md section 6.1 for the supplied Esp32MqttConfig. It is connection configuration, not a payload schema or a blocker for Go implementation. Preserve the proposed Go test contracts. Device-specific MQTT 5 selection depends on the actual C library/build/init; inspect TLS/CA handling, role-specific credentials and single publish_topic limitations during device integration. Do not modify existing C projects. Separate MQTT 3.1.1 payload interoperability from native MQTT 5 feature evidence.
+
+Mandatory completion gate: after implementing this sprint, run its listed automated and manual checks plus relevant regression checks, inspect the final changes for correctness/scope/secrets/compatibility, fix failures and rerun affected checks. Report exact checks, real outcomes, environment, and every acceptance criterion as PASS/FAIL/BLOCKED. Missing required checks are BLOCKED, never a pass. Do not mark the sprint complete with unresolved failures/blockers or bypass verification when continuing to another sprint.
+```

docs/sprints/05-reliability.md

@@ -0,0 +1,81 @@
+# Sprint 5 — Reliability scenarios and compatibility evidence
+
+Status: implemented, with live ACK and reliability checks PASS; full completion BLOCKED by the server container-exec failure preventing the broker restart verifier. See [verification evidence](../VERIFICATION.md). Read [the plan](../PLAN.md), [sprint index and shared rules](../SPRINTS.md), and [README](../../README.md).
+
+
+### Goal and scope
+
+Turn the working local environment into a repeatable test harness with explicit, evidence-backed feature results.
+
+### Deliverables and likely files
+
+- `tests/integration/` or equivalent opt-in Go integration suite; minimal scenario helpers.
+- Simulator controls for delayed/missing ACKs and controlled harmless duplicate tests.
+- `docs/VERIFICATION.md` scenario checklist and known limitations; event/result output consumable by logs and current-process viewer.
+
+### Tasks
+
+1. Implement explicit scenario result records with feature, environment, endpoint/version, status, observation and time; unknown remains not_tested.
+2. Exercise broker graceful restart, network interruption, session resume/loss/expiry and subscription restoration.
+3. Test queued command expiry separately from device-side expiry after delivery; synchronize clocks and use bounded test deadlines.
+4. Test duplicate requests, conflicting IDs, wrong/late/reordered/missing ACKs, invalid payloads and overloaded buffers.
+5. Verify Will after abrupt client stop versus graceful offline status, and stale-status handling.
+6. Test auth/TLS failures and unrestricted cross-device topic access (the user explicitly disabled ACLs for this test broker). Test and native Response Topic/Correlation Data echo. Attempt actual QoS 1 retransmission with controlled interruption; label unreproduced behavior honestly.
+7. Surface structured scenario verdicts in logs and viewer without adding persistent accounting. Document why repeated application publication is different from wire-level DUP retransmission.
+
+### Added scope: backend receipt ACKs (2026-10-07)
+
+Implement device publication → backend receipt → device application ACK, separate from broker PUBACK and device replies to commands. Raw observation continues to accept arbitrary payloads without schema, timestamp or device-ID checks.
+
+- Enable with `receipts.enabled=true` in raw observe mode; Docker enables it.
+- Subscribe on the device to `<namespace>/<topic-device>/acks` before publishing. Route by the incoming topic, not JSON or username. A valid in-namespace MQTT 5 Response Topic overrides the default, with exact Correlation Data echoed.
+- Reply at QoS 1, retain false: `type=receipt_ack`, `ok=true`, original `source_topic`, observed time, payload byte length and SHA-256; echo optional message_id/request_id without requiring them.
+- Never reply to commands, ACK/receipt topics, receipt_ack payloads or retained replays. Repeated live publications may each get a receipt ACK; receipt confirmation does not execute a device action.
+- Keep reply queue bounded and log receipt sent/failed/dropped. Broker acknowledgement of the reply does not prove the device consumed it. Do not claim persistence or exactly-once handling.
+- Test raw text, empty/invalid JSON, optional IDs, mismatched JSON identity, unchanged timestamp placeholder, native reply routing/correlation, duplicate delivery, reply failure, queue overload, cancellation and ACK-loop prevention. Run live independent subscriber/publisher checks over MQTT 3.1.1 and MQTT 5 with both plaintext/TLS, then inspect UI inbound/outbound events.
+
+### Dependencies and required inputs
+
+Sprints 1–4; local broker control and a safe way to interrupt only test connections/containers. Do not change host-wide firewall or unrelated networking.
+
+### Acceptance criteria
+
+- Scenarios are bounded, repeatable, isolated, and leave no unrelated processes/network changes.
+- Resume and session-loss paths both recover subscriptions and readiness correctly.
+- Expired commands are not executed; repeated valid IDs execute once within the documented in-memory retention window.
+- Every compatibility verdict contains evidence/environment; simulator results are not marked physical-device results.
+- Logs/browser explain failures, gaps, timeouts, late responses and limits without secrets.
+
+### Automated checks
+
+Unit/race suite and opt-in real-broker scenarios with explicit pass/fail/skip outcomes and cleanup. Avoid timing-only assertions where an observable state can synchronize the test.
+
+### Manual verification
+
+Follow the checklist with the viewer open; observe reconnect, expiry, Will and late ACK events. Review the resulting feature table and distinguish unsupported, blocked and failed.
+
+### Risks and unresolved decisions
+
+QoS 1 wire retransmission may be nondeterministic; report not reproduced rather than substituting an application duplicate. Broker graceful persistence is not crash-durability proof.
+
+### Mandatory end-of-sprint testing and completion gate
+
+After implementing this sprint, run its automated checks and manual verification above. Inspect the final changes for correctness, scope, secret exposure, and compatibility with completed sprints. Fix failures and rerun affected checks before declaring completion.
+
+Record the exact commands/checks, actual outcomes, verification environment, and each acceptance criterion as PASS, FAIL, or BLOCKED. Include applicable regression checks for previously completed behavior. A skipped/unavailable required check is BLOCKED, never PASS; do not claim full sprint completion while required checks fail or remain blocked. Do not run unrelated or production-project tests.
+
+Report the results and remaining limitations to the user. Wait for authorization before the next sprint unless continued execution was explicitly authorized. Even with permission to continue, do not bypass this verification gate.
+
+### Copy-ready implementation prompt
+
+```text
+Implement Sprint 5 only in /home/ryan/Media/Code/Code/robot-market/test-mqtt-5 after authorization. Read applicable AGENTS.md, /home/ryan/.codex/RTK.md, README.md, docs/PLAN.md and docs/sprints/05-reliability.md. Use rtk. Never alter production brokers, siblings, or host-wide networking.
+
+Build a focused opt-in real-broker reliability/compatibility harness around the existing Go backend, simulator and local viewer. Cover graceful broker restart, isolated network interruption, session resume versus loss/expiry, subscription restoration, broker-queued Message Expiry, device application deadlines, duplicates/conflicting request IDs, wrong/late/reordered/missing ACKs, Will versus graceful offline state, validation/overload and TLS/auth failures and unrestricted authenticated topic access. Prove native Response Topic/Correlation Data behavior with actual properties. Attempt controlled QoS 1 retransmission separately from deterministic repeated application requests and report unreproduced cases honestly.
+
+Emit explicit feature/environment/evidence/status results to logs and the existing viewer, leaving untested features not_tested. Keep all state and test operations bounded, provide cleanup, and avoid a database or monitoring stack. Run meaningful unit/race and isolated integration checks. Update docs/VERIFICATION.md and known limitations; distinguish local simulator, remote and hardware evidence. Report results and wait.
+
+ESP32 reference: read docs/PLAN.md section 6.1 for the supplied Esp32MqttConfig. It is connection configuration, not a payload schema or a blocker for Go implementation. Preserve the proposed Go test contracts. Device-specific MQTT 5 selection depends on the actual C library/build/init; inspect TLS/CA handling, role-specific credentials and single publish_topic limitations during device integration. Do not modify existing C projects. Separate MQTT 3.1.1 payload interoperability from native MQTT 5 feature evidence.
+
+Mandatory completion gate: after implementing this sprint, run its listed automated and manual checks plus relevant regression checks, inspect the final changes for correctness/scope/secrets/compatibility, fix failures and rerun affected checks. Report exact checks, real outcomes, environment, and every acceptance criterion as PASS/FAIL/BLOCKED. Missing required checks are BLOCKED, never a pass. Do not mark the sprint complete with unresolved failures/blockers or bypass verification when continuing to another sprint.
+```

docs/sprints/06-remote-broker.md

@@ -0,0 +1,68 @@
+# Sprint 6 — Remote test-broker deployment and connectivity
+
+Status: proposed; not implemented. Read [the plan](../PLAN.md), [sprint index and shared rules](../SPRINTS.md), and [README](../../README.md).
+
+
+### Goal and scope
+
+Deploy the locally validated broker as a separate remote test instance and verify outbound-only local clients against it.
+
+### Deliverables and likely files
+
+- Environment-specific deployment examples with placeholders and final pinned digest.
+- `docs/DEPLOYMENT.md` complete remote certificate/firewall/operations runbook.
+- Remote entries in `docs/VERIFICATION.md`; no secrets committed.
+
+### Tasks
+
+1. Obtain OS/CPU, Docker/Compose support, available resources/port, DNS/SANs, access method and certificate renewal constraints without requesting secret values in chat.
+2. Inspect current listeners/services read-only; choose an isolated instance/port and validate artifacts locally before remote mutation.
+3. Prepare test-only credentials/ACLs, certificate provisioning/renewal, data/config mounts, health check, ownership, bounded logs and firewall requirements.
+4. Present exact remote operations and obtain any required authorization. Then deploy only the authorized test instance.
+5. Verify hostname/certificate/auth/ACL behavior from the development computer, simulator exchange, restart/session recovery and health.
+6. Document backup/update/rollback with compatible persistence snapshot and renewal verification. Browser viewer remains local.
+
+### Dependencies and required inputs
+
+Sprint 5 local verification; remote details/access and explicit authorization for remote changes. If unavailable, finish locally reviewable artifacts and report remote tests blocked.
+
+### Acceptance criteria
+
+- Separate remote TLS instance works without changing production listeners/configuration or requiring inbound development-computer access.
+- Local backend/simulator exchange messages and show them in logs/viewer through outbound connections.
+- Remote security and restart checks have actual evidence; runbook is complete.
+- If access is unavailable, the sprint is explicitly incomplete for deployment, not marked verified.
+
+### Automated checks
+
+Local artifact validation before deployment; remote health and focused MQTT integration scenarios after authorization. Repeat checks only where remote environment changes behavior.
+
+### Manual verification
+
+Inspect remote listening ports, certificate chain/SAN, log bounds, access controls and persistence. Walk through documented update/rollback steps using the test instance where authorized.
+
+### Risks and unresolved decisions
+
+Unknown remote environment, existing port conflicts, certificate renewal method and network restrictions. Native deployment is an explicit alternative if Compose is unsuitable.
+
+### Mandatory end-of-sprint testing and completion gate
+
+After implementing this sprint, run its automated checks and manual verification above. Inspect the final changes for correctness, scope, secret exposure, and compatibility with completed sprints. Fix failures and rerun affected checks before declaring completion.
+
+Record the exact commands/checks, actual outcomes, verification environment, and each acceptance criterion as PASS, FAIL, or BLOCKED. Include applicable regression checks for previously completed behavior. A skipped/unavailable required check is BLOCKED, never PASS; do not claim full sprint completion while required checks fail or remain blocked. Do not run unrelated or production-project tests.
+
+Report the results and remaining limitations to the user. Wait for authorization before the next sprint unless continued execution was explicitly authorized. Even with permission to continue, do not bypass this verification gate.
+
+### Copy-ready implementation prompt
+
+```text
+Implement Sprint 6 only in /home/ryan/Media/Code/Code/robot-market/test-mqtt-5 after authorization. Read applicable AGENTS.md, /home/ryan/.codex/RTK.md, README.md, docs/PLAN.md and docs/sprints/06-remote-broker.md. Use rtk. Existing projects, production listeners and credentials must remain untouched.
+
+First obtain/inspect remote OS, CPU, Docker/Compose support, free test port, DNS/certificate constraints and access method without exposing secrets. Prepare and locally validate the dedicated TLS Mosquitto deployment from prior sprints, pinned by compatible digest, with separate test credentials, per-device ACLs, persistence, restrictive secret permissions, bounded logs and authenticated health check. Complete certificate issuance/renewal, firewall, backup/update/rollback procedures. Present concrete remote operations before requesting any still-required authorization. Do not treat documentation authorization as deployment authorization.
+
+Only after required authorization, deploy the isolated remote test instance and verify outbound-only local backend/simulator connectivity, hostname validation, bad credentials/ACL behavior, restart and health. Keep the browser viewer on local loopback. If access/details are missing, preserve complete local artifacts and report remote work blocked/unverified. Update deployment/verification docs, report evidence by environment and wait.
+
+ESP32 reference: read docs/PLAN.md section 6.1 for the supplied Esp32MqttConfig. It is connection configuration, not a payload schema or a blocker for Go implementation. Preserve the proposed Go test contracts. Device-specific MQTT 5 selection depends on the actual C library/build/init; inspect TLS/CA handling, role-specific credentials and single publish_topic limitations during device integration. Do not modify existing C projects. Separate MQTT 3.1.1 payload interoperability from native MQTT 5 feature evidence.
+
+Mandatory completion gate: after implementing this sprint, run its listed automated and manual checks plus relevant regression checks, inspect the final changes for correctness/scope/secrets/compatibility, fix failures and rerun affected checks. Report exact checks, real outcomes, environment, and every acceptance criterion as PASS/FAIL/BLOCKED. Missing required checks are BLOCKED, never a pass. Do not mark the sprint complete with unresolved failures/blockers or bypass verification when continuing to another sprint.
+```

docs/sprints/07-device-compatibility.md

@@ -0,0 +1,71 @@
+# Sprint 7 — Conditional real-device compatibility
+
+Status: proposed; not implemented. Read [the plan](../PLAN.md), [sprint index and shared rules](../SPRINTS.md), and [README](../../README.md).
+
+
+### Goal and scope
+
+Determine and verify actual device behavior without modifying existing device projects or operating real machinery.
+
+### Deliverables and likely files
+
+- `docs/DEVICE-SETUP.md` with verified framework/client/protocol and test configuration.
+- Device-specific compatibility results in `docs/VERIFICATION.md`.
+- Only if separately approved: a small independent test firmware/application under `device/` inside this project.
+
+### Tasks
+
+- Inspect how Esp32MqttConfig is consumed: host versus URI, ca_cert PEM versus path, TLS hostname checks, selected protocol and library version, and whether publish calls can target telemetry/status/ACK topics separately. Propose only necessary device-side settings; this struct is not a payload contract.
+
+1. Inspect supplied Linux device and/or ESP32 source read-only; identify framework, exact client version, configured MQTT protocol, TLS/session handling, identities, payloads and command paths.
+2. Determine direct ESP32-to-broker versus gateway architecture and select one initial endpoint with the user.
+3. Map supported existing behavior to the test contract. Identify required source changes explicitly; do not modify existing projects or silently add a compatibility bridge.
+4. Use a dedicated safe test instance/board. If required, propose an isolated test application in this project and obtain authorization before flashing hardware or writing that new device implementation.
+5. Configure test credentials/CA/clock/topic identity; verify actual protocol and native-property evidence, telemetry/status, harmless command lifecycle, reconnect, expiry and duplicate handling where supported.
+6. Update comparison results and limitations. Leave unavailable features unsupported/not_tested/blocked with reasons.
+
+### Dependencies and required inputs
+
+Sprint 5 minimum; Sprint 6 if using the remote broker. Actual device repositories, board model, hardware/access, framework/toolchain and explicit authorization for flashing/test-device changes. Hardware unavailable means this sprint remains blocked; prior simulator work remains useful.
+
+### Acceptance criteria
+
+- “Device backend” is resolved using actual source and deployment evidence.
+- Exact tested SDK/client/protocol and environment are recorded.
+- Only harmless commands run; production/device repositories remain unchanged.
+- Native MQTT 5 features are marked passed only with actual end-to-end evidence. JSON-only interoperability is labeled separately.
+- No simulator result is substituted for hardware verification.
+
+### Automated checks
+
+Approved isolated device builds/contract checks and backend integration tests appropriate to the identified stack. Do not install arbitrary SDKs before choosing the actual target.
+
+### Manual verification
+
+Observe device traffic in backend logs/browser; issue ping/get_version; exercise approved device reconnect/restart and record actual outcomes. Inspect test-broker/client evidence of protocol selection.
+
+### Risks and unresolved decisions
+
+Hardware availability, flashing permissions, clock/certificate resources, SDK support and unknown device contract. Existing firmware may require changes outside authorized scope; stop at a concrete change proposal rather than modifying it.
+
+### Mandatory end-of-sprint testing and completion gate
+
+After implementing this sprint, run its automated checks and manual verification above. Inspect the final changes for correctness, scope, secret exposure, and compatibility with completed sprints. Fix failures and rerun affected checks before declaring completion.
+
+Record the exact commands/checks, actual outcomes, verification environment, and each acceptance criterion as PASS, FAIL, or BLOCKED. Include applicable regression checks for previously completed behavior. A skipped/unavailable required check is BLOCKED, never PASS; do not claim full sprint completion while required checks fail or remain blocked. Do not run unrelated or production-project tests.
+
+Report the results and remaining limitations to the user. Wait for authorization before the next sprint unless continued execution was explicitly authorized. Even with permission to continue, do not bypass this verification gate.
+
+### Copy-ready implementation prompt
+
+```text
+Perform Sprint 7 only after explicit authorization, using /home/ryan/Media/Code/Code/robot-market/test-mqtt-5 as the only writable project. Read applicable AGENTS.md, /home/ryan/.codex/RTK.md, README.md, docs/PLAN.md and docs/sprints/07-device-compatibility.md. Prefix shell commands with rtk. Existing device/server projects are read-only. No vending, motor, payment, credentials-change or OTA commands.
+
+Obtain the actual Linux device/ESP32 repositories and board/access details. Inspect framework, exact MQTT client/version, configured protocol, TLS/session/topic/identity/correlation behavior. Resolve whether the target is direct ESP32 MQTT or a gateway and identify one initial test endpoint. Map compatibility to the isolated test contract; do not infer MQTT 5 support from broker capability or message delivery. If device changes are needed, present the concrete changes and prefer a separately authorized independent test application inside this project. Do not modify existing firmware or flash hardware without authorization.
+
+On authorized dedicated test hardware, verify TLS/time, telemetry/status, ping/get_version ACK stages, native Response Topic/Correlation Data, reconnect/session/expiry/dedup behavior where supported. Inspect real connection/property evidence. Document exact versions, setup, results and limits in docs/DEVICE-SETUP.md and docs/VERIFICATION.md. If hardware/source/access is missing, report blocked/not_tested rather than substituting simulator evidence. Report acceptance status and stop.
+
+ESP32 reference: read docs/PLAN.md section 6.1 for the supplied Esp32MqttConfig. It is connection configuration, not a payload schema or a blocker for Go implementation. Preserve the proposed Go test contracts. Device-specific MQTT 5 selection depends on the actual C library/build/init; inspect TLS/CA handling, role-specific credentials and single publish_topic limitations during device integration. Do not modify existing C projects. Separate MQTT 3.1.1 payload interoperability from native MQTT 5 feature evidence.
+
+Mandatory completion gate: after implementing this sprint, run its listed automated and manual checks plus relevant regression checks, inspect the final changes for correctness/scope/secrets/compatibility, fix failures and rerun affected checks. Report exact checks, real outcomes, environment, and every acceptance criterion as PASS/FAIL/BLOCKED. Missing required checks are BLOCKED, never a pass. Do not mark the sprint complete with unresolved failures/blockers or bypass verification when continuing to another sprint.
+```