mqtt-5-test/docs/PLAN.md

# MQTT 5 test backend plan

Planning date: 2026-10-06. Status: Sprints 1–4 implemented and verified, including the local browser viewer; later sprints await authorization.

## 1. Objective and authorization

Build a small Go test backend that receives telemetry/status, sends harmless commands, correlates application acknowledgements, and makes every relevant input/output and compatibility result understandable through logs and an optional local browser viewer.

The user authorized Sprints 1–4, then directed Sprint 2 toward a server Compose package they will deploy themselves, including optional plaintext MQTT alongside TLS. Sprint 1 is complete; Sprint 2 artifacts are validated locally, the user deployed the server broker, and server runtime plus external connection checks passed. Existing projects remain untouched. Sprint 3 MQTT 5 transport/backend/simulator, logs, deduplication, deadlines and graceful shutdown are implemented and verified locally and against the dedicated remote broker. Later sprints still require authorization.

Project root: `/home/ryan/Media/Code/Code/robot-market/test-mqtt-5`.

Follow applicable AGENTS.md instructions and `/home/ryan/.codex/RTK.md`; prefix shell commands with `rtk`. The existing server repository's build/test restrictions apply when working there; do not build or test that reference project as part of this work.

## 2. Findings and evidence

Reference root: `../robot-market-vending-backend/server_backend` (relative to the new project root).

| Evidence | Finding | Limit |
| --- | --- | --- |
| `go-back/go.mod` | Go 1.25.0, `github.com/eclipse/paho.mqtt.golang v1.5.0` | This is the MQTT 3 client, not the MQTT 5 implementation |
| `go-back/internal/mqtt/client.go` | TLS >=1.2, optional CA/client certificates, 30-second keepalive, auto reconnect, role-based client IDs | Does not establish the remote broker's active configuration |
| `go-back/internal/config/config.go` | Default QoS 1 and CleanSession=false | Defaults are not evidence of deployed values |
| `go-back/cmd/ingest/main.go` and `internal/ingest/subscriber.go` | Startup subscription registration; connection callback primarily logs | No explicit subscription rebuilding on lost broker session in this inspected path |
| `docs/contracts/handoffs/server-mqtt-correlation-ack-handoff-fa.md` | JSON `requestId` or `message_id`/`sync_job_id`; explicit `ok:true`; exact response-topic matching | These are application fields, not native MQTT 5 properties |
| `docs/phases/phase2/12-device-apt-repo-foundation.md` | Bookworm ARM64 packages for `app-core`, `io`, `mqtt-bridge`, `storage`, `payment` | Evidence of a Linux device service; its source/library/version was not available |
| `docs/architecture/runtime-and-deploy.md` | Mosquitto deployment references external `vending/docker` | Actual deployment artifacts, OS, CPU, listeners and certificates unavailable |
| Repository instructions | FN identifies the board; SN identifies the machine | This project must not allocate or mutate those identities |

Existing topics include `devices/...`, `device/{plate}/...`, and `vending/{serial}/...`. Some synchronization responses use `device/{plate}/sync/resp/{message_id}`. The new namespace intentionally isolates test traffic.

No ESP32 firmware, PlatformIO project, ESP-IDF configuration, or device MQTT implementation was found in the supplied parent project tree during discovery. “Device backend” appears to include a Linux service; the relationship to ESP32 remains unknown. Do not infer direct ESP32 MQTT connectivity or firmware capabilities from these documents.

Do not read/copy production secrets. No existing code changes are planned.

## 3. Architecture

```mermaid
flowchart LR
    Browser[Local browser] -->|Loopback HTTP + SSE| Backend[Go test backend]
    Backend -->|Outbound MQTT 5 / TLS| Broker[Isolated Mosquitto]
    Simulator[Local Go simulator] -->|Outbound MQTT 5 / TLS| Broker
    Device[Test hardware or device service] -->|Outbound TLS / verified protocol| Broker
```

The diagram arrows show connection initiation. MQTT messages travel in both directions over those connections. No public inbound access to the development computer is required.

The current broker deliverable is a server deployment package under deploy/. The user will deploy a separate broker instance using available TLS and optional plaintext test ports; existing production listeners stay untouched. No permanent local broker is required. If local isolation is later needed, bind host ports to loopback.

### Go structure and runtime

Planned entry points: `cmd/test-mqtt5` and `cmd/device-sim`. The backend has `observe` and `command` CLI modes; `command` supports `ping` and `get_version`. The viewer belongs to long-running observation mode. An observer can display command/ack traffic from a separate command-mode client but must not claim that traffic as its own measured pending-request latency.

Use focused `internal/config`, `internal/contracts`, `internal/mqtt`, `internal/backend`, `internal/simulator`, and `internal/events` packages; add `internal/web` only with the viewer sprint. Package names may be simplified if responsibilities remain clear. Use standard `flag`, `net/http`, `embed`, `encoding/json`, and `log/slog` rather than frameworks.

Construct dependencies explicitly. Use contexts and operation deadlines, contextual wrapped errors, fixed worker counts, bounded channels, and clear mutex/event-loop ownership. Interfaces only support actual testing or transport boundaries. Callbacks copy necessary data and enqueue quickly; they do not do HTTP, disk work, slow logging, or application execution. Connection hooks signal a supervised worker rather than starting unbounded goroutines.

On SIGINT/SIGTERM: stop admitting commands, stop HTTP/SSE, resolve pending requests as interrupted/unknown, publish graceful offline status where applicable within a bounded deadline, disconnect MQTT, cancel and join workers. Return a meaningful exit status.

### Resource defaults

Initial lab defaults: one simulator device, telemetry every 5 seconds, 16 KiB maximum application payload, 128 pending commands, 256 queued inbound events, 1,000 displayed events, 4 KiB payload preview, eight SSE viewers, and a 64-event queue per viewer. Make necessary limits configurable and validate them; do not make a flag for every setting.

On telemetry/event overflow: count and expose a gap/drop. On command-processing overload: never execute work that was not admitted; emit a rejection when feasible, otherwise allow a visible application timeout. MQTT receipt/transport ACK is not proof of admission. Event-stream loss must not affect command tracking. Clamp negotiated MQTT receive/packet limits to supported values and test the actual pinned client's behavior.

## 4. Dependencies and version policy

These are proposal pins established during discovery, not a tested lockfile. Before implementation, confirm tag existence, Go requirements, security advisories, and image platform/digest availability. Report any required version change instead of silently changing the approved design.

| Dependency | Proposed version | Source |
| --- | --- | --- |
| Go toolchain | 1.27.1 | https://go.dev/dl/ |
| `github.com/eclipse/paho.golang` including `autopaho` | v0.23.0 (requires Go >=1.24) | https://github.com/eclipse-paho/paho.golang/releases/tag/v0.23.0 |
| `github.com/knadh/koanf/v2` | v2.3.7 (requires Go >=1.23) | https://github.com/knadh/koanf/releases/tag/v2.3.7 |
| `github.com/knadh/koanf/parsers/toml/v2` | v2.2.2 | https://pkg.go.dev/github.com/knadh/koanf/parsers/toml/v2@v2.2.2 |
| `github.com/knadh/koanf/providers/file` | v1.2.1 | https://pkg.go.dev/github.com/knadh/koanf/providers/file@v1.2.1 |
| `github.com/knadh/koanf/providers/env` | v1.1.0 | https://pkg.go.dev/github.com/knadh/koanf/providers/env@v1.1.0 |
| Mosquitto | Verified `eclipse-mosquitto:2.1.2-alpine@sha256:38c0da4f2ef84284d47b3b3eeea1cb3bdeabe81ee10caf0cd5c5ff61ee3ea408` | https://mosquitto.org/download/ |
| ESP32 SDK/client | Not selected until device source/board is known | https://github.com/espressif/esp-mqtt |

Paho is released but still pre-1.0: API stability must not be assumed. Commit exact module versions and go.sum when implementation begins. Do not use `paho.mqtt.golang` in the new MQTT 5 implementation. No Node/npm dependency is needed for the viewer.

Additional official references:

- Autopaho controls and lifecycle: https://raw.githubusercontent.com/eclipse-paho/paho.golang/v0.23.0/autopaho/auto.go
- koanf loading/providers: https://github.com/knadh/koanf
- MQTT semantics: https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html
- Mosquitto settings: https://mosquitto.org/man/mosquitto-conf-5.html
- Mosquitto 2.1 migration notes: https://mosquitto.org/blog/2026/01/version-2-1-0-released/
- ESP-MQTT MQTT 5 example: https://components.espressif.com/components/espressif/mqtt/versions/1.0.0/examples/mqtt5?language=en

ESP-MQTT can support MQTT 5, but build configuration and runtime protocol selection must be checked. Do not choose or migrate an ESP32 framework without inspecting the actual device project.

## 5. Configuration

Precedence: defaults < TOML file < environment < explicitly supplied CLI overrides. Use typed structs and load once at startup. A supplied missing/malformed file is an error. Only flags actually visited override merged values.

Prefix: `TEST_MQTT5_`. Strip that prefix, lowercase, and map `__` to nested `.` while preserving single underscores: `TEST_MQTT5_MQTT__CLIENT_ID` -> `mqtt.client_id`. Reject unknown keys under this prefix. Treat empty secrets deliberately; never silently fall back to a different credential.

Sections cover broker URL, username/password or password-file, TLS CA/client certificate paths, client ID, topic namespace/device ID, QoS, initial Clean Start, Session Expiry, keepalive/backoff, operation/shutdown deadlines, logging, event limits, simulator behavior, and optional web viewer. Mutually exclusive secret sources receive actionable validation errors.

Use duration strings such as `5s` and `5m`, explicit decode hooks/conversion, and overflow/range checks before converting MQTT seconds to uint32. Reject bare numeric durations, invalid QoS, unsupported URL schemes, partial certificate pairs, wildcard/path injection in IDs, and deadlines inconsistent with deduplication retention. Unknown TOML keys should fail startup.

Flags: `--config`, `--device-id`, `--log-level`, and optionally `--web` are sufficient. Command arguments select the harmless operation and request ID. Never provide secrets through flags or log complete configuration objects.

TLS remains the default for broker connections. Per user request, the server package also offers an explicit Compose override publishing authenticated plaintext MQTT. Sprint 3 adds an explicit test-only mqtt.allow_plaintext setting (default false) to permit a mqtt:// URL when selected; implemented Sprint 3 validation supports plaintext only with the explicit opt-in. TLS connections always verify hostnames/SANs, using system roots and optionally a test CA. Client certificate/key is optional. Never set InsecureSkipVerify. Ignore local TOML, generated certificates, private keys, broker data, and credential files in Git. Examples contain placeholders only.

## 6. Topics and payloads

### 6.1 ESP32 connection configuration (supplied by user)

The user supplied this C connection configuration and clarified that it is not the application payload contract:

```c
typedef struct
{
    const char* wifi_ssid;
    const char* wifi_password;
    const char* mqtt_server;
    int         mqtt_port;
    const char* mqtt_user;
    const char* mqtt_pass;
    const char* subscribe_topic;
    const char* publish_topic;
    const char* ca_cert;
} Esp32MqttConfig;
```

This struct does not constrain the Go application's message structs or establish payload encoding. Go backend/simulator implementation is not blocked on a C serializer. The topic/payload examples in section 6.2 remain the proposed test contract; actual device payload compatibility is verified when the device code is available.

| C setting | Meaning for this environment | Go/backend impact |
| --- | --- | --- |
| wifi_ssid / wifi_password | ESP32 network connection | Device-only; never copy Wi-Fi credentials into Go config or logs |
| mqtt_server / mqtt_port | Address of the isolated test broker and TLS listener | Go uses its own TLS URL for the same reachable broker; confirm whether C expects hostname or URI |
| mqtt_user / mqtt_pass | Dedicated device credentials | Go uses separate backend credentials; ACLs reflect each role |
| subscribe_topic | Device's test command subscription | Set to `robot-market/test/v1/{device_id}/commands` for the proposed contract |
| publish_topic | A configured device publication topic | One string alone does not establish support for telemetry, status and ACK topics; inspect publish call sites before choosing a device adaptation |
| ca_cert | Device trust material | Inspect whether this is PEM text or a path; Go uses system roots/its own CA path. Never copy C pointers as configuration values |

The existing fields can still describe MQTT 5 broker connectivity. No struct change is required solely to move application payloads to MQTT 5. This struct has no explicit protocol-version field, so it cannot establish that the device uses MQTT 3.1.1 or MQTT 5: inspect the actual library/version, build options and client initialization. Port and certificate settings alone also do not prove TLS is enabled or hostname verification is enforced.

If the library supports MQTT 5, select it in client initialization/build configuration. Only add a protocol-version field to Esp32MqttConfig if runtime selection between 3.1.1 and 5 is actually needed. Do not prescribe enum values or exact library calls before identifying the C library. MQTT 5 Session Expiry, Clean Start, Will, Response Topic, Correlation Data and Message Expiry are configured through that library's supported APIs and need not all become fields in this small struct. Preserve JSON/application correlation for any 3.1.1 compatibility tests; never label it native MQTT 5 metadata. Reference: [OASIS MQTT 5 specification](https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html).

For the proposed four-topic test contract, inspect whether existing publish calls accept a topic per message. If yes, derive telemetry/status/ACK topics from a test namespace and device ID. If publication is fixed to publish_topic, propose the smallest device-only extension (for example namespace/device ID or explicit additional topic settings) before implementing it. Do not silently change the Go contract to one multiplexed topic, or subscribe to production topics. No C changes are authorized by documenting these alternatives.

Verification for device integration: isolated broker host/port, CA representation, TLS enablement and hostname verification, device clock, separate credential/ACL behavior, exact subscription/publication topics, library protocol selection and actual message/property exchange. For a 3.1.1 endpoint, ordinary payload interoperability with the Go MQTT 5 backend is testable through Mosquitto, but native property handling remains unsupported unless that endpoint is upgraded. The Go backend and simulator continue to use Paho MQTT 5 as planned.

### 6.2 Proposed test topics and payloads

Namespace: `robot-market/test/v1/{device_id}`. Test IDs use a restricted safe alphabet, such as `sim-001` or `esp32-lab-001`. Optional FN/SN metadata is informational only.

| Suffix | Direction | Default QoS | Retain |
| --- | --- | --- | --- |
| telemetry | Device -> backend | 1 | false |
| status | Device -> backend | 1 | true |
| commands | Backend -> device | 1 | false |
| acks | Device -> backend | 1 | false |

The proposed test messages have `schema_version:1` and `device_id`; match identity against the topic. Esp32MqttConfig does not define these payloads. Real-device payload support must be checked separately. Reject oversized input before decoding. Validate JSON objects, command allowlists, bounded strings, required identifiers, RFC3339 UTC timestamps, supported stages, and reasonable deadline ranges. MQTT payload-format/content-type properties can label JSON, but payload validation remains necessary.

Telemetry example:

```json
{"schema_version":1,"device_id":"sim-001","message_id":"evt-001","boot_id":"boot-001","sequence":1,"timestamp":"2026-10-06T10:00:00Z","readings":{"uptime_s":12}}
```

Online status example:

```json
{"schema_version":1,"device_id":"sim-001","boot_id":"boot-001","state":"online","timestamp":"2026-10-06T10:00:00Z","implementation":"go-simulator","reported_protocol":"5"}
```

Reported protocol is self-reported metadata, not independent proof. An offline Will carries `state:"offline"`, boot ID and `will_configured_at`; it must not label a preconstructed timestamp as the disconnect time. The observer adds its own reception time.

Command and completion examples:

```json
{"schema_version":1,"device_id":"sim-001","request_id":"req-001","issued_at":"2026-10-06T10:00:00Z","expires_at":"2026-10-06T10:00:15Z","command":"ping"}
```

```json
{"schema_version":1,"device_id":"sim-001","request_id":"req-001","timestamp":"2026-10-06T10:00:01Z","stage":"completed","ok":true,"result":{"pong":true}}
```

Acceptance is a separate ACK with `stage:"accepted"`. Terminal stages are `completed`, `rejected`, and `failed`; rejection/failure contains a bounded error code/message. Out-of-order or repeated ACKs must not regress terminal state. A completion received before acceptance still resolves the request correctly.

MQTT 5 command properties: Response Topic equals the exact device `acks` topic; Correlation Data is bounded opaque bytes and is echoed unchanged. Compare native bytes and JSON request ID when native correlation is enabled. Reject response topics outside the exact allowed topic, including wildcards. A documented JSON-only compatibility mode is separately selected and cannot count as a native-property test.

These contracts are test-only and are not derived from Esp32MqttConfig. Existing `requestId`, `message_id`, `/resp` and sync behavior is not silently reinterpreted; any real-device payload adaptation requires an explicit mapping, and no production namespace bridge is included.

## 7. Reliability and session policy

- Client IDs include lab/role/instance, e.g. `rm5-lab-observer-01`, `rm5-lab-command-01`, `rm5-lab-device-sim-001`. Concurrent processes need distinct configured instance IDs. Never reuse a production ID.
- Initial Clean Start=true and Session Expiry=300 seconds. Within a running process, reconnect with Clean Start=false. Client-process restart deliberately starts fresh because client protocol/pending/dedup state is initially in memory. Cross-process durable client recovery is outside this baseline.
- On every CONNACK inspect Session Present. When false, restore required subscriptions and check SUBACK reasons before readiness. Callback handlers must already exist when a resumed session can deliver queued messages. If session exists, retain its subscriptions rather than blindly replaying them.
- Reconnect backoff uses jitter and a bounded 1–30-second range; context cancellation always stops it. Initial connection has a deadline. Denials are observable and do not produce tight retry loops.
- Publish retained online status only after command subscription readiness. Configure retained QoS 1 offline Will with explicit zero Will Delay. Graceful shutdown attempts offline publication before normal disconnect. Network failure detection still depends on keepalive; absence/staleness is unknown, not proof of current online state.
- Commands default to a 15-second application lifetime and MQTT Message Expiry consistent with remaining lifetime. Expired messages are never newly published or executed. A delayed delivery must be checked against expires_at even if broker expiry was set.
- No unbounded offline application publish queue. If disconnected, reject new CLI commands visibly. Already submitted QoS traffic may still complete or be retransmitted after the caller times out.
- Distinguish publish attempt, successful transport acknowledgement, application acceptance, and completion. Inspect MQTT reason codes; a returned publish result is not automatically success.
- Register correlation before publishing and after ACK subscription readiness. Local timeout means unknown delivery/execution outcome; late ACKs are visible and classified late.
- Intentional retry reuses request ID, operation, and original deadline. The simulator deduplicates by device/request ID, remembers in-progress and final results, and rejects same-ID/different-content conflicts. Do not rely on the MQTT DUP bit as the application deduplication key.
- Dedup cache is bounded (initially 1,024 requests, ten-minute retention; command lifetime capped below retention). Do not evict active/unexpired protected entries to accept more work: reject overload. Cache loss on restart is documented; no exactly-once side-effect guarantee is claimed.
- Message Expiry limits broker delivery; it cannot cancel already delivered work. The device checks application deadlines before work and applies its own cancellation policy. Device time must be synchronized before TLS/deadline hardware tests.

## 8. Logs and input/output inspection

Use one sanitized event model for logs and browser display. Event fields include event ID, observed UTC timestamp, direction (`in`, `out`, `internal`), local client/device identity, topic, QoS, retain/DUP flags when available, payload byte length, bounded preview, MQTT properties, request ID, outcome/reason, and measured latency where locally attributable.

Differentiate events such as `publish_attempt`, `transport_ack`, `message_received`, `validation_failed`, `command_accepted`, `command_completed`, `command_timeout`, `late_ack`, `duplicate_request`, `connection_up`, `connection_down`, and `event_gap`. Never display a publish attempt as proven device receipt.

Default safe previews show the explicitly allowed harmless test fields. Redact secret/token/password fields before logging; do not dump arbitrary invalid JSON or binary payloads. Invalid payload events show metadata and validation errors without echoing raw input. Diagnostic preview opt-in remains bounded and sanitized. Never log credentials, TLS private material, config dumps, or URL userinfo.

Provide text logs for humans and JSON logs for scripts. Basic counters: attempts/connections/reconnects, publish attempts/transport successes/failures, received messages, validation failures, ACK stages, own-request acceptance/completion latency, timeouts, duplicate requests/ACKs, and dropped inspection events. Periodic and shutdown summaries are enough; no metrics server is required.

Usage labels are explicit: application payload bytes exclude MQTT/TCP/TLS overhead; subscriber counts describe its deliveries, not total broker traffic; $SYS generally measures the broker; in-memory counters are not durable accounting. Telemetry history means only the bounded current-process event buffer unless separately approved.

## 9. Optional browser viewer (Sprint 4 complete)

Use Go net/http + embedded HTML/CSS/vanilla JavaScript. Default disabled; `--web` enables `127.0.0.1:8080`. No CDN, frontend build tool, or separate service. Changing to a public bind is outside this baseline.

Read-only pages show connection/readiness, counters, an event timeline with direction/topic/device/request filters, payload/property details, command lifecycle, and a compatibility table. Show empty, disconnected, truncated, and event-gap states. Pause freezes display, not backend processing.

Use GET endpoints for initial sanitized event snapshot and Server-Sent Events for live updates. Event IDs allow replay from the bounded ring on reconnect; if history is gone, emit an explicit gap/reset. Slow browser clients use bounded queues and are disconnected or notified of gaps without blocking MQTT. Clean up subscriptions on close. No durable browser history or command-publishing HTTP endpoint.

Render text via textContent, never payload-supplied HTML. Add a restrictive CSP, same-origin behavior, expected Host/Origin checks, no permissive CORS, and no remote script loading. This is a local developer tool, not a public authenticated dashboard. SSE reconnects must not duplicate event counters or multiply handlers.

## 10. Compatibility reporting

Track each feature per endpoint with `not_tested`, `passed`, `failed`, `unsupported`, or `blocked`, plus evidence, timestamp, endpoint identity, client version when known, and environment (`local_simulator`, `remote_simulator`, `physical_device`). A protocol claim alone cannot mark a feature passed.

| Feature | Go simulator | Existing Linux device service / ESP32 |
| --- | --- | --- |
| TLS, topic/payload contract, QoS 1 | Planned automated proof | Unknown until actual client/source and configuration inspected |
| JSON command/ACK correlation | Planned | Requires test contract support; old payloads are not automatically compatible |
| Native Response Topic / Correlation Data | Planned end-to-end echo verification | Requires MQTT 5 client and property handling |
| Session expiry and subscription restoration | Planned controlled outage scenarios | Requires configured client behavior and observed results |
| Broker Message Expiry | Planned queued-delivery scenario | Broker-side behavior plus device deadline enforcement tested separately |
| Will/retained online state | Planned crash and graceful-stop scenarios | Requires test configuration and freshness interpretation |
| Duplicate request handling | Planned deterministic repeat plus optional wire retransmission | Requires actual application idempotency behavior |

A backend cannot infer another subscriber's MQTT version merely from a delivered message. Use device client source/configuration plus test-broker connection evidence and successful native-property behavior. Cross-version delivery alone proves neither MQTT 5 support nor property preservation.

Initially, live events and scenario results are separate concepts: merely seeing traffic does not auto-pass every compatibility row. Scenario tests emit explicit verdicts. The browser can show supported live evidence and loaded/available current-process results; untested rows remain not_tested. No upload/import framework or persistent report database is planned.

## 11. Broker deployment and operations

Prefer Docker Compose on a supported server. If Docker is unavailable, propose an exact native package/systemd alternative before implementation. Confirm OS/CPU, permissions, existing port use, DNS/SANs, certificate provisioning, and firewall first.

Use dedicated test data/configuration and a verified image digest. Enable authenticated TLS and disable anonymous access. The base Compose file publishes only TLS (default server port 8884); the user-requested insecure override additionally publishes plaintext MQTT (default 1884), with the same credentials/ACLs and explicit firewall scoping. Plaintext is not a certificate-verification bypass. Mosquitto 2.1 has new password/ACL plugin guidance; validate the exact pinned release syntax and plugin paths in the container rather than copying a 2.0 recipe.

Backend can publish only test commands for allowed devices and read their telemetry/status/ACKs. Each device can read only its commands and publish only its own telemetry/status/ACKs. A health-check account can only access a separate health topic. Deny production topics, cross-device writes, and unnecessary $SYS access. MQTT authentication does not remove the need for topic/payload identity validation.

Persist broker sessions/retained messages for restart tests; configure queue, packet and client limits. Restrict config/certificate/password-file ownership and modes to the actual image UID/GID. Mount secrets read-only; avoid exposing them in command arguments or Compose output. Bound container logs, document persistence checkpoint/crash-loss limits, and avoid promising crash durability from a graceful-restart test.

Health check performs authenticated TLS subscribe/publish/receive with a unique ID and timeout on the health topic. It should detect broken listener/auth/ACL behavior, not merely a live process. Clients must still perform independent remote hostname verification.

Firewall allows only the selected TLS port for intended clients where feasible. Public-CA certificate issuance/renewal or private test CA is chosen after DNS/network details are known. Renewal copies readable cert material safely, reloads/restarts the isolated broker as required, and verifies the served certificate. Do not weaken hostname checks for IP-only access; use a matching IP SAN or proper DNS.

Runbook includes start, stop, logs, health, config validation, backup, update, and rollback. Back up configuration and a consistent broker data snapshot before update. Rollback restores the prior image and compatible data snapshot; newer persistence data may not be backward-compatible. No production broker changes are authorized.

## 12. Verification and implementation sequence

Sequence: config/contracts -> server Compose/setup package and operator deployment -> backend/simulator using the provided test-broker endpoint -> browser viewer -> reliability/compatibility scenarios -> remote operational review -> conditional physical-device integration. Detailed sprints are separate files under `docs/sprints/`, linked from [SPRINTS.md](SPRINTS.md). After each sprint, execute its automated and manual checks, review changes, fix failures and rerun affected verification. Report every acceptance criterion with evidence as PASS/FAIL/BLOCKED. Required checks that cannot run block full completion; never count a skip as a pass.

Unit checks cover precedence/typed decoding, validation, correlation including wrong device/topic/properties, terminal-state ordering, timeout and late ACKs, deadline checks, duplicate/conflicting IDs, limits, redaction, cancellation, and shutdown. Use gofmt, go vet, focused tests and race detection in the new project only.

Real-broker scenarios cover initial exchange, bad credentials, forbidden topics, invalid CA/hostname, broker restart, network interruption, session resume versus loss/expiry, subscription recovery, queued command expiry, missing/delayed ACK, Will, graceful shutdown, and bounded resource behavior. Deterministic repeated application messages test idempotency; actual QoS 1 retransmission needs controlled transport interruption and may not reproduce on every run. Record that distinction.

Browser verification covers live traffic, escaped hostile strings, redaction/truncation, filters, reconnection/replay/gap, slow clients, and shutdown. Compare rendered events to logs and avoid double-counting observer traffic as sender-owned latency.

Every report separates verified locally, verified against remote broker, verified on physical hardware, and not verified. Sprint 1 local configuration, payload validation, safe previews and offline CLI behavior are verified. Sprint 2 Compose/setup/certificate/configuration checks are verified locally. Server-container MQTT security and graceful persistence checks passed in user-provided output. External MQTT 5 authenticated publish/receive passed from this development computer over verified TLS and plaintext on 2026-10-06. Sprint 3 unit/race checks, real CLI log/signal checks, TLS/plaintext command/telemetry exchanges, native correlation, duplicate/conflict handling, timeout/late ACK/unknown outcomes and reconnect with retained/lost sessions passed. Physical hardware remains unverified.

## 13. Dependencies, risks and open inputs

| Input/risk | Effect | Response |
| --- | --- | --- |
| ESP32 MQTT initialization/library unknown | Blocks exact device-side MQTT 5 selection and TLS/topic verification, not Go implementation | Inspect the C client setup before the device sprint; use supplied Esp32MqttConfig as connectivity reference |
| Missing device source/board | Blocks real-device client/version/contract decisions | Continue with simulator; obtain paths and board details before hardware sprint |
| Missing remote OS/CPU/access/DNS | Blocks exact remote deployment/cert choices | Validate local artifacts first; collect details without requesting secrets in chat |
| Local container tooling unavailable | Blocks real-broker tests | Report prerequisite; propose native isolated broker if appropriate |
| Client-ID collision | Disconnects another lab client | Configured role/instance IDs; separate observer and command instances |
| Device clock drift | TLS or deadline failures | Synchronize clock and report clock-dependent tests honestly |
| In-memory state loss | Lost history/dedup/pending requests on restart | Explicit limits; fresh client session on process restart; harmless operations only |
| Browser overload or hostile payload | Inspection instability or injection | Bounded fan-out and text-only sanitized rendering |
| Dependency/image availability | Proposed pins may need revision | Recheck official metadata and report material changes before adopting them |

User inputs needed before their dependent work: device service/firmware paths and available boards; remote OS/CPU/tooling/free port/access and certificate situation; whether usage beyond local counters and bounded recent history is required. None blocks writing these documents or reviewing the local-first sprints.