Deploy MQTT test backend with private viewer and Gitea CI/CD
This commit is contained in:
@@ -0,0 +1,99 @@
|
||||
# Sprint 3 — MQTT 5 backend, simulator and visible message flow
|
||||
|
||||
Status: COMPLETE on 2026-10-06; local unit/race, CLI and dedicated remote-broker checks passed. Read [the plan](../PLAN.md), [sprint index and shared rules](../SPRINTS.md), and [README](../../README.md).
|
||||
|
||||
|
||||
### Goal and scope
|
||||
|
||||
Run the complete harmless local command/telemetry loop with structured input/output logs and counters.
|
||||
|
||||
### Deliverables and likely files
|
||||
|
||||
- `cmd/device-sim/main.go`; `cmd/test-mqtt5` observe/command modes.
|
||||
- `internal/mqtt/`, `internal/backend/`, `internal/simulator/`; event buffer and counters.
|
||||
- Focused unit/integration tests, README quick start and contract updates.
|
||||
|
||||
### Tasks
|
||||
|
||||
- Support the user-requested plaintext test connection through explicit mqtt.allow_plaintext=false-by-default and a mqtt:// URL only when enabled. Keep TLS verification intact, preserve test namespace/ACLs, and update configuration tests/example/docs.
|
||||
|
||||
1. Integrate `github.com/eclipse/paho.golang v0.23.0`/autopaho with verified TLS, deadlines, bounded jittered backoff and configured role/instance IDs.
|
||||
2. Implement explicit initial Clean Start/session expiry and subscription readiness/restoration. Copy callback input into bounded workers; define visible overflow behavior.
|
||||
3. Simulator publishes telemetry, retained online status and an offline Will; handles ping/get_version and emits accepted/completed ACKs.
|
||||
4. Register pending correlation before publish; use exact Response Topic and native Correlation Data; validate device/topic/request ID/properties and transport reason codes.
|
||||
5. Enforce application deadline plus MQTT Message Expiry; implement bounded deduplication with in-progress entries, cached ACK replay and conflicting-ID rejection.
|
||||
6. Produce readable text/JSON events for inbound/outbound payload previews/properties and lifecycle outcomes. Separate attempted publication from transport ACK and device completion.
|
||||
7. Add own-request latency/counters, late ACK classification, bounded buffers and graceful shutdown. No offline command outbox or automatic ambiguous retries.
|
||||
|
||||
### Dependencies and required inputs
|
||||
|
||||
Sprint 1, Sprint 2 server package deployed and verified, broker hostname/ports, private test backend/device credential files and public CA. No physical device is required.
|
||||
|
||||
### Acceptance criteria
|
||||
|
||||
- Observer receives valid telemetry/status; command mode receives correlated ping/version results from the simulator.
|
||||
- Logs show safe payload preview, topic, direction, QoS, request ID and native properties; no credentials leak.
|
||||
- Wrong-topic/device/correlation ACKs cannot complete a pending request; late/out-of-order ACKs do not corrupt state.
|
||||
- An intentional repeated request ID does not repeat simulated execution; conflicting reuse is rejected.
|
||||
- Missing ACK and disconnect produce truthful timeout/unknown outcomes. SIGINT/SIGTERM exits within the configured bound.
|
||||
- Session settings, queue/dedup limits and restart limitations are documented.
|
||||
|
||||
### Automated checks
|
||||
|
||||
gofmt, go vet, unit tests and race detector. Real-broker integration for initial exchange, correlation, timeout, duplicate/conflicting request and shutdown. Scope full fault matrix to Sprint 5.
|
||||
|
||||
### Manual verification
|
||||
|
||||
Run observer and simulator in separate terminals, then run command mode with a distinct client ID. Follow one request through publish_attempt, transport_ack, accepted and completed. Compare safe previews to actual test payloads.
|
||||
|
||||
### Risks and unresolved decisions
|
||||
|
||||
Autopaho callback/ACK timing and concurrent state ownership; inspect pinned behavior instead of assuming backpressure guarantees. Transport success is never execution proof.
|
||||
|
||||
### Mandatory end-of-sprint testing and completion gate
|
||||
|
||||
After implementing this sprint, run its automated checks and manual verification above. Inspect the final changes for correctness, scope, secret exposure, and compatibility with completed sprints. Fix failures and rerun affected checks before declaring completion.
|
||||
|
||||
Record the exact commands/checks, actual outcomes, verification environment, and each acceptance criterion as PASS, FAIL, or BLOCKED. Include applicable regression checks for previously completed behavior. A skipped/unavailable required check is BLOCKED, never PASS; do not claim full sprint completion while required checks fail or remain blocked. Do not run unrelated or production-project tests.
|
||||
|
||||
Report the results and remaining limitations to the user. Wait for authorization before the next sprint unless continued execution was explicitly authorized. Even with permission to continue, do not bypass this verification gate.
|
||||
|
||||
### Copy-ready implementation prompt
|
||||
|
||||
```text
|
||||
Implement Sprint 3 only in /home/ryan/Media/Code/Code/robot-market/test-mqtt-5 after authorization. Read applicable AGENTS.md, /home/ryan/.codex/RTK.md, README.md, docs/PLAN.md and docs/sprints/03-backend-simulator.md. Use rtk for shell commands. Do not edit siblings, production infrastructure, or physical firmware.
|
||||
|
||||
Using the existing configuration/contracts and isolated deployed test broker, implement a Go MQTT 5 observer/command CLI and device simulator with github.com/eclipse/paho.golang v0.23.0/autopaho, never paho.mqtt.golang. Support only ping/get_version plus telemetry/status. Use distinct predictable client IDs, TLS hostname verification, bounded workers/state, initial Clean Start=true, 300-second session expiry, subscription readiness/restoration, bounded jittered reconnect, retained online/offline status and explicit Will semantics. Commands are non-retained QoS 1 with a 15-second default deadline and MQTT Message Expiry. Correlate exact device/topic/request ID and native Response Topic/Correlation Data; inspect MQTT reason codes. Implement bounded request deduplication, cached ACK replay, conflict rejection, late ACKs, and graceful signal shutdown.
|
||||
|
||||
Make inputs/outputs visible in text/JSON slog events with sanitized bounded payload previews and MQTT properties. Distinguish publication attempt, transport acknowledgement, application acceptance and completion; expose counters and own-request latency. Do not introduce a database, offline outbox, HTTP viewer or dangerous commands. Run meaningful unit/race and real-broker happy-path/correlation/timeout/dedup/shutdown tests. Update quick start and contracts, report verified environment and limitations, then wait.
|
||||
|
||||
ESP32 reference: read docs/PLAN.md section 6.1 for the supplied Esp32MqttConfig. It is connection configuration, not a payload schema or a blocker for Go implementation. Preserve the proposed Go test contracts. Device-specific MQTT 5 selection depends on the actual C library/build/init; inspect TLS/CA handling, role-specific credentials and single publish_topic limitations during device integration. Do not modify existing C projects. Separate MQTT 3.1.1 payload interoperability from native MQTT 5 feature evidence.
|
||||
|
||||
Mandatory completion gate: after implementing this sprint, run its listed automated and manual checks plus relevant regression checks, inspect the final changes for correctness/scope/secrets/compatibility, fix failures and rerun affected checks. Report exact checks, real outcomes, environment, and every acceptance criterion as PASS/FAIL/BLOCKED. Missing required checks are BLOCKED, never a pass. Do not mark the sprint complete with unresolved failures/blockers or bypass verification when continuing to another sprint.
|
||||
```
|
||||
|
||||
## Completion evidence
|
||||
|
||||
Implemented only in the isolated test project; existing projects and server configuration were not changed. Added pinned Paho v0.23.0/autopaho, shared bounded transport, observer and safe command CLI, simulator, event history/counters and [RUNNING.md](../RUNNING.md). Plaintext requires explicit opt-in; TLS verifies CA and hostname. No browser, database, offline application outbox or physical-device implementation was added.
|
||||
|
||||
| Acceptance criterion | Status | Evidence |
|
||||
| --- | --- | --- |
|
||||
| Observer telemetry/status and correlated ping/version | PASS | Remote race tests on TLS 8884 and plaintext 1884; real CLI observer/simulator/command check |
|
||||
| Safe input/output logs, properties, lifecycle and counters | PASS | CLI JSON logs contain previews, native correlation and publish/transport/accepted/completed stages; copied credential passwords absent; bounded history tests |
|
||||
| Wrong topic/device/correlation, late/out-of-order ACK handling | PASS | Unit tests reject wrong identity/topic/correlation/retention/response/result/deadline; completion before acceptance remains terminal; remote wrong-correlation and late-ACK tests |
|
||||
| Repeated request ID and conflicting reuse | PASS | Unit and remote tests replay cached completion with one execution; changed command with same ID is rejected |
|
||||
| Timeout, disconnect unknown outcome and bounded shutdown | PASS | Remote deadline/late-ACK tests; abrupt relay connection loss returns unknown; CLI SIGTERM and simulator cancellation complete within configured 5-second bound |
|
||||
| Session/queue/dedup/restart settings documented | PASS | RUNNING.md records limits, overflow, in-memory state and no automatic application retry; real reconnect tests cover Session Present=true and false with subscription restoration |
|
||||
|
||||
Checks run from the new project with Go 1.27.1 and temporary module/build caches:
|
||||
|
||||
- `gofmt -l cmd internal`: no unformatted files.
|
||||
- `go vet ./...`: PASS.
|
||||
- `go test -race ./...`: PASS; default remote tests explicitly skip without the opt-in variable, separately verified below.
|
||||
- `go mod verify`: all modules verified.
|
||||
- `go build -buildvcs=false` for both CLI binaries: PASS.
|
||||
- `MQTT5_INTEGRATION_CREDENTIAL_DIR=/home/ryan/mqtt-test-credentials go test -race -v -count=1 -timeout=90s ./internal/integration`: PASS against 109.122.252.129. Covers TLS/plaintext happy paths, native properties, dedup/conflict, deadline/late ACK, local disconnect and actual TLS connection loss/reconnect with retained and lost broker sessions. The relay touches only its own loopback client connection; no broker restart/configuration change.
|
||||
- Real built CLI processes: observer/simulator/command ping; JSON previews/native properties/lifecycle; no credential output; successful bounded SIGTERM shutdown: PASS. Temporary check script/output lived under /tmp; the reproducible manual commands are in RUNNING.md.
|
||||
- Structural source review with graphify AST extraction: 20 Go files, 204 nodes and 660 extracted relationships; source and concurrency review completed without adding graph artifacts to the project.
|
||||
|
||||
Verified environment: Go MQTT 5 backend and local simulator against the dedicated remote Mosquitto 2.1.2 broker, both TLS and plaintext. Larger fault/compatibility matrix, crash durability, durable client/dedup recovery, browser viewing and physical ESP32 interoperability remain unverified or scheduled for later sprints. All Sprint 3 acceptance criteria passed; Sprint 4 requires authorization.
|
||||
Reference in New Issue
Block a user