commit b234bd910816d93e90ce3d1bcdc91881d37c8bd1 Author: ryan Date: Thu Oct 8 13:45:52 2026 +0330 first commit diff --git a/.gitignore b/.gitignore new file mode 100644 index 0000000..0af6bd2 --- /dev/null +++ b/.gitignore @@ -0,0 +1,2 @@ +/lab/secrets/ +/.env diff --git a/.idea/.gitignore b/.idea/.gitignore new file mode 100644 index 0000000..30cf57e --- /dev/null +++ b/.idea/.gitignore @@ -0,0 +1,10 @@ +# Default ignored files +/shelf/ +/workspace.xml +# Editor-based HTTP Client requests +/httpRequests/ +# Ignored default folder with query files +/queries/ +# Datasource local storage ignored files +/dataSources/ +/dataSources.local.xml diff --git a/.idea/go.imports.xml b/.idea/go.imports.xml new file mode 100644 index 0000000..644cdf0 --- /dev/null +++ b/.idea/go.imports.xml @@ -0,0 +1,10 @@ + + + + + + \ No newline at end of file diff --git a/.idea/modules.xml b/.idea/modules.xml new file mode 100644 index 0000000..df70510 --- /dev/null +++ b/.idea/modules.xml @@ -0,0 +1,8 @@ + + + + + + + + \ No newline at end of file diff --git a/.idea/mqtt-5-learning.iml b/.idea/mqtt-5-learning.iml new file mode 100644 index 0000000..5e764c4 --- /dev/null +++ b/.idea/mqtt-5-learning.iml @@ -0,0 +1,9 @@ + + + + + + + + + \ No newline at end of file diff --git a/.idea/vcs.xml b/.idea/vcs.xml new file mode 100644 index 0000000..c9c2a22 --- /dev/null +++ b/.idea/vcs.xml @@ -0,0 +1,7 @@ + + + + + + + \ No newline at end of file diff --git a/COURSE.md b/COURSE.md new file mode 100644 index 0000000..1fb1ed4 --- /dev/null +++ b/COURSE.md @@ -0,0 +1,327 @@ +# Go MQTT 5 backend course for Robot Market + +## Starting point and destination + +You already understand MQTT 3.1.1 concepts. This course concentrates on writing, debugging, testing, and maintaining Go services that communicate with IoT devices. Go and basic Docker syntax are assumed. Explain MQTT semantics where code depends on them, including packet flows, guarantees, failure boundaries, and the purpose and effect of MQTT or advanced Docker commands. Each experiment starts with a prediction and ends with an explanation, not merely successful copied output. + +You will build a small vending backend and a Go device simulator, then use that experience to understand the MQTT 5 backend delivered to you. Work through the numbered sections in order. Each checkbox is an exercise you implement; the tutor provides explanations, small examples, debugging help, and code review. A section can take several sessions. There is no need to finish a whole section in one sitting. + +This is the requested course and implementation checklist. It does not require another round of sprint planning. Ask the tutor to start a numbered exercise when ready; full solutions should wait until you have attempted it or requested one. + +All implementation belongs in `/home/ryan/Media/Code/Code/robot-market/mqtt-5-learning`. Use a dedicated local broker, synthetic data, lab credentials, and simulated devices. Reference the existing backend read-only. + +## How this course matches your backend + +The local backend was inspected on 2026-10-07. These are current source observations, not assumptions about the future MQTT 5 delivery: + +| Observed technology or responsibility | Course connection | +| --- | --- | +| Go 1.25.0 and `github.com/eclipse/paho.mqtt.golang v1.5.0` in `go.mod` | Learn the separate MQTT 5 Paho APIs and later review migration differences | +| `net/http`, API entry point, separate ingestion entry point | Build HTTP and MQTT flows together first, then split processes | +| `database/sql` with pgx stdlib in the DB connector; pgx/v5 and lib/pq dependencies | Add PostgreSQL repositories, transactions, and migrations after in-memory exercises | +| `godotenv` and environment-based configuration | Learn typed TOML + koanf for this lab; understand the configuration boundary at handover | +| JWT dependency, authentication middleware, device permission modules | Apply user authorization to device operations separately from broker permissions | +| Product push envelopes carrying message/job IDs, contract version, and revisions | Practice catalog synchronization and correlation | +| Sales ingestion validates identity, saves reports, and sends application responses | Practice durable ingestion, deduplication, and acknowledgement timing | +| Topic routing includes health, logs, inventory, configuration, updates, and command responses | Build representative small flows before studying the larger routing surface | + +Source reading map, relative to `../robot-market-vending-backend/server_backend/go-back/`: + +- `go.mod`, `internal/config/config.go`, `internal/db/db.go` — dependencies, configuration, database setup. +- `internal/mqtt/client.go` — existing connection configuration and MQTT 3 client lifecycle. +- `cmd/api/main.go`, `cmd/ingest/main.go` — process composition. +- `internal/ingest/subscriber.go` — inbound topic routing. +- `cmd/api/mqtt_products_push.go`, `internal/ingest/handler_products_push_resp.go` — product push and response paths. +- `internal/ingest/handler.go`, `internal/contracts/sales.go`, `internal/sales/` — sales reports and persistence. +- `internal/auth/`, `internal/devices/rbac.go`, `internal/devices/identity.go` — authorization and identity. + +The future delivered backend must be inspected again. Do not assume it will use the lab's package layout, topic names, or configuration system. Board identity (FN), machine identity (SN), and other identifiers must be understood from the delivered contracts rather than treated as interchangeable. + +## Course rules and working structure + +- Build one evolving application, adding only the structure needed by the current section. +- Start with `cmd/backend` and `cmd/simulator`; introduce API and ingestion executables in Section 15. +- Add `internal/config`, `internal/mqtt`, `internal/contracts`, and domain packages as they become useful. Keep interfaces small and driven by callers. +- Keep upstream code under `references/paho.golang/` unchanged. Read `autopaho/examples/basics` first; use `backoff`, `queue`, `router`, and `rpc` when their lessons arrive. Lower-level examples also live under `paho/cmd/`. +- Select a compatible tagged Paho release in Section 1 and pin it. The downloaded commit recorded in `references/README.md` is a reference snapshot, not an automatic release choice. Read API documentation and examples matching the selected version. +- Use TOML and `github.com/knadh/koanf/v2`, with `defaults < TOML < environment < CLI` and prefix `TEST_MQTT5_`. CLI defaults must not accidentally overwrite configured values when flags were not supplied. +- Use memory until Section 13. Use harmless `ping`, `get_status`, and simulated configuration changes throughout. +- For each exercise record the change, command, expected result, actual result, and your explanation in a learning journal. Keep payloads synthetic and omit secrets. +- Move on when the section's completion checks pass and you can explain the results without reading the implementation. + +## Section 1 — Prepare an isolated, reproducible lab + +Detailed lesson: [Section 1 instructions and examples](lessons/01-lab-setup.md). + +**Outcome:** A local MQTT 5 broker you can reach, plus a recorded toolchain and dependency choice. + +1. [ ] Check your Go and Docker installation. Record `go version`, `docker version`, and `docker compose version`. If Docker is unavailable, establish an equivalent isolated local Mosquitto setup before continuing. +2. [ ] Read the local Paho README and the basics example. Identify which responsibilities belong to AutoPaho and which belong to the underlying Paho client. Check a tagged release's Go requirement and release notes before pinning it. +3. [ ] Initialize a Go module in this learning project. Record the selected Go, Paho, and Mosquitto versions; avoid floating `latest` versions for reproducible exercises. +4. [ ] Create a Compose definition with a project-specific name and a broker listener published only on loopback, using an unused lab port such as `127.0.0.1:18883`. Use a separate persistence volume and configuration directory. +5. [ ] Disable anonymous access. Create distinct lab backend and simulator credentials and an initial ACL allowing only the lab topic namespace. Keep passwords and generated secrets out of version control. +6. [ ] Use Mosquitto CLI tools with MQTT version 5 explicitly selected to subscribe and publish a synthetic message. Check the installed tools' help for version and authentication flags. Repeat with incorrect credentials. + +**Initial commands:** `docker compose -p mqtt5-learning config`, then `docker compose -p mqtt5-learning up -d`, `docker compose -p mqtt5-learning ps`, and `docker compose -p mqtt5-learning logs --tail=50`. Run these after you have written the lab files; never use a production Compose directory. Do not share resolved Compose output if it contains secrets. + +**Done when:** A valid subscriber receives the exact test payload; invalid credentials are rejected; you can stop and restart only the lab broker. Record why MQTT 5 must be selected explicitly in your CLI check. + +**Local simplification:** Loopback-only plaintext is acceptable for this initial lab. Credentials are not encrypted on a plaintext connection. Section 11 adds verified TLS before any nonlocal deployment. + +## Section 2 — Write your first Go MQTT 5 connection + +Detailed lesson: [Section 2 connection lifecycle and code](lessons/02-go-connection.md). + +**Outcome:** A small executable that connects, reports connection state, and exits cleanly. + +1. [ ] Create `cmd/backend/main.go` and use AutoPaho with a unique, explicit client ID. Explain the MQTT 5 CONNECT/CONNACK exchange and inspect failure reasons exposed by the selected API. +2. [ ] Add a root context and signal cancellation. Bound initial connection waiting so a stopped broker does not leave startup hanging indefinitely. +3. [ ] Log connection up/down events without credentials. Distinguish application readiness from a running process. +4. [ ] Implement shutdown using the selected release's lifecycle API and wait for background activity to finish with a deadline. +5. [ ] Try wrong credentials, an unreachable port, and Ctrl+C during connection attempts. + +**Done when:** Each failure has a useful diagnosis; cancellation stops retries and exits promptly. Explain why starting the connection manager does not alone prove the broker is ready. + +**Read:** `references/paho.golang/autopaho/examples/basics/` and the selected release's connection-manager documentation. + +## Section 3 — Add typed configuration + +Detailed lesson: [Section 3 TOML and koanf implementation](lessons/03-typed-configuration.md). + +**Outcome:** Both executables can be configured predictably without editing Go code. + +1. [ ] Add typed TOML + koanf configuration for broker URL, client ID, credential inputs, timeouts, log level, and simulator identity. +2. [ ] Implement the required precedence. Define an unambiguous mapping from prefixed environment variables to nested keys. +3. [ ] Validate URLs, durations, required credentials, and conflicting settings before opening connections. +4. [ ] Write small table-driven tests for precedence and invalid configuration. Include the case where a CLI flag is absent. +5. [ ] Print a redacted startup summary and document a sample TOML file containing placeholders only. + +**Done when:** You can demonstrate every override layer, and bad settings fail before connecting. Explain why configuration errors and transient network failures need different handling. + +## Section 4 — Publish, subscribe, and simulate a device + +Detailed lesson: [Section 4 Paho transport, backend, and simulator](lessons/04-publish-subscribe-simulator.md). + +**Outcome:** Two Go processes exchange messages through the broker. + +1. [ ] Create a simulator with its own identity and client ID. Give the backend and simulator different topic permissions. +2. [ ] Define a lab namespace, for example `lab/v1/devices/{deviceID}/...`, with separate telemetry, commands, and responses topics. These are lab contracts, not changes to existing production topics. +3. [ ] Subscribe, inspect SUBACK results, and wait for subscription readiness before publishing a message that must be received. +4. [ ] Publish at QoS 0 and 1. Inspect errors and acknowledgements rather than assuming a nil transport error proves the device acted. +5. [ ] Send heartbeats from the simulator; receive a harmless command and return a simple response. + +**Done when:** You can trace CONNECT, SUBSCRIBE/SUBACK, PUBLISH, and QoS 1 PUBACK for both directions. Explain why a PUBACK is not a business response and why QoS is evaluated per delivery hop. + +## Section 5 — Design contracts and validate inbound messages + +Detailed lesson: [Section 5 instructions](lessons/05-contracts-validation.md). + +**Outcome:** A malformed or misleading message cannot corrupt application state. + +1. [ ] Define versioned JSON structures for heartbeat, command, command result, and synthetic sales report. Include device identity, stable message ID, and relevant timestamps. +2. [ ] Validate topic shape, topic/body identity agreement, payload size, schema version, required values, and timestamps. Decide explicitly how unknown fields and unsupported versions behave. +3. [ ] Separate topic parsing and decoding from business handlers. Introduce Content Type and Payload Format Indicator as metadata; still validate the actual payload. +4. [ ] Add table-driven tests for malformed JSON, identity mismatch, unknown commands, missing IDs, and boundary sizes. +5. [ ] Return safe application errors without echoing confidential content. + +**Done when:** Valid messages route correctly; invalid messages have consistent outcomes. Explain why a body field claiming a device ID does not authenticate the sender. + +## Section 6 — Manage callbacks, concurrency, and cancellation + +Detailed lesson: [Section 6 instructions](lessons/06-concurrency-cancellation.md). + +**Outcome:** Message handling remains responsive under concurrent traffic. + +1. [ ] Inspect the selected Paho release's callback execution and acknowledgement behavior. Document which work happens before and after callback return. +2. [ ] Add bounded work queues and a small worker pool. Define an overload policy separately for disposable telemetry and important commands/reports. +3. [ ] Protect shared state with a deliberate ownership or locking model. Copy message data before retaining it if required by the API's ownership contract. +4. [ ] Propagate lifecycle contexts and operation deadlines; avoid unbounded goroutines and per-message background contexts. +5. [ ] Add a slow handler and a burst generator; observe queue saturation and shutdown while work is pending. + +**Done when:** The learning module passes `go test -race ./...`, and overload behavior is observable and bounded. Explain the possible loss window if MQTT acknowledges a message before queued application work is durable. + +## Section 7 — Build reliable command/response behavior + +Detailed lesson: [Section 7 explanations and experiments](lessons/07-command-response.md). + +**Outcome:** Concurrent commands are matched to the correct device responses. + +1. [ ] Add MQTT 5 Response Topic and Correlation Data. Subscribe to authorized response topics before sending requests and register pending requests before publishing. +2. [ ] Define command states and legal transitions: created, pending delivery, accepted, completed, rejected, and response timeout/unknown outcome. Track broker publish acknowledgement separately. +3. [ ] Match responses using command identity, expected device, and correlation. Handle malformed, duplicate, late, and out-of-order responses without regressing terminal state. +4. [ ] Use deadlines and remove expired in-memory waiters. A timeout must not imply the device definitely did nothing. +5. [ ] Make simulator effects idempotent by stable command ID; replay the recorded result on a repeated command. Reject unexpected response destinations. + +**Done when:** Many concurrent commands receive the right results; duplicate commands produce one simulated effect; late responses follow a documented policy. Explain why MQTT Correlation Data alone supplies neither authorization nor deduplication. + +**Read:** AutoPaho's `rpc` example as a mechanism reference, then compare its scope with your application state machine. + +## Section 8 — Expose device commands through HTTP + +Detailed lesson: [Section 8 instructions](lessons/08-http-commands.md). + +**Outcome:** An API caller can create a command and query its status. + +1. [ ] Use `net/http` for `POST /devices/{id}/commands` and `GET /commands/{id}`. Return a command ID and asynchronous acceptance when appropriate. +2. [ ] Separate HTTP decoding, command service, and MQTT transport. Keep command processing independent of a disconnected HTTP caller after acceptance. +3. [ ] Add request-body limits, JSON error responses, server timeouts, and controlled shutdown. +4. [ ] Distinguish HTTP request idempotency keys from device command IDs and MQTT packet identifiers. Test a retried HTTP request. +5. [ ] Use `httptest` and a small fake transport to test invalid input, unavailable transport, and command-status responses. + +**Done when:** An HTTP command reaches the simulator and status reflects device evidence. A closed HTTP connection does not silently cancel an already accepted command. + +## Section 9 — Learn MQTT 5 sessions through failure experiments + +Detailed lesson: [Section 9 instructions](lessons/09-sessions-recovery.md). + +**Outcome:** You can predict and explain reconnect and offline delivery behavior. + +1. [ ] Compare stable and changing client IDs, Clean Start choices, zero/nonzero Session Expiry Interval, and the broker's Session Present result. +2. [ ] Disconnect the simulator, publish a QoS 1 command, and reconnect before and after expiry. Record the subscription and session prerequisites for queued delivery. +3. [ ] Test subscription restoration when a session exists and when it does not. Observe reconnect callbacks and avoid accidental repeated local handler registration. +4. [ ] Repeat with broker restart, first with persistence disabled and then configured. Record the limits of the actual broker persistence behavior. +5. [ ] Compare client session storage, the outgoing publication queue, broker session state, and your application state in a four-column recovery table. + +**Done when:** Each experiment has a predicted and observed result. Explain what survives network loss versus backend, simulator, or broker process loss. + +## Section 10 — Handle expiry, presence, and desired state + +Detailed lesson: [Section 10 explanations and experiments](lessons/10-expiry-presence-state.md). + +**Outcome:** Stale work and stale device status are handled deliberately. + +1. [ ] Add Message Expiry Interval to time-sensitive commands and an application deadline checked by the simulator. Account for time spent in local queues and clock assumptions. +2. [ ] Compare retained desired configuration with nonretained one-shot commands. Clear a retained message deliberately in the lab and observe the result. +3. [ ] Add heartbeat freshness and a Last Will. Experiment with Will Delay, graceful disconnect, abrupt termination, and reconnect; explain their interaction with sessions. +4. [ ] Introduce desired and reported configuration revisions. Detect a stale report without replacing newer state. + +**Done when:** An expired queued command produces no effect; late telemetry does not make an offline device appear reliably healthy. Explain why Last Will and heartbeat freshness provide different evidence. + +## Section 11 — Secure device communication and API access + +Detailed lesson: [Section 11 explanations and experiments](lessons/11-security-boundaries.md). + +**Outcome:** Unauthorized clients cannot control another device or observe its private traffic. + +1. [ ] Add a lab CA and server certificate with the correct host identity. Verify the certificate chain and hostname; never set `InsecureSkipVerify: true`. +2. [ ] Refine backend/device ACLs. Bind each device credential to its own namespace and restrict response-topic destinations. Test wildcard subscriptions and cross-device publication attempts. +3. [ ] Add API authentication and per-device authorization, informed by the existing JWT and permission code. Use established libraries; validate relevant token claims and expiry. +4. [ ] Test a valid user with insufficient device permissions, expired credentials, a wrong CA, and a hostname mismatch. +5. [ ] Explain credential rotation, optional mTLS, redaction, and why broker identity and HTTP user identity are different trust boundaries. + +**Done when:** Positive and negative access tests behave as designed. Record the observed reason/error for denied operations; do not assume every broker denial is reported identically. + +## Section 12 — Implement representative vending workflows + +**Outcome:** You understand the backend's main device-facing jobs using synthetic data. + +1. [ ] Add synthetic sales reports and application responses keyed by report ID. Deduplicate in memory and identify the restart limitation. +2. [ ] Push a small product catalog with contract version, message ID, job ID, and revision. Have the simulator report which revision it applied. +3. [ ] Implement a device-initiated configuration fetch and a backend-initiated simulated update. Define conflict and stale-revision policies. +4. [ ] Ingest health and bounded log events with basic sanitization. Distinguish transport connectivity, application responsiveness, and device health. +5. [ ] Trace equivalent source paths in the current backend and write a short comparison of its contracts with your learning contracts. + +**Done when:** You demonstrate each flow and explain its identity, acknowledgement, retry, and deduplication rules. Keep remote delivery, real payments, OTP/SMS, and firmware changes simulated or read-only study topics. + +## Section 13 — Persist reports and command state in PostgreSQL + +**Outcome:** Important state and deduplication survive application restart. + +1. [ ] Add a dedicated lab PostgreSQL instance and migrations for devices, commands, command results, and sales reports. Use `database/sql` with pgx stdlib to match the observed connector style. +2. [ ] Move important in-memory repositories behind small interfaces. Use contexts, query deadlines, bounded connection pools, and transaction boundaries. +3. [ ] Enforce uniqueness for device/report IDs and command IDs. Persist a deduplication decision and its corresponding database effect atomically. +4. [ ] Define precisely when to send an application-level saved/accepted response. Investigate manual MQTT acknowledgement support before choosing an acknowledgement policy; never assume callback return proves persistence. +5. [ ] Use isolated DB integration tests to simulate duplicates, rollback, database outage, and a crash after commit but before the response reaches the device. + +**Done when:** Replaying a report after restart does not double-count it, and the simulator can safely retry a missing response. Explain why MQTT QoS 2 does not make database effects or physical actions automatically exactly once. + +## Section 14 — Close database/MQTT failure gaps + +**Outcome:** Accepted work can be recovered across failures between storage and publication. + +1. [ ] Draw the failure windows between saving a command, publishing it, receiving PUBACK, and recording the device result. +2. [ ] Implement a small transactional outbox: save the command and intent to publish in one transaction, then publish from a worker with bounded retries and expiry. +3. [ ] Reuse the same command ID across retries. Model possible duplicate publication when the process crashes after publish but before marking the outbox entry sent. +4. [ ] Persist result processing and idempotency records. Separate retryable failures, permanent rejection, expiry, and unknown outcome. +5. [ ] Inject each crash window. Keep synthetic device effects and their deduplication state recoverable for the restart experiment; discuss why physical effects require device-specific reconciliation. + +**Done when:** Accepted durable work is eventually processed or reaches an explainable terminal/unknown state. Explain the separate purposes of an outbox, a Paho outgoing queue, and MQTT session storage. + +## Section 15 — Split API and ingestion processes + +**Outcome:** Your application resembles the backend's separate process responsibilities. + +1. [ ] Extract `cmd/api` and `cmd/ingest` with separate lifecycle contexts and client IDs. Decide explicitly which process or worker owns publication. +2. [ ] Let the API create durable commands and query status while ingestion validates and records device responses. +3. [ ] Eliminate reliance on a pending-request map shared across processes. Use persisted status as the baseline; add notification only if it has a demonstrated purpose. +4. [ ] Stop and restart either process independently. Verify that the surviving process behaves predictably and shutdown preserves recoverable work. +5. [ ] Document dependency readiness, ownership of subscriptions, and startup order without relying on arbitrary sleeps. + +**Done when:** HTTP status remains correct when responses arrive in another process, and independent restarts do not lose durable command tracking. + +## Section 16 — Explore advanced MQTT 5 controls and scaling + +Detailed lesson: [Section 16 explanations and experiments](lessons/16-mqtt5-controls.md). + +**Outcome:** You can choose MQTT features based on measured behavior. + +1. [ ] Experiment with User Properties and subscription options: No Local, Retain As Published, and Retain Handling. Record the relevant packets and observed behavior. +2. [ ] Use shared subscriptions with two ingestion instances. Observe distribution and failure recovery; do not assume per-device ordering or instance affinity. +3. [ ] Test Receive Maximum and maximum packet-size negotiation in both directions. Distinguish protocol inflight limits from application queue capacity and QoS 0 traffic. +4. [ ] Study topic aliases and their connection-scoped state. Verify selected library/broker support before using them as an optimization. +5. [ ] Review negative acknowledgements, negotiated capabilities, reconnect backoff/jitter, and unavailable or unsupported features. Confirm that shared consumers coordinate idempotency through durable state. + +**Done when:** You have a feature matrix with purpose, support/version, experiment, result, and decision. You can explain which limits belong to MQTT, the broker, the client library, and your application. + +## Section 17 — Operate and diagnose the service + +**Outcome:** You can identify whether a failure is in HTTP, the application, broker, device, or database. + +1. [ ] Add structured logs with stable command/report IDs and safe error categories. Add metrics for reconnects, rejected messages, queue depth, duplicate handling, publish latency, response latency, timeouts, and DB failures. +2. [ ] Define liveness and readiness for each process. Avoid high-cardinality device/command IDs as unrestricted metric labels. +3. [ ] Create a repeatable simulator load scenario and measure throughput, latency, memory, and queue growth. Profile only after observing a bottleneck. +4. [ ] Run a failure matrix: broker down/restart, network loss, simulator offline/restart, expired commands, slow DB, duplicate reports, denied topics, and shutdown under load. +5. [ ] Write a local runbook: symptom, evidence to collect, likely layer, recovery action, and how to verify recovery. Include draining/recovery behavior for rolling restarts. + +**Done when:** Another person could reproduce and diagnose a failed command using your runbook without reading all your code. Relevant Go tests and race checks pass for the learning application. + +## Section 18 — Prepare for the MQTT 5 backend handover + +**Outcome:** You can inspect, explain, and safely maintain the delivered system. + +1. [ ] Reinspect the delivered commit, dependencies, broker configuration, device protocol contracts, and deployment topology. Build an explicit MQTT 3.1.1-to-5 behavior comparison rather than an import-replacement checklist. +2. [ ] Trace one HTTP command to publication and device response, one sale to persistence and application response, and one catalog/configuration synchronization flow. +3. [ ] Document device identities and permissions, topic ownership, payload versions, command states, retry rules, expiry, persistence, and process boundaries. Resolve FN/SN/plate terminology from authoritative contracts. +4. [ ] Run authorized staging acceptance checks with simulators. Check mixed MQTT 3.1.1/5 interoperability and feature loss where relevant; do not assume legacy devices receive or understand MQTT 5 properties. +5. [ ] Review release notes and compatibility before upgrading the client. Propose one narrow maintenance change with reproduction, expected behavior, regression coverage, rollout checks, and rollback criteria. +6. [ ] Produce a handover question list for facts the code cannot prove: device acknowledgement semantics, broker persistence policy, recovery targets, ownership, credentials rotation, and operational escalation. + +**Done when:** You can explain each traced flow, diagnose an injected failure, and justify a small fix with evidence. Production migration or rollout is separate work requiring its own scope. + +## Completion checkpoints + +| After section | Demonstration | +| --- | --- | +| 4 | Go backend and simulated device exchange MQTT 5 messages | +| 8 | HTTP command reaches the device and exposes an accurate status | +| 12 | Secured lab demonstrates sales, catalog, configuration, and health flows | +| 15 | Persistent API/ingestion processes recover independently | +| 18 | You can trace, test, diagnose, and review the delivered backend | + +## How to start a lesson with the tutor + +> Read `prompt.md` and `COURSE.md`. Start Section 1, exercise 1. I know MQTT 3.1.1 concepts. Explain only what I need for this exercise, give me a small task and expected result, then wait for my attempt. Review my result before moving on. Do not generate the complete application. + +For later sessions, replace the section and exercise number and include your current code, command output, or question. The tutor should ask about Go experience only where it affects the current exercise; Docker availability is established in Section 1. + +## Official references + +Consult version-matched API documentation during each lesson. The upstream Paho documentation recommends AutoPaho as the starting point and explains its relationship to the lower-level client; Mosquitto's documentation links the broker configuration and CLI manuals. + +- [Paho repository and overview](https://github.com/eclipse-paho/paho.golang) +- [AutoPaho API](https://pkg.go.dev/github.com/eclipse/paho.golang/autopaho) +- [Paho API](https://pkg.go.dev/github.com/eclipse/paho.golang/paho) +- [MQTT 5 specification](https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html) +- [Mosquitto documentation and manuals](https://mosquitto.org/documentation/) + +The course intentionally leaves exact version-sensitive method signatures to each lesson's verification step. Local source references are available without changing the downloaded checkout. diff --git a/cmd/main.go b/cmd/main.go new file mode 100644 index 0000000..6db2604 --- /dev/null +++ b/cmd/main.go @@ -0,0 +1,93 @@ +package main + +import ( + "context" + "fmt" + "log" + "net/url" + "os" + "os/signal" + "syscall" + "time" + + "github.com/eclipse/paho.golang/autopaho" + "github.com/eclipse/paho.golang/paho" +) + +func main() { + if err := run(); err != nil { + log.Fatal(err) + } +} + +func run() error { + ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM) + defer stop() + + user := os.Getenv("TEST_MQTT5_MQTT_USERNAME") + pass := os.Getenv("TEST_MQTT5_MQTT_PASSWORD") + + if user == "" || pass == "" { + return fmt.Errorf("TEST_MQTT5_MQTT_USERNAME and TEST_MQTT5_MQTT_PASSWORD environment variables not set") + } + + broker, err := url.Parse("mqtt://127.0.0.1:18883") + if err != nil { + return fmt.Errorf("failed to parse broker url: %w", err) + } + + cfg := autopaho.ClientConfig{ + ServerUrls: []*url.URL{broker}, + KeepAlive: 20, + CleanStartOnInitialConnection: true, + SessionExpiryInterval: 0, + ConnectUsername: user, + ConnectPassword: []byte(pass), + OnConnectionUp: func(_ *autopaho.ConnectionManager, ack *paho.Connack) { + log.Printf("connected: session_present=%t reason=0x%02x", ack.SessionPresent, ack.ReasonCode) + }, + OnConnectionDown: func() bool { + log.Print("connection down; AutoPaho will retry") + return true + }, + OnConnectError: func(err error) { + log.Printf("connect attempt failed: %v", err) + }, + ClientConfig: paho.ClientConfig{ + ClientID: "lab-backend", + OnClientError: func(err error) { + log.Printf("client error: %v", err) + }, + OnServerDisconnect: func(packet *paho.Disconnect) { + log.Printf("server disconnected: reason=0x%02x", packet.ReasonCode) + }, + }, + } + + manager, err := autopaho.NewConnection(ctx, cfg) + if err != nil { + return fmt.Errorf("failed to create connection manager: %w", err) + } + + defer func() { + shutdownCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second) + defer cancel() + if err := manager.Disconnect(shutdownCtx); err != nil { + log.Printf("shutdown did not finish within deadline: %v", err) + } + }() + + startup, cancel := context.WithTimeout(ctx, 10*time.Second) + err = manager.AwaitConnection(startup) + cancel() + + if err != nil { + return fmt.Errorf("failed to connect to broker: %w", err) + } + + log.Print("connected to broker; pres Ctrl+C to stop") + + <-ctx.Done() + + return nil +} diff --git a/compose.yml b/compose.yml new file mode 100644 index 0000000..eca9d98 --- /dev/null +++ b/compose.yml @@ -0,0 +1,15 @@ +services: + broker: + image: eclipse-mosquitto:2.0.22 + ports: + - "127.0.0.1:18883:1883" + volumes: + - ./lab/mosquitto/mosquitto.conf:/mosquitto/config/mosquitto.conf:ro + - ./lab/mosquitto/acl:/mosquitto/config/acl:ro + - broker-secrets:/mosquitto/secrets:ro + - broker-data:/mosquitto/data +volumes: + broker-data: + broker-secrets: + external: true + name: mqtt5-learning_broker-secrets diff --git a/go.mod b/go.mod new file mode 100644 index 0000000..27aeada --- /dev/null +++ b/go.mod @@ -0,0 +1,10 @@ +module example.com/mqtt5-learning + +go 1.27.1 + +require github.com/eclipse/paho.golang v0.23.0 + +require ( + github.com/gorilla/websocket v1.5.3 // indirect + golang.org/x/net v0.43.0 // indirect +) diff --git a/go.sum b/go.sum new file mode 100644 index 0000000..ec10061 --- /dev/null +++ b/go.sum @@ -0,0 +1,18 @@ +github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c= +github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38= +github.com/eclipse/paho.golang v0.23.0 h1:KHgl2wz6EJo7cMBmkuhpt7C576vP+kpPv7jjvSyR6Mk= +github.com/eclipse/paho.golang v0.23.0/go.mod h1:nQRhTkoZv8EAiNs5UU0/WdQIx2NrnWUpL9nsGJTQN04= +github.com/google/go-cmp v0.7.0 h1:wk8382ETsv4JYUZwIsn6YpYiWiBsYLSJiTsyBybVuN8= +github.com/google/go-cmp v0.7.0/go.mod h1:pXiqmnSA92OHEEa9HXL2W4E7lf9JzCmGVUdgjX3N/iU= +github.com/gorilla/websocket v1.5.3 h1:saDtZ6Pbx/0u+bgYQ3q96pZgCzfhKXGPqt7kZ72aNNg= +github.com/gorilla/websocket v1.5.3/go.mod h1:YR8l580nyteQvAITg2hZ9XVh4b55+EU/adAjf1fMHhE= +github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM= +github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4= +github.com/stretchr/testify v1.11.1 h1:7s2iGBzp5EwR7/aIZr8ao5+dra3wiQyKjjFuvgVKu7U= +github.com/stretchr/testify v1.11.1/go.mod h1:wZwfW3scLgRK+23gO65QZefKpKQRnfz6sD981Nm4B6U= +go.uber.org/goleak v1.3.0 h1:2K3zAYmnTNqV73imy9J1T3WC+gmCePx2hEGkimedGto= +go.uber.org/goleak v1.3.0/go.mod h1:CoHD4mav9JJNrW/WLlf7HGZPjdw8EucARQHekz1X6bE= +golang.org/x/net v0.43.0 h1:lat02VYK2j4aLzMzecihNvTlJNQUq316m2Mr9rnM6YE= +golang.org/x/net v0.43.0/go.mod h1:vhO1fvI4dGsIjh73sWfUVjj3N7CA9WkKJNQm2svM6Jg= +gopkg.in/yaml.v3 v3.0.1 h1:fxVm/GzAzEWqLHuvctI91KS9hhNmmWOoWu0XTYJS7CA= +gopkg.in/yaml.v3 v3.0.1/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM= diff --git a/lab/mosquitto/acl b/lab/mosquitto/acl new file mode 100644 index 0000000..08f5527 --- /dev/null +++ b/lab/mosquitto/acl @@ -0,0 +1,9 @@ +user backend +topic write lab/v1/devices/+/commands +topic read lab/v1/devices/+/telemetry +topic read lab/v1/devices/+/responses + +user simulator +topic read lab/v1/devices/device-001/commands +topic write lab/v1/devices/device-001/telemetry +topic write lab/v1/devices/device-001/responses diff --git a/lab/mosquitto/mosquitto.conf b/lab/mosquitto/mosquitto.conf new file mode 100644 index 0000000..434d9ef --- /dev/null +++ b/lab/mosquitto/mosquitto.conf @@ -0,0 +1,8 @@ +listener 1883 +allow_anonymous false +password_file /mosquitto/secrets/passwords +acl_file /mosquitto/config/acl +persistence true +persistence_location /mosquitto/data/ +log_dest stdout +connection_messages true \ No newline at end of file diff --git a/lessons/01-lab-setup.md b/lessons/01-lab-setup.md new file mode 100644 index 0000000..50fa190 --- /dev/null +++ b/lessons/01-lab-setup.md @@ -0,0 +1,355 @@ +# Section 1 — Prepare the MQTT 5 lab + +Read this first. You will create the files and run the commands yourself. All paths below are relative to the learning project root. The examples use the module path `example.com/mqtt5-learning`; if you change it, change the imports in later lessons too. + +**Version baseline:** Paho `v0.23.0`, verified from upstream release tags on 2026-10-07. It requires Go 1.24 or newer; Go 1.25 is compatible. The matching local reference is `references/paho.golang-v0.23.0/` (commit `1aa03969c5932464359b3e4ed752d9ac61abccc9`). + + +## Working directory and file placement + +Run every shell command in this lesson from the learning project root: + +```sh +cd /home/ryan/Media/Code/Code/robot-market/mqtt-5-learning +``` + +Paths such as `cmd/backend/main.go` are relative to this directory. Markdown files under `lessons/` explain the work; create the application files at the paths shown, not inside `lessons/` or `references/`. Code blocks labeled Go, TOML, YAML, or conf are file contents; shell blocks are terminal commands. Create parent directories before saving files. + +## 1. Check the tools and create the module + +```sh +go version +docker version +docker compose version +go mod init example.com/mqtt5-learning +go get github.com/eclipse/paho.golang@v0.23.0 +``` + +`go version` reports the installed compiler. `docker version` reports both CLI and daemon versions: a missing server response means the CLI cannot reach the Docker engine. `docker compose version` checks the Compose plugin, which reads `compose.yaml` and manages the application's services. Having Docker installed does not prove the daemon is running. + +`go mod init example.com/mqtt5-learning` creates `go.mod`, declaring this application's import-path prefix. It does not create the source directories. `go get ...@v0.23.0` downloads a selected dependency and records module requirements and checksums in `go.mod`/`go.sum`; it may also update transitive requirements needed by that dependency. + +Run `go mod init` only once. `go get` pins the dependency in `go.mod`; commit `go.mod` and `go.sum` when you initialize version control. Do not use a local `replace` directive to substitute the newer default-branch reference. A repository checkout is reading material, while a module requirement selects the compiled library. + +The old backend's `github.com/eclipse/paho.mqtt.golang` is a different MQTT 3 client. MQTT 5 uses `github.com/eclipse/paho.golang/autopaho` and `/paho`. There is no `SetProtocolVersion(5)` needed in these MQTT 5 examples. + +## 2. Create the broker files + +Create these directories: + +```sh +mkdir -p lab/mosquitto +mkdir -p lab/secrets +``` + +`mkdir` creates directories; `-p` creates missing parents and tolerates an already-existing directory. These commands do not create configuration files. + +Create `compose.yaml` directly in the project root, next to `go.mod`. Its full path is `/home/ryan/Media/Code/Code/robot-market/mqtt-5-learning/compose.yaml`. The `./lab/...` volume paths below are relative to this Compose file: + +```text +mqtt-5-learning/ +├── go.mod +├── compose.yaml +└── lab/ + ├── mosquitto/ + │ ├── mosquitto.conf + │ └── acl + └── secrets/ + └── passwords # generated in step 3, not written by hand +``` + +Your existing `compose.yml` is also a supported Compose filename; keep it rather than creating a competing `compose.yaml`. Use one primary Compose file consistently. YAML nesting is controlled by spaces: the final `volumes:` declaration must align with `services:`, not with `broker:`. If nested under `services`, Compose treats `volumes` as another service and reports `services.volumes additional properties 'broker-data' not allowed`. + +Paste this YAML into your primary Compose file: + +```yaml +services: + broker: + image: eclipse-mosquitto:2.0.22 + ports: + - "127.0.0.1:18883:1883" + volumes: + - ./lab/mosquitto/mosquitto.conf:/mosquitto/config/mosquitto.conf:ro + - ./lab/mosquitto/acl:/mosquitto/config/acl:ro + - ./lab/secrets:/mosquitto/secrets:ro + - broker-data:/mosquitto/data +volumes: + broker-data: +``` + +This is a pinned Mosquitto 2 lab image, not a claim that it is the newest broker release. Host port 18883 maps to container port 1883. Containers use `broker:1883`; Go programs on your host use `127.0.0.1:18883`. Loopback binding keeps this listener off your LAN. Persistence belongs to this Compose project only. + +### Understand the Compose definition + +`services` lists the containers in this application; `broker` is the service name you use in commands and the DNS name on this Compose network. `image` selects a prebuilt image; Docker downloads it if needed. There is no `build` section because this lesson does not build a broker image. + +`ports` uses `host-address:host-port:container-port`. `127.0.0.1:18883:1883` forwards connections from this computer's loopback port 18883 to the container's listener at 1883. A host client cannot use container port 1883 unless that port is separately published. `broker:1883` works for containers on the same Compose network, not arbitrary host programs or remote devices. + +Each `volumes` entry is `source:container-destination[:mode]`. Sources beginning with `./` are host files/directories (bind mounts); `:ro` makes them read-only to this container. Create the configuration files before starting Docker: missing file paths can result in directories being mounted where files were expected. `broker-data` is a Docker-managed named volume, declared at the bottom, for writable broker persistence. It is separate from the image and survives ordinary container replacement. + +Create `lab/mosquitto/mosquitto.conf`: + +```conf +listener 1883 +allow_anonymous false +password_file /mosquitto/secrets/passwords +acl_file /mosquitto/config/acl +persistence true +persistence_location /mosquitto/data/ +log_dest stdout +connection_messages true +``` + +### Understand each broker setting + +| Setting | Purpose and operational consequence | +| --- | --- | +| `listener 1883` | Opens the MQTT listener inside the container; host exposure is controlled separately by Compose | +| `allow_anonymous false` | Requires authentication; it does not grant topic permissions | +| `password_file ...` | Reads the generated username/password hashes at this container path | +| `acl_file ...` | Reads topic permissions; a correct password alone does not authorize every topic | +| `persistence true` | Enables saving broker state, including eligible retained/session information; it does not persist Go application state | +| `persistence_location ...` | Places saved broker data in the named volume rather than the disposable container filesystem | +| `log_dest stdout` | Sends broker logs to Docker's log stream for `compose logs` | +| `connection_messages true` | Logs client connection/disconnection events for troubleshooting | + +Persistence is not a database transaction for your application. Recently changed broker state may not survive an abrupt crash until it has been saved. Session settings and message QoS still decide which offline data is eligible to survive. + +Create `lab/mosquitto/acl`: + +```conf +user backend +topic write lab/v1/devices/+/commands +topic read lab/v1/devices/+/telemetry +topic read lab/v1/devices/+/responses + +user simulator +topic read lab/v1/devices/device-001/commands +topic write lab/v1/devices/device-001/telemetry +topic write lab/v1/devices/device-001/responses +``` + +`user` starts rules for that authenticated username. `topic write` permits publishing; `topic read` permits receiving matching topics. `+` matches exactly one topic level, so the backend rules cover one device-ID segment. The simulator's rules use an exact device ID. Access outside the allowed patterns is denied by this simple ACL configuration. The text file has no extension because Mosquitto reads the path you configure, regardless of its filename suffix. + +The simulator can act only as `device-001`. The backend can operate all lab devices. ACL checks authorize topics; they do not validate JSON or prove that a payload's identity is truthful. The CLI checks below must use a direction permitted for that username. + +Add to `.gitignore` (preserve any existing rules): + +```gitignore +/lab/secrets/ +/.env +``` + +Git ignore patterns are file contents, not terminal commands. `/lab/secrets/` ignores the root-level directory and its contents; `/.env` ignores a root environment file. Ignoring a file does not untrack a secret already committed. Keep real secrets out of the initial commit as well as later changes. + +## 3. Create passwords interactively + +These commands create a lab password file without putting the password literal in your shell history. Run them from the project root: + +```sh +docker run --rm -it --user 0 -v "$PWD/lab/secrets:/work" eclipse-mosquitto:2.0.22 mosquitto_passwd -c /work/passwords backend +docker run --rm -it --user 0 -v "$PWD/lab/secrets:/work" eclipse-mosquitto:2.0.22 mosquitto_passwd /work/passwords simulator +docker run --rm --user 0 -v "$PWD/lab/secrets:/work" eclipse-mosquitto:2.0.22 chown mosquitto:mosquitto /work/passwords +docker run --rm --user 0 -v "$PWD/lab/secrets:/work" eclipse-mosquitto:2.0.22 chmod 600 /work/passwords +``` + +`-c` creates/replaces the file. Use it only for the first username; using it again would erase the backend entry. The last two commands make the file readable by the broker account in this image. Container-root use is limited to setting up this mounted lab file. + +### Why four Docker commands? + +There are two jobs here: generate password hashes for two users, then give the broker permission to read the resulting file. Docker adds a wrapper because we use the tool bundled with the pinned broker image instead of installing Mosquitto utilities on your computer. Each command runs a temporary utility container; none starts the broker. + +| Part of the command | What it does | +| --- | --- | +| `docker run` | Creates and runs a new container for this one operation | +| `--rm` | Removes that container when its command exits; the mounted host files remain | +| `-it` | Keeps standard input open and supplies a terminal for the password prompt | +| `--user 0` | Runs this setup operation as container root, permitting file creation and ownership changes on the mount | +| `-v "$PWD/lab/secrets:/work"` | Makes the host project's `lab/secrets` directory available at `/work` inside the container; this mount is writable | +| `eclipse-mosquitto:2.0.22` | Selects the same pinned image used by the broker | +| `mosquitto_passwd -c /work/passwords backend` | Creates the password file and prompts for the backend user's password | +| `mosquitto_passwd /work/passwords simulator` | Adds the simulator user to that same file, without replacing the backend entry | +| `chown mosquitto:mosquitto /work/passwords` | Assigns file ownership to the broker account as defined inside this image | +| `chmod 600 /work/passwords` | Gives the owner read/write access and removes group/other access | + +`$PWD` is your current terminal directory, so run these from the project root. `/work/passwords` in the setup container and `lab/secrets/passwords` on your host are the same file. Later, Compose mounts that directory at `/mosquitto/secrets` with `:ro`, and `mosquitto.conf` reads `/mosquitto/secrets/passwords`. The setup mount is writable; the running broker mount is read-only. + +The generated file stores password hashes, not the plaintext passwords you type. Your backend and simulator still need their original passwords to connect; a client cannot use the stored hash as its login password. Keep the generated file out of Git. The ACL file separately determines which topics each authenticated username may use. + +Linux checks numeric user/group IDs. `chown` inside this image picks the broker's IDs; your host may display a numeric owner or a different name for those same IDs. With mode `600`, your regular host user may be unable to read the file afterward. That is expected: the broker must read it, and subsequent password maintenance can use the same temporary root utility container. Do not make the file world-readable to work around this. + +### What is usual practice? + +For a small Mosquitto 2.0 installation, generating a password file with `mosquitto_passwd`, configuring `password_file`, and restricting file access is a conventional approach. The official [authentication guide](https://mosquitto.org/documentation/authentication-methods/) documents creating the first user with `-c` and adding or updating users without it. Docker is simply how this lesson obtains that utility. + +- **Local Docker lab:** Generate the file once with a temporary container, mount it into the broker, and repeat the update command only when adding users or rotating passwords. You do not rerun all four commands on each broker start. After modifying the file, reapply the ownership and mode if the utility replaced it, then reload authentication or restart only the lab broker. +- **Host-installed Mosquitto:** Run the installed `mosquitto_passwd` utility directly, then set permissions for the broker's service account. The core operations are the same; there is no container wrapper. +- **Repeated deployment:** Put the provisioning steps in a small reviewed setup script or dedicated setup job. Supply credentials through the deployment's secret-management mechanism and mount the resulting authentication file read-only. A secret mount still needs the correct broker-readable ownership and permissions. Avoid embedding passwords in image layers, committed Compose files, or command arguments. +- **Larger device fleets:** Evaluate managed authentication and authorization, such as Mosquitto's [Dynamic Security plugin](https://mosquitto.org/documentation/dynamic-security/), rather than maintaining a large static file manually. This is a deployment choice, not something needed for this two-user exercise. + +The four explicit commands make the setup visible for learning. Once you understand them, wrapping them in a setup script is reasonable. We keep interactive prompts here so password literals do not appear in shell history or process arguments. + +### Verify password setup before starting + +Do not start the broker until both password prompts and both permission commands have completed successfully. The two accounts must exist in the generated file; creating an empty file does not create usable credentials. + +Verify existence and access without printing hashes: + +```sh +docker run --rm --user mosquitto -v "$PWD/lab/secrets:/work:ro" eclipse-mosquitto:2.0.22 sh -c 'test -s /work/passwords && test -r /work/passwords' +``` + +`--user mosquitto` tests access as the image's broker account. `sh -c` runs the quoted checks inside the utility container; `test -s` requires a nonempty file, `test -r` requires read access, and `&&` runs the second check only if the first succeeds. Success is quiet with exit status zero. Failure means you must fix the file/path/permissions before starting. This checks file access, not whether the typed client password is correct. + +### If permission changes do not stick on the host filesystem + +Verify final metadata with `stat -c '%a %u:%g %n' lab/secrets/passwords`. `stat` reports mode, numeric owner/group, and pathname without exposing hashes. Also use `findmnt -T lab/secrets/passwords -o TARGET,FSTYPE,OPTIONS` to identify the underlying mount. Some mounted filesystems synthesize permissions: `chmod`/`chown` can appear successful while the metadata remains unchanged. Do not conclude the file is protected from the commands' exit status alone. + +This workspace exhibited that behavior on its `fuseblk` mount: the password file remained `755` and `1000:1000`. The current `compose.yml` therefore uses a separately provisioned Docker volume for credentials. To reproduce that fallback after generating the users: + +```sh +docker run --rm --user 0 -v "$PWD/lab/secrets:/source:ro" -v mqtt5-learning_broker-secrets:/target eclipse-mosquitto:2.0.22 sh -c 'cp /source/passwords /target/passwords && chown mosquitto:mosquitto /target/passwords && chmod 600 /target/passwords && stat -c "%a %u:%g" /target/passwords' +``` + +The host source is read-only. The new Docker-managed target is writable; `cp` copies the existing file and the following commands apply permissions there. This command replaces the target password file, so use it only for initial provisioning or an intentional replacement. The verified result in this image is `600 1883:1883`; always inspect the actual result for your environment. A named volume is not automatically secure or encrypted: its actual filesystem and file permissions still matter. + +Replace the service's `./lab/secrets:/mosquitto/secrets:ro` mount with `broker-secrets:/mosquitto/secrets:ro`. Declare this additional volume at the root, alongside `broker-data`: + +```yaml +volumes: + broker-data: + broker-secrets: + external: true + name: mqtt5-learning_broker-secrets +``` + +`name` specifies the already-provisioned Docker volume's exact name. `external: true` tells Compose to use it rather than create/delete it. The preceding utility run creates it if missing. Apply the changed mount with `up -d broker` and inspect recent logs. Verify access with the earlier read-check command but replace its host mount with `-v mqtt5-learning_broker-secrets:/work:ro`. + +For every subsequent add-user, rotation, or permission command, mount `mqtt5-learning_broker-secrets:/work` rather than `$PWD/lab/secrets:/work`: the broker now reads the volume copy. Editing the old host file does not update the live credential store. Include this external volume in backups; Compose `down -v` does not delete external volumes. The old host copy remains sensitive and ignored by Git; it has been preserved rather than automatically deleted. + +### Understand password-tool and entrypoint warnings + +During creation or update, `mosquitto_passwd` can warn about world-readable permissions or ownership. Do not assume these warnings prove the final file is correct: run the ownership/mode commands afterward and verify access. For this pinned Mosquitto 2.0.22 image, use `mosquitto:mosquitto` ownership and mode `600`. A password file needs no executable permission. Blindly following a warning suggesting root ownership with owner-only access can prevent the unprivileged broker from reading it. Review requirements again when upgrading rather than treating a future-version warning as this version's deployment instructions. + +The image entrypoint can also print `chown: ... Read-only file system` for the intentional `:ro` mounts. Those lines alone are not a broker authentication failure. Keep configuration and secrets read-only; identify the subsequent Mosquitto messages. `Error opening password file` is fatal and must be fixed. A directory can exist while the required `passwords` file inside it is missing. + +Start the broker for the first time: + +```sh +docker compose -p mqtt5-learning config +docker compose -p mqtt5-learning up -d +docker compose -p mqtt5-learning ps +docker compose -p mqtt5-learning logs --tail=50 broker +``` + +| Command part | What happens and what to check | +| --- | --- | +| `-p mqtt5-learning` | `-p` means project name in Compose, not network port. Selects the Compose project name, keeping its containers/network/volume names separate from other projects; use this same name for every lifecycle command | +| `config` | Parses and resolves Compose configuration; catches YAML/interpolation errors, but does not test broker authentication or every mounted file's contents | +| `up -d` | Creates/updates the required resources and starts containers in the background; `-d` means detached | +| `ps` | Shows container state and published ports; a running container is not an MQTT readiness check | +| `logs --tail=50 broker` | Shows the last 50 log lines for this service without opening a shell inside it | + +If a container was already created or ran before you fixed its password file, explicitly retry startup: + +```sh +docker compose -p mqtt5-learning restart broker +docker compose -p mqtt5-learning logs --since=1m broker +docker compose -p mqtt5-learning ps +``` + +`restart` stops/starts the existing service even if it has stopped after an error. `up -d` is not a guaranteed restart of an unchanged running container. `--since=1m` restricts logs to the last minute so earlier failures do not get mistaken for a fresh failure. If there is no existing container, use the initial `up -d` sequence instead. If you changed Compose ports or mounts, use `up -d` to reconcile them; if a replaced single-file mount still points at old content, use `up -d --force-recreate broker`. + +Expect a listening broker and no new password/ACL file errors. A running container alone does not prove authentication works. + +For this project, resource names commonly include `mqtt5-learning-broker-1`, `mqtt5-learning_default`, and `mqtt5-learning_broker-data`. Keep `-p mqtt5-learning` on every lifecycle command so they refer to the same project. In `mosquitto_pub/sub`, `-p` instead means port; in `mkdir -p`, it means create parent directories. Flags belong to their commands, not to a universal vocabulary. + +In `ps`, the local-only mapping should show `127.0.0.1:18883->1883/tcp`. A mapping showing `0.0.0.0` or `[::]` exposes that published port on all corresponding host interfaces. Change the Compose mapping to the loopback form and apply it with `up -d broker`; restarting alone does not change port bindings. + +## 4. Verify MQTT 5 without Go first + +Use host-installed `mosquitto_sub` and `mosquitto_pub`, or run their equivalents through `docker compose exec broker` with port 1883 inside the container. Keep a subscription open in terminal A: + +```sh +mosquitto_sub -h 127.0.0.1 -p 18883 -V mqttv5 -u backend -P '' -t 'lab/v1/devices/device-001/telemetry' -q 1 -v -d +``` + +In terminal B: + +```sh +mosquitto_pub -h 127.0.0.1 -p 18883 -V mqttv5 -u simulator -P '' -t 'lab/v1/devices/device-001/telemetry' -q 1 -m '{"online":true}' -d +``` + +`-h` chooses the broker host; `-p` here chooses the network port (unlike Compose's project-name flag). `-u` supplies a username, `-P` its password, `-t` the subscription filter or publication topic, and `-m` the message body. Single quotes preserve the JSON bytes instead of letting the shell interpret them. Start the subscriber first so it is ready before the nonretained publication. `mosquitto_sub` keeps running; Ctrl+C stops this subscriber. `mosquitto_pub` normally exits after completing its publication. + +If using `docker compose -p mqtt5-learning exec broker`, that prefix executes the utility inside the already-running broker container. Use `-h 127.0.0.1 -p 1883` there, because loopback refers to that container. The other MQTT flags remain the same. `exec` requires a running service; the earlier `docker run` setup commands do not. + +The placeholders are not literal passwords. `-V mqttv5` selects the protocol; `-q 1` selects QoS; `-d` shows packet activity; `-v` prints topic plus payload. Expect the subscriber to print the telemetry JSON. This is a synthetic message, not a production heartbeat contract. + +For the CLI only, `-P` exposes its argument to shell history/process inspection. Use temporary lab credentials and avoid pasting real credentials; consult your installed client's options for a protected configuration file if you need repeated CLI checks. Go examples read the password from the environment and never print it. + +Repeat with a wrong password. Also try publishing as `simulator` to `lab/v1/devices/device-002/telemetry`; this must not reach an authorized observer. Check the MQTT reason and broker logs rather than assuming every denied operation has the same client error. + +### Read the CLI exchange, not just the payload + +With `-d`, identify CONNECT/CONNACK first, then SUBSCRIBE/SUBACK in the subscriber terminal. Only publish after SUBACK. In the publisher terminal, find PUBLISH/PUBACK at QoS 1. In the subscriber terminal, find the separately forwarded PUBLISH and its acknowledgement. Debug formatting varies by client version; learn the packet names rather than memorizing one output line. + +Predict before running: stop the subscriber, publish again with the same command, then restart the subscriber. This nonretained message is not a history entry for a new subscriber. A successful publisher acknowledgement does not create a future subscription or a device response. [Lesson 4](04-publish-subscribe-simulator.md) maps these exchanges to Go calls; [Lesson 9](09-sessions-recovery.md) introduces the different case of an existing offline session. + +
+Check your explanation: why start the subscriber first? + +The broker needs an established matching subscription when this nonretained message arrives. Starting two terminals almost simultaneously leaves a race between SUBSCRIBE and PUBLISH. Seeing the subscriber process running is weaker evidence than seeing its successful SUBACK. + +
+ +## API preview: connect each setting to its purpose + +```go +broker, err := url.Parse("mqtt://127.0.0.1:18883") +// Check err before using broker. +cfg := autopaho.ClientConfig{ + ServerUrls: []*url.URL{broker}, + KeepAlive: 20, + ClientConfig: paho.ClientConfig{ClientID: "lab-backend"}, +} +``` + +This is an illustrative fragment, not a standalone program. The URL chooses the transport endpoint, while Client ID selects the broker session identity. `KeepAlive` is an integer number of seconds, not a `time.Duration`. The embedded `ClientConfig` describes the underlying MQTT client; the outer configuration describes managed connections. + +## Exercises and questions + +1. Explain why switching the Go library is more than changing one import. +2. Predict whether the simulator may subscribe to another device's commands; verify it. +3. Identify which packet proves a QoS 1 publication reached its immediate receiver, and why it does not prove application processing. +4. Stop and start only this lab with `docker compose -p mqtt5-learning stop` and `docker compose -p mqtt5-learning start`. + +**Completion:** CLI publish/subscribe works using MQTT 5, invalid credentials fail, and cross-device writes are denied. Do not use `down -v` unless you deliberately want to erase the lab's broker persistence. + +## Broker administration you should practice + +Read [Broker operations](broker-operations.md) for explained stop/start, restart versus recreate, password rotation, backups, restore, upgrades, and the steps required before remote deployment. Complete its local lifecycle and credential-rotation exercises before treating Section 1 as finished. Later Section 9 measures session recovery, and Section 11 in the course adds TLS and stronger access controls. + +## Common failures + +| Symptom | Check | +| --- | --- | +| Port is already allocated | Choose another loopback host port and update all host clients | +| Connection refused | Container state, listener logs, and host/container port distinction | +| Password file missing | Complete password generation before startup; verify `lab/secrets/passwords` exists and is nonempty | +| Password file unreadable | Verify broker-account read access, ownership/mode, mounted path, and directory traversal permissions | +| Read-only `chown` warnings | Check subsequent broker logs; retain `:ro` mounts and fix real read failures separately | +| Old error still appears in logs | Use `logs --since=1m` immediately after retrying startup | +| `services.volumes` validation error | Align the named-volume declaration with root-level `services:` | +| Port shows `0.0.0.0` or `[::]` | Set `127.0.0.1:18883:1883` and apply with `up -d broker` | +| Not authorized | Username, password, and exact permitted topic direction | +| Pub/sub works but Go fails later | Go URL/credentials, client ID, then tagged API version | + +## Sources + +- [Paho v0.23.0 release](https://github.com/eclipse-paho/paho.golang/releases/tag/v0.23.0) +- [Mosquitto broker configuration](https://mosquitto.org/man/mosquitto-conf-5.html) +- [Password-file tool](https://mosquitto.org/man/mosquitto_passwd-1.html) +- [Mosquitto publish tool](https://mosquitto.org/man/mosquitto_pub-1.html) + +Next: [Section 2](02-go-connection.md). diff --git a/lessons/02-go-connection.md b/lessons/02-go-connection.md new file mode 100644 index 0000000..097d854 --- /dev/null +++ b/lessons/02-go-connection.md @@ -0,0 +1,221 @@ +# Section 2 — Understand a Go MQTT 5 connection + +Prerequisite: Section 1's broker and backend credentials work. This section contains a complete small program. Type it in stages, compile after each meaningful change, and explain the code before running the failure exercises. + +**Version baseline:** Paho `v0.23.0`, verified from upstream release tags on 2026-10-07. It requires Go 1.24 or newer; Go 1.25 is compatible. The matching local reference is `references/paho.golang-v0.23.0/` (commit `1aa03969c5932464359b3e4ed752d9ac61abccc9`). + + +## Working directory and file placement + +Run every shell command in this lesson from the learning project root: + +```sh +cd /home/ryan/Media/Code/Code/robot-market/mqtt-5-learning +``` + +Paths such as `cmd/backend/main.go` are relative to this directory. Markdown files under `lessons/` explain the work; create the application files at the paths shown, not inside `lessons/` or `references/`. Code blocks labeled Go, TOML, YAML, or conf are file contents; shell blocks are terminal commands. Create parent directories before saving files. + +## 1. Build the connection program + +Create the directory first: + +```sh +mkdir -p cmd/backend +``` + +Create `cmd/backend/main.go` and paste the complete program below into it. If the file already exists, replace its contents rather than appending a second program: + +```go +package main + +import ( + "context" + "fmt" + "log" + "net/url" + "os" + "os/signal" + "syscall" + "time" + + "github.com/eclipse/paho.golang/autopaho" + "github.com/eclipse/paho.golang/paho" +) + +func main() { + if err := run(); err != nil { + log.Print(err) + os.Exit(1) + } +} + +func run() error { + ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM) + defer stop() + + user := os.Getenv("TEST_MQTT5_MQTT_USERNAME") + password := os.Getenv("TEST_MQTT5_MQTT_PASSWORD") + if user == "" || password == "" { + return fmt.Errorf("set the lab username and password environment variables") + } + broker, err := url.Parse("mqtt://127.0.0.1:18883") + if err != nil { + return fmt.Errorf("parse broker URL: %w", err) + } + + cfg := autopaho.ClientConfig{ + ServerUrls: []*url.URL{broker}, + KeepAlive: 20, + CleanStartOnInitialConnection: true, + SessionExpiryInterval: 0, + ConnectUsername: user, + ConnectPassword: []byte(password), + OnConnectionUp: func(_ *autopaho.ConnectionManager, ack *paho.Connack) { + log.Printf("connected: session_present=%t reason=0x%02x", ack.SessionPresent, ack.ReasonCode) + }, + OnConnectionDown: func() bool { + log.Print("connection down; AutoPaho will retry") + return true + }, + OnConnectError: func(err error) { + // Keep errors free of credentials; never log cfg or broker userinfo. + log.Printf("connect attempt failed: %v", err) + }, + ClientConfig: paho.ClientConfig{ + ClientID: "lab-backend", + OnClientError: func(err error) { log.Printf("client error: %v", err) }, + OnServerDisconnect: func(packet *paho.Disconnect) { + log.Printf("server disconnect: reason=0x%02x", packet.ReasonCode) + }, + }, + } + + manager, err := autopaho.NewConnection(ctx, cfg) + if err != nil { + return fmt.Errorf("create connection manager: %w", err) + } + defer func() { + shutdown, cancel := context.WithTimeout(context.Background(), 5*time.Second) + defer cancel() + if err := manager.Disconnect(shutdown); err != nil { + log.Printf("shutdown did not finish within deadline: %v", err) + } + }() + + startup, cancel := context.WithTimeout(ctx, 10*time.Second) + err = manager.AwaitConnection(startup) + cancel() + if err != nil { + return fmt.Errorf("initial connection: %w", err) + } + log.Print("connection ready; press Ctrl+C to stop") + <-ctx.Done() + return nil +} +``` + + +## 2. Understand the connection lifecycle + +Before reading the full program, predict what evidence you need at each stage: + +```text +Go process → AutoPaho retry loop → TCP connection → CONNECT → CONNACK + → subscriptions (Lesson 4) +``` + +AutoPaho owns reconnect attempts and creates the underlying Paho client for a connection. Paho implements MQTT packets and protocol exchanges. Your application owns readiness, authorization policy, command state, and business retries. A running goroutine or open TCP socket is too early to declare MQTT ready: the broker can still reject CONNECT. + +| Setting or call | MQTT meaning and reason for this choice | +| --- | --- | +| `ServerUrls` | Where to dial; the lab host endpoint is different from the Compose service endpoint | +| `ClientID` | Broker session identity; two simultaneous processes with this ID compete for one connection | +| `ConnectUsername`, `ConnectPassword` | CONNECT authentication inputs; independent of Client ID and payload identity | +| `KeepAlive: 20` | Maximum idle interval between client control packets, in seconds; Paho handles PINGREQ/PINGRESP when needed | +| `CleanStartOnInitialConnection: true` | Discard a previous broker session on the manager's first successful connection | +| `SessionExpiryInterval: 0` | Do not retain this broker session after the connection closes | +| `NewConnection` | Start connection management; a nil error does not establish a successful CONNACK | +| `AwaitConnection` | Wait for connection availability; it does not wait for SUBACK or a device heartbeat | +| `OnConnectionUp` | Observe a successful connection, including reconnection; inspect CONNACK evidence here | +| `OnConnectError` | Observe failed attempts, including refused connections or unsuccessful CONNACK | +| `OnConnectionDown` | Decide whether reconnect attempts should continue; returning false ends that retry lifecycle | +| `OnServerDisconnect` | Inspect a DISCONNECT actually sent by the broker; a severed cable cannot send this packet | + +Keep Alive checks the MQTT connection, not the health of the vending application. With an effective Keep Alive of 20 seconds, the broker's protocol inactivity limit is 30 seconds without a client control packet; an already-detected closed socket can be handled earlier. A CONNACK Server Keep Alive can override the requested value. Your three-second heartbeat in Lesson 4 carries application evidence and has a separate purpose. + +There are three different contexts here. The signal context controls the program lifetime. `startup` limits the initial wait to ten seconds. `shutdown` is a fresh context because the signal context is already canceled during shutdown. The manager receives the long-lived context, not the startup deadline; passing the startup context to `NewConnection` would terminate your service when that deadline expires. + +`CleanStartOnInitialConnection: true` and `SessionExpiryInterval: 0` choose a disposable early-lab session. This is convenient for first experiments and intentionally offers no offline broker session after disconnect. Later lessons change these settings and test recovery. Do not copy them as a durable production policy. + +Paho v0.23.0's `Disconnect(ctx)` cancels the manager and waits for manager shutdown with the provided deadline. Its `Done()` documentation notes that underlying MQTT cleanup may still be completing; neither API is proof that application data was persisted or device work was finished. + +## 3. Run it + +In zsh, read your backend password without echoing it: + +```sh +export TEST_MQTT5_MQTT_USERNAME=backend +read -rs 'TEST_MQTT5_MQTT_PASSWORD?Backend lab password: ' +export TEST_MQTT5_MQTT_PASSWORD +go run ./cmd/backend +``` + +`export NAME=value` puts a variable into this terminal's environment so child processes such as `go run` inherit it. The `read` line uses zsh syntax: `-r` preserves backslashes, `-s` hides typed characters, and `NAME?prompt` selects the variable and prompt. Enter the password you created in Lesson 1; this does not create a broker user. The second `export` exposes that newly read variable to the Go process. In bash the prompt syntax differs; do not copy this zsh-specific line unchanged. + +`go run ./cmd/backend` compiles the package and runs it in the foreground. `./` means relative to the project root. Ctrl+C sends an interrupt to the foreground program; `signal.NotifyContext` turns it into context cancellation, allowing your cleanup code to run. After the process exits, `unset TEST_MQTT5_MQTT_PASSWORD` removes the credential from this shell's environment; it does not rotate the broker credential. + +Environment variables avoid password literals in this command but are not a secrets vault. Use only lab credentials. Expect `connected` and `connection ready`; callback log order can vary. Press Ctrl+C and verify exit within the shutdown deadline. + +Run syntax checks after creating the file: + +```sh +gofmt -w cmd/backend/main.go +go build ./cmd/backend +go vet ./cmd/backend +``` + +`gofmt -w` rewrites the named source file using Go formatting. `go build` checks compilation and linking; `go vet` checks suspicious code patterns without running the application. A successful vet run is normally quiet. + +`go build` emits a binary in the current directory; remove or ignore it as appropriate. A build verifies syntax/types, not connection behavior. + +## 4. Observe the protocol and failures + +CONNECT carries your client ID, authentication fields, keepalive, and session properties. A successful CONNACK is connection evidence. Bad credentials produce an unsuccessful connection attempt; the rejected CONNACK is handled by the connection-error path rather than your success callback. + +1. Stop the broker before running the program. Expect retry diagnostics and initial-wait failure near ten seconds, followed by cleanup. +2. Start the broker; run the program; stop and restart the broker. Expect connection-down and connection-up events. You have not restored subscriptions yet because this program has none. +3. Supply a wrong password. Explain why retrying cannot repair an incorrect credential. +4. Run two copies using the same client ID. Observe connection takeover/reconnect behavior, then stop both. Unique IDs are necessary for concurrently running clients. +5. Press Ctrl+C while the broker is unavailable. Cancellation should interrupt the wait; do not wait for the full startup deadline. + +## Cautions and troubleshooting + +- AutoPaho handles connection management; it does not invent topic authorization, business retries, or durable command storage. +- Do not call `log.Fatal` from callbacks. It terminates the process without running deferred cleanup. +- Callbacks should be short. Never add an indefinite channel send, database operation, or sleep to `OnConnectionDown`. +- Keep client IDs stable when you later want session resumption, but distinct across different simultaneously running clients. +- Logging entire config structs or URL userinfo can reveal passwords. These examples keep credentials separate from the URL. +- A connection-up event is not subscription readiness, and neither is device readiness. + +**Completion:** Explain every field in `cfg`, distinguish the three contexts, and demonstrate successful startup, failed startup, reconnect, and cancellation. + +## Questions to answer before Section 3 + +1. Why can `NewConnection` return before a usable broker connection exists? +2. What would happen if you passed `startup` to `NewConnection`? +3. Why does shutdown use `context.Background()` as its parent? +4. What does returning `false` from `OnConnectionDown` change? + +
+Check your reasoning after answering + +`NewConnection` starts a background lifecycle; CONNACK is later evidence. A startup context passed as the manager lifetime would stop the manager when that context expires or is canceled. Shutdown needs a fresh bounded context because the signal context is already canceled. Returning false from `OnConnectionDown` stops further automatic reconnect attempts for that manager; it does not repair the connection or preserve application state. + +
+ +## Sources + +- [AutoPaho API v0.23.0](https://pkg.go.dev/github.com/eclipse/paho.golang@v0.23.0/autopaho) +- [Tagged connection manager source](https://github.com/eclipse-paho/paho.golang/blob/v0.23.0/autopaho/auto.go) + +Next: [Section 3](03-typed-configuration.md). diff --git a/lessons/03-typed-configuration.md b/lessons/03-typed-configuration.md new file mode 100644 index 0000000..9d43d0b --- /dev/null +++ b/lessons/03-typed-configuration.md @@ -0,0 +1,310 @@ +# Section 3 — Configure Go clients with TOML and koanf + +Prerequisite: Your Section 2 program connects and stops cleanly. Here you move the hard-coded settings into validated configuration. The module remains `example.com/mqtt5-learning`. + +**Version baseline:** Paho `v0.23.0`, verified from upstream release tags on 2026-10-07. It requires Go 1.24 or newer; Go 1.25 is compatible. The matching local reference is `references/paho.golang-v0.23.0/` (commit `1aa03969c5932464359b3e4ed752d9ac61abccc9`). + + +## Working directory and file placement + +Run every shell command in this lesson from the learning project root: + +```sh +cd /home/ryan/Media/Code/Code/robot-market/mqtt-5-learning +``` + +Paths such as `cmd/backend/main.go` are relative to this directory. Markdown files under `lessons/` explain the work; create the application files at the paths shown, not inside `lessons/` or `references/`. Code blocks labeled Go, TOML, YAML, or conf are file contents; shell blocks are terminal commands. Create parent directories before saving files. + +## Why configuration changes MQTT behavior + +The URL chooses a broker; Client ID chooses session identity at that broker; username/password select credentials; device ID chooses the application's topic namespace. Changing one does not change the others. If you change `device_id` to `device-002` but keep the simulator credential, constructing a new topic string does not grant permission to use it. + +Your connect timeout bounds how long startup waits. It is neither Keep Alive (connection inactivity) nor Session Expiry (broker state after disconnect), and it cannot delete a command already sent. Later add those settings as separate typed fields with explicit units. A session expiry of `60` is seconds in MQTT; a Go `time.Duration(60)` is 60 nanoseconds. + +Predict two failures: a malformed URL should fail local validation before dialing; a valid URL with a wrong password reaches the broker but fails authentication. Retrying the latter forever does not fix the configured secret. The loader's purpose is to make the effective policy predictable before AutoPaho starts background activity. + +## 1. Add koanf's core, provider, and parser modules + +```sh +go get github.com/knadh/koanf/v2@v2.3.7 +go get github.com/knadh/koanf/providers/confmap@v1.0.0 +go get github.com/knadh/koanf/providers/file@v1.2.0 +go get github.com/knadh/koanf/parsers/toml/v2@v2.2.0 +mkdir -p internal/config configs cmd/configcheck +``` + +These supporting module versions are a reproducible lesson baseline; the requested latest-release selection concerns Paho. Providers and parsers are separate Go modules. `confmap` supplies defaults, `file` reads bytes, and `toml/v2` interprets those bytes as TOML. The loader uses explicit environment mappings and standard Go flags so you can see precedence clearly. + +## 2. Create a nonsecret TOML file + +Create `configs/lab.toml` in the `configs/` directory created above. This is separate from the broker configuration at `lab/mosquitto/mosquitto.conf`; it configures your Go clients: + +```toml +[mqtt] +url = "mqtt://127.0.0.1:18883" +client_id = "lab-backend" +username = "backend" +device_id = "device-001" + +[app] +connect_timeout = "10s" +``` + +Keep the password in `TEST_MQTT5_MQTT_PASSWORD`. The loader supports a TOML password field for precedence demonstration, but do not store a real password in the committed example. TOML tables map to nested struct fields. + +## 3. Write the loader + +Create `internal/config/config.go`: + +```go +package config + +import ( + "flag" + "fmt" + "net/url" + "os" + "strings" + "time" + + "github.com/knadh/koanf/parsers/toml/v2" + "github.com/knadh/koanf/providers/confmap" + "github.com/knadh/koanf/providers/file" + "github.com/knadh/koanf/v2" +) + +type Config struct { + MQTT struct { + URL string `koanf:"url"` + ClientID string `koanf:"client_id"` + Username string `koanf:"username"` + Password string `koanf:"password"` + DeviceID string `koanf:"device_id"` + } `koanf:"mqtt"` + App struct { + ConnectTimeout string `koanf:"connect_timeout"` + } `koanf:"app"` +} + +// Keep duration input as text; validate/parse it explicitly after merging. +func (c Config) ConnectTimeout() time.Duration { + d, _ := time.ParseDuration(c.App.ConnectTimeout) // Load already validated it. + return d +} + +func Load(args []string) (Config, error) { + var result Config + flags := flag.NewFlagSet("mqtt5-learning", flag.ContinueOnError) + filename := flags.String("config", "configs/lab.toml", "TOML file") + broker := flags.String("broker", "", "broker URL override") + clientID := flags.String("client-id", "", "client ID override") + username := flags.String("username", "", "lab username override") + deviceID := flags.String("device-id", "", "device identity override") + timeout := flags.String("connect-timeout", "", "startup deadline override") + if err := flags.Parse(args); err != nil { + return result, err + } + if flags.NArg() != 0 { + return result, fmt.Errorf("unexpected positional arguments") + } + + k := koanf.New(".") + defaults := map[string]any{ + "mqtt.url": "mqtt://127.0.0.1:18883", + "mqtt.client_id": "lab-backend", + "mqtt.username": "", "mqtt.password": "", + "mqtt.device_id": "device-001", + "app.connect_timeout": "10s", + } + if err := k.Load(confmap.Provider(defaults, "."), nil); err != nil { + return result, fmt.Errorf("defaults: %w", err) + } + // A requested/missing config file is an error, not a silent fallback. + if err := k.Load(file.Provider(*filename), toml.Parser()); err != nil { + return result, fmt.Errorf("TOML: %w", err) + } + + // Explicit mapping preserves underscores inside names such as client_id. + mapping := map[string]string{ + "TEST_MQTT5_MQTT_URL": "mqtt.url", + "TEST_MQTT5_MQTT_CLIENT_ID": "mqtt.client_id", + "TEST_MQTT5_MQTT_USERNAME": "mqtt.username", + "TEST_MQTT5_MQTT_PASSWORD": "mqtt.password", + "TEST_MQTT5_MQTT_DEVICE_ID": "mqtt.device_id", + "TEST_MQTT5_APP_CONNECT_TIMEOUT": "app.connect_timeout", + } + for name, key := range mapping { + if value, exists := os.LookupEnv(name); exists { + if err := k.Set(key, value); err != nil { + return result, err + } + } + } + overrides := map[string]string{} + flags.Visit(func(f *flag.Flag) { + switch f.Name { + case "broker": + overrides["mqtt.url"] = *broker + case "client-id": + overrides["mqtt.client_id"] = *clientID + case "username": + overrides["mqtt.username"] = *username + case "device-id": + overrides["mqtt.device_id"] = *deviceID + case "connect-timeout": + overrides["app.connect_timeout"] = *timeout + } + }) + for key, value := range overrides { + if err := k.Set(key, value); err != nil { + return result, err + } + } + if err := k.Unmarshal("", &result); err != nil { + return result, fmt.Errorf("decode configuration: %w", err) + } + if err := result.Validate(); err != nil { + return result, err + } + return result, nil +} + +func (c Config) Validate() error { + u, err := url.Parse(c.MQTT.URL) + if err != nil || u == nil || u.Hostname() == "" || u.Port() == "" { + return fmt.Errorf("mqtt.url must contain a valid host and explicit port") + } + // Early lab only. TLS configuration is deliberately introduced later. + if u.Scheme != "mqtt" || u.User != nil || u.RawQuery != "" || u.Fragment != "" || u.Path != "" { + return fmt.Errorf("early lab requires mqtt://host:port without credentials or extra URL components") + } + if u.Hostname() != "127.0.0.1" && u.Hostname() != "localhost" { + return fmt.Errorf("plaintext learning broker must be on loopback") + } + for _, value := range []string{c.MQTT.ClientID, c.MQTT.Username, c.MQTT.Password} { + if strings.TrimSpace(value) == "" { + return fmt.Errorf("client ID and credentials are required") + } + } + if c.MQTT.DeviceID == "" || strings.ContainsAny(c.MQTT.DeviceID, "/+# \t\r\n") { + return fmt.Errorf("device ID must be a nonempty single topic segment") + } + d, err := time.ParseDuration(c.App.ConnectTimeout) + if err != nil || d <= 0 || d > time.Minute { + return fmt.Errorf("connect timeout must be a duration greater than zero and at most 1m") + } + return nil +} +``` + + +## 4. Understand effective configuration and precedence + +The loader composes one effective configuration before connecting. `koanf.New(".")` interprets `mqtt.client_id` as a nested key; the `koanf` field tags map those keys into the typed configuration consumed by the transport. Later `Load`/`Set` calls override earlier values. The map provider already supplies structured values, so it needs no text parser. + +The order is defaults → TOML → environment → explicitly supplied CLI flags. `flags.Visit` visits only supplied flags, preventing the empty CLI defaults from overwriting TOML/environment values. `os.LookupEnv` distinguishes an absent variable from an explicitly empty one; an empty password deliberately overrides a file value and then fails validation. + +We map `TEST_MQTT5_MQTT_CLIENT_ID` explicitly to `mqtt.client_id`. Replacing every underscore with a dot would produce `mqtt.client.id`, a different key. This allowlist also makes supported environment variables visible; unexpected prefixed variables are ignored in this small example. + +Duration text is validated with `time.ParseDuration`, avoiding reliance on implicit conversion. The helper returns a `time.Duration` after validation. URL and topic validation are intentionally narrow for the loopback-only early lab. Later TLS lessons extend the schema and validation instead of disabling certificate checks. + +## 5. Add a configuration inspection command + +Create `cmd/configcheck/main.go`: + +```go +package main + +import ( + "example.com/mqtt5-learning/internal/config" + "log" + "os" +) + +func main() { + cfg, err := config.Load(os.Args[1:]) + if err != nil { + log.Print(err) + os.Exit(1) + } + log.Printf("client_id=%s device_id=%s username=%s timeout=%s password=[redacted]", + cfg.MQTT.ClientID, cfg.MQTT.DeviceID, cfg.MQTT.Username, cfg.ConnectTimeout()) +} +``` + + +```sh +go run ./cmd/configcheck +go run ./cmd/configcheck -client-id lab-config-test +go run ./cmd/configcheck -connect-timeout 0s +``` + +Arguments after `./cmd/configcheck` belong to your program. `-client-id` overrides only that configured field; `-connect-timeout 0s` supplies a deliberately invalid zero-second duration. The inspection command loads and validates configuration without establishing an MQTT connection. Keep the backend password exported in this terminal for these checks. + +The first two commands should print a redacted summary. The last must fail before any broker connection. Do not print the whole struct: it contains a password. + +## 6. Integrate it into Section 2 + +Edit the existing `cmd/backend/main.go`. Keep one `package main`, one import block, and one copy of each function; the following integration steps modify the Section 2 program rather than create another executable. Run from the project root so the default `configs/lab.toml` path resolves correctly. + +In `cmd/backend/main.go`, import `example.com/mqtt5-learning/internal/config`. At the beginning of `run`, call: + +```go +settings, err := config.Load(os.Args[1:]) +if err != nil { + return err +} +``` + +Remove the direct environment lookup/credential check and use these replacements: + +| Old expression | Replacement | +| --- | --- | +| `url.Parse("mqtt://127.0.0.1:18883")` | `url.Parse(settings.MQTT.URL)` | +| `ConnectUsername: user` | `ConnectUsername: settings.MQTT.Username` | +| `ConnectPassword: []byte(password)` | `ConnectPassword: []byte(settings.MQTT.Password)` | +| `ClientID: "lab-backend"` | `ClientID: settings.MQTT.ClientID` | +| `context.WithTimeout(ctx, 10*time.Second)` | `context.WithTimeout(ctx, settings.ConnectTimeout())` | + +The snippets are edits to Section 2's complete program, not additional standalone executables. Compile after integrating. Section 4 provides a complete replacement backend once you understand the loader. + +## Exercises: prove the precedence + +1. Set TOML timeout to `8s`; unset `TEST_MQTT5_APP_CONNECT_TIMEOUT`; expect `8s`. +2. Export `TEST_MQTT5_APP_CONNECT_TIMEOUT=6s`; expect `6s`. +3. Pass `-connect-timeout 4s`; expect `4s`. +4. Remove the flag; expect `6s`, proving an absent flag cannot override the environment. +5. Try an empty password, missing TOML file, invalid URL, zero duration, and device ID `device/001`. +6. Write a table-driven test using a temporary TOML file and `t.Setenv`. Clear relevant inherited environment values in each test; environment-mutating tests must not use `t.Parallel`. + +`unset NAME` removes a terminal environment override, revealing the TOML/default layer underneath; it does not change the file. `export NAME=value` sets the environment layer for child processes. A flag is temporary for that invocation. Restore your sample TOML timeout after the experiment so later lessons have a predictable starting point. + +Create `internal/config/config_test.go` for the table-driven cases. A table-driven test loops over named inputs and expected outcomes with `t.Run`. `t.TempDir()` provides a temporary fixture directory that Go cleans up; `t.Setenv` restores the previous environment after the test. Use these to prevent test fixtures from rewriting your real configuration. + +Example assertion inside such a test: + +```go +t.Setenv("TEST_MQTT5_APP_CONNECT_TIMEOUT", "6s") +cfg, err := Load([]string{"-config", filename, "-connect-timeout", "4s"}) +if err != nil { t.Fatal(err) } +if cfg.ConnectTimeout() != 4*time.Second { t.Fatalf("wrong precedence: %s", cfg.ConnectTimeout()) } +``` + +`filename` is the temporary fixture you create; the fragment belongs inside a `Test...` function in package `config`. Also test no flag, malformed values, and an empty password. + +## Common mistakes + +- TOML paths are relative to the process working directory. Run from the project root or pass an explicit `-config` path. +- Exporting backend credentials globally will also affect the simulator. Set simulator credentials in its own terminal. +- Command-line password flags expose secrets to process inspection, so this loader deliberately omits one. +- Configuration precedence is a merge decision, not an MQTT feature. Paho receives only the final validated values. +- These samples do not reject all unknown TOML keys. Add strict key validation as an exercise if typo detection becomes important. + +**Completion:** Demonstrate the override sequence and invalid-startup cases, then explain how the final values reach AutoPaho. + +## Sources + +- [koanf concepts, providers, and unmarshalling](https://pkg.go.dev/github.com/knadh/koanf/v2@v2.3.7) +- [AutoPaho configuration v0.23.0](https://pkg.go.dev/github.com/eclipse/paho.golang@v0.23.0/autopaho#ClientConfig) + +Next: [Section 4](04-publish-subscribe-simulator.md). diff --git a/lessons/04-publish-subscribe-simulator.md b/lessons/04-publish-subscribe-simulator.md new file mode 100644 index 0000000..94d48d8 --- /dev/null +++ b/lessons/04-publish-subscribe-simulator.md @@ -0,0 +1,481 @@ +# Section 4 — Publish, subscribe, and build a Go device simulator + +Prerequisites: Sections 1–3 complete. Keep Section 3's configuration package and TOML file. This lesson supplies complete files for a reusable MQTT transport, backend, and simulator. Implement each file in stages and explain it before running both programs. + +**Version baseline:** Paho `v0.23.0`, verified from upstream release tags on 2026-10-07. It requires Go 1.24 or newer; Go 1.25 is compatible. The matching local reference is `references/paho.golang-v0.23.0/` (commit `1aa03969c5932464359b3e4ed752d9ac61abccc9`). + + +## 1. Map the conversation + +| Topic | Publisher | Subscriber | Example | +| --- | --- | --- | --- | +| `lab/v1/devices/device-001/telemetry` | Simulator | Backend | Timestamped heartbeat | +| `lab/v1/devices/device-001/commands` | Backend | Simulator | Harmless ping with ID | +| `lab/v1/devices/device-001/responses` | Simulator | Backend | Pong echoing that ID | + +A broker subscription and a local Go callback are different things. SUBSCRIBE asks the broker which messages to forward. `OnPublishReceived` tells your local client how to process forwarded PUBLISH packets. You need both. This lab's topic convention is deliberately separate from the production contracts. + +### Trace two independent delivery hops + +```mermaid +sequenceDiagram + participant B as Go backend + participant M as Broker + participant S as Go simulator + S->>M: SUBSCRIBE commands, maximum QoS 1 + M-->>S: SUBACK with grant or rejection + B->>M: PUBLISH command, QoS 1 + M-->>B: PUBACK for backend-to-broker hop + M->>S: PUBLISH command, granted delivery QoS + S-->>M: PUBACK for broker-to-simulator hop + S->>M: PUBLISH application result + M->>B: PUBLISH application result +``` + +This diagram groups exchanges by purpose; it does not promise the backend sees PUBACK before the simulator's response. Each side has its own connection and packet identifiers. The broker routes by topic, not by JSON fields. A result is another publication on another topic; MQTT does not create a response automatically. + +### Choose QoS by the guarantee you need + +| QoS | Successful exchange on one hop | What the Go application must still handle | +| --- | --- | --- | +| 0 | PUBLISH, no MQTT acknowledgement | Loss is possible; nil publish error does not prove receipt | +| 1 | PUBLISH → PUBACK | Redelivery is possible; use stable business IDs | +| 2 | PUBLISH → PUBREC → PUBREL → PUBCOMP | Protocol duplicate suppression needs session state; application retries and external effects still need idempotency | + +The subscriber requests a **maximum** delivery QoS. A QoS 1 publication cannot become QoS 2 merely because the subscription requests 2. With a QoS 0 subscription grant, a publisher's QoS 1 exchange with the broker still ends in PUBACK while the broker-to-subscriber hop has no PUBACK. QoS 2 does not make a database update and a physical dispense one transaction. + +For a QoS 2 experiment, change both publication and requested subscription QoS to 2, accept SUBACK grants 0–2, and inspect the four-packet exchange. The original helper intentionally accepts only grants up to its requested QoS 1. Do not change only the publish field and call the resulting lower-QoS subscriber path an end-to-end QoS 2 test. + +### Topic filters are routing rules + +`lab/v1/devices/+/telemetry` matches one device segment; `lab/v1/devices/#` matches that subtree. `+` occupies an entire level and `#` is the final level. Topics are case-sensitive, so `device-001` and `Device-001` differ. Wildcards belong in subscriptions, never publish topic names. A root `#` filter does not include `$SYS/...`; broker system topics require an explicit matching filter and permission. + +A SUBACK proves the subscription request received a result. Topic ACLs still govern delivery: an accepted broad filter does not prove access to every matching topic. Check an authorized observer, reason codes, and broker logs when testing denial. + +## 2. Create the transport helper + +```sh +mkdir -p internal/mqtt cmd/simulator +``` + +Create `internal/mqtt/client.go`: + +```go +package mqtt + +import ( + "context" + "fmt" + "log" + "net/url" + "time" + + "example.com/mqtt5-learning/internal/config" + "github.com/eclipse/paho.golang/autopaho" + "github.com/eclipse/paho.golang/paho" +) + +type Message struct { + Topic string + Payload []byte + Retain bool +} + +func Start(ctx context.Context, cfg config.Config, topics []string, inbox chan<- Message) (*autopaho.ConnectionManager, error) { + broker, err := url.Parse(cfg.MQTT.URL) + if err != nil { + return nil, err + } + initialSubscription := make(chan error, 1) + manager, err := autopaho.NewConnection(ctx, autopaho.ClientConfig{ + ServerUrls: []*url.URL{broker}, + KeepAlive: 20, + CleanStartOnInitialConnection: true, + SessionExpiryInterval: 0, + ConnectUsername: cfg.MQTT.Username, + ConnectPassword: []byte(cfg.MQTT.Password), + OnConnectionUp: func(cm *autopaho.ConnectionManager, _ *paho.Connack) { + // This runs again on reconnect; reestablish broker subscriptions. + subCtx, cancel := context.WithTimeout(ctx, 5*time.Second) + defer cancel() + subscriptions := make([]paho.SubscribeOptions, 0, len(topics)) + for _, topic := range topics { + subscriptions = append(subscriptions, paho.SubscribeOptions{Topic: topic, QoS: 1, RetainAsPublished: true}) + } + ack, subErr := cm.Subscribe(subCtx, &paho.Subscribe{Subscriptions: subscriptions}) + if subErr == nil { + if ack == nil || len(ack.Reasons) != len(topics) { + subErr = fmt.Errorf("missing or incomplete SUBACK") + } else { + for i, reason := range ack.Reasons { + if reason > 1 { // We requested QoS 1; valid grants are 0 or 1. + subErr = fmt.Errorf("subscription %s rejected: 0x%02x", topics[i], reason) + break + } + } + } + } + if subErr != nil { + log.Printf("subscribe failed: %v", subErr) + } else { + log.Printf("subscriptions ready: %v", topics) + } + select { + case initialSubscription <- subErr: + default: + } + }, + OnConnectionDown: func() bool { log.Print("disconnected; reconnecting"); return true }, + OnConnectError: func(err error) { log.Printf("connect error: %v", err) }, + ClientConfig: paho.ClientConfig{ + ClientID: cfg.MQTT.ClientID, + OnClientError: func(err error) { log.Printf("client error: %v", err) }, + OnServerDisconnect: func(d *paho.Disconnect) { log.Printf("disconnect reason=0x%02x", d.ReasonCode) }, + OnPublishReceived: []func(paho.PublishReceived) (bool, error){ + func(received paho.PublishReceived) (bool, error) { + packet := received.Packet + message := Message{Topic: packet.Topic, Payload: append([]byte(nil), packet.Payload...), Retain: packet.Retain} + select { + case inbox <- message: + default: + // Deliberately lossy early demo: automatic MQTT ACK is still possible. + log.Printf("lab inbox full; message dropped topic=%s", packet.Topic) + } + return true, nil + }, + }, + }, + }) + if err != nil { + return nil, fmt.Errorf("start AutoPaho: %w", err) + } + startup, cancel := context.WithTimeout(ctx, cfg.ConnectTimeout()) + defer cancel() + if err := manager.AwaitConnection(startup); err != nil { + Stop(manager) + return nil, fmt.Errorf("await connection: %w", err) + } + select { + case err := <-initialSubscription: + if err != nil { + Stop(manager) + return nil, err + } + case <-startup.Done(): + Stop(manager) + return nil, fmt.Errorf("await subscriptions: %w", startup.Err()) + } + return manager, nil +} + +func Publish(ctx context.Context, manager *autopaho.ConnectionManager, topic string, payload []byte) error { + operation, cancel := context.WithTimeout(ctx, 3*time.Second) + defer cancel() + ack, err := manager.Publish(operation, &paho.Publish{ + Topic: topic, QoS: 1, Retain: false, Payload: payload, + Properties: &paho.PublishProperties{ContentType: "application/json"}, + }) + if err != nil { + return fmt.Errorf("publish: %w", err) + } + if ack == nil { + return fmt.Errorf("missing QoS 1 publication response") + } + if ack.ReasonCode >= 0x80 { + return fmt.Errorf("publish rejected: 0x%02x", ack.ReasonCode) + } + log.Printf("broker publish acknowledgement topic=%s reason=0x%02x", topic, ack.ReasonCode) + return nil +} + +func Stop(manager *autopaho.ConnectionManager) { + shutdown, cancel := context.WithTimeout(context.Background(), 5*time.Second) + defer cancel() + if err := manager.Disconnect(shutdown); err != nil { + log.Printf("shutdown: %v", err) + } +} +``` + + +## 3. Map the helper to MQTT behavior + +| Code decision | Why it matters | +| --- | --- | +| Subscribe inside `OnConnectionUp` | The early disposable session loses its broker subscriptions on disconnect | +| Inspect every `ack.Reasons` entry | Each requested filter has its own result; connection success is insufficient | +| `RetainAsPublished: true` | Preserve the publisher's RETAIN flag on live forwarding, so the simulator can reject live retained commands as well as retained replay | +| Copy payload into `Message` | The application queue owns bytes that will be processed after the receive callback returns | +| Nonblocking inbox send | Bound callback time and memory; dropping on overflow is deliberately lossy | +| `return true, nil` | Report local handling to subsequent callbacks; this is not a negative/positive business acknowledgement | +| `QoS: 1`, `Retain: false` | Ask for a broker acknowledgement without storing the command as the topic's retained value | +| `ContentType: "application/json"` | Describe the encoding; neither broker nor library validates the application's schema for you | +| Initial SUBACK channel | Delay startup until this client's receive path is established; it says nothing about the peer's readiness | + +SUBACK grants `0`, `1`, or `2` for maximum delivery QoS. Since this program requests QoS 1, it accepts only 0 or 1. Failure reason codes start at `0x80`. Receiving a successful call return without inspecting subscription results can leave you connected but receiving no useful messages. + +For QoS 1, `Publish` returns a response describing the immediate broker acknowledgement. Success codes can include `0x10` (no matching subscribers); this does not prove the simulator is online. Transport errors and context timeouts can leave the delivery outcome uncertain. Never retry a real action with a new business ID just because your waiting context expired. + +`return true, nil` means the callback handled the message for local routing. It is not a device acceptance/completion response or durable-storage guarantee. With automatic acknowledgement, Paho can acknowledge after callback processing while your inbox has not yet been handled. Inspect `paho/client.go` in the tagged source to understand callback/ack order. + +The initial-subscription channel allows startup to wait for SUBACK as well as CONNACK. The callback subscribes again on reconnect. This helper checks initial readiness only; reconnect subscription failures are logged, not represented as continuous health state. Add that state later rather than treating initial readiness as permanent. + +## 4. Replace the backend main program + +Create `cmd/backend/main.go`: + +```go +package main + +import ( + "context" + "encoding/json" + "fmt" + "log" + "os" + "os/signal" + "syscall" + "time" + + "example.com/mqtt5-learning/internal/config" + mqtt "example.com/mqtt5-learning/internal/mqtt" +) + +func main() { + if err := run(); err != nil { + log.Print(err) + os.Exit(1) + } +} + +func run() error { + cfg, err := config.Load(os.Args[1:]) + if err != nil { + return err + } + ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM) + defer stop() + base := "lab/v1/devices/" + cfg.MQTT.DeviceID + inbox := make(chan mqtt.Message, 16) + manager, err := mqtt.Start(ctx, cfg, []string{base + "/telemetry", base + "/responses"}, inbox) + if err != nil { + return err + } + defer mqtt.Stop(manager) + ticker := time.NewTicker(10 * time.Second) + defer ticker.Stop() + runID := time.Now().UnixNano() + sequence := 0 + for { + select { + case <-ctx.Done(): + return nil + case message := <-inbox: + // Synthetic lab payloads only; production payload logging needs redaction. + log.Printf("received topic=%s retained=%t payload=%s", message.Topic, message.Retain, message.Payload) + case <-ticker.C: + sequence++ + payload, err := json.Marshal(struct { + ID string `json:"id"` + Command string `json:"command"` + }{ID: fmt.Sprintf("ping-%d-%d", runID, sequence), Command: "ping"}) + if err != nil { + return err + } + if err := mqtt.Publish(ctx, manager, base+"/commands", payload); err != nil { + log.Printf("command publication failed or outcome uncertain: %v", err) + } + } + } +} +``` + + +The ticker sends one ping every ten seconds. It starts after the backend's own subscriptions are ready. A timestamp and sequence make IDs distinct enough for this demonstration, not a production idempotency system. The backend logs the response but does not yet implement pending requests or validate completion; Section 7 adds those semantics. + +## 5. Create the device simulator + +Create `cmd/simulator/main.go`: + +```go +package main + +import ( + "context" + "encoding/json" + "log" + "os" + "os/signal" + "syscall" + "time" + + "example.com/mqtt5-learning/internal/config" + mqtt "example.com/mqtt5-learning/internal/mqtt" +) + +func main() { + if err := run(); err != nil { + log.Print(err) + os.Exit(1) + } +} + +func run() error { + cfg, err := config.Load(os.Args[1:]) + if err != nil { + return err + } + ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM) + defer stop() + base := "lab/v1/devices/" + cfg.MQTT.DeviceID + inbox := make(chan mqtt.Message, 16) + manager, err := mqtt.Start(ctx, cfg, []string{base + "/commands"}, inbox) + if err != nil { + return err + } + defer mqtt.Stop(manager) + ticker := time.NewTicker(3 * time.Second) + defer ticker.Stop() + for { + select { + case <-ctx.Done(): + return nil + case <-ticker.C: + payload, err := json.Marshal(struct { + DeviceID string `json:"device_id"` + Time string `json:"time"` + }{cfg.MQTT.DeviceID, time.Now().UTC().Format(time.RFC3339)}) + if err != nil { + return err + } + if err := mqtt.Publish(ctx, manager, base+"/telemetry", payload); err != nil { + log.Printf("heartbeat: %v", err) + } + case message := <-inbox: + var command struct { + ID string `json:"id"` + Command string `json:"command"` + } + if message.Retain { + log.Print("ignoring retained command") + continue + } + if err := json.Unmarshal(message.Payload, &command); err != nil || command.ID == "" || command.Command != "ping" { + log.Print("ignoring malformed or unsupported command") + continue + } + response, err := json.Marshal(struct { + ID string `json:"id"` + DeviceID string `json:"device_id"` + Status string `json:"status"` + }{command.ID, cfg.MQTT.DeviceID, "pong"}) + if err != nil { + return err + } + if err := mqtt.Publish(ctx, manager, base+"/responses", response); err != nil { + log.Printf("response: %v", err) + } + } + } +} +``` + + +`json.Marshal` serializes Go structs to JSON bytes. Tags choose field names. `json.Unmarshal(message.Payload, &command)` needs a pointer so the decoder can fill the struct. The simulator's processing loop publishes responses outside the receive callback, keeping the callback short and avoiding blocking it on a network round trip. + +This simulator sends a heartbeat every three seconds, receives ping commands, and publishes a pong carrying the same ID. It rejects packets marked retained to avoid treating stored state as a fresh one-shot action. The subscription sets `RetainAsPublished: true`: with the default false, live forwarding can clear RETAIN and defeat a guard that checks only `message.Retain`. Keep command publishers nonretaining too; this guard does not remove a retained value already stored at the broker. Full contract validation and duplicate handling arrive in later sections; this only demonstrates the transport lifecycle. + +## 6. Run two separate identities + +Terminal A, simulator first: + +```sh +export TEST_MQTT5_MQTT_USERNAME=simulator +read -rs 'TEST_MQTT5_MQTT_PASSWORD?Simulator lab password: ' +export TEST_MQTT5_MQTT_PASSWORD +go run ./cmd/simulator -client-id lab-simulator-device-001 +``` + +Terminal B, backend: + +```sh +export TEST_MQTT5_MQTT_USERNAME=backend +read -rs 'TEST_MQTT5_MQTT_PASSWORD?Backend lab password: ' +export TEST_MQTT5_MQTT_PASSWORD +go run ./cmd/backend -client-id lab-backend +``` + +Open two independent terminal sessions and change both to the project root. Their environments are separate, so the simulator's exported credentials do not overwrite the backend's. The zsh `read -rs` prompt hides the password; `export` passes it to the child process. `go run` runs each executable in the foreground; `-client-id` is a program flag overriding the TOML default, not a Docker setting. Stop each program with Ctrl+C and unset its password when finished. + +Use the same configured `device-001` in both. Distinct username/password pairs authorize different topic directions. Distinct client IDs prevent connection takeover. A username, client ID, and payload device ID serve different purposes; the ACL is what binds this simulator credential to its permitted device namespace. + +Expected backend output, abbreviated (timestamps and IDs vary): + +```text +subscriptions ready: [lab/v1/devices/device-001/telemetry lab/v1/devices/device-001/responses] +received topic=lab/v1/devices/device-001/telemetry ... payload={"device_id":"device-001","time":"..."} +broker publish acknowledgement topic=lab/v1/devices/device-001/commands reason=0x00 +received topic=lab/v1/devices/device-001/responses ... payload={"id":"ping-...","device_id":"device-001","status":"pong"} +``` + +Log order is not guaranteed: the device response may be observed around the publication acknowledgement. Separate connections and goroutines progress independently. If the simulator is not subscribed yet, a ping may have no recipient; this is expected with these zero-expiry early-lab sessions. + +## 7. Experiments to learn the API + +1. Replace the backend's exact telemetry subscription with `lab/v1/devices/+/telemetry`. Explain the topic filter versus a concrete publish topic. Never publish to a topic containing `+` or `#`. +2. Add `Retain: true` only to synthetic telemetry; restart the backend and observe a retained value. Restore `false` afterward. Do not retain one-shot commands. +3. Change publication QoS to 0. Modify the helper's acknowledgement check: QoS 0 has no broker PUBACK; any returned response is not evidence of broker confirmation. Compare CLI packet traces at QoS 0 and 1. +4. Stop the simulator while the backend continues. Observe broker acknowledgement without a pong. Explain why the broker cannot tell you the vending application completed a ping. +5. Restart the broker. Observe reconnect and renewed subscription readiness. Ping delivery during the reconnect interval is not guaranteed by this helper. +6. Change simulator `-device-id device-002`. Its existing ACL must prevent the new subscriptions/publications; do not broaden its ACL to hide the failure. +7. Reduce inbox capacity and add a slow consumer. Observe explicit drops; explain why automatic acknowledgement can make those drops irreversible. + +
+Self-check: why does QoS 1 still lose a command in the full-inbox experiment? + +The immediate receiver is the MQTT client. Its automatic acknowledgement can complete delivery even though your callback drops the application copy. QoS cannot recover work your own receive path has acknowledged and discarded. Preserve important work before acknowledging, with an explicit overload and recovery policy. + +
+ +
+Self-check: why does publishing with Retain false not clear the previous retained value? + +The new publication does not ask the broker to change retained storage. Delete the stored value using a zero-byte retained publication to the same exact topic, then test with a fresh subscription. Stop any retained publisher during that check so it cannot immediately replace the value. + +
+ +## Cautions: limits of this early example + +- The inbox is deliberately bounded and lossy. It makes overload visible without blocking receive callbacks. It is unsuitable for durable sales reports or real commands; later sections replace this policy with persistence and deliberate acknowledgement/backpressure rules. +- A publish call can block this single application loop for up to three seconds; this is acceptable for a tiny demonstration. Later sections add workers and separate business processing. +- QoS 1 can duplicate delivery. Ping is harmless, but real effects need stable command IDs and device-side deduplication. +- Payload validation here is intentionally minimal. Content Type is descriptive metadata, not authentication or schema enforcement. +- Fresh sessions and zero expiry mean offline commands are not a durable workflow. Do not infer restart guarantees from a successful reconnect. +- Raw payload logs are allowed here only for small synthetic lab messages; production logs need redaction and size limits. +- Never close the inbox while Paho may still send to it. These programs let the context and manager stop the producer; the channel need not be closed for process exit. + +## Verification and completion + +```sh +gofmt -w cmd internal +go test ./... +go vet ./... +go build ./cmd/backend ./cmd/simulator ./cmd/configcheck +``` + +`mkdir -p` above creates missing parent directories without creating Go files. `gofmt -w cmd internal` formats Go files under those directories. `go test ./...` runs tests in all packages in this module; `...` is Go's recursive package pattern. `go vet ./...` performs static checks. `go build` with several package paths checks all those executables without writing one combined binary. Run these from the root, after creating all referenced packages. + +For the retain experiment, publish a zero-byte retained payload to that exact telemetry topic to clear its retained value; merely switching future publications back to nonretained does not erase the broker's old retained message. Publish with an identity authorized to write the topic. For broker restart, use the explained lifecycle command in [Broker operations](broker-operations.md), then watch both clients reconnect and resubscribe. + +Packages without tests do not demonstrate runtime correctness. Also run both programs, perform the failure experiments, and verify clean Ctrl+C behavior. + +**Completion:** Explain the complete path from backend Go struct → JSON bytes → MQTT PUBLISH → simulator callback → application loop → response PUBLISH → backend callback. Distinguish broker acknowledgement from a device response, and identify the lossy queue and offline-session limitations yourself. + +## Sources + +- [Paho client API v0.23.0](https://pkg.go.dev/github.com/eclipse/paho.golang@v0.23.0/paho) +- [Tagged AutoPaho source](https://github.com/eclipse-paho/paho.golang/blob/v0.23.0/autopaho/auto.go) +- [Tagged receive callback source](https://github.com/eclipse-paho/paho.golang/blob/v0.23.0/paho/client.go) +- [MQTT 5 specification](https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html) + +Next: [Section 5 — Contracts and validation](05-contracts-validation.md). diff --git a/lessons/05-contracts-validation.md b/lessons/05-contracts-validation.md new file mode 100644 index 0000000..52699c3 --- /dev/null +++ b/lessons/05-contracts-validation.md @@ -0,0 +1,86 @@ +# Section 5 — Design contracts and validate inbound messages + +Prerequisite: Section 4 exchanges heartbeats and ping/pong. + +Run all commands from `/home/ryan/Media/Code/Code/robot-market/mqtt-5-learning`. Create files at the project-relative paths shown below, not under `lessons/`. Continue the same application and pinned Paho v0.23.0 dependency. These are guided implementation exercises; attempt each step before requesting a full solution. + +## Follow one message through the trust boundaries + +Suppose the backend receives topic `lab/v1/devices/device-001/sales` with a body claiming `device_id: device-002`. The broker may have correctly authorized the sender to publish to the topic. It has not established that the JSON is truthful. Reject the mismatch before selecting a device record or changing a balance. + +```text +PUBLISH → size limit → topic parser → JSON decoder → schema checks + → topic/body identity check → business validation → handler +``` + +Each step answers a different question. The size limit bounds decoding cost. Parsing establishes the intended route. JSON decoding proves structure only. Business validation rejects a syntactically valid negative quantity. None of these replaces broker authentication. + +The 16 KiB application limit applies to `Payload`, not the entire MQTT packet. Topic and properties add bytes; the library has already received the packet before this validator runs. Lesson 16 adds protocol packet-size limits to bound an earlier layer. + +### Map encoding metadata into Paho + +This fragment belongs where you build a `paho.Publish`; `payload` is the JSON produced by your contract encoder: + +```go +utf8Payload := byte(1) +properties := &paho.PublishProperties{ + ContentType: "application/json", + PayloadFormat: &utf8Payload, +} +_ = properties // attach as Publish.Properties in your implementation +``` + +Payload Format Indicator 1 declares UTF-8 character data. It does not mean JSON, schema version 1, or trusted content. Content Type describes the media type. For example, `hello` is UTF-8 but fails your JSON object contract. Property pointers distinguish an omitted property from an explicitly supplied value; inspect nil before dereferencing incoming fields. + +Keep these distinctions in your transport type when needed: packet metadata belongs alongside payload bytes, while `schema_version` and `command_id` belong to your application contract. Do not make handlers depend on a mutable packet owned elsewhere merely to read metadata. + +### Predict, modify, explain + +Send a valid heartbeat, then change only its body device ID, then change only its schema version. Predict which validation stage rejects each. Finally publish valid JSON with an extra trailing object: one successful decoder call is insufficient unless you also require EOF. + +
+Why can an invalid message still get a successful MQTT acknowledgement? + +MQTT receipt and application validity are separate. With the early automatic-acknowledgement helper, returning from the callback allows acknowledgement even when later validation rejects the queued message. Record rejection in application evidence; do not report the PUBACK as successful business processing. Only respond to a command after establishing a safe, authorized response destination. + +
+ +## 1. Create the contract package + +```sh +mkdir -p internal/contracts +``` + +Create `internal/contracts/messages.go`, `topics.go`, `decode.go`, and `decode_test.go`. Edit the existing backend and simulator files to use this package. These are exercises: implement the functions described below; this lesson does not supply a complete replacement program. + +Define JSON structs for Heartbeat, Command, CommandResult, and SalesReport. Use `schema_version`, `message_id`, `device_id`, and `occurred_at` consistently. Command also needs `command_id` and `command`; CommandResult needs the same command ID and a status. A sales report needs a stable report ID and synthetic line items with positive quantities and integer minor-unit amounts. Keep command identity stable across retries; an MQTT packet identifier is not that identity. + +## 2. Parse topics before routing + +Implement a parser accepting exactly `lab/v1/devices/{deviceID}/{kind}`. Initially permit telemetry, commands, responses, and sales. Reject empty device IDs, extra segments, wildcards, and unknown kinds. Return a typed device ID and kind rather than making every handler split strings itself. Extend the lab ACL explicitly for sales only when you start publishing reports. + +## 3. Decode and validate + +Apply a 16 KiB payload limit before decoding. Use `json.Decoder` with `DisallowUnknownFields`, then require EOF after the first JSON value. Validate schema version 1, required IDs, supported commands, topic/body device agreement, RFC3339 timestamps, and field lengths. Choose and document a future-clock tolerance, such as 30 seconds; inject the current time into validation to keep tests deterministic. Historical sales may legitimately be old, so do not apply heartbeat freshness rules to every report. + +Treat Content Type and Payload Format Indicator as descriptive metadata. Validate bytes even when they claim to contain JSON. A payload device ID is a claim; broker credentials and ACLs supply the separate authorization boundary. + +## 4. Integrate and verify + +Replace Section 4's anonymous JSON structs in both programs with these contracts. Validate before updating state or sending a result. Log the error category and bounded identifiers, without dumping rejected payloads. Unsupported or malformed commands must have no simulated effect. + +```sh +gofmt -w internal/contracts cmd +go test ./internal/contracts +go test ./... +``` + +`mkdir -p` creates the package directory; you still create the listed `.go` files yourself. `gofmt -w` rewrites source formatting, `go test ./internal/contracts` isolates contract tests, and `go test ./...` checks the whole evolving module for integration regressions. Go discovers test functions named `Test...` in `_test.go` files. No broker is needed for pure parser/validator tests. + +When adding sales, edit `lab/mosquitto/acl`: add `topic read lab/v1/devices/+/sales` under backend and `topic write lab/v1/devices/device-001/sales` under simulator. Restart the lab broker to read the changed ACL, check logs, and test allowed and cross-device-denied publications. Permissions enable a topic direction; they do not replace sales validation. + +Test valid messages, malformed JSON, trailing JSON, unknown fields, missing IDs, unsupported versions, identity mismatch, unknown commands, invalid timestamps, and payloads exactly at and above the limit. Keep the simulator and backend contracts in sync when changing the wire format. + +**Done when:** Both programs still exchange ping/pong, every rejection has a predictable reason, and invalid sales or commands leave state unchanged. Explain why schema validation does not authenticate a device. + +Previous: [04-publish-subscribe-simulator.md](04-publish-subscribe-simulator.md). Next: [06-concurrency-cancellation.md](06-concurrency-cancellation.md). diff --git a/lessons/06-concurrency-cancellation.md b/lessons/06-concurrency-cancellation.md new file mode 100644 index 0000000..06e8e6b --- /dev/null +++ b/lessons/06-concurrency-cancellation.md @@ -0,0 +1,84 @@ +# Section 6 — Manage callbacks, concurrency, and cancellation + +Prerequisite: Section 5 validates messages before handling them. + +Run all commands from `/home/ryan/Media/Code/Code/robot-market/mqtt-5-learning`. Create files at the project-relative paths shown below, not under `lessons/`. Continue the same application and pinned Paho v0.23.0 dependency. These are guided implementation exercises; attempt each step before requesting a full solution. + +## Understand the acknowledgement boundary before adding workers + +For the pinned client's automatic mode, `routePublishPackets` invokes the registered handlers, then calls its session acknowledgement path for nonzero QoS. The boolean and error returned by a handler are passed to later handlers; returning an error does not turn the callback into an application-controlled MQTT NACK or force redelivery. + +Trace the early QoS 1 implementation: + +```text +receive PUBLISH → copy into RAM queue → callback returns → MQTT ACK + → worker validates/applies effect later +``` + +The worker may finish before or after the ACK. If the process crashes after the ACK but before the effect, the broker can consider its delivery complete while the application loses the work. Increasing channel capacity increases the possible backlog; it does not close that gap. + +In manual mode, `EnableManualAcknowledgment` changes who releases acknowledgement. For QoS 1, the intended durable-ingestion sequence is receive → validate → commit durable admission → call `received.Client.Ack(received.Packet)`. Preserve the receiving client and original packet for this operation; a payload-only copy is insufficient. Check Ack's error and study the pinned source before implementing it. Do not reuse this QoS 1 explanation as a claim that every QoS 2 handshake stage waits for your database. + +The client's tracker sends acknowledgements in receipt order. If work 3 finishes before 1, acknowledging 3 can remain buffered behind 1. Slow work therefore consumes inflight capacity even with several workers. Manual acknowledgement delays protocol completion; it does not persist your Go queue, remove duplicate effects, or automatically schedule retries while the same connection stays open. + +### Choose what the callback promises + +| Message class | Reasonable admission choice | Meaning of a successful admission | +| --- | --- | --- | +| Disposable heartbeat | Bounded RAM queue, counted drops | A recent sample may be processed while this process survives | +| Important report | Durable inbox before acknowledgement, introduced in Section 13 | Work can be recovered after this process restarts | +| Command | Validate identity/deadline and record dedup/admission state | Device acceptance still needs an explicit application result | + +For this in-memory lesson, keep the loss limitation visible. Use a fake acknowledgement recorder to test ordering of admission and acknowledgement without pretending that the fake verifies broker behavior. + +### Predict the overload + +Two workers taking 100 ms per item can complete about 20 items/second before other overhead. A burst of 100 items arriving much faster cannot fit into two busy workers plus 16 waiting slots. Predict rejection, then measure actual accepted/completed/rejected counts; scheduler timing affects the exact split. Increasing workers may reorder effects for the same device. If revision order matters, serialize that device's work or enforce revision checks. + +Shutdown also has a dependency: if draining work must publish results, keep its MQTT connection alive while draining. Stop new admission first, drain accepted work with a deadline, then disconnect transport and join workers. If you disconnect first, record unsent results for recovery; an in-memory-only lesson cannot promise to deliver them. The signal context in the early example cancels AutoPaho immediately, so a real drain policy needs a separately controlled transport lifetime. + +
+Does returning false or an error prevent the broker from receiving PUBACK? + +Not in the pinned automatic callback path. These return values describe local handler processing. A deliberate acknowledgement policy uses the client's supported manual mechanism and recovery design. It must also handle invalid permanent failures so one bad message cannot occupy capacity forever. + +
+ +## 1. Trace callback and acknowledgement order + +Read `references/paho.golang-v0.23.0/paho/client.go`, especially receive callback processing, `EnableManualAcknowledgment`, and `Ack`. Write the observed order in `journal/06-concurrency.md`. Inspect this pinned source before choosing an acknowledgement policy. + +Section 4 copies payloads into a bounded channel and drops messages when full. Automatic MQTT acknowledgement can still occur before your application finishes work. This is a deliberate loss window, not reliable ingestion. + +## 2. Build a bounded dispatcher + +Create `internal/dispatch/pool.go` and `pool_test.go`. Start with two workers and capacity 16. Expose a context-aware submit operation and explicit accepted/full/stopping outcomes. Count accepted, rejected, completed, and failed work. Keep telemetry and important work in separate bounded queues so a heartbeat burst cannot occupy every command slot. + +Choose a policy per class: disposable telemetry may be dropped with a counter; commands and sales must produce an explicit admission failure rather than silently disappear. Admission failure under automatic MQTT acknowledgement still loses application work. Record this limitation until durable admission is introduced in Section 13. Do not claim manual acknowledgement alone makes an in-memory queue durable. + +## 3. Define ownership and shutdown + +Copy retained byte slices, including future correlation metadata. Protect shared maps with a mutex or give one goroutine ownership. Never hold a state mutex while publishing over the network. Use operation deadlines derived from the application lifecycle. + +On shutdown, stop admitting new work and let accepted work finish within a separate bounded drain context. Keep MQTT available during the drain if workers must publish results, then stop the transport producers. Cancel remaining work when the drain deadline expires. Never close a queue while a producer can still send; waiting for worker completion is part of shutdown. Document what happens to unfinished in-memory work. + +## 4. Integrate and stress it + +Edit the backend and simulator processing loops to submit validated work. Keep transport callbacks short. Add a configurable artificial handler delay and a finite synthetic burst generator; do not create one goroutine per incoming message. Keep these switches in typed TOML + koanf configuration and preserve existing precedence. + +```sh +mkdir -p internal/dispatch journal +gofmt -w internal/dispatch cmd +go test -race ./... +go vet ./... +``` + +Create `internal/dispatch` and `journal` before saving their files; the `mkdir -p` command is safe to run earlier than the final check block. `go test -race ./...` instruments the test binaries to report unsynchronized memory access during executed tests. It is slower than normal testing and cannot prove unexecuted paths are race-free. `go vet` checks static patterns; it does not replace the race run. + +A finite burst generator publishes a configured count and then stops; record its rate, queue capacity, worker count, and artificial delay. A bounded queue has fixed capacity: when workers are slower than arrival, overload becomes measurable instead of consuming memory indefinitely. Worker cancellation is cooperative: handlers must check contexts and use deadlines on blocking operations. Observe accepted/rejected counts and total shutdown time, not just whether the process eventually exits. + +Use channels in tests to hold a worker busy deterministically, fill the queue, and assert the next submission fails. Test cancellation, concurrent submissions, pending-work shutdown, and accounting. Avoid tests that rely only on sleeps. + +**Done when:** A burst causes observable bounded overload, normal ping/pong recovers afterward, shutdown finishes by its deadline, and the race detector passes. Explain the difference between broker acknowledgement, queue admission, and completed application work. + +Previous: [05-contracts-validation.md](05-contracts-validation.md). Next: [07-command-response.md](07-command-response.md). diff --git a/lessons/07-command-response.md b/lessons/07-command-response.md new file mode 100644 index 0000000..6bfe549 --- /dev/null +++ b/lessons/07-command-response.md @@ -0,0 +1,94 @@ +# Section 7 — Build reliable command and response behavior + +Prerequisite: Sections 5–6 validate messages and bound concurrent processing. + +Run all commands from `/home/ryan/Media/Code/Code/robot-market/mqtt-5-learning`. Create files at the project-relative paths shown below, not under `lessons/`. Continue the same application and pinned Paho v0.23.0 dependency. These are guided implementation exercises; attempt each step before requesting a full solution. + +## MQTT request/response is an application conversation + +Response Topic tells the responder where a result should be published. Correlation Data lets the requester recognize which pending request it belongs to. The broker forwards these properties; it does not run the command, create a result, or enforce your timeout. + +This fragment is the packet-construction portion of a service method, not a standalone program. `commandJSON` is an encoded validated command and `correlation` is owned by its pending record: + +```go +request := &paho.Publish{ + Topic: "lab/v1/devices/device-001/commands", + QoS: 1, + Retain: false, + Payload: commandJSON, + Properties: &paho.PublishProperties{ + ContentType: "application/json", + ResponseTopic: "lab/v1/devices/device-001/responses", + CorrelationData: append([]byte(nil), correlation...), + }, +} +``` + +Pass `request` to the bounded publication operation after registering pending state. The simulator validates `ResponseTopic`, publishes the result to it, and copies Correlation Data unchanged into the result's properties. Correlation bytes are opaque; they need not be text. Do not trust an arbitrary supplied topic as a response destination. + +### Trace a fast response and a lost response + +1. Backend establishes its response subscription and checks SUBACK. +2. Backend stores the expected device, command ID, correlation, and deadline. +3. Backend publishes. The simulator can respond before the publish call returns. +4. Backend validates and applies device evidence independently of the broker acknowledgement. +5. If the waiting deadline expires, backend removes the waiter and records unknown outcome. A later result follows the documented late-result policy. + +Registering after step 3 loses the race with a fast response. Holding the command-map mutex across publishing can block the callback needed to record that response. A context timeout stops a local wait; it cannot recall bytes already delivered to a device. + +| Identifier | Owner and lifetime | Why it cannot replace the others | +| --- | --- | --- | +| MQTT packet ID | Library, per protocol exchange/session direction, reusable | Identifies a transport handshake, not a permanent command | +| Correlation Data | Request/response mechanism | Matches a response; does not authenticate it or prevent repeated effects | +| Command ID | Application, stable across retries | Keys the dedup record and business status | + +### Make duplicate behavior observable + +Use a synthetic effect counter. Send command C twice with the same content, then C with changed content. Expect one effect, a replayed recorded result, then a conflict. Also send the same C in two separate publish calls: application duplicates need not carry MQTT DUP. The DUP flag describes protocol retransmission, so deduplicating only packets with DUP set is wrong. + +Concurrent duplicates need atomic admission: a lookup followed by an unlocked effect allows two workers to both observe "not seen". Reserve the ID with its fingerprint before executing, and define what a second request sees while the first is in progress. A real effect followed by a crash before storing its result remains a separate failure window; the in-memory simulator only demonstrates the idea. + +
+The broker acknowledged, but no device result arrived. Which status is justified? + +Only the transport acknowledgement is established. The device may be offline, may have rejected the request, may still be executing, or may have executed with a lost response. Once its deadline expires, expose unknown outcome. Retry only under the same business identity and a deliberate idempotency policy. + +
+ +## 1. Preserve MQTT 5 request metadata + +Create `internal/commands/service.go`, `store.go`, and `service_test.go`. Edit `internal/mqtt/client.go` to carry Response Topic and a copied Correlation Data slice in received messages. Add an outbound request form that accepts these properties. Retain the existing bounded publish timeout and reason-code checks. + +Read `references/paho.golang-v0.23.0/paho/cp_publish.go` for `paho.PublishProperties`. Set `ResponseTopic` to `lab/v1/devices/device-001/responses` and `CorrelationData` to opaque bytes associated with the pending command. Establish the backend response subscription before sending. Register pending state before publishing so an immediate response cannot beat registration. + +## 2. Model evidence explicitly + +Store command ID, expected device, correlation bytes, creation/deadline times, business status, and independent transport acknowledgement evidence. Start with created → pending delivery → accepted → completed or rejected. A ping simulator can send accepted followed by completed. Response timeout means unknown outcome; it does not establish that no effect happened. + +Protect concurrent access and define allowed transitions. A completed or rejected command must not regress on a duplicate accepted response. Compare command identity, topic/body device identity, and correlation before applying a response. Keep timed-out records for a bounded retention period even after removing their waiters, so status can explain the uncertainty. + +## 3. Make the simulator idempotent + +Create `internal/simulator/dedup.go` and its tests. Index effects by stable command ID and retain a fingerprint of the command. For an identical duplicate, replay the recorded result without repeating the effect. Reject reuse of the ID with different content. Bound cache lifetime and size; state clearly that restarting the simulator loses this in-memory protection. + +Allow a Response Topic only when it exactly matches the authorized response topic for this simulator. Correlation is matching data, not authorization. Continue rejecting retained one-shot commands. + +## 4. Exercise timing and failures + +Replace the backend's unconditional ticker with an explicit finite command exercise or service calls. Send 20 concurrent synthetic pings with separate IDs. Introduce delayed, duplicate, wrong-device, wrong-correlation, and out-of-order responses. Check every command receives only its own evidence. Retry a timed-out request with the same stable ID; never generate a new ID merely because publication timed out. + +```sh +mkdir -p internal/commands internal/simulator +gofmt -w internal cmd +go test -race ./... +``` + +Run `mkdir -p` before saving the service and simulator package files. `gofmt` formats your edits and `go test -race ./...` executes the module's tests with concurrency instrumentation. A fake transport is a small in-process implementation of the caller's transport interface: it lets tests deliberately return errors or responses without needing a broker. An injected clock makes deadline tests deterministic without waiting in real time. + +When copying Correlation Data, allocate a new byte slice so later processing owns stable bytes. Comparing correlation bytes means comparing their contents, not pointer addresses. Before updating a shared command map, validate the response and hold its lock only for the state transition. Publish outside the lock so a network delay cannot freeze every command. + +Test responses arriving before publish returns, publish failure with uncertain outcome, timeout cleanup, deduplication, and terminal-state protection. Use an injectable clock and fake transport for deterministic service tests. + +**Done when:** Concurrent requests match correctly, identical duplicates cause one effect, unexpected destinations are rejected, and late responses follow a documented policy. Explain why PUBACK and Correlation Data cannot prove business completion. + +Previous: [06-concurrency-cancellation.md](06-concurrency-cancellation.md). Next: [08-http-commands.md](08-http-commands.md). diff --git a/lessons/08-http-commands.md b/lessons/08-http-commands.md new file mode 100644 index 0000000..293d633 --- /dev/null +++ b/lessons/08-http-commands.md @@ -0,0 +1,77 @@ +# Section 8 — Expose device commands through HTTP + +Prerequisite: Section 7 provides a command service and queryable in-memory states. + +Run all commands from `/home/ryan/Media/Code/Code/robot-market/mqtt-5-learning`. Create files at the project-relative paths shown below, not under `lessons/`. Continue the same application and pinned Paho v0.23.0 dependency. These are guided implementation exercises; attempt each step before requesting a full solution. + +## Translate evidence between HTTP and MQTT + +HTTP has one response to this request. MQTT has an asynchronous stream where the device may reply after the HTTP connection closes. Returning 202 after admission gives the caller a stable command ID; a later GET reports evidence accumulated by the command service. + +```text +POST + retry key → admit command C → HTTP 202 + C + → MQTT publication → device result → status C +GET C --------------------------------------------------------------→ status C +``` + +Do not convert PUBACK directly into HTTP "device completed". Also do not detach work using an unbounded `context.Background()` per command: accepted work needs the service lifecycle, its own deadline, and tracked shutdown. The HTTP request context controls parsing and admission, while the service controls work after acceptance. + +Predict a lost HTTP response: the server admits C, but the caller never sees 202 and retries. The same retry key and payload should return C, not create D. Reusing the key with a changed payload is a conflict. Later, scope keys by authenticated caller so two users cannot accidentally share a command record. + +
+If the backend restarts after returning 202, can this lesson recover C? + +No. The lesson's in-memory command and retry-key maps disappear. HTTP 202 describes admission under the current implementation, not disk durability. Sections 13–15 add database admission/outbox and process recovery; until then, document this restart limit in the lab API. + +
+ +## 1. Add HTTP configuration and handlers + +Create `internal/httpapi/server.go`, `handlers.go`, and `handlers_test.go`. Extend `internal/config/config.go` and `configs/lab.toml` with a loopback listener such as `127.0.0.1:18080`, request-body limit, and server timeout settings. Preserve defaults < TOML < environment < explicit CLI flags. Validate these settings before starting listeners. + +Compose HTTP and MQTT in the existing `cmd/backend/main.go`. Keep the simulator separate. Section 15 introduces separate API and ingestion processes. + +## 2. Define the API contract + +Implement `POST /devices/{id}/commands` for a JSON command such as `{"command":"ping"}` and `GET /commands/{id}` for status. Limit bodies, reject trailing JSON and unknown fields, validate device IDs, and return JSON errors consistently. Keep the listener local; this lesson does not implement production user authorization. + +Return HTTP 202 with a command ID and a Location pointing to its status after service admission. Document malformed-input 400, oversized-body 413, unknown-command/device 404 or a consistent chosen alternative, conflict 409, and admission-unavailable 503 behavior. Transport uncertainty after acceptance belongs in command status rather than silently discarding an accepted command. + +## 3. Implement HTTP retry identity + +Require an `Idempotency-Key` header for POST. Map the key and request fingerprint to the existing command ID. An identical retry returns the same ID; changed content under the same key returns 409. Serialize concurrent admission for the same key. Bound key retention and document that process restart loses this in-memory guarantee. This key, the stable device command ID, and MQTT packet IDs serve separate purposes. + +Use the HTTP context for request parsing and admission. Once accepted, execute under the application's lifecycle and command deadline, so client disconnect does not cancel the command. Inject the command service through a small interface and keep MQTT calls out of handlers. + +## 4. Run and verify + +Start the simulator and backend using Section 4's separate terminal credentials. In a third terminal: + +```sh +curl -i -X POST http://127.0.0.1:18080/devices/device-001/commands -H 'Content-Type: application/json' -H 'Idempotency-Key: lesson8-ping-001' --data '{"command":"ping"}' +``` + +`curl` is an HTTP client. `-i` prints response headers alongside the body; `-X POST` chooses the method; each `-H` sets a request header; `--data` sends the quoted JSON bytes. Content-Type tells the server how to interpret the body, while Idempotency-Key identifies this caller retry. This local exercise sends no production authorization token. + +Copy the returned ID into `GET /commands/{id}`. Repeat the identical POST and confirm the same ID. Use a new key for a genuinely new command. + +For example, if the returned ID is `cmd-example`, use `curl -i http://127.0.0.1:18080/commands/cmd-example`; without `--data` or an explicit method, curl performs GET. Replace that example ID with the actual returned value. HTTP 202 means admitted for asynchronous processing; inspect later status for device completion. `httptest` runs handlers against an in-process request/recorder or test server, letting you assert headers, status, and body without the live broker. + +Set ReadHeaderTimeout, ReadTimeout, WriteTimeout, and IdleTimeout deliberately. Coordinate bounded HTTP shutdown with the MQTT and worker lifecycle. + +```sh +mkdir -p internal/httpapi +gofmt -w internal cmd +go test -race ./... +go vet ./... +``` + +ReadHeaderTimeout bounds header reading; ReadTimeout bounds reading the request including its body; WriteTimeout bounds response writing; IdleTimeout bounds waiting for another request on an idle keep-alive connection. Choose values that fit this asynchronous API rather than waiting in POST for a device result. Server `Shutdown` stops accepting work and waits for active HTTP requests within its context deadline; your application must separately drain accepted command work. + +`mkdir -p` creates the HTTP package directory, `gofmt` formats edits, and the test/race/vet commands check behavior, executed concurrent access, and suspicious static patterns respectively. Run directory creation before writing the files even though the check block collects the commands together. + +Use `httptest` with a fake service for decoding, size limits, idempotency, unavailable admission, unknown IDs, and status responses. Add a service-level check that accepted work continues after the caller context is canceled. + +**Done when:** HTTP creates a ping, status reflects device evidence, retries reuse its ID, and closing the caller connection does not erase accepted work. + +Previous: [07-command-response.md](07-command-response.md). Next: [09-sessions-recovery.md](09-sessions-recovery.md). diff --git a/lessons/09-sessions-recovery.md b/lessons/09-sessions-recovery.md new file mode 100644 index 0000000..32fe19c --- /dev/null +++ b/lessons/09-sessions-recovery.md @@ -0,0 +1,98 @@ +# Section 9 — Learn MQTT 5 sessions through failure experiments + +Prerequisite: Section 8 exposes commands and their evidence; use only synthetic ping effects. + +Run all commands from `/home/ryan/Media/Code/Code/robot-market/mqtt-5-learning`. Create files at the project-relative paths shown below, not under `lessons/`. Continue the same application and pinned Paho v0.23.0 dependency. These are guided implementation exercises; attempt each step before requesting a full solution. + +## A connection is temporary; a session may outlive it + +The network connection carries packets now. A broker session can preserve subscriptions and eligible pending deliveries after that connection closes. Your Go command map lives in a different place and is not restored by CONNACK. + +| Clean Start on this connection | Prior unexpired session under the same Client ID | Expected Session Present | Meaning | +| --- | --- | --- | --- | +| true | Either | false | Discard old state and begin a new session | +| false | Yes | true | Resume that broker session | +| false | No | false | Create a new session; restore subscriptions | + +Session Expiry answers a later question: how long may the session remain after disconnect? Setting Clean Start true with expiry 60 creates a new session that can subsequently survive a short outage. Setting Clean Start false with expiry 0 can resume an existing session now, but asks for no persistence after this connection ends. They are independent controls, not two names for the same switch. + +In AutoPaho v0.23.0, `CleanStartOnInitialConnection` affects the manager's first successful connection; automatic reconnects use false. Restarting the Go process creates a new manager and applies the initial choice again. Therefore "network reconnect works" does not establish "process restart resumes the old session." + +### Work through an offline command + +At t=0 the simulator has subscribed at QoS 1 with expiry 60. At t=5 it disconnects. At t=10 the backend publishes a nonretained QoS 1 ping. At t=20 the same simulator ID reconnects with Clean Start false. Assuming the broker retained the session, accepted the publication, and no shorter message expiry or resource limit intervened, the subscription allows queued delivery. Reconnect after the session expires and the old queue is gone. + +Creating a new subscriber after publication is different: it had no prior subscription/session to receive that queued nonretained message. Retained state, studied next, is associated with a topic and can initialize a new subscriber. + +### Put each recovery responsibility in the correct process + +| State | Early lesson implementation | Network disconnect | Owning process restart | +| --- | --- | --- | --- | +| Client protocol session | Paho memory state for inflight handshakes | Manager can reuse state subject to broker session | Lost unless persistent client stores are configured | +| Client outgoing queue | Early code uses direct `Publish`, not `PublishViaQueue` | Direct publish can fail while disconnected | No durable outgoing backlog exists in this helper | +| Broker session | Broker memory plus configured persistence, gated by expiry | Can preserve subscription/eligible deliveries | Depends on saved broker state, expiry, and recovery | +| Application records | Go command and dedup maps | Survive while the Go process lives | Lost until database-backed recovery is implemented | + +`PublishViaQueue` is a separate AutoPaho API. A nil result means local enqueue succeeded; its API does not return eventual broker/device status. A file-backed outgoing queue and file-backed protocol session stores solve different problems. Neither replaces an application outbox or device deduplication. + +
+Session Present is true, but GET command status says unknown. Is the broker broken? + +Not necessarily. The broker resumed its session while the backend's in-memory application records were lost on process restart. MQTT has no knowledge of your HTTP command map. Record both pieces of evidence rather than using Session Present as a general recovery-success flag. + +
+ +## 1. Add session controls + +Extend the typed configuration with Clean Start and Session Expiry Interval in seconds, and pass them into AutoPaho. Edit `internal/mqtt/client.go` to log `SessionPresent` from CONNACK without logging credentials. Record connection and subscription readiness separately. Read the pinned `autopaho/auto.go`: `CleanStartOnInitialConnection` applies to the first connection of that manager; automatic reconnects behave differently from constructing a new manager. + +Use unique stable IDs for backend and simulator. Run one process per client ID. A changing ID creates a different session, while simultaneous clients sharing an ID can take over each other's connection. + +Session Expiry is seconds in the MQTT property, whereas Go operation deadlines use `time.Duration`. A nonzero expiry preserves eligible broker session state after disconnect for that interval; Clean Start discards an old session at connection start. Session Present reports whether the broker resumed prior session state, not whether your Go command records survived. + +Create the `journal` directory before saving the experiment record. Use Ctrl+C for a graceful simulator shutdown first; log whether DISCONNECT was sent and whether your configured expiry remains nonzero. For an abrupt-process experiment, run a built simulator binary and record its PID, then terminate that exact lab process; killing a `go run` wrapper can make the process under test ambiguous. Record the shutdown method because a graceful disconnect and lost network connection are different evidence. + +## 2. Run an experiment matrix + +Create `journal/09-sessions.md`. For each row record settings, prediction, exact stop/start steps, CONNACK Session Present, SUBACK, received command IDs, duplicates, and observed result. + +| Experiment | Simulator setup | Observation to explain | +| --- | --- | --- | +| Fresh connection | Clean Start true, expiry 0 | No offline-session guarantee | +| Short outage | Clean Start false, expiry 60, stable ID | Reconnect before expiry and inspect queued QoS 1 delivery | +| Expired session | Same settings, outage longer than 60 seconds | Inspect Session Present and missing queued work | +| New identity | Change only client ID | Previous session is not this new client's session | +| Broker restart | Persistent lab broker | Compare network recovery with broker-process recovery | + +For offline delivery, first establish the simulator subscription at QoS 1. Disconnect it while preserving a nonzero-expiry session, then publish a nonretained QoS 1 ping from the still-connected backend. Reconnect before expiry using the same ID and Clean Start false. Observe broker evidence; do not infer delivery merely from a successful backend publish. Choose a command response deadline long enough for the experiment, and record late-result policy when exceeding it. + +## 3. Audit subscription restoration + +The early helper subscribes on every connection-up event. Keeping this policy is acceptable for the experiment; log each SUBACK and do not register another local callback on every reconnect. Test both session-present and fresh-session paths. A connected process whose subscriptions failed is not ready to receive commands. + +## 4. Compare broker persistence safely + +Use only this project's Compose commands. Keep the default persistence volume intact. For a nonpersistent comparison, stop the lab, temporarily change only `lab/mosquitto/mosquitto.conf` to disable persistence, and use a separate empty lab data volume in a Compose override so old state cannot contaminate the result. Restore the original configuration afterward. Do not delete volumes to run this experiment. + +A graceful broker shutdown and an abrupt broker kill may preserve different amounts of recently changed state. Record which you used and the broker's persistence settings rather than generalizing one observation to every crash. + +A Compose override is an additional YAML file layered over `compose.yaml` using `-f compose.yaml -f ` before the subcommand. Create it at the root and use that same pair for config/up/ps/stop commands throughout the comparison. Replace the broker's `/mosquitto/data` source with a new declared named volume; Compose merges mount entries by container destination. Verify the resolved `config` output shows the new source before starting it. Stop the comparison, then run `up -d` with the original file alone to restore the original mounts; `start` would only reuse the comparison container. Refer to [Broker operations](broker-operations.md) for volume identity and recreation. + +Turning persistence off does not erase existing in-memory sessions or remove old disk data. Use the separate empty volume and a recreated container to avoid that ambiguity. After restoring settings and mounts, inspect logs and repeat normal authentication/ping checks. + +## 5. Separate the recovery layers + +Write a table with four columns: client session storage, client outgoing publication queue, broker session state, and application command/dedup state. For each, state current storage, lifetime, and what survives network interruption or process restart. Read the pinned Paho session and AutoPaho queue documentation; neither broker persistence nor QoS 1 makes your in-memory command records durable. + +```sh +gofmt -w internal cmd +go test -race ./... +docker compose -p mqtt5-learning restart broker +docker compose -p mqtt5-learning logs --tail=50 broker +``` + +`restart broker` stops/starts only that project's broker container, interrupting its client connections; it does not apply edited Compose mounts or image settings. `logs --tail=50 broker` inspects the last 50 service log lines. Observe client reconnection and Session Present as well as broker logs. `gofmt` rewrites edited Go formatting; `go test -race` exercises tests with race detection. For mount/config changes requiring recreation, follow the preceding override instructions instead of assuming restart applies them. + +**Done when:** Every matrix row has a prediction and measured result, you can distinguish automatic reconnect from a new process, and you can explain each recovery layer. An offline ping delivered after your backend timed out must remain consistent with Section 7's unknown-outcome policy. + +Previous: [08-http-commands.md](08-http-commands.md). Next: [Section 10 — Expiry, presence, and retained state](10-expiry-presence-state.md). diff --git a/lessons/10-expiry-presence-state.md b/lessons/10-expiry-presence-state.md new file mode 100644 index 0000000..4b3d592 --- /dev/null +++ b/lessons/10-expiry-presence-state.md @@ -0,0 +1,104 @@ +# Section 10 — Expiry, presence, and retained state + +Prerequisite: complete the session experiments in Lesson 9. Continue the same lab and pinned Paho v0.23.0. These are incremental exercises; the fragments below belong in your existing transport and simulator, not new standalone programs. + +## 1. Give every timer one responsibility + +| Timer | What expires | What it cannot prove | +| --- | --- | --- | +| Go publish context | Your wait for the publication operation | That the device never received it | +| Application command deadline | Permission to begin a stale action under your contract | That an already-started action was undone | +| MQTT Message Expiry | A publication's remaining delivery lifetime | That a delivered message stops executing | +| Session Expiry | Broker session after disconnect | That application records were saved | +| Keep Alive | Acceptable MQTT connection inactivity | That the application is healthy | +| Will Delay | Delay before an eligible Will is published | That the device is physically broken | + +A vending configuration command that was useful five minutes ago may be wrong now. Use Message Expiry to limit stale delivery and an application deadline to reject stale execution after delivery. A response timeout merely tells the caller it lacks a result. + +This fragment builds MQTT metadata for a five-second publication lifetime: + +```go +expirySeconds := uint32(5) +properties := &paho.PublishProperties{ + ContentType: "application/json", + MessageExpiry: &expirySeconds, +} +``` + +Attach `properties` to the outbound PUBLISH. A nil MessageExpiry omits the property; an explicit zero is not an unlimited lifetime. The broker reduces the remaining lifetime when forwarding queued work. Time spent waiting in your own outgoing queue needs a separate policy: check the absolute application deadline before transmission and execution. Do not reissue an old command with a fresh five seconds indefinitely. + +**Experiment:** preserve the simulator session for 60 seconds, disconnect it, publish one five-second QoS 1 ping, and reconnect after ten seconds. Expect no expired command delivery. Repeat with a much longer message expiry and reconnect within it: expect delivery if the other session/ACL prerequisites hold. Log IDs and timestamps, not just a message count. Allow timing margin around expiry boundaries. + +## 2. Retained state is a latest value, not an event history + +A retained publication replaces the stored value for one exact topic. It can initialize a newly created subscription even when that client has never connected before. Offline session queues instead belong to an existing session and its subscriptions. Unsubscribing or deleting a session does not delete a retained value. + +Use a desired-configuration topic for state such as revision 7 and a temperature setting. Use nonretained command topics for one-shot actions. A retained "dispense" would risk replay when someone subscribes later. + +Extend the ACL deliberately before introducing topics: backend writes `lab/v1/devices/+/desired`, simulator reads its exact `.../device-001/desired`; simulator writes `.../device-001/reported`, backend reads `.../+/reported`. Reload/restart the lab and verify both allowed and denied directions. Extend Lesson 5's topic parser and contracts as well. + +Implement desired and reported revision records. Store the highest applied revision in the simulator and explain the initial in-memory restart limit. Receiving revision 6 after 7 must not overwrite 7. A backend PUBACK for revision 7 proves neither application nor reported revision 7; wait for the device's report. + +### Clear the retained telemetry from Lesson 4 + +Stop the simulator's periodic telemetry first so it cannot replace the value during your observation. Run this with the simulator's temporary lab password, using the same CLI credential caveat as Lesson 1: + +```sh +mosquitto_pub -h 127.0.0.1 -p 18883 -V mqttv5 -u simulator -P '' -t 'lab/v1/devices/device-001/telemetry' -q 1 -r -n -d +``` + +`-r` sets RETAIN; `-n` supplies a zero-byte payload. Together they delete the retained value at that exact topic. An empty JSON object or the two characters `""` are nonempty payloads and do not delete it. Existing subscribers can see the empty publication; a fresh subscriber should receive no retained replay. Removing the stored value does not retract copies already delivered or queued elsewhere. Restore ordinary nonretained telemetry after checking. + +## 3. Model presence as evidence + +The simulator supplies its Will during CONNECT. The broker later publishes it when the connection ends in a way that requires a Will; it is not code that the dead simulator executes. First add simulator write/backend read ACLs for its `.../presence` topic and teach the topic parser about it. + +Add these fields to the simulator's AutoPaho configuration, alongside its existing settings: + +```go +WillMessage: &paho.WillMessage{ + Topic: "lab/v1/devices/device-001/presence", + QoS: 1, + Retain: true, + Payload: []byte(`{"status":"offline"}`), +}, +WillProperties: &paho.WillProperties{ + WillDelayInterval: paho.Uint32(5), + ContentType: "application/json", +}, +``` + +`paho.Uint32(5)` supplies the optional MQTT property as a pointer; the unit is seconds. This is a minimal presence payload for the experiment. Extend it with your schema and a connection/boot identity for the real lesson implementation. A Will payload is prepared at connection time: a timestamp embedded there is not the later time of disconnection. Record backend receipt time separately. + +Publish retained online presence after successful simulator startup. Publish an explicit retained offline value before a planned disconnect if observers need that signal. Normal DISCONNECT suppresses the Will, so otherwise a retained online value can remain after a clean shutdown. A special MQTT 5 disconnect reason can request the Will, but the early helper uses the normal path. + +Will publication occurs when its delay elapses or the session ends, whichever happens first. With Session Expiry 0, ending the session can remove the intended five-second grace period. With a preserved session, resuming that session before the delay ends can suppress the pending Will. Test with stable identity, Clean Start false, and a session expiry longer than the delay. + +### Separate experiments that look similar + +| Action | What it tests | Prediction | +| --- | --- | --- | +| Graceful simulator shutdown | DISCONNECT and explicit offline policy | Normal disconnect does not itself publish the Will | +| Abruptly terminate the exact simulator binary | Lost client without DISCONNECT | Broker may publish Will after detection and applicable delay | +| Reconnect into preserved session before Will Delay | Transient outage | Pending Will can be suppressed | +| Stop broker | Broker availability | A stopped broker cannot publish Wills while it is down | +| Keep MQTT traffic alive but stall heartbeat production | Application health | Keep Alive can succeed while freshness fails | + +For an abrupt test, build a dedicated lab binary with `go build -o /tmp/mqtt5-lab-simulator ./cmd/simulator`, run it with your usual simulator flags, and record its PID. Send `kill -KILL ` from another terminal only after verifying that PID. KILL prevents Go cleanup and DISCONNECT; it is different from Ctrl+C. Do not target all Go processes or assume a `go run` wrapper is the actual client. Restart the binary afterward and inspect both retained presence and fresh heartbeats. + +A killed process usually closes its socket promptly. It does not reproduce a silent network black hole that must be detected through Keep Alive. Record the actual fault injected and avoid claiming the measured delay is a universal offline-detection time. + +## 4. Explain the result before moving on + +
+A fresh subscriber receives retained online, but no heartbeat follows. Is the machine healthy? + +You only know the broker stored an online assertion. The producer may have exited cleanly without replacing it, or the retained value may be old. Combine presence with receipt time and heartbeat freshness; use connection/revision identity to reject stale evidence. + +
+ +**Done when:** demonstrate an expired offline command, retained replacement and deletion, graceful versus abrupt presence behavior, and rejection of an older desired revision. Record predictions and observations in `journal/10-expiry-presence.md`. + +Sources: [MQTT 5 packet and session rules](https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html), [Mosquitto publication flags](https://mosquitto.org/man/mosquitto_pub-1.html), and local `references/paho.golang-v0.23.0/paho/cp_publish.go`, `paho/cp_connect.go`, and `autopaho/auto.go`. + +Previous: [Lesson 9](09-sessions-recovery.md). Next: [Lesson 11](11-security-boundaries.md). diff --git a/lessons/11-security-boundaries.md b/lessons/11-security-boundaries.md new file mode 100644 index 0000000..3d43770 --- /dev/null +++ b/lessons/11-security-boundaries.md @@ -0,0 +1,71 @@ +# Section 11 — Understand MQTT security boundaries + +Continue the same isolated lab. This lesson explains the MQTT security design and gives integration exercises; certificate provisioning depends on your chosen lab hostname. Keep remote deployment separate until these checks work. + +## 1. Trace the checks in order + +```text +TCP → TLS certificate verification → MQTT CONNECT authentication + → topic authorization → application identity/schema validation + → permission to perform the business operation +``` + +TLS protects the connection and verifies the server identity. A username/password authenticates an MQTT client. ACLs restrict topic actions. Application validation checks the payload. HTTP user authorization determines whether an API caller may command that device. Passing one check never substitutes for the next. + +A Client ID is session identity, not a secret. A payload `device_id` is a claim. The broker normally does not attach the authenticated publisher username to each forwarded PUBLISH, so the backend cannot authenticate the sender merely by reading a JSON field. Bind a unique device credential to that device's permitted topics and validate topic/body agreement. + +## 2. Configure verified TLS in Go + +Provision a lab CA and server certificate for the exact DNS name or IP used in the client URL. An IP connection needs that IP in the certificate's Subject Alternative Name. Keep the CA private key outside the broker; the broker needs its server key and certificate. Mount files with broker-readable permissions. Configure Mosquitto's TLS listener using `certfile` and `keyfile`; require client certificates only when deliberately introducing mutual TLS. + +This fragment belongs in your configuration/transport setup and requires `crypto/tls`, `crypto/x509`, `fmt`, and `os`. `caPath` is a configured path to the public lab CA certificate and `brokerHostname` is the validated URL hostname (without a port): + +```go +caPEM, err := os.ReadFile(caPath) +if err != nil { + return fmt.Errorf("read lab CA: %w", err) +} +roots := x509.NewCertPool() +if !roots.AppendCertsFromPEM(caPEM) { + return fmt.Errorf("lab CA contains no usable certificate") +} +tlsConfig := &tls.Config{ + RootCAs: roots, + ServerName: brokerHostname, + MinVersion: tls.VersionTLS12, +} +``` + +Assign this configuration to AutoPaho's `TlsCfg` and use a `tls://` broker URL for the TLS listener. Set `ServerName` explicitly from the validated URL host so certificate verification also has the intended identity on the pinned proxy transport path. Do not disable verification to make a hostname failure disappear; correct the certificate or URL. Update Lesson 3's URL validation deliberately: its early allowlist accepts only the plaintext `mqtt://` scheme. + +Server-only TLS can still use the existing MQTT credentials and ACLs. Mutual TLS additionally authenticates a client certificate; mapping certificate identity to topic permissions is another deliberate broker policy. Do not assume enabling encryption creates per-device authorization. + +Use the official [Mosquitto TLS configuration reference](https://mosquitto.org/man/mosquitto-conf-5.html) and [Go TLS configuration API](https://pkg.go.dev/crypto/tls#Config) while provisioning. Match broker options to the pinned 2.0.22 image. + +## 3. Test failures at the intended layer + +| Change one input | Expected failure layer | Evidence to collect | +| --- | --- | --- | +| Wrong CA | TLS verification | No authenticated MQTT session established | +| Valid CA, wrong server name | TLS identity verification | Hostname/IP mismatch | +| Valid TLS, wrong MQTT password | CONNECT authentication | Connection error / unsuccessful CONNACK | +| Valid login, other device's publication topic | Topic authorization | Rejection/delivery denial plus authorized observer and broker logs | +| Permitted topic, mismatched body device | Application validation | Rejection before any state mutation | +| Valid API user, unauthorized target device | HTTP business authorization | Denial before creating/publishing a command | + +Never fix an ACL test by replacing narrow permissions with `#`. A successful broad SUBACK alone is not proof of cross-device access: test actual delivery. Restrict Response Topic before replying so a device cannot be used to publish a response into an unexpected namespace. + +## 4. Rotate credentials without confusing state + +Updating a password file affects authentication policy; it does not prove every existing connection was disconnected. A fresh connection using the old password must fail, a new one must succeed, and the device must regain its subscriptions. [Broker operations](broker-operations.md) explains which credential volume this workspace actually uses and why editing the old host copy does nothing. + +Treat MQTT 5 enhanced authentication (`AUTH` plus an agreed authentication method) as an optional separate extension. It requires compatible client and broker mechanisms; adding User Properties or renaming a username does not enable it. The two-user password-file lab does not demonstrate enhanced authentication. + +
+Why is TLS with a shared fleet password still insufficient for per-device isolation? + +TLS protects traffic, but a shared credential does not distinguish individual devices for authorization. A compromised device can exercise that shared identity's permissions. Use separate identities and topic policies, then validate application claims independently. + +
+ +**Done when:** explain every boundary, run the failure matrix against the lab, and show that a denied user or device cannot produce a simulated effect. Continue with Section 12 in [COURSE.md](../COURSE.md); the [advanced MQTT lesson](16-mqtt5-controls.md) belongs after the persistence/process exercises. diff --git a/lessons/16-mqtt5-controls.md b/lessons/16-mqtt5-controls.md new file mode 100644 index 0000000..c2cc7a9 --- /dev/null +++ b/lessons/16-mqtt5-controls.md @@ -0,0 +1,105 @@ +# Section 16 — MQTT 5 controls you can explain and measure + +Study this after Sections 12–15 in [COURSE.md](../COURSE.md), or read it earlier as a conceptual preview. Continue using Paho v0.23.0 and the isolated broker. Each experiment changes one behavior; record the broker/client versions and observed result instead of assuming a field's existence proves complete support. + +## 1. Subscription options change what arrives + +These fields belong to each `paho.SubscribeOptions` entry, not to the PUBLISH payload: + +| Field | Meaning | Experiment | +| --- | --- | --- | +| `NoLocal` | Suppress publications originating on the same client connection identity | With a lab identity authorized to publish and subscribe to one topic, compare false/true | +| `RetainAsPublished` | Preserve the incoming publication's RETAIN bit when forwarding live messages | Compare a live retained publication with replay after resubscription | +| `RetainHandling: 0` | Send matching retained state when subscribing | Resubscribe and observe replay | +| `RetainHandling: 1` | Send retained state only when this subscription is new | Repeat the same subscription in the existing session | +| `RetainHandling: 2` | Do not send retained state at subscription time | Subscribe after storing a retained value; then publish a live update | + +Retain Handling 2 does not prevent future live retained publications. Retain As Published does not decide whether the broker stores retained state. No Local is not an ACL or a device-ID filter, and must not be set for shared subscriptions. Lesson 4 deliberately enables Retain As Published so the simulator's retained-command guard also works for live forwarding. + +An UNSUBSCRIBE removes a broker subscription after its UNSUBACK exchange; it does not unregister your local Go callback or erase retained values. Use `manager.Unsubscribe(ctx, &paho.Unsubscribe{Topics: []string{filter}})` with an operation deadline and inspect the returned reasons. `filter` is the exact previously subscribed filter. Already delivered work may still be in the local queue. + +## 2. Shared subscriptions distribute work + +Ordinary subscriptions on two clients each receive a matching publication. With `$share/ingest/lab/v1/devices/+/telemetry`, both clients join the `ingest` group for that filter and the broker selects a group member for a matching delivery. Give each process a unique Client ID; sharing a group is different from sharing a Client ID. + +Create two authorized lab consumers, check their SUBACK results, then publish numbered synthetic telemetry. Record which instance handles each ID. Stop one consumer and repeat. Use a fresh lab group/session for the experiment so old offline state does not obscure the result. Verify the broker's shared-subscription support and effective ACL behavior before drawing conclusions. + +Distribution is not guaranteed round-robin, per-device affinity, or business exactly-once execution. A restarted consumer can repeat work, and parallel workers can finish out of order. Shared consumers need shared durable deduplication for important reports. Keep command-response routing deliberate: a response delivered to another instance will not find a waiter stored only in your instance's RAM. + +## 3. Distinguish wire limits from application limits + +The CONNECT properties describe what this client can receive. The CONNACK properties describe what the server can receive or supports. Direction matters: changing the client's Receive Maximum constrains broker-to-client QoS 1/2 inflight publications, not the backend's outgoing command rate. + +Add this field to your existing `autopaho.ClientConfig`; it is a fragment, with `net/url` and Paho imports supplied by that file: + +```go +ConnectPacketBuilder: func(cp *paho.Connect, _ *url.URL) (*paho.Connect, error) { + if cp.Properties == nil { + cp.Properties = &paho.ConnectProperties{} + } + receiveMaximum := uint16(4) + maximumPacketSize := uint32(32768) + cp.Properties.ReceiveMaximum = &receiveMaximum + cp.Properties.MaximumPacketSize = &maximumPacketSize + return cp, nil +}, +``` + +The nil check preserves AutoPaho's existing properties, including configured session expiry. Replacing the whole property struct would silently discard them. Receive Maximum must be nonzero. Maximum Packet Size counts the complete MQTT packet, including topic and properties, not only JSON payload. Paho maps these fields to CONNECT on each connection. + +In `OnConnectionUp`, inspect `ack.Properties` safely and record server limits/capabilities such as Receive Maximum, Maximum Packet Size, Maximum QoS, and retained/shared-subscription availability. A nil optional pointer is absence with a protocol-defined default, not automatically zero. Do not print the complete authentication configuration as a shortcut. + +**Experiment:** compare QoS 1 inflight traffic with automatic acknowledgements and with Lesson 6's deliberately delayed manual acknowledgements. A queue of 16 can still fill even with Receive Maximum 4 if automatic ACK frees inflight slots as soon as callbacks return. Repeat at QoS 0: Receive Maximum is not its backpressure mechanism. Bound application queues and broker resources separately. + +For packet-size testing, use harmless payloads below and above a deliberately small configured limit. Record whether the sender library rejects locally, the peer rejects/disconnects, or delivery is absent; the precise path depends on direction and support. No callback arrival alone does not identify the failure layer. + +## 4. Properties carry context, not trust + +User Properties are UTF-8 key/value pairs and may repeat a key. Use them for synthetic trace context or schema hints, with size/count limits. They are not authenticated claims by themselves and are not HTTP headers that the broker interprets for your application. + +```go +properties := &paho.PublishProperties{ + User: paho.UserProperties{ + {Key: "trace-id", Value: "lab-trace-001"}, + {Key: "producer", Value: "simulator"}, + }, +} +``` + +Attach this fragment to a publication and copy the metadata you need at your receive boundary. Change `producer` to `backend` while publishing as the simulator: the string changes, but the authenticated identity does not. Avoid putting secrets in properties because observers may log them. + +Subscription Identifiers let a subscriber label a subscription and identify which subscription(s) matched a received message. They do not identify the publisher or deduplicate business events. The pinned public `PublishProperties.SubscriptionIdentifier` is a single `*int`; overlapping-subscription cases require checking the library's representation/support rather than assuming it exposes every possible identifier as a list. + +## 5. Topic aliases optimize one connection + +An alias associates a small integer with a topic name for one direction of one connection. The sender first establishes the mapping with the topic and alias; subsequent publications may use the alias to avoid resending the long topic string. This reduces bytes, not permission checks or payload size. + +The receiver advertises its Topic Alias Maximum. Zero or absent support must not be treated as permission to send arbitrary aliases. A reconnect loses the mapping even when an MQTT session resumes. Never send an alias-only first publication after reconnect. + +Read the pinned `paho/extensions/topicaliases/` implementation and its tests, plus the receiving path in `paho/client.go`, before enabling aliases. The extension uses a publish hook to manage outbound mappings. Integrate reset/recreation into the connection lifecycle and verify both initial and reconnected traffic; the presence of a `TopicAlias` field alone is insufficient evidence. Keep the lab's default full-topic publications until that experiment passes. + +## 6. Read failures as protocol evidence + +Reason codes belong to a packet context. A SUBACK contains a result for each filter. A QoS 1 PUBACK describes the immediate publication hop. A DISCONNECT may describe why the broker ended a connection. None is a device business status. + +| Observation | Interpretation | Next action | +| --- | --- | --- | +| Publish response `0x10` | No matching subscribers; a non-error MQTT result | Check peer session/subscription and await application evidence | +| Reason code at least `0x80` in these ACK results | Protocol-level failure | Classify the packet/reason before deciding to retry | +| Connection error without MQTT reason | May have failed before a packet response arrived | Inspect network/TLS and broker evidence | +| Context deadline exceeded | Local wait ended | Preserve outcome uncertainty if transmission may have happened | + +Use bounded reconnect backoff with jitter for transient failures so many devices do not reconnect in lockstep after an outage. The pinned `ReconnectBackoff` field accepts a function of attempt number; inspect the tagged `backoff` example before selecting parameters. Authentication/configuration errors need repair or escalation, not a faster retry loop. Reconnect retries a connection; it does not authorize repeating a business action under a new ID. + +
+Why can Receive Maximum 4 coexist with 100 queued application items? + +The inflight quota counts unacknowledged protocol deliveries, not all work the application has admitted. If the callback queues and acknowledges quickly, each ACK permits another delivery while workers remain behind. Separate durable admission, queue capacity, and protocol flow control. + +
+ +**Done when:** for each feature record purpose, Go API field, packet/direction, prerequisite/ACL, prediction, actual observation, and whether you would use it. A reading-only feature stays marked “not experimentally verified.” + +Sources: [MQTT 5 specification](https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html) and pinned local `paho/cp_subscribe.go`, `paho/cp_unsubscribe.go`, `paho/cp_publish.go`, `paho/cp_connect.go`, `paho/cp_connack.go`, and `autopaho/backoff.go` under `references/paho.golang-v0.23.0/`. + +Next: Section 17 in [COURSE.md](../COURSE.md). diff --git a/lessons/README.md b/lessons/README.md new file mode 100644 index 0000000..eaef247 --- /dev/null +++ b/lessons/README.md @@ -0,0 +1,60 @@ +# MQTT in Go: lessons and concept coverage + +Broker deployment and administration are part of the learning goals. Read [Broker operations](broker-operations.md) alongside Lesson 1; it explains lifecycle, credential maintenance, backup/restore, upgrades, and remote-deployment preparation. Commands are explained in the lesson that introduces them. + +Follow these lessons in order from the learning project root. Read the concept and packet flow first, predict the experiment, then implement and compare the result. Go and basic Docker syntax are assumed; explanations focus on MQTT behavior and operational consequences. Full early examples remain available as reference solutions. + +1. [Set up the broker and MQTT 5 lab](01-lab-setup.md) +2. [Write and understand a Go MQTT 5 connection](02-go-connection.md) +3. [Add typed TOML and koanf configuration](03-typed-configuration.md) +4. [Publish, subscribe, and build a device simulator](04-publish-subscribe-simulator.md) +5. [Design contracts and validate inbound messages](05-contracts-validation.md) +6. [Manage callbacks, concurrency, and cancellation](06-concurrency-cancellation.md) +7. [Build reliable command and response behavior](07-command-response.md) +8. [Expose device commands through HTTP](08-http-commands.md) +9. [Learn MQTT 5 sessions through failure experiments](09-sessions-recovery.md) +10. [Expiry, presence, and retained state](10-expiry-presence-state.md) +11. [MQTT security boundaries](11-security-boundaries.md) + +Continue Sections 12–15 in the course before implementing [Section 16 — MQTT 5 controls](16-mqtt5-controls.md). The course's application/persistence/operations Sections 12–15 and 17–18 remain guided checklists, not complete reference programs. + +Paho is pinned to **v0.23.0**, the selected course baseline. Use the matching checkout under `references/paho.golang-v0.23.0/`. This release requires Go 1.24 or newer; the lessons' Go 1.25 baseline is compatible. + +The files are learning instructions and reference solutions. Application code has not been installed into the learning project: create it yourself as each lesson directs. Section 2 supplies the first backend program; Section 3 adds configuration; Section 4 replaces the backend program and adds shared transport plus a simulator. Do not combine both versions of `main.go` in the same package. + +## Previously recorded verification (before this review) + +The complete Go examples were extracted into a separate temporary module and checked with Go 1.27.1 against the pinned dependencies. They passed compilation, `go vet`, and configuration precedence/validation tests, including a race-enabled test run. Smaller explanatory fragments are identified as fragments and require their surrounding program or test. + +An isolated temporary Mosquitto 2.0.22 lab verified the documented Compose configuration, password-file permissions, ACL file loading, initial subscriptions, Go heartbeats, QoS 1 ping/pong, incorrect-password rejection, and subscription restoration after broker restart. The temporary broker and clients were removed afterward. The lesson exercises still require you to verify behavior in your own implementation; this check does not prove production reliability. + +Sections 5–11 and 16 are guided exercises, not complete tested reference programs. Their implementation checks are for you to run after writing the code. + +## Verification for the 2026-10-08 teaching review + +Extracted the complete Lesson 2–4 programs and the new Paho/TLS fragments into a temporary module. They passed `go build -buildvcs=false ./...` and `go vet -buildvcs=false ./...` against the pinned dependencies. The flag disables Git revision stamping because this environment cannot read usable repository metadata; it does not change MQTT behavior. Fragment wrappers supply their surrounding variables/imports for type checking. + +Checked local Markdown links, source paths, code fences, and expandable answer blocks. Reviewed protocol explanations against official MQTT/Mosquitto documentation and the matching local Paho source. No live broker, TLS, failure-injection, or load experiments were run in this review, and Mermaid rendering was not verified. The earlier runtime results above are historical records, not fresh results for the revised examples. + +## Coverage map: explain it, then prove it + +| Concept | Where it is explained | Evidence you should produce | +| --- | --- | --- | +| Broker routing, topic names/filters, wildcards | 1 and 4 | Trace CLI packets and change one topic filter | +| CONNECT/CONNACK, identity, Keep Alive, reconnect | 2 and 3 | Explain connection versus subscription/device readiness | +| QoS 0/1/2, per-hop acknowledgements, DUP and business IDs | 4 and 7 | Predict no-subscriber, lower-QoS, and duplicate outcomes | +| Callback/ACK order, manual acknowledgement, bounded queues | 6 | Show the RAM loss window and measurable overload | +| Payload validation, Content Type, Payload Format | 5 | Reject valid-looking but invalid messages before state changes | +| Response Topic, Correlation Data, timeouts, idempotency | 7 and 8 | Match concurrent results and retry without another effect | +| Clean Start, Session Expiry, Session Present, recovery stores | 9 | Predict network versus process restart outcomes | +| Message Expiry, retained deletion, Will/Will Delay, freshness | 10 | Observe stale command suppression and presence limitations | +| TLS, authentication, ACLs, identity and response destinations | 1 and 11 | Fail each security boundary deliberately | +| Subscription options, unsubscribe, shared consumers | 16 | Compare live/replayed messages and consumer distribution | +| Receive Maximum, packet size, properties, aliases, capabilities | 16 | Record packet direction, negotiated limit, and measured behavior | +| Docker mount/volume identity, recreation, overrides, crashes | Broker operations | Explain exactly which state survives each command | + +Coverage means these concepts are explained and have exercises. It does not mean you have completed the experiments or that this course implements every optional MQTT extension. Enhanced authentication is introduced conceptually; broker-specific authentication plugins and bridges are outside the lab's implementation scope. + +For each experiment, write: **prediction → operation and reason → observed packet/application evidence → explanation → restoration step**. Expand the answer blocks only after explaining the outcome yourself. + +See [COURSE.md](../COURSE.md) for the complete sequence and the connection to Robot Market's backend. diff --git a/lessons/broker-operations.md b/lessons/broker-operations.md new file mode 100644 index 0000000..212323c --- /dev/null +++ b/lessons/broker-operations.md @@ -0,0 +1,140 @@ +# Broker operations — maintain and deploy your lab + +Use this alongside Lesson 1, then revisit it during Section 9. All commands run from `/home/ryan/Media/Code/Code/robot-market/mqtt-5-learning` and target the `mqtt5-learning` project. These are instructions to perform in your lab; writing this guide has not changed or started a broker. + +## Lifecycle: know which state you are changing + +| Operation | Example | Effect | +| --- | --- | --- | +| Stop | `docker compose -p mqtt5-learning stop broker` | Gracefully stops the container; retains it and its mounts | +| Start | `docker compose -p mqtt5-learning start broker` | Starts that existing container; does not apply edited Compose settings | +| Restart | `docker compose -p mqtt5-learning restart broker` | Stops and starts the existing container; interrupts clients and rereads broker startup configuration | +| Apply Compose changes | `docker compose -p mqtt5-learning up -d broker` | Reconciles the service definition and recreates the container when needed; keeps the named volume | +| Force recreation | `docker compose -p mqtt5-learning up -d --force-recreate broker` | Creates a fresh container even if Compose detects no change; useful after replacing a bind-mounted file | +| Remove containers/network | `docker compose -p mqtt5-learning down` | Stops and removes project containers/network; the declared named data volume remains | +| Delete data too | `docker compose -p mqtt5-learning down -v` | Also removes declared named volumes; this erases lab broker persistence | + +`-p mqtt5-learning` selects the Compose project name, grouping this lab's service, network, and volume resources. It does not select a port. Keep it identical across all commands. Your existing `compose.yml` is valid; the lesson's `compose.yaml` is an alternative filename, not an additional required file. + +After a password-file repair, explicitly restart an existing container and inspect `logs --since=1m broker` so historical failures are distinguished from the new attempt. `up -d` may leave an unchanged running container alone. Read-only entrypoint ownership warnings do not establish failure; verify the broker's own listener/error messages and authenticated messaging. + +Use `restart` after editing broker configuration that requires a restart. Use `up -d` after editing ports, image tags, mounts, or container environment in Compose. Docker's [restart documentation](https://docs.docker.com/reference/cli/docker/compose/restart/) explains that restart does not apply Compose changes. For single-file bind mounts, an editor that replaces a file rather than writing in place can leave an existing container referring to the old file; recreation establishes fresh mounts. + +After every change, inspect `ps`, inspect `logs --tail=50 broker`, and repeat an authenticated publish/subscribe check. These check container state, broker startup, and actual messaging respectively. `logs -f broker` follows new logs until Ctrl+C; Ctrl+C stops log following, not the detached broker. + +## Choose the active credential store before maintenance + +The basic lesson uses a host bind mount. This workspace's current `compose.yml` uses the external `mqtt5-learning_broker-secrets` volume because host permission changes did not stick. Inspect the service mount first. For this workspace, replace `-v "$PWD/lab/secrets:/work"` in the rotation commands below with `-v mqtt5-learning_broker-secrets:/work`. Update and back up the active volume; modifying the preserved host copy has no effect on the broker. See Lesson 1's filesystem fallback for provisioning and access checks. External secret volumes survive Compose `down -v`, but declared nonexternal broker-data volumes do not. + +## Rotate a lab credential + +Use the Lesson 1 utility command without `-c` to update the existing backend user: + +```sh +docker run --rm -it --user 0 -v "$PWD/lab/secrets:/work" eclipse-mosquitto:2.0.22 mosquitto_passwd /work/passwords backend +docker run --rm --user 0 -v "$PWD/lab/secrets:/work" eclipse-mosquitto:2.0.22 chown mosquitto:mosquitto /work/passwords +docker run --rm --user 0 -v "$PWD/lab/secrets:/work" eclipse-mosquitto:2.0.22 chmod 600 /work/passwords +docker compose -p mqtt5-learning restart broker +``` + +The first command prompts for a new password and replaces only this user's hash. The next commands restore broker ownership and owner-only access. Restart reloads the file and disconnects existing clients. Update the backend terminal's password environment variable, reconnect, and test that the new password succeeds and the old one fails on a fresh connection. Simply editing a password file does not establish that already-connected clients have been revoked. + +Mosquitto also supports reloading password-file configuration with SIGHUP; see its [authentication guide](https://mosquitto.org/documentation/authentication-methods/). Restart is the simpler explicit lab operation. For deployed systems, plan client credential rollout and reconnection before rotation to avoid unexpectedly stranding devices. Adding/removing a user also requires reviewing that user's ACL entries. `mosquitto_passwd -D /work/passwords username` removes a user; `-D` means delete, so use it only for an account you intend to remove. + +## Backup and restore: what actually needs saving? + +Save three things: the Compose/broker/ACL configuration, the protected password file, and the broker data volume. The Go application's in-memory records are a fourth, separate concern and are not covered by this backup. Password hashes and any TLS private keys remain sensitive backup material. + +For a simple consistent lab backup, stop the broker gracefully first so it can write persistence state. Confirm `ps` reports it stopped. Use `docker volume ls` to list volumes and identify the project's broker-data volume; `ls` lists names, not contents. Verify it with `docker volume inspect ` before using it. Replace placeholders with the observed name; do not infer a production volume from an example name. + +Create a private backup directory outside Git, archive the stopped volume with a temporary utility container mounting it read-only, and save configuration plus secrets with their permissions preserved. A typical volume-copy utility needs a source mount and a backup-output mount; run it with the permissions required to read the broker's files. The data volume can contain `mosquitto.db`, while credentials live in the separate host secrets directory. Copying only one of them is incomplete. + +Restore into a separate empty lab volume while the target broker is stopped. Restore ownership/permissions, connect that volume to an isolated broker using the recorded image/configuration, and verify authentication, ACL denial, retained state, and session behavior. Do not extract an archive over an active broker data directory. A backup is useful only after a restore exercise succeeds; record the commands, actual volume name, date, image version, and observed recovered state in your journal. + +## Upgrade with an observable rollback + +Record the current image version and back up configuration, secrets, and stopped broker state. Read the target release's migration notes. Test the new pinned image against a restored copy in a separate lab project with a different loopback port; repeat authentication, ACL, reconnect/session, and retained-state checks before changing the primary lab. + +`docker compose -p mqtt5-learning pull broker` downloads the image selected by the current Compose file; it does not replace the running container. After selecting a tested new image tag, `up -d broker` applies it. Keep the previous tag and backup: restoring an older image over data rewritten by a newer version is not a guaranteed rollback. For reproducibility beyond a version tag, record the resolved image digest as well. + +## Preparing an actual remote deployment + +The initial lesson exposes only loopback plaintext, so remote devices cannot connect. Do not merely change the bind address to make it reachable. Work through Section 11 before remote use: create a server certificate with the correct DNS/IP identity, configure a TLS listener, mount certificate/key files with broker-readable permissions, and make clients validate the CA and server identity. A password hash on disk does not encrypt credentials traveling over plaintext MQTT. + +Choose the device-facing hostname, TLS port, firewall policy, per-device credentials/ACLs, persistent storage, backup/restore process, log retention, monitoring, and credential renewal responsibilities. Add a deliberate container restart policy such as `restart: unless-stopped` to the service if the deployment should recover after daemon/host restarts; the initial lab omits it to keep stop/start experiments explicit. A restart policy helps process availability but does not prove messaging readiness or provide high availability. + +Container logs need a bounded retention/rotation policy so a busy broker cannot fill the host disk. Monitor connection failures, permission failures, disk use, and application heartbeat/command evidence. Test broker readiness through an authorized synthetic message round trip; a TCP port check alone cannot prove MQTT authentication or subscriptions work. + +Keep provisioning repeatable with documented setup steps or a reviewed script, use the platform's secret provisioning mechanism, and plan certificate/password rotation and client rollout. Do not bake credentials or private keys into the image. Deployment-specific values belong in that environment's configuration, not in the learning project's committed secrets. + +## Advanced Docker operations: predict the MQTT consequence + +### Inspect the configuration without starting anything + +```sh +docker compose -p mqtt5-learning config -q +docker compose -p mqtt5-learning ps -a +docker compose -p mqtt5-learning logs --since=2m --tail=100 broker +docker volume inspect mqtt5-learning_broker-data +``` + +`config -q` validates the resolved Compose model without printing it or contacting MQTT. `ps -a` includes stopped containers, helping distinguish "never created" from "exited on startup." The log filters restrict evidence to recent lines; they do not clear old logs. `volume inspect` shows volume metadata and its Docker host mountpoint, not the broker's session contents. The volume name above is the expected default for this project: if inspection fails, use `docker volume ls` and identify the actual lab volume rather than creating one to hide the mismatch. + +Ask three separate questions: is the container running, did Mosquitto load the intended files, and can the authorized clients exchange packets? These commands answer only parts of that chain. Finish with the authenticated CLI round trip from Lesson 1. + +### Understand execution context + +```sh +docker compose -p mqtt5-learning exec -T broker mosquitto_pub --help +``` + +This runs the image's CLI in the existing broker container. `-T` disables the pseudo-terminal, useful for scripted output. `--help` is passed to `mosquitto_pub`; it performs no publish. When adapting the actual CLI test to this form, use container port 1883, not host port 18883. A new utility container started with `docker run` has a different loopback interface and does not reach the broker merely by using `127.0.0.1`. + +### Isolate the session-persistence comparison + +For Lesson 9's nonpersistent comparison, create `compose.session-test.yaml`: + +```yaml +services: + broker: + volumes: + - broker-data-session-test:/mosquitto/data +volumes: + broker-data-session-test: +``` + +First stop the original lab. Temporarily set `persistence false` in its broker configuration, then inspect and apply the override: + +```sh +docker compose -p mqtt5-learning stop broker +docker compose -p mqtt5-learning -f compose.yml -f compose.session-test.yaml config +docker compose -p mqtt5-learning -f compose.yml -f compose.session-test.yaml up -d --force-recreate broker +``` + +Use the actual primary filename if yours is `compose.yaml`. The later `-f` file overrides the data mount by its container destination `/mosquitto/data`, while keeping configuration and credential mounts. Verify that merged result before `up`. `--force-recreate` replaces the container so the changed file/mount is actually used; it does not delete the original data volume. Use an unused comparison volume name on repeat experiments so previous state cannot contaminate the result. + +After measuring, stop the comparison with the same two `-f` files, restore `persistence true`, and run `docker compose -p mqtt5-learning -f compose.yml up -d --force-recreate broker`. This restores the original data mount. Inspect logs and verify authentication plus messaging again. The comparison volume remains for inspection; no volume deletion is required. + +### Crash is different from graceful stop + +For a deliberate broker crash experiment only: + +```sh +docker compose -p mqtt5-learning kill -s SIGKILL broker +docker compose -p mqtt5-learning start broker +docker compose -p mqtt5-learning logs --since=2m broker +``` + +SIGKILL bypasses the broker's graceful persistence flush and interrupts every lab client. `start` runs the same container with the same mounted data; inspect what was actually recovered. Compare against `stop`, which permits graceful shutdown before Docker's stop timeout. A named volume survives both, but only state already written to it can be recovered after the crash. Do not infer crash durability from the existence of the volume. + +These commands target the broker, so they test broker recovery. To test a device Will, terminate the simulator instead and keep the broker alive, as explained in [Lesson 10](10-expiry-presence-state.md). + +## Practice and explain + +1. Stop and start the lab; explain which resources survived. +2. Rotate one lab password and test both passwords on fresh connections. +3. Recreate the container and confirm the data volume remains mounted. +4. Perform an isolated backup/restore exercise and record what was recovered. +5. Explain what would change for TLS, remote access, automatic restart, and log retention. + +Read the official [Compose up](https://docs.docker.com/reference/cli/docker/compose/up/), [Compose down](https://docs.docker.com/reference/cli/docker/compose/down/), and [Mosquitto configuration](https://mosquitto.org/man/mosquitto-conf-5.html) documentation when planning changes. Their latest pages may describe features beyond this lab's pinned Mosquitto 2.0.22; check version compatibility before adopting them. diff --git a/prompt.md b/prompt.md new file mode 100644 index 0000000..0278e25 --- /dev/null +++ b/prompt.md @@ -0,0 +1,196 @@ +Act as my senior Go backend engineer and MQTT 5 tutor. + +My goal +I want to understand and implement an MQTT 5 backend in Go using Eclipse Paho. Teach me through a small, separate learning project inspired by my existing vending-machine backend. + +Help me understand the protocol, write the code myself, debug failures, and explain my design decisions. Do not simply generate a finished application for me. + +Current learner context and course +I already understand MQTT 3.1.1 concepts. Focus on practical Go backend development and MQTT 5 differences rather than repeating the introductory MQTT syllabus. +My goal is to maintain Robot Market after its MQTT 5 version is delivered to me. Use the existing backend as a read-only reference, and reinspect the delivered version at handover. +Follow COURSE.md in this project, section by section and exercise by exercise. It is the requested ordered learning course; do not restart planning or require another sprint breakdown before beginning a lesson I request. +Detailed instructions for Sections 1–11 and 16 live in lessons/. Sections 1–4 include reference programs; later lessons provide explained fragments and guided exercises. Use lessons/README.md for concept coverage and the remaining course sections. Teach from them incrementally. The learner explicitly requested code in these lesson files; do not hide those examples, while still allowing them to attempt the exercises themselves. +The lesson baseline uses github.com/eclipse/paho.golang v0.23.0, the selected course baseline, requiring Go 1.24 or newer. Matching source is in references/paho.golang-v0.23.0/. Verify releases again before any future upgrade. + +Existing project context +My project is Robot Market, a backend for managing vending machines. + +Its main responsibilities include: +- HTTP APIs for device management and backend operations. +- Publishing commands, product catalogs, and settings to devices. +- Receiving device messages such as sales reports, logs, heartbeats, and synchronization requests. +- Storing device and business data in PostgreSQL. +- Authentication and authorization using JWT, sessions, and role-based access control. + +Current technologies: +- Go 1.25. +- Standard-library net/http for HTTP APIs. +- PostgreSQL, with database/sql and PostgreSQL libraries including pgx/v5 and lib/pq. +- Eclipse Paho github.com/eclipse/paho.mqtt.golang v1.5.0 for existing MQTT communication. +- Environment-variable and .env configuration using godotenv. +- Separate API and MQTT ingestion processes. +- Database migrations and Go tests. + +Important distinction: +The existing github.com/eclipse/paho.mqtt.golang library supports MQTT 3.1/3.1.1. +For this learning project, use: +- github.com/eclipse/paho.golang/autopaho +- github.com/eclipse/paho.golang/paho + +These are separate client APIs. Do not assume migration is just changing imports. + +Learning environment +Use the separate learning project at /home/ryan/Media/Code/Code/robot-market/mqtt-5-learning. +Do not change my existing backend, production broker, database, or devices. + +Use: +- Go and Eclipse Paho AutoPaho. +- A local Mosquitto broker, preferably through Docker Compose. +- A Go device simulator so I can learn without physical hardware. +- net/http when we introduce HTTP endpoints. +- TOML configuration with github.com/knadh/koanf/v2. +- Typed configuration and startup validation. +- Configuration precedence: defaults < TOML < environment < CLI. +- Environment-variable prefix TEST_MQTT5_. + +TOML and koanf are choices for the new learning project, not technologies already used by the existing backend. + +Begin with in-memory application state. Introduce PostgreSQL only when persistence has a clear learning purpose. + +How to teach me +Always show commands for me to run as normal shell commands without an rtk prefix. Use rtk only for commands you execute through your own tools. + +1. Assume I know Go and basic Docker syntax. Explain MQTT behavior at the code boundary rather than re-teaching pointers, structs, channels, or basic shell commands. Revisit MQTT 3.1.1 concepts when needed to explain guarantees or MQTT 5 differences. + - Before code, explain the packet flow, which process owns each state, and why the API choices implement the policy. + - For MQTT commands and advanced Docker operations, explain purpose, relevant flags, execution context, changed state, expected evidence, and how to restore the lab. + - Ask me to predict outcomes, change one variable, then explain observed results. Include short self-check answers separately from full solutions. + - Never treat successful copy/paste, PUBACK, or a passing build as proof of device completion or understanding. +2. Use COURSE.md as the learning plan and ordered implementation checklist. +3. Start the section or exercise I request. Let me implement it; show a full solution or write application code only when I ask. +4. Teach one lesson or exercise at a time. +5. For each lesson: + - Explain the concept in plain language. + - Connect it to a vending-machine example. + - Explain the relevant MQTT packets and properties. + - Show a small amount of code and explain its important parts. + - Give me an exercise, commands to run, and expected results. + - Explain common mistakes and how to diagnose them. + - Ask a few questions that check my understanding. +6. Let me attempt exercises before showing the full solution, unless I request it. +7. Review code I share for correctness, concurrency, reliability, and clarity. +8. Introduce production concerns gradually and explain why each matters. + +Learning progression +Cover these topics in a sensible order: + +A. MQTT fundamentals +- Broker, client, topics, subscriptions, and wildcards. +- Publish/subscribe versus HTTP request/response. +- QoS 0, 1, and 2, including their actual guarantees. +- Retained messages versus queued messages. +- Client IDs, keepalive, and Last Will. +- How MQTT 5 differs from MQTT 3.1.1. + +B. First Go MQTT 5 application +- Choosing and pinning a current compatible Paho release. +- AutoPaho versus the lower-level Paho client. +- Connecting, subscribing, publishing, and receiving messages. +- context.Context, cancellation, and graceful shutdown. +- Safe callback handling and concurrent application state. +- Checking connection and publish errors and reason codes. + +C. MQTT 5 features through experiments +- Clean Start and Session Expiry Interval. +- Message Expiry Interval. +- Reason codes and negative acknowledgements. +- User Properties, Content Type, and Payload Format Indicator. +- Response Topic and Correlation Data. +- Subscription options and shared subscriptions. +- Will Delay Interval. +- Receive Maximum, packet-size limits, and topic aliases. +- Which behavior belongs to the protocol, broker, client library, or application. + +D. A small vending backend +Build this flow incrementally: +HTTP command request → Go backend → MQTT broker → simulated device +→ application response → Go backend → command status. + +Use harmless commands such as ping, get_status, or simulated configuration updates. + +Design: +- A clear topic naming scheme. +- Versioned JSON message contracts. +- Device identity and topic authorization. +- Command IDs and correlation. +- Timeouts and command status transitions. +- Duplicate handling and idempotency. +- Separation between HTTP handlers, MQTT transport, and business logic. + +Explicitly distinguish: +- MQTT publish acknowledgement. +- Device acceptance of a command. +- Device completion of a command. + +An MQTT acknowledgement does not prove that a vending machine performed the requested action. + +E. Reliability and security +- Reconnection and backoff. +- Subscription restoration and session resumption. +- QoS 1 duplicates. +- Broker restart, network interruption, and process restart. +- Client session storage versus outgoing queues versus application persistence. +- Stale commands and message expiry. +- Bounded queues and backpressure. +- TLS certificate verification. +- Separate backend and device credentials. +- Disabled anonymous access and least-privilege ACLs. +- Safe logging without credentials or secrets. + +Do not use InsecureSkipVerify: true. +Explain any simplified local-lab setup and how it evolves into secure deployment. + +F. Verification +Use meaningful experiments and tests: +- Publish/subscribe between backend and simulator. +- Duplicate command delivery without duplicate effects. +- Offline device reconnect and session behavior. +- Expired commands. +- Application response timeout. +- Broker restart and reconnection. +- Unauthorized topic access. +- Graceful shutdown and cancellation. +- Race detection when shared state is introduced. + +Documentation and accuracy +Local library source and example sources are available in references/paho.golang/. +Upstream repository: https://github.com/eclipse-paho/paho.golang +Downloaded reference commit: 145831e41db6f515957b518d5ef81a9fe21a894f. + +Use these local references when teaching and reviewing code: +- references/paho.golang/README.md for the library overview. +- references/paho.golang/autopaho/ for connection management source. +- references/paho.golang/paho/ for the lower-level MQTT 5 client source. +- references/paho.golang/packets/ for MQTT packet implementations. +- references/paho.golang/autopaho/examples/ for upstream examples, including basics, backoff, queue, router, rpc, and docker. +- Upstream tests alongside the source for concrete behavior and edge cases. + +Treat the downloaded repository as reference material. Keep learning exercises and application code outside it, and preserve upstream licenses. Do not run examples or their Docker environments automatically; introduce them as deliberate lesson exercises. +Explain the relevant parts of an example before adapting a small excerpt. Let me implement the exercise before showing a full solution. +The downloaded default-branch commit is a source snapshot, not automatically the dependency version for the learning application. When we select and pin a compatible release, compare its APIs with this snapshot and consult the matching tagged source and documentation. Record any differences; do not silently update the reference checkout. + +Check current official documentation before relying on version-sensitive APIs. +Prefer: +- https://github.com/eclipse-paho/paho.golang +- https://github.com/eclipse-paho/paho.golang/tree/master/autopaho/examples +- https://pkg.go.dev/github.com/eclipse/paho.golang/autopaho +- https://pkg.go.dev/github.com/eclipse/paho.golang/paho +- https://www.hivemq.com/mqtt/mqtt-5/ +- https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html +- https://mosquitto.org/documentation/ + +Match examples and API documentation to the selected dependency version. +Explain relevant known limitations rather than assuming complete protocol support. +If you cannot verify something, say so clearly. + +Start with the requested exercise in COURSE.md, or Section 1, exercise 1 if no exercise is specified. +Do not generate the complete application yet. diff --git a/references/README.md b/references/README.md new file mode 100644 index 0000000..86ae0fe --- /dev/null +++ b/references/README.md @@ -0,0 +1,24 @@ +# Paho source reference + +## Lesson release source + +`paho.golang-v0.23.0/` is a separate shallow checkout of the latest tagged release verified on 2026-10-07. Sections 1–4 use this version. + +- Tag: `v0.23.0` +- Commit: `1aa03969c5932464359b3e4ed752d9ac61abccc9` +- Minimum Go version: 1.24 +- [Release notes](https://github.com/eclipse-paho/paho.golang/releases/tag/v0.23.0) + +The original default-branch checkout below remains unchanged and can differ from the lesson release. + +## Original default-branch snapshot + +The `paho.golang/` directory is a shallow clone of https://github.com/eclipse-paho/paho.golang, downloaded on 2026-10-07. + +- Snapshot commit: `145831e41db6f515957b518d5ef81a9fe21a894f` +- Library sources: `paho.golang/paho/`, `paho.golang/autopaho/`, and `paho.golang/packets/` +- Example sources: `paho.golang/autopaho/examples/` +- Upstream overview: `paho.golang/README.md` +- Licensing: `paho.golang/LICENSE`, `paho.golang/epl-v20`, and `paho.golang/edl-v10` + +Keep this checkout as upstream reference material and put learning application code outside it. Its default-branch snapshot does not select a dependency release for the learning project. Match source and examples to the release chosen during learning before relying on version-sensitive APIs. diff --git a/references/paho.golang b/references/paho.golang new file mode 160000 index 0000000..145831e --- /dev/null +++ b/references/paho.golang @@ -0,0 +1 @@ +Subproject commit 145831e41db6f515957b518d5ef81a9fe21a894f diff --git a/references/paho.golang-v0.23.0 b/references/paho.golang-v0.23.0 new file mode 160000 index 0000000..1aa0396 --- /dev/null +++ b/references/paho.golang-v0.23.0 @@ -0,0 +1 @@ +Subproject commit 1aa03969c5932464359b3e4ed752d9ac61abccc9