mqtt-5-learning/COURSE.md

Go MQTT 5 backend course for Robot Market

Starting point and destination

You already understand MQTT 3.1.1 concepts. This course concentrates on writing, debugging, testing, and maintaining Go services that communicate with IoT devices. Go and basic Docker syntax are assumed. Explain MQTT semantics where code depends on them, including packet flows, guarantees, failure boundaries, and the purpose and effect of MQTT or advanced Docker commands. Each experiment starts with a prediction and ends with an explanation, not merely successful copied output.

You will build a small vending backend and a Go device simulator, then use that experience to understand the MQTT 5 backend delivered to you. Work through the numbered sections in order. Each checkbox is an exercise you implement; the tutor provides explanations, small examples, debugging help, and code review. A section can take several sessions. There is no need to finish a whole section in one sitting.

This is the requested course and implementation checklist. It does not require another round of sprint planning. Ask the tutor to start a numbered exercise when ready; full solutions should wait until you have attempted it or requested one.

Start with Lesson 0 — The broker, Go backend, and device lifecycle for the end-to-end story. Lessons 1–8 now begin with the system story and a guiding question before each implementation step.

All implementation belongs in /home/ryan/Media/Code/Code/robot-market/mqtt-5-learning. Use a dedicated local broker, synthetic data, lab credentials, and simulated devices. Reference the existing backend read-only.

How this course matches your backend

The local backend was inspected on 2026-10-07. These are current source observations, not assumptions about the future MQTT 5 delivery:

Observed technology or responsibility	Course connection
Go 1.25.0 and github.com/eclipse/paho.mqtt.golang v1.5.0 in go.mod	Learn the separate MQTT 5 Paho APIs and later review migration differences
net/http, API entry point, separate ingestion entry point	Build HTTP and MQTT flows together first, then split processes
database/sql with pgx stdlib in the DB connector; pgx/v5 and lib/pq dependencies	Add PostgreSQL repositories, transactions, and migrations after in-memory exercises
godotenv and environment-based configuration	Learn typed TOML + koanf for this lab; understand the configuration boundary at handover
JWT dependency, authentication middleware, device permission modules	Apply user authorization to device operations separately from broker permissions
Product push envelopes carrying message/job IDs, contract version, and revisions	Practice catalog synchronization and correlation
Sales ingestion validates identity, saves reports, and sends application responses	Practice durable ingestion, deduplication, and acknowledgement timing
Topic routing includes health, logs, inventory, configuration, updates, and command responses	Build representative small flows before studying the larger routing surface

Source reading map, relative to ../robot-market-vending-backend/server_backend/go-back/:

• go.mod, internal/config/config.go, internal/db/db.go — dependencies, configuration, database setup.
• internal/mqtt/client.go — existing connection configuration and MQTT 3 client lifecycle.
• cmd/api/main.go, cmd/ingest/main.go — process composition.
• internal/ingest/subscriber.go — inbound topic routing.
• cmd/api/mqtt_products_push.go, internal/ingest/handler_products_push_resp.go — product push and response paths.
• internal/ingest/handler.go, internal/contracts/sales.go, internal/sales/ — sales reports and persistence.
• internal/auth/, internal/devices/rbac.go, internal/devices/identity.go — authorization and identity.

The future delivered backend must be inspected again. Do not assume it will use the lab's package layout, topic names, or configuration system. Board identity (FN), machine identity (SN), and other identifiers must be understood from the delivered contracts rather than treated as interchangeable.

Course rules and working structure

• Build one evolving application, adding only the structure needed by the current section.
• Start with cmd/backend and cmd/simulator; introduce API and ingestion executables in Section 15.
• Add internal/config, internal/mqtt, internal/contracts, and domain packages as they become useful. Keep interfaces small and driven by callers.
• Keep upstream code under references/paho.golang/ unchanged. Read autopaho/examples/basics first; use backoff, queue, router, and rpc when their lessons arrive. Lower-level examples also live under paho/cmd/.
• Select a compatible tagged Paho release in Section 1 and pin it. The downloaded commit recorded in references/README.md is a reference snapshot, not an automatic release choice. Read API documentation and examples matching the selected version.
• Use TOML and github.com/knadh/koanf/v2, with defaults < TOML < environment < CLI and prefix TEST_MQTT5_. CLI defaults must not accidentally overwrite configured values when flags were not supplied.
• Use memory until Section 13. Use harmless ping, get_status, and simulated configuration changes throughout.
• For each exercise record the change, command, expected result, actual result, and your explanation in a learning journal. Keep payloads synthetic and omit secrets.
• Move on when the section's completion checks pass and you can explain the results without reading the implementation.

Section 1 — Prepare an isolated, reproducible lab

Detailed lesson: Section 1 instructions and examples.

Start with the broker's role and setup: which configuration belongs to Mosquitto, how client accounts are supplied, and what local CLI evidence confirms the broker is ready for Go clients.

Outcome: A local MQTT 5 broker you can reach, plus a recorded toolchain and dependency choice.

1. Check your Go and Docker installation. Record go version, docker version, and docker compose version. If Docker is unavailable, establish an equivalent isolated local Mosquitto setup before continuing.
2. Read the local Paho README and the basics example. Identify which responsibilities belong to AutoPaho and which belong to the underlying Paho client. Check a tagged release's Go requirement and release notes before pinning it.
3. Initialize a Go module in this learning project. Record the selected Go, Paho, and Mosquitto versions; avoid floating latest versions for reproducible exercises.
4. Create a Compose definition with a project-specific name and a broker listener published only on loopback, using an unused lab port such as 127.0.0.1:18883. Use a separate persistence volume and configuration directory.
5. Disable anonymous access. Create distinct lab backend and simulator usernames and an initial ACL allowing only the lab topic namespace. For this disposable local test lab, store the shared password in configs/lab.toml; never reuse a real credential.
6. Use Mosquitto CLI tools with MQTT version 5 explicitly selected to subscribe and publish a synthetic message. Check the installed tools' help for version and authentication flags. Repeat with incorrect credentials.

Initial commands: docker compose -p mqtt5-learning config, then docker compose -p mqtt5-learning up -d, docker compose -p mqtt5-learning ps, and docker compose -p mqtt5-learning logs --tail=50. Run these after you have written the lab files; never use a production Compose directory. Do not share resolved Compose output if it contains secrets.

Done when: A valid subscriber receives the exact test payload; invalid credentials are rejected; you can stop and restart only the lab broker. Record why MQTT 5 must be selected explicitly in your CLI check.

Local simplification: Loopback-only plaintext is acceptable for this initial lab. Credentials are not encrypted on a plaintext connection. Section 11 adds verified TLS before any nonlocal deployment.

Section 2 — Write your first Go MQTT 5 connection

Detailed lesson: Section 2 connection lifecycle and code.

Follow the backend from configuration through AutoPaho, Paho, CONNECT, and CONNACK. The story question is what proves connection acceptance and how startup differs from a running process.

Outcome: A small executable that connects, reports connection state, and exits cleanly.

1. Create cmd/backend/main.go and use AutoPaho with a unique, explicit client ID. Explain the MQTT 5 CONNECT/CONNACK exchange and inspect failure reasons exposed by the selected API.
2. Add a root context and signal cancellation. Bound initial connection waiting so a stopped broker does not leave startup hanging indefinitely.
3. Log connection up/down events without credentials. Distinguish application readiness from a running process.
4. Implement shutdown using the selected release's lifecycle API and wait for background activity to finish with a deadline.
5. Try wrong credentials, an unreachable port, and Ctrl+C during connection attempts.

Done when: Each failure has a useful diagnosis; cancellation stops retries and exits promptly. Explain why starting the connection manager does not alone prove the broker is ready.

Read: references/paho.golang/autopaho/examples/basics/ and the selected release's connection-manager documentation.

Section 3 — Add typed configuration

Detailed lesson: Section 3 TOML and koanf implementation.

Trace how each Go process gets its own identity and credentials, how configuration sources merge, and when validation must finish before MQTT startup.

Outcome: Both executables can be configured predictably without editing Go code.

1. Add typed TOML + koanf configuration for broker URL, client ID, credential inputs, timeouts, log level, and simulator identity.
2. Implement the required precedence. Define an unambiguous mapping from prefixed environment variables to nested keys.
3. Validate URLs, durations, required credentials, and conflicting settings before opening connections.
4. Write small table-driven tests for precedence and invalid configuration. Include the case where a CLI flag is absent.
5. Print a redacted startup summary and document a sample TOML file containing placeholders only.

Done when: You can demonstrate every override layer, and bad settings fail before connecting. Explain why configuration errors and transient network failures need different handling.

Section 4 — Publish, subscribe, and simulate a device

Detailed lesson: Section 4 Paho transport, backend, and simulator.

Start with the story of how the backend and simulator connect, subscribe, exchange a command, and report a device-level response before building the transport helper.

Outcome: Two Go processes exchange messages through the broker.

1. Create a simulator with its own identity and client ID. Give the backend and simulator different topic permissions.
2. Define a lab namespace, for example lab/v1/devices/{deviceID}/..., with separate telemetry, commands, and responses topics. These are lab contracts, not changes to existing production topics.
3. Subscribe, inspect SUBACK results, and wait for subscription readiness before publishing a message that must be received.
4. Publish at QoS 0 and 1. Inspect errors and acknowledgements rather than assuming a nil transport error proves the device acted.
5. Send heartbeats from the simulator; receive a harmless command and return a simple response.

Done when: You can trace CONNECT, SUBSCRIBE/SUBACK, PUBLISH, and QoS 1 PUBACK for both directions. Explain why a PUBACK is not a business response and why QoS is evaluated per delivery hop.

Section 5 — Design contracts and validate inbound messages

Detailed lesson: Section 5 instructions.

Outcome: A malformed or misleading message cannot corrupt application state.

1. Define versioned JSON structures for heartbeat, command, command result, and synthetic sales report. Include device identity, stable message ID, and relevant timestamps.
2. Validate topic shape, topic/body identity agreement, payload size, schema version, required values, and timestamps. Decide explicitly how unknown fields and unsupported versions behave.
3. Separate topic parsing and decoding from business handlers. Introduce Content Type and Payload Format Indicator as metadata; still validate the actual payload.
4. Add table-driven tests for malformed JSON, identity mismatch, unknown commands, missing IDs, and boundary sizes.
5. Return safe application errors without echoing confidential content.

Done when: Valid messages route correctly; invalid messages have consistent outcomes. Explain why a body field claiming a device ID does not authenticate the sender.

Section 6 — Manage callbacks, concurrency, and cancellation

Detailed lesson: Section 6 instructions.

Outcome: Message handling remains responsive under concurrent traffic.

1. Inspect the selected Paho release's callback execution and acknowledgement behavior. Document which work happens before and after callback return.
2. Add bounded work queues and a small worker pool. Define an overload policy separately for disposable telemetry and important commands/reports.
3. Protect shared state with a deliberate ownership or locking model. Copy message data before retaining it if required by the API's ownership contract.
4. Propagate lifecycle contexts and operation deadlines; avoid unbounded goroutines and per-message background contexts.
5. Add a slow handler and a burst generator; observe queue saturation and shutdown while work is pending.

Done when: The learning module passes go test -race ./..., and overload behavior is observable and bounded. Explain the possible loss window if MQTT acknowledges a message before queued application work is durable.

Section 7 — Build reliable command/response behavior

Detailed lesson: Section 7 explanations and experiments.

Outcome: Concurrent commands are matched to the correct device responses.

1. Add MQTT 5 Response Topic and Correlation Data. Subscribe to authorized response topics before sending requests and register pending requests before publishing.
2. Define command states and legal transitions: created, pending delivery, accepted, completed, rejected, and response timeout/unknown outcome. Track broker publish acknowledgement separately.
3. Match responses using command identity, expected device, and correlation. Handle malformed, duplicate, late, and out-of-order responses without regressing terminal state.
4. Use deadlines and remove expired in-memory waiters. A timeout must not imply the device definitely did nothing.
5. Make simulator effects idempotent by stable command ID; replay the recorded result on a repeated command. Reject unexpected response destinations.

Done when: Many concurrent commands receive the right results; duplicate commands produce one simulated effect; late responses follow a documented policy. Explain why MQTT Correlation Data alone supplies neither authorization nor deduplication.

Read: AutoPaho's rpc example as a mechanism reference, then compare its scope with your application state machine.

Section 8 — Expose device commands through HTTP

Detailed lesson: Section 8 instructions.

Outcome: An API caller can create a command and query its status.

1. Use net/http for POST /devices/{id}/commands and GET /commands/{id}. Return a command ID and asynchronous acceptance when appropriate.
2. Separate HTTP decoding, command service, and MQTT transport. Keep command processing independent of a disconnected HTTP caller after acceptance.
3. Add request-body limits, JSON error responses, server timeouts, and controlled shutdown.
4. Distinguish HTTP request idempotency keys from device command IDs and MQTT packet identifiers. Test a retried HTTP request.
5. Use httptest and a small fake transport to test invalid input, unavailable transport, and command-status responses.

Done when: An HTTP command reaches the simulator and status reflects device evidence. A closed HTTP connection does not silently cancel an already accepted command.

Section 9 — Learn MQTT 5 sessions through failure experiments

Detailed lesson: Section 9 instructions.

Outcome: You can predict and explain reconnect and offline delivery behavior.

1. Compare stable and changing client IDs, Clean Start choices, zero/nonzero Session Expiry Interval, and the broker's Session Present result.
2. Disconnect the simulator, publish a QoS 1 command, and reconnect before and after expiry. Record the subscription and session prerequisites for queued delivery.
3. Test subscription restoration when a session exists and when it does not. Observe reconnect callbacks and avoid accidental repeated local handler registration.
4. Repeat with broker restart, first with persistence disabled and then configured. Record the limits of the actual broker persistence behavior.
5. Compare client session storage, the outgoing publication queue, broker session state, and your application state in a four-column recovery table.

Done when: Each experiment has a predicted and observed result. Explain what survives network loss versus backend, simulator, or broker process loss.

Section 10 — Handle expiry, presence, and desired state

Detailed lesson: Section 10 explanations and experiments.

Outcome: Stale work and stale device status are handled deliberately.

1. Add Message Expiry Interval to time-sensitive commands and an application deadline checked by the simulator. Account for time spent in local queues and clock assumptions.
2. Compare retained desired configuration with nonretained one-shot commands. Clear a retained message deliberately in the lab and observe the result.
3. Add heartbeat freshness and a Last Will. Experiment with Will Delay, graceful disconnect, abrupt termination, and reconnect; explain their interaction with sessions.
4. Introduce desired and reported configuration revisions. Detect a stale report without replacing newer state.

Done when: An expired queued command produces no effect; late telemetry does not make an offline device appear reliably healthy. Explain why Last Will and heartbeat freshness provide different evidence.

Section 11 — Secure device communication and API access

Detailed lesson: Section 11 explanations and experiments.

Outcome: Unauthorized clients cannot control another device or observe its private traffic.

1. Add a lab CA and server certificate with the correct host identity. Verify the certificate chain and hostname; never set InsecureSkipVerify: true.
2. Refine backend/device ACLs. Bind each device credential to its own namespace and restrict response-topic destinations. Test wildcard subscriptions and cross-device publication attempts.
3. Add API authentication and per-device authorization, informed by the existing JWT and permission code. Use established libraries; validate relevant token claims and expiry.
4. Test a valid user with insufficient device permissions, expired credentials, a wrong CA, and a hostname mismatch.
5. Explain credential rotation, optional mTLS, redaction, and why broker identity and HTTP user identity are different trust boundaries.

Done when: Positive and negative access tests behave as designed. Record the observed reason/error for denied operations; do not assume every broker denial is reported identically.

Section 12 — Implement representative vending workflows

Outcome: You understand the backend's main device-facing jobs using synthetic data.

1. Add synthetic sales reports and application responses keyed by report ID. Deduplicate in memory and identify the restart limitation.
2. Push a small product catalog with contract version, message ID, job ID, and revision. Have the simulator report which revision it applied.
3. Implement a device-initiated configuration fetch and a backend-initiated simulated update. Define conflict and stale-revision policies.
4. Ingest health and bounded log events with basic sanitization. Distinguish transport connectivity, application responsiveness, and device health.
5. Trace equivalent source paths in the current backend and write a short comparison of its contracts with your learning contracts.

Done when: You demonstrate each flow and explain its identity, acknowledgement, retry, and deduplication rules. Keep remote delivery, real payments, OTP/SMS, and firmware changes simulated or read-only study topics.

Section 13 — Persist reports and command state in PostgreSQL

Outcome: Important state and deduplication survive application restart.

1. Add a dedicated lab PostgreSQL instance and migrations for devices, commands, command results, and sales reports. Use database/sql with pgx stdlib to match the observed connector style.
2. Move important in-memory repositories behind small interfaces. Use contexts, query deadlines, bounded connection pools, and transaction boundaries.
3. Enforce uniqueness for device/report IDs and command IDs. Persist a deduplication decision and its corresponding database effect atomically.
4. Define precisely when to send an application-level saved/accepted response. Investigate manual MQTT acknowledgement support before choosing an acknowledgement policy; never assume callback return proves persistence.
5. Use isolated DB integration tests to simulate duplicates, rollback, database outage, and a crash after commit but before the response reaches the device.

Done when: Replaying a report after restart does not double-count it, and the simulator can safely retry a missing response. Explain why MQTT QoS 2 does not make database effects or physical actions automatically exactly once.

Section 14 — Close database/MQTT failure gaps

Outcome: Accepted work can be recovered across failures between storage and publication.

1. Draw the failure windows between saving a command, publishing it, receiving PUBACK, and recording the device result.
2. Implement a small transactional outbox: save the command and intent to publish in one transaction, then publish from a worker with bounded retries and expiry.
3. Reuse the same command ID across retries. Model possible duplicate publication when the process crashes after publish but before marking the outbox entry sent.
4. Persist result processing and idempotency records. Separate retryable failures, permanent rejection, expiry, and unknown outcome.
5. Inject each crash window. Keep synthetic device effects and their deduplication state recoverable for the restart experiment; discuss why physical effects require device-specific reconciliation.

Done when: Accepted durable work is eventually processed or reaches an explainable terminal/unknown state. Explain the separate purposes of an outbox, a Paho outgoing queue, and MQTT session storage.

Section 15 — Split API and ingestion processes

Outcome: Your application resembles the backend's separate process responsibilities.

1. Extract cmd/api and cmd/ingest with separate lifecycle contexts and client IDs. Decide explicitly which process or worker owns publication.
2. Let the API create durable commands and query status while ingestion validates and records device responses.
3. Eliminate reliance on a pending-request map shared across processes. Use persisted status as the baseline; add notification only if it has a demonstrated purpose.
4. Stop and restart either process independently. Verify that the surviving process behaves predictably and shutdown preserves recoverable work.
5. Document dependency readiness, ownership of subscriptions, and startup order without relying on arbitrary sleeps.

Done when: HTTP status remains correct when responses arrive in another process, and independent restarts do not lose durable command tracking.

Section 16 — Explore advanced MQTT 5 controls and scaling

Detailed lesson: Section 16 explanations and experiments.

Outcome: You can choose MQTT features based on measured behavior.

1. Experiment with User Properties and subscription options: No Local, Retain As Published, and Retain Handling. Record the relevant packets and observed behavior.
2. Use shared subscriptions with two ingestion instances. Observe distribution and failure recovery; do not assume per-device ordering or instance affinity.
3. Test Receive Maximum and maximum packet-size negotiation in both directions. Distinguish protocol inflight limits from application queue capacity and QoS 0 traffic.
4. Study topic aliases and their connection-scoped state. Verify selected library/broker support before using them as an optimization.
5. Review negative acknowledgements, negotiated capabilities, reconnect backoff/jitter, and unavailable or unsupported features. Confirm that shared consumers coordinate idempotency through durable state.

Done when: You have a feature matrix with purpose, support/version, experiment, result, and decision. You can explain which limits belong to MQTT, the broker, the client library, and your application.

Section 17 — Operate and diagnose the service

Outcome: You can identify whether a failure is in HTTP, the application, broker, device, or database.

1. Add structured logs with stable command/report IDs and safe error categories. Add metrics for reconnects, rejected messages, queue depth, duplicate handling, publish latency, response latency, timeouts, and DB failures.
2. Define liveness and readiness for each process. Avoid high-cardinality device/command IDs as unrestricted metric labels.
3. Create a repeatable simulator load scenario and measure throughput, latency, memory, and queue growth. Profile only after observing a bottleneck.
4. Run a failure matrix: broker down/restart, network loss, simulator offline/restart, expired commands, slow DB, duplicate reports, denied topics, and shutdown under load.
5. Write a local runbook: symptom, evidence to collect, likely layer, recovery action, and how to verify recovery. Include draining/recovery behavior for rolling restarts.

Done when: Another person could reproduce and diagnose a failed command using your runbook without reading all your code. Relevant Go tests and race checks pass for the learning application.

Section 18 — Prepare for the MQTT 5 backend handover

Outcome: You can inspect, explain, and safely maintain the delivered system.

1. Reinspect the delivered commit, dependencies, broker configuration, device protocol contracts, and deployment topology. Build an explicit MQTT 3.1.1-to-5 behavior comparison rather than an import-replacement checklist.
2. Trace one HTTP command to publication and device response, one sale to persistence and application response, and one catalog/configuration synchronization flow.
3. Document device identities and permissions, topic ownership, payload versions, command states, retry rules, expiry, persistence, and process boundaries. Resolve FN/SN/plate terminology from authoritative contracts.
4. Run authorized staging acceptance checks with simulators. Check mixed MQTT 3.1.1/5 interoperability and feature loss where relevant; do not assume legacy devices receive or understand MQTT 5 properties.
5. Review release notes and compatibility before upgrading the client. Propose one narrow maintenance change with reproduction, expected behavior, regression coverage, rollout checks, and rollback criteria.
6. Produce a handover question list for facts the code cannot prove: device acknowledgement semantics, broker persistence policy, recovery targets, ownership, credentials rotation, and operational escalation.

Done when: You can explain each traced flow, diagnose an injected failure, and justify a small fix with evidence. Production migration or rollout is separate work requiring its own scope.

Completion checkpoints

After section	Demonstration
4	Go backend and simulated device exchange MQTT 5 messages
8	HTTP command reaches the device and exposes an accurate status
12	Secured lab demonstrates sales, catalog, configuration, and health flows
15	Persistent API/ingestion processes recover independently
18	You can trace, test, diagnose, and review the delivered backend

How to start a lesson with the tutor

Read prompt.md and COURSE.md. Start Section 1, exercise 1. I know MQTT 3.1.1 concepts. Explain only what I need for this exercise, give me a small task and expected result, then wait for my attempt. Review my result before moving on. Do not generate the complete application.

For later sessions, replace the section and exercise number and include your current code, command output, or question. The tutor should ask about Go experience only where it affects the current exercise; Docker availability is established in Section 1.

Official references

Consult version-matched API documentation during each lesson. The upstream Paho documentation recommends AutoPaho as the starting point and explains its relationship to the lower-level client; Mosquitto's documentation links the broker configuration and CLI manuals.

• Paho repository and overview
• AutoPaho API
• Paho API
• MQTT 5 specification
• Mosquitto documentation and manuals

The course intentionally leaves exact version-sensitive method signatures to each lesson's verification step. Local source references are available without changing the downloaded checkout.