Robot-Market/mqtt-5-learning
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Refactor MQTT course and bound backend message processing

Browse Source 
Move lesson source into the teaching structure, align the course path with the learner's Go and DevOps experience, and add the interactive lesson guides. Add advanced DevOps lifecycle study as a separate section.

Bound MQTT message work and drain accepted jobs during shutdown; align the backend implementation with the revised course through Section 6.
This commit is contained in:
Ryan
2026-10-22 10:19:56 +03:30
parent f9ab91ed60
commit fb30e8e884
70 changed files with 7492 additions and 312 deletions
Download Patch File Download Diff File Expand all files Collapse all files
teaching/course-source/00-system-lifecycle.md
+254
View File
@@ -0,0 +1,254 @@
# 🧭 Lesson 0 — The broker, Go backend, and device lifecycle
> [!toc]- 📑 Contents
>
> - [[#🌐 The whole system|🌐 The whole system]]
> - [[#🖥️ 1. Set up the broker|🖥️ Broker setup]]
> - [[#⚙️ 2. Configure the backend and device|⚙️ Client configuration]]
> - [[#🔌 3. Connect both clients|🔌 Connections]]
> - [[#📬 4. Register subscriptions|📬 Subscriptions]]
> - [[#📨 5. Publish and route messages|📨 Message flow]]
> - [[#🔄 6. Recover and stop|🔄 Recovery and shutdown]]
> - [[#🗺️ Where each part is covered|🗺️ Lesson coverage map]]
> - [[#🧪 Understanding check|🧪 Understanding check]]
> - [[#🔗 Related lessons|🔗 Related lessons]]
## 🌐 The whole system
Picture three independently running programs: a broker, a Go backend, and a device. The backend and device are both MQTT clients. The broker accepts their connections, checks credentials and permissions, stores subscriptions, and routes publications.
The backend may have more application responsibilities, but MQTT does not give it a special connection type. This course applies the MQTT fundamentals you already know to a Go application using MQTT 5.
```mermaid
flowchart LR
C[Broker configuration<br/>listener, accounts, ACL] --> B[Mosquitto broker]
G[Go backend configuration<br/>URL, client ID, credentials] --> GL[Go backend + Paho]
D[Device configuration<br/>URL, client ID, credentials] --> DEV[Device application + MQTT client]
GL <-->|MQTT connection and messages| B
DEV <-->|MQTT connection and messages| B
```
The broker has its own setup and runtime. Each client has its own configuration and runtime. The backend and device do not need each other's network address; each connects to the broker.
---
## 🖥️ 1. Set up the broker
**❓ Question:** **Before a client arrives, how does the broker know where to listen, which credentials to accept, and which topics each identity may use?**
**💬 Answer:** Broker configuration supplies the listener, authentication source, and topic permissions. The clients present credentials; they do not create their own broker accounts.
Those decisions belong to broker configuration or to an authentication/authorization service configured by the broker. In this lab, Mosquitto is configured with a listener, a password file, and an ACL file. The `backend` and `simulator` accounts are created before the clients connect.
```mermaid
flowchart TD
OP[Operator prepares lab files] --> CONF[Listener and authentication config]
OP --> USERS[Backend and simulator accounts]
OP --> ACL[Per-identity topic permissions]
CONF -->|broker startup| B[Mosquitto]
USERS --> B
ACL --> B
```
The broker authenticates a client when it connects. It checks authorization when the client asks to subscribe or publish. Successful authentication does not automatically grant access to every topic. The broker can load local files, or a deployment can configure an external authentication provider; clients do not create their own broker accounts as they arrive.
For this learning environment, the broker is isolated and uses synthetic lab data. Its setup is described in [Lesson 1](01-lab-setup.md) and [Broker operations](broker-operations.md).
📚 **Covered in:** [Lesson 1 — Lab setup](01-lab-setup.md) configures Mosquitto, accounts, and ACLs. [Broker operations](broker-operations.md) covers maintenance; [Lesson 11 — Security](11-security-boundaries.md) develops authentication and authorization boundaries.
---
## ⚙️ 2. Configure the backend and device
**❓ Question:** **If both programs are MQTT clients, what distinguishes the backend's connection from the device's connection?**
**💬 Answer:** Each process has its own Client ID, credentials, configuration, and subscriptions. In this lab, the username selects broker ACL rules; a Client ID identifies an MQTT connection/session and is not a password.
Each process has its own configuration and client identity. Both need a broker URL and credentials accepted by that broker. They also need distinct Client IDs while running at the same time. Their usernames determine which broker ACL rules apply in this lab; Client ID identifies the MQTT connection/session and is not a password.
```mermaid
flowchart LR
T[TOML defaults] --> M[Configuration loader]
E[Environment secrets and overrides] --> M
F[Explicit CLI overrides] --> M
M --> V[Validate effective settings]
V -->|backend identity| BC[Backend client config]
V -->|simulator identity| DC[Device client config]
```
The Go project uses TOML for ordinary settings and `TEST_MQTT5_...` environment variables for credentials and overrides. `config.Load(...)` merges and validates the settings before MQTT startup. The simulator is another program with its own configuration; it is not a child session inside the backend.
Connection settings are distinct from message choices. Keep Alive, session behavior, and a Will are connection options. Topic filters and requested QoS belong to subscriptions. Topic, payload, QoS, and Retain belong to publications.
📚 **Covered in:** [Lesson 2 — Connection](02-go-connection.md) introduces the client settings; [Lesson 3 — Typed configuration](03-typed-configuration.md) builds the loader, precedence, and validation.
---
## 🔌 3. Connect both clients
**❓ Question:** **Once the Go process has configuration, what must it prove before it can claim to be connected?**
**💬 Answer:** It needs an accepted CONNACK. Creating a connection manager is not enough, and a successful connection does not prove that subscriptions or business operations succeeded.
The program parses the broker URL, creates a Paho/AutoPaho configuration, and starts the connection manager. AutoPaho manages the connection lifecycle and reconnect attempts; Paho implements MQTT protocol exchanges. The manager being created is not yet proof of a successful connection. The broker must accept MQTT `CONNECT` with `CONNACK`, and the application can wait for that state with `AwaitConnection`.
```mermaid
sequenceDiagram
participant G as Go backend / Paho
participant B as Broker
participant D as Device / MQTT client
G->>B: TCP connection, then MQTT CONNECT
B->>B: Authenticate backend identity
B-->>G: CONNACK accepted or rejected
D->>B: TCP connection, then MQTT CONNECT
B->>B: Authenticate device identity
B-->>D: CONNACK accepted or rejected
```
The backend and device go through this independently. A successful `CONNACK` proves that a client connection was accepted. It does not prove a subscription was accepted, a device is online, or an application operation completed.
📚 **Covered in:** [Lesson 2 — Connection](02-go-connection.md) follows CONNECT, CONNACK, reconnect, and shutdown. [Lesson 4 — Publish and subscribe](04-publish-subscribe-simulator.md) connects both Go processes.
---
## 📬 4. Register subscriptions
**❓ Question:** **The broker accepted the connection. How does it learn which messages this client wants to receive?**
**💬 Answer:** The client sends SUBSCRIBE topic filters, and the broker reports each filter's result in SUBACK. The local Paho handler is a separate step that processes publications the broker delivers.
The client sends `SUBSCRIBE` requests containing topic filters. The broker checks each request against its rules and returns `SUBACK` results. In the Go application, a local callback is also configured so Paho knows what to do when a matching publication arrives. The broker-side subscription and the Go-side callback are two separate parts of receiving a message.
```mermaid
sequenceDiagram
participant G as Go backend
participant B as Broker
participant D as Device
G->>B: SUBSCRIBE device telemetry and responses
B-->>G: SUBACK per-filter result
D->>B: SUBSCRIBE device commands
B-->>D: SUBACK per-filter result
```
The backend commonly listens for telemetry and responses. The device listens for its commands. The client should inspect the subscription result and establish required subscriptions before publishing work that depends on a receiver already listening.
📚 **Covered in:** [Lesson 4 — Publish and subscribe](04-publish-subscribe-simulator.md) builds the subscriptions and checks readiness. [Lesson 9 — Sessions and recovery](09-sessions-recovery.md) explores when subscriptions survive a reconnect; [Lesson 16 — MQTT 5 controls](16-mqtt5-controls.md) examines subscription options.
---
## 📨 5. Publish and route messages
**❓ Question:** **Once subscriptions exist, how does a command travel from backend code to the device application?**
**💬 Answer:** The backend publishes bytes to a topic, the broker routes them to matching subscriptions, and the device's MQTT client passes the publication to application code. QoS acknowledgements cover a delivery hop; the device's application response carries business evidence.
The backend serializes an application value into bytes, publishes those bytes to a concrete topic, and the broker routes the publication to matching subscribed clients. Paho's receive callback runs inside the client process; the application can copy and enqueue the message for its main loop to handle.
```mermaid
sequenceDiagram
participant G as Go backend
participant B as Broker
participant D as Device
G->>B: PUBLISH device-001/commands, QoS 1
B-->>G: PUBACK for backend-to-broker hop
B->>D: PUBLISH matching command
D-->>B: PUBACK for broker-to-device hop
D->>D: Callback copies/enqueues message
D->>D: Application validates and handles command
D->>B: PUBLISH response on response topic
B->>G: PUBLISH matching response
```
There are separate MQTT delivery hops. A `PUBACK` on the backend's publication confirms the QoS 1 exchange with the broker; it is not a device response. The response is a new application publication. Similarly, receipt by a Go callback does not prove a database write or physical device action.
📚 **Covered in:** [Lesson 4](04-publish-subscribe-simulator.md) implements message exchange. [Lesson 5 — Contracts](05-contracts-validation.md) validates topics and payloads; [Lesson 5A](05a-backend-message-processing.md) turns validated heartbeats into an in-memory device view; [Lesson 6 — Concurrency](06-concurrency-cancellation.md) covers callbacks, queues, and acknowledgement timing; [Lesson 7 — Command/response](07-command-response.md) tracks device results; [Lesson 8 — HTTP commands](08-http-commands.md) exposes the flow to an API caller.
---
## 🔄 6. Recover and stop
**❓ Question:** **What should happen if the broker or network disappears while the backend process remains alive?**
**💬 Answer:** The client can reconnect, but the application must ensure needed subscriptions are present. Offline delivery depends on session settings, QoS, expiry, and broker state, so reconnect alone does not guarantee that every message is recovered.
AutoPaho can reconnect according to its lifecycle policy. After a connection comes back, the application must ensure required subscriptions are present—either because the broker resumed a session or because the client re-subscribed. In this lab, Lesson 4's helper explicitly subscribes again on connection-up. Whether offline publications arrive depends on MQTT session settings, QoS, broker state, and expiry; reconnect alone does not promise recovery of every message.
On shutdown, the application cancels its root context, stops background work, and requests a bounded disconnect. Clean MQTT shutdown, broker session deletion, and completion of business work are different events.
```mermaid
stateDiagram-v2
[*] --> Configured
Configured --> Connecting
Connecting --> Connected: accepted CONNACK
Connecting --> Retry: connect error / rejected attempt
Retry --> Connecting: AutoPaho retries
Connected --> Subscribing: connection-up handler
Subscribing --> Ready: acceptable SUBACK
Ready --> Retry: connection lost
Ready --> Stopping: cancellation signal
Stopping --> [*]: bounded disconnect
```
📚 **Covered in:** [Lesson 2](02-go-connection.md) introduces reconnect and bounded shutdown; [Lesson 4](04-publish-subscribe-simulator.md) restores subscriptions after connection-up; [Lesson 6](06-concurrency-cancellation.md) coordinates worker draining and shutdown. [Lesson 9](09-sessions-recovery.md) measures session recovery, and [Lesson 10](10-expiry-presence-state.md) covers expiry and presence.
---
## 🗺️ Where each part is covered
| Lifecycle area | Lessons | Evidence to understand |
|---|---|---|
| Broker setup, credentials, and topic permissions | [1](01-lab-setup.md), [11](11-security-boundaries.md), [11B](11b-dynamic-security.md) optional | Mosquitto owns broker authentication and topic ACL enforcement; optional live per-device provisioning |
| Go client configuration | [2](02-go-connection.md), [3](03-typed-configuration.md) | Effective settings, identity, validation, and precedence |
| CONNECT, CONNACK, reconnect, and shutdown | [2](02-go-connection.md), [4](04-publish-subscribe-simulator.md), [9](09-sessions-recovery.md) | What connection acceptance proves and what must be restored |
| Topic names, filters, wildcards, and routing | [1](01-lab-setup.md), [4](04-publish-subscribe-simulator.md), [16](16-mqtt5-controls.md) | Concrete topic names, `+`/`#` filter matching, broker routing, and subscription options |
| Subscriptions and broker routing | [4](04-publish-subscribe-simulator.md), [9](09-sessions-recovery.md), [16](16-mqtt5-controls.md) | SUBACK results, matching filters, session restoration, shared subscriptions, and options |
| QoS 0/1/2, duplicates, and acknowledgement scope | [4](04-publish-subscribe-simulator.md), [6](06-concurrency-cancellation.md), [7](07-command-response.md), [16](16-mqtt5-controls.md) | Per-hop packet evidence, DUP/retries, and why QoS does not make business effects exactly once |
| Topic and payload validation in Go | [5](05-contracts-validation.md) | `ParseTopic` checks topic shape and kind; `Decode` validates JSON, schema, route/body device identity, and message values before handling |
| Application state after validation | [5A](05a-backend-message-processing.md) | A valid heartbeat updates a simple in-memory device view; freshness is distinct from connection and machine health |
| Callback work, queues, and overload | [6](06-concurrency-cancellation.md) | Admission, acknowledgement timing, bounded capacity, and shutdown |
| Reliable command/response and HTTP | [7](07-command-response.md), [8](08-http-commands.md) | Correlation, idempotency, timeout, and status reporting |
| Sessions, retained state, expiry, and presence | [9](09-sessions-recovery.md), [10](10-expiry-presence-state.md) | Offline delivery prerequisites, retained desired state, message expiry, Will, and heartbeat freshness |
| MQTT 5 flow controls and capabilities | [16](16-mqtt5-controls.md) | Receive Maximum, packet limits, reason codes, aliases, and capability boundaries |
The table points to the lessons that explain and exercise each part of the lifecycle. Lessons 1–11 and 16 contain detailed instructions; Sections 12–15 and 17–18 in [the course plan](../../COURSE.md) are currently guided checklists that extend the story into vending workflows, persistence, process boundaries, and operations.
### What validates a topic?
Lesson 4 introduces `contracts.DeviceTopic`, fleet-wide `contracts.DeviceFilter`, and exact-device `contracts.DeviceSubscription` in `internal/contracts/topic.go`; Lesson 5 adds `contracts.ParseTopic` there and message `Validate` methods in `message.go`. The route rejects malformed paths and selects the expected payload contract, which then checks that its claimed `device_id` matches the topic before application handling.
Those checks do not authorize the publisher. A client with valid credentials could still try to publish to another device's topic, so Mosquitto must enforce a least-privilege ACL for both publish and subscribe operations. Lesson 11 tests that broker boundary and the separate HTTP-user authorization boundary; optional Lesson 11B covers per-device ACL provisioning. Authentication identifies a broker client, ACLs authorize topic operations, and Lesson 5 validates message content. Use all three boundaries.
---
## 🧪 Understanding check
1. What does a successful `CONNACK` prove, and what does it not prove?
2. How are a topic name, a subscription filter, and an ACL rule different?
3. Why does the backend need both a broker subscription and a local receive handler?
4. Which acknowledgement tells the backend that the device application completed a command?
5. What is the difference between backend topic parsing/validation and broker ACL authorization?
6. What is the difference between a client credential and a Client ID in this lab?
> [!answer]- 📋 Answers
>
> **1.** It proves the broker accepted that MQTT connection. It does not prove a subscription was accepted, a device is online, or business work completed.
>
> **2.** A topic name labels one publication route. A filter can match multiple topic names when the broker routes to a subscription; `+` matches one level and `#` matches the remaining levels. An ACL rule permits or denies a client's publish or subscribe operation.
>
> **3.** The broker subscription controls routing; the local handler controls what Paho/the application does with a delivered PUBLISH.
>
> **4.** None of the MQTT packet acknowledgements proves application completion. The device must publish an application-level response, and the backend must validate it.
>
> **5.** Backend validation checks message shape and consistency after delivery. Broker ACLs decide whether that client may publish or subscribe to the topic at all. Payload validation does not replace authorization.
>
> **6.** Credentials authenticate a client and select its ACL identity here. Client ID identifies the MQTT connection/session and can be configured as an ACL input, but it is not inherently authentication.
---
## 🔗 Related lessons
- [Lesson 1 — Isolated lab setup](01-lab-setup.md)
- [Lesson 2 — Go connection](02-go-connection.md)
- [Lesson 3 — Typed configuration](03-typed-configuration.md)
- [Lesson 4 — Publish, subscribe, simulator](04-publish-subscribe-simulator.md)
- [Course index](../../COURSE.md)
teaching/course-source/01-lab-setup.md
+399
View File
@@ -0,0 +1,399 @@
# 🛠️ Section 1 — Prepare the MQTT 5 lab
Read this first. You will create the files and run the commands yourself. All paths below are relative to the learning project root. The examples use the module path `example.com/mqtt5-learning`; if you change it, change the imports in later lessons too.
**Version baseline:** Paho `v0.23.0`, verified from upstream release tags on 2026-10-07. It requires Go 1.24 or newer; Go 1.25 is compatible. The matching local reference is `references/paho.golang-v0.23.0/` (commit `1aa03969c5932464359b3e4ed752d9ac61abccc9`).
## 🎬 Start with the system story
> 💡 **Story question:** Before we write a Go client, what must already exist so the backend and a simulated device can connect safely and exchange messages?
The broker is a separately configured service. It needs a listener, an authentication source, topic permissions, and storage settings. The backend and device are separate clients with their own connection settings and credentials. This lesson prepares the local broker and verifies its behavior with MQTT 5 CLI clients before adding Go to the path.
```mermaid
flowchart LR
O[Prepare Compose and Mosquitto config] --> B[Mosquitto broker starts]
P[Create backend and simulator credentials] --> B
A[Define ACL topic permissions] --> B
G[Go backend comes in a later lesson] -. connects .-> B
D[Go simulator comes in a later lesson] -. connects .-> B
C[Mosquitto CLI verifies MQTT 5] --> B
```
The Compose file starts and isolates the broker; the Mosquitto configuration tells the broker how to listen and enforce authentication/ACLs. A Docker volume preserves broker-owned state. None of these files configures the Go clients—that happens in later lessons.
For the vending-backend goal, this first lesson establishes only the broker boundary: which client may connect, which topics it may use, and how to observe broker activity. It does not yet store a device's status, ingest its diagnostic logs, or execute a vending command. Those are application responsibilities built after the Go clients can exchange messages.
Before each setup step, ask which process owns the setting and what observation will prove it took effect.
## 📁 Working directory and file placement
Run every shell command in this lesson from the learning project root:
```sh
cd /home/ryan/Media/Code/Code/robot-market/mqtt-5-learning
```
Paths such as `cmd/backend/main.go` are relative to the repository root. The Markdown implementation lessons are under `teaching/course-source/`; create application files at the paths shown, not inside `teaching/course-source/` or `references/`. Code blocks labeled Go, TOML, YAML, or conf are file contents; shell blocks are terminal commands. Create parent directories before saving files.
## 🛠️ 1. Check the tools and create the module
> 💡 **Story question:** What must be available on this machine to build the Go client and start the isolated broker, and which Paho API speaks MQTT 5?
Check the Go toolchain and both Docker components, then pin the MQTT 5 module in this learning project. The old `paho.mqtt.golang` client is a separate MQTT 3 API.
```sh
go version
docker version
docker compose version
go mod init example.com/mqtt5-learning
go get github.com/eclipse/paho.golang@v0.23.0
```
`go version` reports the installed compiler. `docker version` reports both CLI and daemon versions: a missing server response means the CLI cannot reach the Docker engine. `docker compose version` checks the Compose plugin, which reads `compose.yaml` and manages the application's services. Having Docker installed does not prove the daemon is running.
`go mod init example.com/mqtt5-learning` creates `go.mod`, declaring this application's import-path prefix. It does not create the source directories. `go get ...@v0.23.0` downloads a selected dependency and records module requirements and checksums in `go.mod`/`go.sum`; it may also update transitive requirements needed by that dependency.
Run `go mod init` only once. `go get` pins the dependency in `go.mod`; commit `go.mod` and `go.sum` when you initialize version control. Do not use a local `replace` directive to substitute the newer default-branch reference. A repository checkout is reading material, while a module requirement selects the compiled library.
The old backend's `github.com/eclipse/paho.mqtt.golang` is a different MQTT 3 client. MQTT 5 uses `github.com/eclipse/paho.golang/autopaho` and `/paho`. There is no `SetProtocolVersion(5)` needed in these MQTT 5 examples.
## 🦟 2. Create the broker files
> 💡 **Story question:** What configuration belongs to Mosquitto itself, and how will host applications reach the broker without exposing the lab listener to the network?
Compose controls the container boundary and port mapping. Mosquitto's own files control its listener, credentials, ACL, and persistence. Keep those roles distinct as you create the files below.
Create these directories:
```sh
mkdir -p lab/mosquitto
mkdir -p lab/secrets
```
`mkdir` creates directories; `-p` creates missing parents and tolerates an already-existing directory. These commands do not create configuration files.
Create `compose.yaml` directly in the project root, next to `go.mod`. Its full path is `/home/ryan/Media/Code/Code/robot-market/mqtt-5-learning/compose.yaml`. The `./lab/...` volume paths below are relative to this Compose file:
```text
mqtt-5-learning/
├── go.mod
├── compose.yaml
└── lab/
├── mosquitto/
│ ├── mosquitto.conf
│ └── acl
└── secrets/
└── passwords # generated in step 3, not written by hand
```
Your existing `compose.yml` is also a supported Compose filename; keep it rather than creating a competing `compose.yaml`. Use one primary Compose file consistently. YAML nesting is controlled by spaces: the final `volumes:` declaration must align with `services:`, not with `broker:`. If nested under `services`, Compose treats `volumes` as another service and reports `services.volumes additional properties 'broker-data' not allowed`.
Paste this YAML into your primary Compose file:
```yaml
services:
broker:
image: eclipse-mosquitto:2.0.22
ports:
- "127.0.0.1:18883:1883"
volumes:
- ./lab/mosquitto/mosquitto.conf:/mosquitto/config/mosquitto.conf:ro
- ./lab/mosquitto/acl:/mosquitto/config/acl:ro
- ./lab/secrets:/mosquitto/secrets:ro
- broker-data:/mosquitto/data
volumes:
broker-data:
```
This is a pinned Mosquitto 2 lab image, not a claim that it is the newest broker release. Host port 18883 maps to container port 1883. Containers use `broker:1883`; Go programs on your host use `127.0.0.1:18883`. Loopback binding keeps this listener off your LAN. Persistence belongs to this Compose project only.
### 🐳 Understand the Compose definition
`services` lists the containers in this application; `broker` is the service name you use in commands and the DNS name on this Compose network. `image` selects a prebuilt image; Docker downloads it if needed. There is no `build` section because this lesson does not build a broker image.
`ports` uses `host-address:host-port:container-port`. `127.0.0.1:18883:1883` forwards connections from this computer's loopback port 18883 to the container's listener at 1883. A host client cannot use container port 1883 unless that port is separately published. `broker:1883` works for containers on the same Compose network, not arbitrary host programs or remote devices.
Each `volumes` entry is `source:container-destination[:mode]`. Sources beginning with `./` are host files/directories (bind mounts); `:ro` makes them read-only to this container. Create the configuration files before starting Docker: missing file paths can result in directories being mounted where files were expected. `broker-data` is a Docker-managed named volume, declared at the bottom, for writable broker persistence. It is separate from the image and survives ordinary container replacement.
Create `lab/mosquitto/mosquitto.conf`:
```conf
listener 1883
allow_anonymous false
password_file /mosquitto/secrets/passwords
acl_file /mosquitto/config/acl
persistence true
persistence_location /mosquitto/data/
log_dest stdout
connection_messages true
```
### 🦟 Understand each broker setting
| Setting | Purpose and operational consequence |
| --- | --- |
| `listener 1883` | Opens the MQTT listener inside the container; host exposure is controlled separately by Compose |
| `allow_anonymous false` | Requires authentication; it does not grant topic permissions |
| `password_file ...` | Reads the generated username/password hashes at this container path |
| `acl_file ...` | Reads topic permissions; a correct password alone does not authorize every topic |
| `persistence true` | Enables saving broker state, including eligible retained/session information; it does not persist Go application state |
| `persistence_location ...` | Places saved broker data in the named volume rather than the disposable container filesystem |
| `log_dest stdout` | Sends broker logs to Docker's log stream for `compose logs` |
| `connection_messages true` | Logs client connection/disconnection events for troubleshooting |
Persistence is not a database transaction for your application. Recently changed broker state may not survive an abrupt crash until it has been saved. Session settings and message QoS still decide which offline data is eligible to survive.
Create `lab/mosquitto/acl`:
```conf
user backend
topic write lab/v1/devices/+/commands
topic read lab/v1/devices/+/telemetry
topic read lab/v1/devices/+/responses
user simulator
topic read lab/v1/devices/device-001/commands
topic write lab/v1/devices/device-001/telemetry
topic write lab/v1/devices/device-001/responses
```
`user` starts rules for that authenticated username. `topic write` permits publishing; `topic read` permits receiving matching topics. `+` matches exactly one topic level, so the backend rules cover one device-ID segment. The simulator's rules use an exact device ID. Access outside the allowed patterns is denied by this simple ACL configuration. The text file has no extension because Mosquitto reads the path you configure, regardless of its filename suffix.
These ACL patterns intentionally mirror the `lab/v1/devices/{deviceID}/{kind}` hierarchy that Section 4 will define in `internal/contracts/topic.go`. The broker reads its own ACL text; it does not import the Go package. Keep both sides aligned and test denied cross-device access, because a valid Go `TopicName` is not a broker permission.
The simulator can act only as `device-001`. The backend can operate all lab devices. ACL checks authorize topics; they do not validate JSON or prove that a payload's identity is truthful. The CLI checks below must use a direction permitted for that username.
Add to `.gitignore` (preserve any existing rules):
```gitignore
/lab/secrets/
/.env
```
Git ignore patterns are file contents, not terminal commands. `/lab/secrets/` ignores the root-level directory and its contents; `/.env` ignores a root environment file. Ignoring a file does not untrack a secret already committed. Keep real secrets out of the initial commit as well as later changes.
## 🔐 3. Create passwords interactively
> 💡 **Story question:** How will the broker recognize the backend and simulator when they connect, and how can those identities receive different topic permissions?
Create separate broker accounts and ACL rules before starting clients. The client supplies credentials; the broker validates them against its configured source. Authentication permits connection, while ACL rules govern topic operations.
These commands create a lab password file without putting the password literal in your shell history. Run them from the project root:
```sh
docker run --rm -it --user 0 -v "$PWD/lab/secrets:/work" eclipse-mosquitto:2.0.22 mosquitto_passwd -c /work/passwords backend
docker run --rm -it --user 0 -v "$PWD/lab/secrets:/work" eclipse-mosquitto:2.0.22 mosquitto_passwd /work/passwords simulator
docker run --rm --user 0 -v "$PWD/lab/secrets:/work" eclipse-mosquitto:2.0.22 chown mosquitto:mosquitto /work/passwords
docker run --rm --user 0 -v "$PWD/lab/secrets:/work" eclipse-mosquitto:2.0.22 chmod 600 /work/passwords
```
`-c` creates/replaces the file. Use it only for the first username; using it again would erase the backend entry. The last two commands make the file readable by the broker account in this image. Container-root use is limited to setting up this mounted lab file.
### 🔧 Why four Docker commands?
There are two jobs here: generate password hashes for two users, then give the broker permission to read the resulting file. Docker adds a wrapper because we use the tool bundled with the pinned broker image instead of installing Mosquitto utilities on your computer. Each command runs a temporary utility container; none starts the broker.
| Part of the command | What it does |
| --- | --- |
| `docker run` | Creates and runs a new container for this one operation |
| `--rm` | Removes that container when its command exits; the mounted host files remain |
| `-it` | Keeps standard input open and supplies a terminal for the password prompt |
| `--user 0` | Runs this setup operation as container root, permitting file creation and ownership changes on the mount |
| `-v "$PWD/lab/secrets:/work"` | Makes the host project's `lab/secrets` directory available at `/work` inside the container; this mount is writable |
| `eclipse-mosquitto:2.0.22` | Selects the same pinned image used by the broker |
| `mosquitto_passwd -c /work/passwords backend` | Creates the password file and prompts for the backend user's password |
| `mosquitto_passwd /work/passwords simulator` | Adds the simulator user to that same file, without replacing the backend entry |
| `chown mosquitto:mosquitto /work/passwords` | Assigns file ownership to the broker account as defined inside this image |
| `chmod 600 /work/passwords` | Gives the owner read/write access and removes group/other access |
`$PWD` is your current terminal directory, so run these from the project root. `/work/passwords` in the setup container and `lab/secrets/passwords` on your host are the same file. Later, Compose mounts that directory at `/mosquitto/secrets` with `:ro`, and `mosquitto.conf` reads `/mosquitto/secrets/passwords`. The setup mount is writable; the running broker mount is read-only.
The generated file stores password hashes, not the plaintext passwords you type. Your backend and simulator still need their original passwords to connect; a client cannot use the stored hash as its login password. Keep the generated file out of Git. The ACL file separately determines which topics each authenticated username may use.
Linux checks numeric user/group IDs. `chown` inside this image picks the broker's IDs; your host may display a numeric owner or a different name for those same IDs. With mode `600`, your regular host user may be unable to read the file afterward. That is expected: the broker must read it, and subsequent password maintenance can use the same temporary root utility container. Do not make the file world-readable to work around this.
### 💡 What is usual practice?
For a small Mosquitto 2.0 installation, generating a password file with `mosquitto_passwd`, configuring `password_file`, and restricting file access is a conventional approach. The official [authentication guide](https://mosquitto.org/documentation/authentication-methods/) documents creating the first user with `-c` and adding or updating users without it. Docker is simply how this lesson obtains that utility.
- **Local Docker lab:** Generate the file once with a temporary container, mount it into the broker, and repeat the update command only when adding users or rotating passwords. You do not rerun all four commands on each broker start. After modifying the file, reapply the ownership and mode if the utility replaced it, then reload authentication or restart only the lab broker.
- **Host-installed Mosquitto:** Run the installed `mosquitto_passwd` utility directly, then set permissions for the broker's service account. The core operations are the same; there is no container wrapper.
- **Repeated deployment:** Put the provisioning steps in a small reviewed setup script or dedicated setup job. Supply credentials through the deployment's secret-management mechanism and mount the resulting authentication file read-only. A secret mount still needs the correct broker-readable ownership and permissions. Avoid embedding passwords in image layers, committed Compose files, or command arguments.
- **Larger device fleets:** Evaluate managed authentication and authorization, such as Mosquitto's [Dynamic Security plugin](https://mosquitto.org/documentation/dynamic-security/), rather than maintaining a large static file manually. This is a deployment choice, not something needed for this two-user exercise.
The four explicit commands make the setup visible for learning. Once you understand them, wrapping them in a setup script is reasonable. We keep interactive prompts here so password literals do not appear in shell history or process arguments.
### 🧪 Verify password setup before starting
Do not start the broker until both password prompts and both permission commands have completed successfully. The two accounts must exist in the generated file; creating an empty file does not create usable credentials.
Verify existence and access without printing hashes:
```sh
docker run --rm --user mosquitto -v "$PWD/lab/secrets:/work:ro" eclipse-mosquitto:2.0.22 sh -c 'test -s /work/passwords && test -r /work/passwords'
```
`--user mosquitto` tests access as the image's broker account. `sh -c` runs the quoted checks inside the utility container; `test -s` requires a nonempty file, `test -r` requires read access, and `&&` runs the second check only if the first succeeds. Success is quiet with exit status zero. Failure means you must fix the file/path/permissions before starting. This checks file access, not whether the typed client password is correct.
### 🔐 If permission changes do not stick on the host filesystem
Verify final metadata with `stat -c '%a %u:%g %n' lab/secrets/passwords`. `stat` reports mode, numeric owner/group, and pathname without exposing hashes. Also use `findmnt -T lab/secrets/passwords -o TARGET,FSTYPE,OPTIONS` to identify the underlying mount. Some mounted filesystems synthesize permissions: `chmod`/`chown` can appear successful while the metadata remains unchanged. Do not conclude the file is protected from the commands' exit status alone.
This workspace exhibited that behavior on its `fuseblk` mount: the password file remained `755` and `1000:1000`. The current `compose.yml` therefore uses a separately provisioned Docker volume for credentials. To reproduce that fallback after generating the users:
```sh
docker run --rm --user 0 -v "$PWD/lab/secrets:/source:ro" -v mqtt5-learning_broker-secrets:/target eclipse-mosquitto:2.0.22 sh -c 'cp /source/passwords /target/passwords && chown mosquitto:mosquitto /target/passwords && chmod 600 /target/passwords && stat -c "%a %u:%g" /target/passwords'
```
The host source is read-only. The new Docker-managed target is writable; `cp` copies the existing file and the following commands apply permissions there. This command replaces the target password file, so use it only for initial provisioning or an intentional replacement. The verified result in this image is `600 1883:1883`; always inspect the actual result for your environment. A named volume is not automatically secure or encrypted: its actual filesystem and file permissions still matter.
Replace the service's `./lab/secrets:/mosquitto/secrets:ro` mount with `broker-secrets:/mosquitto/secrets:ro`. Declare this additional volume at the root, alongside `broker-data`:
```yaml
volumes:
broker-data:
broker-secrets:
external: true
name: mqtt5-learning_broker-secrets
```
`name` specifies the already-provisioned Docker volume's exact name. `external: true` tells Compose to use it rather than create/delete it. The preceding utility run creates it if missing. Apply the changed mount with `up -d broker` and inspect recent logs. Verify access with the earlier read-check command but replace its host mount with `-v mqtt5-learning_broker-secrets:/work:ro`.
For every subsequent add-user, rotation, or permission command, mount `mqtt5-learning_broker-secrets:/work` rather than `$PWD/lab/secrets:/work`: the broker now reads the volume copy. Editing the old host file does not update the live credential store. Include this external volume in backups; Compose `down -v` does not delete external volumes. The old host copy remains sensitive and ignored by Git; it has been preserved rather than automatically deleted.
### 🔐 Understand password-tool and entrypoint warnings
During creation or update, `mosquitto_passwd` can warn about world-readable permissions or ownership. Do not assume these warnings prove the final file is correct: run the ownership/mode commands afterward and verify access. For this pinned Mosquitto 2.0.22 image, use `mosquitto:mosquitto` ownership and mode `600`. A password file needs no executable permission. Blindly following a warning suggesting root ownership with owner-only access can prevent the unprivileged broker from reading it. Review requirements again when upgrading rather than treating a future-version warning as this version's deployment instructions.
The image entrypoint can also print `chown: ... Read-only file system` for the intentional `:ro` mounts. Those lines alone are not a broker authentication failure. Keep configuration and secrets read-only; identify the subsequent Mosquitto messages. `Error opening password file` is fatal and must be fixed. A directory can exist while the required `passwords` file inside it is missing.
Start the broker for the first time:
```sh
docker compose -p mqtt5-learning config
docker compose -p mqtt5-learning up -d
docker compose -p mqtt5-learning ps
docker compose -p mqtt5-learning logs --tail=50 broker
```
| Command part | What happens and what to check |
| --- | --- |
| `-p mqtt5-learning` | `-p` means project name in Compose, not network port. Selects the Compose project name, keeping its containers/network/volume names separate from other projects; use this same name for every lifecycle command |
| `config` | Parses and resolves Compose configuration; catches YAML/interpolation errors, but does not test broker authentication or every mounted file's contents |
| `up -d` | Creates/updates the required resources and starts containers in the background; `-d` means detached |
| `ps` | Shows container state and published ports; a running container is not an MQTT readiness check |
| `logs --tail=50 broker` | Shows the last 50 log lines for this service without opening a shell inside it |
If a container was already created or ran before you fixed its password file, explicitly retry startup:
```sh
docker compose -p mqtt5-learning restart broker
docker compose -p mqtt5-learning logs --since=1m broker
docker compose -p mqtt5-learning ps
```
`restart` stops/starts the existing service even if it has stopped after an error. `up -d` is not a guaranteed restart of an unchanged running container. `--since=1m` restricts logs to the last minute so earlier failures do not get mistaken for a fresh failure. If there is no existing container, use the initial `up -d` sequence instead. If you changed Compose ports or mounts, use `up -d` to reconcile them; if a replaced single-file mount still points at old content, use `up -d --force-recreate broker`.
Expect a listening broker and no new password/ACL file errors. A running container alone does not prove authentication works.
For this project, resource names commonly include `mqtt5-learning-broker-1`, `mqtt5-learning_default`, and `mqtt5-learning_broker-data`. Keep `-p mqtt5-learning` on every lifecycle command so they refer to the same project. In `mosquitto_pub/sub`, `-p` instead means port; in `mkdir -p`, it means create parent directories. Flags belong to their commands, not to a universal vocabulary.
In `ps`, the local-only mapping should show `127.0.0.1:18883->1883/tcp`. A mapping showing `0.0.0.0` or `[::]` exposes that published port on all corresponding host interfaces. Change the Compose mapping to the loopback form and apply it with `up -d broker`; restarting alone does not change port bindings.
## 🧪 4. Verify MQTT 5 without Go first
> 💡 **Story question:** How can we prove that the broker is listening, speaks the intended MQTT version, accepts the right credentials, and enforces topic permissions before debugging Go code?
Use Mosquitto CLI clients as a controlled first pair of MQTT clients. Their packet traces isolate broker reachability, protocol negotiation, authentication, authorization, and routing from application code.
Use host-installed `mosquitto_sub` and `mosquitto_pub`, or run their equivalents through `docker compose exec broker` with port 1883 inside the container. Keep a subscription open in terminal A:
```sh
mosquitto_sub -h 127.0.0.1 -p 18883 -V mqttv5 -u backend -P '<backend-lab-password>' -t 'lab/v1/devices/device-001/telemetry' -q 1 -v -d
```
In terminal B:
```sh
mosquitto_pub -h 127.0.0.1 -p 18883 -V mqttv5 -u simulator -P '<simulator-lab-password>' -t 'lab/v1/devices/device-001/telemetry' -q 1 -m '{"online":true}' -d
```
`-h` chooses the broker host; `-p` here chooses the network port (unlike Compose's project-name flag). `-u` supplies a username, `-P` its password, `-t` the subscription filter or publication topic, and `-m` the message body. Single quotes preserve the JSON bytes instead of letting the shell interpret them. Start the subscriber first so it is ready before the nonretained publication. `mosquitto_sub` keeps running; Ctrl+C stops this subscriber. `mosquitto_pub` normally exits after completing its publication.
If using `docker compose -p mqtt5-learning exec broker`, that prefix executes the utility inside the already-running broker container. Use `-h 127.0.0.1 -p 1883` there, because loopback refers to that container. The other MQTT flags remain the same. `exec` requires a running service; the earlier `docker run` setup commands do not.
The placeholders are not literal passwords. `-V mqttv5` selects the protocol; `-q 1` selects QoS; `-d` shows packet activity; `-v` prints topic plus payload. Expect the subscriber to print the telemetry JSON. This is a synthetic message, not a production heartbeat contract.
For the CLI only, `-P` exposes its argument to shell history/process inspection. Use temporary lab credentials and avoid pasting real credentials; consult your installed client's options for a protected configuration file if you need repeated CLI checks. Go examples read the password from the environment and never print it.
Repeat with a wrong password. Also try publishing as `simulator` to `lab/v1/devices/device-002/telemetry`; this must not reach an authorized observer. Check the MQTT reason and broker logs rather than assuming every denied operation has the same client error.
### 📘 Read the CLI exchange, not just the payload
With `-d`, identify CONNECT/CONNACK first, then SUBSCRIBE/SUBACK in the subscriber terminal. Only publish after SUBACK. In the publisher terminal, find PUBLISH/PUBACK at QoS 1. In the subscriber terminal, find the separately forwarded PUBLISH and its acknowledgement. Debug formatting varies by client version; learn the packet names rather than memorizing one output line.
Predict before running: stop the subscriber, publish again with the same command, then restart the subscriber. This nonretained message is not a history entry for a new subscriber. A successful publisher acknowledgement does not create a future subscription or a device response. [Lesson 4](04-publish-subscribe-simulator.md) maps these exchanges to Go calls; [Lesson 9](09-sessions-recovery.md) introduces the different case of an existing offline session.
<details>
<summary>Check your explanation: why start the subscriber first?</summary>
The broker needs an established matching subscription when this nonretained message arrives. Starting two terminals almost simultaneously leaves a race between SUBSCRIBE and PUBLISH. Seeing the subscriber process running is weaker evidence than seeing its successful SUBACK.
</details>
## ⚙️ API preview: connect each setting to its purpose
```go
broker, err := url.Parse("mqtt://127.0.0.1:18883")
// Check err before using broker.
cfg := autopaho.ClientConfig{
ServerUrls: []*url.URL{broker},
KeepAlive: 20,
ClientConfig: paho.ClientConfig{ClientID: "lab-backend"},
}
```
This is an illustrative fragment, not a standalone program. The URL chooses the transport endpoint, while Client ID selects the broker session identity. `KeepAlive` is an integer number of seconds, not a `time.Duration`. The embedded `ClientConfig` describes the underlying MQTT client; the outer configuration describes managed connections.
## 🧪 Exercises and questions
1. Explain why switching the Go library is more than changing one import.
2. Predict whether the simulator may subscribe to another device's commands; verify it.
3. Identify which packet proves a QoS 1 publication reached its immediate receiver, and why it does not prove application processing.
4. Stop and start only this lab with `docker compose -p mqtt5-learning stop` and `docker compose -p mqtt5-learning start`.
**Completion:** CLI publish/subscribe works using MQTT 5, invalid credentials fail, and cross-device writes are denied. Do not use `down -v` unless you deliberately want to erase the lab's broker persistence.
## ✅ Result — the story so far
The broker is now the meeting point for the lab, listening only on this computer. It checks each client's username and password, then uses the ACL to decide which topic directions that identity may use. MQTT 5 CLI clients can prove the broker routes an allowed message and rejects an unauthorized one. The Go backend and simulator have not joined the story yet; this lesson prepares the place they will meet.
## 🦟 Broker administration you should practice
Read [Broker operations](broker-operations.md) for explained stop/start, restart versus recreate, password rotation, backups, restore, upgrades, and the steps required before remote deployment. Complete its local lifecycle and credential-rotation exercises before treating Section 1 as finished. Later Section 9 measures session recovery, and Section 11 in the course adds TLS and stronger access controls.
## 🔎 Common failures
| Symptom | Check |
| --- | --- |
| Port is already allocated | Choose another loopback host port and update all host clients |
| Connection refused | Container state, listener logs, and host/container port distinction |
| Password file missing | Complete password generation before startup; verify `lab/secrets/passwords` exists and is nonempty |
| Password file unreadable | Verify broker-account read access, ownership/mode, mounted path, and directory traversal permissions |
| Read-only `chown` warnings | Check subsequent broker logs; retain `:ro` mounts and fix real read failures separately |
| Old error still appears in logs | Use `logs --since=1m` immediately after retrying startup |
| `services.volumes` validation error | Align the named-volume declaration with root-level `services:` |
| Port shows `0.0.0.0` or `[::]` | Set `127.0.0.1:18883:1883` and apply with `up -d broker` |
| Not authorized | Username, password, and exact permitted topic direction |
| Pub/sub works but Go fails later | Go URL/credentials, client ID, then tagged API version |
## 🔗 Sources
- [Paho v0.23.0 release](https://github.com/eclipse-paho/paho.golang/releases/tag/v0.23.0)
- [Mosquitto broker configuration](https://mosquitto.org/man/mosquitto-conf-5.html)
- [Password-file tool](https://mosquitto.org/man/mosquitto_passwd-1.html)
- [Mosquitto publish tool](https://mosquitto.org/man/mosquitto_pub-1.html)
Next: [Section 2](02-go-connection.md).
teaching/course-source/02-go-connection.md
+276
View File
@@ -0,0 +1,276 @@
# 🔌 Section 2 — Understand a Go MQTT 5 connection
Prerequisite: Section 1's broker and backend credentials work. This section contains a complete small program. Type it in stages, compile after each meaningful change, and explain the code before running the failure exercises.
**Version baseline:** Paho `v0.23.0`, verified from upstream release tags on 2026-10-07. It requires Go 1.24 or newer; Go 1.25 is compatible. The matching local reference is `references/paho.golang-v0.23.0/` (commit `1aa03969c5932464359b3e4ed752d9ac61abccc9`).
## 🎬 Start with the system story
> 💡 **Story question:** The broker is running and has accepted CLI clients. How does the Go backend become another MQTT client, and what event proves the broker accepted its connection?
The Go process loads its connection values, gives AutoPaho a client configuration, and starts a connection manager. AutoPaho manages the connection lifecycle; the underlying Paho client performs MQTT 5 protocol exchanges. The manager can start before the network connection succeeds, so the application waits for a successful CONNACK through `AwaitConnection`.
```mermaid
sequenceDiagram
participant App as Go backend application
participant Auto as AutoPaho manager
participant Paho as Paho MQTT 5 client
participant Broker as Mosquitto broker
App->>Auto: NewConnection(lifecycle context, config)
Auto->>Paho: create and manage client connection
Paho->>Broker: TCP connection, then CONNECT
Broker->>Broker: authenticate client credentials
Broker-->>Paho: CONNACK accepted or rejected
Paho-->>Auto: connection state / callback
App->>Auto: AwaitConnection(startup deadline)
Auto-->>App: connected or startup error
```
The backend is an MQTT client, just like the device. Its application role does not make the broker handshake different. A successful CONNACK proves connection acceptance; subscriptions, device availability, and business completion are later evidence. See [Lesson 0](00-system-lifecycle.md) for the complete lifecycle.
## 📁 Working directory and file placement
Run every shell command in this lesson from the learning project root:
```sh
cd /home/ryan/Media/Code/Code/robot-market/mqtt-5-learning
```
Paths such as `cmd/backend/main.go` are relative to the repository root. The Markdown implementation lessons are under `teaching/course-source/`; create application files at the paths shown, not inside `teaching/course-source/` or `references/`. Code blocks labeled Go, TOML, YAML, or conf are file contents; shell blocks are terminal commands. Create parent directories before saving files.
## 🔌 1. Build the connection program
> 💡 **Story question:** Which settings and lifecycle controls must the Go process pass to the client library so a connection attempt can start and stop predictably?
The program builds one AutoPaho configuration from the broker URL, credentials, Client ID, connection options, and callbacks. The root context owns the manager lifetime; a separate startup context bounds how long the application waits for the first connection.
Create the directory first:
```sh
mkdir -p cmd/backend
```
Create `cmd/backend/main.go` and paste the complete program below into it. If the file already exists, replace its contents rather than appending a second program:
```go
package main
import (
"context"
"fmt"
"log"
"net/url"
"os"
"os/signal"
"syscall"
"time"
"github.com/eclipse/paho.golang/autopaho"
"github.com/eclipse/paho.golang/paho"
)
func main() {
if err := run(); err != nil {
log.Print(err)
os.Exit(1)
}
}
func run() error {
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
defer stop()
user := os.Getenv("TEST_MQTT5_MQTT_USERNAME")
password := os.Getenv("TEST_MQTT5_MQTT_PASSWORD")
if user == "" || password == "" {
return fmt.Errorf("set the lab username and password environment variables")
}
broker, err := url.Parse("mqtt://127.0.0.1:18883")
if err != nil {
return fmt.Errorf("parse broker URL: %w", err)
}
cfg := autopaho.ClientConfig{
ServerUrls: []*url.URL{broker},
KeepAlive: 20,
CleanStartOnInitialConnection: true,
SessionExpiryInterval: 0,
ConnectUsername: user,
ConnectPassword: []byte(password),
OnConnectionUp: func(_ *autopaho.ConnectionManager, ack *paho.Connack) {
log.Printf("connected: session_present=%t reason=0x%02x", ack.SessionPresent, ack.ReasonCode)
},
OnConnectionDown: func() bool {
log.Print("connection down; AutoPaho will retry")
return true
},
OnConnectError: func(err error) {
// Keep errors free of credentials; never log cfg or broker userinfo.
log.Printf("connect attempt failed: %v", err)
},
ClientConfig: paho.ClientConfig{
ClientID: "lab-backend",
OnClientError: func(err error) { log.Printf("client error: %v", err) },
OnServerDisconnect: func(packet *paho.Disconnect) {
log.Printf("server disconnect: reason=0x%02x", packet.ReasonCode)
},
},
}
manager, err := autopaho.NewConnection(ctx, cfg)
if err != nil {
return fmt.Errorf("create connection manager: %w", err)
}
defer func() {
shutdown, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
if err := manager.Disconnect(shutdown); err != nil {
log.Printf("shutdown did not finish within deadline: %v", err)
}
}()
startup, cancel := context.WithTimeout(ctx, 10*time.Second)
err = manager.AwaitConnection(startup)
cancel()
if err != nil {
return fmt.Errorf("initial connection: %w", err)
}
log.Print("connection ready; press Ctrl+C to stop")
<-ctx.Done()
return nil
}
```
### 🔍 `cmd/backend/main.go` — file overview
This file is the first executable backend. It creates the process lifecycle, validates connection inputs, starts AutoPaho, waits for a successful CONNACK, and disconnects within a deadline. It does not subscribe to devices or run a business use case yet.
#### 🚪 `main` — report process failure
**Summary:** Calls `run` and exits with a nonzero status if setup or runtime returns an error.
**Details:** `main` keeps process-level error handling out of the connection logic. It logs the returned error and calls `os.Exit(1)` only after `run` has returned; do not move `os.Exit` into a callback, because that would bypass deferred cleanup.
#### ⚙️ `run` — own the connection lifecycle
**Summary:** Loads credentials, configures AutoPaho, waits for the first connection, then waits for shutdown.
**Details:** The signal context owns the manager lifetime. A separate startup context bounds `AwaitConnection`, while a fresh shutdown context gives `Disconnect` time after the signal context has already been cancelled. The callbacks report connection events; they do not make the application ready for device work. Lesson 4 later adds subscriptions and a receive loop.
## 🔌 2. Understand the connection lifecycle
> 💡 **Story question:** What is happening between creating the manager and logging “connection ready,” and why are those moments not equivalent?
Trace the background attempt through transport connection, MQTT CONNECT, broker authentication, CONNACK, and `AwaitConnection`. Then trace what AutoPaho does when the connection drops.
Before reading the full program, predict what evidence you need at each stage:
```text
Go process → AutoPaho retry loop → TCP connection → CONNECT → CONNACK
→ subscriptions (Lesson 4)
```
AutoPaho owns reconnect attempts and creates the underlying Paho client for a connection. Paho implements MQTT packets and protocol exchanges. Your application owns readiness, authorization policy, command state, and business retries. A running goroutine or open TCP socket is too early to declare MQTT ready: the broker can still reject CONNECT.
| Setting or call | MQTT meaning and reason for this choice |
| --- | --- |
| `ServerUrls` | Where to dial; the lab host endpoint is different from the Compose service endpoint |
| `ClientID` | Broker session identity; two simultaneous processes with this ID compete for one connection |
| `ConnectUsername`, `ConnectPassword` | CONNECT authentication inputs; independent of Client ID and payload identity |
| `KeepAlive: 20` | Maximum idle interval between client control packets, in seconds; Paho handles PINGREQ/PINGRESP when needed |
| `CleanStartOnInitialConnection: true` | Discard a previous broker session on the manager's first successful connection |
| `SessionExpiryInterval: 0` | Do not retain this broker session after the connection closes |
| `NewConnection` | Start connection management; a nil error does not establish a successful CONNACK |
| `AwaitConnection` | Wait for connection availability; it does not wait for SUBACK or a device heartbeat |
| `OnConnectionUp` | Observe a successful connection, including reconnection; inspect CONNACK evidence here |
| `OnConnectError` | Observe failed attempts, including refused connections or unsuccessful CONNACK |
| `OnConnectionDown` | Decide whether reconnect attempts should continue; returning false ends that retry lifecycle |
| `OnServerDisconnect` | Inspect a DISCONNECT actually sent by the broker; a severed cable cannot send this packet |
Keep Alive checks the MQTT connection, not the health of the vending application. With an effective Keep Alive of 20 seconds, the broker's protocol inactivity limit is 30 seconds without a client control packet; an already-detected closed socket can be handled earlier. A CONNACK Server Keep Alive can override the requested value. Your three-second heartbeat in Lesson 4 carries application evidence and has a separate purpose.
There are three different contexts here. The signal context controls the program lifetime. `startup` limits the initial wait to ten seconds. `shutdown` is a fresh context because the signal context is already canceled during shutdown. The manager receives the long-lived context, not the startup deadline; passing the startup context to `NewConnection` would terminate your service when that deadline expires.
`CleanStartOnInitialConnection: true` and `SessionExpiryInterval: 0` choose a disposable early-lab session. This is convenient for first experiments and intentionally offers no offline broker session after disconnect. Later lessons change these settings and test recovery. Do not copy them as a durable production policy.
Paho v0.23.0's `Disconnect(ctx)` cancels the manager and waits for manager shutdown with the provided deadline. Its `Done()` documentation notes that underlying MQTT cleanup may still be completing; neither API is proof that application data was persisted or device work was finished.
## ▶️ 3. Run it
> 💡 **Story question:** Which output tells you that the broker accepted the client, and what does a clean Ctrl+C demonstrate about lifecycle ownership?
Run the process with valid lab credentials, observe the acceptance evidence, then stop it and compare graceful disconnect with a process that simply disappears.
The backend reads its username and password from `configs/lab.toml`:
```sh
go run ./cmd/backend
```
`go run ./cmd/backend` compiles the package and runs it in the foreground. `./` means relative to the project root. Ctrl+C sends an interrupt to the foreground program; `signal.NotifyContext` turns it into context cancellation, allowing your cleanup code to run.
The TOML file contains the disposable local lab password. Use only lab credentials. Expect `connected` and `connection ready`; callback log order can vary. Press Ctrl+C and verify exit within the shutdown deadline.
Run syntax checks after creating the file:
```sh
gofmt -w cmd/backend/main.go
go build ./cmd/backend
go vet ./cmd/backend
```
`gofmt -w` rewrites the named source file using Go formatting. `go build` checks compilation and linking; `go vet` checks suspicious code patterns without running the application. A successful vet run is normally quiet.
`go build` emits a binary in the current directory; remove or ignore it as appropriate. A build verifies syntax/types, not connection behavior.
## ⚠️ 4. Observe the protocol and failures
> 💡 **Story question:** How can you tell whether a failure happened before MQTT, during broker authentication, or after an established connection was lost?
Change one condition at a time and compare the connection callbacks, reason information, retry behavior, and shutdown outcome.
CONNECT carries your client ID, authentication fields, keepalive, and session properties. A successful CONNACK is connection evidence. Bad credentials produce an unsuccessful connection attempt; the rejected CONNACK is handled by the connection-error path rather than your success callback.
1. Stop the broker before running the program. Expect retry diagnostics and initial-wait failure near ten seconds, followed by cleanup.
2. Start the broker; run the program; stop and restart the broker. Expect connection-down and connection-up events. You have not restored subscriptions yet because this program has none.
3. Supply a wrong password. Explain why retrying cannot repair an incorrect credential.
4. Run two copies using the same client ID. Observe connection takeover/reconnect behavior, then stop both. Unique IDs are necessary for concurrently running clients.
5. Press Ctrl+C while the broker is unavailable. Cancellation should interrupt the wait; do not wait for the full startup deadline.
## 🔎 Cautions and troubleshooting
- AutoPaho handles connection management; it does not invent topic authorization, business retries, or durable command storage.
- Do not call `log.Fatal` from callbacks. It terminates the process without running deferred cleanup.
- Callbacks should be short. Never add an indefinite channel send, database operation, or sleep to `OnConnectionDown`.
- Keep client IDs stable when you later want session resumption, but distinct across different simultaneously running clients.
- Logging entire config structs or URL userinfo can reveal passwords. These examples keep credentials separate from the URL.
- A connection-up event is not subscription readiness, and neither is device readiness.
**Completion:** Explain every field in `cfg`, distinguish the three contexts, and demonstrate successful startup, failed startup, reconnect, and cancellation.
## ✅ Result — the story so far
The backend now joins the broker as an authenticated MQTT 5 client. AutoPaho starts and manages the connection; the application waits until CONNACK confirms the broker accepted it, then keeps running and retries after a dropped connection. The backend has no subscriptions yet, so it is connected but is not listening for device messages.
## ❓ Questions to answer before Section 3
1. Why can `NewConnection` return before a usable broker connection exists?
2. What would happen if you passed `startup` to `NewConnection`?
3. Why does shutdown use `context.Background()` as its parent?
4. What does returning `false` from `OnConnectionDown` change?
<details>
<summary>Check your reasoning after answering</summary>
`NewConnection` starts a background lifecycle; CONNACK is later evidence. A startup context passed as the manager lifetime would stop the manager when that context expires or is canceled. Shutdown needs a fresh bounded context because the signal context is already canceled. Returning false from `OnConnectionDown` stops further automatic reconnect attempts for that manager; it does not repair the connection or preserve application state.
</details>
## 🔗 Sources
- [AutoPaho API v0.23.0](https://pkg.go.dev/github.com/eclipse/paho.golang@v0.23.0/autopaho)
- [Tagged connection manager source](https://github.com/eclipse-paho/paho.golang/blob/v0.23.0/autopaho/auto.go)
Next: [Section 3](03-typed-configuration.md).
teaching/course-source/03-typed-configuration.md
+394
View File
@@ -0,0 +1,394 @@
# ⚙️ Section 3 — Configure Go clients with TOML and koanf
Prerequisite: Your Section 2 program connects and stops cleanly. Here you move the hard-coded settings into validated configuration. The module remains `example.com/mqtt5-learning`.
**Version baseline:** Paho `v0.23.0`, verified from upstream release tags on 2026-10-07. It requires Go 1.24 or newer; Go 1.25 is compatible. The matching local reference is `references/paho.golang-v0.23.0/` (commit `1aa03969c5932464359b3e4ed752d9ac61abccc9`).
## 🎬 Start with the system story
> 💡 **Story question:** The backend, device simulator, and broker each have settings. How can each Go process get the right connection identity and credentials without baking them into source code?
The broker configuration controls the service and its account/ACL source. Each Go process separately loads its client ID, broker URL, username, password, device ID, and timeout. The configuration loader merges ordinary defaults, TOML, environment overrides, and explicitly supplied CLI flags, validates the final values, then hands one typed result to the MQTT startup code.
```mermaid
flowchart LR
DEF[Defaults] --> LOAD[Go configuration loader]
TOML[Local TOML settings<br/>including test password] --> LOAD
ENV[Supported TEST_MQTT5_ overrides<br/>password excluded] --> LOAD
CLI[Explicit CLI overrides] --> LOAD
LOAD --> VALID[Validate effective client settings]
VALID --> AUTO[AutoPaho ClientConfig]
AUTO -->|CONNECT credentials and Client ID| BROKER[Mosquitto broker]
```
Configuration errors should stop before network activity. A valid configuration with an unreachable broker is a different failure: it passes local validation but cannot complete the connection. See [Lesson 0](00-system-lifecycle.md) for how the validated values fit into the full system lifecycle.
## 📁 Working directory and file placement
Run every shell command in this lesson from the learning project root:
```sh
cd /home/ryan/Media/Code/Code/robot-market/mqtt-5-learning
```
Paths such as `cmd/backend/main.go` are relative to the repository root. The Markdown implementation lessons are under `teaching/course-source/`; create application files at the paths shown, not inside `teaching/course-source/` or `references/`. Code blocks labeled Go, TOML, YAML, or conf are file contents; shell blocks are terminal commands. Create parent directories before saving files.
## ⚙️ Why configuration changes MQTT behavior
> 💡 **Story question:** If a URL, Client ID, username, device ID, or timeout changes, which part of the broker/backend/device story changes with it?
The URL chooses a broker; Client ID chooses session identity at that broker; username/password select credentials; device ID chooses the application's topic namespace. Changing one does not change the others. If you change `device_id` to `device-002` but keep the simulator credential, constructing a new topic string does not grant permission to use it.
Your connect timeout bounds how long startup waits. It is neither Keep Alive (connection inactivity) nor Session Expiry (broker state after disconnect), and it cannot delete a command already sent. Later add those settings as separate typed fields with explicit units. A session expiry of `60` is seconds in MQTT; a Go `time.Duration(60)` is 60 nanoseconds.
Predict two failures: a malformed URL should fail local validation before dialing; a valid URL with a wrong password reaches the broker but fails authentication. Retrying the latter forever does not fix the configured secret. The loader's purpose is to make the effective policy predictable before AutoPaho starts background activity.
## ➕ 1. Add koanf's core, provider, and parser modules
> 💡 **Story question:** Which pieces of the loader provide defaults, read the TOML file, and parse its syntax into values the Go application can use?
Koanf combines those providers and parser into one effective configuration; these modules configure the application, not the MQTT broker.
```sh
go get github.com/knadh/koanf/v2@v2.3.7
go get github.com/knadh/koanf/providers/confmap@v1.0.0
go get github.com/knadh/koanf/providers/file@v1.2.0
go get github.com/knadh/koanf/parsers/toml/v2@v2.2.0
mkdir -p internal/config configs cmd/configcheck
```
These supporting module versions are a reproducible lesson baseline; the requested latest-release selection concerns Paho. Providers and parsers are separate Go modules. `confmap` supplies defaults, `file` reads bytes, and `toml/v2` interprets those bytes as TOML. The loader uses explicit environment mappings and standard Go flags so you can see precedence clearly.
## 🛠️ 2. Create the local lab TOML file
> 💡 **Story question:** Which client settings belong in the local TOML file, and which identity values may differ between backend and simulator processes?
Store the broker address, test password, example Client ID, and default backend username in TOML. Override the username for the simulator process.
Create `configs/lab.toml` in the `configs/` directory created above. This is separate from the broker configuration at `lab/mosquitto/mosquitto.conf`; it configures your Go clients:
```toml
[mqtt]
url = "mqtt://127.0.0.1:18883"
client_id = "lab-backend"
username = "backend"
password = "your-local-lab-password"
device_id = "device-001"
[app]
connect_timeout = "10s"
```
For this disposable local lab, put the same test password used for the broker accounts in `mqtt.password`. The application reads the password from TOML only; username and other settings can still be overridden. Do not reuse a personal or production password. TOML tables map to nested struct fields.
## 🛠️ 3. Write the loader
> 💡 **Story question:** How do several configuration sources become one validated set of values before AutoPaho starts its background connection work?
The loader merges each source in precedence order, maps environment names explicitly, decodes into typed fields, and validates the result. Follow the stages below before tracing individual lines.
Create `internal/config/config.go`:
```go
package config
import (
"flag"
"fmt"
"net/url"
"os"
"strings"
"time"
"github.com/knadh/koanf/parsers/toml/v2"
"github.com/knadh/koanf/providers/confmap"
"github.com/knadh/koanf/providers/file"
"github.com/knadh/koanf/v2"
)
type Config struct {
MQTT struct {
URL string `koanf:"url"`
ClientID string `koanf:"client_id"`
Username string `koanf:"username"`
Password string `koanf:"password"`
DeviceID string `koanf:"device_id"`
} `koanf:"mqtt"`
App struct {
ConnectTimeout string `koanf:"connect_timeout"`
} `koanf:"app"`
}
// Keep duration input as text; validate/parse it explicitly after merging.
func (c Config) ConnectTimeout() time.Duration {
d, _ := time.ParseDuration(c.App.ConnectTimeout) // Load already validated it.
return d
}
func Load(args []string) (Config, error) {
var result Config
flags := flag.NewFlagSet("mqtt5-learning", flag.ContinueOnError)
filename := flags.String("config", "configs/lab.toml", "TOML file")
broker := flags.String("broker", "", "broker URL override")
clientID := flags.String("client-id", "", "client ID override")
username := flags.String("username", "", "lab username override")
deviceID := flags.String("device-id", "", "device identity override")
timeout := flags.String("connect-timeout", "", "startup deadline override")
if err := flags.Parse(args); err != nil {
return result, err
}
if flags.NArg() != 0 {
return result, fmt.Errorf("unexpected positional arguments")
}
k := koanf.New(".")
defaults := map[string]any{
"mqtt.url": "mqtt://127.0.0.1:18883",
"mqtt.client_id": "lab-backend",
"mqtt.username": "", "mqtt.password": "",
"mqtt.device_id": "device-001",
"app.connect_timeout": "10s",
}
if err := k.Load(confmap.Provider(defaults, "."), nil); err != nil {
return result, fmt.Errorf("defaults: %w", err)
}
// A requested/missing config file is an error, not a silent fallback.
if err := k.Load(file.Provider(*filename), toml.Parser()); err != nil {
return result, fmt.Errorf("TOML: %w", err)
}
// Explicit mapping preserves underscores inside names such as client_id.
mapping := map[string]string{
"TEST_MQTT5_MQTT_URL": "mqtt.url",
"TEST_MQTT5_MQTT_CLIENT_ID": "mqtt.client_id",
"TEST_MQTT5_MQTT_USERNAME": "mqtt.username",
"TEST_MQTT5_MQTT_DEVICE_ID": "mqtt.device_id",
"TEST_MQTT5_APP_CONNECT_TIMEOUT": "app.connect_timeout",
}
for name, key := range mapping {
if value, exists := os.LookupEnv(name); exists {
if err := k.Set(key, value); err != nil {
return result, err
}
}
}
overrides := map[string]string{}
flags.Visit(func(f *flag.Flag) {
switch f.Name {
case "broker":
overrides["mqtt.url"] = *broker
case "client-id":
overrides["mqtt.client_id"] = *clientID
case "username":
overrides["mqtt.username"] = *username
case "device-id":
overrides["mqtt.device_id"] = *deviceID
case "connect-timeout":
overrides["app.connect_timeout"] = *timeout
}
})
for key, value := range overrides {
if err := k.Set(key, value); err != nil {
return result, err
}
}
if err := k.Unmarshal("", &result); err != nil {
return result, fmt.Errorf("decode configuration: %w", err)
}
if err := result.Validate(); err != nil {
return result, err
}
return result, nil
}
func (c Config) Validate() error {
u, err := url.Parse(c.MQTT.URL)
if err != nil || u == nil || u.Hostname() == "" || u.Port() == "" {
return fmt.Errorf("mqtt.url must contain a valid host and explicit port")
}
// Early lab only. TLS configuration is deliberately introduced later.
if u.Scheme != "mqtt" || u.User != nil || u.RawQuery != "" || u.Fragment != "" || u.Path != "" {
return fmt.Errorf("early lab requires mqtt://host:port without credentials or extra URL components")
}
if u.Hostname() != "127.0.0.1" && u.Hostname() != "localhost" {
return fmt.Errorf("plaintext learning broker must be on loopback")
}
for _, value := range []string{c.MQTT.ClientID, c.MQTT.Username, c.MQTT.Password} {
if strings.TrimSpace(value) == "" {
return fmt.Errorf("client ID and credentials are required")
}
}
if !validDeviceID(c.MQTT.DeviceID) {
return fmt.Errorf("device ID must be 1-128 ASCII letters, digits, dots, underscores, or hyphens")
}
d, err := time.ParseDuration(c.App.ConnectTimeout)
if err != nil || d <= 0 || d > time.Minute {
return fmt.Errorf("connect timeout must be a duration greater than zero and at most 1m")
}
return nil
}
// Keep config validation aligned with the topic builder's identifier rules.
func validDeviceID(value string) bool {
if value == "" || len(value) > 128 {
return false
}
for _, r := range value {
if !(r >= 'a' && r <= 'z' || r >= 'A' && r <= 'Z' || r >= '0' && r <= '9' || strings.ContainsRune("._-", r)) {
return false
}
}
return true
}
```
### 🔍 `config.go` — file overview
This file converts defaults, TOML, environment overrides, and explicitly supplied flags into one validated `Config`. MQTT startup receives only this effective value, so invalid local settings fail before the application starts network activity.
#### ⏱️ `ConnectTimeout` — expose the validated duration
**Summary:** Converts the configured duration string into `time.Duration` for startup code.
**Details:** `Load` has already validated the string, so this helper can discard the parse error. It does not represent MQTT Keep Alive or Session Expiry; those settings answer different lifecycle questions.
#### 📥 `Load` — merge and validate configuration
**Summary:** Parses flags, loads defaults and TOML, applies supported environment and explicit flag overrides, then returns a typed configuration.
**Details:** The flag set uses `ContinueOnError` so callers receive parse errors. `flags.Visit` identifies only flags the user supplied, avoiding empty defaults overwriting TOML. Password is intentionally TOML-only. `Unmarshal` maps nested koanf keys into `Config`, and `Validate` runs before any caller can start AutoPaho.
#### ✅ `Validate` — reject unsafe local settings
**Summary:** Checks URL shape and loopback use, required identities and credentials, device topic-segment safety, and timeout bounds.
**Details:** Validation prevents malformed or out-of-scope config from reaching the transport. It cannot prove the broker is reachable or that its ACL permits the client; those are runtime checks.
## ⚙️ 4. Understand effective configuration and precedence
> 💡 **Story question:** If both TOML and the environment set a timeout, which value reaches AutoPaho—and why should an absent CLI flag not replace it?
Trace the merge order and the explicit-flag detection. The broker sees only the final connection settings sent in MQTT packets; it does not know which local source supplied them.
The loader composes one effective configuration before connecting. `koanf.New(".")` interprets `mqtt.client_id` as a nested key; the `koanf` field tags map those keys into the typed configuration consumed by the transport. Later `Load`/`Set` calls override earlier values. The map provider already supplies structured values, so it needs no text parser.
For ordinary settings the order is defaults → TOML → environment → explicitly supplied CLI flags. The password is read from TOML only, so an old exported `TEST_MQTT5_MQTT_PASSWORD` cannot silently replace it. `flags.Visit` visits only supplied flags, preventing empty CLI defaults from overwriting configured values. `os.LookupEnv` distinguishes an absent variable from an explicitly empty one for supported environment overrides.
We map `TEST_MQTT5_MQTT_CLIENT_ID` explicitly to `mqtt.client_id`. Replacing every underscore with a dot would produce `mqtt.client.id`, a different key. This allowlist also makes supported environment variables visible; unexpected prefixed variables are ignored in this small example.
Duration text is validated with `time.ParseDuration`, avoiding reliance on implicit conversion. The helper returns a `time.Duration` after validation. URL and topic validation are intentionally narrow for the loopback-only early lab. Later TLS lessons extend the schema and validation instead of disabling certificate checks.
## ⚙️ 5. Add a configuration inspection command
> 💡 **Story question:** How can you check the final effective configuration without opening a broker connection or exposing the password?
The inspection executable reuses the same loader as the backend and prints a redacted summary, making source precedence testable independently of network state.
Create `cmd/configcheck/main.go`:
```go
package main
import (
"example.com/mqtt5-learning/internal/config"
"log"
"os"
)
func main() {
cfg, err := config.Load(os.Args[1:])
if err != nil {
log.Print(err)
os.Exit(1)
}
log.Printf("client_id=%s device_id=%s username=%s timeout=%s password=[redacted]",
cfg.MQTT.ClientID, cfg.MQTT.DeviceID, cfg.MQTT.Username, cfg.ConnectTimeout())
}
```
```sh
go run ./cmd/configcheck
go run ./cmd/configcheck -client-id lab-config-test
go run ./cmd/configcheck -connect-timeout 0s
```
Arguments after `./cmd/configcheck` belong to your program. `-client-id` overrides only that configured field; `-connect-timeout 0s` supplies a deliberately invalid zero-second duration. The inspection command loads and validates configuration without establishing an MQTT connection. Keep the backend password exported in this terminal for these checks.
The first two commands should print a redacted summary. The last must fail before any broker connection. The password comes from `configs/lab.toml`; do not print the whole struct because it contains the password.
## 🔗 6. Integrate it into Section 2
> 💡 **Story question:** At what point should the settings cross from application configuration into AutoPaho, and what should happen if validation fails?
Load and validate before creating the connection manager. Then replace hard-coded connection values with the typed configuration fields.
Edit the existing `cmd/backend/main.go`. Keep one `package main`, one import block, and one copy of each function; the following integration steps modify the Section 2 program rather than create another executable. Run from the project root so the default `configs/lab.toml` path resolves correctly.
In `cmd/backend/main.go`, import `example.com/mqtt5-learning/internal/config`. At the beginning of `run`, call:
```go
settings, err := config.Load(os.Args[1:])
if err != nil {
return err
}
```
Remove the direct environment lookup/credential check and use these replacements:
| Old expression | Replacement |
| --- | --- |
| `url.Parse("mqtt://127.0.0.1:18883")` | `url.Parse(settings.MQTT.URL)` |
| `ConnectUsername: user` | `ConnectUsername: settings.MQTT.Username` |
| `ConnectPassword: []byte(password)` | `ConnectPassword: []byte(settings.MQTT.Password)` |
| `ClientID: "lab-backend"` | `ClientID: settings.MQTT.ClientID` |
| `context.WithTimeout(ctx, 10*time.Second)` | `context.WithTimeout(ctx, settings.ConnectTimeout())` |
The snippets are edits to Section 2's complete program, not additional standalone executables. Compile after integrating. Section 4 provides a complete replacement backend once you understand the loader.
## 🧪 Exercises: prove the precedence
1. Set TOML timeout to `8s`; unset `TEST_MQTT5_APP_CONNECT_TIMEOUT`; expect `8s`.
2. Export `TEST_MQTT5_APP_CONNECT_TIMEOUT=6s`; expect `6s`.
3. Pass `-connect-timeout 4s`; expect `4s`.
4. Remove the flag; expect `6s`, proving an absent flag cannot override the environment.
5. Remove `mqtt.password` from a temporary TOML file, try a missing file, invalid URL, zero duration, and device IDs `device/001` and `device!001`.
6. Write a table-driven test using a temporary TOML file and `t.Setenv`. Clear relevant inherited environment values in each test; environment-mutating tests must not use `t.Parallel`.
`unset NAME` removes a terminal environment override, revealing the TOML/default layer underneath; it does not change the file. `export NAME=value` sets the environment layer for child processes. A flag is temporary for that invocation. Restore your sample TOML timeout after the experiment so later lessons have a predictable starting point.
Create `internal/config/config_test.go` for the table-driven cases. A table-driven test loops over named inputs and expected outcomes with `t.Run`. `t.TempDir()` provides a temporary fixture directory that Go cleans up; `t.Setenv` restores the previous environment after the test. Use these to prevent test fixtures from rewriting your real configuration.
Example assertion inside such a test:
```go
t.Setenv("TEST_MQTT5_APP_CONNECT_TIMEOUT", "6s")
cfg, err := Load([]string{"-config", filename, "-connect-timeout", "4s"})
if err != nil { t.Fatal(err) }
if cfg.ConnectTimeout() != 4*time.Second { t.Fatalf("wrong precedence: %s", cfg.ConnectTimeout()) }
```
`filename` is the temporary fixture you create; the fragment belongs inside a `Test...` function in package `config`. Also test no flag, malformed values, and a missing password in the TOML fixture.
## ⚠️ Common mistakes
- TOML paths are relative to the process working directory. Run from the project root or pass an explicit `-config` path.
- An exported username can override the TOML username; use `-username simulator` when starting the simulator.
- Command-line password flags expose secrets to process inspection, so this loader deliberately omits one.
- Configuration precedence is a merge decision, not an MQTT feature. Paho receives only the final validated values.
- These samples do not reject all unknown TOML keys. Add strict key validation as an exercise if typo detection becomes important.
**Completion:** Demonstrate the override sequence and invalid-startup cases, then explain how the final values reach AutoPaho.
## 🔗 Sources
- [koanf concepts, providers, and unmarshalling](https://pkg.go.dev/github.com/knadh/koanf/v2@v2.3.7)
- [AutoPaho configuration v0.23.0](https://pkg.go.dev/github.com/eclipse/paho.golang@v0.23.0/autopaho#ClientConfig)
## ✅ Result — the story so far
The backend's connection details now come from a predictable, validated configuration: defaults, TOML, environment, then explicit flags. Before opening an MQTT connection, the program has checked that its broker URL, identity, device ID, and timeouts make sense. The simulator can select its own username and client ID while using the same lab settings; message exchange still comes in the next lesson.
Next: [Section 4](04-publish-subscribe-simulator.md).
teaching/course-source/04-publish-subscribe-simulator.md
+785
View File
@@ -0,0 +1,785 @@
# 📡 Section 4 — Publish, subscribe, and build a Go device simulator
Prerequisites: Sections 1–3 complete. Keep Section 3's configuration package and TOML file. This lesson supplies complete files for a reusable MQTT transport, backend, and simulator. Implement each file in stages and explain it before running both programs.
**Version baseline:** Paho `v0.23.0`, verified from upstream release tags on 2026-10-07. It requires Go 1.24 or newer; Go 1.25 is compatible. The matching local reference is `references/paho.golang-v0.23.0/` (commit `1aa03969c5932464359b3e4ed752d9ac61abccc9`).
## 🎬 Start with the system story
> 💡 **Story question:** The backend and device are separate MQTT clients. How do they exchange a command without knowing each other's address, and how can the backend tell whether the device actually handled it?
The broker is the meeting point. The backend connects and subscribes to telemetry and responses; the simulator connects with its own identity and subscribes to commands. Each publication goes to the broker, which routes it to matching subscriptions. A device response is a new application message, separate from the broker's acknowledgement of the backend's publish.
```mermaid
sequenceDiagram
participant G as Go backend
participant B as Mosquitto broker
participant D as Go device simulator
G->>B: CONNECT
B-->>G: CONNACK accepted
D->>B: CONNECT
B-->>D: CONNACK accepted
G->>B: SUBSCRIBE telemetry, responses
B-->>G: SUBACK
D->>B: SUBSCRIBE commands
B-->>D: SUBACK
G->>B: PUBLISH ping, QoS 1
par Backend-to-broker acknowledgement
B-->>G: PUBACK
and Broker-to-device delivery
B->>D: PUBLISH ping, QoS 1
D->>D: Paho callback queues message
D-->>B: PUBACK after callback returns
end
D->>D: Application handles ping
D->>B: PUBLISH pong response, QoS 1
par Device-to-broker acknowledgement
B-->>D: PUBACK
and Broker-to-backend delivery
B->>G: PUBLISH pong response, QoS 1
G->>G: Paho callback queues message
G-->>B: PUBACK after callback returns
end
```
The broker, backend, and device have distinct setup. Mosquitto owns its listener, password source, and ACL. Each Go process loads its own broker URL, client ID, and username. The backend and simulator use different usernames for their broker permissions; both use the disposable test password from `configs/lab.toml`. The code below first builds the shared transport that connects, subscribes, receives, and publishes; then the two application loops use it.
At the end of this lesson, ping/pong proves the transport path works, but the backend still treats received bytes as demonstrations. Section 5 adds a shared contract; Section 5A turns a validated heartbeat into a device-view update. Keep this order clear: a network callback receives packets, a contract gives them shape, and an application service gives them meaning.
Before each implementation part, use the question there to predict which process or protocol layer is responsible. Follow [Lesson 0](00-system-lifecycle.md) for the full lifecycle map.
## 🗺️ 1. Map the conversation
> 💡 **Story question:** Which client publishes each kind of message, which client needs it, and what does the broker use to route it?
Map the telemetry, command, and response paths first. This identifies which process must subscribe before we write transport code.
The lab has one deliberately small hierarchy: `lab` names the isolated environment, `v1` is the topic-contract version, `devices` is the entity collection, `{deviceID}` scopes device traffic, and the last level names a flow. `v1` versions the route vocabulary; the payload's `schema_version` versions JSON shape, so the two versions can evolve independently. A `TopicName` is one concrete publish destination; a `TopicFilter` is a subscription pattern. The broker accepts `+` and `#` only in filters, so the Go API keeps these concepts as different types. This hierarchy is the learning contract, not a proposed migration of the existing vending backend.
| Topic | Publisher | Subscriber | Example |
| --- | --- | --- | --- |
| `lab/v1/devices/device-001/telemetry` | Simulator | Backend | Timestamped heartbeat |
| `lab/v1/devices/device-001/commands` | Backend | Simulator | Harmless ping with ID |
| `lab/v1/devices/device-001/responses` | Simulator | Backend | Pong echoing that ID |
A broker subscription and a local Go callback are different things. SUBSCRIBE asks the broker which messages to forward. `OnPublishReceived` tells your local client how to process forwarded PUBLISH packets. You need both. This lab's topic convention is deliberately separate from the production contracts.
### 🔄 Trace two independent delivery hops
```mermaid
sequenceDiagram
participant B as Go backend
participant M as Broker
participant S as Go simulator
S->>M: SUBSCRIBE commands, maximum QoS 1
M-->>S: SUBACK with grant or rejection
B->>M: PUBLISH command, QoS 1
par Backend publish exchange
M-->>B: PUBACK
and Broker forwards to simulator
M->>S: PUBLISH command, granted delivery QoS
S-->>M: PUBACK for delivery hop
end
S->>S: Application handles command
S->>M: PUBLISH application result, QoS 1
par Simulator publish exchange
M-->>S: PUBACK
and Broker forwards result to backend
M->>B: PUBLISH application result
B-->>M: PUBACK for delivery hop
end
```
The `par` blocks show that a publisher-side PUBACK and broker-to-subscriber delivery are independent work on separate client connections; their observed order can vary. Each QoS 1 PUBACK confirms only its own hop. The application result is a separate publication—the broker does not create it automatically, and neither PUBACK proves that business work completed.
### 📨 Choose QoS by the guarantee you need
| QoS | Successful exchange on one hop | What the Go application must still handle |
| --- | --- | --- |
| 0 | PUBLISH, no MQTT acknowledgement | Loss is possible; nil publish error does not prove receipt |
| 1 | PUBLISH → PUBACK | Redelivery is possible; use stable business IDs |
| 2 | PUBLISH → PUBREC → PUBREL → PUBCOMP | Protocol duplicate suppression needs session state; application retries and external effects still need idempotency |
The subscriber requests a **maximum** delivery QoS. A QoS 1 publication cannot become QoS 2 merely because the subscription requests 2. With a QoS 0 subscription grant, a publisher's QoS 1 exchange with the broker still ends in PUBACK while the broker-to-subscriber hop has no PUBACK. QoS 2 does not make a database update and a physical dispense one transaction.
For a QoS 2 experiment, change both publication and requested subscription QoS to 2, accept SUBACK grants 0–2, and inspect the four-packet exchange. The original helper intentionally accepts only grants up to its requested QoS 1. Do not change only the publish field and call the resulting lower-QoS subscriber path an end-to-end QoS 2 test.
### 🏷️ Topic filters are routing rules
`lab/v1/devices/+/telemetry` matches one device segment; `lab/v1/devices/#` matches that subtree. `+` occupies an entire level and `#` is the final level. Topics are case-sensitive, so `device-001` and `Device-001` differ. Wildcards belong in subscriptions, never publish topic names. A root `#` filter does not include `$SYS/...`; broker system topics require an explicit matching filter and permission.
A SUBACK proves the subscription request received a result. Topic ACLs still govern delivery: an accepted broad filter does not prove access to every matching topic. Check an authorized observer, reason codes, and broker logs when testing denial.
## 🛠️ 2. Create the transport helper
> 💡 **Story question:** What should be shared between the backend and simulator, and what must remain specific to each running client?
The transport helper can share connection lifecycle, subscription handling, message copying, and publishing. Each caller still supplies its configured username, Client ID, topic list, and receive queue. Keeping those identities separate lets the broker apply different permissions and prevents Client ID takeover.
Topic construction belongs in `internal/contracts/topic.go`, which is introduced here and reused by later lessons. Application composition still chooses which flows each process subscribes to; the contract package only builds and parses names and filters. Mosquitto ACLs remain a separate broker authorization layer.
```sh
mkdir -p internal/contracts internal/mqtt cmd/simulator
```
Create `internal/contracts/topic.go`:
```go
package contracts
import (
"fmt"
"strings"
)
type TopicName string
type TopicFilter string
type Kind string
const (
KindTelemetry Kind = "telemetry"
KindCommands Kind = "commands"
KindResponses Kind = "responses"
KindSales Kind = "sales"
)
const topicPrefix = "lab/v1/devices/"
type Topic struct {
DeviceID string
Kind Kind
}
func (t TopicName) String() string { return string(t) }
func (f TopicFilter) String() string { return string(f) }
func DeviceTopic(deviceID string, kind Kind) (TopicName, error) {
if !validID(deviceID) {
return "", fmt.Errorf("invalid device ID %q", deviceID)
}
if !supportedKind(kind) {
return "", fmt.Errorf("unsupported topic kind %q", kind)
}
return TopicName(topicPrefix + deviceID + "/" + string(kind)), nil
}
func DeviceFilter(kind Kind) (TopicFilter, error) {
if !supportedKind(kind) {
return "", fmt.Errorf("unsupported topic kind %q", kind)
}
return TopicFilter(topicPrefix + "+/" + string(kind)), nil
}
func DeviceSubscription(deviceID string, kind Kind) (TopicFilter, error) {
topic, err := DeviceTopic(deviceID, kind)
if err != nil {
return "", err
}
return TopicFilter(topic.String()), nil
}
func ParseTopic(value string) (Topic, error) {
if !strings.HasPrefix(value, topicPrefix) {
return Topic{}, fmt.Errorf("topic must be %s{deviceID}/{kind}", topicPrefix)
}
parts := strings.Split(strings.TrimPrefix(value, topicPrefix), "/")
if len(parts) != 2 || !validID(parts[0]) {
return Topic{}, fmt.Errorf("topic must be %s{deviceID}/{kind}", topicPrefix)
}
kind := Kind(parts[1])
if !supportedKind(kind) {
return Topic{}, fmt.Errorf("unsupported topic kind %q", kind)
}
return Topic{DeviceID: parts[0], Kind: kind}, nil
}
func supportedKind(kind Kind) bool {
switch kind {
case KindTelemetry, KindCommands, KindResponses, KindSales:
return true
default:
return false
}
}
func validID(value string) bool {
if value == "" || len(value) > 128 {
return false
}
for _, r := range value {
if !(r >= 'a' && r <= 'z' || r >= 'A' && r <= 'Z' || r >= '0' && r <= '9' || strings.ContainsRune("._-", r)) {
return false
}
}
return true
}
```
### 🔍 `topic.go` — file overview
This file is the source of truth for the lab's MQTT routing shape. `DeviceTopic` creates a concrete publish name; `DeviceFilter` creates a fleet-wide per-flow subscription filter; `DeviceSubscription` creates an exact per-device filter; and `ParseTopic` turns an incoming concrete topic into routing data. `TopicName` and `TopicFilter` are distinct string types so code must consciously convert them at the Paho boundary.
#### 🧱 `DeviceTopic` — build one publish destination
**Summary:** Validates the device ID and flow kind, then returns a topic with no wildcard levels.
**Details:** Device IDs must fit one topic level and may contain only ASCII letters, digits, dot, underscore, and hyphen. This avoids slash injection and prevents a wildcard from accidentally becoming part of a publish name. It is a shape check, not proof that a device is authorized to publish there.
#### 🔎 `DeviceFilter` and `DeviceSubscription` — choose filter scope
**Summary:** `DeviceFilter` subscribes to one flow across devices; `DeviceSubscription` subscribes to one exact device and flow.
**Details:** These functions control wildcard placement instead of letting callers concatenate arbitrary strings. The backend uses fleet filters where it must ingest multiple devices; the simulator uses an exact device subscription for least privilege. Mosquitto ACLs still decide whether either role may subscribe.
#### 🧭 `ParseTopic` — extract route identity
**Summary:** Checks the shared `topicPrefix`, then validates the remaining device ID and flow kind.
**Details:** `ParseTopic` first requires the incoming name to start with `topicPrefix`, then splits only the suffix into exactly two levels. The parser does not repeat the prefix's individual segments, so changing the constant updates builders and parsing together. The returned device ID is routing metadata to compare with the payload claim; it does not authenticate the publishing client.
#### ✅ `supportedKind` and `validID` — share the contract rules
**Summary:** Centralize the allowed final segments and safe single-level identifier syntax.
**Details:** Both building and parsing use these rules, preventing the producer and consumer from quietly accepting different topic vocabularies.
Create `internal/mqtt/client.go`:
```go
package mqtt
import (
"context"
"fmt"
"log"
"net/url"
"time"
"example.com/mqtt5-learning/internal/config"
"example.com/mqtt5-learning/internal/contracts"
"github.com/eclipse/paho.golang/autopaho"
"github.com/eclipse/paho.golang/paho"
)
type Message struct {
Topic string
Payload []byte
Retain bool
}
// Start creates the MQTT connection, subscribes to the requested topics, and
// waits until the initial connection and subscriptions are ready.
func Start(ctx context.Context, cfg config.Config, topics []contracts.TopicFilter, inbox chan<- Message) (*autopaho.ConnectionManager, error) {
// Parse the configured broker URL before creating the reconnecting client.
broker, err := url.Parse(cfg.MQTT.URL)
if err != nil {
return nil, err
}
// The callback reports its first SUBACK so startup can fail early if needed.
initialSubscription := make(chan error, 1)
manager, err := autopaho.NewConnection(ctx, autopaho.ClientConfig{
ServerUrls: []*url.URL{broker},
KeepAlive: 20,
CleanStartOnInitialConnection: true,
SessionExpiryInterval: 0,
ConnectUsername: cfg.MQTT.Username,
ConnectPassword: []byte(cfg.MQTT.Password),
// Recreate subscriptions after every successful connection or reconnect.
OnConnectionUp: func(cm *autopaho.ConnectionManager, _ *paho.Connack) {
// This runs again on reconnect; reestablish broker subscriptions.
subCtx, cancel := context.WithTimeout(ctx, 5*time.Second)
defer cancel()
subscriptions := make([]paho.SubscribeOptions, 0, len(topics))
for _, topic := range topics {
subscriptions = append(subscriptions, paho.SubscribeOptions{Topic: topic.String(), QoS: 1, RetainAsPublished: true})
}
// Ask for QoS 1 and preserve RETAIN so consumers can identify stored data.
ack, subErr := cm.Subscribe(subCtx, &paho.Subscribe{Subscriptions: subscriptions})
if subErr == nil {
if ack == nil || len(ack.Reasons) != len(topics) {
subErr = fmt.Errorf("missing or incomplete SUBACK")
} else {
for i, reason := range ack.Reasons {
if reason > 1 { // We requested QoS 1; valid grants are 0 or 1.
subErr = fmt.Errorf("subscription %s rejected: 0x%02x", topics[i], reason)
break
}
}
}
}
if subErr != nil {
log.Printf("subscribe failed: %v", subErr)
} else {
log.Printf("subscriptions ready: %v", topics)
}
select {
case initialSubscription <- subErr:
default:
}
},
OnConnectionDown: func() bool { log.Print("disconnected; reconnecting"); return true },
OnConnectError: func(err error) { log.Printf("connect error: %v", err) },
// Keep the packet callback short: copy the message and enqueue it for work.
ClientConfig: paho.ClientConfig{
ClientID: cfg.MQTT.ClientID,
OnClientError: func(err error) { log.Printf("client error: %v", err) },
OnServerDisconnect: func(d *paho.Disconnect) { log.Printf("disconnect reason=0x%02x", d.ReasonCode) },
OnPublishReceived: []func(paho.PublishReceived) (bool, error){
func(received paho.PublishReceived) (bool, error) {
packet := received.Packet
message := Message{Topic: packet.Topic, Payload: append([]byte(nil), packet.Payload...), Retain: packet.Retain}
// Never block the network callback when the bounded inbox is full.
select {
case inbox <- message:
default:
// Deliberately lossy early demo: automatic MQTT ACK is still possible.
log.Printf("lab inbox full; message dropped topic=%s", packet.Topic)
}
return true, nil
},
},
},
})
if err != nil {
return nil, fmt.Errorf("start AutoPaho: %w", err)
}
// Bound initial connection and subscription setup by the configured timeout.
startup, cancel := context.WithTimeout(ctx, cfg.ConnectTimeout())
defer cancel()
if err := manager.AwaitConnection(startup); err != nil {
Stop(manager)
return nil, fmt.Errorf("await connection: %w", err)
}
select {
case err := <-initialSubscription:
if err != nil {
Stop(manager)
return nil, err
}
case <-startup.Done():
Stop(manager)
return nil, fmt.Errorf("await subscriptions: %w", startup.Err())
}
return manager, nil
}
func Publish(ctx context.Context, manager *autopaho.ConnectionManager, topic contracts.TopicName, payload []byte) error {
// Publishing waits only for this bounded operation, not for device completion.
operation, cancel := context.WithTimeout(ctx, 3*time.Second)
defer cancel()
ack, err := manager.Publish(operation, &paho.Publish{
Topic: topic.String(), QoS: 1, Retain: false, Payload: payload,
Properties: &paho.PublishProperties{ContentType: "application/json"},
})
if err != nil {
return fmt.Errorf("publish: %w", err)
}
if ack == nil {
return fmt.Errorf("missing QoS 1 publication response")
}
if ack.ReasonCode >= 0x80 {
return fmt.Errorf("publish rejected: 0x%02x", ack.ReasonCode)
}
log.Printf("broker publish acknowledgement topic=%s reason=0x%02x", topic, ack.ReasonCode)
return nil
}
func Stop(manager *autopaho.ConnectionManager) {
// Give the MQTT disconnect handshake a short, independent shutdown window.
shutdown, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
if err := manager.Disconnect(shutdown); err != nil {
log.Printf("shutdown: %v", err)
}
}
```
### 🔍 `internal/mqtt/client.go` — file overview
This file is the transport adapter shared by the backend and simulator. It starts AutoPaho, restores subscriptions after connection-up, copies received packet bytes into an application-owned inbox, publishes QoS 1 messages with a deadline, and disconnects cleanly. It does not validate vending contracts or decide what a device message means.
#### 🔌 `Start` — connect and establish subscriptions
**Summary:** Creates the connection manager, subscribes to the requested filters, checks each SUBACK reason, and waits for initial readiness.
**Details:** Connection-up can happen again after reconnect, so subscription setup belongs in that callback. The initial readiness channel lets the caller distinguish CONNACK from successful subscriptions. The publish-receive callback copies the payload before it leaves the packet context and uses a nonblocking inbox send; if the bounded inbox is full, this early example drops the message and logs the loss.
#### 📤 `Publish` — publish one bounded operation
**Summary:** Sends a QoS 1 JSON message and returns only after the publish operation reports its broker acknowledgement or an error.
**Details:** The operation context bounds the wait. A successful acknowledgement covers this publish hop; it does not prove a subscriber received, validated, or applied the message. Timeout can leave the delivery outcome uncertain.
#### 🛑 `Stop` — disconnect within a deadline
**Summary:** Gives AutoPaho a fresh, bounded shutdown context and logs a disconnect error.
**Details:** The fresh context remains usable after the signal context has been cancelled. This stops transport lifecycle work; it does not drain a future application worker queue, which Section 6 adds.
## 🔗 3. Map the helper to MQTT behavior
> 💡 **Story question:** When `Start` says subscriptions are ready, which broker exchange has happened, and what is the local callback's separate job?
The broker has received `SUBSCRIBE` and returned an acceptable `SUBACK`; the callback is configured in the Go process to receive broker-forwarded PUBLISH packets. Read the implementation against these two sides of the receive path.
| Code decision | Why it matters |
| --- | --- |
| Subscribe inside `OnConnectionUp` | The early disposable session loses its broker subscriptions on disconnect |
| Inspect every `ack.Reasons` entry | Each requested filter has its own result; connection success is insufficient |
| `RetainAsPublished: true` | Preserve the publisher's RETAIN flag on live forwarding, so the simulator can reject live retained commands as well as retained replay |
| Copy payload into `Message` | The application queue owns bytes that will be processed after the receive callback returns |
| Nonblocking inbox send | Bound callback time and memory; dropping on overflow is deliberately lossy |
| `return true, nil` | Report local handling to subsequent callbacks; this is not a negative/positive business acknowledgement |
| `QoS: 1`, `Retain: false` | Ask for a broker acknowledgement without storing the command as the topic's retained value |
| `ContentType: "application/json"` | Describe the encoding; neither broker nor library validates the application's schema for you |
| Initial SUBACK channel | Delay startup until this client's receive path is established; it says nothing about the peer's readiness |
SUBACK grants `0`, `1`, or `2` for maximum delivery QoS. Since this program requests QoS 1, it accepts only 0 or 1. Failure reason codes start at `0x80`. Receiving a successful call return without inspecting subscription results can leave you connected but receiving no useful messages.
For QoS 1, `Publish` returns a response describing the immediate broker acknowledgement. Success codes can include `0x10` (no matching subscribers); this does not prove the simulator is online. Transport errors and context timeouts can leave the delivery outcome uncertain. Never retry a real action with a new business ID just because your waiting context expired.
`return true, nil` means the callback handled the message for local routing. It is not a device acceptance/completion response or durable-storage guarantee. With automatic acknowledgement, Paho can acknowledge after callback processing while your inbox has not yet been handled. Inspect `paho/client.go` in the tagged source to understand callback/ack order.
The initial-subscription channel allows startup to wait for SUBACK as well as CONNACK. The callback subscribes again on reconnect. This helper checks initial readiness only; reconnect subscription failures are logged, not represented as continuous health state. Add that state later rather than treating initial readiness as permanent.
## 💻 4. Replace the backend main program
> 💡 **Story question:** After configuration, connection, and subscriptions are ready, how does a backend timer become an MQTT command?
### 🦟 Trace one ping from Go value to broker
On each timer tick, the backend first builds an ordinary Go value. `json.Marshal` turns that value into the byte sequence sent as the MQTT payload:
```go
struct {
ID string `json:"id"`
Command string `json:"command"`
}{ID: "ping-123-1", Command: "ping"}
```
The JSON field tags set the wire names, so the payload bytes represent:
```json
{"id":"ping-123-1","command":"ping"}
```
The backend asks `contracts.DeviceTopic(deviceID, contracts.KindCommands)` for the concrete destination. That constructor validates the device ID as one safe topic level. It passes the resulting `TopicName` and JSON bytes to `mqtt.Publish`; that helper builds a Paho `paho.Publish` with **QoS 1**, **Retain false**, and Content Type `application/json`, then calls AutoPaho's `manager.Publish`.
`manager.Publish` waits for the QoS 1 publish exchange or an error/timeout. When it receives a successful `PUBACK`, the helper logs the reason code and returns nil. That confirms the backend-to-broker QoS exchange. A success reason of `0x10` means the broker had no matching subscribers; it is still a successful MQTT acknowledgement, so it does not prove the simulator received anything. A timeout can also leave the outcome uncertain.
The broker independently forwards a matching PUBLISH to the simulator's `.../commands` subscription. The simulator's Paho callback copies it into an inbox; the simulator's main loop decodes the JSON, checks that it is a non-retained `ping` with an ID, and publishes a `pong` carrying that same ID to `.../responses`. That new response is application-level evidence the simulator handled the ping. The backend is subscribed to the response topic and currently logs the received bytes; matching responses to pending commands is added in Lesson 7.
```text
Go struct → JSON payload bytes → QoS 1 PUBLISH → broker PUBACK
└→ simulator receives and handles ping
→ separate pong PUBLISH → backend receives
```
The two arrows after the backend's PUBLISH are separate outcomes: PUBACK may arrive before or after the simulator's response. Retain is false because a ping is a one-time action; storing it as retained state could replay the command to a later subscriber.
Create `cmd/backend/main.go`:
```go
package main
import (
"context"
"encoding/json"
"fmt"
"log"
"os"
"os/signal"
"syscall"
"time"
"example.com/mqtt5-learning/internal/config"
"example.com/mqtt5-learning/internal/contracts"
mqtt "example.com/mqtt5-learning/internal/mqtt"
)
func main() {
// Convert a returned application error into a visible nonzero process exit.
if err := run(); err != nil {
log.Print(err)
os.Exit(1)
}
}
func run() error {
// Load settings and bind cancellation to Ctrl+C or the container stop signal.
cfg, err := config.Load(os.Args[1:])
if err != nil {
return err
}
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
defer stop()
// Build subscription filters and a concrete command destination.
telemetryFilter, err := contracts.DeviceFilter(contracts.KindTelemetry)
if err != nil { return err }
responsesFilter, err := contracts.DeviceFilter(contracts.KindResponses)
if err != nil { return err }
commandTopic, err := contracts.DeviceTopic(cfg.MQTT.DeviceID, contracts.KindCommands)
if err != nil { return err }
inbox := make(chan mqtt.Message, 16)
manager, err := mqtt.Start(ctx, cfg, []contracts.TopicFilter{telemetryFilter, responsesFilter}, inbox)
if err != nil {
return err
}
defer mqtt.Stop(manager)
// Publish synthetic ping commands periodically while processing received data.
ticker := time.NewTicker(10 * time.Second)
defer ticker.Stop()
runID := time.Now().UnixNano()
sequence := 0
for {
// The select keeps shutdown, inbound messages, and periodic commands responsive.
select {
case <-ctx.Done():
return nil
case message := <-inbox:
// Synthetic lab payloads only; production payload logging needs redaction.
log.Printf("received topic=%s retained=%t payload=%s", message.Topic, message.Retain, message.Payload)
case <-ticker.C:
sequence++
payload, err := json.Marshal(struct {
ID string `json:"id"`
Command string `json:"command"`
}{ID: fmt.Sprintf("ping-%d-%d", runID, sequence), Command: "ping"})
if err != nil {
return err
}
if err := mqtt.Publish(ctx, manager, commandTopic, payload); err != nil {
log.Printf("command publication failed or outcome uncertain: %v", err)
}
}
}
}
```
#### 🧭 `main` and `run` — assemble the backend process
**Summary:** `main` turns a returned error into a failing process; `run` loads settings, starts MQTT, then handles inbound messages and periodic pings until cancellation.
**Details:** `main` is intentionally small so startup errors have one exit path. `run` owns resources in reverse order: it creates the signal context, starts the manager, and defers shutdown; it also stops the ticker. Its `select` keeps messages, timer work, and cancellation responsive. A publish error is logged as uncertain because the client may not know whether the broker accepted the packet.
The ticker sends one ping every ten seconds. It starts after the backend's own subscriptions are ready. A timestamp and sequence make IDs distinct enough for this demonstration, not a production idempotency system. The backend logs the response but does not yet implement pending requests or validate completion; Section 7 adds those semantics.
## 🛠️ 5. Create the device simulator
> 💡 **Story question:** How does a received MQTT packet become a device action, and how does the simulator report its result without blocking Paho's network callback?
The callback copies the packet into a bounded inbox. The simulator's main loop reads it, validates the harmless ping, and publishes a pong. This draws a deliberate boundary between packet reception and application behavior; later lessons examine the loss window and acknowledgement timing.
Create `cmd/simulator/main.go`:
```go
package main
import (
"context"
"encoding/json"
"log"
"os"
"os/signal"
"syscall"
"time"
"example.com/mqtt5-learning/internal/config"
"example.com/mqtt5-learning/internal/contracts"
mqtt "example.com/mqtt5-learning/internal/mqtt"
)
func main() {
// Convert a returned application error into a visible nonzero process exit.
if err := run(); err != nil {
log.Print(err)
os.Exit(1)
}
}
func run() error {
// Load settings and bind cancellation to Ctrl+C or the container stop signal.
cfg, err := config.Load(os.Args[1:])
if err != nil {
return err
}
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
defer stop()
// Build one command filter and this device's concrete publish destinations.
commandFilter, err := contracts.DeviceSubscription(cfg.MQTT.DeviceID, contracts.KindCommands)
if err != nil { return err }
telemetryTopic, err := contracts.DeviceTopic(cfg.MQTT.DeviceID, contracts.KindTelemetry)
if err != nil { return err }
responseTopic, err := contracts.DeviceTopic(cfg.MQTT.DeviceID, contracts.KindResponses)
if err != nil { return err }
inbox := make(chan mqtt.Message, 16)
manager, err := mqtt.Start(ctx, cfg, []contracts.TopicFilter{commandFilter}, inbox)
if err != nil {
return err
}
defer mqtt.Stop(manager)
// Heartbeats are periodic; command handling remains active between ticks.
ticker := time.NewTicker(3 * time.Second)
defer ticker.Stop()
for {
// Handle cancellation, telemetry publication, and inbound commands together.
select {
case <-ctx.Done():
return nil
case <-ticker.C:
payload, err := json.Marshal(struct {
DeviceID string `json:"device_id"`
Time string `json:"time"`
}{cfg.MQTT.DeviceID, time.Now().UTC().Format(time.RFC3339)})
if err != nil {
return err
}
if err := mqtt.Publish(ctx, manager, telemetryTopic, payload); err != nil {
log.Printf("heartbeat: %v", err)
}
case message := <-inbox:
// Decode into a local command value only after rejecting retained messages.
var command struct {
ID string `json:"id"`
Command string `json:"command"`
}
if message.Retain {
log.Print("ignoring retained command")
continue
}
if err := json.Unmarshal(message.Payload, &command); err != nil || command.ID == "" || command.Command != "ping" {
log.Print("ignoring malformed or unsupported command")
continue
}
// Echo the command ID so the backend can associate this pong with its ping.
response, err := json.Marshal(struct {
ID string `json:"id"`
DeviceID string `json:"device_id"`
Status string `json:"status"`
}{command.ID, cfg.MQTT.DeviceID, "pong"})
if err != nil {
return err
}
if err := mqtt.Publish(ctx, manager, responseTopic, response); err != nil {
log.Printf("response: %v", err)
}
}
}
}
```
#### 🧭 `main` and `run` — simulate a vending device
**Summary:** `main` reports process failure; `run` publishes a heartbeat every three seconds and handles commands from the device's command topic.
**Details:** The simulator uses its own identity and subscribes only to its command namespace. The receive callback hands messages to `inbox`; the main loop validates and responds outside that callback. Cancellation returns cleanly and deferred cleanup stops the ticker and MQTT manager. This simulator is a learning stand-in, not a real vending device driver.
`json.Marshal` serializes Go structs to JSON bytes. Tags choose field names. `json.Unmarshal(message.Payload, &command)` needs a pointer so the decoder can fill the struct. The simulator's processing loop publishes responses outside the receive callback, keeping the callback short and avoiding blocking it on a network round trip.
This simulator sends a heartbeat every three seconds, receives ping commands, and publishes a pong carrying the same ID. It rejects packets marked retained to avoid treating stored state as a fresh one-shot action. The subscription sets `RetainAsPublished: true`: with the default false, live forwarding can clear RETAIN and defeat a guard that checks only `message.Retain`. Keep command publishers nonretaining too; this guard does not remove a retained value already stored at the broker. Full contract validation and duplicate handling arrive in later sections; this only demonstrates the transport lifecycle.
## ▶️ 6. Run two separate identities
> 💡 **Story question:** Why start two processes with different usernames and Client IDs, and what output proves each stage of the path?
The simulator and backend represent independent clients. Their connection logs, subscription results, broker publish acknowledgement, and pong are separate evidence. Start the simulator first so its command subscription exists before the backend publishes.
Terminal A, simulator first:
```sh
go run ./cmd/simulator -username simulator -client-id lab-simulator-device-001
```
Terminal B, backend:
```sh
go run ./cmd/backend -username backend -client-id lab-backend
```
Open two independent terminal sessions and change both to the project root. The shared lab password comes from `configs/lab.toml`; `-username` selects the broker account for each process, and `-client-id` overrides the TOML client ID. Stop each foreground program with Ctrl+C when finished.
Use the same configured `device-001` in both. Distinct usernames authorize different topic directions; both local accounts use the test password in `configs/lab.toml`. Distinct client IDs prevent connection takeover. A username, client ID, and payload device ID serve different purposes; the ACL is what binds this simulator credential to its permitted device namespace.
Expected backend output, abbreviated (timestamps and IDs vary):
```text
subscriptions ready: [lab/v1/devices/device-001/telemetry lab/v1/devices/device-001/responses]
received topic=lab/v1/devices/device-001/telemetry ... payload={"device_id":"device-001","time":"..."}
broker publish acknowledgement topic=lab/v1/devices/device-001/commands reason=0x00
received topic=lab/v1/devices/device-001/responses ... payload={"id":"ping-...","device_id":"device-001","status":"pong"}
```
Log order is not guaranteed: the device response may be observed around the publication acknowledgement. Separate connections and goroutines progress independently. If the simulator is not subscribed yet, a ping may have no recipient; this is expected with these zero-expiry early-lab sessions.
## 🧪 7. Experiments to learn the API
> 💡 **Story question:** Which part of the system changes in each experiment—the broker's routing/state, the client's delivery setting, or the application's handling policy?
Predict the effect at that boundary before changing the code, then use packet or application evidence to explain what happened.
1. Replace the backend's exact telemetry subscription with `lab/v1/devices/+/telemetry`. Explain the topic filter versus a concrete publish topic. Never publish to a topic containing `+` or `#`.
2. Add `Retain: true` only to synthetic telemetry; restart the backend and observe a retained value. Restore `false` afterward. Do not retain one-shot commands.
3. Change publication QoS to 0. Modify the helper's acknowledgement check: QoS 0 has no broker PUBACK; any returned response is not evidence of broker confirmation. Compare CLI packet traces at QoS 0 and 1.
4. Stop the simulator while the backend continues. Observe broker acknowledgement without a pong. Explain why the broker cannot tell you the vending application completed a ping.
5. Restart the broker. Observe reconnect and renewed subscription readiness. Ping delivery during the reconnect interval is not guaranteed by this helper.
6. Change simulator `-device-id device-002`. Its existing ACL must prevent the new subscriptions/publications; do not broaden its ACL to hide the failure.
7. Reduce inbox capacity and add a slow consumer. Observe explicit drops; explain why automatic acknowledgement can make those drops irreversible.
<details>
<summary>Self-check: why does QoS 1 still lose a command in the full-inbox experiment?</summary>
The immediate receiver is the MQTT client. Its automatic acknowledgement can complete delivery even though your callback drops the application copy. QoS cannot recover work your own receive path has acknowledged and discarded. Preserve important work before acknowledging, with an explicit overload and recovery policy.
</details>
<details>
<summary>Self-check: why does publishing with Retain false not clear the previous retained value?</summary>
The new publication does not ask the broker to change retained storage. Delete the stored value using a zero-byte retained publication to the same exact topic, then test with a fresh subscription. Stop any retained publisher during that check so it cannot immediately replace the value.
</details>
## ⚠️ Cautions: limits of this early example
- The inbox is deliberately bounded and lossy. It makes overload visible without blocking receive callbacks. It is unsuitable for durable sales reports or real commands; later sections replace this policy with persistence and deliberate acknowledgement/backpressure rules.
- A publish call can block this single application loop for up to three seconds; this is acceptable for a tiny demonstration. Later sections add workers and separate business processing.
- QoS 1 can duplicate delivery. Ping is harmless, but real effects need stable command IDs and device-side deduplication.
- Payload validation here is intentionally minimal. Content Type is descriptive metadata, not authentication or schema enforcement.
- Fresh sessions and zero expiry mean offline commands are not a durable workflow. Do not infer restart guarantees from a successful reconnect.
- Raw payload logs are allowed here only for small synthetic lab messages; production logs need redaction and size limits.
- Never close the inbox while Paho may still send to it. These programs let the context and manager stop the producer; the channel need not be closed for process exit.
## ✅ Verification and completion
```sh
gofmt -w cmd internal
go test ./...
go vet ./...
go build ./cmd/backend ./cmd/simulator ./cmd/configcheck
```
`mkdir -p` above creates missing parent directories without creating Go files. `gofmt -w cmd internal` formats Go files under those directories. `go test ./...` runs tests in all packages in this module; `...` is Go's recursive package pattern. `go vet ./...` performs static checks. `go build` with several package paths checks all those executables without writing one combined binary. Run these from the root, after creating all referenced packages.
For the retain experiment, publish a zero-byte retained payload to that exact telemetry topic to clear its retained value; merely switching future publications back to nonretained does not erase the broker's old retained message. Publish with an identity authorized to write the topic. For broker restart, use the explained lifecycle command in [Broker operations](broker-operations.md), then watch both clients reconnect and resubscribe.
Packages without tests do not demonstrate runtime correctness. Also run both programs, perform the failure experiments, and verify clean Ctrl+C behavior.
**Completion:** Explain the complete path from backend Go struct → JSON bytes → MQTT PUBLISH → simulator callback → application loop → response PUBLISH → backend callback. Distinguish broker acknowledgement from a device response, and identify the lossy queue and offline-session limitations yourself.
## 🔗 Sources
- [Paho client API v0.23.0](https://pkg.go.dev/github.com/eclipse/paho.golang@v0.23.0/paho)
- [Tagged AutoPaho source](https://github.com/eclipse-paho/paho.golang/blob/v0.23.0/autopaho/auto.go)
- [Tagged receive callback source](https://github.com/eclipse-paho/paho.golang/blob/v0.23.0/paho/client.go)
- [MQTT 5 specification](https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html)
## ✅ Result — the story so far
The backend now subscribes to device telemetry and responses, while the simulator subscribes to that device's commands. The simulator publishes heartbeats for the backend; when the backend sends a QoS 1 ping, the broker forwards it to the simulator, which handles it and publishes a pong carrying the same command ID. The PUBACKs confirm their individual MQTT hops; the pong is the separate application evidence that the simulator handled the ping.
Next: [Section 5 — Contracts and validation](05-contracts-validation.md).
teaching/course-source/05-contracts-validation.md
+714
View File
@@ -0,0 +1,714 @@
# 📋 Section 5 — Design contracts and validate inbound messages
## 🎬 Start with the system story
> **❓ Question:** **The broker can route a message because its topic matches. How does the Go backend decide that the topic and payload describe a valid message from the expected device?**
**💬 Answer:** The topic provides the route, but it does not verify the payload. The backend parses the topic, validates the JSON, and compares the claimed device ID before business handling. Broker authentication and ACLs separately control who may publish to a topic.
```mermaid
flowchart LR
B[Broker routes matching topic] --> P[Paho delivers topic + bytes]
P --> T[Parse topic and device scope]
T --> J[Decode versioned JSON]
J --> V[Validate identity, IDs, size, time]
V -->|valid| H[Business handler]
V -->|invalid| E[Safe rejection / metric]
```
**Before each implementation step, ask:** What can be trusted at this boundary, and what must be checked before the application acts? Follow [Lesson 0](00-system-lifecycle.md) for the complete message path.
Prerequisite: Section 4 exchanges heartbeats and ping/pong. The clients share the disposable password in `configs/lab.toml`; select the simulator account with `-username simulator` when starting it.
Run all commands from `/home/ryan/Media/Code/Code/robot-market/mqtt-5-learning`. Save the complete code blocks at their project-relative paths, not under `teaching/course-source/`. This lesson provides a full working implementation with the pinned Paho v0.23.0 dependency; use the exercises to inspect and verify its behavior.
## 📨 Follow one message through the trust boundaries
**❓ Question:** **The backend receives a sales report on device-001's topic, but its JSON says `device_id: device-002`. What could go wrong if the backend accepts it?**
**💬 Answer:** The report could be applied to the wrong device. The broker may allow the publish, but it does not compare the topic with the JSON. The backend compares both IDs and rejects a mismatch before changing data.
```text
PUBLISH → size limit → topic parser → JSON decoder → schema checks
→ topic/body identity check → business validation → handler
```
The backend applies several checks because each catches a different problem. A size limit bounds the payload it decodes. The topic parser identifies the device and message type. JSON decoding checks the data's shape, while business validation checks its values. For example, a sales report can be valid JSON but contain a negative quantity. The backend rejects that value. These checks do not replace broker authentication or ACLs.
**❓ Question:** **Does the 16 KiB application limit cap the whole MQTT packet?**
**💬 Answer:** No. It caps the payload the application decodes. The topic and properties add packet bytes, and Paho has already received the packet before this check runs. Lesson 16 covers an earlier packet-size limit.
### 📨 Map encoding metadata into Paho
This fragment belongs where you build a `paho.Publish`; `payload` is the JSON produced by your contract encoder:
```go
utf8Payload := byte(1)
// Describe the payload's encoding and media type for MQTT 5 consumers.
properties := &paho.PublishProperties{
ContentType: "application/json",
PayloadFormat: &utf8Payload,
}
// Attach these properties to the PUBLISH packet built by your transport helper.
_ = properties
```
**❓ Question:** **A message declares JSON, but its payload is `hello`. Is it valid JSON?**
**💬 Answer:** No. Content Type describes the media type; Payload Format Indicator 1 says the bytes are UTF-8 text. Neither validates the content or proves its schema. Decode the bytes, and check an incoming property pointer for nil before reading it; nil means that property was omitted.
Keep MQTT properties with the received transport message when the application needs them. Fields such as `schema_version` and `command_id` belong in the JSON contract type. This lets handlers use the needed data without depending on a packet object that another part of the program may change.
### 🧪 Predict, modify, explain
Send a valid heartbeat, then change only its body device ID, then change only its schema version. Predict which validation stage rejects each. Finally publish valid JSON with an extra trailing object: one successful decoder call is insufficient unless you also require EOF.
<details>
<summary>Why can an invalid message still get a successful MQTT acknowledgement?</summary>
**❓ Question:** **The broker delivers a sales report with quantity `-3`. Can the MQTT client acknowledge it even if the application rejects it?**
**💬 Answer:** Yes. In this helper, the callback can return and the MQTT client may acknowledge delivery. The backend then records the validation failure and rejects the sale. PUBACK confirms the MQTT delivery step; it does not mean the sale succeeded.
</details>
## 📋 1. Create the contract package
> **❓ Question:** **What shared vocabulary lets the backend and device agree on the shape and meaning of each message?**
The contract package gives both programs the same field names and meanings. Its `topic.go` file owns the hierarchy and distinguishes a concrete publish name from a subscription filter; `message.go` owns JSON payload types and their checks. For example, if the simulator sends `device` while the backend expects `device_id`, the message may be rejected or misunderstood.
```sh
mkdir -p internal/contracts
```
The complete implementation for this step is shown below. Save each block at its stated path, then update the backend and simulator to use these contracts before they handle or publish messages.
Save this as `internal/contracts/topic.go`. Section 4 introduced its basic purpose; here it is included in full so this lesson can be used on its own.
### 🔍 `topic.go` — file overview
This file owns the lab hierarchy `lab/v1/devices/{deviceID}/{kind}`. It builds concrete `TopicName` values for PUBLISH, fleet-wide and exact-device `TopicFilter` values for SUBSCRIBE, and parses an incoming name into a device ID and flow kind. Composition code still chooses the role's subscriptions, and Mosquitto ACLs still authorize them.
```go
package contracts
import (
"fmt"
"strings"
)
type TopicName string
type TopicFilter string
type Kind string
const (
KindTelemetry Kind = "telemetry"
KindCommands Kind = "commands"
KindResponses Kind = "responses"
KindSales Kind = "sales"
)
const topicPrefix = "lab/v1/devices/"
type Topic struct {
DeviceID string
Kind Kind
}
func (t TopicName) String() string { return string(t) }
func (f TopicFilter) String() string { return string(f) }
func DeviceTopic(deviceID string, kind Kind) (TopicName, error) {
if !validID(deviceID) {
return "", fmt.Errorf("invalid device ID %q", deviceID)
}
if !supportedKind(kind) {
return "", fmt.Errorf("unsupported topic kind %q", kind)
}
return TopicName(topicPrefix + deviceID + "/" + string(kind)), nil
}
func DeviceFilter(kind Kind) (TopicFilter, error) {
if !supportedKind(kind) {
return "", fmt.Errorf("unsupported topic kind %q", kind)
}
return TopicFilter(topicPrefix + "+/" + string(kind)), nil
}
func DeviceSubscription(deviceID string, kind Kind) (TopicFilter, error) {
topic, err := DeviceTopic(deviceID, kind)
if err != nil {
return "", err
}
return TopicFilter(topic.String()), nil
}
func ParseTopic(value string) (Topic, error) {
if !strings.HasPrefix(value, topicPrefix) {
return Topic{}, fmt.Errorf("topic must be %s{deviceID}/{kind}", topicPrefix)
}
parts := strings.Split(strings.TrimPrefix(value, topicPrefix), "/")
if len(parts) != 2 || !validID(parts[0]) {
return Topic{}, fmt.Errorf("topic must be %s{deviceID}/{kind}", topicPrefix)
}
kind := Kind(parts[1])
if !supportedKind(kind) {
return Topic{}, fmt.Errorf("unsupported topic kind %q", kind)
}
return Topic{DeviceID: parts[0], Kind: kind}, nil
}
func supportedKind(kind Kind) bool {
switch kind {
case KindTelemetry, KindCommands, KindResponses, KindSales:
return true
default:
return false
}
}
func validID(value string) bool {
if value == "" || len(value) > 128 {
return false
}
for _, r := range value {
if !(r >= 'a' && r <= 'z' || r >= 'A' && r <= 'Z' || r >= '0' && r <= '9' || strings.ContainsRune("._-", r)) {
return false
}
}
return true
}
```
#### 🧱 `DeviceTopic`, `DeviceFilter`, and `DeviceSubscription` — build the right kind of path
**Summary:** Build a concrete publish name, a fleet-wide flow filter, or an exact per-device subscription filter.
**Details:** `DeviceTopic` never inserts wildcard characters. `DeviceFilter` places one `+` wildcard at the device level for a known flow; `DeviceSubscription` reuses the validated name to produce an exact filter. The backend uses fleet-wide filters, while the simulator subscribes only to its configured device. These Go types reduce accidental mixing; broker ACLs remain the authorization boundary.
#### 🧭 `ParseTopic` — recover routing data
**Summary:** Check the shared `topicPrefix`, then validate the remaining device ID and flow kind.
**Details:** `ParseTopic` first requires the incoming name to start with `topicPrefix`, then splits only the suffix into exactly two levels. The parser does not repeat the prefix's individual segments, so changing the constant updates builders and parsing together. Parsing establishes which contract to decode and lets the application compare topic identity with payload identity. It does not prove which client published the message.
Save the next block as `internal/contracts/message.go`. This file defines the versioned payloads and value checks. Topic hierarchy and parsing live in `topic.go`. A command ID remains stable across retries; an MQTT packet identifier identifies only one protocol exchange.
### 🔍 `message.go` — file overview
This file defines JSON payload types and their validation rules. Its validation functions check the relationship between parsed topic data and the body before a decoded value can reach a handler.
```go
package contracts
import (
"fmt"
"time"
)
const SchemaVersion = 1
// These types define the JSON fields shared by backend and simulator.
type Heartbeat struct {
SchemaVersion int `json:"schema_version"`
MessageID string `json:"message_id"`
DeviceID string `json:"device_id"`
OccurredAt string `json:"occurred_at"`
Temperature *float64 `json:"temperature,omitempty"`
}
type Command struct {
SchemaVersion int `json:"schema_version"`
MessageID string `json:"message_id"`
DeviceID string `json:"device_id"`
OccurredAt string `json:"occurred_at"`
CommandID string `json:"command_id"`
Command string `json:"command"`
}
type CommandResult struct {
SchemaVersion int `json:"schema_version"`
MessageID string `json:"message_id"`
DeviceID string `json:"device_id"`
OccurredAt string `json:"occurred_at"`
CommandID string `json:"command_id"`
Status string `json:"status"`
Detail string `json:"detail,omitempty"`
}
type SalesLine struct {
SKU string `json:"sku"`
Quantity int64 `json:"quantity"`
AmountMinor int64 `json:"amount_minor"`
}
type SalesReport struct {
SchemaVersion int `json:"schema_version"`
ReportID string `json:"report_id"`
DeviceID string `json:"device_id"`
OccurredAt string `json:"occurred_at"`
Items []SalesLine `json:"items"`
}
// validateEnvelope checks fields shared by every versioned message.
func validateEnvelope(topic Topic, version int, messageID, deviceID, occurredAt string, now time.Time) error {
if version != SchemaVersion {
return fmt.Errorf("unsupported schema_version %d", version)
}
if !validID(messageID) || !validID(deviceID) {
return fmt.Errorf("message_id and device_id must be valid IDs")
}
if topic.DeviceID != deviceID {
return fmt.Errorf("topic device %q does not match payload device_id %q", topic.DeviceID, deviceID)
}
stamp, err := time.Parse(time.RFC3339, occurredAt)
if err != nil {
return fmt.Errorf("occurred_at must be RFC3339: %w", err)
}
if stamp.After(now.Add(30 * time.Second)) {
return fmt.Errorf("occurred_at is more than 30 seconds in the future")
}
return nil
}
// Each Validate method adds checks specific to one message kind.
func (m Heartbeat) Validate(topic Topic, now time.Time) error {
if topic.Kind != KindTelemetry {
return fmt.Errorf("heartbeat requires telemetry topic")
}
if err := validateEnvelope(topic, m.SchemaVersion, m.MessageID, m.DeviceID, m.OccurredAt, now); err != nil {
return err
}
if m.Temperature != nil && (*m.Temperature < -50 || *m.Temperature > 100) {
return fmt.Errorf("temperature is outside the supported range")
}
return nil
}
func (m Command) Validate(topic Topic, now time.Time) error {
if topic.Kind != KindCommands {
return fmt.Errorf("command requires commands topic")
}
if err := validateEnvelope(topic, m.SchemaVersion, m.MessageID, m.DeviceID, m.OccurredAt, now); err != nil {
return err
}
if !validID(m.CommandID) || m.Command != "ping" {
return fmt.Errorf("command_id is invalid or command is unsupported")
}
return nil
}
func (m CommandResult) Validate(topic Topic, now time.Time) error {
if topic.Kind != KindResponses {
return fmt.Errorf("command result requires responses topic")
}
if err := validateEnvelope(topic, m.SchemaVersion, m.MessageID, m.DeviceID, m.OccurredAt, now); err != nil {
return err
}
if !validID(m.CommandID) || (m.Status != "accepted" && m.Status != "completed" && m.Status != "rejected") {
return fmt.Errorf("command result has invalid command_id or status")
}
return nil
}
func (m SalesReport) Validate(topic Topic, now time.Time) error {
if topic.Kind != KindSales {
return fmt.Errorf("sales report requires sales topic")
}
if err := validateEnvelope(topic, m.SchemaVersion, m.ReportID, m.DeviceID, m.OccurredAt, now); err != nil {
return err
}
if len(m.Items) == 0 || len(m.Items) > 100 {
return fmt.Errorf("sales report must contain 1 to 100 items")
}
for i, item := range m.Items {
if !validID(item.SKU) || item.Quantity <= 0 || item.AmountMinor < 0 {
return fmt.Errorf("item %d has invalid SKU, quantity, or amount_minor", i)
}
}
return nil
}
```
#### 🪪 `validateEnvelope` — check shared payload fields
**Summary:** Apply identifier syntax, schema-version, topic/body device identity, and timestamp checks used by several message types.
**Details:** `validateEnvelope` checks fields shared by all payload contracts and takes `now` as an argument, allowing deterministic time validation. Topic-segment ID syntax is owned by `topic.go`. Neither helper authenticates the MQTT publisher; broker credentials and ACLs perform a separate job.
#### ✅ Each `Validate` method — check one contract's rules
**Summary:** Enforce that the JSON type matches the topic kind and that message-specific values are supported.
**Details:** `Heartbeat.Validate` checks optional temperature bounds; `Command.Validate` accepts only the current harmless `ping`; `CommandResult.Validate` permits known result states; `SalesReport.Validate` checks item count and each SKU, quantity, and amount. The methods are called by `Decode`; handlers should not bypass them.
`internal/contracts/decode.go` bounds the payload before decoding, rejects unknown fields, requires the JSON value to consume the entire input, and calls the decoded contract's `Validate` method before returning it. `MQTTMessage` restricts the generic decoder to types with that method. The parsed topic and clock are explicit inputs so validation can check route/body identity and remain deterministic in tests.
```go
package contracts
import (
"bytes"
"encoding/json"
"fmt"
"io"
"time"
)
const MaxPayloadBytes = 16 * 1024
// Decode bounds input, rejects unknown fields, requires one JSON value,
// and validates it against the topic before returning it.
func Decode[T MQTTMessage](payload []byte, topic Topic, now time.Time) (T, error) {
var value T
if len(payload) > MaxPayloadBytes {
return value, fmt.Errorf("payload exceeds %d bytes", MaxPayloadBytes)
}
decoder := json.NewDecoder(bytes.NewReader(payload))
decoder.DisallowUnknownFields()
if err := decoder.Decode(&value); err != nil {
return value, fmt.Errorf("decode JSON: %w", err)
}
var trailing any
// Decode reads one JSON value at a time. A second call distinguishes
// whitespace (EOF) from a second value or malformed trailing bytes.
if err := decoder.Decode(&trailing); err != io.EOF {
if err == nil {
return value, fmt.Errorf("payload contains more than one JSON value")
}
return value, fmt.Errorf("trailing data after JSON value: %w", err)
}
if err := value.Validate(topic, now); err != nil {
return value, fmt.Errorf("validate message: %w", err)
}
return value, nil
}
```
### 🔍 `decode.go` — file overview and function guide
This file performs bounded JSON decoding and contract validation. `MQTTMessage` is the type constraint: only message types with a `Validate` method can be passed to `Decode`.
#### 📥 `Decode` — accept one valid contract value
**Summary:** Enforces the byte limit, decodes exactly one JSON object, rejects unknown fields, then calls the selected type's validator.
**Details:** The second decoder call distinguishes trailing whitespace from a second JSON value or malformed trailing bytes. The function returns no validated value until every structural and business check succeeds. This protects application state from malformed data, but a valid value still needs an application service to decide what to do with it; Section 5A adds that missing step.
`internal/contracts/decode_test.go` checks malformed boundaries without a broker:
```go
package contracts
import (
"strings"
"testing"
"time"
)
// Topic tests reject malformed routes and unsupported message kinds.
func TestParseTopic(t *testing.T) {
topic, err := ParseTopic("lab/v1/devices/device-001/sales")
if err != nil { t.Fatal(err) }
if topic.DeviceID != "device-001" || topic.Kind != KindSales {
t.Fatalf("unexpected topic: %#v", topic)
}
for _, invalid := range []string{"lab/v1/devices/+/sales", "lab/v1/devices/device-001/unknown", "lab/v1/devices/device-001/sales/extra"} {
if _, err := ParseTopic(invalid); err == nil { t.Errorf("ParseTopic(%q) succeeded", invalid) }
}
}
// These cases cover strict JSON shape and single-value decoding.
func TestDecodeRejectsUnknownAndTrailingValues(t *testing.T) {
topic := Topic{DeviceID: "device-001", Kind: KindTelemetry}
now := time.Date(2026, 10, 13, 12, 0, 0, 0, time.UTC)
if _, err := Decode[Heartbeat]([]byte(`{"schema_version":1,"unknown":true}`), topic, now); err == nil {
t.Fatal("unknown field was accepted")
}
if _, err := Decode[Heartbeat]([]byte(`{"schema_version":1,"message_id":"m-1","device_id":"device-001","occurred_at":"2026-10-13T12:00:00Z"} {"message_id":"m-2"}`), topic, now); err == nil {
t.Fatal("trailing JSON value was accepted")
}
}
// This case ensures the application limit is applied before decoding.
func TestDecodeRejectsOversizedPayload(t *testing.T) {
payload := []byte(`"` + strings.Repeat("x", MaxPayloadBytes) + `"`)
if _, err := Decode[Heartbeat](payload, topic, now); err == nil {
t.Fatal("oversized payload was accepted")
}
}
// A body cannot override the device selected by the topic.
func TestCommandRejectsTopicIdentityMismatch(t *testing.T) {
topic, err := ParseTopic("lab/v1/devices/device-001/commands")
if err != nil { t.Fatal(err) }
command := Command{SchemaVersion: SchemaVersion, MessageID: "m-1", DeviceID: "device-002", OccurredAt: time.Now().UTC().Format(time.RFC3339), CommandID: "c-1", Command: "ping"}
if err := command.Validate(topic, time.Now().UTC()); err == nil {
t.Fatal("topic/body identity mismatch was accepted")
}
}
```
## 🏷️ 2. Parse topics before routing
> **❓ Question:** **A message arrives with both a topic and a payload. Why decide which device and message kind the topic represents before running a handler?**
`contracts.ParseTopic` in `topic.go` accepts exactly `lab/v1/devices/{deviceID}/{kind}`. It rejects unsafe IDs, extra segments, wildcard characters, and unknown message kinds. Use it before decoding so the validator can compare the route's device ID with the payload claim. Add the sales ACL rules only when you publish sales reports.
## ✅ 3. Decode and validate
> **❓ Question:** **JSON decoding succeeded. What else must be true before this message is safe for application state?**
**💬 Answer:** Decoding checks syntax, not whether the message is safe to use. The backend also checks size, schema, required IDs, timestamps, values, and agreement between the topic and claimed device before business handling.
`contracts.Decode[T]` in `decode.go` applies the 16 KiB payload limit, rejects unknown JSON fields, requires EOF after one value, and calls `Validate(topic, now)` before returning. `MQTTMessage` restricts `T` to contract types that implement validation. Validation checks schema version 1, required IDs, supported commands, topic/body device agreement, RFC3339 timestamps, and message-specific values. Passing `now` explicitly lets callers use a fixed clock in tests. Historical sales are allowed; the validator does not apply heartbeat freshness rules to reports.
Metadata describes the payload, and the ID in it is only a claim. The backend validates the bytes and compares that ID with the topic. Broker credentials and ACLs separately authorize which sender may publish to the topic.
## 🧪 4. Integrate and verify
> **❓ Question:** **Where should validation sit so malformed messages cannot trigger a response or change state?**
Run validation after receiving the message but before business handling. Otherwise, a malformed command could already change state or trigger a reply. Test accepted and rejected messages, then test broker ACL permissions separately; application validation does not decide who may publish to a topic.
For one sales message, the processing order is:
```go
topic, err := contracts.ParseTopic(message.Topic)
if err != nil {
return err
}
report, err := contracts.Decode[contracts.SalesReport](message.Payload, topic, time.Now().UTC())
if err != nil {
return err
}
// Only a validated report reaches the business handler.
```
`Decode` reads one JSON value per call. After decoding the expected report, it calls `Decode` again: `io.EOF` means only whitespace remained, another successful decode means there was a second JSON value, and a different error means malformed trailing bytes. Only after that structural check does it call the report's `Validate` method.
Replace Section 4's anonymous JSON structs in both programs with these contracts. The complete executable files are below; each validates inbound data before logging, changing state, or sending a result. Rejections log a bounded error, not the full payload.
`cmd/backend/main.go`:
```go
package main
import (
"context"
"encoding/json"
"fmt"
"log"
"os"
"os/signal"
"syscall"
"time"
"example.com/mqtt5-learning/internal/config"
"example.com/mqtt5-learning/internal/contracts"
"example.com/mqtt5-learning/internal/mqtt"
)
func main() {
if err := run(); err != nil {
log.Fatal(err)
}
}
func run() error {
cfg, err := config.Load(os.Args[1:])
if err != nil {
return fmt.Errorf("load config: %w", err)
}
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
defer stop()
telemetryFilter, err := contracts.DeviceFilter(contracts.KindTelemetry)
if err != nil { return fmt.Errorf("build telemetry filter: %w", err) }
responsesFilter, err := contracts.DeviceFilter(contracts.KindResponses)
if err != nil { return fmt.Errorf("build responses filter: %w", err) }
salesFilter, err := contracts.DeviceFilter(contracts.KindSales)
if err != nil { return fmt.Errorf("build sales filter: %w", err) }
commandTopic, err := contracts.DeviceTopic(cfg.MQTT.DeviceID, contracts.KindCommands)
if err != nil { return fmt.Errorf("build command topic: %w", err) }
// Start subscriptions before handling any delivered message.
inbox := make(chan mqtt.Message, 16)
manager, err := mqtt.Start(ctx, cfg, []contracts.TopicFilter{telemetryFilter, responsesFilter, salesFilter}, inbox)
if err != nil {
return fmt.Errorf("start MQTT: %w", err)
}
defer mqtt.Stop(manager)
ticker := time.NewTicker(10 * time.Second)
defer ticker.Stop()
sequence := 0
for {
select {
case <-ctx.Done():
return nil
case message := <-inbox:
if err := handleMessage(message); err != nil {
log.Printf("rejected topic=%q reason=%v", message.Topic, err)
}
case <-ticker.C:
sequence++
id := fmt.Sprintf("ping-%d-%d", time.Now().UTC().Unix(), sequence)
command := contracts.Command{SchemaVersion: contracts.SchemaVersion, MessageID: id, DeviceID: cfg.MQTT.DeviceID, OccurredAt: time.Now().UTC().Format(time.RFC3339), CommandID: id, Command: "ping"}
payload, err := json.Marshal(command)
if err != nil {
return fmt.Errorf("encode command: %w", err)
}
if err := mqtt.Publish(ctx, manager, commandTopic, payload); err != nil {
log.Printf("command publish failed or uncertain id=%s: %v", id, err)
}
}
}
}
func handleMessage(message mqtt.Message) error {
// Parse and validate before a message can reach application handling.
topic, err := contracts.ParseTopic(message.Topic)
if err != nil {
return err
}
now := time.Now().UTC()
switch topic.Kind {
case contracts.KindTelemetry:
value, err := contracts.Decode[contracts.Heartbeat](message.Payload, topic, now)
if err != nil { return err }
log.Printf("valid heartbeat device=%s message=%s", value.DeviceID, value.MessageID)
case contracts.KindResponses:
value, err := contracts.Decode[contracts.CommandResult](message.Payload, topic, now)
if err != nil { return err }
log.Printf("command result device=%s command=%s status=%s", value.DeviceID, value.CommandID, value.Status)
case contracts.KindSales:
value, err := contracts.Decode[contracts.SalesReport](message.Payload, topic, now)
if err != nil { return err }
log.Printf("valid sales report device=%s report=%s items=%d", value.DeviceID, value.ReportID, len(value.Items))
default:
return fmt.Errorf("backend does not consume topic kind %q", topic.Kind)
}
return nil
}
```
#### 🧭 `main`, `run`, and `handleMessage` — validate backend traffic
**Summary:** `main` reports startup failure, `run` owns the connection and receive loop, and `handleMessage` routes each topic kind to its typed decoder.
**Details:** The receive loop logs rejected messages without crashing the process. `handleMessage` parses the topic before decoding; that parsed kind chooses the expected payload type, and `Decode` checks the body against the route. The default branch refuses kinds the backend does not consume. At this stage successful sales and results are only logged; Section 5A creates application state, while Sections 7 and 13 add matching and durable sales storage.
`cmd/simulator/main.go`:
```go
package main
import (
"context"
"encoding/json"
"fmt"
"log"
"os"
"os/signal"
"syscall"
"time"
"example.com/mqtt5-learning/internal/config"
"example.com/mqtt5-learning/internal/contracts"
"example.com/mqtt5-learning/internal/mqtt"
"github.com/eclipse/paho.golang/autopaho"
)
func main() {
if err := run(); err != nil { log.Fatal(err) }
}
func run() error {
cfg, err := config.Load(os.Args[1:])
if err != nil { return fmt.Errorf("load config: %w", err) }
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
defer stop()
// Construct routes before connecting so invalid identity fails at startup.
commandFilter, err := contracts.DeviceSubscription(cfg.MQTT.DeviceID, contracts.KindCommands)
if err != nil { return err }
telemetryTopic, err := contracts.DeviceTopic(cfg.MQTT.DeviceID, contracts.KindTelemetry)
if err != nil { return err }
responseTopic, err := contracts.DeviceTopic(cfg.MQTT.DeviceID, contracts.KindResponses)
if err != nil { return err }
inbox := make(chan mqtt.Message, 16)
manager, err := mqtt.Start(ctx, cfg, []contracts.TopicFilter{commandFilter}, inbox)
if err != nil { return fmt.Errorf("start MQTT: %w", err) }
defer mqtt.Stop(manager)
ticker := time.NewTicker(3 * time.Second)
defer ticker.Stop()
sequence := 0
for {
select {
case <-ctx.Done():
return nil
case <-ticker.C:
sequence++
heartbeat := contracts.Heartbeat{SchemaVersion: contracts.SchemaVersion, MessageID: fmt.Sprintf("heartbeat-%d-%d", time.Now().Unix(), sequence), DeviceID: cfg.MQTT.DeviceID, OccurredAt: time.Now().UTC().Format(time.RFC3339)}
payload, err := json.Marshal(heartbeat)
if err != nil { return fmt.Errorf("encode heartbeat: %w", err) }
if err := mqtt.Publish(ctx, manager, telemetryTopic, payload); err != nil { log.Printf("heartbeat publish: %v", err) }
case message := <-inbox:
if message.Retain { log.Printf("reject retained command topic=%q", message.Topic); continue }
if err := respond(ctx, manager, cfg.MQTT.DeviceID, message); err != nil { log.Printf("reject command topic=%q reason=%v", message.Topic, err) }
}
}
}
func respond(ctx context.Context, manager *autopaho.ConnectionManager, deviceID string, message mqtt.Message) error {
// Check route and body identity before constructing a command result.
topic, err := contracts.ParseTopic(message.Topic)
if err != nil { return err }
command, err := contracts.Decode[contracts.Command](message.Payload, topic, time.Now().UTC())
if err != nil { return err }
if topic.DeviceID != deviceID { return fmt.Errorf("command is for another device") }
result := contracts.CommandResult{SchemaVersion: contracts.SchemaVersion, MessageID: "result-"+command.CommandID, DeviceID: deviceID, OccurredAt: time.Now().UTC().Format(time.RFC3339), CommandID: command.CommandID, Status: "completed"}
payload, err := json.Marshal(result)
if err != nil { return fmt.Errorf("encode result: %w", err) }
responseTopic, err := contracts.DeviceTopic(deviceID, contracts.KindResponses)
if err != nil { return fmt.Errorf("build response topic: %w", err) }
return mqtt.Publish(ctx, manager, responseTopic, payload)
}
```
#### 📟 `respond` — validate a command and publish its result
**Summary:** Parses the command topic, validates its payload, confirms the target device, then publishes a completed result.
**Details:** Each check returns before result construction if the topic or body is invalid. The simulator only demonstrates a harmless ping here; it does not execute a real vending-machine operation. Section 7 adds retained-message rejection, response-topic authorization, Correlation Data, and duplicate suppression.
#### 🧭 `main` and `run` — operate the simulator
**Summary:** `main` turns failures into a nonzero exit; `run` loads the simulator identity, subscribes to commands, publishes heartbeats, and calls `respond` for each inbound packet.
**Details:** `run` owns the signal context, manager, ticker, and receive loop. It keeps network callback work separate from contract validation and result publication. The heartbeat ticker is independent of command reception, so an idle device still reports recent contact. Section 6 moves potentially slow message work into bounded workers.
```sh
gofmt -w internal/contracts cmd
go test ./internal/contracts
go test ./...
```
`mkdir -p` creates the package directory. `gofmt -w` formats the complete files above. `go test ./internal/contracts` runs the supplied pure parser/validator tests without a broker; `go test ./...` checks the rest of the application too.
When adding sales, edit `lab/mosquitto/acl`: add `topic read lab/v1/devices/+/sales` under backend and `topic write lab/v1/devices/device-001/sales` under simulator. Restart the lab broker to read the changed ACL, check logs, and test allowed and cross-device-denied publications. Permissions enable a topic direction; they do not replace sales validation.
The supplied tests cover topic parsing, malformed JSON boundaries, payload size, and topic/body identity. Add a case when you change a contract so the test captures the new rule. Keep the simulator and backend contracts in sync when changing the wire format.
**Done when:** Both programs still exchange ping/pong, every rejection has a predictable reason, and invalid sales or commands leave state unchanged. Explain why schema validation does not authenticate a device.
## ✅ Result — the story so far
The backend and simulator now have a shared vocabulary for heartbeats, commands, results, and sales reports. When a message arrives, the receiver checks its topic, size, JSON shape, schema version, identifiers, timestamp, and topic/body device match before changing application state. A broker-routed message can still be rejected by the application; successful MQTT delivery alone does not make its contents valid or trusted.
Previous: [04-publish-subscribe-simulator.md](04-publish-subscribe-simulator.md). Next: [05A — Backend message processing](05a-backend-message-processing.md).
teaching/course-source/05a-backend-message-processing.md
+352
View File
@@ -0,0 +1,352 @@
# 🧭 Section 5A — Turn a valid message into backend state
## 🎬 Start with the system story
> **❓ Question:** **The backend received a valid heartbeat. What changes in the application, and how can another part of the backend ask whether the device was recently heard from?**
Section 5 proves that a packet has a valid topic and payload. That is a trust boundary, not a business action. A backend still needs an application service that decides what the valid message means, a place to record the result, and a safe way to report the outcome.
```mermaid
flowchart LR
B[Mosquitto delivers PUBLISH] --> R[Receive loop]
R --> V[Parse topic and validate contract]
V -->|valid heartbeat| S[Application service]
S --> M[In-memory device view]
S --> L[Structured application log]
M --> Q[Status query: recent heartbeat?]
V -->|invalid| E[Log bounded rejection; no state change]
```
The exercise adds one small vertical slice: a valid heartbeat updates the backend's last-seen view. It uses an in-memory store so the data flow is visible before concurrency and PostgreSQL arrive. A fresh heartbeat is evidence of recent application traffic; it is not proof that the MQTT connection is currently up or that every part of the vending machine is healthy.
**Before coding, predict:** Which facts come from the broker, which come from validation, and which are created by the application? A topic and schema-valid body are not yet a stored status, and a `PUBACK` is not evidence that this store was updated.
Prerequisite: [Section 4](04-publish-subscribe-simulator.md) introduced the complete `internal/contracts/topic.go` hierarchy: `DeviceTopic` builds concrete names, `DeviceFilter` builds fleet filters, `DeviceSubscription` builds exact-device filters, and `ParseTopic` extracts the route. [Section 5](05-contracts-validation.md) adds `contracts.Decode` and `contracts.Heartbeat`. The code below assumes the Paho message-copying receive helper and `mqtt.Message` from Section 4; topics must use the shared contract rather than hand-built strings.
## 🧩 1. Separate validation from application meaning
> **❓ Question:** **Why should the callback not also decide what “online” means and update shared device state?**
The callback owns packet delivery. The application service owns the meaning of a validated message. Keeping those responsibilities separate lets the backend test status behavior without a broker and later move the service behind Section 6's worker pool.
Trace one heartbeat:
```text
PUBLISH → callback copies topic and bytes → inbox
→ service parses topic → Decode validates contract
→ store updates last-seen snapshot → structured log records the outcome
```
The service gives each existing contract an honest current behavior: heartbeats update a last-seen view, while sales reports and command results are validated and logged but not yet persisted or correlated. Section 7 adds command matching; Section 13 adds durable sales storage; Section 12 adds bounded device-log and richer health workflows. This boundary prevents a successful decode from being presented as completed business work.
## 🗃️ 2. Create a small in-memory device view
> **❓ Question:** **Which timestamp should control freshness: when the device says it observed the heartbeat, or when this backend received it?**
Store both. Use the device timestamp to avoid replacing a newer report with an older delayed one. Use the backend receive timestamp to answer whether a heartbeat arrived recently; the device clock does not decide when this backend last heard from it.
Save this complete file as `internal/deviceview/store.go`.
### 🔍 `store.go` — file overview
This file stores the newest accepted heartbeat per device and derives a freshness view. The map is protected because Section 6 will call the application service from multiple workers. The store is still volatile: a process restart clears every snapshot.
```go
package deviceview
import (
"fmt"
"sync"
"time"
"example.com/mqtt5-learning/internal/contracts"
)
type Snapshot struct {
DeviceID string
MessageID string
ReportedAt time.Time
ReceivedAt time.Time
Temperature float64
TemperatureSet bool
}
type Status struct {
Snapshot Snapshot
HeartbeatFresh bool
}
type Store struct {
mu sync.RWMutex
latest map[string]Snapshot
}
func NewStore() *Store {
return &Store{latest: make(map[string]Snapshot)}
}
func (s *Store) RecordHeartbeat(message contracts.Heartbeat, receivedAt time.Time) (bool, error) {
reportedAt, err := time.Parse(time.RFC3339, message.OccurredAt)
if err != nil {
// Decode already validates this timestamp. Keep the store defensive too.
return false, fmt.Errorf("parse heartbeat timestamp: %w", err)
}
snapshot := Snapshot{
DeviceID: message.DeviceID,
MessageID: message.MessageID,
ReportedAt: reportedAt,
ReceivedAt: receivedAt,
}
if message.Temperature != nil {
snapshot.Temperature = *message.Temperature
snapshot.TemperatureSet = true
}
s.mu.Lock()
defer s.mu.Unlock()
previous, exists := s.latest[message.DeviceID]
if exists && !reportedAt.After(previous.ReportedAt) {
return false, nil
}
s.latest[message.DeviceID] = snapshot
return true, nil
}
func (s *Store) GetStatus(deviceID string, now time.Time, freshFor time.Duration) (Status, bool) {
s.mu.RLock()
snapshot, exists := s.latest[deviceID]
s.mu.RUnlock()
if !exists {
return Status{}, false
}
fresh := freshFor > 0 && !now.Before(snapshot.ReceivedAt) && now.Sub(snapshot.ReceivedAt) <= freshFor
return Status{Snapshot: snapshot, HeartbeatFresh: fresh}, true
}
```
#### 🏗️ `NewStore` — initialize the state owner
**Summary:** Creates the map that holds one latest snapshot per device.
**Details:** The map starts empty, so a status query before the first valid heartbeat returns `exists == false`. The store does not invent an offline record for a device the backend has never seen.
#### 📥 `RecordHeartbeat` — apply a validated observation
**Summary:** Converts the contract timestamp, copies optional temperature data, and replaces the stored snapshot only when the report is newer.
**Details:** `Decode` has already checked the timestamp, but the store parses it again because its own method should not silently accept malformed values if another caller uses it later. The device's `OccurredAt` orders reports; `receivedAt` is captured by the backend and supports freshness queries. The mutex protects the map while concurrent workers update it. The temperature is stored by value, and `TemperatureSet` preserves the distinction between a missing reading and a reported zero. This also avoids sharing a mutable pointer with either the incoming message or a status caller.
If a duplicate or delayed older heartbeat arrives, the method returns `false` and leaves the newer state untouched. This is a small ordering rule, not durable deduplication: after restart the store has no history, and equal timestamps are treated as non-newer.
#### 🔎 `GetStatus` — query recent application evidence
**Summary:** Returns the latest snapshot and whether the backend received it within the caller's freshness window.
**Details:** `HeartbeatFresh` is computed from backend receipt time, not device time. It means “a valid heartbeat arrived recently,” not “the broker connection is up” or “the machine is operational.” A missing snapshot returns `false` as the second result. Since the snapshot contains only values, returning its map copy does not expose mutable store-owned data.
## 🧠 3. Add the application service
> **❓ Question:** **Where do topic parsing, payload validation, state updates, and operational logging meet?**
The service coordinates those steps. It does not own a Paho client or publish MQTT packets; it takes a transport-neutral message value and applies one use case. Save this complete file as `internal/deviceview/service.go`.
### 🔍 `service.go` — file overview
This file turns an incoming message into an application decision. Heartbeats update the device view; sales reports and command results are validated and logged until their later persistence/correlation lessons. Invalid topics or payloads return errors before the store changes. Structured logs include identifiers and outcomes, not the raw payload.
```go
package deviceview
import (
"context"
"fmt"
"log/slog"
"time"
"example.com/mqtt5-learning/internal/contracts"
"example.com/mqtt5-learning/internal/mqtt"
)
type Service struct {
store *Store
logger *slog.Logger
freshFor time.Duration
now func() time.Time
}
func NewService(store *Store, logger *slog.Logger, freshFor time.Duration) (*Service, error) {
if store == nil {
return nil, fmt.Errorf("device-view store is required")
}
if logger == nil {
return nil, fmt.Errorf("logger is required")
}
if freshFor <= 0 {
return nil, fmt.Errorf("heartbeat freshness window must be positive")
}
return &Service{store: store, logger: logger, freshFor: freshFor, now: time.Now}, nil
}
func (s *Service) HandleMessage(ctx context.Context, message mqtt.Message) error {
if err := ctx.Err(); err != nil {
return err
}
topic, err := contracts.ParseTopic(message.Topic)
if err != nil {
return fmt.Errorf("parse topic: %w", err)
}
receivedAt := s.now().UTC()
switch topic.Kind {
case contracts.KindTelemetry:
heartbeat, err := contracts.Decode[contracts.Heartbeat](message.Payload, topic, receivedAt)
if err != nil {
return fmt.Errorf("validate heartbeat: %w", err)
}
updated, err := s.store.RecordHeartbeat(heartbeat, receivedAt)
if err != nil {
return fmt.Errorf("record heartbeat: %w", err)
}
if !updated {
s.logger.Info("older or duplicate heartbeat ignored",
"device_id", heartbeat.DeviceID,
"message_id", heartbeat.MessageID,
)
return nil
}
s.logger.Info("heartbeat recorded",
"device_id", heartbeat.DeviceID,
"message_id", heartbeat.MessageID,
"received_at", receivedAt,
)
return nil
case contracts.KindSales:
report, err := contracts.Decode[contracts.SalesReport](message.Payload, topic, receivedAt)
if err != nil {
return fmt.Errorf("validate sales report: %w", err)
}
// Validation is not durable admission; Section 13 stores reports.
s.logger.Info("sales report validated; persistence not implemented",
"device_id", report.DeviceID,
"report_id", report.ReportID,
"item_count", len(report.Items),
)
return nil
case contracts.KindResponses:
result, err := contracts.Decode[contracts.CommandResult](message.Payload, topic, receivedAt)
if err != nil {
return fmt.Errorf("validate command result: %w", err)
}
// Section 7 adds pending-command and correlation matching.
s.logger.Info("command result validated; matching not implemented",
"device_id", result.DeviceID,
"command_id", result.CommandID,
"status", result.Status,
)
return nil
default:
return fmt.Errorf("device-view service does not handle topic kind %q", topic.Kind)
}
}
func (s *Service) Status(deviceID string) (Status, bool) {
return s.store.GetStatus(deviceID, s.now().UTC(), s.freshFor)
}
```
#### 🧰 `NewService` — check application dependencies
**Summary:** Builds a service only when its store, logger, and freshness policy are configured.
**Details:** These checks fail at startup instead of allowing a nil pointer panic when the first device message arrives. `freshFor` is an application policy; it is not MQTT Keep Alive or Session Expiry. Choose a value that accounts for the simulator's heartbeat interval and expected delay.
#### 🔄 `HandleMessage` — validate, route, apply, and record the result
**Summary:** Checks cancellation, parses the topic, then validates and routes each supported contract to its current application behavior.
**Details:** For telemetry, each early return happens before the store update, so malformed messages cannot alter status. The service captures one backend receipt time and reuses it for validation, storage, and logging. The store's `false` result represents an older or duplicate report; it is a handled condition, not a transport failure. For sales and responses, successful decoding is logged as validation only: no sale is saved and no command is matched yet. The logger records stable identifiers but avoids printing untrusted payload contents.
This is an application outcome. The MQTT receive callback may already have returned and automatic acknowledgement may already have occurred. Returning an error here does not reverse a `PUBACK`; Section 6 makes that boundary measurable under worker load.
#### 🔎 `Status` — expose the current device view to callers
**Summary:** Queries the store and evaluates heartbeat freshness using the service's clock and configured window.
**Details:** The method returns the latest snapshot only if one exists. It does not issue an MQTT request to the device. A future API can expose this method; Section 8 builds HTTP endpoints, while Sections 10 and 12 add richer presence and vending-health semantics.
## 🔌 4. Connect the service to the receive loop
> **❓ Question:** **What changes when validated messages now update application state instead of only producing a “valid message” log line?**
Create one store and service during process startup, then call the service for telemetry messages from the existing backend message switch:
Add these imports to `cmd/backend/main.go`:
```go
"log/slog"
"time"
"example.com/mqtt5-learning/internal/deviceview"
```
```go
// In run(), after config.Load and before the receive loop starts:
store := deviceview.NewStore()
deviceView, err := deviceview.NewService(store, slog.Default(), 90*time.Second)
if err != nil {
return fmt.Errorf("create device-view service: %w", err)
}
// Replace the old validation-only handler with this service call.
func handleMessage(ctx context.Context, deviceView *deviceview.Service, message mqtt.Message) error {
return deviceView.HandleMessage(ctx, message)
}
```
#### 🔌 `run` and `handleMessage` — connect startup to application work
**Summary:** `run` constructs one shared store/service at startup; `handleMessage` delegates each received message to that service.
**Details:** Create the service after configuration loads and before the receive loop starts. Replace the validation-only handler with the shown `handleMessage` function and update the receive loop to pass its context and shared service. The service updates heartbeat state and validates/logs sales and responses for now; Section 7 adds response matching and Section 13 adds sales persistence.
Use this receive-loop call and function signature to make the required data flow explicit:
```go
case message := <-inbox:
if err := handleMessage(ctx, deviceView, message); err != nil {
slog.Error("message handling failed", "topic", message.Topic, "error", err)
}
```
```go
func handleMessage(ctx context.Context, deviceView *deviceview.Service, message mqtt.Message) error {
return deviceView.HandleMessage(ctx, message)
}
```
### 🧪 Observe the application result
1. Start the lab backend and simulator. Confirm structured `heartbeat recorded` events include the device and message IDs but no raw JSON payload.
2. Query `deviceView.Status("device-001")` from a temporary diagnostic call. After a heartbeat, expect a snapshot and `HeartbeatFresh == true`:
```go
status, exists := deviceView.Status("device-001")
if exists {
slog.Info("device view", "last_seen", status.Snapshot.ReceivedAt, "heartbeat_fresh", status.HeartbeatFresh)
}
```
3. Stop the simulator, wait longer than the 90-second example window, and query again. The snapshot remains, but freshness becomes false. No MQTT packet declared the device offline; the backend inferred staleness from elapsed time.
4. Deliver an older `occurred_at` after a newer heartbeat. The service logs that it ignored the older observation, and the stored snapshot does not move backward.
5. Change the topic/body device IDs or corrupt the payload. Validation returns an error and the stored snapshot remains unchanged.
Do not confuse backend logs with device diagnostic logs. These `slog` entries explain what the backend did. A device log stream needs its own contract, size/frequency limits, and storage policy; the later vending-workflow lesson covers that. Likewise, a fresh heartbeat proves recent application traffic, not machine health or a live MQTT socket.
### ✅ Result — what the first five lessons were missing
You now have the missing application step between “valid packet” and “concurrent worker”: a service applies a specific use case to a validated message, a store exposes the resulting view, and a structured log records the decision. The status is deliberately simple and volatile. Section 6 adds bounded concurrent handling; later lessons add durable records, richer health/log behavior, and real vending workflows.
**Primary references:** [MQTT 5.0 OASIS Standard](https://docs.oasis-open.org/mqtt/mqtt/v5.0/mqtt-v5.0.html) for the distinction between protocol delivery and application payload handling; [Go `log/slog`](https://pkg.go.dev/log/slog) for structured logging; [Paho Go v0.23.0 API](https://pkg.go.dev/github.com/eclipse/paho.golang/paho@v0.23.0) for the transport API used by the surrounding lesson.
Previous: [05-contracts-validation.md](05-contracts-validation.md). Next: [06-concurrency-cancellation.md](06-concurrency-cancellation.md).
teaching/course-source/06-concurrency-cancellation.md
+465
View File
@@ -0,0 +1,465 @@
# 🧵 Section 6 — Manage callbacks, concurrency, and cancellation
## 🎬 Start with the system story
> **❓ Question:** **A burst of broker-forwarded messages arrives while the backend is doing slow work. Who is receiving packets, who is doing business work, and what happens when the application cannot keep up?**
Paho's network/client machinery receives packets and invokes configured handlers. In this lesson, a **callback** means the application function that Paho calls when a matching incoming PUBLISH reaches the client. The callback runs as part of Paho's receive path; it is not the broker, and it is not a background worker. Your application decides how work is scheduled, bounded, cancelled, and acknowledged. If a callback blocks or a queue fills, that can affect unrelated deliveries; if a message is acknowledged before durable handling and then dropped from RAM, MQTT cannot recover that application copy by itself.
```mermaid
flowchart LR
B[Broker] -->|PUBLISH| P[Paho receive path]
P --> C[Short callback copies/enqueues]
C --> Q[Bounded application queue]
Q --> W[Worker handles and persists]
Q -->|full| O[Explicit overload policy]
W --> A[Application outcome]
```
**Before each implementation step, ask:** What work must happen before the MQTT acknowledgement, what can happen afterward, and what is the defined behavior when capacity is exhausted? Follow [Lesson 0](00-system-lifecycle.md) for the broker-to-callback-to-application path.
Prerequisite: Section 5 validates messages, and Section 5A applies them through an application service to an in-memory device view. This lesson moves that same service behind workers. The backend and simulator continue to read their shared disposable password from `configs/lab.toml`; select each broker username with `-username` when starting the process.
Run all commands from `/home/ryan/Media/Code/Code/robot-market/mqtt-5-learning`. Save the complete source blocks at the project-relative paths shown below. This lesson includes the full dispatcher implementation; the exercises ask you to run it and inspect its admission, overload, and shutdown behavior.
## ✅ Understand the acknowledgement boundary before adding workers
### 📥 The publisher sends a message to the broker
For QoS 1, the publisher sends `PUBLISH` to the broker. The broker sends `PUBACK` to the publisher to complete that delivery hop. This says nothing about whether subscribers received or processed the message.
### 📬 The broker delivers to the backend
When the backend is a subscriber, the broker sends it a separate `PUBLISH`. Paho receives the packet and calls the registered callback. In automatic acknowledgement mode, Paho sends `PUBACK` to the broker after the callback returns.
If the callback enqueues the message and returns, the backend can acknowledge it before a worker processes it. That worker may later reject the message or the process may crash before handling it.
### ⚠️ What if the callback returns an error?
| Paho mode | What Paho does after the callback returns an error | Does the error request redelivery? |
| --- | --- | --- |
| Automatic acknowledgement | Sends `PUBACK` for QoS 1 after the callback returns | No. The error is local; it does not ask the broker to resend. |
| Manual acknowledgement | Does not automatically send `PUBACK` | No. Your code must call `Ack` when its policy says to acknowledge. |
In manual mode, leaving the message unacknowledged keeps that delivery in flight and can slow later QoS 1/2 deliveries. It does not itself guarantee a retry on the same connection. The broker may redeliver after a disconnect only if session, QoS, expiry, and broker state allow recovery.
For QoS 0, there is no `PUBACK`. QoS 2 has a multi-packet handshake, but completing that handshake still does not prove the subscriber's business action succeeded.
**❓ Question:** **The broker delivers a sales report with quantity `-3`. Can MQTT acknowledge it even if a worker later rejects it?**
**💬 Answer:** Yes. With automatic acknowledgement, Paho can send `PUBACK` after the callback returns, before the worker checks the report. The worker may then reject the sale. Record that application result separately; `PUBACK` does not mean the sale was accepted.
Trace the early QoS 1 implementation:
```text
publisher --QoS 1--> broker --QoS 1--> this subscriber's client
└-------> other subscriber clients independently
this client receives PUBLISH → callback copies into RAM queue → callback returns → sends PUBACK
└→ worker validates/applies effect later
```
The worker may finish before or after the ACK. If the process crashes after the ACK but before the effect, the broker can consider its delivery complete while the application loses the work. Increasing channel capacity increases the possible backlog; it does not close that gap.
Read the arrows as two independent delivery hops. The publisher's QoS 1 exchange ends when the broker acknowledges that publisher. The broker then starts a separate delivery to this subscriber, and other subscribers receive their own independent deliveries. The callback's copy puts bytes into process memory; returning from that callback is the event that permits automatic acknowledgement in this example. The worker's later validation and side effect belong to the application lifecycle, not to the completed MQTT exchange. This is why queue admission and business completion need separate evidence and, for important work, separate durable records.
In manual mode, `EnableManualAcknowledgment` means Paho does not automatically acknowledge a QoS 1 message when the callback returns. Your application must call `received.Client.Ack(received.Packet)` to release it. Returning an error alone does not acknowledge or reject it, and does not request a retry. Unless your code calls `Ack`, that delivery stays unacknowledged on the connection. This can occupy an in-flight slot and slow later QoS 1/2 deliveries. For important work, the intended sequence is receive → validate → commit durable admission → call `Ack`. Preserve the receiving client and original packet for this operation; a payload-only copy is insufficient. Check Ack's error and study the pinned source before implementing it. Do not reuse this QoS 1 explanation as a claim that every QoS 2 handshake stage waits for your database.
The client's tracker sends acknowledgements in receipt order. If work 3 finishes before 1, acknowledging 3 can remain buffered behind 1. Slow work therefore consumes inflight capacity even with several workers. Manual acknowledgement delays protocol completion; it does not persist your Go queue, remove duplicate effects, or automatically schedule retries while the same connection stays open.
The practical choice is about the promise made to the broker. Automatic acknowledgement is simple, but returning after an in-memory enqueue accepts a possible loss window. Manual acknowledgement lets the application delay completion until its chosen checkpoint, but the checkpoint should be durable if a process crash must not lose work. Because acknowledgements are tracked in receipt order, finishing jobs concurrently does not guarantee that later packets can be acknowledged ahead of an earlier one. Keep that protocol limit in mind when choosing worker parallelism and Receive Maximum.
### ↩️ Choose what the callback promises
| Message class | Reasonable admission choice | Meaning of a successful admission |
| --- | --- | --- |
| Disposable heartbeat | Bounded RAM queue, counted drops | A recent sample may be processed while this process survives |
| Important report | Durable inbox before acknowledgement, introduced in Section 13 | Work can be recovered after this process restarts |
| Command | Validate identity/deadline and record dedup/admission state | Device acceptance still needs an explicit application result |
For this in-memory lesson, keep the loss limitation visible. Use a fake acknowledgement recorder to test ordering of admission and acknowledgement without pretending that the fake verifies broker behavior.
Use the table as a policy map, not as a claim that all message types need the same storage. A heartbeat is usually useful as a recent sample, so bounded memory plus a drop counter can be an acceptable contract. A report that drives accounting or inventory needs durable admission before the broker delivery is acknowledged; Section 13 builds that inbox. A command needs identity, expiry/deadline, and duplicate handling before the backend can claim it was accepted. Even then, admission is not device acceptance: the device must return an application-level result. A fake recorder can verify that code calls its acknowledgement hook only after the selected admission step; broker/session behavior still needs an integration experiment.
### 🧪 Predict the overload
Two workers taking 100 ms per item can complete about 20 items/second before other overhead. A burst of 100 items arriving much faster cannot fit into two busy workers plus 16 waiting slots. Predict rejection, then measure actual accepted/completed/rejected counts; scheduler timing affects the exact split. Increasing workers may reorder effects for the same device. If revision order matters, serialize that device's work or enforce revision checks.
The estimate follows from service time: each worker can process about ten 100 ms jobs per second, so two workers provide roughly twenty per second when work is uniformly sized. With both workers busy and a queue capacity of 16, at most 18 jobs can be active or waiting at once; additional arrivals encounter `Full` until a worker frees capacity. A fast finite burst may therefore produce a different accepted count depending on scheduling and exactly when arrivals occur. The counters tell you what happened in that run. More workers increase throughput but also allow operations for one device to overlap; use per-device serialization or revision checks when order is part of the data contract.
Shutdown also has a dependency: if draining work must publish results, keep its MQTT connection alive while draining. Stop new admission first, drain accepted work with a deadline, then disconnect transport and join workers. If you disconnect first, record unsent results for recovery; an in-memory-only lesson cannot promise to deliver them. The Section 4 baseline uses one signal context for AutoPaho; the complete Section 6 integration below separates signal and transport lifetimes so it can drain first.
Treat shutdown as an ordering protocol among producers, queues, workers, and transport. First stop the receive loop from submitting new jobs; then close pool admission. Give workers a bounded interval to finish while MQTT remains available for replies. Only after the drain should that transport be disconnected. If the deadline expires, cancellation asks cooperative handlers to stop, but unfinished in-memory jobs may be lost. Keep signal, drain, and transport lifetimes separate.
<details>
<summary><strong>❓ Question: What does a callback error do to QoS 1 acknowledgement?</strong></summary>
**💬 Answer:**
- **Automatic mode:** Paho sends `PUBACK` after the callback returns, even when it returns an error. The error does not request redelivery.
- **Manual mode:** Paho does not send `PUBACK` just because the callback returns. Your application must call `Ack`; otherwise the delivery remains in flight.
In neither mode does returning an error tell the broker to reject or retry the message.
</details>
## ↩️ 1. Trace callback and acknowledgement order
> **❓ Question:** **Does returning from the receive callback mean the business operation is durable, or only that the MQTT client may acknowledge delivery?**
Read the pinned source and write the observed sequence down before choosing a queue or acknowledgement policy.
Read `references/paho.golang-v0.23.0/paho/client.go`, especially receive callback processing, `EnableManualAcknowledgment`, and `Ack`. Write the observed order in `journal/06-concurrency.md`. Inspect this pinned source before choosing an acknowledgement policy.
Section 4 copies payloads into a bounded channel and drops messages when full. With automatic acknowledgement, Paho can send PUBACK for this client's delivery after the callback returns—even if the application later discovers that the queue was full. This is a deliberate loss window, not reliable ingestion.
For this source-reading step, follow the packet from the receive handler to callback invocation and then to the acknowledgement decision. Record which condition enables manual mode, whether callback errors change the automatic-ack path, and what `Ack` needs to identify the original delivery. These observations explain why a channel send is an application decision rather than an MQTT acknowledgement. The written notes become a reference for the implementation choices in the next section; do not infer behavior from a method name alone when the pinned client source is available.
## 🛠️ 2. Build a bounded dispatcher
> **❓ Question:** **When messages arrive faster than handlers can finish, where is excess work stored and how does the application report that capacity is exhausted?**
Set a finite capacity and make admission outcomes explicit for each message class.
### 🔍 `pool.go` — file overview
This file implements one bounded job queue and a fixed set of workers. Callers submit jobs without waiting for free capacity, inspect admission and processing counters, then stop submissions and drain work during shutdown. The pool limits waiting jobs and simultaneous jobs; it does not save them to disk.
Save this complete file as `internal/dispatch/pool.go`.
```go
package dispatch
import (
"context"
"errors"
"sync"
"sync/atomic"
)
type Outcome uint8
// Outcomes make queue admission success and overload visible to callers.
const (
Accepted Outcome = iota
Full
Stopping
)
type Job func(context.Context) error
type Stats struct {
Accepted uint64
Rejected uint64
Completed uint64
Failed uint64
}
// Pool owns a bounded queue, worker lifecycle, and admission counters.
type Pool struct {
mu sync.Mutex
accepting bool
jobs chan Job
ctx context.Context
cancel context.CancelFunc
workers sync.WaitGroup
done chan struct{}
accepted atomic.Uint64
rejected atomic.Uint64
completed atomic.Uint64
failed atomic.Uint64
}
// New starts a fixed worker set and ties job cancellation to parent.
func New(parent context.Context, workers, capacity int) *Pool {
if workers < 1 { workers = 1 }
if capacity < 0 { capacity = 0 }
ctx, cancel := context.WithCancel(parent)
p := &Pool{accepting: true, jobs: make(chan Job, capacity), ctx: ctx, cancel: cancel, done: make(chan struct{})}
p.workers.Add(workers)
for i := 0; i < workers; i++ { go p.worker() }
go func() { p.workers.Wait(); close(p.done) }()
return p
}
// Submit never waits for queue capacity. The caller gets an explicit Full result.
// The caller context can cancel admission before it enters the queue.
func (p *Pool) Submit(ctx context.Context, job Job) (Outcome, error) {
if job == nil { return Stopping, errors.New("nil job") }
if err := ctx.Err(); err != nil { return Stopping, err }
p.mu.Lock()
defer p.mu.Unlock()
if !p.accepting { return Stopping, nil }
select {
case p.jobs <- job:
p.accepted.Add(1)
return Accepted, nil
default:
p.rejected.Add(1)
return Full, nil
}
}
func (p *Pool) StopAdmission() {
p.mu.Lock()
defer p.mu.Unlock()
if p.accepting {
p.accepting = false
close(p.jobs)
}
}
// Shutdown stops new submissions and drains queued work until ctx expires.
// Jobs must observe their context so deadline cancellation can stop slow work.
func (p *Pool) Shutdown(ctx context.Context) error {
p.StopAdmission()
select {
case <-p.done:
p.cancel()
return nil
case <-ctx.Done():
p.cancel()
return ctx.Err()
}
}
func (p *Pool) Stats() Stats {
return Stats{Accepted: p.accepted.Load(), Rejected: p.rejected.Load(), Completed: p.completed.Load(), Failed: p.failed.Load()}
}
// Workers execute admitted jobs and count their final result.
func (p *Pool) worker() {
defer p.workers.Done()
for {
select {
case <-p.ctx.Done():
return
case job, ok := <-p.jobs:
if !ok { return }
if err := job(p.ctx); err != nil { p.failed.Add(1); continue }
p.completed.Add(1)
}
}
}
```
#### 🧩 Types at a glance
- `Outcome` reports whether a job was `Accepted`, rejected as `Full`, or refused because the pool is `Stopping`.
- `Job` is the function a worker runs. Its context lets the job observe pool cancellation.
- `Stats` reports admission counts separately from worker results: accepted, rejected, completed, and failed.
- `Pool` owns the queue, workers, lifecycle context, synchronization, and counters.
#### 🚀 `New` — start the pool
**Summary:** Creates the bounded queue and starts the requested number of workers.
**Details:** Worker counts below one become one; negative capacity becomes zero. The derived context connects parent cancellation to jobs. `done` closes when every worker exits, allowing shutdown to wait for the whole worker set. A zero-capacity channel admits a job only when a worker can receive it immediately; a positive capacity holds that many waiting jobs.
#### 📥 `Submit` — attempt immediate admission
**Summary:** Validates the request and either enqueues the job immediately or returns a reason it was not admitted.
**Details:** A nil job or already-cancelled caller context returns an error. Under the mutex, the method rejects submissions after shutdown begins. The `select` with a `default` case never waits for queue space: a successful send increments `Accepted`; no available slot increments `Rejected` and returns `Full`. The caller context is checked only before admission; the job itself receives the pool context when a worker runs it.
#### 🛑 `StopAdmission` — close submissions safely
**Summary:** Prevents future submissions and closes the job channel once.
**Details:** The mutex is shared with `Submit`, so the channel cannot be closed between the submitter's state check and send. Workers can still read already-queued jobs after closure. Stop message producers before calling this method so no producer continues trying to submit.
#### ⏳ `Shutdown` — drain until a deadline
**Summary:** Stops admission, waits for workers, and cancels jobs if the deadline expires.
**Details:** When `done` closes first, all workers have exited and shutdown returns nil. If the caller context expires first, the pool cancels the job context and returns that context error. Cancellation is cooperative: a job must check its context or use context-aware I/O. On timeout, workers may abandon queued jobs when they observe cancellation, so this RAM-only queue cannot promise those jobs completed.
**What does `ctx.Err()` contain during shutdown?** The `ctx` passed to `Pool.Shutdown` is the drain context. If its deadline expires before the workers finish, `ctx.Err()` returns `context.DeadlineExceeded`; if it is canceled for another reason, it returns `context.Canceled`. If all workers finish before cancellation, `Pool.Shutdown` returns `nil` instead of `ctx.Err()`. The pool uses this error as the shutdown result and cancels the worker job context when the wait times out.
#### 📊 `Stats` — read counters
**Summary:** Returns a concurrency-safe snapshot of the pool's four counters.
**Details:** Atomic loads allow worker updates and reads to happen concurrently. `Accepted` is queue admission, not successful processing; `Completed` and `Failed` are outcomes after a worker runs a job.
#### 👷 `worker` — execute jobs
**Summary:** Repeatedly receives jobs and records each job's result.
**Details:** Each worker exits when the pool context is cancelled or the closed queue is drained. A job error increments `Failed`, then that worker continues to the next job. A nil error increments `Completed`. The worker count therefore caps parallel job execution.
### 🔍 `receiver.go` — file overview
This file routes each validated topic to the right pool and transfers message ownership safely to a worker. Commands and sales use the important-work pool; telemetry uses its own pool so a telemetry burst cannot fill the important queue.
#### 🧩 `Processor` and `Receiver` — declare the route
**Summary:** `Processor` is the business function; `Receiver` holds it and the two pools.
**Details:** The processor receives the worker context and message. The context supports cooperative shutdown. Separate pool fields make the capacity policy visible at the receiver boundary.
#### 📬 `Receiver.Submit` — validate, route, and enqueue
**Summary:** Checks configuration and topic, picks a pool, copies payload bytes, then delegates admission to `Pool.Submit`.
**Details:** A missing processor, invalid topic, or missing pool returns an error before worker execution. Commands and sales select `Important`; other recognized kinds select `Telemetry`. The payload copy prevents the worker from reading bytes that the receive path may reuse or mutate after the method returns. If the message type later adds other mutable slices, maps, or pointers, those need an ownership decision too. The closure passes the pool's worker context to the processor.
### 🔌 Process wiring — connect receive and worker paths
**Summary:** Creates isolated pools, installs the application processor, then submits each received message without doing slow work in the receive loop.
**Details:** `dispatch.New(context.Background(), 2, 16)` configures two workers and sixteen waiting slots for each traffic class. The independent parent context keeps a signal from cancelling workers before the drain; `Pool.Shutdown` cancels jobs after the drain deadline. The processor closure calls the backend handler; in the simulator it calls `respond(...)`. The `case` receives from `inbox`, calls `receiver.Submit` with the signal context, logs errors, and handles non-`Accepted` outcomes. `Full` is an application admission rejection; logging it does not undo an automatic QoS 1 acknowledgement. A command or sales rejection needs an explicit application policy, and durable admission is required if a process crash must not lose accepted jobs.
Save this complete router as `internal/dispatch/receiver.go`. It sends commands, command responses, and sales to the important-work pool, telemetry to its own pool, and rejects unsupported topic kinds before a worker sees them.
```go
package dispatch
import (
"context"
"fmt"
"example.com/mqtt5-learning/internal/contracts"
"example.com/mqtt5-learning/internal/mqtt"
)
type Processor func(context.Context, mqtt.Message) error
// Receiver validates the route, selects a pool, and copies payload ownership.
type Receiver struct {
Telemetry *Pool
Important *Pool
Process Processor
}
func (r *Receiver) Submit(ctx context.Context, message mqtt.Message) (Outcome, error) {
if r.Process == nil { return Stopping, fmt.Errorf("message processor is required") }
topic, err := contracts.ParseTopic(message.Topic)
if err != nil { return Stopping, err }
pool := r.Telemetry
if topic.Kind == contracts.KindCommands || topic.Kind == contracts.KindResponses || topic.Kind == contracts.KindSales { pool = r.Important }
if pool == nil { return Stopping, fmt.Errorf("no queue configured for topic kind %q", topic.Kind) }
copyOfMessage := message
copyOfMessage.Payload = append([]byte(nil), message.Payload...)
return pool.Submit(ctx, func(jobCtx context.Context) error { return r.Process(jobCtx, copyOfMessage) })
}
```
Wire the receiver into each process after creating the pools. In the backend, use the Section 5A service as the processor. In the simulator, use its command-response handler as the processor. The receive loop below keeps the callback path short and reports full queues explicitly:
```go
// Backend: keep telemetry and important work from consuming each other's queue slots.
telemetryPool := dispatch.New(context.Background(), 2, 16)
importantPool := dispatch.New(context.Background(), 2, 16)
receiver := &dispatch.Receiver{Telemetry: telemetryPool, Important: importantPool, Process: deviceView.HandleMessage}
case message := <-inbox:
// Report overload locally; an MQTT acknowledgement cannot undo this loss.
outcome, err := receiver.Submit(ctx, message)
if err != nil { log.Printf("message admission error: %v", err); continue }
if outcome != dispatch.Accepted { log.Printf("message not admitted: outcome=%d topic=%s", outcome, message.Topic) }
```
In the simulator process, construct the same receiver with a processor that runs the existing command handler using the worker pool's context:
```go
var resultSequence atomic.Uint64
receiver := &dispatch.Receiver{
Telemetry: telemetryPool,
Important: importantPool,
Process: func(jobCtx context.Context, message mqtt.Message) error {
resultID := fmt.Sprintf("result-%d-%d", runID, resultSequence.Add(1))
return respond(jobCtx, manager, cfg.MQTT.DeviceID, resultID, message)
},
}
```
The simulator subscribes to command topics, which the receiver routes to the important-work pool. The backend routes telemetry, sales, and responses through the Section 5A service until later lessons replace its temporary sales/response behavior.
Choose a policy per class: disposable telemetry may be dropped with a counter; commands and sales must produce an explicit admission failure rather than silently disappear. Admission failure under automatic MQTT acknowledgement still loses application work. Record this limitation until durable admission is introduced in Section 13. Do not claim manual acknowledgement alone makes an in-memory queue durable.
When adapting this wiring, keep the pool limits near process startup so they are visible and configurable. Check the returned error before interpreting the outcome: parse/configuration failures are errors, while queue-full and stopped states are outcomes. Decide and record the policy for each message class; the in-memory/automatic-ACK setup cannot guarantee that a rejected command or sales report will be retried.
## 🛑 3. Define ownership and shutdown
> **❓ Question:** **If a worker is still processing when shutdown begins, which components must stay alive until the work finishes or its deadline expires?**
Stop admission before disconnecting MQTT, then let workers finish queued jobs under one shared shutdown deadline. Save this helper as `internal/dispatch/shutdown.go`:
In each complete process listing, the signal case logs and returns from `run`; it does not call the drain itself. The deferred `dispatch.Shutdown` then runs as `run` returns. Since that defer is registered after `defer mqtt.Stop(manager)`, Go's last-in-first-out defer order drains the workers before disconnecting MQTT.
### 📄 `shutdown.go` — file overview
This file coordinates shutdown across all dispatch pools. It stops every pool from accepting work before it waits for any pool, so one queue cannot continue growing while another drains.
```go
package dispatch
import (
"context"
"errors"
)
func Shutdown(ctx context.Context, pools ...*Pool) error {
// Close admission on every pool before waiting for any worker to drain.
for _, pool := range pools { if pool != nil { pool.StopAdmission() } }
var result error
// Share one deadline across the drain so total shutdown time stays bounded.
for _, pool := range pools {
if pool != nil { result = errors.Join(result, pool.Shutdown(ctx)) }
}
return result
}
```
#### ⏳ `Shutdown` — stop and drain all pools
**Summary:** Closes admission on each non-nil pool, then waits for all of them using the same deadline.
**Details:** The first loop calls `StopAdmission` on every pool before the second loop waits. This prevents a later pool from continuing to accept while an earlier one drains. Each pool receives the same context, so the caller's deadline applies to the whole operation rather than restarting per pool. `errors.Join` returns failures from all pool shutdowns, and nil entries are skipped. Stop producers before calling this helper; keep MQTT connected through the drain if workers need it to publish results.
Copy retained byte slices, including future correlation metadata. Protect shared maps with a mutex or give one goroutine ownership. Never hold a state mutex while publishing over the network. Use operation deadlines derived from the application lifecycle.
🧠 **Why these rules matter:** A slice refers to mutable backing storage, so copy bytes that must outlive the callback. Shared maps need a mutex or a single goroutine owner. A network publish can block, so do not hold a state mutex while publishing. Derive operation deadlines from the application lifecycle so workers cannot wait forever on network or storage calls during shutdown.
On shutdown, stop admitting new work and let accepted work finish within a separate bounded drain context. Keep MQTT available during the drain if workers must publish results, then stop the transport producers. Cancel remaining work when the drain deadline expires. Never close a queue while a producer can still send; waiting for worker completion is part of shutdown. Document what happens to unfinished in-memory work.
## 🔗 4. Integrate and stress it
> **❓ Question:** **What observations would show that overload is bounded and shutdown is controlled, rather than merely that the process eventually exits?**
Measure accepted, rejected, completed work, queue capacity, and shutdown time under a finite burst.
Use the receiver wiring above in both processes, with `deviceView.HandleMessage` for the backend and a processor that calls the simulator's `respond(...)` for commands. Keep transport callbacks short, and send messages to the appropriate pool. A finite burst generator should stop after its configured count; do not create one goroutine per incoming message. Record accepted, rejected, completed, and failed counts with the configured worker count and queue capacity.
Apply the same dispatcher boundaries in both programs, but connect each to its own business behavior: the backend's handler and the simulator's ping/pong responder. The finite burst should have a known message count and stop on its own, making accepted/rejected results comparable across runs. Creating a goroutine for every publish would move the unbounded backlog from the channel into goroutine stacks, defeating the pool's memory bound. Include worker count, queue capacity, artificial processing delay, and the four counters in the run record so another person can interpret throughput and overload.
```sh
mkdir -p internal/dispatch journal
gofmt -w internal/dispatch cmd
go test -race ./...
go vet ./...
```
Create `internal/dispatch` and `journal` before saving their files; the `mkdir -p` command is safe to run earlier than the final check block. `go test -race ./...` instruments the test binaries to report unsynchronized memory access during executed tests. It is slower than normal testing and cannot prove unexecuted paths are race-free. `go vet` checks static patterns; it does not replace the race run.
🧰 **What each command does:** `mkdir -p` creates the source and journal directories if needed. `gofmt` formats the Go implementation and command packages. `go test -race ./...` runs the repository tests with race detection for executed paths. `go vet ./...` checks for suspicious Go patterns. Race detection and vet provide different evidence; neither proves paths that were not exercised.
A finite burst generator publishes a configured count and then stops; record its rate, queue capacity, worker count, and artificial delay. A bounded queue has fixed capacity: when workers are slower than arrival, overload becomes measurable instead of consuming memory indefinitely. Worker cancellation is cooperative: handlers must check contexts and use deadlines on blocking operations. Observe accepted/rejected counts and total shutdown time, not just whether the process eventually exits.
The measurements answer different questions: accepted and rejected show admission behavior, completed and failed show worker outcomes, and elapsed shutdown time shows whether the drain respected its bound. A finite generator makes the offered load known; the artificial delay makes the service rate slow enough to trigger pressure. The queue prevents waiting jobs from growing without limit, but it cannot stop one handler from blocking forever unless that handler checks cancellation or uses deadlines for its own network/database calls. Process exit alone is weak evidence because forced termination can discard queued work without reporting it.
To inspect overload, temporarily make a worker slow and publish a finite burst larger than the queue plus active workers. Confirm the measured `Full` outcomes, then reduce the burst and confirm normal handling resumes. Observe shutdown time under the deadline; do not infer that queued work completed solely because the process exited.
The slow-worker run should push arrivals beyond the pool's total immediate capacity: active worker slots plus waiting queue slots. `Full` outcomes confirm that backpressure is bounded and visible. A smaller burst should fit under available capacity or be drained quickly enough to avoid rejection, showing that the system recovers after pressure rather than remaining stuck. During shutdown, compare elapsed time to the configured deadline and inspect completed/failed counts; jobs still queued or cancelled at expiry were not proven complete. Then compare that application evidence with the MQTT acknowledgement trace from the first section.
**Done when:** A burst causes observable bounded overload, normal ping/pong recovers afterward, shutdown finishes by its deadline, and the race detector passes. Explain the difference between broker acknowledgement, queue admission, and completed application work.
## ✅ Result — the story so far
The receive callback now has a defined small job: copy and admit work into bounded queues, then return. Workers validate and handle admitted messages, while counters make overload visible and shutdown follows a deliberate stop-admission, drain, and disconnect order. In this lesson's in-memory design, automatic MQTT acknowledgement can happen before worker completion, so a process crash can still lose queued work; durable admission is a later step.
Previous: [05A — Backend message processing](05a-backend-message-processing.md). Next: [07-command-response.md](07-command-response.md).
teaching/course-source/07-command-response.md
+752
View File
@@ -0,0 +1,752 @@
# 🔁 Section 7 — Build reliable command and response behavior
## 🎬 Start with the system story
> **❓ Question:** **The broker acknowledged a backend's command publication. How can the backend match a later device response to the right request—and what should it report if no response arrives?**
The MQTT acknowledgement covers delivery on one hop. The device's application response is a separate publication. The backend therefore needs application-level command identity, correlation, a pending state registered before publication, and a timeout policy that does not claim the device definitely did nothing.
```mermaid
sequenceDiagram
participant H as HTTP/backend caller
participant G as Go command service
participant B as Broker
participant D as Device
H->>G: Request command
G->>G: Create command ID and pending waiter
G->>B: PUBLISH command with correlation data
B-->>G: PUBACK: broker hop accepted
B->>D: PUBLISH command
D->>D: Validate, deduplicate, act
D->>B: PUBLISH correlated application result
B->>G: PUBLISH result
G->>G: Match identity/correlation; update state
G-->>H: Accepted / completed / rejected / unknown
```
**Before each implementation step, ask:** Which event changes command state: broker acknowledgement, device acceptance, device completion, or timeout? Keep those events separate. Follow [Lesson 0](00-system-lifecycle.md) for the full client and broker lifecycle.
Prerequisite: Section 5 validates messages, Section 5A turns a heartbeat into an application status view, and Section 6 bounds concurrent processing. The backend and simulator continue to read their shared disposable password from `configs/lab.toml`; select each broker username with `-username` when starting the process.
**Scope:** This section uses an in-memory command store. The record is registered before publication to prevent a fast-response race, but process restart loses pending and completed command records. An HTTP request is not introduced until Section 8 and is not durable acceptance until Sections 13–14 add database state and recoverable publication intent.
Run all commands from `/home/ryan/Media/Code/Code/robot-market/mqtt-5-learning`. Save the complete implementation blocks at the project-relative paths shown below. This lesson provides the command-state and simulator-deduplication code; the exercises ask you to trace its evidence and failure behavior.
## ↩️ MQTT request/response is an application conversation
Response Topic tells the responder where a result should be published. Correlation Data lets the requester recognize which pending request it belongs to. The broker forwards these properties; it does not run the command, create a result, or enforce your timeout.
This first command is intentionally a harmless `ping`: it teaches how to track request identity, broker acknowledgement, device result, timeout, and duplicate delivery. It does not change a vending machine's catalog or configuration. Sections 10 and 12 build desired state and versioned vending updates on top of this command lifecycle.
This fragment is the packet-construction portion of a service method, not a standalone program. `commandJSON` is an encoded validated command and `correlation` is owned by its pending record:
```go
commandTopic, err := contracts.DeviceTopic("device-001", contracts.KindCommands)
if err != nil { return err }
responseTopic, err := contracts.DeviceTopic("device-001", contracts.KindResponses)
if err != nil { return err }
// MQTT 5 carries the reply destination and request identity in packet properties.
request := &paho.Publish{
Topic: commandTopic.String(),
QoS: 1,
Retain: false,
Payload: commandJSON,
Properties: &paho.PublishProperties{
ContentType: "application/json",
ResponseTopic: responseTopic.String(),
CorrelationData: append([]byte(nil), correlation...),
},
}
// Send this packet only after the pending request has been registered.
```
Pass `request` to the bounded publication operation after registering pending state. The simulator reads `ResponseTopic` to find where to publish its result and copies Correlation Data unchanged into the result's properties. Correlation bytes are opaque; they need not be text.
**❓ Question:** **A command for device-001 asks the simulator to reply on `lab/v1/devices/device-002/responses`. What could go wrong if it publishes there?**
**💬 Answer:** The reply could appear in device-002's message stream. The simulator checks that `ResponseTopic` exactly matches its permitted topic, `lab/v1/devices/device-001/responses`. If it does not match, the simulator rejects the command and publishes no reply. Correlation Data matches a result to a request; it does not authorize the destination.
### ↩️ Trace a fast response and a lost response
1. Backend establishes its response subscription and checks SUBACK.
2. Backend stores the expected device, command ID, correlation, and deadline.
3. Backend publishes. The simulator can respond before the publish call returns.
4. Backend validates and applies device evidence independently of the broker acknowledgement.
5. If the waiting deadline expires, the backend records unknown outcome. In this reference store, a later validated result with matching device and correlation can resolve `unknown`; `completed` and `rejected` remain terminal.
Registering after step 3 loses the race with a fast response. Holding the command-map mutex across publishing can block the callback needed to record that response. A context timeout stops a local wait; it cannot recall bytes already delivered to a device.
| Identifier | Owner and lifetime | Why it cannot replace the others |
| --- | --- | --- |
| MQTT packet ID | Library, per protocol exchange/session direction, reusable | Identifies a transport handshake, not a permanent command |
| Correlation Data | Request/response mechanism | Matches a response; does not authenticate it or prevent repeated effects |
| Command ID | Application, stable across retries | Keys the dedup record and business status |
### 🧭 Define the application state model
The intended lifecycle is `created → pending delivery → accepted → completed/rejected`; a response deadline moves unresolved work to `unknown`. A valid late device result may resolve `unknown` in this reference implementation, while `completed` and `rejected` remain terminal. `BrokerAcked` is an independent transport fact and does not advance that lifecycle.
The example inserts a record directly as `Pending` before calling PUBLISH, so the moment of creation is not a separately queryable status. This keeps the in-memory teaching implementation small while preserving the important transition and failure boundaries. The exercises should state this modeling choice and its limits.
### ♻️ Make duplicate behavior observable
Use a synthetic effect counter. Send command C twice with the same content, then C with changed content. Expect one effect, a replayed recorded result, then a conflict. Also send the same C in two separate publish calls: application duplicates need not carry MQTT DUP. The DUP flag describes protocol retransmission, so deduplicating only packets with DUP set is wrong.
Concurrent duplicates need atomic admission: a lookup followed by an unlocked effect allows two workers to both observe "not seen". Reserve the ID with its fingerprint before executing, and define what a second request sees while the first is in progress. A real effect followed by a crash before storing its result remains a separate failure window; the in-memory simulator only demonstrates the idea.
<details>
<summary><strong>❓ Question: The broker acknowledged, but no device result arrived. Which status is justified?</strong></summary>
**💬 Answer:** Only the transport acknowledgement is established. The device may be offline, may have rejected the request, may still be executing, or may have executed with a lost response. Once its deadline expires, expose unknown outcome. Retry only under the same business identity and a deliberate idempotency policy.
</details>
## 💾 1. Preserve MQTT 5 request metadata
> **❓ Question:** **How can a responder know where to send a reply and the requester know which command the reply belongs to?**
Use the MQTT 5 Response Topic and Correlation Data as transport metadata, while retaining a stable application command ID and checking that the response destination is authorized.
Save the following complete files as `internal/commands/store.go` and `service.go`. The service stores pending state before publishing, keeps broker acknowledgement separate from device status, and matches a response by device, command ID, and Correlation Data.
### 🔍 `store.go` — file overview
This file stores in-memory command records and enforces correlation and status transitions. Its record keeps broker acknowledgement separately from device-reported status.
`internal/commands/store.go`:
```go
package commands
import (
"fmt"
"bytes"
"sync"
"time"
"example.com/mqtt5-learning/internal/contracts"
)
type Status string
// Status records application progress independently from BrokerAcked.
const (
Pending Status = "pending"
Accepted Status = "accepted"
Completed Status = "completed"
Rejected Status = "rejected"
Unknown Status = "unknown"
)
type Record struct {
CommandID string
DeviceID string
CorrelationData []byte
CreatedAt time.Time
Deadline time.Time
Status Status
BrokerAcked bool
AckReasonCode byte
Detail string
}
type Store struct {
mu sync.RWMutex
records map[string]Record
}
// NewStore creates the in-memory command record table.
func NewStore() *Store { return &Store{records: make(map[string]Record)} }
func key(deviceID, commandID string) string { return deviceID + "/" + commandID }
func (s *Store) Insert(record Record) error {
s.mu.Lock()
defer s.mu.Unlock()
k := key(record.DeviceID, record.CommandID)
if _, exists := s.records[k]; exists { return fmt.Errorf("command already exists") }
record.CorrelationData = append([]byte(nil), record.CorrelationData...)
s.records[k] = record
return nil
}
// Get and Update copy correlation bytes so callers cannot mutate stored state.
func (s *Store) Get(deviceID, commandID string) (Record, bool) {
s.mu.RLock()
defer s.mu.RUnlock()
record, ok := s.records[key(deviceID, commandID)]
record.CorrelationData = append([]byte(nil), record.CorrelationData...)
return record, ok
}
func (s *Store) Update(record Record) error {
s.mu.Lock()
defer s.mu.Unlock()
k := key(record.DeviceID, record.CommandID)
if _, exists := s.records[k]; !exists { return fmt.Errorf("command not found") }
record.CorrelationData = append([]byte(nil), record.CorrelationData...)
s.records[k] = record
return nil
}
func (s *Store) MarkBrokerAck(deviceID, commandID string, reason byte) (Record, error) {
s.mu.Lock()
defer s.mu.Unlock()
k := key(deviceID, commandID)
record, ok := s.records[k]
if !ok { return Record{}, fmt.Errorf("command not found") }
record.BrokerAcked, record.AckReasonCode = true, reason
s.records[k] = record
return record, nil
}
// MarkUnknownIfPending preserves any application result that arrived first.
func (s *Store) MarkUnknownIfPending(deviceID, commandID, detail string) (Record, error) {
s.mu.Lock()
defer s.mu.Unlock()
k := key(deviceID, commandID)
record, ok := s.records[k]
if !ok { return Record{}, fmt.Errorf("command not found") }
if record.Status == Pending || record.Status == Accepted { record.Status, record.Detail = Unknown, detail }
s.records[k] = record
return record, nil
}
// ApplyResult checks both device scope and correlation before changing status.
func (s *Store) ApplyResult(deviceID string, correlation []byte, result contracts.CommandResult) (Record, error) {
s.mu.Lock()
defer s.mu.Unlock()
k := key(deviceID, result.CommandID)
record, ok := s.records[k]
if !ok { return Record{}, fmt.Errorf("no pending command matches result") }
if result.DeviceID != record.DeviceID || !bytes.Equal(correlation, record.CorrelationData) {
return Record{}, fmt.Errorf("result identity or correlation does not match pending command")
}
if record.Status == Completed || record.Status == Rejected { return record, nil }
switch result.Status {
case "accepted": record.Status = Accepted
case "completed": record.Status = Completed
case "rejected": record.Status = Rejected
default: return Record{}, fmt.Errorf("unsupported result status %q", result.Status)
}
record.Detail = result.Detail
s.records[k] = record
return record, nil
}
func (s *Store) List() []Record {
s.mu.RLock()
defer s.mu.RUnlock()
result := make([]Record, 0, len(s.records))
for _, record := range s.records {
record.CorrelationData = append([]byte(nil), record.CorrelationData...)
result = append(result, record)
}
return result
}
```
#### 🏗️ `NewStore` — initialize command state
**Summary:** Creates the map that indexes records by device and command ID.
**Details:** The store is process-local; a restart loses pending and completed records. Section 13 replaces this volatile lifetime for durable command state.
#### ➕ `Insert`, `Get`, and `Update` — manage owned records
**Summary:** Add a unique command, retrieve a copy, or replace an existing record.
**Details:** The mutex protects the map, and each method copies `CorrelationData` so a caller cannot mutate stored bytes after the call. The composite key prevents two devices using the same command ID from colliding.
#### 📬 `MarkBrokerAck` — record transport evidence
**Summary:** Stores that the broker acknowledged the command publication and its reason code.
**Details:** This method changes `BrokerAcked`, not the device `Status`. A successful broker acknowledgement does not mean the device accepted or completed the command.
#### ⏳ `MarkUnknownIfPending` — preserve uncertain outcomes
**Summary:** Marks a pending or accepted command as unknown while preserving terminal results.
**Details:** A response may arrive before a publish timeout or expiry path runs. The status check prevents a later timeout from overwriting `Completed` or `Rejected`.
#### 🔗 `ApplyResult` — match and apply device evidence
**Summary:** Requires a known command, matching device, and matching Correlation Data before changing its application status.
**Details:** Correlation bytes are compared by content. Terminal results do not regress when a duplicate accepted response arrives. Correlation is a matching mechanism; topic ACLs and device identity validation remain separate authorization checks.
#### 📋 `List` — copy a snapshot for iteration
**Summary:** Returns all command records without exposing stored correlation byte slices.
**Details:** The snapshot lets expiry logic iterate without holding the store lock while it performs subsequent updates.
`internal/commands/service.go`:
### 🔍 `service.go` — file overview
This file coordinates command state and the transport adapter. It creates and stores pending state before publishing, handles device results, and records deadline expiry as uncertainty.
```go
package commands
import (
"context"
"crypto/rand"
"fmt"
"time"
"example.com/mqtt5-learning/internal/contracts"
)
type Publisher interface {
PublishCommand(context.Context, string, contracts.Command, contracts.TopicName, []byte) (byte, error)
}
// Service coordinates command identity, storage, publication, and deadlines.
type Service struct {
store *Store
publisher Publisher
now func() time.Time
timeout time.Duration
}
func NewService(store *Store, publisher Publisher, timeout time.Duration) *Service {
return &Service{store: store, publisher: publisher, now: func() time.Time { return time.Now().UTC() }, timeout: timeout}
}
func (s *Service) CreateAndPublish(ctx context.Context, deviceID, commandID string) (Record, error) {
responseTopic, err := contracts.DeviceTopic(deviceID, contracts.KindResponses)
if err != nil { return Record{}, fmt.Errorf("build response topic: %w", err) }
// Store pending state first so a fast device response can match immediately.
now := s.now()
correlation := make([]byte, 16)
if _, err := rand.Read(correlation); err != nil { return Record{}, fmt.Errorf("create correlation data: %w", err) }
record := Record{CommandID: commandID, DeviceID: deviceID, CorrelationData: correlation, CreatedAt: now, Deadline: now.Add(s.timeout), Status: Pending}
if err := s.store.Insert(record); err != nil { return Record{}, err }
command := contracts.Command{SchemaVersion: contracts.SchemaVersion, MessageID: "message-"+commandID, DeviceID: deviceID, OccurredAt: now.Format(time.RFC3339), CommandID: commandID, Command: "ping"}
ctx, cancel := context.WithDeadline(ctx, record.Deadline)
defer cancel()
reason, err := s.publisher.PublishCommand(ctx, deviceID, command, responseTopic, append([]byte(nil), correlation...))
if err != nil {
updated, _ := s.store.MarkUnknownIfPending(deviceID, commandID, "publish outcome uncertain: "+err.Error())
return updated, err
}
updated, err := s.store.MarkBrokerAck(deviceID, commandID, reason)
return updated, err
}
func (s *Service) HandleResult(deviceID string, correlation []byte, result contracts.CommandResult) (Record, error) {
return s.store.ApplyResult(deviceID, correlation, result)
}
// Expire records uncertainty; it does not claim that the device did no work.
func (s *Service) Expire(now time.Time) {
for _, record := range s.store.List() {
if (record.Status == Pending || record.Status == Accepted) && !now.Before(record.Deadline) {
_, _ = s.store.MarkUnknownIfPending(record.DeviceID, record.CommandID, "response deadline expired; execution outcome is unknown")
}
}
}
func (s *Service) Get(commandID string) (Record, bool) {
for _, record := range s.store.List() {
if record.CommandID == commandID { return record, true }
}
return Record{}, false
}
```
#### 🧰 `NewService` — connect command dependencies
**Summary:** Binds a command store, publisher, clock, and response timeout into one service.
**Details:** The service coordinates business state; the `Publisher` interface hides the MQTT implementation so state transitions can be reasoned about independently of the client.
#### 🚀 `CreateAndPublish` — register before sending
**Summary:** Creates correlation bytes and a pending record, stores it, then publishes with a deadline.
**Details:** Inserting first handles a fast response that arrives before `PublishCommand` returns. If publication errors or times out, the broker may still have received it, so the method marks the outcome unknown rather than asserting the device did nothing. A successful return records the broker acknowledgement separately.
#### 📩 `HandleResult` — apply a matched result
**Summary:** Delegates response identity and correlation checks to the store.
**Details:** Call it only after the receiver has parsed the topic and validated the result contract. The store then verifies that this application result belongs to the expected pending command.
#### ⏰ `Expire` — mark response timeouts as unknown
**Summary:** Scans commands whose deadlines passed and changes pending/accepted records to unknown.
**Details:** Expiry describes what the backend knows, not what the device did. A late device result can still arrive and needs the course's documented late-result policy.
#### 🔎 `Get` — query a command record
**Summary:** Finds a command by its application command ID.
**Details:** The method returns the application record for HTTP/status use; it does not query the device or the broker.
Read `references/paho.golang-v0.23.0/paho/cp_publish.go` for `paho.PublishProperties`. Build the response destination with `contracts.DeviceTopic(deviceID, contracts.KindResponses)`, then pass its string form to Paho's `ResponseTopic` field. Attach `CorrelationData` as opaque bytes associated with the pending command. Establish the backend response subscription before sending. Register pending state before publishing so an immediate response cannot beat registration.
### 🔍 `command_publisher.go` — file overview
This file adapts the command service to AutoPaho. It checks the requested reply destination, encodes the command, and publishes one MQTT 5 request with Response Topic and Correlation Data.
Save this complete adapter as `internal/mqtt/command_publisher.go`.
```go
package mqtt
import (
"context"
"encoding/json"
"fmt"
"time"
"example.com/mqtt5-learning/internal/contracts"
"github.com/eclipse/paho.golang/autopaho"
"github.com/eclipse/paho.golang/paho"
)
type CommandPublisher struct {
Manager *autopaho.ConnectionManager
}
func (p CommandPublisher) PublishCommand(ctx context.Context, deviceID string, command contracts.Command, responseTopic contracts.TopicName, correlation []byte) (byte, error) {
// Authorize the reply destination before putting it in the MQTT packet.
authorized, err := contracts.DeviceTopic(deviceID, contracts.KindResponses)
if err != nil { return 0, fmt.Errorf("build authorized response topic: %w", err) }
if responseTopic != authorized { return 0, fmt.Errorf("unauthorized response topic") }
if command.DeviceID != deviceID { return 0, fmt.Errorf("command device does not match publish destination") }
commandTopic, err := contracts.DeviceTopic(deviceID, contracts.KindCommands)
if err != nil { return 0, fmt.Errorf("build command topic: %w", err) }
payload, err := json.Marshal(command)
if err != nil { return 0, fmt.Errorf("encode command: %w", err) }
operation, cancel := context.WithTimeout(ctx, 3*time.Second)
defer cancel()
// Correlation Data is copied so the packet owns stable bytes for this publish.
ack, err := p.Manager.Publish(operation, &paho.Publish{
Topic: commandTopic.String(),
QoS: 1,
Payload: payload,
Properties: &paho.PublishProperties{
ContentType: "application/json",
ResponseTopic: responseTopic.String(),
CorrelationData: append([]byte(nil), correlation...),
},
})
if err != nil { return 0, fmt.Errorf("publish command: %w", err) }
if ack == nil { return 0, fmt.Errorf("missing QoS 1 PUBACK") }
if ack.ReasonCode >= 0x80 { return ack.ReasonCode, fmt.Errorf("broker rejected publish: reason 0x%02x", ack.ReasonCode) }
return ack.ReasonCode, nil
}
```
#### 📤 `PublishCommand` — send the transport request
**Summary:** Verifies device and reply-topic agreement, encodes the command, and waits for a bounded QoS 1 publish result.
**Details:** It copies correlation bytes into the packet so the request owns stable data. A returned reason code is broker-hop evidence only. It is returned separately from the device's eventual application result, which arrives later on the response subscription.
The receive callback must copy packet properties while the packet is available. Add these fields to the existing `mqtt.Message` and set them in its receive callback:
```go
type Message struct {
Topic string
Payload []byte
Retain bool
ResponseTopic string
CorrelationData []byte
}
// Copy properties alongside payload before the receive packet leaves the callback.
properties := packet.Properties
message := Message{Topic: packet.Topic, Payload: append([]byte(nil), packet.Payload...), Retain: packet.Retain}
if properties != nil {
message.ResponseTopic = properties.ResponseTopic
message.CorrelationData = append([]byte(nil), properties.CorrelationData...)
}
```
### 🔍 `receive.go` — file overview
This file is the boundary between an incoming response packet and command state. It checks topic kind and payload contract before passing device identity and Correlation Data to the service.
Save this complete result receiver as `internal/commands/receive.go`.
```go
package commands
import (
"fmt"
"time"
"example.com/mqtt5-learning/internal/contracts"
"example.com/mqtt5-learning/internal/mqtt"
)
func HandleMessage(service *Service, message mqtt.Message) error {
// Validate the route and result body before matching the pending request.
topic, err := contracts.ParseTopic(message.Topic)
if err != nil { return err }
if topic.Kind != contracts.KindResponses { return fmt.Errorf("expected a response topic") }
result, err := contracts.Decode[contracts.CommandResult](message.Payload, topic, time.Now().UTC())
if err != nil { return err }
_, err = service.HandleResult(topic.DeviceID, append([]byte(nil), message.CorrelationData...), result)
return err
}
```
#### 📩 `HandleMessage` — validate and match a result
**Summary:** Parses a response topic, decodes a `CommandResult`, then asks the service to match and apply it.
**Details:** Malformed or wrong-kind messages stop before the command store is touched. The device ID comes from the parsed topic and is checked against the body and pending record. Correlation Data is copied before it enters state handling.
## 🧩 2. Model evidence explicitly
> **❓ Question:** **What should the backend show after a PUBACK but before any device response arrives?**
Represent broker delivery evidence separately from device acceptance, completion, rejection, and timeout/unknown outcome.
Store command ID, expected device, correlation bytes, creation/deadline times, business status, and independent transport acknowledgement evidence. Start with created → pending delivery → accepted → completed or rejected. A ping simulator can send accepted followed by completed. Response timeout means unknown outcome; it does not establish that no effect happened.
Protect concurrent access and define allowed transitions. A completed or rejected command must not regress on a duplicate accepted response. Compare command identity, topic/body device identity, and correlation before applying a response. Keep timed-out records for a bounded retention period even after removing their waiters, so status can explain the uncertainty.
## ♻️ 3. Make the simulator idempotent
> **❓ Question:** **If QoS 1 or an application retry delivers the same command again, how does the device avoid repeating a non-idempotent effect?**
Record results by stable command ID and replay the prior result for duplicates instead of applying the simulated effect twice.
### 🔍 `dedup.go` — file overview
This file records a bounded set of command IDs and their outcomes in simulator memory. It ensures the demonstration does not repeat the same synthetic effect while this process remains alive.
Save this complete in-memory simulator cache as `internal/simulator/dedup.go`.
```go
package simulator
import (
"crypto/sha256"
"encoding/json"
"fmt"
"sync"
"example.com/mqtt5-learning/internal/contracts"
)
type cachedResult struct {
fingerprint [32]byte
result contracts.CommandResult
}
type Deduplicator struct {
mu sync.Mutex
capacity int
results map[string]cachedResult
}
// NewDeduplicator bounds how many command results this simulator retains.
func NewDeduplicator(capacity int) *Deduplicator {
if capacity < 1 { capacity = 1 }
return &Deduplicator{capacity: capacity, results: make(map[string]cachedResult)}
}
func AuthorizedResponseTopic(deviceID, requested string) bool {
expected, err := contracts.DeviceTopic(deviceID, contracts.KindResponses)
return err == nil && requested == expected.String()
}
// Apply executes effect at most once for a device/command ID while this process lives.
func (d *Deduplicator) Apply(command contracts.Command, effect func() (contracts.CommandResult, error)) (contracts.CommandResult, bool, error) {
if effect == nil { return contracts.CommandResult{}, false, fmt.Errorf("effect is required") }
encoded, err := json.Marshal(command)
if err != nil { return contracts.CommandResult{}, false, fmt.Errorf("fingerprint command: %w", err) }
fingerprint := sha256.Sum256(encoded)
key := command.DeviceID + "/" + command.CommandID
d.mu.Lock()
defer d.mu.Unlock()
if previous, ok := d.results[key]; ok {
if previous.fingerprint != fingerprint { return contracts.CommandResult{}, false, fmt.Errorf("command ID was reused with different content") }
return previous.result, true, nil
}
if len(d.results) >= d.capacity { return contracts.CommandResult{}, false, fmt.Errorf("deduplication cache is full") }
result, err := effect()
if err != nil { return contracts.CommandResult{}, false, err }
d.results[key] = cachedResult{fingerprint: fingerprint, result: result}
return result, false, nil
}
```
#### 🏗️ `NewDeduplicator` — create bounded replay state
**Summary:** Allocates the cache and raises a too-small capacity to one.
**Details:** Capacity prevents this example from growing its map forever. The cache is volatile; restarting the simulator forgets the IDs, so it does not provide durable idempotency.
#### 🔐 `AuthorizedResponseTopic` — constrain reply routing
**Summary:** Allows only the response topic assigned to this device ID.
**Details:** Correlation bytes match a response to a request, but do not authorize where the response is sent. The simulator checks the destination before publishing.
#### ♻️ `Apply` — execute or replay one command
**Summary:** Fingerprints the command, checks the device/command key under a lock, and runs the effect only when no prior result exists.
**Details:** An identical retry receives the stored result; reusing an ID for different content fails; a full cache refuses new work instead of evicting a result that still protects against a duplicate. Holding the lock across the synthetic effect makes admission atomic in this small example. A real network call or long-running operation should not run under this lock.
Use the deduplicator in the complete simulator handler below. Save it as `internal/simulator/respond.go`; it checks the requested destination before applying the effect, then publishes the result with the original Correlation Data.
```go
package simulator
import (
"context"
"encoding/json"
"fmt"
"time"
"example.com/mqtt5-learning/internal/contracts"
"example.com/mqtt5-learning/internal/mqtt"
"github.com/eclipse/paho.golang/autopaho"
"github.com/eclipse/paho.golang/paho"
)
func Respond(ctx context.Context, manager *autopaho.ConnectionManager, cache *Deduplicator, deviceID string, message mqtt.Message) error {
// Reject retained or misrouted requests before validating or applying an effect.
if message.Retain { return fmt.Errorf("retained commands are rejected") }
topic, err := contracts.ParseTopic(message.Topic)
if err != nil { return err }
if topic.Kind != contracts.KindCommands || topic.DeviceID != deviceID { return fmt.Errorf("command topic is not authorized for this device") }
command, err := contracts.Decode[contracts.Command](message.Payload, topic, time.Now().UTC())
if err != nil { return err }
if !AuthorizedResponseTopic(deviceID, message.ResponseTopic) { return fmt.Errorf("unauthorized response topic") }
// Cache the simulated result by stable command ID to suppress repeat effects.
result, _, err := cache.Apply(command, func() (contracts.CommandResult, error) {
return contracts.CommandResult{SchemaVersion: contracts.SchemaVersion, MessageID: "result-"+command.CommandID, DeviceID: deviceID, OccurredAt: time.Now().UTC().Format(time.RFC3339), CommandID: command.CommandID, Status: "completed"}, nil
})
if err != nil { return err }
payload, err := json.Marshal(result)
if err != nil { return fmt.Errorf("encode result: %w", err) }
operation, cancel := context.WithTimeout(ctx, 3*time.Second)
defer cancel()
// Reply only to the authorized destination and echo correlation bytes unchanged.
ack, err := manager.Publish(operation, &paho.Publish{Topic: message.ResponseTopic, QoS: 1, Payload: payload, Properties: &paho.PublishProperties{ContentType: "application/json", CorrelationData: append([]byte(nil), message.CorrelationData...)}})
if err != nil { return fmt.Errorf("publish response: %w", err) }
if ack == nil { return fmt.Errorf("missing response PUBACK") }
if ack.ReasonCode >= 0x80 { return fmt.Errorf("response publish rejected: reason 0x%02x", ack.ReasonCode) }
return nil
}
```
#### 📩 `Respond` — validate, deduplicate, and reply
**Summary:** Rejects retained or misrouted commands, validates the contract and reply destination, obtains a cached result, then publishes it with the request's correlation bytes.
**Details:** Route and destination checks happen before the simulated effect so an unauthorized request cannot trigger it. `cache.Apply` ensures a repeated command reuses its first result while this process remains alive. The bounded publish context prevents a stuck response operation from occupying a worker indefinitely. PUBACK for the result still proves only the broker hop; the backend must receive and match the response separately.
Replace the Section 5 simulator entrypoint with this complete version to use the deduplication and response handler. It retains the validated heartbeat publisher and routes every received command through `Respond`.
```go
package main
import (
"context"
"encoding/json"
"fmt"
"log"
"os"
"os/signal"
"syscall"
"time"
"example.com/mqtt5-learning/internal/config"
"example.com/mqtt5-learning/internal/contracts"
"example.com/mqtt5-learning/internal/mqtt"
"example.com/mqtt5-learning/internal/simulator"
)
func main() {
if err := run(); err != nil { log.Fatal(err) }
}
func run() error {
cfg, err := config.Load(os.Args[1:])
if err != nil { return fmt.Errorf("load config: %w", err) }
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
defer stop()
commandFilter, err := contracts.DeviceSubscription(cfg.MQTT.DeviceID, contracts.KindCommands)
if err != nil { return err }
telemetryTopic, err := contracts.DeviceTopic(cfg.MQTT.DeviceID, contracts.KindTelemetry)
if err != nil { return err }
inbox := make(chan mqtt.Message, 16)
manager, err := mqtt.Start(ctx, cfg, []contracts.TopicFilter{commandFilter}, inbox)
if err != nil { return fmt.Errorf("start MQTT: %w", err) }
defer mqtt.Stop(manager)
// Keep recent command results so repeated delivery does not repeat the effect.
cache := simulator.NewDeduplicator(1024)
ticker := time.NewTicker(3 * time.Second)
defer ticker.Stop()
sequence := 0
for {
select {
case <-ctx.Done():
return nil
case <-ticker.C:
sequence++
heartbeat := contracts.Heartbeat{SchemaVersion: contracts.SchemaVersion, MessageID: fmt.Sprintf("heartbeat-%d-%d", time.Now().Unix(), sequence), DeviceID: cfg.MQTT.DeviceID, OccurredAt: time.Now().UTC().Format(time.RFC3339)}
payload, err := json.Marshal(heartbeat)
if err != nil { return fmt.Errorf("encode heartbeat: %w", err) }
if err := mqtt.Publish(ctx, manager, telemetryTopic, payload); err != nil { log.Printf("heartbeat publish: %v", err) }
case message := <-inbox:
if err := simulator.Respond(ctx, manager, cache, cfg.MQTT.DeviceID, message); err != nil { log.Printf("command rejected or response failed topic=%q reason=%v", message.Topic, err) }
}
}
}
```
#### 🧭 `main` and `run` — operate the simulator lifecycle
**Summary:** `main` reports a fatal error; `run` starts the device client, creates the bounded deduplication cache, publishes heartbeats, and delegates commands to `Respond`.
**Details:** One cache is created per process so duplicate deliveries share replay state. The ticker is stopped and MQTT manager closed on return. The receive loop invokes `Respond` outside the callback, keeping packet reception separate from validation, effect handling, and reply publication. Restarting the simulator clears this cache, which is why it demonstrates process-local idempotency rather than durable device guarantees.
The device checks that the requested response destination exactly matches its own response topic before publishing. Correlation Data matches the result to the pending request; it does not authorize a destination. Retained one-shot commands are rejected before parsing.
## 🧪 4. Exercise timing and failures
> **❓ Question:** **If a response is late or missing, what can the backend safely conclude, and what should happen to a later result?**
Exercise timeouts and duplicates while keeping uncertainty visible; a timeout is not proof that the device did nothing.
Use the completed command service and simulator handler to trace 20 concurrent pings with separate IDs. Observe delayed, duplicate, wrong-device, wrong-correlation, and out-of-order responses. Check that each command receives only matching evidence. A retry keeps the same stable command ID; a publish timeout alone does not prove that the device received nothing.
```sh
mkdir -p internal/commands internal/simulator
gofmt -w internal cmd
go test -race ./...
```
Run `mkdir -p` before saving the service and simulator package files. `gofmt` formats your edits and `go test -race ./...` executes the module's tests with concurrency instrumentation. A fake transport is a small in-process implementation of the caller's transport interface: it lets tests deliberately return errors or responses without needing a broker. An injected clock makes deadline tests deterministic without waiting in real time.
When copying Correlation Data, allocate a new byte slice so later processing owns stable bytes. Comparing correlation bytes means comparing their contents, not pointer addresses. Before updating a shared command map, validate the response and hold its lock only for the state transition. Publish outside the lock so a network delay cannot freeze every command.
The implementation handles responses that arrive before the publish call returns because pending state is inserted first and broker acknowledgement updates only its own fields. Inspect the matching, timeout, deduplication, and terminal-state branches in the complete files above; a fake transport can isolate each case without a broker.
**Done when:** Concurrent requests match correctly, identical duplicates cause one effect, unexpected destinations are rejected, and late responses follow a documented policy. Explain why PUBACK and Correlation Data cannot prove business completion.
## ✅ Result — the story so far
The backend now treats a ping as a tracked command: it records the command ID, expected device, correlation data, and deadline before publishing. The simulator checks the command and authorized response destination, applies the simulated effect once for that stable ID, and publishes a result. The backend matches that result to the pending command and updates its status. A broker PUBACK still means only that the broker acknowledged its hop; a missing response leaves the outcome unknown, not necessarily unexecuted.
Previous: [06-concurrency-cancellation.md](06-concurrency-cancellation.md). Next: [08-http-commands.md](08-http-commands.md).
teaching/course-source/08-http-commands.md
+412
View File
@@ -0,0 +1,412 @@
# 🌐 Section 8 — Expose device commands through HTTP
## 🎬 Start with the system story
> **❓ Question:** **A user clicks “ping device” in an HTTP API. How can the API respond promptly while the MQTT command and device response happen asynchronously?**
The HTTP handler validates the request and asks an application service to create a command. The service records its identity and pending status, publishes through the MQTT client, then exposes later device evidence through a status endpoint. HTTP request lifetime, MQTT delivery, and the device operation each have their own acknowledgement and failure boundary.
```mermaid
sequenceDiagram
participant U as API caller
participant H as Go HTTP handler
participant S as Command service
participant B as MQTT broker
participant D as Device
U->>H: POST /devices/{id}/commands
H->>S: Validate and create command
S->>S: Store command ID and pending state
S->>B: Publish MQTT command
B-->>S: PUBACK for broker hop
S-->>H: Accepted with command ID
H-->>U: HTTP 202 + command ID
B->>D: Deliver command
D->>B: Publish application result
B->>S: Deliver result
U->>H: GET /commands/{id}
H->>S: Read current status
S-->>H: Accepted/completed/rejected/unknown
H-->>U: HTTP status representation
```
**Before each implementation step, ask:** Is this response evidence that the HTTP request was accepted, that the broker acknowledged a publication, or that the device completed the work? Follow [Lesson 0](00-system-lifecycle.md) for the complete MQTT lifecycle.
Prerequisite: Section 7 provides a command service and queryable in-memory states. The backend and simulator continue to read their shared disposable password from `configs/lab.toml`; select each broker username with `-username` when starting the process.
**Scope and trust boundary:** This lesson's HTTP endpoint is unauthenticated and binds to loopback for the disposable lab. Do not expose it to another network; Section 11 adds API authentication and per-device authorization. HTTP 202 means the command was admitted to process-local memory, not durably stored. A restart loses the command and retry-key maps until Sections 13–14 add persistence and recoverable publication intent.
Run all commands from `/home/ryan/Media/Code/Code/robot-market/mqtt-5-learning`. Save the complete HTTP implementation blocks at the project-relative paths shown below. This lesson includes the handlers and server setup; the remaining exercises trace request, retry, and device-result behavior.
## 🌐 Translate evidence between HTTP and MQTT
HTTP has one response to this request. MQTT has an asynchronous stream where the device may reply after the HTTP connection closes. Returning 202 after admission gives the caller a stable command ID; a later GET reports evidence accumulated by the command service.
```text
POST + retry key → admit command C → HTTP 202 + C
→ MQTT publication → device result → status C
GET C --------------------------------------------------------------→ status C
```
**❓ Question:** **The backend publishes a ping and receives PUBACK. Does that mean the device accepted or completed it?**
**💬 Answer:** No. PUBACK confirms the broker acknowledged the backend's publication. It does not confirm device delivery or business acceptance. The HTTP API returns a command ID, and a later status request reports the device's application result if one arrives. The handler uses the request context while parsing. After validation, it passes work to the command service with the application lifecycle context and the command's own deadline. A disconnected caller therefore does not cancel accepted work.
Predict a lost HTTP response: the server admits C, but the caller never sees 202 and retries. The same retry key and payload should return C, not create D. Reusing the key with a changed payload is a conflict. Later, scope keys by authenticated caller so two users cannot accidentally share a command record.
<details>
<summary>If the backend restarts after returning 202, can this lesson recover C?</summary>
No. The lesson's in-memory command and retry-key maps disappear. HTTP 202 describes admission under the current implementation, not disk durability. Sections 13–14 add database admission and the outbox; Section 15 separates the processes. Until then, document this restart limit in the lab API.
</details>
## ⚙️ 1. Add HTTP configuration and handlers
> **❓ Question:** **Which process accepts the user's HTTP request, and how does that process reach the command service without putting MQTT logic in the handler?**
Wire the HTTP server to the application service and preserve bounded parsing, deadlines, and controlled shutdown.
Save the complete API implementation below as `internal/httpapi/handlers.go` and `server.go`. It binds only to loopback, limits and strictly decodes request bodies, requires an idempotency key, and sets finite HTTP timeouts. The lab listener address is fixed in the complete backend entrypoint below; production authorization is introduced later.
`internal/httpapi/handlers.go`:
```go
package httpapi
import (
"context"
"crypto/rand"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
"strings"
"sync"
"example.com/mqtt5-learning/internal/commands"
)
const maxRequestBytes = 4 * 1024
// The API depends on command operations, not on the MQTT client itself.
type commandService interface {
CreateAndPublish(ctx context.Context, deviceID, commandID string) (commands.Record, error)
Get(commandID string) (commands.Record, bool)
}
type retryEntry struct {
fingerprint [32]byte
record commands.Record
}
type API struct {
commands commandService
serviceCtx context.Context
mu sync.Mutex
retries map[string]retryEntry
}
// NewAPI keeps retries in memory and ties accepted work to the app lifecycle.
func NewAPI(serviceCtx context.Context, service commandService) *API {
return &API{commands: service, serviceCtx: serviceCtx, retries: make(map[string]retryEntry)}
}
type createRequest struct { Command string `json:"command"` }
type commandResponse struct {
CommandID string `json:"command_id"`
DeviceID string `json:"device_id"`
Status string `json:"status"`
BrokerAcked bool `json:"broker_acked"`
Detail string `json:"detail,omitempty"`
}
func (a *API) Handler() http.Handler {
// Keep the public surface small: create commands and read their status.
mux := http.NewServeMux()
mux.HandleFunc("POST /devices/{deviceID}/commands", a.createCommand)
mux.HandleFunc("GET /commands/{commandID}", a.getCommand)
return mux
}
func (a *API) createCommand(w http.ResponseWriter, r *http.Request) {
// Validate caller input before reserving an idempotency key or publishing.
deviceID := r.PathValue("deviceID")
if !validPathID(deviceID) { http.Error(w, "invalid device ID", http.StatusBadRequest); return }
key := strings.TrimSpace(r.Header.Get("Idempotency-Key"))
if key == "" || len(key) > 128 { http.Error(w, "valid Idempotency-Key required", http.StatusBadRequest); return }
var input createRequest
if err := decodeRequest(w, r, &input); err != nil {
var tooLarge *http.MaxBytesError
if errors.As(err, &tooLarge) { http.Error(w, "request body too large", http.StatusRequestEntityTooLarge); return }
http.Error(w, err.Error(), http.StatusBadRequest)
return
}
if input.Command != "ping" { http.Error(w, "unsupported command", http.StatusBadRequest); return }
// The fingerprint detects reuse of one key for different request content.
fingerprint := sha256.Sum256([]byte(deviceID + "\x00" + input.Command))
// This in-memory lesson serializes admissions to make same-key retries atomic.
a.mu.Lock()
defer a.mu.Unlock()
if previous, ok := a.retries[key]; ok {
if previous.fingerprint != fingerprint { http.Error(w, "idempotency key reused with different request", http.StatusConflict); return }
writeCommand(w, previous.record, http.StatusAccepted)
return
}
commandID, err := newCommandID()
if err != nil { http.Error(w, "could not create command ID", http.StatusInternalServerError); return }
record, publishErr := a.commands.CreateAndPublish(a.serviceCtx, deviceID, commandID)
if record.CommandID == "" {
if publishErr != nil { http.Error(w, "command admission unavailable", http.StatusServiceUnavailable); return }
http.Error(w, "command admission failed", http.StatusServiceUnavailable)
return
}
a.retries[key] = retryEntry{fingerprint: fingerprint, record: record}
// The command record exists even if the publish outcome is uncertain.
writeCommand(w, record, http.StatusAccepted)
}
func (a *API) getCommand(w http.ResponseWriter, r *http.Request) {
record, ok := a.commands.Get(r.PathValue("commandID"))
if !ok { http.Error(w, "command not found", http.StatusNotFound); return }
writeCommand(w, record, http.StatusOK)
}
// decodeRequest enforces a byte limit, strict fields, and exactly one JSON value.
func decodeRequest(w http.ResponseWriter, r *http.Request, target any) error {
r.Body = http.MaxBytesReader(w, r.Body, maxRequestBytes)
decoder := json.NewDecoder(r.Body)
decoder.DisallowUnknownFields()
if err := decoder.Decode(target); err != nil { return fmt.Errorf("invalid JSON request: %w", err) }
var trailing any
if err := decoder.Decode(&trailing); err != io.EOF {
if err == nil { return errors.New("request must contain one JSON value") }
return fmt.Errorf("trailing request data: %w", err)
}
return nil
}
func writeCommand(w http.ResponseWriter, record commands.Record, status int) {
w.Header().Set("Content-Type", "application/json")
w.Header().Set("Location", "/commands/"+record.CommandID)
w.WriteHeader(status)
_ = json.NewEncoder(w).Encode(commandResponse{CommandID: record.CommandID, DeviceID: record.DeviceID, Status: string(record.Status), BrokerAcked: record.BrokerAcked, Detail: record.Detail})
}
// Path IDs use the same restricted alphabet as contract and topic IDs.
func validPathID(value string) bool {
if value == "" || len(value) > 128 { return false }
for _, r := range value {
if !(r >= 'a' && r <= 'z' || r >= 'A' && r <= 'Z' || r >= '0' && r <= '9' || strings.ContainsRune("._-", r)) { return false }
}
return true
}
func newCommandID() (string, error) {
var raw [16]byte
if _, err := rand.Read(raw[:]); err != nil { return "", err }
return "cmd-" + hex.EncodeToString(raw[:]), nil
}
```
`internal/httpapi/server.go`:
```go
package httpapi
import (
"fmt"
"net"
"net/http"
"time"
)
func NewServer(address string, api *API) (*http.Server, error) {
// The teaching API listens only on loopback and uses finite I/O timeouts.
host, port, err := net.SplitHostPort(address)
if err != nil || (host != "127.0.0.1" && host != "localhost") || port == "" {
return nil, fmt.Errorf("HTTP lab address must be loopback with an explicit port")
}
return &http.Server{
Addr: address, Handler: api.Handler(),
ReadHeaderTimeout: 5 * time.Second,
ReadTimeout: 10 * time.Second,
WriteTimeout: 10 * time.Second,
IdleTimeout: 60 * time.Second,
}, nil
}
```
The complete final `cmd/backend/main.go` is below. It starts MQTT before HTTP, routes response messages into the command service, and shuts down the HTTP server before disconnecting MQTT. It replaces the Section 5 backend entrypoint.
```go
package main
import (
"context"
"errors"
"fmt"
"log"
"net/http"
"os"
"os/signal"
"syscall"
"time"
"example.com/mqtt5-learning/internal/commands"
"example.com/mqtt5-learning/internal/config"
"example.com/mqtt5-learning/internal/contracts"
"example.com/mqtt5-learning/internal/httpapi"
"example.com/mqtt5-learning/internal/mqtt"
)
func main() {
if err := run(); err != nil { log.Fatal(err) }
}
func run() error {
cfg, err := config.Load(os.Args[1:])
if err != nil { return fmt.Errorf("load config: %w", err) }
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
defer stop()
inbox := make(chan mqtt.Message, 16)
telemetryFilter, err := contracts.DeviceFilter(contracts.KindTelemetry)
if err != nil { return err }
responsesFilter, err := contracts.DeviceFilter(contracts.KindResponses)
if err != nil { return err }
salesFilter, err := contracts.DeviceFilter(contracts.KindSales)
if err != nil { return err }
topics := []contracts.TopicFilter{telemetryFilter, responsesFilter, salesFilter}
manager, err := mqtt.Start(ctx, cfg, topics, inbox)
if err != nil { return fmt.Errorf("start MQTT: %w", err) }
defer mqtt.Stop(manager)
// Wire the command service before exposing its HTTP endpoints.
store := commands.NewStore()
publisher := mqtt.CommandPublisher{Manager: manager}
service := commands.NewService(store, publisher, 15*time.Second)
api := httpapi.NewAPI(ctx, service)
server, err := httpapi.NewServer("127.0.0.1:18080", api)
if err != nil { return err }
serverErrors := make(chan error, 1)
go func() { serverErrors <- server.ListenAndServe() }()
log.Printf("HTTP API listening on %s", server.Addr)
expirationTicker := time.NewTicker(time.Second)
defer expirationTicker.Stop()
// Keep response processing and expiration active for the server lifetime.
for {
select {
case <-ctx.Done():
shutdown, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
if err := server.Shutdown(shutdown); err != nil { return fmt.Errorf("HTTP shutdown: %w", err) }
return nil
case err := <-serverErrors:
if !errors.Is(err, http.ErrServerClosed) { return fmt.Errorf("HTTP server: %w", err) }
return nil
case <-expirationTicker.C:
service.Expire(time.Now().UTC())
case message := <-inbox:
topic, err := contracts.ParseTopic(message.Topic)
if err != nil { log.Printf("rejected topic=%q reason=%v", message.Topic, err); continue }
switch topic.Kind {
case contracts.KindResponses:
if err := commands.HandleMessage(service, message); err != nil { log.Printf("rejected response topic=%q reason=%v", message.Topic, err) }
case contracts.KindTelemetry:
var heartbeat contracts.Heartbeat
heartbeat, err = contracts.Decode[contracts.Heartbeat](message.Payload, topic, time.Now().UTC())
if err != nil { log.Printf("rejected telemetry topic=%q reason=%v", message.Topic, err); continue }
log.Printf("valid heartbeat device=%s message=%s", heartbeat.DeviceID, heartbeat.MessageID)
case contracts.KindSales:
var report contracts.SalesReport
report, err = contracts.Decode[contracts.SalesReport](message.Payload, topic, time.Now().UTC())
if err != nil { log.Printf("rejected sales topic=%q reason=%v", message.Topic, err); continue }
log.Printf("valid sales report device=%s report=%s", report.DeviceID, report.ReportID)
default:
log.Printf("ignored topic kind=%s", topic.Kind)
}
}
}
}
```
## 📋 2. Define the API contract
> **❓ Question:** **What should the caller receive immediately when device work will finish later?**
Return an accepted command ID, then provide a status resource that reports later evidence.
The complete handlers above implement `POST /devices/{id}/commands` and `GET /commands/{id}`. They limit bodies, reject trailing JSON and unknown fields, validate the path ID, require an idempotency key, and return consistent JSON status. The listener remains local; production user authorization is introduced later.
Return HTTP 202 with a command ID and a Location pointing to its status after service admission. Document malformed-input 400, oversized-body 413, unknown-command/device 404 or a consistent chosen alternative, conflict 409, and admission-unavailable 503 behavior. Transport uncertainty after acceptance belongs in command status rather than silently discarding an accepted command.
## 🌐 3. Implement HTTP retry identity
> **❓ Question:** **If the server accepted a request but the HTTP response was lost, how can the caller retry without creating a second device action?**
Use an HTTP idempotency key with stable request semantics; keep it distinct from the command ID and MQTT packet identifier.
The handler keeps the key, request fingerprint, and command record together in memory. An identical retry returns the original ID; changed content under that key returns 409. The mutex makes admission atomic in this single-process lesson. A restart loses this map, so this is not a durable idempotency guarantee. The HTTP key, command ID, and MQTT packet ID have different lifetimes and purposes.
Use the HTTP context for request parsing and admission. Once accepted, execute under the application's lifecycle and command deadline, so client disconnect does not cancel the command. Inject the command service through a small interface and keep MQTT calls out of handlers.
## 🧪 4. Run and verify
> **❓ Question:** **What evidence demonstrates each boundary in the round trip—from HTTP acceptance to broker delivery to device result?**
Observe the returned command ID, MQTT acknowledgement, simulator response, and status endpoint separately.
Start the simulator and backend in separate terminals from the project root. The test password is already loaded from `configs/lab.toml`; `-username` selects the account for each process:
```sh
go run ./cmd/simulator -username simulator -client-id lab-simulator-device-001
```
```sh
go run ./cmd/backend -username backend -client-id lab-backend
```
In a third terminal, send the HTTP command:
```sh
curl -i -X POST http://127.0.0.1:18080/devices/device-001/commands -H 'Content-Type: application/json' -H 'Idempotency-Key: lesson8-ping-001' --data '{"command":"ping"}'
# Replace cmd-example with the ID returned by the POST response.
curl -i http://127.0.0.1:18080/commands/cmd-example
```
`curl` is an HTTP client. `-i` prints response headers alongside the body; `-X POST` chooses the method; each `-H` sets a request header; `--data` sends the quoted JSON bytes. Content-Type tells the server how to interpret the body, while Idempotency-Key identifies this caller retry. This local exercise sends no production authorization token.
Copy the returned ID into `GET /commands/{id}`. Repeat the identical POST and confirm the same ID. Use a new key for a genuinely new command.
For example, if the returned ID is `cmd-example`, use `curl -i http://127.0.0.1:18080/commands/cmd-example`; without `--data` or an explicit method, curl performs GET. Replace that example ID with the actual returned value. HTTP 202 means admitted for asynchronous processing; inspect later status for device completion. `httptest` runs handlers against an in-process request/recorder or test server, letting you assert headers, status, and body without the live broker.
Set ReadHeaderTimeout, ReadTimeout, WriteTimeout, and IdleTimeout deliberately. Coordinate bounded HTTP shutdown with the MQTT and worker lifecycle.
```sh
mkdir -p internal/httpapi
gofmt -w internal cmd
go test -race ./...
go vet ./...
```
ReadHeaderTimeout bounds header reading; ReadTimeout bounds reading the request including its body; WriteTimeout bounds response writing; IdleTimeout bounds waiting for another request on an idle keep-alive connection. Choose values that fit this asynchronous API rather than waiting in POST for a device result. Server `Shutdown` stops accepting work and waits for active HTTP requests within its context deadline; your application must separately drain accepted command work.
`mkdir -p` creates the HTTP package directory, `gofmt` formats edits, and the test/race/vet commands check behavior, executed concurrent access, and suspicious static patterns respectively. Run directory creation before writing the files even though the check block collects the commands together.
The complete handler performs the same checks an `httptest` would exercise: strict decoding, body limits, idempotency, unavailable admission, unknown IDs, and status responses. Run the supplied local `curl` flow to observe them through the real HTTP server. Accepted work is owned by the command service; device completion arrives later through MQTT.
**Done when:** HTTP creates a ping, status reflects device evidence, retries reuse its ID, and closing the caller connection does not erase accepted work.
## ✅ Result — the story so far
An API caller can now ask the backend to ping a device with HTTP. The handler validates the request and admits it through the command service; the caller receives HTTP 202 and a command ID without waiting for the device. The backend publishes the MQTT command, the simulator returns its application result, and a later GET reports the command's current evidence. Retrying the same POST with the same idempotency key returns the original command instead of starting a second effect. HTTP acceptance, MQTT acknowledgement, and device completion remain separate events.
Previous: [07-command-response.md](07-command-response.md). Next: [09-sessions-recovery.md](09-sessions-recovery.md).
teaching/course-source/09-sessions-recovery.md
+123
View File
@@ -0,0 +1,123 @@
# Section 9 — Learn MQTT 5 sessions through failure experiments
Prerequisite: Section 8 exposes commands and their evidence; use only synthetic ping effects.
Run all commands from `/home/ryan/Media/Code/Code/robot-market/mqtt-5-learning`. Create files at the project-relative paths shown below, not under `teaching/course-source/`. Continue the same application and pinned Paho v0.23.0 dependency. These are guided implementation exercises; attempt each step before requesting a full solution.
## A connection is temporary; a session may outlive it
**❓ Question:** **The simulator loses its network connection after subscribing. What must still exist for a queued ping to reach it when it reconnects?**
**💬 Answer:** The simulator must reconnect with the same Client ID to a broker session that still exists, and that session must contain its QoS 1 subscription. The backend's PUBACK only confirms the broker accepted that publication on this hop; it does not prove the simulator received or completed the ping. Your Go command map lives in a different place and is not restored by CONNACK.
The network connection carries packets now. A broker session can preserve subscriptions and eligible pending deliveries after that connection closes.
| Clean Start on this connection | Prior unexpired session under the same Client ID | Expected Session Present | Meaning |
| --- | --- | --- | --- |
| true | Either | false | Discard old state and begin a new session |
| false | Yes | true | Resume that broker session |
| false | No | false | Create a new session; restore subscriptions |
Session Expiry answers a later question: how long may the session remain after disconnect? Setting Clean Start true with expiry 60 creates a new session that can subsequently survive a short outage. Setting Clean Start false with expiry 0 can resume an existing session now, but asks for no persistence after this connection ends. They are independent controls, not two names for the same switch.
In AutoPaho v0.23.0, `CleanStartOnInitialConnection` affects the manager's first successful connection; automatic reconnects use false. Restarting the Go process creates a new manager and applies the initial choice again. Therefore "network reconnect works" does not establish "process restart resumes the old session."
### Work through an offline command
**❓ Question:** **Why does a new Client ID not receive the offline ping sent for the old simulator ID?**
**💬 Answer:** Client ID selects the broker session. A new ID starts or resumes a different session, so it has no old subscription or queued delivery. A `device_id` in the payload is only an application claim and does not select or authenticate a broker session.
At t=0 the simulator has subscribed at QoS 1 with expiry 60. At t=5 it disconnects. At t=10 the backend publishes a nonretained QoS 1 ping. At t=20 the same simulator ID reconnects with Clean Start false. Assuming the broker retained the session, accepted the publication, and no shorter message expiry or resource limit intervened, the subscription allows queued delivery. Reconnect after the session expires and the old queue is gone.
Creating a new subscriber after publication is different: it had no prior subscription/session to receive that queued nonretained message. Retained state is associated with a topic and can initialize a new subscriber, even when it has no prior session.
### Compare queued events with retained latest state
The broker session queue and retained-message store answer different questions:
| Mechanism | What it represents | Who can receive it later |
| --- | --- | --- |
| Nonretained QoS 1 publication to an offline session | A one-shot event the broker may queue for an existing matching subscription | That session, while its subscription and expiry remain valid |
| Retained publication | The latest retained value for a topic, replacing the prior retained value | A subscriber that newly subscribes to the topic, including one with a new Client ID |
Run this comparison only against the disposable broker and simulator. First finish the queued-ping experiment and reconnect the original simulator. Keep it online so the retained probe cannot also accumulate in its offline session. Publish a synthetic retained `ping` to its lab command topic with the MQTT 5 CLI and authorized lab backend identity; confirm the Section 7 simulator rejects it before executing it. Then subscribe with an observer using a fresh Client ID and the lab simulator identity; it should receive the stored latest value immediately despite having no prior session. Use no physical device or production credential.
Immediately clear the retained value by publishing an empty payload with RETAIN set to the same topic (Mosquitto CLI uses `-r -n`). Verify a fresh subscription receives no replay. Do not leave retained one-shot commands in the lab. The point is to see that retained delivery initializes the subscriber with the latest value; it is not an event log and must not be used to queue executable commands.
### Put each recovery responsibility in the correct process
| State | Early lesson implementation | Network disconnect | Owning process restart |
| --- | --- | --- | --- |
| Client protocol session | Paho memory state for inflight handshakes | Manager can reuse state subject to broker session | Lost unless persistent client stores are configured |
| Client outgoing queue | Early code uses direct `Publish`, not `PublishViaQueue` | Direct publish can fail while disconnected | No durable outgoing backlog exists in this helper |
| Broker session | Broker memory plus configured persistence, gated by expiry | Can preserve subscription/eligible deliveries | Depends on saved broker state, expiry, and recovery |
| Application records | Go command and dedup maps | Survive while the Go process lives | Lost until database-backed recovery is implemented |
`PublishViaQueue` is a separate AutoPaho API. A nil result means local enqueue succeeded; its API does not return eventual broker/device status. A file-backed outgoing queue and file-backed protocol session stores solve different problems. Neither replaces an application outbox or device deduplication.
<details>
<summary>Session Present is true, but GET command status says unknown. Is the broker broken?</summary>
Not necessarily. The broker resumed its session while the backend's in-memory application records were lost on process restart. MQTT has no knowledge of your HTTP command map. Record both pieces of evidence rather than using Session Present as a general recovery-success flag.
</details>
## 1. Add session controls
**❓ Question:** **What does `Session Present` tell you after CONNACK, and what does it leave unknown?**
**💬 Answer:** It reports whether the broker resumed prior session state for this connection. It does not report whether the backend's command record survived, whether a device acted, or whether every queued application job completed.
Extend the typed configuration with Clean Start and Session Expiry Interval in seconds, and pass them into AutoPaho. Edit `internal/mqtt/client.go` to log `SessionPresent` from CONNACK without logging credentials. Record connection and subscription readiness separately. Read the pinned `autopaho/auto.go`: `CleanStartOnInitialConnection` applies to the first connection of that manager; automatic reconnects behave differently from constructing a new manager.
Use unique stable IDs for backend and simulator. Run one process per client ID. A changing ID creates a different session, while simultaneous clients sharing an ID can take over each other's connection.
Session Expiry is seconds in the MQTT property, whereas Go operation deadlines use `time.Duration`. A nonzero expiry preserves eligible broker session state after disconnect for that interval; Clean Start discards an old session at connection start. Session Present reports whether the broker resumed prior session state, not whether your Go command records survived.
Create the `journal` directory before saving the experiment record. Use Ctrl+C for a graceful simulator shutdown first; log whether DISCONNECT was sent and whether your configured expiry remains nonzero. For an abrupt-process experiment, run a built simulator binary and record its PID, then terminate that exact lab process; killing a `go run` wrapper can make the process under test ambiguous. Record the shutdown method because a graceful disconnect and lost network connection are different evidence.
## 2. Run an experiment matrix
Create `journal/09-sessions.md`. For each row record settings, prediction, exact stop/start steps, CONNACK Session Present, SUBACK, received command IDs, duplicates, and observed result.
| Experiment | Simulator setup | Observation to explain |
| --- | --- | --- |
| Fresh connection | Clean Start true, expiry 0 | No offline-session guarantee |
| Short outage | Clean Start false, expiry 60, stable ID | Reconnect before expiry and inspect queued QoS 1 delivery |
| Expired session | Same settings, outage longer than 60 seconds | Inspect Session Present and missing queued work |
| New identity | Change only client ID | Previous session is not this new client's session |
| Broker restart | Persistent lab broker | Compare network recovery with broker-process recovery |
For offline delivery, first establish the simulator subscription at QoS 1. Disconnect it while preserving a nonzero-expiry session, then publish a nonretained QoS 1 ping from the still-connected backend. Reconnect before expiry using the same ID and Clean Start false. Observe broker evidence; do not infer delivery merely from a successful backend publish. Choose a command response deadline long enough for the experiment, and record late-result policy when exceeding it.
## 3. Audit subscription restoration
The early helper subscribes on every connection-up event. Keeping this policy is acceptable for the experiment; log each SUBACK and do not register another local callback on every reconnect. Test both session-present and fresh-session paths. A connected process whose subscriptions failed is not ready to receive commands.
## 4. Compare broker persistence safely
Use only this project's Compose commands. Keep the default persistence volume intact. For a nonpersistent comparison, stop the lab, temporarily change only `lab/mosquitto/mosquitto.conf` to disable persistence, and use a separate empty lab data volume in a Compose override so old state cannot contaminate the result. Restore the original configuration afterward. Do not delete volumes to run this experiment.
A graceful broker shutdown and an abrupt broker kill may preserve different amounts of recently changed state. Record which you used and the broker's persistence settings rather than generalizing one observation to every crash.
A Compose override is an additional YAML file layered over `compose.yaml` using `-f compose.yaml -f <override-file>` before the subcommand. Create it at the root and use that same pair for config/up/ps/stop commands throughout the comparison. Replace the broker's `/mosquitto/data` source with a new declared named volume; Compose merges mount entries by container destination. Verify the resolved `config` output shows the new source before starting it. Stop the comparison, then run `up -d` with the original file alone to restore the original mounts; `start` would only reuse the comparison container. Refer to [Broker operations](broker-operations.md) for volume identity and recreation.
Turning persistence off does not erase existing in-memory sessions or remove old disk data. Use the separate empty volume and a recreated container to avoid that ambiguity. After restoring settings and mounts, inspect logs and repeat normal authentication/ping checks.
## 5. Separate the recovery layers
Write a table with four columns: client session storage, client outgoing publication queue, broker session state, and application command/dedup state. For each, state current storage, lifetime, and what survives network interruption or process restart. Read the pinned Paho session and AutoPaho queue documentation; neither broker persistence nor QoS 1 makes your in-memory command records durable.
```sh
gofmt -w internal cmd
go test -race ./...
docker compose -p mqtt5-learning restart broker
docker compose -p mqtt5-learning logs --tail=50 broker
```
`restart broker` stops/starts only that project's broker container, interrupting its client connections; it does not apply edited Compose mounts or image settings. `logs --tail=50 broker` inspects the last 50 service log lines. Observe client reconnection and Session Present as well as broker logs. `gofmt` rewrites edited Go formatting; `go test -race` exercises tests with race detection. For mount/config changes requiring recreation, follow the preceding override instructions instead of assuming restart applies them.
**Done when:** Every matrix row has a prediction and measured result, you can distinguish automatic reconnect from a new process, and you can explain each recovery layer. An offline ping delivered after your backend timed out must remain consistent with Section 7's unknown-outcome policy.
Previous: [08-http-commands.md](08-http-commands.md). Next: [Section 10 — Expiry, presence, and retained state](10-expiry-presence-state.md).
teaching/course-source/10-expiry-presence-state.md
+204
View File
@@ -0,0 +1,204 @@
# Section 10 — Expiry, presence, and retained state
Prerequisite: complete the session experiments in Lesson 9. Continue the same lab and pinned Paho v0.23.0. These are incremental exercises; the fragments below belong in your existing transport and simulator, not new standalone programs.
## 1. Give every timer one responsibility
**❓ Question:** **A ping is queued while a device is offline, then reaches it after the operator has changed the desired setting. Which timer can stop stale delivery, and which check must still run on the device?**
**💬 Answer:** MQTT Message Expiry limits how long the broker may forward the publication. The device must also check the application's absolute deadline before starting the action, because work may have spent time in a local queue or arrived just before expiry. Neither timer can reverse an action already started.
| Timer | What expires | What it cannot prove |
| --- | --- | --- |
| Go publish context | Your wait for the publication operation | That the device never received it |
| Application command deadline | Permission to begin a stale action under your contract | That an already-started action was undone |
| MQTT Message Expiry | A publication's remaining delivery lifetime | That a delivered message stops executing |
| Session Expiry | Broker session after disconnect | That application records were saved |
| Keep Alive | Acceptable MQTT connection inactivity | That the application is healthy |
| Will Delay | Delay before an eligible Will is published | That the device is physically broken |
A vending configuration command that was useful five minutes ago may be wrong now. Use Message Expiry to limit stale delivery and an application deadline to reject stale execution after delivery. A response timeout merely tells the caller it lacks a result. A PUBACK can arrive before the simulator validates the command; if the application rejects it, that MQTT acknowledgement remains valid for the protocol step it acknowledged.
This fragment builds MQTT metadata for a five-second publication lifetime:
```go
expirySeconds := uint32(5)
properties := &paho.PublishProperties{
ContentType: "application/json",
MessageExpiry: &expirySeconds,
}
```
Attach `properties` to the outbound PUBLISH. A nil MessageExpiry omits the property; an explicit zero is not an unlimited lifetime. The broker reduces the remaining lifetime when forwarding queued work. Time spent waiting in your own outgoing queue needs a separate policy: check the absolute application deadline before transmission and execution. Do not reissue an old command with a fresh five seconds indefinitely.
**Experiment:** preserve the simulator session for 60 seconds, disconnect it, publish one five-second QoS 1 ping, and reconnect after ten seconds. Expect no expired command delivery. Repeat with a much longer message expiry and reconnect within it: expect delivery if the other session/ACL prerequisites hold. Log IDs and timestamps, not just a message count. Allow timing margin around expiry boundaries.
## 2. Retained state is a latest value, not an event history
**❓ Question:** **A new simulator subscribes and receives a retained `dispense` command from last week. What design choice caused this replay?**
**💬 Answer:** A one-shot action was published with RETAIN set. Retained publications are stored by topic for future subscribers, so use retained messages for replaceable state such as desired configuration and nonretained messages for one-time actions.
A retained publication replaces the stored value for one exact topic. It can initialize a newly created subscription even when that client has never connected before. Offline session queues instead belong to an existing session and its subscriptions. Unsubscribing or deleting a session does not delete a retained value.
Use a desired-configuration topic for state such as revision 7 and a temperature setting. Use nonretained command topics for one-shot actions. A retained "dispense" would risk replay when someone subscribes later.
Extend the ACL and the shared topic contract before using these flows: backend writes `lab/v1/devices/+/desired`, simulator reads its exact `.../device-001/desired`; simulator writes `.../device-001/reported`, backend reads `.../+/reported`. The complete `topic.go` update below adds those flow kinds to `DeviceTopic`, `DeviceFilter`, `DeviceSubscription`, and `ParseTopic`. Reload/restart the lab and verify both allowed and denied directions; the broker ACL remains its own source of authorization.
Implement desired and reported revision records. Store the highest applied revision in the simulator and explain the initial in-memory restart limit. Receiving revision 6 after 7 must not overwrite 7. A backend PUBACK for revision 7 proves neither application nor reported revision 7; wait for the device's report.
### Clear the retained telemetry from Lesson 4
Stop the simulator's periodic telemetry first so it cannot replace the value during your observation. Run this with the simulator's temporary lab password, using the same CLI credential caveat as Lesson 1:
```sh
mosquitto_pub -h 127.0.0.1 -p 18883 -V mqttv5 -u simulator -P '<simulator-lab-password>' -t 'lab/v1/devices/device-001/telemetry' -q 1 -r -n -d
```
`-r` sets RETAIN; `-n` supplies a zero-byte payload. Together they delete the retained value at that exact topic. An empty JSON object or the two characters `""` are nonempty payloads and do not delete it. Existing subscribers can see the empty publication; a fresh subscriber should receive no retained replay. Removing the stored value does not retract copies already delivered or queued elsewhere. Restore ordinary nonretained telemetry after checking.
## 3. Model presence as evidence
**❓ Question:** **The broker's retained presence says `online`, but the last heartbeat is old. What can the backend safely conclude?**
**💬 Answer:** It can conclude only that the broker currently stores an online assertion. It cannot conclude the device is healthy now. Track backend receipt time and heartbeat freshness, and compare a connection or boot identity so delayed state does not replace newer evidence.
The simulator supplies its Will during CONNECT. The broker later publishes it when the connection ends in a way that requires a Will; it is not code that the dead simulator executes. First add simulator write/backend read ACLs for its `.../presence` topic, then extend the allowlist in `internal/contracts/topic.go`. Keep using `DeviceTopic` so the Will destination follows the same validated hierarchy as ordinary publishes.
Replace `internal/contracts/topic.go` with this complete version. It extends the allowlist with the flows used in this lesson while keeping the existing hierarchy and constructors:
```go
package contracts
import (
"fmt"
"strings"
)
type TopicName string
type TopicFilter string
type Kind string
const (
KindTelemetry Kind = "telemetry"
KindCommands Kind = "commands"
KindResponses Kind = "responses"
KindSales Kind = "sales"
KindPresence Kind = "presence"
KindDesired Kind = "desired"
KindReported Kind = "reported"
)
const topicPrefix = "lab/v1/devices/"
type Topic struct {
DeviceID string
Kind Kind
}
func (t TopicName) String() string { return string(t) }
func (f TopicFilter) String() string { return string(f) }
func DeviceTopic(deviceID string, kind Kind) (TopicName, error) {
if !validID(deviceID) { return "", fmt.Errorf("invalid device ID %q", deviceID) }
if !supportedKind(kind) { return "", fmt.Errorf("unsupported topic kind %q", kind) }
return TopicName(topicPrefix + deviceID + "/" + string(kind)), nil
}
func DeviceFilter(kind Kind) (TopicFilter, error) {
if !supportedKind(kind) { return "", fmt.Errorf("unsupported topic kind %q", kind) }
return TopicFilter(topicPrefix + "+/" + string(kind)), nil
}
func DeviceSubscription(deviceID string, kind Kind) (TopicFilter, error) {
topic, err := DeviceTopic(deviceID, kind)
if err != nil { return "", err }
return TopicFilter(topic.String()), nil
}
func ParseTopic(value string) (Topic, error) {
if !strings.HasPrefix(value, topicPrefix) {
return Topic{}, fmt.Errorf("topic must be %s{deviceID}/{kind}", topicPrefix)
}
parts := strings.Split(strings.TrimPrefix(value, topicPrefix), "/")
if len(parts) != 2 || !validID(parts[0]) {
return Topic{}, fmt.Errorf("topic must be %s{deviceID}/{kind}", topicPrefix)
}
kind := Kind(parts[1])
if !supportedKind(kind) { return Topic{}, fmt.Errorf("unsupported topic kind %q", kind) }
return Topic{DeviceID: parts[0], Kind: kind}, nil
}
func supportedKind(kind Kind) bool {
switch kind {
case KindTelemetry, KindCommands, KindResponses, KindSales,
KindPresence, KindDesired, KindReported:
return true
default:
return false
}
}
func validID(value string) bool {
if value == "" || len(value) > 128 { return false }
for _, r := range value {
if !(r >= 'a' && r <= 'z' || r >= 'A' && r <= 'Z' || r >= '0' && r <= '9' || strings.ContainsRune("._-", r)) { return false }
}
return true
}
```
All three builders now accept these new kinds through the shared allowlist. Do not add arbitrary user-supplied path segments or wildcards to `DeviceTopic`.
Build the will topic before creating the simulator connection, then add these fields to its AutoPaho configuration. The surrounding `run` function already imports `contracts`:
```go
presenceTopic, err := contracts.DeviceTopic(cfg.MQTT.DeviceID, contracts.KindPresence)
if err != nil { return fmt.Errorf("build presence topic: %w", err) }
WillMessage: &paho.WillMessage{
Topic: presenceTopic.String(),
QoS: 1,
Retain: true,
Payload: []byte(`{"status":"offline"}`),
},
WillProperties: &paho.WillProperties{
WillDelayInterval: paho.Uint32(5),
ContentType: "application/json",
},
```
`paho.Uint32(5)` supplies the optional MQTT property as a pointer; the unit is seconds. This is a minimal presence payload for the experiment. Extend it with your schema and a connection/boot identity for the real lesson implementation. A Will payload is prepared at connection time: a timestamp embedded there is not the later time of disconnection. Record backend receipt time separately.
Publish retained online presence after successful simulator startup. Publish an explicit retained offline value before a planned disconnect if observers need that signal. Normal DISCONNECT suppresses the Will, so otherwise a retained online value can remain after a clean shutdown. A special MQTT 5 disconnect reason can request the Will, but the early helper uses the normal path.
Will publication occurs when its delay elapses or the session ends, whichever happens first. With Session Expiry 0, ending the session can remove the intended five-second grace period. With a preserved session, resuming that session before the delay ends can suppress the pending Will. Test with stable identity, Clean Start false, and a session expiry longer than the delay.
### Separate experiments that look similar
| Action | What it tests | Prediction |
| --- | --- | --- |
| Graceful simulator shutdown | DISCONNECT and explicit offline policy | Normal disconnect does not itself publish the Will |
| Abruptly terminate the exact simulator binary | Lost client without DISCONNECT | Broker may publish Will after detection and applicable delay |
| Reconnect into preserved session before Will Delay | Transient outage | Pending Will can be suppressed |
| Stop broker | Broker availability | A stopped broker cannot publish Wills while it is down |
| Keep MQTT traffic alive but stall heartbeat production | Application health | Keep Alive can succeed while freshness fails |
For an abrupt test, build a dedicated lab binary with `go build -o /tmp/mqtt5-lab-simulator ./cmd/simulator`, run it with your usual simulator flags, and record its PID. Send `kill -KILL <recorded-lab-PID>` from another terminal only after verifying that PID. KILL prevents Go cleanup and DISCONNECT; it is different from Ctrl+C. Do not target all Go processes or assume a `go run` wrapper is the actual client. Restart the binary afterward and inspect both retained presence and fresh heartbeats.
A killed process usually closes its socket promptly. It does not reproduce a silent network black hole that must be detected through Keep Alive. Record the actual fault injected and avoid claiming the measured delay is a universal offline-detection time.
## 4. Explain the result before moving on
<details>
<summary>A fresh subscriber receives retained online, but no heartbeat follows. Is the machine healthy?</summary>
You only know the broker stored an online assertion. The producer may have exited cleanly without replacing it, or the retained value may be old. Combine presence with receipt time and heartbeat freshness; use connection/revision identity to reject stale evidence.
</details>
**Done when:** demonstrate an expired offline command, retained replacement and deletion, graceful versus abrupt presence behavior, and rejection of an older desired revision. Record predictions and observations in `journal/10-expiry-presence.md`.
Sources: [MQTT 5 packet and session rules](https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html), [Mosquitto publication flags](https://mosquitto.org/man/mosquitto_pub-1.html), and local `references/paho.golang-v0.23.0/paho/cp_publish.go`, `paho/cp_connect.go`, and `autopaho/auto.go`.
Previous: [Lesson 9](09-sessions-recovery.md). Next: [Lesson 11](11-security-boundaries.md).
teaching/course-source/11-security-boundaries.md
+79
View File
@@ -0,0 +1,79 @@
# Section 11 — Understand MQTT security boundaries
Continue the same isolated lab. This lesson explains the MQTT security design and gives integration exercises; certificate provisioning depends on your chosen lab hostname. Keep remote deployment separate until these checks work.
## 1. Trace the checks in order
**❓ Question:** **A device publishes `{"device_id":"device-001"}` on another device's topic. Which component can establish who sent it, and which component rejects the mismatch?**
**💬 Answer:** The broker authenticates the MQTT client and applies that client's topic ACL before routing. Paho receives the packet and invokes the handler; the application then validates topic and payload identity agreement. The JSON field is a claim, not proof of identity. If the topic is allowed for the authenticated client but the body conflicts, the application rejects it. A prior PUBACK or other MQTT acknowledgement does not get reversed by that application rejection.
```text
TCP → TLS certificate verification → MQTT CONNECT authentication
→ topic authorization → application identity/schema validation
→ permission to perform the business operation
```
TLS protects the connection and verifies the server identity. A username/password authenticates an MQTT client. ACLs restrict topic actions. Application validation checks the payload. HTTP user authorization determines whether an API caller may command that device. Passing one check never substitutes for the next.
A Client ID is session identity, not a secret. A payload `device_id` is a claim. The broker normally does not attach the authenticated publisher username to each forwarded PUBLISH, so the backend cannot authenticate the sender merely by reading a JSON field. Bind a unique device credential to that device's permitted topics and validate topic/body agreement.
## 2. Configure verified TLS in Go
**❓ Question:** **The CA is trusted, but the certificate is for `broker.lab` and the client connects to `127.0.0.1`. Why can verification still fail?**
**💬 Answer:** Trust-chain validation and server-name verification are separate checks. The certificate must include the DNS name or IP the client verifies. Fix the URL or certificate identity; do not disable verification.
Provision a lab CA and server certificate for the exact DNS name or IP used in the client URL. An IP connection needs that IP in the certificate's Subject Alternative Name. Keep the CA private key outside the broker; the broker needs its server key and certificate. Mount files with broker-readable permissions. Configure Mosquitto's TLS listener using `certfile` and `keyfile`; require client certificates only when deliberately introducing mutual TLS.
This fragment belongs in your configuration/transport setup and requires `crypto/tls`, `crypto/x509`, `fmt`, and `os`. `caPath` is a configured path to the public lab CA certificate and `brokerHostname` is the validated URL hostname (without a port):
```go
caPEM, err := os.ReadFile(caPath)
if err != nil {
return fmt.Errorf("read lab CA: %w", err)
}
roots := x509.NewCertPool()
if !roots.AppendCertsFromPEM(caPEM) {
return fmt.Errorf("lab CA contains no usable certificate")
}
tlsConfig := &tls.Config{
RootCAs: roots,
ServerName: brokerHostname,
MinVersion: tls.VersionTLS12,
}
```
Assign this configuration to AutoPaho's `TlsCfg` and use a `tls://` broker URL for the TLS listener. Set `ServerName` explicitly from the validated URL host so certificate verification also has the intended identity on the pinned proxy transport path. Do not disable verification to make a hostname failure disappear; correct the certificate or URL. Update Lesson 3's URL validation deliberately: its early allowlist accepts only the plaintext `mqtt://` scheme.
Server-only TLS can still use the existing MQTT credentials and ACLs. Mutual TLS additionally authenticates a client certificate; mapping certificate identity to topic permissions is another deliberate broker policy. Do not assume enabling encryption creates per-device authorization.
Use the official [Mosquitto TLS configuration reference](https://mosquitto.org/man/mosquitto-conf-5.html) and [Go TLS configuration API](https://pkg.go.dev/crypto/tls#Config) while provisioning. Match broker options to the pinned 2.0.22 image.
## 3. Test failures at the intended layer
| Change one input | Expected failure layer | Evidence to collect |
| --- | --- | --- |
| Wrong CA | TLS verification | No authenticated MQTT session established |
| Valid CA, wrong server name | TLS identity verification | Hostname/IP mismatch |
| Valid TLS, wrong MQTT password | CONNECT authentication | Connection error / unsuccessful CONNACK |
| Valid login, other device's publication topic | Topic authorization | Rejection/delivery denial plus authorized observer and broker logs |
| Permitted topic, mismatched body device | Application validation | Rejection before any state mutation |
| Valid API user, unauthorized target device | HTTP business authorization | Denial before creating/publishing a command |
Never fix an ACL test by replacing narrow permissions with `#`. A successful broad SUBACK alone is not proof of cross-device access: test actual delivery. The device or simulator checks that a command's Response Topic is exactly the response destination authorized for that device before publishing. The command service separately matches the resulting application response to its pending command. Correlation Data identifies the request to match; it does not authorize the destination or authenticate the publisher.
## 4. Rotate credentials without confusing state
Updating a password file affects authentication policy; it does not prove every existing connection was disconnected. A fresh connection using the old password must fail, a new one must succeed, and the device must regain its subscriptions. [Broker operations](broker-operations.md) explains which credential volume this workspace actually uses and why editing the old host copy does nothing.
Treat MQTT 5 enhanced authentication (`AUTH` plus an agreed authentication method) as an optional separate extension. It requires compatible client and broker mechanisms; adding User Properties or renaming a username does not enable it. The two-user password-file lab does not demonstrate enhanced authentication.
<details>
<summary>Why is TLS with a shared fleet password still insufficient for per-device isolation?</summary>
TLS protects traffic, but a shared credential does not distinguish individual devices for authorization. A compromised device can exercise that shared identity's permissions. Use separate identities and topic policies, then validate application claims independently.
</details>
**Done when:** explain every boundary, run the failure matrix against the lab, and show that a denied user or device cannot produce a simulated effect. Continue with Section 12 in [COURSE.md](../../COURSE.md); the [advanced MQTT lesson](16-mqtt5-controls.md) belongs after the persistence/process exercises.
teaching/course-source/11b-dynamic-security.md
+166
View File
@@ -0,0 +1,166 @@
# 🔐 Lesson 11B — Provision device ACLs with Mosquitto Dynamic Security
This is an optional implementation extension to [Section 11](11-security-boundaries.md). It replaces the lab's static password and ACL files with Mosquitto's Dynamic Security plugin in a dedicated lab configuration. Do not combine the two authentication systems casually: first understand which component owns each account and authorization decision.
## 🎬 Start with the provisioning story
> **❓ Question:** **A new device is installed after the broker has started. How can it receive its own credentials and topic permissions without editing files and reloading Mosquitto?**
> **💬 Answer:** A controlled provisioning operator or service authenticates with a separate management identity and asks the broker plugin to create the device client and assign least-privilege roles. The device then connects with its own credentials. The broker checks those credentials and topic permissions; the Go application still validates the delivered topic and payload.
A provisioning service (or a controlled operator job) authenticates to the broker through a separate management identity. It asks the plugin to create/enable the device client and attach least-privilege roles. The device then connects with its own credentials and can use only the topics those roles permit.
```mermaid
sequenceDiagram
participant P as Provisioning service
participant B as Mosquitto + Dynamic Security
participant D as Device
participant A as Go application
P->>B: Admin MQTT connection (TLS for remote use)
P->>B: Create device client; assign device role
B-->>P: Management result
D->>B: CONNECT with device credentials
B->>B: Authenticate client and check ACLs
D->>B: PUBLISH only to its permitted device topic
B->>A: Route topic and payload to backend subscription
A->>A: Validate topic, payload, and business rules
```
The Go backend does **not** learn the publisher username from an ordinary forwarded `PUBLISH`. The broker enforces the credential-to-topic boundary before routing. The backend still checks that the topic's device ID agrees with the payload and validates the message contract before acting.
Dynamic Security is a Mosquitto plugin, not a Go MQTT library feature. It manages broker clients, groups, roles, and ACLs through a special MQTT control API. `mosquitto_ctrl` is the command-line client for that API. Changes apply while the broker runs and are saved to the plugin's `dynamic-security.json` state file. [Official Dynamic Security guide](https://mosquitto.org/documentation/dynamic-security/)
## 🧱 1. Give the plugin its own broker configuration and state
> **❓ Question:** **What must change before the broker can accept live account-management requests?**
> **💬 Answer:** Mosquitto must load its Dynamic Security plugin and a writable, persistent plugin state file. The static password and ACL files do not automatically become plugin-managed accounts.
The current lab uses Mosquitto 2.0.22 with `password_file` and `acl_file`; it does **not** load Dynamic Security. Keep the working lab intact while learning. For this exercise create a separate Compose override/configuration and state volume, and mount `dynamic-security.json` at a persistent, broker-writable path. The broker image/version must include the plugin; verify the library path inside the pinned image rather than assuming a distribution path.
The plugin's broker configuration has this shape (replace the library path with the verified path in your image):
```conf
listener 1883
allow_anonymous false
per_listener_settings false
plugin /verified/path/mosquitto_dynamic_security.so
plugin_opt_config_file /mosquitto/data/dynamic-security.json
persistence true
persistence_location /mosquitto/data/
log_dest stdout
connection_messages true
```
For this listener, Dynamic Security supplies both username/password authentication and topic authorization. Do not also configure `password_file` or `acl_file` for the same identities as if they were one unified policy; choose a clear owner for authentication and ACL decisions. The JSON state path must be writable by the broker process and backed by persistent storage with restricted permissions.
Mosquitto 2.0 requires an initial plugin state file. Generate one with the matching `mosquitto_ctrl` utility, for example:
```sh
mosquitto_ctrl dynsec init ./lab/dynamic-security.json dynsec-admin
```
This creates the initial administrator client interactively. Protect that state file: it contains security configuration and credential hashes. Keep a restricted backup, and test restore as part of broker operations. Do not commit the generated file or put it in an image layer. Newer Mosquitto versions may initialize a missing file differently; follow the documentation for the exact pinned broker version.
Start only the isolated Dynamic Security lab and verify the plugin loaded in broker logs before creating device accounts. If startup fails, check plugin path, file mount, writable ownership, and version compatibility. Do not fall back to anonymous access to make the experiment pass.
## 🪪 2. Model identity and reusable permissions
> **❓ Question:** **How can a rule be reusable for many devices without giving them access to one another's topics?**
> **💬 Answer:** Give each device a unique authenticated username and use `%u` only as a complete topic segment in role ACL patterns. The broker substitutes the authenticated username when it evaluates the rule. A payload ID or client-selected Client ID is not a substitute for that credential binding.
Use one unique Dynamic Security client per device username, and bind its configured client ID when the device has a stable ID and the client library can consistently use it. A username is the plugin's primary client lookup key; a client ID alone is not authentication. Never share one fleet credential between devices.
Create a device role whose ACL patterns substitute `%u` with the authenticated username. In this example the username itself is the device segment, such as `device-001`:
```text
Role: device
publishClientSend lab/v1/devices/%u/telemetry allow
publishClientSend lab/v1/devices/%u/responses allow
subscribePattern lab/v1/devices/%u/commands allow
publishClientReceive lab/v1/devices/%u/commands allow
```
The separation is intentional: `publishClientSend` controls device-to-broker publications; `subscribePattern` controls what filters a device may request; `publishClientReceive` controls what broker-forwarded publications it may receive. A permitted subscription by itself is not always the complete receive policy.
Create a separate backend role. The backend may publish commands and subscribe to device telemetry/responses, but it should not use the device role. Keep the management administrator separate from both roles.
Dynamic Security's initial defaults are not deny-all: publishing to the broker and subscribing are denied by default, while receiving broker-forwarded messages is allowed by default. Set `publishClientReceive` to deny explicitly, then grant only the device's command topic through the role. Add only explicit patterns needed by each role; avoid broad grants such as application-wide `#`. Review ACL event types and wildcard behavior in the official guide before relying on a pattern.
## 🛠️ 3. Create a role and assign a device
> **❓ Question:** **What exact broker changes happen when inventory says device-001 is ready to join?**
> **💬 Answer:** The operator creates a client keyed by its unique username, binds the intended Client ID when required, and assigns the reusable device role. The device's own credential then selects that client's policy at CONNECT.
The following illustrates the `mosquitto_ctrl dynsec` command family. `mosquitto_ctrl` connects to the running broker using a separate administrator account; the device credential being created is not the admin credential.
```sh
# Create a reusable device role once.
mosquitto_ctrl -h 127.0.0.1 -p 18883 -u dynsec-admin -P "$DYNSEC_ADMIN_PASSWORD" \
dynsec createRole device
# Make receive permissions default-deny; the device role grants its command topic below.
mosquitto_ctrl -h 127.0.0.1 -p 18883 -u dynsec-admin -P "$DYNSEC_ADMIN_PASSWORD" \
dynsec setDefaultACLAccess publishClientReceive deny
# Add each role ACL once. Repeat only for ACLs your design actually needs.
mosquitto_ctrl -h 127.0.0.1 -p 18883 -u dynsec-admin -P "$DYNSEC_ADMIN_PASSWORD" \
dynsec addRoleACL device publishClientSend 'lab/v1/devices/%u/telemetry' allow 10
mosquitto_ctrl -h 127.0.0.1 -p 18883 -u dynsec-admin -P "$DYNSEC_ADMIN_PASSWORD" \
dynsec addRoleACL device publishClientSend 'lab/v1/devices/%u/responses' allow 10
mosquitto_ctrl -h 127.0.0.1 -p 18883 -u dynsec-admin -P "$DYNSEC_ADMIN_PASSWORD" \
dynsec addRoleACL device subscribePattern 'lab/v1/devices/%u/commands' allow 10
mosquitto_ctrl -h 127.0.0.1 -p 18883 -u dynsec-admin -P "$DYNSEC_ADMIN_PASSWORD" \
dynsec addRoleACL device publishClientReceive 'lab/v1/devices/%u/commands' allow 10
# Provision the per-device client, then attach the reusable role.
mosquitto_ctrl -h 127.0.0.1 -p 18883 -u dynsec-admin -P "$DYNSEC_ADMIN_PASSWORD" \
dynsec createClient device-001 -i lab-device-001
mosquitto_ctrl -h 127.0.0.1 -p 18883 -u dynsec-admin -P "$DYNSEC_ADMIN_PASSWORD" \
dynsec addClientRole device-001 device 10
```
The tool prompts for the new device password during client creation. Check the exact prompts/options with the pinned `mosquitto_ctrl` version. **Do not put real admin passwords in command arguments**: arguments may appear in process listings or shell history. The `-P "$DYNSEC_ADMIN_PASSWORD"` form above is a disposable-lab placeholder to make the connection explicit, not a production secret-delivery pattern. In a real deployment, run the controller from a restricted job/service and deliver secrets through the platform's protected credential mechanism; use TLS for remote management connections.
To remove the device's access, disable it immediately; delete it when the account is no longer needed:
```sh
mosquitto_ctrl -h 127.0.0.1 -p 18883 -u dynsec-admin -P "$DYNSEC_ADMIN_PASSWORD" \
dynsec disableClient device-001
```
Disabling/deleting a client disconnects its currently connected sessions and prevents future authentication. Record the provisioning request, target device, operator/service identity, result, and time. Make operations idempotent: a retry should detect an existing client/role rather than accidentally broadening permissions.
## 🧪 4. Prove both sides of the boundary
> **❓ Question:** **How will we know the plugin blocks cross-device access instead of merely starting successfully?**
> **💬 Answer:** Exercise both allowed and denied paths using the actual per-device credentials, then inspect reason codes, delivery, and broker logs. Startup only proves the plugin loaded; it does not prove the topic policy is correct.
Use separate device and backend credentials and test both permitted and denied operations:
1. Connect `device-001` with its configured username, password, and client ID. Publish telemetry to `lab/v1/devices/device-001/telemetry`; expect the broker to accept it.
2. Try publishing as that same client to `lab/v1/devices/device-002/telemetry`; expect the broker to deny it.
3. Try subscribing to `lab/v1/devices/device-002/commands`; expect the subscription result to show rejection.
4. Publish an allowed topic with a body claiming a different `device_id`; confirm the Go backend rejects the topic/body mismatch before changing state.
5. Disable the client while it is connected. Confirm the broker disconnects it and a later reconnect is rejected.
6. Restart the broker and verify the client/role/ACL state remains, demonstrating that the plugin state file is persisted.
Capture broker logs and client-side MQTT reason codes where available. A command's successful exit or a loaded plugin alone is not proof that the ACL is correct. Restore the isolated lab to its prior state when finished.
## 🏗️ Production design checklist
- Keep device credentials unique, revocable, rotated, and delivered through a protected provisioning channel.
- Keep Dynamic Security administrator credentials out of device firmware and ordinary message-processing code. A dedicated provisioning service/job with narrowly controlled access is usually easier to audit than giving the backend's main runtime account administrator power.
- Expose the broker management listener only where needed; require TLS and restrict network reachability. On a local-only lab, loopback access is appropriate.
- Grant per-role least privilege, separate publish from subscribe/receive, and review every wildcard. Keep default-deny behavior.
- Use a stable identity mapping. `%u` is useful only when the authenticated username is unique and provisioned to represent that device; never trust an arbitrary client-selected topic or JSON ID as identity proof.
- Persist and back up Dynamic Security state securely. Test recovery, broker upgrades, plugin compatibility, revocation, and credential rotation.
- Audit provisioning changes and alert on repeated denied access. Protect the audit stream from modification by device clients.
- Consider an external authentication/authorization service when policy must integrate with a device registry or central identity system. Dynamic Security is a broker-local role system, not a device inventory database.
**Done when:** You can explain who authenticates a device, which component decides its topic permissions, what the backend can infer from the delivered topic, and how revocation affects an already-connected client.
**Official references:** [Dynamic Security plugin](https://mosquitto.org/documentation/dynamic-security/), [Mosquitto authentication methods](https://mosquitto.org/documentation/authentication-methods/), [Dynamic Security `mosquitto_ctrl` commands](https://www.mosquitto.org/man/mosquitto_ctrl_dynsec-1.html).
teaching/course-source/12-vending-workflows.md
+110
View File
@@ -0,0 +1,110 @@
# 🥤 Lesson 12 — Implement representative vending workflows
## 🎬 Start with the system story
**❓ Question:** **A vending machine reports a sale, receives a new catalog, and asks for configuration. Which side owns each fact, and which MQTT message proves that a change reached the device?**
**💬 Answer:** The device owns what it observed or applied locally; the backend owns the accepted business record and the version it intends the device to use. A broker PUBACK proves only a QoS delivery hop. The device must publish an application result or reported revision to show what it accepted or applied.
```mermaid
flowchart LR
D[Device simulator] -->|sales report / health / logs| B[Mosquitto]
B -->|delivered publication| I[Backend ingestion]
I -->|validate, deduplicate, record| S[In-memory state]
A[Backend workflow] -->|catalog or config request| B
B -->|command / desired state| D
D -->|accepted / completed / reported revision| B
```
This lesson extends the contracts and command/response flow with synthetic vending data. Keep all data disposable and simulated. Do not connect the exercises to production devices, payment processing, or a production database.
## 💰 1. Ingest and deduplicate sales
> **❓ Question:** **The simulator republishes report `sale-0042` after losing its response. How do you avoid counting the same sale twice?**
The backend parses the topic and validates the report as Lesson 5 established. Then it uses a stable application identity such as `(device_id, report_id)` to detect repeats. An MQTT packet identifier is scoped to a protocol exchange and is not a durable sale ID.
For this lesson, keep the deduplication map in memory and make its limitation explicit: a process restart forgets the map. For each new report, record the report ID, its accepted/rejected result, and a synthetic effect count. For a duplicate, return or log the prior outcome without repeating the effect. Keep malformed or mismatched reports out of business state.
The simulator may publish an application-level receipt keyed by report ID. That receipt can mean “the backend saved this report in its current state”; it must not imply durable storage until Lesson 13 adds PostgreSQL. MQTT PUBACK and the report receipt are separate signals.
### 🧪 Predict, then try it
1. Publish the same valid report twice with the same report ID. Predict the effect counter and the two MQTT deliveries.
2. Repeat with a changed amount but the same ID. Define whether the backend rejects ID reuse with different content or returns the original result; do not silently apply a second meaning.
3. Restart the backend and replay the report. Observe that the in-memory implementation cannot deduplicate across restart.
**Expected evidence:** Two broker deliveries, one business effect before restart, and a documented loss of deduplication knowledge after restart.
## 📦 2. Deliver a versioned product catalog
> **❓ Question:** **If a device receives catalog revision 8 after revision 9, should it replace its current catalog?**
Treat a catalog as versioned application data. Give it a schema version, a stable message ID, a catalog/job ID, and a monotonically increasing revision. The backend publishes the intended catalog; the simulator validates the topic and payload, compares revisions, and reports the revision it applied. Older revisions must not overwrite newer local state.
Use a nonretained command when each delivery represents a one-time operation. If the catalog is modeled as current desired state, Lesson 10's retained-state rules may be more appropriate: publish a retained snapshot keyed by device, and let the device report the revision it has applied. Decide which model you are testing; do not mix “deliver this job once” with “this is the latest state.”
```text
backend publishes catalog revision 9
broker PUBACK → confirms backend-to-broker QoS hop only
device receives catalog → validates and compares revision
device publishes applied revision 9 → application evidence
```
An application response should include the job or catalog ID and revision, so the backend can match it to the intended update. Do not claim that a successful publish or PUBACK means the catalog was installed.
### 🧪 Predict, then try it
1. Publish revision 9, then revision 8. Predict the simulator's reported revision.
2. Send revision 9 twice with the same job ID. Confirm the second delivery does not reapply the update.
3. Restart the simulator and explain which revision state is lost in this in-memory stage.
## ⚙️ 3. Model configuration fetch and update
> **❓ Question:** **What should happen when a device asks for configuration while the backend is preparing a newer revision?**
Give each configuration snapshot a device identity, revision, and schema version. For device-initiated fetch, the device publishes a request with a request ID; the backend replies on an authorized response destination or the defined device configuration topic. For a backend-initiated update, the backend publishes the desired revision and the device reports the applied revision.
Define a conflict rule before coding. A simple lab rule is “accept only revisions newer than the simulator's current revision.” If requests can race with updates, include the request ID or expected revision in the response and discard stale responses. Correlation Data can associate an MQTT response with a request, but it is not authorization and does not replace the application revision.
```text
device → config/request (request_id, known_revision)
backend → validate request and select current revision
backend → config/response (request_id, revision, settings)
device → validate request_id and revision; apply only if current
device → reported revision (application confirmation)
```
Use synthetic settings only. Never put credentials or secrets in a retained configuration message; retained data may be replayed to later subscribers.
## ❤️ 4. Add health and bounded logs
> **❓ Question:** **Does a recent heartbeat prove that a machine is healthy and ready to vend?**
No. Keep three observations distinct: MQTT connection status, recent application heartbeat, and device-reported health. A connected client can be malfunctioning; a fresh heartbeat can report an error; a Will indicates an unexpected connection loss but does not diagnose the device.
For log events, bound payload size and event frequency, sanitize control characters and secrets, and avoid logging full untrusted payloads. Decide which events are disposable telemetry and which require durable retention. Section 13 introduces durable persistence for important reports.
## 🔎 5. Compare with the existing backend
Use the existing Robot Market backend as a read-only reference. Trace the source path for sales ingestion, catalog/config publication, and device health. Record the actual topic, contract identity, acknowledgement, retry, and deduplication behavior you observe. Do not infer that a field or identifier has the same meaning as the lab just because the names look similar.
### 🧪 Review questions
1. Why is an MQTT packet ID unsuitable as a sales report ID?
2. What does a broker PUBACK prove in the catalog flow, and what proves device application?
3. How does the simulator prevent an older revision from replacing a newer one?
4. Which state disappears when the backend or simulator restarts at this stage?
5. Why are connection status, heartbeat freshness, and device health different signals?
> [!answer]- 📋 Short answers
>
> 1. Packet IDs are protocol-scoped and reusable; business identity must remain stable across retries and restarts.
> 2. PUBACK proves the publisher-to-broker QoS exchange; a validated device result or reported revision is application evidence.
> 3. It stores the highest applied revision and rejects older revisions.
> 4. In-memory deduplication, business effects, command state, and applied simulator revision.
> 5. Each measures a different boundary: transport connection, recent application message, and the device's own reported condition.
**Done when:** A duplicate sales report causes one in-memory effect, stale catalog/config revisions are rejected, and your notes distinguish transport acknowledgement from device application evidence.
**Next:** [Lesson 13 — Persist reports and command state](13-persist-reports-command-state.md) replaces restart-losing business state with PostgreSQL transactions and durable MQTT admission.
teaching/course-source/13-persist-reports-command-state.md
+131
View File
@@ -0,0 +1,131 @@
# 🗄️ Lesson 13 — Persist reports and command state in PostgreSQL
## 🎬 Start with the system story
**❓ Question:** **The backend receives a QoS 1 sales report, commits it to PostgreSQL, then crashes before acknowledging it. What should happen when the broker delivers it again?**
**💬 Answer:** The database must recognize the stable report identity and avoid applying the sale twice. The backend can then acknowledge the repeated delivery and return the previously recorded application outcome. MQTT manages protocol delivery; PostgreSQL owns durable business state.
```mermaid
sequenceDiagram
participant D as Device
participant B as Broker
participant I as Backend ingestion
participant DB as PostgreSQL
D->>B: QoS 1 PUBLISH report_id
B->>I: QoS 1 PUBLISH
I->>I: Parse topic and validate payload
I->>DB: Transaction: dedup key + business effect
DB-->>I: COMMIT
I->>B: PUBACK
Note over I,B: Crash before PUBACK can cause a duplicate delivery
```
The acceptance boundary for important work is a successful database commit. The callback may hand work to a bounded worker, but keep the original Paho packet and client available until acknowledgement. A payload copy alone cannot be acknowledged later.
## 📥 1. Decide when to acknowledge
> **❓ Question:** **What does Paho do if a callback returns an error? Does that tell the broker to retry?**
For the pinned Paho v0.23.0 client, automatic acknowledgement sends PUBACK for a QoS 1 message after the callback returns—even if it returned an error. The error is local to the callback chain; it is not an MQTT negative acknowledgement or a retry request. If the callback enqueues the message in RAM and returns, the broker may consider the backend's delivery complete before a worker processes it.
With `EnableManualAcknowledgment`, Paho does not automatically acknowledge a QoS 1 message when the callback returns. Your application calls `received.Client.Ack(received.Packet)` after reaching its policy boundary. For reports and command results, use this order:
1. Parse the MQTT topic and validate the message contract (Lesson 5).
2. Submit the message to a bounded worker queue while retaining its packet/client acknowledgement handle.
3. In one PostgreSQL transaction, record the stable identity, apply the business effect or command-state transition, and store the application outcome.
4. Commit the transaction.
5. Call `Ack` and check its error.
If the process crashes after commit but before PUBACK, the broker may deliver the message again. The unique constraint makes the repeated transaction a no-op for business effects; the backend acknowledges after it recognizes the duplicate. If it crashes before commit, there is no durable admission, so the message must remain recoverable from the MQTT session/broker according to QoS, expiry, and persistence settings.
### ❓ What does a callback error mean in each mode?
| Mode | Callback returns `nil` | Callback returns an error | What causes a retry? |
| --- | --- | --- | --- |
| Automatic | Paho sends PUBACK after callback return | Paho still sends PUBACK; error does not request redelivery | Not the callback error. The delivery is complete at the protocol layer. |
| Manual | Paho does not automatically send PUBACK | Paho still does not automatically send PUBACK | Your code must choose recovery; return error alone is not a retry signal. |
In manual mode, leaving a QoS 1 delivery unacknowledged can occupy an in-flight slot and slow later QoS 1/2 deliveries. Paho's acknowledgement tracker sends acknowledgements in receipt order, so acknowledging a later message can wait behind an earlier one. Disconnect/reconnect redelivery is conditional on session continuity, broker persistence, QoS, and message expiry. Manual acknowledgement is not permanent broker storage.
For permanently invalid messages, persist a bounded rejection record or a quarantine entry, then acknowledge after that disposition commits. Do not leave poison messages unacknowledged forever. For transient database failures, do not record permanent rejection; stop or back off consumption using a deliberate connection/recovery policy so work remains recoverable. A callback error alone does neither.
📚 **Pinned implementation:** `references/paho.golang-v0.23.0/paho/client.go` documents callback invocation, manual acknowledgement, `Ack`, and ordered acknowledgement tracking. MQTT 5 Receive Maximum bounds the QoS 1/2 publications that can remain unacknowledged on a connection; it does not bound your Go queue.
## 🔁 2. Make database effects idempotent
> **❓ Question:** **Two ingestion workers receive the same report concurrently. What prevents both transactions from counting its items?**
Use a stable business key and a database uniqueness constraint. Examples include `(device_id, report_id)` for sales reports and `(device_id, command_id)` for device command outcomes. Do not use the MQTT packet identifier; it belongs to an exchange and can be reused.
Inside one transaction:
1. Attempt to insert the deduplication/business record under its unique key.
2. If it is new, apply the corresponding inventory/revenue or command-state effect.
3. Store the application response/outcome needed for a repeat.
4. Commit all related changes together.
If the key already exists, load the previously stored outcome and do not apply the effect again. If the same key arrives with materially different content, record a conflict instead of silently assigning the ID a new meaning. The database constraint arbitrates concurrent workers and remains after process restart; an in-memory map cannot do either reliably.
### 🧪 Predict, then try it
- Send one report twice sequentially and once concurrently. Expect one business effect and an identifiable duplicate outcome.
- Force a transaction rollback after recording the dedup key but before the business effect. Verify that neither half is committed.
- Commit the transaction, terminate the process before `Ack`, reconnect under the same eligible session, and observe duplicate delivery without a duplicate effect.
## 🧹 3. Persist invalid-message disposition
> **❓ Question:** **Should permanently invalid JSON occupy a Receive Maximum slot indefinitely?**
No. After safe validation fails, write a quarantine/rejection record with bounded metadata: topic, reason category, receipt time, and a safe message identifier if one could be parsed. Avoid storing secrets or dumping arbitrary payloads. Commit that record, then acknowledge the delivery. For transient infrastructure failures, keep the work recoverable instead of classifying it as permanently invalid.
This creates a clear distinction:
| Result | Database action | MQTT action |
| --- | --- | --- |
| Valid, new message | Store identity and business effect atomically | Ack after commit |
| Valid duplicate | Load prior outcome; no repeated effect | Ack after duplicate is recognized |
| Permanently invalid | Commit bounded rejection/quarantine record | Ack after disposition commits |
| Transient DB/storage failure | No false success or permanent rejection | Do not Ack; invoke recovery/backoff policy |
## 🧱 4. Add PostgreSQL to the lab
> **❓ Question:** **Which state needs to survive a backend restart, and which state can remain temporary?**
Persist accepted sales reports, deduplication keys, command state/results, and any outbox intent needed by the next lesson. Keep connection objects, worker queues, and in-flight Paho packet handles in memory; they cannot be made database records. On restart, recover work from durable application rows and eligible MQTT session deliveries rather than pretending the old callback is still alive.
Use the course's dedicated lab PostgreSQL service only. Add migrations, bounded connection-pool settings, query contexts/deadlines, and repository interfaces used by business code. Use PostgreSQL unique indexes as the final deduplication authority. Define transaction boundaries around a single business decision rather than spreading a report effect over unrelated commits.
Learner commands, run from the project root after adding the DB files:
```sh
docker compose -p mqtt5-learning config --quiet
docker compose -p mqtt5-learning up -d postgres
go test ./internal/ingest ./internal/sales ./internal/commands
go test -race ./...
```
The first command validates the lab Compose configuration. The second starts only the project-named PostgreSQL service; inspect its health/logs before running integration tests. Tests must use the dedicated lab database and synthetic data. Stop that service with the matching Compose project when finished; do not point these commands at production.
## 🔄 5. Separate MQTT acknowledgement from application responses
> **❓ Question:** **After the backend commits a sales report and sends PUBACK, what does the device know?**
PUBACK completes the device-to-broker QoS 1 delivery hop; it does not say that the backend committed the report. If the device needs application-level evidence, publish a separate receipt after the transaction commits. That receipt should include the stable report ID and a status the device can interpret. The report may be committed even if the receipt is lost, so a retry must receive the stored prior outcome.
For commands, keep broker publish acknowledgement, device acceptance, and device completion as distinct command states. A database commit that records a command result is backend evidence; it does not retroactively change the meaning of the earlier command PUBACK.
If database commit and MQTT response publication must survive a crash between them, write an outbox row in the same database transaction as the business state. Lesson 14 handles that publication gap.
## 🧪 Failure experiments and done criteria
1. Send a duplicate report and verify the database effect occurs once.
2. Interrupt PostgreSQL before the transaction commits; verify the message is not acknowledged as accepted.
3. Crash after commit but before manual Ack; verify replay returns the previous outcome without repeating the effect.
4. Send a permanently invalid message; verify a safe rejection/quarantine record commits before Ack.
5. Delay acknowledgements until Receive Maximum is reached; observe delivery pressure, then free a slot and confirm delivery resumes.
6. Crash after commit but before publishing an application response. Record the failure window that Lesson 14's outbox will close.
**Done when:** Replaying a report after restart does not double-count it, database failures do not become false acknowledgements, invalid messages have a durable disposition, and you can explain why QoS 2 does not make database or physical effects exactly once.
**Next:** [Lesson 14 — Close database/MQTT failure gaps](14-database-mqtt-failure-gaps.md) makes publication intent recoverable with a transactional outbox.
teaching/course-source/14-database-mqtt-failure-gaps.md
+119
View File
@@ -0,0 +1,119 @@
# 🔄 Lesson 14 — Close database/MQTT failure gaps
## 🎬 Start with the system story
**❓ Question:** **The API commits a command to PostgreSQL, then the process crashes before publishing it. How will the device ever receive the accepted command?**
**💬 Answer:** The database must persist both the command and the intent to publish it in the same transaction. A worker can later publish unsent outbox rows. MQTT session storage and a Paho outgoing queue do not replace this application-level record.
```mermaid
flowchart LR
H[HTTP/API request] --> T[DB transaction]
T --> C[Command row]
T --> O[Outbox row]
O --> W[Retrying publisher worker]
W -->|PUBLISH QoS 1| B[Mosquitto]
B --> D[Device simulator]
D -->|application result| B
B --> I[Ingestion]
I --> R[Persist result and command state]
```
The database owns accepted business intent. The outbox worker owns attempts to transmit that intent. The broker owns eligible MQTT session delivery. The device owns its deduplication/effect record. Each layer can repeat work, so stable IDs and idempotency remain necessary.
## 🪟 1. Draw every failure window
> **❓ Question:** **What if the publish reaches the broker, but the backend crashes before marking the outbox row sent?**
Walk each boundary separately and write the durable state before and after a crash:
| Failure point | Durable state | Recovery behavior |
| --- | --- | --- |
| Before command transaction commits | No accepted command/outbox | Caller can retry using its HTTP idempotency key |
| After commit, before worker publishes | Command and pending outbox row | Worker publishes after restart |
| After PUBLISH, before broker PUBACK | Outbox still pending/attempting | Retry may duplicate; preserve command ID |
| After PUBACK, before outbox marked sent | Broker accepted that hop; row may still be pending | Worker may publish again; device deduplicates |
| After device acts, before its result reaches backend | Device may have acted; backend status is uncertain | Retry/reconcile by stable command ID; do not infer failure from timeout |
| After result commit, before application receipt reaches device | Result is stored; response may be lost | Duplicate request/result returns stored prior outcome |
`PUBACK` lets the backend record evidence about a publisher-to-broker hop. It is not proof that a device received or completed the command. A timeout after publishing is an unknown business outcome unless the device later supplies evidence.
## 📤 2. Use a transactional outbox
> **❓ Question:** **How can the API accept a command and guarantee the intent to publish is not lost in a crash?**
In one database transaction, create the command row and an outbox row containing the serialized/versioned publication intent, target topic, stable command ID, expiry/deadline, and attempt state. Commit both together. A worker claims pending rows, publishes with a bounded context, records the returned MQTT result, and retries eligible failures with backoff. Keep the same command ID across every retry.
The outbox closes the “database committed, publish intent forgotten” gap. It does not make the remote publish and local sent marker atomic. If the broker accepts the publication and the backend crashes before updating the row, a later worker can publish again. The device must deduplicate by command ID and replay the prior application result.
Do not hold a database transaction open while waiting for a network publish. Claim work with a short transaction, publish outside it, then record the attempt outcome. Use a lease/claim strategy so two workers do not continuously publish the same row at once; still assume duplicates can occur around crashes.
## 🧾 3. Persist device results and idempotency
> **❓ Question:** **A device result arrives twice, or arrives after the command timed out. Should status regress?**
Validate the response topic, payload, device identity, command ID, and Correlation Data before applying it. In one transaction, store the result under a unique identity and update command state only through allowed transitions. A duplicate terminal result should not repeat an effect or regress completed/rejected state. A late result can resolve an earlier unknown outcome if it matches the command; preserve the history that a timeout had occurred.
Classify outcomes explicitly:
- **Retryable transport failure:** no reliable evidence that the peer accepted the publish; retry within expiry/deadline policy.
- **Permanent rejection:** broker or application reports a non-retryable reason; retain that reason safely.
- **Expired command:** do not publish stale work; mark it expired and expose that outcome.
- **Unknown device outcome:** publication may have arrived, but the application response is missing. Do not claim the device did nothing.
- **Completed/rejected by device:** validated application result is persisted; duplicates do not change the terminal result.
Correlation Data helps match a response to a request but is not authorization or deduplication. The application command ID and device-side deduplication state remain necessary.
## 🔀 4. Keep the three queue/state mechanisms distinct
> **❓ Question:** **Can a file-backed MQTT queue replace the PostgreSQL outbox?**
No. These mechanisms protect different state:
| Mechanism | Owner | What it can preserve | What it does not establish |
| --- | --- | --- | --- |
| Transactional outbox | Application database | Business intent and retry state atomically with command state | Device receipt or completion |
| Paho outgoing queue | MQTT client library/application configuration | Publications queued for transport according to that queue's semantics | Atomicity with the business DB transaction or device effect |
| MQTT session state | Client and broker | Protocol/session continuity and eligible queued QoS delivery | Application business state, deduplication, or physical action |
Review the pinned AutoPaho queue example and persistence API before enabling a file-backed queue. Record exactly where its files live and what a successful enqueue result means. Do not infer broker delivery from local enqueue success.
## ⏳ 5. Apply retries, expiry, and backpressure
> **❓ Question:** **Should an old price update or expired ping be published after several hours of retry?**
Give the command an application deadline and MQTT 5 Message Expiry Interval when appropriate. The worker checks the application deadline before publishing; broker expiry limits queued MQTT delivery after publication. The device checks its own deadline too, because time can be spent in the outbox, network, broker session, and device queue. These are separate clocks and checks.
Use bounded retry counts or retry windows with backoff and jitter. Stop or park permanently rejected and expired rows. Bound concurrent publishes and database claims; observe pending outbox count, oldest row age, attempts, publish latency, PUBACK results, application response latency, duplicate results, and unknown outcomes.
## 🧪 6. Inject crash windows
Run only against the dedicated learning broker and database, with synthetic commands:
1. Commit a command/outbox row, stop the publisher worker, restart it, and verify the pending row is attempted.
2. Stop the backend immediately after publish but before marking the outbox row sent. Restart and verify the duplicate publication uses the same command ID and produces one simulated effect.
3. Drop the device response after the simulated effect. Confirm status becomes unknown or remains pending according to the documented deadline policy; it must not report definite failure.
4. Expire a command before publish and verify the worker does not send it.
5. Restart the broker and compare broker session recovery with the still-pending application outbox row.
For each run, record prediction, injected failure, database rows, MQTT packet evidence, device effect counter, final status, and cleanup. Restore the lab by stopping only the project-named learning services and clearing only test data you created.
### 🧠 Review questions
1. Which transaction removes the lost-intent gap between accepting a command and publishing it?
2. Why can the device receive a duplicate after the broker already sent PUBACK to the backend?
3. What can a Paho outgoing queue prove, and what can it not prove?
4. Which state is owned by the broker session versus the command database?
5. What should the API report when the device may have acted but no result arrived?
> [!answer]- 📋 Short answers
>
> 1. Commit the command and outbox intent atomically.
> 2. The backend can crash after PUBACK but before marking the outbox row sent, then publish again after restart.
> 3. It can prove local queue admission according to its API; it cannot prove broker delivery or business completion.
> 4. Broker session state tracks MQTT subscriptions/in-flight/eligible queued delivery; the command database tracks application intent, status, and results.
> 5. An unknown outcome, until validated device evidence resolves it.
**Done when:** Every outbox crash window has an explainable recovery path, repeated publication causes one simulated effect, expiry prevents stale delivery, and command status never treats PUBACK as device completion.
**Next:** [Section 15](../../COURSE.md#section-15--split-api-and-ingestion-processes) moves API and ingestion into separate processes while keeping persisted status as the shared source of truth.
teaching/course-source/16-mqtt5-controls.md
+129
View File
@@ -0,0 +1,129 @@
# Section 16 — MQTT 5 controls you can explain and measure
Study this after Sections 12–15 in [COURSE.md](../../COURSE.md), or read it earlier as a conceptual preview. Continue using Paho v0.23.0 and the isolated broker. Each experiment changes one behavior; record the broker/client versions and observed result instead of assuming a field's existence proves complete support.
## 1. Subscription options change what arrives
**❓ Question:** **A device publishes a retained status update and then subscribes to that same topic. Which option controls retained replay, and which option controls the RETAIN bit on forwarded live traffic?**
**💬 Answer:** Retain Handling controls retained replay at subscription time. Retain As Published controls whether the broker preserves the incoming RETAIN bit when forwarding a live publication. Neither option changes the broker's ACL decision or filters payload identity.
These fields belong to each `paho.SubscribeOptions` entry, not to the PUBLISH payload:
| Field | Meaning | Experiment |
| --- | --- | --- |
| `NoLocal` | Suppress publications originating on the same client connection identity | With a lab identity authorized to publish and subscribe to one topic, compare false/true |
| `RetainAsPublished` | Preserve the incoming publication's RETAIN bit when forwarding live messages | Compare a live retained publication with replay after resubscription |
| `RetainHandling: 0` | Send matching retained state when subscribing | Resubscribe and observe replay |
| `RetainHandling: 1` | Send retained state only when this subscription is new | Repeat the same subscription in the existing session |
| `RetainHandling: 2` | Do not send retained state at subscription time | Subscribe after storing a retained value; then publish a live update |
Retain Handling 2 does not prevent future live retained publications. Retain As Published does not decide whether the broker stores retained state. No Local is not an ACL or a device-ID filter, and must not be set for shared subscriptions. Lesson 4 deliberately enables Retain As Published so the simulator's retained-command guard also works for live forwarding.
An UNSUBSCRIBE removes a broker subscription after its UNSUBACK exchange; it does not unregister your local Go callback or erase retained values. Use `manager.Unsubscribe(ctx, &paho.Unsubscribe{Topics: []string{filter}})` with an operation deadline and inspect the returned reasons. `filter` is the exact previously subscribed filter. Already delivered work may still be in the local queue.
## 2. Shared subscriptions distribute work
**❓ Question:** **Two ingestion processes share a subscription group, and a response reaches the process that did not create the pending waiter. What fails?**
**💬 Answer:** The receiving process may have no matching in-memory command record. A shared subscription distributes each matching delivery to a group member; it does not share process memory or guarantee device affinity. Keep response correlation state shared or route responses to a common durable store.
Ordinary subscriptions on two clients each receive a matching publication. Build the base filter with `contracts.DeviceFilter(contracts.KindTelemetry)`, then wrap its string form as `$share/ingest/` plus that filter to create the shared subscription. Both clients join the `ingest` group, and the broker selects a group member for each matching delivery. Convert the final shared filter to `contracts.TopicFilter` when passing it through the lab transport helper, or to `string` at the Paho packet boundary. Give each process a unique Client ID; sharing a group is different from sharing a Client ID.
Create two authorized lab consumers, check their SUBACK results, then publish numbered synthetic telemetry. Record which instance handles each ID. Stop one consumer and repeat. Use a fresh lab group/session for the experiment so old offline state does not obscure the result. Verify the broker's shared-subscription support and effective ACL behavior before drawing conclusions.
Distribution is not guaranteed round-robin, per-device affinity, or business exactly-once execution. A restarted consumer can repeat work, and parallel workers can finish out of order. Shared consumers need shared durable deduplication for important reports. Keep command-response routing deliberate: a response delivered to another instance will not find a waiter stored only in your instance's RAM.
## 3. Distinguish wire limits from application limits
**❓ Question:** **The backend sets Receive Maximum to four. Does that cap the number of telemetry jobs waiting in its Go worker queue at four?**
**💬 Answer:** No. This CONNECT limit constrains the broker's concurrent QoS 1/2 publications awaiting protocol acknowledgement on the broker-to-client direction. If callbacks acknowledge quickly after queueing, the application queue can continue to grow until its own bound is reached.
The CONNECT properties describe what this client can receive. The CONNACK properties describe what the server can receive or supports. Direction matters: changing the client's Receive Maximum constrains broker-to-client QoS 1/2 inflight publications, not the backend's outgoing command rate.
Add this field to your existing `autopaho.ClientConfig`; it is a fragment, with `net/url` and Paho imports supplied by that file:
```go
ConnectPacketBuilder: func(cp *paho.Connect, _ *url.URL) (*paho.Connect, error) {
if cp.Properties == nil {
cp.Properties = &paho.ConnectProperties{}
}
receiveMaximum := uint16(4)
maximumPacketSize := uint32(32768)
cp.Properties.ReceiveMaximum = &receiveMaximum
cp.Properties.MaximumPacketSize = &maximumPacketSize
return cp, nil
},
```
The nil check preserves AutoPaho's existing properties, including configured session expiry. Replacing the whole property struct would silently discard them. Receive Maximum must be nonzero. Maximum Packet Size counts the complete MQTT packet, including topic and properties, not only JSON payload. Paho maps these fields to CONNECT on each connection.
In `OnConnectionUp`, inspect `ack.Properties` safely and record server limits/capabilities such as Receive Maximum, Maximum Packet Size, Maximum QoS, and retained/shared-subscription availability. A nil optional pointer is absence with a protocol-defined default, not automatically zero. Do not print the complete authentication configuration as a shortcut.
**Experiment:** compare QoS 1 inflight traffic with automatic acknowledgements and with Lesson 6's deliberately delayed manual acknowledgements. A queue of 16 can still fill even with Receive Maximum 4 if automatic ACK frees inflight slots as soon as callbacks return. Repeat at QoS 0: Receive Maximum is not its backpressure mechanism. Bound application queues and broker resources separately.
For packet-size testing, use harmless payloads below and above a deliberately small configured limit. Record whether the sender library rejects locally, the peer rejects/disconnects, or delivery is absent; the precise path depends on direction and support. No callback arrival alone does not identify the failure layer.
## 4. Properties carry context, not trust
**❓ Question:** **A simulator sets User Property `producer=device-001`. Has the broker authenticated that device identity?**
**💬 Answer:** No. User Properties are forwarded metadata. The broker authenticates the MQTT client and applies its topic ACL; the application validates the topic, payload, and any metadata it chooses to use. Never treat a property or payload ID as proof of the publisher.
User Properties are UTF-8 key/value pairs and may repeat a key. Use them for synthetic trace context or schema hints, with size/count limits. They are not authenticated claims by themselves and are not HTTP headers that the broker interprets for your application.
```go
properties := &paho.PublishProperties{
User: paho.UserProperties{
{Key: "trace-id", Value: "lab-trace-001"},
{Key: "producer", Value: "simulator"},
},
}
```
Attach this fragment to a publication and copy the metadata you need at your receive boundary. Change `producer` to `backend` while publishing as the simulator: the string changes, but the authenticated identity does not. Avoid putting secrets in properties because observers may log them.
Subscription Identifiers let a subscriber label a subscription and identify which subscription(s) matched a received message. They do not identify the publisher or deduplicate business events. The pinned public `PublishProperties.SubscriptionIdentifier` is a single `*int`; overlapping-subscription cases require checking the library's representation/support rather than assuming it exposes every possible identifier as a list.
## 5. Topic aliases optimize one connection
**❓ Question:** **The broker restarted and the MQTT session resumed. Can the client safely publish using an alias-only packet from before the restart?**
**💬 Answer:** No. Topic Alias mappings belong to one network connection and direction. Reestablish the mapping after reconnect; session resumption does not restore it.
An alias associates a small integer with a topic name for one direction of one connection. The sender first establishes the mapping with the topic and alias; subsequent publications may use the alias to avoid resending the long topic string. This reduces bytes, not permission checks or payload size.
The receiver advertises its Topic Alias Maximum. Zero or absent support must not be treated as permission to send arbitrary aliases. A reconnect loses the mapping even when an MQTT session resumes. Never send an alias-only first publication after reconnect.
Read the pinned `paho/extensions/topicaliases/` implementation and its tests, plus the receiving path in `paho/client.go`, before enabling aliases. The extension uses a publish hook to manage outbound mappings. Integrate reset/recreation into the connection lifecycle and verify both initial and reconnected traffic; the presence of a `TopicAlias` field alone is insufficient evidence. Keep the lab's default full-topic publications until that experiment passes.
## 6. Read failures as protocol evidence
**❓ Question:** **A QoS 1 PUBACK has reason code `0x10` (No matching subscribers). Did the device accept the command?**
**💬 Answer:** No. The reason describes the broker's handling of that publication hop. It is not an application response from the device. The command service must wait for and validate a correlated device response, and must preserve an unknown outcome if none arrives.
Reason codes belong to a packet context. A SUBACK contains a result for each filter. A QoS 1 PUBACK describes the immediate publication hop. A DISCONNECT may describe why the broker ended a connection. None is a device business status.
| Observation | Interpretation | Next action |
| --- | --- | --- |
| Publish response `0x10` | No matching subscribers; a non-error MQTT result | Check peer session/subscription and await application evidence |
| Reason code at least `0x80` in these ACK results | Protocol-level failure | Classify the packet/reason before deciding to retry |
| Connection error without MQTT reason | May have failed before a packet response arrived | Inspect network/TLS and broker evidence |
| Context deadline exceeded | Local wait ended | Preserve outcome uncertainty if transmission may have happened |
Use bounded reconnect backoff with jitter for transient failures so many devices do not reconnect in lockstep after an outage. The pinned `ReconnectBackoff` field accepts a function of attempt number; inspect the tagged `backoff` example before selecting parameters. Authentication/configuration errors need repair or escalation, not a faster retry loop. Reconnect retries a connection; it does not authorize repeating a business action under a new ID.
<details>
<summary>Why can Receive Maximum 4 coexist with 100 queued application items?</summary>
The inflight quota counts unacknowledged protocol deliveries, not all work the application has admitted. If the callback queues and acknowledges quickly, each ACK permits another delivery while workers remain behind. Separate durable admission, queue capacity, and protocol flow control.
</details>
**Done when:** for each feature record purpose, Go API field, packet/direction, prerequisite/ACL, prediction, actual observation, and whether you would use it. A reading-only feature stays marked “not experimentally verified.”
Sources: [MQTT 5 specification](https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html) and pinned local `paho/cp_subscribe.go`, `paho/cp_unsubscribe.go`, `paho/cp_publish.go`, `paho/cp_connect.go`, `paho/cp_connack.go`, and `autopaho/backoff.go` under `references/paho.golang-v0.23.0/`.
Next: Section 17 in [COURSE.md](../../COURSE.md).
teaching/course-source/17a-advanced-devops-lifecycle.md
+180
View File
@@ -0,0 +1,180 @@
# Section 17A — Advanced DevOps service lifecycle
> [!toc]- Contents
>
> - [Purpose and boundaries](#purpose-and-boundaries)
> - [The service lifecycle](#the-service-lifecycle)
> - [1. Map ownership and failure domains](#1-map-ownership-and-failure-domains)
> - [2. Build trustworthy CI gates](#2-build-trustworthy-ci-gates)
> - [3. Package and promote artifacts](#3-package-and-promote-artifacts)
> - [4. Configure each environment safely](#4-configure-each-environment-safely)
> - [5. Release compatible services and schemas](#5-release-compatible-services-and-schemas)
> - [6. Deploy, observe, and recover](#6-deploy-observe-and-recover)
> - [7. Operate through incidents and change](#7-operate-through-incidents-and-change)
> - [Capstone exercise](#capstone-exercise)
> - [Self-check](#self-check)
## Purpose and boundaries
This lesson is for a backend engineer with junior-level DevOps experience who wants to own a service through its full delivery and operational lifecycle. It builds on the backend, simulator, PostgreSQL state, MQTT broker, and runbooks from earlier sections.
The goal is to practice production-grade reasoning in an isolated learning environment. Do not deploy to Robot Market production or connect real vending machines. Kubernetes is not required: use the lab's Docker Compose setup to practice the same lifecycle decisions, then compare how an orchestrator would implement them.
The lesson does not prescribe a CI vendor, cloud, or deployment platform. The practices apply across those choices. Each decision should name its owner, evidence, failure mode, and recovery action.
## The service lifecycle
**❓ Question:** What must be true before a code change can safely become a running service?
**💬 Answer:** The change needs reviewable source, automated checks, a known build artifact, environment-specific configuration, an explicit rollout, and evidence that the service works afterward. Operators also need a way to detect failure and recover without guessing which state was lost.
```mermaid
flowchart LR
S[Source change] --> C[CI checks]
C --> A[Versioned artifact]
A --> P[Promote same artifact]
P --> D[Deploy with configuration]
D --> V[Verify health and behavior]
V --> O[Observe service indicators]
O --> R[Recover, roll forward, or roll back]
R --> L[Review and improve]
L --> S
```
Promotion means moving the same artifact through environments. Rebuilding separately for each environment makes it harder to know whether the tested bits are the deployed bits. Configuration and secrets change by environment; the application artifact should not.
## 1. Map ownership and failure domains
**❓ Question:** If the ingestion process restarts, which facts can it reconstruct, and which system owns them?
Start with a state inventory before designing deployment. A successful MQTT reconnection does not restore application data, and a database restart does not restore a broker's session state.
| State or responsibility | Owner in this course | Recovery evidence |
| --- | --- | --- |
| Durable commands, report results, deduplication | PostgreSQL | Query after application restart; replay a duplicate safely |
| MQTT subscriptions and broker session state | Broker, subject to session and persistence configuration | Session experiment and broker restart check |
| Pending callback work and local device view | Service process memory until explicitly persisted | Process restart shows what disappears |
| Desired configuration snapshot | Application database or retained MQTT state, as explicitly designed | Compare the authoritative revision after restart |
| Credentials and environment settings | Deployment configuration/secret source | Redacted effective configuration and rotation procedure |
| Build and release identity | Source revision and artifact registry | Map a running image digest back to source and pipeline result |
Draw the system's dependencies and failure domains: API, ingestion, PostgreSQL, broker, simulator/device, and deployment environment. For each, record what happens when it is unavailable, whether work queues or fails, and which owner responds.
Do not use “MQTT is up” or “the container is running” as a complete health statement. A process can run while subscriptions are missing, the database is unavailable, or accepted work is stuck.
## 2. Build trustworthy CI gates
**❓ Question:** Which checks are fast and repeatable on every change, and which require a controlled environment?
Organize checks by the evidence they provide:
| Gate | Example evidence | What it does not prove |
| --- | --- | --- |
| Formatting and static checks | `gofmt`, `go vet`, lint results | Correct runtime behavior |
| Unit tests | Contract, state transition, and retry behavior | Broker or database interoperability |
| Concurrency checks | `go test -race ./...` | Absence of all logical races or production overload |
| Integration tests | Isolated PostgreSQL and Mosquitto tests | Production network, credentials, or capacity |
| Acceptance checks | API request reaches simulator and accurate result is queryable | Safe deployment or production device compatibility |
| Image checks | Build, user, metadata, dependency and vulnerability review | Runtime configuration or readiness |
Make each gate deterministic where possible: pin tool and dependency versions, use synthetic data, isolate test infrastructure, set timeouts, and collect logs on failure. Keep broker/device acceptance checks separate from fast unit tests so their failures identify the right layer.
A green pipeline is evidence for the checks it ran, not a guarantee of safety. Record skipped checks and required human review explicitly. A failing quality or security gate should stop artifact promotion until the failure is understood or an authorized exception is documented.
## 3. Package and promote artifacts
**❓ Question:** How can an operator identify exactly which program is running and reproduce that artifact?
Build versioned service images from a pinned Go toolchain and dependencies. Prefer a multi-stage build that compiles in a build stage and copies only the required binary and runtime assets into the final image. Run as a non-root user, use a narrow writable filesystem, and include no source credentials, `.env` files, or lab password files in image layers.
Attach traceable metadata to each artifact: source revision, build job, dependency versions, and immutable image digest. A human-readable version tag helps people; the digest identifies the exact content. Promote the same digest from test to staging. Do not rebuild it with different build arguments to “make a staging image.”
Treat base-image and Go dependency updates as planned changes: review release notes, run the normal gates, scan for known vulnerabilities, and retain the prior deployable artifact for recovery. A scan finding needs triage based on affected code, exposure, and available fix; a scan alone is not a complete security review.
For this exercise, add a deployment definition for the learning services and inspect the resolved configuration before starting them. Confirm the final image user, entrypoint, signal behavior, mounted paths, and exposed ports. Keep the broker listener on loopback unless the experiment explicitly requires another network boundary.
## 4. Configure each environment safely
**❓ Question:** What should change between local, test, and staging environments, and what should stay identical?
The artifact stays identical. Broker addresses, database endpoints, resource limits, and secret references vary by environment. Use the configuration loader's validation before opening MQTT or database connections. Log a redacted configuration summary so an operator can diagnose the effective settings without revealing passwords, tokens, or private keys.
Use distinct identities for API, ingestion, broker, database, and deployment automation. Grant each only the access it needs. Supply secrets at runtime through the environment's secret mechanism; never place real values in source, image layers, test output, or resolved configuration shared in an issue.
Plan credential rotation as an operational change: issue a replacement, allow the intended overlap, update consumers, verify successful connections and authorization, then revoke the old credential. Define how to recover if one process fails to adopt the new value. For broker ACL changes, test both allowed and denied topics before promotion.
Configuration validation should distinguish a missing or invalid setting from a transient dependency outage. Invalid configuration should fail clearly before serving traffic; temporary broker/database failure should follow the documented retry and readiness policy.
## 5. Release compatible services and schemas
**❓ Question:** How do you roll out a database change when old and new service versions may run at the same time?
Use an expand/migrate/contract sequence:
1. **Expand:** add a schema capability that old code can tolerate, such as a nullable column or new table.
2. **Migrate:** deploy code that can work with the old and expanded schema; backfill data in bounded, observable batches if needed.
3. **Switch:** verify the new code path and data before relying on it.
4. **Contract:** remove old schema only after all old binaries and rollback paths no longer need it.
Database rollback is not always the inverse of a schema migration. Destructive changes or new data may make old code incompatible. State whether a release can roll back, needs a forward fix, or requires restoring data; rehearse the selected path with synthetic data.
Use immutable artifacts and explicit release records. A release should identify source revision, image digests, schema version, configuration change, approvals, rollout time, verification evidence, and operator. Keep changes small enough to identify which change caused an incident.
## 6. Deploy, observe, and recover
**❓ Question:** What signals show that a new version is ready to receive work, and what should happen to accepted work during termination?
Define health states per process:
- **Liveness:** the process can make progress; a failed check may justify restarting it.
- **Readiness:** the process is prepared to receive its assigned traffic or messages; a failed check should remove it from new work where the platform supports that behavior.
- **Dependency status:** broker and database reachability, reported separately so an operator can see which dependency is failing.
Readiness should reflect required initialization, not merely an open TCP port. For ingestion, verify required subscriptions and durable-store availability before claiming readiness. During shutdown, stop admitting new work, drain accepted work within a deadline, preserve work that cannot finish, and disconnect cleanly. Document what happens when the deadline expires.
Roll out progressively where the platform permits it. Set a small release scope, monitor error and latency indicators, and define pause/rollback thresholds before deployment. For multiple ingestion replicas, include shared-subscription behavior, duplicate handling, ordering assumptions, and durable idempotency in the rollout plan.
Define useful service indicators: successful API requests, MQTT reconnects, subscription readiness, queue depth and age, command completion/unknown outcomes, report processing lag, database errors, and resource pressure. Avoid unbounded device or command identifiers as metric labels. Correlate logs with stable IDs while redacting secrets and untrusted payloads.
For state recovery, define example recovery-point and recovery-time objectives (RPO/RTO). Backups only count when restore works: rehearse PostgreSQL restore and inspect broker persistence/session behavior separately. Record which data can be rebuilt, which work may replay, and how duplicate effects are prevented.
## 7. Operate through incidents and change
**❓ Question:** During an incident, what should the responder establish before restarting services or changing configuration?
Follow a repeatable sequence:
1. **Detect:** identify the user-visible symptom and the alert or report that exposed it.
2. **Scope:** determine affected process, devices, commands/reports, time window, and release version.
3. **Stabilize:** stop harmful retries or further rollout; preserve evidence and durable work.
4. **Diagnose:** compare application, broker, database, network, and device evidence. Separate transport acknowledgement from business completion.
5. **Recover:** choose restart, rollback, forward fix, replay, or restore based on state ownership and data compatibility.
6. **Verify:** confirm service indicators recover and affected commands/reports reach a known state.
7. **Learn:** record timeline, cause, contributing conditions, and a follow-up change with an owner.
Runbooks should state commands in context, expected evidence, access required, risks, and restoration steps. Keep incident notes factual and avoid credentials or private payloads. Schedule restore drills, dependency upgrades, certificate rotation, and access reviews rather than waiting for an incident to reveal they are overdue.
## Capstone exercise
Prepare and release a simulated change that adds a harmless command result field or compatible database column. Use only the isolated lab.
Before starting, predict the behavior if the API is restarted during rollout, PostgreSQL is unavailable, a new image fails readiness, and rollback is requested after the schema migration. Then:
1. Draw the pipeline and state-ownership diagram.
2. Define automated gates and the evidence each gate produces.
3. Build and identify one immutable service artifact.
4. Deploy it with lab-only configuration and verify readiness, a command flow, and report persistence.
5. Inject one failure, pause rollout, recover using the documented path, and verify durable state.
6. Record the release and incident evidence, including what remains unverified outside the lab.
**Deliverables:** pipeline diagram, release checklist, artifact identity, redacted deployment configuration, rollout/rollback decision, recovery record, and a short incident review.
## Self-check
1. Why is promoting one image digest safer than rebuilding for each environment?
2. Why can a successful database backup still leave recovery unproven?
3. Which work should an ingestion readiness check confirm before the service accepts messages?
4. Why might rolling back a binary be unsafe after a database migration?
5. Which evidence distinguishes a broker delivery acknowledgement from a completed vending command?
**Answer guide:** A digest identifies the same tested bits across environments; a restore drill proves backup usability; readiness checks required subscriptions and storage dependencies; schema/data changes can make old code incompatible; a device/application result and persisted command state provide business evidence beyond MQTT acknowledgement.
teaching/course-source/README.md
+72
View File
@@ -0,0 +1,72 @@
# MQTT in Go: Markdown course source
Begin with [Lesson 0 — The broker, Go backend, and device lifecycle](00-system-lifecycle.md) for the complete system story. Lessons 1–8 open with a lifecycle question and diagram where useful, then provide a guiding question before each implementation step.
Broker deployment and administration are part of the learning goals. Read [Broker operations](broker-operations.md) alongside Lesson 1; it explains lifecycle, credential maintenance, backup/restore, upgrades, and remote-deployment preparation. Commands are explained in the lesson that introduces them.
Follow these lessons in order from the learning project root. Read the concept and packet flow first, predict the experiment, then implement and compare the result. Go and basic Docker syntax are assumed; explanations focus on MQTT behavior and operational consequences. Full early examples remain available as reference solutions.
The shared lab topic contract lives in `internal/contracts/topic.go`: use its concrete topic builders for publishing, fleet filters for backend subscriptions, exact filters for simulator subscriptions, and parser for inbound routing. Mosquitto ACL text remains a separate authorization configuration.
1. [Set up the broker and MQTT 5 lab](01-lab-setup.md)
2. [Write and understand a Go MQTT 5 connection](02-go-connection.md)
3. [Add typed TOML and koanf configuration](03-typed-configuration.md)
4. [Publish, subscribe, and build a device simulator](04-publish-subscribe-simulator.md)
5. [Design contracts and validate inbound messages](05-contracts-validation.md)
5A. [Turn validated messages into backend state](05a-backend-message-processing.md)
6. [Manage callbacks, concurrency, and cancellation](06-concurrency-cancellation.md)
7. [Build reliable command and response behavior](07-command-response.md)
8. [Expose device commands through HTTP](08-http-commands.md)
9. [Learn MQTT 5 sessions through failure experiments](09-sessions-recovery.md)
10. [Expiry, presence, and retained state](10-expiry-presence-state.md)
11. [MQTT security boundaries](11-security-boundaries.md)
11B. [Provision device ACLs with Mosquitto Dynamic Security](11b-dynamic-security.md) — optional implementation extension to Section 11
12. [Implement representative vending workflows](12-vending-workflows.md)
13. [Persist reports and command state in PostgreSQL](13-persist-reports-command-state.md)
14. [Close database/MQTT failure gaps with an outbox](14-database-mqtt-failure-gaps.md)
17A. [Practice the advanced DevOps service lifecycle](17a-advanced-devops-lifecycle.md) — standalone lifecycle and release practices lesson
The interactive HTML sequence in `../lessons/` continues beyond the Markdown implementation lessons: [Lesson 15 — split API and ingestion](../lessons/0015-process-boundaries.html), [Lesson 17 — operate and diagnose](../lessons/0017-operations-diagnostics.html), and [Lesson 18 — backend handover](../lessons/0018-backend-handover.html) turn those `COURSE.md` checklists into guided exercises. Section 16 has both a [Markdown source lesson](16-mqtt5-controls.md) and an [interactive guide](../lessons/0016-mqtt5-controls.html). Section 17A is a separate Markdown lesson for advanced DevOps lifecycle practices.
Paho is pinned to **v0.23.0**, the selected course baseline. Use the matching checkout under `references/paho.golang-v0.23.0/`. This release requires Go 1.24 or newer; the lessons' Go 1.25 baseline is compatible.
The files are learning instructions and reference solutions. Application code has not been installed into the learning project: create it yourself as each lesson directs. Section 2 supplies the first backend program; Section 3 adds configuration; Section 4 replaces the backend program and adds shared transport plus a simulator. Do not combine both versions of `main.go` in the same package.
## Previously recorded verification (before this review)
The complete Go examples were extracted into a separate temporary module and checked with Go 1.27.1 against the pinned dependencies. They passed compilation, `go vet`, and configuration precedence/validation tests, including a race-enabled test run. Smaller explanatory fragments are identified as fragments and require their surrounding program or test.
An isolated temporary Mosquitto 2.0.22 lab verified the documented Compose configuration, password-file permissions, ACL file loading, initial subscriptions, Go heartbeats, QoS 1 ping/pong, incorrect-password rejection, and subscription restoration after broker restart. The temporary broker and clients were removed afterward. The lesson exercises still require you to verify behavior in your own implementation; this check does not prove production reliability.
Lessons 5–11, 12–14, and 16 are guided exercises, not complete tested reference programs. Their implementation checks are for you to run after writing the code.
## Verification for the 2026-10-08 teaching review
Extracted the complete Lesson 2–4 programs and the new Paho/TLS fragments into a temporary module. They passed `go build -buildvcs=false ./...` and `go vet -buildvcs=false ./...` against the pinned dependencies. The flag disables Git revision stamping because this environment cannot read usable repository metadata; it does not change MQTT behavior. Fragment wrappers supply their surrounding variables/imports for type checking.
Checked local Markdown links, source paths, code fences, and expandable answer blocks. Reviewed protocol explanations against official MQTT/Mosquitto documentation and the matching local Paho source. No live broker, TLS, failure-injection, or load experiments were run in this review, and Mermaid rendering was not verified. The earlier runtime results above are historical records, not fresh results for the revised examples.
## Coverage map: explain it, then prove it
| Concept | Where it is explained | Evidence you should produce |
| --- | --- | --- |
| Broker routing, topic names/filters, wildcards | 1 and 4 | Trace CLI packets and change one topic filter |
| CONNECT/CONNACK, identity, Keep Alive, reconnect | 2 and 3 | Explain connection versus subscription/device readiness |
| QoS 0/1/2, per-hop acknowledgements, DUP and business IDs | 4 and 7 | Predict no-subscriber, lower-QoS, and duplicate outcomes |
| Callback/ACK order, manual acknowledgement, bounded queues | 6 | Show the RAM loss window and measurable overload |
| Payload validation, Content Type, Payload Format | 5 | Reject valid-looking but invalid messages before state changes |
| Validated message to application service and device view | 5A | Update last-seen state only after validation; distinguish freshness from health and transport |
| Response Topic, Correlation Data, timeouts, idempotency | 7 and 8 | Match concurrent results and retry without another effect |
| Clean Start, Session Expiry, Session Present, recovery stores | 9 | Predict network versus process restart outcomes |
| Message Expiry, retained deletion, Will/Will Delay, freshness | 10 | Observe stale command suppression and presence limitations |
| TLS, authentication, ACLs, identity and response destinations | 1 and 11 | Fail each security boundary deliberately |
| Live per-device provisioning and revocation | 11B (optional) | Deny cross-device topics, then disable a connected client |
| Subscription options, unsubscribe, shared consumers | 16 | Compare live/replayed messages and consumer distribution |
| Receive Maximum, packet size, properties, aliases, capabilities | 16 | Record packet direction, negotiated limit, and measured behavior |
| Docker mount/volume identity, recreation, overrides, crashes | Broker operations | Explain exactly which state survives each command |
Coverage means these concepts are explained and have exercises. It does not mean you have completed the experiments or that this course implements every optional MQTT extension. Enhanced authentication is introduced conceptually; broker-specific authentication plugins and bridges are outside the lab's implementation scope.
For each experiment, write: **prediction → operation and reason → observed packet/application evidence → explanation → restoration step**. Expand the answer blocks only after explaining the outcome yourself.
See [COURSE.md](../../COURSE.md) for the complete sequence and the connection to Robot Market's backend.
teaching/course-source/broker-operations.md
+140
View File
@@ -0,0 +1,140 @@
# Broker operations — maintain and deploy your lab
Use this alongside Lesson 1, then revisit it during Section 9. All commands run from `/home/ryan/Media/Code/Code/robot-market/mqtt-5-learning` and target the `mqtt5-learning` project. These are instructions to perform in your lab; writing this guide has not changed or started a broker.
## Lifecycle: know which state you are changing
| Operation | Example | Effect |
| --- | --- | --- |
| Stop | `docker compose -p mqtt5-learning stop broker` | Gracefully stops the container; retains it and its mounts |
| Start | `docker compose -p mqtt5-learning start broker` | Starts that existing container; does not apply edited Compose settings |
| Restart | `docker compose -p mqtt5-learning restart broker` | Stops and starts the existing container; interrupts clients and rereads broker startup configuration |
| Apply Compose changes | `docker compose -p mqtt5-learning up -d broker` | Reconciles the service definition and recreates the container when needed; keeps the named volume |
| Force recreation | `docker compose -p mqtt5-learning up -d --force-recreate broker` | Creates a fresh container even if Compose detects no change; useful after replacing a bind-mounted file |
| Remove containers/network | `docker compose -p mqtt5-learning down` | Stops and removes project containers/network; the declared named data volume remains |
| Delete data too | `docker compose -p mqtt5-learning down -v` | Also removes declared named volumes; this erases lab broker persistence |
`-p mqtt5-learning` selects the Compose project name, grouping this lab's service, network, and volume resources. It does not select a port. Keep it identical across all commands. Your existing `compose.yml` is valid; the lesson's `compose.yaml` is an alternative filename, not an additional required file.
After a password-file repair, explicitly restart an existing container and inspect `logs --since=1m broker` so historical failures are distinguished from the new attempt. `up -d` may leave an unchanged running container alone. Read-only entrypoint ownership warnings do not establish failure; verify the broker's own listener/error messages and authenticated messaging.
Use `restart` after editing broker configuration that requires a restart. Use `up -d` after editing ports, image tags, mounts, or container environment in Compose. Docker's [restart documentation](https://docs.docker.com/reference/cli/docker/compose/restart/) explains that restart does not apply Compose changes. For single-file bind mounts, an editor that replaces a file rather than writing in place can leave an existing container referring to the old file; recreation establishes fresh mounts.
After every change, inspect `ps`, inspect `logs --tail=50 broker`, and repeat an authenticated publish/subscribe check. These check container state, broker startup, and actual messaging respectively. `logs -f broker` follows new logs until Ctrl+C; Ctrl+C stops log following, not the detached broker.
## Choose the active credential store before maintenance
The basic lesson uses a host bind mount. This workspace's current `compose.yml` uses the external `mqtt5-learning_broker-secrets` volume because host permission changes did not stick. Inspect the service mount first. For this workspace, replace `-v "$PWD/lab/secrets:/work"` in the rotation commands below with `-v mqtt5-learning_broker-secrets:/work`. Update and back up the active volume; modifying the preserved host copy has no effect on the broker. See Lesson 1's filesystem fallback for provisioning and access checks. External secret volumes survive Compose `down -v`, but declared nonexternal broker-data volumes do not.
## Rotate a lab credential
Use the Lesson 1 utility command without `-c` to update the existing backend user:
```sh
docker run --rm -it --user 0 -v "$PWD/lab/secrets:/work" eclipse-mosquitto:2.0.22 mosquitto_passwd /work/passwords backend
docker run --rm --user 0 -v "$PWD/lab/secrets:/work" eclipse-mosquitto:2.0.22 chown mosquitto:mosquitto /work/passwords
docker run --rm --user 0 -v "$PWD/lab/secrets:/work" eclipse-mosquitto:2.0.22 chmod 600 /work/passwords
docker compose -p mqtt5-learning restart broker
```
The first command prompts for a new password and replaces only this user's hash. The next commands restore broker ownership and owner-only access. Restart reloads the file and disconnects existing clients. Update the backend terminal's password environment variable, reconnect, and test that the new password succeeds and the old one fails on a fresh connection. Simply editing a password file does not establish that already-connected clients have been revoked.
Mosquitto also supports reloading password-file configuration with SIGHUP; see its [authentication guide](https://mosquitto.org/documentation/authentication-methods/). Restart is the simpler explicit lab operation. For deployed systems, plan client credential rollout and reconnection before rotation to avoid unexpectedly stranding devices. Adding/removing a user also requires reviewing that user's ACL entries. `mosquitto_passwd -D /work/passwords username` removes a user; `-D` means delete, so use it only for an account you intend to remove.
## Backup and restore: what actually needs saving?
Save three things: the Compose/broker/ACL configuration, the protected password file, and the broker data volume. The Go application's in-memory records are a fourth, separate concern and are not covered by this backup. Password hashes and any TLS private keys remain sensitive backup material.
For a simple consistent lab backup, stop the broker gracefully first so it can write persistence state. Confirm `ps` reports it stopped. Use `docker volume ls` to list volumes and identify the project's broker-data volume; `ls` lists names, not contents. Verify it with `docker volume inspect <actual-volume-name>` before using it. Replace placeholders with the observed name; do not infer a production volume from an example name.
Create a private backup directory outside Git, archive the stopped volume with a temporary utility container mounting it read-only, and save configuration plus secrets with their permissions preserved. A typical volume-copy utility needs a source mount and a backup-output mount; run it with the permissions required to read the broker's files. The data volume can contain `mosquitto.db`, while credentials live in the separate host secrets directory. Copying only one of them is incomplete.
Restore into a separate empty lab volume while the target broker is stopped. Restore ownership/permissions, connect that volume to an isolated broker using the recorded image/configuration, and verify authentication, ACL denial, retained state, and session behavior. Do not extract an archive over an active broker data directory. A backup is useful only after a restore exercise succeeds; record the commands, actual volume name, date, image version, and observed recovered state in your journal.
## Upgrade with an observable rollback
Record the current image version and back up configuration, secrets, and stopped broker state. Read the target release's migration notes. Test the new pinned image against a restored copy in a separate lab project with a different loopback port; repeat authentication, ACL, reconnect/session, and retained-state checks before changing the primary lab.
`docker compose -p mqtt5-learning pull broker` downloads the image selected by the current Compose file; it does not replace the running container. After selecting a tested new image tag, `up -d broker` applies it. Keep the previous tag and backup: restoring an older image over data rewritten by a newer version is not a guaranteed rollback. For reproducibility beyond a version tag, record the resolved image digest as well.
## Preparing an actual remote deployment
The initial lesson exposes only loopback plaintext, so remote devices cannot connect. Do not merely change the bind address to make it reachable. Work through Section 11 before remote use: create a server certificate with the correct DNS/IP identity, configure a TLS listener, mount certificate/key files with broker-readable permissions, and make clients validate the CA and server identity. A password hash on disk does not encrypt credentials traveling over plaintext MQTT.
Choose the device-facing hostname, TLS port, firewall policy, per-device credentials/ACLs, persistent storage, backup/restore process, log retention, monitoring, and credential renewal responsibilities. Add a deliberate container restart policy such as `restart: unless-stopped` to the service if the deployment should recover after daemon/host restarts; the initial lab omits it to keep stop/start experiments explicit. A restart policy helps process availability but does not prove messaging readiness or provide high availability.
Container logs need a bounded retention/rotation policy so a busy broker cannot fill the host disk. Monitor connection failures, permission failures, disk use, and application heartbeat/command evidence. Test broker readiness through an authorized synthetic message round trip; a TCP port check alone cannot prove MQTT authentication or subscriptions work.
Keep provisioning repeatable with documented setup steps or a reviewed script, use the platform's secret provisioning mechanism, and plan certificate/password rotation and client rollout. Do not bake credentials or private keys into the image. Deployment-specific values belong in that environment's configuration, not in the learning project's committed secrets.
## Advanced Docker operations: predict the MQTT consequence
### Inspect the configuration without starting anything
```sh
docker compose -p mqtt5-learning config -q
docker compose -p mqtt5-learning ps -a
docker compose -p mqtt5-learning logs --since=2m --tail=100 broker
docker volume inspect mqtt5-learning_broker-data
```
`config -q` validates the resolved Compose model without printing it or contacting MQTT. `ps -a` includes stopped containers, helping distinguish "never created" from "exited on startup." The log filters restrict evidence to recent lines; they do not clear old logs. `volume inspect` shows volume metadata and its Docker host mountpoint, not the broker's session contents. The volume name above is the expected default for this project: if inspection fails, use `docker volume ls` and identify the actual lab volume rather than creating one to hide the mismatch.
Ask three separate questions: is the container running, did Mosquitto load the intended files, and can the authorized clients exchange packets? These commands answer only parts of that chain. Finish with the authenticated CLI round trip from Lesson 1.
### Understand execution context
```sh
docker compose -p mqtt5-learning exec -T broker mosquitto_pub --help
```
This runs the image's CLI in the existing broker container. `-T` disables the pseudo-terminal, useful for scripted output. `--help` is passed to `mosquitto_pub`; it performs no publish. When adapting the actual CLI test to this form, use container port 1883, not host port 18883. A new utility container started with `docker run` has a different loopback interface and does not reach the broker merely by using `127.0.0.1`.
### Isolate the session-persistence comparison
For Lesson 9's nonpersistent comparison, create `compose.session-test.yaml`:
```yaml
services:
broker:
volumes:
- broker-data-session-test:/mosquitto/data
volumes:
broker-data-session-test:
```
First stop the original lab. Temporarily set `persistence false` in its broker configuration, then inspect and apply the override:
```sh
docker compose -p mqtt5-learning stop broker
docker compose -p mqtt5-learning -f compose.yml -f compose.session-test.yaml config
docker compose -p mqtt5-learning -f compose.yml -f compose.session-test.yaml up -d --force-recreate broker
```
Use the actual primary filename if yours is `compose.yaml`. The later `-f` file overrides the data mount by its container destination `/mosquitto/data`, while keeping configuration and credential mounts. Verify that merged result before `up`. `--force-recreate` replaces the container so the changed file/mount is actually used; it does not delete the original data volume. Use an unused comparison volume name on repeat experiments so previous state cannot contaminate the result.
After measuring, stop the comparison with the same two `-f` files, restore `persistence true`, and run `docker compose -p mqtt5-learning -f compose.yml up -d --force-recreate broker`. This restores the original data mount. Inspect logs and verify authentication plus messaging again. The comparison volume remains for inspection; no volume deletion is required.
### Crash is different from graceful stop
For a deliberate broker crash experiment only:
```sh
docker compose -p mqtt5-learning kill -s SIGKILL broker
docker compose -p mqtt5-learning start broker
docker compose -p mqtt5-learning logs --since=2m broker
```
SIGKILL bypasses the broker's graceful persistence flush and interrupts every lab client. `start` runs the same container with the same mounted data; inspect what was actually recovered. Compare against `stop`, which permits graceful shutdown before Docker's stop timeout. A named volume survives both, but only state already written to it can be recovered after the crash. Do not infer crash durability from the existence of the volume.
These commands target the broker, so they test broker recovery. To test a device Will, terminate the simulator instead and keep the broker alive, as explained in [Lesson 10](10-expiry-presence-state.md).
## Practice and explain
1. Stop and start the lab; explain which resources survived.
2. Rotate one lab password and test both passwords on fresh connections.
3. Recreate the container and confirm the data volume remains mounted.
4. Perform an isolated backup/restore exercise and record what was recovered.
5. Explain what would change for TLS, remote access, automatic restart, and log retention.
Read the official [Compose up](https://docs.docker.com/reference/cli/docker/compose/up/), [Compose down](https://docs.docker.com/reference/cli/docker/compose/down/), and [Mosquitto configuration](https://mosquitto.org/man/mosquitto-conf-5.html) documentation when planning changes. Their latest pages may describe features beyond this lab's pinned Mosquitto 2.0.22; check version compatibility before adopting them.