Refactor MQTT course and bound backend message processing
Move lesson source into the teaching structure, align the course path with the learner's Go and DevOps experience, and add the interactive lesson guides. Add advanced DevOps lifecycle study as a separate section. Bound MQTT message work and drain accepted jobs during shutdown; align the backend implementation with the revised course through Section 6.
This commit is contained in:
@@ -0,0 +1,399 @@
|
||||
# 🛠️ Section 1 — Prepare the MQTT 5 lab
|
||||
|
||||
Read this first. You will create the files and run the commands yourself. All paths below are relative to the learning project root. The examples use the module path `example.com/mqtt5-learning`; if you change it, change the imports in later lessons too.
|
||||
|
||||
**Version baseline:** Paho `v0.23.0`, verified from upstream release tags on 2026-10-07. It requires Go 1.24 or newer; Go 1.25 is compatible. The matching local reference is `references/paho.golang-v0.23.0/` (commit `1aa03969c5932464359b3e4ed752d9ac61abccc9`).
|
||||
|
||||
## 🎬 Start with the system story
|
||||
|
||||
> 💡 **Story question:** Before we write a Go client, what must already exist so the backend and a simulated device can connect safely and exchange messages?
|
||||
|
||||
The broker is a separately configured service. It needs a listener, an authentication source, topic permissions, and storage settings. The backend and device are separate clients with their own connection settings and credentials. This lesson prepares the local broker and verifies its behavior with MQTT 5 CLI clients before adding Go to the path.
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
O[Prepare Compose and Mosquitto config] --> B[Mosquitto broker starts]
|
||||
P[Create backend and simulator credentials] --> B
|
||||
A[Define ACL topic permissions] --> B
|
||||
G[Go backend comes in a later lesson] -. connects .-> B
|
||||
D[Go simulator comes in a later lesson] -. connects .-> B
|
||||
C[Mosquitto CLI verifies MQTT 5] --> B
|
||||
```
|
||||
|
||||
The Compose file starts and isolates the broker; the Mosquitto configuration tells the broker how to listen and enforce authentication/ACLs. A Docker volume preserves broker-owned state. None of these files configures the Go clients—that happens in later lessons.
|
||||
|
||||
For the vending-backend goal, this first lesson establishes only the broker boundary: which client may connect, which topics it may use, and how to observe broker activity. It does not yet store a device's status, ingest its diagnostic logs, or execute a vending command. Those are application responsibilities built after the Go clients can exchange messages.
|
||||
|
||||
Before each setup step, ask which process owns the setting and what observation will prove it took effect.
|
||||
|
||||
|
||||
## 📁 Working directory and file placement
|
||||
|
||||
Run every shell command in this lesson from the learning project root:
|
||||
|
||||
```sh
|
||||
cd /home/ryan/Media/Code/Code/robot-market/mqtt-5-learning
|
||||
```
|
||||
|
||||
Paths such as `cmd/backend/main.go` are relative to the repository root. The Markdown implementation lessons are under `teaching/course-source/`; create application files at the paths shown, not inside `teaching/course-source/` or `references/`. Code blocks labeled Go, TOML, YAML, or conf are file contents; shell blocks are terminal commands. Create parent directories before saving files.
|
||||
|
||||
## 🛠️ 1. Check the tools and create the module
|
||||
|
||||
> 💡 **Story question:** What must be available on this machine to build the Go client and start the isolated broker, and which Paho API speaks MQTT 5?
|
||||
|
||||
Check the Go toolchain and both Docker components, then pin the MQTT 5 module in this learning project. The old `paho.mqtt.golang` client is a separate MQTT 3 API.
|
||||
|
||||
```sh
|
||||
go version
|
||||
docker version
|
||||
docker compose version
|
||||
go mod init example.com/mqtt5-learning
|
||||
go get github.com/eclipse/paho.golang@v0.23.0
|
||||
```
|
||||
|
||||
`go version` reports the installed compiler. `docker version` reports both CLI and daemon versions: a missing server response means the CLI cannot reach the Docker engine. `docker compose version` checks the Compose plugin, which reads `compose.yaml` and manages the application's services. Having Docker installed does not prove the daemon is running.
|
||||
|
||||
`go mod init example.com/mqtt5-learning` creates `go.mod`, declaring this application's import-path prefix. It does not create the source directories. `go get ...@v0.23.0` downloads a selected dependency and records module requirements and checksums in `go.mod`/`go.sum`; it may also update transitive requirements needed by that dependency.
|
||||
|
||||
Run `go mod init` only once. `go get` pins the dependency in `go.mod`; commit `go.mod` and `go.sum` when you initialize version control. Do not use a local `replace` directive to substitute the newer default-branch reference. A repository checkout is reading material, while a module requirement selects the compiled library.
|
||||
|
||||
The old backend's `github.com/eclipse/paho.mqtt.golang` is a different MQTT 3 client. MQTT 5 uses `github.com/eclipse/paho.golang/autopaho` and `/paho`. There is no `SetProtocolVersion(5)` needed in these MQTT 5 examples.
|
||||
|
||||
## 🦟 2. Create the broker files
|
||||
|
||||
> 💡 **Story question:** What configuration belongs to Mosquitto itself, and how will host applications reach the broker without exposing the lab listener to the network?
|
||||
|
||||
Compose controls the container boundary and port mapping. Mosquitto's own files control its listener, credentials, ACL, and persistence. Keep those roles distinct as you create the files below.
|
||||
|
||||
Create these directories:
|
||||
|
||||
```sh
|
||||
mkdir -p lab/mosquitto
|
||||
mkdir -p lab/secrets
|
||||
```
|
||||
|
||||
`mkdir` creates directories; `-p` creates missing parents and tolerates an already-existing directory. These commands do not create configuration files.
|
||||
|
||||
Create `compose.yaml` directly in the project root, next to `go.mod`. Its full path is `/home/ryan/Media/Code/Code/robot-market/mqtt-5-learning/compose.yaml`. The `./lab/...` volume paths below are relative to this Compose file:
|
||||
|
||||
```text
|
||||
mqtt-5-learning/
|
||||
├── go.mod
|
||||
├── compose.yaml
|
||||
└── lab/
|
||||
├── mosquitto/
|
||||
│ ├── mosquitto.conf
|
||||
│ └── acl
|
||||
└── secrets/
|
||||
└── passwords # generated in step 3, not written by hand
|
||||
```
|
||||
|
||||
Your existing `compose.yml` is also a supported Compose filename; keep it rather than creating a competing `compose.yaml`. Use one primary Compose file consistently. YAML nesting is controlled by spaces: the final `volumes:` declaration must align with `services:`, not with `broker:`. If nested under `services`, Compose treats `volumes` as another service and reports `services.volumes additional properties 'broker-data' not allowed`.
|
||||
|
||||
Paste this YAML into your primary Compose file:
|
||||
|
||||
```yaml
|
||||
services:
|
||||
broker:
|
||||
image: eclipse-mosquitto:2.0.22
|
||||
ports:
|
||||
- "127.0.0.1:18883:1883"
|
||||
volumes:
|
||||
- ./lab/mosquitto/mosquitto.conf:/mosquitto/config/mosquitto.conf:ro
|
||||
- ./lab/mosquitto/acl:/mosquitto/config/acl:ro
|
||||
- ./lab/secrets:/mosquitto/secrets:ro
|
||||
- broker-data:/mosquitto/data
|
||||
volumes:
|
||||
broker-data:
|
||||
```
|
||||
|
||||
This is a pinned Mosquitto 2 lab image, not a claim that it is the newest broker release. Host port 18883 maps to container port 1883. Containers use `broker:1883`; Go programs on your host use `127.0.0.1:18883`. Loopback binding keeps this listener off your LAN. Persistence belongs to this Compose project only.
|
||||
|
||||
### 🐳 Understand the Compose definition
|
||||
|
||||
`services` lists the containers in this application; `broker` is the service name you use in commands and the DNS name on this Compose network. `image` selects a prebuilt image; Docker downloads it if needed. There is no `build` section because this lesson does not build a broker image.
|
||||
|
||||
`ports` uses `host-address:host-port:container-port`. `127.0.0.1:18883:1883` forwards connections from this computer's loopback port 18883 to the container's listener at 1883. A host client cannot use container port 1883 unless that port is separately published. `broker:1883` works for containers on the same Compose network, not arbitrary host programs or remote devices.
|
||||
|
||||
Each `volumes` entry is `source:container-destination[:mode]`. Sources beginning with `./` are host files/directories (bind mounts); `:ro` makes them read-only to this container. Create the configuration files before starting Docker: missing file paths can result in directories being mounted where files were expected. `broker-data` is a Docker-managed named volume, declared at the bottom, for writable broker persistence. It is separate from the image and survives ordinary container replacement.
|
||||
|
||||
Create `lab/mosquitto/mosquitto.conf`:
|
||||
|
||||
```conf
|
||||
listener 1883
|
||||
allow_anonymous false
|
||||
password_file /mosquitto/secrets/passwords
|
||||
acl_file /mosquitto/config/acl
|
||||
persistence true
|
||||
persistence_location /mosquitto/data/
|
||||
log_dest stdout
|
||||
connection_messages true
|
||||
```
|
||||
|
||||
### 🦟 Understand each broker setting
|
||||
|
||||
| Setting | Purpose and operational consequence |
|
||||
| --- | --- |
|
||||
| `listener 1883` | Opens the MQTT listener inside the container; host exposure is controlled separately by Compose |
|
||||
| `allow_anonymous false` | Requires authentication; it does not grant topic permissions |
|
||||
| `password_file ...` | Reads the generated username/password hashes at this container path |
|
||||
| `acl_file ...` | Reads topic permissions; a correct password alone does not authorize every topic |
|
||||
| `persistence true` | Enables saving broker state, including eligible retained/session information; it does not persist Go application state |
|
||||
| `persistence_location ...` | Places saved broker data in the named volume rather than the disposable container filesystem |
|
||||
| `log_dest stdout` | Sends broker logs to Docker's log stream for `compose logs` |
|
||||
| `connection_messages true` | Logs client connection/disconnection events for troubleshooting |
|
||||
|
||||
Persistence is not a database transaction for your application. Recently changed broker state may not survive an abrupt crash until it has been saved. Session settings and message QoS still decide which offline data is eligible to survive.
|
||||
|
||||
Create `lab/mosquitto/acl`:
|
||||
|
||||
```conf
|
||||
user backend
|
||||
topic write lab/v1/devices/+/commands
|
||||
topic read lab/v1/devices/+/telemetry
|
||||
topic read lab/v1/devices/+/responses
|
||||
|
||||
user simulator
|
||||
topic read lab/v1/devices/device-001/commands
|
||||
topic write lab/v1/devices/device-001/telemetry
|
||||
topic write lab/v1/devices/device-001/responses
|
||||
```
|
||||
|
||||
`user` starts rules for that authenticated username. `topic write` permits publishing; `topic read` permits receiving matching topics. `+` matches exactly one topic level, so the backend rules cover one device-ID segment. The simulator's rules use an exact device ID. Access outside the allowed patterns is denied by this simple ACL configuration. The text file has no extension because Mosquitto reads the path you configure, regardless of its filename suffix.
|
||||
|
||||
These ACL patterns intentionally mirror the `lab/v1/devices/{deviceID}/{kind}` hierarchy that Section 4 will define in `internal/contracts/topic.go`. The broker reads its own ACL text; it does not import the Go package. Keep both sides aligned and test denied cross-device access, because a valid Go `TopicName` is not a broker permission.
|
||||
|
||||
The simulator can act only as `device-001`. The backend can operate all lab devices. ACL checks authorize topics; they do not validate JSON or prove that a payload's identity is truthful. The CLI checks below must use a direction permitted for that username.
|
||||
|
||||
Add to `.gitignore` (preserve any existing rules):
|
||||
|
||||
```gitignore
|
||||
/lab/secrets/
|
||||
/.env
|
||||
```
|
||||
|
||||
Git ignore patterns are file contents, not terminal commands. `/lab/secrets/` ignores the root-level directory and its contents; `/.env` ignores a root environment file. Ignoring a file does not untrack a secret already committed. Keep real secrets out of the initial commit as well as later changes.
|
||||
|
||||
## 🔐 3. Create passwords interactively
|
||||
|
||||
> 💡 **Story question:** How will the broker recognize the backend and simulator when they connect, and how can those identities receive different topic permissions?
|
||||
|
||||
Create separate broker accounts and ACL rules before starting clients. The client supplies credentials; the broker validates them against its configured source. Authentication permits connection, while ACL rules govern topic operations.
|
||||
|
||||
These commands create a lab password file without putting the password literal in your shell history. Run them from the project root:
|
||||
|
||||
```sh
|
||||
docker run --rm -it --user 0 -v "$PWD/lab/secrets:/work" eclipse-mosquitto:2.0.22 mosquitto_passwd -c /work/passwords backend
|
||||
docker run --rm -it --user 0 -v "$PWD/lab/secrets:/work" eclipse-mosquitto:2.0.22 mosquitto_passwd /work/passwords simulator
|
||||
docker run --rm --user 0 -v "$PWD/lab/secrets:/work" eclipse-mosquitto:2.0.22 chown mosquitto:mosquitto /work/passwords
|
||||
docker run --rm --user 0 -v "$PWD/lab/secrets:/work" eclipse-mosquitto:2.0.22 chmod 600 /work/passwords
|
||||
```
|
||||
|
||||
`-c` creates/replaces the file. Use it only for the first username; using it again would erase the backend entry. The last two commands make the file readable by the broker account in this image. Container-root use is limited to setting up this mounted lab file.
|
||||
|
||||
### 🔧 Why four Docker commands?
|
||||
|
||||
There are two jobs here: generate password hashes for two users, then give the broker permission to read the resulting file. Docker adds a wrapper because we use the tool bundled with the pinned broker image instead of installing Mosquitto utilities on your computer. Each command runs a temporary utility container; none starts the broker.
|
||||
|
||||
| Part of the command | What it does |
|
||||
| --- | --- |
|
||||
| `docker run` | Creates and runs a new container for this one operation |
|
||||
| `--rm` | Removes that container when its command exits; the mounted host files remain |
|
||||
| `-it` | Keeps standard input open and supplies a terminal for the password prompt |
|
||||
| `--user 0` | Runs this setup operation as container root, permitting file creation and ownership changes on the mount |
|
||||
| `-v "$PWD/lab/secrets:/work"` | Makes the host project's `lab/secrets` directory available at `/work` inside the container; this mount is writable |
|
||||
| `eclipse-mosquitto:2.0.22` | Selects the same pinned image used by the broker |
|
||||
| `mosquitto_passwd -c /work/passwords backend` | Creates the password file and prompts for the backend user's password |
|
||||
| `mosquitto_passwd /work/passwords simulator` | Adds the simulator user to that same file, without replacing the backend entry |
|
||||
| `chown mosquitto:mosquitto /work/passwords` | Assigns file ownership to the broker account as defined inside this image |
|
||||
| `chmod 600 /work/passwords` | Gives the owner read/write access and removes group/other access |
|
||||
|
||||
`$PWD` is your current terminal directory, so run these from the project root. `/work/passwords` in the setup container and `lab/secrets/passwords` on your host are the same file. Later, Compose mounts that directory at `/mosquitto/secrets` with `:ro`, and `mosquitto.conf` reads `/mosquitto/secrets/passwords`. The setup mount is writable; the running broker mount is read-only.
|
||||
|
||||
The generated file stores password hashes, not the plaintext passwords you type. Your backend and simulator still need their original passwords to connect; a client cannot use the stored hash as its login password. Keep the generated file out of Git. The ACL file separately determines which topics each authenticated username may use.
|
||||
|
||||
Linux checks numeric user/group IDs. `chown` inside this image picks the broker's IDs; your host may display a numeric owner or a different name for those same IDs. With mode `600`, your regular host user may be unable to read the file afterward. That is expected: the broker must read it, and subsequent password maintenance can use the same temporary root utility container. Do not make the file world-readable to work around this.
|
||||
|
||||
### 💡 What is usual practice?
|
||||
|
||||
For a small Mosquitto 2.0 installation, generating a password file with `mosquitto_passwd`, configuring `password_file`, and restricting file access is a conventional approach. The official [authentication guide](https://mosquitto.org/documentation/authentication-methods/) documents creating the first user with `-c` and adding or updating users without it. Docker is simply how this lesson obtains that utility.
|
||||
|
||||
- **Local Docker lab:** Generate the file once with a temporary container, mount it into the broker, and repeat the update command only when adding users or rotating passwords. You do not rerun all four commands on each broker start. After modifying the file, reapply the ownership and mode if the utility replaced it, then reload authentication or restart only the lab broker.
|
||||
- **Host-installed Mosquitto:** Run the installed `mosquitto_passwd` utility directly, then set permissions for the broker's service account. The core operations are the same; there is no container wrapper.
|
||||
- **Repeated deployment:** Put the provisioning steps in a small reviewed setup script or dedicated setup job. Supply credentials through the deployment's secret-management mechanism and mount the resulting authentication file read-only. A secret mount still needs the correct broker-readable ownership and permissions. Avoid embedding passwords in image layers, committed Compose files, or command arguments.
|
||||
- **Larger device fleets:** Evaluate managed authentication and authorization, such as Mosquitto's [Dynamic Security plugin](https://mosquitto.org/documentation/dynamic-security/), rather than maintaining a large static file manually. This is a deployment choice, not something needed for this two-user exercise.
|
||||
|
||||
The four explicit commands make the setup visible for learning. Once you understand them, wrapping them in a setup script is reasonable. We keep interactive prompts here so password literals do not appear in shell history or process arguments.
|
||||
|
||||
### 🧪 Verify password setup before starting
|
||||
|
||||
Do not start the broker until both password prompts and both permission commands have completed successfully. The two accounts must exist in the generated file; creating an empty file does not create usable credentials.
|
||||
|
||||
Verify existence and access without printing hashes:
|
||||
|
||||
```sh
|
||||
docker run --rm --user mosquitto -v "$PWD/lab/secrets:/work:ro" eclipse-mosquitto:2.0.22 sh -c 'test -s /work/passwords && test -r /work/passwords'
|
||||
```
|
||||
|
||||
`--user mosquitto` tests access as the image's broker account. `sh -c` runs the quoted checks inside the utility container; `test -s` requires a nonempty file, `test -r` requires read access, and `&&` runs the second check only if the first succeeds. Success is quiet with exit status zero. Failure means you must fix the file/path/permissions before starting. This checks file access, not whether the typed client password is correct.
|
||||
|
||||
### 🔐 If permission changes do not stick on the host filesystem
|
||||
|
||||
Verify final metadata with `stat -c '%a %u:%g %n' lab/secrets/passwords`. `stat` reports mode, numeric owner/group, and pathname without exposing hashes. Also use `findmnt -T lab/secrets/passwords -o TARGET,FSTYPE,OPTIONS` to identify the underlying mount. Some mounted filesystems synthesize permissions: `chmod`/`chown` can appear successful while the metadata remains unchanged. Do not conclude the file is protected from the commands' exit status alone.
|
||||
|
||||
This workspace exhibited that behavior on its `fuseblk` mount: the password file remained `755` and `1000:1000`. The current `compose.yml` therefore uses a separately provisioned Docker volume for credentials. To reproduce that fallback after generating the users:
|
||||
|
||||
```sh
|
||||
docker run --rm --user 0 -v "$PWD/lab/secrets:/source:ro" -v mqtt5-learning_broker-secrets:/target eclipse-mosquitto:2.0.22 sh -c 'cp /source/passwords /target/passwords && chown mosquitto:mosquitto /target/passwords && chmod 600 /target/passwords && stat -c "%a %u:%g" /target/passwords'
|
||||
```
|
||||
|
||||
The host source is read-only. The new Docker-managed target is writable; `cp` copies the existing file and the following commands apply permissions there. This command replaces the target password file, so use it only for initial provisioning or an intentional replacement. The verified result in this image is `600 1883:1883`; always inspect the actual result for your environment. A named volume is not automatically secure or encrypted: its actual filesystem and file permissions still matter.
|
||||
|
||||
Replace the service's `./lab/secrets:/mosquitto/secrets:ro` mount with `broker-secrets:/mosquitto/secrets:ro`. Declare this additional volume at the root, alongside `broker-data`:
|
||||
|
||||
```yaml
|
||||
volumes:
|
||||
broker-data:
|
||||
broker-secrets:
|
||||
external: true
|
||||
name: mqtt5-learning_broker-secrets
|
||||
```
|
||||
|
||||
`name` specifies the already-provisioned Docker volume's exact name. `external: true` tells Compose to use it rather than create/delete it. The preceding utility run creates it if missing. Apply the changed mount with `up -d broker` and inspect recent logs. Verify access with the earlier read-check command but replace its host mount with `-v mqtt5-learning_broker-secrets:/work:ro`.
|
||||
|
||||
For every subsequent add-user, rotation, or permission command, mount `mqtt5-learning_broker-secrets:/work` rather than `$PWD/lab/secrets:/work`: the broker now reads the volume copy. Editing the old host file does not update the live credential store. Include this external volume in backups; Compose `down -v` does not delete external volumes. The old host copy remains sensitive and ignored by Git; it has been preserved rather than automatically deleted.
|
||||
|
||||
### 🔐 Understand password-tool and entrypoint warnings
|
||||
|
||||
During creation or update, `mosquitto_passwd` can warn about world-readable permissions or ownership. Do not assume these warnings prove the final file is correct: run the ownership/mode commands afterward and verify access. For this pinned Mosquitto 2.0.22 image, use `mosquitto:mosquitto` ownership and mode `600`. A password file needs no executable permission. Blindly following a warning suggesting root ownership with owner-only access can prevent the unprivileged broker from reading it. Review requirements again when upgrading rather than treating a future-version warning as this version's deployment instructions.
|
||||
|
||||
The image entrypoint can also print `chown: ... Read-only file system` for the intentional `:ro` mounts. Those lines alone are not a broker authentication failure. Keep configuration and secrets read-only; identify the subsequent Mosquitto messages. `Error opening password file` is fatal and must be fixed. A directory can exist while the required `passwords` file inside it is missing.
|
||||
|
||||
Start the broker for the first time:
|
||||
|
||||
```sh
|
||||
docker compose -p mqtt5-learning config
|
||||
docker compose -p mqtt5-learning up -d
|
||||
docker compose -p mqtt5-learning ps
|
||||
docker compose -p mqtt5-learning logs --tail=50 broker
|
||||
```
|
||||
|
||||
| Command part | What happens and what to check |
|
||||
| --- | --- |
|
||||
| `-p mqtt5-learning` | `-p` means project name in Compose, not network port. Selects the Compose project name, keeping its containers/network/volume names separate from other projects; use this same name for every lifecycle command |
|
||||
| `config` | Parses and resolves Compose configuration; catches YAML/interpolation errors, but does not test broker authentication or every mounted file's contents |
|
||||
| `up -d` | Creates/updates the required resources and starts containers in the background; `-d` means detached |
|
||||
| `ps` | Shows container state and published ports; a running container is not an MQTT readiness check |
|
||||
| `logs --tail=50 broker` | Shows the last 50 log lines for this service without opening a shell inside it |
|
||||
|
||||
If a container was already created or ran before you fixed its password file, explicitly retry startup:
|
||||
|
||||
```sh
|
||||
docker compose -p mqtt5-learning restart broker
|
||||
docker compose -p mqtt5-learning logs --since=1m broker
|
||||
docker compose -p mqtt5-learning ps
|
||||
```
|
||||
|
||||
`restart` stops/starts the existing service even if it has stopped after an error. `up -d` is not a guaranteed restart of an unchanged running container. `--since=1m` restricts logs to the last minute so earlier failures do not get mistaken for a fresh failure. If there is no existing container, use the initial `up -d` sequence instead. If you changed Compose ports or mounts, use `up -d` to reconcile them; if a replaced single-file mount still points at old content, use `up -d --force-recreate broker`.
|
||||
|
||||
Expect a listening broker and no new password/ACL file errors. A running container alone does not prove authentication works.
|
||||
|
||||
For this project, resource names commonly include `mqtt5-learning-broker-1`, `mqtt5-learning_default`, and `mqtt5-learning_broker-data`. Keep `-p mqtt5-learning` on every lifecycle command so they refer to the same project. In `mosquitto_pub/sub`, `-p` instead means port; in `mkdir -p`, it means create parent directories. Flags belong to their commands, not to a universal vocabulary.
|
||||
|
||||
In `ps`, the local-only mapping should show `127.0.0.1:18883->1883/tcp`. A mapping showing `0.0.0.0` or `[::]` exposes that published port on all corresponding host interfaces. Change the Compose mapping to the loopback form and apply it with `up -d broker`; restarting alone does not change port bindings.
|
||||
|
||||
## 🧪 4. Verify MQTT 5 without Go first
|
||||
|
||||
> 💡 **Story question:** How can we prove that the broker is listening, speaks the intended MQTT version, accepts the right credentials, and enforces topic permissions before debugging Go code?
|
||||
|
||||
Use Mosquitto CLI clients as a controlled first pair of MQTT clients. Their packet traces isolate broker reachability, protocol negotiation, authentication, authorization, and routing from application code.
|
||||
|
||||
Use host-installed `mosquitto_sub` and `mosquitto_pub`, or run their equivalents through `docker compose exec broker` with port 1883 inside the container. Keep a subscription open in terminal A:
|
||||
|
||||
```sh
|
||||
mosquitto_sub -h 127.0.0.1 -p 18883 -V mqttv5 -u backend -P '<backend-lab-password>' -t 'lab/v1/devices/device-001/telemetry' -q 1 -v -d
|
||||
```
|
||||
|
||||
In terminal B:
|
||||
|
||||
```sh
|
||||
mosquitto_pub -h 127.0.0.1 -p 18883 -V mqttv5 -u simulator -P '<simulator-lab-password>' -t 'lab/v1/devices/device-001/telemetry' -q 1 -m '{"online":true}' -d
|
||||
```
|
||||
|
||||
`-h` chooses the broker host; `-p` here chooses the network port (unlike Compose's project-name flag). `-u` supplies a username, `-P` its password, `-t` the subscription filter or publication topic, and `-m` the message body. Single quotes preserve the JSON bytes instead of letting the shell interpret them. Start the subscriber first so it is ready before the nonretained publication. `mosquitto_sub` keeps running; Ctrl+C stops this subscriber. `mosquitto_pub` normally exits after completing its publication.
|
||||
|
||||
If using `docker compose -p mqtt5-learning exec broker`, that prefix executes the utility inside the already-running broker container. Use `-h 127.0.0.1 -p 1883` there, because loopback refers to that container. The other MQTT flags remain the same. `exec` requires a running service; the earlier `docker run` setup commands do not.
|
||||
|
||||
The placeholders are not literal passwords. `-V mqttv5` selects the protocol; `-q 1` selects QoS; `-d` shows packet activity; `-v` prints topic plus payload. Expect the subscriber to print the telemetry JSON. This is a synthetic message, not a production heartbeat contract.
|
||||
|
||||
For the CLI only, `-P` exposes its argument to shell history/process inspection. Use temporary lab credentials and avoid pasting real credentials; consult your installed client's options for a protected configuration file if you need repeated CLI checks. Go examples read the password from the environment and never print it.
|
||||
|
||||
Repeat with a wrong password. Also try publishing as `simulator` to `lab/v1/devices/device-002/telemetry`; this must not reach an authorized observer. Check the MQTT reason and broker logs rather than assuming every denied operation has the same client error.
|
||||
|
||||
### 📘 Read the CLI exchange, not just the payload
|
||||
|
||||
With `-d`, identify CONNECT/CONNACK first, then SUBSCRIBE/SUBACK in the subscriber terminal. Only publish after SUBACK. In the publisher terminal, find PUBLISH/PUBACK at QoS 1. In the subscriber terminal, find the separately forwarded PUBLISH and its acknowledgement. Debug formatting varies by client version; learn the packet names rather than memorizing one output line.
|
||||
|
||||
Predict before running: stop the subscriber, publish again with the same command, then restart the subscriber. This nonretained message is not a history entry for a new subscriber. A successful publisher acknowledgement does not create a future subscription or a device response. [Lesson 4](04-publish-subscribe-simulator.md) maps these exchanges to Go calls; [Lesson 9](09-sessions-recovery.md) introduces the different case of an existing offline session.
|
||||
|
||||
<details>
|
||||
<summary>Check your explanation: why start the subscriber first?</summary>
|
||||
|
||||
The broker needs an established matching subscription when this nonretained message arrives. Starting two terminals almost simultaneously leaves a race between SUBSCRIBE and PUBLISH. Seeing the subscriber process running is weaker evidence than seeing its successful SUBACK.
|
||||
|
||||
</details>
|
||||
|
||||
## ⚙️ API preview: connect each setting to its purpose
|
||||
|
||||
```go
|
||||
broker, err := url.Parse("mqtt://127.0.0.1:18883")
|
||||
// Check err before using broker.
|
||||
cfg := autopaho.ClientConfig{
|
||||
ServerUrls: []*url.URL{broker},
|
||||
KeepAlive: 20,
|
||||
ClientConfig: paho.ClientConfig{ClientID: "lab-backend"},
|
||||
}
|
||||
```
|
||||
|
||||
This is an illustrative fragment, not a standalone program. The URL chooses the transport endpoint, while Client ID selects the broker session identity. `KeepAlive` is an integer number of seconds, not a `time.Duration`. The embedded `ClientConfig` describes the underlying MQTT client; the outer configuration describes managed connections.
|
||||
|
||||
## 🧪 Exercises and questions
|
||||
|
||||
1. Explain why switching the Go library is more than changing one import.
|
||||
2. Predict whether the simulator may subscribe to another device's commands; verify it.
|
||||
3. Identify which packet proves a QoS 1 publication reached its immediate receiver, and why it does not prove application processing.
|
||||
4. Stop and start only this lab with `docker compose -p mqtt5-learning stop` and `docker compose -p mqtt5-learning start`.
|
||||
|
||||
**Completion:** CLI publish/subscribe works using MQTT 5, invalid credentials fail, and cross-device writes are denied. Do not use `down -v` unless you deliberately want to erase the lab's broker persistence.
|
||||
|
||||
## ✅ Result — the story so far
|
||||
|
||||
The broker is now the meeting point for the lab, listening only on this computer. It checks each client's username and password, then uses the ACL to decide which topic directions that identity may use. MQTT 5 CLI clients can prove the broker routes an allowed message and rejects an unauthorized one. The Go backend and simulator have not joined the story yet; this lesson prepares the place they will meet.
|
||||
|
||||
## 🦟 Broker administration you should practice
|
||||
|
||||
Read [Broker operations](broker-operations.md) for explained stop/start, restart versus recreate, password rotation, backups, restore, upgrades, and the steps required before remote deployment. Complete its local lifecycle and credential-rotation exercises before treating Section 1 as finished. Later Section 9 measures session recovery, and Section 11 in the course adds TLS and stronger access controls.
|
||||
|
||||
## 🔎 Common failures
|
||||
|
||||
| Symptom | Check |
|
||||
| --- | --- |
|
||||
| Port is already allocated | Choose another loopback host port and update all host clients |
|
||||
| Connection refused | Container state, listener logs, and host/container port distinction |
|
||||
| Password file missing | Complete password generation before startup; verify `lab/secrets/passwords` exists and is nonempty |
|
||||
| Password file unreadable | Verify broker-account read access, ownership/mode, mounted path, and directory traversal permissions |
|
||||
| Read-only `chown` warnings | Check subsequent broker logs; retain `:ro` mounts and fix real read failures separately |
|
||||
| Old error still appears in logs | Use `logs --since=1m` immediately after retrying startup |
|
||||
| `services.volumes` validation error | Align the named-volume declaration with root-level `services:` |
|
||||
| Port shows `0.0.0.0` or `[::]` | Set `127.0.0.1:18883:1883` and apply with `up -d broker` |
|
||||
| Not authorized | Username, password, and exact permitted topic direction |
|
||||
| Pub/sub works but Go fails later | Go URL/credentials, client ID, then tagged API version |
|
||||
|
||||
## 🔗 Sources
|
||||
|
||||
- [Paho v0.23.0 release](https://github.com/eclipse-paho/paho.golang/releases/tag/v0.23.0)
|
||||
- [Mosquitto broker configuration](https://mosquitto.org/man/mosquitto-conf-5.html)
|
||||
- [Password-file tool](https://mosquitto.org/man/mosquitto_passwd-1.html)
|
||||
- [Mosquitto publish tool](https://mosquitto.org/man/mosquitto_pub-1.html)
|
||||
|
||||
Next: [Section 2](02-go-connection.md).
|
||||
Reference in New Issue
Block a user