mqtt-5-learning/teaching/course-source/broker-operations.md

# Broker operations — maintain and deploy your lab

Use this alongside Lesson 1, then revisit it during Section 9. All commands run from `/home/ryan/Media/Code/Code/robot-market/mqtt-5-learning` and target the `mqtt5-learning` project. These are instructions to perform in your lab; writing this guide has not changed or started a broker.

## Lifecycle: know which state you are changing

| Operation | Example | Effect |
| --- | --- | --- |
| Stop | `docker compose -p mqtt5-learning stop broker` | Gracefully stops the container; retains it and its mounts |
| Start | `docker compose -p mqtt5-learning start broker` | Starts that existing container; does not apply edited Compose settings |
| Restart | `docker compose -p mqtt5-learning restart broker` | Stops and starts the existing container; interrupts clients and rereads broker startup configuration |
| Apply Compose changes | `docker compose -p mqtt5-learning up -d broker` | Reconciles the service definition and recreates the container when needed; keeps the named volume |
| Force recreation | `docker compose -p mqtt5-learning up -d --force-recreate broker` | Creates a fresh container even if Compose detects no change; useful after replacing a bind-mounted file |
| Remove containers/network | `docker compose -p mqtt5-learning down` | Stops and removes project containers/network; the declared named data volume remains |
| Delete data too | `docker compose -p mqtt5-learning down -v` | Also removes declared named volumes; this erases lab broker persistence |

`-p mqtt5-learning` selects the Compose project name, grouping this lab's service, network, and volume resources. It does not select a port. Keep it identical across all commands. Your existing `compose.yml` is valid; the lesson's `compose.yaml` is an alternative filename, not an additional required file.

After a password-file repair, explicitly restart an existing container and inspect `logs --since=1m broker` so historical failures are distinguished from the new attempt. `up -d` may leave an unchanged running container alone. Read-only entrypoint ownership warnings do not establish failure; verify the broker's own listener/error messages and authenticated messaging.

Use `restart` after editing broker configuration that requires a restart. Use `up -d` after editing ports, image tags, mounts, or container environment in Compose. Docker's [restart documentation](https://docs.docker.com/reference/cli/docker/compose/restart/) explains that restart does not apply Compose changes. For single-file bind mounts, an editor that replaces a file rather than writing in place can leave an existing container referring to the old file; recreation establishes fresh mounts.

After every change, inspect `ps`, inspect `logs --tail=50 broker`, and repeat an authenticated publish/subscribe check. These check container state, broker startup, and actual messaging respectively. `logs -f broker` follows new logs until Ctrl+C; Ctrl+C stops log following, not the detached broker.

## Choose the active credential store before maintenance

The basic lesson uses a host bind mount. This workspace's current `compose.yml` uses the external `mqtt5-learning_broker-secrets` volume because host permission changes did not stick. Inspect the service mount first. For this workspace, replace `-v "$PWD/lab/secrets:/work"` in the rotation commands below with `-v mqtt5-learning_broker-secrets:/work`. Update and back up the active volume; modifying the preserved host copy has no effect on the broker. See Lesson 1's filesystem fallback for provisioning and access checks. External secret volumes survive Compose `down -v`, but declared nonexternal broker-data volumes do not.

## Rotate a lab credential

Use the Lesson 1 utility command without `-c` to update the existing backend user:

```sh
docker run --rm -it --user 0 -v "$PWD/lab/secrets:/work" eclipse-mosquitto:2.0.22 mosquitto_passwd /work/passwords backend
docker run --rm --user 0 -v "$PWD/lab/secrets:/work" eclipse-mosquitto:2.0.22 chown mosquitto:mosquitto /work/passwords
docker run --rm --user 0 -v "$PWD/lab/secrets:/work" eclipse-mosquitto:2.0.22 chmod 600 /work/passwords
docker compose -p mqtt5-learning restart broker
```

The first command prompts for a new password and replaces only this user's hash. The next commands restore broker ownership and owner-only access. Restart reloads the file and disconnects existing clients. Update the backend terminal's password environment variable, reconnect, and test that the new password succeeds and the old one fails on a fresh connection. Simply editing a password file does not establish that already-connected clients have been revoked.

Mosquitto also supports reloading password-file configuration with SIGHUP; see its [authentication guide](https://mosquitto.org/documentation/authentication-methods/). Restart is the simpler explicit lab operation. For deployed systems, plan client credential rollout and reconnection before rotation to avoid unexpectedly stranding devices. Adding/removing a user also requires reviewing that user's ACL entries. `mosquitto_passwd -D /work/passwords username` removes a user; `-D` means delete, so use it only for an account you intend to remove.

## Backup and restore: what actually needs saving?

Save three things: the Compose/broker/ACL configuration, the protected password file, and the broker data volume. The Go application's in-memory records are a fourth, separate concern and are not covered by this backup. Password hashes and any TLS private keys remain sensitive backup material.

For a simple consistent lab backup, stop the broker gracefully first so it can write persistence state. Confirm `ps` reports it stopped. Use `docker volume ls` to list volumes and identify the project's broker-data volume; `ls` lists names, not contents. Verify it with `docker volume inspect <actual-volume-name>` before using it. Replace placeholders with the observed name; do not infer a production volume from an example name.

Create a private backup directory outside Git, archive the stopped volume with a temporary utility container mounting it read-only, and save configuration plus secrets with their permissions preserved. A typical volume-copy utility needs a source mount and a backup-output mount; run it with the permissions required to read the broker's files. The data volume can contain `mosquitto.db`, while credentials live in the separate host secrets directory. Copying only one of them is incomplete.

Restore into a separate empty lab volume while the target broker is stopped. Restore ownership/permissions, connect that volume to an isolated broker using the recorded image/configuration, and verify authentication, ACL denial, retained state, and session behavior. Do not extract an archive over an active broker data directory. A backup is useful only after a restore exercise succeeds; record the commands, actual volume name, date, image version, and observed recovered state in your journal.

## Upgrade with an observable rollback

Record the current image version and back up configuration, secrets, and stopped broker state. Read the target release's migration notes. Test the new pinned image against a restored copy in a separate lab project with a different loopback port; repeat authentication, ACL, reconnect/session, and retained-state checks before changing the primary lab.

`docker compose -p mqtt5-learning pull broker` downloads the image selected by the current Compose file; it does not replace the running container. After selecting a tested new image tag, `up -d broker` applies it. Keep the previous tag and backup: restoring an older image over data rewritten by a newer version is not a guaranteed rollback. For reproducibility beyond a version tag, record the resolved image digest as well.

## Preparing an actual remote deployment

The initial lesson exposes only loopback plaintext, so remote devices cannot connect. Do not merely change the bind address to make it reachable. Work through Section 11 before remote use: create a server certificate with the correct DNS/IP identity, configure a TLS listener, mount certificate/key files with broker-readable permissions, and make clients validate the CA and server identity. A password hash on disk does not encrypt credentials traveling over plaintext MQTT.

Choose the device-facing hostname, TLS port, firewall policy, per-device credentials/ACLs, persistent storage, backup/restore process, log retention, monitoring, and credential renewal responsibilities. Add a deliberate container restart policy such as `restart: unless-stopped` to the service if the deployment should recover after daemon/host restarts; the initial lab omits it to keep stop/start experiments explicit. A restart policy helps process availability but does not prove messaging readiness or provide high availability.

Container logs need a bounded retention/rotation policy so a busy broker cannot fill the host disk. Monitor connection failures, permission failures, disk use, and application heartbeat/command evidence. Test broker readiness through an authorized synthetic message round trip; a TCP port check alone cannot prove MQTT authentication or subscriptions work.

Keep provisioning repeatable with documented setup steps or a reviewed script, use the platform's secret provisioning mechanism, and plan certificate/password rotation and client rollout. Do not bake credentials or private keys into the image. Deployment-specific values belong in that environment's configuration, not in the learning project's committed secrets.

## Advanced Docker operations: predict the MQTT consequence

### Inspect the configuration without starting anything

```sh
docker compose -p mqtt5-learning config -q
docker compose -p mqtt5-learning ps -a
docker compose -p mqtt5-learning logs --since=2m --tail=100 broker
docker volume inspect mqtt5-learning_broker-data
```

`config -q` validates the resolved Compose model without printing it or contacting MQTT. `ps -a` includes stopped containers, helping distinguish "never created" from "exited on startup." The log filters restrict evidence to recent lines; they do not clear old logs. `volume inspect` shows volume metadata and its Docker host mountpoint, not the broker's session contents. The volume name above is the expected default for this project: if inspection fails, use `docker volume ls` and identify the actual lab volume rather than creating one to hide the mismatch.

Ask three separate questions: is the container running, did Mosquitto load the intended files, and can the authorized clients exchange packets? These commands answer only parts of that chain. Finish with the authenticated CLI round trip from Lesson 1.

### Understand execution context

```sh
docker compose -p mqtt5-learning exec -T broker mosquitto_pub --help
```

This runs the image's CLI in the existing broker container. `-T` disables the pseudo-terminal, useful for scripted output. `--help` is passed to `mosquitto_pub`; it performs no publish. When adapting the actual CLI test to this form, use container port 1883, not host port 18883. A new utility container started with `docker run` has a different loopback interface and does not reach the broker merely by using `127.0.0.1`.

### Isolate the session-persistence comparison

For Lesson 9's nonpersistent comparison, create `compose.session-test.yaml`:

```yaml
services:
  broker:
    volumes:
      - broker-data-session-test:/mosquitto/data
volumes:
  broker-data-session-test:
```

First stop the original lab. Temporarily set `persistence false` in its broker configuration, then inspect and apply the override:

```sh
docker compose -p mqtt5-learning stop broker
docker compose -p mqtt5-learning -f compose.yml -f compose.session-test.yaml config
docker compose -p mqtt5-learning -f compose.yml -f compose.session-test.yaml up -d --force-recreate broker
```

Use the actual primary filename if yours is `compose.yaml`. The later `-f` file overrides the data mount by its container destination `/mosquitto/data`, while keeping configuration and credential mounts. Verify that merged result before `up`. `--force-recreate` replaces the container so the changed file/mount is actually used; it does not delete the original data volume. Use an unused comparison volume name on repeat experiments so previous state cannot contaminate the result.

After measuring, stop the comparison with the same two `-f` files, restore `persistence true`, and run `docker compose -p mqtt5-learning -f compose.yml up -d --force-recreate broker`. This restores the original data mount. Inspect logs and verify authentication plus messaging again. The comparison volume remains for inspection; no volume deletion is required.

### Crash is different from graceful stop

For a deliberate broker crash experiment only:

```sh
docker compose -p mqtt5-learning kill -s SIGKILL broker
docker compose -p mqtt5-learning start broker
docker compose -p mqtt5-learning logs --since=2m broker
```

SIGKILL bypasses the broker's graceful persistence flush and interrupts every lab client. `start` runs the same container with the same mounted data; inspect what was actually recovered. Compare against `stop`, which permits graceful shutdown before Docker's stop timeout. A named volume survives both, but only state already written to it can be recovered after the crash. Do not infer crash durability from the existence of the volume.

These commands target the broker, so they test broker recovery. To test a device Will, terminate the simulator instead and keep the broker alive, as explained in [Lesson 10](10-expiry-presence-state.md).

## Practice and explain

1. Stop and start the lab; explain which resources survived.
2. Rotate one lab password and test both passwords on fresh connections.
3. Recreate the container and confirm the data volume remains mounted.
4. Perform an isolated backup/restore exercise and record what was recovered.
5. Explain what would change for TLS, remote access, automatic restart, and log retention.

Read the official [Compose up](https://docs.docker.com/reference/cli/docker/compose/up/), [Compose down](https://docs.docker.com/reference/cli/docker/compose/down/), and [Mosquitto configuration](https://mosquitto.org/man/mosquitto-conf-5.html) documentation when planning changes. Their latest pages may describe features beyond this lab's pinned Mosquitto 2.0.22; check version compatibility before adopting them.