Robot-Market/mqtt-5-learning
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Refactor MQTT course and bound backend message processing

Browse Source 
Move lesson source into the teaching structure, align the course path with the learner's Go and DevOps experience, and add the interactive lesson guides. Add advanced DevOps lifecycle study as a separate section.

Bound MQTT message work and drain accepted jobs during shutdown; align the backend implementation with the revised course through Section 6.
This commit is contained in:
Ryan
2026-10-22 10:19:56 +03:30
parent f9ab91ed60
commit fb30e8e884
70 changed files with 7492 additions and 312 deletions
Download Patch File Download Diff File Expand all files Collapse all files
COURSE.md
+82 -52
View File
@@ -2,13 +2,13 @@
## Starting point and destination
You already understand MQTT 3.1.1 concepts. This course concentrates on writing, debugging, testing, and maintaining Go services that communicate with IoT devices. Go and basic Docker syntax are assumed. Explain MQTT semantics where code depends on them, including packet flows, guarantees, failure boundaries, and the purpose and effect of MQTT or advanced Docker commands. Each experiment starts with a prediction and ends with an explanation, not merely successful copied output.
You have four years of Go backend experience and are confident with Go syntax and core language concepts. You also understand MQTT 3.1.1. Treat both as established skills: do not spend lesson time teaching Go basics or introductory MQTT. Focus on designing, debugging, testing, and operating Go services that communicate with IoT devices, including the MQTT 5 differences and the places where protocol behavior affects application guarantees. Explain Go details only when they materially affect a design decision, such as ownership, concurrency, cancellation, error handling, or API shape. Explain packet flows, failure boundaries, and the purpose and effect of MQTT or advanced Docker commands. Each experiment starts with a prediction and ends with an explanation, not merely successful copied output.
You will build a small vending backend and a Go device simulator, then use that experience to understand the MQTT 5 backend delivered to you. Work through the numbered sections in order. Each checkbox is an exercise you implement; the tutor provides explanations, small examples, debugging help, and code review. A section can take several sessions. There is no need to finish a whole section in one sitting.
This is the requested course and implementation checklist. It does not require another round of sprint planning. Ask the tutor to start a numbered exercise when ready; full solutions should wait until you have attempted it or requested one.
Start with [Lesson 0 — The broker, Go backend, and device lifecycle](lessons/00-system-lifecycle.md) for the end-to-end story. Lessons 1–8 now begin with the system story and a guiding question before each implementation step.
Start with [Lesson 0 — The broker, Go backend, and device lifecycle](teaching/course-source/00-system-lifecycle.md) for the end-to-end story. Lessons 1–8 now begin with the system story and a guiding question before each implementation step.
All implementation belongs in `/home/ryan/Media/Code/Code/robot-market/mqtt-5-learning`. Use a dedicated local broker, synthetic data, lab credentials, and simulated devices. Reference the existing backend read-only.
@@ -42,6 +42,8 @@ The future delivered backend must be inspected again. Do not assume it will use
## Course rules and working structure
- Build one evolving application, adding only the structure needed by the current section.
- Assume fluency with Go syntax, pointers, interfaces, channels, standard-library patterns, and ordinary testing. Spend explanation time on system boundaries, concurrency/lifecycle decisions, tradeoffs, failure modes, and evidence; revisit a Go concept only when its precise behavior affects the exercise.
- Keep implementation exercises substantive: ask the learner to make and defend a design choice, implement the relevant slice, and explain what the observed result proves and does not prove. Do not replace implementation practice with a conceptual overview.
- Start with `cmd/backend` and `cmd/simulator`; introduce API and ingestion executables in Section 15.
- Add `internal/config`, `internal/mqtt`, `internal/contracts`, and domain packages as they become useful. Keep interfaces small and driven by callers.
- Keep upstream code under `references/paho.golang/` unchanged. Read `autopaho/examples/basics` first; use `backoff`, `queue`, `router`, and `rpc` when their lessons arrive. Lower-level examples also live under `paho/cmd/`.
@@ -53,7 +55,7 @@ The future delivered backend must be inspected again. Do not assume it will use
## Section 1 — Prepare an isolated, reproducible lab
Detailed lesson: [Section 1 instructions and examples](lessons/01-lab-setup.md).
Detailed lesson: [Section 1 instructions and examples](teaching/course-source/01-lab-setup.md).
Start with the broker's role and setup: which configuration belongs to Mosquitto, how client accounts are supplied, and what local CLI evidence confirms the broker is ready for Go clients.
@@ -74,7 +76,7 @@ Start with the broker's role and setup: which configuration belongs to Mosquitto
## Section 2 — Write your first Go MQTT 5 connection
Detailed lesson: [Section 2 connection lifecycle and code](lessons/02-go-connection.md).
Detailed lesson: [Section 2 connection lifecycle and code](teaching/course-source/02-go-connection.md).
Follow the backend from configuration through AutoPaho, Paho, CONNECT, and CONNACK. The story question is what proves connection acceptance and how startup differs from a running process.
@@ -92,7 +94,7 @@ Follow the backend from configuration through AutoPaho, Paho, CONNECT, and CONNA
## Section 3 — Add typed configuration
Detailed lesson: [Section 3 TOML and koanf implementation](lessons/03-typed-configuration.md).
Detailed lesson: [Section 3 TOML and koanf implementation](teaching/course-source/03-typed-configuration.md).
Trace how each Go process gets its own identity and credentials, how configuration sources merge, and when validation must finish before MQTT startup.
@@ -108,51 +110,67 @@ Trace how each Go process gets its own identity and credentials, how configurati
## Section 4 — Publish, subscribe, and simulate a device
Detailed lesson: [Section 4 Paho transport, backend, and simulator](lessons/04-publish-subscribe-simulator.md).
Detailed lesson: [Section 4 Paho transport, backend, and simulator](teaching/course-source/04-publish-subscribe-simulator.md).
Start with the story of how the backend and simulator connect, subscribe, exchange a command, and report a device-level response before building the transport helper.
**Outcome:** Two Go processes exchange messages through the broker.
1. [ ] Create a simulator with its own identity and client ID. Give the backend and simulator different topic permissions.
2. [ ] Define a lab namespace, for example `lab/v1/devices/{deviceID}/...`, with separate telemetry, commands, and responses topics. These are lab contracts, not changes to existing production topics.
2. [ ] Define the lab hierarchy `lab/v1/devices/{deviceID}/{kind}` in `internal/contracts/topic.go`. Use distinct Go types/builders for concrete publish names, fleet-wide filters, and exact-device filters; keep role-specific subscription choices at process wiring and broker authorization in Mosquitto ACLs. These are lab contracts, not changes to existing production topics.
3. [ ] Subscribe, inspect SUBACK results, and wait for subscription readiness before publishing a message that must be received.
4. [ ] Publish at QoS 0 and 1. Inspect errors and acknowledgements rather than assuming a nil transport error proves the device acted.
4. [ ] Compare QoS 0, 1, and 2 for a lab publication. Trace the QoS 2 handshake and compare its protocol guarantee with application deduplication and business completion; inspect errors and acknowledgements rather than assuming a nil transport error proves the device acted.
5. [ ] Send heartbeats from the simulator; receive a harmless command and return a simple response.
**Done when:** You can trace CONNECT, SUBSCRIBE/SUBACK, PUBLISH, and QoS 1 PUBACK for both directions. Explain why a PUBACK is not a business response and why QoS is evaluated per delivery hop.
## Section 5 — Design contracts and validate inbound messages
Detailed lesson: [Section 5 instructions](lessons/05-contracts-validation.md).
Detailed lesson: [Section 5 instructions](teaching/course-source/05-contracts-validation.md).
**Outcome:** A malformed or misleading message cannot corrupt application state.
1. [ ] Define versioned JSON structures for heartbeat, command, command result, and synthetic sales report. Include device identity, stable message ID, and relevant timestamps.
2. [ ] Validate topic shape, topic/body identity agreement, payload size, schema version, required values, and timestamps. Decide explicitly how unknown fields and unsupported versions behave.
2. [ ] Use `contracts.ParseTopic` from `topic.go`, then validate topic/body identity agreement, payload size, schema version, required values, and timestamps. Decide explicitly how unknown fields and unsupported versions behave.
3. [ ] Separate topic parsing and decoding from business handlers. Introduce Content Type and Payload Format Indicator as metadata; still validate the actual payload.
4. [ ] Add table-driven tests for malformed JSON, identity mismatch, unknown commands, missing IDs, and boundary sizes.
5. [ ] Return safe application errors without echoing confidential content.
**Done when:** Valid messages route correctly; invalid messages have consistent outcomes. Explain why a body field claiming a device ID does not authenticate the sender.
## Section 5A — Turn validated messages into backend state
Detailed lesson: [Section 5A — Backend message processing](teaching/course-source/05a-backend-message-processing.md).
This is the missing application step between the first five lessons and concurrency. Section 5 proves a message is structurally valid; this section routes it to a use case, updates a small in-memory device view from heartbeats, and records safe structured outcomes. It deliberately does not implement durable sales storage, full device-log ingestion, or vending configuration workflows.
**Outcome:** You can explain and implement the path from a valid MQTT message to an application state change, and distinguish a recent heartbeat from a live broker connection or full device-health result.
1. [ ] Separate transport receipt, contract validation, application service, state store, and application logging.
2. [ ] Store the newest heartbeat observation per device; compare device report time for ordering and backend receive time for freshness.
3. [ ] Expose a status query that reports whether a heartbeat arrived within a configured window without claiming the device is definitely offline.
4. [ ] Log stable identifiers and processing outcomes without dumping arbitrary payloads.
5. [ ] State the RAM-loss boundary; keep sales persistence, command correlation, and diagnostic-log workflows for their later sections.
**Done when:** A valid heartbeat updates the device view; malformed and older messages do not replace it; you can explain why MQTT PUBACK, heartbeat freshness, device health, and business completion are separate evidence.
## Section 6 — Manage callbacks, concurrency, and cancellation
Detailed lesson: [Section 6 instructions](lessons/06-concurrency-cancellation.md).
Detailed lesson: [Section 6 instructions](teaching/course-source/06-concurrency-cancellation.md).
**Outcome:** Message handling remains responsive under concurrent traffic.
1. [ ] Inspect the selected Paho release's callback execution and acknowledgement behavior. Document which work happens before and after callback return.
2. [ ] Add bounded work queues and a small worker pool. Define an overload policy separately for disposable telemetry and important commands/reports.
3. [ ] Protect shared state with a deliberate ownership or locking model. Copy message data before retaining it if required by the API's ownership contract.
2. [ ] Move Section 5A's application service behind bounded work queues and a small worker pool. Define an overload policy separately for telemetry and important commands/reports.
3. [ ] Protect the shared device view with a deliberate ownership or locking model. Copy message data before retaining it if required by the API's ownership contract.
4. [ ] Propagate lifecycle contexts and operation deadlines; avoid unbounded goroutines and per-message background contexts.
5. [ ] Add a slow handler and a burst generator; observe queue saturation and shutdown while work is pending.
5. [ ] Add a slow handler and a burst generator; observe queue saturation and shutdown while work is pending. Record queue depth, processing latency, and dropped or rejected work so overload is visible before the operations section.
**Done when:** The learning module passes `go test -race ./...`, and overload behavior is observable and bounded. Explain the possible loss window if MQTT acknowledges a message before queued application work is durable.
## Section 7 — Build reliable command/response behavior
Detailed lesson: [Section 7 explanations and experiments](lessons/07-command-response.md).
Detailed lesson: [Section 7 explanations and experiments](teaching/course-source/07-command-response.md).
**Outcome:** Concurrent commands are matched to the correct device responses.
@@ -164,14 +182,18 @@ Detailed lesson: [Section 7 explanations and experiments](lessons/07-command-res
**Done when:** Many concurrent commands receive the right results; duplicate commands produce one simulated effect; late responses follow a documented policy. Explain why MQTT Correlation Data alone supplies neither authorization nor deduplication.
**State boundary:** Command waiters and command state are in memory at this stage. A process restart can lose them; an HTTP/API acceptance here is not durable acceptance. Sections 13–14 add persisted command state and recoverable publication intent.
**Read:** AutoPaho's `rpc` example as a mechanism reference, then compare its scope with your application state machine.
## Section 8 — Expose device commands through HTTP
Detailed lesson: [Section 8 instructions](lessons/08-http-commands.md).
Detailed lesson: [Section 8 instructions](teaching/course-source/08-http-commands.md).
**Outcome:** An API caller can create a command and query its status.
This endpoint is a local lab exercise. Keep it bound to the lab environment and treat it as unauthenticated until Section 11 adds API authentication and per-device authorization. Command state is still in memory, so do not describe an accepted request as durable until Sections 13–14.
1. [ ] Use `net/http` for `POST /devices/{id}/commands` and `GET /commands/{id}`. Return a command ID and asynchronous acceptance when appropriate.
2. [ ] Separate HTTP decoding, command service, and MQTT transport. Keep command processing independent of a disconnected HTTP caller after acceptance.
3. [ ] Add request-body limits, JSON error responses, server timeouts, and controlled shutdown.
@@ -182,12 +204,12 @@ Detailed lesson: [Section 8 instructions](lessons/08-http-commands.md).
## Section 9 — Learn MQTT 5 sessions through failure experiments
Detailed lesson: [Section 9 instructions](lessons/09-sessions-recovery.md).
Detailed lesson: [Section 9 instructions](teaching/course-source/09-sessions-recovery.md).
**Outcome:** You can predict and explain reconnect and offline delivery behavior.
1. [ ] Compare stable and changing client IDs, Clean Start choices, zero/nonzero Session Expiry Interval, and the broker's Session Present result.
2. [ ] Disconnect the simulator, publish a QoS 1 command, and reconnect before and after expiry. Record the subscription and session prerequisites for queued delivery.
2. [ ] Disconnect the simulator, publish a QoS 1 command, and reconnect before and after expiry. Record the subscription and session prerequisites for queued delivery. Compare this queued one-shot command with a retained publication, which represents the latest retained value rather than an offline event history.
3. [ ] Test subscription restoration when a session exists and when it does not. Observe reconnect callbacks and avoid accidental repeated local handler registration.
4. [ ] Repeat with broker restart, first with persistence disabled and then configured. Record the limits of the actual broker persistence behavior.
5. [ ] Compare client session storage, the outgoing publication queue, broker session state, and your application state in a four-column recovery table.
@@ -196,7 +218,7 @@ Detailed lesson: [Section 9 instructions](lessons/09-sessions-recovery.md).
## Section 10 — Handle expiry, presence, and desired state
Detailed lesson: [Section 10 explanations and experiments](lessons/10-expiry-presence-state.md).
Detailed lesson: [Section 10 explanations and experiments](teaching/course-source/10-expiry-presence-state.md).
**Outcome:** Stale work and stale device status are handled deliberately.
@@ -209,7 +231,7 @@ Detailed lesson: [Section 10 explanations and experiments](lessons/10-expiry-pre
## Section 11 — Secure device communication and API access
Detailed lesson: [Section 11 explanations and experiments](lessons/11-security-boundaries.md).
Detailed lesson: [Section 11 explanations and experiments](teaching/course-source/11-security-boundaries.md).
**Outcome:** Unauthorized clients cannot control another device or observe its private traffic.
@@ -223,7 +245,7 @@ Detailed lesson: [Section 11 explanations and experiments](lessons/11-security-b
### Optional Section 11B — Provision device ACLs with Mosquitto Dynamic Security
Detailed lesson: [Section 11B Dynamic Security implementation](lessons/11b-dynamic-security.md).
Detailed lesson: [Section 11B Dynamic Security implementation](teaching/course-source/11b-dynamic-security.md).
Use this extension when you want to replace the lab's static password/ACL files with live per-device provisioning. It uses a separate broker configuration and state file so the working Section 1 lab remains intact.
@@ -236,6 +258,8 @@ Use this extension when you want to replace the lab's static password/ACL files
## Section 12 — Implement representative vending workflows
Detailed lesson: [Lesson 12 — Representative vending workflows](teaching/course-source/12-vending-workflows.md).
**Outcome:** You understand the backend's main device-facing jobs using synthetic data.
1. [ ] Add synthetic sales reports and application responses keyed by report ID. Deduplicate in memory and identify the restart limitation.
@@ -248,37 +272,11 @@ Use this extension when you want to replace the lab's static password/ACL files
## Section 13 — Persist reports and command state in PostgreSQL
Detailed lesson: [Lesson 13 — Persist reports and command state](teaching/course-source/13-persist-reports-command-state.md).
**Outcome:** Important state and deduplication survive application restart.
### 📥 Choose the acknowledgement boundary for important work
The MQTT client receives the broker's QoS 1 PUBLISH and invokes the registered callback. In automatic acknowledgement mode, Paho sends PUBACK after the callback returns—even if the callback returns an error. Returning an error does not ask the broker to retry. A callback that places a message in a RAM queue and returns can therefore acknowledge it before a worker handles it; a process crash can then lose work the broker considers delivered.
With Paho's manual acknowledgement enabled, the application calls `received.Client.Ack(received.Packet)` when its chosen acceptance boundary has been reached. For important reports, make that boundary a successful database transaction: validate the topic and payload, then atomically save the message/deduplication key and the business effect, and only then acknowledge. Keep the original client and packet available to the worker until `Ack` succeeds. Manual acknowledgement delays protocol completion; it does not make an in-memory queue durable.
> **❓ Question:** **If the database is unavailable, should an important report be acknowledged?**
>
> **💬 Answer:** Not before the application has safely recorded it. Leave it unacknowledged and follow a deliberate connection/recovery policy, or reject/quarantine it durably if it is permanently invalid. Returning a callback error alone neither acknowledges nor requests redelivery in manual mode. Avoid leaving poison messages unacknowledged forever: they consume in-flight capacity and can stall later QoS 1/2 deliveries.
### 🔁 Make database effects idempotent
QoS 1 can deliver a message again when a connection fails before the broker receives PUBACK. Application retries and crash recovery can also repeat work. Use a stable application-level identity—such as `(device_id, report_id)`—and enforce it with a database unique constraint. In one transaction, decide whether this is new work, record the deduplication key, and apply the business effect. A duplicate should return or reproduce the prior application result without applying the effect twice. MQTT packet identifiers are temporary protocol identifiers, so they are not suitable business deduplication keys.
For sales reports, commit the report record and its inventory/revenue effect atomically. For command results, commit the validated result and state transition together. A stable ID helps the application recognize repeats; the database constraint makes that decision safe across concurrent workers and process restarts.
### 🧹 Give invalid messages a durable disposition
Topic parsing and payload validation happen before business state changes. If a message is permanently invalid, record a bounded rejection reason and enough safe identity/context to diagnose it, or move it to a durable quarantine. After that disposition commits, acknowledge it so it does not block the stream. Do not log credentials or dump untrusted payloads. If failure is transient—for example, the database is temporarily unavailable—do not mark the message permanently rejected; keep it recoverable and apply a bounded retry/reconnect policy.
### ⚖️ Match QoS and queue policy to the data
QoS 1 is a practical choice for important ingestion when the application is built to tolerate duplicates. QoS 2 adds a protocol handshake; it does not make a database transaction or physical device effect exactly once. For disposable telemetry, automatic acknowledgement and bounded best-effort processing can be acceptable when drops are counted and freshness matters more than replay. For reports and command state, prefer durable admission and idempotent processing.
Keep the callback short: copy the needed message data and submit it to a bounded worker queue. For important work, preserve the acknowledgement handle with the queued item and acknowledge only after the durable transaction commits. Bound queue capacity and configure MQTT 5 Receive Maximum; monitor queue depth, unacknowledged deliveries, processing failures, duplicate counts, and acknowledgement latency. Receive Maximum limits protocol in-flight QoS 1/2 deliveries, not the size of the Go queue.
If the transaction must also trigger a later MQTT publication, do not commit the database and then rely on an immediate publish alone: a crash between the commit and publication can lose the response. Store an outbox entry in the same transaction and publish it from a retrying worker. Section 14 develops that crash-recovery pattern.
📚 **Builds on:** [Lesson 5](lessons/05-contracts-validation.md) validates topics and payloads; [Lesson 6](lessons/06-concurrency-cancellation.md) explains callbacks, manual acknowledgement, and bounded queues; [Lesson 7](lessons/07-command-response.md) introduces stable IDs and idempotent effects; [Lesson 16](lessons/16-mqtt5-controls.md) covers Receive Maximum. Section 14 adds the transactional outbox.
📚 **Detailed teaching and experiments:** [Lesson 13](teaching/course-source/13-persist-reports-command-state.md).
1. [ ] Add a dedicated lab PostgreSQL instance and migrations for devices, commands, command results, and sales reports. Use `database/sql` with pgx stdlib to match the observed connector style.
2. [ ] Move important in-memory repositories behind small interfaces. Use contexts, query deadlines, bounded connection pools, and transaction boundaries.
@@ -290,6 +288,9 @@ If the transaction must also trigger a later MQTT publication, do not commit the
## Section 14 — Close database/MQTT failure gaps
Detailed lesson: [Lesson 14 — Close database/MQTT failure gaps](teaching/course-source/14-database-mqtt-failure-gaps.md).
Interactive guide: [Lesson 14 — transactional outbox](teaching/lessons/0014-outbox-failure-gaps.html).
**Outcome:** Accepted work can be recovered across failures between storage and publication.
1. [ ] Draw the failure windows between saving a command, publishing it, receiving PUBACK, and recording the device result.
@@ -302,6 +303,8 @@ If the transaction must also trigger a later MQTT publication, do not commit the
## Section 15 — Split API and ingestion processes
Interactive guide: [Lesson 15 — Split API and ingestion](teaching/lessons/0015-process-boundaries.html).
**Outcome:** Your application resembles the backend's separate process responsibilities.
1. [ ] Extract `cmd/api` and `cmd/ingest` with separate lifecycle contexts and client IDs. Decide explicitly which process or worker owns publication.
@@ -314,7 +317,8 @@ If the transaction must also trigger a later MQTT publication, do not commit the
## Section 16 — Explore advanced MQTT 5 controls and scaling
Detailed lesson: [Section 16 explanations and experiments](lessons/16-mqtt5-controls.md).
Detailed lesson: [Section 16 explanations and experiments](teaching/course-source/16-mqtt5-controls.md).
Interactive guide: [Lesson 16 — MQTT 5 controls](teaching/lessons/0016-mqtt5-controls.html).
**Outcome:** You can choose MQTT features based on measured behavior.
@@ -328,6 +332,8 @@ Detailed lesson: [Section 16 explanations and experiments](lessons/16-mqtt5-cont
## Section 17 — Operate and diagnose the service
Interactive guide: [Lesson 17 — Operate and diagnose](teaching/lessons/0017-operations-diagnostics.html).
**Outcome:** You can identify whether a failure is in HTTP, the application, broker, device, or database.
1. [ ] Add structured logs with stable command/report IDs and safe error categories. Add metrics for reconnects, rejected messages, queue depth, duplicate handling, publish latency, response latency, timeouts, and DB failures.
@@ -338,8 +344,31 @@ Detailed lesson: [Section 16 explanations and experiments](lessons/16-mqtt5-cont
**Done when:** Another person could reproduce and diagnose a failed command using your runbook without reading all your code. Relevant Go tests and race checks pass for the learning application.
## Section 17A — Practice the advanced DevOps service lifecycle
Detailed lesson: [Advanced DevOps service lifecycle](teaching/course-source/17a-advanced-devops-lifecycle.md).
This separate lesson builds on the junior-level DevOps knowledge assumed by the course. It follows the backend from a source change through validation, artifact creation, deployment, rollout, operation, recovery, and improvement. Use the isolated learning project and synthetic devices; Kubernetes is not required, and production deployment is outside this course.
**Outcome:** You can design and explain a repeatable release lifecycle for the backend, simulator, broker configuration, and database changes, with evidence and a recovery path at each stage.
1. [ ] Map source, CI, build artifact, configuration/secrets, runtime dependencies, and operator ownership. Identify which state belongs to PostgreSQL, the broker, and each service process.
2. [ ] Define pipeline gates for formatting, static checks, unit tests, race checks where relevant, and isolated broker/database integration checks. Separate checks that can run on every change from environment-dependent acceptance checks.
3. [ ] Package the Go services as reproducible, versioned images. Run as a non-root user, keep credentials out of the image and build logs, handle termination signals, and define resource and health-check behavior.
4. [ ] Design environment configuration and secret injection, validation, rotation, and least-privilege runtime identities. Make development, test, and staging differences explicit without copying production secrets into the lab.
5. [ ] Plan compatible releases for services and database schema changes. Practice an expand/migrate/contract sequence, immutable artifact promotion, rollout gates, and rollback limits when data has already changed.
6. [ ] Exercise deployment and shutdown with readiness, draining, in-flight command behavior, and independent API/ingestion restarts. Decide which changes can roll back automatically and which require operator judgment.
7. [ ] Define service indicators and objectives, alert conditions, dashboards, and an incident sequence from detection through mitigation, recovery, and review. Use the telemetry introduced in Sections 6 and 17.
8. [ ] Write and rehearse backup/restore and disaster-recovery procedures for PostgreSQL and broker state. Set example recovery-point and recovery-time objectives, then collect evidence from a restore drill rather than treating a successful backup as proof.
9. [ ] Review dependency and base-image updates, vulnerability findings, artifact provenance, access to deployment credentials, and audit records. Decide how a release is paused when a gate fails.
10. [ ] Produce a release package: pipeline diagram, deployment checklist, rollback/recovery plan, runbook links, and a post-deployment verification record for a simulated release.
**Done when:** A teammate can follow your documented path from a code change to a verified lab release, identify the owner of each state boundary, and recover from one injected deployment or dependency failure without losing durable commands or reports.
## Section 18 — Prepare for the MQTT 5 backend handover
Interactive guide: [Lesson 18 — Backend handover](teaching/lessons/0018-backend-handover.html).
**Outcome:** You can inspect, explain, and safely maintain the delivered system.
1. [ ] Reinspect the delivered commit, dependencies, broker configuration, device protocol contracts, and deployment topology. Build an explicit MQTT 3.1.1-to-5 behavior comparison rather than an import-replacement checklist.
@@ -359,13 +388,14 @@ Detailed lesson: [Section 16 explanations and experiments](lessons/16-mqtt5-cont
| 8 | HTTP command reaches the device and exposes an accurate status |
| 12 | Secured lab demonstrates sales, catalog, configuration, and health flows |
| 15 | Persistent API/ingestion processes recover independently |
| 17A | A simulated release passes its gates, is verified after rollout, and has a rehearsed recovery path |
| 18 | You can trace, test, diagnose, and review the delivered backend |
## How to start a lesson with the tutor
> Read `prompt.md` and `COURSE.md`. Start Section 1, exercise 1. I know MQTT 3.1.1 concepts. Explain only what I need for this exercise, give me a small task and expected result, then wait for my attempt. Review my result before moving on. Do not generate the complete application.
For later sessions, replace the section and exercise number and include your current code, command output, or question. The tutor should ask about Go experience only where it affects the current exercise; Docker availability is established in Section 1.
For later sessions, replace the section and exercise number and include your current code, command output, or question. Assume the learner is a mid-level Go backend engineer with four years of experience; do not recheck their Go level or explain syntax unless asked. Docker availability is established in Section 1.
## Official references