feat: expand MQTT contract and reliability lessons

COURSE.md

@@ -221,6 +221,19 @@ Detailed lesson: [Section 11 explanations and experiments](lessons/11-security-b
 
 **Done when:** Positive and negative access tests behave as designed. Record the observed reason/error for denied operations; do not assume every broker denial is reported identically.
 
+### Optional Section 11B — Provision device ACLs with Mosquitto Dynamic Security
+
+Detailed lesson: [Section 11B Dynamic Security implementation](lessons/11b-dynamic-security.md).
+
+Use this extension when you want to replace the lab's static password/ACL files with live per-device provisioning. It uses a separate broker configuration and state file so the working Section 1 lab remains intact.
+
+1. [ ] Enable the plugin in an isolated Mosquitto configuration and persist its state file with broker-readable permissions.
+2. [ ] Create least-privilege device/backend roles, using a trusted identity-to-topic rule and default-deny behavior.
+3. [ ] Provision one unique device client and assign its role through a separately protected management identity.
+4. [ ] Test allowed and denied publish/subscribe operations, mismatched payload identity, revocation, and state persistence after broker restart.
+
+**Done when:** You can trace provisioning, broker-side ACL enforcement, backend validation, and revocation of a connected device. The production runtime should not receive broker-administrator credentials merely to process messages.
+
 ## Section 12 — Implement representative vending workflows
 
 **Outcome:** You understand the backend's main device-facing jobs using synthetic data.
@@ -237,6 +250,36 @@ Detailed lesson: [Section 11 explanations and experiments](lessons/11-security-b
 
 **Outcome:** Important state and deduplication survive application restart.
 
+### 📥 Choose the acknowledgement boundary for important work
+
+The MQTT client receives the broker's QoS 1 PUBLISH and invokes the registered callback. In automatic acknowledgement mode, Paho sends PUBACK after the callback returns—even if the callback returns an error. Returning an error does not ask the broker to retry. A callback that places a message in a RAM queue and returns can therefore acknowledge it before a worker handles it; a process crash can then lose work the broker considers delivered.
+
+With Paho's manual acknowledgement enabled, the application calls `received.Client.Ack(received.Packet)` when its chosen acceptance boundary has been reached. For important reports, make that boundary a successful database transaction: validate the topic and payload, then atomically save the message/deduplication key and the business effect, and only then acknowledge. Keep the original client and packet available to the worker until `Ack` succeeds. Manual acknowledgement delays protocol completion; it does not make an in-memory queue durable.
+
+> **❓ Question:** **If the database is unavailable, should an important report be acknowledged?**
+>
+> **💬 Answer:** Not before the application has safely recorded it. Leave it unacknowledged and follow a deliberate connection/recovery policy, or reject/quarantine it durably if it is permanently invalid. Returning a callback error alone neither acknowledges nor requests redelivery in manual mode. Avoid leaving poison messages unacknowledged forever: they consume in-flight capacity and can stall later QoS 1/2 deliveries.
+
+### 🔁 Make database effects idempotent
+
+QoS 1 can deliver a message again when a connection fails before the broker receives PUBACK. Application retries and crash recovery can also repeat work. Use a stable application-level identity—such as `(device_id, report_id)`—and enforce it with a database unique constraint. In one transaction, decide whether this is new work, record the deduplication key, and apply the business effect. A duplicate should return or reproduce the prior application result without applying the effect twice. MQTT packet identifiers are temporary protocol identifiers, so they are not suitable business deduplication keys.
+
+For sales reports, commit the report record and its inventory/revenue effect atomically. For command results, commit the validated result and state transition together. A stable ID helps the application recognize repeats; the database constraint makes that decision safe across concurrent workers and process restarts.
+
+### 🧹 Give invalid messages a durable disposition
+
+Topic parsing and payload validation happen before business state changes. If a message is permanently invalid, record a bounded rejection reason and enough safe identity/context to diagnose it, or move it to a durable quarantine. After that disposition commits, acknowledge it so it does not block the stream. Do not log credentials or dump untrusted payloads. If failure is transient—for example, the database is temporarily unavailable—do not mark the message permanently rejected; keep it recoverable and apply a bounded retry/reconnect policy.
+
+### ⚖️ Match QoS and queue policy to the data
+
+QoS 1 is a practical choice for important ingestion when the application is built to tolerate duplicates. QoS 2 adds a protocol handshake; it does not make a database transaction or physical device effect exactly once. For disposable telemetry, automatic acknowledgement and bounded best-effort processing can be acceptable when drops are counted and freshness matters more than replay. For reports and command state, prefer durable admission and idempotent processing.
+
+Keep the callback short: copy the needed message data and submit it to a bounded worker queue. For important work, preserve the acknowledgement handle with the queued item and acknowledge only after the durable transaction commits. Bound queue capacity and configure MQTT 5 Receive Maximum; monitor queue depth, unacknowledged deliveries, processing failures, duplicate counts, and acknowledgement latency. Receive Maximum limits protocol in-flight QoS 1/2 deliveries, not the size of the Go queue.
+
+If the transaction must also trigger a later MQTT publication, do not commit the database and then rely on an immediate publish alone: a crash between the commit and publication can lose the response. Store an outbox entry in the same transaction and publish it from a retrying worker. Section 14 develops that crash-recovery pattern.
+
+📚 **Builds on:** [Lesson 5](lessons/05-contracts-validation.md) validates topics and payloads; [Lesson 6](lessons/06-concurrency-cancellation.md) explains callbacks, manual acknowledgement, and bounded queues; [Lesson 7](lessons/07-command-response.md) introduces stable IDs and idempotent effects; [Lesson 16](lessons/16-mqtt5-controls.md) covers Receive Maximum. Section 14 adds the transactional outbox.
+
 1. [ ] Add a dedicated lab PostgreSQL instance and migrations for devices, commands, command results, and sales reports. Use `database/sql` with pgx stdlib to match the observed connector style.
 2. [ ] Move important in-memory repositories behind small interfaces. Use contexts, query deadlines, bounded connection pools, and transaction boundaries.
 3. [ ] Enforce uniqueness for device/report IDs and command IDs. Persist a deduplication decision and its corresponding database effect atomically.