updated and tested till end of lesson 4

2026-10-11 15:00:58 +03:30
parent d73090946b
commit 2d639d78eb
18 changed files with 1120 additions and 187 deletions
@@ -8,6 +8,8 @@ You will build a small vending backend and a Go device simulator, then use that
 This is the requested course and implementation checklist. It does not require another round of sprint planning. Ask the tutor to start a numbered exercise when ready; full solutions should wait until you have attempted it or requested one.
 Start with [Lesson 0 — The broker, Go backend, and device lifecycle](lessons/00-system-lifecycle.md) for the end-to-end story. Lessons 1–8 now begin with the system story and a guiding question before each implementation step.
 All implementation belongs in `/home/ryan/Media/Code/Code/robot-market/mqtt-5-learning`. Use a dedicated local broker, synthetic data, lab credentials, and simulated devices. Reference the existing backend read-only.
 ## How this course matches your backend
@@ -53,13 +55,15 @@ The future delivered backend must be inspected again. Do not assume it will use
 Detailed lesson: [Section 1 instructions and examples](lessons/01-lab-setup.md).
 Start with the broker's role and setup: which configuration belongs to Mosquitto, how client accounts are supplied, and what local CLI evidence confirms the broker is ready for Go clients.
 **Outcome:** A local MQTT 5 broker you can reach, plus a recorded toolchain and dependency choice.
 1. [ ] Check your Go and Docker installation. Record `go version`, `docker version`, and `docker compose version`. If Docker is unavailable, establish an equivalent isolated local Mosquitto setup before continuing.
 2. [ ] Read the local Paho README and the basics example. Identify which responsibilities belong to AutoPaho and which belong to the underlying Paho client. Check a tagged release's Go requirement and release notes before pinning it.
 3. [ ] Initialize a Go module in this learning project. Record the selected Go, Paho, and Mosquitto versions; avoid floating `latest` versions for reproducible exercises.
 4. [ ] Create a Compose definition with a project-specific name and a broker listener published only on loopback, using an unused lab port such as `127.0.0.1:18883`. Use a separate persistence volume and configuration directory.
-5. [ ] Disable anonymous access. Create distinct lab backend and simulator credentials and an initial ACL allowing only the lab topic namespace. Keep passwords and generated secrets out of version control.
+5. [ ] Disable anonymous access. Create distinct lab backend and simulator usernames and an initial ACL allowing only the lab topic namespace. For this disposable local test lab, store the shared password in `configs/lab.toml`; never reuse a real credential.
 6. [ ] Use Mosquitto CLI tools with MQTT version 5 explicitly selected to subscribe and publish a synthetic message. Check the installed tools' help for version and authentication flags. Repeat with incorrect credentials.
 **Initial commands:** `docker compose -p mqtt5-learning config`, then `docker compose -p mqtt5-learning up -d`, `docker compose -p mqtt5-learning ps`, and `docker compose -p mqtt5-learning logs --tail=50`. Run these after you have written the lab files; never use a production Compose directory. Do not share resolved Compose output if it contains secrets.
@@ -72,6 +76,8 @@ Detailed lesson: [Section 1 instructions and examples](lessons/01-lab-setup.md).
 Detailed lesson: [Section 2 connection lifecycle and code](lessons/02-go-connection.md).
 Follow the backend from configuration through AutoPaho, Paho, CONNECT, and CONNACK. The story question is what proves connection acceptance and how startup differs from a running process.
 **Outcome:** A small executable that connects, reports connection state, and exits cleanly.
 1. [ ] Create `cmd/backend/main.go` and use AutoPaho with a unique, explicit client ID. Explain the MQTT 5 CONNECT/CONNACK exchange and inspect failure reasons exposed by the selected API.
@@ -88,6 +94,8 @@ Detailed lesson: [Section 2 connection lifecycle and code](lessons/02-go-connect
 Detailed lesson: [Section 3 TOML and koanf implementation](lessons/03-typed-configuration.md).
 Trace how each Go process gets its own identity and credentials, how configuration sources merge, and when validation must finish before MQTT startup.
 **Outcome:** Both executables can be configured predictably without editing Go code.
 1. [ ] Add typed TOML + koanf configuration for broker URL, client ID, credential inputs, timeouts, log level, and simulator identity.
@@ -102,6 +110,8 @@ Detailed lesson: [Section 3 TOML and koanf implementation](lessons/03-typed-conf
 Detailed lesson: [Section 4 Paho transport, backend, and simulator](lessons/04-publish-subscribe-simulator.md).
 Start with the story of how the backend and simulator connect, subscribe, exchange a command, and report a device-level response before building the transport helper.
 **Outcome:** Two Go processes exchange messages through the broker.
 1. [ ] Create a simulator with its own identity and client ID. Give the backend and simulator different topic permissions.
@@ -2,16 +2,16 @@ package main
 import (
 	"context"
 	"encoding/json"
 	"fmt"
 	"log"
 	"net/url"
 	"os"
 	"os/signal"
 	"syscall"
 	"time"
-	"github.com/eclipse/paho.golang/autopaho"
+	"example.com/mqtt5-learning/internal/config"
-	"github.com/eclipse/paho.golang/paho"
+	"example.com/mqtt5-learning/internal/mqtt"
 )
 func main() {
@@ -21,71 +21,52 @@ func main() {
 }
 func run() error {
 	cfg, err := config.Load(os.Args[1:])
 	if err != nil {
 		return fmt.Errorf("failed to load config: %w", err)
 	}
 	ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
 	defer stop()
-	user := os.Getenv("TEST_MQTT5_MQTT_USERNAME")
+	base := "lab/v1/devices/" + cfg.MQTT.DeviceID
-	pass := os.Getenv("TEST_MQTT5_MQTT_PASSWORD")
+	inbox := make(chan mqtt.Message, 16)
-	if user == "" || pass == "" {
+	manager, err := mqtt.Start(ctx, cfg, []string{base + "/telemetry", base + "/responses"}, inbox)
 		return fmt.Errorf("TEST_MQTT5_MQTT_USERNAME and TEST_MQTT5_MQTT_PASSWORD environment variables not set")
 	}
 	broker, err := url.Parse("mqtt://127.0.0.1:18883")
 	if err != nil {
-		return fmt.Errorf("failed to parse broker url: %w", err)
+		return fmt.Errorf("failed to start MQTT client: %w", err)
 	}
 	defer mqtt.Stop(manager)
-	cfg := autopaho.ClientConfig{
+	ticker := time.NewTicker(10 * time.Second)
-		ServerUrls:                    []*url.URL{broker},
+	defer ticker.Stop()
 		KeepAlive:                     20,
 		CleanStartOnInitialConnection: true,
 		SessionExpiryInterval:         0,
 		ConnectUsername:               user,
 		ConnectPassword:               []byte(pass),
 		OnConnectionUp: func(_ *autopaho.ConnectionManager, ack *paho.Connack) {
 			log.Printf("connected: session_present=%t reason=0x%02x", ack.SessionPresent, ack.ReasonCode)
 		},
 		OnConnectionDown: func() bool {
 			log.Print("connection down; AutoPaho will retry")
 			return true
 		},
 		OnConnectError: func(err error) {
 			log.Printf("connect attempt failed: %v", err)
 		},
 		ClientID: "lab-backend",
 		OnClientError: func(err error) {
 			log.Printf("client error: %v", err)
 		},
 		OnServerDisconnect: func(packet *paho.Disconnect) {
 			log.Printf("server disconnected: reason=0x%02x", packet.ReasonCode)
 		},
 	}
-	manager, err := autopaho.NewConnection(ctx, cfg)
+	runID := time.Now().UnixNano()
-	if err != nil {
+	sequence := 0
 		return fmt.Errorf("failed to create connection manager: %w", err)
 	}
 	defer func() {
 		shutdownCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
 		defer cancel()
 		if err := manager.Disconnect(shutdownCtx); err != nil {
 			log.Printf("shutdown did not finish within deadline: %v", err)
 		}
 	}()
 	startup, cancel := context.WithTimeout(ctx, 10*time.Second)
 	err = manager.AwaitConnection(startup)
 	cancel()
 	if err != nil {
 		return fmt.Errorf("failed to connect to broker: %w", err)
 	}
 	log.Print("connected to broker; pres Ctrl+C to stop")
 	<-ctx.Done()
 	for {
 		select {
 		case <-ctx.Done():
 			log.Print("shutting down")
 			return nil
 		case msg := <-inbox:
 			log.Printf("received message on topic %s retained=%t payload=%s", msg.Topic, msg.Retain, string(msg.Payload))
 		case <-ticker.C:
 			sequence++
 			payload, err := json.Marshal(struct {
 				ID      string `json:"id"`
 				Command string `json:"command"`
 			}{
 				ID: fmt.Sprintf("ping-%d-%d", runID, sequence), Command: "ping",
 			})
 			if err != nil {
 				return fmt.Errorf("failed to marshal payload: %w", err)
 			}
 			if err := mqtt.Publish(ctx, manager, base+"/commands", payload); err != nil {
 				log.Printf("command publication failed or outcome uncertain: %v", err)
 			}
 		}
 	}
 }
@@ -0,0 +1,18 @@
 package main
 import (
 	"log"
 	"os"
 	"example.com/mqtt5-learning/internal/config"
 )
 func main() {
 	cfg, err := config.Load(os.Args[1:])
 	if err != nil {
 		log.Print(err)
 		os.Exit(1)
 	}
 	log.Printf("client_id=%s device_id=%s username=%s timeout=%s password=[redacted]",
 		cfg.MQTT.ClientID, cfg.MQTT.DeviceID, cfg.MQTT.Username, cfg.ConnectTimeout())
 }
@@ -0,0 +1,87 @@
 package main
 import (
 	"context"
 	"encoding/json"
 	"log"
 	"os"
 	"os/signal"
 	"syscall"
 	"time"
 	"example.com/mqtt5-learning/internal/config"
 	mqtt "example.com/mqtt5-learning/internal/mqtt"
 )
 func main() {
 	// Convert a returned application error into a visible nonzero process exit.
 	if err := run(); err != nil {
 		log.Print(err)
 		os.Exit(1)
 	}
 }
 func run() error {
 	// Load settings and bind cancellation to Ctrl+C or the container stop signal.
 	cfg, err := config.Load(os.Args[1:])
 	if err != nil {
 		return err
 	}
 	ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
 	defer stop()
 	// Build this device's topic namespace and bounded receive queue.
 	base := "lab/v1/devices/" + cfg.MQTT.DeviceID
 	inbox := make(chan mqtt.Message, 16)
 	manager, err := mqtt.Start(ctx, cfg, []string{base + "/commands"}, inbox)
 	if err != nil {
 		return err
 	}
 	defer mqtt.Stop(manager)
 	// Heartbeats are periodic; command handling remains active between ticks.
 	ticker := time.NewTicker(3 * time.Second)
 	defer ticker.Stop()
 	for {
 		// Handle cancellation, telemetry publication, and inbound commands together.
 		select {
 		case <-ctx.Done():
 			return nil
 		case <-ticker.C:
 			payload, err := json.Marshal(struct {
 				DeviceID string `json:"device_id"`
 				Time     string `json:"time"`
 			}{cfg.MQTT.DeviceID, time.Now().UTC().Format(time.RFC3339)})
 			if err != nil {
 				return err
 			}
 			if err := mqtt.Publish(ctx, manager, base+"/telemetry", payload); err != nil {
 				log.Printf("heartbeat: %v", err)
 			}
 		case message := <-inbox:
 			// Decode into a local command value only after rejecting retained messages.
 			var command struct {
 				ID      string `json:"id"`
 				Command string `json:"command"`
 			}
 			if message.Retain {
 				log.Print("ignoring retained command")
 				continue
 			}
 			if err := json.Unmarshal(message.Payload, &command); err != nil || command.ID == "" || command.Command != "ping" {
 				log.Print("ignoring malformed or unsupported command")
 				continue
 			}
 			// Echo the command ID so the backend can associate this pong with its ping.
 			response, err := json.Marshal(struct {
 				ID       string `json:"id"`
 				DeviceID string `json:"device_id"`
 				Status   string `json:"status"`
 			}{command.ID, cfg.MQTT.DeviceID, "pong"})
 			if err != nil {
 				return err
 			}
 			if err := mqtt.Publish(ctx, manager, base+"/responses", response); err != nil {
 				log.Printf("response: %v", err)
 			}
 		}
 	}
 }
@@ -2,7 +2,9 @@
 url = "mqtt://127.0.0.1:18883"
 client_id = "lab-backend"
 username = "backend"
 # Disposable local lab password shared by the backend and simulator accounts.
 password = "fItD5H2HRJ6qNaI7Y7O6IFz5-H8sqKZ2"
 device_id = "device-001"
 [app]
-connect_timeout = "10s"
+connect_timeout = "8s"
@@ -70,7 +70,6 @@ func Load(args []string) (Config, error) {
 		"TEST_MQTT5_MQTT_URL":            "mqtt.url",
 		"TEST_MQTT5_MQTT_CLIENT_ID":      "mqtt.client_id",
 		"TEST_MQTT5_MQTT_USERNAME":       "mqtt.username",
 		"TEST_MQTT5_MQTT_PASSWORD":       "mqtt.password",
 		"TEST_MQTT5_MQTT_DEVICE_ID":      "mqtt.device_id",
 		"TEST_MQTT5_APP_CONNECT_TIMEOUT": "app.connect_timeout",
 	}
@@ -122,10 +121,14 @@ func (c Config) Validate() error {
 	if u.Hostname() != "127.0.0.1" && u.Hostname() != "localhost" {
 		return fmt.Errorf("plaintext learning broker must be on loopback")
 	}
-	for _, value := range []string{c.MQTT.ClientID, c.MQTT.Username, c.MQTT.Password} {
+	if strings.TrimSpace(c.MQTT.ClientID) == "" {
-		if strings.TrimSpace(value) == "" {
+		return fmt.Errorf("mqtt.client_id is required")
 			return fmt.Errorf("client ID and credentials are required")
 	}
 	if strings.TrimSpace(c.MQTT.Username) == "" {
 		return fmt.Errorf("mqtt.username is required")
 	}
 	if strings.TrimSpace(c.MQTT.Password) == "" {
 		return fmt.Errorf("mqtt.password is required in the TOML configuration")
 	}
 	if c.MQTT.DeviceID == "" || strings.ContainsAny(c.MQTT.DeviceID, "/+# \t\r\n") {
 		return fmt.Errorf("device ID must be a nonempty single topic segment")
@@ -0,0 +1,143 @@
 package mqtt
 import (
 	"context"
 	"fmt"
 	"log"
 	"net/url"
 	"time"
 	"example.com/mqtt5-learning/internal/config"
 	"github.com/eclipse/paho.golang/autopaho"
 	"github.com/eclipse/paho.golang/paho"
 )
 type Message struct {
 	Topic   string
 	Payload []byte
 	Retain  bool
 }
 func Start(ctx context.Context, cfg config.Config, topics []string, inbox chan<- Message) (*autopaho.ConnectionManager, error) {
 	broker, err := url.Parse(cfg.MQTT.URL)
 	if err != nil {
 		return nil, err
 	}
 	initialSubscription := make(chan error, 1)
 	manager, err := autopaho.NewConnection(ctx, autopaho.ClientConfig{
 		ServerUrls:                    []*url.URL{broker},
 		KeepAlive:                     20,
 		CleanStartOnInitialConnection: true,
 		SessionExpiryInterval:         0,
 		ConnectUsername:               cfg.MQTT.Username,
 		ConnectPassword:               []byte(cfg.MQTT.Password),
 		OnConnectionUp: func(cm *autopaho.ConnectionManager, _ *paho.Connack) {
 			// This runs again on reconnect; reestablish broker subscriptions.
 			subCtx, cancel := context.WithTimeout(ctx, 5*time.Second)
 			defer cancel()
 			subscriptions := make([]paho.SubscribeOptions, 0, len(topics))
 			for _, topic := range topics {
 				subscriptions = append(subscriptions, paho.SubscribeOptions{Topic: topic, QoS: 1, RetainAsPublished: true})
 			}
 			ack, subErr := cm.Subscribe(subCtx, &paho.Subscribe{Subscriptions: subscriptions})
 			if subErr == nil {
 				if ack == nil || len(ack.Reasons) != len(topics) {
 					subErr = fmt.Errorf("missing or incomplete SUBACK")
 				} else {
 					for i, reason := range ack.Reasons {
 						if reason > 1 { // We requested QoS 1; valid grants are 0 or 1.
 							subErr = fmt.Errorf("subscription %s rejected: 0x%02x", topics[i], reason)
 							break
 						}
 					}
 				}
 			}
 			if subErr != nil {
 				log.Printf("subscribe failed: %v", subErr)
 			} else {
 				log.Printf("subscriptions ready: %v", topics)
 			}
 			select {
 			case initialSubscription <- subErr:
 			default:
 			}
 		},
 		OnConnectionDown: func() bool { log.Print("disconnected; reconnecting"); return true },
 		OnConnectError:   func(err error) { log.Printf("connect error: %v", err) },
 			ClientID:           cfg.MQTT.ClientID,
 			OnClientError:      func(err error) { log.Printf("client error: %v", err) },
 			OnServerDisconnect: func(d *paho.Disconnect) { log.Printf("disconnect reason=0x%02x", d.ReasonCode) },
 			OnPublishReceived: []func(paho.PublishReceived) (bool, error){
 				func(received paho.PublishReceived) (bool, error) {
 					packet := received.Packet
 					message := Message{Topic: packet.Topic, Payload: append([]byte(nil), packet.Payload...), Retain: packet.Retain}
 					select {
 					case inbox <- message:
 					default:
 						// Deliberately lossy early demo: automatic MQTT ACK is still possible.
 						log.Printf("lab inbox full; message dropped topic=%s", packet.Topic)
 					}
 					return true, nil
 				},
 			},
 	})
 	if err != nil {
 		return nil, fmt.Errorf("start AutoPaho: %w", err)
 	}
 	startup, cancel := context.WithTimeout(ctx, cfg.ConnectTimeout())
 	defer cancel()
 	if err = manager.AwaitConnection(startup); err != nil {
 		Stop(manager)
 		return nil, fmt.Errorf("await connection: %w", err)
 	}
 	select {
 	case err = <-initialSubscription:
 		if err != nil {
 			Stop(manager)
 			return nil, err
 		}
 	case <-startup.Done():
 		Stop(manager)
 		return nil, fmt.Errorf("await subscriptions: %w", startup.Err())
 	}
 	return manager, nil
 }
 func Publish(ctx context.Context, manager *autopaho.ConnectionManager, topic string, payload []byte) error {
 	operation, cancel := context.WithTimeout(ctx, 3*time.Second)
 	defer cancel()
 	ack, err := manager.Publish(operation, &paho.Publish{
 		Topic:   topic,
 		Payload: payload,
 		QoS:     1,
 		Retain:  false,
 		Properties: &paho.PublishProperties{
 			ContentType: "application/json",
 		},
 	})
 	if err != nil {
 		return fmt.Errorf("publish: %w", err)
 	}
 	if ack == nil {
 		return fmt.Errorf("missing Qos 1 publication response")
 	}
 	if ack.ReasonCode >= 0x80 {
 		return fmt.Errorf("publish rejected: 0x%02x", ack.ReasonCode)
 	}
 	log.Printf("broker publish acknowledgement topic=%s reason=0x%02x", topic, ack.ReasonCode)
 	return nil
 }
 func Stop(manager *autopaho.ConnectionManager) {
 	shutdown, cancel := context.WithTimeout(context.Background(), 5*time.Second)
 	defer cancel()
 	if err:=manager.Disconnect(shutdown); err != nil {
 		log.Printf("shutdown did not finish within deadline: %v", err)
 	}
 }
@@ -0,0 +1,226 @@
 # 🧭 Lesson 0 — The broker, Go backend, and device lifecycle
 > [!toc]- 📑 Contents
 >
 > - [[#🌐 The whole system|🌐 The whole system]]
 > - [[#🖥️ 1. Set up the broker|🖥️ Broker setup]]
 > - [[#⚙️ 2. Configure the backend and device|⚙️ Client configuration]]
 > - [[#🔌 3. Connect both clients|🔌 Connections]]
 > - [[#📬 4. Register subscriptions|📬 Subscriptions]]
 > - [[#📨 5. Publish and route messages|📨 Message flow]]
 > - [[#🔄 6. Recover and stop|🔄 Recovery and shutdown]]
 > - [[#🗺️ Where each part is covered|🗺️ Lesson coverage map]]
 > - [[#🧪 Understanding check|🧪 Understanding check]]
 > - [[#🔗 Related lessons|🔗 Related lessons]]
 ## 🌐 The whole system
 Picture three independently running programs: a broker, a Go backend, and a device. The backend and device are both MQTT clients. The broker accepts their connections, checks credentials and permissions, stores subscriptions, and routes publications.
 The backend may have more application responsibilities, but MQTT does not give it a special connection type. This course applies the MQTT fundamentals you already know to a Go application using MQTT 5.
 ```mermaid
 flowchart LR
    C[Broker configuration<br/>listener, accounts, ACL] --> B[Mosquitto broker]
    G[Go backend configuration<br/>URL, client ID, credentials] --> GL[Go backend + Paho]
    D[Device configuration<br/>URL, client ID, credentials] --> DEV[Device application + MQTT client]
    GL <-->|MQTT connection and messages| B
    DEV <-->|MQTT connection and messages| B
 ```
 The broker has its own setup and runtime. Each client has its own configuration and runtime. The backend and device do not need each other's network address; each connects to the broker.
 ---
 ## 🖥️ 1. Set up the broker
 > 💡 **Story question:** Before a client arrives, how does the broker know where to listen, which credentials to accept, and which topics each identity may use?
 Those decisions belong to broker configuration or to an authentication/authorization service configured by the broker. In this lab, Mosquitto is configured with a listener, a password file, and an ACL file. The `backend` and `simulator` accounts are created before the clients connect.
 ```mermaid
 flowchart TD
    OP[Operator prepares lab files] --> CONF[Listener and authentication config]
    OP --> USERS[Backend and simulator accounts]
    OP --> ACL[Per-identity topic permissions]
    CONF -->|broker startup| B[Mosquitto]
    USERS --> B
    ACL --> B
 ```
 The broker authenticates a client when it connects. It checks authorization when the client asks to subscribe or publish. Successful authentication does not automatically grant access to every topic. The broker can load local files, or a deployment can configure an external authentication provider; clients do not create their own broker accounts as they arrive.
 For this learning environment, the broker is isolated and uses synthetic lab data. Its setup is described in [Lesson 1](01-lab-setup.md) and [Broker operations](broker-operations.md).
 📚 **Covered in:** [Lesson 1 — Lab setup](01-lab-setup.md) configures Mosquitto, accounts, and ACLs. [Broker operations](broker-operations.md) covers maintenance; [Lesson 11 — Security](11-security-boundaries.md) develops authentication and authorization boundaries.
 ---
 ## ⚙️ 2. Configure the backend and device
 > 💡 **Story question:** If both programs are MQTT clients, what distinguishes the backend's connection from the device's connection?
 Each process has its own configuration and client identity. Both need a broker URL and credentials accepted by that broker. They also need distinct Client IDs while running at the same time. Their usernames determine which broker ACL rules apply in this lab; Client ID identifies the MQTT connection/session and is not a password.
 ```mermaid
 flowchart LR
    T[TOML defaults] --> M[Configuration loader]
    E[Environment secrets and overrides] --> M
    F[Explicit CLI overrides] --> M
    M --> V[Validate effective settings]
    V -->|backend identity| BC[Backend client config]
    V -->|simulator identity| DC[Device client config]
 ```
 The Go project uses TOML for ordinary settings and `TEST_MQTT5_...` environment variables for credentials and overrides. `config.Load(...)` merges and validates the settings before MQTT startup. The simulator is another program with its own configuration; it is not a child session inside the backend.
 Connection settings are distinct from message choices. Keep Alive, session behavior, and a Will are connection options. Topic filters and requested QoS belong to subscriptions. Topic, payload, QoS, and Retain belong to publications.
 📚 **Covered in:** [Lesson 2 — Connection](02-go-connection.md) introduces the client settings; [Lesson 3 — Typed configuration](03-typed-configuration.md) builds the loader, precedence, and validation.
 ---
 ## 🔌 3. Connect both clients
 > 💡 **Story question:** Once the Go process has configuration, what must it prove before it can claim to be connected?
 The program parses the broker URL, creates a Paho/AutoPaho configuration, and starts the connection manager. AutoPaho manages the connection lifecycle and reconnect attempts; Paho implements MQTT protocol exchanges. The manager being created is not yet proof of a successful connection. The broker must accept MQTT `CONNECT` with `CONNACK`, and the application can wait for that state with `AwaitConnection`.
 ```mermaid
 sequenceDiagram
    participant G as Go backend / Paho
    participant B as Broker
    participant D as Device / MQTT client
    G->>B: TCP connection, then MQTT CONNECT
    B->>B: Authenticate backend identity
    B-->>G: CONNACK accepted or rejected
    D->>B: TCP connection, then MQTT CONNECT
    B->>B: Authenticate device identity
    B-->>D: CONNACK accepted or rejected
 ```
 The backend and device go through this independently. A successful `CONNACK` proves that a client connection was accepted. It does not prove a subscription was accepted, a device is online, or an application operation completed.
 📚 **Covered in:** [Lesson 2 — Connection](02-go-connection.md) follows CONNECT, CONNACK, reconnect, and shutdown. [Lesson 4 — Publish and subscribe](04-publish-subscribe-simulator.md) connects both Go processes.
 ---
 ## 📬 4. Register subscriptions
 > 💡 **Story question:** The broker accepted the connection. How does it learn which messages this client wants to receive?
 The client sends `SUBSCRIBE` requests containing topic filters. The broker checks each request against its rules and returns `SUBACK` results. In the Go application, a local callback is also configured so Paho knows what to do when a matching publication arrives. The broker-side subscription and the Go-side callback are two separate parts of receiving a message.
 ```mermaid
 sequenceDiagram
    participant G as Go backend
    participant B as Broker
    participant D as Device
    G->>B: SUBSCRIBE device telemetry and responses
    B-->>G: SUBACK per-filter result
    D->>B: SUBSCRIBE device commands
    B-->>D: SUBACK per-filter result
 ```
 The backend commonly listens for telemetry and responses. The device listens for its commands. The client should inspect the subscription result and establish required subscriptions before publishing work that depends on a receiver already listening.
 📚 **Covered in:** [Lesson 4 — Publish and subscribe](04-publish-subscribe-simulator.md) builds the subscriptions and checks readiness. [Lesson 9 — Sessions and recovery](09-sessions-recovery.md) explores when subscriptions survive a reconnect; [Lesson 16 — MQTT 5 controls](16-mqtt5-controls.md) examines subscription options.
 ---
 ## 📨 5. Publish and route messages
 > 💡 **Story question:** Once subscriptions exist, how does a command travel from backend code to the device application?
 The backend serializes an application value into bytes, publishes those bytes to a concrete topic, and the broker routes the publication to matching subscribed clients. Paho's receive callback runs inside the client process; the application can copy and enqueue the message for its main loop to handle.
 ```mermaid
 sequenceDiagram
    participant G as Go backend
    participant B as Broker
    participant D as Device
    G->>B: PUBLISH device-001/commands, QoS 1
    B-->>G: PUBACK for backend-to-broker hop
    B->>D: PUBLISH matching command
    D-->>B: PUBACK for broker-to-device hop
    D->>D: Callback copies/enqueues message
    D->>D: Application validates and handles command
    D->>B: PUBLISH response on response topic
    B->>G: PUBLISH matching response
 ```
 There are separate MQTT delivery hops. A `PUBACK` on the backend's publication confirms the QoS 1 exchange with the broker; it is not a device response. The response is a new application publication. Similarly, receipt by a Go callback does not prove a database write or physical device action.
 📚 **Covered in:** [Lesson 4](04-publish-subscribe-simulator.md) implements message exchange. [Lesson 5 — Contracts](05-contracts-validation.md) validates topics and payloads; [Lesson 6 — Concurrency](06-concurrency-cancellation.md) covers callbacks, queues, and acknowledgement timing; [Lesson 7 — Command/response](07-command-response.md) tracks device results; [Lesson 8 — HTTP commands](08-http-commands.md) exposes the flow to an API caller.
 ---
 ## 🔄 6. Recover and stop
 > 💡 **Story question:** What should happen if the broker or network disappears while the backend process remains alive?
 AutoPaho can reconnect according to its lifecycle policy. After a connection comes back, the application must ensure required subscriptions are present—either because the broker resumed a session or because the client re-subscribed. In this lab, Lesson 4's helper explicitly subscribes again on connection-up. Whether offline publications arrive depends on MQTT session settings, QoS, broker state, and expiry; reconnect alone does not promise recovery of every message.
 On shutdown, the application cancels its root context, stops background work, and requests a bounded disconnect. Clean MQTT shutdown, broker session deletion, and completion of business work are different events.
 ```mermaid
 stateDiagram-v2
    [*] --> Configured
    Configured --> Connecting
    Connecting --> Connected: accepted CONNACK
    Connecting --> Retry: connect error / rejected attempt
    Retry --> Connecting: AutoPaho retries
    Connected --> Subscribing: connection-up handler
    Subscribing --> Ready: acceptable SUBACK
    Ready --> Retry: connection lost
    Ready --> Stopping: cancellation signal
    Stopping --> [*]: bounded disconnect
 ```
 📚 **Covered in:** [Lesson 2](02-go-connection.md) introduces reconnect and bounded shutdown; [Lesson 4](04-publish-subscribe-simulator.md) restores subscriptions after connection-up; [Lesson 6](06-concurrency-cancellation.md) coordinates worker draining and shutdown. [Lesson 9](09-sessions-recovery.md) measures session recovery, and [Lesson 10](10-expiry-presence-state.md) covers expiry and presence.
 ---
 ## 🗺️ Where each part is covered
 | Lifecycle area | Lessons | Evidence to understand |
 |---|---|---|
 | Broker setup, credentials, and topic permissions | [1](01-lab-setup.md), [11](11-security-boundaries.md) | Which process owns each setting; authentication versus authorization |
 | Go client configuration | [2](02-go-connection.md), [3](03-typed-configuration.md) | Effective settings, identity, validation, and precedence |
 | CONNECT, CONNACK, reconnect, and shutdown | [2](02-go-connection.md), [4](04-publish-subscribe-simulator.md), [9](09-sessions-recovery.md) | What connection acceptance proves and what must be restored |
 | Subscriptions and broker routing | [4](04-publish-subscribe-simulator.md), [9](09-sessions-recovery.md), [16](16-mqtt5-controls.md) | SUBACK results, matching filters, session restoration, and options |
 | Publish, delivery, and application response | [4](04-publish-subscribe-simulator.md), [5](05-contracts-validation.md) | Per-hop QoS acknowledgements versus validated business evidence |
 | Callback work, queues, and overload | [6](06-concurrency-cancellation.md) | Admission, acknowledgement timing, bounded capacity, and shutdown |
 | Reliable command/response and HTTP | [7](07-command-response.md), [8](08-http-commands.md) | Correlation, idempotency, timeout, and status reporting |
 | Expiry and device presence | [10](10-expiry-presence-state.md) | Stale messages and the limits of retained state |
 The table links each part of this lifecycle to the lessons that explain or exercise it. Sections 12–15 and 17–18 in [the course plan](../COURSE.md) extend the story into persistence, process boundaries, and operations.
 ---
 ## 🧪 Understanding check
 1. What does a successful `CONNACK` prove, and what does it not prove?
 2. Why does the backend need both a broker subscription and a local receive handler?
 3. Which acknowledgement tells the backend that the device application completed a command?
 4. What is the difference between a client credential and a Client ID in this lab?
 > [!answer]- 📋 Answers
 >
 > **1.** It proves the broker accepted that MQTT connection. It does not prove a subscription was accepted, a device is online, or business work completed.
 >
 > **2.** The broker subscription controls routing; the local handler controls what Paho/the application does with a delivered PUBLISH.
 >
 > **3.** None of the MQTT packet acknowledgements proves application completion. The device must publish an application-level response, and the backend must validate it.
 >
 > **4.** Credentials authenticate a client and select its ACL identity here. Client ID identifies the MQTT connection/session and can be configured as an ACL input, but it is not inherently authentication.
 ---
 ## 🔗 Related lessons
 - [Lesson 1 — Isolated lab setup](01-lab-setup.md)
 - [Lesson 2 — Go connection](02-go-connection.md)
 - [Lesson 3 — Typed configuration](03-typed-configuration.md)
 - [Lesson 4 — Publish, subscribe, simulator](04-publish-subscribe-simulator.md)
 - [Course index](../COURSE.md)
@@ -1,11 +1,31 @@
-# Section 1 — Prepare the MQTT 5 lab
+# 🛠️ Section 1 — Prepare the MQTT 5 lab
 Read this first. You will create the files and run the commands yourself. All paths below are relative to the learning project root. The examples use the module path `example.com/mqtt5-learning`; if you change it, change the imports in later lessons too.
 **Version baseline:** Paho `v0.23.0`, verified from upstream release tags on 2026-10-07. It requires Go 1.24 or newer; Go 1.25 is compatible. The matching local reference is `references/paho.golang-v0.23.0/` (commit `1aa03969c5932464359b3e4ed752d9ac61abccc9`).
 ## 🎬 Start with the system story
-## Working directory and file placement
+> 💡 **Story question:** Before we write a Go client, what must already exist so the backend and a simulated device can connect safely and exchange messages?
 The broker is a separately configured service. It needs a listener, an authentication source, topic permissions, and storage settings. The backend and device are separate clients with their own connection settings and credentials. This lesson prepares the local broker and verifies its behavior with MQTT 5 CLI clients before adding Go to the path.
 ```mermaid
 flowchart LR
    O[Prepare Compose and Mosquitto config] --> B[Mosquitto broker starts]
    P[Create backend and simulator credentials] --> B
    A[Define ACL topic permissions] --> B
    G[Go backend comes in a later lesson] -. connects .-> B
    D[Go simulator comes in a later lesson] -. connects .-> B
    C[Mosquitto CLI verifies MQTT 5] --> B
 ```
 The Compose file starts and isolates the broker; the Mosquitto configuration tells the broker how to listen and enforce authentication/ACLs. A Docker volume preserves broker-owned state. None of these files configures the Go clients—that happens in later lessons.
 Before each setup step, ask which process owns the setting and what observation will prove it took effect.
 ## 📁 Working directory and file placement
 Run every shell command in this lesson from the learning project root:
@@ -15,7 +35,11 @@ cd /home/ryan/Media/Code/Code/robot-market/mqtt-5-learning
 Paths such as `cmd/backend/main.go` are relative to this directory. Markdown files under `lessons/` explain the work; create the application files at the paths shown, not inside `lessons/` or `references/`. Code blocks labeled Go, TOML, YAML, or conf are file contents; shell blocks are terminal commands. Create parent directories before saving files.
-## 1. Check the tools and create the module
+## 🛠️ 1. Check the tools and create the module
 > 💡 **Story question:** What must be available on this machine to build the Go client and start the isolated broker, and which Paho API speaks MQTT 5?
 Check the Go toolchain and both Docker components, then pin the MQTT 5 module in this learning project. The old `paho.mqtt.golang` client is a separate MQTT 3 API.
 ```sh
 go version
@@ -33,7 +57,11 @@ Run `go mod init` only once. `go get` pins the dependency in `go.mod`; commit `g
 The old backend's `github.com/eclipse/paho.mqtt.golang` is a different MQTT 3 client. MQTT 5 uses `github.com/eclipse/paho.golang/autopaho` and `/paho`. There is no `SetProtocolVersion(5)` needed in these MQTT 5 examples.
-## 2. Create the broker files
+## 🦟 2. Create the broker files
 > 💡 **Story question:** What configuration belongs to Mosquitto itself, and how will host applications reach the broker without exposing the lab listener to the network?
 Compose controls the container boundary and port mapping. Mosquitto's own files control its listener, credentials, ACL, and persistence. Keep those roles distinct as you create the files below.
 Create these directories:
@@ -79,7 +107,7 @@ volumes:
 This is a pinned Mosquitto 2 lab image, not a claim that it is the newest broker release. Host port 18883 maps to container port 1883. Containers use `broker:1883`; Go programs on your host use `127.0.0.1:18883`. Loopback binding keeps this listener off your LAN. Persistence belongs to this Compose project only.
-### Understand the Compose definition
+### 🐳 Understand the Compose definition
 `services` lists the containers in this application; `broker` is the service name you use in commands and the DNS name on this Compose network. `image` selects a prebuilt image; Docker downloads it if needed. There is no `build` section because this lesson does not build a broker image.
@@ -100,7 +128,7 @@ log_dest stdout
 connection_messages true
 ```
-### Understand each broker setting
+### 🦟 Understand each broker setting
 | Setting | Purpose and operational consequence |
 | --- | --- |
@@ -142,7 +170,11 @@ Add to `.gitignore` (preserve any existing rules):
 Git ignore patterns are file contents, not terminal commands. `/lab/secrets/` ignores the root-level directory and its contents; `/.env` ignores a root environment file. Ignoring a file does not untrack a secret already committed. Keep real secrets out of the initial commit as well as later changes.
-## 3. Create passwords interactively
+## 🔐 3. Create passwords interactively
 > 💡 **Story question:** How will the broker recognize the backend and simulator when they connect, and how can those identities receive different topic permissions?
 Create separate broker accounts and ACL rules before starting clients. The client supplies credentials; the broker validates them against its configured source. Authentication permits connection, while ACL rules govern topic operations.
 These commands create a lab password file without putting the password literal in your shell history. Run them from the project root:
@@ -155,7 +187,7 @@ docker run --rm --user 0 -v "$PWD/lab/secrets:/work" eclipse-mosquitto:2.0.22 ch
 `-c` creates/replaces the file. Use it only for the first username; using it again would erase the backend entry. The last two commands make the file readable by the broker account in this image. Container-root use is limited to setting up this mounted lab file.
-### Why four Docker commands?
+### 🔧 Why four Docker commands?
 There are two jobs here: generate password hashes for two users, then give the broker permission to read the resulting file. Docker adds a wrapper because we use the tool bundled with the pinned broker image instead of installing Mosquitto utilities on your computer. Each command runs a temporary utility container; none starts the broker.
@@ -178,7 +210,7 @@ The generated file stores password hashes, not the plaintext passwords you type.
 Linux checks numeric user/group IDs. `chown` inside this image picks the broker's IDs; your host may display a numeric owner or a different name for those same IDs. With mode `600`, your regular host user may be unable to read the file afterward. That is expected: the broker must read it, and subsequent password maintenance can use the same temporary root utility container. Do not make the file world-readable to work around this.
-### What is usual practice?
+### 💡 What is usual practice?
 For a small Mosquitto 2.0 installation, generating a password file with `mosquitto_passwd`, configuring `password_file`, and restricting file access is a conventional approach. The official [authentication guide](https://mosquitto.org/documentation/authentication-methods/) documents creating the first user with `-c` and adding or updating users without it. Docker is simply how this lesson obtains that utility.
@@ -189,7 +221,7 @@ For a small Mosquitto 2.0 installation, generating a password file with `mosquit
 The four explicit commands make the setup visible for learning. Once you understand them, wrapping them in a setup script is reasonable. We keep interactive prompts here so password literals do not appear in shell history or process arguments.
-### Verify password setup before starting
+### 🧪 Verify password setup before starting
 Do not start the broker until both password prompts and both permission commands have completed successfully. The two accounts must exist in the generated file; creating an empty file does not create usable credentials.
@@ -201,7 +233,7 @@ docker run --rm --user mosquitto -v "$PWD/lab/secrets:/work:ro" eclipse-mosquitt
 `--user mosquitto` tests access as the image's broker account. `sh -c` runs the quoted checks inside the utility container; `test -s` requires a nonempty file, `test -r` requires read access, and `&&` runs the second check only if the first succeeds. Success is quiet with exit status zero. Failure means you must fix the file/path/permissions before starting. This checks file access, not whether the typed client password is correct.
-### If permission changes do not stick on the host filesystem
+### 🔐 If permission changes do not stick on the host filesystem
 Verify final metadata with `stat -c '%a %u:%g %n' lab/secrets/passwords`. `stat` reports mode, numeric owner/group, and pathname without exposing hashes. Also use `findmnt -T lab/secrets/passwords -o TARGET,FSTYPE,OPTIONS` to identify the underlying mount. Some mounted filesystems synthesize permissions: `chmod`/`chown` can appear successful while the metadata remains unchanged. Do not conclude the file is protected from the commands' exit status alone.
@@ -227,7 +259,7 @@ volumes:
 For every subsequent add-user, rotation, or permission command, mount `mqtt5-learning_broker-secrets:/work` rather than `$PWD/lab/secrets:/work`: the broker now reads the volume copy. Editing the old host file does not update the live credential store. Include this external volume in backups; Compose `down -v` does not delete external volumes. The old host copy remains sensitive and ignored by Git; it has been preserved rather than automatically deleted.
-### Understand password-tool and entrypoint warnings
+### 🔐 Understand password-tool and entrypoint warnings
 During creation or update, `mosquitto_passwd` can warn about world-readable permissions or ownership. Do not assume these warnings prove the final file is correct: run the ownership/mode commands afterward and verify access. For this pinned Mosquitto 2.0.22 image, use `mosquitto:mosquitto` ownership and mode `600`. A password file needs no executable permission. Blindly following a warning suggesting root ownership with owner-only access can prevent the unprivileged broker from reading it. Review requirements again when upgrading rather than treating a future-version warning as this version's deployment instructions.
@@ -266,7 +298,11 @@ For this project, resource names commonly include `mqtt5-learning-broker-1`, `mq
 In `ps`, the local-only mapping should show `127.0.0.1:18883->1883/tcp`. A mapping showing `0.0.0.0` or `[::]` exposes that published port on all corresponding host interfaces. Change the Compose mapping to the loopback form and apply it with `up -d broker`; restarting alone does not change port bindings.
-## 4. Verify MQTT 5 without Go first
+## 🧪 4. Verify MQTT 5 without Go first
 > 💡 **Story question:** How can we prove that the broker is listening, speaks the intended MQTT version, accepts the right credentials, and enforces topic permissions before debugging Go code?
 Use Mosquitto CLI clients as a controlled first pair of MQTT clients. Their packet traces isolate broker reachability, protocol negotiation, authentication, authorization, and routing from application code.
 Use host-installed `mosquitto_sub` and `mosquitto_pub`, or run their equivalents through `docker compose exec broker` with port 1883 inside the container. Keep a subscription open in terminal A:
@@ -290,7 +326,7 @@ For the CLI only, `-P` exposes its argument to shell history/process inspection.
 Repeat with a wrong password. Also try publishing as `simulator` to `lab/v1/devices/device-002/telemetry`; this must not reach an authorized observer. Check the MQTT reason and broker logs rather than assuming every denied operation has the same client error.
-### Read the CLI exchange, not just the payload
+### 📘 Read the CLI exchange, not just the payload
 With `-d`, identify CONNECT/CONNACK first, then SUBSCRIBE/SUBACK in the subscriber terminal. Only publish after SUBACK. In the publisher terminal, find PUBLISH/PUBACK at QoS 1. In the subscriber terminal, find the separately forwarded PUBLISH and its acknowledgement. Debug formatting varies by client version; learn the packet names rather than memorizing one output line.
@@ -303,7 +339,7 @@ The broker needs an established matching subscription when this nonretained mess
 </details>
-## API preview: connect each setting to its purpose
+## ⚙️ API preview: connect each setting to its purpose
 ```go
 broker, err := url.Parse("mqtt://127.0.0.1:18883")
@@ -317,7 +353,7 @@ cfg := autopaho.ClientConfig{
 This is an illustrative fragment, not a standalone program. The URL chooses the transport endpoint, while Client ID selects the broker session identity. `KeepAlive` is an integer number of seconds, not a `time.Duration`. The embedded `ClientConfig` describes the underlying MQTT client; the outer configuration describes managed connections.
-## Exercises and questions
+## 🧪 Exercises and questions
 1. Explain why switching the Go library is more than changing one import.
 2. Predict whether the simulator may subscribe to another device's commands; verify it.
@@ -326,11 +362,15 @@ This is an illustrative fragment, not a standalone program. The URL chooses the
 **Completion:** CLI publish/subscribe works using MQTT 5, invalid credentials fail, and cross-device writes are denied. Do not use `down -v` unless you deliberately want to erase the lab's broker persistence.
-## Broker administration you should practice
+## ✅ Result — the story so far
 The broker is now the meeting point for the lab, listening only on this computer. It checks each client's username and password, then uses the ACL to decide which topic directions that identity may use. MQTT 5 CLI clients can prove the broker routes an allowed message and rejects an unauthorized one. The Go backend and simulator have not joined the story yet; this lesson prepares the place they will meet.
 ## 🦟 Broker administration you should practice
 Read [Broker operations](broker-operations.md) for explained stop/start, restart versus recreate, password rotation, backups, restore, upgrades, and the steps required before remote deployment. Complete its local lifecycle and credential-rotation exercises before treating Section 1 as finished. Later Section 9 measures session recovery, and Section 11 in the course adds TLS and stronger access controls.
-## Common failures
+## 🔎 Common failures
 | Symptom | Check |
 | --- | --- |
@@ -345,7 +385,7 @@ Read [Broker operations](broker-operations.md) for explained stop/start, restart
 | Not authorized | Username, password, and exact permitted topic direction |
 | Pub/sub works but Go fails later | Go URL/credentials, client ID, then tagged API version |
-## Sources
+## 🔗 Sources
 - [Paho v0.23.0 release](https://github.com/eclipse-paho/paho.golang/releases/tag/v0.23.0)
 - [Mosquitto broker configuration](https://mosquitto.org/man/mosquitto-conf-5.html)
@@ -1,11 +1,35 @@
-# Section 2 — Understand a Go MQTT 5 connection
+# 🔌 Section 2 — Understand a Go MQTT 5 connection
 Prerequisite: Section 1's broker and backend credentials work. This section contains a complete small program. Type it in stages, compile after each meaningful change, and explain the code before running the failure exercises.
 **Version baseline:** Paho `v0.23.0`, verified from upstream release tags on 2026-10-07. It requires Go 1.24 or newer; Go 1.25 is compatible. The matching local reference is `references/paho.golang-v0.23.0/` (commit `1aa03969c5932464359b3e4ed752d9ac61abccc9`).
 ## 🎬 Start with the system story
-## Working directory and file placement
+> 💡 **Story question:** The broker is running and has accepted CLI clients. How does the Go backend become another MQTT client, and what event proves the broker accepted its connection?
 The Go process loads its connection values, gives AutoPaho a client configuration, and starts a connection manager. AutoPaho manages the connection lifecycle; the underlying Paho client performs MQTT 5 protocol exchanges. The manager can start before the network connection succeeds, so the application waits for a successful CONNACK through `AwaitConnection`.
 ```mermaid
 sequenceDiagram
    participant App as Go backend application
    participant Auto as AutoPaho manager
    participant Paho as Paho MQTT 5 client
    participant Broker as Mosquitto broker
    App->>Auto: NewConnection(lifecycle context, config)
    Auto->>Paho: create and manage client connection
    Paho->>Broker: TCP connection, then CONNECT
    Broker->>Broker: authenticate client credentials
    Broker-->>Paho: CONNACK accepted or rejected
    Paho-->>Auto: connection state / callback
    App->>Auto: AwaitConnection(startup deadline)
    Auto-->>App: connected or startup error
 ```
 The backend is an MQTT client, just like the device. Its application role does not make the broker handshake different. A successful CONNACK proves connection acceptance; subscriptions, device availability, and business completion are later evidence. See [Lesson 0](00-system-lifecycle.md) for the complete lifecycle.
 ## 📁 Working directory and file placement
 Run every shell command in this lesson from the learning project root:
@@ -15,7 +39,11 @@ cd /home/ryan/Media/Code/Code/robot-market/mqtt-5-learning
 Paths such as `cmd/backend/main.go` are relative to this directory. Markdown files under `lessons/` explain the work; create the application files at the paths shown, not inside `lessons/` or `references/`. Code blocks labeled Go, TOML, YAML, or conf are file contents; shell blocks are terminal commands. Create parent directories before saving files.
-## 1. Build the connection program
+## 🔌 1. Build the connection program
 > 💡 **Story question:** Which settings and lifecycle controls must the Go process pass to the client library so a connection attempt can start and stop predictably?
 The program builds one AutoPaho configuration from the broker URL, credentials, Client ID, connection options, and callbacks. The root context owns the manager lifetime; a separate startup context bounds how long the application waits for the first connection.
 Create the directory first:
@@ -115,7 +143,11 @@ func run() error {
 ```
-## 2. Understand the connection lifecycle
+## 🔌 2. Understand the connection lifecycle
 > 💡 **Story question:** What is happening between creating the manager and logging “connection ready,” and why are those moments not equivalent?
 Trace the background attempt through transport connection, MQTT CONNECT, broker authentication, CONNACK, and `AwaitConnection`. Then trace what AutoPaho does when the connection drops.
 Before reading the full program, predict what evidence you need at each stage:
@@ -149,22 +181,21 @@ There are three different contexts here. The signal context controls the program
 Paho v0.23.0's `Disconnect(ctx)` cancels the manager and waits for manager shutdown with the provided deadline. Its `Done()` documentation notes that underlying MQTT cleanup may still be completing; neither API is proof that application data was persisted or device work was finished.
-## 3. Run it
+## ▶️ 3. Run it
-In zsh, read your backend password without echoing it:
+> 💡 **Story question:** Which output tells you that the broker accepted the client, and what does a clean Ctrl+C demonstrate about lifecycle ownership?
 Run the process with valid lab credentials, observe the acceptance evidence, then stop it and compare graceful disconnect with a process that simply disappears.
 The backend reads its username and password from `configs/lab.toml`:
 ```sh
 export TEST_MQTT5_MQTT_USERNAME=backend
 read -rs 'TEST_MQTT5_MQTT_PASSWORD?Backend lab password: '
 export TEST_MQTT5_MQTT_PASSWORD
 go run ./cmd/backend
 ```
-`export NAME=value` puts a variable into this terminal's environment so child processes such as `go run` inherit it. The `read` line uses zsh syntax: `-r` preserves backslashes, `-s` hides typed characters, and `NAME?prompt` selects the variable and prompt. Enter the password you created in Lesson 1; this does not create a broker user. The second `export` exposes that newly read variable to the Go process. In bash the prompt syntax differs; do not copy this zsh-specific line unchanged.
+`go run ./cmd/backend` compiles the package and runs it in the foreground. `./` means relative to the project root. Ctrl+C sends an interrupt to the foreground program; `signal.NotifyContext` turns it into context cancellation, allowing your cleanup code to run.
-`go run ./cmd/backend` compiles the package and runs it in the foreground. `./` means relative to the project root. Ctrl+C sends an interrupt to the foreground program; `signal.NotifyContext` turns it into context cancellation, allowing your cleanup code to run. After the process exits, `unset TEST_MQTT5_MQTT_PASSWORD` removes the credential from this shell's environment; it does not rotate the broker credential.
+The TOML file contains the disposable local lab password. Use only lab credentials. Expect `connected` and `connection ready`; callback log order can vary. Press Ctrl+C and verify exit within the shutdown deadline.
 Environment variables avoid password literals in this command but are not a secrets vault. Use only lab credentials. Expect `connected` and `connection ready`; callback log order can vary. Press Ctrl+C and verify exit within the shutdown deadline.
 Run syntax checks after creating the file:
@@ -178,7 +209,11 @@ go vet ./cmd/backend
 `go build` emits a binary in the current directory; remove or ignore it as appropriate. A build verifies syntax/types, not connection behavior.
-## 4. Observe the protocol and failures
+## ⚠️ 4. Observe the protocol and failures
 > 💡 **Story question:** How can you tell whether a failure happened before MQTT, during broker authentication, or after an established connection was lost?
 Change one condition at a time and compare the connection callbacks, reason information, retry behavior, and shutdown outcome.
 CONNECT carries your client ID, authentication fields, keepalive, and session properties. A successful CONNACK is connection evidence. Bad credentials produce an unsuccessful connection attempt; the rejected CONNACK is handled by the connection-error path rather than your success callback.
@@ -188,7 +223,7 @@ CONNECT carries your client ID, authentication fields, keepalive, and session pr
 4. Run two copies using the same client ID. Observe connection takeover/reconnect behavior, then stop both. Unique IDs are necessary for concurrently running clients.
 5. Press Ctrl+C while the broker is unavailable. Cancellation should interrupt the wait; do not wait for the full startup deadline.
-## Cautions and troubleshooting
+## 🔎 Cautions and troubleshooting
 - AutoPaho handles connection management; it does not invent topic authorization, business retries, or durable command storage.
 - Do not call `log.Fatal` from callbacks. It terminates the process without running deferred cleanup.
@@ -199,7 +234,11 @@ CONNECT carries your client ID, authentication fields, keepalive, and session pr
 **Completion:** Explain every field in `cfg`, distinguish the three contexts, and demonstrate successful startup, failed startup, reconnect, and cancellation.
-## Questions to answer before Section 3
+## ✅ Result — the story so far
 The backend now joins the broker as an authenticated MQTT 5 client. AutoPaho starts and manages the connection; the application waits until CONNACK confirms the broker accepted it, then keeps running and retries after a dropped connection. The backend has no subscriptions yet, so it is connected but is not listening for device messages.
 ## ❓ Questions to answer before Section 3
 1. Why can `NewConnection` return before a usable broker connection exists?
 2. What would happen if you passed `startup` to `NewConnection`?
@@ -213,7 +252,7 @@ CONNECT carries your client ID, authentication fields, keepalive, and session pr
 </details>
-## Sources
+## 🔗 Sources
 - [AutoPaho API v0.23.0](https://pkg.go.dev/github.com/eclipse/paho.golang@v0.23.0/autopaho)
 - [Tagged connection manager source](https://github.com/eclipse-paho/paho.golang/blob/v0.23.0/autopaho/auto.go)
@@ -1,11 +1,30 @@
-# Section 3 — Configure Go clients with TOML and koanf
+# ⚙️ Section 3 — Configure Go clients with TOML and koanf
 Prerequisite: Your Section 2 program connects and stops cleanly. Here you move the hard-coded settings into validated configuration. The module remains `example.com/mqtt5-learning`.
 **Version baseline:** Paho `v0.23.0`, verified from upstream release tags on 2026-10-07. It requires Go 1.24 or newer; Go 1.25 is compatible. The matching local reference is `references/paho.golang-v0.23.0/` (commit `1aa03969c5932464359b3e4ed752d9ac61abccc9`).
 ## 🎬 Start with the system story
-## Working directory and file placement
+> 💡 **Story question:** The backend, device simulator, and broker each have settings. How can each Go process get the right connection identity and credentials without baking them into source code?
 The broker configuration controls the service and its account/ACL source. Each Go process separately loads its client ID, broker URL, username, password, device ID, and timeout. The configuration loader merges ordinary defaults, TOML, environment overrides, and explicitly supplied CLI flags, validates the final values, then hands one typed result to the MQTT startup code.
 ```mermaid
 flowchart LR
    DEF[Defaults] --> LOAD[Go configuration loader]
    TOML[Local TOML settings<br/>including test password] --> LOAD
    ENV[Supported TEST_MQTT5_ overrides<br/>password excluded] --> LOAD
    CLI[Explicit CLI overrides] --> LOAD
    LOAD --> VALID[Validate effective client settings]
    VALID --> AUTO[AutoPaho ClientConfig]
    AUTO -->|CONNECT credentials and Client ID| BROKER[Mosquitto broker]
 ```
 Configuration errors should stop before network activity. A valid configuration with an unreachable broker is a different failure: it passes local validation but cannot complete the connection. See [Lesson 0](00-system-lifecycle.md) for how the validated values fit into the full system lifecycle.
 ## 📁 Working directory and file placement
 Run every shell command in this lesson from the learning project root:
@@ -15,7 +34,9 @@ cd /home/ryan/Media/Code/Code/robot-market/mqtt-5-learning
 Paths such as `cmd/backend/main.go` are relative to this directory. Markdown files under `lessons/` explain the work; create the application files at the paths shown, not inside `lessons/` or `references/`. Code blocks labeled Go, TOML, YAML, or conf are file contents; shell blocks are terminal commands. Create parent directories before saving files.
-## Why configuration changes MQTT behavior
+## ⚙️ Why configuration changes MQTT behavior
 > 💡 **Story question:** If a URL, Client ID, username, device ID, or timeout changes, which part of the broker/backend/device story changes with it?
 The URL chooses a broker; Client ID chooses session identity at that broker; username/password select credentials; device ID chooses the application's topic namespace. Changing one does not change the others. If you change `device_id` to `device-002` but keep the simulator credential, constructing a new topic string does not grant permission to use it.
@@ -23,7 +44,11 @@ Your connect timeout bounds how long startup waits. It is neither Keep Alive (co
 Predict two failures: a malformed URL should fail local validation before dialing; a valid URL with a wrong password reaches the broker but fails authentication. Retrying the latter forever does not fix the configured secret. The loader's purpose is to make the effective policy predictable before AutoPaho starts background activity.
-## 1. Add koanf's core, provider, and parser modules
+## ➕ 1. Add koanf's core, provider, and parser modules
 > 💡 **Story question:** Which pieces of the loader provide defaults, read the TOML file, and parse its syntax into values the Go application can use?
 Koanf combines those providers and parser into one effective configuration; these modules configure the application, not the MQTT broker.
 ```sh
 go get github.com/knadh/koanf/v2@v2.3.7
@@ -35,7 +60,11 @@ mkdir -p internal/config configs cmd/configcheck
 These supporting module versions are a reproducible lesson baseline; the requested latest-release selection concerns Paho. Providers and parsers are separate Go modules. `confmap` supplies defaults, `file` reads bytes, and `toml/v2` interprets those bytes as TOML. The loader uses explicit environment mappings and standard Go flags so you can see precedence clearly.
-## 2. Create a nonsecret TOML file
+## 🛠️ 2. Create the local lab TOML file
 > 💡 **Story question:** Which client settings belong in the local TOML file, and which identity values may differ between backend and simulator processes?
 Store the broker address, test password, example Client ID, and default backend username in TOML. Override the username for the simulator process.
 Create `configs/lab.toml` in the `configs/` directory created above. This is separate from the broker configuration at `lab/mosquitto/mosquitto.conf`; it configures your Go clients:
@@ -44,15 +73,20 @@ Create `configs/lab.toml` in the `configs/` directory created above. This is sep
 url = "mqtt://127.0.0.1:18883"
 client_id = "lab-backend"
 username = "backend"
 password = "your-local-lab-password"
 device_id = "device-001"
 [app]
 connect_timeout = "10s"
 ```
-Keep the password in `TEST_MQTT5_MQTT_PASSWORD`. The loader supports a TOML password field for precedence demonstration, but do not store a real password in the committed example. TOML tables map to nested struct fields.
+For this disposable local lab, put the same test password used for the broker accounts in `mqtt.password`. The application reads the password from TOML only; username and other settings can still be overridden. Do not reuse a personal or production password. TOML tables map to nested struct fields.
-## 3. Write the loader
+## 🛠️ 3. Write the loader
 > 💡 **Story question:** How do several configuration sources become one validated set of values before AutoPaho starts its background connection work?
 The loader merges each source in precedence order, maps environment names explicitly, decodes into typed fields, and validates the result. Follow the stages below before tracing individual lines.
 Create `internal/config/config.go`:
@@ -129,7 +163,6 @@ func Load(args []string) (Config, error) {
 		"TEST_MQTT5_MQTT_URL":            "mqtt.url",
 		"TEST_MQTT5_MQTT_CLIENT_ID":      "mqtt.client_id",
 		"TEST_MQTT5_MQTT_USERNAME":       "mqtt.username",
 		"TEST_MQTT5_MQTT_PASSWORD":       "mqtt.password",
 		"TEST_MQTT5_MQTT_DEVICE_ID":      "mqtt.device_id",
 		"TEST_MQTT5_APP_CONNECT_TIMEOUT": "app.connect_timeout",
 	}
@@ -198,17 +231,25 @@ func (c Config) Validate() error {
 ```
-## 4. Understand effective configuration and precedence
+## ⚙️ 4. Understand effective configuration and precedence
 > 💡 **Story question:** If both TOML and the environment set a timeout, which value reaches AutoPaho—and why should an absent CLI flag not replace it?
 Trace the merge order and the explicit-flag detection. The broker sees only the final connection settings sent in MQTT packets; it does not know which local source supplied them.
 The loader composes one effective configuration before connecting. `koanf.New(".")` interprets `mqtt.client_id` as a nested key; the `koanf` field tags map those keys into the typed configuration consumed by the transport. Later `Load`/`Set` calls override earlier values. The map provider already supplies structured values, so it needs no text parser.
-The order is defaults → TOML → environment → explicitly supplied CLI flags. `flags.Visit` visits only supplied flags, preventing the empty CLI defaults from overwriting TOML/environment values. `os.LookupEnv` distinguishes an absent variable from an explicitly empty one; an empty password deliberately overrides a file value and then fails validation.
+For ordinary settings the order is defaults → TOML → environment → explicitly supplied CLI flags. The password is read from TOML only, so an old exported `TEST_MQTT5_MQTT_PASSWORD` cannot silently replace it. `flags.Visit` visits only supplied flags, preventing empty CLI defaults from overwriting configured values. `os.LookupEnv` distinguishes an absent variable from an explicitly empty one for supported environment overrides.
 We map `TEST_MQTT5_MQTT_CLIENT_ID` explicitly to `mqtt.client_id`. Replacing every underscore with a dot would produce `mqtt.client.id`, a different key. This allowlist also makes supported environment variables visible; unexpected prefixed variables are ignored in this small example.
 Duration text is validated with `time.ParseDuration`, avoiding reliance on implicit conversion. The helper returns a `time.Duration` after validation. URL and topic validation are intentionally narrow for the loopback-only early lab. Later TLS lessons extend the schema and validation instead of disabling certificate checks.
-## 5. Add a configuration inspection command
+## ⚙️ 5. Add a configuration inspection command
 > 💡 **Story question:** How can you check the final effective configuration without opening a broker connection or exposing the password?
 The inspection executable reuses the same loader as the backend and prints a redacted summary, making source precedence testable independently of network state.
 Create `cmd/configcheck/main.go`:
@@ -241,9 +282,13 @@ go run ./cmd/configcheck -connect-timeout 0s
 Arguments after `./cmd/configcheck` belong to your program. `-client-id` overrides only that configured field; `-connect-timeout 0s` supplies a deliberately invalid zero-second duration. The inspection command loads and validates configuration without establishing an MQTT connection. Keep the backend password exported in this terminal for these checks.
-The first two commands should print a redacted summary. The last must fail before any broker connection. Do not print the whole struct: it contains a password.
+The first two commands should print a redacted summary. The last must fail before any broker connection. The password comes from `configs/lab.toml`; do not print the whole struct because it contains the password.
-## 6. Integrate it into Section 2
+## 🔗 6. Integrate it into Section 2
 > 💡 **Story question:** At what point should the settings cross from application configuration into AutoPaho, and what should happen if validation fails?
 Load and validate before creating the connection manager. Then replace hard-coded connection values with the typed configuration fields.
 Edit the existing `cmd/backend/main.go`. Keep one `package main`, one import block, and one copy of each function; the following integration steps modify the Section 2 program rather than create another executable. Run from the project root so the default `configs/lab.toml` path resolves correctly.
@@ -268,13 +313,13 @@ Remove the direct environment lookup/credential check and use these replacements
 The snippets are edits to Section 2's complete program, not additional standalone executables. Compile after integrating. Section 4 provides a complete replacement backend once you understand the loader.
-## Exercises: prove the precedence
+## 🧪 Exercises: prove the precedence
 1. Set TOML timeout to `8s`; unset `TEST_MQTT5_APP_CONNECT_TIMEOUT`; expect `8s`.
 2. Export `TEST_MQTT5_APP_CONNECT_TIMEOUT=6s`; expect `6s`.
 3. Pass `-connect-timeout 4s`; expect `4s`.
 4. Remove the flag; expect `6s`, proving an absent flag cannot override the environment.
-5. Try an empty password, missing TOML file, invalid URL, zero duration, and device ID `device/001`.
+5. Remove `mqtt.password` from a temporary TOML file, try a missing file, invalid URL, zero duration, and device ID `device/001`.
 6. Write a table-driven test using a temporary TOML file and `t.Setenv`. Clear relevant inherited environment values in each test; environment-mutating tests must not use `t.Parallel`.
 `unset NAME` removes a terminal environment override, revealing the TOML/default layer underneath; it does not change the file. `export NAME=value` sets the environment layer for child processes. A flag is temporary for that invocation. Restore your sample TOML timeout after the experiment so later lessons have a predictable starting point.
@@ -290,21 +335,25 @@ if err != nil { t.Fatal(err) }
 if cfg.ConnectTimeout() != 4*time.Second { t.Fatalf("wrong precedence: %s", cfg.ConnectTimeout()) }
 ```
-`filename` is the temporary fixture you create; the fragment belongs inside a `Test...` function in package `config`. Also test no flag, malformed values, and an empty password.
+`filename` is the temporary fixture you create; the fragment belongs inside a `Test...` function in package `config`. Also test no flag, malformed values, and a missing password in the TOML fixture.
-## Common mistakes
+## ⚠️ Common mistakes
 - TOML paths are relative to the process working directory. Run from the project root or pass an explicit `-config` path.
- Exporting backend credentials globally will also affect the simulator. Set simulator credentials in its own terminal.
+- An exported username can override the TOML username; use `-username simulator` when starting the simulator.
 - Command-line password flags expose secrets to process inspection, so this loader deliberately omits one.
 - Configuration precedence is a merge decision, not an MQTT feature. Paho receives only the final validated values.
 - These samples do not reject all unknown TOML keys. Add strict key validation as an exercise if typo detection becomes important.
 **Completion:** Demonstrate the override sequence and invalid-startup cases, then explain how the final values reach AutoPaho.
-## Sources
+## 🔗 Sources
 - [koanf concepts, providers, and unmarshalling](https://pkg.go.dev/github.com/knadh/koanf/v2@v2.3.7)
 - [AutoPaho configuration v0.23.0](https://pkg.go.dev/github.com/eclipse/paho.golang@v0.23.0/autopaho#ClientConfig)
 ## ✅ Result — the story so far
 The backend's connection details now come from a predictable, validated configuration: defaults, TOML, environment, then explicit flags. Before opening an MQTT connection, the program has checked that its broker URL, identity, device ID, and timeouts make sense. The simulator can select its own username and client ID while using the same lab settings; message exchange still comes in the next lesson.
 Next: [Section 4](04-publish-subscribe-simulator.md).
@@ -1,11 +1,57 @@
-# Section 4 — Publish, subscribe, and build a Go device simulator
+# 📡 Section 4 — Publish, subscribe, and build a Go device simulator
 Prerequisites: Sections 1–3 complete. Keep Section 3's configuration package and TOML file. This lesson supplies complete files for a reusable MQTT transport, backend, and simulator. Implement each file in stages and explain it before running both programs.
 **Version baseline:** Paho `v0.23.0`, verified from upstream release tags on 2026-10-07. It requires Go 1.24 or newer; Go 1.25 is compatible. The matching local reference is `references/paho.golang-v0.23.0/` (commit `1aa03969c5932464359b3e4ed752d9ac61abccc9`).
-## 1. Map the conversation
+## 🎬 Start with the system story
 > 💡 **Story question:** The backend and device are separate MQTT clients. How do they exchange a command without knowing each other's address, and how can the backend tell whether the device actually handled it?
 The broker is the meeting point. The backend connects and subscribes to telemetry and responses; the simulator connects with its own identity and subscribes to commands. Each publication goes to the broker, which routes it to matching subscriptions. A device response is a new application message, separate from the broker's acknowledgement of the backend's publish.
 ```mermaid
 sequenceDiagram
    participant G as Go backend
    participant B as Mosquitto broker
    participant D as Go device simulator
    G->>B: CONNECT
    B-->>G: CONNACK accepted
    D->>B: CONNECT
    B-->>D: CONNACK accepted
    G->>B: SUBSCRIBE telemetry, responses
    B-->>G: SUBACK
    D->>B: SUBSCRIBE commands
    B-->>D: SUBACK
    G->>B: PUBLISH ping, QoS 1
    par Backend-to-broker acknowledgement
        B-->>G: PUBACK
    and Broker-to-device delivery
        B->>D: PUBLISH ping, QoS 1
        D->>D: Paho callback queues message
        D-->>B: PUBACK after callback returns
    end
    D->>D: Application handles ping
    D->>B: PUBLISH pong response, QoS 1
    par Device-to-broker acknowledgement
        B-->>D: PUBACK
    and Broker-to-backend delivery
        B->>G: PUBLISH pong response, QoS 1
        G->>G: Paho callback queues message
        G-->>B: PUBACK after callback returns
    end
 ```
 The broker, backend, and device have distinct setup. Mosquitto owns its listener, password source, and ACL. Each Go process loads its own broker URL, client ID, and username. The backend and simulator use different usernames for their broker permissions; both use the disposable test password from `configs/lab.toml`. The code below first builds the shared transport that connects, subscribes, receives, and publishes; then the two application loops use it.
 Before each implementation part, use the question there to predict which process or protocol layer is responsible. Follow [Lesson 0](00-system-lifecycle.md) for the full lifecycle map.
 ## 🗺️ 1. Map the conversation
 > 💡 **Story question:** Which client publishes each kind of message, which client needs it, and what does the broker use to route it?
 Map the telemetry, command, and response paths first. This identifies which process must subscribe before we write transport code.
 | Topic | Publisher | Subscriber | Example |
 | --- | --- | --- | --- |
@@ -15,7 +61,7 @@ Prerequisites: Sections 1–3 complete. Keep Section 3's configuration package a
 A broker subscription and a local Go callback are different things. SUBSCRIBE asks the broker which messages to forward. `OnPublishReceived` tells your local client how to process forwarded PUBLISH packets. You need both. This lab's topic convention is deliberately separate from the production contracts.
-### Trace two independent delivery hops
+### 🔄 Trace two independent delivery hops
 ```mermaid
 sequenceDiagram
@@ -25,16 +71,25 @@ sequenceDiagram
    S->>M: SUBSCRIBE commands, maximum QoS 1
    M-->>S: SUBACK with grant or rejection
    B->>M: PUBLISH command, QoS 1
-    M-->>B: PUBACK for backend-to-broker hop
+    par Backend publish exchange
        M-->>B: PUBACK
    and Broker forwards to simulator
        M->>S: PUBLISH command, granted delivery QoS
-    S-->>M: PUBACK for broker-to-simulator hop
+        S-->>M: PUBACK for delivery hop
-    S->>M: PUBLISH application result
+    end
    S->>S: Application handles command
    S->>M: PUBLISH application result, QoS 1
    par Simulator publish exchange
        M-->>S: PUBACK
    and Broker forwards result to backend
        M->>B: PUBLISH application result
        B-->>M: PUBACK for delivery hop
    end
 ```
-This diagram groups exchanges by purpose; it does not promise the backend sees PUBACK before the simulator's response. Each side has its own connection and packet identifiers. The broker routes by topic, not by JSON fields. A result is another publication on another topic; MQTT does not create a response automatically.
+The `par` blocks show that a publisher-side PUBACK and broker-to-subscriber delivery are independent work on separate client connections; their observed order can vary. Each QoS 1 PUBACK confirms only its own hop. The application result is a separate publication—the broker does not create it automatically, and neither PUBACK proves that business work completed.
-### Choose QoS by the guarantee you need
+### 📨 Choose QoS by the guarantee you need
 | QoS | Successful exchange on one hop | What the Go application must still handle |
 | --- | --- | --- |
@@ -46,13 +101,17 @@ The subscriber requests a **maximum** delivery QoS. A QoS 1 publication cannot b
 For a QoS 2 experiment, change both publication and requested subscription QoS to 2, accept SUBACK grants 0–2, and inspect the four-packet exchange. The original helper intentionally accepts only grants up to its requested QoS 1. Do not change only the publish field and call the resulting lower-QoS subscriber path an end-to-end QoS 2 test.
-### Topic filters are routing rules
+### 🏷️ Topic filters are routing rules
 `lab/v1/devices/+/telemetry` matches one device segment; `lab/v1/devices/#` matches that subtree. `+` occupies an entire level and `#` is the final level. Topics are case-sensitive, so `device-001` and `Device-001` differ. Wildcards belong in subscriptions, never publish topic names. A root `#` filter does not include `$SYS/...`; broker system topics require an explicit matching filter and permission.
 A SUBACK proves the subscription request received a result. Topic ACLs still govern delivery: an accepted broad filter does not prove access to every matching topic. Check an authorized observer, reason codes, and broker logs when testing denial.
-## 2. Create the transport helper
+## 🛠️ 2. Create the transport helper
 > 💡 **Story question:** What should be shared between the backend and simulator, and what must remain specific to each running client?
 The transport helper can share connection lifecycle, subscription handling, message copying, and publishing. Each caller still supplies its configured username, Client ID, topic list, and receive queue. Keeping those identities separate lets the broker apply different permissions and prevents Client ID takeover.
 ```sh
 mkdir -p internal/mqtt cmd/simulator
@@ -81,11 +140,15 @@ type Message struct {
 	Retain  bool
 }
 // Start creates the MQTT connection, subscribes to the requested topics, and
 // waits until the initial connection and subscriptions are ready.
 func Start(ctx context.Context, cfg config.Config, topics []string, inbox chan<- Message) (*autopaho.ConnectionManager, error) {
 	// Parse the configured broker URL before creating the reconnecting client.
 	broker, err := url.Parse(cfg.MQTT.URL)
 	if err != nil {
 		return nil, err
 	}
 	// The callback reports its first SUBACK so startup can fail early if needed.
 	initialSubscription := make(chan error, 1)
 	manager, err := autopaho.NewConnection(ctx, autopaho.ClientConfig{
 		ServerUrls:                    []*url.URL{broker},
@@ -94,6 +157,7 @@ func Start(ctx context.Context, cfg config.Config, topics []string, inbox chan<-
 		SessionExpiryInterval:         0,
 		ConnectUsername:               cfg.MQTT.Username,
 		ConnectPassword:               []byte(cfg.MQTT.Password),
 		// Recreate subscriptions after every successful connection or reconnect.
 		OnConnectionUp: func(cm *autopaho.ConnectionManager, _ *paho.Connack) {
 			// This runs again on reconnect; reestablish broker subscriptions.
 			subCtx, cancel := context.WithTimeout(ctx, 5*time.Second)
@@ -102,6 +166,7 @@ func Start(ctx context.Context, cfg config.Config, topics []string, inbox chan<-
 			for _, topic := range topics {
 				subscriptions = append(subscriptions, paho.SubscribeOptions{Topic: topic, QoS: 1, RetainAsPublished: true})
 			}
 			// Ask for QoS 1 and preserve RETAIN so consumers can identify stored data.
 			ack, subErr := cm.Subscribe(subCtx, &paho.Subscribe{Subscriptions: subscriptions})
 			if subErr == nil {
 				if ack == nil || len(ack.Reasons) != len(topics) {
@@ -127,6 +192,7 @@ func Start(ctx context.Context, cfg config.Config, topics []string, inbox chan<-
 		},
 		OnConnectionDown: func() bool { log.Print("disconnected; reconnecting"); return true },
 		OnConnectError:   func(err error) { log.Printf("connect error: %v", err) },
 		// Keep the packet callback short: copy the message and enqueue it for work.
 		ClientConfig: paho.ClientConfig{
 			ClientID:           cfg.MQTT.ClientID,
 			OnClientError:      func(err error) { log.Printf("client error: %v", err) },
@@ -135,6 +201,8 @@ func Start(ctx context.Context, cfg config.Config, topics []string, inbox chan<-
 				func(received paho.PublishReceived) (bool, error) {
 					packet := received.Packet
 					message := Message{Topic: packet.Topic, Payload: append([]byte(nil), packet.Payload...), Retain: packet.Retain}
 					// Never block the network callback when the bounded inbox is full.
 					select {
 					case inbox <- message:
 					default:
@@ -149,6 +217,7 @@ func Start(ctx context.Context, cfg config.Config, topics []string, inbox chan<-
 	if err != nil {
 		return nil, fmt.Errorf("start AutoPaho: %w", err)
 	}
 	// Bound initial connection and subscription setup by the configured timeout.
 	startup, cancel := context.WithTimeout(ctx, cfg.ConnectTimeout())
 	defer cancel()
 	if err := manager.AwaitConnection(startup); err != nil {
@@ -169,6 +238,7 @@ func Start(ctx context.Context, cfg config.Config, topics []string, inbox chan<-
 }
 func Publish(ctx context.Context, manager *autopaho.ConnectionManager, topic string, payload []byte) error {
 	// Publishing waits only for this bounded operation, not for device completion.
 	operation, cancel := context.WithTimeout(ctx, 3*time.Second)
 	defer cancel()
 	ack, err := manager.Publish(operation, &paho.Publish{
@@ -189,6 +259,7 @@ func Publish(ctx context.Context, manager *autopaho.ConnectionManager, topic str
 }
 func Stop(manager *autopaho.ConnectionManager) {
 	// Give the MQTT disconnect handshake a short, independent shutdown window.
 	shutdown, cancel := context.WithTimeout(context.Background(), 5*time.Second)
 	defer cancel()
 	if err := manager.Disconnect(shutdown); err != nil {
@@ -198,7 +269,11 @@ func Stop(manager *autopaho.ConnectionManager) {
 ```
-## 3. Map the helper to MQTT behavior
+## 🔗 3. Map the helper to MQTT behavior
 > 💡 **Story question:** When `Start` says subscriptions are ready, which broker exchange has happened, and what is the local callback's separate job?
 The broker has received `SUBSCRIBE` and returned an acceptable `SUBACK`; the callback is configured in the Go process to receive broker-forwarded PUBLISH packets. Read the implementation against these two sides of the receive path.
 | Code decision | Why it matters |
 | --- | --- |
@@ -220,7 +295,40 @@ For QoS 1, `Publish` returns a response describing the immediate broker acknowle
 The initial-subscription channel allows startup to wait for SUBACK as well as CONNACK. The callback subscribes again on reconnect. This helper checks initial readiness only; reconnect subscription failures are logged, not represented as continuous health state. Add that state later rather than treating initial readiness as permanent.
-## 4. Replace the backend main program
+## 💻 4. Replace the backend main program
 > 💡 **Story question:** After configuration, connection, and subscriptions are ready, how does a backend timer become an MQTT command?
 ### 🦟 Trace one ping from Go value to broker
 On each timer tick, the backend first builds an ordinary Go value. `json.Marshal` turns that value into the byte sequence sent as the MQTT payload:
 ```go
 struct {
    ID      string `json:"id"`
    Command string `json:"command"`
 }{ID: "ping-123-1", Command: "ping"}
 ```
 The JSON field tags set the wire names, so the payload bytes represent:
 ```json
 {"id":"ping-123-1","command":"ping"}
 ```
 The backend chooses the concrete topic `lab/v1/devices/device-001/commands` using its configured device ID. It passes that topic and the JSON bytes to `mqtt.Publish`; that helper builds a Paho `paho.Publish` with **QoS 1**, **Retain false**, and Content Type `application/json`, then calls AutoPaho's `manager.Publish`.
 `manager.Publish` waits for the QoS 1 publish exchange or an error/timeout. When it receives a successful `PUBACK`, the helper logs the reason code and returns nil. That confirms the backend-to-broker QoS exchange. A success reason of `0x10` means the broker had no matching subscribers; it is still a successful MQTT acknowledgement, so it does not prove the simulator received anything. A timeout can also leave the outcome uncertain.
 The broker independently forwards a matching PUBLISH to the simulator's `.../commands` subscription. The simulator's Paho callback copies it into an inbox; the simulator's main loop decodes the JSON, checks that it is a non-retained `ping` with an ID, and publishes a `pong` carrying that same ID to `.../responses`. That new response is application-level evidence the simulator handled the ping. The backend is subscribed to the response topic and currently logs the received bytes; matching responses to pending commands is added in Lesson 7.
 ```text
 Go struct → JSON payload bytes → QoS 1 PUBLISH → broker PUBACK
                                            └→ simulator receives and handles ping
                                               → separate pong PUBLISH → backend receives
 ```
 The two arrows after the backend's PUBLISH are separate outcomes: PUBACK may arrive before or after the simulator's response. Retain is false because a ping is a one-time action; storing it as retained state could replay the command to a later subscriber.
 Create `cmd/backend/main.go`:
@@ -242,6 +350,7 @@ import (
 )
 func main() {
 	// Convert a returned application error into a visible nonzero process exit.
 	if err := run(); err != nil {
 		log.Print(err)
 		os.Exit(1)
@@ -249,12 +358,14 @@ func main() {
 }
 func run() error {
 	// Load settings and bind cancellation to Ctrl+C or the container stop signal.
 	cfg, err := config.Load(os.Args[1:])
 	if err != nil {
 		return err
 	}
 	ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
 	defer stop()
 	// Build this device's topic namespace and bounded receive queue.
 	base := "lab/v1/devices/" + cfg.MQTT.DeviceID
 	inbox := make(chan mqtt.Message, 16)
 	manager, err := mqtt.Start(ctx, cfg, []string{base + "/telemetry", base + "/responses"}, inbox)
@@ -262,11 +373,13 @@ func run() error {
 		return err
 	}
 	defer mqtt.Stop(manager)
 	// Publish synthetic ping commands periodically while processing received data.
 	ticker := time.NewTicker(10 * time.Second)
 	defer ticker.Stop()
 	runID := time.Now().UnixNano()
 	sequence := 0
 	for {
 		// The select keeps shutdown, inbound messages, and periodic commands responsive.
 		select {
 		case <-ctx.Done():
 			return nil
@@ -293,7 +406,11 @@ func run() error {
 The ticker sends one ping every ten seconds. It starts after the backend's own subscriptions are ready. A timestamp and sequence make IDs distinct enough for this demonstration, not a production idempotency system. The backend logs the response but does not yet implement pending requests or validate completion; Section 7 adds those semantics.
-## 5. Create the device simulator
+## 🛠️ 5. Create the device simulator
 > 💡 **Story question:** How does a received MQTT packet become a device action, and how does the simulator report its result without blocking Paho's network callback?
 The callback copies the packet into a bounded inbox. The simulator's main loop reads it, validates the harmless ping, and publishes a pong. This draws a deliberate boundary between packet reception and application behavior; later lessons examine the loss window and acknowledgement timing.
 Create `cmd/simulator/main.go`:
@@ -314,6 +431,7 @@ import (
 )
 func main() {
 	// Convert a returned application error into a visible nonzero process exit.
 	if err := run(); err != nil {
 		log.Print(err)
 		os.Exit(1)
@@ -321,12 +439,14 @@ func main() {
 }
 func run() error {
 	// Load settings and bind cancellation to Ctrl+C or the container stop signal.
 	cfg, err := config.Load(os.Args[1:])
 	if err != nil {
 		return err
 	}
 	ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
 	defer stop()
 	// Build this device's topic namespace and bounded receive queue.
 	base := "lab/v1/devices/" + cfg.MQTT.DeviceID
 	inbox := make(chan mqtt.Message, 16)
 	manager, err := mqtt.Start(ctx, cfg, []string{base + "/commands"}, inbox)
@@ -334,9 +454,11 @@ func run() error {
 		return err
 	}
 	defer mqtt.Stop(manager)
 	// Heartbeats are periodic; command handling remains active between ticks.
 	ticker := time.NewTicker(3 * time.Second)
 	defer ticker.Stop()
 	for {
 		// Handle cancellation, telemetry publication, and inbound commands together.
 		select {
 		case <-ctx.Done():
 			return nil
@@ -352,6 +474,7 @@ func run() error {
 				log.Printf("heartbeat: %v", err)
 			}
 		case message := <-inbox:
 			// Decode into a local command value only after rejecting retained messages.
 			var command struct {
 				ID      string `json:"id"`
 				Command string `json:"command"`
@@ -364,6 +487,7 @@ func run() error {
 				log.Print("ignoring malformed or unsupported command")
 				continue
 			}
 			// Echo the command ID so the backend can associate this pong with its ping.
 			response, err := json.Marshal(struct {
 				ID       string `json:"id"`
 				DeviceID string `json:"device_id"`
@@ -385,29 +509,27 @@ func run() error {
 This simulator sends a heartbeat every three seconds, receives ping commands, and publishes a pong carrying the same ID. It rejects packets marked retained to avoid treating stored state as a fresh one-shot action. The subscription sets `RetainAsPublished: true`: with the default false, live forwarding can clear RETAIN and defeat a guard that checks only `message.Retain`. Keep command publishers nonretaining too; this guard does not remove a retained value already stored at the broker. Full contract validation and duplicate handling arrive in later sections; this only demonstrates the transport lifecycle.
-## 6. Run two separate identities
+## ▶️ 6. Run two separate identities
 > 💡 **Story question:** Why start two processes with different usernames and Client IDs, and what output proves each stage of the path?
 The simulator and backend represent independent clients. Their connection logs, subscription results, broker publish acknowledgement, and pong are separate evidence. Start the simulator first so its command subscription exists before the backend publishes.
 Terminal A, simulator first:
 ```sh
-export TEST_MQTT5_MQTT_USERNAME=simulator
+go run ./cmd/simulator -username simulator -client-id lab-simulator-device-001
 read -rs 'TEST_MQTT5_MQTT_PASSWORD?Simulator lab password: '
 export TEST_MQTT5_MQTT_PASSWORD
 go run ./cmd/simulator -client-id lab-simulator-device-001
 ```
 Terminal B, backend:
 ```sh
-export TEST_MQTT5_MQTT_USERNAME=backend
+go run ./cmd/backend -username backend -client-id lab-backend
 read -rs 'TEST_MQTT5_MQTT_PASSWORD?Backend lab password: '
 export TEST_MQTT5_MQTT_PASSWORD
 go run ./cmd/backend -client-id lab-backend
 ```
-Open two independent terminal sessions and change both to the project root. Their environments are separate, so the simulator's exported credentials do not overwrite the backend's. The zsh `read -rs` prompt hides the password; `export` passes it to the child process. `go run` runs each executable in the foreground; `-client-id` is a program flag overriding the TOML default, not a Docker setting. Stop each program with Ctrl+C and unset its password when finished.
+Open two independent terminal sessions and change both to the project root. The shared lab password comes from `configs/lab.toml`; `-username` selects the broker account for each process, and `-client-id` overrides the TOML client ID. Stop each foreground program with Ctrl+C when finished.
-Use the same configured `device-001` in both. Distinct username/password pairs authorize different topic directions. Distinct client IDs prevent connection takeover. A username, client ID, and payload device ID serve different purposes; the ACL is what binds this simulator credential to its permitted device namespace.
+Use the same configured `device-001` in both. Distinct usernames authorize different topic directions; both local accounts use the test password in `configs/lab.toml`. Distinct client IDs prevent connection takeover. A username, client ID, and payload device ID serve different purposes; the ACL is what binds this simulator credential to its permitted device namespace.
 Expected backend output, abbreviated (timestamps and IDs vary):
@@ -420,7 +542,11 @@ received topic=lab/v1/devices/device-001/responses ... payload={"id":"ping-...",
 Log order is not guaranteed: the device response may be observed around the publication acknowledgement. Separate connections and goroutines progress independently. If the simulator is not subscribed yet, a ping may have no recipient; this is expected with these zero-expiry early-lab sessions.
-## 7. Experiments to learn the API
+## 🧪 7. Experiments to learn the API
 > 💡 **Story question:** Which part of the system changes in each experiment—the broker's routing/state, the client's delivery setting, or the application's handling policy?
 Predict the effect at that boundary before changing the code, then use packet or application evidence to explain what happened.
 1. Replace the backend's exact telemetry subscription with `lab/v1/devices/+/telemetry`. Explain the topic filter versus a concrete publish topic. Never publish to a topic containing `+` or `#`.
 2. Add `Retain: true` only to synthetic telemetry; restart the backend and observe a retained value. Restore `false` afterward. Do not retain one-shot commands.
@@ -444,7 +570,7 @@ The new publication does not ask the broker to change retained storage. Delete t
 </details>
-## Cautions: limits of this early example
+## ⚠️ Cautions: limits of this early example
 - The inbox is deliberately bounded and lossy. It makes overload visible without blocking receive callbacks. It is unsuitable for durable sales reports or real commands; later sections replace this policy with persistence and deliberate acknowledgement/backpressure rules.
 - A publish call can block this single application loop for up to three seconds; this is acceptable for a tiny demonstration. Later sections add workers and separate business processing.
@@ -454,7 +580,7 @@ The new publication does not ask the broker to change retained storage. Delete t
 - Raw payload logs are allowed here only for small synthetic lab messages; production logs need redaction and size limits.
 - Never close the inbox while Paho may still send to it. These programs let the context and manager stop the producer; the channel need not be closed for process exit.
-## Verification and completion
+## ✅ Verification and completion
 ```sh
 gofmt -w cmd internal
@@ -471,11 +597,15 @@ Packages without tests do not demonstrate runtime correctness. Also run both pro
 **Completion:** Explain the complete path from backend Go struct → JSON bytes → MQTT PUBLISH → simulator callback → application loop → response PUBLISH → backend callback. Distinguish broker acknowledgement from a device response, and identify the lossy queue and offline-session limitations yourself.
-## Sources
+## 🔗 Sources
 - [Paho client API v0.23.0](https://pkg.go.dev/github.com/eclipse/paho.golang@v0.23.0/paho)
 - [Tagged AutoPaho source](https://github.com/eclipse-paho/paho.golang/blob/v0.23.0/autopaho/auto.go)
 - [Tagged receive callback source](https://github.com/eclipse-paho/paho.golang/blob/v0.23.0/paho/client.go)
 - [MQTT 5 specification](https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html)
 ## ✅ Result — the story so far
 The backend now subscribes to device telemetry and responses, while the simulator subscribes to that device's commands. The simulator publishes heartbeats for the backend; when the backend sends a QoS 1 ping, the broker forwards it to the simulator, which handles it and publishes a pong carrying the same command ID. The PUBACKs confirm their individual MQTT hops; the pong is the separate application evidence that the simulator handled the ping.
 Next: [Section 5 — Contracts and validation](05-contracts-validation.md).
@@ -1,10 +1,28 @@
-# Section 5 — Design contracts and validate inbound messages
+# 📋 Section 5 — Design contracts and validate inbound messages
-Prerequisite: Section 4 exchanges heartbeats and ping/pong.
+## 🎬 Start with the system story
 > 💡 **Story question:** The broker can route a message because its topic matches. How does the Go backend decide that the topic and payload describe a valid message from the expected device?
 Routing is not validation or identity proof. A message arrives as a topic plus bytes; the application must parse the route, decode the payload, and check that fields and identity make sense before changing state. A device ID inside JSON is a claim, so broker authentication and topic authorization remain separate safeguards.
 ```mermaid
 flowchart LR
    B[Broker routes matching topic] --> P[Paho delivers topic + bytes]
    P --> T[Parse topic and device scope]
    T --> J[Decode versioned JSON]
    J --> V[Validate identity, IDs, size, time]
    V -->|valid| H[Business handler]
    V -->|invalid| E[Safe rejection / metric]
 ```
 **Before each implementation step, ask:** What can be trusted at this boundary, and what must be checked before the application acts? Follow [Lesson 0](00-system-lifecycle.md) for the complete message path.
 Prerequisite: Section 4 exchanges heartbeats and ping/pong. The clients share the disposable password in `configs/lab.toml`; select the simulator account with `-username simulator` when starting it.
 Run all commands from `/home/ryan/Media/Code/Code/robot-market/mqtt-5-learning`. Create files at the project-relative paths shown below, not under `lessons/`. Continue the same application and pinned Paho v0.23.0 dependency. These are guided implementation exercises; attempt each step before requesting a full solution.
-## Follow one message through the trust boundaries
+## 📨 Follow one message through the trust boundaries
 Suppose the backend receives topic `lab/v1/devices/device-001/sales` with a body claiming `device_id: device-002`. The broker may have correctly authorized the sender to publish to the topic. It has not established that the JSON is truthful. Reject the mismatch before selecting a device record or changing a balance.
@@ -17,24 +35,27 @@ Each step answers a different question. The size limit bounds decoding cost. Par
 The 16 KiB application limit applies to `Payload`, not the entire MQTT packet. Topic and properties add bytes; the library has already received the packet before this validator runs. Lesson 16 adds protocol packet-size limits to bound an earlier layer.
-### Map encoding metadata into Paho
+### 📨 Map encoding metadata into Paho
 This fragment belongs where you build a `paho.Publish`; `payload` is the JSON produced by your contract encoder:
 ```go
 utf8Payload := byte(1)
 // Describe the payload's encoding and media type for MQTT 5 consumers.
 properties := &paho.PublishProperties{
    ContentType:   "application/json",
    PayloadFormat: &utf8Payload,
 }
-_ = properties // attach as Publish.Properties in your implementation
+// Attach these properties to the PUBLISH packet built by your transport helper.
 _ = properties
 ```
 Payload Format Indicator 1 declares UTF-8 character data. It does not mean JSON, schema version 1, or trusted content. Content Type describes the media type. For example, `hello` is UTF-8 but fails your JSON object contract. Property pointers distinguish an omitted property from an explicitly supplied value; inspect nil before dereferencing incoming fields.
 Keep these distinctions in your transport type when needed: packet metadata belongs alongside payload bytes, while `schema_version` and `command_id` belong to your application contract. Do not make handlers depend on a mutable packet owned elsewhere merely to read metadata.
-### Predict, modify, explain
+### 🧪 Predict, modify, explain
 Send a valid heartbeat, then change only its body device ID, then change only its schema version. Predict which validation stage rejects each. Finally publish valid JSON with an extra trailing object: one successful decoder call is insufficient unless you also require EOF.
@@ -45,7 +66,11 @@ MQTT receipt and application validity are separate. With the early automatic-ack
 </details>
-## 1. Create the contract package
+## 📋 1. Create the contract package
 > 💡 **Story question:** What shared vocabulary lets the backend and device agree on the shape and meaning of each message?
 The contract package defines those wire-level names and fields so neither client silently invents a different JSON shape.
 ```sh
 mkdir -p internal/contracts
@@ -55,23 +80,37 @@ Create `internal/contracts/messages.go`, `topics.go`, `decode.go`, and `decode_t
 Define JSON structs for Heartbeat, Command, CommandResult, and SalesReport. Use `schema_version`, `message_id`, `device_id`, and `occurred_at` consistently. Command also needs `command_id` and `command`; CommandResult needs the same command ID and a status. A sales report needs a stable report ID and synthetic line items with positive quantities and integer minor-unit amounts. Keep command identity stable across retries; an MQTT packet identifier is not that identity.
-## 2. Parse topics before routing
+## 🏷️ 2. Parse topics before routing
 > 💡 **Story question:** A message arrives with both a topic and a payload. Why decide which device and message kind the topic represents before running a handler?
 Parsing the topic establishes the intended route from broker metadata; it also gives validation a value to compare against any device identity claimed in the payload.
 Implement a parser accepting exactly `lab/v1/devices/{deviceID}/{kind}`. Initially permit telemetry, commands, responses, and sales. Reject empty device IDs, extra segments, wildcards, and unknown kinds. Return a typed device ID and kind rather than making every handler split strings itself. Extend the lab ACL explicitly for sales only when you start publishing reports.
-## 3. Decode and validate
+## ✅ 3. Decode and validate
 > 💡 **Story question:** JSON decoding succeeded. What else must be true before this message is safe for application state?
 Structure is only one check. Bound the input, enforce the supported schema and required IDs, validate timestamps and values, and compare topic identity with the body claim.
 Apply a 16 KiB payload limit before decoding. Use `json.Decoder` with `DisallowUnknownFields`, then require EOF after the first JSON value. Validate schema version 1, required IDs, supported commands, topic/body device agreement, RFC3339 timestamps, and field lengths. Choose and document a future-clock tolerance, such as 30 seconds; inject the current time into validation to keep tests deterministic. Historical sales may legitimately be old, so do not apply heartbeat freshness rules to every report.
 Treat Content Type and Payload Format Indicator as descriptive metadata. Validate bytes even when they claim to contain JSON. A payload device ID is a claim; broker credentials and ACLs supply the separate authorization boundary.
-## 4. Integrate and verify
+## 🧪 4. Integrate and verify
 > 💡 **Story question:** Where should validation sit so malformed messages cannot trigger a response or change state?
 Run it between transport reception and business handling. Verify both accepted messages and rejection cases, and inspect broker ACLs separately when testing sender permissions.
 Replace Section 4's anonymous JSON structs in both programs with these contracts. Validate before updating state or sending a result. Log the error category and bounded identifiers, without dumping rejected payloads. Unsupported or malformed commands must have no simulated effect.
 ```sh
 gofmt -w internal/contracts cmd
 go test ./internal/contracts
 go test ./...
 ```
@@ -83,4 +122,8 @@ Test valid messages, malformed JSON, trailing JSON, unknown fields, missing IDs,
 **Done when:** Both programs still exchange ping/pong, every rejection has a predictable reason, and invalid sales or commands leave state unchanged. Explain why schema validation does not authenticate a device.
 ## ✅ Result — the story so far
 The backend and simulator now have a shared vocabulary for heartbeats, commands, results, and sales reports. When a message arrives, the receiver checks its topic, size, JSON shape, schema version, identifiers, timestamp, and topic/body device match before changing application state. A broker-routed message can still be rejected by the application; successful MQTT delivery alone does not make its contents valid or trusted.
 Previous: [04-publish-subscribe-simulator.md](04-publish-subscribe-simulator.md). Next: [06-concurrency-cancellation.md](06-concurrency-cancellation.md).
@@ -1,10 +1,28 @@
-# Section 6 — Manage callbacks, concurrency, and cancellation
+# 🧵 Section 6 — Manage callbacks, concurrency, and cancellation
-Prerequisite: Section 5 validates messages before handling them.
+## 🎬 Start with the system story
 > 💡 **Story question:** A burst of broker-forwarded messages arrives while the backend is doing slow work. Who is receiving packets, who is doing business work, and what happens when the application cannot keep up?
 Paho's network/client machinery receives packets and invokes configured handlers. Your application decides how that work is scheduled, bounded, cancelled, and acknowledged. If a callback blocks or a queue fills, that can affect unrelated deliveries; if a message is acknowledged before durable handling and then dropped from RAM, MQTT cannot recover that application copy by itself.
 ```mermaid
 flowchart LR
    B[Broker] -->|PUBLISH| P[Paho receive path]
    P --> C[Short callback copies/enqueues]
    C --> Q[Bounded application queue]
    Q --> W[Worker handles and persists]
    Q -->|full| O[Explicit overload policy]
    W --> A[Application outcome]
 ```
 **Before each implementation step, ask:** What work must happen before the MQTT acknowledgement, what can happen afterward, and what is the defined behavior when capacity is exhausted? Follow [Lesson 0](00-system-lifecycle.md) for the broker-to-callback-to-application path.
 Prerequisite: Section 5 validates messages before handling them. The backend and simulator continue to read their shared disposable password from `configs/lab.toml`; select each broker username with `-username` when starting the process.
 Run all commands from `/home/ryan/Media/Code/Code/robot-market/mqtt-5-learning`. Create files at the project-relative paths shown below, not under `lessons/`. Continue the same application and pinned Paho v0.23.0 dependency. These are guided implementation exercises; attempt each step before requesting a full solution.
-## Understand the acknowledgement boundary before adding workers
+## ✅ Understand the acknowledgement boundary before adding workers
 For the pinned client's automatic mode, `routePublishPackets` invokes the registered handlers, then calls its session acknowledgement path for nonzero QoS. The boolean and error returned by a handler are passed to later handlers; returning an error does not turn the callback into an application-controlled MQTT NACK or force redelivery.
@@ -21,7 +39,7 @@ In manual mode, `EnableManualAcknowledgment` changes who releases acknowledgemen
 The client's tracker sends acknowledgements in receipt order. If work 3 finishes before 1, acknowledging 3 can remain buffered behind 1. Slow work therefore consumes inflight capacity even with several workers. Manual acknowledgement delays protocol completion; it does not persist your Go queue, remove duplicate effects, or automatically schedule retries while the same connection stays open.
-### Choose what the callback promises
+### ↩️ Choose what the callback promises
 | Message class | Reasonable admission choice | Meaning of a successful admission |
 | --- | --- | --- |
@@ -31,7 +49,7 @@ The client's tracker sends acknowledgements in receipt order. If work 3 finishes
 For this in-memory lesson, keep the loss limitation visible. Use a fake acknowledgement recorder to test ordering of admission and acknowledgement without pretending that the fake verifies broker behavior.
-### Predict the overload
+### 🧪 Predict the overload
 Two workers taking 100 ms per item can complete about 20 items/second before other overhead. A burst of 100 items arriving much faster cannot fit into two busy workers plus 16 waiting slots. Predict rejection, then measure actual accepted/completed/rejected counts; scheduler timing affects the exact split. Increasing workers may reorder effects for the same device. If revision order matters, serialize that device's work or enforce revision checks.
@@ -44,32 +62,51 @@ Not in the pinned automatic callback path. These return values describe local ha
 </details>
-## 1. Trace callback and acknowledgement order
+## ↩️ 1. Trace callback and acknowledgement order
 > 💡 **Story question:** Does returning from the receive callback mean the business operation is durable, or only that the MQTT client may acknowledge delivery?
 Read the pinned source and write the observed sequence down before choosing a queue or acknowledgement policy.
 Read `references/paho.golang-v0.23.0/paho/client.go`, especially receive callback processing, `EnableManualAcknowledgment`, and `Ack`. Write the observed order in `journal/06-concurrency.md`. Inspect this pinned source before choosing an acknowledgement policy.
 Section 4 copies payloads into a bounded channel and drops messages when full. Automatic MQTT acknowledgement can still occur before your application finishes work. This is a deliberate loss window, not reliable ingestion.
-## 2. Build a bounded dispatcher
+## 🛠️ 2. Build a bounded dispatcher
 > 💡 **Story question:** When messages arrive faster than handlers can finish, where is excess work stored and how does the application report that capacity is exhausted?
 Set a finite capacity and make admission outcomes explicit for each message class.
 Create `internal/dispatch/pool.go` and `pool_test.go`. Start with two workers and capacity 16. Expose a context-aware submit operation and explicit accepted/full/stopping outcomes. Count accepted, rejected, completed, and failed work. Keep telemetry and important work in separate bounded queues so a heartbeat burst cannot occupy every command slot.
 Choose a policy per class: disposable telemetry may be dropped with a counter; commands and sales must produce an explicit admission failure rather than silently disappear. Admission failure under automatic MQTT acknowledgement still loses application work. Record this limitation until durable admission is introduced in Section 13. Do not claim manual acknowledgement alone makes an in-memory queue durable.
-## 3. Define ownership and shutdown
+## 🛑 3. Define ownership and shutdown
 > 💡 **Story question:** If a worker is still processing when shutdown begins, which components must stay alive until the work finishes or its deadline expires?
 Define buffer ownership, stop admission, drain accepted work, then stop the MQTT producer path in a bounded order.
 Copy retained byte slices, including future correlation metadata. Protect shared maps with a mutex or give one goroutine ownership. Never hold a state mutex while publishing over the network. Use operation deadlines derived from the application lifecycle.
 On shutdown, stop admitting new work and let accepted work finish within a separate bounded drain context. Keep MQTT available during the drain if workers must publish results, then stop the transport producers. Cancel remaining work when the drain deadline expires. Never close a queue while a producer can still send; waiting for worker completion is part of shutdown. Document what happens to unfinished in-memory work.
-## 4. Integrate and stress it
+## 🔗 4. Integrate and stress it
 > 💡 **Story question:** What observations would show that overload is bounded and shutdown is controlled, rather than merely that the process eventually exits?
 Measure accepted, rejected, completed work, queue capacity, and shutdown time under a finite burst.
 Edit the backend and simulator processing loops to submit validated work. Keep transport callbacks short. Add a configurable artificial handler delay and a finite synthetic burst generator; do not create one goroutine per incoming message. Keep these switches in typed TOML + koanf configuration and preserve existing precedence.
 ```sh
 mkdir -p internal/dispatch journal
 gofmt -w internal/dispatch cmd
 go test -race ./...
 go vet ./...
 ```
@@ -81,4 +118,8 @@ Use channels in tests to hold a worker busy deterministically, fill the queue, a
 **Done when:** A burst causes observable bounded overload, normal ping/pong recovers afterward, shutdown finishes by its deadline, and the race detector passes. Explain the difference between broker acknowledgement, queue admission, and completed application work.
 ## ✅ Result — the story so far
 The receive callback now has a defined small job: copy and admit work into bounded queues, then return. Workers validate and handle admitted messages, while counters make overload visible and shutdown follows a deliberate stop-admission, drain, and disconnect order. In this lesson's in-memory design, automatic MQTT acknowledgement can happen before worker completion, so a process crash can still lose queued work; durable admission is a later step.
 Previous: [05-contracts-validation.md](05-contracts-validation.md). Next: [07-command-response.md](07-command-response.md).
@@ -1,10 +1,36 @@
-# Section 7 — Build reliable command and response behavior
+# 🔁 Section 7 — Build reliable command and response behavior
-Prerequisite: Sections 5–6 validate messages and bound concurrent processing.
+## 🎬 Start with the system story
 > 💡 **Story question:** The broker acknowledged a backend's command publication. How can the backend match a later device response to the right request—and what should it report if no response arrives?
 The MQTT acknowledgement covers delivery on one hop. The device's application response is a separate publication. The backend therefore needs application-level command identity, correlation, a pending state registered before publication, and a timeout policy that does not claim the device definitely did nothing.
 ```mermaid
 sequenceDiagram
    participant H as HTTP/backend caller
    participant G as Go command service
    participant B as Broker
    participant D as Device
    H->>G: Request command
    G->>G: Create command ID and pending waiter
    G->>B: PUBLISH command with correlation data
    B-->>G: PUBACK: broker hop accepted
    B->>D: PUBLISH command
    D->>D: Validate, deduplicate, act
    D->>B: PUBLISH correlated application result
    B->>G: PUBLISH result
    G->>G: Match identity/correlation; update state
    G-->>H: Accepted / completed / rejected / unknown
 ```
 **Before each implementation step, ask:** Which event changes command state: broker acknowledgement, device acceptance, device completion, or timeout? Keep those events separate. Follow [Lesson 0](00-system-lifecycle.md) for the full client and broker lifecycle.
 Prerequisite: Sections 5–6 validate messages and bound concurrent processing. The backend and simulator continue to read their shared disposable password from `configs/lab.toml`; select each broker username with `-username` when starting the process.
 Run all commands from `/home/ryan/Media/Code/Code/robot-market/mqtt-5-learning`. Create files at the project-relative paths shown below, not under `lessons/`. Continue the same application and pinned Paho v0.23.0 dependency. These are guided implementation exercises; attempt each step before requesting a full solution.
-## MQTT request/response is an application conversation
+## ↩️ MQTT request/response is an application conversation
 Response Topic tells the responder where a result should be published. Correlation Data lets the requester recognize which pending request it belongs to. The broker forwards these properties; it does not run the command, create a result, or enforce your timeout.
@@ -22,11 +48,13 @@ request := &paho.Publish{
        CorrelationData: append([]byte(nil), correlation...),
    },
 }
 // Send this packet only after the pending request has been registered.
 ```
 Pass `request` to the bounded publication operation after registering pending state. The simulator validates `ResponseTopic`, publishes the result to it, and copies Correlation Data unchanged into the result's properties. Correlation bytes are opaque; they need not be text. Do not trust an arbitrary supplied topic as a response destination.
-### Trace a fast response and a lost response
+### ↩️ Trace a fast response and a lost response
 1. Backend establishes its response subscription and checks SUBACK.
 2. Backend stores the expected device, command ID, correlation, and deadline.
@@ -42,7 +70,7 @@ Registering after step 3 loses the race with a fast response. Holding the comman
 | Correlation Data | Request/response mechanism | Matches a response; does not authenticate it or prevent repeated effects |
 | Command ID | Application, stable across retries | Keys the dedup record and business status |
-### Make duplicate behavior observable
+### ♻️ Make duplicate behavior observable
 Use a synthetic effect counter. Send command C twice with the same content, then C with changed content. Expect one effect, a replayed recorded result, then a conflict. Also send the same C in two separate publish calls: application duplicates need not carry MQTT DUP. The DUP flag describes protocol retransmission, so deduplicating only packets with DUP set is wrong.
@@ -55,31 +83,49 @@ Only the transport acknowledgement is established. The device may be offline, ma
 </details>
-## 1. Preserve MQTT 5 request metadata
+## 💾 1. Preserve MQTT 5 request metadata
 > 💡 **Story question:** How can a responder know where to send a reply and the requester know which command the reply belongs to?
 Use the MQTT 5 Response Topic and Correlation Data as transport metadata, while retaining a stable application command ID and checking that the response destination is authorized.
 Create `internal/commands/service.go`, `store.go`, and `service_test.go`. Edit `internal/mqtt/client.go` to carry Response Topic and a copied Correlation Data slice in received messages. Add an outbound request form that accepts these properties. Retain the existing bounded publish timeout and reason-code checks.
 Read `references/paho.golang-v0.23.0/paho/cp_publish.go` for `paho.PublishProperties`. Set `ResponseTopic` to `lab/v1/devices/device-001/responses` and `CorrelationData` to opaque bytes associated with the pending command. Establish the backend response subscription before sending. Register pending state before publishing so an immediate response cannot beat registration.
-## 2. Model evidence explicitly
+## 🧩 2. Model evidence explicitly
 > 💡 **Story question:** What should the backend show after a PUBACK but before any device response arrives?
 Represent broker delivery evidence separately from device acceptance, completion, rejection, and timeout/unknown outcome.
 Store command ID, expected device, correlation bytes, creation/deadline times, business status, and independent transport acknowledgement evidence. Start with created → pending delivery → accepted → completed or rejected. A ping simulator can send accepted followed by completed. Response timeout means unknown outcome; it does not establish that no effect happened.
 Protect concurrent access and define allowed transitions. A completed or rejected command must not regress on a duplicate accepted response. Compare command identity, topic/body device identity, and correlation before applying a response. Keep timed-out records for a bounded retention period even after removing their waiters, so status can explain the uncertainty.
-## 3. Make the simulator idempotent
+## ♻️ 3. Make the simulator idempotent
 > 💡 **Story question:** If QoS 1 or an application retry delivers the same command again, how does the device avoid repeating a non-idempotent effect?
 Record results by stable command ID and replay the prior result for duplicates instead of applying the simulated effect twice.
 Create `internal/simulator/dedup.go` and its tests. Index effects by stable command ID and retain a fingerprint of the command. For an identical duplicate, replay the recorded result without repeating the effect. Reject reuse of the ID with different content. Bound cache lifetime and size; state clearly that restarting the simulator loses this in-memory protection.
 Allow a Response Topic only when it exactly matches the authorized response topic for this simulator. Correlation is matching data, not authorization. Continue rejecting retained one-shot commands.
-## 4. Exercise timing and failures
+## 🧪 4. Exercise timing and failures
 > 💡 **Story question:** If a response is late or missing, what can the backend safely conclude, and what should happen to a later result?
 Exercise timeouts and duplicates while keeping uncertainty visible; a timeout is not proof that the device did nothing.
 Replace the backend's unconditional ticker with an explicit finite command exercise or service calls. Send 20 concurrent synthetic pings with separate IDs. Introduce delayed, duplicate, wrong-device, wrong-correlation, and out-of-order responses. Check every command receives only its own evidence. Retry a timed-out request with the same stable ID; never generate a new ID merely because publication timed out.
 ```sh
 mkdir -p internal/commands internal/simulator
 gofmt -w internal cmd
 go test -race ./...
 ```
@@ -91,4 +137,8 @@ Test responses arriving before publish returns, publish failure with uncertain o
 **Done when:** Concurrent requests match correctly, identical duplicates cause one effect, unexpected destinations are rejected, and late responses follow a documented policy. Explain why PUBACK and Correlation Data cannot prove business completion.
 ## ✅ Result — the story so far
 The backend now treats a ping as a tracked command: it records the command ID, expected device, correlation data, and deadline before publishing. The simulator checks the command and authorized response destination, applies the simulated effect once for that stable ID, and publishes a result. The backend matches that result to the pending command and updates its status. A broker PUBACK still means only that the broker acknowledged its hop; a missing response leaves the outcome unknown, not necessarily unexecuted.
 Previous: [06-concurrency-cancellation.md](06-concurrency-cancellation.md). Next: [08-http-commands.md](08-http-commands.md).
@@ -1,10 +1,41 @@
-# Section 8 — Expose device commands through HTTP
+# 🌐 Section 8 — Expose device commands through HTTP
-Prerequisite: Section 7 provides a command service and queryable in-memory states.
+## 🎬 Start with the system story
 > 💡 **Story question:** A user clicks “ping device” in an HTTP API. How can the API respond promptly while the MQTT command and device response happen asynchronously?
 The HTTP handler validates the request and asks an application service to create a command. The service records its identity and pending status, publishes through the MQTT client, then exposes later device evidence through a status endpoint. HTTP request lifetime, MQTT delivery, and the device operation each have their own acknowledgement and failure boundary.
 ```mermaid
 sequenceDiagram
    participant U as API caller
    participant H as Go HTTP handler
    participant S as Command service
    participant B as MQTT broker
    participant D as Device
    U->>H: POST /devices/{id}/commands
    H->>S: Validate and create command
    S->>S: Store command ID and pending state
    S->>B: Publish MQTT command
    B-->>S: PUBACK for broker hop
    S-->>H: Accepted with command ID
    H-->>U: HTTP 202 + command ID
    B->>D: Deliver command
    D->>B: Publish application result
    B->>S: Deliver result
    U->>H: GET /commands/{id}
    H->>S: Read current status
    S-->>H: Accepted/completed/rejected/unknown
    H-->>U: HTTP status representation
 ```
 **Before each implementation step, ask:** Is this response evidence that the HTTP request was accepted, that the broker acknowledged a publication, or that the device completed the work? Follow [Lesson 0](00-system-lifecycle.md) for the complete MQTT lifecycle.
 Prerequisite: Section 7 provides a command service and queryable in-memory states. The backend and simulator continue to read their shared disposable password from `configs/lab.toml`; select each broker username with `-username` when starting the process.
 Run all commands from `/home/ryan/Media/Code/Code/robot-market/mqtt-5-learning`. Create files at the project-relative paths shown below, not under `lessons/`. Continue the same application and pinned Paho v0.23.0 dependency. These are guided implementation exercises; attempt each step before requesting a full solution.
-## Translate evidence between HTTP and MQTT
+## 🌐 Translate evidence between HTTP and MQTT
 HTTP has one response to this request. MQTT has an asynchronous stream where the device may reply after the HTTP connection closes. Returning 202 after admission gives the caller a stable command ID; a later GET reports evidence accumulated by the command service.
@@ -25,30 +56,59 @@ No. The lesson's in-memory command and retry-key maps disappear. HTTP 202 descri
 </details>
-## 1. Add HTTP configuration and handlers
+## ⚙️ 1. Add HTTP configuration and handlers
-Create `internal/httpapi/server.go`, `handlers.go`, and `handlers_test.go`. Extend `internal/config/config.go` and `configs/lab.toml` with a loopback listener such as `127.0.0.1:18080`, request-body limit, and server timeout settings. Preserve defaults < TOML < environment < explicit CLI flags. Validate these settings before starting listeners.
+> 💡 **Story question:** Which process accepts the user's HTTP request, and how does that process reach the command service without putting MQTT logic in the handler?
 Wire the HTTP server to the application service and preserve bounded parsing, deadlines, and controlled shutdown.
 Create `internal/httpapi/server.go`, `handlers.go`, and `handlers_test.go`. Extend `internal/config/config.go` and `configs/lab.toml` with a loopback listener such as `127.0.0.1:18080`, request-body limit, and server timeout settings. Preserve defaults < TOML < environment < explicit CLI flags for supported settings; keep `mqtt.password` TOML-only. Validate these settings before starting listeners.
 Compose HTTP and MQTT in the existing `cmd/backend/main.go`. Keep the simulator separate. Section 15 introduces separate API and ingestion processes.
-## 2. Define the API contract
+## 📋 2. Define the API contract
 > 💡 **Story question:** What should the caller receive immediately when device work will finish later?
 Return an accepted command ID, then provide a status resource that reports later evidence.
 Implement `POST /devices/{id}/commands` for a JSON command such as `{"command":"ping"}` and `GET /commands/{id}` for status. Limit bodies, reject trailing JSON and unknown fields, validate device IDs, and return JSON errors consistently. Keep the listener local; this lesson does not implement production user authorization.
 Return HTTP 202 with a command ID and a Location pointing to its status after service admission. Document malformed-input 400, oversized-body 413, unknown-command/device 404 or a consistent chosen alternative, conflict 409, and admission-unavailable 503 behavior. Transport uncertainty after acceptance belongs in command status rather than silently discarding an accepted command.
-## 3. Implement HTTP retry identity
+## 🌐 3. Implement HTTP retry identity
 > 💡 **Story question:** If the server accepted a request but the HTTP response was lost, how can the caller retry without creating a second device action?
 Use an HTTP idempotency key with stable request semantics; keep it distinct from the command ID and MQTT packet identifier.
 Require an `Idempotency-Key` header for POST. Map the key and request fingerprint to the existing command ID. An identical retry returns the same ID; changed content under the same key returns 409. Serialize concurrent admission for the same key. Bound key retention and document that process restart loses this in-memory guarantee. This key, the stable device command ID, and MQTT packet IDs serve separate purposes.
 Use the HTTP context for request parsing and admission. Once accepted, execute under the application's lifecycle and command deadline, so client disconnect does not cancel the command. Inject the command service through a small interface and keep MQTT calls out of handlers.
-## 4. Run and verify
+## 🧪 4. Run and verify
-Start the simulator and backend using Section 4's separate terminal credentials. In a third terminal:
+> 💡 **Story question:** What evidence demonstrates each boundary in the round trip—from HTTP acceptance to broker delivery to device result?
 Observe the returned command ID, MQTT acknowledgement, simulator response, and status endpoint separately.
 Start the simulator and backend in separate terminals from the project root. The test password is already loaded from `configs/lab.toml`; `-username` selects the account for each process:
 ```sh
 go run ./cmd/simulator -username simulator -client-id lab-simulator-device-001
 ```
 ```sh
 go run ./cmd/backend -username backend -client-id lab-backend
 ```
 In a third terminal, send the HTTP command:
 ```sh
 curl -i -X POST http://127.0.0.1:18080/devices/device-001/commands -H 'Content-Type: application/json' -H 'Idempotency-Key: lesson8-ping-001' --data '{"command":"ping"}'
 # Replace cmd-example with the ID returned by the POST response.
 curl -i http://127.0.0.1:18080/commands/cmd-example
 ```
 `curl` is an HTTP client. `-i` prints response headers alongside the body; `-X POST` chooses the method; each `-H` sets a request header; `--data` sends the quoted JSON bytes. Content-Type tells the server how to interpret the body, while Idempotency-Key identifies this caller retry. This local exercise sends no production authorization token.
@@ -61,8 +121,11 @@ Set ReadHeaderTimeout, ReadTimeout, WriteTimeout, and IdleTimeout deliberately.
 ```sh
 mkdir -p internal/httpapi
 gofmt -w internal cmd
 go test -race ./...
 go vet ./...
 ```
@@ -74,4 +137,8 @@ Use `httptest` with a fake service for decoding, size limits, idempotency, unava
 **Done when:** HTTP creates a ping, status reflects device evidence, retries reuse its ID, and closing the caller connection does not erase accepted work.
 ## ✅ Result — the story so far
 An API caller can now ask the backend to ping a device with HTTP. The handler validates the request and admits it through the command service; the caller receives HTTP 202 and a command ID without waiting for the device. The backend publishes the MQTT command, the simulator returns its application result, and a later GET reports the command's current evidence. Retrying the same POST with the same idempotency key returns the original command instead of starting a second effect. HTTP acceptance, MQTT acknowledgement, and device completion remain separate events.
 Previous: [07-command-response.md](07-command-response.md). Next: [09-sessions-recovery.md](09-sessions-recovery.md).
@@ -1,5 +1,7 @@
 # MQTT in Go: lessons and concept coverage
 Begin with [Lesson 0 — The broker, Go backend, and device lifecycle](00-system-lifecycle.md) for the complete system story. Lessons 1–8 open with a lifecycle question and diagram where useful, then provide a guiding question before each implementation step.
 Broker deployment and administration are part of the learning goals. Read [Broker operations](broker-operations.md) alongside Lesson 1; it explains lifecycle, credential maintenance, backup/restore, upgrades, and remote-deployment preparation. Commands are explained in the lesson that introduces them.
 Follow these lessons in order from the learning project root. Read the concept and packet flow first, predict the experiment, then implement and compare the result. Go and basic Docker syntax are assumed; explanations focus on MQTT behavior and operational consequences. Full early examples remain available as reference solutions.
@@ -69,6 +69,8 @@ Always show commands for me to run as normal shell commands without an rtk prefi
 3. Start the section or exercise I request. Let me implement it; show a full solution or write application code only when I ask.
 4. Teach one lesson or exercise at a time.
 5. For each lesson:
   - Begin with the end-to-end system story: which processes are involved, who owns the relevant state, what message or connection moves between them, and why the implementation step is needed.
   - Open each implementation step with one question that helps me predict the behavior before I read or write code. Explain the story and wait for my response when we are working interactively.
   - Explain the concept in plain language.
   - Connect it to a vending-machine example.
   - Explain the relevant MQTT packets and properties.