updated and tested till end of lesson 4

2026-10-11 15:00:58 +03:30
parent d73090946b
commit 2d639d78eb
18 changed files with 1120 additions and 187 deletions
@@ -1,11 +1,57 @@
-# Section 4 — Publish, subscribe, and build a Go device simulator
+# 📡 Section 4 — Publish, subscribe, and build a Go device simulator

 Prerequisites: Sections 1–3 complete. Keep Section 3's configuration package and TOML file. This lesson supplies complete files for a reusable MQTT transport, backend, and simulator. Implement each file in stages and explain it before running both programs.

 **Version baseline:** Paho `v0.23.0`, verified from upstream release tags on 2026-10-07. It requires Go 1.24 or newer; Go 1.25 is compatible. The matching local reference is `references/paho.golang-v0.23.0/` (commit `1aa03969c5932464359b3e4ed752d9ac61abccc9`).


-## 1. Map the conversation
+## 🎬 Start with the system story
+
+> 💡 **Story question:** The backend and device are separate MQTT clients. How do they exchange a command without knowing each other's address, and how can the backend tell whether the device actually handled it?
+
+The broker is the meeting point. The backend connects and subscribes to telemetry and responses; the simulator connects with its own identity and subscribes to commands. Each publication goes to the broker, which routes it to matching subscriptions. A device response is a new application message, separate from the broker's acknowledgement of the backend's publish.
+
+```mermaid
+sequenceDiagram
+    participant G as Go backend
+    participant B as Mosquitto broker
+    participant D as Go device simulator
+    G->>B: CONNECT
+    B-->>G: CONNACK accepted
+    D->>B: CONNECT
+    B-->>D: CONNACK accepted
+    G->>B: SUBSCRIBE telemetry, responses
+    B-->>G: SUBACK
+    D->>B: SUBSCRIBE commands
+    B-->>D: SUBACK
+    G->>B: PUBLISH ping, QoS 1
+    par Backend-to-broker acknowledgement
+        B-->>G: PUBACK
+    and Broker-to-device delivery
+        B->>D: PUBLISH ping, QoS 1
+        D->>D: Paho callback queues message
+        D-->>B: PUBACK after callback returns
+    end
+    D->>D: Application handles ping
+    D->>B: PUBLISH pong response, QoS 1
+    par Device-to-broker acknowledgement
+        B-->>D: PUBACK
+    and Broker-to-backend delivery
+        B->>G: PUBLISH pong response, QoS 1
+        G->>G: Paho callback queues message
+        G-->>B: PUBACK after callback returns
+    end
+```
+
+The broker, backend, and device have distinct setup. Mosquitto owns its listener, password source, and ACL. Each Go process loads its own broker URL, client ID, and username. The backend and simulator use different usernames for their broker permissions; both use the disposable test password from `configs/lab.toml`. The code below first builds the shared transport that connects, subscribes, receives, and publishes; then the two application loops use it.
+
+Before each implementation part, use the question there to predict which process or protocol layer is responsible. Follow [Lesson 0](00-system-lifecycle.md) for the full lifecycle map.
+
+## 🗺️ 1. Map the conversation
+
+> 💡 **Story question:** Which client publishes each kind of message, which client needs it, and what does the broker use to route it?
+
+Map the telemetry, command, and response paths first. This identifies which process must subscribe before we write transport code.

 | Topic | Publisher | Subscriber | Example |
 | --- | --- | --- | --- |
@@ -15,7 +61,7 @@ Prerequisites: Sections 1–3 complete. Keep Section 3's configuration package a

 A broker subscription and a local Go callback are different things. SUBSCRIBE asks the broker which messages to forward. `OnPublishReceived` tells your local client how to process forwarded PUBLISH packets. You need both. This lab's topic convention is deliberately separate from the production contracts.

-### Trace two independent delivery hops
+### 🔄 Trace two independent delivery hops

 ```mermaid
 sequenceDiagram
@@ -25,16 +71,25 @@ sequenceDiagram
    S->>M: SUBSCRIBE commands, maximum QoS 1
    M-->>S: SUBACK with grant or rejection
    B->>M: PUBLISH command, QoS 1
-    M-->>B: PUBACK for backend-to-broker hop
-    M->>S: PUBLISH command, granted delivery QoS
-    S-->>M: PUBACK for broker-to-simulator hop
-    S->>M: PUBLISH application result
-    M->>B: PUBLISH application result
+    par Backend publish exchange
+        M-->>B: PUBACK
+    and Broker forwards to simulator
+        M->>S: PUBLISH command, granted delivery QoS
+        S-->>M: PUBACK for delivery hop
+    end
+    S->>S: Application handles command
+    S->>M: PUBLISH application result, QoS 1
+    par Simulator publish exchange
+        M-->>S: PUBACK
+    and Broker forwards result to backend
+        M->>B: PUBLISH application result
+        B-->>M: PUBACK for delivery hop
+    end
 ```

-This diagram groups exchanges by purpose; it does not promise the backend sees PUBACK before the simulator's response. Each side has its own connection and packet identifiers. The broker routes by topic, not by JSON fields. A result is another publication on another topic; MQTT does not create a response automatically.
+The `par` blocks show that a publisher-side PUBACK and broker-to-subscriber delivery are independent work on separate client connections; their observed order can vary. Each QoS 1 PUBACK confirms only its own hop. The application result is a separate publication—the broker does not create it automatically, and neither PUBACK proves that business work completed.

-### Choose QoS by the guarantee you need
+### 📨 Choose QoS by the guarantee you need

 | QoS | Successful exchange on one hop | What the Go application must still handle |
 | --- | --- | --- |
@@ -46,13 +101,17 @@ The subscriber requests a **maximum** delivery QoS. A QoS 1 publication cannot b

 For a QoS 2 experiment, change both publication and requested subscription QoS to 2, accept SUBACK grants 0–2, and inspect the four-packet exchange. The original helper intentionally accepts only grants up to its requested QoS 1. Do not change only the publish field and call the resulting lower-QoS subscriber path an end-to-end QoS 2 test.

-### Topic filters are routing rules
+### 🏷️ Topic filters are routing rules

 `lab/v1/devices/+/telemetry` matches one device segment; `lab/v1/devices/#` matches that subtree. `+` occupies an entire level and `#` is the final level. Topics are case-sensitive, so `device-001` and `Device-001` differ. Wildcards belong in subscriptions, never publish topic names. A root `#` filter does not include `$SYS/...`; broker system topics require an explicit matching filter and permission.

 A SUBACK proves the subscription request received a result. Topic ACLs still govern delivery: an accepted broad filter does not prove access to every matching topic. Check an authorized observer, reason codes, and broker logs when testing denial.

-## 2. Create the transport helper
+## 🛠️ 2. Create the transport helper
+
+> 💡 **Story question:** What should be shared between the backend and simulator, and what must remain specific to each running client?
+
+The transport helper can share connection lifecycle, subscription handling, message copying, and publishing. Each caller still supplies its configured username, Client ID, topic list, and receive queue. Keeping those identities separate lets the broker apply different permissions and prevents Client ID takeover.

 ```sh
 mkdir -p internal/mqtt cmd/simulator
@@ -81,11 +140,15 @@ type Message struct {
 	Retain  bool
 }

+// Start creates the MQTT connection, subscribes to the requested topics, and
+// waits until the initial connection and subscriptions are ready.
 func Start(ctx context.Context, cfg config.Config, topics []string, inbox chan<- Message) (*autopaho.ConnectionManager, error) {
+	// Parse the configured broker URL before creating the reconnecting client.
 	broker, err := url.Parse(cfg.MQTT.URL)
 	if err != nil {
 		return nil, err
 	}
+	// The callback reports its first SUBACK so startup can fail early if needed.
 	initialSubscription := make(chan error, 1)
 	manager, err := autopaho.NewConnection(ctx, autopaho.ClientConfig{
 		ServerUrls:                    []*url.URL{broker},
@@ -94,6 +157,7 @@ func Start(ctx context.Context, cfg config.Config, topics []string, inbox chan<-
 		SessionExpiryInterval:         0,
 		ConnectUsername:               cfg.MQTT.Username,
 		ConnectPassword:               []byte(cfg.MQTT.Password),
+		// Recreate subscriptions after every successful connection or reconnect.
 		OnConnectionUp: func(cm *autopaho.ConnectionManager, _ *paho.Connack) {
 			// This runs again on reconnect; reestablish broker subscriptions.
 			subCtx, cancel := context.WithTimeout(ctx, 5*time.Second)
@@ -102,6 +166,7 @@ func Start(ctx context.Context, cfg config.Config, topics []string, inbox chan<-
 			for _, topic := range topics {
 				subscriptions = append(subscriptions, paho.SubscribeOptions{Topic: topic, QoS: 1, RetainAsPublished: true})
 			}
+			// Ask for QoS 1 and preserve RETAIN so consumers can identify stored data.
 			ack, subErr := cm.Subscribe(subCtx, &paho.Subscribe{Subscriptions: subscriptions})
 			if subErr == nil {
 				if ack == nil || len(ack.Reasons) != len(topics) {
@@ -127,6 +192,7 @@ func Start(ctx context.Context, cfg config.Config, topics []string, inbox chan<-
 		},
 		OnConnectionDown: func() bool { log.Print("disconnected; reconnecting"); return true },
 		OnConnectError:   func(err error) { log.Printf("connect error: %v", err) },
+		// Keep the packet callback short: copy the message and enqueue it for work.
 		ClientConfig: paho.ClientConfig{
 			ClientID:           cfg.MQTT.ClientID,
 			OnClientError:      func(err error) { log.Printf("client error: %v", err) },
@@ -135,6 +201,8 @@ func Start(ctx context.Context, cfg config.Config, topics []string, inbox chan<-
 				func(received paho.PublishReceived) (bool, error) {
 					packet := received.Packet
 					message := Message{Topic: packet.Topic, Payload: append([]byte(nil), packet.Payload...), Retain: packet.Retain}
+
+					// Never block the network callback when the bounded inbox is full.
 					select {
 					case inbox <- message:
 					default:
@@ -149,6 +217,7 @@ func Start(ctx context.Context, cfg config.Config, topics []string, inbox chan<-
 	if err != nil {
 		return nil, fmt.Errorf("start AutoPaho: %w", err)
 	}
+	// Bound initial connection and subscription setup by the configured timeout.
 	startup, cancel := context.WithTimeout(ctx, cfg.ConnectTimeout())
 	defer cancel()
 	if err := manager.AwaitConnection(startup); err != nil {
@@ -169,6 +238,7 @@ func Start(ctx context.Context, cfg config.Config, topics []string, inbox chan<-
 }

 func Publish(ctx context.Context, manager *autopaho.ConnectionManager, topic string, payload []byte) error {
+	// Publishing waits only for this bounded operation, not for device completion.
 	operation, cancel := context.WithTimeout(ctx, 3*time.Second)
 	defer cancel()
 	ack, err := manager.Publish(operation, &paho.Publish{
@@ -189,6 +259,7 @@ func Publish(ctx context.Context, manager *autopaho.ConnectionManager, topic str
 }

 func Stop(manager *autopaho.ConnectionManager) {
+	// Give the MQTT disconnect handshake a short, independent shutdown window.
 	shutdown, cancel := context.WithTimeout(context.Background(), 5*time.Second)
 	defer cancel()
 	if err := manager.Disconnect(shutdown); err != nil {
@@ -198,7 +269,11 @@ func Stop(manager *autopaho.ConnectionManager) {
 ```


-## 3. Map the helper to MQTT behavior
+## 🔗 3. Map the helper to MQTT behavior
+
+> 💡 **Story question:** When `Start` says subscriptions are ready, which broker exchange has happened, and what is the local callback's separate job?
+
+The broker has received `SUBSCRIBE` and returned an acceptable `SUBACK`; the callback is configured in the Go process to receive broker-forwarded PUBLISH packets. Read the implementation against these two sides of the receive path.

 | Code decision | Why it matters |
 | --- | --- |
@@ -220,7 +295,40 @@ For QoS 1, `Publish` returns a response describing the immediate broker acknowle

 The initial-subscription channel allows startup to wait for SUBACK as well as CONNACK. The callback subscribes again on reconnect. This helper checks initial readiness only; reconnect subscription failures are logged, not represented as continuous health state. Add that state later rather than treating initial readiness as permanent.

-## 4. Replace the backend main program
+## 💻 4. Replace the backend main program
+
+> 💡 **Story question:** After configuration, connection, and subscriptions are ready, how does a backend timer become an MQTT command?
+
+### 🦟 Trace one ping from Go value to broker
+
+On each timer tick, the backend first builds an ordinary Go value. `json.Marshal` turns that value into the byte sequence sent as the MQTT payload:
+
+```go
+struct {
+    ID      string `json:"id"`
+    Command string `json:"command"`
+}{ID: "ping-123-1", Command: "ping"}
+```
+
+The JSON field tags set the wire names, so the payload bytes represent:
+
+```json
+{"id":"ping-123-1","command":"ping"}
+```
+
+The backend chooses the concrete topic `lab/v1/devices/device-001/commands` using its configured device ID. It passes that topic and the JSON bytes to `mqtt.Publish`; that helper builds a Paho `paho.Publish` with **QoS 1**, **Retain false**, and Content Type `application/json`, then calls AutoPaho's `manager.Publish`.
+
+`manager.Publish` waits for the QoS 1 publish exchange or an error/timeout. When it receives a successful `PUBACK`, the helper logs the reason code and returns nil. That confirms the backend-to-broker QoS exchange. A success reason of `0x10` means the broker had no matching subscribers; it is still a successful MQTT acknowledgement, so it does not prove the simulator received anything. A timeout can also leave the outcome uncertain.
+
+The broker independently forwards a matching PUBLISH to the simulator's `.../commands` subscription. The simulator's Paho callback copies it into an inbox; the simulator's main loop decodes the JSON, checks that it is a non-retained `ping` with an ID, and publishes a `pong` carrying that same ID to `.../responses`. That new response is application-level evidence the simulator handled the ping. The backend is subscribed to the response topic and currently logs the received bytes; matching responses to pending commands is added in Lesson 7.
+
+```text
+Go struct → JSON payload bytes → QoS 1 PUBLISH → broker PUBACK
+                                            └→ simulator receives and handles ping
+                                               → separate pong PUBLISH → backend receives
+```
+
+The two arrows after the backend's PUBLISH are separate outcomes: PUBACK may arrive before or after the simulator's response. Retain is false because a ping is a one-time action; storing it as retained state could replay the command to a later subscriber.

 Create `cmd/backend/main.go`:

@@ -242,6 +350,7 @@ import (
 )

 func main() {
+	// Convert a returned application error into a visible nonzero process exit.
 	if err := run(); err != nil {
 		log.Print(err)
 		os.Exit(1)
@@ -249,12 +358,14 @@ func main() {
 }

 func run() error {
+	// Load settings and bind cancellation to Ctrl+C or the container stop signal.
 	cfg, err := config.Load(os.Args[1:])
 	if err != nil {
 		return err
 	}
 	ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
 	defer stop()
+	// Build this device's topic namespace and bounded receive queue.
 	base := "lab/v1/devices/" + cfg.MQTT.DeviceID
 	inbox := make(chan mqtt.Message, 16)
 	manager, err := mqtt.Start(ctx, cfg, []string{base + "/telemetry", base + "/responses"}, inbox)
@@ -262,11 +373,13 @@ func run() error {
 		return err
 	}
 	defer mqtt.Stop(manager)
+	// Publish synthetic ping commands periodically while processing received data.
 	ticker := time.NewTicker(10 * time.Second)
 	defer ticker.Stop()
 	runID := time.Now().UnixNano()
 	sequence := 0
 	for {
+		// The select keeps shutdown, inbound messages, and periodic commands responsive.
 		select {
 		case <-ctx.Done():
 			return nil
@@ -293,7 +406,11 @@ func run() error {

 The ticker sends one ping every ten seconds. It starts after the backend's own subscriptions are ready. A timestamp and sequence make IDs distinct enough for this demonstration, not a production idempotency system. The backend logs the response but does not yet implement pending requests or validate completion; Section 7 adds those semantics.

-## 5. Create the device simulator
+## 🛠️ 5. Create the device simulator
+
+> 💡 **Story question:** How does a received MQTT packet become a device action, and how does the simulator report its result without blocking Paho's network callback?
+
+The callback copies the packet into a bounded inbox. The simulator's main loop reads it, validates the harmless ping, and publishes a pong. This draws a deliberate boundary between packet reception and application behavior; later lessons examine the loss window and acknowledgement timing.

 Create `cmd/simulator/main.go`:

@@ -314,6 +431,7 @@ import (
 )

 func main() {
+	// Convert a returned application error into a visible nonzero process exit.
 	if err := run(); err != nil {
 		log.Print(err)
 		os.Exit(1)
@@ -321,12 +439,14 @@ func main() {
 }

 func run() error {
+	// Load settings and bind cancellation to Ctrl+C or the container stop signal.
 	cfg, err := config.Load(os.Args[1:])
 	if err != nil {
 		return err
 	}
 	ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
 	defer stop()
+	// Build this device's topic namespace and bounded receive queue.
 	base := "lab/v1/devices/" + cfg.MQTT.DeviceID
 	inbox := make(chan mqtt.Message, 16)
 	manager, err := mqtt.Start(ctx, cfg, []string{base + "/commands"}, inbox)
@@ -334,9 +454,11 @@ func run() error {
 		return err
 	}
 	defer mqtt.Stop(manager)
+	// Heartbeats are periodic; command handling remains active between ticks.
 	ticker := time.NewTicker(3 * time.Second)
 	defer ticker.Stop()
 	for {
+		// Handle cancellation, telemetry publication, and inbound commands together.
 		select {
 		case <-ctx.Done():
 			return nil
@@ -352,6 +474,7 @@ func run() error {
 				log.Printf("heartbeat: %v", err)
 			}
 		case message := <-inbox:
+			// Decode into a local command value only after rejecting retained messages.
 			var command struct {
 				ID      string `json:"id"`
 				Command string `json:"command"`
@@ -364,6 +487,7 @@ func run() error {
 				log.Print("ignoring malformed or unsupported command")
 				continue
 			}
+			// Echo the command ID so the backend can associate this pong with its ping.
 			response, err := json.Marshal(struct {
 				ID       string `json:"id"`
 				DeviceID string `json:"device_id"`
@@ -385,29 +509,27 @@ func run() error {

 This simulator sends a heartbeat every three seconds, receives ping commands, and publishes a pong carrying the same ID. It rejects packets marked retained to avoid treating stored state as a fresh one-shot action. The subscription sets `RetainAsPublished: true`: with the default false, live forwarding can clear RETAIN and defeat a guard that checks only `message.Retain`. Keep command publishers nonretaining too; this guard does not remove a retained value already stored at the broker. Full contract validation and duplicate handling arrive in later sections; this only demonstrates the transport lifecycle.

-## 6. Run two separate identities
+## ▶️ 6. Run two separate identities
+
+> 💡 **Story question:** Why start two processes with different usernames and Client IDs, and what output proves each stage of the path?
+
+The simulator and backend represent independent clients. Their connection logs, subscription results, broker publish acknowledgement, and pong are separate evidence. Start the simulator first so its command subscription exists before the backend publishes.

 Terminal A, simulator first:

 ```sh
-export TEST_MQTT5_MQTT_USERNAME=simulator
-read -rs 'TEST_MQTT5_MQTT_PASSWORD?Simulator lab password: '
-export TEST_MQTT5_MQTT_PASSWORD
-go run ./cmd/simulator -client-id lab-simulator-device-001
+go run ./cmd/simulator -username simulator -client-id lab-simulator-device-001
 ```

 Terminal B, backend:

 ```sh
-export TEST_MQTT5_MQTT_USERNAME=backend
-read -rs 'TEST_MQTT5_MQTT_PASSWORD?Backend lab password: '
-export TEST_MQTT5_MQTT_PASSWORD
-go run ./cmd/backend -client-id lab-backend
+go run ./cmd/backend -username backend -client-id lab-backend
 ```

-Open two independent terminal sessions and change both to the project root. Their environments are separate, so the simulator's exported credentials do not overwrite the backend's. The zsh `read -rs` prompt hides the password; `export` passes it to the child process. `go run` runs each executable in the foreground; `-client-id` is a program flag overriding the TOML default, not a Docker setting. Stop each program with Ctrl+C and unset its password when finished.
+Open two independent terminal sessions and change both to the project root. The shared lab password comes from `configs/lab.toml`; `-username` selects the broker account for each process, and `-client-id` overrides the TOML client ID. Stop each foreground program with Ctrl+C when finished.

-Use the same configured `device-001` in both. Distinct username/password pairs authorize different topic directions. Distinct client IDs prevent connection takeover. A username, client ID, and payload device ID serve different purposes; the ACL is what binds this simulator credential to its permitted device namespace.
+Use the same configured `device-001` in both. Distinct usernames authorize different topic directions; both local accounts use the test password in `configs/lab.toml`. Distinct client IDs prevent connection takeover. A username, client ID, and payload device ID serve different purposes; the ACL is what binds this simulator credential to its permitted device namespace.

 Expected backend output, abbreviated (timestamps and IDs vary):

@@ -420,7 +542,11 @@ received topic=lab/v1/devices/device-001/responses ... payload={"id":"ping-...",

 Log order is not guaranteed: the device response may be observed around the publication acknowledgement. Separate connections and goroutines progress independently. If the simulator is not subscribed yet, a ping may have no recipient; this is expected with these zero-expiry early-lab sessions.

-## 7. Experiments to learn the API
+## 🧪 7. Experiments to learn the API
+
+> 💡 **Story question:** Which part of the system changes in each experiment—the broker's routing/state, the client's delivery setting, or the application's handling policy?
+
+Predict the effect at that boundary before changing the code, then use packet or application evidence to explain what happened.

 1. Replace the backend's exact telemetry subscription with `lab/v1/devices/+/telemetry`. Explain the topic filter versus a concrete publish topic. Never publish to a topic containing `+` or `#`.
 2. Add `Retain: true` only to synthetic telemetry; restart the backend and observe a retained value. Restore `false` afterward. Do not retain one-shot commands.
@@ -444,7 +570,7 @@ The new publication does not ask the broker to change retained storage. Delete t

 </details>

-## Cautions: limits of this early example
+## ⚠️ Cautions: limits of this early example

 - The inbox is deliberately bounded and lossy. It makes overload visible without blocking receive callbacks. It is unsuitable for durable sales reports or real commands; later sections replace this policy with persistence and deliberate acknowledgement/backpressure rules.
 - A publish call can block this single application loop for up to three seconds; this is acceptable for a tiny demonstration. Later sections add workers and separate business processing.
@@ -454,7 +580,7 @@ The new publication does not ask the broker to change retained storage. Delete t
 - Raw payload logs are allowed here only for small synthetic lab messages; production logs need redaction and size limits.
 - Never close the inbox while Paho may still send to it. These programs let the context and manager stop the producer; the channel need not be closed for process exit.

-## Verification and completion
+## ✅ Verification and completion

 ```sh
 gofmt -w cmd internal
@@ -471,11 +597,15 @@ Packages without tests do not demonstrate runtime correctness. Also run both pro

 **Completion:** Explain the complete path from backend Go struct → JSON bytes → MQTT PUBLISH → simulator callback → application loop → response PUBLISH → backend callback. Distinguish broker acknowledgement from a device response, and identify the lossy queue and offline-session limitations yourself.

-## Sources
+## 🔗 Sources

 - [Paho client API v0.23.0](https://pkg.go.dev/github.com/eclipse/paho.golang@v0.23.0/paho)
 - [Tagged AutoPaho source](https://github.com/eclipse-paho/paho.golang/blob/v0.23.0/autopaho/auto.go)
 - [Tagged receive callback source](https://github.com/eclipse-paho/paho.golang/blob/v0.23.0/paho/client.go)
 - [MQTT 5 specification](https://docs.oasis-open.org/mqtt/mqtt/v5.0/os/mqtt-v5.0-os.html)

+## ✅ Result — the story so far
+
+The backend now subscribes to device telemetry and responses, while the simulator subscribes to that device's commands. The simulator publishes heartbeats for the backend; when the backend sends a QoS 1 ping, the broker forwards it to the simulator, which handles it and publishes a pong carrying the same command ID. The PUBACKs confirm their individual MQTT hops; the pong is the separate application evidence that the simulator handled the ping.
+
 Next: [Section 5 — Contracts and validation](05-contracts-validation.md).