Robot-Market/mqtt-5-learning
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Refactor MQTT course and bound backend message processing

Browse Source 
Move lesson source into the teaching structure, align the course path with the learner's Go and DevOps experience, and add the interactive lesson guides. Add advanced DevOps lifecycle study as a separate section.

Bound MQTT message work and drain accepted jobs during shutdown; align the backend implementation with the revised course through Section 6.
This commit is contained in:
Ryan
2026-10-22 10:19:56 +03:30
parent f9ab91ed60
commit fb30e8e884
70 changed files with 7492 additions and 312 deletions
Download Patch File Download Diff File Expand all files Collapse all files
teaching/lessons/0005-contract-validation.html
+670
View File
@@ -0,0 +1,670 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Lesson 5 — Validate the Message Before Acting</title>
<link rel="stylesheet" href="../assets/lesson.css">
<script src="../assets/quiz.js" defer></script>
</head>
<body>
<main class="page">
<p class="eyebrow">Go MQTT Vending Backend · Lesson 5 · 7 minutes</p>
<h1>Validate the message before acting</h1>
<p class="dek">A broker-routed message still contains untrusted bytes. Build the boundary that stops a wrong or malformed vending report before it can change backend state.</p>
<div class="meta"><span class="pill">🎯 One skill: validate route and body together</span><span class="pill">🧠 Predict, then check</span><span class="pill">🧪 No broker required</span></div>
<section class="card callout">
<p class="question">❓ Start with the story</p>
<p>A publication arrives on <code>lab/v1/devices/device-001/sales</code>. Its valid JSON body claims <code>device_id: device-002</code>. Should the backend accept it?</p>
<p><strong>Say what could go wrong before revealing the model:</strong> the route and the body disagree about which machine owns this report.</p>
</section>
<h2>🗺️ First, see the full boundary</h2>
<div class="flow" aria-label="message validation sequence">
<div class="step">Broker delivers<br><strong>topic + bytes</strong></div><span class="arrow">→</span>
<div class="step">Parse route<br><strong>device + kind</strong></div><span class="arrow">→</span>
<div class="step">Decode contract<br><strong>shape + values</strong></div><span class="arrow">→</span>
<div class="step">Only then<br><strong>application action</strong></div>
</div>
<p>The topic tells the backend where the broker routed the message. The payload claims what the message means. The backend must compare them; neither a matching topic nor valid JSON authenticates the sender. Mosquitto credentials and ACLs are a separate access-control boundary.</p>
<h2>🔎 Read the code by responsibility</h2>
<div class="two-col">
<article class="card">
<h3>📄 What the files do</h3>
<p><code>internal/contracts/topic.go</code> defines the topic hierarchy, publish-name/filter types, builders, and parser. <code>message.go</code> defines payload structures and their checks. <code>decode.go</code> limits and decodes JSON, then invokes the correct contract validator.</p>
</article>
<article class="card">
<h3>📥 `Decode` — short summary</h3>
<p>Turns bytes into one typed message only after the size, JSON shape, topic identity, schema, timestamp, and message-specific rules pass.</p>
</article>
</div>
<section class="card">
<h3>🧭 Function overview: follow the boundary in order</h3>
<p><strong><code>ParseTopic</code> summary:</strong> checks the shared <code>topicPrefix</code>, then validates the remaining device ID and message kind.</p>
<p><strong><code>Decode[T]</code> summary:</strong> checks payload size and JSON shape, decodes one typed value, then calls that value's <code>Validate</code> method.</p>
<p><strong><code>Validate</code> summary:</strong> confirms that the message type matches the topic kind, shared envelope fields are valid, topic and body device IDs agree, and message-specific values are allowed.</p>
<h4>🔎 Details: each layer narrows what the next layer must trust</h4>
<ol>
<li><code>ParseTopic</code> checks the configured prefix first, then splits and validates only the two remaining levels. Changing the shared prefix does not require rewriting the parser's level checks. It rejects malformed routes before selecting a payload type and does not prove which client sent the packet.</li>
<li><code>Decode</code> rejects oversized bytes before allocating/parsing JSON, disallows unknown object fields, and rejects a second trailing JSON value. Only then does it construct <code>T</code>.</li>
<li><code>validateEnvelope</code>, called by each contract's <code>Validate</code>, checks schema version, safe IDs, topic/body identity agreement, and timestamp bounds.</li>
<li>Each concrete <code>Validate</code> method adds its own checks: telemetry temperature, supported command, response status, or sales-line values. A structurally valid message can still fail these domain rules.</li>
</ol>
<p>The backend entrypoint must stop on any returned error; only the typed value returned by successful <code>Decode</code> is eligible for application logic. Authentication and authorization still come from the broker/client configuration, not these payload checks.</p>
</section>
<article class="card">
<h3>🧩 Details: route first, decode second</h3>
<pre><code>topic, err := contracts.ParseTopic(message.Topic)
if err != nil {
return err // malformed route: no body handler runs
}
report, err := contracts.Decode[contracts.SalesReport](
message.Payload, topic, time.Now().UTC(),
)
if err != nil {
return err // bad shape, identity, schema, time, or values
}
log.Printf("valid sales report device=%s report=%s", report.DeviceID, report.ReportID)</code></pre>
<p><strong>Why this order?</strong> The parsed topic chooses the expected contract type. `Decode` rejects oversized input before parsing JSON, rejects unknown fields and extra JSON values, and calls `Validate`. Only a valid typed value reaches application handling. This section logs the validated report; Section 13 later persists sales.</p>
<p class="small">This focused fragment shows the order for the validation-stage handler. The current <code>cmd/backend/main.go</code> still logs raw payloads; the complete target entrypoints and contract files are included in the full-code section below.</p>
</article>
<h2>🧪 Check your model</h2>
<form class="card quiz" data-quiz data-correct="b" data-explanation="Right. The topic supplies routing scope, and the backend still validates the body against that route before using it." data-retry="Separate the broker's routing choice from what the JSON body claims.">
<fieldset class="question-block">
<legend><strong>1. What do topic and payload prove together?</strong></legend>
<label class="option"><input type="radio" name="q1" value="a"> Payload ID is only an untrusted claim</label>
<label class="option"><input type="radio" name="q1" value="b"> Topic gives route; validate the body too</label>
<label class="option"><input type="radio" name="q1" value="c"> Valid JSON proves no sender device identity</label>
</fieldset>
<button type="submit">Check answer</button><p class="feedback" data-feedback aria-live="polite"></p>
</form>
<form class="card quiz" data-quiz data-correct="c" data-explanation="Correct. The app limit checks payload bytes before JSON decoding; it is not a cap on the complete MQTT packet." data-retry="Which bytes does contracts.Decode receive? What packet bytes were already handled by Paho?">
<fieldset class="question-block">
<legend><strong>2. What does the 16 KiB application limit bound?</strong></legend>
<label class="option"><input type="radio" name="q2" value="a"> Payload limit caps the full MQTT packet</label>
<label class="option"><input type="radio" name="q2" value="b"> The broker rejects all oversized packet fields</label>
<label class="option"><input type="radio" name="q2" value="c"> Application cap checks payload bytes before decoding</label>
</fieldset>
<button type="submit">Check answer</button><p class="feedback" data-feedback aria-live="polite"></p>
</form>
<form class="card quiz" data-quiz data-correct="c" data-explanation="Yes. MQTT transport acknowledgement and application validation are different events; a returned application error does not itself ask the broker to retry." data-retry="Does an application error change a packet exchange that the MQTT client already acknowledged automatically?">
<fieldset class="question-block">
<legend><strong>3. What can an invalid message still receive?</strong></legend>
<label class="option"><input type="radio" name="q3" value="a"> Returning handler errors automatically requests broker retry</label>
<label class="option"><input type="radio" name="q3" value="b"> Validation failure alone makes MQTT PUBACK impossible</label>
<label class="option"><input type="radio" name="q3" value="c"> MQTT delivery and application validation are separate</label>
</fieldset>
<button type="submit">Check answer</button><p class="feedback" data-feedback aria-live="polite"></p>
</form>
<h2>🛠️ Your tiny practice</h2>
<div class="card">
<p>Predict which check rejects each input before looking at the implementation:</p>
<ol>
<li>Correct sales topic, body has an unknown field.</li>
<li>Correct topic, valid JSON, body names a different device.</li>
<li>Correct topic and identity, but one sale line has quantity zero.</li>
</ol>
<p>Write the validation stage for each: <em>topic parse</em>, <em>JSON/schema</em>, <em>shared identity</em>, or <em>message-specific value</em>. Then compare with <a href="../course-source/05-contracts-validation.md">the complete Section 5 contract code ↗</a>.</p>
</div>
<details class="card"><summary><strong>📦 Section 5 complete implementation files</strong></summary><p>Complete source files for this lesson are included here. The code blocks earlier in the lesson focus on the behavior being taught; use these full files when implementing it.</p><article class="card"><h3>📄 internal/contracts/topic.go</h3><pre><code>package contracts
import (
"fmt"
"strings"
)
// TopicName is a concrete MQTT topic used by PUBLISH. It never contains MQTT
// subscription wildcards.
type TopicName string
// TopicFilter is an MQTT subscription pattern. This lab deliberately supports
// one device-level wildcard rather than accepting arbitrary filters.
type TopicFilter string
// Kind identifies the message flow in the final topic level.
type Kind string
const (
KindTelemetry Kind = "telemetry"
KindCommands Kind = "commands"
KindResponses Kind = "responses"
KindSales Kind = "sales"
)
const topicPrefix = "lab/v1/devices/"
// Topic is the parsed device identity and flow from a concrete topic name.
type Topic struct {
DeviceID string
Kind Kind
}
func (t TopicName) String() string { return string(t) }
func (f TopicFilter) String() string { return string(f) }
// DeviceTopic builds the concrete topic name for one device and supported flow.
func DeviceTopic(deviceID string, kind Kind) (TopicName, error) {
if !validID(deviceID) {
return "", fmt.Errorf("invalid device ID %q", deviceID)
}
if !supportedKind(kind) {
return "", fmt.Errorf("unsupported topic kind %q", kind)
}
return TopicName(topicPrefix + deviceID + "/" + string(kind)), nil
}
// DeviceFilter builds a subscription filter for one flow across lab devices.
func DeviceFilter(kind Kind) (TopicFilter, error) {
if !supportedKind(kind) {
return "", fmt.Errorf("unsupported topic kind %q", kind)
}
return TopicFilter(topicPrefix + "+/" + string(kind)), nil
}
// DeviceSubscription builds an exact-topic filter for one device and flow.
func DeviceSubscription(deviceID string, kind Kind) (TopicFilter, error) {
topic, err := DeviceTopic(deviceID, kind)
if err != nil {
return "", err
}
return TopicFilter(topic.String()), nil
}
// ParseTopic turns broker routing metadata into values handlers can compare.
func ParseTopic(value string) (Topic, error) {
if !strings.HasPrefix(value, topicPrefix) {
return Topic{}, fmt.Errorf("topic must be %s{deviceID}/{kind}", topicPrefix)
}
parts := strings.Split(strings.TrimPrefix(value, topicPrefix), "/")
if len(parts) != 2 || !validID(parts[0]) {
return Topic{}, fmt.Errorf("topic must be %s{deviceID}/{kind}", topicPrefix)
}
kind := Kind(parts[1])
if !supportedKind(kind) {
return Topic{}, fmt.Errorf("unsupported topic kind %q", kind)
}
return Topic{DeviceID: parts[0], Kind: kind}, nil
}
func supportedKind(kind Kind) bool {
switch kind {
case KindTelemetry, KindCommands, KindResponses, KindSales:
return true
default:
return false
}
}
// validID limits identifiers to safe, single-segment topic values.
func validID(value string) bool {
if value == "" || len(value) &gt; 128 {
return false
}
for _, r := range value {
if !(r &gt;= 'a' &amp;&amp; r &lt;= 'z' || r &gt;= 'A' &amp;&amp; r &lt;= 'Z' || r &gt;= '0' &amp;&amp; r &lt;= '9' || strings.ContainsRune("._-", r)) {
return false
}
}
return true
}
</code></pre></article><article class="card"><h3>📄 internal/contracts/message.go</h3><pre><code>package contracts
import (
"fmt"
"time"
)
const SchemaVersion = 1
// MQTTMessage is implemented by each versioned message contract.
// Validation uses the parsed route and caller-supplied time for deterministic checks.
type MQTTMessage interface {
Validate(topic Topic, now time.Time) error
}
// These types define the JSON fields shared by backend and simulator.
type Heartbeat struct {
SchemaVersion int `json:"schema_version"`
MessageID string `json:"message_id"`
DeviceID string `json:"device_id"`
OccurredAt string `json:"occurred_at"`
Temperature *float64 `json:"temperature,omitempty"`
}
type Command struct {
SchemaVersion int `json:"schema_version"`
MessageID string `json:"message_id"`
DeviceID string `json:"device_id"`
OccurredAt string `json:"occurred_at"`
CommandID string `json:"command_id"`
Command string `json:"command"`
}
type CommandResult struct {
SchemaVersion int `json:"schema_version"`
MessageID string `json:"message_id"`
DeviceID string `json:"device_id"`
OccurredAt string `json:"occurred_at"`
CommandID string `json:"command_id"`
Status string `json:"status"`
Detail string `json:"detail,omitempty"`
}
type SalesLine struct {
SKU string `json:"sku"`
Quantity int64 `json:"quantity"`
AmountMinor int64 `json:"amount_minor"`
}
type SalesReport struct {
SchemaVersion int `json:"schema_version"`
ReportID string `json:"report_id"`
DeviceID string `json:"device_id"`
OccurredAt string `json:"occurred_at"`
Items []SalesLine `json:"items"`
}
// validateEnvelope checks fields shared by every versioned message.
func validateEnvelope(topic Topic, version int, messageID, deviceID, occurredAt string, now time.Time) error {
if version != SchemaVersion {
return fmt.Errorf("unsupported schema_version %d", version)
}
if !validID(messageID) || !validID(deviceID) {
return fmt.Errorf("message_id and device_id must be valid IDs")
}
if topic.DeviceID != deviceID {
return fmt.Errorf("topic device %q does not match payload device_id %q", topic.DeviceID, deviceID)
}
stamp, err := time.Parse(time.RFC3339, occurredAt)
if err != nil {
return fmt.Errorf("occurred_at must be RFC3339: %w", err)
}
if stamp.After(now.Add(30 * time.Second)) {
return fmt.Errorf("occurred_at is more than 30 seconds in the future")
}
return nil
}
// Each Validate method adds checks specific to one message kind.
func (m Heartbeat) Validate(topic Topic, now time.Time) error {
if topic.Kind != KindTelemetry {
return fmt.Errorf("heartbeat requires telemetry topic")
}
if err := validateEnvelope(topic, m.SchemaVersion, m.MessageID, m.DeviceID, m.OccurredAt, now); err != nil {
return err
}
if m.Temperature != nil &amp;&amp; (*m.Temperature &lt; -50 || *m.Temperature &gt; 100) {
return fmt.Errorf("temperature is outside the supported range")
}
return nil
}
func (m Command) Validate(topic Topic, now time.Time) error {
if topic.Kind != KindCommands {
return fmt.Errorf("command requires commands topic")
}
if err := validateEnvelope(topic, m.SchemaVersion, m.MessageID, m.DeviceID, m.OccurredAt, now); err != nil {
return err
}
if !validID(m.CommandID) || m.Command != "ping" {
return fmt.Errorf("command_id is invalid or command is unsupported")
}
return nil
}
func (m CommandResult) Validate(topic Topic, now time.Time) error {
if topic.Kind != KindResponses {
return fmt.Errorf("command result requires responses topic")
}
if err := validateEnvelope(topic, m.SchemaVersion, m.MessageID, m.DeviceID, m.OccurredAt, now); err != nil {
return err
}
if !validID(m.CommandID) || (m.Status != "accepted" &amp;&amp; m.Status != "completed" &amp;&amp; m.Status != "rejected") {
return fmt.Errorf("command result has invalid command_id or status")
}
return nil
}
func (m SalesReport) Validate(topic Topic, now time.Time) error {
if topic.Kind != KindSales {
return fmt.Errorf("sales report requires sales topic")
}
if err := validateEnvelope(topic, m.SchemaVersion, m.ReportID, m.DeviceID, m.OccurredAt, now); err != nil {
return err
}
if len(m.Items) == 0 || len(m.Items) &gt; 100 {
return fmt.Errorf("sales report must contain 1 to 100 items")
}
for i, item := range m.Items {
if !validID(item.SKU) || item.Quantity &lt;= 0 || item.AmountMinor &lt; 0 {
return fmt.Errorf("item %d has invalid SKU, quantity, or amount_minor", i)
}
}
return nil
}
</code></pre></article>
<article class="card"><h3>📄 internal/contracts/decode.go</h3><pre><code>package contracts
import (
"bytes"
"encoding/json"
"fmt"
"io"
"time"
)
const MaxPayloadBytes = 16 * 1024
// Decode bounds input, rejects unknown fields, requires one JSON value,
// and validates it against the topic before returning it.
func Decode[T MQTTMessage](payload []byte, topic Topic, now time.Time) (T, error) {
var value T
if len(payload) &gt; MaxPayloadBytes {
return value, fmt.Errorf("payload exceeds %d bytes", MaxPayloadBytes)
}
decoder := json.NewDecoder(bytes.NewReader(payload))
decoder.DisallowUnknownFields()
if err := decoder.Decode(&amp;value); err != nil {
return value, fmt.Errorf("decode JSON: %w", err)
}
var trailing any
// Decode reads one JSON value at a time. A second call distinguishes
// whitespace (EOF) from a second value or malformed trailing bytes.
if err := decoder.Decode(&amp;trailing); err != io.EOF {
if err == nil {
return value, fmt.Errorf("payload contains more than one JSON value")
}
return value, fmt.Errorf("trailing data after JSON value: %w", err)
}
if err := value.Validate(topic, now); err != nil {
return value, fmt.Errorf("validate message: %w", err)
}
return value, nil
}
</code></pre></article>
<article class="card"><h3>📄 internal/contracts/decode_test.go</h3><pre><code>package contracts
import (
"strings"
"testing"
"time"
)
// Topic tests reject malformed routes and unsupported message kinds.
func TestParseTopic(t *testing.T) {
topic, err := ParseTopic("lab/v1/devices/device-001/sales")
if err != nil {
t.Fatal(err)
}
if topic.DeviceID != "device-001" || topic.Kind != KindSales {
t.Fatalf("unexpected topic: %#v", topic)
}
for _, invalid := range []string{"lab/v1/devices/+/sales", "lab/v1/devices/device-001/unknown", "lab/v1/devices/device-001/sales/extra"} {
if _, err := ParseTopic(invalid); err == nil {
t.Errorf("ParseTopic(%q) succeeded", invalid)
}
}
}
// These cases cover strict JSON shape and single-value decoding.
func TestDecodeRejectsUnknownAndTrailingValues(t *testing.T) {
topic := Topic{DeviceID: "device-001", Kind: KindTelemetry}
now := time.Date(2026, 10, 13, 12, 0, 0, 0, time.UTC)
if _, err := Decode[Heartbeat]([]byte(`{"schema_version":1,"unknown":true}`), topic, now); err == nil {
t.Fatal("unknown field was accepted")
}
if _, err := Decode[Heartbeat]([]byte(`{"schema_version":1,"message_id":"m-1","device_id":"device-001","occurred_at":"2026-10-13T12:00:00Z"} {"message_id":"m-2"}`), topic, now); err == nil {
t.Fatal("trailing JSON value was accepted")
}
}
// This case ensures the application limit is applied before decoding.
func TestDecodeRejectsOversizedPayload(t *testing.T) {
payload := []byte(`"` + strings.Repeat("x", MaxPayloadBytes) + `"`)
if _, err := Decode[Heartbeat](payload, topic, now); err == nil {
t.Fatal("oversized payload was accepted")
}
}
// A body cannot override the device selected by the topic.
func TestCommandRejectsTopicIdentityMismatch(t *testing.T) {
topic, err := ParseTopic("lab/v1/devices/device-001/commands")
if err != nil {
t.Fatal(err)
}
command := Command{SchemaVersion: SchemaVersion, MessageID: "m-1", DeviceID: "device-002", OccurredAt: time.Now().UTC().Format(time.RFC3339), CommandID: "c-1", Command: "ping"}
if err := command.Validate(topic, time.Now().UTC()); err == nil {
t.Fatal("topic/body identity mismatch was accepted")
}
}
</code></pre></article>
<article class="card"><h3>📄 cmd/backend/main.go — validation-stage target</h3><pre><code>package main
import (
"context"
"encoding/json"
"fmt"
"log"
"os"
"os/signal"
"syscall"
"time"
"example.com/mqtt5-learning/internal/config"
"example.com/mqtt5-learning/internal/contracts"
"example.com/mqtt5-learning/internal/mqtt"
)
func main() {
if err := run(); err != nil {
log.Fatal(err)
}
}
func run() error {
cfg, err := config.Load(os.Args[1:])
if err != nil {
return fmt.Errorf("load config: %w", err)
}
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
defer stop()
telemetryFilter, err := contracts.DeviceFilter(contracts.KindTelemetry)
if err != nil { return err }
responsesFilter, err := contracts.DeviceFilter(contracts.KindResponses)
if err != nil { return err }
salesFilter, err := contracts.DeviceFilter(contracts.KindSales)
if err != nil { return err }
commandTopic, err := contracts.DeviceTopic(cfg.MQTT.DeviceID, contracts.KindCommands)
if err != nil { return err }
// Start subscriptions before handling any delivered message.
inbox := make(chan mqtt.Message, 16)
manager, err := mqtt.Start(ctx, cfg, []contracts.TopicFilter{telemetryFilter, responsesFilter, salesFilter}, inbox)
if err != nil {
return fmt.Errorf("start MQTT: %w", err)
}
defer mqtt.Stop(manager)
ticker := time.NewTicker(10 * time.Second)
defer ticker.Stop()
sequence := 0
for {
select {
case &lt;-ctx.Done():
return nil
case message := &lt;-inbox:
if err := handleMessage(message); err != nil {
log.Printf("rejected topic=%q reason=%v", message.Topic, err)
}
case &lt;-ticker.C:
sequence++
id := fmt.Sprintf("ping-%d-%d", time.Now().UTC().Unix(), sequence)
command := contracts.Command{SchemaVersion: contracts.SchemaVersion, MessageID: id, DeviceID: cfg.MQTT.DeviceID, OccurredAt: time.Now().UTC().Format(time.RFC3339), CommandID: id, Command: "ping"}
payload, err := json.Marshal(command)
if err != nil {
return fmt.Errorf("encode command: %w", err)
}
if err := mqtt.Publish(ctx, manager, commandTopic, payload); err != nil {
log.Printf("command publish failed or uncertain id=%s: %v", id, err)
}
}
}
}
func handleMessage(message mqtt.Message) error {
// Parse and validate before a message can reach application handling.
topic, err := contracts.ParseTopic(message.Topic)
if err != nil {
return err
}
now := time.Now().UTC()
switch topic.Kind {
case contracts.KindTelemetry:
value, err := contracts.Decode[contracts.Heartbeat](message.Payload, topic, now)
if err != nil {
return err
}
log.Printf("valid heartbeat device=%s message=%s", value.DeviceID, value.MessageID)
case contracts.KindResponses:
value, err := contracts.Decode[contracts.CommandResult](message.Payload, topic, now)
if err != nil {
return err
}
log.Printf("command result device=%s command=%s status=%s", value.DeviceID, value.CommandID, value.Status)
case contracts.KindSales:
value, err := contracts.Decode[contracts.SalesReport](message.Payload, topic, now)
if err != nil {
return err
}
log.Printf("valid sales report device=%s report=%s items=%d", value.DeviceID, value.ReportID, len(value.Items))
default:
return fmt.Errorf("backend does not consume topic kind %q", topic.Kind)
}
return nil
}
</code></pre></article>
<article class="card"><h3>📄 cmd/simulator/main.go — contract-stage target</h3><pre><code>package main
import (
"context"
"encoding/json"
"fmt"
"log"
"os"
"os/signal"
"syscall"
"time"
"example.com/mqtt5-learning/internal/config"
"example.com/mqtt5-learning/internal/contracts"
"example.com/mqtt5-learning/internal/mqtt"
"github.com/eclipse/paho.golang/autopaho"
)
func main() {
if err := run(); err != nil {
log.Fatal(err)
}
}
func run() error {
cfg, err := config.Load(os.Args[1:])
if err != nil {
return fmt.Errorf("load config: %w", err)
}
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
defer stop()
// Publish this device's heartbeat and subscribe only to its command topic.
commandFilter, err := contracts.DeviceSubscription(cfg.MQTT.DeviceID, contracts.KindCommands)
if err != nil { return err }
telemetryTopic, err := contracts.DeviceTopic(cfg.MQTT.DeviceID, contracts.KindTelemetry)
if err != nil { return err }
responseTopic, err := contracts.DeviceTopic(cfg.MQTT.DeviceID, contracts.KindResponses)
if err != nil { return err }
inbox := make(chan mqtt.Message, 16)
manager, err := mqtt.Start(ctx, cfg, []contracts.TopicFilter{commandFilter}, inbox)
if err != nil {
return fmt.Errorf("start MQTT: %w", err)
}
defer mqtt.Stop(manager)
ticker := time.NewTicker(3 * time.Second)
defer ticker.Stop()
sequence := 0
for {
select {
case &lt;-ctx.Done():
return nil
case &lt;-ticker.C:
sequence++
heartbeat := contracts.Heartbeat{SchemaVersion: contracts.SchemaVersion, MessageID: fmt.Sprintf("heartbeat-%d-%d", time.Now().Unix(), sequence), DeviceID: cfg.MQTT.DeviceID, OccurredAt: time.Now().UTC().Format(time.RFC3339)}
payload, err := json.Marshal(heartbeat)
if err != nil {
return fmt.Errorf("encode heartbeat: %w", err)
}
if err := mqtt.Publish(ctx, manager, telemetryTopic, payload); err != nil {
log.Printf("heartbeat publish: %v", err)
}
case message := &lt;-inbox:
if message.Retain {
log.Printf("reject retained command topic=%q", message.Topic)
continue
}
if err := respond(ctx, manager, cfg.MQTT.DeviceID, message); err != nil {
log.Printf("reject command topic=%q reason=%v", message.Topic, err)
}
}
}
}
func respond(ctx context.Context, manager *autopaho.ConnectionManager, deviceID string, message mqtt.Message) error {
// Check route and body identity before constructing a command result.
topic, err := contracts.ParseTopic(message.Topic)
if err != nil {
return err
}
command, err := contracts.Decode[contracts.Command](message.Payload, topic, time.Now().UTC())
if err != nil {
return err
}
if topic.DeviceID != deviceID {
return fmt.Errorf("command is for another device")
}
result := contracts.CommandResult{SchemaVersion: contracts.SchemaVersion, MessageID: "result-" + command.CommandID, DeviceID: deviceID, OccurredAt: time.Now().UTC().Format(time.RFC3339), CommandID: command.CommandID, Status: "completed"}
payload, err := json.Marshal(result)
if err != nil {
return fmt.Errorf("encode result: %w", err)
}
responseTopic, err := contracts.DeviceTopic(deviceID, contracts.KindResponses)
if err != nil { return fmt.Errorf("build response topic: %w", err) }
return mqtt.Publish(ctx, manager, responseTopic, payload)
}
</code></pre></article></details>
<h2>🏗️ Build and run the validation boundary</h2>
<p>After saving the contract files and updating both process entrypoints, run these from the repository root:</p>
<pre><code>go test ./internal/contracts
go build -o /tmp/mqtt5-lab-backend ./cmd/backend
go build -o /tmp/mqtt5-lab-simulator ./cmd/simulator
docker compose -p mqtt5-learning up -d broker
docker compose -p mqtt5-learning ps</code></pre>
<p>The contract tests exercise the topic and payload cases in the test file. Building each process checks that the typed contract is wired through the real programs. The broker command starts only the isolated learning service.</p>
<p>Run the simulator and backend in separate terminals:</p>
<pre><code>/tmp/mqtt5-lab-simulator -username simulator -client-id lab-simulator-device-001</code></pre>
<pre><code>/tmp/mqtt5-lab-backend -username backend -client-id lab-backend</code></pre>
<p>Expect the backend to log validated heartbeat identifiers and the simulator to remain connected. Then stop both clients with Ctrl+C and stop the broker with <code>docker compose -p mqtt5-learning stop broker</code>. This smoke run checks valid traffic; the unit tests cover malformed, oversized, and identity-mismatch inputs without relying on broker delivery. Ask me a follow-up if the result differs from your prediction.</p>
<section class="card callout"><h2>✅ Takeaway</h2><p><strong>Route identifies a topic scope; contract validation checks a payload claim; broker ACLs decide who may publish.</strong> Pass only a validated typed value to application work.</p><p>Ask me a follow-up if any of those boundaries still blur together.</p></section>
<footer>
<p>Start: <a href="../course-source/04-publish-subscribe-simulator.md">Section 4 source lesson</a> · Next: <a href="0005a-backend-message-processing.html">Lesson 5A — apply valid messages</a> · Reference: <a href="../reference/message-processing-lifecycle.html">message-processing lifecycle</a></p>
<p>Primary sources: <a href="https://docs.oasis-open.org/mqtt/mqtt/v5.0/mqtt-v5.0.html">OASIS MQTT 5.0 Standard</a>; <a href="https://pkg.go.dev/encoding/json">Go `encoding/json` documentation</a>. See <a href="../RESOURCES.md">workspace resources</a>.</p>
</footer>
</main>
</body>
</html>
teaching/lessons/0005a-backend-message-processing.html
+420
View File
@@ -0,0 +1,420 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Lesson 5A — Add the Backend Message-Processing Step</title>
<link rel="stylesheet" href="../assets/lesson.css">
<script src="../assets/quiz.js" defer></script>
</head>
<body>
<main class="page">
<p class="eyebrow">Go MQTT Vending Backend · Lesson 5A · 15 minutes</p>
<h1>Add the backend message-processing step</h1>
<p class="dek">Section 5 validates incoming bytes. In 5A, add the application code that turns a valid heartbeat into a small, queryable backend state.</p>
<div class="meta"><span class="pill">🎯 One skill: apply a validated message</span><span class="pill">🧠 Separate device time from receive time</span><span class="pill">🧪 In-memory lab</span></div>
<section class="card callout">
<p class="question">❓ Start with the repository as it is</p>
<p>The current <code>cmd/backend/main.go</code> receives <code>mqtt.Message</code> values from <code>internal/mqtt</code> and logs their topic and payload. The current codebase does <strong>not</strong> yet contain <code>internal/deviceview</code>, <code>Store</code>, <code>Service</code>, or the methods in this lesson.</p>
<p>Section 5A in <code>teaching/course-source/05a-backend-message-processing.md</code> is an implementation assignment: create those files and replace the raw-message log path with contract validation and application behavior. Every code block below is a complete file, including imports and wiring. The new code is the lesson's target, not code that already exists.</p>
</section>
<h2>🗺️ First, see what the files do</h2>
<div class="two-col">
<article class="card">
<h3>✅ Existing files used by this lesson</h3>
<ul>
<li><code>internal/contracts/topic.go</code> defines <code>TopicName</code>, <code>TopicFilter</code>, <code>Kind</code>, concrete-topic builders, exact and fleet-wide subscription builders, and <code>ParseTopic</code>. Concrete publish names and subscription filters use distinct types.</li>
<li><code>internal/contracts/message.go</code> defines the payload contracts and their topic/body validation methods.</li>
<li><code>internal/contracts/decode.go</code> decodes and validates a payload against that parsed topic.</li>
<li><code>internal/mqtt/client.go</code> defines <code>mqtt.Message</code> and copies received topic and payload into the inbox.</li>
<li><code>cmd/backend/main.go</code> owns startup and the receive loop; today it logs received payloads.</li>
</ul>
</article>
<article class="card">
<h3>🛠️ Files this lesson adds or changes</h3>
<ul>
<li>Create <code>internal/deviceview/store.go</code> for an in-memory latest-heartbeat view.</li>
<li>Create <code>internal/deviceview/service.go</code> to validate messages and apply their current use cases.</li>
<li>Edit <code>cmd/backend/main.go</code> to create the service and delegate inbox messages to it.</li>
</ul>
<p><code>Store</code>, <code>Service</code>, <code>RecordHeartbeat</code>, and the other device-view methods are new lesson code. They are not present in the checkout before you implement this section.</p>
</article>
</div>
<div class="flow" aria-label="message processing boundary">
<div class="step">Existing receive loop<br><strong>topic + bytes</strong></div><span class="arrow">→</span>
<div class="step">Existing contracts<br><strong>parse + validate</strong></div><span class="arrow">→</span>
<div class="step">New service<br><strong>choose application behavior</strong></div><span class="arrow">→</span>
<div class="step">New store<br><strong>latest heartbeat view</strong></div>
</div>
<p>Validation is a trust boundary, not a business action. The service applies the meaning of a valid message. The store owns the resulting view. Keeping these roles separate lets Section 6 move service work to workers without mixing application state with Paho callback code.</p>
<h2>🗃️ Then, create `internal/deviceview/store.go`</h2>
<p><strong>File overview:</strong> this file owns one latest heartbeat snapshot per device. It keeps the map in process memory, so a backend restart clears the view.</p>
<p><strong>`NewStore` summary:</strong> initializes the empty map. A query before the first heartbeat can report “not seen yet” rather than inventing an offline record.</p>
<p><strong>`RecordHeartbeat` summary:</strong> parses device-reported time, copies optional temperature, and replaces the snapshot only when the report is newer.</p>
<p><strong>`GetStatus` summary:</strong> returns the snapshot, plus whether the backend received it within the requested freshness window.</p>
<h3>🧩 Complete file</h3>
<pre><code>package deviceview
import (
"fmt"
"sync"
"time"
"example.com/mqtt5-learning/internal/contracts"
)
type Snapshot struct {
DeviceID string
MessageID string
ReportedAt time.Time
ReceivedAt time.Time
Temperature float64
TemperatureSet bool
}
type Status struct {
Snapshot Snapshot
HeartbeatFresh bool
}
type Store struct {
mu sync.RWMutex
latest map[string]Snapshot
}
func NewStore() *Store {
return &amp;Store{latest: make(map[string]Snapshot)}
}
func (s *Store) RecordHeartbeat(message contracts.Heartbeat, receivedAt time.Time) (bool, error) {
reportedAt, err := time.Parse(time.RFC3339, message.OccurredAt)
if err != nil {
// Decode already validates this timestamp. Keep the store defensive too.
return false, fmt.Errorf("parse heartbeat timestamp: %w", err)
}
snapshot := Snapshot{
DeviceID: message.DeviceID,
MessageID: message.MessageID,
ReportedAt: reportedAt,
ReceivedAt: receivedAt,
}
if message.Temperature != nil {
snapshot.Temperature = *message.Temperature
snapshot.TemperatureSet = true
}
s.mu.Lock()
defer s.mu.Unlock()
previous, exists := s.latest[message.DeviceID]
if exists &amp;&amp; !reportedAt.After(previous.ReportedAt) {
return false, nil
}
s.latest[message.DeviceID] = snapshot
return true, nil
}
func (s *Store) GetStatus(deviceID string, now time.Time, freshFor time.Duration) (Status, bool) {
s.mu.RLock()
snapshot, exists := s.latest[deviceID]
s.mu.RUnlock()
if !exists {
return Status{}, false
}
fresh := freshFor &gt; 0 &amp;&amp; !now.Before(snapshot.ReceivedAt) &amp;&amp; now.Sub(snapshot.ReceivedAt) &lt;= freshFor
return Status{Snapshot: snapshot, HeartbeatFresh: fresh}, true
}
</code></pre>
<div class="card">
<h3>🔎 Details: how the store methods work</h3>
<ul>
<li><strong><code>NewStore</code>:</strong> the map starts empty; the second return from <code>GetStatus</code> is false until a heartbeat is recorded.</li>
<li><strong><code>RecordHeartbeat</code>:</strong> contract decoding already checks <code>OccurredAt</code>, but the store parses it defensively too. <code>ReportedAt</code> orders observations; backend-supplied <code>ReceivedAt</code> measures freshness.</li>
<li><strong>Optional temperature:</strong> the store keeps temperature as a value and uses <code>TemperatureSet</code> to distinguish a missing reading from a reported zero. The map snapshot copy is therefore independent without pointer-copy logic.</li>
<li><strong>Locking:</strong> use an exclusive lock while comparing and replacing the map entry. Section 6 will allow multiple workers to call the service. A duplicate or older timestamp returns <code>false, nil</code> and leaves the newer view unchanged.</li>
<li><strong><code>GetStatus</code>:</strong> the read lock protects the map lookup. Freshness requires a positive window, a receive time not in the future, and an age no greater than that window.</li>
</ul>
<p>Ordering is not durable deduplication. Equal timestamps count as not newer, and process restart removes all snapshots.</p>
</div>
<h2>🧭 Next, create `internal/deviceview/service.go`</h2>
<p><strong>File overview:</strong> this file receives the existing transport-neutral <code>mqtt.Message</code>, validates it with <code>contracts</code>, applies the behavior assigned to its message kind, and logs a safe outcome. It does not own a Paho client or publish MQTT messages.</p>
<p><strong>`NewService` summary:</strong> checks that the store, logger, and positive freshness window are configured before message handling begins.</p>
<p><strong>`HandleMessage` summary:</strong> checks cancellation, parses the route, captures one backend receive time, then validates and handles the message according to its kind.</p>
<p><strong>`Status` summary:</strong> asks the store for the latest view using the service clock and configured freshness window.</p>
<h3>🧩 Complete file</h3>
<pre><code>package deviceview
import (
"context"
"fmt"
"log/slog"
"time"
"example.com/mqtt5-learning/internal/contracts"
"example.com/mqtt5-learning/internal/mqtt"
)
type Service struct {
store *Store
logger *slog.Logger
freshFor time.Duration
now func() time.Time
}
func NewService(store *Store, logger *slog.Logger, freshFor time.Duration) (*Service, error) {
if store == nil {
return nil, fmt.Errorf("device-view store is required")
}
if logger == nil {
return nil, fmt.Errorf("logger is required")
}
if freshFor &lt;= 0 {
return nil, fmt.Errorf("heartbeat freshness window must be positive")
}
return &amp;Service{store: store, logger: logger, freshFor: freshFor, now: time.Now}, nil
}
func (s *Service) HandleMessage(ctx context.Context, message mqtt.Message) error {
if err := ctx.Err(); err != nil {
return err
}
topic, err := contracts.ParseTopic(message.Topic)
if err != nil {
return fmt.Errorf("parse topic: %w", err)
}
receivedAt := s.now().UTC()
switch topic.Kind {
case contracts.KindTelemetry:
heartbeat, err := contracts.Decode[contracts.Heartbeat](message.Payload, topic, receivedAt)
if err != nil {
return fmt.Errorf("validate heartbeat: %w", err)
}
updated, err := s.store.RecordHeartbeat(heartbeat, receivedAt)
if err != nil {
return fmt.Errorf("record heartbeat: %w", err)
}
if !updated {
s.logger.Info("older or duplicate heartbeat ignored",
"device_id", heartbeat.DeviceID,
"message_id", heartbeat.MessageID,
)
return nil
}
s.logger.Info("heartbeat recorded",
"device_id", heartbeat.DeviceID,
"message_id", heartbeat.MessageID,
"received_at", receivedAt,
)
return nil
case contracts.KindSales:
report, err := contracts.Decode[contracts.SalesReport](message.Payload, topic, receivedAt)
if err != nil {
return fmt.Errorf("validate sales report: %w", err)
}
// Validation is not durable admission; Section 13 stores reports.
s.logger.Info("sales report validated; persistence not implemented",
"device_id", report.DeviceID,
"report_id", report.ReportID,
"item_count", len(report.Items),
)
return nil
case contracts.KindResponses:
result, err := contracts.Decode[contracts.CommandResult](message.Payload, topic, receivedAt)
if err != nil {
return fmt.Errorf("validate command result: %w", err)
}
// Section 7 adds pending-command and correlation matching.
s.logger.Info("command result validated; matching not implemented",
"device_id", result.DeviceID,
"command_id", result.CommandID,
"status", result.Status,
)
return nil
default:
return fmt.Errorf("device-view service does not handle topic kind %q", topic.Kind)
}
}
func (s *Service) Status(deviceID string) (Status, bool) {
return s.store.GetStatus(deviceID, s.now().UTC(), s.freshFor)
}
</code></pre>
<div class="card">
<h3>🔎 Details: how the service methods work</h3>
<ul>
<li><strong><code>NewService</code>:</strong> rejects missing dependencies and invalid policy at startup, not later as a nil-pointer failure. Its freshness window is application policy, separate from MQTT Keep Alive and Session Expiry.</li>
<li><strong><code>HandleMessage</code>:</strong> returns immediately for cancelled work or an invalid topic. It captures backend receive time once and reuses it in validation, storage, and logs.</li>
<li><strong>Telemetry branch:</strong> decodes a heartbeat, asks the store to advance its snapshot, and logs either “recorded” or “older or duplicate ignored.” Invalid input never reaches the store.</li>
<li><strong>Sales and response branches:</strong> decode and log identifiers only. Sales are not persisted here, and command responses are not correlated here. Those later outcomes must not be implied by a successful decode.</li>
<li><strong><code>Status</code>:</strong> returns backend state without querying the device. A fresh heartbeat means recent valid application traffic, not a live broker connection or healthy vending hardware.</li>
</ul>
</div>
<h2>🔌 Finally, edit `cmd/backend/main.go`</h2>
<p><strong>File overview:</strong> this existing file loads configuration, starts the MQTT client, owns the inbox receive loop, and periodically publishes the lab's ping command. The edited version keeps those responsibilities and delegates inbound messages to the new service instead of printing raw payloads.</p>
<p><strong><code>run</code> summary:</strong> constructs one store and service before MQTT startup, then passes each received message to <code>handleMessage</code>.</p>
<p><strong><code>handleMessage</code> summary:</strong> keeps the select loop focused on receiving events by delegating application behavior to <code>Service</code>.</p>
<h3>🧩 Complete edited file</h3>
<pre><code>package main
import (
"context"
"encoding/json"
"fmt"
"log"
"log/slog"
"os"
"os/signal"
"syscall"
"time"
"example.com/mqtt5-learning/internal/config"
"example.com/mqtt5-learning/internal/deviceview"
"example.com/mqtt5-learning/internal/mqtt"
)
func main() {
if err := run(); err != nil {
log.Fatal(err)
}
}
func run() error {
cfg, err := config.Load(os.Args[1:])
if err != nil {
return fmt.Errorf("failed to load config: %w", err)
}
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
defer stop()
// Keep one shared in-memory view for the lifetime of this backend process.
store := deviceview.NewStore()
deviceView, err := deviceview.NewService(store, slog.Default(), 90*time.Second)
if err != nil {
return fmt.Errorf("create device-view service: %w", err)
}
telemetryFilter, err := contracts.DeviceFilter(contracts.KindTelemetry)
if err != nil { return err }
responsesFilter, err := contracts.DeviceFilter(contracts.KindResponses)
if err != nil { return err }
commandTopic, err := contracts.DeviceTopic(cfg.MQTT.DeviceID, contracts.KindCommands)
if err != nil { return err }
inbox := make(chan mqtt.Message, 16)
manager, err := mqtt.Start(ctx, cfg, []contracts.TopicFilter{telemetryFilter, responsesFilter}, inbox)
if err != nil {
return fmt.Errorf("failed to start MQTT client: %w", err)
}
defer mqtt.Stop(manager)
ticker := time.NewTicker(10 * time.Second)
defer ticker.Stop()
runID := time.Now().UnixNano()
sequence := 0
for {
select {
case &lt;-ctx.Done():
log.Print("shutting down")
return nil
case message := &lt;-inbox:
if err := handleMessage(ctx, deviceView, message); err != nil {
slog.Error("message handling failed", "topic", message.Topic, "error", err)
}
case &lt;-ticker.C:
sequence++
id := fmt.Sprintf("ping-%d-%d", runID, sequence)
payload, err := json.Marshal(contracts.Command{
SchemaVersion: contracts.SchemaVersion,
MessageID: id,
DeviceID: cfg.MQTT.DeviceID,
OccurredAt: time.Now().UTC().Format(time.RFC3339),
CommandID: id,
Command: "ping",
})
if err != nil {
return fmt.Errorf("failed to marshal payload: %w", err)
}
if err := mqtt.Publish(ctx, manager, commandTopic, payload); err != nil {
log.Printf("command publication failed or outcome uncertain: %v", err)
}
}
}
}
func handleMessage(ctx context.Context, deviceView *deviceview.Service, message mqtt.Message) error {
return deviceView.HandleMessage(ctx, message)
}
</code></pre>
<div class="card">
<h3>🔎 Details: why the integration is placed here</h3>
<ul>
<li>Construct the store once before the loop. Creating it per message would discard prior snapshots.</li>
<li>The 90-second freshness window is the course example policy; it should be chosen relative to the heartbeat interval and expected network delay.</li>
<li>The service handles inbound telemetry and response messages. The existing ticker and outbound ping remain intact.</li>
<li>Errors are logged with the topic and error, while the raw payload is no longer printed. Invalid or mismatched input therefore does not leak into a general log line.</li>
<li>The MQTT callback copies into the bounded inbox and returns before this receive loop applies the use case. Under automatic acknowledgement, a queue drop or later application error does not reverse an acknowledgement already sent. Section 6 studies that loss boundary and adds explicit overload handling.</li>
</ul>
</div>
<h2>🧪 Check your model</h2>
<form class="card quiz" data-quiz data-correct="b" data-explanation="Correct. Section 5A adds the deviceview package and wires it into the existing main loop." data-retry="Which directory exists now, and which package does the lesson ask you to add?">
<fieldset class="question-block"><legend><strong>1. What is the status of `internal/deviceview`?</strong></legend>
<label class="option"><input type="radio" name="q1" value="a"> Current code already stores persisted device status</label>
<label class="option"><input type="radio" name="q1" value="b"> Section 5A asks you to add it</label>
<label class="option"><input type="radio" name="q1" value="c"> Paho creates this package when client connects</label>
</fieldset><button type="submit">Check answer</button><p class="feedback" data-feedback aria-live="polite"></p>
</form>
<form class="card quiz" data-quiz data-correct="a" data-explanation="Right. Device time orders observations; backend receipt time measures how recently this service received a heartbeat." data-retry="Which timestamp comes from the device, and which one can only be recorded by this backend?">
<fieldset class="question-block"><legend><strong>2. Which timestamp measures heartbeat freshness?</strong></legend>
<label class="option"><input type="radio" name="q2" value="a"> Backend receipt time measures heartbeat freshness</label>
<label class="option"><input type="radio" name="q2" value="b"> Device report time measures backend freshness</label>
<label class="option"><input type="radio" name="q2" value="c"> MQTT packet ID measures backend freshness</label>
</fieldset><button type="submit">Check answer</button><p class="feedback" data-feedback aria-live="polite"></p>
</form>
<form class="card quiz" data-quiz data-correct="c" data-explanation="Correct. Freshness describes recent valid heartbeat traffic observed by the backend. It is not a health check for every subsystem." data-retry="What observation does the store actually have: recent heartbeat, live socket, or full physical machine health?">
<fieldset class="question-block"><legend><strong>3. What does `HeartbeatFresh` mean?</strong></legend>
<label class="option"><input type="radio" name="q3" value="a"> All vending hardware is healthy right now</label>
<label class="option"><input type="radio" name="q3" value="b"> Broker confirms that this device remains connected</label>
<label class="option"><input type="radio" name="q3" value="c"> A valid heartbeat arrived within the window</label>
</fieldset><button type="submit">Check answer</button><p class="feedback" data-feedback aria-live="polite"></p>
</form>
<h2>🛠️ Your tiny practice</h2>
<article class="card">
<p>Predict the final store contents for these arrivals:</p>
<ol><li>The device reports 12:00:05; the backend receives it at 12:00:06.</li><li>The device reports 12:00:02; the backend receives it at 12:00:09.</li></ol>
<p>Which snapshot remains? At 12:00:09, is that snapshot fresh under a 90-second window? Explain why the later arrival does not move device-observation order backward.</p>
</article>
<h2>🏗️ Build and run the vertical slice</h2>
<p>After saving the new <code>internal/deviceview</code> files and updating <code>cmd/backend/main.go</code>, compile the real programs and start the isolated broker. Run these from the repository root:</p>
<pre><code>go test ./...
go build -o /tmp/mqtt5-lab-backend ./cmd/backend
go build -o /tmp/mqtt5-lab-simulator ./cmd/simulator
docker compose -p mqtt5-learning up -d broker
docker compose -p mqtt5-learning ps</code></pre>
<p><code>go test ./...</code> runs the tests that exist; it does not prove every device-view branch is covered. The two builds catch compile and link errors in the actual processes. Check that the project-named broker is running before connecting clients.</p>
<p>Start each binary in its own terminal:</p>
<pre><code>/tmp/mqtt5-lab-simulator -username simulator -client-id lab-simulator-device-001</code></pre>
<pre><code>/tmp/mqtt5-lab-backend -username backend -client-id lab-backend</code></pre>
<p>Within a few seconds, expect backend log entries for recorded heartbeats with device and message IDs. The backend also sends a ping every ten seconds; the simulator replies, and the backend validates that response. This confirms a real MQTT message reached the application service. It does not establish command correlation, which Lesson 7 adds.</p>
<p>Stop both clients with Ctrl+C, then run <code>docker compose -p mqtt5-learning stop broker</code>. The device view is in memory and disappears when the backend exits. The lesson's temporary <code>deviceView.Status</code> diagnostic can show the snapshot and freshness flag; no HTTP status endpoint exists yet. Ask me a follow-up if a build error or log result is unclear.</p>
<section class="card callout"><h2>✅ Takeaway</h2><p><strong>The existing code receives MQTT messages; the new service validates and applies them; the new store holds the latest heartbeat view.</strong> The implementation is complete above, but its state is RAM-only and a fresh heartbeat does not prove device health.</p><p>Ask me a follow-up if you want to trace one message through the code line by line.</p></section>
<footer>
<p>Previous: <a href="0005-contract-validation.html">Lesson 5 — validate the message</a> · Next: <a href="0006-bounded-message-work.html">Lesson 6 — bounded message work</a> · <a href="../reference/message-processing-lifecycle.html">message lifecycle reference</a></p>
<p>Source implementation: <a href="../course-source/05a-backend-message-processing.md">Section 5A in the course</a>. Primary references: <a href="https://pkg.go.dev/log/slog">Go structured logging</a>; <a href="https://pkg.go.dev/sync">Go synchronization</a>; <a href="https://docs.oasis-open.org/mqtt/mqtt/v5.0/mqtt-v5.0.html">OASIS MQTT 5.0 Standard</a>.</p>
</footer>
</main>
</body>
</html>
teaching/lessons/0006-bounded-message-work.html
+540
View File
@@ -0,0 +1,540 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8"><meta name="viewport" content="width=device-width, initial-scale=1">
<title>Lesson 6 — Bound Message Work</title>
<link rel="stylesheet" href="../assets/lesson.css"><script src="../assets/quiz.js" defer></script>
</head>
<body><main class="page">
<p class="eyebrow">Go MQTT Vending Backend · Lesson 6 · 8 minutes</p>
<h1>Bound message work; make overload visible</h1>
<p class="dek">Move application work out of the MQTT receive callback without creating an unlimited backlog or hiding queue rejection.</p>
<div class="meta"><span class="pill">🎯 One skill: reason about admission</span><span class="pill">🧠 Predict overload behavior</span><span class="pill">🧪 In-memory queue</span></div>
<section class="card callout"><p class="question">❓ Start with the story</p><p>A burst of vending heartbeats arrives while workers are slow. The callback returns, but the telemetry queue is full. What happened to the packet, and what can the backend truthfully claim?</p><p>Pause before reading: separate MQTT delivery acknowledgement from the backend's queue decision.</p></section>
<section class="card"><h2>📍 Files to implement in the codebase</h2><p><code>internal/dispatch/pool.go</code> is already present. Read it first: this lesson teaches its admission, worker, and shutdown behavior, then aligns the routing, pool coordination, and both process entrypoints with the listings below.</p><table><thead><tr><th>Path</th><th>Action</th><th>What you will learn or change</th></tr></thead><tbody><tr><td><code>internal/dispatch/pool.go</code></td><td>Read and understand existing file</td><td>Fixed workers, bounded queue, immediate admission outcomes, counters, bounded drain</td></tr><tr><td><code>internal/dispatch/receiver.go</code></td><td>Review and align</td><td>Route by parsed topic kind and copy payload ownership before enqueue</td></tr><tr><td><code>internal/dispatch/shutdown.go</code></td><td>Review and align</td><td>Stop every pool before waiting under one shared deadline</td></tr><tr><td><code>cmd/backend/main.go</code></td><td>Replace with complete target file below</td><td>Wire Section 5A service to separate bounded pools and preserve MQTT during drain</td></tr><tr><td><code>cmd/simulator/main.go</code></td><td>Replace with complete target file below</td><td>Handle device commands through bounded work and drain before disconnect</td></tr></tbody></table><p>The routing and shutdown files are already present in this checkout. Compare them with the complete listings below and preserve the same admission and error semantics.</p></section>
<h2>📄 File overview: `pool.go` controls finite work</h2>
<p><code>internal/dispatch/pool.go</code> owns a fixed worker set and a bounded jobs channel. It limits simultaneous work and waiting work. It does not make queued messages durable, and it cannot ask MQTT to reverse an acknowledgement already sent.</p>
<div class="flow" aria-label="dispatch lifecycle"><div class="step">MQTT callback<br><strong>copy + submit</strong></div><span class="arrow">→</span><div class="step">Finite queue<br><strong>admit or reject</strong></div><span class="arrow">→</span><div class="step">Fixed workers<br><strong>run service</strong></div></div>
<article class="card">
<h3>📥 `Submit` — short summary</h3>
<p>Returns immediately with an explicit outcome: accepted, full, or stopping.</p>
<pre><code>func (p *Pool) Submit(ctx context.Context, job Job) (Outcome, error) {
if job == nil { return Stopping, errors.New("nil job") }
if err := ctx.Err(); err != nil { return Stopping, err }
p.mu.Lock()
defer p.mu.Unlock()
if !p.accepting { return Stopping, nil }
select {
case p.jobs &lt;- job:
p.accepted.Add(1)
return Accepted, nil
default:
p.rejected.Add(1)
return Full, nil
}
}</code></pre>
<p class="small">This is the focused <code>Submit</code> method excerpt from <code>Pool</code>. The complete file defines the pool fields, <code>Outcome</code> values, worker loop, and shutdown behavior.</p>
<h3>🧩 Details: why each branch exists</h3>
<ul><li>Reject nil or already-cancelled work before touching the queue.</li><li>The mutex makes the accepting check and channel send atomic with shutdown closing the channel.</li><li>The <code>default</code> branch means “no slot now”: this method never waits for capacity.</li><li><code>Accepted</code> means queued in memory, not completed or saved to disk.</li><li><code>Full</code> is an application overload decision. With automatic acknowledgement, returning it does not undo PUBACK or request redelivery.</li></ul>
</article>
<h2>⚖️ Choose a policy per traffic class</h2>
<div class="two-col"><article class="card"><h3>📈 Telemetry</h3><p>If the business allows it, drop a stale heartbeat and count the rejection. Freshness will age out. Make the loss visible.</p></article><article class="card"><h3>🧾 Sales and commands</h3><p>Silently dropping a report or command is usually unacceptable. Return an explicit failure and add durable admission before claiming crash-safe processing.</p></article></div>
<p>Separate pools can prevent telemetry from consuming every waiting slot reserved for important work. That is capacity isolation, not durability. Shutdown order matters too: stop producers, stop admission, drain workers to a deadline while needed transport remains alive, then disconnect.</p>
<section class="card"><h2>🛑 What does <code>ctx.Err()</code> contain during shutdown?</h2><p>The <code>ctx</code> passed to <code>Pool.Shutdown</code> is the drain context. If its deadline expires before workers finish, <code>ctx.Err()</code> returns <code>context.DeadlineExceeded</code>. If something cancels it earlier, it returns <code>context.Canceled</code>. If workers finish first, <code>Pool.Shutdown</code> returns <code>nil</code> instead. On timeout, the pool uses that context error as its shutdown result and cancels the worker job context. Cancellation is cooperative, so a handler must check its context or use context-aware I/O to stop promptly.</p></section>
<h2>🧭 File and function map</h2>
<p><strong>File overview:</strong> Section 6 adds three pieces: <code>pool.go</code> owns bounded execution, <code>receiver.go</code> routes each message to the right pool, and <code>shutdown.go</code> closes all pools before waiting for their workers. The backend entrypoint below wires those pieces to the Section 5A service and existing MQTT inbox.</p>
<div class="two-col">
<article class="card"><h3>📄 <code>internal/dispatch/pool.go</code></h3>
<p><strong><code>New</code> summary:</strong> starts a fixed worker count and a queue with a fixed capacity.</p>
<p><strong><code>Submit</code> summary:</strong> reports <code>Accepted</code>, <code>Full</code>, or <code>Stopping</code> without waiting for queue room.</p>
<p><strong><code>StopAdmission</code> / <code>Shutdown</code> summary:</strong> close the submission path, wait for workers up to a deadline, then cancel remaining work.</p>
<p><strong><code>Stats</code> / <code>worker</code> summary:</strong> expose admission and completion counts; workers run jobs and count success or failure.</p>
<h4>🔎 Details</h4><p>The mutex protects both the accepting flag and channel close, preventing a send-on-closed-channel race. The nonblocking <code>select</code> makes overload immediate. Closing the jobs channel lets workers finish buffered jobs; the shared deadline keeps draining bounded. If it expires, cancellation is cooperative: a job that ignores its context can keep running. Counters describe process-local events, not durable business records.</p>
</article>
<article class="card"><h3>📄 <code>internal/dispatch/receiver.go</code></h3>
<p><strong><code>Receiver.Submit</code> summary:</strong> parses the topic, selects the telemetry or important pool, copies payload bytes, and submits a closure that calls the application processor.</p>
<h4>🔎 Details</h4><p>Routing on a parsed <code>Kind</code> keeps traffic policy out of the MQTT callback. Commands, sales, and command responses use the important pool; only telemetry uses the telemetry pool. Copying the payload transfers ownership: the worker no longer depends on Paho or a reused inbox buffer. The job closure captures that owned copy and runs later with the worker's context.</p>
</article>
<article class="card"><h3>📄 <code>internal/dispatch/shutdown.go</code></h3>
<p><strong><code>Shutdown</code> summary:</strong> stops submissions to every pool first, then drains them using one shared deadline.</p>
<h4>🔎 Details</h4><p>Stopping all pools before waiting prevents one pool from accepting new work while the other drains. Reusing one context bounds the total shutdown time instead of allowing a fresh timeout for each pool. <code>errors.Join</code> preserves timeout failures from multiple pools. Keep MQTT alive during this interval if jobs need to publish their result; disconnect only after the drain attempt.</p>
</article>
</div>
<h3>🔌 Integration overview: edit <code>cmd/backend/main.go</code></h3>
<p><strong><code>run</code> summary:</strong> builds the service and pools once, starts MQTT, sends each inbox message through <code>Receiver</code>, and drains workers when the signal context ends.</p>
<p><strong>Details:</strong> the signal context stops the main loop, while a separate transport context keeps MQTT available during the bounded drain. The signal case itself only logs and returns; as <code>run</code> returns, deferred cleanup executes in reverse registration order. The dispatcher drain defer was registered after <code>mqtt.Stop</code>, so it runs first. Pool jobs use an independent parent context so the signal does not cancel them before draining. Once drain completes or times out, <code>mqtt.Stop</code> disconnects the client. The inbox and callback remain bounded; if more messages arrive during shutdown they can still be dropped under automatic acknowledgement, which this lesson reports rather than hiding.</p>
<details class="card" open><summary><strong>🧩 Complete Section 6 backend wiring: <code>cmd/backend/main.go</code></strong></summary><p>This is the full target entrypoint layered on the Section 5A service. The three dispatcher files follow it below.</p><pre><code>package main
import (
"context"
"encoding/json"
"fmt"
"log"
"log/slog"
"os"
"os/signal"
"syscall"
"time"
"example.com/mqtt5-learning/internal/config"
"example.com/mqtt5-learning/internal/contracts"
"example.com/mqtt5-learning/internal/deviceview"
"example.com/mqtt5-learning/internal/dispatch"
"example.com/mqtt5-learning/internal/mqtt"
)
func main() {
if err := run(); err != nil {
log.Fatal(err)
}
}
func run() error {
cfg, err := config.Load(os.Args[1:])
if err != nil {
return fmt.Errorf("failed to load config: %w", err)
}
lifecycleCtx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
defer stop()
transportCtx, stopTransport := context.WithCancel(context.Background())
defer stopTransport()
store := deviceview.NewStore()
service, err := deviceview.NewService(store, slog.Default(), 90*time.Second)
if err != nil {
return fmt.Errorf("create device-view service: %w", err)
}
telemetryFilter, err := contracts.DeviceFilter(contracts.KindTelemetry)
if err != nil { return err }
responsesFilter, err := contracts.DeviceFilter(contracts.KindResponses)
if err != nil { return err }
commandTopic, err := contracts.DeviceTopic(cfg.MQTT.DeviceID, contracts.KindCommands)
if err != nil { return err }
inbox := make(chan mqtt.Message, 16)
manager, err := mqtt.Start(transportCtx, cfg, []contracts.TopicFilter{telemetryFilter, responsesFilter}, inbox)
if err != nil {
return fmt.Errorf("failed to start MQTT client: %w", err)
}
defer mqtt.Stop(manager)
telemetryPool := dispatch.New(context.Background(), 2, 16)
importantPool := dispatch.New(context.Background(), 2, 32)
receiver := &amp;dispatch.Receiver{
Telemetry: telemetryPool,
Important: importantPool,
Process: service.HandleMessage,
}
defer func() {
drainCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
started := time.Now()
if err := dispatch.Shutdown(drainCtx, telemetryPool, importantPool); err != nil {
log.Printf("dispatcher shutdown incomplete: %v", err)
}
log.Printf("dispatcher drain finished elapsed=%s telemetry=%+v important=%+v",
time.Since(started), telemetryPool.Stats(), importantPool.Stats())
}()
ticker := time.NewTicker(10 * time.Second)
defer ticker.Stop()
runID := time.Now().UnixNano()
sequence := 0
for {
select {
case &lt;-lifecycleCtx.Done():
log.Print("shutting down; draining accepted work before MQTT disconnect")
return nil
case message := &lt;-inbox:
outcome, submitErr := receiver.Submit(lifecycleCtx, message)
if submitErr != nil || outcome != dispatch.Accepted {
slog.Warn("message admission rejected", "topic", message.Topic, "outcome", outcome, "error", submitErr)
}
case &lt;-ticker.C:
sequence++
id := fmt.Sprintf("ping-%d-%d", runID, sequence)
payload, marshalErr := json.Marshal(contracts.Command{
SchemaVersion: contracts.SchemaVersion,
MessageID: id,
DeviceID: cfg.MQTT.DeviceID,
OccurredAt: time.Now().UTC().Format(time.RFC3339),
CommandID: id,
Command: "ping",
})
if marshalErr != nil {
return fmt.Errorf("failed to marshal payload: %w", marshalErr)
}
if publishErr := mqtt.Publish(lifecycleCtx, manager, commandTopic, payload); publishErr != nil {
log.Printf("command publication failed or outcome uncertain: %v", publishErr)
}
}
}
}
</code></pre></details>
<h3>🤖 Simulator integration overview: edit <code>cmd/simulator/main.go</code></h3>
<p><strong><code>run</code> summary:</strong> starts the device client, publishes heartbeats, and submits received commands to bounded workers instead of handling them inline in the main loop.</p>
<p><strong><code>respond</code> summary:</strong> rejects retained or misrouted commands, validates the contract, and publishes the pong using the worker context.</p>
<p><strong>Details:</strong> the simulator attempts to drain accepted command jobs before closing MQTT so a worker can still publish its response. The drain has a five-second limit; if it expires, remaining in-memory work may not complete. A full important queue is logged as an application rejection; automatic acknowledgement may already have completed the subscriber delivery, so this RAM-only example cannot promise redelivery.</p>
<details class="card" open><summary><strong>🧩 Complete Section 6 simulator wiring: <code>cmd/simulator/main.go</code></strong></summary><p>This complete target file carries the Section 5 contract-aware simulator forward and moves inbound command work to the dispatcher.</p><pre><code>package main
import (
"context"
"encoding/json"
"fmt"
"log"
"os"
"os/signal"
"sync/atomic"
"syscall"
"time"
"example.com/mqtt5-learning/internal/config"
"example.com/mqtt5-learning/internal/contracts"
"example.com/mqtt5-learning/internal/dispatch"
"example.com/mqtt5-learning/internal/mqtt"
"github.com/eclipse/paho.golang/autopaho"
)
func main() {
if err := run(); err != nil {
log.Fatal(err)
}
}
func run() error {
cfg, err := config.Load(os.Args[1:])
if err != nil {
return fmt.Errorf("load config: %w", err)
}
lifecycleCtx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
defer stop()
transportCtx, stopTransport := context.WithCancel(context.Background())
defer stopTransport()
commandFilter, err := contracts.DeviceSubscription(cfg.MQTT.DeviceID, contracts.KindCommands)
if err != nil { return err }
telemetryTopic, err := contracts.DeviceTopic(cfg.MQTT.DeviceID, contracts.KindTelemetry)
if err != nil { return err }
inbox := make(chan mqtt.Message, 16)
manager, err := mqtt.Start(transportCtx, cfg, []contracts.TopicFilter{commandFilter}, inbox)
if err != nil {
return fmt.Errorf("start MQTT: %w", err)
}
defer mqtt.Stop(manager)
runID := time.Now().UnixNano()
telemetryPool := dispatch.New(context.Background(), 2, 16)
importantPool := dispatch.New(context.Background(), 2, 16)
var resultSequence atomic.Uint64
receiver := &amp;dispatch.Receiver{
Telemetry: telemetryPool,
Important: importantPool,
Process: func(jobCtx context.Context, message mqtt.Message) error {
resultID := fmt.Sprintf("result-%d-%d", runID, resultSequence.Add(1))
return respond(jobCtx, manager, cfg.MQTT.DeviceID, resultID, message)
},
}
defer func() {
drainCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
started := time.Now()
if err := dispatch.Shutdown(drainCtx, telemetryPool, importantPool); err != nil {
log.Printf("dispatcher shutdown incomplete: %v", err)
}
log.Printf("dispatcher drain finished elapsed=%s telemetry=%+v important=%+v",
time.Since(started), telemetryPool.Stats(), importantPool.Stats())
}()
ticker := time.NewTicker(3 * time.Second)
defer ticker.Stop()
sequence := uint64(0)
for {
select {
case &lt;-lifecycleCtx.Done():
log.Print("shutting down; draining accepted command work before MQTT disconnect")
return nil
case &lt;-ticker.C:
sequence++
heartbeat := contracts.Heartbeat{
SchemaVersion: contracts.SchemaVersion,
MessageID: fmt.Sprintf("heartbeat-%d-%d", time.Now().Unix(), sequence),
DeviceID: cfg.MQTT.DeviceID,
OccurredAt: time.Now().UTC().Format(time.RFC3339),
}
payload, marshalErr := json.Marshal(heartbeat)
if marshalErr != nil {
return fmt.Errorf("encode heartbeat: %w", marshalErr)
}
if publishErr := mqtt.Publish(lifecycleCtx, manager, telemetryTopic, payload); publishErr != nil {
log.Printf("heartbeat publish: %v", publishErr)
}
case message := &lt;-inbox:
outcome, submitErr := receiver.Submit(lifecycleCtx, message)
if submitErr != nil || outcome != dispatch.Accepted {
log.Printf("command admission rejected outcome=%d topic=%q error=%v", outcome, message.Topic, submitErr)
}
}
}
}
func respond(ctx context.Context, manager *autopaho.ConnectionManager, deviceID, resultMessageID string, message mqtt.Message) error {
if message.Retain {
return fmt.Errorf("retained command rejected")
}
topic, err := contracts.ParseTopic(message.Topic)
if err != nil {
return err
}
if topic.Kind != contracts.KindCommands || topic.DeviceID != deviceID {
return fmt.Errorf("command topic is not authorized for this device")
}
command, err := contracts.Decode[contracts.Command](message.Payload, topic, time.Now().UTC())
if err != nil {
return err
}
result := contracts.CommandResult{
SchemaVersion: contracts.SchemaVersion,
MessageID: resultMessageID,
DeviceID: deviceID,
OccurredAt: time.Now().UTC().Format(time.RFC3339),
CommandID: command.CommandID,
Status: "completed",
}
payload, err := json.Marshal(result)
if err != nil {
return fmt.Errorf("encode result: %w", err)
}
responseTopic, err := contracts.DeviceTopic(deviceID, contracts.KindResponses)
if err != nil { return fmt.Errorf("build response topic: %w", err) }
return mqtt.Publish(ctx, manager, responseTopic, payload)
}
</code></pre></details>
<h2>🧪 Check your model</h2>
<form class="card quiz" data-quiz data-correct="b" data-explanation="Correct. Full describes local queue admission; automatic MQTT acknowledgement is a separate protocol decision and is not rolled back." data-retry="Ask what Submit controls and what Paho controls. Are those the same state machine?"><fieldset class="question-block"><legend><strong>1. The bounded queue returns `Full`. What does that change?</strong></legend><label class="option"><input type="radio" name="q1" value="a"> The broker must retransmit this message automatically now</label><label class="option"><input type="radio" name="q1" value="b"> The app rejected admission; PUBACK may already stand</label><label class="option"><input type="radio" name="q1" value="c"> The device command has completed with a failure</label></fieldset><button type="submit">Check answer</button><p class="feedback" data-feedback aria-live="polite"></p></form>
<form class="card quiz" data-quiz data-correct="c" data-explanation="Right. Accepted means the queue received an in-memory job. A worker has not necessarily completed it, and a process crash can lose it." data-retry="Which counter records enqueue, and which evidence records worker completion?"><fieldset class="question-block"><legend><strong>2. What does `Accepted` mean for a queue job?</strong></legend><label class="option"><input type="radio" name="q2" value="a"> The worker completed the vending business operation</label><label class="option"><input type="radio" name="q2" value="b"> The broker durably stored the application result</label><label class="option"><input type="radio" name="q2" value="c"> The job entered the process's in-memory queue</label></fieldset><button type="submit">Check answer</button><p class="feedback" data-feedback aria-live="polite"></p></form>
<form class="card quiz" data-quiz data-correct="a" data-explanation="Correct. Stop admission, drain workers while needed transport remains available, then close it. A drain deadline can still leave RAM-only work unfinished." data-retry="Could a worker publish a response after MQTT has already been disconnected?"><fieldset class="question-block"><legend><strong>3. What should shutdown do before MQTT disconnect?</strong></legend><label class="option"><input type="radio" name="q3" value="a"> Stop admission, drain workers, then close the transport</label><label class="option"><input type="radio" name="q3" value="b"> Disconnect MQTT, then let workers finish replies later</label><label class="option"><input type="radio" name="q3" value="c"> Keep admitting jobs until the process exits fully</label></fieldset><button type="submit">Check answer</button><p class="feedback" data-feedback aria-live="polite"></p></form>
<h2>🛠️ Your tiny practice</h2><div class="card"><p>When capacity is full, write one sentence and one metric for each class: heartbeat, sales report, command result. Then state the exact loss window created by automatic acknowledgement plus an in-memory queue. Compare with <a href="../course-source/06-concurrency-cancellation.md">Section 6's dispatcher and shutdown code ↗</a>.</p></div>
<details class="card" open><summary><strong>📦 Section 6 complete dispatcher files</strong></summary><p>Complete source files for the added dispatcher package. The complete backend integration appears immediately above.</p><article class="card"><h3>📄 internal/dispatch/pool.go</h3><pre><code>package dispatch
import (
"context"
"errors"
"sync"
"sync/atomic"
)
type Outcome uint8
// Outcomes make queue admission success and overload visible to callers.
const (
Accepted Outcome = iota
Full
Stopping
)
type Job func(context.Context) error
type Stats struct {
Accepted uint64
Rejected uint64
Completed uint64
Failed uint64
}
// Pool owns a bounded queue, worker lifecycle, and admission counters.
type Pool struct {
mu sync.Mutex
accepting bool
jobs chan Job
ctx context.Context
cancel context.CancelFunc
workers sync.WaitGroup
done chan struct{}
accepted atomic.Uint64
rejected atomic.Uint64
completed atomic.Uint64
failed atomic.Uint64
}
// New starts a fixed worker set and ties job cancellation to parent.
func New(parent context.Context, workers, capacity int) *Pool {
if workers &lt; 1 {
workers = 1
}
if capacity &lt; 0 {
capacity = 0
}
ctx, cancel := context.WithCancel(parent)
p := &amp;Pool{accepting: true, jobs: make(chan Job, capacity), ctx: ctx, cancel: cancel, done: make(chan struct{})}
p.workers.Add(workers)
for i := 0; i &lt; workers; i++ {
go p.worker()
}
go func() { p.workers.Wait(); close(p.done) }()
return p
}
// Submit never waits for queue capacity. The caller gets an explicit Full result.
// The caller context can cancel admission before it enters the queue.
func (p *Pool) Submit(ctx context.Context, job Job) (Outcome, error) {
if job == nil {
return Stopping, errors.New("nil job")
}
if err := ctx.Err(); err != nil {
return Stopping, err
}
p.mu.Lock()
defer p.mu.Unlock()
if !p.accepting {
return Stopping, nil
}
select {
case p.jobs &lt;- job:
p.accepted.Add(1)
return Accepted, nil
default:
p.rejected.Add(1)
return Full, nil
}
}
func (p *Pool) StopAdmission() {
p.mu.Lock()
defer p.mu.Unlock()
if p.accepting {
p.accepting = false
close(p.jobs)
}
}
// Shutdown stops new submissions and drains queued work until ctx expires.
// Jobs must observe their context so deadline cancellation can stop slow work.
func (p *Pool) Shutdown(ctx context.Context) error {
p.StopAdmission()
select {
case &lt;-p.done:
p.cancel()
return nil
case &lt;-ctx.Done():
p.cancel()
return ctx.Err()
}
}
func (p *Pool) Stats() Stats {
return Stats{Accepted: p.accepted.Load(), Rejected: p.rejected.Load(), Completed: p.completed.Load(), Failed: p.failed.Load()}
}
// Workers execute admitted jobs and count their final result.
func (p *Pool) worker() {
defer p.workers.Done()
for {
select {
case &lt;-p.ctx.Done():
return
case job, ok := &lt;-p.jobs:
if !ok {
return
}
if err := job(p.ctx); err != nil {
p.failed.Add(1)
continue
}
p.completed.Add(1)
}
}
}
</code></pre></article>
<article class="card"><h3>📄 internal/dispatch/receiver.go</h3><pre><code>package dispatch
import (
"context"
"fmt"
"example.com/mqtt5-learning/internal/contracts"
"example.com/mqtt5-learning/internal/mqtt"
)
type Processor func(context.Context, mqtt.Message) error
// Receiver validates the route, selects a pool, and copies payload ownership.
type Receiver struct {
Telemetry *Pool
Important *Pool
Process Processor
}
func (r *Receiver) Submit(ctx context.Context, message mqtt.Message) (Outcome, error) {
if r.Process == nil {
return Stopping, fmt.Errorf("message processor is required")
}
topic, err := contracts.ParseTopic(message.Topic)
if err != nil {
return Stopping, err
}
pool := r.Telemetry
if topic.Kind == contracts.KindCommands || topic.Kind == contracts.KindResponses || topic.Kind == contracts.KindSales {
pool = r.Important
}
if pool == nil {
return Stopping, fmt.Errorf("no queue configured for topic kind %q", topic.Kind)
}
copyOfMessage := message
copyOfMessage.Payload = append([]byte(nil), message.Payload...)
return pool.Submit(ctx, func(jobCtx context.Context) error { return r.Process(jobCtx, copyOfMessage) })
}
</code></pre></article>
<article class="card"><h3>📄 internal/dispatch/shutdown.go</h3><pre><code>package dispatch
import (
"context"
"errors"
)
func Shutdown(ctx context.Context, pools ...*Pool) error {
// Close admission on every pool before waiting for any worker to drain.
for _, pool := range pools {
if pool != nil {
pool.StopAdmission()
}
}
var result error
// Share one deadline across the drain so total shutdown time stays bounded.
for _, pool := range pools {
if pool != nil {
result = errors.Join(result, pool.Shutdown(ctx))
}
}
return result
}
</code></pre></article></details>
<h2>🏗️ Build and smoke-run the updated processes</h2>
<p>After saving all three <code>internal/dispatch</code> files and wiring both entrypoints, run these commands from the repository root:</p>
<pre><code>go test ./...
go test -race ./...
go build -o /tmp/mqtt5-lab-backend ./cmd/backend
go build -o /tmp/mqtt5-lab-simulator ./cmd/simulator</code></pre>
<p>The race detector can report races only on paths the tests execute; it does not prove every worker schedule safe. The builds confirm both programs use the updated wiring.</p>
<p>Start only the named lab broker, then run the two binaries in separate terminals:</p>
<pre><code>docker compose -p mqtt5-learning up -d broker
docker compose -p mqtt5-learning ps</code></pre>
<pre><code>/tmp/mqtt5-lab-simulator -username simulator -client-id lab-simulator-device-001</code></pre>
<pre><code>/tmp/mqtt5-lab-backend -username backend -client-id lab-backend</code></pre>
<p>Confirm heartbeats and ping responses still complete while passing through worker queues. On Ctrl+C, observe the shutdown summary and compare accepted, rejected, completed, failed, and drain-time values if your wiring logs them. This ordinary traffic is a smoke run, not an overload test; use a finite burst larger than active workers plus queue capacity to observe <code>Full</code> outcomes. Stop the clients, then stop the lab broker with <code>docker compose -p mqtt5-learning stop broker</code>.</p>
<p>Ask me a follow-up if the counters or shutdown behavior differ from your prediction.</p>
<section class="card callout"><h2>✅ Takeaway</h2><p><strong>A bounded queue protects process memory; an explicit overload policy protects business meaning. Neither makes work durable.</strong></p><p>Tell me which message class you would admit first and why.</p></section>
<footer><p>Previous: <a href="0005a-backend-message-processing.html">Lesson 5A</a> · Next: <a href="0007-command-state.html">Lesson 7 — command identity and state</a> · <a href="../reference/message-processing-lifecycle.html">lifecycle reference</a></p><p>Sources: <a href="https://pkg.go.dev/context">Go `context`</a>; <a href="https://pkg.go.dev/sync">Go `sync`</a>; <a href="https://pkg.go.dev/github.com/eclipse/paho.golang/autopaho/examples/queue@v0.23.0">Paho Go v0.23.0 queue example</a>; <a href="../course-source/06-concurrency-cancellation.md">repository Section 6</a>.</p></footer>
</main></body></html>
teaching/lessons/0007-command-state.html
+745
View File
@@ -0,0 +1,745 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8"><meta name="viewport" content="width=device-width, initial-scale=1">
<title>Lesson 7 — Match Command Results to the Right Request</title>
<link rel="stylesheet" href="../assets/lesson.css"><script src="../assets/quiz.js" defer></script>
</head>
<body><main class="page">
<p class="eyebrow">Go MQTT Vending Backend · Lesson 7 · 8 minutes</p>
<h1>Match command results to the right request</h1>
<p class="dek">Keep broker acknowledgement, device result, and timeout uncertainty separate while several vending commands are in flight.</p>
<div class="meta"><span class="pill">🎯 One skill: correlate safely</span><span class="pill">🧠 Predict a fast response</span><span class="pill">🧪 Simulated device</span></div>
<section class="card callout"><p class="question">❓ Start with the story</p><p>The backend publishes command <code>C-42</code>. PUBACK arrives, but no device result arrives before the deadline. Can the backend say the command failed or that the machine did nothing?</p><p>State what is known, what is unknown, and which ID can tie a later result to this command.</p></section>
<section class="card"><h2>📍 Checkout status</h2><p>The current codebase does not yet contain <code>internal/commands</code> or <code>internal/simulator</code>, and its <code>mqtt.Message</code> has no Response Topic or Correlation Data fields. Section 7 creates those files and extends the MQTT message copy; complete source files are included below.</p><p><strong>Durability boundary:</strong> command records and simulator deduplication are process-local. Restarting either process loses its in-memory state. Sections 13–14 add durable records and recoverable publication intent.</p></section>
<h2>🗺️ File overview: command state outlives a packet</h2>
<p><code>internal/commands/store.go</code> owns command records and state transitions. <code>service.go</code> registers state before publishing and records timeouts. The MQTT adapter carries Response Topic and Correlation Data; the simulator checks its reply destination and suppresses repeated effects by command ID.</p>
<div class="flow" aria-label="command result lifecycle"><div class="step">Create stable ID<br><strong>pending record</strong></div><span class="arrow">→</span><div class="step">Publish MQTT<br><strong>PUBACK evidence</strong></div><span class="arrow">→</span><div class="step">Match result<br><strong>device evidence</strong></div><span class="arrow">→</span><div class="step">No result by deadline<br><strong>unknown</strong></div></div>
<article class="card"><h3>🚀 `CreateAndPublish` — short summary</h3><p>Creates correlation bytes, stores pending state, publishes with a deadline, then records broker acknowledgement separately.</p><h3>🧩 Details: insert before publish</h3><pre><code>record := Record{
CommandID: commandID,
DeviceID: deviceID,
CorrelationData: correlation,
Status: Pending,
Deadline: now.Add(timeout),
}
if err := store.Insert(record); err != nil {
return Record{}, err
}
// Only after pending state exists:
replyTopic, err := contracts.DeviceTopic(deviceID, contracts.KindResponses)
if err != nil { return Record{}, err }
reason, err := publisher.PublishCommand(ctx, deviceID, command, replyTopic, correlation)
if err != nil {
return store.MarkUnknownIfPending(deviceID, commandID, "publish outcome uncertain")
}
return store.MarkBrokerAck(deviceID, commandID, reason)</code></pre><p>A device can reply before the publish call returns, so pending state must exist first. If the publish call times out, the backend may not know whether the broker accepted it; mark the outcome unknown rather than asserting no effect. A late response still needs an explicit policy.</p></article>
<h2>🪪 Three identifiers have different jobs</h2>
<article class="card"><ul><li><strong>MQTT packet ID:</strong> protocol bookkeeping for an exchange; it can be reused.</li><li><strong>Correlation Data:</strong> opaque bytes to match one response to one pending request; it is not authorization.</li><li><strong>Application command ID:</strong> stable business identity across retries; it supports device-side deduplication and status lookup.</li></ul><p>The responder must also check that the Response Topic is authorized for that device. Matching identity is not proof of identity; credentials and broker ACLs remain separate controls.</p></article>
<h2>🧭 Keep the state meanings honest</h2>
<table><thead><tr><th>Evidence or event</th><th>Safe interpretation</th></tr></thead><tbody><tr><td>Created and stored</td><td>Pending publish attempt</td></tr><tr><td>Successful PUBACK</td><td>Broker acknowledged this publish hop</td></tr><tr><td>Validated device result</td><td>Apply accepted, completed, or rejected evidence</td></tr><tr><td>Deadline with no result</td><td>Unknown outcome; the effect may or may not have happened</td></tr></tbody></table>
<p>Conceptually, the lifecycle is <code>created → pending delivery → accepted → completed/rejected</code>; a deadline moves unresolved work to <code>unknown</code>. A valid late result may resolve <code>unknown</code> here, while <code>completed</code> and <code>rejected</code> remain terminal. The reference store inserts directly as <code>Pending</code> before PUBLISH, so creation is not a separate queryable status. <code>BrokerAcked</code> is orthogonal to application status.</p>
<h2>🧭 File and function map</h2>
<p><strong>File overview:</strong> <code>commands/store.go</code> owns in-memory records and transition checks; <code>commands/service.go</code> coordinates state and publishing; <code>mqtt/command_publisher.go</code> translates a command into an MQTT 5 packet; <code>commands/receive.go</code> validates incoming results. The simulator files validate, deduplicate, and answer commands. The modified <code>mqtt/client.go</code> copies MQTT 5 response properties into the app-owned message.</p>
<div class="two-col">
<article class="card"><h3>📦 <code>internal/commands/store.go</code></h3>
<p><strong>Function summaries:</strong> <code>NewStore</code> creates the table; <code>Insert</code>, <code>Get</code>, <code>Update</code>, and <code>List</code> manage records; <code>MarkBrokerAck</code> records transport evidence; <code>MarkUnknownIfPending</code> records uncertainty; <code>ApplyResult</code> matches device evidence.</p>
<h4>🔎 Details</h4><p>Keys include both device and command ID so equal IDs from different devices do not collide. The lock protects concurrent HTTP, MQTT, and timeout activity. Correlation bytes are copied at storage boundaries. <code>ApplyResult</code> checks device scope and correlation before changing state, and preserves terminal <code>Completed</code>/<code>Rejected</code> results against duplicates. Broker acknowledgement remains an independent field, not a command status.</p>
</article>
<article class="card"><h3>🧠 <code>internal/commands/service.go</code></h3>
<p><strong>Function summaries:</strong> <code>NewService</code> wires the store, publisher, clock, and response timeout; <code>CreateAndPublish</code> stores pending state before MQTT publish; <code>HandleResult</code> delegates validated result matching; <code>Expire</code> marks overdue work unknown; <code>Get</code> reads current state.</p>
<h4>🔎 Details</h4><p>The pending insert must happen before publishing because a fast simulator can answer before <code>PublishCommand</code> returns. A publish timeout can mean “ack not observed,” not “broker definitely did not accept it,” so the service marks the outcome unknown. <code>Expire</code> uses the same uncertainty meaning and never reports that the physical action did not occur. In this teaching version, the clock and records are process-local.</p>
</article>
<article class="card"><h3>📡 MQTT adapter and result receiver</h3>
<p><strong><code>PublishCommand</code> summary:</strong> validates the device and reply destination, encodes the contract, publishes QoS 1 with Response Topic and Correlation Data, and returns the broker reason code.</p>
<p><strong><code>HandleMessage</code> summary:</strong> parses the response route, decodes a <code>CommandResult</code>, and passes copied correlation bytes to the service.</p>
<p><strong><code>mqtt.Start</code> summary:</strong> the Section 7 target copies Response Topic and Correlation Data inside the callback, while the packet properties are still available.</p>
<h4>🔎 Details</h4><p>The adapter verifies that the reply topic belongs to the target device before publishing. A response is not accepted just because it arrived on a response-shaped topic: the contract must validate and the store must match its device and correlation to a pending record. Correlation matches a request; broker ACLs and device credentials authorize who can publish.</p>
</article>
<article class="card"><h3>🤖 Simulator files</h3>
<p><strong><code>AuthorizedResponseTopic</code> summary:</strong> permits only that device's response topic. <strong><code>Deduplicator.Apply</code> summary:</strong> runs an effect once per device/command ID while its bounded cache retains the result. <strong><code>Respond</code> summary:</strong> validates route, payload, retained flag, and response destination, then publishes the result with the original correlation bytes. <strong><code>run</code> summary:</strong> owns MQTT startup, heartbeat publication, command receive, and cancellation.</p>
<h4>🔎 Details</h4><p>The cache fingerprints command content. Reusing an ID with changed content is an error; repeating identical content replays the saved result without repeating the effect. Because the cache is in memory and bounded, this is a simulator guarantee only while the entry remains cached and the process stays alive—not production durable exactly-once execution.</p>
</article>
</div>
<h2>🧪 Check your model</h2>
<form class="card quiz" data-quiz data-correct="c" data-explanation="Correct. PUBACK is transport evidence for the publication exchange; it does not establish device delivery or business completion." data-retry="Which component sends PUBACK, and which component creates an application result?"><fieldset class="question-block"><legend><strong>1. What can the backend conclude from PUBACK?</strong></legend><label class="option"><input type="radio" name="q1" value="a"> The device accepted the vending command for execution</label><label class="option"><input type="radio" name="q1" value="b"> The device completed the vending command successfully now</label><label class="option"><input type="radio" name="q1" value="c"> The broker acknowledged this client's publication hop only</label></fieldset><button type="submit">Check answer</button><p class="feedback" data-feedback aria-live="polite"></p></form>
<form class="card quiz" data-quiz data-correct="a" data-explanation="Yes. No response means the backend lacks a device outcome. A timeout records uncertainty; it does not prove non-execution." data-retry="What does the lack of a response say about a physical effect that may have happened?"><fieldset class="question-block"><legend><strong>2. What should timeout mean without a response?</strong></legend><label class="option"><input type="radio" name="q2" value="a"> Backend outcome is unknown; device work may have happened</label><label class="option"><input type="radio" name="q2" value="b"> The command definitely never reached the vending device now</label><label class="option"><input type="radio" name="q2" value="c"> The broker can never undo any command delivered already</label></fieldset><button type="submit">Check answer</button><p class="feedback" data-feedback aria-live="polite"></p></form>
<form class="card quiz" data-quiz data-correct="b" data-explanation="Correct. Correlation matches a response to a request. Topic authorization and device identity need separate checks." data-retry="Can matching bytes prove a device is allowed to publish to a chosen response topic?"><fieldset class="question-block"><legend><strong>3. What does Correlation Data not provide?</strong></legend><label class="option"><input type="radio" name="q3" value="a"> A match between one response and one pending request</label><label class="option"><input type="radio" name="q3" value="b"> Authorization for the response topic chosen by device itself</label><label class="option"><input type="radio" name="q3" value="c"> Opaque request metadata copied into the result packet unchanged</label></fieldset><button type="submit">Check answer</button><p class="feedback" data-feedback aria-live="polite"></p></form>
<h2>🛠️ Your tiny practice</h2><div class="card"><p>Trace this race: pending record inserted → PUBLISH begins → device responds → PUBACK returns. Which state fields change at each step? Then let a command time out, deliver a valid late result with matching device and correlation, and verify the reference policy resolves <code>unknown</code> without regressing <code>completed</code> or <code>rejected</code>. Finally send the same command ID twice and explain which component prevents the simulated effect from running twice.</p><p>Compare with <a href="../course-source/07-command-response.md">Section 7's complete command store and simulator code ↗</a>.</p></div>
<details class="card" open><summary><strong>📦 Section 7 complete command and simulator files</strong></summary><p>Complete source files for this lesson are included here. The code blocks earlier in the lesson focus on the behavior being taught; use these full files when implementing it.</p><article class="card"><h3>📄 internal/commands/store.go</h3><pre><code>package commands
import (
"bytes"
"fmt"
"sync"
"time"
"example.com/mqtt5-learning/internal/contracts"
)
type Status string
// Status records application progress independently from BrokerAcked.
const (
Pending Status = "pending"
Accepted Status = "accepted"
Completed Status = "completed"
Rejected Status = "rejected"
Unknown Status = "unknown"
)
type Record struct {
CommandID string
DeviceID string
CorrelationData []byte
CreatedAt time.Time
Deadline time.Time
Status Status
BrokerAcked bool
AckReasonCode byte
Detail string
}
type Store struct {
mu sync.RWMutex
records map[string]Record
}
// NewStore creates the in-memory command record table.
func NewStore() *Store { return &amp;Store{records: make(map[string]Record)} }
func key(deviceID, commandID string) string { return deviceID + "/" + commandID }
func (s *Store) Insert(record Record) error {
s.mu.Lock()
defer s.mu.Unlock()
k := key(record.DeviceID, record.CommandID)
if _, exists := s.records[k]; exists {
return fmt.Errorf("command already exists")
}
record.CorrelationData = append([]byte(nil), record.CorrelationData...)
s.records[k] = record
return nil
}
// Get and Update copy correlation bytes so callers cannot mutate stored state.
func (s *Store) Get(deviceID, commandID string) (Record, bool) {
s.mu.RLock()
defer s.mu.RUnlock()
record, ok := s.records[key(deviceID, commandID)]
record.CorrelationData = append([]byte(nil), record.CorrelationData...)
return record, ok
}
func (s *Store) Update(record Record) error {
s.mu.Lock()
defer s.mu.Unlock()
k := key(record.DeviceID, record.CommandID)
if _, exists := s.records[k]; !exists {
return fmt.Errorf("command not found")
}
record.CorrelationData = append([]byte(nil), record.CorrelationData...)
s.records[k] = record
return nil
}
func (s *Store) MarkBrokerAck(deviceID, commandID string, reason byte) (Record, error) {
s.mu.Lock()
defer s.mu.Unlock()
k := key(deviceID, commandID)
record, ok := s.records[k]
if !ok {
return Record{}, fmt.Errorf("command not found")
}
record.BrokerAcked, record.AckReasonCode = true, reason
s.records[k] = record
return record, nil
}
// MarkUnknownIfPending preserves any application result that arrived first.
func (s *Store) MarkUnknownIfPending(deviceID, commandID, detail string) (Record, error) {
s.mu.Lock()
defer s.mu.Unlock()
k := key(deviceID, commandID)
record, ok := s.records[k]
if !ok {
return Record{}, fmt.Errorf("command not found")
}
if record.Status == Pending || record.Status == Accepted {
record.Status, record.Detail = Unknown, detail
}
s.records[k] = record
return record, nil
}
// ApplyResult checks both device scope and correlation before changing status.
func (s *Store) ApplyResult(deviceID string, correlation []byte, result contracts.CommandResult) (Record, error) {
s.mu.Lock()
defer s.mu.Unlock()
k := key(deviceID, result.CommandID)
record, ok := s.records[k]
if !ok {
return Record{}, fmt.Errorf("no pending command matches result")
}
if result.DeviceID != record.DeviceID || !bytes.Equal(correlation, record.CorrelationData) {
return Record{}, fmt.Errorf("result identity or correlation does not match pending command")
}
if record.Status == Completed || record.Status == Rejected {
return record, nil
}
switch result.Status {
case "accepted":
record.Status = Accepted
case "completed":
record.Status = Completed
case "rejected":
record.Status = Rejected
default:
return Record{}, fmt.Errorf("unsupported result status %q", result.Status)
}
record.Detail = result.Detail
s.records[k] = record
return record, nil
}
func (s *Store) List() []Record {
s.mu.RLock()
defer s.mu.RUnlock()
result := make([]Record, 0, len(s.records))
for _, record := range s.records {
record.CorrelationData = append([]byte(nil), record.CorrelationData...)
result = append(result, record)
}
return result
}
</code></pre></article>
<article class="card"><h3>📄 internal/commands/service.go</h3><pre><code>package commands
import (
"context"
"crypto/rand"
"fmt"
"time"
"example.com/mqtt5-learning/internal/contracts"
)
type Publisher interface {
PublishCommand(context.Context, string, contracts.Command, contracts.TopicName, []byte) (byte, error)
}
// Service coordinates command identity, storage, publication, and deadlines.
type Service struct {
store *Store
publisher Publisher
now func() time.Time
timeout time.Duration
}
func NewService(store *Store, publisher Publisher, timeout time.Duration) *Service {
return &amp;Service{store: store, publisher: publisher, now: func() time.Time { return time.Now().UTC() }, timeout: timeout}
}
func (s *Service) CreateAndPublish(ctx context.Context, deviceID, commandID string) (Record, error) {
responseTopic, err := contracts.DeviceTopic(deviceID, contracts.KindResponses)
if err != nil { return Record{}, fmt.Errorf("build response topic: %w", err) }
// Store pending state first so a fast device response can match immediately.
now := s.now()
correlation := make([]byte, 16)
if _, err := rand.Read(correlation); err != nil {
return Record{}, fmt.Errorf("create correlation data: %w", err)
}
record := Record{CommandID: commandID, DeviceID: deviceID, CorrelationData: correlation, CreatedAt: now, Deadline: now.Add(s.timeout), Status: Pending}
if err := s.store.Insert(record); err != nil {
return Record{}, err
}
command := contracts.Command{SchemaVersion: contracts.SchemaVersion, MessageID: "message-" + commandID, DeviceID: deviceID, OccurredAt: now.Format(time.RFC3339), CommandID: commandID, Command: "ping"}
ctx, cancel := context.WithDeadline(ctx, record.Deadline)
defer cancel()
reason, err := s.publisher.PublishCommand(ctx, deviceID, command, responseTopic, append([]byte(nil), correlation...))
if err != nil {
updated, _ := s.store.MarkUnknownIfPending(deviceID, commandID, "publish outcome uncertain: "+err.Error())
return updated, err
}
updated, err := s.store.MarkBrokerAck(deviceID, commandID, reason)
return updated, err
}
func (s *Service) HandleResult(deviceID string, correlation []byte, result contracts.CommandResult) (Record, error) {
return s.store.ApplyResult(deviceID, correlation, result)
}
// Expire records uncertainty; it does not claim that the device did no work.
func (s *Service) Expire(now time.Time) {
for _, record := range s.store.List() {
if (record.Status == Pending || record.Status == Accepted) &amp;&amp; !now.Before(record.Deadline) {
_, _ = s.store.MarkUnknownIfPending(record.DeviceID, record.CommandID, "response deadline expired; execution outcome is unknown")
}
}
}
func (s *Service) Get(commandID string) (Record, bool) {
for _, record := range s.store.List() {
if record.CommandID == commandID {
return record, true
}
}
return Record{}, false
}
</code></pre></article>
<article class="card"><h3>📄 internal/mqtt/command_publisher.go</h3><pre><code>package mqtt
import (
"context"
"encoding/json"
"fmt"
"time"
"example.com/mqtt5-learning/internal/contracts"
"github.com/eclipse/paho.golang/autopaho"
"github.com/eclipse/paho.golang/paho"
)
type CommandPublisher struct {
Manager *autopaho.ConnectionManager
}
func (p CommandPublisher) PublishCommand(ctx context.Context, deviceID string, command contracts.Command, responseTopic contracts.TopicName, correlation []byte) (byte, error) {
// Authorize the reply destination before putting it in the MQTT packet.
authorized, err := contracts.DeviceTopic(deviceID, contracts.KindResponses)
if err != nil { return 0, fmt.Errorf("build response topic: %w", err) }
if responseTopic != authorized {
return 0, fmt.Errorf("unauthorized response topic")
}
if command.DeviceID != deviceID {
return 0, fmt.Errorf("command device does not match publish destination")
}
commandTopic, err := contracts.DeviceTopic(deviceID, contracts.KindCommands)
if err != nil { return 0, fmt.Errorf("build command topic: %w", err) }
payload, err := json.Marshal(command)
if err != nil {
return 0, fmt.Errorf("encode command: %w", err)
}
operation, cancel := context.WithTimeout(ctx, 3*time.Second)
defer cancel()
// Correlation Data is copied so the packet owns stable bytes for this publish.
ack, err := p.Manager.Publish(operation, &amp;paho.Publish{
Topic: commandTopic.String(),
QoS: 1,
Payload: payload,
Properties: &amp;paho.PublishProperties{
ContentType: "application/json",
ResponseTopic: responseTopic.String(),
CorrelationData: append([]byte(nil), correlation...),
},
})
if err != nil {
return 0, fmt.Errorf("publish command: %w", err)
}
if ack == nil {
return 0, fmt.Errorf("missing QoS 1 PUBACK")
}
if ack.ReasonCode &gt;= 0x80 {
return ack.ReasonCode, fmt.Errorf("broker rejected publish: reason 0x%02x", ack.ReasonCode)
}
return ack.ReasonCode, nil
}
</code></pre></article>
<article class="card"><h3>📄 internal/commands/receive.go</h3><pre><code>package commands
import (
"fmt"
"time"
"example.com/mqtt5-learning/internal/contracts"
"example.com/mqtt5-learning/internal/mqtt"
)
func HandleMessage(service *Service, message mqtt.Message) error {
// Validate the route and result body before matching the pending request.
topic, err := contracts.ParseTopic(message.Topic)
if err != nil {
return err
}
if topic.Kind != contracts.KindResponses {
return fmt.Errorf("expected a response topic")
}
result, err := contracts.Decode[contracts.CommandResult](message.Payload, topic, time.Now().UTC())
if err != nil {
return err
}
_, err = service.HandleResult(topic.DeviceID, append([]byte(nil), message.CorrelationData...), result)
return err
}
</code></pre></article>
<article class="card"><h3>📄 internal/simulator/dedup.go</h3><pre><code>package simulator
import (
"crypto/sha256"
"encoding/json"
"fmt"
"sync"
"example.com/mqtt5-learning/internal/contracts"
)
type cachedResult struct {
fingerprint [32]byte
result contracts.CommandResult
}
type Deduplicator struct {
mu sync.Mutex
capacity int
results map[string]cachedResult
}
// NewDeduplicator bounds how many command results this simulator retains.
func NewDeduplicator(capacity int) *Deduplicator {
if capacity &lt; 1 {
capacity = 1
}
return &amp;Deduplicator{capacity: capacity, results: make(map[string]cachedResult)}
}
func AuthorizedResponseTopic(deviceID, requested string) bool {
expected, err := contracts.DeviceTopic(deviceID, contracts.KindResponses)
return err == nil && requested == expected.String()
}
// Apply executes effect at most once for a device/command ID while this process lives.
func (d *Deduplicator) Apply(command contracts.Command, effect func() (contracts.CommandResult, error)) (contracts.CommandResult, bool, error) {
if effect == nil {
return contracts.CommandResult{}, false, fmt.Errorf("effect is required")
}
encoded, err := json.Marshal(command)
if err != nil {
return contracts.CommandResult{}, false, fmt.Errorf("fingerprint command: %w", err)
}
fingerprint := sha256.Sum256(encoded)
key := command.DeviceID + "/" + command.CommandID
d.mu.Lock()
defer d.mu.Unlock()
if previous, ok := d.results[key]; ok {
if previous.fingerprint != fingerprint {
return contracts.CommandResult{}, false, fmt.Errorf("command ID was reused with different content")
}
return previous.result, true, nil
}
if len(d.results) &gt;= d.capacity {
return contracts.CommandResult{}, false, fmt.Errorf("deduplication cache is full")
}
result, err := effect()
if err != nil {
return contracts.CommandResult{}, false, err
}
d.results[key] = cachedResult{fingerprint: fingerprint, result: result}
return result, false, nil
}
</code></pre></article>
<article class="card"><h3>📄 internal/simulator/respond.go</h3><pre><code>package simulator
import (
"context"
"encoding/json"
"fmt"
"time"
"example.com/mqtt5-learning/internal/contracts"
"example.com/mqtt5-learning/internal/mqtt"
"github.com/eclipse/paho.golang/autopaho"
"github.com/eclipse/paho.golang/paho"
)
func Respond(ctx context.Context, manager *autopaho.ConnectionManager, cache *Deduplicator, deviceID string, message mqtt.Message) error {
// Reject retained or misrouted requests before validating or applying an effect.
if message.Retain {
return fmt.Errorf("retained commands are rejected")
}
topic, err := contracts.ParseTopic(message.Topic)
if err != nil {
return err
}
if topic.Kind != contracts.KindCommands || topic.DeviceID != deviceID {
return fmt.Errorf("command topic is not authorized for this device")
}
command, err := contracts.Decode[contracts.Command](message.Payload, topic, time.Now().UTC())
if err != nil {
return err
}
if !AuthorizedResponseTopic(deviceID, message.ResponseTopic) {
return fmt.Errorf("unauthorized response topic")
}
// Cache the simulated result by stable command ID to suppress repeat effects.
result, _, err := cache.Apply(command, func() (contracts.CommandResult, error) {
return contracts.CommandResult{SchemaVersion: contracts.SchemaVersion, MessageID: "result-" + command.CommandID, DeviceID: deviceID, OccurredAt: time.Now().UTC().Format(time.RFC3339), CommandID: command.CommandID, Status: "completed"}, nil
})
if err != nil {
return err
}
payload, err := json.Marshal(result)
if err != nil {
return fmt.Errorf("encode result: %w", err)
}
operation, cancel := context.WithTimeout(ctx, 3*time.Second)
defer cancel()
// Reply only to the authorized destination and echo correlation bytes unchanged.
ack, err := manager.Publish(operation, &amp;paho.Publish{Topic: message.ResponseTopic, QoS: 1, Payload: payload, Properties: &amp;paho.PublishProperties{ContentType: "application/json", CorrelationData: append([]byte(nil), message.CorrelationData...)}})
if err != nil {
return fmt.Errorf("publish response: %w", err)
}
if ack == nil {
return fmt.Errorf("missing response PUBACK")
}
if ack.ReasonCode &gt;= 0x80 {
return fmt.Errorf("response publish rejected: reason 0x%02x", ack.ReasonCode)
}
return nil
}
</code></pre></article>
<article class="card"><h3>📄 cmd/simulator/main.go</h3><pre><code>package main
import (
"context"
"encoding/json"
"fmt"
"log"
"os"
"os/signal"
"syscall"
"time"
"example.com/mqtt5-learning/internal/config"
"example.com/mqtt5-learning/internal/contracts"
"example.com/mqtt5-learning/internal/mqtt"
"example.com/mqtt5-learning/internal/simulator"
)
func main() {
if err := run(); err != nil {
log.Fatal(err)
}
}
func run() error {
cfg, err := config.Load(os.Args[1:])
if err != nil {
return fmt.Errorf("load config: %w", err)
}
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
defer stop()
commandFilter, err := contracts.DeviceSubscription(cfg.MQTT.DeviceID, contracts.KindCommands)
if err != nil { return err }
telemetryTopic, err := contracts.DeviceTopic(cfg.MQTT.DeviceID, contracts.KindTelemetry)
if err != nil { return err }
inbox := make(chan mqtt.Message, 16)
manager, err := mqtt.Start(ctx, cfg, []contracts.TopicFilter{commandFilter}, inbox)
if err != nil {
return fmt.Errorf("start MQTT: %w", err)
}
defer mqtt.Stop(manager)
// Keep recent command results so repeated delivery does not repeat the effect.
cache := simulator.NewDeduplicator(1024)
ticker := time.NewTicker(3 * time.Second)
defer ticker.Stop()
sequence := 0
for {
select {
case &lt;-ctx.Done():
return nil
case &lt;-ticker.C:
sequence++
heartbeat := contracts.Heartbeat{SchemaVersion: contracts.SchemaVersion, MessageID: fmt.Sprintf("heartbeat-%d-%d", time.Now().Unix(), sequence), DeviceID: cfg.MQTT.DeviceID, OccurredAt: time.Now().UTC().Format(time.RFC3339)}
payload, err := json.Marshal(heartbeat)
if err != nil {
return fmt.Errorf("encode heartbeat: %w", err)
}
if err := mqtt.Publish(ctx, manager, telemetryTopic, payload); err != nil {
log.Printf("heartbeat publish: %v", err)
}
case message := &lt;-inbox:
if err := simulator.Respond(ctx, manager, cache, cfg.MQTT.DeviceID, message); err != nil {
log.Printf("command rejected or response failed topic=%q reason=%v", message.Topic, err)
}
}
}
}
</code></pre></article><article class="card"><h3>📄 internal/mqtt/client.go — complete file with Section 7 message properties</h3><p>This target version extends the existing MQTT transport message and copies MQTT 5 response properties inside the callback, before the received packet goes out of scope.</p><pre><code>package mqtt
import (
"context"
"fmt"
"log"
"net/url"
"time"
"example.com/mqtt5-learning/internal/config"
"example.com/mqtt5-learning/internal/contracts"
"github.com/eclipse/paho.golang/autopaho"
"github.com/eclipse/paho.golang/paho"
)
type Message struct {
Topic string
Payload []byte
Retain bool
ResponseTopic string
CorrelationData []byte
}
func Start(ctx context.Context, cfg config.Config, topics []contracts.TopicFilter, inbox chan&lt;- Message) (*autopaho.ConnectionManager, error) {
broker, err := url.Parse(cfg.MQTT.URL)
if err != nil {
return nil, err
}
initialSubscription := make(chan error, 1)
manager, err := autopaho.NewConnection(ctx, autopaho.ClientConfig{
ServerUrls: []*url.URL{broker},
KeepAlive: 20,
CleanStartOnInitialConnection: true,
SessionExpiryInterval: 0,
ConnectUsername: cfg.MQTT.Username,
ConnectPassword: []byte(cfg.MQTT.Password),
OnConnectionUp: func(cm *autopaho.ConnectionManager, _ *paho.Connack) {
// This runs again on reconnect; reestablish broker subscriptions.
subCtx, cancel := context.WithTimeout(ctx, 5*time.Second)
defer cancel()
subscriptions := make([]paho.SubscribeOptions, 0, len(topics))
for _, topic := range topics {
subscriptions = append(subscriptions, paho.SubscribeOptions{Topic: topic.String(), QoS: 1, RetainAsPublished: true})
}
ack, subErr := cm.Subscribe(subCtx, &amp;paho.Subscribe{Subscriptions: subscriptions})
if subErr == nil {
if ack == nil || len(ack.Reasons) != len(topics) {
subErr = fmt.Errorf("missing or incomplete SUBACK")
} else {
for i, reason := range ack.Reasons {
if reason &gt; 1 { // We requested QoS 1; valid grants are 0 or 1.
subErr = fmt.Errorf("subscription %s rejected: 0x%02x", topics[i], reason)
break
}
}
}
}
if subErr != nil {
log.Printf("subscribe failed: %v", subErr)
} else {
log.Printf("subscriptions ready: %v", topics)
}
select {
case initialSubscription &lt;- subErr:
default:
}
},
OnConnectionDown: func() bool { log.Print("disconnected; reconnecting"); return true },
OnConnectError: func(err error) { log.Printf("connect error: %v", err) },
ClientID: cfg.MQTT.ClientID,
OnClientError: func(err error) { log.Printf("client error: %v", err) },
OnServerDisconnect: func(d *paho.Disconnect) { log.Printf("disconnect reason=0x%02x", d.ReasonCode) },
OnPublishReceived: []func(paho.PublishReceived) (bool, error){
func(received paho.PublishReceived) (bool, error) {
packet := received.Packet
message := Message{Topic: packet.Topic, Payload: append([]byte(nil), packet.Payload...), Retain: packet.Retain}
if properties := packet.Properties; properties != nil {
message.ResponseTopic = properties.ResponseTopic
message.CorrelationData = append([]byte(nil), properties.CorrelationData...)
}
select {
case inbox &lt;- message:
default:
// Deliberately lossy early demo: automatic MQTT ACK is still possible.
log.Printf("lab inbox full; message dropped topic=%s", packet.Topic)
}
return true, nil
},
},
})
if err != nil {
return nil, fmt.Errorf("start AutoPaho: %w", err)
}
startup, cancel := context.WithTimeout(ctx, cfg.ConnectTimeout())
defer cancel()
if err = manager.AwaitConnection(startup); err != nil {
Stop(manager)
return nil, fmt.Errorf("await connection: %w", err)
}
select {
case err = &lt;-initialSubscription:
if err != nil {
Stop(manager)
return nil, err
}
case &lt;-startup.Done():
Stop(manager)
return nil, fmt.Errorf("await subscriptions: %w", startup.Err())
}
return manager, nil
}
func Publish(ctx context.Context, manager *autopaho.ConnectionManager, topic contracts.TopicName, payload []byte) error {
operation, cancel := context.WithTimeout(ctx, 3*time.Second)
defer cancel()
ack, err := manager.Publish(operation, &amp;paho.Publish{
Topic: topic.String(),
Payload: payload,
QoS: 1,
Retain: false,
Properties: &amp;paho.PublishProperties{
ContentType: "application/json",
},
})
if err != nil {
return fmt.Errorf("publish: %w", err)
}
if ack == nil {
return fmt.Errorf("missing Qos 1 publication response")
}
if ack.ReasonCode &gt;= 0x80 {
return fmt.Errorf("publish rejected: 0x%02x", ack.ReasonCode)
}
log.Printf("broker publish acknowledgement topic=%s reason=0x%02x", topic, ack.ReasonCode)
return nil
}
func Stop(manager *autopaho.ConnectionManager) {
shutdown, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
if err := manager.Disconnect(shutdown); err != nil {
log.Printf("shutdown did not finish within deadline: %v", err)
}
}
</code></pre></article>
</details>
<h2>🏗️ Build and run the command-state stage</h2>
<p>After saving the Section 7 files and updating <code>internal/mqtt/client.go</code>, check the repository and build both processes from its root:</p>
<pre><code>go test ./...
go test -race ./...
go build -o /tmp/mqtt5-lab-backend ./cmd/backend
go build -o /tmp/mqtt5-lab-simulator ./cmd/simulator</code></pre>
<p>These commands run the tests that currently exist and check compilation. The race detector only covers schedules exercised by tests. Section 7 has no HTTP command entrypoint yet, so starting the processes verifies MQTT startup, subscriptions, and simulator heartbeats; it does not trigger a command/response round trip by itself.</p>
<p>For that smoke run, start the named broker and inspect its state:</p>
<pre><code>docker compose -p mqtt5-learning up -d broker
docker compose -p mqtt5-learning ps</code></pre>
<p>Then run these in separate terminals:</p>
<pre><code>/tmp/mqtt5-lab-simulator -username simulator -client-id lab-simulator-device-001</code></pre>
<pre><code>/tmp/mqtt5-lab-backend -username backend -client-id lab-backend</code></pre>
<p>Expect successful connection/subscription logs and simulator heartbeats. Stop both with Ctrl+C, then stop the broker using <code>docker compose -p mqtt5-learning stop broker</code>. Lesson 8 adds the HTTP endpoint that can create a ping and exercise correlation end to end. Ask me a follow-up if your build or startup evidence is unexpected.</p>
<section class="card callout"><h2>✅ Takeaway</h2><p><strong>Register before publish. Match device, command ID, and correlation. Treat a missing result as unknown.</strong></p><p>Tell me how you handled the race; we’ll carry that state into an HTTP endpoint next.</p></section>
<footer><p>Previous: <a href="0006-bounded-message-work.html">Lesson 6</a> · Next: <a href="0008-http-acceptance-vs-completion.html">Lesson 8 — expose command state over HTTP</a> · <a href="../reference/message-processing-lifecycle.html">lifecycle reference</a></p><p>Sources: <a href="https://docs.oasis-open.org/mqtt/mqtt/v5.0/mqtt-v5.0.html">OASIS MQTT 5.0 Standard</a>; <a href="https://pkg.go.dev/github.com/eclipse/paho.golang/paho@v0.23.0">Paho Go v0.23.0 API</a>; <a href="../course-source/07-command-response.md">repository Section 7</a>.</p></footer>
</main></body></html>
teaching/lessons/0008-http-acceptance-vs-completion.html
+510
View File
@@ -0,0 +1,510 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta name="theme-color" content="#fbfaf6">
<title>Lesson 8 — HTTP Acceptance Is Not Device Completion</title>
<link rel="stylesheet" href="../assets/lesson.css">
<script src="../assets/quiz.js" defer></script>
</head>
<body>
<main class="page">
<p class="eyebrow">Go MQTT Vending Backend · Lesson 8 · 6 minutes</p>
<h1>HTTP acceptance is not device completion</h1>
<p class="dek">Build an honest response for a vending command that travels from an HTTP caller through your Go service and Mosquitto to a device.</p>
<div class="meta"><span class="pill">🎯 One skill: name each acknowledgement</span><span class="pill">🧠 Recall before reveal</span><span class="pill">🧪 Disposable broker lab</span></div>
<section class="card callout">
<p class="question">❓ Start with the system story</p>
<p>A dashboard user asks the backend to ping a vending machine. Your API admitted a command record to process-local memory and published the MQTT request. The broker returned PUBACK, but the machine has not replied yet. What can you tell the caller right now?</p>
<p><strong>Pause and answer aloud before reading on.</strong></p>
</section>
<section class="card"><h2>📍 Checkout status</h2><p><code>internal/httpapi/handlers.go</code> and <code>server.go</code> are Section 8 files to create; they do not yet exist in the current codebase. The complete target files and edited backend entrypoint are included below. They depend on the command service introduced in Section 7.</p><p><strong>Lab boundary:</strong> the API binds to loopback and has no authentication. Do not expose it to another network; Section 11 adds API authentication and per-device authorization. HTTP 202 admits work into RAM only. A process restart loses command and retry-key state until Sections 13–14 add durable storage and a recoverable outbox.</p></section>
<h2>🗺️ First, see the whole path</h2>
<p>The API request, MQTT exchange, and device operation are separate stages. Each can succeed or fail independently.</p>
<div class="flow" aria-label="HTTP-to-device command flow">
<div class="step">Caller sends<br><code>POST</code></div><span class="arrow">→</span>
<div class="step">Backend records command<br><strong>API admission</strong></div><span class="arrow">→</span>
<div class="step">Broker replies<br><strong>PUBACK</strong></div><span class="arrow">→</span>
<div class="step">Device reports later<br><strong>application result</strong></div>
</div>
<p>HTTP 202 means the server accepted the request for processing; the processing is not complete. A QoS 1 PUBACK is evidence for its MQTT publication exchange. The device's later result is application evidence. The API should return a command ID now and let the caller query status later. <a href="../reference/http-mqtt-evidence.html">Keep the evidence map nearby ↗</a></p>
<h2>🔎 Then, read the function through its purpose</h2>
<div class="two-col">
<article class="card">
<h3>📄 What the file does</h3>
<p>The new <code>internal/httpapi/handlers.go</code> will translate HTTP input into command-service calls and command records into HTTP responses. It will not publish MQTT packets itself.</p>
</article>
<article class="card">
<h3>🚀 `createCommand` — short summary</h3>
<p>Validates the request and idempotency key, asks the command service to admit one command, then returns its ID and current evidence.</p>
</article>
</div>
<article class="card">
<h3>🧩 Details that affect behavior</h3>
<ul>
<li><code>POST /devices/{deviceID}/commands</code> identifies the operation and device. The handler validates the path ID and command before creating work.</li>
<li><code>Idempotency-Key</code> lets a caller retry after a lost HTTP response without intentionally creating a second command. The key is not an MQTT packet ID or the command ID.</li>
<li>The command service owns the command record and MQTT publish. Keeping MQTT out of the handler makes the HTTP boundary easier to inspect and replace.</li>
<li>HTTP 202 can include the command ID and a <code>Location</code> pointing to <code>GET /commands/{id}</code>. That status resource reports the latest evidence known by the backend.</li>
<li>In this lab, retry keys and command records are in memory. A process restart loses them; later persistence lessons add a durable boundary.</li>
</ul>
</article>
<article class="card">
<h3>📦 `writeCommand` — shape the status response</h3>
<p><strong>Summary:</strong> Sets response metadata, writes the chosen HTTP status, then encodes a command record as JSON.</p>
<pre><code>type commandResponse struct {
CommandID string `json:"command_id"`
DeviceID string `json:"device_id"`
Status string `json:"status"`
BrokerAcked bool `json:"broker_acked"`
}
func writeCommand(w http.ResponseWriter, record commands.Record, status int) {
w.Header().Set("Content-Type", "application/json")
w.Header().Set("Location", "/commands/"+record.CommandID)
w.WriteHeader(status)
_ = json.NewEncoder(w).Encode(commandResponse{
CommandID: record.CommandID,
DeviceID: record.DeviceID,
Status: string(record.Status),
BrokerAcked: record.BrokerAcked,
})
}
</code></pre>
<p><strong>Details:</strong> <code>Location</code> tells the client where to read status. The HTTP status is an argument because the same mapper serves both POST admission and GET status. <code>record.Status</code> and <code>BrokerAcked</code> remain separate fields: the first is application progress, while the second records MQTT publish acknowledgement. The caller should query the returned resource for later device evidence.</p>
<p class="small">This function excerpt focuses on response construction. The full handler file, including imports, request handling, and supporting functions, is included below.</p>
</article>
<p class="small">The route patterns shown in this course use Go's method-aware <code>ServeMux</code> syntax and <code>Request.PathValue</code>, available from Go 1.22. This repository's <code>go.mod</code> targets Go 1.27.1. <a href="https://go.dev/blog/routing-enhancements">Go team: routing enhancements ↗</a></p>
<h2>🗺️ File and function map</h2>
<p><strong>File overview:</strong> <code>handlers.go</code> owns HTTP routes, request validation, retry-key handling, and JSON responses. <code>server.go</code> builds a loopback-only server with I/O deadlines. The backend <code>main.go</code> wires HTTP to the command service while continuing to receive MQTT results and expire old commands.</p>
<div class="two-col">
<article class="card">
<h3>📄 <code>internal/httpapi/handlers.go</code></h3>
<p><strong>Function summaries:</strong> <code>NewAPI</code> stores dependencies; <code>Handler</code> registers the two routes; <code>createCommand</code> validates and admits a request; <code>getCommand</code> reads current status; <code>decodeRequest</code> bounds and strictly decodes JSON; <code>writeCommand</code> shapes the JSON response; <code>validPathID</code> checks safe path IDs; <code>newCommandID</code> generates a random stable ID.</p>
<h4>🔎 Details</h4>
<ul>
<li><code>createCommand</code> validates device, idempotency key, body size/shape, and allowed command before calling the service. It hashes device plus command as a request fingerprint. Reusing the same key and same input returns the saved record; reusing it with different input returns conflict.</li>
<li>The retry map is protected by a mutex, and this simple implementation keeps the lock during service admission. That makes same-key retries atomic in one process, but serializes concurrent creates and loses entries on restart.</li>
<li>If the service returns a record plus a publish error, the handler still stores that record and returns <code>202</code>: the command exists, but its publish outcome may be uncertain. The status body keeps <code>status</code> and <code>broker_acked</code> separate.</li>
<li><code>decodeRequest</code> caps the body, rejects unknown JSON fields, and requires exactly one JSON value. <code>writeCommand</code> sets <code>Location</code> before writing the status and body; after <code>WriteHeader</code>, headers cannot be changed.</li>
</ul>
</article>
<article class="card">
<h3>🌐 <code>internal/httpapi/server.go</code></h3>
<p><strong><code>NewServer</code> summary:</strong> accepts only a loopback address with an explicit port and attaches the API handler plus finite read/write/idle timeouts.</p>
<h4>🔎 Details</h4><p>Loopback keeps this teaching endpoint local; it is not an internet-facing production server. Header, body, response, and idle timeouts limit resource use from slow clients. Production exposure would need an intentional network binding, authentication/authorization, TLS termination, and operational configuration.</p>
</article>
<article class="card">
<h3>🔌 <code>cmd/backend/main.go</code></h3>
<p><strong><code>run</code> summary:</strong> starts MQTT, wires store → command service → HTTP API, runs HTTP in a goroutine, processes MQTT results and expiry ticks, then shuts down HTTP on cancellation.</p>
<p><strong><code>main</code> summary:</strong> reports a startup/runtime error and exits nonzero.</p>
<h4>🔎 Details</h4><p>MQTT starts before the API is exposed, so a request cannot be accepted when the command publisher has no initialized connection manager. The main select loop continues processing response messages while HTTP serves requests; otherwise commands could never advance to device-reported results. The ticker changes pending/accepted commands to unknown after deadlines. On a signal, <code>Server.Shutdown</code> stops accepting new HTTP requests and waits up to five seconds for in-flight handlers. The in-memory store still loses all status on process restart.</p>
</article>
</div>
<h2>🧪 Check your model</h2>
<p>Choose the evidence each event actually provides. Options in each question have the same word count; the correct answer moves between positions.</p>
<form class="card quiz" data-quiz data-correct="a" data-explanation="Correct. HTTP 202 says this API accepted the request for processing; it does not say MQTT or device work completed." data-retry="Think about which component sends the HTTP response and which component has not replied yet.">
<fieldset class="question-block">
<legend><strong>1. What does HTTP 202 support?</strong></legend>
<label class="option"><input type="radio" name="q1" value="a"> The API accepted work for processing</label>
<label class="option"><input type="radio" name="q1" value="b"> The broker acknowledged this MQTT publication</label>
<label class="option"><input type="radio" name="q1" value="c"> Device completed its requested vending operation</label>
</fieldset>
<button type="submit">Check answer</button>
<p class="feedback" data-feedback aria-live="polite"></p>
</form>
<form class="card quiz" data-quiz data-correct="b" data-explanation="Correct. PUBACK is the QoS 1 acknowledgement for this MQTT hop. It does not prove device delivery or business completion." data-retry="Trace the MQTT sender and receiver for this publication. Which component sends PUBACK?">
<fieldset class="question-block">
<legend><strong>2. What does a successful QoS 1 PUBACK establish?</strong></legend>
<label class="option"><input type="radio" name="q2" value="a"> Device completed the vending operation after publication</label>
<label class="option"><input type="radio" name="q2" value="b"> The broker acknowledged this MQTT client publication</label>
<label class="option"><input type="radio" name="q2" value="c"> HTTP caller received a command status resource</label>
</fieldset>
<button type="submit">Check answer</button>
<p class="feedback" data-feedback aria-live="polite"></p>
</form>
<form class="card quiz" data-quiz data-correct="c" data-explanation="Correct. The GET endpoint exposes the latest application evidence the backend has recorded; it does not query or guarantee a live device connection." data-retry="A status endpoint reads backend state. Does it necessarily contact the device now?">
<fieldset class="question-block">
<legend><strong>3. What can GET /commands/{id} return later?</strong></legend>
<label class="option"><input type="radio" name="q3" value="a"> The device's current network connection status only</label>
<label class="option"><input type="radio" name="q3" value="b"> The broker's acknowledgement of the HTTP request</label>
<label class="option"><input type="radio" name="q3" value="c"> The latest application evidence backend has stored</label>
</fieldset>
<button type="submit">Check answer</button>
<p class="feedback" data-feedback aria-live="polite"></p>
</form>
<h2>🛠️ Your tiny design task</h2>
<div class="card">
<p>Open the Section 8 implementation and find <code>writeCommand</code>. Without changing code yet, write the response a caller should see immediately after command admission:</p>
<ol>
<li>Which HTTP status should it return?</li>
<li>Which stable identifier should the caller save?</li>
<li>What URL should the <code>Location</code> header name?</li>
<li>Which status field must not be described as “completed” just because PUBACK arrived?</li>
</ol>
<p>Then compare your answer to the existing implementation in <a href="../course-source/08-http-commands.md">Section 8 — Expose device commands through HTTP ↗</a>. If it matches, explain why in your own words. If it does not, identify which component owns the evidence.</p>
</div>
<details class="card" open><summary><strong>📦 Section 8 complete HTTP files</strong></summary><p>Complete source files for this lesson are included here. The code blocks earlier in the lesson focus on the behavior being taught; use these full files when implementing it.</p><article class="card"><h3>📄 internal/httpapi/handlers.go</h3><pre><code>package httpapi
import (
"context"
"crypto/rand"
"crypto/sha256"
"encoding/hex"
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
"strings"
"sync"
"example.com/mqtt5-learning/internal/commands"
)
const maxRequestBytes = 4 * 1024
// The API depends on command operations, not on the MQTT client itself.
type commandService interface {
CreateAndPublish(ctx context.Context, deviceID, commandID string) (commands.Record, error)
Get(commandID string) (commands.Record, bool)
}
type retryEntry struct {
fingerprint [32]byte
record commands.Record
}
type API struct {
commands commandService
serviceCtx context.Context
mu sync.Mutex
retries map[string]retryEntry
}
// NewAPI keeps retries in memory and ties accepted work to the app lifecycle.
func NewAPI(serviceCtx context.Context, service commandService) *API {
return &amp;API{commands: service, serviceCtx: serviceCtx, retries: make(map[string]retryEntry)}
}
type createRequest struct {
Command string `json:"command"`
}
type commandResponse struct {
CommandID string `json:"command_id"`
DeviceID string `json:"device_id"`
Status string `json:"status"`
BrokerAcked bool `json:"broker_acked"`
Detail string `json:"detail,omitempty"`
}
func (a *API) Handler() http.Handler {
// Keep the public surface small: create commands and read their status.
mux := http.NewServeMux()
mux.HandleFunc("POST /devices/{deviceID}/commands", a.createCommand)
mux.HandleFunc("GET /commands/{commandID}", a.getCommand)
return mux
}
func (a *API) createCommand(w http.ResponseWriter, r *http.Request) {
// Validate caller input before reserving an idempotency key or publishing.
deviceID := r.PathValue("deviceID")
if !validPathID(deviceID) {
http.Error(w, "invalid device ID", http.StatusBadRequest)
return
}
key := strings.TrimSpace(r.Header.Get("Idempotency-Key"))
if key == "" || len(key) &gt; 128 {
http.Error(w, "valid Idempotency-Key required", http.StatusBadRequest)
return
}
var input createRequest
if err := decodeRequest(w, r, &amp;input); err != nil {
var tooLarge *http.MaxBytesError
if errors.As(err, &amp;tooLarge) {
http.Error(w, "request body too large", http.StatusRequestEntityTooLarge)
return
}
http.Error(w, err.Error(), http.StatusBadRequest)
return
}
if input.Command != "ping" {
http.Error(w, "unsupported command", http.StatusBadRequest)
return
}
// The fingerprint detects reuse of one key for different request content.
fingerprint := sha256.Sum256([]byte(deviceID + "\x00" + input.Command))
// This in-memory lesson serializes admissions to make same-key retries atomic.
a.mu.Lock()
defer a.mu.Unlock()
if previous, ok := a.retries[key]; ok {
if previous.fingerprint != fingerprint {
http.Error(w, "idempotency key reused with different request", http.StatusConflict)
return
}
writeCommand(w, previous.record, http.StatusAccepted)
return
}
commandID, err := newCommandID()
if err != nil {
http.Error(w, "could not create command ID", http.StatusInternalServerError)
return
}
record, publishErr := a.commands.CreateAndPublish(a.serviceCtx, deviceID, commandID)
if record.CommandID == "" {
if publishErr != nil {
http.Error(w, "command admission unavailable", http.StatusServiceUnavailable)
return
}
http.Error(w, "command admission failed", http.StatusServiceUnavailable)
return
}
a.retries[key] = retryEntry{fingerprint: fingerprint, record: record}
// The command record exists even if the publish outcome is uncertain.
writeCommand(w, record, http.StatusAccepted)
}
func (a *API) getCommand(w http.ResponseWriter, r *http.Request) {
record, ok := a.commands.Get(r.PathValue("commandID"))
if !ok {
http.Error(w, "command not found", http.StatusNotFound)
return
}
writeCommand(w, record, http.StatusOK)
}
// decodeRequest enforces a byte limit, strict fields, and exactly one JSON value.
func decodeRequest(w http.ResponseWriter, r *http.Request, target any) error {
r.Body = http.MaxBytesReader(w, r.Body, maxRequestBytes)
decoder := json.NewDecoder(r.Body)
decoder.DisallowUnknownFields()
if err := decoder.Decode(target); err != nil {
return fmt.Errorf("invalid JSON request: %w", err)
}
var trailing any
if err := decoder.Decode(&amp;trailing); err != io.EOF {
if err == nil {
return errors.New("request must contain one JSON value")
}
return fmt.Errorf("trailing request data: %w", err)
}
return nil
}
func writeCommand(w http.ResponseWriter, record commands.Record, status int) {
w.Header().Set("Content-Type", "application/json")
w.Header().Set("Location", "/commands/"+record.CommandID)
w.WriteHeader(status)
_ = json.NewEncoder(w).Encode(commandResponse{CommandID: record.CommandID, DeviceID: record.DeviceID, Status: string(record.Status), BrokerAcked: record.BrokerAcked, Detail: record.Detail})
}
// Path IDs use the same restricted alphabet as contract and topic IDs.
func validPathID(value string) bool {
if value == "" || len(value) &gt; 128 {
return false
}
for _, r := range value {
if !(r &gt;= 'a' &amp;&amp; r &lt;= 'z' || r &gt;= 'A' &amp;&amp; r &lt;= 'Z' || r &gt;= '0' &amp;&amp; r &lt;= '9' || strings.ContainsRune("._-", r)) {
return false
}
}
return true
}
func newCommandID() (string, error) {
var raw [16]byte
if _, err := rand.Read(raw[:]); err != nil {
return "", err
}
return "cmd-" + hex.EncodeToString(raw[:]), nil
}
</code></pre></article>
<article class="card"><h3>📄 internal/httpapi/server.go</h3><pre><code>package httpapi
import (
"fmt"
"net"
"net/http"
"time"
)
func NewServer(address string, api *API) (*http.Server, error) {
// The teaching API listens only on loopback and uses finite I/O timeouts.
host, port, err := net.SplitHostPort(address)
if err != nil || (host != "127.0.0.1" &amp;&amp; host != "localhost") || port == "" {
return nil, fmt.Errorf("HTTP lab address must be loopback with an explicit port")
}
return &amp;http.Server{
Addr: address, Handler: api.Handler(),
ReadHeaderTimeout: 5 * time.Second,
ReadTimeout: 10 * time.Second,
WriteTimeout: 10 * time.Second,
IdleTimeout: 60 * time.Second,
}, nil
}
</code></pre></article>
<article class="card"><h3>📄 cmd/backend/main.go</h3><pre><code>package main
import (
"context"
"errors"
"fmt"
"log"
"net/http"
"os"
"os/signal"
"syscall"
"time"
"example.com/mqtt5-learning/internal/commands"
"example.com/mqtt5-learning/internal/config"
"example.com/mqtt5-learning/internal/contracts"
"example.com/mqtt5-learning/internal/httpapi"
"example.com/mqtt5-learning/internal/mqtt"
)
func main() {
if err := run(); err != nil {
log.Fatal(err)
}
}
func run() error {
cfg, err := config.Load(os.Args[1:])
if err != nil {
return fmt.Errorf("load config: %w", err)
}
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
defer stop()
inbox := make(chan mqtt.Message, 16)
telemetryFilter, err := contracts.DeviceFilter(contracts.KindTelemetry)
if err != nil { return err }
responsesFilter, err := contracts.DeviceFilter(contracts.KindResponses)
if err != nil { return err }
salesFilter, err := contracts.DeviceFilter(contracts.KindSales)
if err != nil { return err }
topics := []contracts.TopicFilter{telemetryFilter, responsesFilter, salesFilter}
manager, err := mqtt.Start(ctx, cfg, topics, inbox)
if err != nil {
return fmt.Errorf("start MQTT: %w", err)
}
defer mqtt.Stop(manager)
// Wire the command service before exposing its HTTP endpoints.
store := commands.NewStore()
publisher := mqtt.CommandPublisher{Manager: manager}
service := commands.NewService(store, publisher, 15*time.Second)
api := httpapi.NewAPI(ctx, service)
server, err := httpapi.NewServer("127.0.0.1:18080", api)
if err != nil {
return err
}
serverErrors := make(chan error, 1)
go func() { serverErrors &lt;- server.ListenAndServe() }()
log.Printf("HTTP API listening on %s", server.Addr)
expirationTicker := time.NewTicker(time.Second)
defer expirationTicker.Stop()
// Keep response processing and expiration active for the server lifetime.
for {
select {
case &lt;-ctx.Done():
shutdown, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
if err := server.Shutdown(shutdown); err != nil {
return fmt.Errorf("HTTP shutdown: %w", err)
}
return nil
case err := &lt;-serverErrors:
if !errors.Is(err, http.ErrServerClosed) {
return fmt.Errorf("HTTP server: %w", err)
}
return nil
case &lt;-expirationTicker.C:
service.Expire(time.Now().UTC())
case message := &lt;-inbox:
topic, err := contracts.ParseTopic(message.Topic)
if err != nil {
log.Printf("rejected topic=%q reason=%v", message.Topic, err)
continue
}
switch topic.Kind {
case contracts.KindResponses:
if err := commands.HandleMessage(service, message); err != nil {
log.Printf("rejected response topic=%q reason=%v", message.Topic, err)
}
case contracts.KindTelemetry:
var heartbeat contracts.Heartbeat
heartbeat, err = contracts.Decode[contracts.Heartbeat](message.Payload, topic, time.Now().UTC())
if err != nil {
log.Printf("rejected telemetry topic=%q reason=%v", message.Topic, err)
continue
}
log.Printf("valid heartbeat device=%s message=%s", heartbeat.DeviceID, heartbeat.MessageID)
case contracts.KindSales:
var report contracts.SalesReport
report, err = contracts.Decode[contracts.SalesReport](message.Payload, topic, time.Now().UTC())
if err != nil {
log.Printf("rejected sales topic=%q reason=%v", message.Topic, err)
continue
}
log.Printf("valid sales report device=%s report=%s", report.DeviceID, report.ReportID)
default:
log.Printf("ignored topic kind=%s", topic.Kind)
}
}
}
}
</code></pre></article></details>
<h2>🏗️ Build and run the HTTP-to-device path</h2>
<p>After saving the Section 7 command files plus the Section 8 HTTP files and backend wiring, build and start the lab from the repository root:</p>
<pre><code>go test ./...
go build -o /tmp/mqtt5-lab-backend ./cmd/backend
go build -o /tmp/mqtt5-lab-simulator ./cmd/simulator
docker compose -p mqtt5-learning up -d broker
docker compose -p mqtt5-learning ps</code></pre>
<p>The tests cover only the tests present in the checkout; the builds verify the complete API and simulator programs compile. Run the simulator and backend in separate terminals:</p>
<pre><code>/tmp/mqtt5-lab-simulator -username simulator -client-id lab-simulator-device-001</code></pre>
<pre><code>/tmp/mqtt5-lab-backend -username backend -client-id lab-backend</code></pre>
<p>Once the backend reports that the HTTP API is listening, submit a harmless ping. Copy the returned <code>command_id</code> into the GET request:</p>
<pre><code>curl -i -X POST http://127.0.0.1:18080/devices/device-001/commands \
-H 'Content-Type: application/json' \
-H 'Idempotency-Key: lesson8-ping-001' \
--data '{"command":"ping"}'
# Replace cmd-example with the command_id returned by POST.
curl -i http://127.0.0.1:18080/commands/cmd-example</code></pre>
<p>Expect HTTP <code>202</code> from POST, then later application status from GET after the simulator responds. Repeat the same POST with the same idempotency key and body; it should return the same command ID. PUBACK alone is not device completion. Stop both clients with Ctrl+C, then stop the broker with <code>docker compose -p mqtt5-learning stop broker</code>. Ask me a follow-up if the observed status or retry result differs from your prediction.</p>
<section class="card callout">
<h2>✅ Takeaway</h2>
<p><strong>202 = API admission. PUBACK = MQTT publication acknowledgement. Device result = application-reported outcome.</strong></p>
<p>Keep those facts separate in logs, API fields, dashboards, and incident reports. Ask me a follow-up question if any boundary still feels fuzzy.</p>
</section>
<footer>
<p>↩️ Previous: <a href="../course-source/07-command-response.md">Section 7 — command/response</a> · 📚 Reference: <a href="../reference/http-mqtt-evidence.html">HTTP and MQTT evidence</a></p>
<p>Previous: <a href="0007-command-state.html">Lesson 7 — command identity and state</a> · Next: <a href="0009-session-recovery.html">Lesson 9 — session recovery</a> · Full implementation: <a href="../course-source/08-http-commands.md">Section 8 source lesson</a>.</p>
<p>Primary sources: <a href="https://datatracker.ietf.org/doc/rfc9110/">RFC 9110 §15.3.3 — 202 Accepted</a>; <a href="https://docs.oasis-open.org/mqtt/mqtt/v5.0/mqtt-v5.0.html">OASIS MQTT 5.0 Standard — PUBACK</a>; <a href="https://pkg.go.dev/net/http">Go `net/http` documentation</a>. See <a href="../RESOURCES.md">all workspace resources</a>.</p>
</footer>
</main>
</body>
</html>
teaching/lessons/0009-session-recovery.html
+181
View File
@@ -0,0 +1,181 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8"><meta name="viewport" content="width=device-width, initial-scale=1">
<title>Lesson 9 — MQTT Sessions and Recovery</title>
<link rel="stylesheet" href="../assets/lesson.css"><script src="../assets/quiz.js" defer></script>
</head>
<body><main class="page">
<p class="eyebrow">Go MQTT Vending Backend · Lesson 9 · 10 minutes</p>
<h1>Recover broker session work across disconnects</h1>
<p class="dek">Make MQTT session policy visible in the connection code, then test what survives network loss and process restart.</p>
<div class="meta"><span class="pill">🎯 One skill: trace session ownership</span><span class="pill">🧠 Predict reconnect behavior</span><span class="pill">🧪 Disposable broker lab</span></div>
<section class="card callout"><p class="question">❓ Start with the system story</p><p>A simulator disconnects after subscribing. The backend publishes a QoS 1 ping. What must the broker still have when the simulator reconnects?</p><p>Predict the Client ID, session expiry, subscription, and CONNACK Session Present before reading the implementation.</p></section>
<h2>🗺️ File overview</h2>
<p><code>internal/mqtt/client.go</code> owns the AutoPaho connection, MQTT subscriptions, and the inbox callback. This lesson edits that file to use a stable-session profile and record the broker's CONNACK evidence. The backend and simulator already load distinct, stable Client IDs from configuration.</p>
<h3>🔌 <code>Start</code> — short summary</h3>
<p>Creates AutoPaho with Clean Start disabled for the initial connection and a 60-second Session Expiry Interval. It logs Session Present on every connection and restores the configured subscriptions.</p>
<h3>📤 <code>Publish</code> and <code>Stop</code> — short summary</h3>
<p><code>Publish</code> sends QoS 1 and checks the PUBACK reason. <code>Stop</code> performs a bounded graceful disconnect.</p>
<h2>🧩 Complete file: <code>internal/mqtt/client.go</code></h2>
<p>Replace the file with this full version. It preserves the lesson's bounded inbox and adds session logging; it does not make application command records durable.</p>
<pre><code>package mqtt
import (
"context"
"fmt"
"log"
"net/url"
"time"
"example.com/mqtt5-learning/internal/config"
"example.com/mqtt5-learning/internal/contracts"
"github.com/eclipse/paho.golang/autopaho"
"github.com/eclipse/paho.golang/paho"
)
type Message struct {
Topic string
Payload []byte
Retain bool
}
const sessionExpirySeconds uint32 = 60
func Start(ctx context.Context, cfg config.Config, topics []contracts.TopicFilter, inbox chan&lt;- Message) (*autopaho.ConnectionManager, error) {
broker, err := url.Parse(cfg.MQTT.URL)
if err != nil {
return nil, fmt.Errorf("parse broker URL: %w", err)
}
initialSubscription := make(chan error, 1)
manager, err := autopaho.NewConnection(ctx, autopaho.ClientConfig{
ServerUrls: []*url.URL{broker},
KeepAlive: 20,
CleanStartOnInitialConnection: false, // Resume this Client ID's session when one exists.
SessionExpiryInterval: sessionExpirySeconds,
ConnectUsername: cfg.MQTT.Username,
ConnectPassword: []byte(cfg.MQTT.Password),
ClientID: cfg.MQTT.ClientID,
OnConnectionUp: func(cm *autopaho.ConnectionManager, ack *paho.Connack) {
if ack != nil {
log.Printf("mqtt connected client_id=%s session_present=%t", cfg.MQTT.ClientID, ack.SessionPresent)
}
subCtx, cancel := context.WithTimeout(ctx, 5*time.Second)
defer cancel()
subscriptions := make([]paho.SubscribeOptions, 0, len(topics))
for _, topic := range topics {
subscriptions = append(subscriptions, paho.SubscribeOptions{
Topic: topic.String(), QoS: 1, RetainAsPublished: true,
})
}
suback, subErr := cm.Subscribe(subCtx, &amp;paho.Subscribe{Subscriptions: subscriptions})
if subErr == nil {
if suback == nil || len(suback.Reasons) != len(topics) {
subErr = fmt.Errorf("missing or incomplete SUBACK")
} else {
for i, reason := range suback.Reasons {
if reason &gt; 1 {
subErr = fmt.Errorf("subscription %s rejected: 0x%02x", topics[i], reason)
break
}
}
}
}
if subErr != nil {
log.Printf("mqtt subscription failed: %v", subErr)
} else {
log.Printf("mqtt subscriptions ready client_id=%s count=%d", cfg.MQTT.ClientID, len(topics))
}
select {
case initialSubscription &lt;- subErr:
default: // Reconnect callbacks must never block on this one-time startup signal.
}
},
OnConnectionDown: func() bool {
log.Printf("mqtt connection down client_id=%s; reconnecting", cfg.MQTT.ClientID)
return true
},
OnConnectError: func(err error) { log.Printf("mqtt connect error: %v", err) },
OnClientError: func(err error) { log.Printf("mqtt client error: %v", err) },
OnServerDisconnect: func(packet *paho.Disconnect) {
log.Printf("mqtt server disconnect reason=0x%02x", packet.ReasonCode)
},
OnPublishReceived: []func(paho.PublishReceived) (bool, error){
func(received paho.PublishReceived) (bool, error) {
packet := received.Packet
message := Message{
Topic: packet.Topic, Payload: append([]byte(nil), packet.Payload...), Retain: packet.Retain,
}
select {
case inbox &lt;- message:
default:
// Automatic acknowledgement may still complete this delivery.
log.Printf("mqtt inbox full; application message dropped topic=%s", packet.Topic)
}
return true, nil
},
},
})
if err != nil {
return nil, fmt.Errorf("start AutoPaho: %w", err)
}
startup, cancel := context.WithTimeout(ctx, cfg.ConnectTimeout())
defer cancel()
if err := manager.AwaitConnection(startup); err != nil {
Stop(manager)
return nil, fmt.Errorf("await MQTT connection: %w", err)
}
select {
case err := &lt;-initialSubscription:
if err != nil {
Stop(manager)
return nil, fmt.Errorf("await MQTT subscriptions: %w", err)
}
case &lt;-startup.Done():
Stop(manager)
return nil, fmt.Errorf("await MQTT subscriptions: %w", startup.Err())
}
return manager, nil
}
func Publish(ctx context.Context, manager *autopaho.ConnectionManager, topic contracts.TopicName, payload []byte) error {
operation, cancel := context.WithTimeout(ctx, 3*time.Second)
defer cancel()
ack, err := manager.Publish(operation, &amp;paho.Publish{
Topic: topic.String(), Payload: payload, QoS: 1,
Properties: &amp;paho.PublishProperties{ContentType: "application/json"},
})
if err != nil {
return fmt.Errorf("publish: %w", err)
}
if ack == nil {
return fmt.Errorf("missing QoS 1 publish acknowledgement")
}
if ack.ReasonCode &gt;= 0x80 {
return fmt.Errorf("publish rejected: reason=0x%02x", ack.ReasonCode)
}
log.Printf("mqtt publish acknowledged topic=%s reason=0x%02x", topic, ack.ReasonCode)
return nil
}
func Stop(manager *autopaho.ConnectionManager) {
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
if err := manager.Disconnect(ctx); err != nil {
log.Printf("MQTT shutdown did not finish within deadline: %v", err)
}
}
</code></pre>
<div class="card"><h3>🔎 Details: the session guarantees</h3><ul><li><code>CleanStartOnInitialConnection: false</code> asks AutoPaho to resume this Client ID's session when available. AutoPaho uses Clean Start false on subsequent reconnects regardless; process restart creates a new manager and applies the configured initial behavior again.</li><li><code>SessionExpiryInterval</code> is seconds. A 60-second session can disappear after the interval or broker loss if persistence is disabled.</li><li><code>SessionPresent</code> reports broker session resumption only. This callback logs it separately from subscription readiness.</li><li>Subscribing again on connection-up is deliberate for a simple lab, but duplicate SUBSCRIBE behavior must be observed. It does not register another local Go callback.</li><li>The helper still uses automatic acknowledgement and a RAM inbox. Session continuity cannot repair a message dropped after callback admission, and it cannot restore the backend's in-memory command record.</li></ul></div>
<h2>🧪 Run the experiment matrix</h2>
<p>Rebuild both programs after changing the constant or Clean Start setting. Use one simulator per Client ID. Record prediction, broker settings, CONNACK Session Present, SUBACK results, and command delivery in <code>journal/09-sessions.md</code>.</p>
<ol><li>Connect and subscribe, stop the simulator gracefully, publish a QoS 1 ping while it is offline, then reconnect with the same Client ID before 60 seconds.</li><li>Repeat with Clean Start true on the initial connection, then with a changed Client ID, then after the expiry interval.</li><li>Compare broker restart with persistence disabled and enabled, using the isolated Compose override described in the full source lesson.</li><li>Keep the backend command ID and simulator effect counter separate from Session Present.</li></ol>
<h2>🧭 Session queue versus retained latest value</h2>
<table><thead><tr><th>Delivery mechanism</th><th>What it represents</th><th>Who can receive it later</th></tr></thead><tbody><tr><td>Queued nonretained QoS 1 publication</td><td>One event for an existing matching session subscription</td><td>That session before expiry, if broker state and QoS permit</td></tr><tr><td>Retained publication</td><td>The latest value stored for a topic</td><td>A new subscriber, even with a fresh Client ID</td></tr></tbody></table>
<div class="card"><p><strong>Safe lab comparison:</strong> After the queued-ping run, reconnect the original simulator and keep it online so the retained probe cannot also queue in its offline session. Publish a synthetic retained ping to its lab command topic with the authorized lab backend identity; the Lesson 7 simulator must reject it before any effect. Subscribe a fresh observer Client ID with the lab simulator identity; it should receive the stored value immediately without a prior session. Clear it immediately with an empty retained publication (`mosquitto_pub -r -n`), then verify a new subscription gets no replay. Use no physical device or production credential, and do not leave retained commands in the lab.</p><p>A retained message is the latest value, not event history. Do not use retained publications for one-shot commands.</p></div>
<div class="card"><h3>✅ Done when</h3><p>You can explain which subscriptions and eligible queued messages the broker preserved and which Go state was lost. Session Present is never used as proof of command completion.</p></div>
<details class="card"><summary><strong>📋 Short self-check answers</strong></summary><ol><li>Stable Client ID, resumable unexpired session, matching QoS subscription, eligible nonexpired QoS publication, and available broker state.</li><li>Session Present means the broker resumed session state, not application memory or device effect.</li><li>A fresh session after process restart is different from an automatic reconnect within the existing manager.</li></ol></details>
<p>📚 Full session experiment details: <a href="../course-source/09-sessions-recovery.md">Section 9 source lesson</a>. Primary references: <a href="https://docs.oasis-open.org/mqtt/mqtt/v5.0/mqtt-v5.0.html">OASIS MQTT 5.0 Standard</a> and <a href="https://pkg.go.dev/github.com/eclipse/paho.golang/autopaho@v0.23.0">Paho AutoPaho v0.23.0 API</a>.</p>
<p>If the observed session behavior differs from your prediction, ask me to trace the CONNECT/CONNACK and SUBSCRIBE/SUBACK evidence with you.</p>
<footer><p>Previous: <a href="0008-http-acceptance-vs-completion.html">Lesson 8</a> · Next: <a href="0010-expiry-presence-state.html">Lesson 10 — expiry and presence</a>.</p></footer>
</main></body></html>
teaching/lessons/0010-expiry-presence-state.html
+290
View File
@@ -0,0 +1,290 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8"><meta name="viewport" content="width=device-width, initial-scale=1">
<title>Lesson 10 — Expiry, Presence, and Desired State</title>
<link rel="stylesheet" href="../assets/lesson.css"><script src="../assets/quiz.js" defer></script>
</head>
<body><main class="page">
<p class="eyebrow">Go MQTT Vending Backend · Lesson 10 · 12 minutes</p>
<h1>Give stale work an expiry; treat presence as evidence</h1>
<p class="dek">Use separate clocks for publication lifetime, application deadlines, sessions, and freshness. Keep desired state distinct from one-shot commands.</p>
<div class="meta"><span class="pill">🎯 One skill: assign timer ownership</span><span class="pill">🧠 Predict stale delivery</span><span class="pill">🧪 Simulator + broker</span></div>
<section class="card callout"><p class="question">❓ Start with the system story</p><p>A vending machine has been offline for an hour. A queued “apply price update” arrives after it reconnects, but the price is obsolete. Which timer prevents stale delivery, and which check prevents late execution after delivery?</p><p>Answer before reading: broker policy and device policy are separate checks.</p></section>
<h2>⏱️ One timer, one responsibility</h2>
<div class="two-col"><article class="card"><h3>MQTT Message Expiry</h3><p>Bounds how long a PUBLISH remains eligible for onward delivery. The broker may discard expired queued work.</p></article><article class="card"><h3>Application deadline</h3><p>An absolute deadline in the command contract lets the device reject work that is already delivered but too old to execute.</p></article><article class="card"><h3>Session Expiry</h3><p>Controls how long the broker may preserve the client's session after disconnect.</p></article><article class="card"><h3>Heartbeat freshness</h3><p>Backend receive time measures how recently an application report arrived. It is not a broker connection probe or health diagnosis.</p></article></div>
<p>Keep Alive detects a lack of MQTT control traffic; it does not show that the application is producing heartbeats. A publish context timeout ends your wait, not necessarily the remote operation. Will Delay delays an eligible Will; it does not certify why the device disappeared.</p>
<form class="card quiz" data-quiz data-correct="b" data-explanation="Message Expiry bounds queued delivery; the device deadline rejects work that is already delivered but too old to execute." data-retry="One policy bounds broker delivery; the other is enforced by the application before the action.">
<p><strong>🧠 Check the boundary:</strong> What combination protects against a command that waits in the broker and then spends time in a local queue?</p>
<label><input type="radio" name="q" value="a"> Session Expiry and Retain Handling protect delayed command execution</label><br><label><input type="radio" name="q" value="b"> MQTT Message Expiry and device deadline protect delayed command execution</label><br><label><input type="radio" name="q" value="c"> Keep Alive and Will Delay protect delayed command execution</label><br><button type="submit">Check answer</button><p data-feedback aria-live="polite"></p>
</form>
<h2>📌 Retained desired state is not an event queue</h2>
<p>A retained publication stores the latest value for one topic and can initialize a new subscriber. It is useful for replaceable desired configuration, such as catalog revision 9. A retained “dispense” or “run calibration once” action can replay to a later subscriber, so one-shot commands should be nonretained.</p>
<p>For desired state, keep the highest revision at the simulator and reject revision 8 after revision 9. The device reports the revision it applied. A PUBACK only confirms the relevant QoS hop; it does not prove application. An empty retained publication removes the stored value for that topic; it does not retract copies already delivered.</p>
<h3>📄 Complete file: <code>internal/contracts/topic.go</code></h3>
<p><strong>File overview:</strong> This is the single source of truth for concrete publish names, controlled subscription filters, and inbound topic parsing. The added kinds support presence and desired/reported state.</p>
<p><strong><code>DeviceTopic</code> / <code>DeviceFilter</code> summary:</strong> Build validated exact names and fleet filters. <strong><code>ParseTopic</code> summary:</strong> Derive the device and kind from the shared prefix and reject unknown routes.</p>
<details class="card" open><summary><strong>🧩 Full replacement source</strong></summary><pre><code>package contracts
import (
"fmt"
"strings"
)
type TopicName string
type TopicFilter string
type Kind string
const (
KindTelemetry Kind = "telemetry"
KindCommands Kind = "commands"
KindResponses Kind = "responses"
KindSales Kind = "sales"
KindPresence Kind = "presence"
KindDesired Kind = "desired"
KindReported Kind = "reported"
)
const topicPrefix = "lab/v1/devices/"
type Topic struct {
DeviceID string
Kind Kind
}
func (t TopicName) String() string { return string(t) }
func (f TopicFilter) String() string { return string(f) }
func DeviceTopic(deviceID string, kind Kind) (TopicName, error) {
if !validID(deviceID) {
return "", fmt.Errorf("invalid device ID %q", deviceID)
}
if !supportedKind(kind) {
return "", fmt.Errorf("unsupported topic kind %q", kind)
}
return TopicName(topicPrefix + deviceID + "/" + string(kind)), nil
}
func DeviceFilter(kind Kind) (TopicFilter, error) {
if !supportedKind(kind) {
return "", fmt.Errorf("unsupported topic kind %q", kind)
}
return TopicFilter(topicPrefix + "+/" + string(kind)), nil
}
func DeviceSubscription(deviceID string, kind Kind) (TopicFilter, error) {
topic, err := DeviceTopic(deviceID, kind)
if err != nil {
return "", err
}
return TopicFilter(topic.String()), nil
}
func ParseTopic(value string) (Topic, error) {
if !strings.HasPrefix(value, topicPrefix) {
return Topic{}, fmt.Errorf("topic must be %s{deviceID}/{kind}", topicPrefix)
}
parts := strings.Split(strings.TrimPrefix(value, topicPrefix), "/")
if len(parts) != 2 || !validID(parts[0]) {
return Topic{}, fmt.Errorf("topic must be %s{deviceID}/{kind}", topicPrefix)
}
kind := Kind(parts[1])
if !supportedKind(kind) {
return Topic{}, fmt.Errorf("unsupported topic kind %q", kind)
}
return Topic{DeviceID: parts[0], Kind: kind}, nil
}
func supportedKind(kind Kind) bool {
switch kind {
case KindTelemetry, KindCommands, KindResponses, KindSales,
KindPresence, KindDesired, KindReported:
return true
default:
return false
}
}
func validID(value string) bool {
if value == "" || len(value) &gt; 128 {
return false
}
for _, r := range value {
if !(r &gt;= 'a' &amp;&amp; r &lt;= 'z' || r &gt;= 'A' &amp;&amp; r &lt;= 'Z' ||
r &gt;= '0' &amp;&amp; r &lt;= '9' || strings.ContainsRune("._-", r)) {
return false
}
}
return true
}
</code></pre></details>
<h2>🧩 Complete file: expiry and command deadline policy</h2>
<p><strong>File overview:</strong> Create <code>internal/mqtt/expiry.go</code> to build a QoS 1 publication with explicit expiry, and <code>internal/deviceview/deadline.go</code> to reject stale commands at execution time. Each file is complete and has its own imports, types, and errors.</p>
<details class="card" open><summary><strong>📄 <code>internal/mqtt/expiry.go</code> — complete file</strong></summary><pre><code>package mqtt
import (
"fmt"
"example.com/mqtt5-learning/internal/contracts"
"github.com/eclipse/paho.golang/paho"
)
// ExpiringPublish builds a publication whose broker-queued lifetime is bounded.
// The device must still check its application deadline before executing it.
func ExpiringPublish(topic contracts.TopicName, payload []byte, expirySeconds uint32) (*paho.Publish, error) {
if topic == "" {
return nil, fmt.Errorf("topic is required")
}
if expirySeconds == 0 {
return nil, fmt.Errorf("expirySeconds must be greater than zero")
}
seconds := expirySeconds
return &amp;paho.Publish{
Topic: topic.String(), Payload: append([]byte(nil), payload...), QoS: 1,
Properties: &amp;paho.PublishProperties{
ContentType: "application/json",
MessageExpiry: &amp;seconds,
},
}, nil
}
</code></pre></details>
<details class="card" open><summary><strong>📄 <code>internal/deviceview/deadline.go</code> — complete file</strong></summary><pre><code>package deviceview
import (
"fmt"
"time"
)
// CheckCommandDeadline rejects a command before the simulator starts its effect.
// The absolute deadline protects against time spent in local queues after delivery.
func CheckCommandDeadline(deadline string, now time.Time) error {
if deadline == "" {
return fmt.Errorf("command deadline is required")
}
when, err := time.Parse(time.RFC3339, deadline)
if err != nil {
return fmt.Errorf("parse command deadline: %w", err)
}
if !now.Before(when) {
return fmt.Errorf("command deadline has passed")
}
return nil
}
</code></pre></details>
<h2>🧩 Complete file: simulator's monotonic desired state</h2>
<p><strong>File overview:</strong> Create <code>internal/deviceview/revisions.go</code> to keep a per-device revision and prevent older desired state from replacing newer applied state.</p>
<p><strong><code>Apply</code> summary:</strong> Lock, compare the incoming revision, and update only when it is newer. Return an explicit outcome so duplicate and stale deliveries are observable.</p>
<details class="card" open><summary><strong>📄 <code>internal/deviceview/revisions.go</code> — complete file</strong></summary><pre><code>package deviceview
import "sync"
type RevisionOutcome uint8
const (
RevisionApplied RevisionOutcome = iota + 1
RevisionDuplicate
RevisionStale
)
type RevisionStore struct {
mu sync.Mutex
current map[string]uint64
}
func NewRevisionStore() *RevisionStore {
return &amp;RevisionStore{current: make(map[string]uint64)}
}
func (s *RevisionStore) Apply(deviceID string, revision uint64) RevisionOutcome {
s.mu.Lock()
defer s.mu.Unlock()
current, exists := s.current[deviceID]
if exists &amp;&amp; revision == current {
return RevisionDuplicate
}
if exists &amp;&amp; revision &lt; current {
return RevisionStale
}
s.current[deviceID] = revision
return RevisionApplied
}
func (s *RevisionStore) Current(deviceID string) (uint64, bool) {
s.mu.Lock()
defer s.mu.Unlock()
revision, ok := s.current[deviceID]
return revision, ok
}
</code></pre></details>
<h2>🧭 File overview: <code>internal/mqtt/presence.go</code></h2>
<p>Create this complete helper file for the Last Will portion. <code>OfflineWill</code> builds the message prepared during CONNECT; <code>OnlinePresence</code> builds the retained publication sent after a successful connection. Both use one validated presence topic and stable device/boot identity.</p>
<h3>📴 <code>OfflineWill</code> — short summary</h3><p>Returns a retained QoS 1 offline Will with a five-second Will Delay property.</p>
<h3>🟢 <code>OnlinePresence</code> — short summary</h3><p>Creates the retained online publication for the same device and boot identity.</p>
<details class="card" open><summary><strong>📄 <code>internal/mqtt/presence.go</code> — complete file</strong></summary><pre><code>package mqtt
import (
"encoding/json"
"fmt"
"time"
"example.com/mqtt5-learning/internal/contracts"
"github.com/eclipse/paho.golang/paho"
)
type Presence struct {
SchemaVersion int `json:"schema_version"`
MessageID string `json:"message_id"`
DeviceID string `json:"device_id"`
BootID string `json:"boot_id"`
Status string `json:"status"`
OccurredAt string `json:"occurred_at"`
}
func OfflineWill(deviceID, bootID string) (*paho.WillMessage, *paho.WillProperties, error) {
if deviceID == "" || bootID == "" {
return nil, nil, fmt.Errorf("device ID and boot ID are required")
}
topic, err := contracts.DeviceTopic(deviceID, contracts.KindPresence)
if err != nil {
return nil, nil, fmt.Errorf("build presence topic: %w", err)
}
body, err := json.Marshal(Presence{
SchemaVersion: contracts.SchemaVersionOne,
MessageID: "will-" + bootID, DeviceID: deviceID, BootID: bootID,
Status: "offline", OccurredAt: time.Now().UTC().Format(time.RFC3339),
})
if err != nil { return nil, nil, fmt.Errorf("encode offline Will: %w", err) }
delay := uint32(5)
return &amp;paho.WillMessage{Topic: topic.String(), QoS: 1, Retain: true, Payload: body},
&amp;paho.WillProperties{WillDelayInterval: &amp;delay, ContentType: "application/json"}, nil
}
func OnlinePresence(deviceID, bootID string, now time.Time) (*paho.Publish, error) {
if deviceID == "" || bootID == "" {
return nil, fmt.Errorf("device ID and boot ID are required")
}
topic, err := contracts.DeviceTopic(deviceID, contracts.KindPresence)
if err != nil { return nil, fmt.Errorf("build presence topic: %w", err) }
body, err := json.Marshal(Presence{
SchemaVersion: contracts.SchemaVersionOne,
MessageID: "online-" + bootID, DeviceID: deviceID, BootID: bootID,
Status: "online", OccurredAt: now.UTC().Format(time.RFC3339),
})
if err != nil { return nil, fmt.Errorf("encode online presence: %w", err) }
return &amp;paho.Publish{
Topic: topic.String(), Payload: body, QoS: 1, Retain: true,
Properties: &amp;paho.PublishProperties{ContentType: "application/json"},
}, nil
}
</code></pre></details>
<div class="card"><h3>🔎 Details: a Will timestamp is not a failure timestamp</h3><p>The Will body is prepared at CONNECT, so <code>OccurredAt</code> above is the preparation time. The backend should record its own receipt time and use it when evaluating freshness. Include a boot ID so stale presence from a previous process can be distinguished. The simulator's planned shutdown must publish retained offline presence before sending normal DISCONNECT if the application needs that final state; graceful DISCONNECT suppresses the Will.</p></div>
<h2>🧪 Predict, then observe</h2>
<ol><li>Set a persistent simulator session, disconnect it, publish a QoS 1 command with a short Message Expiry Interval, then reconnect after expiry.</li><li>Repeat with a longer expiry but an already-passed application deadline. The simulator should receive and reject it as stale.</li><li>Publish desired revision 9, then revision 8; verify reported state remains at 9.</li><li>Compare graceful shutdown, abrupt simulator termination, reconnect before Will Delay, and broker stop. Record the exact fault and evidence.</li><li>Clear the retained value and subscribe fresh. Verify no retained replay.</li></ol>
<div class="card"><h3>❤️ Presence has limits</h3><p>Retained “online” is the broker's latest stored assertion. A Last Will is published by the broker after an eligible unexpected loss; it is not sent by the dead process. A clean DISCONNECT normally suppresses the Will. Combine connection events with backend-received heartbeat freshness and device-reported health rather than treating one as the other.</p><p><strong>What could go wrong?</strong> A stale retained online value can outlive a graceful shutdown; Will Delay may interact with session expiry; a command delivered just before its expiry may still sit in a local queue; clocks can disagree. Record backend receive time and define the device deadline policy.</p></div>
<h2>🔎 Details: ownership and limitations</h2><p><code>ExpiringPublish</code> creates protocol metadata only; wire it into the existing publisher and handle the returned PUBACK as transport evidence. <code>CheckCommandDeadline</code> must run immediately before the simulator effect, after local queue wait. <code>RevisionStore</code> is intentionally RAM-only, so a simulator restart loses its revision; PostgreSQL or device flash is needed for durable state. The will is a broker publication prepared during CONNECT; normal DISCONNECT suppresses it. These helpers do not replace ACL updates, topic/body validation, or device response reporting.</p>
<h2>✅ Done when</h2><p>Demonstrate expired queued work, device-side stale rejection, retained replacement/deletion, stale revision rejection, and the difference between graceful and unexpected disconnect evidence.</p>
<details class="card"><summary><strong>📋 Short self-check answers</strong></summary><ol><li>Broker expiry limits onward delivery; an application deadline limits whether the device starts the action.</li><li>Retained messages represent latest state, not historical event delivery.</li><li>Heartbeat freshness says a recent report arrived, not that the device is healthy or the broker currently has a connection.</li></ol></details>
<p>📚 Detailed experiments: <a href="../course-source/10-expiry-presence-state.md">Section 10 source lesson</a>. Primary references: <a href="https://docs.oasis-open.org/mqtt/mqtt/v5.0/mqtt-v5.0.html">OASIS MQTT 5.0 Standard</a>, <a href="https://pkg.go.dev/github.com/eclipse/paho.golang/paho@v0.23.0">Paho v0.23.0 API</a>, and <a href="https://mosquitto.org/man/mosquitto_pub-1.html">Mosquitto publish client manual</a>.</p>
<p>Ask me to trace an expiry or presence result if the packet evidence differs from your prediction.</p>
<footer><p>Previous: <a href="0009-session-recovery.html">Lesson 9 — session recovery</a> · Next: <a href="0011-security-boundaries.html">Lesson 11 — security boundaries</a>.</p></footer>
</main></body></html>
teaching/lessons/0011-security-boundaries.html
+71
View File
@@ -0,0 +1,71 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8"><meta name="viewport" content="width=device-width, initial-scale=1">
<title>Lesson 11 — MQTT Security Boundaries</title>
<link rel="stylesheet" href="../assets/lesson.css"><script src="../assets/quiz.js" defer></script>
</head>
<body><main class="page">
<p class="eyebrow">Go MQTT Vending Backend · Lesson 11 · 12 minutes</p>
<h1>Trace security checks from socket to business action</h1>
<p class="dek">TLS, MQTT credentials, topic ACLs, payload validation, and HTTP user permissions answer different questions.</p>
<div class="meta"><span class="pill">🎯 One skill: locate a trust boundary</span><span class="pill">🧠 Predict which layer rejects</span><span class="pill">🧪 Negative tests</span></div>
<section class="card callout"><p class="question">❓ Start with the system story</p><p>A client with a valid username publishes to another device's topic and claims that device's ID in JSON. Which component can reject the topic access, and which component checks the body claim?</p><p>Pause and trace the request from connection setup through business authorization.</p></section>
<h2>🔐 The checks run in sequence</h2>
<div class="flow" aria-label="security checks"><div class="step">TLS<br><strong>server identity + encryption</strong></div><span class="arrow">→</span><div class="step">MQTT CONNECT<br><strong>client authentication</strong></div><span class="arrow">→</span><div class="step">Broker ACL<br><strong>topic authorization</strong></div><span class="arrow">→</span><div class="step">Go service<br><strong>contract + business rules</strong></div><span class="arrow">→</span><div class="step">HTTP API<br><strong>user's device permission</strong></div></div>
<p>A JSON <code>device_id</code> is a claim, not authenticated identity. A Client ID names a session; it is not a secret. The broker usually does not attach the MQTT publisher username to a forwarded PUBLISH, so bind unique client credentials to narrow topic ACLs, then validate topic/body agreement in the application.</p>
<form class="card quiz" data-quiz data-correct="a" data-explanation="The broker authenticates the MQTT client and enforces its topic ACL. The application validates topic and payload identity agreement after delivery; neither a payload field nor TLS does both jobs." data-retry="Distinguish broker authorization from application contract validation.">
<p><strong>🧠 Check the boundary:</strong> Which check enforces that a device credential can publish only to its assigned device namespace?</p>
<label><input type="radio" name="q" value="a"> Broker topic ACL tied to the authenticated client identity</label><br><label><input type="radio" name="q" value="b"> TLS encryption alone</label><br><label><input type="radio" name="q" value="c"> The JSON device_id field alone</label><br><button type="submit">Check answer</button><p data-feedback aria-live="polite"></p>
</form>
<h2>🌐 Verify the broker before sending credentials</h2>
<p>For TLS, the certificate chain must lead to a trusted CA and its subject alternative name must match the host/IP used by the client. A trusted certificate for <code>broker.lab</code> can still fail when the client verifies <code>127.0.0.1</code>. Correct the URL or certificate identity; never turn off certificate verification to make the experiment pass.</p>
<p>Server TLS encrypts the channel and verifies the broker. Username/password authenticates the MQTT client. Per-device ACLs authorize topics. Mutual TLS adds client-certificate authentication only when the broker maps that identity to a deliberate policy. These choices do not authorize an HTTP user to operate a particular vending machine.</p>
<h3>📄 Complete new file: <code>internal/mqtt/tls.go</code></h3>
<p><strong>File overview:</strong> This file reads a public CA certificate and builds a client TLS configuration that verifies both the chain and broker hostname. It does not load client private keys or disable verification.</p>
<p><strong><code>NewVerifiedTLSConfig</code> summary:</strong> Rejects empty inputs and invalid PEM, then returns TLS 1.2+ settings with an explicit <code>ServerName</code>.</p>
<p><strong>Details:</strong> The caller supplies the DNS name or IP address used by the broker certificate SAN. This lesson's implementation is deliberately a complete, standalone trust-config helper; it does not replace the existing MQTT client or config loader. The CA file is public trust material; never embed a private CA key in the client.</p>
<details class="card" open><summary><strong>🧩 Full file contents</strong></summary><pre><code>package mqtt
import (
"crypto/tls"
"crypto/x509"
"fmt"
"os"
"strings"
)
// NewVerifiedTLSConfig builds a client config that verifies the broker's
// certificate chain and the exact DNS name or IP used by the client.
func NewVerifiedTLSConfig(caPath, serverName string) (*tls.Config, error) {
if strings.TrimSpace(caPath) == "" {
return nil, fmt.Errorf("CA certificate path is required")
}
if strings.TrimSpace(serverName) == "" {
return nil, fmt.Errorf("broker server name is required")
}
caPEM, err := os.ReadFile(caPath)
if err != nil {
return nil, fmt.Errorf("read broker CA: %w", err)
}
roots := x509.NewCertPool()
if !roots.AppendCertsFromPEM(caPEM) {
return nil, fmt.Errorf("broker CA file contains no valid certificates")
}
return &amp;tls.Config{
RootCAs: roots,
ServerName: serverName,
MinVersion: tls.VersionTLS12,
}, nil
}
</code></pre></details>
<p><strong>Integration exercise:</strong> In the existing <code>autopaho.ClientConfig</code>, pass this helper's result through <code>TlsCfg</code> and make the configured broker URL and certificate SAN name agree. Treat changes to the client and config loader as separate files: show each whole updated file in your work, including imports and validation, rather than pasting a partial field diff. Do not use <code>InsecureSkipVerify</code>. Until both files are updated, this helper alone does not make the running client use TLS.</p>
<h2>🧪 Change one input at a time</h2>
<table><thead><tr><th>Change</th><th>Expected boundary</th><th>Evidence</th></tr></thead><tbody><tr><td>Wrong CA</td><td>TLS trust</td><td>No authenticated MQTT session</td></tr><tr><td>Wrong certificate hostname</td><td>TLS server identity</td><td>Name mismatch</td></tr><tr><td>Wrong password</td><td>CONNECT authentication</td><td>Connection rejected</td></tr><tr><td>Valid login, cross-device topic</td><td>Broker ACL</td><td>Denied publish/subscribe and broker evidence</td></tr><tr><td>Allowed topic, body ID mismatch</td><td>Application validation</td><td>Rejected before state mutation</td></tr><tr><td>Valid HTTP user, unauthorized target</td><td>API authorization</td><td>Denied before command creation/publication</td></tr></tbody></table>
<div class="card"><h3>📬 Response Topic is also a trust boundary</h3><p>The simulator should compare a command's Response Topic with its exact authorized response topic before publishing. Correlation Data helps match a response to a request; it does not authorize the destination or authenticate the publisher. An application-level denial also does not reverse an MQTT acknowledgement already sent.</p><p>Rotate credentials by verifying fresh connections with both old and new credentials and checking subscription restoration. Updating a password file alone does not prove existing connections were closed.</p></div>
<h2>✅ Done when</h2><p>Run the negative matrix in the disposable lab and show that each failure is rejected at its intended boundary. A broad <code>#</code> ACL is not a fix for an authorization failure.</p>
<details class="card"><summary><strong>📋 Short self-check answers</strong></summary><ol><li>Broker ACL: which authenticated MQTT client may use which topic.</li><li>Application: whether the delivered topic and payload claims agree and whether the operation is valid.</li><li>API authorization: whether this HTTP principal may act on the requested device.</li></ol></details>
<p>📚 Full failure matrix: <a href="../course-source/11-security-boundaries.md">Section 11 source lesson</a>. Primary references: <a href="https://pkg.go.dev/crypto/tls#Config">Go TLS Config API</a>, <a href="https://mosquitto.org/man/mosquitto-conf-5.html">Mosquitto configuration manual</a>, and <a href="https://docs.oasis-open.org/mqtt/mqtt/v5.0/mqtt-v5.0.html">OASIS MQTT 5.0 Standard</a>.</p>
<p>Ask me to inspect a TLS or ACL failure with you; the layer and evidence should agree.</p>
<footer><p>Previous: <a href="0010-expiry-presence-state.html">Lesson 10 — expiry and presence</a> · Optional: <a href="0011b-dynamic-security.html">Lesson 11B — Dynamic Security</a> · Next: <a href="0012-vending-workflows.html">Lesson 12 — vending workflows</a>.</p></footer>
</main></body></html>
teaching/lessons/0011b-dynamic-security.html
+76
View File
@@ -0,0 +1,76 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8"><meta name="viewport" content="width=device-width, initial-scale=1">
<title>Lesson 11B — Mosquitto Dynamic Security</title>
<link rel="stylesheet" href="../assets/lesson.css"><script src="../assets/quiz.js" defer></script>
</head>
<body><main class="page">
<p class="eyebrow">Go MQTT Vending Backend · Lesson 11B · Optional extension</p>
<h1>Provision per-device broker permissions at runtime</h1>
<p class="dek">Use Mosquitto Dynamic Security to create device clients and roles without editing static password and ACL files.</p>
<div class="meta"><span class="pill">🎯 One skill: separate provisioning from runtime traffic</span><span class="pill">🧠 Predict revocation behavior</span><span class="pill">🧪 Isolated broker config</span></div>
<section class="card callout"><p class="question">❓ Start with the provisioning story</p><p>A new vending machine is installed after Mosquitto starts. How can it receive an individual credential and narrow topic permissions while the broker is running?</p><p>Name the privileged actor and the separate device identity before continuing.</p></section>
<h2>🧭 Three identities, three responsibilities</h2>
<ol><li><strong>Provisioning operator/service:</strong> authenticates with a protected management identity and uses the Dynamic Security control API.</li><li><strong>Device client:</strong> receives its own account and a role limited to its topic namespace.</li><li><strong>Backend:</strong> receives ordinary device publications, validates their contracts, and applies application policy. It should not hold broker-administrator credentials just to process traffic.</li></ol>
<p>Dynamic Security is a Mosquitto plugin and broker control API, not a Go MQTT 5 library feature. The plugin manages clients, groups, roles, and ACLs and persists its configuration state. The ordinary backend/device listener and a separate management identity keep operations clearly scoped.</p>
<form class="card quiz" data-quiz data-correct="b" data-explanation="A protected provisioning identity manages plugin state; the device uses only its own ordinary runtime account, and the backend keeps validating messages." data-retry="Do not grant the backend runtime process the broker's management identity by default.">
<p><strong>🧠 Check the boundary:</strong> Which credential should the device use to publish telemetry?</p>
<label><input type="radio" name="q" value="a"> The Dynamic Security administrator credential</label><br><label><input type="radio" name="q" value="b"> Its own provisioned client credential with a narrow role</label><br><label><input type="radio" name="q" value="c"> A shared backend credential used by every device</label><br><button type="submit">Check answer</button><p data-feedback aria-live="polite"></p>
</form>
<h2>🧱 Keep the extension isolated</h2>
<p>The normal course lab uses Mosquitto password and ACL files and does not load this plugin. Create a separate broker configuration and persistent, broker-writable Dynamic Security state file. Verify the plugin path inside the pinned image. Do not silently mix plugin-managed accounts with the existing static authentication setup.</p>
<h3>📄 Complete broker configuration: <code>lab/dynsec/mosquitto.conf</code></h3>
<p><strong>File overview:</strong> This isolated configuration loads only Dynamic Security for the lab listener, disables anonymous access, and stores plugin state under the persistent broker data mount. The Linux plugin path below is the standard Debian multiarch path; confirm it inside the exact image before starting the comparison broker because the container build may package it elsewhere.</p>
<details class="card" open><summary><strong>🧩 Full configuration file</strong></summary><pre><code>listener 1883
allow_anonymous false
per_listener_settings false
plugin /usr/lib/x86_64-linux-gnu/mosquitto_dynamic_security.so
plugin_opt_config_file /mosquitto/data/dynamic-security.json
persistence true
persistence_location /mosquitto/data/
log_dest stdout
connection_messages true
</code></pre></details>
<p>Before writing that file, inspect the pinned image's plugin path with a one-off container command such as <code>docker run --rm --entrypoint sh eclipse-mosquitto:2.0.22 -c 'find /usr -name mosquitto_dynamic_security.so -print'</code>. The command searches inside the image and removes the temporary container; it does not start or modify the course broker. If the path differs, update the single <code>plugin</code> line and record the observed path. I could not inspect this host's Docker image because the Docker socket denied access.</p>
<h3>🪪 Complete device and backend role rules</h3>
<p><strong>File overview:</strong> These are the full topic permissions to install through the management API. Device ACL patterns bind the authenticated username as one device-ID segment. Backend permissions are fleet scoped. Set receive default-deny explicitly before grants.</p>
<details class="card"><summary><strong>📋 Role matrix and complete control commands</strong></summary><pre><code># These commands run against the dedicated Dynamic Security broker.
# Supply the admin secret through your protected local shell/session.
mosquitto_ctrl -h 127.0.0.1 -p 18883 -u dynsec-admin -P "$DYNSEC_ADMIN_PASSWORD" \
dynsec setDefaultACLAccess publishClientReceive deny
mosquitto_ctrl -h 127.0.0.1 -p 18883 -u dynsec-admin -P "$DYNSEC_ADMIN_PASSWORD" \
dynsec createRole device
mosquitto_ctrl -h 127.0.0.1 -p 18883 -u dynsec-admin -P "$DYNSEC_ADMIN_PASSWORD" \
dynsec addRoleACL device publishClientSend 'lab/v1/devices/%u/telemetry' allow 10
mosquitto_ctrl -h 127.0.0.1 -p 18883 -u dynsec-admin -P "$DYNSEC_ADMIN_PASSWORD" \
dynsec addRoleACL device publishClientSend 'lab/v1/devices/%u/responses' allow 10
mosquitto_ctrl -h 127.0.0.1 -p 18883 -u dynsec-admin -P "$DYNSEC_ADMIN_PASSWORD" \
dynsec addRoleACL device subscribePattern 'lab/v1/devices/%u/commands' allow 10
mosquitto_ctrl -h 127.0.0.1 -p 18883 -u dynsec-admin -P "$DYNSEC_ADMIN_PASSWORD" \
dynsec addRoleACL device publishClientReceive 'lab/v1/devices/%u/commands' allow 10
mosquitto_ctrl -h 127.0.0.1 -p 18883 -u dynsec-admin -P "$DYNSEC_ADMIN_PASSWORD" \
dynsec createRole backend
mosquitto_ctrl -h 127.0.0.1 -p 18883 -u dynsec-admin -P "$DYNSEC_ADMIN_PASSWORD" \
dynsec addRoleACL backend publishClientSend 'lab/v1/devices/+/commands' allow 10
mosquitto_ctrl -h 127.0.0.1 -p 18883 -u dynsec-admin -P "$DYNSEC_ADMIN_PASSWORD" \
dynsec addRoleACL backend subscribePattern 'lab/v1/devices/+/telemetry' allow 10
mosquitto_ctrl -h 127.0.0.1 -p 18883 -u dynsec-admin -P "$DYNSEC_ADMIN_PASSWORD" \
dynsec addRoleACL backend publishClientReceive 'lab/v1/devices/+/telemetry' allow 10
mosquitto_ctrl -h 127.0.0.1 -p 18883 -u dynsec-admin -P "$DYNSEC_ADMIN_PASSWORD" \
dynsec createClient device-001 -i lab-device-001
mosquitto_ctrl -h 127.0.0.1 -p 18883 -u dynsec-admin -P "$DYNSEC_ADMIN_PASSWORD" \
dynsec addClientRole device-001 device 10
</code></pre><p>Run the commands in order and only once per role/ACL. The client-creation command prompts for the device password. For a repeated setup, query existing plugin state first and handle already-created roles/ACLs explicitly rather than widening access.</p></details>
<p>Start with default-deny behavior for the relevant direction, grant only the required device publish/backend subscribe paths, and test real message delivery. A successful SUBACK or an ACL line alone does not prove a cross-device publication is blocked.</p>
<h2>🧪 Provision, test, revoke</h2>
<ol><li>Create the isolated plugin configuration and initialize its state using the secured management procedure documented in the full lesson.</li><li>Provision one test device account and a least-privilege role. Connect it and verify its own telemetry succeeds.</li><li>Attempt cross-device publish and subscription, then send a permitted topic with a mismatched payload identity. Identify whether the broker or application rejects each case.</li><li>Revoke/disable the client and observe behavior for both a new connection and an already-connected device.</li><li>Restart the broker and confirm the plugin state persists from its dedicated volume.</li></ol>
<div class="card"><h3>⚠️ What this experiment does not prove</h3><p>Successful provisioning does not establish secure remote administration unless transport and management access are protected. A plugin role does not replace application schema checks. Revoking an account may not by itself terminate every existing connection; verify the live-client behavior for the pinned broker/plugin.</p></div>
<h2>✅ Done when</h2><p>You can trace provisioning, broker authentication, topic authorization, application validation, and revocation as distinct steps. The normal static lab remains available as a separate configuration.</p>
<p>📚 Full failure and revocation experiments: <a href="../course-source/11b-dynamic-security.md">Section 11B source lesson</a>. Primary references: the official <a href="https://mosquitto.org/documentation/dynamic-security/">Mosquitto Dynamic Security guide</a> and <a href="https://mosquitto.org/man/mosquitto_ctrl_dynsec-1.html">mosquitto_ctrl DynSec manual</a>.</p>
<p>Ask me to troubleshoot a denied operation using its client reason and broker log evidence.</p>
<footer><p>Previous: <a href="0011-security-boundaries.html">Lesson 11 — security boundaries</a> · Continue: <a href="0012-vending-workflows.html">Lesson 12 — vending workflows</a>.</p></footer>
</main></body></html>
teaching/lessons/0012-vending-workflows.html
+231
View File
@@ -0,0 +1,231 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8"><meta name="viewport" content="width=device-width, initial-scale=1">
<title>Lesson 12 — Vending Workflows</title>
<link rel="stylesheet" href="../assets/lesson.css"><script src="../assets/quiz.js" defer></script>
</head>
<body><main class="page">
<p class="eyebrow">Go MQTT Vending Backend · Lesson 12 · 12 minutes</p>
<h1>Connect MQTT patterns to vending workflows</h1>
<p class="dek">Apply stable business identities and explicit application evidence to sales, catalog, configuration, health, and device logs.</p>
<div class="meta"><span class="pill">🎯 One skill: define workflow evidence</span><span class="pill">🧠 Predict duplicate and stale effects</span><span class="pill">🧪 Synthetic data only</span></div>
<section class="card callout"><p class="question">❓ Start with the system story</p><p>A simulator sends the same sale report twice, receives a new product catalog, and later asks for configuration. Which side owns each fact, and what proves the device applied the catalog?</p><p>Pause: separate backend business records from device-reported state.</p></section>
<h2>💰 Sales: stable identity survives retries</h2>
<p>Deduplicate with a business key such as <code>(device_id, report_id)</code>. MQTT packet identifiers belong to protocol exchanges and can be reused; they are not sale IDs. In this lesson's in-memory stage, replay the same report without repeating its business effect, and state clearly that a backend restart forgets the deduplication map.</p>
<h2>📦 Catalog: send intent, receive application evidence</h2>
<div class="flow" aria-label="catalog delivery"><div class="step">Backend<br><strong>desired revision 9</strong></div><span class="arrow">→</span><div class="step">Broker<br><strong>PUBACK for QoS hop</strong></div><span class="arrow">→</span><div class="step">Simulator<br><strong>validate + apply</strong></div><span class="arrow">→</span><div class="step">Backend<br><strong>reported revision 9</strong></div></div>
<p>Include schema version, stable message/job ID, and monotonic revision. Have the simulator reject revision 8 after applying revision 9. If you model catalog as replaceable desired state, retained state may fit; if each delivery is a one-time job, publish it nonretained. Pick one semantic model and explain it.</p>
<form class="card quiz" data-quiz data-correct="c" data-explanation="The simulator's validated application response or reported revision is evidence that it applied the catalog. PUBACK only acknowledges the relevant MQTT delivery step." data-retry="Ask which process produced the evidence and whether it saw the application effect.">
<p><strong>🧠 Check the evidence:</strong> What proves the device applied catalog revision 9?</p>
<label><input type="radio" name="q" value="a"> Backend publish returned without an error</label><br><label><input type="radio" name="q" value="b"> Broker PUBACK</label><br><label><input type="radio" name="q" value="c"> A validated device response/report of applied revision 9</label><br><button type="submit">Check answer</button><p data-feedback aria-live="polite"></p>
</form>
<h2>⚙️ Configuration: make races visible</h2>
<p>A device fetch request should carry a request ID and its known revision. The backend returns the selected version to an authorized response destination; the device validates the request/revision and applies only according to the agreed revision rule. Correlation Data can match a response to a request, but it does not authorize a Response Topic.</p>
<h2>❤️ Health and logs answer different questions</h2>
<p>Track broker connection status, heartbeat freshness, and device-reported health as separate observations. Bound log size and frequency, sanitize untrusted content, and avoid dumping raw payloads. Decide which records are disposable telemetry and which deserve durable storage in Lesson 13.</p>
<h2>🧭 File overview: <code>internal/vending/store.go</code></h2>
<p>Create this complete file for the Lesson 12 in-memory stage. It owns accepted synthetic sales identities and the latest catalog revision per device. It deliberately does not publish MQTT messages; the existing transport and Section 7 command path remain responsible for wire delivery.</p>
<h3>💰 <code>RecordSale</code> — short summary</h3><p>Stores a validated report under <code>(device_id, report_id)</code> and returns whether the effect was new, a duplicate, or an ID conflict.</p>
<h3>📦 <code>ApplyCatalog</code> — short summary</h3><p>Applies only a newer catalog revision and returns a copy of its product list.</p>
<details class="card" open><summary><strong>🧩 Complete file contents</strong></summary><pre><code>package vending
import (
"crypto/sha256"
"encoding/json"
"fmt"
"strings"
"sync"
"time"
"example.com/mqtt5-learning/internal/contracts"
)
type SaleOutcome uint8
const (
SaleAccepted SaleOutcome = iota + 1
SaleDuplicate
SaleIDConflict
)
type CatalogItem struct {
SKU string `json:"sku"`
Price int64 `json:"price_minor"`
}
type Catalog struct {
SchemaVersion int `json:"schema_version"`
JobID string `json:"job_id"`
DeviceID string `json:"device_id"`
Revision uint64 `json:"revision"`
Items []CatalogItem `json:"items"`
}
type Configuration struct {
SchemaVersion int `json:"schema_version"`
RequestID string `json:"request_id"`
DeviceID string `json:"device_id"`
Revision uint64 `json:"revision"`
Settings map[string]string `json:"settings"`
}
type HealthObservation struct {
DeviceID string
Status string
ReceivedAt time.Time
}
type Store struct {
mu sync.Mutex
sales map[string][32]byte
effects uint64
catalogs map[string]Catalog
configs map[string]Configuration
health map[string]HealthObservation
}
func NewStore() *Store {
return &amp;Store{
sales: make(map[string][32]byte), catalogs: make(map[string]Catalog),
configs: make(map[string]Configuration), health: make(map[string]HealthObservation),
}
}
// RecordSale uses stable business identity; MQTT packet IDs are not accepted here.
func (s *Store) RecordSale(report contracts.SalesReport) (SaleOutcome, error) {
if report.DeviceID == "" || report.ReportID == "" {
return 0, fmt.Errorf("device ID and report ID are required")
}
encoded, err := json.Marshal(report)
if err != nil {
return 0, fmt.Errorf("encode report identity: %w", err)
}
digest := sha256.Sum256(encoded)
key := report.DeviceID + "/" + report.ReportID
s.mu.Lock()
defer s.mu.Unlock()
if previous, exists := s.sales[key]; exists {
if previous != digest {
return SaleIDConflict, nil
}
return SaleDuplicate, nil
}
s.sales[key] = digest
s.effects++ // Stand-in for one synthetic business effect in this lesson.
return SaleAccepted, nil
}
func (s *Store) EffectCount() uint64 {
s.mu.Lock()
defer s.mu.Unlock()
return s.effects
}
// ApplyCatalog replaces device state only for a strictly newer revision.
func (s *Store) ApplyCatalog(catalog Catalog) (bool, error) {
if catalog.DeviceID == "" || catalog.JobID == "" || catalog.Revision == 0 {
return false, fmt.Errorf("device ID, job ID, and positive revision are required")
}
for _, item := range catalog.Items {
if item.SKU == "" || item.Price &lt; 0 {
return false, fmt.Errorf("catalog item has invalid SKU or price")
}
}
s.mu.Lock()
defer s.mu.Unlock()
previous, exists := s.catalogs[catalog.DeviceID]
if exists &amp;&amp; catalog.Revision &lt;= previous.Revision {
return false, nil
}
catalog.Items = append([]CatalogItem(nil), catalog.Items...)
s.catalogs[catalog.DeviceID] = catalog
return true, nil
}
func (s *Store) CatalogFor(deviceID string) (Catalog, bool) {
s.mu.Lock()
defer s.mu.Unlock()
catalog, exists := s.catalogs[deviceID]
if !exists {
return Catalog{}, false
}
catalog.Items = append([]CatalogItem(nil), catalog.Items...)
return catalog, true
}
// ApplyConfiguration rejects equal or older revisions to make retries idempotent.
func (s *Store) ApplyConfiguration(config Configuration) (bool, error) {
if config.DeviceID == "" || config.RequestID == "" || config.Revision == 0 || len(config.Settings) == 0 {
return false, fmt.Errorf("device, request ID, positive revision, and settings are required")
}
for key, value := range config.Settings {
if key == "" || len(key) &gt; 64 || len(value) &gt; 256 || strings.ContainsAny(key, "\r\n") {
return false, fmt.Errorf("configuration contains an invalid key or oversized value")
}
}
s.mu.Lock()
defer s.mu.Unlock()
previous, exists := s.configs[config.DeviceID]
if exists &amp;&amp; config.Revision &lt;= previous.Revision {
return false, nil
}
config.Settings = cloneSettings(config.Settings)
s.configs[config.DeviceID] = config
return true, nil
}
func (s *Store) ConfigurationFor(deviceID string) (Configuration, bool) {
s.mu.Lock()
defer s.mu.Unlock()
config, exists := s.configs[deviceID]
if !exists { return Configuration{}, false }
config.Settings = cloneSettings(config.Settings)
return config, true
}
func (s *Store) RecordHealth(deviceID, status string, receivedAt time.Time) error {
if deviceID == "" || (status != "ok" &amp;&amp; status != "warning" &amp;&amp; status != "fault") {
return fmt.Errorf("device ID or health status is invalid")
}
s.mu.Lock()
s.health[deviceID] = HealthObservation{DeviceID: deviceID, Status: status, ReceivedAt: receivedAt.UTC()}
s.mu.Unlock()
return nil
}
func (s *Store) HealthFor(deviceID string) (HealthObservation, bool) {
s.mu.Lock()
defer s.mu.Unlock()
observation, ok := s.health[deviceID]
return observation, ok
}
// SanitizeLogLine bounds untrusted device log text before it reaches a log sink.
func SanitizeLogLine(message string) string {
clean := strings.Map(func(r rune) rune {
if r == '\r' || r == '\n' || r == 0 { return ' ' }
if r &lt; 0x20 &amp;&amp; r != '\t' { return -1 }
return r
}, message)
clean = strings.TrimSpace(clean)
if len(clean) &gt; 512 { clean = clean[:512] }
return clean
}
func cloneSettings(input map[string]string) map[string]string {
copy := make(map[string]string, len(input))
for key, value := range input { copy[key] = value }
return copy
}
</code></pre></details>
<div class="card"><h3>🔎 Details: what this file proves</h3><p>The SHA-256 digest makes a reused report ID with changed content visible as a conflict. The mutex makes state safe inside this process, but it is not durable across restart and not shared between processes. <code>EffectCount</code> is a lab observation, not a revenue ledger. Catalog slices and configuration maps are copied on write/read so callers cannot mutate stored state through shared backing data. <code>SanitizeLogLine</code> removes line breaks/control bytes and caps log text; it does not identify secrets, so never log arbitrary payloads without an allowlist.</p></div>
<h2>🧪 Build the workflow matrix</h2>
<ol><li>Replay one report ID twice and then reuse it with changed content. Define whether changed content is a conflict.</li><li>Apply catalog revision 9, then 8, then a duplicate 9. Record the simulator's reported state and deduplication effect.</li><li>Request config at revision 4 while the backend is preparing revision 5. Define which response is valid.</li><li>Compare a fresh heartbeat with a device health report that says “fault”. Explain why freshness is not health.</li><li>Inspect the existing backend read-only and record its actual contract/identity behavior without assuming lab names match production.</li></ol>
<div class="card"><h3>🔎 What each result proves</h3><p>Two MQTT deliveries prove the broker delivered twice; one in-memory effect shows this process deduplicated. A backend PUBACK proves its QoS exchange. A device result/report proves application-level evidence it chose to report. None of these in-memory exercises prove crash-safe deduplication; PostgreSQL arrives next.</p></div>
<h2>✅ Done when</h2><p>Duplicates do not repeat the simulated business effect, stale revisions do not replace newer state, and your workflow notes name the evidence boundary for each operation.</p>
<details class="card"><summary><strong>📋 Short self-check answers</strong></summary><ol><li>Use stable application report identity, not MQTT packet ID.</li><li>Device-reported applied revision is application evidence; PUBACK is transport evidence.</li><li>Fresh heartbeat, current MQTT connection, and device health measure different facts.</li></ol></details>
<p>📚 Full workflow exercise and comparison questions: <a href="../course-source/12-vending-workflows.md">Lesson 12 source lesson</a>. Primary reference: <a href="https://docs.oasis-open.org/mqtt/mqtt/v5.0/mqtt-v5.0.html">OASIS MQTT 5.0 Standard</a> for the transport acknowledgement boundary.</p>
<p>Ask me to review a duplicate-report or stale-revision result with you before moving on to PostgreSQL.</p>
<footer><p>Previous: <a href="0011-security-boundaries.html">Lesson 11 — security boundaries</a> · Next: <a href="0013-persist-reports-command-state.html">Lesson 13 — durable persistence</a>.</p></footer>
</main></body></html>
teaching/lessons/0013-persist-reports-command-state.html
+280
View File
@@ -0,0 +1,280 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8"><meta name="viewport" content="width=device-width, initial-scale=1">
<title>Lesson 13 — Persist Reports and Command State</title>
<link rel="stylesheet" href="../assets/lesson.css"><script src="../assets/quiz.js" defer></script>
</head>
<body><main class="page">
<p class="eyebrow">Go MQTT Vending Backend · Lesson 13 · 15 minutes</p>
<h1>Commit business state before acknowledging important work</h1>
<p class="dek">Use PostgreSQL uniqueness and transactions to make report effects durable across duplicate delivery and backend restart.</p>
<div class="meta"><span class="pill">🎯 One skill: choose the durable acceptance boundary</span><span class="pill">🧠 Predict crash recovery</span><span class="pill">🧪 Dedicated lab database only</span></div>
<section class="card callout"><p class="question">❓ Start with the system story</p><p>The backend commits a QoS 1 sales report to PostgreSQL, then crashes before sending PUBACK. What should happen when the broker delivers the report again?</p><p>Pause before continuing: identify the durable idempotency key and the order of transaction and acknowledgement.</p></section>
<h2>📥 Acknowledge after the policy boundary</h2>
<div class="flow" aria-label="durable ingestion sequence"><div class="step">Receive QoS 1<br><strong>validate</strong></div><span class="arrow">→</span><div class="step">PostgreSQL<br><strong>dedupe + effect transaction</strong></div><span class="arrow">→</span><div class="step">COMMIT<br><strong>durable outcome</strong></div><span class="arrow">→</span><div class="step">Manual Ack<br><strong>check error</strong></div></div>
<p>With Paho v0.23.0 automatic acknowledgement, a callback error does not request MQTT retry; Paho acknowledges QoS 1 after callback return. Enabling manual acknowledgement changes who calls Ack, but returning an error still does not cause retry by itself. Keep the packet/client acknowledgement handle until the durable policy decision, not just a copied payload in a RAM queue.</p>
<form class="card quiz" data-quiz data-correct="b" data-explanation="If commit succeeded but PUBACK was lost, redelivery is safe: the unique business key finds the existing outcome, no effect repeats, and the backend can acknowledge the duplicate." data-retry="Consider the crash after COMMIT but before Ack, then ask what survives a process restart.">
<p><strong>🧠 Check crash recovery:</strong> PostgreSQL committed a report, but the backend crashed before PUBACK. What should replay do?</p>
<label><input type="radio" name="q" value="a"> Apply the sale again because MQTT redelivered it</label><br><label><input type="radio" name="q" value="b"> Find the stable unique key, return the stored outcome without repeating the effect, then acknowledge</label><br><label><input type="radio" name="q" value="c"> Treat the earlier callback's error as an automatic retry guarantee</label><br><button type="submit">Check answer</button><p data-feedback aria-live="polite"></p>
</form>
<h2>🔁 Let the database arbitrate duplicates</h2>
<p>Use a unique business key such as <code>(device_id, report_id)</code>. In one transaction, insert the report identity, apply its business effect, and store the application outcome needed for duplicate replay. The unique constraint handles two concurrent workers and remains after restart; an in-memory map cannot.</p>
<p>If the same key arrives with materially different content, record a conflict instead of silently giving one ID two meanings. MQTT packet IDs are not business IDs. QoS 2 cannot make a PostgreSQL transaction and a physical vending action exactly once.</p>
<h2>🧭 File overview: <code>internal/persistence/sales.go</code></h2>
<p>Create this complete repository file for durable sales ingestion. The caller opens a PostgreSQL <code>*sql.DB</code> using the project's pgx stdlib driver and validates the report with <code>contracts.Decode</code> first. This file owns the transaction boundary: report identity and sale lines commit together.</p>
<h3>🏗️ <code>Migrate</code> — short summary</h3><p>Creates a report table with a unique business key and a child line table.</p>
<h3>💾 <code>SaveReport</code> — short summary</h3><p>Inserts the report identity, detects duplicate or conflicting content, inserts lines for a new report, and commits once.</p>
<details class="card" open><summary><strong>🧩 Complete file contents</strong></summary><pre><code>package persistence
import (
"context"
"crypto/sha256"
"database/sql"
"encoding/json"
"errors"
"fmt"
"time"
"example.com/mqtt5-learning/internal/contracts"
)
type ReportOutcome uint8
const (
ReportSaved ReportOutcome = iota + 1
ReportDuplicate
)
var ErrReportIDConflict = errors.New("report ID was reused with different content")
type SalesRepository struct {
db *sql.DB
}
func NewSalesRepository(db *sql.DB) (*SalesRepository, error) {
if db == nil {
return nil, fmt.Errorf("database connection is required")
}
return &amp;SalesRepository{db: db}, nil
}
func (r *SalesRepository) Migrate(ctx context.Context) error {
statements := []string{
`CREATE TABLE IF NOT EXISTS sales_reports (
device_id TEXT NOT NULL,
report_id TEXT NOT NULL,
payload_hash BYTEA NOT NULL,
occurred_at TIMESTAMPTZ NOT NULL,
saved_at TIMESTAMPTZ NOT NULL DEFAULT now(),
PRIMARY KEY (device_id, report_id)
)`,
`CREATE TABLE IF NOT EXISTS sales_report_lines (
device_id TEXT NOT NULL,
report_id TEXT NOT NULL,
line_number INTEGER NOT NULL,
sku TEXT NOT NULL,
quantity BIGINT NOT NULL CHECK (quantity &gt; 0),
amount_minor BIGINT NOT NULL CHECK (amount_minor &gt;= 0),
PRIMARY KEY (device_id, report_id, line_number),
FOREIGN KEY (device_id, report_id)
REFERENCES sales_reports (device_id, report_id) ON DELETE CASCADE
)`,
}
for _, statement := range statements {
if _, err := r.db.ExecContext(ctx, statement); err != nil {
return fmt.Errorf("create sales schema: %w", err)
}
}
return nil
}
func (r *SalesRepository) SaveReport(ctx context.Context, report contracts.SalesReport) (ReportOutcome, error) {
if report.DeviceID == "" || report.ReportID == "" || len(report.Items) == 0 {
return 0, fmt.Errorf("validated report identity and at least one item are required")
}
occurredAt, err := time.Parse(time.RFC3339, report.OccurredAt)
if err != nil {
return 0, fmt.Errorf("parse report time: %w", err)
}
encoded, err := json.Marshal(report)
if err != nil {
return 0, fmt.Errorf("encode report for deduplication: %w", err)
}
hash := sha256.Sum256(encoded)
tx, err := r.db.BeginTx(ctx, &amp;sql.TxOptions{Isolation: sql.LevelReadCommitted})
if err != nil {
return 0, fmt.Errorf("begin report transaction: %w", err)
}
defer func() { _ = tx.Rollback() }() // No effect after a successful commit.
result, err := tx.ExecContext(ctx, `
INSERT INTO sales_reports (device_id, report_id, payload_hash, occurred_at)
VALUES ($1, $2, $3, $4)
ON CONFLICT (device_id, report_id) DO NOTHING`,
report.DeviceID, report.ReportID, hash[:], occurredAt)
if err != nil {
return 0, fmt.Errorf("insert report identity: %w", err)
}
inserted, err := result.RowsAffected()
if err != nil {
return 0, fmt.Errorf("inspect report insert: %w", err)
}
if inserted == 0 {
var storedHash []byte
if err := tx.QueryRowContext(ctx, `
SELECT payload_hash FROM sales_reports WHERE device_id=$1 AND report_id=$2`,
report.DeviceID, report.ReportID).Scan(&amp;storedHash); err != nil {
return 0, fmt.Errorf("load duplicate report: %w", err)
}
if !equalBytes(storedHash, hash[:]) {
return 0, ErrReportIDConflict
}
if err := tx.Commit(); err != nil {
return 0, fmt.Errorf("commit duplicate observation: %w", err)
}
return ReportDuplicate, nil
}
for i, line := range report.Items {
if _, err := tx.ExecContext(ctx, `
INSERT INTO sales_report_lines
(device_id, report_id, line_number, sku, quantity, amount_minor)
VALUES ($1, $2, $3, $4, $5, $6)`,
report.DeviceID, report.ReportID, i, line.SKU, line.Quantity, line.AmountMinor); err != nil {
return 0, fmt.Errorf("insert sales line %d: %w", i, err)
}
}
if err := tx.Commit(); err != nil {
return 0, fmt.Errorf("commit sales report: %w", err)
}
return ReportSaved, nil
}
func equalBytes(left, right []byte) bool {
if len(left) != len(right) {
return false
}
var difference byte
for i := range left {
difference |= left[i] ^ right[i]
}
return difference == 0
}
</code></pre></details>
<div class="card"><h3>🔎 Details: duplicate and transaction behavior</h3><p>The unique constraint handles concurrent transactions for one device/report pair. The winner inserts lines in the same transaction as the report row. A second identical report reads the stored digest and returns <code>ReportDuplicate</code>; different content under the same key returns <code>ErrReportIDConflict</code>. The deferred rollback is harmless after commit and cleans up early returns. Do not Ack the MQTT packet until this method returns a committed saved/duplicate outcome. This repository does not send the application response; publication intent belongs in an outbox when crash-safe receipts are required.</p></div>
<h2>🧭 File overview: <code>internal/persistence/commands.go</code></h2>
<p>Command state also needs durable storage. Create this complete file alongside <code>sales.go</code>. It stores validated device results under the stable command ID and allows a late result to resolve an earlier unknown outcome without losing the separate broker-ack field.</p>
<h3>🏗️ <code>Migrate</code> — short summary</h3><p>Creates the command table and a unique result table.</p>
<h3>📩 <code>ApplyResult</code> — short summary</h3><p>Inserts one result, accepts identical replays, detects conflicting IDs, and updates command status transactionally.</p>
<details class="card" open><summary><strong>📄 <code>internal/persistence/commands.go</code> — complete file</strong></summary><pre><code>package persistence
import (
"context"
"database/sql"
"errors"
"fmt"
"time"
"example.com/mqtt5-learning/internal/contracts"
)
var ErrCommandResultConflict = errors.New("command ID has a conflicting result")
type CommandRepository struct { db *sql.DB }
func NewCommandRepository(db *sql.DB) (*CommandRepository, error) {
if db == nil { return nil, fmt.Errorf("database connection is required") }
return &amp;CommandRepository{db: db}, nil
}
func (r *CommandRepository) Migrate(ctx context.Context) error {
statements := []string{
`CREATE TABLE IF NOT EXISTS device_commands (
command_id TEXT PRIMARY KEY, device_id TEXT NOT NULL, state TEXT NOT NULL,
broker_acked BOOLEAN NOT NULL DEFAULT FALSE,
deadline TIMESTAMPTZ,
created_at TIMESTAMPTZ NOT NULL DEFAULT now(), updated_at TIMESTAMPTZ NOT NULL DEFAULT now())`,
`CREATE TABLE IF NOT EXISTS command_results (
command_id TEXT NOT NULL REFERENCES device_commands(command_id),
device_id TEXT NOT NULL, message_id TEXT NOT NULL, status TEXT NOT NULL,
detail TEXT NOT NULL DEFAULT '', occurred_at TIMESTAMPTZ NOT NULL,
PRIMARY KEY(command_id,message_id), UNIQUE(device_id,message_id))`,
}
for _, statement := range statements {
if _, err := r.db.ExecContext(ctx, statement); err != nil {
return fmt.Errorf("create command schema: %w", err)
}
}
return nil
}
func (r *CommandRepository) Create(ctx context.Context, commandID, deviceID string) error {
if commandID == "" || deviceID == "" { return fmt.Errorf("command ID and device ID are required") }
_, err := r.db.ExecContext(ctx, `INSERT INTO device_commands(command_id,device_id,state)
VALUES ($1,$2,'pending')`, commandID, deviceID)
if err != nil { return fmt.Errorf("create command: %w", err) }
return nil
}
func (r *CommandRepository) MarkBrokerAck(ctx context.Context, commandID string) error {
result, err := r.db.ExecContext(ctx, `UPDATE device_commands SET broker_acked=TRUE,updated_at=now()
WHERE command_id=$1`, commandID)
if err != nil { return fmt.Errorf("record broker acknowledgement: %w", err) }
rows, err := result.RowsAffected()
if err != nil { return fmt.Errorf("inspect command update: %w", err) }
if rows != 1 { return fmt.Errorf("command %q not found", commandID) }
return nil
}
func (r *CommandRepository) ApplyResult(ctx context.Context, result contracts.CommandResult) error {
if result.CommandID == "" || result.DeviceID == "" || result.MessageID == "" {
return fmt.Errorf("validated result identity is required")
}
occurredAt, err := time.Parse(time.RFC3339, result.OccurredAt)
if err != nil { return fmt.Errorf("parse result time: %w", err) }
tx, err := r.db.BeginTx(ctx, nil)
if err != nil { return fmt.Errorf("begin result transaction: %w", err) }
defer func() { _ = tx.Rollback() }()
insert, err := tx.ExecContext(ctx, `INSERT INTO command_results
(command_id,device_id,message_id,status,detail,occurred_at)
VALUES ($1,$2,$3,$4,$5,$6) ON CONFLICT (command_id,message_id) DO NOTHING`,
result.CommandID,result.DeviceID,result.MessageID,result.Status,result.Detail,occurredAt)
if err != nil { return fmt.Errorf("insert command result: %w", err) }
rows, err := insert.RowsAffected()
if err != nil { return fmt.Errorf("inspect result insert: %w", err) }
if rows == 0 {
var oldDevice, oldMessage, oldStatus, oldDetail string
if err := tx.QueryRowContext(ctx, `SELECT device_id,message_id,status,detail
FROM command_results WHERE command_id=$1 AND message_id=$2`, result.CommandID,result.MessageID).Scan(&amp;oldDevice,&amp;oldMessage,&amp;oldStatus,&amp;oldDetail); err != nil {
return fmt.Errorf("load prior command result: %w", err)
}
if oldDevice != result.DeviceID || oldMessage != result.MessageID || oldStatus != result.Status || oldDetail != result.Detail {
return ErrCommandResultConflict
}
return tx.Commit() // Identical replay: keep the prior terminal state.
}
updated, err := tx.ExecContext(ctx, `UPDATE device_commands SET state=$2,updated_at=now()
WHERE command_id=$1 AND device_id=$3 AND state IN ('pending','accepted','unknown')`,
result.CommandID,result.Status,result.DeviceID)
if err != nil { return fmt.Errorf("update command state: %w", err) }
changed, err := updated.RowsAffected()
if err != nil { return fmt.Errorf("inspect command state update: %w", err) }
if changed != 1 { return fmt.Errorf("command missing, device mismatch, or terminal state conflict") }
if err := tx.Commit(); err != nil { return fmt.Errorf("commit command result: %w", err) }
return nil
}
</code></pre></details>
<div class="card"><h3>🔎 Details: legal state transitions</h3><p>The <code>state IN ('pending','accepted','unknown')</code> guard prevents a new result from overwriting an already terminal state such as completed or rejected. The result row and command-state update share a transaction. <code>broker_acked</code> remains a separate transport fact. Before using this example, ensure the command row exists and the result's topic/body identity was validated in the ingestion service.</p></div>
<h2>🧹 Invalid and transient failures need different paths</h2>
<table><thead><tr><th>Outcome</th><th>Durable action</th><th>MQTT decision</th></tr></thead><tbody><tr><td>Valid new report</td><td>Commit identity and effect</td><td>Ack after commit</td></tr><tr><td>Valid duplicate</td><td>Load prior result; no repeated effect</td><td>Ack after recognizing it</td></tr><tr><td>Permanently invalid</td><td>Commit bounded quarantine/rejection metadata</td><td>Ack after disposition commits</td></tr><tr><td>Transient database failure</td><td>Do not claim durable acceptance</td><td>Do not Ack; invoke explicit recovery/backoff</td></tr></tbody></table>
<p>A poison message must not occupy a Receive Maximum slot forever. Record safe metadata such as topic, reason category, and receipt time; avoid secrets and arbitrary payload dumps. A transient database outage is not permanent invalid input.</p>
<h2>🧪 Crash-point experiments</h2>
<ol><li>Send the same report twice sequentially and concurrently. Expect one durable business effect.</li><li>Force a transaction rollback and verify neither dedupe identity nor business effect commits alone.</li><li>Commit, stop the process before manual Ack, reconnect with an eligible broker session, and observe a safe duplicate.</li><li>Send a permanently invalid report and confirm a bounded rejection record commits before Ack.</li><li>Delay acknowledgements to the configured Receive Maximum and observe delivery pressure. Remember this protocol limit does not cap your Go worker queue.</li><li>Record the gap between database commit and publishing an application response; Lesson 14 closes that gap with an outbox.</li></ol>
<div class="card"><h3>🧱 Keep durable and temporary state distinct</h3><p>Persist reports, deduplication keys, command state/results, and any publication intent needed after restart. Keep live connections, worker queues, in-flight Paho packet handles, and process contexts in memory. On restart, recover from durable rows and eligible MQTT sessions; a dead callback cannot resume.</p><p><strong>Evidence boundary:</strong> database COMMIT establishes the backend's durable acceptance policy. PUBACK completes the MQTT protocol delivery. A separate application receipt tells the device the backend outcome, and it can still be lost unless its publication intent is persisted.</p></div>
<h2>✅ Done when</h2><p>After a backend restart, replay does not double-count a report; a database outage is not acknowledged as durable success; invalid data has a durable disposition; and you can explain why manual Ack alone is not permanent broker storage.</p>
<p>Run only against the dedicated lab PostgreSQL service and synthetic data. The full source lesson includes the project-specific Docker commands and Paho acknowledgement references.</p>
<details class="card"><summary><strong>📋 Short self-check answers</strong></summary><ol><li>Unique stable business key plus one transaction for identity/effect/outcome.</li><li>Commit then Ack; after crash-before-Ack, recognize duplicate and avoid repeating the effect.</li><li>Callback error alone does not request retry; recovery is an application policy.</li><li>Manual Ack is not a durable application queue and depends on session/broker behavior.</li></ol></details>
<p>📚 Failure experiments: <a href="../course-source/13-persist-reports-command-state.md">Lesson 13 source lesson</a>. Primary references: <a href="https://pkg.go.dev/database/sql">Go <code>database/sql</code></a>, <a href="https://www.postgresql.org/docs/current/ddl-constraints.html">PostgreSQL constraints</a>, and the <a href="https://docs.oasis-open.org/mqtt/mqtt/v5.0/mqtt-v5.0.html">OASIS MQTT 5.0 Standard</a>.</p>
<p>Ask me to reason through the exact crash point you observed before changing the acknowledgement policy.</p>
<footer><p>Previous: <a href="0012-vending-workflows.html">Lesson 12 — vending workflows</a> · Next: <a href="0014-outbox-failure-gaps.html">Lesson 14 — transactional outbox</a> · Detailed source: <a href="../course-source/14-database-mqtt-failure-gaps.md">Markdown lesson</a>.</p></footer>
</main></body></html>
teaching/lessons/0014-outbox-failure-gaps.html
+149
View File
@@ -0,0 +1,149 @@
<!doctype html>
<html lang="en"><head>
<meta charset="utf-8"><meta name="viewport" content="width=device-width, initial-scale=1">
<title>Lesson 14 — Close Database and MQTT Failure Gaps</title>
<link rel="stylesheet" href="../assets/lesson.css"><script src="../assets/quiz.js" defer></script>
</head><body><main class="page">
<p class="eyebrow">Go MQTT Vending Backend · Lesson 14 · 15 minutes</p>
<h1>Keep accepted publish intent recoverable</h1>
<p class="dek">Use a transactional outbox to bridge the database and MQTT broker while keeping retries, duplicates, and uncertain device outcomes visible.</p>
<div class="meta"><span class="pill">🎯 One skill: reason about cross-system failure</span><span class="pill">🧠 Predict duplicate publication</span><span class="pill">🧪 Synthetic commands only</span></div>
<section class="card callout"><p class="question">❓ Start with the system story</p><p>The API commits a ping command to PostgreSQL, then crashes before publishing it. What durable record lets another process resume the accepted work?</p><p>Name which values must commit together and which process retries them.</p></section>
<h2>🧾 Commit business intent and publish intent together</h2>
<div class="flow" aria-label="transactional outbox flow"><div class="step">HTTP API<br><strong>validate + authorize</strong></div><span class="arrow">→</span><div class="step">One DB transaction<br><strong>command + outbox row</strong></div><span class="arrow">→</span><div class="step">Publisher worker<br><strong>bounded claim + retry</strong></div><span class="arrow">→</span><div class="step">MQTT + device<br><strong>stable ID + dedupe</strong></div></div>
<p>The transaction closes the gap where a command is durably accepted but the intent to publish is lost. A worker claims rows briefly, publishes outside the DB transaction, then records the attempt. Do not hold a database transaction open while waiting on a network operation.</p>
<form class="card quiz" data-quiz data-correct="b" data-explanation="The command and its publish intent are committed atomically. A worker can recover the pending outbox row after restart." data-retry="Which database rows can exist after a crash immediately following command acceptance?"><p><strong>🧠 Check the guarantee:</strong> What makes accepted work publishable after an API process crashes before MQTT publish?</p><label><input type="radio" name="q" value="a"> A successful HTTP 202 by itself</label><br><label><input type="radio" name="q" value="b"> The command and outbox intent committed in the same DB transaction</label><br><label><input type="radio" name="q" value="c"> A retained command message without an application record</label><br><button type="submit">Check answer</button><p data-feedback aria-live="polite"></p></form>
<h2>🪟 The outbox still permits duplicates</h2>
<p>If the broker accepts a PUBLISH and the backend crashes before marking its outbox row sent, the worker may publish it again after restart. Keep one command ID across retries. The simulator stores the prior effect/result by command ID and returns that outcome on a duplicate. A timeout after possible publication is an unknown device outcome, not proof that it did nothing.</p>
<table><thead><tr><th>Crash point</th><th>What remains</th><th>Recovery</th></tr></thead><tbody><tr><td>Before transaction commit</td><td>No accepted command/outbox</td><td>Caller may retry with the same HTTP idempotency key</td></tr><tr><td>After commit, before publish</td><td>Pending outbox row</td><td>Worker publishes after restart</td></tr><tr><td>After publish, before sent marker</td><td>Row may remain pending</td><td>Republish same command ID; device deduplicates</td></tr><tr><td>After device effect, before result</td><td>Device may have acted; backend lacks result</td><td>Expose unknown/pending and reconcile by stable ID</td></tr></tbody></table>
<h2>🧭 File overview: <code>internal/outbox/store.go</code></h2>
<p>Create this complete PostgreSQL repository file for the outbox chapter. The HTTP API calls <code>Enqueue</code> in its acceptance path; a bounded worker uses <code>ClaimOne</code> and records the PUBACK outcome with <code>MarkPublished</code>. No database transaction stays open during the network publish.</p>
<h3>🧾 <code>Enqueue</code> — short summary</h3><p>Commits the command state and serialized publication intent atomically under one command ID.</p>
<h3>🔒 <code>ClaimOne</code> — short summary</h3><p>Uses a short transaction and <code>FOR UPDATE SKIP LOCKED</code> to lease one eligible row for one worker.</p>
<h3>📬 <code>MarkPublished</code> — short summary</h3><p>Records the broker acknowledgement as transport evidence and marks only that command's outbox row sent.</p>
<details class="card" open><summary><strong>🧩 Complete file contents</strong></summary><pre><code>package outbox
import (
"context"
"database/sql"
"errors"
"fmt"
"time"
)
var ErrNoWork = errors.New("no eligible outbox row")
type Command struct {
ID string
DeviceID string
Topic string
Payload []byte
Deadline time.Time
}
type Item struct {
CommandID string
DeviceID string
Topic string
Payload []byte
Deadline time.Time
Attempt int
}
type Store struct { db *sql.DB }
func NewStore(db *sql.DB) (*Store, error) {
if db == nil { return nil, fmt.Errorf("database connection is required") }
return &amp;Store{db: db}, nil
}
func (s *Store) Migrate(ctx context.Context) error {
statements := []string{
`CREATE TABLE IF NOT EXISTS device_commands (
command_id TEXT PRIMARY KEY, device_id TEXT NOT NULL, state TEXT NOT NULL,
deadline TIMESTAMPTZ NOT NULL, created_at TIMESTAMPTZ NOT NULL DEFAULT now())`,
`CREATE TABLE IF NOT EXISTS command_outbox (
command_id TEXT PRIMARY KEY REFERENCES device_commands(command_id),
topic TEXT NOT NULL, payload BYTEA NOT NULL, attempts INTEGER NOT NULL DEFAULT 0,
available_at TIMESTAMPTZ NOT NULL DEFAULT now(), lease_until TIMESTAMPTZ,
published_at TIMESTAMPTZ, last_reason SMALLINT)`,
}
for _, statement := range statements {
if _, err := s.db.ExecContext(ctx, statement); err != nil {
return fmt.Errorf("create outbox schema: %w", err)
}
}
return nil
}
func (s *Store) Enqueue(ctx context.Context, command Command) error {
if command.ID == "" || command.DeviceID == "" || command.Topic == "" || len(command.Payload) == 0 {
return fmt.Errorf("command ID, device, topic, and payload are required")
}
if !command.Deadline.After(time.Now()) {
return fmt.Errorf("command deadline must be in the future")
}
tx, err := s.db.BeginTx(ctx, nil)
if err != nil { return fmt.Errorf("begin command transaction: %w", err) }
defer func() { _ = tx.Rollback() }()
if _, err := tx.ExecContext(ctx, `INSERT INTO device_commands(command_id,device_id,state,deadline)
VALUES ($1,$2,'pending',$3)`, command.ID, command.DeviceID, command.Deadline); err != nil {
return fmt.Errorf("insert command: %w", err)
}
if _, err := tx.ExecContext(ctx, `INSERT INTO command_outbox(command_id,topic,payload)
VALUES ($1,$2,$3)`, command.ID, command.Topic, command.Payload); err != nil {
return fmt.Errorf("insert outbox intent: %w", err)
}
if err := tx.Commit(); err != nil { return fmt.Errorf("commit command and outbox: %w", err) }
return nil
}
func (s *Store) ClaimOne(ctx context.Context, lease time.Duration, now time.Time) (Item, error) {
if lease &lt;= 0 { return Item{}, fmt.Errorf("lease must be positive") }
tx, err := s.db.BeginTx(ctx, nil)
if err != nil { return Item{}, fmt.Errorf("begin outbox claim: %w", err) }
defer func() { _ = tx.Rollback() }()
var item Item
err = tx.QueryRowContext(ctx, `SELECT o.command_id,c.device_id,o.topic,o.payload,c.deadline,o.attempts
FROM command_outbox o JOIN device_commands c USING (command_id)
WHERE o.published_at IS NULL AND o.available_at &lt;= $1
AND (o.lease_until IS NULL OR o.lease_until &lt;= $1)
AND c.deadline &gt; $1
ORDER BY o.available_at,o.command_id
LIMIT 1 FOR UPDATE OF o SKIP LOCKED`, now).Scan(
&amp;item.CommandID, &amp;item.DeviceID, &amp;item.Topic, &amp;item.Payload, &amp;item.Deadline, &amp;item.Attempt)
if errors.Is(err, sql.ErrNoRows) { return Item{}, ErrNoWork }
if err != nil { return Item{}, fmt.Errorf("select outbox work: %w", err) }
item.Payload = append([]byte(nil), item.Payload...)
if _, err := tx.ExecContext(ctx, `UPDATE command_outbox
SET lease_until=$2, attempts=attempts+1 WHERE command_id=$1`, item.CommandID, now.Add(lease)); err != nil {
return Item{}, fmt.Errorf("lease outbox work: %w", err)
}
if err := tx.Commit(); err != nil { return Item{}, fmt.Errorf("commit outbox lease: %w", err) }
item.Attempt++
return item, nil
}
func (s *Store) MarkPublished(ctx context.Context, commandID string, reason byte, at time.Time) error {
result, err := s.db.ExecContext(ctx, `UPDATE command_outbox SET published_at=$2,
lease_until=NULL,last_reason=$3 WHERE command_id=$1 AND published_at IS NULL`, commandID, at, reason)
if err != nil { return fmt.Errorf("mark outbox published: %w", err) }
rows, err := result.RowsAffected()
if err != nil { return fmt.Errorf("inspect outbox update: %w", err) }
if rows != 1 { return fmt.Errorf("outbox command %q was not pending", commandID) }
return nil
}
</code></pre></details>
<div class="card"><h3>🔎 Details: lease and retry policy</h3><p>The lease stops two active workers from claiming the same row at once. A crashed worker leaves a lease that expires; a later worker can claim and republish. A DB transaction makes intent durable, but it cannot atomically include the MQTT PUBLISH. <code>MarkPublished</code> therefore follows the PUBLISH and can race a crash; the device must deduplicate. Production code should add bounded exponential backoff, a retryable/permanent error classification, and a separate terminal outcome for expired rows rather than retrying forever.</p></div>
<h2>📚 Keep three queues separate</h2>
<div class="two-col"><article class="card"><h3>Transactional outbox</h3><p>Application DB owns durable business intent and retry status. It does not prove broker delivery or device completion.</p></article><article class="card"><h3>Paho outgoing queue</h3><p>Client library owns transport queueing under its configured semantics. It is not atomic with the business transaction.</p></article><article class="card"><h3>MQTT session state</h3><p>Client/broker own protocol session continuity and eligible queued deliveries. It does not store business intent or dedupe effects.</p></article></div>
<h2>⏳ Bound retries and stale work</h2>
<p>Set a retry window with backoff and jitter, bound concurrent claims and publishes, and measure pending row count and oldest age. Check the application deadline before sending; use MQTT Message Expiry for broker-queued lifetime; have the device check its deadline before execution. Park expired or permanent failures and preserve unknown outcomes.</p>
<h2>🧪 Inject each failure window</h2>
<ol><li>Commit an outbox row while the publisher is stopped; restart it and confirm the row is attempted.</li><li>Stop the worker after publish but before marking sent. Restart and confirm duplicate publication keeps the command ID and causes one simulated effect.</li><li>Drop the device response after its simulated effect. Verify status does not claim definite failure.</li><li>Expire a row before publish and verify it is not sent.</li><li>Restart the broker and compare the pending application outbox state with broker session recovery.</li></ol>
<div class="card"><h3>✅ Done when</h3><p>Every crash point has a recovery policy; duplicate publication creates one simulated effect; expiry blocks stale work; status separates PUBACK from device completion.</p><p><strong>Evidence to record:</strong> prediction, injected crash point, DB rows, MQTT evidence, simulator effect count, final command state, and cleanup.</p></div>
<details class="card"><summary><strong>📋 Short self-check answers</strong></summary><ol><li>Commit the command and outbox intent atomically.</li><li>The worker can crash after PUBACK but before the sent marker; same-ID retry may duplicate.</li><li>Paho queue admission does not prove broker delivery or business completion.</li><li>Report unknown when the device may have acted but the application response is missing.</li></ol></details>
<p>📚 Full crash experiment plan: <a href="../course-source/14-database-mqtt-failure-gaps.md">Lesson 14 source lesson</a>. Primary references: <a href="https://www.postgresql.org/docs/current/sql-select.html">PostgreSQL locking clauses</a> and <a href="https://docs.oasis-open.org/mqtt/mqtt/v5.0/mqtt-v5.0.html">OASIS MQTT 5.0 Standard</a>.</p>
<p>Ask me to trace a crash window against the durable row and MQTT packet evidence.</p>
<footer><p>Previous: <a href="0013-persist-reports-command-state.html">Lesson 13 — durable persistence</a> · Next: <a href="0015-process-boundaries.html">Lesson 15 — split API and ingestion</a>.</p></footer>
</main></body></html>
teaching/lessons/0015-process-boundaries.html
+71
View File
@@ -0,0 +1,71 @@
<!doctype html>
<html lang="en"><head>
<meta charset="utf-8"><meta name="viewport" content="width=device-width, initial-scale=1">
<title>Lesson 15 — Split API and Ingestion Processes</title>
<link rel="stylesheet" href="../assets/lesson.css"><script src="../assets/quiz.js" defer></script>
</head><body><main class="page">
<p class="eyebrow">Go MQTT Vending Backend · Lesson 15 · 12 minutes</p>
<h1>Split processes around durable ownership</h1>
<p class="dek">Let an HTTP API accept and query durable commands while an independent MQTT ingestion process stores device results.</p>
<div class="meta"><span class="pill">🎯 One skill: design across process boundaries</span><span class="pill">🧠 Predict independent restarts</span><span class="pill">🧪 Local processes</span></div>
<section class="card callout"><p class="question">❓ Start with the system story</p><p>The API process creates a command. A separate ingestion process receives the device result. Which process owns the command's pending waiter, and how can HTTP status remain correct if the API restarts?</p><p>Pause: a Go map shared only in one process cannot be the answer.</p></section>
<h2>🗂️ Put shared truth in PostgreSQL</h2>
<div class="flow" aria-label="separated services"><div class="step">HTTP API<br><strong>authorize + write command</strong></div><span class="arrow">→</span><div class="step">PostgreSQL<br><strong>command/outbox/result state</strong></div><span class="arrow">↔</span><div class="step">Ingestion process<br><strong>subscribe + validate + persist</strong></div></div>
<p>The API serves HTTP, checks user/device permission, creates durable command state and outbox intent, and reads status. Ingestion owns MQTT subscriptions and result processing. Both use the database as their shared source of truth. A pending-request map can still optimize local waiting, but it cannot be required for correctness across processes.</p>
<form class="card quiz" data-quiz data-correct="c" data-explanation="Persisted command/result state is shared across the API and ingestion processes and survives an API restart; notifications can be added for latency but do not replace it." data-retry="Ask what both processes can read after either process restarts."><p><strong>🧠 Check the ownership:</strong> How should API status reflect a result received in another process?</p><label><input type="radio" name="q" value="a"> Copy a Go map between processes</label><br><label><input type="radio" name="q" value="b"> Assume the API process owns every response callback</label><br><label><input type="radio" name="q" value="c"> Read persisted command/result state; use notifications only as an optional latency aid</label><br><button type="submit">Check answer</button><p data-feedback aria-live="polite"></p></form>
<h2>🧩 Give each process its own lifecycle</h2>
<p>Use separate executables, signal contexts, health endpoints, and MQTT Client IDs. The ingestion process owns the response/telemetry subscriptions; the API or a dedicated publisher worker owns command publication. State that ownership explicitly so two processes do not accidentally compete for a device command subscription.</p>
<p>Readiness should mean the process can perform its assigned role: API readiness can include database availability and HTTP dependencies; ingestion readiness can include database availability and restored required MQTT subscriptions. Avoid arbitrary startup sleeps. Expose the dependency that is not ready.</p>
<h2>📄 Complete artifact: <code>docs/process-ownership.md</code></h2>
<p>This is the complete ownership contract to create before extracting executables. Fill every bracketed value from the actual deployment; the template is intentionally not a pretend Go implementation. It makes the process split reviewable and catches duplicate MQTT ownership before deployment.</p>
<details class="card" open><summary><strong>🧭 Full file contents</strong></summary><pre><code># Process ownership
## Runtime identities
| Process | Executable | MQTT Client ID | MQTT responsibilities | HTTP responsibilities |
| --- | --- | --- | --- | --- |
| API | `cmd/api` | `[none or explicit publisher ID]` | `[publish only if assigned; never subscribe to device results]` | Authorize callers, create durable commands, query command state, serve `/livez` and `/readyz` |
| Ingestion | `cmd/ingest` | `[unique stable ingestion ID]` | Own response and report subscriptions; validate and persist inbound messages | Serve `/livez` and `/readyz` if this role exposes health over HTTP |
| Outbox worker | `[same API process or separate executable]` | `[unique publisher ID when separate]` | Claim durable publish intent, publish, record broker outcome | No public command-creation endpoint |
## Durable state contract
- PostgreSQL owns accepted command state, the outbox, deduplication keys, and received results.
- A process-local waiter or cache may reduce latency; it never decides correctness.
- Every retry reuses the original command ID and idempotency key.
- A broker acknowledgment records a transport event. Device acceptance, completion, and physical effect require separate evidence.
- `[Describe migration ownership and how both versions remain compatible during rollout.]`
## Readiness and lifecycle
| Role | Ready when | Shutdown sequence | Recoverable work |
| --- | --- | --- | --- |
| API | Database is reachable and HTTP dependencies are initialized | Stop accepting requests, finish bounded in-flight work, close listeners | Commands and outbox rows committed before shutdown |
| Ingestion | Database is reachable and required subscriptions have successful SUBACKs | Stop admission, finish or safely abandon bounded handlers, close MQTT | Unacknowledged QoS 1 deliveries and dedupe records, according to configured ack policy |
| Outbox worker | Database and assigned MQTT connection are ready | Stop new claims, finish bounded publish, release/expire leases | Unpublished or expired leases are reclaimed after restart |
## Deployment checks
1. Confirm each simultaneously running MQTT process has a distinct Client ID.
2. Confirm only the ingestion owner subscribes to each response/report filter.
3. Confirm API writes command and outbox intent in one database transaction.
4. Stop API while ingestion stores a result; restart API and query the same command ID.
5. Stop ingestion while API remains live; confirm API can accept/query work and readiness/status reports delayed result processing honestly.
6. Restart either process with an outbox lease and an in-flight QoS 1 delivery; inspect durable state and duplicate handling.
7. Record observed readiness, recovery, and shutdown evidence: `[commands, timestamps, logs, and database query results]`.
## Open questions
- Device completion evidence: `[authoritative contract or owner]`
- Broker session persistence and recovery target: `[configuration or owner]`
- Credential ownership and rotation: `[owner and procedure]`
- Maximum shutdown drain period: `[duration and consequence when exceeded]`
</code></pre></details>
<h2>🧪 Restart either process</h2>
<ol><li>Start API and ingestion independently and verify they use distinct Client IDs and assigned responsibilities.</li><li>Create a harmless ping through HTTP; inspect the durable command and outbox records.</li><li>Stop/restart API while ingestion receives the response. Query status after API restart.</li><li>Stop/restart ingestion while API remains available. Confirm the API can create and query commands while clearly reporting delayed result processing.</li><li>Test shutdown with pending outbox or MQTT work; confirm state remains recoverable and subscriptions restore without duplicate local handler registration.</li></ol>
<div class="card"><h3>⚠️ What to avoid</h3><p>Do not make correctness depend on a shared in-memory waiter map. Do not assign the same stable MQTT Client ID to two simultaneous processes unless they are deliberately replacing one another: they can disconnect each other and alter session ownership. Notifications and caches may accelerate reads, but PostgreSQL remains the correctness baseline.</p></div>
<h2>✅ Done when</h2><p>HTTP status remains accurate when ingestion receives the response in a different process. Either process can restart independently without losing durable command tracking. Your run notes state client IDs, subscription ownership, readiness conditions, and recovery behavior.</p>
<p>📚 Section checklist and process boundaries: <a href="../../COURSE.md#section-15--split-api-and-ingestion-processes">COURSE.md — Section 15</a>. Primary references: <a href="https://pkg.go.dev/os/signal#NotifyContext">Go signal lifecycle</a> and <a href="https://www.postgresql.org/docs/current/explicit-locking.html">PostgreSQL locking</a>.</p>
<p>Ask me to review your ownership table against a proposed process layout before you extract the executables.</p>
<footer><p>Previous: <a href="0014-outbox-failure-gaps.html">Lesson 14 — transactional outbox</a> · Next: <a href="0016-mqtt5-controls.html">Lesson 16 — MQTT 5 controls</a>.</p></footer>
</main></body></html>
teaching/lessons/0016-mqtt5-controls.html
+100
View File
@@ -0,0 +1,100 @@
<!doctype html>
<html lang="en"><head>
<meta charset="utf-8"><meta name="viewport" content="width=device-width, initial-scale=1">
<title>Lesson 16 — Advanced MQTT 5 Controls</title>
<link rel="stylesheet" href="../assets/lesson.css"><script src="../assets/quiz.js" defer></script>
</head><body><main class="page">
<p class="eyebrow">Go MQTT Vending Backend · Lesson 16 · 15 minutes</p>
<h1>Choose MQTT controls from measured behavior</h1>
<p class="dek">Experiment with subscription options, shared consumers, flow limits, packet sizes, aliases, and reason codes without confusing protocol controls with application policy.</p>
<div class="meta"><span class="pill">🎯 One skill: measure before adopting a feature</span><span class="pill">🧠 Predict packet effects</span><span class="pill">🧪 Version-pinned lab</span></div>
<section class="card callout"><p class="question">❓ Start with the system story</p><p>Two ingestion processes share a subscription. A command response reaches the instance that did not create the original process-local waiter. Which state design still lets that instance process the result correctly?</p><p>Answer with the shared durable source of truth, then explore the protocol controls.</p></section>
<h2>📬 Subscription options shape delivery</h2>
<table><thead><tr><th>Option</th><th>Changes</th><th>Does not do</th></tr></thead><tbody><tr><td>No Local</td><td>Suppresses same-client-originated publications for that subscription</td><td>Authorize a topic or filter a device identity</td></tr><tr><td>Retain As Published</td><td>Preserves incoming RETAIN bit on forwarded live traffic</td><td>Choose whether broker stores retained state</td></tr><tr><td>Retain Handling</td><td>Controls retained replay when subscription is established</td><td>Block future live retained publications</td></tr><tr><td>UNSUBSCRIBE</td><td>Removes the broker subscription after UNSUBACK</td><td>Erase retained state or unregister a local Go handler</td></tr></tbody></table>
<p>Change one option at a time with a retained status topic. Record whether the publication was a retained replay or live delivery and inspect the packet flags. Never set No Local as a substitute for ACLs.</p>
<form class="card quiz" data-quiz data-correct="a" data-explanation="Shared consumers distribute publications, but any instance may receive one; durable shared command state and idempotent result processing are needed instead of local waiter affinity." data-retry="A shared subscription doesn't route a response back to the instance that created an in-memory waiter."><p><strong>🧠 Check scaling behavior:</strong> What makes a shared-subscription response safe when it reaches a different ingestion instance?</p><label><input type="radio" name="q" value="a"> Persisted command state and durable idempotency shared by the instances</label><br><label><input type="radio" name="q" value="b"> A larger process-local waiter map</label><br><label><input type="radio" name="q" value="c"> Assuming per-device ordering and instance affinity</label><br><button type="submit">Check answer</button><p data-feedback aria-live="polite"></p></form>
<h2>⚖️ Shared subscriptions and limits</h2>
<p>Run two ingestion instances in one shared group. Observe distribution and what happens when one stops. Do not assume per-device ordering or affinity; both workers must use durable idempotency. MQTT Receive Maximum limits unacknowledged QoS 1/2 protocol exchanges in a direction. It is not your Go queue capacity, and it does not limit QoS 0 in the same way.</p>
<p>Advertised Maximum Packet Size limits packet size in the direction advertised. Test client-to-broker and broker-to-client separately. Compare protocol inflight count with application queue depth; they can differ sharply if callbacks acknowledge before queued work is durable.</p>
<h2>🏷️ Aliases, reason codes, and support matrix</h2>
<p>Topic Alias mappings are scoped to a network connection. A reconnect needs a new mapping even if the MQTT session resumes. Confirm the pinned Paho and Mosquitto support and lifecycle reset behavior before treating aliases as an optimization; a TopicAlias field existing in a type is not evidence that a full flow works.</p>
<p>Interpret reason codes in their packet context. PUBACK “No matching subscribers” is not a device response. A SUBACK result applies to its filter. A context deadline means the local wait ended and may leave business outcome uncertain. Use bounded reconnect backoff with jitter for transient failures; repair authentication/configuration errors instead of retrying faster.</p>
<h2>🧭 File overview: <code>internal/mqtt/controls.go</code></h2>
<p>Create this complete helper file to build MQTT 5 CONNECT and SUBSCRIBE options without mixing broker protocol limits with local queue capacity. The caller still inspects actual CONNACK/SUBACK results and checks version-matched Paho support.</p>
<h3>🧰 <code>NewConnectProperties</code> — short summary</h3><p>Builds validated client Receive Maximum and Maximum Packet Size properties.</p>
<h3>📬 <code>NewSubscription</code> — short summary</h3><p>Builds one subscription and rejects invalid QoS, Retain Handling, and No Local on a shared filter.</p>
<h3>🔎 <code>CapabilitiesFromConnack</code> — short summary</h3><p>Copies optional server capability values safely, preserving nil as “not advertised.”</p>
<details class="card" open><summary><strong>📄 Complete file contents</strong></summary><pre><code>package mqtt
import (
"fmt"
"strings"
"example.com/mqtt5-learning/internal/contracts"
"github.com/eclipse/paho.golang/paho"
)
type ServerCapabilities struct {
SessionPresent bool
MaximumQoS *byte
ReceiveMaximum *uint16
MaximumPacketSize *uint32
TopicAliasMaximum *uint16
SharedSubscriptions bool
RetainedMessages bool
}
func NewConnectProperties(receiveMaximum uint16, maximumPacketSize uint32) (*paho.ConnectProperties, error) {
if receiveMaximum == 0 { return nil, fmt.Errorf("receive maximum must be at least one") }
if maximumPacketSize == 0 { return nil, fmt.Errorf("maximum packet size must be at least one byte") }
packetLimit := maximumPacketSize
flightLimit := receiveMaximum
return &amp;paho.ConnectProperties{ReceiveMaximum: &amp;flightLimit, MaximumPacketSize: &amp;packetLimit}, nil
}
func NewSubscription(filter contracts.TopicFilter, qos, retainHandling byte, noLocal, retainAsPublished bool) (paho.SubscribeOptions, error) {
value := filter.String()
if value == "" { return paho.SubscribeOptions{}, fmt.Errorf("subscription filter is required") }
if qos &gt; 2 { return paho.SubscribeOptions{}, fmt.Errorf("QoS must be 0, 1, or 2") }
if retainHandling &gt; 2 { return paho.SubscribeOptions{}, fmt.Errorf("Retain Handling must be 0, 1, or 2") }
if noLocal &amp;&amp; strings.HasPrefix(value, "$share/") {
return paho.SubscribeOptions{}, fmt.Errorf("No Local cannot be used with a shared subscription")
}
return paho.SubscribeOptions{
Topic: value, QoS: qos, RetainHandling: retainHandling,
NoLocal: noLocal, RetainAsPublished: retainAsPublished,
}, nil
}
func CapabilitiesFromConnack(ack *paho.Connack) ServerCapabilities {
if ack == nil { return ServerCapabilities{} }
result := ServerCapabilities{SessionPresent: ack.SessionPresent}
if ack.Properties == nil { return result }
result.MaximumQoS = copyByte(ack.Properties.MaximumQoS)
result.ReceiveMaximum = copyUint16(ack.Properties.ReceiveMaximum)
result.MaximumPacketSize = copyUint32(ack.Properties.MaximumPacketSize)
result.TopicAliasMaximum = copyUint16(ack.Properties.TopicAliasMaximum)
result.SharedSubscriptions = ack.Properties.SharedSubAvailable
result.RetainedMessages = ack.Properties.RetainAvailable
return result
}
func copyByte(value *byte) *byte {
if value == nil { return nil }; copy := *value; return &amp;copy
}
func copyUint16(value *uint16) *uint16 {
if value == nil { return nil }; copy := *value; return &amp;copy
}
func copyUint32(value *uint32) *uint32 {
if value == nil { return nil }; copy := *value; return &amp;copy
}
</code></pre></details>
<div class="card"><h3>🔎 Details: limits and capability defaults</h3><p>The helper does not resize your application queue or guarantee that the broker supports a requested behavior. The server's absent optional numeric properties stay nil; callers must apply the MQTT-defined defaults for the relevant feature. This small struct records the fields needed by the experiment and does not claim to capture every negotiated capability. Topic Alias state must be rebuilt after a new network connection.</p></div>
<h2>🧪 Build a feature decision matrix</h2>
<ol><li>Record Paho and broker versions and the exact option/API field.</li><li>For each experiment record purpose, packet and direction, prediction, actual observation, and whether you would use the feature.</li><li>Test No Local, Retain As Published, and each Retain Handling value with stored and live messages.</li><li>Run two shared consumers; stop one during traffic and inspect distribution, duplicates, and result processing.</li><li>Test Receive Maximum and packet-size negotiation in both directions. Measure protocol inflight and application queue separately.</li><li>Review Topic Alias and reason-code paths against the pinned source; mark reading-only items as not experimentally verified.</li></ol>
<div class="card"><h3>✅ Done when</h3><p>Your matrix distinguishes MQTT specification, broker, client-library, and application behavior. Every feature has evidence or is marked unverified; no optimization is selected just because the API exposes a field.</p></div>
<details class="card"><summary><strong>📋 Short self-check answers</strong></summary><ol><li>Retain Handling controls replay at subscribe time; Retain As Published controls a flag on forwarded live traffic.</li><li>Shared subscriptions distribute work and do not guarantee affinity; durable state lets any instance process safely.</li><li>Receive Maximum limits in-flight QoS 1/2 packets, not an application's total queue.</li><li>Aliases need to be re-established after reconnect.</li></ol></details>
<p>📚 Detailed experiments and Paho v0.23.0 source pointers: <a href="../course-source/16-mqtt5-controls.md">Lesson 16 source lesson</a>. Primary references: <a href="https://docs.oasis-open.org/mqtt/mqtt/v5.0/mqtt-v5.0.html">OASIS MQTT 5.0 Standard</a> and <a href="https://pkg.go.dev/github.com/eclipse/paho.golang/paho@v0.23.0">Paho v0.23.0 API</a>.</p>
<p>Ask me to map an observed control or reason code to its packet and owning layer.</p>
<footer><p>Previous: <a href="0015-process-boundaries.html">Lesson 15 — process boundaries</a> · Next: <a href="0017-operations-diagnostics.html">Lesson 17 — operate and diagnose</a>.</p></footer>
</main></body></html>
teaching/lessons/0017-operations-diagnostics.html
+83
View File
@@ -0,0 +1,83 @@
<!doctype html>
<html lang="en"><head>
<meta charset="utf-8"><meta name="viewport" content="width=device-width, initial-scale=1">
<title>Lesson 17 — Operate and Diagnose the Service</title>
<link rel="stylesheet" href="../assets/lesson.css"><script src="../assets/quiz.js" defer></script>
</head><body><main class="page">
<p class="eyebrow">Go MQTT Vending Backend · Lesson 17 · 15 minutes</p>
<h1>Diagnose the failing boundary with evidence</h1>
<p class="dek">Make each command traceable across HTTP, Go processes, MQTT, broker, device simulator, and PostgreSQL.</p>
<div class="meta"><span class="pill">🎯 One skill: turn symptoms into a diagnostic path</span><span class="pill">🧠 Predict likely fault layer</span><span class="pill">🧪 Repeatable failure matrix</span></div>
<section class="card callout"><p class="question">❓ Start with the system story</p><p>An API returns 202, the backend records PUBACK, but no device result appears. What evidence do you inspect next, and what can you not conclude yet?</p><p>Trace the same command ID through each owner before labeling the device failed.</p></section>
<h2>🔎 Put identifiers in logs, not metric labels</h2>
<p>Use structured logs with stable command/report IDs, process identity, message kind, and safe error categories. Redact credentials and avoid arbitrary payload dumps. Put high-cardinality IDs in logs/traces where appropriate, not unrestricted metric labels that create a new time series per device or command.</p>
<div class="two-col"><article class="card"><h3>Metrics worth watching</h3><p>Reconnects, rejected messages, queue depth/capacity, duplicate outcomes, publish and response latency, timeouts, DB failures, outbox backlog/oldest age, and shutdown drain results.</p></article><article class="card"><h3>Health endpoints</h3><p>Liveness asks whether a process should be restarted. Readiness asks whether that process can serve its assigned role and dependencies. A connected MQTT client alone does not establish useful service readiness.</p></article></div>
<h2>🧭 File overview: <code>internal/observability/health.go</code></h2>
<p>Create this complete standard-library file for the first operational instrumentation slice. It exposes liveness, dependency-aware readiness, and low-cardinality Prometheus text metrics. It does not label metrics with device or command IDs.</p>
<h3>🩺 <code>NewHandler</code> and readiness setters — short summary</h3><p>Create one per process role and update its dependency flags when DB/MQTT readiness changes.</p>
<h3>📈 <code>ServeHTTP</code> — short summary</h3><p>Serves bounded health endpoints and fixed metric names without exposing secrets or arbitrary IDs.</p>
<details class="card" open><summary><strong>📄 Complete file contents</strong></summary><pre><code>package observability
import (
"fmt"
"net/http"
"sync/atomic"
)
type Handler struct {
role string
requiresMQTT bool
databaseReady atomic.Bool
mqttReady atomic.Bool
rejected atomic.Uint64
duplicates atomic.Uint64
reconnects atomic.Uint64
completed atomic.Uint64
}
func NewHandler(role string, requiresMQTT bool) (*Handler, error) {
if role != "api" &amp;&amp; role != "ingest" { return nil, fmt.Errorf("role must be api or ingest") }
return &amp;Handler{role: role, requiresMQTT: requiresMQTT}, nil
}
func (h *Handler) SetDatabaseReady(ready bool) { h.databaseReady.Store(ready) }
func (h *Handler) SetMQTTReady(ready bool) { h.mqttReady.Store(ready) }
func (h *Handler) RecordRejected() { h.rejected.Add(1) }
func (h *Handler) RecordDuplicate() { h.duplicates.Add(1) }
func (h *Handler) RecordReconnect() { h.reconnects.Add(1) }
func (h *Handler) RecordCompleted() { h.completed.Add(1) }
func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
switch r.URL.Path {
case "/livez":
w.WriteHeader(http.StatusNoContent)
case "/readyz":
if !h.databaseReady.Load() || (h.requiresMQTT &amp;&amp; !h.mqttReady.Load()) {
http.Error(w, "required dependency is not ready", http.StatusServiceUnavailable)
return
}
w.WriteHeader(http.StatusNoContent)
case "/metrics":
w.Header().Set("Content-Type", "text/plain; version=0.0.4; charset=utf-8")
fmt.Fprintf(w, "mqtt_learning_messages_rejected_total{role=%q} %d\n", h.role, h.rejected.Load())
fmt.Fprintf(w, "mqtt_learning_messages_duplicate_total{role=%q} %d\n", h.role, h.duplicates.Load())
fmt.Fprintf(w, "mqtt_learning_reconnects_total{role=%q} %d\n", h.role, h.reconnects.Load())
fmt.Fprintf(w, "mqtt_learning_work_completed_total{role=%q} %d\n", h.role, h.completed.Load())
default:
http.NotFound(w, r)
}
}
</code></pre></details>
<div class="card"><h3>🔎 Details: operational meaning</h3><p>Liveness stays independent of database/MQTT so a dependency outage does not trigger a restart loop. Readiness requires PostgreSQL for both processes and MQTT subscriptions for ingestion. The metrics are counters, not full histogram/queue instrumentation; add queue gauges and latency histograms when the service has those measurements. Use process role as a bounded label and keep IDs in structured logs instead.</p></div>
<form class="card quiz" data-quiz data-correct="c" data-explanation="The API and PUBACK are evidence for earlier boundaries. Inspect broker subscription/delivery, ingestion logs and counters, DB result state, and simulator response before deciding where the path stopped." data-retry="Follow the command ID and list what each previous observation actually proves."><p><strong>🧠 Check the diagnosis:</strong> After 202 and PUBACK with no device result, what conclusion is justified?</p><label><input type="radio" name="q" value="a"> The device definitely completed the command</label><br><label><input type="radio" name="q" value="b"> The device definitely received nothing</label><br><label><input type="radio" name="q" value="c"> The outcome is not established; trace delivery, device processing, ingestion, and persisted result evidence</label><br><button type="submit">Check answer</button><p data-feedback aria-live="polite"></p></form>
<h2>🧪 Make a repeatable load and failure scenario</h2>
<p>Start with a synthetic simulator load that has a fixed duration, device count, message rate, and payload mix. Measure throughput, latency percentiles, memory, queue growth, and errors. Profile after evidence identifies a bottleneck; a large queue is not a throughput fix if admission exceeds processing indefinitely.</p>
<table><thead><tr><th>Injected fault</th><th>Evidence to collect</th><th>Recovery check</th></tr></thead><tbody><tr><td>Broker down/restart</td><td>Reconnect state, Session Present, subscription results</td><td>Expected subscriptions return; duplicate handling stays safe</td></tr><tr><td>Simulator offline or restart</td><td>Command state, expiry/deadline, session behavior</td><td>Late work is safe; status remains honest</td></tr><tr><td>Slow/unavailable DB</td><td>Query latency/errors, manual Ack policy, backlog</td><td>No false durable acceptance; recovery resumes</td></tr><tr><td>Duplicate report/result</td><td>Stable ID and duplicate counters</td><td>One business effect; stored outcome remains stable</td></tr><tr><td>Denied topic</td><td>Broker logs/reason and application visibility</td><td>No unauthorized effect or data leakage</td></tr><tr><td>Shutdown under load</td><td>Queue depth, accepted/rejected, drain deadline</td><td>Recoverable work is preserved or loss is explicitly recorded</td></tr></tbody></table>
<h2>📓 Write a runbook another person can follow</h2>
<ol><li>State the symptom and its user-visible impact.</li><li>Collect command/report ID, timestamps, process logs, metrics, broker evidence, and relevant database state without secrets.</li><li>Map the last confirmed evidence to HTTP, application, broker, device, or database boundary.</li><li>Give the bounded recovery action and note the risk of retrying a possibly completed physical effect.</li><li>Verify recovery with a harmless synthetic operation and record the evidence that changed.</li></ol>
<div class="card"><h3>✅ Done when</h3><p>Someone who has not read your code can use the runbook to diagnose one injected command failure. Your load scenario and fault matrix can be repeated, and each metric helps answer an operational question.</p></div>
<details class="card"><summary><strong>📋 Short self-check answers</strong></summary><ol><li>202 is API acceptance; PUBACK is one MQTT hop. Neither proves device completion.</li><li>Readiness reflects dependencies needed for the process's assigned work; liveness only says whether it is functioning enough to keep running.</li><li>Metric labels should remain bounded; use logs/traces for high-cardinality IDs.</li></ol></details>
<p>📚 Section checklist: <a href="../../COURSE.md#section-17--operate-and-diagnose-the-service">COURSE.md — Section 17</a>. Primary references: <a href="https://pkg.go.dev/net/http">Go <code>net/http</code></a> and <a href="https://prometheus.io/docs/instrumenting/exposition_formats/">Prometheus exposition format</a>.</p>
<p>Ask me to help map a runbook symptom to the next useful measurement.</p>
<footer><p>Previous: <a href="0016-mqtt5-controls.html">Lesson 16 — MQTT 5 controls</a> · Next: <a href="0018-backend-handover.html">Lesson 18 — backend handover</a>.</p></footer>
</main></body></html>
teaching/lessons/0018-backend-handover.html
+108
View File
@@ -0,0 +1,108 @@
<!doctype html>
<html lang="en"><head>
<meta charset="utf-8"><meta name="viewport" content="width=device-width, initial-scale=1">
<title>Lesson 18 — Prepare for the MQTT 5 Backend Handover</title>
<link rel="stylesheet" href="../assets/lesson.css"><script src="../assets/quiz.js" defer></script>
</head><body><main class="page">
<p class="eyebrow">Go MQTT Vending Backend · Lesson 18 · Capstone</p>
<h1>Turn course knowledge into a safe backend handover</h1>
<p class="dek">Inspect the delivered system from evidence, trace its business flows, and prepare a narrow maintenance change with a rollback plan.</p>
<div class="meta"><span class="pill">🎯 One skill: explain a system you did not design</span><span class="pill">🧠 Separate source evidence from assumptions</span><span class="pill">🧪 Authorized staging only</span></div>
<section class="card callout"><p class="question">❓ Start with the handover story</p><p>You receive a new MQTT 5 backend repository. The course lab uses different topics, IDs, and packages. What must you inspect before proposing an import change or device rollout?</p><p>Start with commit, dependency and deployment evidence, then authoritative contracts and end-to-end traces.</p></section>
<h2>🧭 Re-establish the actual system</h2>
<p>Inspect the delivered commit, Go modules, broker configuration, deployment topology, and device contracts. Record which observations come from source, tests, broker configuration, staging, or a human owner. Do not carry lab topic names or identity assumptions into production.</p>
<form class="card quiz" data-quiz data-correct="b" data-explanation="The delivered behavior and authoritative contracts define the migration. MQTT 3.1.1 and MQTT 5 clients are different APIs and feature support must be checked per path/device cohort." data-retry="A package import is not evidence that protocol behavior or device contracts are compatible."><p><strong>🧠 Check the handover:</strong> What is the right starting point for a delivered MQTT 5 backend?</p><label><input type="radio" name="q" value="a"> Replace the MQTT 3 import and assume the same behavior</label><br><label><input type="radio" name="q" value="b"> Reinspect exact commit, dependencies, broker, contracts, and deployment; trace behavior before proposing a change</label><br><label><input type="radio" name="q" value="c"> Reuse every topic and identity from the learning lab</label><br><button type="submit">Check answer</button><p data-feedback aria-live="polite"></p></form>
<h2>🔬 Trace three complete business flows</h2>
<ol><li><strong>HTTP command:</strong> API authorization → idempotency/command record → publication and PUBACK → device acceptance/completion → persisted result → API status.</li><li><strong>Sale report:</strong> device report identity → topic/payload validation → deduplication and transaction → MQTT acknowledgment decision → application receipt or retry.</li><li><strong>Catalog/configuration:</strong> backend revision/job → publish → device validation/application → reported revision/result → retry and stale-revision policy.</li></ol>
<p>For each step, name the process and durable state owner. Record actual identifiers, topic ACLs, schema versions, timeout/expiry behavior, and evidence. If source cannot answer a question—such as physical device semantics or broker recovery targets—put it in the handover question list instead of guessing.</p>
<h2>🔀 Compare protocol generations by behavior</h2>
<p>Build an MQTT 3.1.1-to-5 comparison for each real flow: connection/session behavior, reason handling, expiry, properties, request/response, subscription options, acknowledgement timing, and client-library lifecycle. Mark feature support by device cohort. A legacy device may ignore or lack MQTT 5 properties; a server migration does not make the device understand them.</p>
<h2>🛠️ Propose one narrow maintenance change</h2>
<p>Choose a change with a reproducible symptom and bounded blast radius. Define expected behavior, regression coverage, observability, rollout checks, and rollback criteria before implementing. Use authorized staging simulators for acceptance checks; production migration or deployment is separate scope.</p>
<div class="two-col"><article class="card"><h3>Evidence checklist</h3><p>Commit/dependencies, contract source, topic and ACL ownership, command/report identities, persistence boundaries, retry/expiry rules, and process readiness.</p></article><article class="card"><h3>Questions for owners</h3><p>What exactly does device acknowledgment mean? How is broker persistence configured? What recovery target applies? Who rotates credentials and owns escalation?</p></article></div>
<h2>📄 Complete handover artifact: <code>docs/backend-handover.md</code></h2>
<p>Create this report from the delivered checkout. Replace every bracket with evidence or an explicit unanswered question. It is a complete documentation artifact for this non-coding capstone; it does not prescribe Go source changes.</p>
<details class="card" open><summary><strong>🗂️ Full file contents</strong></summary><pre><code># MQTT backend handover
## Scope and evidence
- Repository and commit: `[URL/path and commit SHA]`
- Review date and reviewer: `[date, name]`
- Runtime environments inspected: `[local, staging, production read-only, other]`
- Evidence collected: `[source paths, tests, broker config, logs, queries, owner answers]`
- Facts that remain unverified: `[list; do not infer from the learning lab]`
## Runtime map
| Process | Entry point | MQTT Client ID | Owns subscriptions/publication | Durable dependencies | Readiness evidence |
| --- | --- | --- | --- | --- | --- |
| `[role]` | `[path]` | `[identity]` | `[filters and publish classes]` | `[database/queue]` | `[endpoint or observation]` |
## Wire and identity contract
- MQTT protocol versions by device cohort: `[evidence]`
- Device identity and credential mapping: `[source/config/owner]`
- Topic namespace and ACL owner: `[evidence]`
- Payload schema versions and compatibility policy: `[evidence]`
- Correlation, idempotency, and deduplication keys: `[evidence]`
- MQTT 3.1.1 versus MQTT 5 behavior differences: `[feature, cohort, test result]`
## End-to-end traces
### HTTP command
`[authorization]` → `[durable acceptance]` → `[publish and PUBACK evidence]` → `[device acceptance]` → `[completion/result]` → `[status query]`
For each boundary, record process, stable ID, durable owner, and evidence: `[details]`.
### Sale report
`[device report ID]` → `[topic/payload validation]` → `[transaction and dedupe]` → `[MQTT acknowledgment decision]` → `[application receipt/retry]`
For each boundary, record process, stable ID, durable owner, and evidence: `[details]`.
### Catalog or configuration update
`[revision/job]` → `[publish]` → `[device validation/application]` → `[reported revision/result]` → `[stale/retry policy]`
For each boundary, record process, stable ID, durable owner, and evidence: `[details]`.
## Recovery and operations
- Session persistence, expiry, and reconnect behavior: `[evidence]`
- Database migrations and backup/restore owner: `[evidence]`
- Outbox/retry/dead-letter behavior: `[evidence]`
- Liveness and readiness meaning by process: `[evidence]`
- Failure reproduced and diagnostic path: `[symptom, IDs, evidence, layer, recovery]`
- Physical-effect ambiguity and reconciliation policy: `[evidence or owner question]`
## Proposed maintenance change
- Reproduction and observed result: `[steps and evidence]`
- Expected behavior and acceptance check: `[specific]`
- Files and wire contracts affected: `[paths; distinguish internal from device-facing change]`
- Regression coverage and staging cohort: `[checks and devices]`
- Rollout guardrails and abort signal: `[metrics/threshold/owner]`
- Rollback condition and recovery action: `[specific]`
## Questions requiring an owner
1. Device acknowledgment semantics: `[owner, answer, or unresolved]`
2. Broker persistence and recovery target: `[owner, answer, or unresolved]`
3. Credential rotation and revocation: `[owner, answer, or unresolved]`
4. Escalation and physical reconciliation: `[owner, answer, or unresolved]`
## Handover decision
- Confirmed safe maintenance scope: `[narrow change or none]`
- Explicitly excluded rollout scope: `[devices/environments/actions]`
- Evidence reviewer still needs: `[list]`
</code></pre></details>
<h2>🧪 Capstone review</h2>
<ol><li>Explain one command, one sale, and one catalog/config flow end-to-end with evidence at each boundary.</li><li>Inject one staging failure and diagnose it with the Section 17 runbook.</li><li>Review mixed MQTT 3.1.1/5 interoperability and list where feature loss is expected.</li><li>Present one narrow change with reproduction, behavior, regression test, rollout checks, and rollback condition.</li><li>Deliver a question list for facts that source and staging cannot establish.</li></ol>
<div class="card"><h3>🏁 Course completion</h3><p>You are ready to maintain the delivered system when you can explain the traced flows, diagnose an injected failure, justify a small change with evidence, and distinguish confirmed behavior from unanswered operational questions. This checkpoint does not itself authorize a production rollout.</p></div>
<details class="card"><summary><strong>📋 Short self-check answers</strong></summary><ol><li>Reinspect the delivered checkout and authoritative contracts before selecting APIs.</li><li>A broker ACK, application result, durable business record, and physical effect are distinct evidence.</li><li>Compatibility must be checked per device cohort and behavior, not inferred from the broker/client migration.</li></ol></details>
<p>📚 Course handover checklist: <a href="../../COURSE.md#section-18--prepare-for-the-mqtt-5-backend-handover">COURSE.md — Section 18</a>. Primary references: <a href="https://docs.oasis-open.org/mqtt/mqtt/v5.0/mqtt-v5.0.html">OASIS MQTT 5.0 Standard</a>, <a href="https://docs.oasis-open.org/mqtt/mqtt/v3.1.1/os/mqtt-v3.1.1-os.html">OASIS MQTT 3.1.1 Standard</a>, and <a href="https://pkg.go.dev/cmd/go">Go command documentation</a>.</p>
<p>Ask me to review a completed handover report and help separate confirmed facts from unresolved owner questions.</p>
<footer><p>Previous: <a href="0017-operations-diagnostics.html">Lesson 17 — operations and diagnosis</a> · End of the course plan.</p></footer>
</main></body></html>