Refactor MQTT course and bound backend message processing

Move lesson source into the teaching structure, align the course path with the learner's Go and DevOps experience, and add the interactive lesson guides. Add advanced DevOps lifecycle study as a separate section.

Bound MQTT message work and drain accepted jobs during shutdown; align the backend implementation with the revised course through Section 6.

teaching/lessons/0014-outbox-failure-gaps.html

@@ -0,0 +1,149 @@
+<!doctype html>
+<html lang="en"><head>
+  <meta charset="utf-8"><meta name="viewport" content="width=device-width, initial-scale=1">
+  <title>Lesson 14 — Close Database and MQTT Failure Gaps</title>
+  <link rel="stylesheet" href="../assets/lesson.css"><script src="../assets/quiz.js" defer></script>
+</head><body><main class="page">
+  <p class="eyebrow">Go MQTT Vending Backend · Lesson 14 · 15 minutes</p>
+  <h1>Keep accepted publish intent recoverable</h1>
+  <p class="dek">Use a transactional outbox to bridge the database and MQTT broker while keeping retries, duplicates, and uncertain device outcomes visible.</p>
+  <div class="meta"><span class="pill">🎯 One skill: reason about cross-system failure</span><span class="pill">🧠 Predict duplicate publication</span><span class="pill">🧪 Synthetic commands only</span></div>
+  <section class="card callout"><p class="question">❓ Start with the system story</p><p>The API commits a ping command to PostgreSQL, then crashes before publishing it. What durable record lets another process resume the accepted work?</p><p>Name which values must commit together and which process retries them.</p></section>
+  <h2>🧾 Commit business intent and publish intent together</h2>
+  <div class="flow" aria-label="transactional outbox flow"><div class="step">HTTP API<br><strong>validate + authorize</strong></div><span class="arrow">→</span><div class="step">One DB transaction<br><strong>command + outbox row</strong></div><span class="arrow">→</span><div class="step">Publisher worker<br><strong>bounded claim + retry</strong></div><span class="arrow">→</span><div class="step">MQTT + device<br><strong>stable ID + dedupe</strong></div></div>
+  <p>The transaction closes the gap where a command is durably accepted but the intent to publish is lost. A worker claims rows briefly, publishes outside the DB transaction, then records the attempt. Do not hold a database transaction open while waiting on a network operation.</p>
+  <form class="card quiz" data-quiz data-correct="b" data-explanation="The command and its publish intent are committed atomically. A worker can recover the pending outbox row after restart." data-retry="Which database rows can exist after a crash immediately following command acceptance?"><p><strong>🧠 Check the guarantee:</strong> What makes accepted work publishable after an API process crashes before MQTT publish?</p><label><input type="radio" name="q" value="a"> A successful HTTP 202 by itself</label><br><label><input type="radio" name="q" value="b"> The command and outbox intent committed in the same DB transaction</label><br><label><input type="radio" name="q" value="c"> A retained command message without an application record</label><br><button type="submit">Check answer</button><p data-feedback aria-live="polite"></p></form>
+  <h2>🪟 The outbox still permits duplicates</h2>
+  <p>If the broker accepts a PUBLISH and the backend crashes before marking its outbox row sent, the worker may publish it again after restart. Keep one command ID across retries. The simulator stores the prior effect/result by command ID and returns that outcome on a duplicate. A timeout after possible publication is an unknown device outcome, not proof that it did nothing.</p>
+  <table><thead><tr><th>Crash point</th><th>What remains</th><th>Recovery</th></tr></thead><tbody><tr><td>Before transaction commit</td><td>No accepted command/outbox</td><td>Caller may retry with the same HTTP idempotency key</td></tr><tr><td>After commit, before publish</td><td>Pending outbox row</td><td>Worker publishes after restart</td></tr><tr><td>After publish, before sent marker</td><td>Row may remain pending</td><td>Republish same command ID; device deduplicates</td></tr><tr><td>After device effect, before result</td><td>Device may have acted; backend lacks result</td><td>Expose unknown/pending and reconcile by stable ID</td></tr></tbody></table>
+  <h2>🧭 File overview: <code>internal/outbox/store.go</code></h2>
+  <p>Create this complete PostgreSQL repository file for the outbox chapter. The HTTP API calls <code>Enqueue</code> in its acceptance path; a bounded worker uses <code>ClaimOne</code> and records the PUBACK outcome with <code>MarkPublished</code>. No database transaction stays open during the network publish.</p>
+  <h3>🧾 <code>Enqueue</code> — short summary</h3><p>Commits the command state and serialized publication intent atomically under one command ID.</p>
+  <h3>🔒 <code>ClaimOne</code> — short summary</h3><p>Uses a short transaction and <code>FOR UPDATE SKIP LOCKED</code> to lease one eligible row for one worker.</p>
+  <h3>📬 <code>MarkPublished</code> — short summary</h3><p>Records the broker acknowledgement as transport evidence and marks only that command's outbox row sent.</p>
+  <details class="card" open><summary><strong>🧩 Complete file contents</strong></summary><pre><code>package outbox
+
+import (
+	"context"
+	"database/sql"
+	"errors"
+	"fmt"
+	"time"
+)
+
+var ErrNoWork = errors.New("no eligible outbox row")
+
+type Command struct {
+	ID        string
+	DeviceID  string
+	Topic     string
+	Payload   []byte
+	Deadline  time.Time
+}
+
+type Item struct {
+	CommandID string
+	DeviceID  string
+	Topic     string
+	Payload   []byte
+	Deadline  time.Time
+	Attempt   int
+}
+
+type Store struct { db *sql.DB }
+
+func NewStore(db *sql.DB) (*Store, error) {
+	if db == nil { return nil, fmt.Errorf("database connection is required") }
+	return &amp;Store{db: db}, nil
+}
+
+func (s *Store) Migrate(ctx context.Context) error {
+	statements := []string{
+		`CREATE TABLE IF NOT EXISTS device_commands (
+			command_id TEXT PRIMARY KEY, device_id TEXT NOT NULL, state TEXT NOT NULL,
+			deadline TIMESTAMPTZ NOT NULL, created_at TIMESTAMPTZ NOT NULL DEFAULT now())`,
+		`CREATE TABLE IF NOT EXISTS command_outbox (
+			command_id TEXT PRIMARY KEY REFERENCES device_commands(command_id),
+			topic TEXT NOT NULL, payload BYTEA NOT NULL, attempts INTEGER NOT NULL DEFAULT 0,
+			available_at TIMESTAMPTZ NOT NULL DEFAULT now(), lease_until TIMESTAMPTZ,
+			published_at TIMESTAMPTZ, last_reason SMALLINT)`,
+	}
+	for _, statement := range statements {
+		if _, err := s.db.ExecContext(ctx, statement); err != nil {
+			return fmt.Errorf("create outbox schema: %w", err)
+		}
+	}
+	return nil
+}
+
+func (s *Store) Enqueue(ctx context.Context, command Command) error {
+	if command.ID == "" || command.DeviceID == "" || command.Topic == "" || len(command.Payload) == 0 {
+		return fmt.Errorf("command ID, device, topic, and payload are required")
+	}
+	if !command.Deadline.After(time.Now()) {
+		return fmt.Errorf("command deadline must be in the future")
+	}
+	tx, err := s.db.BeginTx(ctx, nil)
+	if err != nil { return fmt.Errorf("begin command transaction: %w", err) }
+	defer func() { _ = tx.Rollback() }()
+	if _, err := tx.ExecContext(ctx, `INSERT INTO device_commands(command_id,device_id,state,deadline)
+VALUES ($1,$2,'pending',$3)`, command.ID, command.DeviceID, command.Deadline); err != nil {
+		return fmt.Errorf("insert command: %w", err)
+	}
+	if _, err := tx.ExecContext(ctx, `INSERT INTO command_outbox(command_id,topic,payload)
+VALUES ($1,$2,$3)`, command.ID, command.Topic, command.Payload); err != nil {
+		return fmt.Errorf("insert outbox intent: %w", err)
+	}
+	if err := tx.Commit(); err != nil { return fmt.Errorf("commit command and outbox: %w", err) }
+	return nil
+}
+
+func (s *Store) ClaimOne(ctx context.Context, lease time.Duration, now time.Time) (Item, error) {
+	if lease &lt;= 0 { return Item{}, fmt.Errorf("lease must be positive") }
+	tx, err := s.db.BeginTx(ctx, nil)
+	if err != nil { return Item{}, fmt.Errorf("begin outbox claim: %w", err) }
+	defer func() { _ = tx.Rollback() }()
+	var item Item
+	err = tx.QueryRowContext(ctx, `SELECT o.command_id,c.device_id,o.topic,o.payload,c.deadline,o.attempts
+FROM command_outbox o JOIN device_commands c USING (command_id)
+WHERE o.published_at IS NULL AND o.available_at &lt;= $1
+  AND (o.lease_until IS NULL OR o.lease_until &lt;= $1)
+  AND c.deadline &gt; $1
+ORDER BY o.available_at,o.command_id
+LIMIT 1 FOR UPDATE OF o SKIP LOCKED`, now).Scan(
+		&amp;item.CommandID, &amp;item.DeviceID, &amp;item.Topic, &amp;item.Payload, &amp;item.Deadline, &amp;item.Attempt)
+	if errors.Is(err, sql.ErrNoRows) { return Item{}, ErrNoWork }
+	if err != nil { return Item{}, fmt.Errorf("select outbox work: %w", err) }
+	item.Payload = append([]byte(nil), item.Payload...)
+	if _, err := tx.ExecContext(ctx, `UPDATE command_outbox
+SET lease_until=$2, attempts=attempts+1 WHERE command_id=$1`, item.CommandID, now.Add(lease)); err != nil {
+		return Item{}, fmt.Errorf("lease outbox work: %w", err)
+	}
+	if err := tx.Commit(); err != nil { return Item{}, fmt.Errorf("commit outbox lease: %w", err) }
+	item.Attempt++
+	return item, nil
+}
+
+func (s *Store) MarkPublished(ctx context.Context, commandID string, reason byte, at time.Time) error {
+	result, err := s.db.ExecContext(ctx, `UPDATE command_outbox SET published_at=$2,
+lease_until=NULL,last_reason=$3 WHERE command_id=$1 AND published_at IS NULL`, commandID, at, reason)
+	if err != nil { return fmt.Errorf("mark outbox published: %w", err) }
+	rows, err := result.RowsAffected()
+	if err != nil { return fmt.Errorf("inspect outbox update: %w", err) }
+	if rows != 1 { return fmt.Errorf("outbox command %q was not pending", commandID) }
+	return nil
+}
+</code></pre></details>
+  <div class="card"><h3>🔎 Details: lease and retry policy</h3><p>The lease stops two active workers from claiming the same row at once. A crashed worker leaves a lease that expires; a later worker can claim and republish. A DB transaction makes intent durable, but it cannot atomically include the MQTT PUBLISH. <code>MarkPublished</code> therefore follows the PUBLISH and can race a crash; the device must deduplicate. Production code should add bounded exponential backoff, a retryable/permanent error classification, and a separate terminal outcome for expired rows rather than retrying forever.</p></div>
+  <h2>📚 Keep three queues separate</h2>
+  <div class="two-col"><article class="card"><h3>Transactional outbox</h3><p>Application DB owns durable business intent and retry status. It does not prove broker delivery or device completion.</p></article><article class="card"><h3>Paho outgoing queue</h3><p>Client library owns transport queueing under its configured semantics. It is not atomic with the business transaction.</p></article><article class="card"><h3>MQTT session state</h3><p>Client/broker own protocol session continuity and eligible queued deliveries. It does not store business intent or dedupe effects.</p></article></div>
+  <h2>⏳ Bound retries and stale work</h2>
+  <p>Set a retry window with backoff and jitter, bound concurrent claims and publishes, and measure pending row count and oldest age. Check the application deadline before sending; use MQTT Message Expiry for broker-queued lifetime; have the device check its deadline before execution. Park expired or permanent failures and preserve unknown outcomes.</p>
+  <h2>🧪 Inject each failure window</h2>
+  <ol><li>Commit an outbox row while the publisher is stopped; restart it and confirm the row is attempted.</li><li>Stop the worker after publish but before marking sent. Restart and confirm duplicate publication keeps the command ID and causes one simulated effect.</li><li>Drop the device response after its simulated effect. Verify status does not claim definite failure.</li><li>Expire a row before publish and verify it is not sent.</li><li>Restart the broker and compare the pending application outbox state with broker session recovery.</li></ol>
+  <div class="card"><h3>✅ Done when</h3><p>Every crash point has a recovery policy; duplicate publication creates one simulated effect; expiry blocks stale work; status separates PUBACK from device completion.</p><p><strong>Evidence to record:</strong> prediction, injected crash point, DB rows, MQTT evidence, simulator effect count, final command state, and cleanup.</p></div>
+  <details class="card"><summary><strong>📋 Short self-check answers</strong></summary><ol><li>Commit the command and outbox intent atomically.</li><li>The worker can crash after PUBACK but before the sent marker; same-ID retry may duplicate.</li><li>Paho queue admission does not prove broker delivery or business completion.</li><li>Report unknown when the device may have acted but the application response is missing.</li></ol></details>
+  <p>📚 Full crash experiment plan: <a href="../course-source/14-database-mqtt-failure-gaps.md">Lesson 14 source lesson</a>. Primary references: <a href="https://www.postgresql.org/docs/current/sql-select.html">PostgreSQL locking clauses</a> and <a href="https://docs.oasis-open.org/mqtt/mqtt/v5.0/mqtt-v5.0.html">OASIS MQTT 5.0 Standard</a>.</p>
+  <p>Ask me to trace a crash window against the durable row and MQTT packet evidence.</p>
+  <footer><p>Previous: <a href="0013-persist-reports-command-state.html">Lesson 13 — durable persistence</a> · Next: <a href="0015-process-boundaries.html">Lesson 15 — split API and ingestion</a>.</p></footer>
+</main></body></html>