mqtt-5-learning/teaching/lessons/0015-process-boundaries.html

<!doctype html>
<html lang="en"><head>
  <meta charset="utf-8"><meta name="viewport" content="width=device-width, initial-scale=1">
  <title>Lesson 15 — Split API and Ingestion Processes</title>
  <link rel="stylesheet" href="../assets/lesson.css"><script src="../assets/quiz.js" defer></script>
</head><body><main class="page">
  <p class="eyebrow">Go MQTT Vending Backend · Lesson 15 · 12 minutes</p>
  <h1>Split processes around durable ownership</h1>
  <p class="dek">Let an HTTP API accept and query durable commands while an independent MQTT ingestion process stores device results.</p>
  <div class="meta"><span class="pill">🎯 One skill: design across process boundaries</span><span class="pill">🧠 Predict independent restarts</span><span class="pill">🧪 Local processes</span></div>
  <section class="card callout"><p class="question">❓ Start with the system story</p><p>The API process creates a command. A separate ingestion process receives the device result. Which process owns the command's pending waiter, and how can HTTP status remain correct if the API restarts?</p><p>Pause: a Go map shared only in one process cannot be the answer.</p></section>
  <h2>🗂️ Put shared truth in PostgreSQL</h2>
  <div class="flow" aria-label="separated services"><div class="step">HTTP API<br><strong>authorize + write command</strong></div><span class="arrow">→</span><div class="step">PostgreSQL<br><strong>command/outbox/result state</strong></div><span class="arrow">↔</span><div class="step">Ingestion process<br><strong>subscribe + validate + persist</strong></div></div>
  <p>The API serves HTTP, checks user/device permission, creates durable command state and outbox intent, and reads status. Ingestion owns MQTT subscriptions and result processing. Both use the database as their shared source of truth. A pending-request map can still optimize local waiting, but it cannot be required for correctness across processes.</p>
  <form class="card quiz" data-quiz data-correct="c" data-explanation="Persisted command/result state is shared across the API and ingestion processes and survives an API restart; notifications can be added for latency but do not replace it." data-retry="Ask what both processes can read after either process restarts."><p><strong>🧠 Check the ownership:</strong> How should API status reflect a result received in another process?</p><label><input type="radio" name="q" value="a"> Copy a Go map between processes</label><br><label><input type="radio" name="q" value="b"> Assume the API process owns every response callback</label><br><label><input type="radio" name="q" value="c"> Read persisted command/result state; use notifications only as an optional latency aid</label><br><button type="submit">Check answer</button><p data-feedback aria-live="polite"></p></form>
  <h2>🧩 Give each process its own lifecycle</h2>
  <p>Use separate executables, signal contexts, health endpoints, and MQTT Client IDs. The ingestion process owns the response/telemetry subscriptions; the API or a dedicated publisher worker owns command publication. State that ownership explicitly so two processes do not accidentally compete for a device command subscription.</p>
  <p>Readiness should mean the process can perform its assigned role: API readiness can include database availability and HTTP dependencies; ingestion readiness can include database availability and restored required MQTT subscriptions. Avoid arbitrary startup sleeps. Expose the dependency that is not ready.</p>
  <h2>📄 Complete artifact: <code>docs/process-ownership.md</code></h2>
  <p>This is the complete ownership contract to create before extracting executables. Fill every bracketed value from the actual deployment; the template is intentionally not a pretend Go implementation. It makes the process split reviewable and catches duplicate MQTT ownership before deployment.</p>
  <details class="card" open><summary><strong>🧭 Full file contents</strong></summary><pre><code># Process ownership

## Runtime identities

| Process | Executable | MQTT Client ID | MQTT responsibilities | HTTP responsibilities |
| --- | --- | --- | --- | --- |
| API | `cmd/api` | `[none or explicit publisher ID]` | `[publish only if assigned; never subscribe to device results]` | Authorize callers, create durable commands, query command state, serve `/livez` and `/readyz` |
| Ingestion | `cmd/ingest` | `[unique stable ingestion ID]` | Own response and report subscriptions; validate and persist inbound messages | Serve `/livez` and `/readyz` if this role exposes health over HTTP |
| Outbox worker | `[same API process or separate executable]` | `[unique publisher ID when separate]` | Claim durable publish intent, publish, record broker outcome | No public command-creation endpoint |

## Durable state contract

- PostgreSQL owns accepted command state, the outbox, deduplication keys, and received results.
- A process-local waiter or cache may reduce latency; it never decides correctness.
- Every retry reuses the original command ID and idempotency key.
- A broker acknowledgment records a transport event. Device acceptance, completion, and physical effect require separate evidence.
- `[Describe migration ownership and how both versions remain compatible during rollout.]`

## Readiness and lifecycle

| Role | Ready when | Shutdown sequence | Recoverable work |
| --- | --- | --- | --- |
| API | Database is reachable and HTTP dependencies are initialized | Stop accepting requests, finish bounded in-flight work, close listeners | Commands and outbox rows committed before shutdown |
| Ingestion | Database is reachable and required subscriptions have successful SUBACKs | Stop admission, finish or safely abandon bounded handlers, close MQTT | Unacknowledged QoS 1 deliveries and dedupe records, according to configured ack policy |
| Outbox worker | Database and assigned MQTT connection are ready | Stop new claims, finish bounded publish, release/expire leases | Unpublished or expired leases are reclaimed after restart |

## Deployment checks

1. Confirm each simultaneously running MQTT process has a distinct Client ID.
2. Confirm only the ingestion owner subscribes to each response/report filter.
3. Confirm API writes command and outbox intent in one database transaction.
4. Stop API while ingestion stores a result; restart API and query the same command ID.
5. Stop ingestion while API remains live; confirm API can accept/query work and readiness/status reports delayed result processing honestly.
6. Restart either process with an outbox lease and an in-flight QoS 1 delivery; inspect durable state and duplicate handling.
7. Record observed readiness, recovery, and shutdown evidence: `[commands, timestamps, logs, and database query results]`.

## Open questions

- Device completion evidence: `[authoritative contract or owner]`
- Broker session persistence and recovery target: `[configuration or owner]`
- Credential ownership and rotation: `[owner and procedure]`
- Maximum shutdown drain period: `[duration and consequence when exceeded]`
</code></pre></details>
  <h2>🧪 Restart either process</h2>
  <ol><li>Start API and ingestion independently and verify they use distinct Client IDs and assigned responsibilities.</li><li>Create a harmless ping through HTTP; inspect the durable command and outbox records.</li><li>Stop/restart API while ingestion receives the response. Query status after API restart.</li><li>Stop/restart ingestion while API remains available. Confirm the API can create and query commands while clearly reporting delayed result processing.</li><li>Test shutdown with pending outbox or MQTT work; confirm state remains recoverable and subscriptions restore without duplicate local handler registration.</li></ol>
  <div class="card"><h3>⚠️ What to avoid</h3><p>Do not make correctness depend on a shared in-memory waiter map. Do not assign the same stable MQTT Client ID to two simultaneous processes unless they are deliberately replacing one another: they can disconnect each other and alter session ownership. Notifications and caches may accelerate reads, but PostgreSQL remains the correctness baseline.</p></div>
  <h2>✅ Done when</h2><p>HTTP status remains accurate when ingestion receives the response in a different process. Either process can restart independently without losing durable command tracking. Your run notes state client IDs, subscription ownership, readiness conditions, and recovery behavior.</p>
  <p>📚 Section checklist and process boundaries: <a href="../../COURSE.md#section-15--split-api-and-ingestion-processes">COURSE.md — Section 15</a>. Primary references: <a href="https://pkg.go.dev/os/signal#NotifyContext">Go signal lifecycle</a> and <a href="https://www.postgresql.org/docs/current/explicit-locking.html">PostgreSQL locking</a>.</p>
  <p>Ask me to review your ownership table against a proposed process layout before you extract the executables.</p>
  <footer><p>Previous: <a href="0014-outbox-failure-gaps.html">Lesson 14 — transactional outbox</a> · Next: <a href="0016-mqtt5-controls.html">Lesson 16 — MQTT 5 controls</a>.</p></footer>
</main></body></html>