Robot-Market/mqtt-5-learning
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Refactor MQTT course and bound backend message processing

Browse Source 
Move lesson source into the teaching structure, align the course path with the learner's Go and DevOps experience, and add the interactive lesson guides. Add advanced DevOps lifecycle study as a separate section.

Bound MQTT message work and drain accepted jobs during shutdown; align the backend implementation with the revised course through Section 6.
This commit is contained in:
Ryan
2026-10-22 10:19:56 +03:30
parent f9ab91ed60
commit fb30e8e884
70 changed files with 7492 additions and 312 deletions
Download Patch File Download Diff File Expand all files Collapse all files
teaching/lessons/0017-operations-diagnostics.html
+83
View File
@@ -0,0 +1,83 @@
<!doctype html>
<html lang="en"><head>
<meta charset="utf-8"><meta name="viewport" content="width=device-width, initial-scale=1">
<title>Lesson 17 — Operate and Diagnose the Service</title>
<link rel="stylesheet" href="../assets/lesson.css"><script src="../assets/quiz.js" defer></script>
</head><body><main class="page">
<p class="eyebrow">Go MQTT Vending Backend · Lesson 17 · 15 minutes</p>
<h1>Diagnose the failing boundary with evidence</h1>
<p class="dek">Make each command traceable across HTTP, Go processes, MQTT, broker, device simulator, and PostgreSQL.</p>
<div class="meta"><span class="pill">🎯 One skill: turn symptoms into a diagnostic path</span><span class="pill">🧠 Predict likely fault layer</span><span class="pill">🧪 Repeatable failure matrix</span></div>
<section class="card callout"><p class="question">❓ Start with the system story</p><p>An API returns 202, the backend records PUBACK, but no device result appears. What evidence do you inspect next, and what can you not conclude yet?</p><p>Trace the same command ID through each owner before labeling the device failed.</p></section>
<h2>🔎 Put identifiers in logs, not metric labels</h2>
<p>Use structured logs with stable command/report IDs, process identity, message kind, and safe error categories. Redact credentials and avoid arbitrary payload dumps. Put high-cardinality IDs in logs/traces where appropriate, not unrestricted metric labels that create a new time series per device or command.</p>
<div class="two-col"><article class="card"><h3>Metrics worth watching</h3><p>Reconnects, rejected messages, queue depth/capacity, duplicate outcomes, publish and response latency, timeouts, DB failures, outbox backlog/oldest age, and shutdown drain results.</p></article><article class="card"><h3>Health endpoints</h3><p>Liveness asks whether a process should be restarted. Readiness asks whether that process can serve its assigned role and dependencies. A connected MQTT client alone does not establish useful service readiness.</p></article></div>
<h2>🧭 File overview: <code>internal/observability/health.go</code></h2>
<p>Create this complete standard-library file for the first operational instrumentation slice. It exposes liveness, dependency-aware readiness, and low-cardinality Prometheus text metrics. It does not label metrics with device or command IDs.</p>
<h3>🩺 <code>NewHandler</code> and readiness setters — short summary</h3><p>Create one per process role and update its dependency flags when DB/MQTT readiness changes.</p>
<h3>📈 <code>ServeHTTP</code> — short summary</h3><p>Serves bounded health endpoints and fixed metric names without exposing secrets or arbitrary IDs.</p>
<details class="card" open><summary><strong>📄 Complete file contents</strong></summary><pre><code>package observability
import (
"fmt"
"net/http"
"sync/atomic"
)
type Handler struct {
role string
requiresMQTT bool
databaseReady atomic.Bool
mqttReady atomic.Bool
rejected atomic.Uint64
duplicates atomic.Uint64
reconnects atomic.Uint64
completed atomic.Uint64
}
func NewHandler(role string, requiresMQTT bool) (*Handler, error) {
if role != "api" &amp;&amp; role != "ingest" { return nil, fmt.Errorf("role must be api or ingest") }
return &amp;Handler{role: role, requiresMQTT: requiresMQTT}, nil
}
func (h *Handler) SetDatabaseReady(ready bool) { h.databaseReady.Store(ready) }
func (h *Handler) SetMQTTReady(ready bool) { h.mqttReady.Store(ready) }
func (h *Handler) RecordRejected() { h.rejected.Add(1) }
func (h *Handler) RecordDuplicate() { h.duplicates.Add(1) }
func (h *Handler) RecordReconnect() { h.reconnects.Add(1) }
func (h *Handler) RecordCompleted() { h.completed.Add(1) }
func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
switch r.URL.Path {
case "/livez":
w.WriteHeader(http.StatusNoContent)
case "/readyz":
if !h.databaseReady.Load() || (h.requiresMQTT &amp;&amp; !h.mqttReady.Load()) {
http.Error(w, "required dependency is not ready", http.StatusServiceUnavailable)
return
}
w.WriteHeader(http.StatusNoContent)
case "/metrics":
w.Header().Set("Content-Type", "text/plain; version=0.0.4; charset=utf-8")
fmt.Fprintf(w, "mqtt_learning_messages_rejected_total{role=%q} %d\n", h.role, h.rejected.Load())
fmt.Fprintf(w, "mqtt_learning_messages_duplicate_total{role=%q} %d\n", h.role, h.duplicates.Load())
fmt.Fprintf(w, "mqtt_learning_reconnects_total{role=%q} %d\n", h.role, h.reconnects.Load())
fmt.Fprintf(w, "mqtt_learning_work_completed_total{role=%q} %d\n", h.role, h.completed.Load())
default:
http.NotFound(w, r)
}
}
</code></pre></details>
<div class="card"><h3>🔎 Details: operational meaning</h3><p>Liveness stays independent of database/MQTT so a dependency outage does not trigger a restart loop. Readiness requires PostgreSQL for both processes and MQTT subscriptions for ingestion. The metrics are counters, not full histogram/queue instrumentation; add queue gauges and latency histograms when the service has those measurements. Use process role as a bounded label and keep IDs in structured logs instead.</p></div>
<form class="card quiz" data-quiz data-correct="c" data-explanation="The API and PUBACK are evidence for earlier boundaries. Inspect broker subscription/delivery, ingestion logs and counters, DB result state, and simulator response before deciding where the path stopped." data-retry="Follow the command ID and list what each previous observation actually proves."><p><strong>🧠 Check the diagnosis:</strong> After 202 and PUBACK with no device result, what conclusion is justified?</p><label><input type="radio" name="q" value="a"> The device definitely completed the command</label><br><label><input type="radio" name="q" value="b"> The device definitely received nothing</label><br><label><input type="radio" name="q" value="c"> The outcome is not established; trace delivery, device processing, ingestion, and persisted result evidence</label><br><button type="submit">Check answer</button><p data-feedback aria-live="polite"></p></form>
<h2>🧪 Make a repeatable load and failure scenario</h2>
<p>Start with a synthetic simulator load that has a fixed duration, device count, message rate, and payload mix. Measure throughput, latency percentiles, memory, queue growth, and errors. Profile after evidence identifies a bottleneck; a large queue is not a throughput fix if admission exceeds processing indefinitely.</p>
<table><thead><tr><th>Injected fault</th><th>Evidence to collect</th><th>Recovery check</th></tr></thead><tbody><tr><td>Broker down/restart</td><td>Reconnect state, Session Present, subscription results</td><td>Expected subscriptions return; duplicate handling stays safe</td></tr><tr><td>Simulator offline or restart</td><td>Command state, expiry/deadline, session behavior</td><td>Late work is safe; status remains honest</td></tr><tr><td>Slow/unavailable DB</td><td>Query latency/errors, manual Ack policy, backlog</td><td>No false durable acceptance; recovery resumes</td></tr><tr><td>Duplicate report/result</td><td>Stable ID and duplicate counters</td><td>One business effect; stored outcome remains stable</td></tr><tr><td>Denied topic</td><td>Broker logs/reason and application visibility</td><td>No unauthorized effect or data leakage</td></tr><tr><td>Shutdown under load</td><td>Queue depth, accepted/rejected, drain deadline</td><td>Recoverable work is preserved or loss is explicitly recorded</td></tr></tbody></table>
<h2>📓 Write a runbook another person can follow</h2>
<ol><li>State the symptom and its user-visible impact.</li><li>Collect command/report ID, timestamps, process logs, metrics, broker evidence, and relevant database state without secrets.</li><li>Map the last confirmed evidence to HTTP, application, broker, device, or database boundary.</li><li>Give the bounded recovery action and note the risk of retrying a possibly completed physical effect.</li><li>Verify recovery with a harmless synthetic operation and record the evidence that changed.</li></ol>
<div class="card"><h3>✅ Done when</h3><p>Someone who has not read your code can use the runbook to diagnose one injected command failure. Your load scenario and fault matrix can be repeated, and each metric helps answer an operational question.</p></div>
<details class="card"><summary><strong>📋 Short self-check answers</strong></summary><ol><li>202 is API acceptance; PUBACK is one MQTT hop. Neither proves device completion.</li><li>Readiness reflects dependencies needed for the process's assigned work; liveness only says whether it is functioning enough to keep running.</li><li>Metric labels should remain bounded; use logs/traces for high-cardinality IDs.</li></ol></details>
<p>📚 Section checklist: <a href="../../COURSE.md#section-17--operate-and-diagnose-the-service">COURSE.md — Section 17</a>. Primary references: <a href="https://pkg.go.dev/net/http">Go <code>net/http</code></a> and <a href="https://prometheus.io/docs/instrumenting/exposition_formats/">Prometheus exposition format</a>.</p>
<p>Ask me to help map a runbook symptom to the next useful measurement.</p>
<footer><p>Previous: <a href="0016-mqtt5-controls.html">Lesson 16 — MQTT 5 controls</a> · Next: <a href="0018-backend-handover.html">Lesson 18 — backend handover</a>.</p></footer>
</main></body></html>