Refactor MQTT course and bound backend message processing

Move lesson source into the teaching structure, align the course path with the learner's Go and DevOps experience, and add the interactive lesson guides. Add advanced DevOps lifecycle study as a separate section.

Bound MQTT message work and drain accepted jobs during shutdown; align the backend implementation with the revised course through Section 6.
This commit is contained in:
2026-10-22 10:19:56 +03:30
parent f9ab91ed60
commit fb30e8e884
70 changed files with 7492 additions and 312 deletions
@@ -0,0 +1,420 @@
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<title>Lesson 5A — Add the Backend Message-Processing Step</title>
<link rel="stylesheet" href="../assets/lesson.css">
<script src="../assets/quiz.js" defer></script>
</head>
<body>
<main class="page">
<p class="eyebrow">Go MQTT Vending Backend · Lesson 5A · 15 minutes</p>
<h1>Add the backend message-processing step</h1>
<p class="dek">Section 5 validates incoming bytes. In 5A, add the application code that turns a valid heartbeat into a small, queryable backend state.</p>
<div class="meta"><span class="pill">🎯 One skill: apply a validated message</span><span class="pill">🧠 Separate device time from receive time</span><span class="pill">🧪 In-memory lab</span></div>
<section class="card callout">
<p class="question">❓ Start with the repository as it is</p>
<p>The current <code>cmd/backend/main.go</code> receives <code>mqtt.Message</code> values from <code>internal/mqtt</code> and logs their topic and payload. The current codebase does <strong>not</strong> yet contain <code>internal/deviceview</code>, <code>Store</code>, <code>Service</code>, or the methods in this lesson.</p>
<p>Section 5A in <code>teaching/course-source/05a-backend-message-processing.md</code> is an implementation assignment: create those files and replace the raw-message log path with contract validation and application behavior. Every code block below is a complete file, including imports and wiring. The new code is the lesson's target, not code that already exists.</p>
</section>
<h2>🗺️ First, see what the files do</h2>
<div class="two-col">
<article class="card">
<h3>✅ Existing files used by this lesson</h3>
<ul>
<li><code>internal/contracts/topic.go</code> defines <code>TopicName</code>, <code>TopicFilter</code>, <code>Kind</code>, concrete-topic builders, exact and fleet-wide subscription builders, and <code>ParseTopic</code>. Concrete publish names and subscription filters use distinct types.</li>
<li><code>internal/contracts/message.go</code> defines the payload contracts and their topic/body validation methods.</li>
<li><code>internal/contracts/decode.go</code> decodes and validates a payload against that parsed topic.</li>
<li><code>internal/mqtt/client.go</code> defines <code>mqtt.Message</code> and copies received topic and payload into the inbox.</li>
<li><code>cmd/backend/main.go</code> owns startup and the receive loop; today it logs received payloads.</li>
</ul>
</article>
<article class="card">
<h3>🛠️ Files this lesson adds or changes</h3>
<ul>
<li>Create <code>internal/deviceview/store.go</code> for an in-memory latest-heartbeat view.</li>
<li>Create <code>internal/deviceview/service.go</code> to validate messages and apply their current use cases.</li>
<li>Edit <code>cmd/backend/main.go</code> to create the service and delegate inbox messages to it.</li>
</ul>
<p><code>Store</code>, <code>Service</code>, <code>RecordHeartbeat</code>, and the other device-view methods are new lesson code. They are not present in the checkout before you implement this section.</p>
</article>
</div>
<div class="flow" aria-label="message processing boundary">
<div class="step">Existing receive loop<br><strong>topic + bytes</strong></div><span class="arrow">→</span>
<div class="step">Existing contracts<br><strong>parse + validate</strong></div><span class="arrow">→</span>
<div class="step">New service<br><strong>choose application behavior</strong></div><span class="arrow">→</span>
<div class="step">New store<br><strong>latest heartbeat view</strong></div>
</div>
<p>Validation is a trust boundary, not a business action. The service applies the meaning of a valid message. The store owns the resulting view. Keeping these roles separate lets Section 6 move service work to workers without mixing application state with Paho callback code.</p>
<h2>🗃️ Then, create `internal/deviceview/store.go`</h2>
<p><strong>File overview:</strong> this file owns one latest heartbeat snapshot per device. It keeps the map in process memory, so a backend restart clears the view.</p>
<p><strong>`NewStore` summary:</strong> initializes the empty map. A query before the first heartbeat can report “not seen yet” rather than inventing an offline record.</p>
<p><strong>`RecordHeartbeat` summary:</strong> parses device-reported time, copies optional temperature, and replaces the snapshot only when the report is newer.</p>
<p><strong>`GetStatus` summary:</strong> returns the snapshot, plus whether the backend received it within the requested freshness window.</p>
<h3>🧩 Complete file</h3>
<pre><code>package deviceview
import (
"fmt"
"sync"
"time"
"example.com/mqtt5-learning/internal/contracts"
)
type Snapshot struct {
DeviceID string
MessageID string
ReportedAt time.Time
ReceivedAt time.Time
Temperature float64
TemperatureSet bool
}
type Status struct {
Snapshot Snapshot
HeartbeatFresh bool
}
type Store struct {
mu sync.RWMutex
latest map[string]Snapshot
}
func NewStore() *Store {
return &amp;Store{latest: make(map[string]Snapshot)}
}
func (s *Store) RecordHeartbeat(message contracts.Heartbeat, receivedAt time.Time) (bool, error) {
reportedAt, err := time.Parse(time.RFC3339, message.OccurredAt)
if err != nil {
// Decode already validates this timestamp. Keep the store defensive too.
return false, fmt.Errorf("parse heartbeat timestamp: %w", err)
}
snapshot := Snapshot{
DeviceID: message.DeviceID,
MessageID: message.MessageID,
ReportedAt: reportedAt,
ReceivedAt: receivedAt,
}
if message.Temperature != nil {
snapshot.Temperature = *message.Temperature
snapshot.TemperatureSet = true
}
s.mu.Lock()
defer s.mu.Unlock()
previous, exists := s.latest[message.DeviceID]
if exists &amp;&amp; !reportedAt.After(previous.ReportedAt) {
return false, nil
}
s.latest[message.DeviceID] = snapshot
return true, nil
}
func (s *Store) GetStatus(deviceID string, now time.Time, freshFor time.Duration) (Status, bool) {
s.mu.RLock()
snapshot, exists := s.latest[deviceID]
s.mu.RUnlock()
if !exists {
return Status{}, false
}
fresh := freshFor &gt; 0 &amp;&amp; !now.Before(snapshot.ReceivedAt) &amp;&amp; now.Sub(snapshot.ReceivedAt) &lt;= freshFor
return Status{Snapshot: snapshot, HeartbeatFresh: fresh}, true
}
</code></pre>
<div class="card">
<h3>🔎 Details: how the store methods work</h3>
<ul>
<li><strong><code>NewStore</code>:</strong> the map starts empty; the second return from <code>GetStatus</code> is false until a heartbeat is recorded.</li>
<li><strong><code>RecordHeartbeat</code>:</strong> contract decoding already checks <code>OccurredAt</code>, but the store parses it defensively too. <code>ReportedAt</code> orders observations; backend-supplied <code>ReceivedAt</code> measures freshness.</li>
<li><strong>Optional temperature:</strong> the store keeps temperature as a value and uses <code>TemperatureSet</code> to distinguish a missing reading from a reported zero. The map snapshot copy is therefore independent without pointer-copy logic.</li>
<li><strong>Locking:</strong> use an exclusive lock while comparing and replacing the map entry. Section 6 will allow multiple workers to call the service. A duplicate or older timestamp returns <code>false, nil</code> and leaves the newer view unchanged.</li>
<li><strong><code>GetStatus</code>:</strong> the read lock protects the map lookup. Freshness requires a positive window, a receive time not in the future, and an age no greater than that window.</li>
</ul>
<p>Ordering is not durable deduplication. Equal timestamps count as not newer, and process restart removes all snapshots.</p>
</div>
<h2>🧭 Next, create `internal/deviceview/service.go`</h2>
<p><strong>File overview:</strong> this file receives the existing transport-neutral <code>mqtt.Message</code>, validates it with <code>contracts</code>, applies the behavior assigned to its message kind, and logs a safe outcome. It does not own a Paho client or publish MQTT messages.</p>
<p><strong>`NewService` summary:</strong> checks that the store, logger, and positive freshness window are configured before message handling begins.</p>
<p><strong>`HandleMessage` summary:</strong> checks cancellation, parses the route, captures one backend receive time, then validates and handles the message according to its kind.</p>
<p><strong>`Status` summary:</strong> asks the store for the latest view using the service clock and configured freshness window.</p>
<h3>🧩 Complete file</h3>
<pre><code>package deviceview
import (
"context"
"fmt"
"log/slog"
"time"
"example.com/mqtt5-learning/internal/contracts"
"example.com/mqtt5-learning/internal/mqtt"
)
type Service struct {
store *Store
logger *slog.Logger
freshFor time.Duration
now func() time.Time
}
func NewService(store *Store, logger *slog.Logger, freshFor time.Duration) (*Service, error) {
if store == nil {
return nil, fmt.Errorf("device-view store is required")
}
if logger == nil {
return nil, fmt.Errorf("logger is required")
}
if freshFor &lt;= 0 {
return nil, fmt.Errorf("heartbeat freshness window must be positive")
}
return &amp;Service{store: store, logger: logger, freshFor: freshFor, now: time.Now}, nil
}
func (s *Service) HandleMessage(ctx context.Context, message mqtt.Message) error {
if err := ctx.Err(); err != nil {
return err
}
topic, err := contracts.ParseTopic(message.Topic)
if err != nil {
return fmt.Errorf("parse topic: %w", err)
}
receivedAt := s.now().UTC()
switch topic.Kind {
case contracts.KindTelemetry:
heartbeat, err := contracts.Decode[contracts.Heartbeat](message.Payload, topic, receivedAt)
if err != nil {
return fmt.Errorf("validate heartbeat: %w", err)
}
updated, err := s.store.RecordHeartbeat(heartbeat, receivedAt)
if err != nil {
return fmt.Errorf("record heartbeat: %w", err)
}
if !updated {
s.logger.Info("older or duplicate heartbeat ignored",
"device_id", heartbeat.DeviceID,
"message_id", heartbeat.MessageID,
)
return nil
}
s.logger.Info("heartbeat recorded",
"device_id", heartbeat.DeviceID,
"message_id", heartbeat.MessageID,
"received_at", receivedAt,
)
return nil
case contracts.KindSales:
report, err := contracts.Decode[contracts.SalesReport](message.Payload, topic, receivedAt)
if err != nil {
return fmt.Errorf("validate sales report: %w", err)
}
// Validation is not durable admission; Section 13 stores reports.
s.logger.Info("sales report validated; persistence not implemented",
"device_id", report.DeviceID,
"report_id", report.ReportID,
"item_count", len(report.Items),
)
return nil
case contracts.KindResponses:
result, err := contracts.Decode[contracts.CommandResult](message.Payload, topic, receivedAt)
if err != nil {
return fmt.Errorf("validate command result: %w", err)
}
// Section 7 adds pending-command and correlation matching.
s.logger.Info("command result validated; matching not implemented",
"device_id", result.DeviceID,
"command_id", result.CommandID,
"status", result.Status,
)
return nil
default:
return fmt.Errorf("device-view service does not handle topic kind %q", topic.Kind)
}
}
func (s *Service) Status(deviceID string) (Status, bool) {
return s.store.GetStatus(deviceID, s.now().UTC(), s.freshFor)
}
</code></pre>
<div class="card">
<h3>🔎 Details: how the service methods work</h3>
<ul>
<li><strong><code>NewService</code>:</strong> rejects missing dependencies and invalid policy at startup, not later as a nil-pointer failure. Its freshness window is application policy, separate from MQTT Keep Alive and Session Expiry.</li>
<li><strong><code>HandleMessage</code>:</strong> returns immediately for cancelled work or an invalid topic. It captures backend receive time once and reuses it in validation, storage, and logs.</li>
<li><strong>Telemetry branch:</strong> decodes a heartbeat, asks the store to advance its snapshot, and logs either “recorded” or “older or duplicate ignored.” Invalid input never reaches the store.</li>
<li><strong>Sales and response branches:</strong> decode and log identifiers only. Sales are not persisted here, and command responses are not correlated here. Those later outcomes must not be implied by a successful decode.</li>
<li><strong><code>Status</code>:</strong> returns backend state without querying the device. A fresh heartbeat means recent valid application traffic, not a live broker connection or healthy vending hardware.</li>
</ul>
</div>
<h2>🔌 Finally, edit `cmd/backend/main.go`</h2>
<p><strong>File overview:</strong> this existing file loads configuration, starts the MQTT client, owns the inbox receive loop, and periodically publishes the lab's ping command. The edited version keeps those responsibilities and delegates inbound messages to the new service instead of printing raw payloads.</p>
<p><strong><code>run</code> summary:</strong> constructs one store and service before MQTT startup, then passes each received message to <code>handleMessage</code>.</p>
<p><strong><code>handleMessage</code> summary:</strong> keeps the select loop focused on receiving events by delegating application behavior to <code>Service</code>.</p>
<h3>🧩 Complete edited file</h3>
<pre><code>package main
import (
"context"
"encoding/json"
"fmt"
"log"
"log/slog"
"os"
"os/signal"
"syscall"
"time"
"example.com/mqtt5-learning/internal/config"
"example.com/mqtt5-learning/internal/deviceview"
"example.com/mqtt5-learning/internal/mqtt"
)
func main() {
if err := run(); err != nil {
log.Fatal(err)
}
}
func run() error {
cfg, err := config.Load(os.Args[1:])
if err != nil {
return fmt.Errorf("failed to load config: %w", err)
}
ctx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
defer stop()
// Keep one shared in-memory view for the lifetime of this backend process.
store := deviceview.NewStore()
deviceView, err := deviceview.NewService(store, slog.Default(), 90*time.Second)
if err != nil {
return fmt.Errorf("create device-view service: %w", err)
}
telemetryFilter, err := contracts.DeviceFilter(contracts.KindTelemetry)
if err != nil { return err }
responsesFilter, err := contracts.DeviceFilter(contracts.KindResponses)
if err != nil { return err }
commandTopic, err := contracts.DeviceTopic(cfg.MQTT.DeviceID, contracts.KindCommands)
if err != nil { return err }
inbox := make(chan mqtt.Message, 16)
manager, err := mqtt.Start(ctx, cfg, []contracts.TopicFilter{telemetryFilter, responsesFilter}, inbox)
if err != nil {
return fmt.Errorf("failed to start MQTT client: %w", err)
}
defer mqtt.Stop(manager)
ticker := time.NewTicker(10 * time.Second)
defer ticker.Stop()
runID := time.Now().UnixNano()
sequence := 0
for {
select {
case &lt;-ctx.Done():
log.Print("shutting down")
return nil
case message := &lt;-inbox:
if err := handleMessage(ctx, deviceView, message); err != nil {
slog.Error("message handling failed", "topic", message.Topic, "error", err)
}
case &lt;-ticker.C:
sequence++
id := fmt.Sprintf("ping-%d-%d", runID, sequence)
payload, err := json.Marshal(contracts.Command{
SchemaVersion: contracts.SchemaVersion,
MessageID: id,
DeviceID: cfg.MQTT.DeviceID,
OccurredAt: time.Now().UTC().Format(time.RFC3339),
CommandID: id,
Command: "ping",
})
if err != nil {
return fmt.Errorf("failed to marshal payload: %w", err)
}
if err := mqtt.Publish(ctx, manager, commandTopic, payload); err != nil {
log.Printf("command publication failed or outcome uncertain: %v", err)
}
}
}
}
func handleMessage(ctx context.Context, deviceView *deviceview.Service, message mqtt.Message) error {
return deviceView.HandleMessage(ctx, message)
}
</code></pre>
<div class="card">
<h3>🔎 Details: why the integration is placed here</h3>
<ul>
<li>Construct the store once before the loop. Creating it per message would discard prior snapshots.</li>
<li>The 90-second freshness window is the course example policy; it should be chosen relative to the heartbeat interval and expected network delay.</li>
<li>The service handles inbound telemetry and response messages. The existing ticker and outbound ping remain intact.</li>
<li>Errors are logged with the topic and error, while the raw payload is no longer printed. Invalid or mismatched input therefore does not leak into a general log line.</li>
<li>The MQTT callback copies into the bounded inbox and returns before this receive loop applies the use case. Under automatic acknowledgement, a queue drop or later application error does not reverse an acknowledgement already sent. Section 6 studies that loss boundary and adds explicit overload handling.</li>
</ul>
</div>
<h2>🧪 Check your model</h2>
<form class="card quiz" data-quiz data-correct="b" data-explanation="Correct. Section 5A adds the deviceview package and wires it into the existing main loop." data-retry="Which directory exists now, and which package does the lesson ask you to add?">
<fieldset class="question-block"><legend><strong>1. What is the status of `internal/deviceview`?</strong></legend>
<label class="option"><input type="radio" name="q1" value="a"> Current code already stores persisted device status</label>
<label class="option"><input type="radio" name="q1" value="b"> Section 5A asks you to add it</label>
<label class="option"><input type="radio" name="q1" value="c"> Paho creates this package when client connects</label>
</fieldset><button type="submit">Check answer</button><p class="feedback" data-feedback aria-live="polite"></p>
</form>
<form class="card quiz" data-quiz data-correct="a" data-explanation="Right. Device time orders observations; backend receipt time measures how recently this service received a heartbeat." data-retry="Which timestamp comes from the device, and which one can only be recorded by this backend?">
<fieldset class="question-block"><legend><strong>2. Which timestamp measures heartbeat freshness?</strong></legend>
<label class="option"><input type="radio" name="q2" value="a"> Backend receipt time measures heartbeat freshness</label>
<label class="option"><input type="radio" name="q2" value="b"> Device report time measures backend freshness</label>
<label class="option"><input type="radio" name="q2" value="c"> MQTT packet ID measures backend freshness</label>
</fieldset><button type="submit">Check answer</button><p class="feedback" data-feedback aria-live="polite"></p>
</form>
<form class="card quiz" data-quiz data-correct="c" data-explanation="Correct. Freshness describes recent valid heartbeat traffic observed by the backend. It is not a health check for every subsystem." data-retry="What observation does the store actually have: recent heartbeat, live socket, or full physical machine health?">
<fieldset class="question-block"><legend><strong>3. What does `HeartbeatFresh` mean?</strong></legend>
<label class="option"><input type="radio" name="q3" value="a"> All vending hardware is healthy right now</label>
<label class="option"><input type="radio" name="q3" value="b"> Broker confirms that this device remains connected</label>
<label class="option"><input type="radio" name="q3" value="c"> A valid heartbeat arrived within the window</label>
</fieldset><button type="submit">Check answer</button><p class="feedback" data-feedback aria-live="polite"></p>
</form>
<h2>🛠️ Your tiny practice</h2>
<article class="card">
<p>Predict the final store contents for these arrivals:</p>
<ol><li>The device reports 12:00:05; the backend receives it at 12:00:06.</li><li>The device reports 12:00:02; the backend receives it at 12:00:09.</li></ol>
<p>Which snapshot remains? At 12:00:09, is that snapshot fresh under a 90-second window? Explain why the later arrival does not move device-observation order backward.</p>
</article>
<h2>🏗️ Build and run the vertical slice</h2>
<p>After saving the new <code>internal/deviceview</code> files and updating <code>cmd/backend/main.go</code>, compile the real programs and start the isolated broker. Run these from the repository root:</p>
<pre><code>go test ./...
go build -o /tmp/mqtt5-lab-backend ./cmd/backend
go build -o /tmp/mqtt5-lab-simulator ./cmd/simulator
docker compose -p mqtt5-learning up -d broker
docker compose -p mqtt5-learning ps</code></pre>
<p><code>go test ./...</code> runs the tests that exist; it does not prove every device-view branch is covered. The two builds catch compile and link errors in the actual processes. Check that the project-named broker is running before connecting clients.</p>
<p>Start each binary in its own terminal:</p>
<pre><code>/tmp/mqtt5-lab-simulator -username simulator -client-id lab-simulator-device-001</code></pre>
<pre><code>/tmp/mqtt5-lab-backend -username backend -client-id lab-backend</code></pre>
<p>Within a few seconds, expect backend log entries for recorded heartbeats with device and message IDs. The backend also sends a ping every ten seconds; the simulator replies, and the backend validates that response. This confirms a real MQTT message reached the application service. It does not establish command correlation, which Lesson 7 adds.</p>
<p>Stop both clients with Ctrl+C, then run <code>docker compose -p mqtt5-learning stop broker</code>. The device view is in memory and disappears when the backend exits. The lesson's temporary <code>deviceView.Status</code> diagnostic can show the snapshot and freshness flag; no HTTP status endpoint exists yet. Ask me a follow-up if a build error or log result is unclear.</p>
<section class="card callout"><h2>✅ Takeaway</h2><p><strong>The existing code receives MQTT messages; the new service validates and applies them; the new store holds the latest heartbeat view.</strong> The implementation is complete above, but its state is RAM-only and a fresh heartbeat does not prove device health.</p><p>Ask me a follow-up if you want to trace one message through the code line by line.</p></section>
<footer>
<p>Previous: <a href="0005-contract-validation.html">Lesson 5 — validate the message</a> · Next: <a href="0006-bounded-message-work.html">Lesson 6 — bounded message work</a> · <a href="../reference/message-processing-lifecycle.html">message lifecycle reference</a></p>
<p>Source implementation: <a href="../course-source/05a-backend-message-processing.md">Section 5A in the course</a>. Primary references: <a href="https://pkg.go.dev/log/slog">Go structured logging</a>; <a href="https://pkg.go.dev/sync">Go synchronization</a>; <a href="https://docs.oasis-open.org/mqtt/mqtt/v5.0/mqtt-v5.0.html">OASIS MQTT 5.0 Standard</a>.</p>
</footer>
</main>
</body>
</html>