Refactor MQTT course and bound backend message processing
Move lesson source into the teaching structure, align the course path with the learner's Go and DevOps experience, and add the interactive lesson guides. Add advanced DevOps lifecycle study as a separate section. Bound MQTT message work and drain accepted jobs during shutdown; align the backend implementation with the revised course through Section 6.
This commit is contained in:
@@ -0,0 +1,540 @@
|
||||
<!doctype html>
|
||||
<html lang="en">
|
||||
<head>
|
||||
<meta charset="utf-8"><meta name="viewport" content="width=device-width, initial-scale=1">
|
||||
<title>Lesson 6 — Bound Message Work</title>
|
||||
<link rel="stylesheet" href="../assets/lesson.css"><script src="../assets/quiz.js" defer></script>
|
||||
</head>
|
||||
<body><main class="page">
|
||||
<p class="eyebrow">Go MQTT Vending Backend · Lesson 6 · 8 minutes</p>
|
||||
<h1>Bound message work; make overload visible</h1>
|
||||
<p class="dek">Move application work out of the MQTT receive callback without creating an unlimited backlog or hiding queue rejection.</p>
|
||||
<div class="meta"><span class="pill">🎯 One skill: reason about admission</span><span class="pill">🧠 Predict overload behavior</span><span class="pill">🧪 In-memory queue</span></div>
|
||||
|
||||
<section class="card callout"><p class="question">❓ Start with the story</p><p>A burst of vending heartbeats arrives while workers are slow. The callback returns, but the telemetry queue is full. What happened to the packet, and what can the backend truthfully claim?</p><p>Pause before reading: separate MQTT delivery acknowledgement from the backend's queue decision.</p></section>
|
||||
<section class="card"><h2>📍 Files to implement in the codebase</h2><p><code>internal/dispatch/pool.go</code> is already present. Read it first: this lesson teaches its admission, worker, and shutdown behavior, then aligns the routing, pool coordination, and both process entrypoints with the listings below.</p><table><thead><tr><th>Path</th><th>Action</th><th>What you will learn or change</th></tr></thead><tbody><tr><td><code>internal/dispatch/pool.go</code></td><td>Read and understand existing file</td><td>Fixed workers, bounded queue, immediate admission outcomes, counters, bounded drain</td></tr><tr><td><code>internal/dispatch/receiver.go</code></td><td>Review and align</td><td>Route by parsed topic kind and copy payload ownership before enqueue</td></tr><tr><td><code>internal/dispatch/shutdown.go</code></td><td>Review and align</td><td>Stop every pool before waiting under one shared deadline</td></tr><tr><td><code>cmd/backend/main.go</code></td><td>Replace with complete target file below</td><td>Wire Section 5A service to separate bounded pools and preserve MQTT during drain</td></tr><tr><td><code>cmd/simulator/main.go</code></td><td>Replace with complete target file below</td><td>Handle device commands through bounded work and drain before disconnect</td></tr></tbody></table><p>The routing and shutdown files are already present in this checkout. Compare them with the complete listings below and preserve the same admission and error semantics.</p></section>
|
||||
<h2>📄 File overview: `pool.go` controls finite work</h2>
|
||||
<p><code>internal/dispatch/pool.go</code> owns a fixed worker set and a bounded jobs channel. It limits simultaneous work and waiting work. It does not make queued messages durable, and it cannot ask MQTT to reverse an acknowledgement already sent.</p>
|
||||
<div class="flow" aria-label="dispatch lifecycle"><div class="step">MQTT callback<br><strong>copy + submit</strong></div><span class="arrow">→</span><div class="step">Finite queue<br><strong>admit or reject</strong></div><span class="arrow">→</span><div class="step">Fixed workers<br><strong>run service</strong></div></div>
|
||||
|
||||
<article class="card">
|
||||
<h3>📥 `Submit` — short summary</h3>
|
||||
<p>Returns immediately with an explicit outcome: accepted, full, or stopping.</p>
|
||||
<pre><code>func (p *Pool) Submit(ctx context.Context, job Job) (Outcome, error) {
|
||||
if job == nil { return Stopping, errors.New("nil job") }
|
||||
if err := ctx.Err(); err != nil { return Stopping, err }
|
||||
|
||||
p.mu.Lock()
|
||||
defer p.mu.Unlock()
|
||||
if !p.accepting { return Stopping, nil }
|
||||
select {
|
||||
case p.jobs <- job:
|
||||
p.accepted.Add(1)
|
||||
return Accepted, nil
|
||||
default:
|
||||
p.rejected.Add(1)
|
||||
return Full, nil
|
||||
}
|
||||
}</code></pre>
|
||||
<p class="small">This is the focused <code>Submit</code> method excerpt from <code>Pool</code>. The complete file defines the pool fields, <code>Outcome</code> values, worker loop, and shutdown behavior.</p>
|
||||
<h3>🧩 Details: why each branch exists</h3>
|
||||
<ul><li>Reject nil or already-cancelled work before touching the queue.</li><li>The mutex makes the accepting check and channel send atomic with shutdown closing the channel.</li><li>The <code>default</code> branch means “no slot now”: this method never waits for capacity.</li><li><code>Accepted</code> means queued in memory, not completed or saved to disk.</li><li><code>Full</code> is an application overload decision. With automatic acknowledgement, returning it does not undo PUBACK or request redelivery.</li></ul>
|
||||
</article>
|
||||
|
||||
<h2>⚖️ Choose a policy per traffic class</h2>
|
||||
<div class="two-col"><article class="card"><h3>📈 Telemetry</h3><p>If the business allows it, drop a stale heartbeat and count the rejection. Freshness will age out. Make the loss visible.</p></article><article class="card"><h3>🧾 Sales and commands</h3><p>Silently dropping a report or command is usually unacceptable. Return an explicit failure and add durable admission before claiming crash-safe processing.</p></article></div>
|
||||
<p>Separate pools can prevent telemetry from consuming every waiting slot reserved for important work. That is capacity isolation, not durability. Shutdown order matters too: stop producers, stop admission, drain workers to a deadline while needed transport remains alive, then disconnect.</p>
|
||||
<section class="card"><h2>🛑 What does <code>ctx.Err()</code> contain during shutdown?</h2><p>The <code>ctx</code> passed to <code>Pool.Shutdown</code> is the drain context. If its deadline expires before workers finish, <code>ctx.Err()</code> returns <code>context.DeadlineExceeded</code>. If something cancels it earlier, it returns <code>context.Canceled</code>. If workers finish first, <code>Pool.Shutdown</code> returns <code>nil</code> instead. On timeout, the pool uses that context error as its shutdown result and cancels the worker job context. Cancellation is cooperative, so a handler must check its context or use context-aware I/O to stop promptly.</p></section>
|
||||
|
||||
<h2>🧭 File and function map</h2>
|
||||
<p><strong>File overview:</strong> Section 6 adds three pieces: <code>pool.go</code> owns bounded execution, <code>receiver.go</code> routes each message to the right pool, and <code>shutdown.go</code> closes all pools before waiting for their workers. The backend entrypoint below wires those pieces to the Section 5A service and existing MQTT inbox.</p>
|
||||
<div class="two-col">
|
||||
<article class="card"><h3>📄 <code>internal/dispatch/pool.go</code></h3>
|
||||
<p><strong><code>New</code> summary:</strong> starts a fixed worker count and a queue with a fixed capacity.</p>
|
||||
<p><strong><code>Submit</code> summary:</strong> reports <code>Accepted</code>, <code>Full</code>, or <code>Stopping</code> without waiting for queue room.</p>
|
||||
<p><strong><code>StopAdmission</code> / <code>Shutdown</code> summary:</strong> close the submission path, wait for workers up to a deadline, then cancel remaining work.</p>
|
||||
<p><strong><code>Stats</code> / <code>worker</code> summary:</strong> expose admission and completion counts; workers run jobs and count success or failure.</p>
|
||||
<h4>🔎 Details</h4><p>The mutex protects both the accepting flag and channel close, preventing a send-on-closed-channel race. The nonblocking <code>select</code> makes overload immediate. Closing the jobs channel lets workers finish buffered jobs; the shared deadline keeps draining bounded. If it expires, cancellation is cooperative: a job that ignores its context can keep running. Counters describe process-local events, not durable business records.</p>
|
||||
</article>
|
||||
<article class="card"><h3>📄 <code>internal/dispatch/receiver.go</code></h3>
|
||||
<p><strong><code>Receiver.Submit</code> summary:</strong> parses the topic, selects the telemetry or important pool, copies payload bytes, and submits a closure that calls the application processor.</p>
|
||||
<h4>🔎 Details</h4><p>Routing on a parsed <code>Kind</code> keeps traffic policy out of the MQTT callback. Commands, sales, and command responses use the important pool; only telemetry uses the telemetry pool. Copying the payload transfers ownership: the worker no longer depends on Paho or a reused inbox buffer. The job closure captures that owned copy and runs later with the worker's context.</p>
|
||||
</article>
|
||||
<article class="card"><h3>📄 <code>internal/dispatch/shutdown.go</code></h3>
|
||||
<p><strong><code>Shutdown</code> summary:</strong> stops submissions to every pool first, then drains them using one shared deadline.</p>
|
||||
<h4>🔎 Details</h4><p>Stopping all pools before waiting prevents one pool from accepting new work while the other drains. Reusing one context bounds the total shutdown time instead of allowing a fresh timeout for each pool. <code>errors.Join</code> preserves timeout failures from multiple pools. Keep MQTT alive during this interval if jobs need to publish their result; disconnect only after the drain attempt.</p>
|
||||
</article>
|
||||
</div>
|
||||
<h3>🔌 Integration overview: edit <code>cmd/backend/main.go</code></h3>
|
||||
<p><strong><code>run</code> summary:</strong> builds the service and pools once, starts MQTT, sends each inbox message through <code>Receiver</code>, and drains workers when the signal context ends.</p>
|
||||
<p><strong>Details:</strong> the signal context stops the main loop, while a separate transport context keeps MQTT available during the bounded drain. The signal case itself only logs and returns; as <code>run</code> returns, deferred cleanup executes in reverse registration order. The dispatcher drain defer was registered after <code>mqtt.Stop</code>, so it runs first. Pool jobs use an independent parent context so the signal does not cancel them before draining. Once drain completes or times out, <code>mqtt.Stop</code> disconnects the client. The inbox and callback remain bounded; if more messages arrive during shutdown they can still be dropped under automatic acknowledgement, which this lesson reports rather than hiding.</p>
|
||||
<details class="card" open><summary><strong>🧩 Complete Section 6 backend wiring: <code>cmd/backend/main.go</code></strong></summary><p>This is the full target entrypoint layered on the Section 5A service. The three dispatcher files follow it below.</p><pre><code>package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"log"
|
||||
"log/slog"
|
||||
"os"
|
||||
"os/signal"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
"example.com/mqtt5-learning/internal/config"
|
||||
"example.com/mqtt5-learning/internal/contracts"
|
||||
"example.com/mqtt5-learning/internal/deviceview"
|
||||
"example.com/mqtt5-learning/internal/dispatch"
|
||||
"example.com/mqtt5-learning/internal/mqtt"
|
||||
)
|
||||
|
||||
func main() {
|
||||
if err := run(); err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
func run() error {
|
||||
cfg, err := config.Load(os.Args[1:])
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to load config: %w", err)
|
||||
}
|
||||
|
||||
lifecycleCtx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
|
||||
defer stop()
|
||||
transportCtx, stopTransport := context.WithCancel(context.Background())
|
||||
defer stopTransport()
|
||||
|
||||
store := deviceview.NewStore()
|
||||
service, err := deviceview.NewService(store, slog.Default(), 90*time.Second)
|
||||
if err != nil {
|
||||
return fmt.Errorf("create device-view service: %w", err)
|
||||
}
|
||||
|
||||
telemetryFilter, err := contracts.DeviceFilter(contracts.KindTelemetry)
|
||||
if err != nil { return err }
|
||||
responsesFilter, err := contracts.DeviceFilter(contracts.KindResponses)
|
||||
if err != nil { return err }
|
||||
commandTopic, err := contracts.DeviceTopic(cfg.MQTT.DeviceID, contracts.KindCommands)
|
||||
if err != nil { return err }
|
||||
inbox := make(chan mqtt.Message, 16)
|
||||
manager, err := mqtt.Start(transportCtx, cfg, []contracts.TopicFilter{telemetryFilter, responsesFilter}, inbox)
|
||||
if err != nil {
|
||||
return fmt.Errorf("failed to start MQTT client: %w", err)
|
||||
}
|
||||
defer mqtt.Stop(manager)
|
||||
|
||||
telemetryPool := dispatch.New(context.Background(), 2, 16)
|
||||
importantPool := dispatch.New(context.Background(), 2, 32)
|
||||
receiver := &dispatch.Receiver{
|
||||
Telemetry: telemetryPool,
|
||||
Important: importantPool,
|
||||
Process: service.HandleMessage,
|
||||
}
|
||||
defer func() {
|
||||
drainCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
defer cancel()
|
||||
started := time.Now()
|
||||
if err := dispatch.Shutdown(drainCtx, telemetryPool, importantPool); err != nil {
|
||||
log.Printf("dispatcher shutdown incomplete: %v", err)
|
||||
}
|
||||
log.Printf("dispatcher drain finished elapsed=%s telemetry=%+v important=%+v",
|
||||
time.Since(started), telemetryPool.Stats(), importantPool.Stats())
|
||||
}()
|
||||
|
||||
ticker := time.NewTicker(10 * time.Second)
|
||||
defer ticker.Stop()
|
||||
runID := time.Now().UnixNano()
|
||||
sequence := 0
|
||||
|
||||
for {
|
||||
select {
|
||||
case <-lifecycleCtx.Done():
|
||||
log.Print("shutting down; draining accepted work before MQTT disconnect")
|
||||
return nil
|
||||
case message := <-inbox:
|
||||
outcome, submitErr := receiver.Submit(lifecycleCtx, message)
|
||||
if submitErr != nil || outcome != dispatch.Accepted {
|
||||
slog.Warn("message admission rejected", "topic", message.Topic, "outcome", outcome, "error", submitErr)
|
||||
}
|
||||
case <-ticker.C:
|
||||
sequence++
|
||||
id := fmt.Sprintf("ping-%d-%d", runID, sequence)
|
||||
payload, marshalErr := json.Marshal(contracts.Command{
|
||||
SchemaVersion: contracts.SchemaVersion,
|
||||
MessageID: id,
|
||||
DeviceID: cfg.MQTT.DeviceID,
|
||||
OccurredAt: time.Now().UTC().Format(time.RFC3339),
|
||||
CommandID: id,
|
||||
Command: "ping",
|
||||
})
|
||||
if marshalErr != nil {
|
||||
return fmt.Errorf("failed to marshal payload: %w", marshalErr)
|
||||
}
|
||||
if publishErr := mqtt.Publish(lifecycleCtx, manager, commandTopic, payload); publishErr != nil {
|
||||
log.Printf("command publication failed or outcome uncertain: %v", publishErr)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
</code></pre></details>
|
||||
<h3>🤖 Simulator integration overview: edit <code>cmd/simulator/main.go</code></h3>
|
||||
<p><strong><code>run</code> summary:</strong> starts the device client, publishes heartbeats, and submits received commands to bounded workers instead of handling them inline in the main loop.</p>
|
||||
<p><strong><code>respond</code> summary:</strong> rejects retained or misrouted commands, validates the contract, and publishes the pong using the worker context.</p>
|
||||
<p><strong>Details:</strong> the simulator attempts to drain accepted command jobs before closing MQTT so a worker can still publish its response. The drain has a five-second limit; if it expires, remaining in-memory work may not complete. A full important queue is logged as an application rejection; automatic acknowledgement may already have completed the subscriber delivery, so this RAM-only example cannot promise redelivery.</p>
|
||||
<details class="card" open><summary><strong>🧩 Complete Section 6 simulator wiring: <code>cmd/simulator/main.go</code></strong></summary><p>This complete target file carries the Section 5 contract-aware simulator forward and moves inbound command work to the dispatcher.</p><pre><code>package main
|
||||
|
||||
import (
|
||||
"context"
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"log"
|
||||
"os"
|
||||
"os/signal"
|
||||
"sync/atomic"
|
||||
"syscall"
|
||||
"time"
|
||||
|
||||
"example.com/mqtt5-learning/internal/config"
|
||||
"example.com/mqtt5-learning/internal/contracts"
|
||||
"example.com/mqtt5-learning/internal/dispatch"
|
||||
"example.com/mqtt5-learning/internal/mqtt"
|
||||
"github.com/eclipse/paho.golang/autopaho"
|
||||
)
|
||||
|
||||
func main() {
|
||||
if err := run(); err != nil {
|
||||
log.Fatal(err)
|
||||
}
|
||||
}
|
||||
|
||||
func run() error {
|
||||
cfg, err := config.Load(os.Args[1:])
|
||||
if err != nil {
|
||||
return fmt.Errorf("load config: %w", err)
|
||||
}
|
||||
lifecycleCtx, stop := signal.NotifyContext(context.Background(), os.Interrupt, syscall.SIGTERM)
|
||||
defer stop()
|
||||
transportCtx, stopTransport := context.WithCancel(context.Background())
|
||||
defer stopTransport()
|
||||
|
||||
commandFilter, err := contracts.DeviceSubscription(cfg.MQTT.DeviceID, contracts.KindCommands)
|
||||
if err != nil { return err }
|
||||
telemetryTopic, err := contracts.DeviceTopic(cfg.MQTT.DeviceID, contracts.KindTelemetry)
|
||||
if err != nil { return err }
|
||||
inbox := make(chan mqtt.Message, 16)
|
||||
manager, err := mqtt.Start(transportCtx, cfg, []contracts.TopicFilter{commandFilter}, inbox)
|
||||
if err != nil {
|
||||
return fmt.Errorf("start MQTT: %w", err)
|
||||
}
|
||||
defer mqtt.Stop(manager)
|
||||
|
||||
runID := time.Now().UnixNano()
|
||||
telemetryPool := dispatch.New(context.Background(), 2, 16)
|
||||
importantPool := dispatch.New(context.Background(), 2, 16)
|
||||
var resultSequence atomic.Uint64
|
||||
receiver := &dispatch.Receiver{
|
||||
Telemetry: telemetryPool,
|
||||
Important: importantPool,
|
||||
Process: func(jobCtx context.Context, message mqtt.Message) error {
|
||||
resultID := fmt.Sprintf("result-%d-%d", runID, resultSequence.Add(1))
|
||||
return respond(jobCtx, manager, cfg.MQTT.DeviceID, resultID, message)
|
||||
},
|
||||
}
|
||||
defer func() {
|
||||
drainCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
|
||||
defer cancel()
|
||||
started := time.Now()
|
||||
if err := dispatch.Shutdown(drainCtx, telemetryPool, importantPool); err != nil {
|
||||
log.Printf("dispatcher shutdown incomplete: %v", err)
|
||||
}
|
||||
log.Printf("dispatcher drain finished elapsed=%s telemetry=%+v important=%+v",
|
||||
time.Since(started), telemetryPool.Stats(), importantPool.Stats())
|
||||
}()
|
||||
|
||||
ticker := time.NewTicker(3 * time.Second)
|
||||
defer ticker.Stop()
|
||||
sequence := uint64(0)
|
||||
for {
|
||||
select {
|
||||
case <-lifecycleCtx.Done():
|
||||
log.Print("shutting down; draining accepted command work before MQTT disconnect")
|
||||
return nil
|
||||
case <-ticker.C:
|
||||
sequence++
|
||||
heartbeat := contracts.Heartbeat{
|
||||
SchemaVersion: contracts.SchemaVersion,
|
||||
MessageID: fmt.Sprintf("heartbeat-%d-%d", time.Now().Unix(), sequence),
|
||||
DeviceID: cfg.MQTT.DeviceID,
|
||||
OccurredAt: time.Now().UTC().Format(time.RFC3339),
|
||||
}
|
||||
payload, marshalErr := json.Marshal(heartbeat)
|
||||
if marshalErr != nil {
|
||||
return fmt.Errorf("encode heartbeat: %w", marshalErr)
|
||||
}
|
||||
if publishErr := mqtt.Publish(lifecycleCtx, manager, telemetryTopic, payload); publishErr != nil {
|
||||
log.Printf("heartbeat publish: %v", publishErr)
|
||||
}
|
||||
case message := <-inbox:
|
||||
outcome, submitErr := receiver.Submit(lifecycleCtx, message)
|
||||
if submitErr != nil || outcome != dispatch.Accepted {
|
||||
log.Printf("command admission rejected outcome=%d topic=%q error=%v", outcome, message.Topic, submitErr)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func respond(ctx context.Context, manager *autopaho.ConnectionManager, deviceID, resultMessageID string, message mqtt.Message) error {
|
||||
if message.Retain {
|
||||
return fmt.Errorf("retained command rejected")
|
||||
}
|
||||
topic, err := contracts.ParseTopic(message.Topic)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
if topic.Kind != contracts.KindCommands || topic.DeviceID != deviceID {
|
||||
return fmt.Errorf("command topic is not authorized for this device")
|
||||
}
|
||||
command, err := contracts.Decode[contracts.Command](message.Payload, topic, time.Now().UTC())
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
result := contracts.CommandResult{
|
||||
SchemaVersion: contracts.SchemaVersion,
|
||||
MessageID: resultMessageID,
|
||||
DeviceID: deviceID,
|
||||
OccurredAt: time.Now().UTC().Format(time.RFC3339),
|
||||
CommandID: command.CommandID,
|
||||
Status: "completed",
|
||||
}
|
||||
payload, err := json.Marshal(result)
|
||||
if err != nil {
|
||||
return fmt.Errorf("encode result: %w", err)
|
||||
}
|
||||
responseTopic, err := contracts.DeviceTopic(deviceID, contracts.KindResponses)
|
||||
if err != nil { return fmt.Errorf("build response topic: %w", err) }
|
||||
return mqtt.Publish(ctx, manager, responseTopic, payload)
|
||||
}
|
||||
</code></pre></details>
|
||||
|
||||
<h2>🧪 Check your model</h2>
|
||||
<form class="card quiz" data-quiz data-correct="b" data-explanation="Correct. Full describes local queue admission; automatic MQTT acknowledgement is a separate protocol decision and is not rolled back." data-retry="Ask what Submit controls and what Paho controls. Are those the same state machine?"><fieldset class="question-block"><legend><strong>1. The bounded queue returns `Full`. What does that change?</strong></legend><label class="option"><input type="radio" name="q1" value="a"> The broker must retransmit this message automatically now</label><label class="option"><input type="radio" name="q1" value="b"> The app rejected admission; PUBACK may already stand</label><label class="option"><input type="radio" name="q1" value="c"> The device command has completed with a failure</label></fieldset><button type="submit">Check answer</button><p class="feedback" data-feedback aria-live="polite"></p></form>
|
||||
<form class="card quiz" data-quiz data-correct="c" data-explanation="Right. Accepted means the queue received an in-memory job. A worker has not necessarily completed it, and a process crash can lose it." data-retry="Which counter records enqueue, and which evidence records worker completion?"><fieldset class="question-block"><legend><strong>2. What does `Accepted` mean for a queue job?</strong></legend><label class="option"><input type="radio" name="q2" value="a"> The worker completed the vending business operation</label><label class="option"><input type="radio" name="q2" value="b"> The broker durably stored the application result</label><label class="option"><input type="radio" name="q2" value="c"> The job entered the process's in-memory queue</label></fieldset><button type="submit">Check answer</button><p class="feedback" data-feedback aria-live="polite"></p></form>
|
||||
<form class="card quiz" data-quiz data-correct="a" data-explanation="Correct. Stop admission, drain workers while needed transport remains available, then close it. A drain deadline can still leave RAM-only work unfinished." data-retry="Could a worker publish a response after MQTT has already been disconnected?"><fieldset class="question-block"><legend><strong>3. What should shutdown do before MQTT disconnect?</strong></legend><label class="option"><input type="radio" name="q3" value="a"> Stop admission, drain workers, then close the transport</label><label class="option"><input type="radio" name="q3" value="b"> Disconnect MQTT, then let workers finish replies later</label><label class="option"><input type="radio" name="q3" value="c"> Keep admitting jobs until the process exits fully</label></fieldset><button type="submit">Check answer</button><p class="feedback" data-feedback aria-live="polite"></p></form>
|
||||
|
||||
<h2>🛠️ Your tiny practice</h2><div class="card"><p>When capacity is full, write one sentence and one metric for each class: heartbeat, sales report, command result. Then state the exact loss window created by automatic acknowledgement plus an in-memory queue. Compare with <a href="../course-source/06-concurrency-cancellation.md">Section 6's dispatcher and shutdown code ↗</a>.</p></div>
|
||||
|
||||
<details class="card" open><summary><strong>📦 Section 6 complete dispatcher files</strong></summary><p>Complete source files for the added dispatcher package. The complete backend integration appears immediately above.</p><article class="card"><h3>📄 internal/dispatch/pool.go</h3><pre><code>package dispatch
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
"sync"
|
||||
"sync/atomic"
|
||||
)
|
||||
|
||||
type Outcome uint8
|
||||
|
||||
// Outcomes make queue admission success and overload visible to callers.
|
||||
const (
|
||||
Accepted Outcome = iota
|
||||
Full
|
||||
Stopping
|
||||
)
|
||||
|
||||
type Job func(context.Context) error
|
||||
|
||||
type Stats struct {
|
||||
Accepted uint64
|
||||
Rejected uint64
|
||||
Completed uint64
|
||||
Failed uint64
|
||||
}
|
||||
|
||||
// Pool owns a bounded queue, worker lifecycle, and admission counters.
|
||||
type Pool struct {
|
||||
mu sync.Mutex
|
||||
accepting bool
|
||||
jobs chan Job
|
||||
ctx context.Context
|
||||
cancel context.CancelFunc
|
||||
workers sync.WaitGroup
|
||||
done chan struct{}
|
||||
|
||||
accepted atomic.Uint64
|
||||
rejected atomic.Uint64
|
||||
completed atomic.Uint64
|
||||
failed atomic.Uint64
|
||||
}
|
||||
|
||||
// New starts a fixed worker set and ties job cancellation to parent.
|
||||
func New(parent context.Context, workers, capacity int) *Pool {
|
||||
if workers < 1 {
|
||||
workers = 1
|
||||
}
|
||||
if capacity < 0 {
|
||||
capacity = 0
|
||||
}
|
||||
ctx, cancel := context.WithCancel(parent)
|
||||
p := &Pool{accepting: true, jobs: make(chan Job, capacity), ctx: ctx, cancel: cancel, done: make(chan struct{})}
|
||||
p.workers.Add(workers)
|
||||
for i := 0; i < workers; i++ {
|
||||
go p.worker()
|
||||
}
|
||||
go func() { p.workers.Wait(); close(p.done) }()
|
||||
return p
|
||||
}
|
||||
|
||||
// Submit never waits for queue capacity. The caller gets an explicit Full result.
|
||||
// The caller context can cancel admission before it enters the queue.
|
||||
func (p *Pool) Submit(ctx context.Context, job Job) (Outcome, error) {
|
||||
if job == nil {
|
||||
return Stopping, errors.New("nil job")
|
||||
}
|
||||
if err := ctx.Err(); err != nil {
|
||||
return Stopping, err
|
||||
}
|
||||
p.mu.Lock()
|
||||
defer p.mu.Unlock()
|
||||
if !p.accepting {
|
||||
return Stopping, nil
|
||||
}
|
||||
select {
|
||||
case p.jobs <- job:
|
||||
p.accepted.Add(1)
|
||||
return Accepted, nil
|
||||
default:
|
||||
p.rejected.Add(1)
|
||||
return Full, nil
|
||||
}
|
||||
}
|
||||
|
||||
func (p *Pool) StopAdmission() {
|
||||
p.mu.Lock()
|
||||
defer p.mu.Unlock()
|
||||
if p.accepting {
|
||||
p.accepting = false
|
||||
close(p.jobs)
|
||||
}
|
||||
}
|
||||
|
||||
// Shutdown stops new submissions and drains queued work until ctx expires.
|
||||
// Jobs must observe their context so deadline cancellation can stop slow work.
|
||||
func (p *Pool) Shutdown(ctx context.Context) error {
|
||||
p.StopAdmission()
|
||||
select {
|
||||
case <-p.done:
|
||||
p.cancel()
|
||||
return nil
|
||||
case <-ctx.Done():
|
||||
p.cancel()
|
||||
return ctx.Err()
|
||||
}
|
||||
}
|
||||
|
||||
func (p *Pool) Stats() Stats {
|
||||
return Stats{Accepted: p.accepted.Load(), Rejected: p.rejected.Load(), Completed: p.completed.Load(), Failed: p.failed.Load()}
|
||||
}
|
||||
|
||||
// Workers execute admitted jobs and count their final result.
|
||||
func (p *Pool) worker() {
|
||||
defer p.workers.Done()
|
||||
for {
|
||||
select {
|
||||
case <-p.ctx.Done():
|
||||
return
|
||||
case job, ok := <-p.jobs:
|
||||
if !ok {
|
||||
return
|
||||
}
|
||||
if err := job(p.ctx); err != nil {
|
||||
p.failed.Add(1)
|
||||
continue
|
||||
}
|
||||
p.completed.Add(1)
|
||||
}
|
||||
}
|
||||
}
|
||||
</code></pre></article>
|
||||
<article class="card"><h3>📄 internal/dispatch/receiver.go</h3><pre><code>package dispatch
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
|
||||
"example.com/mqtt5-learning/internal/contracts"
|
||||
"example.com/mqtt5-learning/internal/mqtt"
|
||||
)
|
||||
|
||||
type Processor func(context.Context, mqtt.Message) error
|
||||
|
||||
// Receiver validates the route, selects a pool, and copies payload ownership.
|
||||
type Receiver struct {
|
||||
Telemetry *Pool
|
||||
Important *Pool
|
||||
Process Processor
|
||||
}
|
||||
|
||||
func (r *Receiver) Submit(ctx context.Context, message mqtt.Message) (Outcome, error) {
|
||||
if r.Process == nil {
|
||||
return Stopping, fmt.Errorf("message processor is required")
|
||||
}
|
||||
topic, err := contracts.ParseTopic(message.Topic)
|
||||
if err != nil {
|
||||
return Stopping, err
|
||||
}
|
||||
pool := r.Telemetry
|
||||
if topic.Kind == contracts.KindCommands || topic.Kind == contracts.KindResponses || topic.Kind == contracts.KindSales {
|
||||
pool = r.Important
|
||||
}
|
||||
if pool == nil {
|
||||
return Stopping, fmt.Errorf("no queue configured for topic kind %q", topic.Kind)
|
||||
}
|
||||
copyOfMessage := message
|
||||
copyOfMessage.Payload = append([]byte(nil), message.Payload...)
|
||||
return pool.Submit(ctx, func(jobCtx context.Context) error { return r.Process(jobCtx, copyOfMessage) })
|
||||
}
|
||||
</code></pre></article>
|
||||
<article class="card"><h3>📄 internal/dispatch/shutdown.go</h3><pre><code>package dispatch
|
||||
|
||||
import (
|
||||
"context"
|
||||
"errors"
|
||||
)
|
||||
|
||||
func Shutdown(ctx context.Context, pools ...*Pool) error {
|
||||
// Close admission on every pool before waiting for any worker to drain.
|
||||
for _, pool := range pools {
|
||||
if pool != nil {
|
||||
pool.StopAdmission()
|
||||
}
|
||||
}
|
||||
var result error
|
||||
// Share one deadline across the drain so total shutdown time stays bounded.
|
||||
for _, pool := range pools {
|
||||
if pool != nil {
|
||||
result = errors.Join(result, pool.Shutdown(ctx))
|
||||
}
|
||||
}
|
||||
return result
|
||||
}
|
||||
</code></pre></article></details>
|
||||
|
||||
<h2>🏗️ Build and smoke-run the updated processes</h2>
|
||||
<p>After saving all three <code>internal/dispatch</code> files and wiring both entrypoints, run these commands from the repository root:</p>
|
||||
<pre><code>go test ./...
|
||||
go test -race ./...
|
||||
go build -o /tmp/mqtt5-lab-backend ./cmd/backend
|
||||
go build -o /tmp/mqtt5-lab-simulator ./cmd/simulator</code></pre>
|
||||
<p>The race detector can report races only on paths the tests execute; it does not prove every worker schedule safe. The builds confirm both programs use the updated wiring.</p>
|
||||
<p>Start only the named lab broker, then run the two binaries in separate terminals:</p>
|
||||
<pre><code>docker compose -p mqtt5-learning up -d broker
|
||||
docker compose -p mqtt5-learning ps</code></pre>
|
||||
<pre><code>/tmp/mqtt5-lab-simulator -username simulator -client-id lab-simulator-device-001</code></pre>
|
||||
<pre><code>/tmp/mqtt5-lab-backend -username backend -client-id lab-backend</code></pre>
|
||||
<p>Confirm heartbeats and ping responses still complete while passing through worker queues. On Ctrl+C, observe the shutdown summary and compare accepted, rejected, completed, failed, and drain-time values if your wiring logs them. This ordinary traffic is a smoke run, not an overload test; use a finite burst larger than active workers plus queue capacity to observe <code>Full</code> outcomes. Stop the clients, then stop the lab broker with <code>docker compose -p mqtt5-learning stop broker</code>.</p>
|
||||
<p>Ask me a follow-up if the counters or shutdown behavior differ from your prediction.</p>
|
||||
|
||||
<section class="card callout"><h2>✅ Takeaway</h2><p><strong>A bounded queue protects process memory; an explicit overload policy protects business meaning. Neither makes work durable.</strong></p><p>Tell me which message class you would admit first and why.</p></section>
|
||||
<footer><p>Previous: <a href="0005a-backend-message-processing.html">Lesson 5A</a> · Next: <a href="0007-command-state.html">Lesson 7 — command identity and state</a> · <a href="../reference/message-processing-lifecycle.html">lifecycle reference</a></p><p>Sources: <a href="https://pkg.go.dev/context">Go `context`</a>; <a href="https://pkg.go.dev/sync">Go `sync`</a>; <a href="https://pkg.go.dev/github.com/eclipse/paho.golang/autopaho/examples/queue@v0.23.0">Paho Go v0.23.0 queue example</a>; <a href="../course-source/06-concurrency-cancellation.md">repository Section 6</a>.</p></footer>
|
||||
</main></body></html>
|
||||
Reference in New Issue
Block a user