126 lines
11 KiB
Markdown
126 lines
11 KiB
Markdown
# 🧵 Section 6 — Manage callbacks, concurrency, and cancellation
|
|
|
|
## 🎬 Start with the system story
|
|
|
|
> 💡 **Story question:** A burst of broker-forwarded messages arrives while the backend is doing slow work. Who is receiving packets, who is doing business work, and what happens when the application cannot keep up?
|
|
|
|
Paho's network/client machinery receives packets and invokes configured handlers. Your application decides how that work is scheduled, bounded, cancelled, and acknowledged. If a callback blocks or a queue fills, that can affect unrelated deliveries; if a message is acknowledged before durable handling and then dropped from RAM, MQTT cannot recover that application copy by itself.
|
|
|
|
```mermaid
|
|
flowchart LR
|
|
B[Broker] -->|PUBLISH| P[Paho receive path]
|
|
P --> C[Short callback copies/enqueues]
|
|
C --> Q[Bounded application queue]
|
|
Q --> W[Worker handles and persists]
|
|
Q -->|full| O[Explicit overload policy]
|
|
W --> A[Application outcome]
|
|
```
|
|
|
|
**Before each implementation step, ask:** What work must happen before the MQTT acknowledgement, what can happen afterward, and what is the defined behavior when capacity is exhausted? Follow [Lesson 0](00-system-lifecycle.md) for the broker-to-callback-to-application path.
|
|
|
|
Prerequisite: Section 5 validates messages before handling them. The backend and simulator continue to read their shared disposable password from `configs/lab.toml`; select each broker username with `-username` when starting the process.
|
|
|
|
Run all commands from `/home/ryan/Media/Code/Code/robot-market/mqtt-5-learning`. Create files at the project-relative paths shown below, not under `lessons/`. Continue the same application and pinned Paho v0.23.0 dependency. These are guided implementation exercises; attempt each step before requesting a full solution.
|
|
|
|
## ✅ Understand the acknowledgement boundary before adding workers
|
|
|
|
For the pinned client's automatic mode, `routePublishPackets` invokes the registered handlers, then calls its session acknowledgement path for nonzero QoS. The boolean and error returned by a handler are passed to later handlers; returning an error does not turn the callback into an application-controlled MQTT NACK or force redelivery.
|
|
|
|
Trace the early QoS 1 implementation:
|
|
|
|
```text
|
|
receive PUBLISH → copy into RAM queue → callback returns → MQTT ACK
|
|
→ worker validates/applies effect later
|
|
```
|
|
|
|
The worker may finish before or after the ACK. If the process crashes after the ACK but before the effect, the broker can consider its delivery complete while the application loses the work. Increasing channel capacity increases the possible backlog; it does not close that gap.
|
|
|
|
In manual mode, `EnableManualAcknowledgment` changes who releases acknowledgement. For QoS 1, the intended durable-ingestion sequence is receive → validate → commit durable admission → call `received.Client.Ack(received.Packet)`. Preserve the receiving client and original packet for this operation; a payload-only copy is insufficient. Check Ack's error and study the pinned source before implementing it. Do not reuse this QoS 1 explanation as a claim that every QoS 2 handshake stage waits for your database.
|
|
|
|
The client's tracker sends acknowledgements in receipt order. If work 3 finishes before 1, acknowledging 3 can remain buffered behind 1. Slow work therefore consumes inflight capacity even with several workers. Manual acknowledgement delays protocol completion; it does not persist your Go queue, remove duplicate effects, or automatically schedule retries while the same connection stays open.
|
|
|
|
### ↩️ Choose what the callback promises
|
|
|
|
| Message class | Reasonable admission choice | Meaning of a successful admission |
|
|
| --- | --- | --- |
|
|
| Disposable heartbeat | Bounded RAM queue, counted drops | A recent sample may be processed while this process survives |
|
|
| Important report | Durable inbox before acknowledgement, introduced in Section 13 | Work can be recovered after this process restarts |
|
|
| Command | Validate identity/deadline and record dedup/admission state | Device acceptance still needs an explicit application result |
|
|
|
|
For this in-memory lesson, keep the loss limitation visible. Use a fake acknowledgement recorder to test ordering of admission and acknowledgement without pretending that the fake verifies broker behavior.
|
|
|
|
### 🧪 Predict the overload
|
|
|
|
Two workers taking 100 ms per item can complete about 20 items/second before other overhead. A burst of 100 items arriving much faster cannot fit into two busy workers plus 16 waiting slots. Predict rejection, then measure actual accepted/completed/rejected counts; scheduler timing affects the exact split. Increasing workers may reorder effects for the same device. If revision order matters, serialize that device's work or enforce revision checks.
|
|
|
|
Shutdown also has a dependency: if draining work must publish results, keep its MQTT connection alive while draining. Stop new admission first, drain accepted work with a deadline, then disconnect transport and join workers. If you disconnect first, record unsent results for recovery; an in-memory-only lesson cannot promise to deliver them. The signal context in the early example cancels AutoPaho immediately, so a real drain policy needs a separately controlled transport lifetime.
|
|
|
|
<details>
|
|
<summary>Does returning false or an error prevent the broker from receiving PUBACK?</summary>
|
|
|
|
Not in the pinned automatic callback path. These return values describe local handler processing. A deliberate acknowledgement policy uses the client's supported manual mechanism and recovery design. It must also handle invalid permanent failures so one bad message cannot occupy capacity forever.
|
|
|
|
</details>
|
|
|
|
## ↩️ 1. Trace callback and acknowledgement order
|
|
|
|
> 💡 **Story question:** Does returning from the receive callback mean the business operation is durable, or only that the MQTT client may acknowledge delivery?
|
|
|
|
Read the pinned source and write the observed sequence down before choosing a queue or acknowledgement policy.
|
|
|
|
Read `references/paho.golang-v0.23.0/paho/client.go`, especially receive callback processing, `EnableManualAcknowledgment`, and `Ack`. Write the observed order in `journal/06-concurrency.md`. Inspect this pinned source before choosing an acknowledgement policy.
|
|
|
|
Section 4 copies payloads into a bounded channel and drops messages when full. Automatic MQTT acknowledgement can still occur before your application finishes work. This is a deliberate loss window, not reliable ingestion.
|
|
|
|
## 🛠️ 2. Build a bounded dispatcher
|
|
|
|
> 💡 **Story question:** When messages arrive faster than handlers can finish, where is excess work stored and how does the application report that capacity is exhausted?
|
|
|
|
Set a finite capacity and make admission outcomes explicit for each message class.
|
|
|
|
Create `internal/dispatch/pool.go` and `pool_test.go`. Start with two workers and capacity 16. Expose a context-aware submit operation and explicit accepted/full/stopping outcomes. Count accepted, rejected, completed, and failed work. Keep telemetry and important work in separate bounded queues so a heartbeat burst cannot occupy every command slot.
|
|
|
|
Choose a policy per class: disposable telemetry may be dropped with a counter; commands and sales must produce an explicit admission failure rather than silently disappear. Admission failure under automatic MQTT acknowledgement still loses application work. Record this limitation until durable admission is introduced in Section 13. Do not claim manual acknowledgement alone makes an in-memory queue durable.
|
|
|
|
## 🛑 3. Define ownership and shutdown
|
|
|
|
> 💡 **Story question:** If a worker is still processing when shutdown begins, which components must stay alive until the work finishes or its deadline expires?
|
|
|
|
Define buffer ownership, stop admission, drain accepted work, then stop the MQTT producer path in a bounded order.
|
|
|
|
Copy retained byte slices, including future correlation metadata. Protect shared maps with a mutex or give one goroutine ownership. Never hold a state mutex while publishing over the network. Use operation deadlines derived from the application lifecycle.
|
|
|
|
On shutdown, stop admitting new work and let accepted work finish within a separate bounded drain context. Keep MQTT available during the drain if workers must publish results, then stop the transport producers. Cancel remaining work when the drain deadline expires. Never close a queue while a producer can still send; waiting for worker completion is part of shutdown. Document what happens to unfinished in-memory work.
|
|
|
|
## 🔗 4. Integrate and stress it
|
|
|
|
> 💡 **Story question:** What observations would show that overload is bounded and shutdown is controlled, rather than merely that the process eventually exits?
|
|
|
|
Measure accepted, rejected, completed work, queue capacity, and shutdown time under a finite burst.
|
|
|
|
Edit the backend and simulator processing loops to submit validated work. Keep transport callbacks short. Add a configurable artificial handler delay and a finite synthetic burst generator; do not create one goroutine per incoming message. Keep these switches in typed TOML + koanf configuration and preserve existing precedence.
|
|
|
|
```sh
|
|
mkdir -p internal/dispatch journal
|
|
|
|
gofmt -w internal/dispatch cmd
|
|
|
|
go test -race ./...
|
|
|
|
go vet ./...
|
|
```
|
|
|
|
Create `internal/dispatch` and `journal` before saving their files; the `mkdir -p` command is safe to run earlier than the final check block. `go test -race ./...` instruments the test binaries to report unsynchronized memory access during executed tests. It is slower than normal testing and cannot prove unexecuted paths are race-free. `go vet` checks static patterns; it does not replace the race run.
|
|
|
|
A finite burst generator publishes a configured count and then stops; record its rate, queue capacity, worker count, and artificial delay. A bounded queue has fixed capacity: when workers are slower than arrival, overload becomes measurable instead of consuming memory indefinitely. Worker cancellation is cooperative: handlers must check contexts and use deadlines on blocking operations. Observe accepted/rejected counts and total shutdown time, not just whether the process eventually exits.
|
|
|
|
Use channels in tests to hold a worker busy deterministically, fill the queue, and assert the next submission fails. Test cancellation, concurrent submissions, pending-work shutdown, and accounting. Avoid tests that rely only on sleeps.
|
|
|
|
**Done when:** A burst causes observable bounded overload, normal ping/pong recovers afterward, shutdown finishes by its deadline, and the race detector passes. Explain the difference between broker acknowledgement, queue admission, and completed application work.
|
|
|
|
## ✅ Result — the story so far
|
|
|
|
The receive callback now has a defined small job: copy and admit work into bounded queues, then return. Workers validate and handle admitted messages, while counters make overload visible and shutdown follows a deliberate stop-admission, drain, and disconnect order. In this lesson's in-memory design, automatic MQTT acknowledgement can happen before worker completion, so a process crash can still lose queued work; durable admission is a later step.
|
|
|
|
Previous: [05-contracts-validation.md](05-contracts-validation.md). Next: [07-command-response.md](07-command-response.md).
|