85 lines
8.2 KiB
Markdown
85 lines
8.2 KiB
Markdown
# Section 6 — Manage callbacks, concurrency, and cancellation
|
|
|
|
Prerequisite: Section 5 validates messages before handling them.
|
|
|
|
Run all commands from `/home/ryan/Media/Code/Code/robot-market/mqtt-5-learning`. Create files at the project-relative paths shown below, not under `lessons/`. Continue the same application and pinned Paho v0.23.0 dependency. These are guided implementation exercises; attempt each step before requesting a full solution.
|
|
|
|
## Understand the acknowledgement boundary before adding workers
|
|
|
|
For the pinned client's automatic mode, `routePublishPackets` invokes the registered handlers, then calls its session acknowledgement path for nonzero QoS. The boolean and error returned by a handler are passed to later handlers; returning an error does not turn the callback into an application-controlled MQTT NACK or force redelivery.
|
|
|
|
Trace the early QoS 1 implementation:
|
|
|
|
```text
|
|
receive PUBLISH → copy into RAM queue → callback returns → MQTT ACK
|
|
→ worker validates/applies effect later
|
|
```
|
|
|
|
The worker may finish before or after the ACK. If the process crashes after the ACK but before the effect, the broker can consider its delivery complete while the application loses the work. Increasing channel capacity increases the possible backlog; it does not close that gap.
|
|
|
|
In manual mode, `EnableManualAcknowledgment` changes who releases acknowledgement. For QoS 1, the intended durable-ingestion sequence is receive → validate → commit durable admission → call `received.Client.Ack(received.Packet)`. Preserve the receiving client and original packet for this operation; a payload-only copy is insufficient. Check Ack's error and study the pinned source before implementing it. Do not reuse this QoS 1 explanation as a claim that every QoS 2 handshake stage waits for your database.
|
|
|
|
The client's tracker sends acknowledgements in receipt order. If work 3 finishes before 1, acknowledging 3 can remain buffered behind 1. Slow work therefore consumes inflight capacity even with several workers. Manual acknowledgement delays protocol completion; it does not persist your Go queue, remove duplicate effects, or automatically schedule retries while the same connection stays open.
|
|
|
|
### Choose what the callback promises
|
|
|
|
| Message class | Reasonable admission choice | Meaning of a successful admission |
|
|
| --- | --- | --- |
|
|
| Disposable heartbeat | Bounded RAM queue, counted drops | A recent sample may be processed while this process survives |
|
|
| Important report | Durable inbox before acknowledgement, introduced in Section 13 | Work can be recovered after this process restarts |
|
|
| Command | Validate identity/deadline and record dedup/admission state | Device acceptance still needs an explicit application result |
|
|
|
|
For this in-memory lesson, keep the loss limitation visible. Use a fake acknowledgement recorder to test ordering of admission and acknowledgement without pretending that the fake verifies broker behavior.
|
|
|
|
### Predict the overload
|
|
|
|
Two workers taking 100 ms per item can complete about 20 items/second before other overhead. A burst of 100 items arriving much faster cannot fit into two busy workers plus 16 waiting slots. Predict rejection, then measure actual accepted/completed/rejected counts; scheduler timing affects the exact split. Increasing workers may reorder effects for the same device. If revision order matters, serialize that device's work or enforce revision checks.
|
|
|
|
Shutdown also has a dependency: if draining work must publish results, keep its MQTT connection alive while draining. Stop new admission first, drain accepted work with a deadline, then disconnect transport and join workers. If you disconnect first, record unsent results for recovery; an in-memory-only lesson cannot promise to deliver them. The signal context in the early example cancels AutoPaho immediately, so a real drain policy needs a separately controlled transport lifetime.
|
|
|
|
<details>
|
|
<summary>Does returning false or an error prevent the broker from receiving PUBACK?</summary>
|
|
|
|
Not in the pinned automatic callback path. These return values describe local handler processing. A deliberate acknowledgement policy uses the client's supported manual mechanism and recovery design. It must also handle invalid permanent failures so one bad message cannot occupy capacity forever.
|
|
|
|
</details>
|
|
|
|
## 1. Trace callback and acknowledgement order
|
|
|
|
Read `references/paho.golang-v0.23.0/paho/client.go`, especially receive callback processing, `EnableManualAcknowledgment`, and `Ack`. Write the observed order in `journal/06-concurrency.md`. Inspect this pinned source before choosing an acknowledgement policy.
|
|
|
|
Section 4 copies payloads into a bounded channel and drops messages when full. Automatic MQTT acknowledgement can still occur before your application finishes work. This is a deliberate loss window, not reliable ingestion.
|
|
|
|
## 2. Build a bounded dispatcher
|
|
|
|
Create `internal/dispatch/pool.go` and `pool_test.go`. Start with two workers and capacity 16. Expose a context-aware submit operation and explicit accepted/full/stopping outcomes. Count accepted, rejected, completed, and failed work. Keep telemetry and important work in separate bounded queues so a heartbeat burst cannot occupy every command slot.
|
|
|
|
Choose a policy per class: disposable telemetry may be dropped with a counter; commands and sales must produce an explicit admission failure rather than silently disappear. Admission failure under automatic MQTT acknowledgement still loses application work. Record this limitation until durable admission is introduced in Section 13. Do not claim manual acknowledgement alone makes an in-memory queue durable.
|
|
|
|
## 3. Define ownership and shutdown
|
|
|
|
Copy retained byte slices, including future correlation metadata. Protect shared maps with a mutex or give one goroutine ownership. Never hold a state mutex while publishing over the network. Use operation deadlines derived from the application lifecycle.
|
|
|
|
On shutdown, stop admitting new work and let accepted work finish within a separate bounded drain context. Keep MQTT available during the drain if workers must publish results, then stop the transport producers. Cancel remaining work when the drain deadline expires. Never close a queue while a producer can still send; waiting for worker completion is part of shutdown. Document what happens to unfinished in-memory work.
|
|
|
|
## 4. Integrate and stress it
|
|
|
|
Edit the backend and simulator processing loops to submit validated work. Keep transport callbacks short. Add a configurable artificial handler delay and a finite synthetic burst generator; do not create one goroutine per incoming message. Keep these switches in typed TOML + koanf configuration and preserve existing precedence.
|
|
|
|
```sh
|
|
mkdir -p internal/dispatch journal
|
|
gofmt -w internal/dispatch cmd
|
|
go test -race ./...
|
|
go vet ./...
|
|
```
|
|
|
|
Create `internal/dispatch` and `journal` before saving their files; the `mkdir -p` command is safe to run earlier than the final check block. `go test -race ./...` instruments the test binaries to report unsynchronized memory access during executed tests. It is slower than normal testing and cannot prove unexecuted paths are race-free. `go vet` checks static patterns; it does not replace the race run.
|
|
|
|
A finite burst generator publishes a configured count and then stops; record its rate, queue capacity, worker count, and artificial delay. A bounded queue has fixed capacity: when workers are slower than arrival, overload becomes measurable instead of consuming memory indefinitely. Worker cancellation is cooperative: handlers must check contexts and use deadlines on blocking operations. Observe accepted/rejected counts and total shutdown time, not just whether the process eventually exits.
|
|
|
|
Use channels in tests to hold a worker busy deterministically, fill the queue, and assert the next submission fails. Test cancellation, concurrent submissions, pending-work shutdown, and accounting. Avoid tests that rely only on sleeps.
|
|
|
|
**Done when:** A burst causes observable bounded overload, normal ping/pong recovers afterward, shutdown finishes by its deadline, and the race detector passes. Explain the difference between broker acknowledgement, queue admission, and completed application work.
|
|
|
|
Previous: [05-contracts-validation.md](05-contracts-validation.md). Next: [07-command-response.md](07-command-response.md).
|