mqtt-5-learning/teaching/course-source/09-sessions-recovery.md

Section 9 — Learn MQTT 5 sessions through failure experiments

Prerequisite: Section 8 exposes commands and their evidence; use only synthetic ping effects.

Run all commands from /home/ryan/Media/Code/Code/robot-market/mqtt-5-learning. Create files at the project-relative paths shown below, not under teaching/course-source/. Continue the same application and pinned Paho v0.23.0 dependency. These are guided implementation exercises; attempt each step before requesting a full solution.

A connection is temporary; a session may outlive it

❓ Question: The simulator loses its network connection after subscribing. What must still exist for a queued ping to reach it when it reconnects?

💬 Answer: The simulator must reconnect with the same Client ID to a broker session that still exists, and that session must contain its QoS 1 subscription. The backend's PUBACK only confirms the broker accepted that publication on this hop; it does not prove the simulator received or completed the ping. Your Go command map lives in a different place and is not restored by CONNACK.

The network connection carries packets now. A broker session can preserve subscriptions and eligible pending deliveries after that connection closes.

Clean Start on this connection	Prior unexpired session under the same Client ID	Expected Session Present	Meaning
true	Either	false	Discard old state and begin a new session
false	Yes	true	Resume that broker session
false	No	false	Create a new session; restore subscriptions

Session Expiry answers a later question: how long may the session remain after disconnect? Setting Clean Start true with expiry 60 creates a new session that can subsequently survive a short outage. Setting Clean Start false with expiry 0 can resume an existing session now, but asks for no persistence after this connection ends. They are independent controls, not two names for the same switch.

In AutoPaho v0.23.0, CleanStartOnInitialConnection affects the manager's first successful connection; automatic reconnects use false. Restarting the Go process creates a new manager and applies the initial choice again. Therefore "network reconnect works" does not establish "process restart resumes the old session."

Work through an offline command

❓ Question: Why does a new Client ID not receive the offline ping sent for the old simulator ID?

💬 Answer: Client ID selects the broker session. A new ID starts or resumes a different session, so it has no old subscription or queued delivery. A device_id in the payload is only an application claim and does not select or authenticate a broker session.

At t=0 the simulator has subscribed at QoS 1 with expiry 60. At t=5 it disconnects. At t=10 the backend publishes a nonretained QoS 1 ping. At t=20 the same simulator ID reconnects with Clean Start false. Assuming the broker retained the session, accepted the publication, and no shorter message expiry or resource limit intervened, the subscription allows queued delivery. Reconnect after the session expires and the old queue is gone.

Creating a new subscriber after publication is different: it had no prior subscription/session to receive that queued nonretained message. Retained state is associated with a topic and can initialize a new subscriber, even when it has no prior session.

Compare queued events with retained latest state

The broker session queue and retained-message store answer different questions:

Mechanism	What it represents	Who can receive it later
Nonretained QoS 1 publication to an offline session	A one-shot event the broker may queue for an existing matching subscription	That session, while its subscription and expiry remain valid
Retained publication	The latest retained value for a topic, replacing the prior retained value	A subscriber that newly subscribes to the topic, including one with a new Client ID

Run this comparison only against the disposable broker and simulator. First finish the queued-ping experiment and reconnect the original simulator. Keep it online so the retained probe cannot also accumulate in its offline session. Publish a synthetic retained ping to its lab command topic with the MQTT 5 CLI and authorized lab backend identity; confirm the Section 7 simulator rejects it before executing it. Then subscribe with an observer using a fresh Client ID and the lab simulator identity; it should receive the stored latest value immediately despite having no prior session. Use no physical device or production credential.

Immediately clear the retained value by publishing an empty payload with RETAIN set to the same topic (Mosquitto CLI uses -r -n). Verify a fresh subscription receives no replay. Do not leave retained one-shot commands in the lab. The point is to see that retained delivery initializes the subscriber with the latest value; it is not an event log and must not be used to queue executable commands.

Put each recovery responsibility in the correct process

State	Early lesson implementation	Network disconnect	Owning process restart
Client protocol session	Paho memory state for inflight handshakes	Manager can reuse state subject to broker session	Lost unless persistent client stores are configured
Client outgoing queue	Early code uses direct Publish, not PublishViaQueue	Direct publish can fail while disconnected	No durable outgoing backlog exists in this helper
Broker session	Broker memory plus configured persistence, gated by expiry	Can preserve subscription/eligible deliveries	Depends on saved broker state, expiry, and recovery
Application records	Go command and dedup maps	Survive while the Go process lives	Lost until database-backed recovery is implemented

PublishViaQueue is a separate AutoPaho API. A nil result means local enqueue succeeded; its API does not return eventual broker/device status. A file-backed outgoing queue and file-backed protocol session stores solve different problems. Neither replaces an application outbox or device deduplication.

Session Present is true, but GET command status says unknown. Is the broker broken?

Not necessarily. The broker resumed its session while the backend's in-memory application records were lost on process restart. MQTT has no knowledge of your HTTP command map. Record both pieces of evidence rather than using Session Present as a general recovery-success flag.

1. Add session controls

❓ Question: What does Session Present tell you after CONNACK, and what does it leave unknown?

💬 Answer: It reports whether the broker resumed prior session state for this connection. It does not report whether the backend's command record survived, whether a device acted, or whether every queued application job completed.

Extend the typed configuration with Clean Start and Session Expiry Interval in seconds, and pass them into AutoPaho. Edit internal/mqtt/client.go to log SessionPresent from CONNACK without logging credentials. Record connection and subscription readiness separately. Read the pinned autopaho/auto.go: CleanStartOnInitialConnection applies to the first connection of that manager; automatic reconnects behave differently from constructing a new manager.

Use unique stable IDs for backend and simulator. Run one process per client ID. A changing ID creates a different session, while simultaneous clients sharing an ID can take over each other's connection.

Session Expiry is seconds in the MQTT property, whereas Go operation deadlines use time.Duration. A nonzero expiry preserves eligible broker session state after disconnect for that interval; Clean Start discards an old session at connection start. Session Present reports whether the broker resumed prior session state, not whether your Go command records survived.

Create the journal directory before saving the experiment record. Use Ctrl+C for a graceful simulator shutdown first; log whether DISCONNECT was sent and whether your configured expiry remains nonzero. For an abrupt-process experiment, run a built simulator binary and record its PID, then terminate that exact lab process; killing a go run wrapper can make the process under test ambiguous. Record the shutdown method because a graceful disconnect and lost network connection are different evidence.

2. Run an experiment matrix

Create journal/09-sessions.md. For each row record settings, prediction, exact stop/start steps, CONNACK Session Present, SUBACK, received command IDs, duplicates, and observed result.

Experiment	Simulator setup	Observation to explain
Fresh connection	Clean Start true, expiry 0	No offline-session guarantee
Short outage	Clean Start false, expiry 60, stable ID	Reconnect before expiry and inspect queued QoS 1 delivery
Expired session	Same settings, outage longer than 60 seconds	Inspect Session Present and missing queued work
New identity	Change only client ID	Previous session is not this new client's session
Broker restart	Persistent lab broker	Compare network recovery with broker-process recovery

For offline delivery, first establish the simulator subscription at QoS 1. Disconnect it while preserving a nonzero-expiry session, then publish a nonretained QoS 1 ping from the still-connected backend. Reconnect before expiry using the same ID and Clean Start false. Observe broker evidence; do not infer delivery merely from a successful backend publish. Choose a command response deadline long enough for the experiment, and record late-result policy when exceeding it.

3. Audit subscription restoration

The early helper subscribes on every connection-up event. Keeping this policy is acceptable for the experiment; log each SUBACK and do not register another local callback on every reconnect. Test both session-present and fresh-session paths. A connected process whose subscriptions failed is not ready to receive commands.

4. Compare broker persistence safely

Use only this project's Compose commands. Keep the default persistence volume intact. For a nonpersistent comparison, stop the lab, temporarily change only lab/mosquitto/mosquitto.conf to disable persistence, and use a separate empty lab data volume in a Compose override so old state cannot contaminate the result. Restore the original configuration afterward. Do not delete volumes to run this experiment.

A graceful broker shutdown and an abrupt broker kill may preserve different amounts of recently changed state. Record which you used and the broker's persistence settings rather than generalizing one observation to every crash.

A Compose override is an additional YAML file layered over compose.yaml using -f compose.yaml -f <override-file> before the subcommand. Create it at the root and use that same pair for config/up/ps/stop commands throughout the comparison. Replace the broker's /mosquitto/data source with a new declared named volume; Compose merges mount entries by container destination. Verify the resolved config output shows the new source before starting it. Stop the comparison, then run up -d with the original file alone to restore the original mounts; start would only reuse the comparison container. Refer to Broker operations for volume identity and recreation.

Turning persistence off does not erase existing in-memory sessions or remove old disk data. Use the separate empty volume and a recreated container to avoid that ambiguity. After restoring settings and mounts, inspect logs and repeat normal authentication/ping checks.

5. Separate the recovery layers

Write a table with four columns: client session storage, client outgoing publication queue, broker session state, and application command/dedup state. For each, state current storage, lifetime, and what survives network interruption or process restart. Read the pinned Paho session and AutoPaho queue documentation; neither broker persistence nor QoS 1 makes your in-memory command records durable.

gofmt -w internal cmd
go test -race ./...
docker compose -p mqtt5-learning restart broker
docker compose -p mqtt5-learning logs --tail=50 broker

restart broker stops/starts only that project's broker container, interrupting its client connections; it does not apply edited Compose mounts or image settings. logs --tail=50 broker inspects the last 50 service log lines. Observe client reconnection and Session Present as well as broker logs. gofmt rewrites edited Go formatting; go test -race exercises tests with race detection. For mount/config changes requiring recreation, follow the preceding override instructions instead of assuming restart applies them.

Done when: Every matrix row has a prediction and measured result, you can distinguish automatic reconnect from a new process, and you can explain each recovery layer. An offline ping delivered after your backend timed out must remain consistent with Section 7's unknown-outcome policy.

Previous: 08-http-commands.md. Next: Section 10 — Expiry, presence, and retained state.