mqtt-5-learning/teaching/course-source/10-expiry-presence-state.md

Section 10 — Expiry, presence, and retained state

Prerequisite: complete the session experiments in Lesson 9. Continue the same lab and pinned Paho v0.23.0. These are incremental exercises; the fragments below belong in your existing transport and simulator, not new standalone programs.

1. Give every timer one responsibility

❓ Question: A ping is queued while a device is offline, then reaches it after the operator has changed the desired setting. Which timer can stop stale delivery, and which check must still run on the device?

💬 Answer: MQTT Message Expiry limits how long the broker may forward the publication. The device must also check the application's absolute deadline before starting the action, because work may have spent time in a local queue or arrived just before expiry. Neither timer can reverse an action already started.

Timer	What expires	What it cannot prove
Go publish context	Your wait for the publication operation	That the device never received it
Application command deadline	Permission to begin a stale action under your contract	That an already-started action was undone
MQTT Message Expiry	A publication's remaining delivery lifetime	That a delivered message stops executing
Session Expiry	Broker session after disconnect	That application records were saved
Keep Alive	Acceptable MQTT connection inactivity	That the application is healthy
Will Delay	Delay before an eligible Will is published	That the device is physically broken

A vending configuration command that was useful five minutes ago may be wrong now. Use Message Expiry to limit stale delivery and an application deadline to reject stale execution after delivery. A response timeout merely tells the caller it lacks a result. A PUBACK can arrive before the simulator validates the command; if the application rejects it, that MQTT acknowledgement remains valid for the protocol step it acknowledged.

This fragment builds MQTT metadata for a five-second publication lifetime:

expirySeconds := uint32(5)
properties := &paho.PublishProperties{
    ContentType:   "application/json",
    MessageExpiry: &expirySeconds,
}

Attach properties to the outbound PUBLISH. A nil MessageExpiry omits the property; an explicit zero is not an unlimited lifetime. The broker reduces the remaining lifetime when forwarding queued work. Time spent waiting in your own outgoing queue needs a separate policy: check the absolute application deadline before transmission and execution. Do not reissue an old command with a fresh five seconds indefinitely.

Experiment: preserve the simulator session for 60 seconds, disconnect it, publish one five-second QoS 1 ping, and reconnect after ten seconds. Expect no expired command delivery. Repeat with a much longer message expiry and reconnect within it: expect delivery if the other session/ACL prerequisites hold. Log IDs and timestamps, not just a message count. Allow timing margin around expiry boundaries.

2. Retained state is a latest value, not an event history

❓ Question: A new simulator subscribes and receives a retained dispense command from last week. What design choice caused this replay?

💬 Answer: A one-shot action was published with RETAIN set. Retained publications are stored by topic for future subscribers, so use retained messages for replaceable state such as desired configuration and nonretained messages for one-time actions.

A retained publication replaces the stored value for one exact topic. It can initialize a newly created subscription even when that client has never connected before. Offline session queues instead belong to an existing session and its subscriptions. Unsubscribing or deleting a session does not delete a retained value.

Use a desired-configuration topic for state such as revision 7 and a temperature setting. Use nonretained command topics for one-shot actions. A retained "dispense" would risk replay when someone subscribes later.

Extend the ACL and the shared topic contract before using these flows: backend writes lab/v1/devices/+/desired, simulator reads its exact .../device-001/desired; simulator writes .../device-001/reported, backend reads .../+/reported. The complete topic.go update below adds those flow kinds to DeviceTopic, DeviceFilter, DeviceSubscription, and ParseTopic. Reload/restart the lab and verify both allowed and denied directions; the broker ACL remains its own source of authorization.

Implement desired and reported revision records. Store the highest applied revision in the simulator and explain the initial in-memory restart limit. Receiving revision 6 after 7 must not overwrite 7. A backend PUBACK for revision 7 proves neither application nor reported revision 7; wait for the device's report.

Clear the retained telemetry from Lesson 4

Stop the simulator's periodic telemetry first so it cannot replace the value during your observation. Run this with the simulator's temporary lab password, using the same CLI credential caveat as Lesson 1:

mosquitto_pub -h 127.0.0.1 -p 18883 -V mqttv5 -u simulator -P '<simulator-lab-password>' -t 'lab/v1/devices/device-001/telemetry' -q 1 -r -n -d

-r sets RETAIN; -n supplies a zero-byte payload. Together they delete the retained value at that exact topic. An empty JSON object or the two characters "" are nonempty payloads and do not delete it. Existing subscribers can see the empty publication; a fresh subscriber should receive no retained replay. Removing the stored value does not retract copies already delivered or queued elsewhere. Restore ordinary nonretained telemetry after checking.

3. Model presence as evidence

❓ Question: The broker's retained presence says online, but the last heartbeat is old. What can the backend safely conclude?

💬 Answer: It can conclude only that the broker currently stores an online assertion. It cannot conclude the device is healthy now. Track backend receipt time and heartbeat freshness, and compare a connection or boot identity so delayed state does not replace newer evidence.

The simulator supplies its Will during CONNECT. The broker later publishes it when the connection ends in a way that requires a Will; it is not code that the dead simulator executes. First add simulator write/backend read ACLs for its .../presence topic, then extend the allowlist in internal/contracts/topic.go. Keep using DeviceTopic so the Will destination follows the same validated hierarchy as ordinary publishes.

Replace internal/contracts/topic.go with this complete version. It extends the allowlist with the flows used in this lesson while keeping the existing hierarchy and constructors:

package contracts

import (
	"fmt"
	"strings"
)

type TopicName string
type TopicFilter string
type Kind string

const (
	KindTelemetry Kind = "telemetry"
	KindCommands  Kind = "commands"
	KindResponses Kind = "responses"
	KindSales     Kind = "sales"
	KindPresence  Kind = "presence"
	KindDesired   Kind = "desired"
	KindReported  Kind = "reported"
)

const topicPrefix = "lab/v1/devices/"

type Topic struct {
	DeviceID string
	Kind     Kind
}

func (t TopicName) String() string   { return string(t) }
func (f TopicFilter) String() string { return string(f) }

func DeviceTopic(deviceID string, kind Kind) (TopicName, error) {
	if !validID(deviceID) { return "", fmt.Errorf("invalid device ID %q", deviceID) }
	if !supportedKind(kind) { return "", fmt.Errorf("unsupported topic kind %q", kind) }
	return TopicName(topicPrefix + deviceID + "/" + string(kind)), nil
}

func DeviceFilter(kind Kind) (TopicFilter, error) {
	if !supportedKind(kind) { return "", fmt.Errorf("unsupported topic kind %q", kind) }
	return TopicFilter(topicPrefix + "+/" + string(kind)), nil
}

func DeviceSubscription(deviceID string, kind Kind) (TopicFilter, error) {
	topic, err := DeviceTopic(deviceID, kind)
	if err != nil { return "", err }
	return TopicFilter(topic.String()), nil
}

func ParseTopic(value string) (Topic, error) {
	if !strings.HasPrefix(value, topicPrefix) {
		return Topic{}, fmt.Errorf("topic must be %s{deviceID}/{kind}", topicPrefix)
	}
	parts := strings.Split(strings.TrimPrefix(value, topicPrefix), "/")
	if len(parts) != 2 || !validID(parts[0]) {
		return Topic{}, fmt.Errorf("topic must be %s{deviceID}/{kind}", topicPrefix)
	}
	kind := Kind(parts[1])
	if !supportedKind(kind) { return Topic{}, fmt.Errorf("unsupported topic kind %q", kind) }
	return Topic{DeviceID: parts[0], Kind: kind}, nil
}

func supportedKind(kind Kind) bool {
	switch kind {
	case KindTelemetry, KindCommands, KindResponses, KindSales,
		KindPresence, KindDesired, KindReported:
		return true
	default:
		return false
	}
}

func validID(value string) bool {
	if value == "" || len(value) > 128 { return false }
	for _, r := range value {
		if !(r >= 'a' && r <= 'z' || r >= 'A' && r <= 'Z' || r >= '0' && r <= '9' || strings.ContainsRune("._-", r)) { return false }
	}
	return true
}

All three builders now accept these new kinds through the shared allowlist. Do not add arbitrary user-supplied path segments or wildcards to DeviceTopic.

Build the will topic before creating the simulator connection, then add these fields to its AutoPaho configuration. The surrounding run function already imports contracts:

presenceTopic, err := contracts.DeviceTopic(cfg.MQTT.DeviceID, contracts.KindPresence)
if err != nil { return fmt.Errorf("build presence topic: %w", err) }

WillMessage: &paho.WillMessage{
    Topic:   presenceTopic.String(),
    QoS:     1,
    Retain:  true,
    Payload: []byte(`{"status":"offline"}`),
},
WillProperties: &paho.WillProperties{
    WillDelayInterval: paho.Uint32(5),
    ContentType:       "application/json",
},

paho.Uint32(5) supplies the optional MQTT property as a pointer; the unit is seconds. This is a minimal presence payload for the experiment. Extend it with your schema and a connection/boot identity for the real lesson implementation. A Will payload is prepared at connection time: a timestamp embedded there is not the later time of disconnection. Record backend receipt time separately.

Publish retained online presence after successful simulator startup. Publish an explicit retained offline value before a planned disconnect if observers need that signal. Normal DISCONNECT suppresses the Will, so otherwise a retained online value can remain after a clean shutdown. A special MQTT 5 disconnect reason can request the Will, but the early helper uses the normal path.

Will publication occurs when its delay elapses or the session ends, whichever happens first. With Session Expiry 0, ending the session can remove the intended five-second grace period. With a preserved session, resuming that session before the delay ends can suppress the pending Will. Test with stable identity, Clean Start false, and a session expiry longer than the delay.

Separate experiments that look similar

Action	What it tests	Prediction
Graceful simulator shutdown	DISCONNECT and explicit offline policy	Normal disconnect does not itself publish the Will
Abruptly terminate the exact simulator binary	Lost client without DISCONNECT	Broker may publish Will after detection and applicable delay
Reconnect into preserved session before Will Delay	Transient outage	Pending Will can be suppressed
Stop broker	Broker availability	A stopped broker cannot publish Wills while it is down
Keep MQTT traffic alive but stall heartbeat production	Application health	Keep Alive can succeed while freshness fails

For an abrupt test, build a dedicated lab binary with go build -o /tmp/mqtt5-lab-simulator ./cmd/simulator, run it with your usual simulator flags, and record its PID. Send kill -KILL <recorded-lab-PID> from another terminal only after verifying that PID. KILL prevents Go cleanup and DISCONNECT; it is different from Ctrl+C. Do not target all Go processes or assume a go run wrapper is the actual client. Restart the binary afterward and inspect both retained presence and fresh heartbeats.

A killed process usually closes its socket promptly. It does not reproduce a silent network black hole that must be detected through Keep Alive. Record the actual fault injected and avoid claiming the measured delay is a universal offline-detection time.

4. Explain the result before moving on

A fresh subscriber receives retained online, but no heartbeat follows. Is the machine healthy?

You only know the broker stored an online assertion. The producer may have exited cleanly without replacing it, or the retained value may be old. Combine presence with receipt time and heartbeat freshness; use connection/revision identity to reject stale evidence.

Done when: demonstrate an expired offline command, retained replacement and deletion, graceful versus abrupt presence behavior, and rejection of an older desired revision. Record predictions and observations in journal/10-expiry-presence.md.

Sources: MQTT 5 packet and session rules, Mosquitto publication flags, and local references/paho.golang-v0.23.0/paho/cp_publish.go, paho/cp_connect.go, and autopaho/auto.go.

Previous: Lesson 9. Next: Lesson 11.