MQTT Message Expiry
Bounds how long a PUBLISH remains eligible for onward delivery. The broker may discard expired queued work.
Go MQTT Vending Backend Β· Lesson 10 Β· 12 minutes
Use separate clocks for publication lifetime, application deadlines, sessions, and freshness. Keep desired state distinct from one-shot commands.
β Start with the system story
A vending machine has been offline for an hour. A queued βapply price updateβ arrives after it reconnects, but the price is obsolete. Which timer prevents stale delivery, and which check prevents late execution after delivery?
Answer before reading: broker policy and device policy are separate checks.
Bounds how long a PUBLISH remains eligible for onward delivery. The broker may discard expired queued work.
An absolute deadline in the command contract lets the device reject work that is already delivered but too old to execute.
Controls how long the broker may preserve the client's session after disconnect.
Backend receive time measures how recently an application report arrived. It is not a broker connection probe or health diagnosis.
Keep Alive detects a lack of MQTT control traffic; it does not show that the application is producing heartbeats. A publish context timeout ends your wait, not necessarily the remote operation. Will Delay delays an eligible Will; it does not certify why the device disappeared.
A retained publication stores the latest value for one topic and can initialize a new subscriber. It is useful for replaceable desired configuration, such as catalog revision 9. A retained βdispenseβ or βrun calibration onceβ action can replay to a later subscriber, so one-shot commands should be nonretained.
For desired state, keep the highest revision at the simulator and reject revision 8 after revision 9. The device reports the revision it applied. A PUBACK only confirms the relevant QoS hop; it does not prove application. An empty retained publication removes the stored value for that topic; it does not retract copies already delivered.
internal/contracts/topic.goFile overview: This is the single source of truth for concrete publish names, controlled subscription filters, and inbound topic parsing. The added kinds support presence and desired/reported state.
DeviceTopic / DeviceFilter summary: Build validated exact names and fleet filters. ParseTopic summary: Derive the device and kind from the shared prefix and reject unknown routes.
package contracts
import (
"fmt"
"strings"
)
type TopicName string
type TopicFilter string
type Kind string
const (
KindTelemetry Kind = "telemetry"
KindCommands Kind = "commands"
KindResponses Kind = "responses"
KindSales Kind = "sales"
KindPresence Kind = "presence"
KindDesired Kind = "desired"
KindReported Kind = "reported"
)
const topicPrefix = "lab/v1/devices/"
type Topic struct {
DeviceID string
Kind Kind
}
func (t TopicName) String() string { return string(t) }
func (f TopicFilter) String() string { return string(f) }
func DeviceTopic(deviceID string, kind Kind) (TopicName, error) {
if !validID(deviceID) {
return "", fmt.Errorf("invalid device ID %q", deviceID)
}
if !supportedKind(kind) {
return "", fmt.Errorf("unsupported topic kind %q", kind)
}
return TopicName(topicPrefix + deviceID + "/" + string(kind)), nil
}
func DeviceFilter(kind Kind) (TopicFilter, error) {
if !supportedKind(kind) {
return "", fmt.Errorf("unsupported topic kind %q", kind)
}
return TopicFilter(topicPrefix + "+/" + string(kind)), nil
}
func DeviceSubscription(deviceID string, kind Kind) (TopicFilter, error) {
topic, err := DeviceTopic(deviceID, kind)
if err != nil {
return "", err
}
return TopicFilter(topic.String()), nil
}
func ParseTopic(value string) (Topic, error) {
if !strings.HasPrefix(value, topicPrefix) {
return Topic{}, fmt.Errorf("topic must be %s{deviceID}/{kind}", topicPrefix)
}
parts := strings.Split(strings.TrimPrefix(value, topicPrefix), "/")
if len(parts) != 2 || !validID(parts[0]) {
return Topic{}, fmt.Errorf("topic must be %s{deviceID}/{kind}", topicPrefix)
}
kind := Kind(parts[1])
if !supportedKind(kind) {
return Topic{}, fmt.Errorf("unsupported topic kind %q", kind)
}
return Topic{DeviceID: parts[0], Kind: kind}, nil
}
func supportedKind(kind Kind) bool {
switch kind {
case KindTelemetry, KindCommands, KindResponses, KindSales,
KindPresence, KindDesired, KindReported:
return true
default:
return false
}
}
func validID(value string) bool {
if value == "" || len(value) > 128 {
return false
}
for _, r := range value {
if !(r >= 'a' && r <= 'z' || r >= 'A' && r <= 'Z' ||
r >= '0' && r <= '9' || strings.ContainsRune("._-", r)) {
return false
}
}
return true
}
File overview: Create internal/mqtt/expiry.go to build a QoS 1 publication with explicit expiry, and internal/deviceview/deadline.go to reject stale commands at execution time. Each file is complete and has its own imports, types, and errors.
internal/mqtt/expiry.go β complete filepackage mqtt
import (
"fmt"
"example.com/mqtt5-learning/internal/contracts"
"github.com/eclipse/paho.golang/paho"
)
// ExpiringPublish builds a publication whose broker-queued lifetime is bounded.
// The device must still check its application deadline before executing it.
func ExpiringPublish(topic contracts.TopicName, payload []byte, expirySeconds uint32) (*paho.Publish, error) {
if topic == "" {
return nil, fmt.Errorf("topic is required")
}
if expirySeconds == 0 {
return nil, fmt.Errorf("expirySeconds must be greater than zero")
}
seconds := expirySeconds
return &paho.Publish{
Topic: topic.String(), Payload: append([]byte(nil), payload...), QoS: 1,
Properties: &paho.PublishProperties{
ContentType: "application/json",
MessageExpiry: &seconds,
},
}, nil
}
internal/deviceview/deadline.go β complete filepackage deviceview
import (
"fmt"
"time"
)
// CheckCommandDeadline rejects a command before the simulator starts its effect.
// The absolute deadline protects against time spent in local queues after delivery.
func CheckCommandDeadline(deadline string, now time.Time) error {
if deadline == "" {
return fmt.Errorf("command deadline is required")
}
when, err := time.Parse(time.RFC3339, deadline)
if err != nil {
return fmt.Errorf("parse command deadline: %w", err)
}
if !now.Before(when) {
return fmt.Errorf("command deadline has passed")
}
return nil
}
File overview: Create internal/deviceview/revisions.go to keep a per-device revision and prevent older desired state from replacing newer applied state.
Apply summary: Lock, compare the incoming revision, and update only when it is newer. Return an explicit outcome so duplicate and stale deliveries are observable.
internal/deviceview/revisions.go β complete filepackage deviceview
import "sync"
type RevisionOutcome uint8
const (
RevisionApplied RevisionOutcome = iota + 1
RevisionDuplicate
RevisionStale
)
type RevisionStore struct {
mu sync.Mutex
current map[string]uint64
}
func NewRevisionStore() *RevisionStore {
return &RevisionStore{current: make(map[string]uint64)}
}
func (s *RevisionStore) Apply(deviceID string, revision uint64) RevisionOutcome {
s.mu.Lock()
defer s.mu.Unlock()
current, exists := s.current[deviceID]
if exists && revision == current {
return RevisionDuplicate
}
if exists && revision < current {
return RevisionStale
}
s.current[deviceID] = revision
return RevisionApplied
}
func (s *RevisionStore) Current(deviceID string) (uint64, bool) {
s.mu.Lock()
defer s.mu.Unlock()
revision, ok := s.current[deviceID]
return revision, ok
}
internal/mqtt/presence.goCreate this complete helper file for the Last Will portion. OfflineWill builds the message prepared during CONNECT; OnlinePresence builds the retained publication sent after a successful connection. Both use one validated presence topic and stable device/boot identity.
OfflineWill β short summaryReturns a retained QoS 1 offline Will with a five-second Will Delay property.
OnlinePresence β short summaryCreates the retained online publication for the same device and boot identity.
internal/mqtt/presence.go β complete filepackage mqtt
import (
"encoding/json"
"fmt"
"time"
"example.com/mqtt5-learning/internal/contracts"
"github.com/eclipse/paho.golang/paho"
)
type Presence struct {
SchemaVersion int `json:"schema_version"`
MessageID string `json:"message_id"`
DeviceID string `json:"device_id"`
BootID string `json:"boot_id"`
Status string `json:"status"`
OccurredAt string `json:"occurred_at"`
}
func OfflineWill(deviceID, bootID string) (*paho.WillMessage, *paho.WillProperties, error) {
if deviceID == "" || bootID == "" {
return nil, nil, fmt.Errorf("device ID and boot ID are required")
}
topic, err := contracts.DeviceTopic(deviceID, contracts.KindPresence)
if err != nil {
return nil, nil, fmt.Errorf("build presence topic: %w", err)
}
body, err := json.Marshal(Presence{
SchemaVersion: contracts.SchemaVersionOne,
MessageID: "will-" + bootID, DeviceID: deviceID, BootID: bootID,
Status: "offline", OccurredAt: time.Now().UTC().Format(time.RFC3339),
})
if err != nil { return nil, nil, fmt.Errorf("encode offline Will: %w", err) }
delay := uint32(5)
return &paho.WillMessage{Topic: topic.String(), QoS: 1, Retain: true, Payload: body},
&paho.WillProperties{WillDelayInterval: &delay, ContentType: "application/json"}, nil
}
func OnlinePresence(deviceID, bootID string, now time.Time) (*paho.Publish, error) {
if deviceID == "" || bootID == "" {
return nil, fmt.Errorf("device ID and boot ID are required")
}
topic, err := contracts.DeviceTopic(deviceID, contracts.KindPresence)
if err != nil { return nil, fmt.Errorf("build presence topic: %w", err) }
body, err := json.Marshal(Presence{
SchemaVersion: contracts.SchemaVersionOne,
MessageID: "online-" + bootID, DeviceID: deviceID, BootID: bootID,
Status: "online", OccurredAt: now.UTC().Format(time.RFC3339),
})
if err != nil { return nil, fmt.Errorf("encode online presence: %w", err) }
return &paho.Publish{
Topic: topic.String(), Payload: body, QoS: 1, Retain: true,
Properties: &paho.PublishProperties{ContentType: "application/json"},
}, nil
}
The Will body is prepared at CONNECT, so OccurredAt above is the preparation time. The backend should record its own receipt time and use it when evaluating freshness. Include a boot ID so stale presence from a previous process can be distinguished. The simulator's planned shutdown must publish retained offline presence before sending normal DISCONNECT if the application needs that final state; graceful DISCONNECT suppresses the Will.
Retained βonlineβ is the broker's latest stored assertion. A Last Will is published by the broker after an eligible unexpected loss; it is not sent by the dead process. A clean DISCONNECT normally suppresses the Will. Combine connection events with backend-received heartbeat freshness and device-reported health rather than treating one as the other.
What could go wrong? A stale retained online value can outlive a graceful shutdown; Will Delay may interact with session expiry; a command delivered just before its expiry may still sit in a local queue; clocks can disagree. Record backend receive time and define the device deadline policy.
ExpiringPublish creates protocol metadata only; wire it into the existing publisher and handle the returned PUBACK as transport evidence. CheckCommandDeadline must run immediately before the simulator effect, after local queue wait. RevisionStore is intentionally RAM-only, so a simulator restart loses its revision; PostgreSQL or device flash is needed for durable state. The will is a broker publication prepared during CONNECT; normal DISCONNECT suppresses it. These helpers do not replace ACL updates, topic/body validation, or device response reporting.
Demonstrate expired queued work, device-side stale rejection, retained replacement/deletion, stale revision rejection, and the difference between graceful and unexpected disconnect evidence.
π Detailed experiments: Section 10 source lesson. Primary references: OASIS MQTT 5.0 Standard, Paho v0.23.0 API, and Mosquitto publish client manual.
Ask me to trace an expiry or presence result if the packet evidence differs from your prediction.