Go MQTT Vending Backend Β· Lesson 10 Β· 12 minutes

Give stale work an expiry; treat presence as evidence

Use separate clocks for publication lifetime, application deadlines, sessions, and freshness. Keep desired state distinct from one-shot commands.

🎯 One skill: assign timer ownership🧠 Predict stale deliveryπŸ§ͺ Simulator + broker

❓ Start with the system story

A vending machine has been offline for an hour. A queued β€œapply price update” arrives after it reconnects, but the price is obsolete. Which timer prevents stale delivery, and which check prevents late execution after delivery?

Answer before reading: broker policy and device policy are separate checks.

⏱️ One timer, one responsibility

MQTT Message Expiry

Bounds how long a PUBLISH remains eligible for onward delivery. The broker may discard expired queued work.

Application deadline

An absolute deadline in the command contract lets the device reject work that is already delivered but too old to execute.

Session Expiry

Controls how long the broker may preserve the client's session after disconnect.

Heartbeat freshness

Backend receive time measures how recently an application report arrived. It is not a broker connection probe or health diagnosis.

Keep Alive detects a lack of MQTT control traffic; it does not show that the application is producing heartbeats. A publish context timeout ends your wait, not necessarily the remote operation. Will Delay delays an eligible Will; it does not certify why the device disappeared.

🧠 Check the boundary: What combination protects against a command that waits in the broker and then spends time in a local queue?




πŸ“Œ Retained desired state is not an event queue

A retained publication stores the latest value for one topic and can initialize a new subscriber. It is useful for replaceable desired configuration, such as catalog revision 9. A retained β€œdispense” or β€œrun calibration once” action can replay to a later subscriber, so one-shot commands should be nonretained.

For desired state, keep the highest revision at the simulator and reject revision 8 after revision 9. The device reports the revision it applied. A PUBACK only confirms the relevant QoS hop; it does not prove application. An empty retained publication removes the stored value for that topic; it does not retract copies already delivered.

πŸ“„ Complete file: internal/contracts/topic.go

File overview: This is the single source of truth for concrete publish names, controlled subscription filters, and inbound topic parsing. The added kinds support presence and desired/reported state.

DeviceTopic / DeviceFilter summary: Build validated exact names and fleet filters. ParseTopic summary: Derive the device and kind from the shared prefix and reject unknown routes.

🧩 Full replacement source
package contracts

import (
	"fmt"
	"strings"
)

type TopicName string
type TopicFilter string
type Kind string

const (
	KindTelemetry Kind = "telemetry"
	KindCommands  Kind = "commands"
	KindResponses Kind = "responses"
	KindSales     Kind = "sales"
	KindPresence  Kind = "presence"
	KindDesired   Kind = "desired"
	KindReported  Kind = "reported"
)

const topicPrefix = "lab/v1/devices/"

type Topic struct {
	DeviceID string
	Kind     Kind
}

func (t TopicName) String() string   { return string(t) }
func (f TopicFilter) String() string { return string(f) }

func DeviceTopic(deviceID string, kind Kind) (TopicName, error) {
	if !validID(deviceID) {
		return "", fmt.Errorf("invalid device ID %q", deviceID)
	}
	if !supportedKind(kind) {
		return "", fmt.Errorf("unsupported topic kind %q", kind)
	}
	return TopicName(topicPrefix + deviceID + "/" + string(kind)), nil
}

func DeviceFilter(kind Kind) (TopicFilter, error) {
	if !supportedKind(kind) {
		return "", fmt.Errorf("unsupported topic kind %q", kind)
	}
	return TopicFilter(topicPrefix + "+/" + string(kind)), nil
}

func DeviceSubscription(deviceID string, kind Kind) (TopicFilter, error) {
	topic, err := DeviceTopic(deviceID, kind)
	if err != nil {
		return "", err
	}
	return TopicFilter(topic.String()), nil
}

func ParseTopic(value string) (Topic, error) {
	if !strings.HasPrefix(value, topicPrefix) {
		return Topic{}, fmt.Errorf("topic must be %s{deviceID}/{kind}", topicPrefix)
	}
	parts := strings.Split(strings.TrimPrefix(value, topicPrefix), "/")
	if len(parts) != 2 || !validID(parts[0]) {
		return Topic{}, fmt.Errorf("topic must be %s{deviceID}/{kind}", topicPrefix)
	}
	kind := Kind(parts[1])
	if !supportedKind(kind) {
		return Topic{}, fmt.Errorf("unsupported topic kind %q", kind)
	}
	return Topic{DeviceID: parts[0], Kind: kind}, nil
}

func supportedKind(kind Kind) bool {
	switch kind {
	case KindTelemetry, KindCommands, KindResponses, KindSales,
		KindPresence, KindDesired, KindReported:
		return true
	default:
		return false
	}
}

func validID(value string) bool {
	if value == "" || len(value) > 128 {
		return false
	}
	for _, r := range value {
		if !(r >= 'a' && r <= 'z' || r >= 'A' && r <= 'Z' ||
			r >= '0' && r <= '9' || strings.ContainsRune("._-", r)) {
			return false
		}
	}
	return true
}

🧩 Complete file: expiry and command deadline policy

File overview: Create internal/mqtt/expiry.go to build a QoS 1 publication with explicit expiry, and internal/deviceview/deadline.go to reject stale commands at execution time. Each file is complete and has its own imports, types, and errors.

πŸ“„ internal/mqtt/expiry.go β€” complete file
package mqtt

import (
	"fmt"

	"example.com/mqtt5-learning/internal/contracts"
	"github.com/eclipse/paho.golang/paho"
)

// ExpiringPublish builds a publication whose broker-queued lifetime is bounded.
// The device must still check its application deadline before executing it.
func ExpiringPublish(topic contracts.TopicName, payload []byte, expirySeconds uint32) (*paho.Publish, error) {
	if topic == "" {
		return nil, fmt.Errorf("topic is required")
	}
	if expirySeconds == 0 {
		return nil, fmt.Errorf("expirySeconds must be greater than zero")
	}
	seconds := expirySeconds
	return &paho.Publish{
		Topic: topic.String(), Payload: append([]byte(nil), payload...), QoS: 1,
		Properties: &paho.PublishProperties{
			ContentType: "application/json",
			MessageExpiry: &seconds,
		},
	}, nil
}
πŸ“„ internal/deviceview/deadline.go β€” complete file
package deviceview

import (
	"fmt"
	"time"
)

// CheckCommandDeadline rejects a command before the simulator starts its effect.
// The absolute deadline protects against time spent in local queues after delivery.
func CheckCommandDeadline(deadline string, now time.Time) error {
	if deadline == "" {
		return fmt.Errorf("command deadline is required")
	}
	when, err := time.Parse(time.RFC3339, deadline)
	if err != nil {
		return fmt.Errorf("parse command deadline: %w", err)
	}
	if !now.Before(when) {
		return fmt.Errorf("command deadline has passed")
	}
	return nil
}

🧩 Complete file: simulator's monotonic desired state

File overview: Create internal/deviceview/revisions.go to keep a per-device revision and prevent older desired state from replacing newer applied state.

Apply summary: Lock, compare the incoming revision, and update only when it is newer. Return an explicit outcome so duplicate and stale deliveries are observable.

πŸ“„ internal/deviceview/revisions.go β€” complete file
package deviceview

import "sync"

type RevisionOutcome uint8

const (
	RevisionApplied RevisionOutcome = iota + 1
	RevisionDuplicate
	RevisionStale
)

type RevisionStore struct {
	mu      sync.Mutex
	current map[string]uint64
}

func NewRevisionStore() *RevisionStore {
	return &RevisionStore{current: make(map[string]uint64)}
}

func (s *RevisionStore) Apply(deviceID string, revision uint64) RevisionOutcome {
	s.mu.Lock()
	defer s.mu.Unlock()
	current, exists := s.current[deviceID]
	if exists && revision == current {
		return RevisionDuplicate
	}
	if exists && revision < current {
		return RevisionStale
	}
	s.current[deviceID] = revision
	return RevisionApplied
}

func (s *RevisionStore) Current(deviceID string) (uint64, bool) {
	s.mu.Lock()
	defer s.mu.Unlock()
	revision, ok := s.current[deviceID]
	return revision, ok
}

🧭 File overview: internal/mqtt/presence.go

Create this complete helper file for the Last Will portion. OfflineWill builds the message prepared during CONNECT; OnlinePresence builds the retained publication sent after a successful connection. Both use one validated presence topic and stable device/boot identity.

πŸ“΄ OfflineWill β€” short summary

Returns a retained QoS 1 offline Will with a five-second Will Delay property.

🟒 OnlinePresence β€” short summary

Creates the retained online publication for the same device and boot identity.

πŸ“„ internal/mqtt/presence.go β€” complete file
package mqtt

import (
	"encoding/json"
	"fmt"
	"time"

	"example.com/mqtt5-learning/internal/contracts"
	"github.com/eclipse/paho.golang/paho"
)

type Presence struct {
	SchemaVersion int    `json:"schema_version"`
	MessageID     string `json:"message_id"`
	DeviceID      string `json:"device_id"`
	BootID        string `json:"boot_id"`
	Status        string `json:"status"`
	OccurredAt    string `json:"occurred_at"`
}

func OfflineWill(deviceID, bootID string) (*paho.WillMessage, *paho.WillProperties, error) {
	if deviceID == "" || bootID == "" {
		return nil, nil, fmt.Errorf("device ID and boot ID are required")
	}
	topic, err := contracts.DeviceTopic(deviceID, contracts.KindPresence)
	if err != nil {
		return nil, nil, fmt.Errorf("build presence topic: %w", err)
	}
	body, err := json.Marshal(Presence{
		SchemaVersion: contracts.SchemaVersionOne,
		MessageID: "will-" + bootID, DeviceID: deviceID, BootID: bootID,
		Status: "offline", OccurredAt: time.Now().UTC().Format(time.RFC3339),
	})
	if err != nil { return nil, nil, fmt.Errorf("encode offline Will: %w", err) }
	delay := uint32(5)
	return &paho.WillMessage{Topic: topic.String(), QoS: 1, Retain: true, Payload: body},
		&paho.WillProperties{WillDelayInterval: &delay, ContentType: "application/json"}, nil
}

func OnlinePresence(deviceID, bootID string, now time.Time) (*paho.Publish, error) {
	if deviceID == "" || bootID == "" {
		return nil, fmt.Errorf("device ID and boot ID are required")
	}
	topic, err := contracts.DeviceTopic(deviceID, contracts.KindPresence)
	if err != nil { return nil, fmt.Errorf("build presence topic: %w", err) }
	body, err := json.Marshal(Presence{
		SchemaVersion: contracts.SchemaVersionOne,
		MessageID: "online-" + bootID, DeviceID: deviceID, BootID: bootID,
		Status: "online", OccurredAt: now.UTC().Format(time.RFC3339),
	})
	if err != nil { return nil, fmt.Errorf("encode online presence: %w", err) }
	return &paho.Publish{
		Topic: topic.String(), Payload: body, QoS: 1, Retain: true,
		Properties: &paho.PublishProperties{ContentType: "application/json"},
	}, nil
}

πŸ”Ž Details: a Will timestamp is not a failure timestamp

The Will body is prepared at CONNECT, so OccurredAt above is the preparation time. The backend should record its own receipt time and use it when evaluating freshness. Include a boot ID so stale presence from a previous process can be distinguished. The simulator's planned shutdown must publish retained offline presence before sending normal DISCONNECT if the application needs that final state; graceful DISCONNECT suppresses the Will.

πŸ§ͺ Predict, then observe

  1. Set a persistent simulator session, disconnect it, publish a QoS 1 command with a short Message Expiry Interval, then reconnect after expiry.
  2. Repeat with a longer expiry but an already-passed application deadline. The simulator should receive and reject it as stale.
  3. Publish desired revision 9, then revision 8; verify reported state remains at 9.
  4. Compare graceful shutdown, abrupt simulator termination, reconnect before Will Delay, and broker stop. Record the exact fault and evidence.
  5. Clear the retained value and subscribe fresh. Verify no retained replay.

❀️ Presence has limits

Retained β€œonline” is the broker's latest stored assertion. A Last Will is published by the broker after an eligible unexpected loss; it is not sent by the dead process. A clean DISCONNECT normally suppresses the Will. Combine connection events with backend-received heartbeat freshness and device-reported health rather than treating one as the other.

What could go wrong? A stale retained online value can outlive a graceful shutdown; Will Delay may interact with session expiry; a command delivered just before its expiry may still sit in a local queue; clocks can disagree. Record backend receive time and define the device deadline policy.

πŸ”Ž Details: ownership and limitations

ExpiringPublish creates protocol metadata only; wire it into the existing publisher and handle the returned PUBACK as transport evidence. CheckCommandDeadline must run immediately before the simulator effect, after local queue wait. RevisionStore is intentionally RAM-only, so a simulator restart loses its revision; PostgreSQL or device flash is needed for durable state. The will is a broker publication prepared during CONNECT; normal DISCONNECT suppresses it. These helpers do not replace ACL updates, topic/body validation, or device response reporting.

βœ… Done when

Demonstrate expired queued work, device-side stale rejection, retained replacement/deletion, stale revision rejection, and the difference between graceful and unexpected disconnect evidence.

πŸ“‹ Short self-check answers
  1. Broker expiry limits onward delivery; an application deadline limits whether the device starts the action.
  2. Retained messages represent latest state, not historical event delivery.
  3. Heartbeat freshness says a recent report arrived, not that the device is healthy or the broker currently has a connection.

πŸ“š Detailed experiments: Section 10 source lesson. Primary references: OASIS MQTT 5.0 Standard, Paho v0.23.0 API, and Mosquitto publish client manual.

Ask me to trace an expiry or presence result if the packet evidence differs from your prediction.