mqtt-5-learning/teaching/course-source/05a-backend-message-processing.md

# 🧭 Section 5A — Turn a valid message into backend state

## 🎬 Start with the system story

> **❓ Question:** **The backend received a valid heartbeat. What changes in the application, and how can another part of the backend ask whether the device was recently heard from?**

Section 5 proves that a packet has a valid topic and payload. That is a trust boundary, not a business action. A backend still needs an application service that decides what the valid message means, a place to record the result, and a safe way to report the outcome.

```mermaid
flowchart LR
    B[Mosquitto delivers PUBLISH] --> R[Receive loop]
    R --> V[Parse topic and validate contract]
    V -->|valid heartbeat| S[Application service]
    S --> M[In-memory device view]
    S --> L[Structured application log]
    M --> Q[Status query: recent heartbeat?]
    V -->|invalid| E[Log bounded rejection; no state change]
```

The exercise adds one small vertical slice: a valid heartbeat updates the backend's last-seen view. It uses an in-memory store so the data flow is visible before concurrency and PostgreSQL arrive. A fresh heartbeat is evidence of recent application traffic; it is not proof that the MQTT connection is currently up or that every part of the vending machine is healthy.

**Before coding, predict:** Which facts come from the broker, which come from validation, and which are created by the application? A topic and schema-valid body are not yet a stored status, and a `PUBACK` is not evidence that this store was updated.

Prerequisite: [Section 4](04-publish-subscribe-simulator.md) introduced the complete `internal/contracts/topic.go` hierarchy: `DeviceTopic` builds concrete names, `DeviceFilter` builds fleet filters, `DeviceSubscription` builds exact-device filters, and `ParseTopic` extracts the route. [Section 5](05-contracts-validation.md) adds `contracts.Decode` and `contracts.Heartbeat`. The code below assumes the Paho message-copying receive helper and `mqtt.Message` from Section 4; topics must use the shared contract rather than hand-built strings.

## 🧩 1. Separate validation from application meaning

> **❓ Question:** **Why should the callback not also decide what “online” means and update shared device state?**

The callback owns packet delivery. The application service owns the meaning of a validated message. Keeping those responsibilities separate lets the backend test status behavior without a broker and later move the service behind Section 6's worker pool.

Trace one heartbeat:

```text
PUBLISH → callback copies topic and bytes → inbox
        → service parses topic → Decode validates contract
        → store updates last-seen snapshot → structured log records the outcome
```

The service gives each existing contract an honest current behavior: heartbeats update a last-seen view, while sales reports and command results are validated and logged but not yet persisted or correlated. Section 7 adds command matching; Section 13 adds durable sales storage; Section 12 adds bounded device-log and richer health workflows. This boundary prevents a successful decode from being presented as completed business work.

## 🗃️ 2. Create a small in-memory device view

> **❓ Question:** **Which timestamp should control freshness: when the device says it observed the heartbeat, or when this backend received it?**

Store both. Use the device timestamp to avoid replacing a newer report with an older delayed one. Use the backend receive timestamp to answer whether a heartbeat arrived recently; the device clock does not decide when this backend last heard from it.

Save this complete file as `internal/deviceview/store.go`.

### 🔍 `store.go` — file overview

This file stores the newest accepted heartbeat per device and derives a freshness view. The map is protected because Section 6 will call the application service from multiple workers. The store is still volatile: a process restart clears every snapshot.

```go
package deviceview

import (
	"fmt"
	"sync"
	"time"

	"example.com/mqtt5-learning/internal/contracts"
)

type Snapshot struct {
	DeviceID    string
	MessageID   string
	ReportedAt  time.Time
	ReceivedAt  time.Time
	Temperature    float64
	TemperatureSet bool
}

type Status struct {
	Snapshot       Snapshot
	HeartbeatFresh bool
}

type Store struct {
	mu     sync.RWMutex
	latest map[string]Snapshot
}

func NewStore() *Store {
	return &Store{latest: make(map[string]Snapshot)}
}

func (s *Store) RecordHeartbeat(message contracts.Heartbeat, receivedAt time.Time) (bool, error) {
	reportedAt, err := time.Parse(time.RFC3339, message.OccurredAt)
	if err != nil {
		// Decode already validates this timestamp. Keep the store defensive too.
		return false, fmt.Errorf("parse heartbeat timestamp: %w", err)
	}

	snapshot := Snapshot{
		DeviceID:   message.DeviceID,
		MessageID:  message.MessageID,
		ReportedAt: reportedAt,
		ReceivedAt: receivedAt,
	}
	if message.Temperature != nil {
		snapshot.Temperature = *message.Temperature
		snapshot.TemperatureSet = true
	}

	s.mu.Lock()
	defer s.mu.Unlock()
	previous, exists := s.latest[message.DeviceID]
	if exists && !reportedAt.After(previous.ReportedAt) {
		return false, nil
	}
	s.latest[message.DeviceID] = snapshot
	return true, nil
}

func (s *Store) GetStatus(deviceID string, now time.Time, freshFor time.Duration) (Status, bool) {
	s.mu.RLock()
	snapshot, exists := s.latest[deviceID]
	s.mu.RUnlock()
	if !exists {
		return Status{}, false
	}

	fresh := freshFor > 0 && !now.Before(snapshot.ReceivedAt) && now.Sub(snapshot.ReceivedAt) <= freshFor
	return Status{Snapshot: snapshot, HeartbeatFresh: fresh}, true
}
```

#### 🏗️ `NewStore` — initialize the state owner

**Summary:** Creates the map that holds one latest snapshot per device.

**Details:** The map starts empty, so a status query before the first valid heartbeat returns `exists == false`. The store does not invent an offline record for a device the backend has never seen.

#### 📥 `RecordHeartbeat` — apply a validated observation

**Summary:** Converts the contract timestamp, copies optional temperature data, and replaces the stored snapshot only when the report is newer.

**Details:** `Decode` has already checked the timestamp, but the store parses it again because its own method should not silently accept malformed values if another caller uses it later. The device's `OccurredAt` orders reports; `receivedAt` is captured by the backend and supports freshness queries. The mutex protects the map while concurrent workers update it. The temperature is stored by value, and `TemperatureSet` preserves the distinction between a missing reading and a reported zero. This also avoids sharing a mutable pointer with either the incoming message or a status caller.

If a duplicate or delayed older heartbeat arrives, the method returns `false` and leaves the newer state untouched. This is a small ordering rule, not durable deduplication: after restart the store has no history, and equal timestamps are treated as non-newer.

#### 🔎 `GetStatus` — query recent application evidence

**Summary:** Returns the latest snapshot and whether the backend received it within the caller's freshness window.

**Details:** `HeartbeatFresh` is computed from backend receipt time, not device time. It means “a valid heartbeat arrived recently,” not “the broker connection is up” or “the machine is operational.” A missing snapshot returns `false` as the second result. Since the snapshot contains only values, returning its map copy does not expose mutable store-owned data.

## 🧠 3. Add the application service

> **❓ Question:** **Where do topic parsing, payload validation, state updates, and operational logging meet?**

The service coordinates those steps. It does not own a Paho client or publish MQTT packets; it takes a transport-neutral message value and applies one use case. Save this complete file as `internal/deviceview/service.go`.

### 🔍 `service.go` — file overview

This file turns an incoming message into an application decision. Heartbeats update the device view; sales reports and command results are validated and logged until their later persistence/correlation lessons. Invalid topics or payloads return errors before the store changes. Structured logs include identifiers and outcomes, not the raw payload.

```go
package deviceview

import (
	"context"
	"fmt"
	"log/slog"
	"time"

	"example.com/mqtt5-learning/internal/contracts"
	"example.com/mqtt5-learning/internal/mqtt"
)

type Service struct {
	store    *Store
	logger   *slog.Logger
	freshFor time.Duration
	now      func() time.Time
}

func NewService(store *Store, logger *slog.Logger, freshFor time.Duration) (*Service, error) {
	if store == nil {
		return nil, fmt.Errorf("device-view store is required")
	}
	if logger == nil {
		return nil, fmt.Errorf("logger is required")
	}
	if freshFor <= 0 {
		return nil, fmt.Errorf("heartbeat freshness window must be positive")
	}
	return &Service{store: store, logger: logger, freshFor: freshFor, now: time.Now}, nil
}

func (s *Service) HandleMessage(ctx context.Context, message mqtt.Message) error {
	if err := ctx.Err(); err != nil {
		return err
	}
	topic, err := contracts.ParseTopic(message.Topic)
	if err != nil {
		return fmt.Errorf("parse topic: %w", err)
	}
	receivedAt := s.now().UTC()
	switch topic.Kind {
	case contracts.KindTelemetry:
		heartbeat, err := contracts.Decode[contracts.Heartbeat](message.Payload, topic, receivedAt)
		if err != nil {
			return fmt.Errorf("validate heartbeat: %w", err)
		}
		updated, err := s.store.RecordHeartbeat(heartbeat, receivedAt)
		if err != nil {
			return fmt.Errorf("record heartbeat: %w", err)
		}
		if !updated {
			s.logger.Info("older or duplicate heartbeat ignored",
				"device_id", heartbeat.DeviceID,
				"message_id", heartbeat.MessageID,
			)
			return nil
		}
		s.logger.Info("heartbeat recorded",
			"device_id", heartbeat.DeviceID,
			"message_id", heartbeat.MessageID,
			"received_at", receivedAt,
		)
		return nil
	case contracts.KindSales:
		report, err := contracts.Decode[contracts.SalesReport](message.Payload, topic, receivedAt)
		if err != nil {
			return fmt.Errorf("validate sales report: %w", err)
		}
		// Validation is not durable admission; Section 13 stores reports.
		s.logger.Info("sales report validated; persistence not implemented",
			"device_id", report.DeviceID,
			"report_id", report.ReportID,
			"item_count", len(report.Items),
		)
		return nil
	case contracts.KindResponses:
		result, err := contracts.Decode[contracts.CommandResult](message.Payload, topic, receivedAt)
		if err != nil {
			return fmt.Errorf("validate command result: %w", err)
		}
		// Section 7 adds pending-command and correlation matching.
		s.logger.Info("command result validated; matching not implemented",
			"device_id", result.DeviceID,
			"command_id", result.CommandID,
			"status", result.Status,
		)
		return nil
	default:
		return fmt.Errorf("device-view service does not handle topic kind %q", topic.Kind)
	}
}

func (s *Service) Status(deviceID string) (Status, bool) {
	return s.store.GetStatus(deviceID, s.now().UTC(), s.freshFor)
}
```

#### 🧰 `NewService` — check application dependencies

**Summary:** Builds a service only when its store, logger, and freshness policy are configured.

**Details:** These checks fail at startup instead of allowing a nil pointer panic when the first device message arrives. `freshFor` is an application policy; it is not MQTT Keep Alive or Session Expiry. Choose a value that accounts for the simulator's heartbeat interval and expected delay.

#### 🔄 `HandleMessage` — validate, route, apply, and record the result

**Summary:** Checks cancellation, parses the topic, then validates and routes each supported contract to its current application behavior.

**Details:** For telemetry, each early return happens before the store update, so malformed messages cannot alter status. The service captures one backend receipt time and reuses it for validation, storage, and logging. The store's `false` result represents an older or duplicate report; it is a handled condition, not a transport failure. For sales and responses, successful decoding is logged as validation only: no sale is saved and no command is matched yet. The logger records stable identifiers but avoids printing untrusted payload contents.

This is an application outcome. The MQTT receive callback may already have returned and automatic acknowledgement may already have occurred. Returning an error here does not reverse a `PUBACK`; Section 6 makes that boundary measurable under worker load.

#### 🔎 `Status` — expose the current device view to callers

**Summary:** Queries the store and evaluates heartbeat freshness using the service's clock and configured window.

**Details:** The method returns the latest snapshot only if one exists. It does not issue an MQTT request to the device. A future API can expose this method; Section 8 builds HTTP endpoints, while Sections 10 and 12 add richer presence and vending-health semantics.

## 🔌 4. Connect the service to the receive loop

> **❓ Question:** **What changes when validated messages now update application state instead of only producing a “valid message” log line?**

Create one store and service during process startup, then call the service for telemetry messages from the existing backend message switch:

Add these imports to `cmd/backend/main.go`:

```go
"log/slog"
"time"

"example.com/mqtt5-learning/internal/deviceview"
```

```go
// In run(), after config.Load and before the receive loop starts:
store := deviceview.NewStore()
deviceView, err := deviceview.NewService(store, slog.Default(), 90*time.Second)
if err != nil {
	return fmt.Errorf("create device-view service: %w", err)
}

// Replace the old validation-only handler with this service call.
func handleMessage(ctx context.Context, deviceView *deviceview.Service, message mqtt.Message) error {
	return deviceView.HandleMessage(ctx, message)
}
```

#### 🔌 `run` and `handleMessage` — connect startup to application work

**Summary:** `run` constructs one shared store/service at startup; `handleMessage` delegates each received message to that service.

**Details:** Create the service after configuration loads and before the receive loop starts. Replace the validation-only handler with the shown `handleMessage` function and update the receive loop to pass its context and shared service. The service updates heartbeat state and validates/logs sales and responses for now; Section 7 adds response matching and Section 13 adds sales persistence.

Use this receive-loop call and function signature to make the required data flow explicit:

```go
case message := <-inbox:
	if err := handleMessage(ctx, deviceView, message); err != nil {
		slog.Error("message handling failed", "topic", message.Topic, "error", err)
	}
```

```go
func handleMessage(ctx context.Context, deviceView *deviceview.Service, message mqtt.Message) error {
	return deviceView.HandleMessage(ctx, message)
}
```

### 🧪 Observe the application result

1. Start the lab backend and simulator. Confirm structured `heartbeat recorded` events include the device and message IDs but no raw JSON payload.
2. Query `deviceView.Status("device-001")` from a temporary diagnostic call. After a heartbeat, expect a snapshot and `HeartbeatFresh == true`:

   ```go
   status, exists := deviceView.Status("device-001")
   if exists {
   	slog.Info("device view", "last_seen", status.Snapshot.ReceivedAt, "heartbeat_fresh", status.HeartbeatFresh)
   }
   ```
3. Stop the simulator, wait longer than the 90-second example window, and query again. The snapshot remains, but freshness becomes false. No MQTT packet declared the device offline; the backend inferred staleness from elapsed time.
4. Deliver an older `occurred_at` after a newer heartbeat. The service logs that it ignored the older observation, and the stored snapshot does not move backward.

5. Change the topic/body device IDs or corrupt the payload. Validation returns an error and the stored snapshot remains unchanged.

Do not confuse backend logs with device diagnostic logs. These `slog` entries explain what the backend did. A device log stream needs its own contract, size/frequency limits, and storage policy; the later vending-workflow lesson covers that. Likewise, a fresh heartbeat proves recent application traffic, not machine health or a live MQTT socket.

### ✅ Result — what the first five lessons were missing

You now have the missing application step between “valid packet” and “concurrent worker”: a service applies a specific use case to a validated message, a store exposes the resulting view, and a structured log records the decision. The status is deliberately simple and volatile. Section 6 adds bounded concurrent handling; later lessons add durable records, richer health/log behavior, and real vending workflows.

**Primary references:** [MQTT 5.0 OASIS Standard](https://docs.oasis-open.org/mqtt/mqtt/v5.0/mqtt-v5.0.html) for the distinction between protocol delivery and application payload handling; [Go `log/slog`](https://pkg.go.dev/log/slog) for structured logging; [Paho Go v0.23.0 API](https://pkg.go.dev/github.com/eclipse/paho.golang/paho@v0.23.0) for the transport API used by the surrounding lesson.

Previous: [05-contracts-validation.md](05-contracts-validation.md). Next: [06-concurrency-cancellation.md](06-concurrency-cancellation.md).