Go MQTT Vending Backend · Lesson 17 · 15 minutes

Diagnose the failing boundary with evidence

Make each command traceable across HTTP, Go processes, MQTT, broker, device simulator, and PostgreSQL.

🎯 One skill: turn symptoms into a diagnostic path🧠 Predict likely fault layer🧪 Repeatable failure matrix

❓ Start with the system story

An API returns 202, the backend records PUBACK, but no device result appears. What evidence do you inspect next, and what can you not conclude yet?

Trace the same command ID through each owner before labeling the device failed.

🔎 Put identifiers in logs, not metric labels

Use structured logs with stable command/report IDs, process identity, message kind, and safe error categories. Redact credentials and avoid arbitrary payload dumps. Put high-cardinality IDs in logs/traces where appropriate, not unrestricted metric labels that create a new time series per device or command.

Metrics worth watching

Reconnects, rejected messages, queue depth/capacity, duplicate outcomes, publish and response latency, timeouts, DB failures, outbox backlog/oldest age, and shutdown drain results.

Health endpoints

Liveness asks whether a process should be restarted. Readiness asks whether that process can serve its assigned role and dependencies. A connected MQTT client alone does not establish useful service readiness.

🧭 File overview: `internal/observability/health.go`

Create this complete standard-library file for the first operational instrumentation slice. It exposes liveness, dependency-aware readiness, and low-cardinality Prometheus text metrics. It does not label metrics with device or command IDs.

🩺 `NewHandler` and readiness setters — short summary

Create one per process role and update its dependency flags when DB/MQTT readiness changes.

📈 `ServeHTTP` — short summary

Serves bounded health endpoints and fixed metric names without exposing secrets or arbitrary IDs.

📄 Complete file contents

package observability

import (
	"fmt"
	"net/http"
	"sync/atomic"
)

type Handler struct {
	role string
	requiresMQTT bool
	databaseReady atomic.Bool
	mqttReady atomic.Bool
	rejected atomic.Uint64
	duplicates atomic.Uint64
	reconnects atomic.Uint64
	completed atomic.Uint64
}

func NewHandler(role string, requiresMQTT bool) (*Handler, error) {
	if role != "api" && role != "ingest" { return nil, fmt.Errorf("role must be api or ingest") }
	return &Handler{role: role, requiresMQTT: requiresMQTT}, nil
}

func (h *Handler) SetDatabaseReady(ready bool) { h.databaseReady.Store(ready) }
func (h *Handler) SetMQTTReady(ready bool) { h.mqttReady.Store(ready) }
func (h *Handler) RecordRejected() { h.rejected.Add(1) }
func (h *Handler) RecordDuplicate() { h.duplicates.Add(1) }
func (h *Handler) RecordReconnect() { h.reconnects.Add(1) }
func (h *Handler) RecordCompleted() { h.completed.Add(1) }

func (h *Handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {
	switch r.URL.Path {
	case "/livez":
		w.WriteHeader(http.StatusNoContent)
	case "/readyz":
		if !h.databaseReady.Load() || (h.requiresMQTT && !h.mqttReady.Load()) {
			http.Error(w, "required dependency is not ready", http.StatusServiceUnavailable)
			return
		}
		w.WriteHeader(http.StatusNoContent)
	case "/metrics":
		w.Header().Set("Content-Type", "text/plain; version=0.0.4; charset=utf-8")
		fmt.Fprintf(w, "mqtt_learning_messages_rejected_total{role=%q} %d\n", h.role, h.rejected.Load())
		fmt.Fprintf(w, "mqtt_learning_messages_duplicate_total{role=%q} %d\n", h.role, h.duplicates.Load())
		fmt.Fprintf(w, "mqtt_learning_reconnects_total{role=%q} %d\n", h.role, h.reconnects.Load())
		fmt.Fprintf(w, "mqtt_learning_work_completed_total{role=%q} %d\n", h.role, h.completed.Load())
	default:
		http.NotFound(w, r)
	}
}

🔎 Details: operational meaning

Liveness stays independent of database/MQTT so a dependency outage does not trigger a restart loop. Readiness requires PostgreSQL for both processes and MQTT subscriptions for ingestion. The metrics are counters, not full histogram/queue instrumentation; add queue gauges and latency histograms when the service has those measurements. Use process role as a bounded label and keep IDs in structured logs instead.

🧪 Make a repeatable load and failure scenario

Start with a synthetic simulator load that has a fixed duration, device count, message rate, and payload mix. Measure throughput, latency percentiles, memory, queue growth, and errors. Profile after evidence identifies a bottleneck; a large queue is not a throughput fix if admission exceeds processing indefinitely.

Injected fault	Evidence to collect	Recovery check
Broker down/restart	Reconnect state, Session Present, subscription results	Expected subscriptions return; duplicate handling stays safe
Simulator offline or restart	Command state, expiry/deadline, session behavior	Late work is safe; status remains honest
Slow/unavailable DB	Query latency/errors, manual Ack policy, backlog	No false durable acceptance; recovery resumes
Duplicate report/result	Stable ID and duplicate counters	One business effect; stored outcome remains stable
Denied topic	Broker logs/reason and application visibility	No unauthorized effect or data leakage
Shutdown under load	Queue depth, accepted/rejected, drain deadline	Recoverable work is preserved or loss is explicitly recorded

📓 Write a runbook another person can follow

State the symptom and its user-visible impact.
Collect command/report ID, timestamps, process logs, metrics, broker evidence, and relevant database state without secrets.
Map the last confirmed evidence to HTTP, application, broker, device, or database boundary.
Give the bounded recovery action and note the risk of retrying a possibly completed physical effect.
Verify recovery with a harmless synthetic operation and record the evidence that changed.

✅ Done when

Someone who has not read your code can use the runbook to diagnose one injected command failure. Your load scenario and fault matrix can be repeated, and each metric helps answer an operational question.

📋 Short self-check answers

202 is API acceptance; PUBACK is one MQTT hop. Neither proves device completion.
Readiness reflects dependencies needed for the process's assigned work; liveness only says whether it is functioning enough to keep running.
Metric labels should remain bounded; use logs/traces for high-cardinality IDs.

📚 Section checklist: COURSE.md — Section 17. Primary references: Go net/http and Prometheus exposition format.

Ask me to help map a runbook symptom to the next useful measurement.