Go MQTT Vending Backend ยท Lesson 13 ยท 15 minutes

Commit business state before acknowledging important work

Use PostgreSQL uniqueness and transactions to make report effects durable across duplicate delivery and backend restart.

๐ŸŽฏ One skill: choose the durable acceptance boundary๐Ÿง  Predict crash recovery๐Ÿงช Dedicated lab database only

โ“ Start with the system story

The backend commits a QoS 1 sales report to PostgreSQL, then crashes before sending PUBACK. What should happen when the broker delivers the report again?

Pause before continuing: identify the durable idempotency key and the order of transaction and acknowledgement.

๐Ÿ“ฅ Acknowledge after the policy boundary

Receive QoS 1
validate
โ†’
PostgreSQL
dedupe + effect transaction
โ†’
COMMIT
durable outcome
โ†’
Manual Ack
check error

With Paho v0.23.0 automatic acknowledgement, a callback error does not request MQTT retry; Paho acknowledges QoS 1 after callback return. Enabling manual acknowledgement changes who calls Ack, but returning an error still does not cause retry by itself. Keep the packet/client acknowledgement handle until the durable policy decision, not just a copied payload in a RAM queue.

๐Ÿง  Check crash recovery: PostgreSQL committed a report, but the backend crashed before PUBACK. What should replay do?




๐Ÿ” Let the database arbitrate duplicates

Use a unique business key such as (device_id, report_id). In one transaction, insert the report identity, apply its business effect, and store the application outcome needed for duplicate replay. The unique constraint handles two concurrent workers and remains after restart; an in-memory map cannot.

If the same key arrives with materially different content, record a conflict instead of silently giving one ID two meanings. MQTT packet IDs are not business IDs. QoS 2 cannot make a PostgreSQL transaction and a physical vending action exactly once.

๐Ÿงญ File overview: internal/persistence/sales.go

Create this complete repository file for durable sales ingestion. The caller opens a PostgreSQL *sql.DB using the project's pgx stdlib driver and validates the report with contracts.Decode first. This file owns the transaction boundary: report identity and sale lines commit together.

๐Ÿ—๏ธ Migrate โ€” short summary

Creates a report table with a unique business key and a child line table.

๐Ÿ’พ SaveReport โ€” short summary

Inserts the report identity, detects duplicate or conflicting content, inserts lines for a new report, and commits once.

๐Ÿงฉ Complete file contents
package persistence

import (
	"context"
	"crypto/sha256"
	"database/sql"
	"encoding/json"
	"errors"
	"fmt"
	"time"

	"example.com/mqtt5-learning/internal/contracts"
)

type ReportOutcome uint8

const (
	ReportSaved ReportOutcome = iota + 1
	ReportDuplicate
)

var ErrReportIDConflict = errors.New("report ID was reused with different content")

type SalesRepository struct {
	db *sql.DB
}

func NewSalesRepository(db *sql.DB) (*SalesRepository, error) {
	if db == nil {
		return nil, fmt.Errorf("database connection is required")
	}
	return &SalesRepository{db: db}, nil
}

func (r *SalesRepository) Migrate(ctx context.Context) error {
	statements := []string{
`CREATE TABLE IF NOT EXISTS sales_reports (
    device_id TEXT NOT NULL,
    report_id TEXT NOT NULL,
    payload_hash BYTEA NOT NULL,
    occurred_at TIMESTAMPTZ NOT NULL,
    saved_at TIMESTAMPTZ NOT NULL DEFAULT now(),
    PRIMARY KEY (device_id, report_id)
)`,
`CREATE TABLE IF NOT EXISTS sales_report_lines (
    device_id TEXT NOT NULL,
    report_id TEXT NOT NULL,
    line_number INTEGER NOT NULL,
    sku TEXT NOT NULL,
    quantity BIGINT NOT NULL CHECK (quantity > 0),
    amount_minor BIGINT NOT NULL CHECK (amount_minor >= 0),
    PRIMARY KEY (device_id, report_id, line_number),
    FOREIGN KEY (device_id, report_id)
        REFERENCES sales_reports (device_id, report_id) ON DELETE CASCADE
	)`,
	}
	for _, statement := range statements {
		if _, err := r.db.ExecContext(ctx, statement); err != nil {
			return fmt.Errorf("create sales schema: %w", err)
		}
	}
	return nil
}

func (r *SalesRepository) SaveReport(ctx context.Context, report contracts.SalesReport) (ReportOutcome, error) {
	if report.DeviceID == "" || report.ReportID == "" || len(report.Items) == 0 {
		return 0, fmt.Errorf("validated report identity and at least one item are required")
	}
	occurredAt, err := time.Parse(time.RFC3339, report.OccurredAt)
	if err != nil {
		return 0, fmt.Errorf("parse report time: %w", err)
	}
	encoded, err := json.Marshal(report)
	if err != nil {
		return 0, fmt.Errorf("encode report for deduplication: %w", err)
	}
	hash := sha256.Sum256(encoded)
	tx, err := r.db.BeginTx(ctx, &sql.TxOptions{Isolation: sql.LevelReadCommitted})
	if err != nil {
		return 0, fmt.Errorf("begin report transaction: %w", err)
	}
	defer func() { _ = tx.Rollback() }() // No effect after a successful commit.

	result, err := tx.ExecContext(ctx, `
INSERT INTO sales_reports (device_id, report_id, payload_hash, occurred_at)
VALUES ($1, $2, $3, $4)
ON CONFLICT (device_id, report_id) DO NOTHING`,
		report.DeviceID, report.ReportID, hash[:], occurredAt)
	if err != nil {
		return 0, fmt.Errorf("insert report identity: %w", err)
	}
	inserted, err := result.RowsAffected()
	if err != nil {
		return 0, fmt.Errorf("inspect report insert: %w", err)
	}
	if inserted == 0 {
		var storedHash []byte
		if err := tx.QueryRowContext(ctx, `
SELECT payload_hash FROM sales_reports WHERE device_id=$1 AND report_id=$2`,
			report.DeviceID, report.ReportID).Scan(&storedHash); err != nil {
			return 0, fmt.Errorf("load duplicate report: %w", err)
		}
		if !equalBytes(storedHash, hash[:]) {
			return 0, ErrReportIDConflict
		}
		if err := tx.Commit(); err != nil {
			return 0, fmt.Errorf("commit duplicate observation: %w", err)
		}
		return ReportDuplicate, nil
	}
	for i, line := range report.Items {
		if _, err := tx.ExecContext(ctx, `
INSERT INTO sales_report_lines
    (device_id, report_id, line_number, sku, quantity, amount_minor)
VALUES ($1, $2, $3, $4, $5, $6)`,
			report.DeviceID, report.ReportID, i, line.SKU, line.Quantity, line.AmountMinor); err != nil {
			return 0, fmt.Errorf("insert sales line %d: %w", i, err)
		}
	}
	if err := tx.Commit(); err != nil {
		return 0, fmt.Errorf("commit sales report: %w", err)
	}
	return ReportSaved, nil
}

func equalBytes(left, right []byte) bool {
	if len(left) != len(right) {
		return false
	}
	var difference byte
	for i := range left {
		difference |= left[i] ^ right[i]
	}
	return difference == 0
}

๐Ÿ”Ž Details: duplicate and transaction behavior

The unique constraint handles concurrent transactions for one device/report pair. The winner inserts lines in the same transaction as the report row. A second identical report reads the stored digest and returns ReportDuplicate; different content under the same key returns ErrReportIDConflict. The deferred rollback is harmless after commit and cleans up early returns. Do not Ack the MQTT packet until this method returns a committed saved/duplicate outcome. This repository does not send the application response; publication intent belongs in an outbox when crash-safe receipts are required.

๐Ÿงญ File overview: internal/persistence/commands.go

Command state also needs durable storage. Create this complete file alongside sales.go. It stores validated device results under the stable command ID and allows a late result to resolve an earlier unknown outcome without losing the separate broker-ack field.

๐Ÿ—๏ธ Migrate โ€” short summary

Creates the command table and a unique result table.

๐Ÿ“ฉ ApplyResult โ€” short summary

Inserts one result, accepts identical replays, detects conflicting IDs, and updates command status transactionally.

๐Ÿ“„ internal/persistence/commands.go โ€” complete file
package persistence

import (
	"context"
	"database/sql"
	"errors"
	"fmt"
	"time"

	"example.com/mqtt5-learning/internal/contracts"
)

var ErrCommandResultConflict = errors.New("command ID has a conflicting result")

type CommandRepository struct { db *sql.DB }

func NewCommandRepository(db *sql.DB) (*CommandRepository, error) {
	if db == nil { return nil, fmt.Errorf("database connection is required") }
	return &CommandRepository{db: db}, nil
}

func (r *CommandRepository) Migrate(ctx context.Context) error {
	statements := []string{
		`CREATE TABLE IF NOT EXISTS device_commands (
			command_id TEXT PRIMARY KEY, device_id TEXT NOT NULL, state TEXT NOT NULL,
			broker_acked BOOLEAN NOT NULL DEFAULT FALSE,
			deadline TIMESTAMPTZ,
			created_at TIMESTAMPTZ NOT NULL DEFAULT now(), updated_at TIMESTAMPTZ NOT NULL DEFAULT now())`,
		`CREATE TABLE IF NOT EXISTS command_results (
			command_id TEXT NOT NULL REFERENCES device_commands(command_id),
			device_id TEXT NOT NULL, message_id TEXT NOT NULL, status TEXT NOT NULL,
			detail TEXT NOT NULL DEFAULT '', occurred_at TIMESTAMPTZ NOT NULL,
			PRIMARY KEY(command_id,message_id), UNIQUE(device_id,message_id))`,
	}
	for _, statement := range statements {
		if _, err := r.db.ExecContext(ctx, statement); err != nil {
			return fmt.Errorf("create command schema: %w", err)
		}
	}
	return nil
}

func (r *CommandRepository) Create(ctx context.Context, commandID, deviceID string) error {
	if commandID == "" || deviceID == "" { return fmt.Errorf("command ID and device ID are required") }
	_, err := r.db.ExecContext(ctx, `INSERT INTO device_commands(command_id,device_id,state)
VALUES ($1,$2,'pending')`, commandID, deviceID)
	if err != nil { return fmt.Errorf("create command: %w", err) }
	return nil
}

func (r *CommandRepository) MarkBrokerAck(ctx context.Context, commandID string) error {
	result, err := r.db.ExecContext(ctx, `UPDATE device_commands SET broker_acked=TRUE,updated_at=now()
WHERE command_id=$1`, commandID)
	if err != nil { return fmt.Errorf("record broker acknowledgement: %w", err) }
	rows, err := result.RowsAffected()
	if err != nil { return fmt.Errorf("inspect command update: %w", err) }
	if rows != 1 { return fmt.Errorf("command %q not found", commandID) }
	return nil
}

func (r *CommandRepository) ApplyResult(ctx context.Context, result contracts.CommandResult) error {
	if result.CommandID == "" || result.DeviceID == "" || result.MessageID == "" {
		return fmt.Errorf("validated result identity is required")
	}
	occurredAt, err := time.Parse(time.RFC3339, result.OccurredAt)
	if err != nil { return fmt.Errorf("parse result time: %w", err) }
	tx, err := r.db.BeginTx(ctx, nil)
	if err != nil { return fmt.Errorf("begin result transaction: %w", err) }
	defer func() { _ = tx.Rollback() }()
	insert, err := tx.ExecContext(ctx, `INSERT INTO command_results
(command_id,device_id,message_id,status,detail,occurred_at)
VALUES ($1,$2,$3,$4,$5,$6) ON CONFLICT (command_id,message_id) DO NOTHING`,
		result.CommandID,result.DeviceID,result.MessageID,result.Status,result.Detail,occurredAt)
	if err != nil { return fmt.Errorf("insert command result: %w", err) }
	rows, err := insert.RowsAffected()
	if err != nil { return fmt.Errorf("inspect result insert: %w", err) }
	if rows == 0 {
		var oldDevice, oldMessage, oldStatus, oldDetail string
		if err := tx.QueryRowContext(ctx, `SELECT device_id,message_id,status,detail
FROM command_results WHERE command_id=$1 AND message_id=$2`, result.CommandID,result.MessageID).Scan(&oldDevice,&oldMessage,&oldStatus,&oldDetail); err != nil {
			return fmt.Errorf("load prior command result: %w", err)
		}
		if oldDevice != result.DeviceID || oldMessage != result.MessageID || oldStatus != result.Status || oldDetail != result.Detail {
			return ErrCommandResultConflict
		}
		return tx.Commit() // Identical replay: keep the prior terminal state.
	}
	updated, err := tx.ExecContext(ctx, `UPDATE device_commands SET state=$2,updated_at=now()
WHERE command_id=$1 AND device_id=$3 AND state IN ('pending','accepted','unknown')`,
		result.CommandID,result.Status,result.DeviceID)
	if err != nil { return fmt.Errorf("update command state: %w", err) }
	changed, err := updated.RowsAffected()
	if err != nil { return fmt.Errorf("inspect command state update: %w", err) }
	if changed != 1 { return fmt.Errorf("command missing, device mismatch, or terminal state conflict") }
	if err := tx.Commit(); err != nil { return fmt.Errorf("commit command result: %w", err) }
	return nil
}

๐Ÿ”Ž Details: legal state transitions

The state IN ('pending','accepted','unknown') guard prevents a new result from overwriting an already terminal state such as completed or rejected. The result row and command-state update share a transaction. broker_acked remains a separate transport fact. Before using this example, ensure the command row exists and the result's topic/body identity was validated in the ingestion service.

๐Ÿงน Invalid and transient failures need different paths

OutcomeDurable actionMQTT decision
Valid new reportCommit identity and effectAck after commit
Valid duplicateLoad prior result; no repeated effectAck after recognizing it
Permanently invalidCommit bounded quarantine/rejection metadataAck after disposition commits
Transient database failureDo not claim durable acceptanceDo not Ack; invoke explicit recovery/backoff

A poison message must not occupy a Receive Maximum slot forever. Record safe metadata such as topic, reason category, and receipt time; avoid secrets and arbitrary payload dumps. A transient database outage is not permanent invalid input.

๐Ÿงช Crash-point experiments

  1. Send the same report twice sequentially and concurrently. Expect one durable business effect.
  2. Force a transaction rollback and verify neither dedupe identity nor business effect commits alone.
  3. Commit, stop the process before manual Ack, reconnect with an eligible broker session, and observe a safe duplicate.
  4. Send a permanently invalid report and confirm a bounded rejection record commits before Ack.
  5. Delay acknowledgements to the configured Receive Maximum and observe delivery pressure. Remember this protocol limit does not cap your Go worker queue.
  6. Record the gap between database commit and publishing an application response; Lesson 14 closes that gap with an outbox.

๐Ÿงฑ Keep durable and temporary state distinct

Persist reports, deduplication keys, command state/results, and any publication intent needed after restart. Keep live connections, worker queues, in-flight Paho packet handles, and process contexts in memory. On restart, recover from durable rows and eligible MQTT sessions; a dead callback cannot resume.

Evidence boundary: database COMMIT establishes the backend's durable acceptance policy. PUBACK completes the MQTT protocol delivery. A separate application receipt tells the device the backend outcome, and it can still be lost unless its publication intent is persisted.

โœ… Done when

After a backend restart, replay does not double-count a report; a database outage is not acknowledged as durable success; invalid data has a durable disposition; and you can explain why manual Ack alone is not permanent broker storage.

Run only against the dedicated lab PostgreSQL service and synthetic data. The full source lesson includes the project-specific Docker commands and Paho acknowledgement references.

๐Ÿ“‹ Short self-check answers
  1. Unique stable business key plus one transaction for identity/effect/outcome.
  2. Commit then Ack; after crash-before-Ack, recognize duplicate and avoid repeating the effect.
  3. Callback error alone does not request retry; recovery is an application policy.
  4. Manual Ack is not a durable application queue and depends on session/broker behavior.

๐Ÿ“š Failure experiments: Lesson 13 source lesson. Primary references: Go database/sql, PostgreSQL constraints, and the OASIS MQTT 5.0 Standard.

Ask me to reason through the exact crash point you observed before changing the acknowledgement policy.