Transactional outbox
Application DB owns durable business intent and retry status. It does not prove broker delivery or device completion.
Go MQTT Vending Backend ยท Lesson 14 ยท 15 minutes
Use a transactional outbox to bridge the database and MQTT broker while keeping retries, duplicates, and uncertain device outcomes visible.
โ Start with the system story
The API commits a ping command to PostgreSQL, then crashes before publishing it. What durable record lets another process resume the accepted work?
Name which values must commit together and which process retries them.
The transaction closes the gap where a command is durably accepted but the intent to publish is lost. A worker claims rows briefly, publishes outside the DB transaction, then records the attempt. Do not hold a database transaction open while waiting on a network operation.
If the broker accepts a PUBLISH and the backend crashes before marking its outbox row sent, the worker may publish it again after restart. Keep one command ID across retries. The simulator stores the prior effect/result by command ID and returns that outcome on a duplicate. A timeout after possible publication is an unknown device outcome, not proof that it did nothing.
| Crash point | What remains | Recovery |
|---|---|---|
| Before transaction commit | No accepted command/outbox | Caller may retry with the same HTTP idempotency key |
| After commit, before publish | Pending outbox row | Worker publishes after restart |
| After publish, before sent marker | Row may remain pending | Republish same command ID; device deduplicates |
| After device effect, before result | Device may have acted; backend lacks result | Expose unknown/pending and reconcile by stable ID |
internal/outbox/store.goCreate this complete PostgreSQL repository file for the outbox chapter. The HTTP API calls Enqueue in its acceptance path; a bounded worker uses ClaimOne and records the PUBACK outcome with MarkPublished. No database transaction stays open during the network publish.
Enqueue โ short summaryCommits the command state and serialized publication intent atomically under one command ID.
ClaimOne โ short summaryUses a short transaction and FOR UPDATE SKIP LOCKED to lease one eligible row for one worker.
MarkPublished โ short summaryRecords the broker acknowledgement as transport evidence and marks only that command's outbox row sent.
package outbox
import (
"context"
"database/sql"
"errors"
"fmt"
"time"
)
var ErrNoWork = errors.New("no eligible outbox row")
type Command struct {
ID string
DeviceID string
Topic string
Payload []byte
Deadline time.Time
}
type Item struct {
CommandID string
DeviceID string
Topic string
Payload []byte
Deadline time.Time
Attempt int
}
type Store struct { db *sql.DB }
func NewStore(db *sql.DB) (*Store, error) {
if db == nil { return nil, fmt.Errorf("database connection is required") }
return &Store{db: db}, nil
}
func (s *Store) Migrate(ctx context.Context) error {
statements := []string{
`CREATE TABLE IF NOT EXISTS device_commands (
command_id TEXT PRIMARY KEY, device_id TEXT NOT NULL, state TEXT NOT NULL,
deadline TIMESTAMPTZ NOT NULL, created_at TIMESTAMPTZ NOT NULL DEFAULT now())`,
`CREATE TABLE IF NOT EXISTS command_outbox (
command_id TEXT PRIMARY KEY REFERENCES device_commands(command_id),
topic TEXT NOT NULL, payload BYTEA NOT NULL, attempts INTEGER NOT NULL DEFAULT 0,
available_at TIMESTAMPTZ NOT NULL DEFAULT now(), lease_until TIMESTAMPTZ,
published_at TIMESTAMPTZ, last_reason SMALLINT)`,
}
for _, statement := range statements {
if _, err := s.db.ExecContext(ctx, statement); err != nil {
return fmt.Errorf("create outbox schema: %w", err)
}
}
return nil
}
func (s *Store) Enqueue(ctx context.Context, command Command) error {
if command.ID == "" || command.DeviceID == "" || command.Topic == "" || len(command.Payload) == 0 {
return fmt.Errorf("command ID, device, topic, and payload are required")
}
if !command.Deadline.After(time.Now()) {
return fmt.Errorf("command deadline must be in the future")
}
tx, err := s.db.BeginTx(ctx, nil)
if err != nil { return fmt.Errorf("begin command transaction: %w", err) }
defer func() { _ = tx.Rollback() }()
if _, err := tx.ExecContext(ctx, `INSERT INTO device_commands(command_id,device_id,state,deadline)
VALUES ($1,$2,'pending',$3)`, command.ID, command.DeviceID, command.Deadline); err != nil {
return fmt.Errorf("insert command: %w", err)
}
if _, err := tx.ExecContext(ctx, `INSERT INTO command_outbox(command_id,topic,payload)
VALUES ($1,$2,$3)`, command.ID, command.Topic, command.Payload); err != nil {
return fmt.Errorf("insert outbox intent: %w", err)
}
if err := tx.Commit(); err != nil { return fmt.Errorf("commit command and outbox: %w", err) }
return nil
}
func (s *Store) ClaimOne(ctx context.Context, lease time.Duration, now time.Time) (Item, error) {
if lease <= 0 { return Item{}, fmt.Errorf("lease must be positive") }
tx, err := s.db.BeginTx(ctx, nil)
if err != nil { return Item{}, fmt.Errorf("begin outbox claim: %w", err) }
defer func() { _ = tx.Rollback() }()
var item Item
err = tx.QueryRowContext(ctx, `SELECT o.command_id,c.device_id,o.topic,o.payload,c.deadline,o.attempts
FROM command_outbox o JOIN device_commands c USING (command_id)
WHERE o.published_at IS NULL AND o.available_at <= $1
AND (o.lease_until IS NULL OR o.lease_until <= $1)
AND c.deadline > $1
ORDER BY o.available_at,o.command_id
LIMIT 1 FOR UPDATE OF o SKIP LOCKED`, now).Scan(
&item.CommandID, &item.DeviceID, &item.Topic, &item.Payload, &item.Deadline, &item.Attempt)
if errors.Is(err, sql.ErrNoRows) { return Item{}, ErrNoWork }
if err != nil { return Item{}, fmt.Errorf("select outbox work: %w", err) }
item.Payload = append([]byte(nil), item.Payload...)
if _, err := tx.ExecContext(ctx, `UPDATE command_outbox
SET lease_until=$2, attempts=attempts+1 WHERE command_id=$1`, item.CommandID, now.Add(lease)); err != nil {
return Item{}, fmt.Errorf("lease outbox work: %w", err)
}
if err := tx.Commit(); err != nil { return Item{}, fmt.Errorf("commit outbox lease: %w", err) }
item.Attempt++
return item, nil
}
func (s *Store) MarkPublished(ctx context.Context, commandID string, reason byte, at time.Time) error {
result, err := s.db.ExecContext(ctx, `UPDATE command_outbox SET published_at=$2,
lease_until=NULL,last_reason=$3 WHERE command_id=$1 AND published_at IS NULL`, commandID, at, reason)
if err != nil { return fmt.Errorf("mark outbox published: %w", err) }
rows, err := result.RowsAffected()
if err != nil { return fmt.Errorf("inspect outbox update: %w", err) }
if rows != 1 { return fmt.Errorf("outbox command %q was not pending", commandID) }
return nil
}
The lease stops two active workers from claiming the same row at once. A crashed worker leaves a lease that expires; a later worker can claim and republish. A DB transaction makes intent durable, but it cannot atomically include the MQTT PUBLISH. MarkPublished therefore follows the PUBLISH and can race a crash; the device must deduplicate. Production code should add bounded exponential backoff, a retryable/permanent error classification, and a separate terminal outcome for expired rows rather than retrying forever.
Application DB owns durable business intent and retry status. It does not prove broker delivery or device completion.
Client library owns transport queueing under its configured semantics. It is not atomic with the business transaction.
Client/broker own protocol session continuity and eligible queued deliveries. It does not store business intent or dedupe effects.
Set a retry window with backoff and jitter, bound concurrent claims and publishes, and measure pending row count and oldest age. Check the application deadline before sending; use MQTT Message Expiry for broker-queued lifetime; have the device check its deadline before execution. Park expired or permanent failures and preserve unknown outcomes.
Every crash point has a recovery policy; duplicate publication creates one simulated effect; expiry blocks stale work; status separates PUBACK from device completion.
Evidence to record: prediction, injected crash point, DB rows, MQTT evidence, simulator effect count, final command state, and cleanup.
๐ Full crash experiment plan: Lesson 14 source lesson. Primary references: PostgreSQL locking clauses and OASIS MQTT 5.0 Standard.
Ask me to trace a crash window against the durable row and MQTT packet evidence.