mqtt-5-learning/teaching/course-source/13-persist-reports-command-state.md

🗄️ Lesson 13 — Persist reports and command state in PostgreSQL

🎬 Start with the system story

❓ Question: The backend receives a QoS 1 sales report, commits it to PostgreSQL, then crashes before acknowledging it. What should happen when the broker delivers it again?

💬 Answer: The database must recognize the stable report identity and avoid applying the sale twice. The backend can then acknowledge the repeated delivery and return the previously recorded application outcome. MQTT manages protocol delivery; PostgreSQL owns durable business state.

sequenceDiagram
    participant D as Device
    participant B as Broker
    participant I as Backend ingestion
    participant DB as PostgreSQL
    D->>B: QoS 1 PUBLISH report_id
    B->>I: QoS 1 PUBLISH
    I->>I: Parse topic and validate payload
    I->>DB: Transaction: dedup key + business effect
    DB-->>I: COMMIT
    I->>B: PUBACK
    Note over I,B: Crash before PUBACK can cause a duplicate delivery

The acceptance boundary for important work is a successful database commit. The callback may hand work to a bounded worker, but keep the original Paho packet and client available until acknowledgement. A payload copy alone cannot be acknowledged later.

📥 1. Decide when to acknowledge

❓ Question: What does Paho do if a callback returns an error? Does that tell the broker to retry?

For the pinned Paho v0.23.0 client, automatic acknowledgement sends PUBACK for a QoS 1 message after the callback returns—even if it returned an error. The error is local to the callback chain; it is not an MQTT negative acknowledgement or a retry request. If the callback enqueues the message in RAM and returns, the broker may consider the backend's delivery complete before a worker processes it.

With EnableManualAcknowledgment, Paho does not automatically acknowledge a QoS 1 message when the callback returns. Your application calls received.Client.Ack(received.Packet) after reaching its policy boundary. For reports and command results, use this order:

1. Parse the MQTT topic and validate the message contract (Lesson 5).
2. Submit the message to a bounded worker queue while retaining its packet/client acknowledgement handle.
3. In one PostgreSQL transaction, record the stable identity, apply the business effect or command-state transition, and store the application outcome.
4. Commit the transaction.
5. Call Ack and check its error.

If the process crashes after commit but before PUBACK, the broker may deliver the message again. The unique constraint makes the repeated transaction a no-op for business effects; the backend acknowledges after it recognizes the duplicate. If it crashes before commit, there is no durable admission, so the message must remain recoverable from the MQTT session/broker according to QoS, expiry, and persistence settings.

❓ What does a callback error mean in each mode?

Mode	Callback returns nil	Callback returns an error	What causes a retry?
Automatic	Paho sends PUBACK after callback return	Paho still sends PUBACK; error does not request redelivery	Not the callback error. The delivery is complete at the protocol layer.
Manual	Paho does not automatically send PUBACK	Paho still does not automatically send PUBACK	Your code must choose recovery; return error alone is not a retry signal.

In manual mode, leaving a QoS 1 delivery unacknowledged can occupy an in-flight slot and slow later QoS 1/2 deliveries. Paho's acknowledgement tracker sends acknowledgements in receipt order, so acknowledging a later message can wait behind an earlier one. Disconnect/reconnect redelivery is conditional on session continuity, broker persistence, QoS, and message expiry. Manual acknowledgement is not permanent broker storage.

For permanently invalid messages, persist a bounded rejection record or a quarantine entry, then acknowledge after that disposition commits. Do not leave poison messages unacknowledged forever. For transient database failures, do not record permanent rejection; stop or back off consumption using a deliberate connection/recovery policy so work remains recoverable. A callback error alone does neither.

📚 Pinned implementation: references/paho.golang-v0.23.0/paho/client.go documents callback invocation, manual acknowledgement, Ack, and ordered acknowledgement tracking. MQTT 5 Receive Maximum bounds the QoS 1/2 publications that can remain unacknowledged on a connection; it does not bound your Go queue.

🔁 2. Make database effects idempotent

❓ Question: Two ingestion workers receive the same report concurrently. What prevents both transactions from counting its items?

Use a stable business key and a database uniqueness constraint. Examples include (device_id, report_id) for sales reports and (device_id, command_id) for device command outcomes. Do not use the MQTT packet identifier; it belongs to an exchange and can be reused.

Inside one transaction:

1. Attempt to insert the deduplication/business record under its unique key.
2. If it is new, apply the corresponding inventory/revenue or command-state effect.
3. Store the application response/outcome needed for a repeat.
4. Commit all related changes together.

If the key already exists, load the previously stored outcome and do not apply the effect again. If the same key arrives with materially different content, record a conflict instead of silently assigning the ID a new meaning. The database constraint arbitrates concurrent workers and remains after process restart; an in-memory map cannot do either reliably.

🧪 Predict, then try it

• Send one report twice sequentially and once concurrently. Expect one business effect and an identifiable duplicate outcome.
• Force a transaction rollback after recording the dedup key but before the business effect. Verify that neither half is committed.
• Commit the transaction, terminate the process before Ack, reconnect under the same eligible session, and observe duplicate delivery without a duplicate effect.

🧹 3. Persist invalid-message disposition

❓ Question: Should permanently invalid JSON occupy a Receive Maximum slot indefinitely?

No. After safe validation fails, write a quarantine/rejection record with bounded metadata: topic, reason category, receipt time, and a safe message identifier if one could be parsed. Avoid storing secrets or dumping arbitrary payloads. Commit that record, then acknowledge the delivery. For transient infrastructure failures, keep the work recoverable instead of classifying it as permanently invalid.

This creates a clear distinction:

Result	Database action	MQTT action
Valid, new message	Store identity and business effect atomically	Ack after commit
Valid duplicate	Load prior outcome; no repeated effect	Ack after duplicate is recognized
Permanently invalid	Commit bounded rejection/quarantine record	Ack after disposition commits
Transient DB/storage failure	No false success or permanent rejection	Do not Ack; invoke recovery/backoff policy

🧱 4. Add PostgreSQL to the lab

❓ Question: Which state needs to survive a backend restart, and which state can remain temporary?

Persist accepted sales reports, deduplication keys, command state/results, and any outbox intent needed by the next lesson. Keep connection objects, worker queues, and in-flight Paho packet handles in memory; they cannot be made database records. On restart, recover work from durable application rows and eligible MQTT session deliveries rather than pretending the old callback is still alive.

Use the course's dedicated lab PostgreSQL service only. Add migrations, bounded connection-pool settings, query contexts/deadlines, and repository interfaces used by business code. Use PostgreSQL unique indexes as the final deduplication authority. Define transaction boundaries around a single business decision rather than spreading a report effect over unrelated commits.

Learner commands, run from the project root after adding the DB files:

docker compose -p mqtt5-learning config --quiet
docker compose -p mqtt5-learning up -d postgres
go test ./internal/ingest ./internal/sales ./internal/commands
go test -race ./...

The first command validates the lab Compose configuration. The second starts only the project-named PostgreSQL service; inspect its health/logs before running integration tests. Tests must use the dedicated lab database and synthetic data. Stop that service with the matching Compose project when finished; do not point these commands at production.

🔄 5. Separate MQTT acknowledgement from application responses

❓ Question: After the backend commits a sales report and sends PUBACK, what does the device know?

PUBACK completes the device-to-broker QoS 1 delivery hop; it does not say that the backend committed the report. If the device needs application-level evidence, publish a separate receipt after the transaction commits. That receipt should include the stable report ID and a status the device can interpret. The report may be committed even if the receipt is lost, so a retry must receive the stored prior outcome.

For commands, keep broker publish acknowledgement, device acceptance, and device completion as distinct command states. A database commit that records a command result is backend evidence; it does not retroactively change the meaning of the earlier command PUBACK.

If database commit and MQTT response publication must survive a crash between them, write an outbox row in the same database transaction as the business state. Lesson 14 handles that publication gap.

🧪 Failure experiments and done criteria

1. Send a duplicate report and verify the database effect occurs once.
2. Interrupt PostgreSQL before the transaction commits; verify the message is not acknowledged as accepted.
3. Crash after commit but before manual Ack; verify replay returns the previous outcome without repeating the effect.
4. Send a permanently invalid message; verify a safe rejection/quarantine record commits before Ack.
5. Delay acknowledgements until Receive Maximum is reached; observe delivery pressure, then free a slot and confirm delivery resumes.
6. Crash after commit but before publishing an application response. Record the failure window that Lesson 14's outbox will close.

Done when: Replaying a report after restart does not double-count it, database failures do not become false acknowledgements, invalid messages have a durable disposition, and you can explain why QoS 2 does not make database or physical effects exactly once.

Next: Lesson 14 — Close database/MQTT failure gaps makes publication intent recoverable with a transactional outbox.