mqtt-5-learning/lessons/07-command-response.md

Section 7 — Build reliable command and response behavior

Prerequisite: Sections 5–6 validate messages and bound concurrent processing.

Run all commands from /home/ryan/Media/Code/Code/robot-market/mqtt-5-learning. Create files at the project-relative paths shown below, not under lessons/. Continue the same application and pinned Paho v0.23.0 dependency. These are guided implementation exercises; attempt each step before requesting a full solution.

MQTT request/response is an application conversation

Response Topic tells the responder where a result should be published. Correlation Data lets the requester recognize which pending request it belongs to. The broker forwards these properties; it does not run the command, create a result, or enforce your timeout.

This fragment is the packet-construction portion of a service method, not a standalone program. commandJSON is an encoded validated command and correlation is owned by its pending record:

request := &paho.Publish{
    Topic:   "lab/v1/devices/device-001/commands",
    QoS:     1,
    Retain:  false,
    Payload: commandJSON,
    Properties: &paho.PublishProperties{
        ContentType:     "application/json",
        ResponseTopic:   "lab/v1/devices/device-001/responses",
        CorrelationData: append([]byte(nil), correlation...),
    },
}

Pass request to the bounded publication operation after registering pending state. The simulator validates ResponseTopic, publishes the result to it, and copies Correlation Data unchanged into the result's properties. Correlation bytes are opaque; they need not be text. Do not trust an arbitrary supplied topic as a response destination.

Trace a fast response and a lost response

1. Backend establishes its response subscription and checks SUBACK.
2. Backend stores the expected device, command ID, correlation, and deadline.
3. Backend publishes. The simulator can respond before the publish call returns.
4. Backend validates and applies device evidence independently of the broker acknowledgement.
5. If the waiting deadline expires, backend removes the waiter and records unknown outcome. A later result follows the documented late-result policy.

Registering after step 3 loses the race with a fast response. Holding the command-map mutex across publishing can block the callback needed to record that response. A context timeout stops a local wait; it cannot recall bytes already delivered to a device.

Identifier	Owner and lifetime	Why it cannot replace the others
MQTT packet ID	Library, per protocol exchange/session direction, reusable	Identifies a transport handshake, not a permanent command
Correlation Data	Request/response mechanism	Matches a response; does not authenticate it or prevent repeated effects
Command ID	Application, stable across retries	Keys the dedup record and business status

Make duplicate behavior observable

Use a synthetic effect counter. Send command C twice with the same content, then C with changed content. Expect one effect, a replayed recorded result, then a conflict. Also send the same C in two separate publish calls: application duplicates need not carry MQTT DUP. The DUP flag describes protocol retransmission, so deduplicating only packets with DUP set is wrong.

Concurrent duplicates need atomic admission: a lookup followed by an unlocked effect allows two workers to both observe "not seen". Reserve the ID with its fingerprint before executing, and define what a second request sees while the first is in progress. A real effect followed by a crash before storing its result remains a separate failure window; the in-memory simulator only demonstrates the idea.

The broker acknowledged, but no device result arrived. Which status is justified?

Only the transport acknowledgement is established. The device may be offline, may have rejected the request, may still be executing, or may have executed with a lost response. Once its deadline expires, expose unknown outcome. Retry only under the same business identity and a deliberate idempotency policy.

1. Preserve MQTT 5 request metadata

Create internal/commands/service.go, store.go, and service_test.go. Edit internal/mqtt/client.go to carry Response Topic and a copied Correlation Data slice in received messages. Add an outbound request form that accepts these properties. Retain the existing bounded publish timeout and reason-code checks.

Read references/paho.golang-v0.23.0/paho/cp_publish.go for paho.PublishProperties. Set ResponseTopic to lab/v1/devices/device-001/responses and CorrelationData to opaque bytes associated with the pending command. Establish the backend response subscription before sending. Register pending state before publishing so an immediate response cannot beat registration.

2. Model evidence explicitly

Store command ID, expected device, correlation bytes, creation/deadline times, business status, and independent transport acknowledgement evidence. Start with created → pending delivery → accepted → completed or rejected. A ping simulator can send accepted followed by completed. Response timeout means unknown outcome; it does not establish that no effect happened.

Protect concurrent access and define allowed transitions. A completed or rejected command must not regress on a duplicate accepted response. Compare command identity, topic/body device identity, and correlation before applying a response. Keep timed-out records for a bounded retention period even after removing their waiters, so status can explain the uncertainty.

3. Make the simulator idempotent

Create internal/simulator/dedup.go and its tests. Index effects by stable command ID and retain a fingerprint of the command. For an identical duplicate, replay the recorded result without repeating the effect. Reject reuse of the ID with different content. Bound cache lifetime and size; state clearly that restarting the simulator loses this in-memory protection.

Allow a Response Topic only when it exactly matches the authorized response topic for this simulator. Correlation is matching data, not authorization. Continue rejecting retained one-shot commands.

4. Exercise timing and failures

Replace the backend's unconditional ticker with an explicit finite command exercise or service calls. Send 20 concurrent synthetic pings with separate IDs. Introduce delayed, duplicate, wrong-device, wrong-correlation, and out-of-order responses. Check every command receives only its own evidence. Retry a timed-out request with the same stable ID; never generate a new ID merely because publication timed out.

mkdir -p internal/commands internal/simulator
gofmt -w internal cmd
go test -race ./...

Run mkdir -p before saving the service and simulator package files. gofmt formats your edits and go test -race ./... executes the module's tests with concurrency instrumentation. A fake transport is a small in-process implementation of the caller's transport interface: it lets tests deliberately return errors or responses without needing a broker. An injected clock makes deadline tests deterministic without waiting in real time.

When copying Correlation Data, allocate a new byte slice so later processing owns stable bytes. Comparing correlation bytes means comparing their contents, not pointer addresses. Before updating a shared command map, validate the response and hold its lock only for the state transition. Publish outside the lock so a network delay cannot freeze every command.

Test responses arriving before publish returns, publish failure with uncertain outcome, timeout cleanup, deduplication, and terminal-state protection. Use an injectable clock and fake transport for deterministic service tests.

Done when: Concurrent requests match correctly, identical duplicates cause one effect, unexpected destinations are rejected, and late responses follow a documented policy. Explain why PUBACK and Correlation Data cannot prove business completion.

Previous: 06-concurrency-cancellation.md. Next: 08-http-commands.md.