first commit

lessons/07-command-response.md

@@ -0,0 +1,94 @@
+# Section 7 — Build reliable command and response behavior
+
+Prerequisite: Sections 5–6 validate messages and bound concurrent processing.
+
+Run all commands from `/home/ryan/Media/Code/Code/robot-market/mqtt-5-learning`. Create files at the project-relative paths shown below, not under `lessons/`. Continue the same application and pinned Paho v0.23.0 dependency. These are guided implementation exercises; attempt each step before requesting a full solution.
+
+## MQTT request/response is an application conversation
+
+Response Topic tells the responder where a result should be published. Correlation Data lets the requester recognize which pending request it belongs to. The broker forwards these properties; it does not run the command, create a result, or enforce your timeout.
+
+This fragment is the packet-construction portion of a service method, not a standalone program. `commandJSON` is an encoded validated command and `correlation` is owned by its pending record:
+
+```go
+request := &paho.Publish{
+    Topic:   "lab/v1/devices/device-001/commands",
+    QoS:     1,
+    Retain:  false,
+    Payload: commandJSON,
+    Properties: &paho.PublishProperties{
+        ContentType:     "application/json",
+        ResponseTopic:   "lab/v1/devices/device-001/responses",
+        CorrelationData: append([]byte(nil), correlation...),
+    },
+}
+```
+
+Pass `request` to the bounded publication operation after registering pending state. The simulator validates `ResponseTopic`, publishes the result to it, and copies Correlation Data unchanged into the result's properties. Correlation bytes are opaque; they need not be text. Do not trust an arbitrary supplied topic as a response destination.
+
+### Trace a fast response and a lost response
+
+1. Backend establishes its response subscription and checks SUBACK.
+2. Backend stores the expected device, command ID, correlation, and deadline.
+3. Backend publishes. The simulator can respond before the publish call returns.
+4. Backend validates and applies device evidence independently of the broker acknowledgement.
+5. If the waiting deadline expires, backend removes the waiter and records unknown outcome. A later result follows the documented late-result policy.
+
+Registering after step 3 loses the race with a fast response. Holding the command-map mutex across publishing can block the callback needed to record that response. A context timeout stops a local wait; it cannot recall bytes already delivered to a device.
+
+| Identifier | Owner and lifetime | Why it cannot replace the others |
+| --- | --- | --- |
+| MQTT packet ID | Library, per protocol exchange/session direction, reusable | Identifies a transport handshake, not a permanent command |
+| Correlation Data | Request/response mechanism | Matches a response; does not authenticate it or prevent repeated effects |
+| Command ID | Application, stable across retries | Keys the dedup record and business status |
+
+### Make duplicate behavior observable
+
+Use a synthetic effect counter. Send command C twice with the same content, then C with changed content. Expect one effect, a replayed recorded result, then a conflict. Also send the same C in two separate publish calls: application duplicates need not carry MQTT DUP. The DUP flag describes protocol retransmission, so deduplicating only packets with DUP set is wrong.
+
+Concurrent duplicates need atomic admission: a lookup followed by an unlocked effect allows two workers to both observe "not seen". Reserve the ID with its fingerprint before executing, and define what a second request sees while the first is in progress. A real effect followed by a crash before storing its result remains a separate failure window; the in-memory simulator only demonstrates the idea.
+
+<details>
+<summary>The broker acknowledged, but no device result arrived. Which status is justified?</summary>
+
+Only the transport acknowledgement is established. The device may be offline, may have rejected the request, may still be executing, or may have executed with a lost response. Once its deadline expires, expose unknown outcome. Retry only under the same business identity and a deliberate idempotency policy.
+
+</details>
+
+## 1. Preserve MQTT 5 request metadata
+
+Create `internal/commands/service.go`, `store.go`, and `service_test.go`. Edit `internal/mqtt/client.go` to carry Response Topic and a copied Correlation Data slice in received messages. Add an outbound request form that accepts these properties. Retain the existing bounded publish timeout and reason-code checks.
+
+Read `references/paho.golang-v0.23.0/paho/cp_publish.go` for `paho.PublishProperties`. Set `ResponseTopic` to `lab/v1/devices/device-001/responses` and `CorrelationData` to opaque bytes associated with the pending command. Establish the backend response subscription before sending. Register pending state before publishing so an immediate response cannot beat registration.
+
+## 2. Model evidence explicitly
+
+Store command ID, expected device, correlation bytes, creation/deadline times, business status, and independent transport acknowledgement evidence. Start with created → pending delivery → accepted → completed or rejected. A ping simulator can send accepted followed by completed. Response timeout means unknown outcome; it does not establish that no effect happened.
+
+Protect concurrent access and define allowed transitions. A completed or rejected command must not regress on a duplicate accepted response. Compare command identity, topic/body device identity, and correlation before applying a response. Keep timed-out records for a bounded retention period even after removing their waiters, so status can explain the uncertainty.
+
+## 3. Make the simulator idempotent
+
+Create `internal/simulator/dedup.go` and its tests. Index effects by stable command ID and retain a fingerprint of the command. For an identical duplicate, replay the recorded result without repeating the effect. Reject reuse of the ID with different content. Bound cache lifetime and size; state clearly that restarting the simulator loses this in-memory protection.
+
+Allow a Response Topic only when it exactly matches the authorized response topic for this simulator. Correlation is matching data, not authorization. Continue rejecting retained one-shot commands.
+
+## 4. Exercise timing and failures
+
+Replace the backend's unconditional ticker with an explicit finite command exercise or service calls. Send 20 concurrent synthetic pings with separate IDs. Introduce delayed, duplicate, wrong-device, wrong-correlation, and out-of-order responses. Check every command receives only its own evidence. Retry a timed-out request with the same stable ID; never generate a new ID merely because publication timed out.
+
+```sh
+mkdir -p internal/commands internal/simulator
+gofmt -w internal cmd
+go test -race ./...
+```
+
+Run `mkdir -p` before saving the service and simulator package files. `gofmt` formats your edits and `go test -race ./...` executes the module's tests with concurrency instrumentation. A fake transport is a small in-process implementation of the caller's transport interface: it lets tests deliberately return errors or responses without needing a broker. An injected clock makes deadline tests deterministic without waiting in real time.
+
+When copying Correlation Data, allocate a new byte slice so later processing owns stable bytes. Comparing correlation bytes means comparing their contents, not pointer addresses. Before updating a shared command map, validate the response and hold its lock only for the state transition. Publish outside the lock so a network delay cannot freeze every command.
+
+Test responses arriving before publish returns, publish failure with uncertain outcome, timeout cleanup, deduplication, and terminal-state protection. Use an injectable clock and fake transport for deterministic service tests.
+
+**Done when:** Concurrent requests match correctly, identical duplicates cause one effect, unexpected destinations are rejected, and late responses follow a documented policy. Explain why PUBACK and Correlation Data cannot prove business completion.
+
+Previous: [06-concurrency-cancellation.md](06-concurrency-cancellation.md). Next: [08-http-commands.md](08-http-commands.md).