Verify server receipt delivery through configurable UI tunnel

DEPLOYMENT-HANDOFF.md

@@ -0,0 +1,112 @@
+# Backend deployment handoff — deployment verified
+
+Resumed 2026-10-10 (Asia/Tehran) at user request. Deployment verification is complete. The original paused-state record below is retained as history; the closeout supersedes its outstanding deployment checklist.
+
+## Closeout — 2026-10-10
+
+- Rechecked current server: image `mqtt5-test-backend:2026-10-08_11-24-19-9a32f634d41f`, healthy, `/healthz` ready.
+- Confirmed duplicate local observer `mqtt5-learning-observer-1` remains exited. No other local containers were stopped or restarted.
+- Restored private tunnel on local18081 -> server127.0.0.1:18080. Task exec session37169; do not assume process persistence in a later chat.
+- Current HTTP latency through the tunnel: readiness0.082s; snapshots0.105s and0.263s. Previous timeout was not reproduced; its exact historical cause is not proven.
+- Updated `scripts/check_receipts.py` with `--viewer-url` and bounded `--viewer-wait` (default15s, maximum60s). Transient HTTP failures are counted and reported; missing exact event evidence still fails. No backend payload validation changed.
+- Independent server-only checks PASS for MQTT3.1.1 and5 over plaintext/TLS; raw text, arbitrary JSON and empty payloads; exact UI incoming/outgoing ACK evidence; no reply loop; native reply-topic and binary correlation; controlled QoS1 wire DUP; anonymous/invalid password and CA/hostname rejection.
+- Command: `rtk python3 scripts/check_receipts.py --credentials esp32-test-kit/connection.txt --ca esp32-test-kit/ca.crt --viewer-url http://127.0.0.1:18081`.
+- Earlier Sprint5 broker restart/persistence gate is a separate unresolved check; this closeout does not mark it passed. Physical ESP32 return-path consumption is still for its developer to confirm in firmware logs.
+- Deployment operates through main Gitea pushes; server UI remains private. See `docs/BACKEND-SERVER.md`.
+
+Original paused-state record saved 2026-10-08 follows.
+
+## Objective
+
+Deploy the existing MQTT test Go backend on the server and configure Gitea CI/CD. User provided repository `ssh://git@git.my-rm.com:2222/Robot-Market/mqtt-5-test.git`, selected private UI access through SSH tunnel, and explicitly approved stopping the duplicate local observer `mqtt5-learning-observer-1`.
+
+## Boundaries and instructions
+
+- Project: `/home/ryan/Media/Code/Code/robot-market/test-mqtt-5`.
+- Production backend and other project files must remain untouched.
+- Prefix local shell commands with `rtk` per `/home/ryan/.codex/RTK.md`. Commands run remotely over SSH use standard Linux commands.
+- Use existing SSH key access; do not print passwords, tokens or private handoff files.
+- Keep raw payload observation and receipt ACKs enabled; user does not want schema/device/timestamp validation for these test messages.
+- No simulator should run continuously. Do not restart the stopped duplicate observer without user instruction.
+
+## Completed deployment
+
+- Initialized Git in the test project, branch `main`, remote `origin` set to the supplied Gitea URL. Repository was empty before the first push.
+- Pushed commits:
+  - `71ef622b4fc975f4ad78b216cfaaf79396d4c412`: backend source, server Compose/deployment script, Gitea workflow and runbook.
+  - `9a32f634d41f6f0d8d723d6345ba32024a82d7a2`: current MQTT readiness endpoint and server health check.
+- Last Git status before creating this file was clean. This handoff file is new and uncommitted; do not assume it is on Gitea.
+- Gitea workflow `.gitea/workflows/backend.yml`: test main PRs/pushes; deploy only main after tests, plus manual dispatch. Existing `ubuntu-latest` runner uses the server Docker daemon. No registry or deployment SSH secret needed.
+- Dockerfile has a `test` target running `go test ./...` and `go vet ./...`. Local race tests were also run.
+- Server runtime: `/opt/test-mqtt-5/backend`, Compose project `mqtt5-test-backend`, service `backend`, container `mqtt5-test-backend-backend-1`.
+- Last verified server image: `mqtt5-test-backend:2026-10-08_11-24-19-9a32f634d41f`, container healthy.
+- Server UI listens ONLY on `127.0.0.1:18080`; MQTT uses verified TLS to `109.122.252.129:8884`, client ID `rm5-server-observer`.
+- Private credentials copied server-side from existing broker deployment to `/opt/test-mqtt-5/backend/secrets/backend.json`; root:10001 mode0640, secrets directory root:10001 mode0750. Public CA copied to runtime `ca.crt`. Contents were not printed or committed.
+- Existing broker is separate in `/opt/test-mqtt-5/deploy`; its ports/config were not changed by this deployment task.
+- Runner `/opt/gitea-runner/config.yaml` now permits only the added backend bind path `/opt/test-mqtt-5/backend` in `container.valid_volumes`; deploy job requests it as `/runtime`. Prior config backed up with `.mqtt5-backup-<epoch>` suffix. Existing frontend mount unchanged. Runner was restarted successfully before pipeline dispatch.
+- Gitea runs 32 and 33 passed (database `action_run.status=1`). Run32 deployed initial commit; run33 deployed readiness correction. Tests/deploy job IDs 38/39 and40/41 respectively.
+- `server/deploy.sh` uses flock, validates Compose, backs up `.env`/Compose, deploys, waits up to60 seconds for healthy, restores previous release after failure. Images retained; no image prune.
+- `/healthz` checks current `client.Ready()` rather than relying on connection events remaining in bounded viewer history. Last HTTP result: `{"status":"ready"}`.
+- The server snapshot through the dedicated tunnel reported connected and only client ID `rm5-server-observer`.
+
+## Local observer and tunnels
+
+- The old observer was actually container `mqtt5-learning-observer-1`, running the test backend against the remote broker with client ID `rm5-docker-observer`. It caused duplicate ACKs during initial independent tests.
+- User explicitly approved stopping ONLY that container; `docker stop mqtt5-learning-observer-1` succeeded. No files or local learning broker were changed.
+- Docker simulator remained exited. Do not mistake Compose `ps` for the full Docker inventory: the duplicate observer belongs to another Compose project.
+- SSH tunnel started for verification: local `18081` -> server `127.0.0.1:18080`, exec session91813. It may still run, but session persistence across stopping the task is not guaranteed.
+- Earlier attempted tunnel local8080 -> server18080 is exec session32787; local8080 conflicted with the old observer, so do not rely on it. Check/clean up only these task-created SSH tunnels when resuming.
+- To open a fresh tunnel (use an available local port):
+
+```sh
+rtk ssh -N -L 18081:127.0.0.1:18080 -o ExitOnForwardFailure=yes root@109.122.252.129
+```
+
+Open http://127.0.0.1:18081. Runbook `docs/BACKEND-SERVER.md` also documents8080/8081 alternatives.
+
+## Verification already completed
+
+- Local full `GOCACHE=/tmp/mqtt5-go-cache go test -race ./...`: PASS.
+- Local Docker `--target test` (tests plus vet): PASS.
+- After readiness change, affected web/CLI/command race tests: PASS.
+- Dedicated readiness tests cover ready/unready independent of retained event history.
+- Deployment success and failed-update rollback were tested with mocked Docker: PASS. No intentional live failed deployment was performed.
+- Both actual Gitea pipelines passed; latest backend container healthy and `/healthz` ready.
+- Source staged before first push was scanned against known private MQTT passwords and excluded runtime directories. ESP32 kit/ZIP, local secrets/certs/.env, IDE data and private handoff doc were excluded from Git.
+- Latest server-only independent test passed plaintext MQTT3.1.1: text, arbitrary JSON, empty payload; device received ACKs; UI incoming/outgoing events matched; no extra reply loop.
+
+## Outstanding work — do not claim final verification complete
+
+The final independent four-combination check did NOT complete. After passing plaintext MQTT3.1.1, it hit an HTTP timeout while reading the server snapshot through the SSH tunnel during the next case. This is not proof of MQTT rejection; diagnose tunnel/server HTTP latency before classifying it. Test process exited1, exec session75330, and is finished.
+
+The checker was a temporary copy `/tmp/mqtt5-deployed-check.py` of `scripts/check_receipts.py` with only the viewer URL changed from local8080 to18081. It uses existing `esp32-test-kit/connection.txt` and `esp32-test-kit/ca.crt` without printing secrets.
+
+```sh
+rtk python3 /tmp/mqtt5-deployed-check.py --credentials esp32-test-kit/connection.txt --ca esp32-test-kit/ca.crt
+```
+
+Recreate the temporary copy if missing. Consider making viewer URL configurable and handling transient HTTP timeouts with bounded retries, but do not hide real failures. Original checker expects viewer8080. Test MQTT3.1.1/5 over plaintext/TLS, native Response Topic/binary Correlation Data, wire DUP retransmission and auth/TLS rejection, confirming exact outgoing ACK in SERVER UI. Previous Sprint5 checks passed these against the local observer, not this final server deployment.
+
+An attempted final local `docker ps` check hit automatic approval-review timeout: `The automatic permission approval review did not finish before its deadline`. It was not a policy rejection and was not retried because the user requested stopping. The earlier explicit `docker stop` succeeded; verify stopped state later.
+
+Remote broker was reported unhealthy by Docker during this task, while MQTT and backend readiness worked. This predates the backend deployment and may relate to earlier broker health/container-exec issues. Do not claim broker health is fixed or restart it without assessing impact. Sprint5's earlier restart/persistence verification gate remains separate and unresolved.
+
+## Resume checklist
+
+1. Read this file and `docs/BACKEND-SERVER.md`; inspect Git and current server/tunnel state because it may have changed.
+2. Confirm latest server container remains healthy and /healthz ready; confirm duplicate local observer remains stopped.
+3. Diagnose the HTTP/tunnel timeout and finish server-only independent receipt tests with correct viewer URL.
+4. If code changes, test affected packages and use the existing main CI/CD workflow; verify resulting run/deployment.
+5. Record final evidence and report deployed URL/tunnel command, CI/CD branch behavior, tests and any remaining limitations. Commit this handoff only if useful; it contains no passwords.
+6. Leave the user's other projects/brokers and credentials intact.
+
+## Operational commands on server
+
+```sh
+cd /opt/test-mqtt-5/backend
+docker compose --env-file .env -f compose.yaml ps
+docker compose --env-file .env -f compose.yaml logs -f backend
+docker compose --env-file .env -f compose.yaml restart backend
+```
+
+Gitea API requires authentication (anonymous status request returned401). Read-only status was obtained using the existing Gitea PostgreSQL container, selecting only this repository's run/job IDs, statuses and commit SHA; no tokens were obtained. Prefer normal authenticated Gitea UI/API access if available on resume.

docs/BACKEND-SERVER.md

@@ -43,3 +43,15 @@ docker compose --env-file .env -f compose.yaml up -d --force-recreate backend
 ```
 
 During deployments do not run a simultaneous manual rollback. Before changing broker credentials, update the server's private `secrets/backend.json` (root:10001, mode 0640, parent directory 0750) and restart backend. Do not rotate the broker's existing credentials as part of ordinary backend deployments.
+
+## Deployment verification — 2026-10-10
+
+The deployed observer was healthy and MQTT-ready. Independent device clients passed MQTT3.1.1 and5 over both plaintext and TLS, receiving receipt ACKs for raw text, arbitrary JSON and empty messages. Exact incoming/outgoing evidence was verified in the SERVER UI through the private SSH tunnel. Native response/correlation, controlled wire DUP and auth/TLS rejection checks passed. Duplicate local observer remained stopped. This is independent test-client evidence, not physical ESP32 firmware proof.
+
+For repeatable checks against a tunnel, use:
+
+```sh
+rtk python3 scripts/check_receipts.py --credentials esp32-test-kit/connection.txt --ca esp32-test-kit/ca.crt --viewer-url http://127.0.0.1:18081
+```
+
+The script allows bounded waiting for UI evidence (default15 seconds), reports transient HTTP failures and still fails if matching evidence never appears. The old tunnel timeout was not reproduced; no historical root cause is claimed. Broker restart/persistence verification remains a separate Sprint5 gate.

scripts/check_receipts.py

@@ -4,6 +4,7 @@ import ssl
 import struct
 import time
 import urllib.request
+import urllib.error
 import uuid
 from pathlib import Path
 
@@ -12,9 +13,13 @@ parser = argparse.ArgumentParser(description='Independent test-device/backend re
 parser.add_argument('--credentials', required=True, help='Private JSON handoff or ESP32 connection.txt file')
 parser.add_argument('--ca', required=True, help='Test broker public CA file')
 parser.add_argument('--host', default='109.122.252.129')
+parser.add_argument('--viewer-url', default='http://127.0.0.1:8080', help='Viewer base URL, including a private SSH tunnel port')
+parser.add_argument('--viewer-wait', type=float, default=15, help='Bounded seconds to wait for exact UI event evidence')
 parser.add_argument('--plaintext-port', type=int, default=1884)
 parser.add_argument('--tls-port', type=int, default=8884)
 args = parser.parse_args()
+if args.viewer_wait <= 0 or args.viewer_wait > 60:
+    parser.error('--viewer-wait must be in (0, 60]')
 HOST = args.host
 private = Path(args.credentials).read_text()
 if args.credentials.endswith('.json'):
@@ -116,14 +121,25 @@ def received(sock, version):
     return body[position:].decode()
 
 def viewer_has(topic, payload, kind):
-    until = time.monotonic() + 5
+    until = time.monotonic() + args.viewer_wait
+    unavailable = 0
     while time.monotonic() < until:
-        with urllib.request.urlopen('http://127.0.0.1:8080/api/snapshot', timeout=3) as response:
+        remaining = until - time.monotonic()
+        if remaining <= 0:
+            break
+        try:
+            with urllib.request.urlopen(args.viewer_url.rstrip('/') + '/api/snapshot', timeout=min(5, remaining)) as response:
                 snapshot = json.load(response)
+        except (TimeoutError, urllib.error.URLError):
+            unavailable += 1
+            time.sleep(min(.1, max(0, until-time.monotonic())))
+            continue
         if any(event.get('topic') == topic and event.get('kind') == kind and event.get('payload', {}).get('text', '') == payload for event in snapshot['events']):
+            if unavailable:
+                print('UI evidence confirmed after ' + str(unavailable) + ' transient HTTP failures')
             return
-        time.sleep(0.1)
-    raise RuntimeError('broker delivered but viewer did not show ' + kind)
+        time.sleep(min(.1, max(0, until-time.monotonic())))
+    raise RuntimeError('exact UI evidence missing: ' + kind + '; HTTP failures=' + str(unavailable))
 
 
 import hashlib