mqtt-5-learning/teaching/course-source/17a-advanced-devops-lifecycle.md

# Section 17A — Advanced DevOps service lifecycle

> [!toc]- Contents
>
> - [Purpose and boundaries](#purpose-and-boundaries)
> - [The service lifecycle](#the-service-lifecycle)
> - [1. Map ownership and failure domains](#1-map-ownership-and-failure-domains)
> - [2. Build trustworthy CI gates](#2-build-trustworthy-ci-gates)
> - [3. Package and promote artifacts](#3-package-and-promote-artifacts)
> - [4. Configure each environment safely](#4-configure-each-environment-safely)
> - [5. Release compatible services and schemas](#5-release-compatible-services-and-schemas)
> - [6. Deploy, observe, and recover](#6-deploy-observe-and-recover)
> - [7. Operate through incidents and change](#7-operate-through-incidents-and-change)
> - [Capstone exercise](#capstone-exercise)
> - [Self-check](#self-check)

## Purpose and boundaries

This lesson is for a backend engineer with junior-level DevOps experience who wants to own a service through its full delivery and operational lifecycle. It builds on the backend, simulator, PostgreSQL state, MQTT broker, and runbooks from earlier sections.

The goal is to practice production-grade reasoning in an isolated learning environment. Do not deploy to Robot Market production or connect real vending machines. Kubernetes is not required: use the lab's Docker Compose setup to practice the same lifecycle decisions, then compare how an orchestrator would implement them.

The lesson does not prescribe a CI vendor, cloud, or deployment platform. The practices apply across those choices. Each decision should name its owner, evidence, failure mode, and recovery action.

## The service lifecycle

**❓ Question:** What must be true before a code change can safely become a running service?

**💬 Answer:** The change needs reviewable source, automated checks, a known build artifact, environment-specific configuration, an explicit rollout, and evidence that the service works afterward. Operators also need a way to detect failure and recover without guessing which state was lost.

```mermaid
flowchart LR
    S[Source change] --> C[CI checks]
    C --> A[Versioned artifact]
    A --> P[Promote same artifact]
    P --> D[Deploy with configuration]
    D --> V[Verify health and behavior]
    V --> O[Observe service indicators]
    O --> R[Recover, roll forward, or roll back]
    R --> L[Review and improve]
    L --> S
```

Promotion means moving the same artifact through environments. Rebuilding separately for each environment makes it harder to know whether the tested bits are the deployed bits. Configuration and secrets change by environment; the application artifact should not.

## 1. Map ownership and failure domains

**❓ Question:** If the ingestion process restarts, which facts can it reconstruct, and which system owns them?

Start with a state inventory before designing deployment. A successful MQTT reconnection does not restore application data, and a database restart does not restore a broker's session state.

| State or responsibility | Owner in this course | Recovery evidence |
| --- | --- | --- |
| Durable commands, report results, deduplication | PostgreSQL | Query after application restart; replay a duplicate safely |
| MQTT subscriptions and broker session state | Broker, subject to session and persistence configuration | Session experiment and broker restart check |
| Pending callback work and local device view | Service process memory until explicitly persisted | Process restart shows what disappears |
| Desired configuration snapshot | Application database or retained MQTT state, as explicitly designed | Compare the authoritative revision after restart |
| Credentials and environment settings | Deployment configuration/secret source | Redacted effective configuration and rotation procedure |
| Build and release identity | Source revision and artifact registry | Map a running image digest back to source and pipeline result |

Draw the system's dependencies and failure domains: API, ingestion, PostgreSQL, broker, simulator/device, and deployment environment. For each, record what happens when it is unavailable, whether work queues or fails, and which owner responds.

Do not use “MQTT is up” or “the container is running” as a complete health statement. A process can run while subscriptions are missing, the database is unavailable, or accepted work is stuck.

## 2. Build trustworthy CI gates

**❓ Question:** Which checks are fast and repeatable on every change, and which require a controlled environment?

Organize checks by the evidence they provide:

| Gate | Example evidence | What it does not prove |
| --- | --- | --- |
| Formatting and static checks | `gofmt`, `go vet`, lint results | Correct runtime behavior |
| Unit tests | Contract, state transition, and retry behavior | Broker or database interoperability |
| Concurrency checks | `go test -race ./...` | Absence of all logical races or production overload |
| Integration tests | Isolated PostgreSQL and Mosquitto tests | Production network, credentials, or capacity |
| Acceptance checks | API request reaches simulator and accurate result is queryable | Safe deployment or production device compatibility |
| Image checks | Build, user, metadata, dependency and vulnerability review | Runtime configuration or readiness |

Make each gate deterministic where possible: pin tool and dependency versions, use synthetic data, isolate test infrastructure, set timeouts, and collect logs on failure. Keep broker/device acceptance checks separate from fast unit tests so their failures identify the right layer.

A green pipeline is evidence for the checks it ran, not a guarantee of safety. Record skipped checks and required human review explicitly. A failing quality or security gate should stop artifact promotion until the failure is understood or an authorized exception is documented.

## 3. Package and promote artifacts

**❓ Question:** How can an operator identify exactly which program is running and reproduce that artifact?

Build versioned service images from a pinned Go toolchain and dependencies. Prefer a multi-stage build that compiles in a build stage and copies only the required binary and runtime assets into the final image. Run as a non-root user, use a narrow writable filesystem, and include no source credentials, `.env` files, or lab password files in image layers.

Attach traceable metadata to each artifact: source revision, build job, dependency versions, and immutable image digest. A human-readable version tag helps people; the digest identifies the exact content. Promote the same digest from test to staging. Do not rebuild it with different build arguments to “make a staging image.”

Treat base-image and Go dependency updates as planned changes: review release notes, run the normal gates, scan for known vulnerabilities, and retain the prior deployable artifact for recovery. A scan finding needs triage based on affected code, exposure, and available fix; a scan alone is not a complete security review.

For this exercise, add a deployment definition for the learning services and inspect the resolved configuration before starting them. Confirm the final image user, entrypoint, signal behavior, mounted paths, and exposed ports. Keep the broker listener on loopback unless the experiment explicitly requires another network boundary.

## 4. Configure each environment safely

**❓ Question:** What should change between local, test, and staging environments, and what should stay identical?

The artifact stays identical. Broker addresses, database endpoints, resource limits, and secret references vary by environment. Use the configuration loader's validation before opening MQTT or database connections. Log a redacted configuration summary so an operator can diagnose the effective settings without revealing passwords, tokens, or private keys.

Use distinct identities for API, ingestion, broker, database, and deployment automation. Grant each only the access it needs. Supply secrets at runtime through the environment's secret mechanism; never place real values in source, image layers, test output, or resolved configuration shared in an issue.

Plan credential rotation as an operational change: issue a replacement, allow the intended overlap, update consumers, verify successful connections and authorization, then revoke the old credential. Define how to recover if one process fails to adopt the new value. For broker ACL changes, test both allowed and denied topics before promotion.

Configuration validation should distinguish a missing or invalid setting from a transient dependency outage. Invalid configuration should fail clearly before serving traffic; temporary broker/database failure should follow the documented retry and readiness policy.

## 5. Release compatible services and schemas

**❓ Question:** How do you roll out a database change when old and new service versions may run at the same time?

Use an expand/migrate/contract sequence:

1. **Expand:** add a schema capability that old code can tolerate, such as a nullable column or new table.
2. **Migrate:** deploy code that can work with the old and expanded schema; backfill data in bounded, observable batches if needed.
3. **Switch:** verify the new code path and data before relying on it.
4. **Contract:** remove old schema only after all old binaries and rollback paths no longer need it.

Database rollback is not always the inverse of a schema migration. Destructive changes or new data may make old code incompatible. State whether a release can roll back, needs a forward fix, or requires restoring data; rehearse the selected path with synthetic data.

Use immutable artifacts and explicit release records. A release should identify source revision, image digests, schema version, configuration change, approvals, rollout time, verification evidence, and operator. Keep changes small enough to identify which change caused an incident.

## 6. Deploy, observe, and recover

**❓ Question:** What signals show that a new version is ready to receive work, and what should happen to accepted work during termination?

Define health states per process:

- **Liveness:** the process can make progress; a failed check may justify restarting it.
- **Readiness:** the process is prepared to receive its assigned traffic or messages; a failed check should remove it from new work where the platform supports that behavior.
- **Dependency status:** broker and database reachability, reported separately so an operator can see which dependency is failing.

Readiness should reflect required initialization, not merely an open TCP port. For ingestion, verify required subscriptions and durable-store availability before claiming readiness. During shutdown, stop admitting new work, drain accepted work within a deadline, preserve work that cannot finish, and disconnect cleanly. Document what happens when the deadline expires.

Roll out progressively where the platform permits it. Set a small release scope, monitor error and latency indicators, and define pause/rollback thresholds before deployment. For multiple ingestion replicas, include shared-subscription behavior, duplicate handling, ordering assumptions, and durable idempotency in the rollout plan.

Define useful service indicators: successful API requests, MQTT reconnects, subscription readiness, queue depth and age, command completion/unknown outcomes, report processing lag, database errors, and resource pressure. Avoid unbounded device or command identifiers as metric labels. Correlate logs with stable IDs while redacting secrets and untrusted payloads.

For state recovery, define example recovery-point and recovery-time objectives (RPO/RTO). Backups only count when restore works: rehearse PostgreSQL restore and inspect broker persistence/session behavior separately. Record which data can be rebuilt, which work may replay, and how duplicate effects are prevented.

## 7. Operate through incidents and change

**❓ Question:** During an incident, what should the responder establish before restarting services or changing configuration?

Follow a repeatable sequence:

1. **Detect:** identify the user-visible symptom and the alert or report that exposed it.
2. **Scope:** determine affected process, devices, commands/reports, time window, and release version.
3. **Stabilize:** stop harmful retries or further rollout; preserve evidence and durable work.
4. **Diagnose:** compare application, broker, database, network, and device evidence. Separate transport acknowledgement from business completion.
5. **Recover:** choose restart, rollback, forward fix, replay, or restore based on state ownership and data compatibility.
6. **Verify:** confirm service indicators recover and affected commands/reports reach a known state.
7. **Learn:** record timeline, cause, contributing conditions, and a follow-up change with an owner.

Runbooks should state commands in context, expected evidence, access required, risks, and restoration steps. Keep incident notes factual and avoid credentials or private payloads. Schedule restore drills, dependency upgrades, certificate rotation, and access reviews rather than waiting for an incident to reveal they are overdue.

## Capstone exercise

Prepare and release a simulated change that adds a harmless command result field or compatible database column. Use only the isolated lab.

Before starting, predict the behavior if the API is restarted during rollout, PostgreSQL is unavailable, a new image fails readiness, and rollback is requested after the schema migration. Then:

1. Draw the pipeline and state-ownership diagram.
2. Define automated gates and the evidence each gate produces.
3. Build and identify one immutable service artifact.
4. Deploy it with lab-only configuration and verify readiness, a command flow, and report persistence.
5. Inject one failure, pause rollout, recover using the documented path, and verify durable state.
6. Record the release and incident evidence, including what remains unverified outside the lab.

**Deliverables:** pipeline diagram, release checklist, artifact identity, redacted deployment configuration, rollout/rollback decision, recovery record, and a short incident review.

## Self-check

1. Why is promoting one image digest safer than rebuilding for each environment?
2. Why can a successful database backup still leave recovery unproven?
3. Which work should an ingestion readiness check confirm before the service accepts messages?
4. Why might rolling back a binary be unsafe after a database migration?
5. Which evidence distinguishes a broker delivery acknowledgement from a completed vending command?

**Answer guide:** A digest identifies the same tested bits across environments; a restore drill proves backup usability; readiness checks required subscriptions and storage dependencies; schema/data changes can make old code incompatible; a device/application result and persisted command state provide business evidence beyond MQTT acknowledgement.