@ybotok/loam 0.1.0-beta.3 → 0.2.0-alpha.2

package/examples/docs/services/platform/identity-service/runbook.md

@@ -0,0 +1,95 @@
+---
+service: identity-service
+status: draft
+owner: platform-identity
+---
+
+# identity-service — runbook
+
+Written for whoever is holding the pager at 03:00, so every step below is one a
+stranger to this service can carry out. `loam list` tracks only that this file
+exists; nothing in it is checked, which means the burden of keeping it true is on
+the same PR that changes the behaviour it describes.
+
+Read this together with `arch.spec.md`: the two incidents below are the failure
+modes ARCH-KEY-ROTATION and ARCH-INTROSPECT-CACHE exist to prevent, and each
+mitigation here is the manual version of a promise the service is supposed to keep
+on its own.
+
+## Deploy
+- Pipeline: `.github/workflows/release.yml` → `deploy-prod`, gated on a manual approval.
+- Config: `config/prod.yaml`; the signing key material is never in it — the service
+  reads keys from the key store at startup and gets its store credentials from Vault
+  (`secret/identity-service`).
+- Rollout is one instance at a time, and the gate is readiness (below). A key store
+  the new revision cannot reach fails readiness and stalls the rollout, which is the
+  intended outcome: half a fleet unable to sign is worse than a paused deploy.
+
+## Health
+- Liveness: `GET /healthz` — the process answers. Nothing behind it is checked, so
+  this never restarts an instance for a dependency's fault.
+- Readiness: `GET /readyz` — the active signing key is loaded and the key store
+  answers. An instance that is up and cannot sign must take no traffic; it would
+  turn every issuance into a 5xx and burn `token_issue_5xx` for a fault that a
+  rolling restart fixes.
+- SLIs and the alert are declared in `health.yaml`; the arch requirement covering
+  them is ARCH-ISSUE-HEALTH.
+
+## Common incidents
+
+### A rotation promoted a key the other instances cannot read
+**Symptom.** Token issuance is fine, and every other service starts rejecting
+tokens: 401s spread across the fleet within seconds of a rotation job running,
+while `token_issue_5xx` stays quiet because *this* service is healthy. The
+consumers are refusing on an introspection verdict of `active: false` — freshly
+minted tokens that this service itself will not vouch for.
+
+**Cause.** The new key was promoted to active before every instance could read it
+from the key store, or the previous key was retired inside one token lifetime.
+Either breaks the overlap ARCH-KEY-ROTATION requires: the instance that mints and
+the instance that answers introspection are different processes, and only the key
+store makes them agree.
+
+**Mitigation.**
+1. Demote to the previous key: `identityctl keys promote --key <previous-kid>`.
+   Tokens already signed with the bad key keep reading inactive; they expire within
+   a lifetime and there is nothing to do about them.
+2. Confirm the key store holds both kids, and that a *different* instance than the
+   one you are on can verify a token minted with the new kid, before promoting
+   anything again.
+3. Do not "fix" it by shortening the token lifetime — that widens the outage.
+
+**Prevention.** Write, wait one full token lifetime, then promote. The rotation job
+does this on its own; a manual promotion is the way it gets skipped.
+
+### The introspection cache stampeded
+**Symptom.** The key store's connection pool saturates and `token_issue_latency_p99_ms`
+climbs past its SLO — issuance signs from the same store, so a flood of introspection
+misses shows up on the issuance SLI even though issuance is not what changed. Usually
+right after a cache restart, a failover, or a mass revocation that cleared many
+entries at once.
+
+**Cause.** Every introspection missed at the same moment and each miss became a
+key store read. The service stays *correct* through this — ARCH-INTROSPECT-CACHE
+requires a miss to produce the same verdict — so what is failing is latency, and
+the cure is to protect the store rather than to bypass the cache.
+
+**Mitigation.**
+1. Check the miss rate and the store's pool metrics before touching anything; if
+   the pool is not saturated, this is a different incident.
+2. Raise the per-key single-flight window (`introspection.singleflight_ms`) so that
+   concurrent misses for one token collapse into one store read.
+3. If the store is already saturated, shed introspection load at the gateway before
+   restarting the cache — a cold restart replays the stampede.
+4. Never disable the cache to "get more capacity": every request then reaches the
+   store.
+
+**Prevention.** Jittered TTLs and single-flight on miss, both of which the cache
+layer owns; a mass revocation should delete entries in batches rather than flush.
+
+## Escalation
+- On-call channel: `#identity-oncall`; the alert `token_issue_5xx` pages it directly.
+- Because every service depends on this one, an incident here is announced in
+  `#fleet-incidents` as soon as it is confirmed, before it is understood — the
+  consumers are the ones who will see the symptom first.
+- Owning team: platform-identity.

package/examples/docs/services/platform/identity-service/spec.md

@@ -0,0 +1,119 @@
+---
+service: identity-service
+status: draft
+owner: platform-identity
+sources:
+  - internal/tokens/
+  - cmd/
+---
+
+# identity-service
+
+Issues the access tokens every other service in this fleet presents, and answers the
+one question those services cannot answer for themselves: is this token still good?
+This is the **living spec** — the complete current state, the operation on its way
+out included. Feature deltas (ADDED/MODIFIED/REMOVED requirements) merge in here on
+`loam archive`.
+
+`status: draft`, and no digest anywhere in the frontmatter: nothing in this directory
+has been vouched. `sources` names the two trees in the service's own repository these
+requirements were written from, so that when `loam validate` runs inside that repo it
+can say whether the code has moved since a human last read it (`sources.current` /
+`sources.stale` / `sources.unvouched`). Writing `status: verified` by hand would be a
+claim about bytes nobody hashed — the exact forgery `loam vouch` exists to make
+impossible, which is why it and it alone writes `sources_digest`, `content_digest`,
+`last_verified` and `vouched_by`.
+
+One requirement below, `Answer legacy token validation`, deliberately draws
+`api.requirement-deprecated`: it governs only an operation this service's OpenAPI
+marks `deprecated: true`. That warning is the point — it is how a fleet sees, from
+`loam validate` rather than from a migration wiki, which promised behaviour is being
+retired and has not finished going.
+
+## Requirements
+
+### Requirement: Issue an access token
+
+Requirement-ID: IDN-ISSUE
+The service SHALL issue a signed, expiring access token to a client that presents
+valid credentials, and SHALL refuse every other caller with one indistinguishable
+refusal — a response that reveals which half of the credential was wrong turns the
+token endpoint into a client-enumeration oracle.
+
+Operations: issueToken
+Capability: identity/tokens
+Realizes: identity/tokens#IDENTITY-ONE-REFUSAL
+
+#### Scenario: A valid client credential is exchanged for a token
+- **Given** a registered client whose secret has not been revoked
+- **When** it posts its credentials to the token endpoint
+- **Then** a token signed by the currently active key is returned
+- **And** the response states the expiry the token was minted with
+
+#### Scenario: An unknown client is refused
+- **Given** a client id that was never registered
+- **When** it posts credentials to the token endpoint
+- **Then** the request is refused with 401 and no token is minted
+- **And** the body says only that authentication failed
+
+#### Scenario: A revoked secret is refused identically
+- **Given** a registered client whose secret was revoked an hour ago
+- **When** it posts its old credentials to the token endpoint
+- **Then** the request is refused with 401
+- **And** the response is byte-identical to the unknown-client refusal, so the two cases cannot be told apart from outside
+
+### Requirement: Introspect a token
+
+Requirement-ID: IDN-INTROSPECT
+The service SHALL report, for any token a caller presents, whether it is active now
+and what it grants — computed from revocation state, never from the token's signature
+alone, which stays valid for the whole of a revoked token's remaining lifetime. Only
+callers holding `service/tokens:introspect` SHALL be answered: an introspection
+endpoint open to anything on the network is a token-validity oracle for stolen
+tokens.
+
+Operations: introspectToken
+Requires: service/tokens:introspect
+Capability: identity/tokens
+Realizes: identity/tokens#IDENTITY-REVOKE-EVERYWHERE
+
+#### Scenario: An active token is described to its caller
+- **Given** a token issued eight minutes ago with a thirty-minute lifetime
+- **When** order-service introspects it
+- **Then** the verdict is active, with the client it was issued to and the scopes it carries
+
+#### Scenario: A revoked token reads inactive while its signature still verifies
+- **Given** a token whose client was revoked after issuance
+- **When** the token is introspected
+- **Then** the verdict is inactive
+- **And** no subject or scope is disclosed for it
+
+#### Scenario: A caller without the introspection permission is told nothing
+- **Given** a caller whose own token does not hold `service/tokens:introspect`
+- **When** it introspects somebody else's token
+- **Then** the request is refused before any verdict is computed
+- **And** the refusal is the same whether or not the presented token exists, so a caller cannot use the refusal itself to test tokens
+
+### Requirement: Answer legacy token validation
+
+Requirement-ID: IDN-VALIDATE-LEGACY
+`validateToken` is superseded by `introspectToken` and is kept alive for exactly one
+reason: payment-service still calls it, and the fleet map says so with an op-linked
+edge. Until that consumer migrates, the service SHALL answer the legacy endpoint with
+the same verdict introspection would give — never a cheaper local signature check —
+and SHALL NOT extend it: no new caller, no new field, no new status code. The
+endpoint is retired by a feature delta that removes the operation and this
+requirement together, not by quietly deleting either.
+
+Operations: validateToken
+
+#### Scenario: The legacy verdict agrees with introspection
+- **Given** a token that introspection reports as active
+- **When** the same token is checked through the legacy endpoint
+- **Then** it is reported valid, with the same expiry introspection gave
+
+#### Scenario: A revoked token is not reported valid by the legacy path
+- **Given** a token whose client was revoked after issuance
+- **When** the legacy endpoint is asked about it
+- **Then** it is reported invalid
+- **And** the answer came from revocation state, not from the token's signature

package/examples/docs/services/platform/notification-service/asyncapi.yaml

@@ -0,0 +1,199 @@
+# The async contract — read-only. Every operation here carries `action: receive`:
+# this service produces no message and exposes no HTTP operation, so the whole of
+# what it promises the fleet is what it does with three topics.
+#
+# AsyncAPI 3.0 only. Its operations are named top-level objects carrying
+# `action: send|receive`, which is what makes a message's direction legible;
+# 2.x spells the same thing as publish/subscribe under a channel, where the two
+# words are ambiguous about whose perspective they take — and a 2.x document
+# reads here as a contract with no messages at all.
+#
+# The name is the spine: the landscape edge's `metadata { consumes }`, the
+# requirement's `Consumes:` line and `name` below are one name spelled three
+# times, and they must be identical. For a consumer the line that grades THIS
+# file is the requirement's — a `Consumes:` naming a message no operation here
+# receives is `spec-event.message-undefined`. The edge's copy answers the other
+# question, the one no single repository can: whether anybody publishes the
+# name at all. Deleting a `Consumes:` line therefore does not move a check onto
+# the landscape, it removes one.
+#
+# WHY THE PAYLOADS ARE HERE AT ALL, and why one of them is different in kind:
+#
+#   * `order.OrderPlaced` and `payment.PaymentCaptured` are produced INSIDE the
+#     fleet, so order-service's and payment-service's own asyncapi.yaml are the
+#     authoritative contracts. What is restated below is the subset this consumer
+#     actually reads — enough that the handler is codeable from this file, not a
+#     second source of truth for the event.
+#
+#   * `crm.CustomerUpdated` is produced by `salesforce`, an `#external` element in
+#     the landscape. There is no services/salesforce/, there never will be, and so
+#     no producer contract exists anywhere in this docs repo. That is a legal
+#     shape — loam recognizes an external producer and does not report
+#     `spine.message-unproduced` — but it shifts the contract question onto the
+#     only file that can answer it: this one. `spine.message-external` (warn)
+#     fires for exactly as long as this document defines no shape for the message,
+#     and stops the moment it does. The payload below is therefore not a
+#     convenience copy; it IS the fleet's contract for that message, and the
+#     reason the finding is absent from this service's validate output.
+asyncapi: 3.0.0
+info:
+  title: notification-service events
+  version: "1.0"
+
+channels:
+  orderEvents:
+    address: order.events.v1
+    description: Order lifecycle events, produced by order-service
+    messages:
+      OrderPlaced:
+        $ref: '#/components/messages/OrderPlaced'
+  paymentEvents:
+    address: payment.events.v1
+    description: Payment lifecycle events, produced by payment-service's outbox relay
+    messages:
+      PaymentCaptured:
+        $ref: "#/components/messages/PaymentCaptured"
+      PaymentRefunded:
+        $ref: "#/components/messages/PaymentRefunded"
+  crmEvents:
+    address: crm.events.v1
+    description: Customer master-data events, produced by salesforce (outside the fleet)
+    messages:
+      CustomerUpdated:
+        $ref: '#/components/messages/CustomerUpdated'
+
+# One operation per message rather than one per channel, so that adding a second
+# message to a topic later is an addition here instead of a silent widening: an
+# operation that lists no messages applies to EVERY message of its channel, and
+# would start receiving contracts nobody read.
+operations:
+  receiveOrderPlaced:
+    action: receive
+    channel:
+      $ref: '#/channels/orderEvents'
+    messages:
+      - $ref: '#/channels/orderEvents/messages/OrderPlaced'
+  receivePaymentCaptured:
+    action: receive
+    channel:
+      $ref: '#/channels/paymentEvents'
+    messages:
+      - $ref: '#/channels/paymentEvents/messages/PaymentCaptured'
+  receiveCustomerUpdated:
+    action: receive
+    channel:
+      $ref: '#/channels/crmEvents'
+    messages:
+      - $ref: '#/channels/crmEvents/messages/CustomerUpdated'
+  receivePaymentRefunded:
+    action: receive
+    channel:
+      $ref: "#/channels/paymentEvents"
+    messages:
+      - $ref: "#/channels/paymentEvents/messages/PaymentRefunded"
+
+components:
+  messages:
+    OrderPlaced:
+      # Namespaced by the domain that OWNS the message, never by the service
+      # reading it: `order.` says order-service is the one producer, and two
+      # services declaring they send this name would be
+      # `asyncapi.message-contested` — which contract a consumer reads must not
+      # be a coin flip.
+      name: order.OrderPlaced
+      title: Order placed
+      payload:
+        type: object
+        # A payload declaring no shape — absent, or a bare `type: object` — is
+        # `asyncapi.payload-undescribed`: the name joins the spine while nothing
+        # says what a handler would read. loam probes for presence and never
+        # reads a field, which is what keeps a later move to Avro a
+        # `schemaFormat:` line here and no change in loam.
+        required: [ orderId, customerId, placedAt ]
+        properties:
+          orderId:
+            type: string
+            format: uuid
+            description: The idempotency key for the confirmation — one send per orderId
+          customerId:
+            type: string
+            format: uuid
+          placedAt:
+            type: string
+            format: date-time
+    PaymentCaptured:
+      name: payment.PaymentCaptured
+      title: Payment captured
+      payload:
+        type: object
+        required: [ paymentId, orderId, capturedAmount, currency, capturedAt ]
+        properties:
+          paymentId:
+            type: string
+            format: uuid
+          orderId:
+            type: string
+            format: uuid
+            description: What joins a receipt to the order the customer was told about
+          capturedAmount:
+            type: integer
+            description: minor units, restated on the receipt exactly as captured
+          currency:
+            type: string
+            pattern: '^[A-Z]{3}$'
+          capturedAt:
+            type: string
+            format: date-time
+    CustomerUpdated:
+      name: crm.CustomerUpdated
+      title: Customer contact details updated
+      payload:
+        type: object
+        # `version` is load-bearing rather than descriptive: the CRM redelivers
+        # and reorders, and the "out-of-order update is ignored" scenario in
+        # spec.md is only implementable because this field is in the contract.
+        # Nothing outside this document defines it.
+        required: [ customerId, version, updatedAt ]
+        properties:
+          customerId:
+            type: string
+            format: uuid
+          version:
+            type: integer
+            description: Monotonic per customer; an update at or below the stored version is
+              dropped
+          email:
+            type: string
+            format: email
+          phone:
+            type: string
+            description: E.164; absent when the customer has no phone on record
+          updatedAt:
+            type: string
+            format: date-time
+    PaymentRefunded:
+      name: payment.PaymentRefunded
+      title: Payment refunded
+      payload:
+        type: object
+        required:
+          - paymentId
+          - orderId
+          - refundedAmount
+          - currency
+          - refundedAt
+        properties:
+          paymentId:
+            type: string
+            format: uuid
+          orderId:
+            type: string
+          refundedAmount:
+            type: integer
+            description: minor units
+          currency:
+            type: string
+            pattern: ^[A-Z]{3}$
+          refundedAt:
+            type: string
+            format: date-time

package/examples/docs/services/platform/notification-service/health.yaml

@@ -0,0 +1,35 @@
+# Service health model. Targets are authored; the dependency graph comes from C4
+# — this service's own model.likec4, which carries everything the service touches.
+# planned: `loam health` composes these across the fleet -> cascade / blast-radius.
+#
+# `loam validate` reads only ids out of this file: the `name` of each SLI and
+# alert, and each dependency's `service:` id.
+#
+# THERE ARE NO `slis:` AND NO `alerts:` HERE, and the absence is the point of this
+# example. `health.uncovered` asks that every declared signal be named by a
+# `Covers: sli:<id>` / `Covers: alert:<id>` line in the service's living
+# arch.spec.md — so declaring nothing is what makes an arch.spec.md genuinely not
+# owed, and the check has nothing to fire on rather than being suppressed. The
+# honest reading of this file today is "nobody has written this service's SLOs
+# yet": consumer lag and provider failure rate are the two obvious ones, and the
+# day either is declared here it also owes an arch requirement covering it. An
+# absent file would say the same thing more quietly and lose the `checks:` and
+# `dependencies:` below, which on-call reads first.
+service: notification-service
+status: draft
+# Probes, not fleet operations: they carry no operationId, no requirement governs
+# them and no service calls them, which is exactly why they live here and not in
+# an openapi.yaml this service does not owe. Nothing below is read by any check —
+# `checks:` is authored prose for whoever is holding the pager.
+checks:
+  liveness: GET /internal/health/liveness
+  readiness: GET /internal/health/readiness    # consumer group joined + both stores reachable
+dependencies:
+  # Resolved against services/notification-service/model.likec4 and never the
+  # landscape (`health.dependency-unmodelled` when nothing there answers to the
+  # name). The delivery log and the contact store are absent on purpose: both
+  # are nested inside the service element and share its lifecycle, so they are
+  # blast radius rather than dependency — identity-service's health.yaml states
+  # the same rule for its two stores.
+  - service: kafka
+    criticality: critical        # no broker -> no events -> nothing to notify anyone about

package/examples/docs/services/platform/notification-service/model.likec4

@@ -0,0 +1,118 @@
+// Per-service C4. Written by an agent from the `loam adopt` brief, then curated.
+//
+// Every .likec4 file is parsed ALONE, so the kinds this document uses have to be
+// declared in this document: a `specification` block here is not a duplicate of
+// the landscape's, it is what makes this file readable without it.
+specification {
+  element softwareSystem
+  element container
+  element database
+  // Declared because the consumption edges below point at CHANNELS rather than
+  // at the broker. Three edges into one `kafka` box would say only "talks to
+  // kafka"; three edges into three topics say which contracts this service is
+  // standing on, and they line up one-for-one with the channels in
+  // asyncapi.yaml.
+  element topic {
+    style {
+      shape queue
+    }
+  }
+}
+
+model {
+  notificationService = softwareSystem 'notification-service' {
+    description 'Turns fleet events into customer email and SMS'
+    // The binding, not the title, is what ties this element to
+    // services/notification-service/. With only a title, renaming the box in a
+    // diagram silently detaches every join that runs through it — health.yaml's
+    // dependencies, `Covers:` entries, the landscape's service checks.
+    metadata {
+      service 'notification-service'
+    }
+
+    consumer = container 'consumer' {
+      description 'Kafka consumer group; one handler per message name'
+      technology 'Go'
+    }
+    sender = container 'sender' {
+      description 'Renders a template and hands it to the email/SMS provider'
+      technology 'Go'
+    }
+    // Both stores are nested INSIDE the service and neither is drawn beside it
+    // in the landscape: a store with one consumer drawn as a fleet-level peer
+    // reads as a system other services may depend on, which is what
+    // `landscape.datastore-private` pushes back on. Neither appears in
+    // health.yaml's dependencies either — they share this service's lifecycle
+    // rather than being something it reaches across a boundary.
+    deliveryLog = database 'delivery-log' {
+      description 'One row per (message id, channel) — the idempotency key that makes redelivery a no-op — carrying the orderId and customer each send was addressed to, so a receipt can find its order and a capture that arrives first has somewhere to wait'
+      technology 'PostgreSQL'
+    }
+    // The state NTF-CONTACT is entirely about, and it cannot be a corner of the
+    // delivery log: that table is keyed by message and holds nothing per
+    // customer, so it has no place to put a `version` to compare an update
+    // against. Without this element the spec promises a versioned copy of every
+    // customer address that the model gives nowhere to live — a divergence no
+    // loam check catches, because loam grades that health.yaml's dependencies
+    // resolve here and never that the requirements do.
+    contacts = database 'contacts' {
+      description 'The copy of each customer email and phone this service sends to, with the CRM version that decides whether an arriving update is newer'
+      technology 'PostgreSQL'
+    }
+
+    consumer -> deliveryLog 'Claims a delivery before sending'
+    consumer -> contacts 'Applies a CRM update only when its version is newer'
+    consumer -> sender 'Hands off a rendered notification'
+    sender -> contacts 'Reads the address a notification goes to'
+    sender -> deliveryLog 'Records delivered / failed'
+  }
+
+  // Everything the service touches belongs in its own model, cross-boundary
+  // dependencies included — the landscape carries only the boundary crossing.
+  // health.yaml names `kafka` as a critical dependency and that id has to resolve
+  // HERE, by element id, `metadata { service }` binding or title
+  // (`health.dependency-unmodelled` otherwise); and a model with elements and no
+  // edges beside a health.yaml that declares dependencies is `c4.no-relationships`
+  // — a service that reaches nothing is almost never true.
+  kafka = softwareSystem 'kafka' {
+    description 'Event backbone (platform)'
+    orderEvents = topic 'order.events'
+    paymentEvents = topic 'payment.events'
+    crmEvents = topic 'crm.events'
+  }
+
+  // The second thing this service reaches across its own boundary, and the one
+  // the runbook spends an incident section on: while the provider is down or
+  // throttling, nothing is delivered and there is no fallback path. It belongs
+  // here for the same reason payment-service's model draws `stripe` — a model
+  // that stopped at the broker would show a worker whose only failure mode is
+  // lag, which is the half of the picture the on-call graphs already give you.
+  provider = softwareSystem 'notification-provider' {
+    description 'Third-party email and SMS gateway — quota-limited, and outside this fleet'
+  }
+
+  // No `metadata { consumes }` on these edges, and adding it would change
+  // nothing: loam reads the event spine off `architecture/landscape.likec4`
+  // alone, never off a service model. What resolves a consumed name against
+  // asyncapi.yaml is each requirement's `Consumes:` line
+  // (`spec-event.message-undefined` when one names a message no `action:
+  // receive` operation declares); what the landscape settles is the other
+  // half — that somebody, inside the fleet or tagged #external, publishes it.
+  // These three edges are here for the picture only, and they earn it: this
+  // model, the three channels in asyncapi.yaml and the three topics the runbook
+  // names agree, or one of the three is out of date.
+  // Drawn topic → consumer, the direction the landscape uses and the one the
+  // event spine requires there: `consumes` binds the edge's TARGET, so the same
+  // edge drawn the other way round binds to nothing at all.
+  kafka.orderEvents -> notificationService.consumer 'OrderPlaced triggers the confirmation'
+  kafka.paymentEvents -> notificationService.consumer 'PaymentCaptured triggers the receipt'
+  kafka.crmEvents -> notificationService.consumer 'CustomerUpdated refreshes contact details'
+
+  notificationService.sender -> provider 'Sends email and SMS, and backs off on 429'
+}
+
+views {
+  view notification_service of notificationService {
+    include *
+  }
+}

package/examples/docs/services/platform/notification-service/runbook.md

@@ -0,0 +1,48 @@
+---
+service: notification-service
+status: draft
+owner: growth-team
+---
+
+# notification-service — runbook
+
+A consumer-only worker: it serves no traffic, so nothing degrades when it stops — it falls
+behind. Every incident below is a lag or a send failure, never a request error.
+
+## Deploy
+- Pipeline: `.gitlab-ci.yml` -> `deploy:prod` (manual gate).
+- Config: `config/`; provider credentials via Vault path `secret/notification-service`.
+- Rolling deploys rebalance the consumer group: expect a lag spike for the length of one
+  rebalance, and do not roll again until it has drained.
+
+## Health
+- Liveness: `GET /internal/health/liveness`
+- Readiness: `GET /internal/health/readiness` (consumer group joined + both stores reachable)
+- There are no SLOs yet — see the comment at the top of `health.yaml`.
+
+## Common incidents
+
+### Consumer lag on one topic
+Symptom: notifications arrive late; `kafka_consumergroup_lag` climbs on `order.events.v1`,
+`payment.events.v1` or `crm.events.v1`.
+- Check which partition is behind — a single hot partition is a slow handler, all of them is
+  the provider (below) or the delivery log.
+- Redelivery is safe on every topic, for two different reasons: `order.events.v1` and
+  `payment.events.v1` because the delivery log is keyed on `(message id, channel)` and a
+  replayed partition sends nothing twice, `crm.events.v1` because an update at or below the
+  `version` already in the contact store is dropped. Resetting an offset FORWARD is safe on
+  none of them — it drops confirmations nobody will re-emit.
+
+### Provider throttling
+Symptom: the sender's 429 rate rises and lag follows it on every topic at once.
+- The sender backs off and retries; do not scale the consumer up, which only queues more
+  work behind the same provider quota.
+- Confirm the quota with the provider before raising concurrency. Sending faster than the
+  quota converts throttling into hard rejections, and a rejected send is a customer who was
+  never told.
+- A held receipt (a capture whose order has not arrived — `NTF-RECEIPT`) looks identical in
+  the lag graph. Check the held-receipt count before treating it as a provider problem.
+
+## Escalation
+On-call channel: `#growth-oncall`. Provider incidents escalate to the vendor's status page
+first — this service has no way to deliver anything while the provider is down.