@ybotok/loam 0.1.0-beta.3 → 0.2.0-alpha.13

package/examples/docs/services/platform/identity-service/arch.spec.md

@@ -0,0 +1,120 @@
+---
+service: identity-service
+status: draft
+owner: platform-identity
+sources:
+  - internal/tokens/
+  - cmd/
+---
+
+# identity-service — architecture spec
+
+The **living architecture spec**: the obligations no business scenario was ever going
+to carry. Nothing in `spec.md` mentions key rotation or a cache TTL, because a caller
+exchanging credentials for a token cannot observe either — and those are exactly the
+places a coding agent cuts a corner unless the obligation is written where a test can
+be generated from it.
+
+Same grammar as `spec.md`, and one line more: `Covers:` is the architecture analog of
+`Operations:`. Where a business requirement declares the operations it governs, an
+architecture requirement declares the model objects its scenarios exercise — a C4
+element id, an edge `source -> target`, or a health signal `sli:<id>` / `alert:<id>`.
+`loam validate` grades the join in both directions: an entry that resolves to nothing
+is `covers.unknown` (a mistyped id silently costs exactly the coverage it was written
+for), and an SLI or alert in `health.yaml` that no requirement here covers is
+`health.uncovered`. Every id below comes from `model.likec4` or `health.yaml` in this
+directory; no entry names another service, because everything this service touches is
+inside its own boundary. The element ids carry the full `marketplace.identityService.`
+path because that is what the ids ARE: `model.likec4` extends the element the fleet map
+declares inside the `marketplace` group, so a container it adds has no shorter spelling.
+SCHEMA.md, "Two shapes of a service model", has the rule.
+
+## Requirements
+
+### Requirement: Signing keys rotate without invalidating live tokens
+
+Requirement-ID: ARCH-KEY-ROTATION
+The service SHALL keep the previous signing key verifiable for at least one full
+token lifetime after promoting a new one, and SHALL never sign with a key that is
+not already readable, from the key store, by every instance that answers
+introspection. Rotation that flips both halves at once invalidates every token in
+flight — an outage of the whole fleet, produced by a routine maintenance job, which
+is why the overlap is a requirement and not a runbook note. The second half is the
+one a single-instance test never catches: the signer and the verifier are different
+processes reading the same store, and a key promoted before it lands there makes
+one instance mint tokens the others report inactive.
+
+Covers: marketplace.identityService.keyStore, marketplace.identityService.tokens -> marketplace.identityService.keyStore
+
+#### Scenario: A rotation keeps tokens minted a minute earlier verifiable
+- **Given** the active key K1 and tokens issued under it with thirty minutes of life left
+- **When** K2 is promoted to active
+- **Then** new tokens are signed with K2
+- **And** tokens signed with K1 keep verifying until the last of them has expired
+
+#### Scenario: A key is readable by every instance before it signs anything
+- **Given** a newly generated key that is not yet in the key store the other instances read
+- **When** the rotation job runs
+- **Then** the key is written to the key store first and only then promoted to active
+- **And** no token is ever signed with a key another instance could not verify
+
+#### Scenario: Rotation is refused while the key store is unreachable
+- **Given** the key store is unavailable
+- **When** the rotation job runs
+- **Then** it fails without promoting anything and the previous active key stays in use
+- **And** the fleet keeps issuing tokens rather than half-rotating
+
+### Requirement: Introspection is cached, and the cache cannot outlive a revocation
+
+Requirement-ID: ARCH-INTROSPECT-CACHE
+The service SHALL answer introspection from a cache whose entries expire within the
+revocation propagation budget, SHALL delete a token's entry when that token is
+revoked, and SHALL fall back to the key store when the cache is unavailable. The
+cache holds derived state only: losing it costs latency, never correctness, and any
+design where a cache miss can produce a *different verdict* is the bug this
+requirement exists to prevent.
+
+Covers: marketplace.identityService.introspectionCache, marketplace.identityService.tokens -> marketplace.identityService.introspectionCache
+
+#### Scenario: A revocation invalidates the cached verdict immediately
+- **Given** a token whose active verdict is cached
+- **When** its client is revoked
+- **Then** the cached entry is deleted as part of the revocation
+- **And** the next introspection of that token reports inactive
+
+#### Scenario: A cache entry cannot outlive the propagation budget
+- **Given** a cached verdict written thirty seconds ago
+- **When** it is read after its TTL has elapsed
+- **Then** it is treated as a miss and the verdict is recomputed from revocation state
+
+#### Scenario: The cache being down degrades latency, not the answer
+- **Given** the introspection cache is unreachable
+- **When** a token is introspected
+- **Then** the verdict is computed from the revocations in the key store and is unchanged
+- **And** the service stays available, paying for the miss in introspection latency alone
+
+### Requirement: Token issuance is measured and paged
+
+Requirement-ID: ARCH-ISSUE-HEALTH
+The service SHALL export an availability and an issuance-latency SLI, and SHALL page
+the identity on-call when issuance starts returning server errors. Issuance failing
+is the one fault every other service in the fleet experiences as its own, so it is
+paged rather than dashboarded.
+
+Covers: sli:availability, sli:token_issue_latency_p99_ms, alert:token_issue_5xx
+
+#### Scenario: Both SLIs are exported while the service serves traffic
+- **Given** the service is serving token requests
+- **When** its metrics endpoint is scraped
+- **Then** `availability` and `token_issue_latency_p99_ms` report values against their SLOs
+
+#### Scenario: Sustained issuance errors page the identity on-call
+- **Given** more than 0.5% of token issuance requests return a server error over five minutes
+- **When** the alert rules are evaluated
+- **Then** `token_issue_5xx` pages the identity on-call
+
+#### Scenario: An instance that cannot sign takes no traffic
+- **Given** an instance whose active signing key failed to load
+- **When** the readiness check is evaluated
+- **Then** it reports not ready and is removed from the pool
+- **And** the failure never reaches a caller as a server error

package/examples/docs/services/platform/identity-service/health.yaml

@@ -0,0 +1,70 @@
+# Service health model: the signals this service promises to emit and the pages it
+# promises to send. Targets and queries are authored here; the dependency graph
+# comes from C4. Planned: `loam health` composes these across the fleet into a
+# cascade / blast-radius view.
+#
+# `loam validate` reads exactly three things out of this file, all of them ids:
+#
+#   * each `slis[].name` — every one wants a `Covers: sli:<id>` line in
+#     arch.spec.md, or `health.uncovered` says nothing tests the signal, and a
+#     signal nothing tests is dashboard decoration;
+#   * each `alerts[].name` — same obligation, through `Covers: alert:<id>`;
+#   * each `dependencies[]` id — which must resolve in this service's OWN
+#     model.likec4, or `health.dependency-unmodelled` fires.
+#
+# Queries, thresholds, pages and the checks below are authored prose that no check
+# reads. They are here because the on-call engineer reads them.
+service: identity-service
+status: draft
+
+slis:
+  - name: availability
+    # Every service that accepts a token calls this one, so this number is a
+    # ceiling on the fleet's: the 0.05% it permits is spent by every consumer at
+    # once. That is why it is set a notch tighter than payment-service's 0.999
+    # rather than matching it — a shared dependency cannot hold the same budget
+    # as the services depending on it and leave them any of their own.
+    query: "sum(rate(http_requests_total{job='identity-service',code!~'5..'}[5m])) / sum(rate(http_requests_total{job='identity-service'}[5m]))"
+    slo: 0.9995
+  - name: token_issue_latency_p99_ms
+    # Issuance alone, not introspection: minting signs with the active key and is
+    # the slow path, while introspection answers from cache. Averaging the two
+    # would let a signing stall hide behind cache hits.
+    query: "histogram_quantile(0.99, sum by (le) (rate(http_request_duration_seconds_bucket{job='identity-service',path='/tokens'}[5m]))) * 1000"
+    slo: 150
+
+alerts:
+  - name: token_issue_5xx
+    # Pages on issuance failing, which is the failure every other service feels as
+    # its own: a caller that cannot get a token cannot call anybody.
+    #
+    # A RATIO, not a raw error rate: ARCH-ISSUE-HEALTH promises a page at "more
+    # than 0.5% of token issuance requests", and `rate(5xx) > 0.005` would instead
+    # page at one error every 200 seconds — a threshold that means one thing at
+    # 10 rps and something else entirely at 1000. The five minutes the scenario
+    # names live in the range vector, so there is no second `for:` delay under it
+    # making the page later than the requirement says it comes.
+    expr: "sum(rate(http_requests_total{job='identity-service',path='/tokens',code=~'5..'}[5m])) / sum(rate(http_requests_total{job='identity-service',path='/tokens'}[5m])) > 0.005"
+    page: identity-oncall
+
+checks:
+  liveness: GET /healthz
+  # Readiness fails while the active signing key cannot be loaded: an instance that
+  # is up and cannot sign should take no traffic, because it turns every issuance
+  # into a 5xx and burns the alert above instead of being rolled out of the pool.
+  readiness: GET /readyz
+
+# There is deliberately NO `dependencies:` key here, and this comment is the
+# difference between that fact and a forgotten section. identity-service is the
+# fleet's root of trust: it calls no other service, publishes to no broker and
+# reaches no third party, so it is named in other services' dependency lists
+# (payment-service carries it as `degradable`) and names none of its own. Its two
+# stores are inside its own boundary in model.likec4 (a private store nested in
+# the service, never a fleet-level peer) and share its lifecycle rather than being
+# something it depends on across a boundary.
+#
+# The day that stops being true, the new entry has to be drawn in
+# services/identity-service/model.likec4 as well — resolution is against this
+# service's own model and never the landscape, because the landscape carries only
+# what crosses the service boundary while the model carries everything the service
+# touches, and the model is the set an on-call file has to agree with.

package/examples/docs/services/platform/identity-service/model.likec4

@@ -0,0 +1,81 @@
+// identity-service's own C4 — what is INSIDE the element the fleet map binds to
+// services/platform/identity-service/: the containers behind the token API and
+// the two stores they reach.
+//
+// An EXTENDING model. There is no `specification` block here and there must not
+// be one: the kinds come from architecture/landscape.likec4, which is read as
+// part of the SAME LikeC4 project (the root `likec4.config.json` excludes
+// `**/node_modules/**` and `features/**`, and nothing under `services/`). A
+// second declaration of a kind the map already declares is a duplicate error
+// blamed on both files. `extend` takes the fully-qualified id the map minted —
+// `marketplace.identityService` — because the element is nested in the
+// `marketplace` group; `extend identityService` resolves to nothing.
+//
+// Where the service sits in the SUBSYSTEM tree changes none of this: the model
+// is found by walking `services/`, and `services/platform/` is a grouping
+// directory, never part of any id. SCHEMA.md, "Two shapes of a service model",
+// has the standalone shape and the recipe for migrating a model to this one.
+model {
+  extend marketplace.identityService {
+    api = container 'api' {
+      description 'HTTP surface: POST /tokens, POST /tokens/introspect, GET /tokens/{token}/valid'
+      technology 'Go'
+    }
+    tokens = container 'tokens' {
+      description 'Mints and signs tokens, and turns a recorded revocation into an inactive verdict'
+      technology 'Go'
+    }
+    // Everything durable is in ONE store on purpose, and the description says
+    // which things, because two documents downstream depend on it: the cache
+    // requirement in arch.spec.md falls back HERE for a verdict, and the
+    // stampede in runbook.md is issuance and introspection contending for this
+    // store's connection pool. Naming only the signing keys here would leave
+    // both of them reaching for state nothing in this model holds.
+    keyStore = database 'key-store' {
+      description 'Signing keys and their rotation state, the client registry, and the revocations a verdict is computed from — the only durable state this service owns'
+      technology 'PostgreSQL'
+    }
+    // Derived state, so a plain container rather than a `database`: losing every
+    // byte in here costs latency and never correctness, and ARCH-INTROSPECT-CACHE
+    // in arch.spec.md is the requirement that says so in a scenario.
+    //
+    // Both stores stay NESTED inside the service boundary, and nothing makes
+    // them: loam grades neither placement, so lifting either into
+    // architecture/landscape.likec4 as a peer would earn no finding. They stay
+    // here because the extending shape makes it the cheapest default — an
+    // internal is written where the service is described, and nobody has to
+    // decide anything. The one consequence to know before choosing is about
+    // LikeC4 and not about taste: a second service's edge resolves only against
+    // a declaration the MAP holds, so a store somebody else will name goes up
+    // onto the landscape, inside `marketplace.identityService`.
+    introspectionCache = container 'introspection-cache' {
+      description 'Short-TTL introspection verdicts, keyed by token id'
+      technology 'Redis'
+    }
+
+    api -> tokens 'Issues tokens, answers introspection and the legacy check'
+    // These three edges are the WHOLE runtime graph of this service:
+    // identity-service is the fleet's root of trust and calls nothing outside
+    // its own boundary, which is why health.yaml carries no `dependencies:`
+    // key, and why this model draws no cross-boundary edge at all. They still
+    // have to be drawn — with four nested elements and no edge joining any of
+    // them this model is `c4.no-relationships`, on the grounds that parts which
+    // reach nothing are almost never the truth. (The check is evidence-gated:
+    // one lone container proves nothing and stays silent, which is why the
+    // warning is about a model this shape and not about any model.)
+    tokens -> keyStore 'Signs with the active key; rotation promotes the next; introspection reads keys and revocations back'
+    tokens -> introspectionCache 'Caches introspection verdicts, invalidates on revocation'
+  }
+}
+
+// Views belong to LikeC4's renderer; no loam check reads anything below. One
+// view is enough here because the whole model fits inside one boundary — the
+// landscape splits its views only because a fleet-wide `include *` is the page
+// nobody can read. View ids are flat across the whole project now that the map
+// and every model share one, so this one is spelled to be unique.
+views {
+  view identity_service of marketplace.identityService {
+    title 'identity-service — containers'
+    include *
+  }
+}

package/examples/docs/services/platform/identity-service/openapi.yaml

@@ -0,0 +1,197 @@
+# The HTTP contract. `operationId` is the join token that ties this file to the
+# other two axes: a landscape edge says `metadata { op 'introspectToken' }` and a
+# requirement says `Operations: introspectToken`, and all three spellings must be
+# byte-identical or the join is silently empty.
+#
+# loam reads operationIds from the eight HTTP methods of a path item and nowhere
+# else — an id parked under a vendor extension (`x-legacy:`) does not exist as far
+# as every check here is concerned, because a phantom id would mask a real
+# contract break rather than document one.
+openapi: 3.1.0
+info:
+  title: identity-service
+  version: "2.0"
+  description: >-
+    Token issuance and introspection for the marketplace fleet. Every service that
+    accepts a caller's token resolves it here; nothing in this contract calls out.
+
+paths:
+  /tokens:
+    post:
+      operationId: issueToken
+      summary: Exchange client credentials for a signed access token
+      requestBody:
+        required: true
+        content:
+          application/json:
+            schema:
+              $ref: '#/components/schemas/TokenRequest'
+      responses:
+        "201":
+          description: A signed token, valid until its stated expiry
+          # Every operation needs at least one response declaring
+          # `content: <media>: schema:` — without one, `openapi.response-undescribed`
+          # says the contract names the operation and nothing about what it
+          # returns, so a reader still has to open the code. 204 and 304 are the
+          # only codes excused, since neither carries a body.
+          content:
+            application/json:
+              schema:
+                $ref: '#/components/schemas/AccessToken'
+        "401":
+          description: The credentials did not authenticate
+          # A declared 4xx/5xx is a promise about a refusal, and loam grades it as
+          # one: this exact number must appear as a bare number inside the body of
+          # a scenario belonging to a requirement whose `Operations:` line names
+          # issueToken, or `api.response-ungoverned` fires. IDN-ISSUE's
+          # "unknown client" and "revoked secret" scenarios are where 401 is paid
+          # for — written as scenarios rather than prose, because a scenario is
+          # what `loam gherkin` turns into a test.
+          content:
+            application/json:
+              schema:
+                $ref: '#/components/schemas/AuthError'
+
+  /tokens/introspect:
+    post:
+      operationId: introspectToken
+      summary: Report whether a token is currently active and what it grants
+      description: >-
+        The authoritative answer, computed from revocation state — not from the
+        token's signature, which stays valid for the whole of a revoked token's
+        remaining lifetime.
+      requestBody:
+        required: true
+        content:
+          application/json:
+            schema:
+              $ref: '#/components/schemas/IntrospectionRequest'
+      responses:
+        # RFC 7662's shape, deliberately: an unknown, expired or revoked token is
+        # `{"active": false}` with a 200, never a 4xx. There is consequently no
+        # failure code to govern here, and `api.response-ungoverned` has nothing
+        # to say about this operation.
+        "200":
+          description: The verdict for the presented token
+          content:
+            application/json:
+              schema:
+                $ref: '#/components/schemas/Introspection'
+
+  /tokens/{token}/valid:
+    get:
+      operationId: validateToken
+      summary: Legacy boolean validity check
+      # `deprecated: true` is standard OpenAPI, and loam turns it into lifecycle
+      # visibility on the spine rather than leaving migration state to a wiki.
+      # Two findings come out of this one line, both warnings, both wanted here:
+      #
+      #   * `spine.op-deprecated` — the landscape's
+      #     `paymentService -> identityService` edge carries
+      #     `metadata { op 'validateToken' }`, and payment-service has not moved
+      #     yet. The finding is filed against the PROVIDER (this service, the
+      #     edge's target) because the contract lives here, and it names the
+      #     consumer that still has to migrate — one warning per inbound edge, so
+      #     a deprecated operation nobody calls stays silent.
+      #   * `api.requirement-deprecated` — IDN-VALIDATE-LEGACY governs only this
+      #     operation, so the behaviour it describes is on its way out.
+      #
+      # Deprecation is the FIRST step of retiring an operation, never the removal
+      # itself: the op stays defined and callable, because deleting it here would
+      # break payment-service's edge with nothing having proposed the break. The
+      # step that ends this state is a feature delta, and its recipe is written
+      # once, in adrs/0001-token-introspection-over-local-validation.md.
+      deprecated: true
+      parameters:
+        - name: token
+          in: path
+          required: true
+          description: The opaque token id being checked
+          schema:
+            type: string
+      responses:
+        "200":
+          description: Whether the token is currently valid
+          content:
+            application/json:
+              schema:
+                $ref: '#/components/schemas/LegacyValidity'
+
+components:
+  # Every schema this document references lives here, in the same file. An
+  # external `$ref` is out of scope on this axis — it does not travel with the
+  # contract when the contract is read on its own — and an internal one that
+  # points at nothing is `openapi.ref-unresolved`: the operation validates while
+  # its payload is silently absent, which reads exactly like a complete contract.
+  schemas:
+    TokenRequest:
+      type: object
+      required: [clientId, clientSecret]
+      properties:
+        clientId:
+          type: string
+        clientSecret:
+          type: string
+          format: password
+        scope:
+          type: string
+          description: Space-separated scopes; the issued token carries the intersection with what the client is granted
+    AccessToken:
+      type: object
+      required: [accessToken, tokenType, expiresIn]
+      properties:
+        accessToken:
+          type: string
+        tokenType:
+          type: string
+          const: Bearer
+        expiresIn:
+          type: integer
+          description: Seconds until expiry, counted from issuance
+        scope:
+          type: string
+    AuthError:
+      type: object
+      required: [error]
+      properties:
+        error:
+          type: string
+          # One value, on purpose: the refusal must not tell a caller whether the
+          # client id exists. IDN-ISSUE's two 401 scenarios are what hold that
+          # promise, and this enum is what makes breaking it a contract change.
+          enum: [invalid_client]
+        errorDescription:
+          type: string
+          description: Human-readable, never distinguishing an unknown client from a bad secret
+    IntrospectionRequest:
+      type: object
+      required: [token]
+      properties:
+        token:
+          type: string
+    Introspection:
+      type: object
+      required: [active]
+      properties:
+        active:
+          type: boolean
+        sub:
+          type: string
+          description: The client the token was issued to; absent when inactive
+        scope:
+          type: string
+        exp:
+          type: integer
+          description: Expiry as seconds since the epoch; absent when inactive
+    LegacyValidity:
+      type: object
+      required: [valid]
+      properties:
+        valid:
+          type: boolean
+        expiresAt:
+          type: string
+          format: date-time
+          description: >-
+            The only field this legacy shape carries beyond the boolean. Callers that
+            need scope or subject have to use introspectToken; this schema is frozen.

package/examples/docs/services/platform/identity-service/runbook.md

@@ -0,0 +1,95 @@
+---
+service: identity-service
+status: draft
+owner: platform-identity
+---
+
+# identity-service — runbook
+
+Written for whoever is holding the pager at 03:00, so every step below is one a
+stranger to this service can carry out. `loam list` tracks only that this file
+exists; nothing in it is checked, which means the burden of keeping it true is on
+the same PR that changes the behaviour it describes.
+
+Read this together with `arch.spec.md`: the two incidents below are the failure
+modes ARCH-KEY-ROTATION and ARCH-INTROSPECT-CACHE exist to prevent, and each
+mitigation here is the manual version of a promise the service is supposed to keep
+on its own.
+
+## Deploy
+- Pipeline: `.github/workflows/release.yml` → `deploy-prod`, gated on a manual approval.
+- Config: `config/prod.yaml`; the signing key material is never in it — the service
+  reads keys from the key store at startup and gets its store credentials from Vault
+  (`secret/identity-service`).
+- Rollout is one instance at a time, and the gate is readiness (below). A key store
+  the new revision cannot reach fails readiness and stalls the rollout, which is the
+  intended outcome: half a fleet unable to sign is worse than a paused deploy.
+
+## Health
+- Liveness: `GET /healthz` — the process answers. Nothing behind it is checked, so
+  this never restarts an instance for a dependency's fault.
+- Readiness: `GET /readyz` — the active signing key is loaded and the key store
+  answers. An instance that is up and cannot sign must take no traffic; it would
+  turn every issuance into a 5xx and burn `token_issue_5xx` for a fault that a
+  rolling restart fixes.
+- SLIs and the alert are declared in `health.yaml`; the arch requirement covering
+  them is ARCH-ISSUE-HEALTH.
+
+## Common incidents
+
+### A rotation promoted a key the other instances cannot read
+**Symptom.** Token issuance is fine, and every other service starts rejecting
+tokens: 401s spread across the fleet within seconds of a rotation job running,
+while `token_issue_5xx` stays quiet because *this* service is healthy. The
+consumers are refusing on an introspection verdict of `active: false` — freshly
+minted tokens that this service itself will not vouch for.
+
+**Cause.** The new key was promoted to active before every instance could read it
+from the key store, or the previous key was retired inside one token lifetime.
+Either breaks the overlap ARCH-KEY-ROTATION requires: the instance that mints and
+the instance that answers introspection are different processes, and only the key
+store makes them agree.
+
+**Mitigation.**
+1. Demote to the previous key: `identityctl keys promote --key <previous-kid>`.
+   Tokens already signed with the bad key keep reading inactive; they expire within
+   a lifetime and there is nothing to do about them.
+2. Confirm the key store holds both kids, and that a *different* instance than the
+   one you are on can verify a token minted with the new kid, before promoting
+   anything again.
+3. Do not "fix" it by shortening the token lifetime — that widens the outage.
+
+**Prevention.** Write, wait one full token lifetime, then promote. The rotation job
+does this on its own; a manual promotion is the way it gets skipped.
+
+### The introspection cache stampeded
+**Symptom.** The key store's connection pool saturates and `token_issue_latency_p99_ms`
+climbs past its SLO — issuance signs from the same store, so a flood of introspection
+misses shows up on the issuance SLI even though issuance is not what changed. Usually
+right after a cache restart, a failover, or a mass revocation that cleared many
+entries at once.
+
+**Cause.** Every introspection missed at the same moment and each miss became a
+key store read. The service stays *correct* through this — ARCH-INTROSPECT-CACHE
+requires a miss to produce the same verdict — so what is failing is latency, and
+the cure is to protect the store rather than to bypass the cache.
+
+**Mitigation.**
+1. Check the miss rate and the store's pool metrics before touching anything; if
+   the pool is not saturated, this is a different incident.
+2. Raise the per-key single-flight window (`introspection.singleflight_ms`) so that
+   concurrent misses for one token collapse into one store read.
+3. If the store is already saturated, shed introspection load at the gateway before
+   restarting the cache — a cold restart replays the stampede.
+4. Never disable the cache to "get more capacity": every request then reaches the
+   store.
+
+**Prevention.** Jittered TTLs and single-flight on miss, both of which the cache
+layer owns; a mass revocation should delete entries in batches rather than flush.
+
+## Escalation
+- On-call channel: `#identity-oncall`; the alert `token_issue_5xx` pages it directly.
+- Because every service depends on this one, an incident here is announced in
+  `#fleet-incidents` as soon as it is confirmed, before it is understood — the
+  consumers are the ones who will see the symptom first.
+- Owning team: platform-identity.