@shortlink-org/portolan 0.1.0

package/plugins/README.md

@@ -0,0 +1,1160 @@
+# Generators
+
+A generator turns the catalog into something else. The built-in generators
+produce Markdown and LLM indexes, standalone Mermaid diagrams, and a Backstage
+Software Catalog bundle; each lives in its own directory beside the plugin.
+
+The contract is one JSON message in and one JSON message out:
+
+```
+→ { "portolanVersion": "0.1.0", "catalog": { ... }, "options": { ... } }
+← { "files": [{ "name": "shop/oms/README.md", "contents": "..." }] }
+```
+
+There is a second question, asked with `"kind": "describe"`, and the answer is
+the plugin itself rather than its work:
+
+```
+→ { "portolanVersion": "0.1.0", "kind": "describe" }
+← { "files": [],
+    "describe": { "name": "extract-go", "summary": "...",
+                  "phases": ["extract"],
+                  "options": { "type": "object", "additionalProperties": false,
+                               "properties": { "context": { ... } } } } }
+```
+
+The options a plugin takes are facts the source does not carry, so only the
+plugin knows what it can be told. `npm run schema` asks all of them and composes
+`schema/portolan.schema.json`, which an editor reads while the manifest is being
+written and `gen` checks before it runs anything. `additionalProperties: false`
+is what makes that worth having: `encoding/json` drops a field it does not
+recognise, so before this a misspelled option was no option at all and nothing
+said so.
+
+A generator **names** files; it never writes them. `scripts/gen.mjs` writes what
+comes back, refuses a name that points outside the output directory, and deletes
+pages that stopped being generated. That is what lets a generator run as a wasm
+module with no directory preopened at all — the sandbox is not a restriction
+worked around, it is the reason the protocol has this shape.
+
+Three obligations, and they are the whole of it:
+
+1. **Valid output.** Every file a generator names is committed and read.
+2. **Determinism.** The same catalog produces byte-identical output, so
+   reviewing generated documentation is reviewing a diff. Sort anything that
+   comes out of a map; never read a clock.
+3. **One decisive response.** A malformed request, incompatible protocol,
+   unsafe filename or invalid response fails the run. Non-fatal extraction
+   notes go to stderr; there is no advisory response property a caller may
+   accidentally ignore. A note that opens with `warning: ` is kept beside the
+   step in `.portolan/build-report.json` and listed on the Settings page.
+
+The repository enforces those obligations with schema/field coverage tests,
+byte-for-byte permutation tests, generated-link and anchor checks, Mermaid
+parser checks, and Backstage relationship validation. A new catalog field must
+either be rendered or be explicitly acknowledged by the relevant exporter.
+
+## Adding one
+
+1. Write it. In Go, a new directory here with a `main` that hands its options
+   type to `plugin.Serve`, which reads the request, answers a describe and calls
+   the work; `catalog.Catalog` from `github.com/shortlink-org/portolan/catalog`
+   is the mirror of the schema, and `internal/goscan` is the tree as syntax -
+   the files parsed once, the import path of each package, the string
+   constants followed to their literals - which River and Watermill share and
+   the next Go extractor should not copy. In Python, the same three things live in
+   `pyplugin/` - `protocol.py`, `source.py` for the tree as syntax, and
+   `catalog.py` for the fragment shapes - and `extract-django` and
+   `extract-celery` are what using them looks like. In any other language,
+   anything that speaks the protocol above.
+2. Describe it. An `options.schema.json` beside the source, embedded with
+   `go:embed` and returned in the descriptor. `schematest.Check` in a test keeps
+   it from drifting from the options struct: a field renamed on one side and not
+   the other fails, and so does an option with no description.
+3. Build it. For a wasm plugin, `GOOS=wasip1 GOARCH=wasm go build`. Add the
+   line to `plugins:build` in `package.json`.
+4. Declare it in `portolan.json`, under `plugins` (how to run it) and
+   `generate` (what to run it on), then run `npm run schema` so the manifest
+   schema learns its options.
+
+```json
+{
+  "plugins": [
+    { "name": "markdown", "wasm": { "url": "file://plugins/gen-markdown.wasm" } }
+  ],
+  "generate": [
+    { "plugin": "markdown", "out": "docs", "options": { "title": "Example estate" } }
+  ]
+}
+```
+
+## Lifecycles
+
+An aggregate whose root has a status gets a `lifecycle` on the catalog: the
+states, the first being where a new root starts, and one move per edge - the
+method that makes it and the event it hands back. All three extractors read it
+off a table the code keeps, never off the branches of the methods, because the
+table is the claim and the methods are held to it: an edge in the table no
+method makes, a move into a state the table lacks, and a status changed
+outside the one way through the table are each reported.
+
+In Go the table is a `go-sdk/fsm` rule set, `var Rules = fsm.TransitionRuleSet{
+StateLive: {EventRevoke: StateRevoked}, …}`, with the states and events as
+string constants; the method whose body calls `TriggerEvent` is the mover, and
+every exported method that hands it a constant makes the edges that constant
+names. The event type a method returns is what its last move publishes - a
+method lapsing a lock and then locking again hands `AccountLocked` back for
+the lock. In TypeScript it is `export const TRANSITIONS = { open: ["checked-out"],
+… }` and the method that assigns `this.status`; see `extract-ts/README.md`. In
+Rust it is `pub const TRANSITIONS: &[(&str, &[&str])] = &[("placed", &["confirmed"]),
+…]` and the method that assigns `self.status`, handed a string or a variant of
+the status enum; see `extract-rust/README.md`. In Python it is that mapping on
+the Django model, beside the `TextChoices` that names the states, and the
+method assigning `self.status` - or django-fsm's `@transition(field=status,
+source=…, target=…)`, which is the same table written one edge at a time; see
+`extract-django/README.md`. In Java it is `TRANSITIONS` beside the status enum
+and the method assigning `this.status`; Java is also the one language here with
+a vocabulary for the model, so the rest of `extract-java` reads what jMolecules
+declares rather than what the layout implies.
+Terminal states are derived on the page - nothing leads out - and never
+written down. A move the clock makes, a session expiring, a lock running out,
+is not a move: nothing runs when it happens, so it is not in the table.
+
+## Enums
+
+An aggregate's fields take some of their values from closed sets - a reason,
+a status, a code - and a consumer of its events switches on them. Those sets
+are the aggregate's `enums`: one entry per set, id `<aggregate>.<slug>`,
+the values in declaration order, each with its doc and a `deprecated` mark
+when the source carries one. A status enum is read here as well as by the
+lifecycle: the lifecycle keeps the moves, the enum keeps the doc on each
+value, and a page may draw both.
+
+What counts as one is a convention per language, and each extractor's README
+says which. In Go, which has no enum, it is a named type over a basic one -
+`type Reason string` - and a const block whose constants are of that type,
+looked for in the aggregate's package, under `vo/`, and under `event/`;
+a value's name is the constant's literal, because that is what the wire
+carries, and the constant's own name only for an iota. A `Deprecated:`
+paragraph in the doc marks the value. In Rust it is a `pub enum` whose every
+variant is a bare name, the literal an `as_str` answers standing in for the
+variant. In Java it is a top-level enum in the aggregate's package. In proto,
+the enums the messages reach through their fields sit on the interface as
+`enums`, with the numbers the wire uses.
+
+## A repository without a domain model
+
+`extract-project` is the baseline extractor for any repository. It reads only
+README and build/deployment manifests, emits a neutral group and component,
+and leaves `aggregates` empty. This is intentional: a package called `domain`
+is not evidence that the project models aggregates. OpenAPI, AsyncAPI,
+GraphQL, proto and SQL extractors merge their facts into the same component.
+
+The language-specific domain extractors are optional enrichments. The local
+setup wizard offers one only when it finds the structure that extractor
+requires; for Go this means an aggregate package in either
+`internal/domain/<aggregate>` or `internal/<aggregate>/domain`, containing the
+root struct named after that aggregate. The extractor follows the same choice
+for application use cases, transport adapters, policies, integration-event
+DTOs, and assembly bindings, so horizontal layers and feature slices can
+coexist while a service is being migrated.
+
+The SQL extractor follows the same migration path. With no `repositories` or
+`projectors` option it discovers both
+`internal/infrastructure/repository/<aggregate>/migrations` and
+`internal/<aggregate>/infrastructure/repository/migrations` (and the matching
+projector forms), merging every package into one store. An explicit root keeps
+the original collection layout for TypeScript, Rust, Java, or custom trees and
+may also point directly at one feature repository.
+
+`extract-river` is another independent enrichment for Go repositories. It
+joins a job argument's `Kind()` to `Client.Insert`/`InsertTx`, the selected
+queue, `Worker[Args].Work`, and `river.AddWorker`. The result is a work-queue
+channel plus a two-hop enqueue/dispatch flow, with payload fields and source
+lines. It does not need aggregates and does not treat a job as a domain event.
+
+`extract-redis` finds runtime construction of go-redis, rueidis and redigo
+clients in non-test Go source. That source evidence adds a service-owned Redis
+store to the catalog even when the repository has no SQL migrations or domain
+model. It follows literals, constants, concatenation, `fmt.Sprintf`, helper
+functions and conditional suffixes into common Redis operations, producing
+key patterns with their read/write/delete use, TTL, value type and source. The
+patterns remain Redis keyspaces rather than being presented as SQL tables.
+
+`extract-watermill` reads Watermill `Router.AddHandler` and
+`AddNoPublisherHandler` registrations. It resolves literal and constant topics,
+plus defaults on env-config structs, follows direct `Publisher.Publish` calls
+and one-hop publishing helpers, and traces JSON marshal/unmarshal values back to
+their Go payload structs. It carries enclosing `if`/`else` conditions and early
+returns through the handler's control flow: proven alternatives become one
+catalog `alt`; publications whose relationship cannot be proven remain separate
+possible routes. The channels merge normally with AsyncAPI declarations by address. Generic
+`NewEventHandler[T]` and `NewCommandHandler[T]` registrations on Watermill CQRS
+processors are also extracted; fixed topic generators and the standard
+event/command-name generator form are resolved from source.
+
+`extract-go-nats` reads nats.go and JetStream calls into the subjects a service
+listens on and publishes to. A call is known by the type it is made on -
+`*nats.Conn`, `nats.JetStreamContext`, `jetstream.JetStream`, `jetstream.Stream` -
+and not by its name, because a service's own bus port has a `Subscribe` too.
+The subject is followed to a literal, a constant, a config default, or a
+parameter; a parameter is followed up to two hops through the callers,
+including calls through an interface the adapter satisfies, which is how a
+port `Subscribe(ctx, topic, name, handler)` reads as the assembly's
+`Subscribe(ctx, cart.Topic, cart.BasketCheckedOut{}.Name(), …)`. When the port
+takes exactly one other string beside the subject, that string is the
+message's name; a direct call names no message, and its direction is in the
+channel's doc. A subject read off a database row is a warning at the call,
+not a channel. Consumer configs give the filter subject and the durable name;
+streams, wildcard subjects and work-queue retention are not read yet.
+
+`extract-python-kafka` is the framework-independent Kafka enrichment for
+Python. It recognizes confluent-kafka, kafka-python and aiokafka by their
+imported client types, follows literal topics through constants, environment
+defaults, settings and local factories, and emits generic message streams with
+producer and consumer flows. Only client-side configuration proven in source
+is retained; authentication values are omitted, while broker-side partitions,
+replication and retention remain explicitly unknown. See
+`extract-python-kafka/README.md` for the supported call shapes.
+
+`extract-wsdl` reads WSDL 1.1 contracts as structured SOAP APIs. It follows
+local WSDL imports and XSD imports/includes without network access, keeps
+distinct services, ports and SOAP 1.1/1.2 bindings, and records operation
+actions, request/response messages, faults, headers and reachable XSD shapes.
+It can describe a contract implemented by the component or a vendored copy for
+an external system (`mode: external`). Remote imports are reported as missing
+evidence rather than fetched during generation.
+
+`extract-http-clients` is the outbound counterpart and does not require a
+domain layout. It reads `net/http` request construction, calls through an
+`oapi-codegen` client, and SOAP `Call`/`CallContext` sites. A generated client
+is joined to the OpenAPI document beside it, so the call uses the document's
+operation id and lands on an external with the contract the document declares.
+SOAP actions are joined to WSDL bindings when the action matches. Generated and
+hand-written wrapper signatures are learned from their call into the SOAP
+transport or their `SOAPAction`/SOAP 1.2 content-type header, so the action,
+request and response positions are taken from code rather than assumed. A
+raw request whose peer or contract cannot be proved is still useful evidence:
+it is emitted as `unresolved`, with its method, path and source line, rather
+than being assigned to a guessed system. Conditions guarding a call and the
+opposite path after an early return are carried into the flow note.
+Calls through local wrappers retain their argument values, including closure
+arguments, so a path and HTTP method declared by a business operation survive
+the trip into the transport. URL-shaped configuration is followed through a
+constructor and client field to the request, and the resulting evidence chain
+is included in the flow instead of presenting a receiver field as an endpoint.
+Inbound composition also follows fixed factories, string-keyed constructor
+maps, capability type assertions, and interface fields wired by composite
+literals, direct assignments, or setters. A standalone flow says whether no
+source caller exists or callers exist but no inbound/asynchronous root was
+proved, so the UI exposes the missing evidence instead of implying a complete
+business path.
+When a provider branch still ends before its transport, the extractor loads
+the module with `go/packages`, builds SSA, and uses `x/tools` VTA to resolve
+calls through interface parameters, function values, return values, and
+interface-typed struct fields. Typed edges are followed only after a concrete
+factory branch is selected: applying context-insensitive VTA to a shared
+dispatcher would otherwise attach every request implementation to every
+endpoint. Factory conditions and HTTP/SOAP meaning continue to come from the
+source extractor. Module loading is read-only and bounded; unavailable private
+dependencies, type errors, or a timeout produce a warning and retain the
+syntax-only result rather than failing generation.
+Routes without a provider factory are also joined to their outbound calls,
+including handlers invoked from closures and methods on locally constructed
+values. A Swagger `@Router` annotation is medium-confidence root evidence for
+handler factories behind custom registries. Direct calls reached through a
+`main → Run`/`Start`/`Bootstrap` assembly path become high-confidence startup
+flows. `AddFunc`, `AfterFunc`, and `Schedule` registrations create scheduled
+roots when their handler reaches an outbound call. Extracted HTTP, callback,
+startup, scheduled, River-job and Watermill-event flows publish that trigger
+provenance; unmatched transport fragments are explicitly `unproven` with low
+confidence.
+
+After fragments merge, Portolan composes them into root-oriented,
+cross-protocol flows. The seam is machine evidence rather than a display-name
+match: an exact source function reached by another extractor, a River job's
+queue plus `Kind()`, or a Watermill transport plus topic address.
+Composition is recursive, so one request can continue through provider
+HTTP/SOAP calls, enqueue a River job, enter its worker, and make further
+outbound calls. Ambiguous handoffs remain separate. Each composed flow records
+the source fragment slugs it includes, and the UI exposes that provenance as a
+`cross-protocol` badge. Trigger-bearing job and event flows stay available as
+standalone entry views; transport-only fragments consumed by a proven root are
+removed from the top-level flow list.
+
+## Flows written by hand
+
+Some flows will always be written by people: the design doc for something not
+built yet, the reconstruction after an incident, the path no test pins. The
+catalog's JSON is the wrong place to write one - a tree of nodes, a unique id
+per step, every lane declared twice - so `extract-flows` reads a text form
+that reads like the sequence diagram it becomes: one file per flow, one line
+per hop, frames closed by `end`. The estate here declares no such step: every
+flow it shows is read out of code that runs, and one written by hand over files
+nobody wrote would be the single page in the catalog nothing holds to account.
+A repository that has a real one points an `extract-flows` step at the directory
+the files live in.
+
+```markdown
+# Order accepted
+owner: shop
+source: services/oms/test/integration/order_accepted_test.go
+
+The narrow slice one integration test pins end to end.
+
+## Participants
+- oms-db: store in shop "oms-db (postgres)"
+- psp-gateway: external "psp-gateway (external)"
+
+## Steps
+shop.oms -> oms-db: insertOrderAndOutboxRow [verified] @internal/oms/adapter/postgres/order_repo.go:141 #a1
+  > The order row and the outbox row commit in one transaction.
+shop.oms -> bus: event shop.oms.order.OrderPlaced [verified]
+shop.oms -> shop.pricing: rpc shop.v1.Pricing/GetQuote as "GetQuote (250 ms)"
+
+alt score below 40 #alt-risk
+  bus -> payments.ledger: event shop.oms.order.OrderPlaced
+else score at or above 40
+  shop.oms -> bus: event shop.oms.order.OrderCancelled
+  stop
+else
+end
+
+par OrderPlaced fan-out
+  bus -> payments.ledger: event shop.oms.order.OrderPlaced
+and
+  bus -> delivery.core: event shop.oms.order.OrderPlaced
+end
+
+loop outbox relay, every 200 ms until the batch is empty
+  shop.oms -> oms-db: SELECT ... FOR UPDATE SKIP LOCKED
+end
+```
+
+The head is the name, `owner:` (the context the flow belongs to), an optional
+`source:` (where it was read from; the file itself when left out) and an
+optional `slug:` (the file's name when left out), then the summary. A hop is
+`from -> to: [call|rpc|event] label-or-ref` - `call` when no kind is written -
+followed in any order by `as "label"`, `[status]`, `@where` and `#id`. An
+event names its ref, `shop.oms.order.OrderPlaced`; an rpc names its call id,
+`shop.v1.Pricing/GetQuote`, or, for a call no interface declares - a webhook
+arriving on a route - just a label. The label of an event or rpc is the last
+segment of its ref unless `as` says otherwise; the status is `declared`
+unless written; `@` is a file and line, or wherever the hop was seen. A note is
+the `>` lines under the hop. Ids are numbered unless given, and giving them
+is what keeps deep links to a step stable across edits.
+
+`alt <when> … else <when> … end` is a choice; `else` alone is "otherwise", a
+branch with nothing in it is allowed, and `stop` as the last line of a branch
+says the flow ends there rather than rejoining. `par [title] … and … end`
+runs its branches side by side; `loop <until> … end` repeats. Frames nest.
+
+Services are known by their `context.service` id, `bus` and `client` by name;
+any other lane is declared under Participants, in the order the lanes should
+be drawn, as `- <id>: <kind> [in <context>] ["label"]`. Lines starting with
+`//` are comments. A mistake fails the run with its file and line, the way a
+compiler would: a flow silently left out is the kind of missing nobody
+notices. A ref that resolves to nothing is caught later, by the validator,
+because only the merged catalog can say.
+
+## Decisions written by hand
+
+Every decision worth keeping is already written down, in a file beside the code
+it constrains, in the MADR shape a person reads. Typing it a second time into
+the catalog's JSON - an id, a slug, a number, a scope and a body with every
+newline escaped - makes the JSON the source and the markdown a copy, and the
+copy is the one that goes stale: the three records that lived in
+`data/catalog.json` before `extract-adr` existed all named files that were
+never in the repository at all.
+
+So the markdown is the source and the fragment is the output. `extract-adr`
+reads `docs/adr/*.md` under a service's root - a `README.md` among them is the
+directory's index, not a decision, and is skipped - and answers with one
+fragment holding the records.
+
+```markdown
+# auth.0003 — Session expiry publishes no event
+
+- **Status:** accepted
+- **Date:** 2026-08-22
+- **Scope:** auth.auth
+- **Superseded by:** auth.0007
+- **Supersedes:** auth.0001, auth.0002
+- **Relates:** auth.auth.session.SessionEnded, shop.cart, checkout
+- **Note:** how a revocation is kept out of the cache was decided again in
+  auth.0010; the drop described below turned out not to be enough.
+
+## Context and Problem Statement
+…
+## Decision Outcome
+…
+```
+
+The title carries the record's id, an em dash, and the title. The id is a
+prefix and four padded digits, and the prefix is whatever the record is about -
+a service (`auth`, `cart`, `oms`), a context (`payments`) or the organisation
+(`org`). The file is named `NNNN-kebab-slug.md` with the same number, because
+the slug the catalog keeps is built from both: the id with its dots opened out,
+then the file's kebab, as in `auth-0003-expiry-publishes-nothing`. A file
+renamed away from its record would silently change the address of a decision
+somebody linked to, so the two are held against each other.
+
+`Status` and `Date` are required, and so is `Scope` unless the step's `scope`
+option names it for the whole tree; the rest are written when there is
+something to write, and a bullet the format does not have fails the record
+rather than being dropped. A bullet may wrap onto the next line, indented under
+itself - the break is the author's line width and closes up into a space.
+`Scope` says what the record is about by how many segments it has: none, or
+`org`, for the organisation, one for a context, two for `<context>.<service>`.
+`Relates` names events, services and flows in one list and they are told apart
+by their shape - a flow by its slug, which has no dots, a service by
+`<context>.<service>`, an event by the aggregate and `Name` after that -
+because an author should not have to remember which of three lists a name
+belongs in. `Note` is prose no other field holds, most often that part of a
+record was decided again somewhere else without the whole of it being
+superseded; it sits in the page's header, above the frozen body.
+
+Everything from the first `##` onward is the record. It goes into the catalog
+exactly as written and comes back out onto the page the same way, headings and
+all: an ADR is frozen history, and nothing on its page is redrawn from the
+model as it stands now. Prose above that first `##` is a mistake - a paragraph
+that drifted up there would be read by a person and dropped by the extractor.
+
+The [adr-tools](https://github.com/npryce/adr-tools) shape is read too: number
+and title on the title line, `Date:` above the record, status as the first
+section with its `Superseded by` / `Supersedes` link. It has no prefix or
+scope, so both come from the step's `scope` option.
+
+`src/catalog.ts` fails the whole app on load if a record breaks any of its
+rules, so the extractor checks first, where the file that caused it can be
+named: ids and slugs unique, an id ending in its own zero-padded number, a date
+that parses, a status from the five, and both halves of a supersession. A file
+that breaks one of them is left out with a warning naming the file and line,
+and the rest of the tree is read; only a supersession with one half recorded
+refuses the whole run. That is why `Superseded by` and `Supersedes` are two
+bullets rather than one derived from the other - supersession is a two-way
+fact, and half of it recorded is a bug. The halves that live in one step's
+tree are held against each other there; a record superseded by one in another
+service's tree is a claim only the merged catalog can check, and the validator
+checks it. The same goes for `Scope` and `Relates`: an extractor sees one root,
+so it validates the shape of a name and leaves whether the thing exists to the
+far side.
+
+The demo estate's org-wide and context-wide records live in `data/adr`, and are
+read by a step that points at that directory with a glob of its own. Root
+`docs/` is where `gen-markdown` writes, so nothing hand-written can live there.
+`in` is the directory of records rather than `data` itself: a step's fragment is
+only left out of its own stamp when the output is *inside* the input root, and
+`in: data` with `out: data` would be stamped from the file it writes.
+
+```json
+{
+  "plugins": [{ "name": "adr", "process": { "cmd": "go run ./plugins/extract-adr" } }],
+  "extract": [
+    {
+      "plugin": "adr",
+      "in": "data/adr",
+      "out": "data",
+      "options": { "files": ["*.md"], "out": "adr.json" }
+    },
+    {
+      "plugin": "adr",
+      "in": "examples/auth",
+      "out": "examples/auth/portolan",
+      "options": { "out": "adr.json" }
+    }
+  ]
+}
+```
+
+## Vocabulary written by hand
+
+A context's glossary is the one file in a service written for a person and read
+by everyone: what a word means inside the boundary it is spoken in. Nothing
+generates it and nothing should - a definition is a decision about language,
+not a fact about a type - so `extract-glossary` only reads. It takes
+`GLOSSARY.md` at a service's root and answers with a fragment of terms.
+
+```markdown
+# Glossary — auth
+
+One meaning per word inside this context.
+
+**Session.** Proof that a user logged in, how long that proof is good for, and
+whether it has been taken away.
+```
+
+A title, an optional line or two saying what the vocabulary covers, then one
+paragraph per term in alphabetical order. The paragraph opens with the term in
+bold and the full stop inside the bold, so `**Email address.**` names a
+two-word term and nothing has to guess where the name ends. Everything after it
+is the definition, carried through as written. Hard wrapping is the author's
+business: a soft break inside a paragraph is a space, here as in every markdown
+renderer.
+
+Nothing reads the definition for structure. A glossary is a person explaining a
+word to another person, and a parser that went looking for shapes inside the
+explanation would be a parser telling an estate how to phrase itself.
+
+The shapes a glossary is otherwise written in are refused by name: a table, a
+bullet list, a heading per term. Also refused: a file that does not open with
+`# Glossary`, an entry that defines nothing, and one word defined twice -
+inside a file or across the files of one step - because a word with two
+meanings in one context is the failure the glossary exists to prevent.
+
+What is merely untidy comes back as a warning and the fragment is still
+written: a file that has drifted out of alphabetical order, a root with no
+glossary at all.
+
+The term's id is `<context>.<slug>` - `auth.session`, `shop.order` - so the
+context has to be told to the step rather than derived from the directory: a
+glossary sits beside a SERVICE, and `examples/shop/oms/GLOSSARY.md` holds
+words that belong to `shop`. The same word in two contexts is two terms, which
+is the point of the id; the same word twice in one context is an error.
+
+```json
+{
+  "plugins": [{ "name": "glossary", "process": { "cmd": "go run ./plugins/extract-glossary" } }],
+  "extract": [
+    {
+      "plugin": "glossary",
+      "in": "examples/auth",
+      "out": "examples/auth/portolan",
+      "options": { "context": "auth", "out": "glossary.json" }
+    }
+  ]
+}
+```
+
+## Commands: what to type
+
+The first thing a reader new to a checkout wants is not the aggregate list but
+`how do I build this, test it, run it`. The README answers that sometimes; the
+runner files answer it always, and they are already in the tree. `extract-commands`
+reads them and puts the answer on the service as a list of commands - the line
+to type, what the file says it is for, what the runner would execute, and where
+it was read.
+
+| File | Runner | Line to type | Description read from |
+| --- | --- | --- | --- |
+| `Makefile` | make | `make <target>` | `## comment` on the target's line, else the `#` block over it |
+| `justfile` | just | `just <recipe>` | `[doc("...")]`, else the `#` block over it |
+| `Taskfile.yml` | task | `task <name>` | `desc`, else `summary` |
+| `package.json` `scripts` | npm, or pnpm/yarn/bun by the lockfile | `npm run <name>`; `npm test`, `npm start` | nothing - the script line is the body |
+| `pyproject.toml` | poe (`[tool.poe.tasks]`), pdm (`[tool.pdm.scripts]`) | `poe <name>`, `pdm run <name>` | `help` |
+| `pom.xml` | mvn, or `./mvnw` when the wrapper is there | `mvn test`, `mvn package`, and the goals of declared plugins: `spring-boot:run`, `quarkus:dev`, `flyway:migrate`, … | the goal's own purpose |
+| `build.gradle(.kts)` | gradle, or `./gradlew` | `gradle build`, `gradle test`; `run` with the application plugin, `bootRun` with Spring Boot; every `tasks.register("...")` | the task's `description` |
+| `.cargo/config.toml` | cargo | `cargo <alias>` for each `[alias]`; when one runs the `xtask` package, `cargo xtask <sub>` for each subcommand of its `main.rs` | `///` over a clap variant, `//` over a match arm |
+
+Left out, because the runner leaves them out too: `_`-prefixed and `[private]`
+entries, `internal` tasks, make's special and pattern targets (`.PHONY`, `%.o`),
+targets spelled through a variable, and `pre`/`post` hooks of a script that is
+itself listed. `[project.scripts]` in a pyproject is not read: those are
+programs the package installs, not tasks a developer runs. `uv` has no task
+section and Go has no runner of its own, so a repository with only those
+declares no commands.
+
+Maven and Gradle are the other way round: the build is the runner, and what
+a project declares is which plugins extend it. So a pom lists the two
+lifecycle phases every pom answers to, `test` and `package`, and then the
+goals a person types that its `<build><plugins>` add - `spring-boot:run` for
+the Spring Boot plugin, `flyway:migrate` for Flyway - from a short table of
+the plugins an estate meets. A plugin bound to a phase, like protobuf
+generation, is run by the phase and is not listed; neither is one under
+`<pluginManagement>`, which pins a version and runs nothing. A Gradle script
+lists `build` and `test`, `run` when it applies the application plugin,
+`bootRun` for Spring Boot, and every task it registers itself, with the
+description it sets. The script is read as text, not run: a task registered
+in a loop or by an unnamed plugin is not here.
+
+Cargo declares commands in one place, `[alias]` in `.cargo/config.toml`, and
+the xtask convention is an alias that runs a package: `xtask = "run --package
+xtask --"`. Each alias is a command with its expansion as the body, and when
+the expansion runs a package and leaves the subcommand to the caller, that
+package's `main.rs` is read for the subcommands - the variants of a clap enum
+deriving `Subcommand`, in kebab-case, with their `///` comment as the doc, and
+the string literals of a `match` on the first argument, with a `//` comment
+over the arm. An alias that names its subcommand already, like `gen = "run -p
+xtask -- gen"`, is listed as the one command it is.
+
+Nothing is evaluated. A target inside an `ifeq` is listed; a name that depends
+on a variable's value is not, because the name it would have is not in the
+file. The pyproject reader is a line scanner that follows the table headers
+and the shapes both runners document, not a TOML parser.
+
+The fragment names the service and lists its commands, and claims nothing
+else: merging puts the list on the service a domain extractor described. The
+runner files sit beside the service, so the step is told which one it is.
+
+```json
+{
+  "plugins": [{ "name": "commands", "process": { "cmd": "go run ./plugins/extract-commands" } }],
+  "extract": [
+    {
+      "plugin": "commands",
+      "in": "examples/shop/pricing",
+      "out": "examples/shop/pricing/portolan",
+      "options": { "context": "shop", "service": "pricing", "out": "commands.json" }
+    }
+  ]
+}
+```
+
+`extract-project` reads the same files for the component it describes, so a
+repository read by that plugin does not need this one.
+
+Downstream, `gen-markdown` draws the list as a Commands table on the service
+page, and `gen-backstage` puts it on the Component twice, because Backstage
+has no field for it: as the `portolan.io/commands` annotation, one command a
+line with its description, and as entity links of type `command`, each
+leading to the line of the runner file it was read from, when `sourceBaseUrl`
+says where the repository is.
+
+## Outside the estate: an external with a contract
+
+A service calls things nobody here builds - a card network, a tax API, a
+carrier. The first answer the catalog gave was `unknown`: the call was
+recorded and left unresolved, the lane drawn dashed and red, and the Problems
+page listed it beside real defects. True, and unhelpful, once the far end is
+Stripe and Stripe publishes a document.
+
+An **external** is what the catalog may claim about such a system, and no
+more: what it answers on, read from the copy of its document vendored beside
+the adapter that calls it, and what the manifest says it is called and is
+for. It sits at the root beside the contexts (`catalog.externals`) with a bare
+id, no context, no aggregates and no repository, and the estate's picture
+draws it outside, muted, exactly where it is.
+
+Two steps describe one, and neither knows the other exists:
+
+```json
+{ "plugin": "openapi", "in": "examples/payments/ledger",
+  "out": "examples/payments/ledger/portolan",
+  "options": { "external": "stripe", "externalName": "Stripe",
+               "externalUrl": "https://docs.stripe.com/api",
+               "spec": "src/main/java/.../infrastructure/stripe/openapi/openapi.yaml",
+               "out": "stripe.json" } },
+{ "plugin": "java-domain", "in": "examples/payments/ledger", "...": "...",
+  "options": { "externals": { "stripe.v1": "stripe" } } }
+```
+
+The first reads the copy and says what `stripe` answers on. The second reads
+the adapter, finds the verb and the route in each call, looks the operation up
+in the same copy, and - told by `externals` that the copy's api id belongs to
+`stripe` - records the call as `stripe.v1/PostPaymentIntents`, declared, on a
+lane of kind `external`. The merge joins the two by the id; a call to an
+operation the copy declares resolves, one it does not is reported by the
+extractor and left out.
+
+Neither line is needed when the tree can say it itself. An `openapi` step
+with no `spec` walks the tree for every document and reads what sits beside
+each one: a server generated from it - oapi-codegen's `ServerInterface`,
+swag's `docs` package, a handler or controller written against it - means the
+service implements it, and the document is what the service provides; a
+client generated from it - `ClientInterface`, a `client.gen.go`, an adapter
+named after a client - means the service calls it, and the document names a
+system this tree does not implement. That system is an external, with the
+id, the name and the summary the document gives itself: a copy titled
+"Gordian Flights & Ancillaries API" becomes `gordian-flights-ancillaries`,
+the trailing "API" dropped because the system is the thing and not its
+interface. `peers` on the same step says which called documents are ours -
+`{"auth.v1": "auth.auth"}` - and those are skipped, because the service that
+implements them describes them; `externals` names a system when the title
+would not, `{"stripe.v1": "stripe"}`. A document with neither a server nor a
+client beside it is reported and left alone, since reading it as either would
+be a guess.
+
+The domain extractor keeps the same rule from its side: a generated HTTP
+client whose api no `peers` line claims is read as calling the system the
+document beside it is titled after, and the call lands on an `external` lane,
+declared, under the id both sides derive from that title. `externals` on the
+domain step overrides the name; a proto client, whose contract names no
+system, stays unresolved until `peers` says who answers.
+
+The copy is **narrow** - the operations the service calls and the schemas they
+answer with, every field verbatim - for the reason org.0001 gives for a proto:
+what is vendored has to be reviewable, and Stripe's whole document is not. It
+carries one line the original does not, `x-portolan-api: stripe.v1` in
+`info`: the copy is already the consumer's translation boundary, so it is the
+one place the estate's name for the document is written, and every reader of
+the copy takes the id from there rather than from Stripe's title and version
+- or from two manifests that would have to agree.
+
+## Verifiers: the third phase
+
+An extractor reads source and runs before there is a catalog; a generator
+reads the catalog and writes pages. A **verifier** sits between them. It reads
+something observed - traces today, a test's record tomorrow - and answers with
+a fragment like an extractor's, but one that only makes sense against the
+merged catalog: "this hop was seen running" names a hop somebody else declared.
+So a `verify` step is handed both `input` and `catalog`, and the catalog it is
+handed leaves out the step's own last output. Without that, what it wrote last
+time would count as evidence this time, and the fragment could never be checked
+against a clean run.
+
+```json
+{
+  "plugins": [{ "name": "otel", "process": { "command": "go", "args": ["run", "./plugins/verify-otel"] } }],
+  "verify": [
+    {
+      "plugin": "otel",
+      "in": "examples/auth",
+      "out": "examples/auth/portolan",
+      "options": { "traces": ["telemetry/traces.jsonl"], "out": "observed.json" }
+    }
+  ]
+}
+```
+
+What comes back is merged like any other source, under two rules that exist
+for it. A flow declared twice is accepted when the second declaration differs
+only in status: `declared` steps become `verified`, and anything else that
+differs - a lane, a hop, a branch - is the conflict it always was. A consumer
+or a call declared twice keeps the first note and takes `verified` if either
+side has it.
+
+### verify-otel
+
+Reads OTLP JSON - one batch per file or one per line, as a collector's file
+exporter writes it - and turns each trace into hops between lanes:
+
+| span | hop |
+| --- | --- |
+| kind server, `http.route` | `client → service`, an rpc; matched to the operation whose `http` verb and path the OpenAPI extractor recorded, which is what opens an endpoint flow |
+| kind client, `rpc.service` + `rpc.method` | `service → provider`, an rpc; `unknown` lane and `unresolved` when nothing provides it, however often it ran |
+| `db.system.name`, `db.operation.name` | `service → its store`, a call; the statement nested under a query is not a second call |
+| kind producer, `event.name` | `service → bus`, the event whose name that is among the service's own; a producer span under another for the same name is the relay's and the same publish |
+| kind consumer, `event.name` | `bus → service`, and a `verified` consumer on the event |
+
+A trace whose root opens a declared flow raises the steps it shows: the call
+in, the events out, the rpcs with a ref. A `call` step is never raised - a
+`SELECT` ran, which is not the same claim as "the repository's `ByEmail` was
+called" - and `unresolved` is never raised, because a trace does not put the
+far end in the catalog. A consumer span inside a trace opens a flow of its own
+and is matched the same way, so one password change verifies both the
+request's flow and the policy's. A root no flow opens is written down as
+`observed-<service>-<route>`, once per shape, with a summary saying how many
+traces showed it.
+
+`service.name` is matched to the one service whose slug it is, `event.name` to
+the one event whose `wire.name` it is, or failing that to the one event of the
+publisher's with that last segment; `services` and `events` in the options say
+otherwise where an estate's names differ. A publish span whose
+`messaging.destination.name` is not the event's `wire.channel` is a warning:
+the event went out, but not where the code says it does.
+
+### verify-codeowners
+
+Reads the `CODEOWNERS` a repository already keeps and says who to ask about
+each service.
+
+"Who do I ask about `shop.oms`" was the question the estate answered worst.
+`owner` on a flow, a store or a module means the bounded context that holds it
+- a grouping, not a team - and there was nothing on any page a reader could act
+on. The answer was already written down, in the one file the forge itself
+enforces: a team that owns a directory is a team that gets the pull request.
+
+```json
+{
+  "plugins": [{ "name": "codeowners", "process": { "command": "go", "args": ["run", "./plugins/verify-codeowners"] } }],
+  "verify": [
+    { "plugin": "codeowners", "in": ".github", "out": "data", "options": { "out": "owners.json" } }
+  ]
+}
+```
+
+Point `in` at the directory the file is in, not at the repository root: the
+host dates a fragment from the last commit to touch the step's input, and the
+subject of this one is the `CODEOWNERS` file. Rooted at the repository, it
+would be restamped by every commit ever made. Left with no `file`, the three
+places a forge looks are tried in order - `CODEOWNERS`, `.github/CODEOWNERS`,
+`docs/CODEOWNERS` - and a `file` that names something absent fails the run,
+because answering "nobody owns anything" to a typo is only noticed a month
+later.
+
+It is a verifier and not an extractor because a rule is a path and only the
+merged catalog knows where each service is; an extractor would have to be told,
+service by service, in the manifest, what the catalog already says. It earns
+the name twice over. A service no rule matches is reported, and so is a rule
+that matches no service - a team believing it owns something the estate does
+not have, which is the one failure a `CODEOWNERS` file can never report about
+itself. A rule that matches and never wins is reported differently, because the
+fix is different: nothing is wrong with the path, everything it covers is just
+claimed by a rule below it.
+
+What lands on the catalog is `owners` on the service: handles exactly as the
+file spells them, `@acme/oms-team`, `@someone`, `dev@acme.io`. Deliberately
+nothing more - resolving a handle to the people currently in it is a call to a
+forge's API, which needs a credential this does not have and answers
+differently tomorrow, and a handle is what a reviewer types anyway. Two sources
+naming owners are unioned, because two rules that both matched are two facts
+and not two answers.
+
+The grammar is gitignore's, minus the parts CODEOWNERS does not have. A
+pattern owns a directory when it names the directory or anything above it, and
+does not when it names only something inside it: `services/oms/internal` is a
+rule about part of a service, and reading it as ownership of the whole would
+hand a team a page it never asked for. Later rules win. A pattern with no
+owners after it wins too - taking ownership back is the only reason anybody
+writes one. GitLab's sections change which rule wins and are read the flatter
+way GitHub means, with a warning saying so, because the difference only ever
+shows up as an owner quietly missing from a page.
+
+The demo estate's file is `data/codeowners/CODEOWNERS`, which is deliberately
+not one of the three places a forge reads: this repository is both the tool and
+the estate it describes, and a real `CODEOWNERS` here would ask GitHub to
+request reviews from teams that do not exist. A real repository puts it where
+the forge looks.
+
+## wasm or process
+
+`wasm` is the default and should stay that way. The module gets no filesystem,
+no network and no environment; a plugin from somebody else's repository can be
+run over your source tree without reading it.
+
+`process` is the escape hatch for a generator that needs a toolchain — one
+reading Go source has to run `go list`, and no wasm module can spawn anything.
+It gets the same protocol and none of the sandbox, which is the trade being made
+and the reason it is not the default. It declares `command` and an `args` array;
+the host never feeds a command string through a shell.
+
+A plugin fetched over `https://` must declare its `sha256`; the host verifies it
+and caches by digest. A `file://` plugin may declare one, but a checksum
+protects a download, not a module built from the source next to it.
+Downloads do not follow redirects and are bounded in time and size. Every run
+has a deadline and bounded stdout/stderr; wasm runs in a worker so even a module
+stuck in a loop can be terminated. Responses reject unknown properties,
+duplicate or unsafe filenames, and non-string contents before anything is
+written.
+
+## Services in other repositories: fetch-git
+
+`fetch-git` is `fetch-bsr` for a repository rather than a registry, and it
+lives by the same four rules. A pin is a repository, a commit and the paths
+actually read; the step fetches exactly those directories at exactly that
+commit and hands them back as files, so the host writes them into the tree
+beside a `git.lock.json` naming the commit and the digest of every file. The
+paths inside the copy are the repository's own, which is the point: the
+extract step that follows points its `in` at the vendored service and reads
+it exactly as it would read that service's checkout.
+
+```json
+{
+  "sources": ["data/*.json", "vendor/repos/*/*/git.repo.json"],
+  "plugins": [{ "name": "git", "process": { "command": "go", "args": ["run", "./plugins/fetch-git"] } }],
+  "extract": [
+    {
+      "plugin": "git",
+      "in": "vendor",
+      "out": "vendor/repos",
+      "options": {
+        "cache": "vendor/repos",
+        "repos": [
+          { "repo": "github.com/acme/shop", "commit": "c1d2e3f4…", "paths": ["services/oms", "proto"] }
+        ]
+      }
+    },
+    {
+      "plugin": "go-domain",
+      "in": "vendor/repos/acme/shop/services/oms",
+      "out": "data/shop",
+      "options": { "context": "shop", "service": "oms", "store": "pg" }
+    }
+  ]
+}
+```
+
+It runs the `git` the host already needs for stamps: a fetch of the one
+commit into a directory that exists for one call, and an archive of the paths
+wanted, read straight into memory. Whatever git is configured to do about
+credentials and hosts - a helper, a netrc entry, an ssh agent - it does here
+too, and the plugin reads none of it. `PORTOLAN_OFFLINE` (or any truthy `CI`)
+replays the committed copies against their locks; a commit the manifest does
+not pin is resolved online with a warning and refused offline; a fetch that
+fails falls back to the committed copy when there is one, and is a red build
+when there is not; a vendored file edited by hand is reported by path.
+
+### What the copy says about itself
+
+Two files land beside every copy, and the difference between them is who reads
+them. `git.lock.json` is for the next run of this step: the commit, and the
+digest of every file, which is what makes replaying the copy equivalent to
+fetching it again. `git.repo.json` is for the estate - a catalog fragment
+holding one line, the repository and the commit it is a copy of - which is why
+it is in `sources` above.
+
+Nothing else can say it. A service says which repository it lives in, and an
+extractor reads a directory as a pure function of what is on disk; neither has
+any idea which commit somebody fetched. Without that line, every source path
+of every vendored service is dead text on the page - the file and the line are
+known, and there is nowhere to send a reader - and every fragment read out of
+the copy is stamped with the commit that VENDORED it, so the service looks
+fresh whenever the fetch is re-run and unchanged when its own repository
+moves. With it, `sourceHref` links the line at the commit it was read at and
+`stampFor` dates the fragment from the code rather than the vendoring.
+
+One more line is needed for the app itself: `SOURCE_GLOBS` in `src/data.ts`,
+where the same patterns are written out a second time because
+`import.meta.glob` resolves at build time and needs literals.
+
+## Schema modules: fetch and parse, kept apart
+
+`fetch-bsr` and `extract-proto` are two plugins on purpose, and the split is
+the whole design.
+
+| | `fetch-bsr` | `extract-proto` |
+| --- | --- | --- |
+| job | registry wire → `.proto` bytes | `.proto` bytes → catalog fragment |
+| network | yes | never |
+| environment | reads `BUF_TOKEN` | never |
+| output | `.proto` files and a `bsr.lock.json` per module | one catalog fragment |
+| deterministic | only because it is pinned and cached | absolutely |
+
+Extraction stays a pure function of the tree. Fetching is the step that can
+fail, need a credential, or come back with something different than it did
+yesterday, and confining that to its own step is what lets everything after it
+be replayed byte-for-byte from a checkout.
+
+**The fetched protos are the plugin's `Response.Files`, not a side effect.** The
+host writes them like any other generated file, so they get a manifest entry,
+are compared by `gen:check`, and are removed when the step stops naming them.
+The cache is not a second copy of anything — it *is* the tree. Refreshing a pin
+produces one pull request holding the pin bump, the proto diff, the lock diff
+and the fragment diff, which is the review worth having.
+
+Declare the fetch step **before** the extract step: steps run in list order, so
+its protos and locks are on disk by the time the parser reads them.
+
+```json
+{
+  "plugins": [
+    { "name": "bsr",   "process": { "command": "go", "args": ["run", "./plugins/fetch-bsr"] } },
+    { "name": "proto", "process": { "command": "go", "args": ["run", "./plugins/extract-proto"] } }
+  ],
+  "extract": [
+    {
+      "plugin": "bsr",
+      "in": "examples/shop",
+      "out": "examples/shop/vendor/proto",
+      "options": {
+        "cache": "examples/shop/vendor/proto",
+        "modules": [
+          { "module": "buf.build/acme/shop", "commit": "c1d2e3f4a5b6c7d8e9f0a1b2c3d4e5f6" }
+        ]
+      }
+    },
+    {
+      "plugin": "proto",
+      "in": "examples/shop",
+      "out": "examples/shop/portolan",
+      "options": {
+        "context": "shop",
+        "service": "oms",
+        "paths": ["vendor/proto/acme/shop"],
+        "vendored": ["internal/infrastructure/pricing"],
+        "peers": { "pricing.v1": "shop.pricing" },
+        "out": "proto.json"
+      }
+    }
+  ]
+}
+```
+
+`cache` repeats the step's `out` because a plugin is never told where its output
+goes. That is the shape of the protocol — a plugin returns files and the host
+decides what to do with them — and not an oversight to work around.
+
+### Why fetch-bsr can never be wasm
+
+It needs a socket and a credential. `process` exists for exactly that trade, and
+`auth.go` is the only file in either plugin that reads the environment. The
+protocol's "no ambient state" rule is about *facts*: nothing about the estate may
+come from anywhere but the request. A credential is not a fact about the estate —
+it decides whether the fetch succeeds, never what the fetch says — and a test
+asserts the output is byte-identical with and without a token.
+
+### Pinning, and the offline rule
+
+Pin every module to a commit. A BSR commit is immutable, so a pinned download is
+byte-reproducible, which is the only reason replaying from disk is equivalent to
+fetching again. An unpinned module is resolved and warned about when online, and
+refused when offline — there is nothing to replay against.
+
+`PORTOLAN_OFFLINE=1` (or any truthy `CI`) turns the fetch off. The step then
+re-emits the committed copies, checked against their locks. Set it in CI; the
+workflow already does.
+
+Four rules govern what happens when a fetch does not:
+
+1. Fetch succeeded → the fetched files and a regenerated lock.
+2. Skipped or failed, cache complete and matching its digests → the cached files
+   byte-identically, plus a warning. Output unchanged, so `--check` stays clean.
+3. Failed **and** no usable cache → a non-zero exit, never a short file list. The
+   host deletes files a step stops naming, and dropping a repository's vendored
+   protos because a laptop went offline is worse than a red build.
+4. A cached file whose digest no longer matches is reported by path — someone
+   edited a vendored copy, which is the drift `docs/adr/org.0001.md` wants seen.
+
+### Why the proto parser is hand-written
+
+`docs/adr/org.0001.md` has consumers keeping *narrowed* vendored copies, and a
+narrowed copy routinely imports a file nobody vendored beside it. A compiler —
+`protocompile`, `protoc` — refuses to produce anything for that input. The whole
+point of reading vendored copies is to describe files that do not build
+standalone, so the parser is tolerant: every construct it declines to model
+(`extend`, a proto2 `group`, an aggregate option body) is named in a diagnostic
+rather than dropped, and only a file that cannot be tokenised is fatal.
+
+It also keeps types **as written** — `[]LineItem`, `map[string]Money`, the
+`optional` keyword, the author's declaration order — the same stance the catalog
+already takes for schemas, and one a descriptor has already thrown away.
+
+### What extract-proto will not claim
+
+`status` is only ever `declared` or `unresolved`, **never `verified`**. Reading a
+`.proto` proves a call was written down. `verified` in the shipped catalog means
+a test exercises it end to end, which is a property of the merged catalog — and
+every extractor runs before one exists.
+
+A message named `OrderPlaced` in an `events.proto` stays an `RpcMessage` and does
+not become a catalog `Event`. `Event.id` is `<service>.<aggregate>.<Name>` and
+this extractor knows the package, not the aggregate; a guess would collide with
+the event `extract-go` already emits, or invent a ghost aggregate that would sit
+beside the real one forever.
+
+## Schema registry subjects: the same split, one topic further
+
+`fetch-csr` and `extract-csr` are the Confluent Schema Registry half of the same
+argument, and they are shaped like `fetch-bsr` and `extract-proto` on purpose:
+one owns the socket and the credential, the other owns the reading, and CI runs
+the reading over a tree it can verify without a registry existing at all.
+
+| | `fetch-csr` | `extract-csr` |
+| --- | --- | --- |
+| job | registry wire → schema files | schema files → catalog fragment |
+| network | yes | never |
+| environment | reads `CSR_API_KEY`/`CSR_API_SECRET`, or `CSR_TOKEN` | never |
+| output | one schema and a `csr.lock.json` per subject | one catalog fragment |
+| deterministic | only because it is pinned and cached | absolutely |
+
+A registered version is immutable: subject `orders-value` at version 3 is the
+same bytes today and next year, and re-registering a changed schema makes
+version 4. That is the promise a BSR commit makes, so the same four rules govern
+a fetch that does not happen, and `PORTOLAN_OFFLINE=1` replays the committed
+copies against their locks exactly as it does there.
+
+A subject a fetched schema **references** is fetched too, pinned by the
+reference rather than by the manifest, and needs no entry of its own — the
+version is part of the bytes we already have, so following one adds no lottery.
+Each lands in its own directory, and the referring subject's lock is what an
+offline run follows to find them.
+
+Avro and JSON schemas arrive minified onto one line. They are written out
+indented — with `json.Indent`, which reformats without reordering, so the file
+still says what the registry said in the order it said it — because a version
+bump that is one unreadable line is a review nobody can do. The digest is over
+the bytes as written, so verifying needs no reformatting of anything.
+
+### The strategy the registry does not record
+
+A subject name is whatever the producer's serializer decided to call the
+registration, and the rule it used — the `SubjectNameStrategy` — is nowhere in
+the registry's answer. `shop.oms.order-value` is a topic plus a suffix under
+`TopicNameStrategy` and a record's full name under `RecordNameStrategy`, and
+nothing but the manifest can say which. So `strategy` is an option, and
+everything `extract-csr` does with a name follows from being told it.
+
+Under `topic-record` the separator is a hyphen and both halves may contain one,
+so the split is made by matching the **schema's own full name** as the suffix
+rather than by searching for a delimiter.
+
+A `-key` subject is kept as a shape and put on no channel: a key is part of
+every message on the topic, not a message on it. Under `record` there is no
+topic at all, and a fragment with no channels is the right answer rather than a
+gap — that strategy exists so a record can be reused across many.
+
+### What extract-csr will not claim
+
+**It emits no events.** An `Event` in the catalog belongs to an aggregate, and a
+registry holds schemas, not domains — it has no idea which aggregate raises
+what. So the shapes land in `defs`, where a shared shape belongs, and the topics
+land in the service's channels beside the ones an AsyncAPI document declares.
+The domain extractor says an aggregate raises `OrderPlaced` and calls it
+`shop.oms.OrderPlaced` on the wire; this says a schema by that name is
+registered against topic `shop.oms.order` and has these fields. Neither knows
+the other exists, and the pages hold the two against each other.
+
+**It does not say who produces.** A registry records no producer and no
+consumer. `direction` is told, per step and per subject, or it would be invented.
+
+**It does not parse protobuf.** That is `extract-proto`'s whole job, and a
+second, worse parser here would be a second answer to one question. A `PROTOBUF`
+subject still names its topic — which is the one thing a `.proto` file cannot
+say — and a diagnostic points at `extract-proto` for the fields.
+
+A field referencing a shape nothing in the estate vendored keeps its **name**
+and loses its **ref**. The catalog validates that every ref resolves, and
+failing a run over a reference that is genuinely true — the shape really does
+live in another estate — would be the wrong end of the trade.
+
+## The bus: a channel is a claim, not an event
+
+`extract-asyncapi` reads an AsyncAPI document and answers with the channels a
+service declares — the address the broker knows, and each message on it with the
+direction it travels. What it does **not** answer with is events.
+
+That looks like a gap and is a boundary. `Event.id` is
+`<service>.<aggregate>.<Name>`, and an AsyncAPI document knows the message on the
+wire, not the aggregate that raised it. An extractor that guessed would either
+collide with the event `extract-go` and `extract-ts` already emit or invent a
+ghost aggregate that would sit beside the real one forever — the same rule
+`extract-proto` keeps about a message called `OrderPlaced`.
+
+So the two sources meet in the merge instead, and the pages hold them against
+each other. The domain says an aggregate raises `BasketCreated` and how it leaves,
+in `wire`; the document says the service sends `cart.BasketCreated` on
+`shop.cart.basket`. Where they agree the catalog says the same thing twice, which
+is worth nothing. Where they disagree it is worth a row on the Problems page,
+because one of the two is stale:
+
+- an event whose channel the document does not declare — a subscriber reading
+  the document does not know the message exists;
+- a channel the document declares and no event names — a promise nothing keeps;
+- a message the document listens for that nothing in the estate publishes.
+
+That last one is the only edge in the catalog that runs from the subscriber
+outwards. Everywhere else a publisher names its consumers; here the subscriber
+names a message and the estate is searched for whoever puts it on the wire. A
+subscription that resolves is how two repositories that never mention each other
+are found to be joined — and a channel that two services both declare a send on
+is a second publisher, which is an error for the reason a second writer in a
+database is.
+
+### 2.x says publish and subscribe backwards
+
+In AsyncAPI 3.x an operation carries `action: send` or `action: receive`, from
+the application's side, and there is nothing to get wrong. In 2.x a channel has
+`publish` and `subscribe`, and both are written from the **client's** side:
+`publish` is what somebody else publishes *to* the application, so the
+application receives it, and `subscribe` is what the application produces for
+somebody else to subscribe to.
+
+Reading 2.x the obvious way puts every arrow in the estate the wrong way round.
+The extractor reads both versions and answers in 3.x's vocabulary, which is the
+one the catalog keeps.
+
+### A work queue is a channel of kind `job`
+
+A task queue is a channel too - an address the broker knows, messages that
+travel on it - with one difference the catalog has to be told: many callers
+put the same job on it by design, and no domain event stands behind a job.
+`kind: "job"` on the channel says so. The merge does not call two senders on a
+job queue rival publishers, and the Problems page does not look for an event
+with the job's wire name. `extract-celery` answers with these for a Python tree:
+one channel per queue, a `send` per task the tree enqueues and a `receive` per
+task it declares, and one flow per task that is both - the call that enqueues
+it, then the worker that runs it. It reads the queue the way Celery decides
+it, the call before the decorator before `task_routes` before the default,
+and `transaction.on_commit(...)` around an enqueue is a note on the step,
+which is the one fact about *when* a message leaves that the code states
+plainly.