@shortlink-org/portolan 0.2.4 → 0.4.0

package/plugins/README.md

@@ -18,7 +18,7 @@ the plugin itself rather than its work:
 → { "portolanVersion": "0.1.0", "kind": "describe" }
 ← { "files": [],
     "describe": { "name": "extract-go", "summary": "...",
-                  "phases": ["extract"],
+                  "category": "code", "phases": ["extract"],
                   "options": { "type": "object", "additionalProperties": false,
                                "properties": { "context": { ... } } } } }
 ```
@@ -31,6 +31,16 @@ is what makes that worth having: `encoding/json` drops a field it does not
 recognise, so before this a misspelled option was no option at all and nothing
 said so.
 
+`category` is what the plugin reads or makes, one of `code`, `contracts`,
+`messaging`, `data`, `infrastructure`, `repository`, `documents`, `evidence`,
+`sources` and `exports` (the constants in `plugin/describe.go`). A phase says
+where in a run a plugin goes; the category says what kind of fact it is after,
+which is how the site's `/plugins` page groups it. `npm run schema` also writes
+`src/lib/plugin-index.json` - every declared plugin as it describes itself,
+with how the manifest runs it - and that page and the landing page's list of
+inputs are rendered from it, so a plugin added to `portolan.json` appears on
+both without being written down twice.
+
 A descriptor may also carry `needs`: what the host must put in the request
 beyond the tree, because a sandboxed module cannot reach it. The one need so
 far is `history` (portolan.0007) - when each file under the root was first
@@ -54,7 +64,10 @@ Three obligations, and they are the whole of it:
    unsafe filename or invalid response fails the run. Non-fatal extraction
    notes go to stderr; there is no advisory response property a caller may
    accidentally ignore. A note that opens with `warning: ` is kept beside the
-   step in `.portolan/build-report.json` and listed on the Settings page.
+   step in `.portolan/build-report.json`. The Settings page classifies and
+   groups those notes by plugin, stable rule and severity. Typed CEL policies
+   in `portolan.json` can suppress reviewed diagnostics only with an explicit
+   reason; the host evaluates them and records the decision in the build report.
 
 The repository enforces those obligations with schema/field coverage tests,
 byte-for-byte permutation tests, generated-link and anchor checks, Mermaid
@@ -78,12 +91,14 @@ either be rendered or be explicitly acknowledged by the relevant exporter.
    `go:embed` and returned in the descriptor. `schematest.Check` in a test keeps
    it from drifting from the options struct: a field renamed on one side and not
    the other fails, and so does an option with no description.
-3. Build it. A built-in Go plugin is a library package with
+3. Build it. A sandboxed built-in Go plugin is a library package with
    `Serve(io.Reader, io.Writer) error`; add it to the map in
    `plugins/cmd/portolan-go/main.go`, and `plugins:build` in `package.json`
    puts it in `plugins/portolan-go.wasm` with the rest (a test keeps the map
-   and `portolan.json` in step). A plugin of your own is its own module:
-   `GOOS=wasip1 GOARCH=wasm go build`.
+   and `portolan.json` in step). A built-in that genuinely needs the Go
+   toolchain gets a small command under `plugins/cmd/` and is declared as a
+   `process`; `http-clients` is the example. A plugin of your own is its own
+   module: `GOOS=wasip1 GOARCH=wasm go build`.
 4. Declare it in `portolan.json`, under `plugins` (how to run it) and
    `generate` (what to run it on), then run `npm run schema` so the manifest
    schema learns its options.
@@ -149,7 +164,14 @@ a value's name is the constant's literal, because that is what the wire
 carries, and the constant's own name only for an iota. A `Deprecated:`
 paragraph in the doc marks the value. In Rust it is a `pub enum` whose every
 variant is a bare name, the literal an `as_str` answers standing in for the
-variant. In Java it is a top-level enum in the aggregate's package. In proto,
+variant. In Java it is a top-level enum in the aggregate's package. In PHP it
+is a backed `enum` anywhere in the module, and on an Eloquent model the
+constants that share a prefix - `STATUS_PENDING = 'pending'` beside
+`STATUS_CLOSED = 'closed'` - with `@deprecated` in a constant's docblock
+marking the value; see `extract-laravel/README.md`. In a PHP tree laid out
+by bounded context (`extract-php-ddd`) it is a backed `enum` under a module's
+`Domain/`. In a .NET modular monolith (`extract-csharp-ddd`) it is a C# `enum`
+in the aggregate's directory, `[Obsolete]` marking a value. In proto,
 the enums the messages reach through their fields sit on the interface as
 `enums`, with the numbers the wire uses.
 
@@ -179,10 +201,23 @@ the original collection layout for TypeScript, Rust, Java, or custom trees and
 may also point directly at one feature repository.
 
 `extract-river` is another independent enrichment for Go repositories. It
-joins a job argument's `Kind()` to `Client.Insert`/`InsertTx`, the selected
-queue, `Worker[Args].Work`, and `river.AddWorker`. The result is a work-queue
-channel plus a two-hop enqueue/dispatch flow, with payload fields and source
-lines. It does not need aggregates and does not treat a job as a domain event.
+joins a job argument's `Kind()` to `Client.Insert`/`InsertTx` and the
+`InsertMany*` batches, the selected queue, `Worker[Args].Work`, and
+`river.AddWorker`. The result is a work-queue channel plus a two-hop
+enqueue/dispatch flow, with payload fields and source lines. It does not need
+aggregates and does not treat a job as a domain event. The job type is
+followed through locals, constructors' results and parameters up to the
+callers, so an `Enqueue(ctx, args river.JobArgs, opts)` wrapper is read once
+per concrete job passed to it, and a worker registered as
+`river.AddWorker(workers, NewSendWorker(deps))` is registered. The queue is
+the `InsertOpts.Queue` at the call (a literal, a constant, a config `default:`
+tag, or what the callers pass), else the args type's own `InsertOpts()`, else
+`default`. What no Insert in the tree feeds is not dropped: a registered
+worker's queue is the one its type chooses or the one queue `river.Config`
+lists, and with several and no choice the worker is kept as a flow whose
+receive is unresolved. The diagnostics keep the two cases apart - an Insert
+whose job type or queue the reader could not follow is said at the call, a
+worker with no Insert in the tree is said at the worker.
 
 `extract-redis` finds runtime construction of go-redis, rueidis and redigo
 clients in non-test Go source. That source evidence adds a service-owned Redis
@@ -193,16 +228,27 @@ key patterns with their read/write/delete use, TTL, value type and source. The
 patterns remain Redis keyspaces rather than being presented as SQL tables.
 
 `extract-watermill` reads Watermill `Router.AddHandler` and
-`AddNoPublisherHandler` registrations. It resolves literal and constant topics,
-plus defaults on env-config structs, follows direct `Publisher.Publish` calls
-and one-hop publishing helpers, and traces JSON marshal/unmarshal values back to
-their Go payload structs. It carries enclosing `if`/`else` conditions and early
-returns through the handler's control flow: proven alternatives become one
-catalog `alt`; publications whose relationship cannot be proven remain separate
-possible routes. The channels merge normally with AsyncAPI declarations by address. Generic
-`NewEventHandler[T]` and `NewCommandHandler[T]` registrations on Watermill CQRS
-processors are also extracted; fixed topic generators and the standard
-event/command-name generator form are resolved from source.
+`AddNoPublisherHandler` registrations, and `Subscriber.Subscribe` calls made
+without a router. It resolves literal and constant topics - typed constants
+through `string(...)`, constants concatenated with `+`, single-return
+helpers - plus defaults on env-config structs, nested (`cfg.Kafka.Orders`) or
+not; a topic, a name or a handler that a registering function takes as a
+parameter is filled in once per caller, so a `handle(r, name, topic, fn)`
+wrapper yields one handler per call. The handler value may be a function
+literal, a function or method of the tree (`h.OnOrder`, `handlers.OnShipped`)
+or a constructor returning a literal. It follows direct `Publisher.Publish`
+calls and one-hop publishing helpers, records what the service publishes
+outside any handler as a `send` on the topic, and traces JSON marshal/unmarshal
+values back to their Go payload structs. It carries enclosing `if`/`else`
+conditions and early returns through the handler's control flow: proven
+alternatives become one catalog `alt`; publications whose relationship cannot
+be proven remain separate possible routes. A handler whose topic does not
+resolve is kept, with its receive step unresolved and no channel claimed, and
+named in a diagnostic. The channels merge normally with AsyncAPI declarations
+by address. Generic `NewEventHandler[T]` and `NewCommandHandler[T]`
+registrations on Watermill CQRS processors are also extracted; fixed topic
+generators and the standard event/command-name generator form are resolved
+from source.
 
 `extract-go-nats` reads nats.go and JetStream calls into the subjects a service
 listens on and publishes to. A call is known by the type it is made on -
@@ -219,6 +265,61 @@ channel's doc. A subject read off a database row is a warning at the call,
 not a channel. Consumer configs give the filter subject and the durable name;
 streams, wildcard subjects and work-queue retention are not read yet.
 
+`extract-go-sqs` is its twin for Amazon SQS through aws-sdk-go-v2. A call is
+known by being made on `*sqs.Client` - `SendMessage`, `SendMessageBatch`,
+`ReceiveMessage` - and the queue is the `QueueUrl` of the input struct,
+written inline or built into a local first. `aws.String` and `aws.ToString`
+are taken off, and the `QueueUrl` of a `GetQueueUrl` result is read as the
+`QueueName` that was asked for. The queue is then followed like a subject: to
+a literal, a constant, a config default, or a parameter up to two hops through
+the callers, and when it sits in a field of the receiver, to the constructor
+that filled the field and on to that constructor's callers - the worker's
+shape, where the queue comes in at `New` and is read at `Run`. A URL becomes
+the queue's name, its last path segment. Channels are `message` streams, not
+domain events: a queue is point to point, and the event model is not asked to
+match it. A port that takes exactly one other string beside the queue names
+the message; a direct call names none. `DeleteMessage`, visibility changes and
+`CreateQueue` say nothing about direction and are not read; SNS is not read.
+
+`extract-terraform` reads the other half: the infrastructure the code runs
+on, from the `*.tf` files of the directory given as `dir` - or the root, or
+the one directory under it that holds any. Nothing is evaluated; the files
+are read as syntax with `hashicorp/hcl`, and a name is followed the way a
+subject is - to a literal, a variable's default, a local, the argument a
+calling `module` block passed in, or a child module's output - and left out
+with a warning when none of those reaches it, never taken from the label.
+Local modules (`source = "./..."`) are read where they are called. A registry
+module the reader knows - `terraform-aws-modules/lambda`, `sqs`, `sns`,
+`s3-bucket` with its `notification` submodule, `dynamodb-table`, whether
+named through the registry or as the git repository behind it with a ref -
+is read as the resource it wraps: its inputs are that resource's attributes
+under the module's names (`environment_variables`, `attributes = [...]`,
+`event_source_mapping = {...}`, `subscriptions = {...}`, `create_dlq`), and
+its outputs (`lambda_function_arn`, `queue_arn` or `sqs_queue_arn`,
+`dead_letter_queue_arn`, `dynamodb_table_stream_arn`) are references to it,
+so `module.queue` reads as `aws_sqs_queue` would; the dead-letter queue the
+queue module makes beside itself is `module.queue.dlq`, and a layer the
+lambda module makes is not a function. Any other registry module is noted
+and skipped. A reference from one resource to another
+(`aws_sqs_queue.orders.arn`) is the edge itself and needs no value. What
+comes out: an `aws_sqs_queue` is a `message` channel and an `aws_sns_topic`
+an `event` channel, each with `messages: []` and the facts in the doc - FIFO,
+the dead-letter queue and its receive count from `redrive_policy`, the SNS
+subscription that fills a queue, the S3 notification that fills either; an
+`aws_lambda_function` is a component of kind `function` in the same context,
+its runtime among its technologies - or the service itself when its name
+slugs to the `service` option; an `aws_lambda_event_source_mapping` puts the
+queue on the function as received, and a topic reaching the function through
+a subscribed queue is listed on it too; a value in the function's
+`environment` that reaches a queue, topic, table or bucket lists that on the
+function with the variable's name, and without a direction, because
+Terraform does not know one. `aws_s3_bucket`, `aws_dynamodb_table` (keys and
+secondary indexes as the table), `aws_db_instance` and `aws_rds_cluster`
+(kind from `engine`) are stores at the root, owned by the `service` option;
+a queue or topic nothing in the module touches is listed on that service as
+well, and says so. Kinesis, EventBridge, API Gateway, Step Functions,
+ElastiCache, MSK and ECS resources are named once as not read yet.
+
 `extract-python-kafka` is the framework-independent Kafka enrichment for
 Python. It recognizes confluent-kafka, kafka-python and aiokafka by their
 imported client types, follows literal topics through constants, environment
@@ -260,16 +361,19 @@ literals, direct assignments, or setters. A standalone flow says whether no
 source caller exists or callers exist but no inbound/asynchronous root was
 proved, so the UI exposes the missing evidence instead of implying a complete
 business path.
-When a provider branch still ends before its transport, the extractor loads
-the module with `go/packages`, builds SSA, and uses `x/tools` VTA to resolve
+When a provider branch still ends before its transport, the extractor runs as
+a workspace-local native sidecar, loads the module with `go/packages`, builds
+SSA, and uses `x/tools` VTA to resolve
 calls through interface parameters, function values, return values, and
 interface-typed struct fields. Typed edges are followed only after a concrete
 factory branch is selected: applying context-insensitive VTA to a shared
 dispatcher would otherwise attach every request implementation to every
 endpoint. Factory conditions and HTTP/SOAP meaning continue to come from the
-source extractor. Module loading is read-only and bounded; unavailable private
-dependencies, type errors, or a timeout produce a warning and retain the
-syntax-only result rather than failing generation.
+source extractor. The sidecar is built from the Portolan package through the
+project's Go toolchain and runs with the project as its working directory.
+Module loading is read-only and bounded; unavailable private dependencies,
+type errors, or a timeout produce a warning and retain the syntax-only result
+rather than failing generation.
 Routes without a provider factory are also joined to their outbound calls,
 including handlers invoked from closures and methods on locally constructed
 values. A Swagger `@Router` annotation is medium-confidence root evidence for
@@ -454,9 +558,10 @@ far side.
 The demo estate's org-wide and context-wide records live in `data/adr`, and are
 read by a step that points at that directory with a glob of its own. Root
 `docs/` is where `gen-markdown` writes, so nothing hand-written can live there.
-`in` is the directory of records rather than `data` itself: a step's fragment is
-only left out of its own stamp when the output is *inside* the input root, and
-`in: data` with `out: data` would be stamped from the file it writes.
+`in` is the directory of records rather than `data` itself: what `gen --check`
+lists as changed inputs is the root less the output directories inside it, and
+`in: data` with `out: data` would list the file the step writes among the
+reasons it was rewritten.
 
 ```json
 {
@@ -621,6 +726,178 @@ line with its description, and as entity links of type `command`, each
 leading to the line of the runner file it was read from, when `sourceBaseUrl`
 says where the repository is.
 
+## Where it answers, what it dials: the manifests, and the cluster
+
+A catalog that knows what a service provides and what it consumes still has
+holes where the two do not meet by route alone: two services answer
+`POST /foo`, and the caller's code says only that it posts to `$LEDGER_URL`.
+What that variable holds is not in the code. It is in the Deployment, the
+ConfigMap it reads, and the Service and Ingress objects that give the far end
+a name. Two plugins read those - `extract-k8s` from the manifests in a
+repository, `fetch-k8s` from a live cluster - and put two lists on the
+service:
+
+- **Hosts** - the names it answers on: a Kubernetes Service's name in its
+  short, namespaced, `svc` and fully qualified forms, and the hosts of the
+  Ingress or Gateway API HTTPRoute in front of it, from a rule whose backend
+  is that Service.
+- **Dials** - the in-cluster names its workload is configured to reach, cut
+  out of its containers' `env` and the ConfigMaps they read.
+
+The merge then has both facts in one place. A call that names a host, or a
+caller whose manifests dial one, resolves against the providers of its route
+when exactly one of them answers on that name, and the evidence says
+`kubernetes-host` as the basis. A host only ever decides between providers of
+the route; it never conjures a provider that does not have it.
+
+### What is never written
+
+A manifest holds configuration, and configuration is where the passwords are.
+The rule that keeps them out is a rule of shape, not a list of words:
+
+- A `Secret`, a `SealedSecret`, an `ExternalSecret`, or any document with
+  `stringData` is passed over at its `kind`, before its body is decoded. A
+  `secretKeyRef` or `secretRef` is not followed.
+- A value from `env` or a ConfigMap is reduced to the host it names, or to
+  nothing. A URL gives up its hostname and keeps nothing else - not the
+  scheme, the port, the path, the query, nor the user and password in front
+  of the host. A bare `host[:port]` gives up the port. `info`, `true`, a
+  token, a DSN written as `key=value` pairs: none of these is a host, and
+  none passes.
+- What is left qualifies only when the cluster resolves it: a form the tree's
+  own Services answer to, or `<name>.<namespace>.svc[.cluster.local]`, the
+  one shape no name outside a cluster has. A bare `cart` from another
+  repository's manifests is not taken, because nothing distinguishes it from
+  a word.
+- Warnings name files and directories, never values, because the build
+  report is committed.
+
+The fragment carries the variable's name no more than its value: the
+`environmentVariable` the HTTP client extractor records stays what it was, and
+no value is attached to it.
+
+### The manifests: extract-k8s
+
+Every `*.yaml` and `*.yml` under the root, or under the `paths` named in the
+options; anything without `apiVersion` and `kind` - a compose file, a
+workflow - is passed over silently. Multi-document files are split. A file
+holding `{{` is a template and not YAML until rendered, so a directory of
+Helm templates is passed over with one warning for the directory; render the
+chart into the tree, or point `paths` at plain manifests, to have it read.
+Kustomize is read without running it: the base and the overlays are all under
+the root, and two documents of one kind and name - a Deployment and the patch
+laid over it - fold into one, their env and labels unioned. `namespace`
+narrows the read to one namespace; an object naming none is read either way.
+
+The workload the step is about is the Deployment, StatefulSet, DaemonSet,
+ReplicaSet, Job or CronJob named like the service, or labelled so with
+`app.kubernetes.io/name` or `app`, or the only one in the tree. A Job or
+CronJob makes the service's kind `job`; a Deployment says nothing about kind,
+because a webapp and a worker both deploy that way. Several workloads and
+none named is a warning, not a guess.
+
+### The cluster: fetch-k8s
+
+The manifests say what should run; the cluster says what does, and for an
+estate deployed by Helm or Argo the rendered objects are in no repository at
+all. `fetch-k8s` asks `kubectl` for the workloads, Services, Ingresses,
+Gateway API routes and ConfigMaps of the namespaces named - never for a
+Secret - and answers with one fragment for every workload it saw, placed
+into contexts and services by label. It runs in the host (portolan.0008)
+because it needs the binary, the socket and kubectl's own credential, which
+comes from the kubeconfig and is never in the manifest.
+
+```json
+{ "name": "k8s-cluster", "host": "fetch-k8s" }
+
+{
+  "plugin": "k8s-cluster",
+  "in": ".",
+  "out": "data",
+  "options": {
+    "kubeContext": "prod-eu",
+    "namespaces": ["shop", "auth", "payments"],
+    "labels": { "context": "app.kubernetes.io/part-of", "service": "app.kubernetes.io/name" },
+    "namespaceContexts": { "shop-jobs": "shop" },
+    "cache": "data",
+    "out": "k8s-cluster.json"
+  }
+}
+```
+
+The context is the value of the `app.kubernetes.io/part-of` label and the
+service the value of `app.kubernetes.io/name`; an estate that labels
+differently names its own keys under `labels`. A workload without the
+context label falls back to `namespaceContexts`; one with neither is counted
+in a warning for its namespace and left out, not guessed at. A workload
+without the service label is named after itself. Several workloads landing
+on one service - a Deployment and the CronJob beside it - fold into one; a
+service that is only jobs is a `job`, one with a Deployment beside them is
+whatever its code says.
+
+The same offline rule as fetch-git: with `PORTOLAN_OFFLINE` set, or in CI,
+the fragment committed at `cache` is used unchanged; when the cluster cannot
+be reached it is used with a warning that names the failed command and its
+exit status, not the server; with no committed copy, that is an error and
+not a short answer. What kubectl itself said on stderr goes to the terminal,
+where the person at the keyboard can read the connection error, and not into
+the build report, which is committed. Nothing but the fragment is written to
+disk: the objects live in the process for the length of one call.
+
+portolan.0011 records why both readers keep names and nothing else.
+
+## Where it runs: the GitOps tree, and the deployer
+
+A service's manifests say what it is; what deploys them says where it
+stands - which environment, cluster and namespace, from which directory of
+which repository, tracking which revision. Two plugins read that, and the
+merge lays one under the other (portolan.0012, portolan.0013):
+
+- **extract-argocd** reads the Argo CD `Application` and `ApplicationSet`
+  objects in a GitOps tree, as wasm over the workspace, and answers with
+  what *should* run: one deployment per Application, basis `manifest`. An
+  ApplicationSet is expanded as far as a tree allows - `list` generators
+  whole, `git` directories and files by walking this repository, `matrix`
+  over those, the template rendered as Go `text/template` or fasttemplate
+  as it declared. A `clusters`, `pullRequest`, `scmProvider`, `merge` or
+  `plugin` generator, or a `git` one pointing at another repository, is a
+  warning naming the ApplicationSet and no rows. When the source path holds
+  a `kustomization.yaml`, the tool is kustomize and the images the overlay
+  pins are the declared images.
+- **fetch-argocd** asks the Argo CD API for what *does* run - the revision
+  synced, the images running, the link - and commits it as a snapshot
+  beside a lock, basis `api` (see the fetchers above).
+
+```json
+{ "name": "argocd-gitops", "wasm": { "url": "file://plugins/portolan-go.wasm" } }
+
+{
+  "plugin": "argocd-gitops",
+  "in": "examples/gitops",
+  "out": "examples/gitops/portolan",
+  "options": {
+    "repo": "github.com/shortlink-org/portolan",
+    "paths": ["bootstrap", "appsets"],
+    "labels": { "context": "app.kubernetes.io/part-of", "service": "app.kubernetes.io/name" },
+    "out": "argocd.json"
+  }
+}
+```
+
+An Application is placed on a service by the labels it carries -
+`app.kubernetes.io/part-of` and `app.kubernetes.io/name`, the same pair
+fetch-k8s reads off a workload, or whichever the manifest names under
+`labels` - and, without them, by deploying from the service's repository
+inside the service's directory. One that matches neither way is listed on
+the Problems page as unclaimed rather than guessed at.
+
+Where both plugins spoke for one Application the merge folds the two into
+one row, basis `both`: the deployer's shape, the tree filling what the
+deployer did not say, and the fields the two disagree on kept as the tree's
+word under `drift` - a `deployment-drift` problem, and a chip on the
+service page that says the difference in words. A row only the tree spoke
+for wears `declared`.
+
 ## Outside the estate: an external with a contract
 
 A service calls things nobody here builds - a card network, a tax API, a
@@ -744,13 +1021,60 @@ called" - and `unresolved` is never raised, because a trace does not put the
 far end in the catalog. A consumer span inside a trace opens a flow of its own
 and is matched the same way, so one password change verifies both the
 request's flow and the policy's. A root no flow opens is written down as
-`observed-<service>-<route>`, once per shape, with a summary saying how many
+`observed-<service>-<route>`, once per opening, with a summary saying how many
 traces showed it.
 
+Every recording that opened a flow is laid over it. A declared step carries
+`seen: { traces }`, how many recordings showed it. An rpc or an event a
+recording showed that the code does not declare is put into the flow after
+the declared step it followed, as a step with an id the code never gave
+(`seen1`, `seen2`) and a note saying in how many recordings; a lane such a
+hop needed is added after the declared ones. A store call is never added -
+the code's word on which repository method ran is the better one - and a
+consumer met on the way belongs to the flow it opens.
+
+Where recordings part, the flow says so. What every recording showed after
+a step is a run of plain steps; where they went different ways is an `alt`
+(`seen-alt1`, or `alt1` in an observed flow) with a branch for each way,
+titled by the hop it starts with and counting the recordings that went it,
+each branch laid over the same way inside; what they all showed again
+after parting comes after the frame, once. A recording that went no
+further where others went on is the branch titled `otherwise`, with no
+steps and its count, so that a reader sees the hop is not always taken.
+Two observed recordings that open the same way are one flow built this
+way, so a happy path and a refusal read as one sequence that parts. The
+merge takes all of this from a second declaration of a flow because it
+says where it came from: `seen` on a step the first declaration does not
+have, or a frame whose every step and branch carries it.
+
+A recording is also kept as an example of each flow it showed, under
+`examples` on the flow: the recording and trace it came from, when it ran,
+how long it took, and for each step the span's name, its length and an
+allowlist of its attributes - `http.route`, `http.request.method`,
+`http.response.status_code`, `rpc.service`, `rpc.method`, `db.operation.name`,
+`db.collection.name`, `messaging.destination.name`, `event.name`,
+`server.address` and a few more of that kind. A query text, a header, a full
+URL or a path with an id in it is somebody's data and is never carried; the
+list is closed in `examples.go`, and a new attribute is added there with the
+argument for it. `examples` in the options says how many recordings a flow
+keeps, the ones that show the most of it first: five when unset, zero for
+none.
+
+A recording added from the page lands under the project's
+`telemetry/recordings/` and widens (or adds) the project's `otel` verify step
+to read that directory, so every run after reads it too; the page runs the
+generator over a copy of the workspace first and shows which flows the
+recording showed before anything is written (portolan.0014).
+
 `service.name` is matched to the one service whose slug it is, `event.name` to
 the one event whose `wire.name` it is, or failing that to the one event of the
 publisher's with that last segment; `services` and `events` in the options say
-otherwise where an estate's names differ. A publish span whose
+otherwise where an estate's names differ. A server span's route is matched to
+the operation whose `http` verb and path template it fits; `routes` in the
+options names an operation for a route the estate spells another way - a
+prefix a gateway adds, a path the document writes differently - as
+`{ "POST /api/v1/sessions": "login" }`, and the name has to be one the
+service's interface declares. A publish span whose
 `messaging.destination.name` is not the event's `wire.channel` is a warning:
 the event went out, but not where the code says it does.
 
@@ -775,9 +1099,9 @@ enforces: a team that owns a directory is a team that gets the pull request.
 ```
 
 Point `in` at the directory the file is in, not at the repository root: the
-host dates a fragment from the last commit to touch the step's input, and the
-subject of this one is the `CODEOWNERS` file. Rooted at the repository, it
-would be restamped by every commit ever made. Left with no `file`, the three
+subject of this step is the `CODEOWNERS` file, and `gen --check` explains a
+changed fragment by what moved under the step's input. Rooted at the
+repository, every commit ever made would be named. Left with no `file`, the three
 places a forge looks are tried in order - `CODEOWNERS`, `.github/CODEOWNERS`,
 `docs/CODEOWNERS` - and a `file` that names something absent fails the run,
 because answering "nobody owns anything" to a typo is only noticed a month
@@ -822,21 +1146,23 @@ the forge looks.
 `wasm` is the default and should stay that way. The module gets no network, no
 environment and no way to start a process. A generator gets no filesystem
 either. An extract or verify step gets the workspace preopened as `/`
-(portolan.0006), which is how the built-in Go extractors read a tree without a
-Go toolchain on the machine: every one of them, and the three generators, is
-the single module `plugins/portolan-go.wasm`, which answers to the plugin name
-the host passes as `argv[0]`. WASI preopens read-write, so an extractor is
+(portolan.0006), which is how syntax-only built-in Go extractors read a tree
+without a Go toolchain on the machine: those extractors and the three generators
+are the single module `plugins/portolan-go.wasm`, which answers to the plugin
+name the host passes as `argv[0]`. WASI preopens read-write, so an extractor is
 trusted not to write the tree it reads, the same trust a process plugin has
 today; a `sha256` pins that trust to a build.
 
-`process` is the escape hatch for a plugin that needs a toolchain: the Rust,
-Java, Python and TypeScript extractors run in their own runtimes, and
-`fetch-bsr` still talks to its registry from Go. It gets the same protocol
+`process` is the escape hatch for a plugin that needs a toolchain: the typed Go
+HTTP client analyzer and the Rust, Java, Python, PHP and TypeScript extractors run in
+their own runtimes. It gets the same protocol
 and none of the sandbox, which is the trade being made and the reason it is
 not the default. It declares `command` and an `args` array; the host never
-feeds a command string through a shell. A built-in Go plugin that still runs
-as a process is the same code reached as
-`go run ./plugins/cmd/portolan-go <name>`.
+feeds a command string through a shell. For a built-in declared as `go run`,
+the adapter compiles a workspace-local native sidecar from the package and
+then runs it with the scanned workspace as its current directory. This keeps
+module resolution in Portolan's shipped source and project loading in the
+project being analyzed.
 
 `host` is for Portolan's own code that needs what only the host has - a git
 binary, a socket - and so runs inside the host process (portolan.0008):
@@ -859,12 +1185,18 @@ written.
 `fetch-git` is `fetch-bsr` for a repository rather than a registry, and it
 lives by the same four rules. A pin is a repository, a commit and the paths
 actually read; the step fetches exactly those directories at exactly that
-commit and hands them back as files, so the host writes them into the tree
+commit and hands their text files back, so the host writes them into the tree
 beside a `git.lock.json` naming the commit and the digest of every file. The
 paths inside the copy are the repository's own, which is the point: the
 extract step that follows points its `in` at the vendored service and reads
 it exactly as it would read that service's checkout.
 
+Binary files are not part of the extractor input. Known binary extensions are
+discarded before their blobs are read; files without a known extension are
+sniffed for NUL bytes and invalid UTF-8 before the response is assembled. The
+lock records only their path, byte size and skip reason, never Base64 content,
+so online fetch and offline replay produce the same bounded text-only copy.
+
 It runs inside the host (`scripts/host-plugins/fetch-git.mjs`,
 portolan.0008) rather than as a module, because it needs a git binary and a
 socket, and a manifest names it with `host` rather than `wasm` or `process`.
@@ -919,11 +1251,12 @@ Nothing else can say it. A service says which repository it lives in, and an
 extractor reads a directory as a pure function of what is on disk; neither has
 any idea which commit somebody fetched. Without that line, every source path
 of every vendored service is dead text on the page - the file and the line are
-known, and there is nowhere to send a reader - and every fragment read out of
-the copy is stamped with the commit that VENDORED it, so the service looks
-fresh whenever the fetch is re-run and unchanged when its own repository
-moves. With it, `sourceHref` links the line at the commit it was read at and
-`stampFor` dates the fragment from the code rather than the vendoring.
+known, and there is nowhere to send a reader. With it, `sourceHref` links the
+line at the commit it was read at. The fragments read out of the copy are
+dated, like every other source, by the commit of THIS repository that last
+changed them (portolan.0010) - which is the fetch that brought the change in,
+and is what "how fresh is this" means to the estate that vendored it; the pin
+is what says which upstream commit that was.
 
 One more line is needed for the app itself: `SOURCE_GLOBS` in `src/data.ts`,
 where the same patterns are written out a second time because
@@ -1020,7 +1353,8 @@ workflow already does.
 
 Four rules govern what happens when a fetch does not:
 
-1. Fetch succeeded → the fetched files and a regenerated lock.
+1. Fetch succeeded → the fetched text files, binary-skip metadata and a
+   regenerated lock.
 2. Skipped or failed, cache complete and matching its digests → the cached files
    byte-identically, plus a warning. Output unchanged, so `--check` stays clean.
 3. Failed **and** no usable cache → a non-zero exit, never a short file list. The
@@ -1202,3 +1536,23 @@ it, the call before the decorator before `task_routes` before the default,
 and `transaction.on_commit(...)` around an enqueue is a note on the step,
 which is the one fact about *when* a message leaves that the code states
 plainly.
+
+### HTTP destination provenance
+
+`extract-http-clients` emits structured `destination` evidence on HTTP consumes
+and flow steps: the request call site and endpoint expression, local path,
+base URL expression, config field and environment variable, source locations,
+and a full path when source proves the join. Functional options that assign a
+URL field to a returned client are followed through adapter constructors.
+Bindings belong to the adapter's client field; conflicting construction sites
+remain unresolved instead of sharing another client's settings.
+
+A Go `default` struct tag is recorded as `config-default`, not as the effective
+runtime configuration. For example, `WithBaseURL(cfg.SettingAddr)` with
+`default:"http://localhost:8000/settings"` and a concatenated
+`/get-admin-settings` yields `/settings/get-admin-settings`. The merge matches
+that full path exactly and never falls back to a suffix if it is absent.
+Legacy suffix matches remain explicitly labeled `unique-suffix` in the evidence
+shown in RPC step details. Later runtime URL modifiers are listed separately;
+the recovered full path describes the static join. Unreadable options may
+leave only the local route available.

package/plugins/cmd/portolan-http-clients/main.go

@@ -0,0 +1,19 @@
+// Command portolan-http-clients is the native sidecar for typed Go call-graph
+// analysis. It intentionally does not compile for WASI: go/packages needs the
+// project Go toolchain to load build tags and dependencies before SSA/VTA can
+// resolve interface calls.
+package main
+
+import (
+	"fmt"
+	"os"
+
+	extracthttpclients "github.com/shortlink-org/portolan/plugins/extract-http-clients"
+)
+
+func main() {
+	if err := extracthttpclients.Serve(os.Stdin, os.Stdout); err != nil {
+		fmt.Fprintln(os.Stderr, "portolan-http-clients:", err)
+		os.Exit(1)
+	}
+}