@shortlink-org/portolan 0.2.4 → 0.4.0

package/README.md

@@ -87,8 +87,9 @@ reference).
 - **Diagrams** — LikeC4 C4 views (estate landscape, every container in the
   estate with its technology and the protocol on each edge, one per context,
   two per service, one dynamic view per flow), an ELK-routed dependency graph,
-  and a context map. The app never draws these itself; `npm run likec4:gen`
-  writes the model from the catalog.
+  and a context map. The app never draws these itself; `npm run gen` writes
+  the model from the catalog as its last step, and holds it to the catalog in
+  `gen:check`, and `npm run likec4:gen` writes it on its own before `dev`.
 - **ER canvases** — per store: tables, views, keys and crow's feet, plus column
   lineage (`from`) drawn dashed; hovering a column lights the whole chain back
   to where the value came from.
@@ -116,7 +117,43 @@ first:
   with a second writer, a table that no longer holds the aggregate it claims,
   a column whose type has drifted from its field's, an outbox with no payload;
 - a channel with a second publisher, an event on a channel its service does not
-  declare, a declared channel no event names, a subscription nothing publishes.
+  declare, a declared channel no event names, a subscription nothing publishes;
+- what the deployer runs from a place no service lives at, and where the GitOps
+  tree and the deployer disagree.
+
+Each check is a rule in `rules/builtin.json`: a passport - id, severity, what
+it looks for and what to do about a row - and the rule itself, in CEL over one
+subject. A subject is one row with every fact a rule would otherwise have to
+look up already on it - a table knows who writes it, a channel knows who else
+publishes there, a call knows whether its peer is in the estate - so a rule is
+one line a reader can read, copy and tighten. **Settings → Rules** lists them
+with the rows each produces now. A shipped rule can be switched off or
+re-graded in `portolan.json`, with a reason, and a rule of your own is written
+there the same way - over a service, a call, a copy, an event, a consumer, a
+channel, a subscription, a table, a column, a deployment, a flow or an
+aggregate - and runs in the page the moment it is saved:
+
+```json
+{
+  "problemRules": [
+    { "id": "shared-store", "enabled": false, "reason": "the estate shares one database by design" },
+    {
+      "id": "team.quiet-event",
+      "over": "event",
+      "severity": "warning",
+      "title": "Event nobody consumes",
+      "when": "size(event.consumers) == 0 && !event.name.endsWith('Audit')",
+      "message": "'nothing consumes ' + event.id",
+      "peer": "event.service"
+    }
+  ]
+}
+```
+
+Expressions are type-checked against the subject's fields when the manifest is
+read - `event.nme` is refused, not shown as an empty page - and again in the
+page, by the same module. The Rules page writes the entry after a preview of
+the rows it would add (portolan.0016, portolan.0017).
 
 ## Where the facts come from
 
@@ -125,17 +162,47 @@ Plugins, one JSON message in and one out (`plugins/README.md`), declared in
 
 | phase | plugins |
 | --- | --- |
-| extract | `extract-project`, `extract-go`, `extract-ts`, `extract-rust`, `extract-java`, `extract-django`, `extract-celery`, `extract-python-kafka`, `extract-openapi`, `extract-wsdl`, `extract-http-clients`, `extract-redis`, `extract-asyncapi`, `extract-graphql`, `extract-proto`, `extract-river`, `extract-watermill`, `extract-go-nats`, `extract-csr`, `extract-sql`, `extract-flows`, `extract-adr`, `extract-glossary`, `extract-commands` |
+| extract | `extract-project`, `extract-go`, `extract-ts`, `extract-rust`, `extract-java`, `extract-django`, `extract-laravel`, `extract-php-ddd`, `extract-csharp-ddd`, `extract-celery`, `extract-python-kafka`, `extract-openapi`, `extract-wsdl`, `extract-http-clients`, `extract-redis`, `extract-asyncapi`, `extract-graphql`, `extract-proto`, `extract-river`, `extract-watermill`, `extract-go-nats`, `extract-go-sqs`, `extract-csr`, `extract-sql`, `extract-flows`, `extract-adr`, `extract-glossary`, `extract-commands`, `extract-k8s`, `extract-argocd` |
 | verify | `verify-otel` — reads traces, marks the hops they show as `verified`; `verify-codeowners` — reads CODEOWNERS, says who to ask about each service |
 | generate | `gen-markdown` — `docs/`, `gen-mermaid` — standalone flow diagrams, `gen-backstage` — Backstage entities |
 
 `fetch-git`, `fetch-bsr` and `fetch-csr` bring in sources from other
 repositories, the Buf Schema Registry and a Confluent Schema Registry, against a
-lock, so a later build can reproduce them without a socket.
+lock, so a later build can reproduce them without a socket. `fetch-k8s` reads a
+live cluster through `kubectl` into the names each service answers on and
+dials, and replays the committed fragment when there is no cluster to ask.
+`fetch-argocd` reads the applications an Argo CD server manages into a snapshot
+of where each service runs - environment, cluster, namespace, revision, images -
+against the same kind of lock; the service page shows it under "Where it runs".
 
 Each plugin describes its own options; `npm run schema` asks all of them and
 composes `schema/portolan.schema.json`, which editors complete against and `gen`
-checks before running anything.
+checks before running anything. The same answers are rendered as the site's
+plugin reference - `/plugins` on any generated site, grouped by what each reads
+or makes, with the options it takes - so the list above is the short form.
+
+Non-fatal extractor diagnostics remain attached to their pipeline step. The
+Settings page groups repetitions by plugin, stable rule and severity, and shows
+the recommended next action. A reviewed limitation can be hidden from the
+active view without discarding it:
+
+```json
+{
+  "warningPolicies": [
+    {
+      "when": "plugin == 'openapi' && rule == 'openapi.missing-operation-id' && project == 'aviacore'",
+      "action": "suppress",
+      "reason": "The partner-owned contract cannot be changed in this repository."
+    }
+  ]
+}
+```
+
+The rule id is shown beside the warning. CEL expressions can read `plugin`,
+`rule`, `severity`, `project`, `phase`, `ref`, `message`, and the integer
+`count` for that rule in the step. Expressions are type-checked when the
+manifest is read, must return `bool`, and run during generation. A suppression
+requires a reason and remains available through the `suppressed` filter.
 
 ## Use it in your project
 
@@ -234,6 +301,107 @@ live in a dedicated architecture repository: `fetch-git` pins the service
 repositories at immutable commits and the normal merge, check, diff, and build
 commands operate on the combined estate.
 
+### A real one: Bagisto
+
+The services under `examples/` are small on purpose. For what the catalog
+looks like on a real application, the site ships a third profile: pick
+**Bagisto** in the catalog selector. [Bagisto](https://github.com/bagisto/bagisto)
+is an open-source e-commerce platform on Laravel - 41 packages, one per
+module, talking to each other through named events - read by
+`extract-laravel` alone, with nothing annotated for the catalog.
+
+What comes out of Bagisto 2.4, pinned in `vendor/repos/bagisto/bagisto`:
+
+| | |
+| --- | --- |
+| model groups | 28, one per package with models, 125 Eloquent models and 22 enums between them |
+| events | 309, most of them named - `Event::dispatch('sales.order.cancel.after', …)` - owned by the package their name says, 50 with a listener |
+| HTTP | 520 operations in 10 interfaces, one per route file, with an inferred OpenAPI document |
+| database | 138 tables and 1300 columns replayed from 189 migrations, 185 foreign keys, 120 places the code reads or writes them |
+| queues | 17 jobs on one queue, each a hop in the flow that dispatches it and a worker flow of its own |
+| flows | 610 |
+
+Go to the service and read it the way the tabs are ordered: **bus** for the
+queue, **data** for the schema laid out by model group - the picker beside
+the arrangement toggle switches groups off, the search brings a table up
+close - and **flows** for `shop-checkout-onepage-orders-store`, the checkout
+followed from the controller through the repository it holds to the events
+and the job it ends in. Every "view source" link opens the file on GitHub at
+the pinned commit.
+
+The warnings the extractor left are findings about Bagisto, not about the
+tool: 218 events under the admin package are dispatched by name and declare
+no payload; `Settings\CurrencyController` answers admin routes and does not
+exist; the sitemap job reads a `pages` table no migration creates; a
+foreign key points at a `theme_customizations` table no migration creates;
+one listener waits for `bagisto.shop.products.price.after`, which nothing
+dispatches, and one for the repository events of a vendored package. How the
+fragments were made, and how to refresh them, is in
+`vendor/repos/bagisto/bagisto/README.md`; the reading rules are in
+`plugins/extract-laravel/README.md`.
+
+### The small one, by the book: CodelyTV php-ddd-example
+
+The fourth profile, **CodelyTV php-ddd-example**, is the opposite kind of
+real: [php-ddd-example](https://github.com/CodelyTV/php-ddd-example) is
+CodelyTV's reference for DDD, hexagonal architecture and CQRS in PHP, a
+Symfony monorepository laid out as `src/<Context>/<Module>/{Domain,Application,Infrastructure}`
+with the deployables under `apps/`, read by `extract-php-ddd` alone - the
+layout is the claim, and nothing is annotated. What comes out, pinned in
+`vendor/repos/CodelyTV/php-ddd-example`: four contexts (Mooc, Backoffice,
+Analytics, Retention), six services, five aggregates with their value
+objects and three domain events on one RabbitMQ exchange, one queue per
+subscriber, the MySQL tables from the Doctrine mappings including a JOINED
+`steps` family, and sixteen flows. The one to read is
+`backoffice-frontend-courses-post`: the back office's form dispatches
+Mooc's `CreateCourseCommand` over an in-memory bus, so the flow crosses to
+`mooc.backend` with a `call` step, records `CourseCreated`, saves the
+course, and the subscribers in Mooc, Backoffice and Analytics pick the event
+up from their own queues.
+
+The warnings are findings about the project: Analytics and Retention have no
+application under `apps/`, so their code deploys nowhere and Analytics'
+subscriber to every event runs nowhere; `TrimVideoCommandHandler` is named
+like a handler and implements neither bus interface; Retention's four modules
+and Mooc's Notifications are empty scaffolding; the front end routes
+`/api/courses` to a controller that does not exist. How the fragments were
+made is in `vendor/repos/CodelyTV/php-ddd-example/README.md`; the reading
+rules are in `plugins/extract-php-ddd/README.md`.
+
+### The same lesson in C#: Modular Monolith with DDD
+
+The fifth profile, **Modular Monolith with DDD**, is Kamil Grzybek's
+[modular-monolith-with-ddd](https://github.com/kgrzybek/modular-monolith-with-ddd),
+the reference for the pattern in .NET: five modules laid out as
+`src/Modules/<Module>/{Domain,Application,Infrastructure,IntegrationEvents}`,
+one API host, one SQL Server database with a schema per module, read by
+`extract-csharp-ddd` alone - through Roslyn, the compiler's own parser, over
+the tree as one compilation and without restoring a single package. What
+comes out, pinned in `vendor/repos/kgrzybek/modular-monolith-with-ddd`: five
+contexts with one service each, nineteen aggregates and five model groups,
+sixty domain events dispatched in process and seven integration events on
+the in-memory bus, sixty-one commands and twenty-nine queries, an internal command queue
+per module, the forty-three tables and seventeen views of the database
+project with the root tables mapped to their aggregates through the EF
+configurations, fifty-one HTTP operations across four documents, the
+project's seventeen decision records, and a hundred and two flows. The one
+to read is `register-new-user`: the User Access controller hands
+`RegisterNewUserCommand` to the Registrations module's facade in process,
+the flow crosses with a `call` step, `NewUserRegisteredDomainEvent` is
+raised and saved, and from there the outbox republishes it to the bus as an
+integration event that Meetings, Administration and Payments each hear and
+turn into a job on their own internal queue.
+
+The warnings are findings about the project: `MemberCreatedIntegrationEvent`
+is declared and published nowhere while User Access subscribes to it;
+Payments subscribes to `MeetingGroupProposalAcceptedIntegrationEvent` and
+has no handler; the `app` schema belongs to no module; the view is created
+as `v_Countriess` and the query reads `v_Countries`; and two modules keep a
+`Domain/Users` directory with a typed id and nothing to hold it. How the
+fragments were made is in
+`vendor/repos/kgrzybek/modular-monolith-with-ddd/README.md`; the reading
+rules are in `plugins/extract-csharp-ddd/README.md`.
+
 ## Develop Portolan itself
 
 ```bash

package/catalog/enum_test.go

@@ -0,0 +1,46 @@
+package catalog
+
+import (
+	"encoding/json"
+	"testing"
+)
+
+// An enum round-trips whole: the aggregate's list, the values with their
+// docs, the deprecation marks - and the proto side's numbers. The fixture is
+// literal rather than read from a catalog so that the test says exactly what
+// the mirror is expected to keep.
+func TestEnumRoundTrip(t *testing.T) {
+	raw := `{"id":"a","slug":"a","name":"A","readme":"","root":"A","entities":[],"valueObjects":[],"operations":[],"events":[],
+		"enums":[{"id":"shop.oms.order.status","slug":"status","name":"Status","doc":"Where an order is.","deprecated":true,
+		"values":[{"name":"placed","doc":"just in"},{"name":"legacy","doc":"","deprecated":true}]}]}`
+
+	var agg Aggregate
+	if err := json.Unmarshal([]byte(raw), &agg); err != nil {
+		t.Fatal(err)
+	}
+	if len(agg.Enums) != 1 || agg.Enums[0].ID != "shop.oms.order.status" || !agg.Enums[0].Deprecated {
+		t.Fatalf("enum not read: %+v", agg.Enums)
+	}
+	if got := agg.Enums[0].Values; len(got) != 2 || got[0].Name != "placed" || !got[1].Deprecated {
+		t.Fatalf("values not read: %+v", got)
+	}
+
+	out, err := json.Marshal(agg)
+	if err != nil {
+		t.Fatal(err)
+	}
+	var before, after any
+	_ = json.Unmarshal([]byte(raw), &before)
+	_ = json.Unmarshal(out, &after)
+	if diffs := compare("aggregate", normalize(before), normalize(after)); len(diffs) != 0 {
+		t.Fatalf("round trip lost something: %v", diffs)
+	}
+
+	var svc RpcService
+	if err := json.Unmarshal([]byte(`{"id":"shop.v1.OrderService","methods":[],"source":"","enums":[{"name":"Status","values":[{"name":"STATUS_UNSPECIFIED","number":0},{"name":"PLACED","number":1,"doc":"in"}]}]}`), &svc); err != nil {
+		t.Fatal(err)
+	}
+	if len(svc.Enums) != 1 || svc.Enums[0].Values[1].Number != 1 || svc.Enums[0].Values[0].Number != 0 {
+		t.Fatalf("proto enum not read: %+v", svc.Enums)
+	}
+}

package/catalog/evidence_test.go

@@ -0,0 +1,35 @@
+package catalog
+
+import (
+	"encoding/json"
+	"reflect"
+	"testing"
+)
+
+func TestRelationEvidenceSurvivesEverySupportedCarrier(t *testing.T) {
+	evidence := []RelationEvidence{{Kind: "binding", Rule: "provider-signature", Source: "di.go:14", Symbol: "Provide", Candidates: []string{"one", "two"}}}
+	before := struct {
+		Step     Step
+		Call     RpcCall
+		Table    Table
+		Persists Persists
+	}{
+		Step: Step{Evidence: evidence}, Call: RpcCall{Evidence: evidence},
+		Table: Table{Evidence: evidence}, Persists: Persists{Evidence: evidence},
+	}
+	raw, err := json.Marshal(before)
+	if err != nil {
+		t.Fatal(err)
+	}
+	after := before
+	after.Step.Evidence = nil
+	after.Call.Evidence = nil
+	after.Table.Evidence = nil
+	after.Persists.Evidence = nil
+	if err := json.Unmarshal(raw, &after); err != nil {
+		t.Fatal(err)
+	}
+	if !reflect.DeepEqual(before, after) {
+		t.Fatalf("evidence changed: %s", raw)
+	}
+}