@mindbricks/genesis-set 0.0.0-stage → 0.2.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (41) hide show
  1. package/CHANGELOG.md +44 -0
  2. package/LICENSE +202 -0
  3. package/README.md +148 -2
  4. package/bin/genesis-set.mjs +153 -0
  5. package/corpus/README.md +25 -0
  6. package/corpus/genesis/compile-results.md +246 -0
  7. package/corpus/genesis/compiler-api.md +265 -0
  8. package/corpus/genesis/custom-code.md +519 -0
  9. package/corpus/genesis/deploy.md +190 -0
  10. package/corpus/genesis/pitfalls.md +231 -0
  11. package/corpus/genesis/procedures/README.md +17 -0
  12. package/corpus/genesis/procedures/agent-hub-blueprint.md +13 -0
  13. package/corpus/genesis/procedures/auth-blueprint.md +14 -0
  14. package/corpus/genesis/procedures/bff-blueprint.md +16 -0
  15. package/corpus/genesis/procedures/blueprint-authoring.md +16 -0
  16. package/corpus/genesis/procedures/howto-mail-to-tasks.md +26 -0
  17. package/corpus/genesis/procedures/howto-payment-flow.md +112 -0
  18. package/corpus/genesis/procedures/howto-subscription.md +33 -0
  19. package/corpus/genesis/procedures/notifications-blueprint.md +13 -0
  20. package/corpus/genesis/procedures/patterns-catalog.md +12 -0
  21. package/corpus/genesis/procedures/project-lifecycle.md +27 -0
  22. package/corpus/genesis/run-locally.md +310 -0
  23. package/corpus/genesis/testing.md +532 -0
  24. package/lib/config-cli.mjs +179 -0
  25. package/lib/install.mjs +639 -0
  26. package/lib/mcp-config.mjs +222 -0
  27. package/lib/semver.mjs +94 -0
  28. package/mcp/blueprint.mjs +64 -0
  29. package/mcp/client.mjs +141 -0
  30. package/mcp/config.mjs +166 -0
  31. package/mcp/key-page-html.mjs +275 -0
  32. package/mcp/key-page.mjs +217 -0
  33. package/mcp/key-store.mjs +204 -0
  34. package/mcp/key-tools.mjs +225 -0
  35. package/mcp/server.mjs +164 -0
  36. package/mcp/tools.mjs +270 -0
  37. package/package.json +44 -4
  38. package/skills/genesis-compiler-api/SKILL.md +56 -0
  39. package/skills/genesis-deploy/SKILL.md +36 -0
  40. package/skills/genesis-generated-code/SKILL.md +48 -0
  41. package/skills/genesis-run-and-test/SKILL.md +43 -0
@@ -0,0 +1,112 @@
1
+ # Genesis additions: How-to: the payment flow
2
+
3
+ These notes add to the mind-set procedure `procedures/howto-payment-flow.md` (installed at
4
+ `~/.mind-set/current/procedures/howto-payment-flow.md`). Follow that procedure; at each place below,
5
+ do what this file says. They hold what is true of the Genesis compiler and the code it
6
+ generates, and the Genesis Compiler API steps.
7
+
8
+ ## 1. In "9. Validate"
9
+
10
+ Add:
11
+
12
+ **Runtime prerequisites.** The service compiled and running; the AUTH
13
+ service compiled AFTER the payment provider was bound (its `sys_paymentCustomer` / `sys_paymentMethod` family
14
+ is generated only then: a `startorderpayment` call 500s through a missing
15
+ `/auth-api/paymentcustomers` until auth is compiled again); a published,
16
+ in-stock product; a verified BUYER session (`pitfalls.md` section 1); a
17
+ `superAdmin` session for the operator rung; a staff-role session for
18
+ ship/deliver. Base paths below are an example (`/order-api`, `<order>` =
19
+ `order`); read yours off the generated service's route list.
20
+
21
+ **Webhook secret step.** After the FIRST compile of a service with a
22
+ payment provider, and after every reset of its database, call once as
23
+ superAdmin:
24
+ `POST /<service>-api/v1/payments/admin/webhooks/provision`, body `{}`.
25
+ It registers the webhook endpoint and stores its signing secret encrypted
26
+ in the service's database (restarts keep it); without it every webhook is
27
+ refused 401 `SIGNATURE_UNVERIFIABLE` and plan invoices / credit notes never
28
+ issue. Done = each `providers[]` entry `provisioned` / `rotated` with
29
+ `persisted: true`, or `already-registered` with `liveInThisProcess: true`
30
+ (after a reset it rotates by itself: `autoRotated: true`). Skip while the
31
+ provider is mock (`mock-status` `mock: true`). Re-run it after every framework
32
+ upgrade too: on a matched endpoint it adds the events a newer framework maps
33
+ and the endpoint lacks (`eventsAdded`, fw #6369; id and secret unchanged,
34
+ nothing removed); `eventsNotAdded` names events the provider refused to add,
35
+ report them to the owner. A call that creates or rotates the endpoint
36
+ also deletes this service's own endpoints at an older URL (`staleRemoved`; a
37
+ plain `already-registered` match deletes nothing), except the endpoint
38
+ of the stored secret and anything in a dev slot (`CONFIG_ENV=dev`): those are
39
+ only listed in `staleNotRemoved` with the reason. An endpoint at the same URL
40
+ with another owner's `mb_*` tags is never touched and is listed in
41
+ `foreignEndpointIds`: report it to the owner; an entry `failed`
42
+ with `errorCode: "WEBHOOK_ENDPOINT_LIMIT"` means the provider account is full:
43
+ report its `endpointsByProject` list to the owner (endpoints are pruned at the
44
+ provider, never by you) and re-run once they are removed.
45
+
46
+ **Payment re-provision after a database reset.** A reset of the service's
47
+ database empties the subscription plan catalog
48
+ (`sys_<X>SubscriptionPlans`), and the boot seed deliberately never writes
49
+ `providerPlanId`, so after EVERY reset each subscription purchase answers
50
+ 409 `SUBSCRIPTION_PLAN_NOT_PROVISIONED`. For each payment service, as
51
+ superAdmin and BEFORE any purchase: first
52
+ `POST /<service>-api/v1/payments/admin/plans/provision`, body `{}` (a
53
+ service with subscription plans), THEN the webhook step above
54
+ (`POST /<service>-api/v1/payments/admin/webhooks/provision`, body `{}`).
55
+ The plan sweep finds the provider prices the earlier provisioning created
56
+ and adopts them; it creates nothing new. Done = `totals.failed: 0` with
57
+ every plan counted as `adopted` (the usual answer after a reset) or
58
+ `provisioned`, then the webhook done-check above.
59
+
60
+ | Rung | Call | Expect |
61
+ |---|---|---|
62
+ | 0 | `GET /order-api/v1/payments/mock-status` | `{ providers: [{ providerName, brand, mock: true, reason, transport: "present", blocking: false, mode }], anyMock: true, blocking: [], health: "ok" }` on a deployment with blank credentials — the mock adapter is the payment path; never call the live provider from tests. `transport`, not `mock`, is the field that says the mock path ANSWERS (see the brand caveat below): `transport: "absent"` puts the provider in `blocking[]` and flips `health` to `"blocked"`. A row that carries `transportFamilies` (Adyen) is covered only for the families with `coverage: "present"`: a call of an `absent` family is refused `501 MOCK_UNHANDLED_PATH` although `transport` reads `"present"` |
63
+ | 1 | buyer: `POST /order-api/cart/items { productId, quantity }`, `GET /order-api/cart`, `POST /order-api/checkout` | 201 with the order (`status: placed`, `paymentStatus: unpaid`, `expiresAt`), items priced from the catalog; `GET /order-api/cart` empty afterwards; an unavailable line answers 422 and creates NO order |
64
+ | 2 | buyer: `PATCH /order-api/v1/startorderpayment/:orderId` body `{ paymentUserParams: { return_url: "<app url>" } }` (add `providerOptions: { mockScenario: "requires_action" }` to exercise the 3-D Secure branch) | 200 with the ticket (`paymentStatus`, `statusLiteral`); the buyer must OWN the order (`403 NOT_OWNER` otherwise). With the default mock scenario the mock settles synchronously and the capture hooks run before the answer (`lifecyclePending: true` when they exceed `PAYMENT_SYNC_SETTLE_TIMEOUT_MS`, default 10 s) |
65
+ | 3a | `POST /order-api/v1/ordermockauthenticate/:orderId` | Mock mode only (404 otherwise): stands in for the customer authenticating and confirms the intent — needed in the `requires_action` scenario |
66
+ | 3b | superAdmin: `POST /order-api/v1/payments/admin/simulate-capture/order/:orderId` | Operator-gated like every `/v1/payments/admin/*` route: `401 LOGIN_REQUIRED` without a session, `403 OPERATOR_REQUIRED` without the `paymentOperator` capability (a plain `staff` role is refused — pilot-verified; `superAdmin` passes). The route is platform-operator-only (`superAdmin` / `admin` / `saasAdmin`; `superAdmin` / `saasAdmin` in a multi-tenant project, which has no role named `admin`, #5701), and that second gate answers differently HERE: a tenant-level operator (`tenantOwner` / `tenantAdmin`), or a caller admitted only through the `paymentOperator` permission, PASSES the operator gate and is then refused `404` rather than the console's `403 PLATFORM_OPERATOR_REQUIRED`, so this sandbox route reads as unmounted - assert 404, not 403, for that actor. Also: 404 with no payment ticket yet ("start the payment first"), 404 on a production deployment or non-test credentials, `409 PAYMENT_NOT_CAPTURABLE` on a terminal ticket; repeat answers 200 `outcome: "already-recorded"` when its event-store lookup completes within the wait budget; if recording has not completed by that deadline, the 200 body instead carries `outcome: "record-pending"` and `lifecyclePending: true` (including on a repeat). Recording continues after the answer; the existing row is reused when found. `PAYMENT_SYNC_SETTLE_TIMEOUT_MS=0` skips waiting and answers `record-pending` immediately |
67
+ | 4 | `GET /order-api/v1/orderpaymentbyorderid/:orderId`; `GET /order-api/my-orders/:orderId` | ticket `statusLiteral` captured (or `paymentStatus: simulated_capture` after 3b); the ORDER shows `status: paid`, `paymentStatus: succeeded`, `paidAt`, `paymentReference` = the provider intent id — proof the `afterPaymentCaptured` hook ran your API. If the ticket is captured but the order did not move, read the service log for `PaymentLifecycleHookFailed` (hook name, trigger, error) — the hook ran and your API refused |
68
+ | 5 | a Kafka consumer on the wire topic (`<codename>-order-service-payment-captured`); the notification service's log or its notification rows in the database | the captured event with your payload keys; the `paymentReceived` mail resolved to `customerUserId` |
69
+ | 6 | staff: `POST /order-api/orders/:orderId/ship { trackingNumber }`, then `.../deliver` | `shipped` with `shippedAt` + tracking; an unpaid or claim-incomplete order answers 422 (static message); `paid` -> `shipped` -> `delivered` are one-way — do NOT retry on a state-shaped 422, pick another open order |
70
+ | 7 | buyer: `POST /order-api/my-orders/:orderId/cancel` on a PAID, unshipped order | 200 `status: cancelled`, `paymentStatus: refund_pending`, the `sys_orderRefund` row exists (`GET /order-api/v1/payments/refunds`), stock released once at the catalog; a shipped order answers 422 with no refund and no release. Then re-read the order: `paymentStatus: refunded` once `afterPaymentRefunded` converged it — if it stays `refund_pending`, the sweep retries every 5 min; record what you observed (the pilot could not climb this rung because the iyzico mock transport did not exist yet; it does now — climb it) |
71
+ | 8 | wait past `expiresAt` on an UNPAID order (or seed one with a past `expiresAt`) | the minute cron flips it to `expired`, publishes `order-expired`, releases nothing; a capture racing the sweep wins (`applyPaymentCapture` refunds a late capture in full and keeps `expired`) |
72
+
73
+ **Brand caveat — READ THE `transport` FIELD (fw #4473, SHIPPED).**
74
+ `mock: true` only says the credentials are blank; whether the mock path
75
+ WORKS is the per-provider `transport` field, and `stripe`, `iyzico` and
76
+ `adyen` all answer `transport: "present"` now. An iyzico-bound object climbs the whole
77
+ ladder on its mock — rungs 2-8 must NOT be logged as skips, and a lane that
78
+ records them as "blocked by #4473" is reporting a condition that no longer
79
+ exists. An Adyen-bound object's mock answers too. Adyen has no
80
+ `GET /payments/{pspReference}`, and since fw #5808 the adapter never calls it:
81
+ a capture, cancel or refund sends the amount and currency recorded on the
82
+ payment (or the refund) itself, and a refresh answers from the ticket and the
83
+ state the notifications already persisted, so the refund of rung 7 and a
84
+ refresh climb on the mock like every other brand - do NOT log them as a
85
+ reasoned skip. Note what stays true of Adyen: a webhook AUTHORISATION leaves
86
+ the payment `authorized` (only a CAPTURE pays), and a shopper-action challenge
87
+ you abandon is not auto-cancelled - a restart inside the action window
88
+ (`ADYEN_ACTION_TTL_MS`, 30 minutes) resumes the same attempt from the ticket,
89
+ and after it the restart answers 409 (there is no automatic unlock yet). An
90
+ iyzico restart of an open payment (fw #6343) answers the SAME step again
91
+ (`pendingIntentReused: true`, `next_action.paymentPageUrl` or
92
+ `next_action.threeDSHtmlContent`, with `next_action.actionExpiresAt`) while
93
+ it has not expired; a restart that names another card (or switches between
94
+ no card and a card) opens a new attempt while the open payment provably
95
+ cannot capture (an expired hosted page nobody started paying on, a 3-D
96
+ Secure payment at `INIT_THREEDS`) and answers `409 PAYMENT_START_INTENT_PENDING` otherwise - on
97
+ Stripe a changed `paymentMethodId` is always that 409. Assert these, do not
98
+ log them as defects. A brand with no mock of its own (paypal, mangopay, ...) still answers
99
+ `transport: "absent"`: the adapter constructs and then refuses every call by
100
+ name, `blocking` lists that provider and `health` is `"blocked"`. Only THAT
101
+ is a reasoned skip, and it is named by the payload rather than assumed.
102
+
103
+ ## 2. In "9. Validate"
104
+
105
+ Add:
106
+
107
+ **Rung fails — where the fix goes.** A 500 on your own route is a design
108
+ fault: fix the blueprint, rebuild (`iteration.md`), never the generated
109
+ code. A pattern route misbehaving with your blueprint clean is a compiler
110
+ fault: capture `requestId` + the exact error from the service log
111
+ and report it with evidence. `column "x" does not exist` after a blueprint
112
+ change is schema drift — rebuild, never hand DDL (`pitfalls.md` section 3).
@@ -0,0 +1,33 @@
1
+ # Genesis additions: How-to: a recurring pass
2
+
3
+ These notes add to the mind-set procedure `procedures/howto-subscription.md` (installed at
4
+ `~/.mind-set/current/procedures/howto-subscription.md`). Follow that procedure; at each place below,
5
+ do what this file says. They hold what is true of the Genesis compiler and the code it
6
+ generates, and the Genesis Compiler API steps.
7
+
8
+ ## 1. In "7. Renewal versus charging"
9
+
10
+ Instead of the mind-set step that starts "## 8. Validate Validate the blueprint after every module you touched: ...":
11
+
12
+ #### 8. Validate, then verify the running service - the ladder
13
+
14
+ Validate every module you touched (`mindset validate`, then the Genesis
15
+ Compiler API's validation), compile, and start the service. Then climb,
16
+ recording what you OBSERVED per rung; log a rung you could not climb as a
17
+ finding, not as success.
18
+
19
+ | Rung | Call | Expect |
20
+ |---|---|---|
21
+ | 0 | `GET /<svc>-api/v1/payments/mock-status` | the provider listed with `mock: true`, `transport: "present"`, `blocking: false`. `transport: "absent"` = the brand has no mock; stop and report |
22
+ | 1 | catalog present: the generated service's route list shows the subscribe route; the plan seed rows exist | the seed was applied (otherwise compile and start the service again) |
23
+ | 2 | `POST .../v1/payments/admin/plans/provision` as superAdmin | `totals.provisioned` >= number of seeded plans; a second call reports them `skipped` |
24
+ | 3 | member: subscribe with the plan code (injected `POST /v1/subscribe<x>/:orderId` or your authored API) | 200 with `statusLiteral` (`active` or `trialing`; `pending_provider` plus a checkout form on iyzico: complete it per `howto-payment-flow.md`) and a `sys_<X>Subscription` row |
25
+ | 4 | read the allowance (`getBalance` or your wrapper) | the first period's allowance is present after an eligible confirmation (section 5); a still-unpaid first payment grants nothing |
26
+ | 5 | member: spend through the wrapper API; spend past zero | decrements; the overspend is refused with the wrapper's own error, the balance unchanged |
27
+ | 6 | trigger the renewal path (call the cron's business API by its manual route, or replay a renewal event) | the allowance is refreshed once; a second trigger with the same period is a no-op (`alreadyProcessed`) |
28
+ | 7 | member: `DELETE /v1/<x>subscription/:subscriptionId` | `canceled` (or cancel-at-period-end); the cancel hook revoked the allowance with a `consume` |
29
+
30
+ Failure reading: `409 SUBSCRIPTION_PLAN_NOT_FOUND` = seed missing;
31
+ `409 SUBSCRIPTION_PLAN_NOT_PROVISIONED` = run rung 2; a hook that "did
32
+ nothing" = read the service log for `PaymentLifecycleHookFailed`; an
33
+ overspend that succeeded = the wrapper reads the wrong owner row.
@@ -0,0 +1,13 @@
1
+ # Genesis additions: Notifications module authoring
2
+
3
+ These notes add to the mind-set procedure `procedures/notifications-blueprint.md` (installed at
4
+ `~/.mind-set/current/procedures/notifications-blueprint.md`). Follow that procedure; at each place below,
5
+ do what this file says. They hold what is true of the Genesis compiler and the code it
6
+ generates, and the Genesis Compiler API steps.
7
+
8
+ ## 1. In "4. Validate, build, gate"
9
+
10
+ Instead of the mind-set step that starts "2. Building the system service **`notification`** and checking its ...":
11
+
12
+ 2. Compile the system service **`notification`** with the Genesis Compiler
13
+ API (the genesis-compiler-api skill) and check its health where you run it.
@@ -0,0 +1,12 @@
1
+ # Genesis additions: Patterns catalog
2
+
3
+ These notes add to the mind-set procedure `procedures/patterns-catalog.md` (installed at
4
+ `~/.mind-set/current/procedures/patterns-catalog.md`). Follow that procedure; at each place below,
5
+ do what this file says. They hold what is true of the Genesis compiler and the code it
6
+ generates, and the Genesis Compiler API steps.
7
+
8
+ ## 1. In "0. Need -> pattern selector"
9
+
10
+ Instead of the mind-set step that starts "| Each user or tenant brings their OWN AI key (BYOK) | ...":
11
+
12
+ | Each user or tenant brings their OWN AI key (BYOK) | `AiProvider.credentialScopes` | nodejs, python, rust (user, tenant, workspace levels); dotnet, java, golang (user and tenant levels); never a key column or custom variable; `pitfalls.md` section 7 |
@@ -0,0 +1,27 @@
1
+ # Genesis additions: Project lifecycle
2
+
3
+ These notes add to the mind-set procedure `procedures/project-lifecycle.md` (installed at
4
+ `~/.mind-set/current/procedures/project-lifecycle.md`). Follow that procedure; at each place below,
5
+ do what this file says. They hold what is true of the Genesis compiler and the code it
6
+ generates, and the Genesis Compiler API steps.
7
+
8
+ ## 1. In "1. Dependency graph — always respect, even on partial requests"
9
+
10
+ Add:
11
+
12
+ **Compile once, after authoring.** With the single exception of auth
13
+ (compiled first because domain authoring reads its live shape), do not
14
+ compile a service just because the Compiler API is at hand right after you
15
+ authored it. Every build is charged by the lines it adds, and compiling a
16
+ domain service before the bff wires `dataViewName` into its APIs costs a
17
+ second, avoidable round: author and validate everything first, THEN compile
18
+ it once.
19
+
20
+ ## 2. In "2. The phases > Phase D — Design (the interview, the requirement docs, the specs)"
21
+
22
+ Add:
23
+
24
+ With genesis-set, that compiler is the Genesis compiler, reached through the
25
+ Genesis Compiler API (the genesis-compiler-api skill): compile `auth` first,
26
+ then every other service once, after the whole blueprint validates, and run
27
+ the result locally or deploy it yourself; the Compiler API only writes code.
@@ -0,0 +1,310 @@
1
+ # Running and debugging the generated services locally
2
+
3
+ This page is for the moment after a build: the service repos are checked out and you want to start
4
+ them, reach one, read its health, reproduce a failure and decide who owns the fault. Everything runs
5
+ on your machine with your own tools. Genesis never runs or deploys anything for you. For shipping
6
+ to a real slot see [deploy.md](deploy.md); for the build answers see [compile-results.md](compile-results.md).
7
+
8
+ ## 1. What you have
9
+
10
+ - One git repo per service, `<codename>-<service>-service`, in your own account. Check them out
11
+ side by side in one parent folder; the compose stack of section 4 expects that layout.
12
+ - The repo is the compiler's output plus your own files. You never edit generated files: the next
13
+ build overwrites them. Extension goes through custom code, see [custom-code.md](custom-code.md).
14
+ - Env files (nodejs): `.dev.env`, `.test.env`, `.stage.env`, `.prod.env` are committed with secret
15
+ keys left blank; `.<slot>.secrets.env` is your gitignored keystore. There is no `.exp.*` file.
16
+ - `Dockerfile.dev` builds the local image; `Dockerfile.prod` is the production image.
17
+ - Treat the checkout as a build artifact: after any new build of that service, pull it before you
18
+ debug. A checkout at an older commit cannot reproduce a failure of a newer one.
19
+
20
+ ## 2. Env resolution (why a value is not what the file says)
21
+
22
+ Order, strongest first:
23
+
24
+ ```
25
+ non-empty variable in the process environment (shell, compose, orchestrator)
26
+ > .<slot>.secrets.env (your local keystore)
27
+ > .<slot>.env (committed slot file)
28
+ ```
29
+
30
+ - `SERVICE_CONFIG` selects the slot file at boot; the default is `test`. The nodejs scripts set it:
31
+ `npm run dev` (ts-node) and `npm run start-dev` (built `dist/`) run the `dev` slot,
32
+ `start-stage` / `start-prod` the others; plain `npm start` runs `test`.
33
+ - The env files load through dotenv, which does not override a variable that is already set. That
34
+ is why one tree runs natively (file values) and in compose (container values) unchanged.
35
+ - A variable the process holds EMPTY is not filled from the slot file, but the keystore still
36
+ fills it. A secret you put in the keystore therefore does not win over a non-empty injected one.
37
+ - If a key looks ignored, print what the process really holds before editing files: the shell
38
+ value or the compose `environment:` entry is usually the one that won.
39
+ - Slots `test`, `stage` and `prod` carry EMPTY peer addresses (`SERVICE_URL`, `AUTH_SERVICE_URL`,
40
+ `SERVICE_<NAME>_URL`, `SERVICE_MCPBFF_URL`); the deployer fills them. `dev` keeps local ones.
41
+ The JWT public key URL is `AUTH_SERVICE_URL` + `/publickey`.
42
+ - Never copy a secrets file into a commit, an issue, a log or a bug report.
43
+
44
+ ## 3. Ports and signing keys
45
+
46
+ Dev ports (every `.dev.env` sibling URL is built from them):
47
+
48
+ | Service | Port |
49
+ |---|---|
50
+ | `auth` | 3001 |
51
+ | `bff` | 3002 |
52
+ | `notification` | 3003 |
53
+ | `document` | 3004 |
54
+ | `agenthub` | 3006 |
55
+ | `mcpbff` | 3600 |
56
+ | business services | 3050, 3051, ... in blueprint order |
57
+
58
+ Check the real value in the service's `.dev.env` (`HTTP_PORT`) before you rely on the table.
59
+
60
+ Signing keys: each service signs its tokens with its own Ed25519 key, in `SERVICE_PRIVATE_KEY_PEM`
61
+ and `SERVICE_KID` (in compose: `<SVC>_SERVICE_PRIVATE_KEY_PEM` and `<SVC>_SERVICE_KID`).
62
+ Without one the service starts degraded and the login service cannot issue sessions.
63
+
64
+ ```sh
65
+ openssl genpkey -algorithm ed25519 -out key.pem # PKCS#8 PEM
66
+ uuidgen # any UUID works as the key id
67
+ ```
68
+
69
+ Put the PEM in double quotes when it goes into a `.env`. Keys are yours; the compiler never
70
+ holds them.
71
+
72
+ ## 4. Start the stack
73
+
74
+ Two ways. Pick by what you need.
75
+
76
+ ### 4a. The `docker-compose` deploy target (all services, one command)
77
+
78
+ Ask for it in the build request (`deployTargets`, see [compiler-api.md](compiler-api.md)). It lands
79
+ in every service repo under `deploy/docker-compose/`:
80
+
81
+ - `docker-compose.yml`, `.env.example`, `.providers.secrets.env.example`, `README.md`.
82
+ - Every deployable service of the build, each built from its own repo at `../../../<repo>`
83
+ (the repo you stand in is `../..`), running its DEV slot image.
84
+ - Infrastructure by need: PostgreSQL (PostGIS 16) and/or MongoDB 7 by the service's `dbType`;
85
+ Redis 7 and Elasticsearch for every backend service; Kafka 3.7 (KRaft) for all.
86
+ They are reachable inside the compose network only; nothing but the services publishes a port.
87
+ - Every password is a `${VAR}` reference; no secret value is written into the files.
88
+
89
+ ```sh
90
+ cd deploy/docker-compose
91
+ cp .env.example .env
92
+ cp .providers.secrets.env.example .providers.secrets.env
93
+ # fill .env: POSTGRES_PASSWORD (and MONGO_PASSWORD), the signing keys per service
94
+ docker compose up --build
95
+ ```
96
+
97
+ - Needs Docker Compose 2.24 or later (optional `env_file`). `.env` and `.providers.secrets.env`
98
+ are gitignored.
99
+ - The MongoDB password goes into a connection URL: use a URL-safe one.
100
+ - Container env wins over the baked slot file. The stack sets, per service: `DB_HOST`, `DB_PORT`,
101
+ `DB_USER`, `DB_PASSWORD` (or `MONGO_URL`), `REDIS_URL`, `ELASTIC_HOST`, `KAFKA_BROKERS`,
102
+ `KAFKA_ENABLED_IN_DEV=true` (the dev slot keeps Kafka off otherwise), `AUTH_SERVICE_URL` and
103
+ `SERVICE_<NAME>_URL` as `http://<service>:<port>` for peers inside the stack.
104
+ - A peer the build does not include is `${KEY:-}`, an empty line you fill in `.env` if you run
105
+ that peer somewhere else. A missing address shows up later as an inter-service error.
106
+ - Provider credentials (payment, mail, storage) are passed to every service from one file. Add the
107
+ keys the other repos list in their own example file. A provider without a credential runs
108
+ degraded; that is expected until you add one.
109
+ - Never ask the user to paste a credential into the chat. Ask for it on the local key page instead:
110
+ 1. `genesis_requestKey` with a name, a label and where the user gets the key.
111
+ 2. Give the user the link it answers.
112
+ 3. Check `genesis_listKeys` until the key is set.
113
+ 4. `genesis_writeKeyToEnvFile` puts it into the gitignored env file (`.env`,
114
+ `.providers.secrets.env` or another `*.secrets.env`) without the value passing through you.
115
+ - Compose project name is the codename. Use `docker compose ps` and `docker compose logs -f <service>`
116
+ with the compose service names from `docker-compose.yml`.
117
+ - Stop with `docker compose down` (volumes kept); `down -v` wipes the data, which is your decision.
118
+
119
+ ### 4b. One service as a native process (fastest loop)
120
+
121
+ Use it when you reproduce one service and its siblings run elsewhere.
122
+
123
+ 1. Start the infrastructure the service's `.dev.env` names (`DB_HOST`, `REDIS_URL`, `ELASTIC_HOST`,
124
+ `KAFKA_BROKERS`, or `MONGO_URL`). The simplest source is the same compose file: bring up only
125
+ the infra services (`docker compose up -d postgres redis elasticsearch kafka`), then publish
126
+ their ports yourself if the compose file does not (it does not by default).
127
+ 2. `npm install` in the repo once, then `npm run dev` (or `npm run build` and `npm run start-dev`).
128
+ 3. The service binds its own port. Every sibling the request path calls must also be running, on
129
+ its own dev port or in compose, and its address must be in this service's env.
130
+ 4. Kafka advertises the in-network name `kafka:9092`. A native process on the host cannot resolve
131
+ it unless `kafka` maps to `127.0.0.1` in `/etc/hosts`. Say so when a Kafka-consuming service runs
132
+ natively and does not get its events.
133
+
134
+ ### 4c. No container runtime on the machine
135
+
136
+ Both ways above need a container runtime for the infrastructure. Check it first:
137
+ `docker info` (or `docker compose version`). When it is missing or the daemon is not running,
138
+ do not guess. Tell the user and offer the two ways below.
139
+
140
+ **1. Install a runtime (recommended).**
141
+
142
+ The user installs it; the installer asks for an administrator password and a licence acceptance,
143
+ which are the user's to give. You may name the download page or prepare the package-manager
144
+ command; run it only when the user says so, and never type their password.
145
+
146
+ | OS | Runtime | Notes |
147
+ |---|---|---|
148
+ | macOS | OrbStack (`brew install --cask orbstack`) or Docker Desktop | OrbStack is lighter; it is paid for commercial use. |
149
+ | Linux | Docker Engine from the distribution's or Docker's packages | The user then joins the `docker` group or uses `sudo`. |
150
+ | Windows | Docker Desktop with the WSL 2 backend | Run the stack inside WSL 2. |
151
+
152
+ - Docker Desktop is paid for larger companies (its subscription terms decide); say so, and
153
+ let the user choose.
154
+ - After the install: start the runtime and run `docker info` again. Then continue with 4a or 4b.
155
+
156
+ **2. One service natively, with the database only.**
157
+
158
+ Only the database is critical at boot (section 6). Redis, Elasticsearch and Kafka are optional,
159
+ so one service can run with just a PostgreSQL (or MongoDB) and no container at all.
160
+
161
+ 1. **A database.** Either one installed on the machine (for example `brew install postgresql@16`
162
+ and `brew services start postgresql@16`) or a hosted one the user creates in their own
163
+ account.
164
+ 2. **Its address.** Put `DB_HOST`, `DB_PORT`, `DB_USER` and `DB_PASSWORD` (for MongoDB,
165
+ `MONGO_URL`) in the gitignored `.dev.secrets.env` or in the shell environment. Both win over
166
+ the committed `.dev.env` (section 2), which you never edit. The service creates its own
167
+ database on the server.
168
+ 3. **Start it.** `npm install`, then `npm run dev` (4b).
169
+
170
+ Say what is missing before you report anything:
171
+
172
+ - Without Redis there is no cache.
173
+ - Without Elasticsearch the reads that use it answer `503 ELASTIC_UNAVAILABLE`.
174
+ - Without Kafka no event is published or consumed, so event-driven flows and their tests cannot
175
+ pass.
176
+
177
+ A test that needs a missing dependency is reported as skipped with the reason, never as passed.
178
+
179
+ ## 5. Resource checklist (before `up`)
180
+
181
+ - RAM: the infrastructure alone takes about 3 GB (Elasticsearch heap is set to 512 MB); each node
182
+ service needs 200-400 MB more. Give Docker 6 GB or more and run only the services the
183
+ reproduction needs.
184
+ - Ports: each service publishes its own dev port; free them first (`lsof -i :<port>`). Two stacks
185
+ of different projects on one machine collide on the same dev ports: run one at a time or change
186
+ the published port in the compose file you own.
187
+ - One heavy job at a time on one machine: not a full stack, a test run and a build together.
188
+ - Report what is left running when you stop.
189
+
190
+ ## 6. Health: alive is not ready
191
+
192
+ `GET http://localhost:<port>/health` answers 200 `{"status":"ok"}` as soon as the port is bound, and
193
+ in every boot state. That is liveness only. Read the whole body:
194
+
195
+ | Field | Meaning |
196
+ |---|---|
197
+ | `readiness.ready` | `false` until the boot chain finished |
198
+ | `readiness.schema` | the schema step outcome; `ready:false` with a failed schema means every request that touches a missing table fails |
199
+ | `boot.state` | `starting`, `ready` or `degraded` |
200
+ | `boot.bootErrors[]` | `step`, `code`, `critical`, `retryable`, `attempts`, `lastAttemptAt`, `nextRetryAt` |
201
+
202
+ - Nothing is "running" until `boot.state` is `ready` and `readiness.ready` is `true`.
203
+ - While starting, every other route answers `503 SERVICE_STARTING`. When a critical step fails,
204
+ every other route answers `503 SERVICE_DEGRADED` with a `requestId` and `Retry-After`, and the
205
+ body does not name the cause. The cause is in `/health` (`code`) and in the service log.
206
+ - The service never exits on a boot error. It logs and keeps answering. Do not wait for a crash.
207
+ - Public codes: `DB_UNREACHABLE`, `DB_AUTH_FAILED`, `SCHEMA_FAILED`, `CONFIG_ENV_INVALID`,
208
+ `CONFIG_INVALID`, `IDENTITY_KEY_MISSING`, `PORT_IN_USE`, `BUILD_INCOMPLETE`,
209
+ `DEPENDENCY_UNSUPPORTED`, `STEP_FAILED`.
210
+ - `SchemaBootstrapFailed` and `ModelSyncFailed` in the log mean the schema did not materialise
211
+ (for example a permission or missing-table fault). Waiting or restarting does not fix it: fix the
212
+ database or the design, then rebuild.
213
+ - A pending migration, an absent optional Elasticsearch or Kafka, never make `ready` false.
214
+ - Do not probe business endpoints to "verify" the boot. They need auth, bodies and fixtures, and
215
+ answer 401/403/400/404 for reasons that teach nothing. Use the health body, then the test suite.
216
+
217
+ Boot verdicts by dependency (nodejs):
218
+
219
+ | Step | Failure | Verdict | Retried |
220
+ |---|---|---|---|
221
+ | `postgres`, `mongodb` | refused or timed out | critical `DB_UNREACHABLE` | yes, background |
222
+ | `postgres`, `mongodb` | credential refused | critical `DB_AUTH_FAILED` | yes |
223
+ | `mongodb` | malformed `MONGO_URL` | critical `CONFIG_INVALID` | no |
224
+ | `migrations` | lock or schema step failed | critical `SCHEMA_FAILED` | yes |
225
+ | `service-identity` | no `SERVICE_KID` / `SERVICE_PRIVATE_KEY_PEM` on a login-minting service | critical `IDENTITY_KEY_MISSING` | no |
226
+ | `models` | a model definition throws | critical `STEP_FAILED` | no (a generated-code defect: report it) |
227
+ | `redis` | unreachable after its attempts | optional; runs without cache | no |
228
+ | `elastic`, `kafka` | unreachable | optional; heals on its own | yes |
229
+ | `elastic` | endpoint is not Elasticsearch | optional `DEPENDENCY_UNSUPPORTED`; reads answer `503 ELASTIC_UNAVAILABLE` | no |
230
+
231
+ `/admin/health` carries the full, redacted detail but is gated; use the log if you cannot reach it.
232
+
233
+ ## 7. Dependencies that start late
234
+
235
+ A cold start where the database is a second behind is normal; the service heals without a restart.
236
+
237
+ - Kafka, Elasticsearch: unbounded background retry, boot never blocked. Env: `KAFKA_RECONNECT_INTERVAL_MS`
238
+ (5000) and `KAFKA_RECONNECT_MAX_INTERVAL_MS` (60000); `ELASTIC_RECONNECT_INTERVAL_MS` (5000) and
239
+ `ELASTIC_RECONNECT_MAX_INTERVAL_MS` (60000). A literal `0` on a `*_MAX_INTERVAL_MS` opts out.
240
+ `/health` echoes `{connected, reason?}` for each, so "still converging" differs from "misconfigured".
241
+ - Redis, Postgres: a short bounded retry inside boot (`REDIS_RECONNECT_*`, `DB_RECONNECT_*`; defaults
242
+ 3 attempts, 1s initial, 8s cap; `*_MAX_ATTEMPTS=1` tries once; the budget is capped at 20). When it is
243
+ spent, Postgres runs degraded and the boot supervisor retries in the background
244
+ (`BOOTSTRAP_RETRY_INTERVAL_MS` 5000 rising to `BOOTSTRAP_RETRY_MAX_INTERVAL_MS` 60000).
245
+ - Blank, negative or non-decimal values count as unset; intervals have a 1000 ms floor.
246
+ - Close budgets (`DB_CLOSE_TIMEOUT_MS`, `ELASTIC_CLOSE_TIMEOUT_MS`, `KAFKA_CLOSE_TIMEOUT_MS`,
247
+ `REDIS_CLOSE_TIMEOUT_MS`, `MONGO_CLOSE_TIMEOUT_MS`, default 5000) only matter for slow shutdowns.
248
+ - Retry lines are `warn`, not `error`; the verdict lines to alert on are `redisStartFailed`,
249
+ `postgresStartFailed` and `BootstrapDegraded`. A permanent Kafka/Elasticsearch outage logs one loud
250
+ line about every tenth attempt once the backoff has settled.
251
+ - Shutdown on `SIGTERM` is graceful and exits 0; that is the only exit.
252
+
253
+ ## 8. Reproduce a failure
254
+
255
+ 1. Capture the failing call exactly: method, path, body, which actor, the response (`status`, the
256
+ error code and message) and its `requestId`. Find the log line with the same `requestId`: it
257
+ names the stage and the generated file that threw. Keep the excerpt as the comparison target.
258
+ 2. Confirm the commit: the checkout you run is the commit of the build you are debugging.
259
+ 3. Decide the running set: every sibling the request path calls must run too. An inter-service error
260
+ that your original failure did not show is your setup, not the defect.
261
+ 4. Bootstrap identity locally: the service seeds its bootstrap super admin from the env it booted
262
+ with; the password is the one in your blueprint or `$ENV:` reference. Customer and staff actors
263
+ come from your own register, verify, login and role-grant calls.
264
+ 5. Replay the one failing case, ideally from your test suite ([testing.md](testing.md)), not by hand.
265
+ Same error and same stack in the local log means reproduced. A different symptom means your local
266
+ setup differs (env, data, a missing sibling); do not go on until they match.
267
+ 6. Instrument on a throwaway copy: log the assembled `dataClause` / `whereClause` in the api manager
268
+ the stack names, run `node --inspect`, rerun the one case. Narrow it to one of three owners
269
+ (section 9).
270
+ 7. Read logs the same way each time: native, the process stdout; compose, `docker compose logs -f`.
271
+ Quote the original line and the local line together.
272
+ 8. Inspect state directly when the log is not enough: query the service's database, its Redis keys,
273
+ its Kafka topic or Elasticsearch index with your own client against the local infrastructure.
274
+ Compose publishes no infra ports by default, so run the client inside the network
275
+ (`docker compose exec <infra> ...`).
276
+
277
+ Payment and subscription runtime checks (webhook, mock provider, state ladder) have their own
278
+ ladders: follow the Genesis additions in the `genesis/procedures/` folder of this corpus (the
279
+ payment and subscription files, and its index) instead of repeating them here.
280
+
281
+ ## 9. Whose fault is it, and where the fix goes
282
+
283
+ | You found | Owner | Action |
284
+ |---|---|---|
285
+ | An authored slot is wrong: an MScript expression, a where column, an action's parameters, a role or route | Blueprint | Fix the blueprint, validate, rebuild, pull, rerun. Discard the local experiment. |
286
+ | Your custom code (a library or edge function body in the blueprint) | Custom code | Fix it in the blueprint ([custom-code.md](custom-code.md)), validate, rebuild. |
287
+ | A file you added to the repo, or under `service-files/` | You | Fix it in place; it survives rebuilds. |
288
+ | A generated framework line mishandles a legal input, or a model/route fails to compile or load | Compiler | Report it with the evidence below. Do not patch the generated file. |
289
+ | Missing env, key, provider credential or peer address | Your deployment | Fix env or keys; nothing to rebuild. |
290
+
291
+ Report a compiler fault with: the request id or build job id of the build, the repo and commit, the
292
+ generated file and line, the input that reaches it, the local log line and the original `requestId`.
293
+ A local reproduction is the strongest report; file it before you ship any workaround.
294
+
295
+ Never:
296
+
297
+ - hand-edit a generated file as a durable fix (it is overwritten and counted at the next build);
298
+ - weaken a blueprint or test just to get green; classify the red first;
299
+ - copy an env or secrets file into a commit, a report or a chat;
300
+ - treat a local edit that confirmed the diagnosis as the fix. The fix lands in the blueprint or in
301
+ your custom code, then a rebuild ships it.
302
+
303
+ ## 10. Checklist
304
+
305
+ - [ ] Checkouts at the commit of the build being debugged; container runtime (`docker info`), RAM and ports checked, or the no-runtime way of 4c chosen with the user.
306
+ - [ ] `.env` filled: passwords and one Ed25519 key and key id per service.
307
+ - [ ] Every service `boot.state: ready` and `readiness.ready: true`, not just 200 OK.
308
+ - [ ] Failure reproduced with the same symptom, or reported as not reproduced.
309
+ - [ ] Owner decided: blueprint, custom code, compiler, deployment; fix routed there.
310
+ - [ ] Stack stopped, or the running state reported.