@mindbricks/genesis-set 0.0.0-stage → 0.2.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (41) hide show
  1. package/CHANGELOG.md +44 -0
  2. package/LICENSE +202 -0
  3. package/README.md +148 -2
  4. package/bin/genesis-set.mjs +153 -0
  5. package/corpus/README.md +25 -0
  6. package/corpus/genesis/compile-results.md +246 -0
  7. package/corpus/genesis/compiler-api.md +265 -0
  8. package/corpus/genesis/custom-code.md +519 -0
  9. package/corpus/genesis/deploy.md +190 -0
  10. package/corpus/genesis/pitfalls.md +231 -0
  11. package/corpus/genesis/procedures/README.md +17 -0
  12. package/corpus/genesis/procedures/agent-hub-blueprint.md +13 -0
  13. package/corpus/genesis/procedures/auth-blueprint.md +14 -0
  14. package/corpus/genesis/procedures/bff-blueprint.md +16 -0
  15. package/corpus/genesis/procedures/blueprint-authoring.md +16 -0
  16. package/corpus/genesis/procedures/howto-mail-to-tasks.md +26 -0
  17. package/corpus/genesis/procedures/howto-payment-flow.md +112 -0
  18. package/corpus/genesis/procedures/howto-subscription.md +33 -0
  19. package/corpus/genesis/procedures/notifications-blueprint.md +13 -0
  20. package/corpus/genesis/procedures/patterns-catalog.md +12 -0
  21. package/corpus/genesis/procedures/project-lifecycle.md +27 -0
  22. package/corpus/genesis/run-locally.md +310 -0
  23. package/corpus/genesis/testing.md +532 -0
  24. package/lib/config-cli.mjs +179 -0
  25. package/lib/install.mjs +639 -0
  26. package/lib/mcp-config.mjs +222 -0
  27. package/lib/semver.mjs +94 -0
  28. package/mcp/blueprint.mjs +64 -0
  29. package/mcp/client.mjs +141 -0
  30. package/mcp/config.mjs +166 -0
  31. package/mcp/key-page-html.mjs +275 -0
  32. package/mcp/key-page.mjs +217 -0
  33. package/mcp/key-store.mjs +204 -0
  34. package/mcp/key-tools.mjs +225 -0
  35. package/mcp/server.mjs +164 -0
  36. package/mcp/tools.mjs +270 -0
  37. package/package.json +44 -4
  38. package/skills/genesis-compiler-api/SKILL.md +56 -0
  39. package/skills/genesis-deploy/SKILL.md +36 -0
  40. package/skills/genesis-generated-code/SKILL.md +48 -0
  41. package/skills/genesis-run-and-test/SKILL.md +43 -0
@@ -0,0 +1,519 @@
1
+ # Extending the generated code: custom code
2
+
3
+ This page is for the moment a blueprint needs behaviour that its patterns cannot express. It says
4
+ where custom code lives, how the compiler turns it into the generated service, what it can call at
5
+ run time, and what survives a rebuild. How to design the library and edge patterns themselves is
6
+ in mind-set (`procedures/blueprint-authoring.md`, section "Library functions"); this page covers
7
+ what the compiler does with them. Related pages: [compiler-api.md](compiler-api.md),
8
+ [compile-results.md](compile-results.md), [testing.md](testing.md), [pitfalls.md](pitfalls.md).
9
+
10
+ ## The central rules
11
+
12
+ 1. **Custom code is part of the blueprint.** Library function bodies and edge function bodies are
13
+ fields of the blueprint (`moduleBody`). The compiler re-injects them into the service at every
14
+ build. To change them, change the blueprint and compile again.
15
+ 2. **Never edit a generated file to change behaviour.** The next compile overwrites it, and the
16
+ overwrite is counted in that build's `addedLines`.
17
+ 3. **Files you add survive.** A file you add to a service repo outside the generated paths stays
18
+ through every rebuild. Service files under `service-files/` are always yours.
19
+ 4. **Where service-file bytes live.** The bytes stay in your service repo. The blueprint
20
+ holds only the manifest (`path`, `visibility`, `contentType`, `size`, `sha256`); the compiler
21
+ reads the bytes from your repo once builds compile onto your repositories. Until then (dry runs)
22
+ a build skips the declared service files and its log says so (`service files skipped
23
+ in Genesis builds`).
24
+ 5. **Custom code runs with the service's full privileges.** There is no sandbox for MScript, arrow
25
+ functions, library functions or edge functions. Review them like any code you ship.
26
+
27
+ ## The four escape hatches
28
+
29
+ Pick the lowest layer that carries the requirement.
30
+
31
+ | Hatch | Scope | Language |
32
+ |---|---|---|
33
+ | MScript | one pattern field (condition, where clause, default, response value) | JavaScript, transpiled to the service language |
34
+ | `ArrowFunctionAction` | one step inside one Business API | JavaScript body, spliced into the service |
35
+ | `ServiceLibrary` function | reused across the service, callable as `LIB.<moduleName>(...)` | the service's own language |
36
+ | `EdgeController` + edge function | a whole route (REST, Kafka) outside the action workflow | the service's own language |
37
+
38
+ Choice rules:
39
+
40
+ - A predicate, derived value or payload field: MScript.
41
+ - A transform of 1 to 10 lines used in one API that MScript cannot express: `ArrowFunctionAction`.
42
+ - Logic used in three places, or longer than about 10 to 30 lines: a library function.
43
+ - A route that must own the whole request (provider webhook with its own signature, binary
44
+ download, a bespoke report, a Kafka consumer): an edge controller. If the operation reads and
45
+ writes data objects in the request, validate, act, respond shape, declare a Business API instead.
46
+ - If it is persisted, declare it on the data object. Custom code is behaviour, not data. Tables
47
+ that a library reaches into outside the blueprint are invisible to migrations and will drift. For a
48
+ derived counter, declare the column and keep it up to date with an update action on a cron API.
49
+
50
+ Only MScript is portable. Library and edge bodies are never transpiled and never bridged: they are
51
+ written in the service's language.
52
+
53
+ ## `LIB` and `API`: the ambient containers
54
+
55
+ In nodejs every hatch reaches the same two globals. No import is needed.
56
+
57
+ | Container | Holds | Call |
58
+ |---|---|---|
59
+ | `LIB` | this service's `library.functions[]` modules (plus native kernels) | `LIB.<moduleName>(...)` |
60
+ | `API` | this service's Business APIs that set `controllers.internalCallSettings.hasInternalCallController: true` | `await API.<apiName>({ ...params })` |
61
+
62
+ They are installed on `globalThis` at boot (`src/index.ts`) and typed by `src/lib/index.ts` and
63
+ `src/global.d.ts`. Rules:
64
+
65
+ - Call them inside the exported function, never at module load time. A top-level `API.x()` runs
66
+ before the install.
67
+ - `API` names are exact and case-sensitive. An unknown name is `INTERNAL_API_NOT_FOUND` and an API
68
+ without the controller is `INTERNAL_API_NO_CONTROLLER`, both at design time. Computed access
69
+ (`API[expr]`) cannot be checked and draws a warning.
70
+ - An internal call runs as the system with a null session. On nodejs and rust the second argument
71
+ carries a session: `await API.getOrder(params, { session: this.session })`. Even then role and
72
+ ownership gates stay bypassed, so the calling code must enforce access. A `session` key inside
73
+ `params` is plain input data.
74
+ - An internal call bypasses object moderation: a get or list of an `onPlace` object returns every
75
+ row, unapproved ones too. The same holds for direct `dbLayer` reads. Filter before relaying to a
76
+ user.
77
+ - Nested internal calls fail with `INTERNAL_CALL_DEPTH_EXCEEDED`.
78
+
79
+ ## Library functions
80
+
81
+ `services.library` holds one library per service, with these partitions:
82
+
83
+ | Partition | Holds | Body key |
84
+ |---|---|---|
85
+ | `functions[]` | reusable functions | `ts` (preferred) or `js` on nodejs; the language key elsewhere |
86
+ | `edgeFunctions[]` | request handlers | same |
87
+ | `templates[]` | render templates | `ejs` |
88
+ | `assets[]` | server-side static text | `json`, `svg`, `txt`, `html` |
89
+ | `public[]` | files served at a public URL | same |
90
+ | `pdfTemplates[]` | PDF templates (page setup, header, footer, metadata) | EJS body plus layout fields |
91
+ | `hooks[]` | binds a `functions[]` module to service startup or shutdown | none |
92
+
93
+ ### How the compiler picks a body
94
+
95
+ `moduleBody` is an object keyed by language, not file extension: `ts`, `js`, `python`, `java`,
96
+ `csharp`, `go`, `rust`. The compiler takes the key of the service's language (nodejs: `ts`, then
97
+ `js`). A module with no body for that language does not fail the build. It is emitted as a stub that
98
+ throws at runtime (for example "lib 'computeShipping' has no nodejs (js/ts) body") and the build
99
+ prints a warning. So read every build warning, and give each language you generate its own body.
100
+ A `php` key is kept but no service reads it.
101
+
102
+ Name rules: `moduleName` is camelCase, unique in its partition, and is the name MScript uses and
103
+ the file base name.
104
+
105
+ ### A nodejs function
106
+
107
+ A module must export one function. `module.exports = { a, b }` and `exports.a = ...` are refused
108
+ (`NO_EXPORT`), because the runtime binds the module itself as `LIB.<moduleName>`.
109
+
110
+ ```ts
111
+ module.exports = function shippingFee(order: { weightKg: number; zone: string; subtotal: number }): number {
112
+ if (!order || order.weightKg <= 0) throw new Error('order weight required');
113
+ if (order.subtotal >= 250) return 0;
114
+ return Math.round(order.weightKg * 12 * 100) / 100;
115
+ }
116
+ ```
117
+
118
+ Call it from a `FunctionCallAction` with `callScript: { "js": "LIB.shippingFee({ ... })" }`. The
119
+ result lands on the workflow context as `this.<contextPropertyName>` (the action `name` when no
120
+ property name is set). Schedule that action before the step that reads it. For example, a data
121
+ clause item that reads it needs the action at `afterCheckBasicAuth`, because the data clause is
122
+ built after that step.
123
+
124
+ Several helpers in one module: declare the function, attach members, export it.
125
+
126
+ ```js
127
+ function textTools() {}
128
+ textTools.slug = (s) => String(s).toLowerCase().replace(/\s+/g, '-');
129
+ module.exports = textTools;
130
+ ```
131
+
132
+ Then `LIB.textTools.slug(this.title)` works. Write `await` yourself where an attached member is
133
+ async; the validator does not check it. The validator trusts attached members only for this plain
134
+ shape (no `this`, `arguments`, `eval`, prototype tricks or reads of `process`). Anything else is
135
+ valid JavaScript, but the members are then not counted and `LIB_BARE_FUNCTION_MEMBER_ACCESS`
136
+ fires.
137
+
138
+ Undeclared names fail the build before anything is committed. The failed build names the function,
139
+ the identifier and the line. Allowed without declaration: `LIB`, `API`, the CommonJS names, Node
140
+ globals and JavaScript built-ins. Reach a runtime-specific name through `globalThis.Deno?.exit()`,
141
+ not `Deno.exit()`. The same check runs in the repo as the `correctness/noUndeclaredVariables` lint rule on
142
+ `src/lib/functions/**` and `src/lib/edge/**`: `npm run lint` shows it.
143
+
144
+ To give a function the whole workflow context (API params, the session, earlier action results),
145
+ pass `this`: `LIB.quoteOrder(this)`. Pass plain fields instead for a pure helper that is easy to
146
+ test. A function that writes rows forwards `{ session, requestId }` to the `dbLayer` helper.
147
+
148
+ ### Publishing Kafka events from a function
149
+
150
+ A function that publishes with `ServicePublisher` lists every topic in `publishes` on the library
151
+ module, named as it goes on the wire (`<codename>-<service>-service-<event>`). The compiler cannot see
152
+ inside a body. Without the declaration, a listener on that topic is reported producerless at boot
153
+ (`KafkaSubscriptionsWithoutProducer`, ERROR) and at design time (`KAFKA_TOPIC_NO_PRODUCER`). Code that
154
+ is not a library module declares on `serviceSettings.serviceOptions.publishes`. A lifecycle hook that must
155
+ announce a fact calls a library function that publishes. An `API.*` call from a hook that already
156
+ runs inside an internal call fails with `INTERNAL_CALL_DEPTH_EXCEEDED`.
157
+
158
+ ### Startup and shutdown hooks
159
+
160
+ `hooks[]` entries (`name`, `trigger` of `startup` or `shutdown`, `moduleName`) run a `functions[]`
161
+ module automatically. Only `functions[]` modules are valid (`HOOK_MODULE_NOT_FOUND`).
162
+
163
+ - Startup hooks run on every boot, so the function must be idempotent (upserts, existence checks,
164
+ version stamps; never blind inserts).
165
+ - They run after the database, migrations, seed data and Kafka wiring, and before the HTTP listener binds.
166
+ - A hook that throws never blocks `/health` and never crashes boot. It is logged as
167
+ `StartupHookFailed` and the service comes up degraded. Success is `StartupHookCompleted`.
168
+ - A startup hook is awaited for at most `LIBRARY_STARTUP_HOOK_BUDGET_MS` (default 30000). After
169
+ that boot proceeds (`StartupHookStillRunning`) while the hook keeps running.
170
+ - Shutdown hooks run best effort on SIGTERM and SIGINT, bounded by
171
+ `LIBRARY_SHUTDOWN_HOOK_BUDGET_MS` (default 10000). A killed process runs nothing, so never
172
+ depend on one for correctness.
173
+
174
+ ### Service files in code
175
+
176
+ On a nodejs service that has at least one service file, `LIB.serviceFiles` exists:
177
+ `text(path)`, `json(path)`, `bytes(path)`, `info(path)`, `has(path)`. All are synchronous (no
178
+ `await`). An unlisted path throws `ServiceFileNotFoundError`; a missing or altered file throws
179
+ `ServiceFileIntegrityError`. Do not name a library module `serviceFiles`. A private file is not secret:
180
+ it sits in the repo and in every image built from it. The manifest design is in mind-set.
181
+
182
+ ## Edge controllers
183
+
184
+ Two blueprint pieces that reference each other by name: an entry in `services.edgeControllers[]`
185
+ and an `edgeFunctions[]` module. `edgeControllerOptions.functionName` must equal the module's
186
+ `moduleName`. The compiler emits an adapter that authenticates, builds the context and calls your
187
+ function.
188
+
189
+ ### Triggers
190
+
191
+ | Block | Fields | Generated |
192
+ |---|---|---|
193
+ | `edgeRestSettings` | `restEnabled`, `method`, `path` | a REST route |
194
+ | `edgeKafkaSettings` | `kafkaEnabled`, `requestTopic`, `responseTopic?` | a Kafka consumer; the return value goes to `responseTopic` |
195
+ | `edgeCronSettings` | `cronEnabled`, `cronExpression` | nothing: no generator schedules it |
196
+
197
+ `cronEnabled: true` draws the warning `EDGE_CRON_NOT_SCHEDULED` and the function is never called by
198
+ a clock. For scheduled work use a service cron job (`Service.cronJobs[]`), a cron-triggered
199
+ Business API, or an external scheduler that calls the REST route.
200
+
201
+ ### Where the route lives
202
+
203
+ `edgeRestSettings.path` is relative to the edge sub-router. The mount prefix is `/edge`, so
204
+ `/generate-invoice` is served at `/edge/generate-invoice`. Never write the prefix into `path`
205
+ (`/edge/x` would be served at `/edge/edge/x`). A deployment can change the prefix with the env var
206
+ `EDGE_MOUNT_PATH` (unset or blank is `/edge`; `/` mounts at the root). The declared paths never
207
+ change. Edge routes stay out of the `/v1` business API routes.
208
+
209
+ ### Kafka consumer group
210
+
211
+ A Kafka edge controller consumes in the group `<SERVICE_CODENAME>-edge-<functionName>`: keyed by
212
+ the function, not the topic. Two functions on the same `requestTopic` both receive every message
213
+ and each publishes its own response. If you want one handler, use one edge controller and branch
214
+ inside it. `requestTopic` defaults to `<functionName>-requests`. Replicas of the same function share
215
+ the partitions.
216
+
217
+ ### The handler context
218
+
219
+ One async export that receives one normalized context, not an Express request:
220
+
221
+ | Field | Meaning |
222
+ |---|---|
223
+ | `body` | REST: `{ ...req.body, ...req.query }`; Kafka: the payload |
224
+ | `headers` | request headers (REST); `{}` on Kafka |
225
+ | `session` | REST: the user session, or `null` on a public or service-token-only route; Kafka: the identity the event arrived with, or `null` |
226
+ | `isM2M`, `m2mIssuer` | true only for a verified, admitted service token, and the issuing service name from the token; always false on Kafka |
227
+ | `requestId` | the correlation id, never empty (a fresh UUID when none came in) |
228
+ | `service`, `function` | for logging |
229
+
230
+ ### Return contract
231
+
232
+ - A plain value is sent as JSON with HTTP 200, even an object that has a `status` member (it is
233
+ just a JSON field).
234
+ - A response envelope `{ status, headers?, body? }` is honoured when a numeric `status` comes with a
235
+ `body` and/or an object `headers`. A `Buffer` or string `body` is sent raw (images, CSV, PDF);
236
+ anything else is JSON. A bare `{ status: 200, ... }` stays a JSON body.
237
+ - To refuse, throw an error with a numeric `.status`; the adapter answers `application/problem+json`
238
+ with that code (500 by default).
239
+ - On a Kafka controller with a `responseTopic`, a plain return is published as is and an envelope
240
+ publishes its `body`. An envelope with `status >= 400` is a refusal: it is logged as
241
+ `EdgeKafkaMessageRejected` and never published, and the offset commits. Without a `responseTopic`
242
+ the return value is ignored.
243
+
244
+ ### Auth
245
+
246
+ The adapter enforces auth before your function runs:
247
+
248
+ | `loginRequired` | `M2MAllowed` | Mode | Needs | Refusal |
249
+ |---|---|---|---|---|
250
+ | true | false | session | a user session | 401 `SESSION_REQUIRED` |
251
+ | false | true | m2m | a verified, admitted service token | 401 `M2M_TOKEN_REQUIRED`, 403 `M2M_CLIENT_NOT_ALLOWED` |
252
+ | true | true | both | a session or a service token | 401 `SESSION_REQUIRED`, 403 `M2M_CLIENT_NOT_ALLOWED` |
253
+ | false | false | public | nothing | none |
254
+
255
+ The credential is the `x-m2m-token` header (an `Authorization: Bearer` token is tried only when
256
+ `x-m2m-token` is absent, and only if it verifies as a service token). Verified means an EdDSA
257
+ service JWT from a service of this project, signed by that service's key and addressed to this
258
+ service. Admitted means it passes `serviceOptions.allowedM2MClients` / `blockedM2MClients`. Kafka
259
+ has no per-message check. A webhook with a provider signature is a public (or M2M) route that
260
+ verifies `ctx.headers[...]` itself and throws a `.status = 401` error on a bad signature.
261
+
262
+ ### Other languages
263
+
264
+ - Library and edge bodies are never bridged. Without a native body there is a warning and a stub
265
+ that fails at runtime.
266
+ - .NET: `moduleBody.csharp`, a global static PascalCase class with
267
+ `Task<Dictionary<string, object?>> Invoke(IDictionary<string, object?> ctx)`. For typed `LIB`
268
+ forwarders a leading comment `// @mindbricks-lib-invoke {"returnType":...,"parameters":[...]}` declares each
269
+ `Invoke` overload (bad metadata is `DOTNET_LIB_INVOKE_METADATA_INVALID`; missing metadata is the warning
270
+ `DOTNET_LIB_INVOKE_METADATA_MISSING`). Call `global::LIB.echo(...)`; `API.getOrder(input)` returns a
271
+ `Task`.
272
+ - Java: `moduleBody.java`; static `API` and `LIB` in the service's `lib` package;
273
+ `API.<name>(params)`. An API name outside the membership is `JAVA_NATIVE_API_UNBOUND`.
274
+ - Rust: `moduleBody.rust`. A function is `src/lib_layer/<snake_name>.rs`; an edge function is
275
+ `src/edge_functions/<snake_name>.rs` with
276
+ `pub async fn handle(ctx: Value, pool: Arc<PgPool>) -> Result<Value, EdgeError>`. There is no
277
+ `LIB` or `API` identifier: use `crate::lib_layer::<snake_name>(...)` and
278
+ `crate::api::<snake_name>(params, session).await` (`crate::api::blocking::...` in a synchronous
279
+ body). Errors `RUST_INTERNAL_API_UNBOUND`, `RUST_INTERNAL_API_VALUE_UNSUPPORTED`,
280
+ `RUST_NATIVE_API_UNBOUND`. Rust edge routes exist for PostgreSQL only. A binary body is a base64 string
281
+ with `"bodyEncoding": "base64"` (`EDGE_BODY_ENCODING_INVALID`, `EDGE_BODY_BASE64_INVALID`).
282
+ - Python: `moduleBody.python`. A function is `src/<package>/lib/functions/<moduleName>.py` and must define a
283
+ top-level function named `<moduleName>`; an edge function is `src/<package>/lib/edge/<moduleName>.py`
284
+ with `handler(ctx)`. `API` and `LIB` are bare names. `await API.<apiName>(params)`; there is no
285
+ session argument. A wrong name raises an error naming `INTERNAL_API_NOT_REGISTERED`.
286
+ - Go: the edge REST route is mounted with auth and answers `501 EDGE_FN_UNSUPPORTED`; its Kafka edge
287
+ transport is absent.
288
+ - Payment lifecycle hooks are not lowered on Rust or Python (`PAYMENT_LIFECYCLE_HOOK_NOT_LOWERED`),
289
+ so an `API` call in a hook does not run there.
290
+ - When converting an existing nodejs blueprint, keep the original `ts`/`js` bodies, add the native
291
+ body, check each missing-body warning, and test the real route. A build that succeeds is not proof
292
+ of behaviour.
293
+
294
+ ## Native MScript expression slots
295
+
296
+ Use the portable `js` expression for a blueprint that targets several languages. A
297
+ nonempty native expression is inserted into its target's generated evaluator.
298
+ Expression slots and library module bodies have different contracts.
299
+
300
+ | Target | Native expression contract |
301
+ | --- | --- |
302
+ | Java | The generated manager evaluates the expression in a synchronous `Supplier`. `this` is the manager. Use a map lookup or `MScriptRuntime.prop(this.order, "status")` for nested values. `prop` returns null for a null receiver. |
303
+ | .NET | The manager evaluates the expression in a synchronous `Func`. Read custom context from `Context["order"]` or `Ms.Get(this, "order")`. The lambda has no top-level `await`. The wrapper completes a returned `Task` or `ValueTask` before using its value. |
304
+ | Rust | In API managers, `this` is the JSON snapshot from `self.this_view()`. Read members with JSON indexing or the MScript `member` helper. The synchronous closure returns the value directly. Value-model slots require a `serde_json::Value`, such as `serde_json::json!(true)`. Other slots can wrap the result in JSON themselves. |
305
+
306
+ For .NET expressions, the first non-nullish key wins in this order
307
+ `csharp`, `cs`, `dotnet`. An empty winning key falls back to translating `js`
308
+ (or `value`) and shadows subsequent aliases. For example, in
309
+ `{ "js": "40 + 2", "csharp": "", "cs": "6 * 7" }`, .NET translates
310
+ `40 + 2`. Remove the empty alias to use `cs`. A .NET library module body
311
+ uses only `moduleBody.csharp`.
312
+
313
+ Design validation, native syntax checking and translation during generation
314
+ each report their own findings. A valid design can still encounter a per-slot
315
+ translation refusal. For example, fiscal lifecycle conditions and scripts
316
+ refuse translator warnings and unsupported waits, while JobStream reports
317
+ translator warnings. Use the failed build's `buildErrors` entry to locate the
318
+ service, slot and source expression. Read [the compile result guide](compile-results.md)
319
+ for diagnosis and repair the exact expression that failed.
320
+
321
+ ## What custom code can call (nodejs)
322
+
323
+ Import aliases, resolved at compile time, at run time and under test:
324
+
325
+ | Alias | Is |
326
+ |---|---|
327
+ | `common` | the framework runtime barrel |
328
+ | `dbLayer` | per-data-object CRUD helpers generated from this service's `dataObjects[]` |
329
+ | `models` | raw Sequelize or Mongoose models (advanced; skips indexing, events, soft delete) |
330
+ | `apiLayer`, `controllers`, `agentLayer` | internal runtime; rarely imported by hand |
331
+ | `bucketLayer` | bucket layer; `bucketLayer/bucket-router` exports `readBucketObject`, `writeBucketObject`, `deleteBucketObject` |
332
+
333
+ There is no `serviceCommon` package and no such alias. `ServicePublisher` and `ElasticIndexer`
334
+ ship from `common`. Code that imports `serviceCommon` must import from `common`.
335
+
336
+ Useful `common` exports: `ServicePublisher` (Kafka event with the caller's session:
337
+ `new ServicePublisher(eventName, data, session, requestId).publish()`), `KafkaListener`,
338
+ `dbEventTopic`, `ServiceIndexer`, `fetchRemote*`, `getRedisClient`, `EntityCache`,
339
+ `QueryCache` and `QueryCacheInvalidator`, `getLogger`, `newUUID`, `runMScript`, and the
340
+ Ed25519 helpers `signServiceToken` / `verifyServiceToken`. There is no OpenAI or Stripe client
341
+ export: AI and payments go through their blueprint settings. The authoritative export list is the
342
+ generated `src/common/index.ts` in the service repo; check there before relying on a name.
343
+
344
+ Two helpers live on the API manager, so MScript and arrow functions reach them as `this.*`:
345
+ `this.hashPassword(plain)` (bcrypt, cost 10) and `this.hashCompare(plain, hashed)`. Both return
346
+ Promises; inside a larger expression write `await` yourself, or the design is refused with
347
+ `MSCRIPT_ASYNC_HELPER_NOT_AWAITED`.
348
+
349
+ ### `dbLayer` helpers
350
+
351
+ Always prefer `dbLayer` to `models`. For an object `order` the names are `<operation><Order>`:
352
+
353
+ - writes: `createOrder(data, context?)`, `createBulkOrder(rows, context?)`,
354
+ `updateOrderById(id, data, context?)`, `updateOrderByIdList(ids, data, context?)`,
355
+ `updateOrderByMQuery(data, mQuery, context?, includeArchived?)` (data clause FIRST),
356
+ `deleteOrderById(id, context?)`, `deleteOrderByMQuery(mQuery, context?, includeArchived?)`;
357
+ - reads: `getOrderById`, `getOrderByMQuery`, `getOrderAggById`, `listOrder`, `listOrderByMQuery`
358
+ (alias `getOrderListByMQuery`), `countOrder`, `getOrderStatsByMQuery`, `getIdListOfOrderByField`,
359
+ `getIdPageOfOrder`, plus `getOrderBy<Field>` for each indexed non-key field.
360
+
361
+ Rules worth knowing:
362
+
363
+ - Two families, one suffix apart. `listOrder`, `countOrder`, `getOrderByQuery` and
364
+ `deleteOrderByQuery` take a raw driver `where`; MQuery `$` operators fail there on a SQL service.
365
+ Only the `*ByMQuery` family lowers MQuery. Validation refuses the raw form in library and edge code
366
+ with `LIB_DBLAYER_RAW_WHERE_USES_MQUERY`. `getOrderListByMQuery` takes a bare MQuery and returns an
367
+ array, not `{ rows, count }`. The `*ByQuery` helpers are deprecated.
368
+ - Return shapes: `createOrder` gives `{ success, data?, error?, existing? }`. `updateOrderById`
369
+ gives `{ success, affectedRows, data }` and answers `success: false` (it does not throw) when no live row
370
+ matches. `deleteOrderById` gives the deleted row, or `null`; there is no `success` flag.
371
+ - Stats: `getOrderStatsByMQuery(mQuery, stats?, bucketBy?)`. `stats` is an ARRAY (`'count'`,
372
+ `'sum(f)'`, `'avg(f)'`, `'min(f)'`, `'max(f)'`). Without `bucketBy` the result is a flat object
373
+ keyed `op-field` (`{ count, 'sum-total' }`, an object even for one stat). With `bucketBy` it is an
374
+ array of buckets, each with `count`.
375
+ - `context` is `{ session?, requestId? }`. Pass the caller's so the Kafka events carry the session.
376
+ Omit it for cron and system work.
377
+ - A bulk or by-query update or delete with no selector besides `isActive` (after the soft-delete gate is
378
+ waived) is refused.
379
+ - All delete helpers honour `onDeleteAction` relations; a live `restrict` child refuses the delete
380
+ with 409 `RELATION_RESTRICTED`.
381
+ - Soft delete: with it on, a delete flips `isActive` to false and stamps `_archivedAt`; reads hide
382
+ those rows unless `includeArchived` is true. Without it, a delete removes the row.
383
+ - `dbLayer` skips the Business API permission and validation chain. Use it only on paths that have
384
+ made their own checks (an edge function with explicit authorization, a cron library call). In a
385
+ Business API use the CRUD actions.
386
+
387
+ Every write is followed by, in order: the Elasticsearch index write (awaited for at most
388
+ `ELASTIC_PROJECTION_WAIT_MS`, default 3000; a timeout is logged `ElasticProjectionWaitExpired` and is
389
+ not fatal), the `elastic-index-<codename>_<object>-<verb>` event, and the
390
+ `<SERVICE_CODENAME>-dbevent-<objectname>-<verb>` event (always emitted). A failed index or publish
391
+ never rolls back the database write. Because the db event is published after the index attempt, a
392
+ consumer that re-reads the projection normally sees the new row.
393
+
394
+ Example (library function for a cron API):
395
+
396
+ ```js
397
+ const { listOrder, updateOrderByIdList } = require('dbLayer');
398
+ const { Op } = require('sequelize');
399
+
400
+ module.exports = async function closeStaleOrders(cutoffDate, context) {
401
+ const { rows } = await listOrder({
402
+ where: { status: 'awaiting_payment', createdAt: { [Op.lt]: cutoffDate } },
403
+ limit: 1000,
404
+ });
405
+ if (rows.length === 0) return { closed: 0 };
406
+ const result = await updateOrderByIdList(rows.map(r => r.id), { status: 'canceled' }, context);
407
+ return { closed: result.count };
408
+ };
409
+ ```
410
+
411
+ ## Reading another service's data
412
+
413
+ Never read another service's database. Read its Elasticsearch projection instead; every
414
+ `dbLayer` write keeps it current. Pick by call site:
415
+
416
+ | Where | Use |
417
+ |---|---|
418
+ | Inside a Business API | `InterserviceCallAction` for one typed call, or a static join that copies the field at write time |
419
+ | Inside a BFF data view | the view's `joins[]` |
420
+ | Library, edge or cron code | `ServiceIndexer` or `fetchRemote*` from `common` |
421
+
422
+ Rules for code you write:
423
+
424
+ - `new ServiceIndexer('User', session, requestId)` takes the bare object name; it adds the project
425
+ codename (`<codename>_user`). `session` and `requestId` are only diagnostic context. They do not
426
+ filter the read.
427
+ - Read methods: `getById`, `getOne`, `getByPage(from, size, query?, sort?, cached?)`, `getCount`
428
+ (each takes `{ onError: 'throw' }`), `getAggregations` (degrades to `null`), `rawSearch` (always
429
+ throws), `checkIndexState()` (`present`, `absent`, `unknown`). By default a failure degrades to
430
+ `null`, `[]` or `0`, which is indistinguishable from "nothing there". Pass `onError: 'throw'`, use
431
+ the `*WithStatus` helpers (`{ data, failure }`), or check `checkIndexState()` whenever the degraded
432
+ value is an answer you act on. An unconfigured Elasticsearch client still returns the empty value
433
+ even with `onError: 'throw'`.
434
+ - `fetchRemoteObjectByMQuery(object, query, ctx?)` and the list, stats and `*ByElasticQuery` variants
435
+ take an optional trailing `{ session, requestId }`. The plain helpers collapse "absent" and "not
436
+ answered" into `null` or `[]`.
437
+ - Not tenant-scoped. The index is namespaced by the project codename only and the session is never
438
+ consulted on a read. Add the tenant column to your query yourself, failing closed:
439
+ `convertMQueryToElasticQuery({ companyId: ctx.session?.tenantId ?? null, status: 'active' })`.
440
+ - Not access-scoped. Every data object is projected, whatever its `dataObjectAccess`, and the
441
+ owner's Business API gates (such as `ownershipCheck`) do not run. For a read that must honour the
442
+ owner's rules, or must see the database, use an `InterserviceCallAction` to the owner's API.
443
+ Custom reads also add no moderation gate: filter on `moderationStatus` yourself.
444
+ - Rows from `fetchRemote*` never carry a property the project marks `hashed` or `isSecret`.
445
+ - Paginate every list read. Prefer one `getByPage` or aggregation to a loop of `getById`.
446
+ - Cross-service reads are async. Wrap them in a library function and call it through a
447
+ `FunctionCallAction`; do not inline them in MScript.
448
+ - The `projection-index-missing` reason: the owning service never indexed that object, or its
449
+ Elasticsearch bootstrap or startup sync failed. Read the owner's boot log for `[SMART-SYNC]` lines, fix the
450
+ cause, restart the owner (its startup sync rebuilds an absent or empty index). A row inserted
451
+ straight into the database by a script is not projected until the next rebuild. Other reasons:
452
+ `elastic-not-configured`, `invalid-arguments`, `query-conversion-failed`, `result-window`,
453
+ `partial-timeout`, `transport-error`.
454
+ - A cross-service `FetchObjectAction` that cannot be answered logs `RemoteFetchNotAnswered` and
455
+ answers 503 when required (it is fail-closed: a gate behind it does not run).
456
+
457
+ ## Packages
458
+
459
+ Third-party dependencies are `serviceSettings.serviceBasics.packages[]` entries: `manager`,
460
+ `name`, `version` (`"*"` is latest), optional `importAs`, optional `options`.
461
+
462
+ | `manager` | Written into |
463
+ |---|---|
464
+ | `npm` | `package.json` dependencies |
465
+ | `pip` | `pyproject.toml` |
466
+ | `maven` (`name` is `groupId:artifactId`) | `pom.xml` |
467
+ | `nuget` | `.csproj` |
468
+ | `gomod` (`name` is the module path) | `go.mod` |
469
+ | `cargo` | `Cargo.toml` (`options` carry crate features) |
470
+ | `composer` | nowhere (no service language generates it) |
471
+ | `vcpkg`, `conan`, `apt` | the build image, for native libraries a kernel links |
472
+
473
+ Each target takes only the entries whose manager matches its language. An npm package used only by
474
+ a `ts` body is not loaded by a non-nodejs service: declare the target language's equivalent and use
475
+ it from the native body. `importAs` also binds the package into MScript, so an expression can say
476
+ `dayjs(this.createdAt)`; without it the package is only installed and the function imports it itself.
477
+ The nodejs compiler also merges `import` and `require` references in your library modules into
478
+ `package.json`. Older `nodejsPackages[]` items are migrated into `packages[]` (`defaultImportConst`
479
+ becomes `importAs`). For `apt`, `name` matches `^[a-z0-9][a-z0-9+.-]+$` and `version` is `"*"` or
480
+ matches `^[0-9][A-Za-z0-9.+:~-]*$`.
481
+
482
+ A native C or C++ library is a `NativeKernel` on `library.nativeKernels[]`, called as
483
+ `LIB.<name>(...)` and linked into the service binary through FFI. Use it only when a specific
484
+ native library must be linked: it adds a native build dependency, a crash takes the process down, and it
485
+ has no database or session access. Keep calls coarse. Its design is in the kernel pattern of mind-set.
486
+
487
+ ## Names a custom-code author needs
488
+
489
+ The compiler's naming rules (the generated repo itself holds the per-generator detail; read the
490
+ generated files for the exact names before you call them):
491
+
492
+ - Data object: model name is PascalCase of the object name (`userGroup` becomes `UserGroup`).
493
+ Cross-service reference is `<service>:<object>`. Postgres table is the English plural of the object
494
+ name with its camel case kept (`order` becomes `orders`, `person` becomes `people`). MongoDB
495
+ collection is the object name verbatim. Objects generated by higher-order patterns (payment, agent hub)
496
+ start with `sys_`.
497
+ - `dbLayer`: `<operation><ModelName>`, imported from `dbLayer`.
498
+ - REST routes sit behind `/<serviceName>-api/v<version>/`. Standard action with its own resource:
499
+ `/<pluralObject>` and `/<pluralObject>/:<object>Id`. Non-standard action: `/<action><Resource>`.
500
+ An explicit `restSettings.routePath` overrides. Standard actions are get, list, create, update,
501
+ delete; there is no automatic CRUD, only declared APIs.
502
+ - Kafka: DB events `<serviceCodename>-dbevent-<objectname>-<verb>` with
503
+ `serviceCodename = <projectCodename>-<serviceName>-service` and verb `created`, `updated` or
504
+ `deleted`; API events `<projectCodename>-<serviceName>-service-<raisedEvent>`, lowercase, raised
505
+ only when `apiOptions.raiseApiEvent` is true; index events `elastic-index-<codename>_<object>-<verb>`.
506
+ - Elasticsearch index: `<codename>_<objectname>` lowercased.
507
+ - Every service that lets users log in also serves `GET /publickey` (its Ed25519 public key) and
508
+ `POST /rawsearch/:index`. The latter needs a login, and stays off (403) until the operator sets
509
+ `RAWSEARCH_INDEXES`; it accepts only a small query subset (no scripts, aggregations or wildcards).
510
+
511
+ ## Testing custom code
512
+
513
+ A generated service collects `**/__tests__/**/*.test.ts` and `**/*.spec.ts`, and nothing else.
514
+ `__tests__/mytest.ts` and `my.test.js` are silently never run. Keep helper modules out of
515
+ `__tests__/` (the convention is a sibling `__testutils__/`), because jest would collect them as
516
+ empty suites and fail the run. Test an edge function by passing a mocked context
517
+ (`{ body, headers, session }`) and asserting the return value; mock `dbLayer` so unit tests never
518
+ touch a database, and mock `ServiceIndexer` for cross-service reads. The aliases `common`,
519
+ `dbLayer` and `models` resolve under jest. More in [testing.md](testing.md).