@flusys/nestjs-entity-builder 9.1.0-rc.1

package/README.md

@@ -0,0 +1,103 @@
+# @flusys/nestjs-entity-builder
+
+Runtime "backendless" entities: define an entity and its fields through the API and get a **real database table** with per-entity permissions. The package only manages entities (definitions, fields, schema changes, table health); it exposes **no record API**. Every read and write of records goes through a **flow** (visual virtual API): its entity steps (create / update / delete / get / find) are the only way data moves, so logic that must run before a write (checks, computed values, other writes in the same transaction) lives in the flow that performs the write, and a flow reuses another through a **Call flow** step. There are no record hooks or separate operations.
+
+Works on PostgreSQL and MySQL (via `SchemaDialectAdapterService`). Import `EntityBuilderModule` **after** `IAMModule` so permission Actions can be provisioned. Swagger: `entityBuilderSwaggerConfig()` from `@flusys/nestjs-entity-builder/docs` (served at `api/docs/entity-builder`).
+
+## Tenancy and company scoping
+
+- **Multi-tenant** (`databaseMode: 'multi-tenant'`): definitions, runtime tables, flows and flow executions all live in the tenant's database - the tenant is the only isolation the package enforces. Services resolved through `ModuleRef` (flow entity access) find the tenant through the request context: `MultiTenantDataSourceService` falls back to it when there is no request.
+- **Company feature** (`enableCompanyFeature`): changes nothing in the tables. Entity and field definitions, flows, their API keys and flow executions are tenant-wide - every company has the same schema and runs the same flows, and permission codes `entity_builder.entity.<code>.*` are global like other permission actions. A runtime table holds only the system columns (`id`, timestamps, soft delete and audit columns when enabled) plus the fields the designer defines; there is no automatic `company_id`, no company filter and no per-company unique.
+- **Scoping records to a company or branch is the entity designer's choice**: add fields such as `company_id` / `branch_id` (plain field codes, not reserved), then map them in the flow - `user.companyId` / `user.branchId` on insert, and the same value in the filter of every get / find / update / delete / lookup step that must stay inside the caller's company. Public and API-key runs have no signed-in caller, so such a flow takes the company from its input instead. A unique field is unique across the whole table.
+- IAM still applies per company: permission checks (`HAS_PERMISSION`, Check permission steps, entity permissions of flows running as the caller) use the caller's own company and branch.
+
+## Endpoints (all POST)
+
+| Path | Purpose | Permission |
+| --- | --- | --- |
+| `entity-builder/entity-definitions/create-entity` | Create entity + table (+ initial fields) in one DDL transaction. `flowEndpoints` (any of the generic API controller's endpoints: `insert`, `insertMany`, `getById`, `getByIds`, `getAll`, `getByFilter`, `update`, `updateMany`, `bulkUpsert`, `delete`) also saves one ready-made flow per endpoint after the commit - slug `<code-with-dashes>-<endpoint-in-kebab-case>` (e.g. `ticket-get-by-ids`), `jwt` auth, runs as the caller (entity permissions apply). Single endpoints are trigger -> entity step -> respond with its result; `getAll` / `getByFilter` take an optional equality filter per scalar field (`getByFilter` answers the first match or 404); `getByIds` takes `ids` (`in` filter); `insertMany` / `updateMany` / `bulkUpsert` have a `list` body type - the request body is the list of records itself (`[{ ... }, { ... }]`), each item checked against the entity fields (insert: the insert inputs; update many: `id` required, the rest optional; bulk upsert: all optional) - loop over `input` (up to 200) in one transaction and answer the saved records in order (`bulkUpsert` updates an item with an `id`, inserts one without). `flowTexts` (`names` per endpoint, `inputLabels` for `id` / `ids` / `search`, `noMatchMessage`) carries the text written into the flows in the creator's language - English for anything left out. Each flow is saved on its own after the commit: one that fails does not stop the rest or the permission set-up and comes back in the response's `warnings` (`entity_builder.schema.warning.endpoint.flow.failed`, with the cause as a nested message ref), next to a failed permission set-up (`...follow.up.failed`). They are ordinary flows the user can change or delete later. Needs `flow_definition.create` too; a slug already in use is a plan blocker | `entity_builder.entity_definition.create` |
+| `entity-builder/entity-definitions/plan-create-entity` | Dry run of the above: the real `CREATE TABLE` SQL, nothing is created | `...entity_definition.create` |
+| `entity-builder/entity-definitions/plan-change` | Dry run of any change below: exact SQL + reverse SQL, data checks, blockers, warnings, affected flows. Plans count records (never show their values), so planning needs the permission that applies the change | `...entity_definition.update` (`...delete` for `drop_entity` / `purge_field`) |
+| `entity-builder/entity-definitions/apply-change` | Apply `add_field`, `update_field`, `deprecate_field`, `restore_field`, `update_entity`, `repair`. Answers the plan of what ran (`downSql` = the undo statements of what ran) plus `result` (the change's summary, e.g. repair's `fixed`) | `...entity_definition.update` |
+| `entity-builder/entity-definitions/apply-destructive-change` | Apply `purge_field` (drop column) or `drop_entity` (drop table) | `...entity_definition.delete` |
+| `entity-builder/entity-definitions/inspect` | Drift check: entity definition vs the real table | `...entity_definition.read` |
+| `entity-builder/entity-definitions/schema-history` | Every schema change with its SQL, failed attempts included | `...entity_definition.read` |
+| `entity-builder/entity-definitions/{get-all,get/:id,get-by-ids,get-by-filter}` | Metadata read (writes only through the endpoints above) | `...entity_definition.read` |
+| `entity-builder/field-definitions/{get-all,get/:id,get-by-ids,get-by-filter}` | Field metadata read | `...field_definition.read` |
+| `api-flows/:slug` | **Call a flow** (a virtual API). Access is the flow's own auth mode; the answer is what its `respond` node says | per flow: public / API key / any user / a permission |
+| `entity-builder/flows/{insert,update,delete,get-all,get/:id,...}` | Flow CRUD - every save is validated as a whole | `...flow_definition.*` |
+| `entity-builder/flows/validate` | Check a draft without saving (errors block a save, warnings do not) | `...flow_definition.read` |
+| `entity-builder/flows/test-run` | Run a saved flow or an unsaved draft, with a per-node trace | `...flow_definition.test` |
+| `entity-builder/flow-api-keys/{create,list,revoke,delete}` | API keys of a flow (the key is shown once). `create` takes `{ flowId, name, expiresAt? }` (ISO, in the future; no zone = UTC); every key has `status` `active` / `revoked` / `expired`. Revoke and delete work whatever the flow's auth mode is | `...flow_definition.update/read` |
+| `entity-builder/flow-executions/{list,get}` | Run history | `...flow_definition.read` |
+
+`entity_builder.entity.*` is a wildcard grant covering every dynamic entity (seeded by `seed:admin`); it covers only dynamic entities, not the static schema-admin permissions (`entity_builder.entity_definition.*`, `entity_builder.field_definition.*`, `entity_builder.flow_definition.*`).
+
+## Guarantees and limits
+
+- **Every structural change is a plan first.** `plan-change` runs the same code as `apply-change` in a dry run (TypeORM SQL-memory mode), so the SQL you preview is the SQL that runs. Data is checked before anything changes: converting `TEXT` to `INTEGER` is refused while any value is not a whole number (with examples), a unique constraint is refused while duplicates exist, and a required field on a table with records needs a default/backfill value. Unsupported conversions are refused with the reason. MySQL has no `USING` clause, so on MySQL single select <-> multi select and text -> boolean are refused too (add a new field, copy the data, deprecate the old one).
+- Supported: add field (optionally required/unique/indexed, with backfill), change type, required/optional, unique, index, default, validation rules, select options, label; **rename a field** (column renamed, flows that use it are rewritten, including `{{ ... }}` template placeholders); deprecate / restore; **permanently delete** a deprecated field; entity label/description/status/soft-delete/audit; **drop entity**; **repair** drift (missing columns, indexes, unique constraints, NOT NULL).
+- Every change value is type-checked (`FieldChangesDto` / `EntityChangesDto`): `nullable: "false"`, `null` for a flag, an unknown field type or a too long label is a 400 `entity_builder.schema.invalid.change.value`. `update_field` can set `relationTargetEntityId` (RELATION fields only; the target must exist and is locked for the change; converting to RELATION needs one, `...blocker.relation.target.required`).
+- **Optimistic concurrency**: send `expectedUpdatedAt` (the `updatedAt` you loaded - of the field for field actions, of the entity otherwise; select `updatedAt` when reading) with `plan-change` / `apply-change`; a newer one answers 409 `entity_builder.schema.stale.definition`. Every run re-reads the definitions inside its own transaction (under the entity lock when applying) and only trusts those; a field renamed or deprecated meanwhile blocks with `...blocker.definition.changed`.
+- `LONG_TEXT`, `JSON` and `MULTI_SELECT` fields cannot be unique or indexed (`entity_builder.schema.not.indexable`). Index and unique names are `idx_` / `uq_` + table + column cut to the driver limit (63 PostgreSQL, 64 MySQL) + a hash of table and column, so they never collide.
+- Dropping an entity is blocked while relation fields of other entities point at it (`...blocker.drop.relation.in.use`, listing `"entity.field"`).
+- Destructive changes (`purge_field`, `drop_entity`, lossy conversions) need `confirm: true` and, for purge/drop, the code typed back (`confirmCode`). They also refuse while flows still use the field/entity.
+- Each change runs in one locked transaction and is logged in `eb_schema_change_log` with the SQL that ran and the reverse SQL (`downSql`). On PostgreSQL a change that fails rolls back completely. MySQL commits every DDL statement at once (plans warn `...warning.mysql.no.rollback`): a failed change is undone by running the reverse of each statement that had run, newest first; the error carries `undo: { reverted, pendingSql, dataNotReverted }` and, when an undo statement fails, the key `entity_builder.schema.failed.not.reverted` with the pending statements also in the `FAILED` log row's `downSql`. Backfilled values are not undone. Either way a `FAILED` log row keeps the statement that failed and the error is mapped to a human-readable one. A follow-up step that fails after commit (permission cleanup) never fails the change; it is reported as a warning.
+- Changes made outside the entity builder (a column dropped by hand) show up in `inspect` and are fixed by `repair`; extra columns are reported and never dropped automatically.
+- The entity/field metadata `update` endpoints are gone on purpose: every edit goes through `apply-change` so the table, the metadata and the schema cache cannot disagree.
+- Identifiers must match `^[a-z][a-z0-9_]{2,59}$`, not be an SQL reserved word, and the table (`eb_<code>`) must not already exist. Every DDL change takes a per-entity lock (`pg_advisory_xact_lock` / `GET_LOCK`, names over 64 characters hashed) - plus the lock of any relation target it points at - and is written to `eb_schema_change_log`.
+- Every filter/sort key of a flow's **Find records** step and of a `lookup` is checked against the entity's real fields; unknown keys are rejected (never interpolated into SQL). A filter value is a plain value (`=`) or `{ op, value? }` with `op` one of `eq`, `ne`, `gt`, `gte`, `lt`, `lte`, `in` (a list of at most 100 scalars; an empty list matches nothing), `is_null`, `not_null`. Operators map to a fixed SQL table and every value is a bound parameter.
+- **Search** (a Find records step's search text) matches every searchable field by its text form, case-insensitively on both drivers (`CAST ... AS TEXT ILIKE` / `CAST ... AS CHAR LIKE`), so non-text fields can be searchable; `%` and `_` in the search text are literal.
+- Each entity gets the IAM actions `entity_builder.entity.<code>.<create|read|update|delete>` (only when IAM is installed); flows running as the caller check them on every entity step. DECIMAL fields are returned as numbers. `RELATION`/`FILE` are plain uuid columns (`char(36)` on MySQL, which has no uuid type) with no foreign-key constraint.
+- **Tenants never share schemas**: every tenant DataSource gets its own copy of the entities array (`getEntityBuilderEntities()` returns a copy), and runtime schemas are registered per DataSource.
+- `forRootAsync` takes `useFactory`, `useClass`, `useExisting` or a static `config`; with none of them it throws at configuration time. The root entry also exports the flow types, errors and `validateFlow`, and the schema helpers (`schema-columns`, `identifier-rules`, `field-type-conversion`).
+
+## Rules
+
+Conditions and expressions are declarative JSON (`RuleGroup` / `RuleExpression`) evaluated by a closed `switch`; rules never evaluate code. Limits: depth 8, 100 conditions per group, 10 `lookup` expressions per evaluation. References: `input.`, `context.<stepId>.`, `vars.`, `loop.`, `request.`, `user.`; another entity's rows are read with a `lookup` expression.
+
+- **Operands**: a condition compares `field` (a reference) or, when set, `left` (any expression) with `value`: a literal, `{ "$field": "<ref>" }` or `{ "$expr": <expression> }`. Old conditions (`field` + literal / `$field`) are unchanged.
+- **Evaluation**: `AND` / `OR` / `NOT` evaluate their children in order and stop at the first decisive one, so an earlier condition can guard a later lookup or permission check. An unknown comparison or group operator fails the evaluation (`rule_engine.unknown.comparison` / `rule_engine.unknown.group.operator`), also inside `NOT`.
+- **Missing values**: `greater_than` / `less_than` / `greater_or_equal` / `less_or_equal` are false when either side is null or missing. `equals` / `not_equals` (and `in`, `is_any_of`, list `contains`) treat null and missing as the same value and compare a `Date` with date text by instant; everything else compares strictly. Date text without a zone (`2026-04-03`, `2026-04-03T09:00`, `2026-04-03 09:00:00`) is read as UTC, never in the server's timezone; other formats need an explicit zone.
+- **Regex**: `matches` / `not_matches` search the text with RE2 (`re2js`): linear time, no backtracking, so a pattern cannot hang the server (no backreferences or lookaround; `(?i)` for case-insensitive). Patterns are at most 500 characters, input at most 10,000. Counted repeats are unrolled by RE2, so `{n,m}` may count at most 100, nested counts may multiply to at most 100, and all counted repeats together may add at most 300 to the pattern's size (`*`, `+`, `?` are unlimited). An invalid or too complex pattern is refused when a flow is saved and fails with `rule_engine.regex.invalid` / `rule_engine.regex.too.complex` at run time.
+- **Date functions** (UTC, plain `Date` math; values are ISO text, `Date`s or epoch ms, `null` in gives `null` out): `DATE_DIFF(a, b, unit)` = a - b in `second` / `minute` / `hour` / `day`, fractional; `START_OF_DAY(date, offsetMinutes?)` / `END_OF_DAY(date, offsetMinutes?)` (optional offset, e.g. `360` for UTC+6, to take "today" in the caller's zone); `ADD_DURATION(date, amount, unit)`; `NOW()`.
+- **Permission functions**: `HAS_PERMISSION(code, branch?)`, `HAS_ANY_PERMISSION(codes, branch?)`, `HAS_ALL_PERMISSIONS(codes, branch?)` give a boolean for the signed-in user (`codes`: a list or comma/space separated text, at most 50; `*` / `prefix.*` grants match). Use them as a condition's `left` with `is_true` ("user has") / `is_false` ("user doesn't have"), or as a value in Set / Switch. `branch` omitted = the user's current branch, null or empty = company-wide grants only, otherwise that branch id. Company and tenant are never arguments: a flow checks in the caller's own company, and the tenant database is the request's. Codes come from `PERMISSION_RESOLVER` (nestjs-iam: cached, rebuilt from roles and direct grants for a branch not loaded yet); without IAM the shared permission cache is used (only scopes the user has loaded), and with neither the check fails with `PermissionSystemUnavailableException`. A flow fetches each branch's codes once per run. No signed-in user = false (the validator warns about permission checks in `public` / `api_key` flows).
+- **Request functions** (flows only; elsewhere `null` / false): `HEADER(name)` (any letter case; `authorization`, `cookie`, `x-api-key`, `proxy-authorization` are never visible), `QUERY_PARAM(name)`, `IP_IN_RANGE(ip, ranges)` (IPv4/IPv6 addresses and CIDR blocks, list or comma separated text, at most 100; an IPv4-mapped IPv6 caller compares as IPv4; a bad range is refused on save and fails with `rule_engine.invalid.ip.range` at run time).
+- **Text functions**: `LOWER`, `UPPER`, `TRIM`, `LENGTH` (text or list), `SPLIT_PART(text, separator, position?)` (1-based, negative counts from the end, trimmed; e.g. the first `x-forwarded-for` address), `REGEX_EXTRACT(text, pattern, group?)` (RE2; default group 1 when the pattern has one, else the whole match), `REPLACE(text, find, with)` (plain text, every occurrence), `TO_NUMBER(value)` (`null` when not numeric).
+- **Lookup** (`{ "type": "lookup", "entity", "filter", "mode": "exists" | "count" | "first", "field"?, "sort"? }`): reads another entity. `filter` maps a column to an expression (`=`) or `{ op, value? }` (the filter operators above). `exists` gives a boolean, `count` a number, `first` the first row (after `sort`) or its `field`, `null` when none. The read goes through the flow's per-entity `read` check (run-as caller) and transaction; a rule evaluated outside a flow has no reader and fails with `rule_engine.lookup.resolver.unavailable`.
+
+## Flows (virtual APIs)
+
+A flow is an endpoint (`POST /api-flows/<slug>`) whose behaviour is a graph of steps, so a backend can be built without writing backend code. Steps: **Request** (trigger), **Validate** (checks that must pass), **Check permission**, **Code** (sandboxed JavaScript), **Set variables**, **If**, **Switch**, **For each**, **Create / Update / Delete / Get / Find record**, **Save progress**, **Call flow**, **HTTP request**, **Publish event**, **Respond**. Each step's result is `context.<stepId>`; the request body is `input.*`; also `vars.*`, `loop.item` / `loop.index` (inside a loop within a loop the outer loop is `loop.parent.item` / `loop.parent.index`, and so on up), `request.*`, `user.*`. `request` holds `ip`, `method`, `path`, `query` (text values, first of a repeated one, at most 50), `headers` (lower-case names, credential headers removed; read dashed names with `HEADER()`), and ready-made `userAgent`, `origin`, `referer`, `contentType`, `language` (first `accept-language` tag) and `receivedAt`. Text can use `{{ scope.path }}` placeholders (the `template` expression). Steps connect by output ports (`out`, `true`/`false`, `each`/`done`, switch cases, `error`); a step marked "carry on if it fails" stores `context.<id>.error` and follows its `error` port. Inside a transaction (transactional flows and every dry run) such a step runs in a savepoint, so its failure does not abort the flow's transaction.
+
+- **Loops hand data on** by `collect` (evaluated after each pass, reading `loop.*` and the body steps; the values become `context.<loop>.results` next to `count` - collect a Set/Code step to hand on several values), by variables (`vars` is flow-wide and survives passes; initialize before the loop), and by body step results (after the loop they hold the last pass).
+- **Reference timing** (validation warnings): a step's result exists for the steps on a path after it; once a loop is done (`done` branch) its whole body's results exist too (their last pass). The validator warns when a step reads `context.<id>` of a step that has not run yet when it runs (later on the path, on another branch, or later in the same loop body), `loop.*` outside a loop's `each` branch (a foreach's own `collect` counts as inside), `loop.parent` beyond the loops around the step, or `context.<loop>.results` inside that loop (only `count` exists until it is done). Unreachable steps are not checked (they already warn).
+
+- **Validate**: `{ checks: [{ id, rule, message, field? }], mode: 'all' | 'first', statusCode? }`. `all` (default) evaluates every check and rejects with every failure; `first` stops at the first. The answer is `statusCode` (default 400, or 403 when every failed check is a permission check) with `errors: [{ field, message }]` (`field` defaults to the check id); one failure uses its message as the top-level message, several use `flow.checks.failed`. Example (attendance check-out): check 1 `left` = lookup `attendance` with `employee_id = input.employeeId`, `out_time is_null`, `in_time gte START_OF_DAY(NOW())`, mode `exists`, `is_true`; check 2 `left` = `DATE_DIFF(NOW(), <same lookup, mode first, field in_time>, 'hour')` `greater_or_equal` 5.
+- **Check permission** (`permission_check`): `{ codes, match: 'any' | 'all', branch?, onDenied: 'reject' | 'branch', statusCode?, message? }`. `reject` (default) answers `statusCode` (403) with `message` (or `flow.permission.denied`) and stops, otherwise follows `out`; `branch` follows `allowed` / `denied` and never rejects. `branch` left out = the caller's current branch, `null` = company-wide grants, an expression = that branch id. Output `{ allowed, missing }`. A caller who is not signed in is denied (and the validator warns in `public` / `api_key` flows).
+- **Code**: `{ code, timeoutMs? }` runs `code` as the body of a synchronous function in a QuickJS WebAssembly sandbox (`quickjs-emscripten`) on a worker thread (a small pool, at most 4), so a busy code step never blocks the server's event loop; the host also kills a worker that overruns its deadline. It reads a deep-frozen JSON copy of `{ input, vars, context, loop, user: { id, email, name, companyId, branchId, permissions? } }` as the global `ctx` (no request headers; `permissions`, the caller's codes in the current branch, is fetched only when the code mentions permissions). `hasPermission(code)`, `hasAnyPermission(...codes)` and `hasAllPermissions(...codes)` are plain JavaScript inside the sandbox over `ctx.user.permissions` with the server's wildcards and `return`s a JSON value, which becomes `context.<id>`. Nothing of the host is reachable (no `require`, `process`, network, filesystem, timers or database); `console.log` is captured into the trace (`trace[].logs`, 50 lines of 500 characters, sensitive-looking keys masked). Every run gets a fresh runtime with a 32 MB memory limit, a 512 KB stack and an interrupt deadline of `timeoutMs` (default 1000, 10-5000) clamped to the time left in the flow; the output may be at most 1 MB. A throw, time-out or memory overflow is a node error (`flow.error.code.*`), so `onError: continue` and the `error` port work. Code nodes also run in dry runs. Saving a flow that contains a code node needs `entity_builder.flow_definition.code`.
+- **Create / Update / Delete record** write several records in one step through `target`: `one` (default, left out) writes one record; `many` writes one per item of `items` (up to `FLOW_LIMITS.MAX_WRITE_ITEMS` = 1000), and the step's per-record values (`fields`, `id`) read the item as `loop.item` / `loop.index` with a loop around the step as `loop.parent`; `filter` (update / delete) writes every record matching `filter` (same shape as Find), read as `loop.item` (e.g. `stock = loop.item.stock - 1`). A filter that resolves to nothing is refused (`flow.error.write.filter.empty`) instead of changing every record, and more matches than `limit` (1-1000, default 100) fail the step (`flow.error.too.many.matches`) instead of changing some. In `many` mode `id` defaults to the item's own id (the item itself when it is text, else `item.id`). `onNotFound`: `error` (default), `skip`, or for update `insert` (upsert: a missing record, or in `many` mode an item with an empty id, is created with the same field values - needs the entity's `create` permission too). Create takes `children` (up to 10): `{ as, entityCode, parentField, items, fields }` saves child records under every saved record, `parentField` filled with the parent's id; `items` is read in the parent's scope (`loop.item.lines` in `many` mode, `input.lines` otherwise) and child `fields` read the child item as `loop.item` and the parent's scope as `loop.parent`. Results: create one = the record plus one array per `as`; create many = `{ items, count }`; update one = the record (`null` when skipped); update many / filter = `{ items, count, updated, created, skipped }`; delete one = `{ id, deleted }`; delete many / filter = `{ ids, count, skipped }` (an id listed twice is deleted once). A create never takes an `id` field (`generic_record.system.column.readonly`), so it cannot overwrite an existing record; an upsert's created record ignores a mapped `id`. A step that writes several records (`many`, `filter`, or any `children`) is **all-or-nothing on its own**: inside a transactional run it uses the run's transaction, otherwise it opens a transaction for just that step and publishes its record events after that commit. Every record written counts against `FLOW_LIMITS.MAX_WRITES_PER_RUN` (5000, shared with called flows).
+- **Save progress** (`commit`, no settings, port `out`): commits everything the run has written so far, publishes the events held back until then, and starts a new transaction for the rest (the statement timeout is set again). A later failure rolls back only what came after it; the run's result then carries `savedUpTo: { nodeId, name, at }` (the last one that committed) and a failed answer's body `savedUpTo: '<step name>'`, so a caller knows the first part is already saved (make a retry check for it). It commits only in the transaction the run opened itself - it is skipped (trace `skipped`, `reasonKey` `flow.skip.commit.*`) in a test run (nothing is ever saved there), in a flow without the transaction setting (every step already saves on its own) and in a flow called inside its caller's transaction (only the caller may commit that one; a called flow that opened its own transaction commits normally). It cannot be set to carry on if it fails (a save error). Save-time warnings: a Save progress step in a flow without the transaction setting, inside a loop (it saves once per item - fine for batch imports) or in a flow other flows call; and, for any flow, two or more write steps without the transaction setting.
+- **Call flow** (`call_flow`): `{ flowSlug, input }` runs another active flow as part of this run and puts its answer in `context.<id>`: its `respond` body, or the output of its last step. The called flow gets `input` checked against its own input schema (a mismatch rejects with 400), the same caller, request and test mode, and shares the run's step, HTTP-call and time budget. It joins the caller's transaction when there is one (so a dry run rolls back its writes too, and its events wait for the caller's commit; a joined call that fails or answers 4xx/5xx drops the events it queued, so a caller that continues past it with `onError: continue` never announces writes the savepoint rolled back); otherwise a transactional called flow commits on its own. A called flow that answers 4xx/5xx, or rejects (validate, permission), passes that answer on as the caller's; any other failure fails the step with `flow.error.called.flow.failed[.at.node]`, so `onError: continue` and the `error` port work. Access: any flow may call a function (`internal`), which then runs as its caller's run does (`runAs` inherited, its own ignored); a flow running as the system may call any flow; one running as the caller may call only what that caller could call directly - `public` and `jwt` flows, and `permission` flows when the caller holds the permission, never an `api_key` flow. A chain that comes back to a running flow and nesting deeper than 5 levels stop the run. On save the step must name an existing flow the author can see (not the flow itself), and what the runtime would always refuse is refused then too: an `api_key` flow from a flow running as the caller (only a warning from a function, which runs as whoever calls it), a required input (without a default) left unmapped, a chain of calls that comes back to this flow, and a chain nested deeper than `FLOW_LIMITS.MAX_CALL_DEPTH`. Calling an inactive flow, and mapping an input the called flow does not declare (it is dropped), are warnings; a flow another flow calls cannot be deleted or have its URL name changed until those callers are changed.
+- **Body type** (`bodyType`, column `body_type`, default `object`): `object` - the body is one object and the input fields are its keys (`input.<name>`); `list` - the body itself is a list, like `insert-many` (`POST api-flows/<slug>` with `[{...}, {...}]`), the input fields describe each item, every item is checked the same way (errors name the index: `[0].sku`; a body that is not a list is refused with `flow.input.body.not.list`), and the flow reads the list as `input` (a For each over `input`, or `input.0.<name>`). With no fields declared a list body is passed through as-is. A Call flow step sends a list-bodied flow one expression, `inputList`, instead of named `input`s; saving checks the step matches the called flow's body type (`node.flow.input.list.required` / `.unexpected`). The test run accepts a list `input` too.
+- **Input**: the flow declares the request fields (type, required, default, min/max, choices). The body is checked and converted with the same validator entity records use; undeclared keys are dropped; every problem is returned at once (400). An `object` field may declare its own `fields`; an `array` field may declare an `itemType` (any type but `array`) that every item must have - min/max/length/options then apply to each item - and a list of objects (`itemType: 'object'`) declares the `fields` of each item. Nested values are checked the same way (undeclared keys inside them are dropped too), errors name their path (`address.city`, `items[0].qty`), and nesting goes at most `FLOW_LIMITS.MAX_INPUT_DEPTH` (4) levels. Without `fields` / `itemType` the value is only checked to be an object / a list. Generated entity flows declare `ids` as a list of ids; their bulk flows use a `list` body whose items are the entity fields.
+- **Who can call it**: `public`, `api_key` (`x-api-key`, only a SHA-256 hash stored, constant-time compare; a key past its `expiresAt` answers 401 `flow.api.key.expired`; keys are only checked while the auth mode is `api_key`, so they stop working when it changes), `jwt` (any signed-in user), `permission` (`entity_builder.flow.<slug>.execute` by default, provisioned as an IAM action) or `internal` - a **function**: no endpoint of its own, run only by other flows' Call flow steps, for logic several flows repeat. Unknown, inactive and `internal` flows all answer 404 on `POST /api-flows/<slug>`. A per-caller-IP rate limit runs **before** credentials are checked (in memory per server instance, fixed one-minute windows, at most 10,000 tracked callers - the oldest window is dropped beyond that).
+- **Find record** filters accept the `{ op, value }` operators (the value side is an expression). A filter entry is an operator only when it is authored as exactly `{ op, value? }` with a known `op` (any other key, or an unknown `op`, and it is not a condition - so an expression such as `{ type: 'arithmetic', op: 'add', ... }` is a plain value). A value resolved at run time (from input, a variable, a step result) is only ever an operand: an object or list is compared for equality and refused as not one value, never read as an operator, so `{ "op": "not_null" }` sent as input cannot widen a Find, lookup, update or delete. On a Find an entry that resolves to nothing is left out; on an update / delete by filter it fails the step (`flow.error.filter.value.missing`) instead of widening the write.
+- **Checks cannot be carried past**: a failed Validate or Check permission step (like Save progress) always ends the run, whatever its `onError` says.
+- **Runs as**: `caller` (every entity step and lookup re-checks `dynamicEntityPermission(entity, action)` = `entity_builder.entity.<entity>.<action>` for the caller) or `system` (no checks - so saving such a flow needs `entity_builder.flow_definition.system`, and a public system flow is flagged in the warnings). It fails closed: any other value is refused on save and at run time, and a system run touches entities only when it was granted the system (a saved flow, or a test run whose tester holds `flow_definition.system`) - otherwise every entity step and lookup answers 403 `flow.error.run.as.not.allowed`.
+- **Transactions**: a transactional flow's entity writes commit or roll back together; entity events and **Publish event** steps are published only after the commit, in order, and never for a run that rolled back - nor for the rows of a step whose savepoint (`onError: continue`) rolled back. Metadata a transactional run needs (entity and field definitions, called flows) is read on the run's own transaction, so a run never waits on a second pooled connection. HTTP calls cannot be rolled back, so a transactional flow with an HTTP step is flagged in the warnings. In a non-transactional flow each step commits on its own, except that a multi-record write step is always all-or-nothing (see above); make the flow transactional when several steps must succeed or fail together, and add **Save progress** steps where the work so far must be kept even if a later step fails.
+- **Test runs** need what running the flow for real would, for a saved flow as much as a draft and in both modes: `flow_definition.system` for run-as system, `flow_definition.code` for a code node. A `live` test run also needs what makes it real: for a draft the permission to save it (`flow_definition.create`, or `update` when it has a `flowId`), for a `permission` flow its own execute permission. The draft is validated like a saved flow (DTO: run-as, auth mode, time-out, rate limit and retention bounds; then `validateFlow`); the server-owned fields of a loaded flow (`id`, `version`, timestamps) are ignored. Test runs of unsaved drafts are logged without a flow id and pruned together, keeping the newest 200.
+- **Test runs**: `dry_run` (default) executes inside a transaction that is always rolled back and skips HTTP and events; `live` does everything. Test runs return the real error text; live callers get a safe summary. `request: { headers?, query?, ip? }` simulates what a caller would send, on top of the designer's own request, so header / query / IP rules can be tried (credential headers are still removed).
+- **Safety limits**: at most 100 nodes, 5000 steps per run (room for a full 1000-item loop with a few steps per item, and for the scaffolded 200-item bulk flows), 1000 items per loop, 10 HTTP calls, 120 s total; loops cannot nest deeper than 3; a flow cannot loop back on itself (use For each). Steps are data walked by a fixed engine; the only author-written code that runs is a code node's, inside the QuickJS sandbox described above.
+- **HTTP steps** never follow redirects, cap the response at 1 MB, and refuse loopback / private / link-local (cloud metadata) addresses **at connect time** (DNS-rebinding safe), including every IPv6 form that embeds or tunnels to one (IPv4-mapped `::ffff:`, IPv4-translated `::ffff:0:`, IPv4-compatible, NAT64 `64:ff9b::/96` and all of the local-use `64:ff9b:1::/48`, 6to4 `2002:`, and all of Teredo `2001::/32`). The timeout is one deadline for the whole request, response body included. For local development only, set `flows.allowPrivateHttp` in the module config or `ENTITY_BUILDER_FLOW_ALLOW_PRIVATE_HTTP=true`.
+- **Runs are logged** (`eb_flow_execution`) with the redacted input, a trace of every step and the output; the newest N (default 200) are kept.
+- **Schema changes are flow-aware**: renaming an entity field rewrites the flows that use it; dropping an entity or field lists (and for a drop, blocks on) the flows that use it. Tracked: the entity step's `fields` / `filter` / `sort`, a create step's `children[]` (`entityCode`, `fields` keys, `parentField`), lookups, and output paths holding records - `context.<id>.<field>` (get / create / update one), `context.<id>.items.<n>.<field>` (Find, many / filter writes), `context.<id>[.items.<n>].<as>.<n>.<field>` (saved children). References reached through a loop over query results (`loop.item.<field>`) are not tracked.
+- Secrets typed into an HTTP header are stored in the flow definition (there is no credential vault yet); flows are readable only with `flow_definition.read`.
+
+## Messages and localization
+
+Every user-facing message is key based (`config/message-keys.ts`, keys `entity_builder.*`): exceptions and responses carry `messageKey` (+ `messageVariables`), and diagnostics (plan `warnings`/`blockers`/step descriptions/`impact`, drift `findings`, flow validation `errors`/`warnings`, run `trace[].error`, record `errors[]`) are `IMessageRef`s from `@flusys/nestjs-shared`, never text. The CRUD controllers use `entityName: 'entity_builder.<resource>'` so the base controller's `*.success` keys match the constants. The English text and the Bengali/Arabic translations live in the app's seed (`flusysnest/src/persistence/localization/entity-builder.localization.ts`, module `entityBuilder`); the frontend `ENTITY_BUILDER_MESSAGES` carries the same English text. Adding a message means adding the constant and both entries.
+
+## Not in v1
+
+Flows: scheduled/event triggers, a merge (join) node, a credential vault, a shared rate limit across instances. Bulk endpoints (`insert-many`, `update-many`, `bulk-upsert`, `get-by-ids`), a send-email flow node, changing a field between unrelated types (e.g. `INTEGER` to `FILE`): add a new field, copy the data, deprecate the old one.

package/config/entity-builder.constants.d.ts

@@ -0,0 +1,6 @@
+export declare const ENTITY_BUILDER_MODULE_OPTIONS = "ENTITY_BUILDER_MODULE_OPTIONS";
+export declare const ENTITY_CODE_PATTERN: RegExp;
+export declare const ENTITY_CODE_MAX_LENGTH = 63;
+export declare const ENTITY_TABLE_PREFIX = "eb_";
+export declare const ENTITY_FLOW_ENDPOINTS: readonly ["insert", "insertMany", "getById", "getByIds", "getAll", "getByFilter", "update", "updateMany", "bulkUpsert", "delete"];
+export type EntityFlowEndpoint = (typeof ENTITY_FLOW_ENDPOINTS)[number];

package/config/event-actions.d.ts

@@ -0,0 +1,8 @@
+export declare const ENTITY_BUILDER_EVENT_ENTITIES: {
+    readonly ENTITY_DEFINITION: "entity_definition";
+    readonly FIELD_DEFINITION: "field_definition";
+    readonly SCHEMA_CHANGE_LOG: "schema_change_log";
+    readonly FLOW_DEFINITION: "flow_definition";
+    readonly FLOW: "flow";
+};
+export declare const ENTITY_BUILDER_EVENT_MODULE = "entity-builder";

package/config/index.d.ts

@@ -0,0 +1,3 @@
+export * from './entity-builder.constants.js';
+export * from './event-actions.js';
+export * from './message-keys.js';