@jskit-ai/agent-docs 0.1.282 → 0.1.284

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (151) hide show
  1. package/guide/agent/app-extras/assistant.md +14 -2
  2. package/guide/agent/app-extras/realtime.md +38 -1
  3. package/guide/agent/app-setup/authentication.md +64 -14
  4. package/guide/agent/app-setup/console.md +8 -0
  5. package/guide/agent/app-setup/database-layer.md +23 -103
  6. package/guide/agent/app-setup/existing-application-migration.md +105 -72
  7. package/guide/agent/app-setup/multi-homing.md +94 -32
  8. package/guide/agent/app-setup/quickstart.md +1 -1
  9. package/guide/agent/app-setup/run-an-accounts-hub.md +1 -1
  10. package/guide/agent/app-setup/upgrading-jskit.md +8 -3
  11. package/guide/agent/app-setup/users.md +8 -5
  12. package/guide/agent/framework/application-operations.md +1 -1
  13. package/guide/agent/framework/crud-operations.md +63 -45
  14. package/guide/agent/framework/data-access.md +1031 -0
  15. package/guide/agent/framework/sanity-checks.md +26 -27
  16. package/guide/agent/framework/typed-features.md +114 -247
  17. package/guide/agent/index.md +2 -0
  18. package/package.json +6 -3
  19. package/patterns/feature-package/PATTERN.md +25 -17
  20. package/patterns/feature-package/example/booking-engine/package.json +1 -1
  21. package/patterns/feature-package/example/variations/InvoiceRollupFeature.js +6 -7
  22. package/patterns/feature-package/example/variations/invoiceResource.js +15 -0
  23. package/patterns/feature-package/example/variations/invoiceRollupActions.js +4 -4
  24. package/patterns/feature-package/example/variations/ownedTableRepository.js +27 -0
  25. package/patterns/feature-package/example/variations/types.js +1 -1
  26. package/patterns/minimal-foundation/example/AGENTS.md +4 -0
  27. package/patterns/minimal-foundation/example/package.json +4 -4
  28. package/patterns/minimal-foundation/example/server.js +4 -1
  29. package/patterns/shell-foundation/example/AGENTS.md +4 -0
  30. package/patterns/shell-foundation/example/package.json +5 -5
  31. package/patterns/shell-foundation/example/packages/main/package.json +1 -1
  32. package/patterns/shell-foundation/example/server.js +4 -1
  33. package/reference/autogen/KERNEL_MAP.md +2 -1
  34. package/reference/autogen/PATTERN_INDEX.md +70 -59
  35. package/reference/autogen/README.md +1 -1
  36. package/reference/autogen/packages/agent-docs.md +7 -3
  37. package/reference/autogen/packages/agent-runtime.md +48 -11
  38. package/reference/autogen/packages/architecture-inspector.md +6 -2
  39. package/reference/autogen/packages/assistant-core.md +4 -4
  40. package/reference/autogen/packages/assistant-runtime.md +33 -13
  41. package/reference/autogen/packages/auth-provider-local-core.md +3 -1
  42. package/reference/autogen/packages/auth-provider-local-db-core.md +65 -14
  43. package/reference/autogen/packages/auth-provider-oidc-core.md +5 -1
  44. package/reference/autogen/packages/auth-provider-supabase-core.md +1 -1
  45. package/reference/autogen/packages/connectors-core.md +22 -2
  46. package/reference/autogen/packages/console-core.md +9 -5
  47. package/reference/autogen/packages/crud-core.md +38 -173
  48. package/reference/autogen/packages/database-runtime.md +8 -32
  49. package/reference/autogen/packages/http-runtime.md +3 -3
  50. package/reference/autogen/packages/http-web.md +12 -0
  51. package/reference/autogen/packages/kernel.md +25 -2
  52. package/reference/autogen/packages/oidc-issuer-core.md +4 -0
  53. package/reference/autogen/packages/payments-core.md +29 -6
  54. package/reference/autogen/packages/realtime.md +15 -6
  55. package/reference/autogen/packages/resource-core.md +30 -5
  56. package/reference/autogen/packages/resource-crud-core.md +30 -34
  57. package/reference/autogen/packages/resource-runtime.md +198 -0
  58. package/reference/autogen/packages/rewarded-core.md +16 -10
  59. package/reference/autogen/packages/shell-web.md +1 -0
  60. package/reference/autogen/packages/users-core.md +26 -14
  61. package/reference/autogen/packages/workspaces-core.md +17 -10
  62. package/reference/autogen/tooling/config-eslint.md +69 -0
  63. package/reference/autogen/tooling/jskit-catalog.md +40 -1
  64. package/reference/autogen/tooling/jskit-release.md +3 -0
  65. package/skills/jskit/SKILL.md +4 -1
  66. package/skills/jskit/references/app-operations.md +1 -1
  67. package/skills/jskit/references/crud-operations.md +62 -44
  68. package/skills/jskit/references/data-access.md +1026 -0
  69. package/skills/jskit/references/database-layer.md +411 -0
  70. package/skills/jskit/references/existing-application-migration.md +105 -72
  71. package/skills/jskit/references/heuristics/INDEX.md +6 -4
  72. package/skills/jskit/references/heuristics/crud-authoring.md +22 -14
  73. package/skills/jskit/references/heuristics/data-access.md +19 -0
  74. package/skills/jskit/references/heuristics/filters.md +6 -6
  75. package/skills/jskit/references/heuristics/row-policies.md +40 -16
  76. package/skills/jskit/references/heuristics/server-search.md +30 -42
  77. package/skills/jskit/references/heuristics/table-resource-fields.md +78 -0
  78. package/skills/jskit/references/heuristics/typed-features.md +3 -3
  79. package/skills/jskit/references/multi-homing.md +94 -32
  80. package/skills/jskit/references/pattern-index.md +70 -59
  81. package/skills/jskit/references/patterns/app/minimal-foundation/example/AGENTS.md +4 -0
  82. package/skills/jskit/references/patterns/app/minimal-foundation/example/package.json +4 -4
  83. package/skills/jskit/references/patterns/app/minimal-foundation/example/server.js +4 -1
  84. package/skills/jskit/references/patterns/app/shell-foundation/example/AGENTS.md +4 -0
  85. package/skills/jskit/references/patterns/app/shell-foundation/example/package.json +5 -5
  86. package/skills/jskit/references/patterns/app/shell-foundation/example/packages/main/package.json +1 -1
  87. package/skills/jskit/references/patterns/app/shell-foundation/example/server.js +4 -1
  88. package/skills/jskit/references/patterns/assistant/assistant-surface/PATTERN.md +5 -2
  89. package/skills/jskit/references/patterns/auth/accounts-hub/example/package.json +3 -2
  90. package/skills/jskit/references/patterns/auth/auth-surface/PATTERN.md +15 -0
  91. package/skills/jskit/references/patterns/auth/oidc-sign-in/example/package.json +2 -2
  92. package/skills/jskit/references/patterns/auth/supabase-auth/example/package.json +1 -1
  93. package/skills/jskit/references/patterns/connectors/calendar-cli/example/package.json +4 -4
  94. package/skills/jskit/references/patterns/crud/crud-contract/PATTERN.md +133 -0
  95. package/skills/jskit/references/patterns/crud/crud-contract/example/bookResource.js +74 -0
  96. package/skills/jskit/references/patterns/crud/{json-api-resource-package → crud-feature-package}/PATTERN.md +69 -37
  97. package/skills/jskit/references/patterns/crud/{json-api-resource-package → crud-feature-package}/example/packages/books/package.json +5 -4
  98. package/skills/jskit/references/patterns/crud/crud-feature-package/example/packages/books/src/server/BooksFeature.js +13 -0
  99. package/skills/jskit/references/patterns/crud/crud-feature-package/example/packages/books/src/shared/bookResource.js +74 -0
  100. package/skills/jskit/references/patterns/crud/crud-feature-package/example/packages/books/src/shared/index.js +1 -0
  101. package/skills/jskit/references/patterns/crud/crud-screen-set/PATTERN.md +26 -8
  102. package/skills/jskit/references/patterns/crud/crud-screen-set/example/books/BookEditPage.vue +2 -2
  103. package/skills/jskit/references/patterns/crud/crud-screen-set/example/books/BookListPage.vue +2 -2
  104. package/skills/jskit/references/patterns/crud/crud-screen-set/example/books/BookNewPage.vue +2 -2
  105. package/skills/jskit/references/patterns/crud/crud-screen-set/example/books/BookViewPage.vue +3 -3
  106. package/skills/jskit/references/patterns/database/mysql-application/PATTERN.md +9 -4
  107. package/skills/jskit/references/patterns/database/mysql-application/example/package.json +2 -1
  108. package/skills/jskit/references/patterns/database/mysql-application/example/scripts/prepare-database.js +8 -0
  109. package/skills/jskit/references/patterns/database/postgres-application/PATTERN.md +8 -4
  110. package/skills/jskit/references/patterns/database/postgres-application/example/package.json +2 -1
  111. package/skills/jskit/references/patterns/database/postgres-application/example/scripts/prepare-database.js +8 -0
  112. package/skills/jskit/references/patterns/database/table-resource/PATTERN.md +117 -0
  113. package/skills/jskit/references/patterns/database/table-resource/example/orderResource.js +25 -0
  114. package/skills/jskit/references/patterns/realtime/realtime-application/PATTERN.md +7 -0
  115. package/skills/jskit/references/patterns/realtime/realtime-application/example/package.json +2 -2
  116. package/skills/jskit/references/patterns/rewards/google-rewarded/PATTERN.md +1 -1
  117. package/skills/jskit/references/patterns/server/feature-package/PATTERN.md +25 -17
  118. package/skills/jskit/references/patterns/server/feature-package/example/booking-engine/package.json +1 -1
  119. package/skills/jskit/references/patterns/server/feature-package/example/variations/InvoiceRollupFeature.js +6 -7
  120. package/skills/jskit/references/patterns/server/feature-package/example/variations/invoiceResource.js +15 -0
  121. package/skills/jskit/references/patterns/server/feature-package/example/variations/invoiceRollupActions.js +4 -4
  122. package/skills/jskit/references/patterns/server/feature-package/example/variations/ownedTableRepository.js +27 -0
  123. package/skills/jskit/references/patterns/server/feature-package/example/variations/types.js +1 -1
  124. package/skills/jskit/references/patterns/users/user-administration-server/PATTERN.md +59 -10
  125. package/skills/jskit/references/patterns/users/user-administration-server/example/packages/users/package.json +5 -4
  126. package/skills/jskit/references/patterns/users/user-administration-server/example/packages/users/src/server/UsersFeature.js +7 -5
  127. package/skills/jskit/references/patterns/users/user-administration-server/example/packages/users/src/shared/index.js +1 -1
  128. package/skills/jskit/references/patterns/users/user-administration-server/example/packages/users/src/shared/userCrud.js +21 -0
  129. package/skills/jskit/references/patterns/users/user-administration-server/example/packages/users-workspace/package.json +11 -5
  130. package/skills/jskit/references/patterns/users/user-administration-server/example/packages/users-workspace/src/server/UsersWorkspaceFeature.js +14 -5
  131. package/skills/jskit/references/patterns/users/user-administration-server/example/packages/users-workspace/src/server/workspaceMembersOnly.js +52 -0
  132. package/skills/jskit/references/patterns/users/user-administration-server/example/packages/users-workspace/src/shared/index.js +1 -1
  133. package/skills/jskit/references/patterns/users/user-administration-server/example/packages/users-workspace/src/shared/userCrud.js +21 -0
  134. package/skills/jskit/references/patterns/workspaces/workspace-server/PATTERN.md +9 -9
  135. package/skills/jskit/references/patterns/workspaces/workspace-surfaces/PATTERN.md +16 -0
  136. package/skills/jskit/references/patterns/workspaces/workspace-surfaces/example/src/pages/home/workspaces/index.vue +2 -0
  137. package/skills/jskit/references/sanity-checks.md +27 -28
  138. package/skills/jskit/references/typed-features.md +114 -247
  139. package/patterns/feature-package/example/variations/customKnexRepository.js +0 -40
  140. package/reference/autogen/packages/json-rest-api-core.md +0 -66
  141. package/skills/jskit/references/heuristics/crud-repository-mapping.md +0 -140
  142. package/skills/jskit/references/patterns/crud/json-api-resource-package/example/packages/books/src/server/BooksFeature.js +0 -13
  143. package/skills/jskit/references/patterns/crud/json-api-resource-package/example/packages/books/src/shared/bookResource.js +0 -83
  144. package/skills/jskit/references/patterns/crud/json-api-resource-package/example/packages/books/src/shared/index.js +0 -1
  145. package/skills/jskit/references/patterns/crud/resource-contract/PATTERN.md +0 -155
  146. package/skills/jskit/references/patterns/crud/resource-contract/example/bookResource.js +0 -87
  147. package/skills/jskit/references/patterns/server/feature-package/example/variations/customKnexRepository.js +0 -40
  148. package/skills/jskit/references/patterns/users/user-administration-server/example/packages/users/src/shared/userResource.js +0 -69
  149. package/skills/jskit/references/patterns/users/user-administration-server/example/packages/users-workspace/src/shared/userResource.js +0 -56
  150. /package/skills/jskit/references/patterns/crud/{json-api-resource-package → crud-feature-package}/example/migrations/20260815000000_books.cjs +0 -0
  151. /package/skills/jskit/references/patterns/crud/{json-api-resource-package → crud-feature-package}/example/packages/books/src/server/types.js +0 -0
@@ -0,0 +1,1026 @@
1
+ <!-- Generated from `packages/agent-docs/site/guide/framework/data-access.md`. -->
2
+
3
+ # Data access
4
+
5
+ Read this before writing code that reads or writes a database table, with or
6
+ without screens. [Typed features](typed-features.md) covers how
7
+ the code around it is shaped; this page covers how it reaches the data.
8
+
9
+ ## The resource is the only door
10
+
11
+ Every table has exactly one **table resource**. Only the feature that owns the
12
+ table reads or writes it, through the resource's handle. Ownership, the row
13
+ policy, validation, ownership stamping, audits and change events run only on
14
+ that path. Knex reaches the table past all of them, and putting it in a file
15
+ named "repository" protects nothing.
16
+
17
+ Bad: a repository raised a version counter on a record id it was handed, with
18
+ no workspace predicate, so one workspace could change another workspace's
19
+ booking.
20
+
21
+ Good: the owner's repository calls `bookings.patch(id, { version }, options)`,
22
+ which changes the row only when the caller's scope owns it.
23
+
24
+ ## The handle's calls
25
+
26
+ One table, one table resource, one owner feature. The owner reaches its table
27
+ through its handle's calls, and every call takes the call's `options` last:
28
+
29
+ | Call | Does |
30
+ | --- | --- |
31
+ | `query({ filters, sort, page, include }, options)` | one page: `{ records, page }` |
32
+ | `queryAll({ filters, sort, include }, options)` | every matching row, for jobs, id lists and one parent's rows |
33
+ | `get(id, params, options)` | one record |
34
+ | `aggregate({ measures, groupBy, filters, sort }, options)` | totals, grouped by fields or date buckets |
35
+ | `create(data, options)`, `findOrCreate({ id, ...data }, options)` | one record; a keyed one, created once |
36
+ | `patch(id, changes, options)`, `delete(id, options)` | one record |
37
+ | `createMany(rows, options)` | many records, or an upsert with `onConflict` |
38
+ | `patchMany(rows, options)` | many records, each with its own changes |
39
+ | `patchWhere(filters, changes, options)`, `deleteWhere(filters, options)` | every matching row |
40
+ | `lock(id, txOptions)`, `lockMany(ids, txOptions)`, `lockWhere(filters, txOptions)` | rows locked until the transaction ends |
41
+ | `lockOrCreate({ id, ...initial }, options, work)` | a keyed record locked, created first when missing, for `work` |
42
+ | `claim({ filters, sort, take }, txOptions)` | the next rows no other worker holds |
43
+ | `withTransaction(options, work)` | one transaction; `work` gets its `txOptions` |
44
+ | `scopedQuery(params, options)` | a read-only builder for a report ([Layer 5](#layer-5-reports-the-resource-cannot-express)) |
45
+
46
+ ```js
47
+ // How many invoices of each status the workspace issued each month, and their total.
48
+ function totalsByMonth(options) {
49
+ return invoices.aggregate({
50
+ groupBy: ["status", { field: "issuedAt", bucket: "month", as: "month" }],
51
+ measures: { invoices: { count: "*" }, amount: { sum: "amount" } },
52
+ sort: ["month"]
53
+ }, options);
54
+ }
55
+ ```
56
+
57
+ ## Words
58
+
59
+ | Word | Meaning |
60
+ | --- | --- |
61
+ | table resource | The one description of one table: fields and their columns, ownership, relationships, search and lock version. `defineTableResource()`, shared with the client. |
62
+ | owner feature | The feature whose `owns` lists the table resource. It is the only feature that reads or writes the table. |
63
+ | handle | What the owner gets for each resource it owns. Every method applies the scope of the call's `options.context`. |
64
+ | repository | The owner's module that holds every query, built on its handles. Its methods have domain names. A plain CRUD feature's repository is its table's handle itself. |
65
+ | service | Orchestration. It calls repositories (its own, or another feature's through that feature's capability: an owner API, or a CRUD feature's `repository`) and other features' services. It never builds a query (no Knex, no `scopedQuery`) and never reaches another feature's table any other way. |
66
+ | owner API | The capability through which a table's owner lets other features read and write its records, such as `users.profiles`. |
67
+ | CRUD contract | The optional HTTP and screen surface on a table resource: `defineCrudContract()`. |
68
+ | CRUD feature | The feature `defineCrudFeature()` builds from a CRUD contract: its repository, service, actions, routes and screens. |
69
+ | ownership | Who owns a row, required in every table resource: `public`, `workspace`, `user`, `workspace_user`, or `workspace_or_shared` (a workspace's own rows, or rows shared with every workspace when the owner is empty). |
70
+ | row policy | A server-only rule narrowing which rows a context sees, applied to every statement, joins, includes and locks included. |
71
+ | scoped query | The read-only Knex builder `scopedQuery()` returns, with ownership and the row policy already applied. |
72
+ | system scope | Why a feature works across every workspace and user, declared with `systemScope`. |
73
+ | system context | The context of that work, from `systemContext(reason)`; `forWorkspace`, `forUser` and `forWorkspaceUser` narrow it. |
74
+ | `txOptions` | What `withTransaction(options, work)` hands `work`: only the call's context and its transaction, `{ context, trx }`, passed to every call inside. |
75
+ | record | What a handle answers: `{ id, ...fields }`, relationships as ids, included records nested. |
76
+
77
+ ## Layer 1: the table resource
78
+
79
+ Every table your app owns has one table resource. The feature that owns the
80
+ table lists it in `owns`, and only that feature's setup receives its handle.
81
+ Every read and write goes through the handle. On every statement the handle
82
+ adds the ownership predicate and the row policy, validates what you write and
83
+ stamps the workspace or user on new rows. A table whose schema stores
84
+ `createdAt` and `updatedAt` as `dateTime` fields gets them stamped by every
85
+ write: both on a create (`createMany` too), `updatedAt` on every change
86
+ (`patch`, `patchMany`, `patchWhere`, an upsert merge, which keeps the stored
87
+ `createdAt`). The stamps replace what any other caller sends, and a CRUD
88
+ contract offers them as output only; only a system context (an import, a
89
+ migration, a seed) sets its own. Pass the call's `options` (its context and
90
+ transaction) to every handle call; a call without a context throws instead of
91
+ guessing.
92
+
93
+ ```js
94
+ // shared/receivalResource.js: the table's facts, shared with the client
95
+ const receivalResource = defineTableResource({
96
+ name: "receivals",
97
+ tableName: "receivals",
98
+ ownership: "workspace",
99
+ schema: {
100
+ status: { type: "string", required: true, search: true },
101
+ arrivedAt: { type: "dateTime", nullable: true }
102
+ }
103
+ });
104
+
105
+ // server/ReceivalsFeature.js: the owner
106
+ const ReceivalsFeature = defineFeature({
107
+ id: "receivals",
108
+ owns: [receivalResource, { resource: receivalNoteResource, rowPolicy: authorOrManager }],
109
+ provides: { receivals: "receivals" },
110
+ setup: ({ resources }) => ({ receivals: createReceivalsService(createReceivalsRepository(resources)) })
111
+ });
112
+
113
+ /**
114
+ * server/receivalsRepository.js: every query, by domain name.
115
+ * @param {{ receivals: import("@jskit-ai/resource-runtime/server/types").ResourceHandle }} handles
116
+ */
117
+ function createReceivalsRepository({ receivals }) {
118
+ /**
119
+ * One page of the open receivals.
120
+ * @param {{ size: number, after?: string }} page
121
+ * @param {import("@jskit-ai/resource-runtime/server/types").ResourceCallOptions} options
122
+ */
123
+ function listOpen(page, options) {
124
+ return receivals.query({ filters: { status: "open" }, page }, options);
125
+ }
126
+
127
+ /**
128
+ * @param {string} id
129
+ * @param {string} arrivedAt
130
+ * @param {import("@jskit-ai/resource-runtime/server/types").ResourceCallOptions} options
131
+ */
132
+ function markArrived(id, arrivedAt, options) {
133
+ return receivals.patch(id, { status: "arrived", arrivedAt }, options);
134
+ }
135
+
136
+ return Object.freeze({
137
+ withTransaction: receivals.withTransaction,
138
+ listOpen,
139
+ markArrived
140
+ });
141
+ }
142
+ ```
143
+
144
+ A package whose features own tables or declare `systemScope` lists
145
+ `runtime.resources` in its descriptor's `capabilities.requires`, or boot
146
+ refuses it; the feature never requires it itself, since it receives its
147
+ handles.
148
+
149
+ Reads take `(params, options)` and writes `(data, options)`: `get(id, {},
150
+ options)`. A call's settings ride in its options, added on the call they are
151
+ for and spread from the options the method received: `{ ...options, missing:
152
+ "skip" }`, `{ ...options, returning: "id" }`. A method passes on only the
153
+ options it received, never a setting it added for one of its own calls, and
154
+ another table gets only `{ context, trx }`: `passedOn(options)` from
155
+ `@jskit-ai/resource-runtime/server`. An owner API, called with another
156
+ feature's options, hands each of its calls `passedOn(options)` plus the
157
+ settings that call needs, never the caller's options whole. `withTransaction`
158
+ hands its work only the context and the transaction, and so does a handle to
159
+ its hooks and its own nested calls.
160
+
161
+ `txOptions` is `{ context, trx }` and nothing else: every other key of the
162
+ method's `options`, the application's own included (a flag that skips an
163
+ email, a `limit`, a per-request cache), is gone inside the work. Read what the
164
+ work needs from `options` itself, never from `txOptions`. A method that wraps
165
+ a write on its own table in `withTransaction` passes that write `{ ...options,
166
+ ...txOptions }`, or the caller's `returning`, `missing` or `format` never
167
+ reaches it and the method answers something its caller did not ask for.
168
+ Another table's call still gets `txOptions`:
169
+
170
+ ```js
171
+ const addLines = (rows, options) => lines.withTransaction(options, async (txOptions) => {
172
+ const added = await lines.createMany(rows, { ...options, ...txOptions });
173
+ await orders.patch(rows[0].orderId, { linesChangedAt: new Date() }, txOptions);
174
+ return added;
175
+ });
176
+ ```
177
+
178
+ A filter key is one of the resource's `search: true` fields or a
179
+ `searchSchema` entry. `id` is every handle's own filter (a resource never
180
+ declares an `id` search entry) and takes one id or a list; an empty list
181
+ matches no row. Another list of values needs an entry with `filterOperator:
182
+ "in"`. A `null` value matches the rows where the field is null: `filters: {
183
+ archivedAt: null }`.
184
+
185
+ A filter on another table's field (`searchSchema: { supplierName: { type:
186
+ "string", actualField: "suppliers.name" } }`) needs that field declared
187
+ `indexed: true` on its own table resource, with a database index on its column,
188
+ even when the join is on the other table's id; `own()` refuses it at startup
189
+ otherwise, naming both.
190
+
191
+ A table keyed by a `uuid` or `string` primary key takes its ids as text; an
192
+ `owns` entry's `normalizeId` is only for an id written some other way.
193
+
194
+ A read answers every field unless `fields` names the ones it needs (a
195
+ belongs-to field by its own key, such as `customerId`, or its relationship);
196
+ the locks (`lock`, `lockMany`, `lockWhere` and `claim`) take it in their
197
+ options, `{ ...txOptions, fields }`.
198
+ Each belongs-to field a read answers costs a check that its related record is
199
+ visible, so a hot path names its fields. A collection relation with
200
+ `hydrateOnList: false` or `hydrateOnView: false` loads its rows on that kind of
201
+ read only when the read includes it.
202
+
203
+ A `hidden: true` field (an ownership id, a bearer token, a storage key) never
204
+ reaches a request: no CRUD route or JSON:API document (`format: "jsonapi"`)
205
+ carries it, even when `fields` names it. The owner's own code reads one by
206
+ naming it in a handle read's `fields`, `get(id, { fields: ["publicToken"] },
207
+ options)` or `{ ...txOptions, fields }` for a lock, which costs one more
208
+ statement per 1,000 records, scoped as the read is; a read without `fields`
209
+ never answers it. It may be an id, string, integer, number or boolean, and
210
+ naming any other kind refuses the read. The repository a CRUD feature gives
211
+ other features, and its own actions, refuses a read naming a hidden field, and
212
+ any `scopedQuery()` (its builder carries every column), and every method it
213
+ holds, `decorateRepository`'s included, answers without the hidden fields of the
214
+ tables its owner owns, unless the call runs in a system context. So the owner
215
+ checks a token in a method `decorateRepository` adds, with the handle it
216
+ receives, and answers only the outcome. Never unhide a field to read it, and
217
+ never read it with `scopedQuery`.
218
+
219
+ `create` and `patch` answer the written record by default, read back through
220
+ the row policy. Server code that does not use it passes `returning: "id"` or
221
+ `"none"`, which skips that read; `delete` answers the id. One rule covers a
222
+ record that is missing or outside the caller's scope: `get`, `patch`, `delete`
223
+ and `lock` throw json-rest-api's not-found error (a 404), and with
224
+ `missing: "skip"` they answer `null` instead, without failing the caller's
225
+ transaction. A row keyed by an id the code computes (a fingerprint, a
226
+ conversation id) is read or created once with `findOrCreate({ id,
227
+ ...attributes }, options)`, outside the transaction that then locks it: inside
228
+ one, losing the race to create it fails the whole transaction. Compute such an
229
+ id from the owner too (the workspace or user id in the fingerprint): an id
230
+ another scope's row holds answers a missing record. That answer still says the
231
+ id is taken, so `findOrCreate` and `lockOrCreate` take only ids the server
232
+ computes or generates, never an id a request supplies.
233
+
234
+ A write that the caller's row policy then hides fails, because its read-back
235
+ finds nothing. The `create` or `patch` answers json-rest-api's 404
236
+ (`REST_API_RESOURCE`, "Resource not found") and its transaction rolls back. A
237
+ staff member who sees only the bookings they run cannot book a colleague this
238
+ way. Decide which way the rule goes. Either the policy lets the writer see
239
+ what they may write, or the action refuses the write before it runs, with a
240
+ 403 or 422 that says why.
241
+
242
+ You never write `where workspace_id = ?`, so you cannot forget it. The row
243
+ policy and write hooks go in the `owns` entry on the server, never in the
244
+ shared definition, which ships to the browser. Another feature's contribution
245
+ to a row policy can only narrow it. A row policy adds one predicate correlated
246
+ with the row being read, on indexed columns, and reads the owner's other
247
+ tables through `ownTable(name)`, a read-only subquery, never by table name. It
248
+ never loads everything the subject was ever linked to into an `IN` list, and
249
+ never calls another owner's check once per item: it runs on every read, write
250
+ read-back and lookup, so every request pays what it costs. A policy is told
251
+ who the call runs for as `caller`: `{ kind: "user", userId }` for a signed-in
252
+ request, `{ kind: "anonymous" }` for one by no one, and `{ kind: "system" }` or
253
+ `{ kind: "job" }` (a context `forWorkspace` and its kin derived), each with its
254
+ `reason`, for background work. A job names a user only as `actorId`, the
255
+ person it was derived for (below), so never read a missing actor as an
256
+ anonymous request.
257
+
258
+ ```js
259
+ // good: one predicate on the row being read
260
+ function noticeVisibilityPolicy({ query, caller, column, value, ownTable }) {
261
+ if (caller.kind !== "user") {
262
+ return caller.kind !== "anonymous";
263
+ }
264
+ query.where(column("recipientId"), value("recipientId", caller.userId)).whereExists(
265
+ ownTable("sources").select(1).whereColumn("sources.id", column("sourceId")).where("sources.status", "published")
266
+ );
267
+ return true;
268
+ }
269
+
270
+ // bad: every source the recipient was ever notified about, checked one by one
271
+ async function noticeVisibilityPolicy({ query, caller, context, column, value }) {
272
+ const readable = [];
273
+ for (const sourceId of await noticeHistory.listSourceIds(caller.userId)) {
274
+ if (await sources.canRead(sourceId, context)) readable.push(sourceId);
275
+ }
276
+ query.whereIn(column("sourceId"), readable.map((id) => value("sourceId", id)));
277
+ return true;
278
+ }
279
+ ```
280
+
281
+ A rule that depends on the caller's role reads `context.permissions` with
282
+ `hasPermission(context.permissions, "bookings.view-all")` from
283
+ `@jskit-ai/kernel/shared/support`, never with `includes()`. A role may hold
284
+ `*` (an owner does) or `bookings.*`, and `includes()` misses both, so the
285
+ owner would see only the rows a narrower rule leaves.
286
+
287
+ ```js
288
+ import { hasPermission } from "@jskit-ai/kernel/shared/support";
289
+
290
+ // good: staff see the bookings they run; bookings.view-all, the owner's * included, sees them all
291
+ function staffSeeTheirOwnBookings({ query, caller, context, column, value }) {
292
+ if (caller.kind !== "user") {
293
+ return caller.kind !== "anonymous";
294
+ }
295
+ if (hasPermission(context.permissions, "bookings.view-all")) {
296
+ return true;
297
+ }
298
+ query.where(column("staffUserId"), value("staffUserId", caller.userId));
299
+ return true;
300
+ }
301
+
302
+ // bad: an owner holding "*" sees only the bookings they run
303
+ if (context.permissions.includes("bookings.view-all")) return true;
304
+ ```
305
+
306
+ A policy trusts `caller`. In a workspace request it also trusts what the
307
+ server resolved from the route's workspace slug and the caller's membership
308
+ before any handle ran:
309
+ - `context.workspace.id`;
310
+ - `context.membership.roleSid`;
311
+ - `context.permissions`, the role's permissions in that workspace.
312
+
313
+ On a surface whose access policy sets `requireWorkspaceMembership: false`, a
314
+ caller who is not a member has `context.membership` null and
315
+ `context.permissions` empty. Never decide on a value from the request's input.
316
+
317
+ A row policy hides a row everywhere, also as the target of a visible record's
318
+ link: an order whose supplier the reader cannot see reads `supplierId: null`.
319
+ A form that saves the order back sends that null, and the write keeps the
320
+ stored link: a null for a link whose target exists but is hidden from the
321
+ writer counts as unchanged. Send that null, never the hidden id: a write naming
322
+ a target the writer cannot see is refused as a missing target, since accepting
323
+ the right hidden id would tell the writer which record is linked. The link is
324
+ still unreadable to that reader, so a row policy says who may see a row at all;
325
+ narrowing what one screen lists is a filter on that screen's read.
326
+
327
+ ```js
328
+ // bad: a row policy to list only the suppliers a user added; every order whose
329
+ // supplier someone else added now reads supplierId: null for that user
330
+ owns: [{ resource: supplierResource, rowPolicy: addedByActor }]
331
+
332
+ // good: the screen's read filters; orders keep their suppliers
333
+ function listAddedByMe(params, options) {
334
+ return suppliers.query({ ...params, filters: { ...params.filters, addedBy: resolveActorId(options.context) } }, options);
335
+ }
336
+ ```
337
+
338
+ ## Layer 2: the CRUD surface
339
+
340
+ When people list, view, create, edit or delete the rows over HTTP, add a CRUD
341
+ contract to the same table resource. It says which operations exist and which
342
+ fields each one takes and returns; `defineCrudFeature` builds the standard
343
+ repository, service, actions, routes and screens from it. A table without
344
+ screens skips this layer and still uses Layer 1. A table never gets a second
345
+ resource for its CRUD. [CRUD operations](crud-operations.md)
346
+ covers this layer in depth.
347
+
348
+ ```js
349
+ const receivalCrud = defineCrudContract(receivalResource, {
350
+ namespace: "receivals",
351
+ apiAccess: "authenticated",
352
+ operations: ["list", "view", "create", "patch"],
353
+ fields: {
354
+ status: { output: "required", create: "required", patch: "optional" },
355
+ arrivedAt: { output: "optional" }
356
+ }
357
+ });
358
+ ```
359
+
360
+ A contract shows and accepts only the fields it names, and so does an included
361
+ record: it shows its id, its relation's `labelKey`, and only what the including
362
+ contract lists, a record it links to only when the list names that
363
+ relationship, at any depth. Another table's contract never shapes it, since
364
+ that contract may be served to another surface or behind another permission:
365
+
366
+ ```js
367
+ fields: {
368
+ ownerId: { output: { required: false, includeFields: ["displayName"] } },
369
+ // ?include=pet,pet.breed shows the pet's name and its breed's name
370
+ petId: { output: { required: false, includeFields: ["name", { breed: ["name"] }] } }
371
+ }
372
+ ```
373
+ A `decorateService` that also needs the record it wrote (for an audit or a
374
+ history row) reads it from its own answer, `simplifyJsonApiResult(answer)`
375
+ from `@jskit-ai/http-runtime/shared`, never with a second read; a relationship
376
+ whose field key is not its name plus `Id` passes
377
+ `{ lookupFieldMap: { owner: "ownerUserId" } }`.
378
+ A second surface on the same table (a customer's view beside the staff's) is
379
+ a second CRUD contract on the same resource, never a second resource, with
380
+ its own namespace. One plain feature owns the table and provides its handle.
381
+ Each surface is a `defineCrudFeature` that serves its contract through that
382
+ handle with `ownerRepository`. A CRUD feature that owns the table provides
383
+ `{ repository, crud, service }` rather than a handle, so it cannot be another
384
+ one's `ownerRepository`.
385
+
386
+ ```js
387
+ const ServicesFeature = defineFeature({
388
+ id: "app.services",
389
+ owns: [serviceResource],
390
+ provides: { services: "app.services" },
391
+ setup: ({ resources }) => ({ services: resources.services })
392
+ });
393
+ const ServiceAdminFeature = defineCrudFeature({
394
+ crud: serviceAdminCrud,
395
+ surface: "admin",
396
+ ownerRepository: "app.services",
397
+ scope: WORKSPACE_CRUD_SCOPE,
398
+ relativePath: "/services"
399
+ });
400
+ const ServiceCatalogFeature = defineCrudFeature({
401
+ crud: serviceCatalogCrud,
402
+ surface: "app",
403
+ ownerRepository: "app.services",
404
+ scope: WORKSPACE_CRUD_SCOPE,
405
+ relativePath: "/service-catalog"
406
+ });
407
+ ```
408
+
409
+ `WORKSPACE_CRUD_SCOPE` is from
410
+ `@jskit-ai/workspaces-core/server/support/workspaceCrud`; the two contracts'
411
+ namespaces differ because `crud.contracts` refuses two contracts under one
412
+ namespace. A field
413
+ fragment several resources spread into their schemas carries only the field:
414
+ each contract that shows it states its operations in its own `fields`.
415
+
416
+ ## Layer 3: many rows at once
417
+
418
+ When one action creates, changes or deletes many rows, use the handle's
419
+ many-row operations: `createMany`, `patchMany`, `patchWhere` and `deleteWhere`.
420
+ Every filter takes `id`, one id or a list, so `patchWhere({ id: ids },
421
+ changes, options)` and `deleteWhere({ id: ids }, options)` change a known
422
+ list. Each checks every row against the resource first, reads which of the
423
+ rows the caller may touch, then writes each chunk with one statement, in one
424
+ transaction. Audits and change events run once for the batch, so the
425
+ statements are the same for 10 rows as for 200. `patchWhere` and
426
+ `deleteWhere` check the scope and filters on each row once they hold its lock,
427
+ so a row gone meanwhile, or changed so that it no longer matches, is skipped,
428
+ and they take no `missing`. That makes them a compare-and-set on their filters
429
+ on every database: of two writers whose filters both matched a row (open to
430
+ booked), only one writes it, and the other's answer says so
431
+ (`returning: "ids"` lists only the rows it wrote). The one that waited for the
432
+ other's lock leaves the row out; SQLite gets there by failing the second writer
433
+ with `SQLITE_BUSY`, and MariaDB with `innodb_snapshot_isolation` fails the
434
+ waiter with errno 1020 ("Record has changed since last read"), which a write
435
+ that owns its transaction runs again. For one row,
436
+ `patch(id, changes, { ...options, onlyIf: { status: "open" } })` does the same.
437
+ A filter on another table (a search field whose `actualField` reaches a
438
+ related table) is refused on MySQL and MariaDB (`RestApiValidationError`, rule
439
+ `selection_cross_table`), even when it matches no row. A search entry's own
440
+ `applyFilter` that reads another table through a subquery is not detected, and
441
+ is no compare-and-set there either. Filter on the row's own columns instead, for example
442
+ on the link after locking its targets with `lockMany` in the same transaction
443
+ (Layer 4), or move the value onto the row itself for `onlyIf`, or use a
444
+ `claim`. A many-row write answers
445
+ nothing unless `returning` asks: `"ids"`, `"records"`, or `"count"`, which
446
+ reads nothing back.
447
+
448
+ Bad: a batch intake patched each record through the resource: 705 statements
449
+ for 10 records, 11,915 for 200. Written with `patchMany`, it sends the same
450
+ number at both sizes.
451
+
452
+ An upsert (`createMany(rows, { ...options, onConflict })`) on an owned table
453
+ names the ownership field among its conflict keys, or it throws: otherwise it
454
+ could update or hide another workspace's row. `merge` is refused under a row
455
+ policy; use `ignore`, then `patchMany`. A table with no owners and no row
456
+ policy, keyed by a natural id such as an email, may key the upsert on the id
457
+ alone: `onConflict: { keys: ["id"], action: "merge" }`, with an id on every
458
+ row, creates the record or replaces the fields given in one statement. Of two
459
+ upserts at once the later stays, so a counter that must count both is created
460
+ plainly and its duplicate retried instead. Inserting and catching the
461
+ duplicate works only outside a transaction: inside one, the failed insert
462
+ fails the whole transaction, so use an upsert there.
463
+
464
+ A transaction whose insert loses a race for a unique key (another request with
465
+ the same idempotency key committed first) runs once more:
466
+ `retryLostInsertRace(options, () => repository.withTransaction(options, work))`
467
+ from `@jskit-ai/database-runtime/shared`. The database reports the duplicate
468
+ only once the winner has committed, so the second run's lookup by the key
469
+ finds the winner's row. A second duplicate is final, and inside a caller's
470
+ transaction nothing runs again: the caller's transaction has failed, and the
471
+ caller's own retry runs the whole of it. Never write this retry by hand.
472
+
473
+ ```js
474
+ // A repeated payment request answers the first one's record.
475
+ const recordPayment = (input, options) => retryLostInsertRace(options, () => payments.withTransaction(options, async (txOptions) => {
476
+ const [paid] = await payments.queryAll({ filters: { requestKey: input.requestKey } }, txOptions);
477
+ return paid ?? payments.create(input, txOptions);
478
+ }));
479
+ ```
480
+
481
+ ## Layer 4: locks
482
+
483
+ When two requests could change the same rows at once (allocating stock,
484
+ numbering documents, moving items into a batch), lock the rows first, inside
485
+ the transaction: `lock(id, txOptions)`, `lockMany(ids, txOptions)` or
486
+ `lockWhere(filters, txOptions)`. The handle locks only rows the caller may
487
+ see, in id order, so nobody locks another workspace's rows: a missing or
488
+ hidden id throws not-found, as for `get`, or with `missing: "skip"` is left
489
+ out. A check-then-write that must not race (is the drier empty? then load it)
490
+ locks the record first, so another lock or write of it waits until the
491
+ transaction ends. Locks cover existing rows only: number with a unique index
492
+ or a counter row taken with `lock(id)`. Every lock needs a transaction; SQLite
493
+ runs one transaction at a time, so there is no row lock to take. Take locks in
494
+ one order on every path, a parent before its children.
495
+
496
+ On MySQL and MariaDB a locking read locks every row it examines, so two rules
497
+ apply there. Lock ids taken from your own scoped reads, never a list straight
498
+ from a request: every id listed is locked, a hidden one too, before it is left
499
+ out. And `lockWhere` needs an index that starts with the ownership columns (on
500
+ a table with no owners, with a column the filters compare); without one it
501
+ refuses with `RestApiBulkWriteUnsupportedError` naming the index to add in a
502
+ migration.
503
+
504
+ A worker taking the next job uses `claim({ filters, sort, take }, txOptions)`,
505
+ which skips rows another worker holds. A record that serializes work on a key
506
+ and may not exist yet (an account's balance, a connection to a provider), on a
507
+ table with no owners, takes `lockOrCreate({ id, ...initial }, options, work)`:
508
+ it locks the record, creating it from the row first when it is missing, and
509
+ runs `work(record, txOptions)` in the same transaction, so failed work leaves
510
+ no record behind. On its own it runs work once, so work may call a provider:
511
+ two first uses of one id at once can collide on the insert, and the loser runs
512
+ again before its work starts.
513
+
514
+ A record that a person edits from a form it read earlier carries a version
515
+ number: the table resource names it with `lockVersionField`, an integer field
516
+ over a non-null integer column with `defaultTo: 0`.
517
+ `patch(id, changes, { ...options, lockVersion })` then locks the row, applies
518
+ `changes` only while it still holds that version, and advances it. A record
519
+ whose version moved on rejects with a 409 coded
520
+ `crud.document.lockVersionConflict` ("This record changed. Reload and try
521
+ again.") and nothing is written; a missing or hidden record follows the
522
+ missing-row rule. The changes leave the version field out. Every update
523
+ advances the version, a plain `patch`, `patchMany`, `patchWhere` or upsert
524
+ merge too, so a form that read version 3 and saves after a background job's
525
+ plain patch of the same record (now version 4) gets the 409 and reloads, even
526
+ though the job changed other fields. `patch(id, changes, { ...options, onlyIf:
527
+ { status: "open" } })` is the same check on stored values, answering `null`
528
+ when they differ.
529
+
530
+ ## Layer 5: reports the resource cannot express
531
+
532
+ Some reads are reports, not records: totals by status and month, the newest
533
+ row per group, distinct active days. Try `aggregate` (counts, sums, buckets by
534
+ day, week, month or year) and an include with `strategy: "window"` (newest per
535
+ group) first. A window include is declared on the table resource's
536
+ relationship, never in the query, and read with `include: ["snapshots"]`:
537
+
538
+ ```js
539
+ relationships: {
540
+ snapshots: { type: "hasMany", target: "snapshots", foreignKey: "booking_id", include: { strategy: "window", limit: 1, orderBy: ["-revision"] } }
541
+ }
542
+ ```
543
+
544
+ When neither can express it, the owner's repository builds it from
545
+ `scopedQuery()`, the table's builder with ownership and the row policy already
546
+ applied:
547
+
548
+ ```js
549
+ // Knex: report - the days each member was active, counted per member
550
+ async function activeDays(memberIds, options) {
551
+ const { query } = await visits.scopedQuery({}, options);
552
+ return query
553
+ .select({ memberId: visits.column("memberId") })
554
+ .countDistinct({ days: visits.raw("DATE(??)", [visits.column("visitedAt")]) })
555
+ .whereIn(visits.column("memberId"), memberIds)
556
+ .groupBy(visits.column("memberId"));
557
+ }
558
+ ```
559
+
560
+ - Each report is one named method in its owner's repository file
561
+ (`*Repository.js`, `repository.js`, or a file under `repositories/`;
562
+ `jskit/knex-reason` refuses `scopedQuery()` anywhere else), under
563
+ `// Knex: <kind> - <reason>`. The kinds:
564
+ - `totals`: sums or counts `aggregate` cannot group;
565
+ - `report`: rows shaped across tables;
566
+ - `newest-per-group`: one row per group, such as each customer's latest
567
+ visit;
568
+ - `subquery`: a scoped set another query joins;
569
+ - `hot-path`: a measured path the handle makes too slow, with its JSKIT
570
+ ticket (`// Knex: hot-path #123 - <reason>`);
571
+ - `runtime-tables`: tables people define while the app runs, which no table
572
+ resource in the code describes.
573
+ - `column()` names columns; `raw` takes one expression as a string literal,
574
+ with its values as bindings.
575
+ - The builder only reads, and only its own table is scoped. A `.join()` of a
576
+ table by name, or a table named inside `raw()`, reads that table's rows past
577
+ its owner and row policy. Nothing at run time stops it: `jskit/no-direct-knex`
578
+ and `jskit/sql-literal` flag it, and `expectScoped` catches it only when the
579
+ test seeds another owner's rows the join matches. Join another table only as
580
+ that table's scoped subquery.
581
+ - When a report needs another owner's rows side by side, that owner adds a
582
+ method answering its scoped rows as a subquery with its own column names,
583
+ and the report joins it in one statement. The method answers
584
+ `{ query, column }` from its `scopedQuery({ alias }, options)`, with
585
+ `column(field)` naming its columns under that alias, wrapped, since a
586
+ builder is thenable and an async function would run it. The report shapes
587
+ that builder (select, group), names it with `.as()` and joins it, so every
588
+ owner's scope and row policy stay applied. The report lives with the main
589
+ table's owner, or in a feature that requires both. A join source carries
590
+ every column, hidden ones too, so an owner answers one only to a feature
591
+ that may read them.
592
+
593
+ ```js
594
+ // billing's owner API: its scoped invoices as a join source, with their column names
595
+ async function invoiceSource(alias, options) {
596
+ const { query } = await invoices.scopedQuery({ alias }, options);
597
+ return { query, column: (field) => invoices.column(field, alias) };
598
+ }
599
+
600
+ // Knex: report - open tasks and billed totals per project, joined on billing's scoped invoices
601
+ async function projectSummary(options) {
602
+ const source = await billing.invoiceSource("i", options);
603
+ const billed = source.query
604
+ .select({ projectId: source.column("projectId") })
605
+ .sum({ billed: source.column("amount") })
606
+ .groupBy(source.column("projectId"))
607
+ .as("b");
608
+ const { query } = await tasks.scopedQuery({ alias: "t" }, options);
609
+ const projectId = tasks.column("projectId", "t");
610
+ return query
611
+ .leftJoin(billed, "b.projectId", projectId)
612
+ .select({ projectId })
613
+ .count({ open: tasks.column("id", "t") })
614
+ .max({ billed: "b.billed" })
615
+ .groupBy(projectId);
616
+ }
617
+ ```
618
+
619
+ - When each owner can answer on its own (keys in, grouped totals out), the
620
+ report is a feature of its own that calls each owner's method and merges the
621
+ answers in memory: one statement per owner, never a call per row.
622
+ - Give each report a scope test and a flat-query test (see [Tests](#tests)).
623
+
624
+ ## Reads shaped for the screen
625
+
626
+ - A list reads one page with `query`; it needs `page.size`, at most the
627
+ resource's `queryMaxLimit`. A scrolling list passes the previous page's
628
+ `page.next` as `page.after`; numbered pages, or restoring a known page, pass
629
+ `page.number` (from 1) and answer `page: { size, number, hasMore, next }`.
630
+ `page.number` reads with OFFSET, which slows as the rows grow, so anything
631
+ that grows pages with `page.after`. Each page is one statement, with no
632
+ count: never walk pages to reach one.
633
+ - A screen list never uses `queryAll`.
634
+ - Related rows come by id list (`filters: { id: ids }`, or an `in` filter on
635
+ another field) or `include`, grouped in memory. Never a query per row, however
636
+ the loop is spelled (`for`, `map` with `Promise.all`, a helper called per
637
+ item).
638
+ - `queryAll` reads every matching row, in cursor pages that skip and repeat
639
+ none. Use it for id lists, one parent's rows and jobs: a read its callers
640
+ treat as complete, which a `query` would silently cut at one page. In a
641
+ request it stops past the request row limit (10,000 rows) with a 400 coded
642
+ `RESOURCE_ROW_LIMIT`.
643
+
644
+ ```js
645
+ // good: one page of invoices, then all of that page's lines in one read
646
+ const { records, page: next } = await invoices.query({ sort: ["-issuedAt"], page }, options);
647
+ const lines = await invoiceLines.queryAll({ filters: { invoiceIds: records.map((invoice) => invoice.id) } }, options);
648
+ const linesByInvoice = Map.groupBy(lines, (line) => line.invoiceId);
649
+
650
+ // bad: one more read, and its row policy, for every invoice
651
+ for (const invoice of records) {
652
+ invoice.lines = await invoiceLines.queryAll({ filters: { invoiceIds: [invoice.id] } }, options);
653
+ }
654
+
655
+ // good: a delivery job reads every active subscriber
656
+ const subscribers = await subscriptions.queryAll({ filters: { newsletterId, status: "active" } }, options);
657
+
658
+ // bad: the first page only, whatever its size
659
+ const { records: firstPage } = await subscriptions.query({ filters: { newsletterId, status: "active" }, page: { size: 20 } }, options);
660
+ ```
661
+
662
+ Bad: a list read 100-row pages in a loop for each of 40 related sets.
663
+
664
+ ## Background work
665
+
666
+ Work over the whole population runs as a background job, never inside one
667
+ person's request. Its feature declares why with `systemScope`, and its setup
668
+ receives `systemContext(reason)`, which names its reason at every call:
669
+
670
+ ```js
671
+ const SettlementFeature = defineFeature({
672
+ id: "leagues.settlement",
673
+ owns: [cohortResource, memberResource],
674
+ systemScope: "settle finished weekly leagues for every user",
675
+ setup: ({ resources, systemContext }) => ({ settlement: createSettlement({ resources, systemContext }) })
676
+ });
677
+ ```
678
+
679
+ - Prefer `systemContext(reason).forWorkspace(id)`, `.forUser(id)` or
680
+ `.forWorkspaceUser(workspaceId, userId)`: each workspace or user is then one
681
+ unit of work, scoped as usual, in a transaction of its own. An event it
682
+ publishes carries that workspace's or user's scope, as a request there would,
683
+ so it reaches that audience, never everyone. An action it runs with
684
+ `actions.execute({ actionId, input, context })` keeps the context's kind:
685
+ inside it, a handle call still reads as the system context or job.
686
+ - Work a job does for one person (a request it carries out later) passes
687
+ `{ actorId }` last: `forWorkspace(id, { actorId: requestedBy })`. It is for
688
+ attribution only: audit fields read it with `resolveAuditActorId(context)`
689
+ (the signed-in user, else the person a job acts for) and the row policy as
690
+ `caller.actorId`. It never changes which rows the job reaches and signs no
691
+ one in: `resolveActorId` and every `authenticated` check still see no one.
692
+ Write audit fields with `resolveAuditActorId`, and decide access with
693
+ `resolveActorId`.
694
+ - A job finds its units with one `aggregate` grouped by the ownership field,
695
+ which a system context may group by (and count distinct) even when the table
696
+ hides it, then runs each unit scoped to it. A request's aggregate still
697
+ refuses a hidden field:
698
+
699
+ ```js
700
+ async function settleDue(options) {
701
+ const due = await cohorts.aggregate({ measures: { cohorts: { count: "id" } }, groupBy: ["workspaceId"], filters: { status: "finished" } }, options);
702
+ for (const { workspaceId } of due) {
703
+ await settleWorkspace({ context: systemContext("settle finished weekly leagues for every user").forWorkspace(workspaceId) });
704
+ }
705
+ }
706
+ ```
707
+ - A job that must learn each row's owner across workspaces names the hidden
708
+ ownership field in its read's `fields`, `queryAll({ filters, fields:
709
+ ["workspaceId", "email"] }, { context })`, as a system context may do through
710
+ another feature's CRUD repository too (see [Layer
711
+ 1](#layer-1-the-table-resource)). Never unhide it for the job.
712
+ - A request uses its own context. The only exception is a console report
713
+ behind a console permission, with bounded output.
714
+ - `jskit check` lists every system scope and every `systemContext` call with
715
+ its reason, read from the string literal at the call: a reason kept in a
716
+ constant reads as none.
717
+ - A system context comes only from `systemScope`, and a context's scope only
718
+ from the request or from `forWorkspace`, `forUser` and `forWorkspaceUser`:
719
+ lint refuses `createSystemContext` and hand-written `scopeValues`, and a
720
+ context naming two workspaces or two users throws.
721
+ - A job that runs on a schedule is a feature whose `boot()` starts a timer and
722
+ whose `shutdown()` stops it; JSKIT ships no scheduler. See
723
+ [typed features, Owned state](typed-features.md#owned-state).
724
+ Every server runs the timer, so a run takes its rows with `claim()`.
725
+
726
+ Bad: a job read every record of every workspace inside a request path, every
727
+ hour.
728
+
729
+ ## Transactions
730
+
731
+ Open a transaction with the repository's `withTransaction(options, work)` and
732
+ pass the `txOptions` it gives `work` to every call inside, spread over the
733
+ method's own `options` for the write the caller's settings are for. Never read the
734
+ transaction out of `options` by hand. A handle refuses every other
735
+ transaction, a raw Knex one or a savepoint, for reads as for writes. Inside an
736
+ open transaction it joins it; otherwise it opens one and runs events and other
737
+ after-effects after commit. Any failed write inside it fails the whole
738
+ transaction.
739
+
740
+ A deadlock victim's transaction (MySQL and MariaDB errno 1213, PostgreSQL
741
+ `40P01`), and one MariaDB fails as "Record has changed since last read" (errno
742
+ 1020, with `innodb_snapshot_isolation` on), runs its whole work again, up to
743
+ three more times, so the work must
744
+ be safe to repeat: an HTTP call or an email goes after commit, or into an
745
+ outbox row written inside it. Work that cannot be repeated passes `{ ...options, retry: false }`, and work that must
746
+ commit before its caller goes on passes `{ ...options, outermost: true }`.
747
+ Code never asks whether it runs in a transaction: `withTransaction` joins the
748
+ caller's, `outermost` refuses one, and `lock`, `lockMany`, `lockWhere` and
749
+ `claim` refuse to run outside one; `lockOrCreate` opens its own (and retries a
750
+ lost insert race only then).
751
+
752
+ When the work throws, the transaction rolls back and the caller gets the
753
+ work's own error, with its status and code, reporting `transactionOutcome:
754
+ "rolledBack"`, so `retryLostInsertRace` (Layer 3) also runs again work that
755
+ turned the duplicate into its own 409. A failed write inside the work, or a failed commit, reaches
756
+ the caller as json-rest-api's `RestApiWriteError` with the transaction's
757
+ outcome. It carries the failure's own `code` and `status` (a driver's
758
+ `ER_CHECKREAD`, a not-found's 404), and `REST_API_WRITE` only when the failure
759
+ has no code. Classify an error by its `code` or `status`,
760
+ never with `instanceof`: a second copy of a package makes a second class.
761
+
762
+ A database error never reaches a client. Its message carries the statement
763
+ and its values, so JSKIT's HTTP handler and the assistant's stream answer one,
764
+ thrown as it is or as a wrapper's cause, with a stand-in
765
+ (`hideDatabaseError` in `@jskit-ai/kernel/server/runtime`) and log it in full:
766
+ 409 `kernel.database.concurrentWrite` when the database refused the transaction for a concurrent one
767
+ (a deadlock or lock wait whose retries ran out, a row changed since the
768
+ transaction's snapshot, a serialization failure), which the client may send
769
+ again, else 500. To give a duplicate key or another database refusal its own
770
+ answer, throw an `AppError` (a `ConflictError`) with the driver's error as its
771
+ `cause`: an `AppError` keeps its answer whatever it wraps.
772
+
773
+ ```js
774
+ const allocate = (input, options) => repository.withTransaction(options, async (txOptions) => {
775
+ const invoices = await repository.lockOpenInvoices(input.invoiceIds, txOptions);
776
+ return repository.recordAllocations(allocationsFor(invoices, input.amount), txOptions);
777
+ });
778
+ ```
779
+
780
+ ## Tables you don't own
781
+
782
+ Ask the owner: its service, or for a JSKIT table its owner API, with the same
783
+ `options`. JSKIT's packages expose their tables this way: users
784
+ (`users.profiles`, `users.settings`), workspaces (`workspaces`,
785
+ `workspaces.memberships`), local sign-in (`auth.local.accounts`,
786
+ `auth.local.sessions`, `auth.local.recovery`) and the console
787
+ (`console.settings`). A realtime audience reaches its users the same way: its
788
+ `audienceUsers({ event, options })` asks the owners and answers user ids, in the
789
+ event's scope: for a customer's or a public surface's event, the workspace the
790
+ request resolved, never one its payload names (see [Realtime](/guide/app-extras/realtime)).
791
+
792
+ An app that had a table resource of its own over a JSKIT table (such as
793
+ `users`) deletes it: its CRUD contract is built on the package's table resource
794
+ (users-core's `userProfileTableResource`) and served through the owner with
795
+ `defineCrudFeature({ crud, ownerRepository: "users.profiles" })` (the
796
+ `users/user-administration-server` pattern). The app's own facts about a
797
+ person (a staff role, a preference) live in the app's own table keyed by the
798
+ user id, owned by the app with ownership `user` or `workspace_user`, never as
799
+ new columns on `users`.
800
+
801
+ A surface over a table you don't own that may show only some of its rows (a
802
+ people picker limited to those who share a workspace with the caller) asks the
803
+ owner for exactly that read, as one owner API method that narrows with one
804
+ predicate over the owner's own tables. It never contributes the rule to the
805
+ owner's row policy, which narrows every read of the table (sign-in's too), and
806
+ never passes an id list of everyone the caller may see: an id list is fine for
807
+ the rows one screen shows, not for a rule over thousands of people. An owner
808
+ without such a method gets one added at the owner; until then the surface uses
809
+ the owner's own listing for that scope, such as
810
+ `workspaces.memberships.listActiveByWorkspaceId(workspaceId, options)`, which
811
+ reads the members with their users in one statement.
812
+
813
+ Bad: an app wrote sign-in sessions and memberships with Knex; the framework's
814
+ own checks and audits never ran.
815
+
816
+ ## Two features that need each other
817
+
818
+ When a parent's owner and its children's owners each need the other (the
819
+ parent's service reads its children, and a child's write must advance the
820
+ parent's version), split what both need into a feature of its own, so
821
+ capabilities point one way:
822
+
823
+ - The parent table's owner keeps the table and provides a small owner API,
824
+ such as `versions.advance(id, txOptions)`, and requires none of the
825
+ children.
826
+ - Each child's owner requires that API and calls it in its own write's
827
+ transaction, from its repository's write method. An `owns` entry's
828
+ `afterWrite` hook cannot do it: it is declared with the table, before any
829
+ capability arrives.
830
+ - The aggregate that reads or changes them together (the parent with its
831
+ children) is a third feature that requires both.
832
+
833
+ A version that moves on with no field changed is the parent's
834
+ `lockVersionField`: `lockMany(ids, txOptions)` locks the parents, and
835
+ `patchMany(ids.map((id) => ({ id })), { ...txOptions, returning: "none" })`
836
+ advances each by one, in one statement, whatever the field is called. An
837
+ advance is an update, so it stamps the parent's `updatedAt` too: the parent as
838
+ a whole changed, and a reader that compares either one sees it. A condition
839
+ named with no value, such as `lockVersion: record.lockVersion` on a field
840
+ called `version`, is refused (`options.lockVersion is undefined`), never run
841
+ without it.
842
+
843
+ ## Check and judge
844
+
845
+ Before calling data work done, apply the
846
+ [sanity checks](sanity-checks.md#query-counts-and-batches) with
847
+ JSKIT's tools: `expectFlatQueries` for statements that grow with rows,
848
+ `expectScoped` for every read and write path, and a look at each `// Knex:`
849
+ method, which `aggregate` or an include might do instead.
850
+
851
+ ## Tests
852
+
853
+ Each data path gets two one-line tests on the real test database, with the
854
+ helpers from `@jskit-ai/resource-runtime/test`:
855
+
856
+ ```js
857
+ import { expectEveryTableOwned, expectFlatQueries, expectScoped, startTestFeatures } from "@jskit-ai/resource-runtime/test";
858
+
859
+ test("allocation stays in its workspace", () => expectScoped({ environment: process.env, handles: [invoices, payments], systemContext, owners: [workspaceA, workspaceB], seed: seedInvoices, run: (owner) => billing.allocate(input, owner) }));
860
+ test("allocation is flat", () => expectFlatQueries({ environment: process.env, knex, seed: seedInvoices, run: (seeded) => billing.allocate(inputFor(seeded), seeded.options) }));
861
+ ```
862
+
863
+ `expectFlatQueries` (from `@jskit-ai/resource-runtime/test`) seeds each size,
864
+ counts the statements the path sends, and fails when a larger size sends more,
865
+ with the counts and the statements repeated most; the later chunks a many-row
866
+ write reports are allowed. `expectScoped` runs the path as owner A before and
867
+ after seeding owner B and fails when the answer changes, so it catches a report
868
+ that sums, counts or joins B's rows, not only one that returns them; it also
869
+ fails when the path changed or removed a row of B's. The path runs twice, so it
870
+ answers the same for the same rows: a write answers what repeats, such as the
871
+ ids it changed, not a fresh id or time.
872
+
873
+ An owner is whatever `seed` and `run` receive. On a workspace path it is a
874
+ signed-in member's options, built as the request pipeline builds them:
875
+ `{ context: { actor: { id: userId }, workspace: { id: workspaceId }, permissions } }`,
876
+ where `permissions` holds the member's role permissions. Seed with a job's
877
+ context, `{ context: systemContext(reason).forWorkspace(workspaceId) }`. A row
878
+ policy sees that context as `caller.kind` `"job"`, never as the member.
879
+
880
+ A data test never skips: without `TEST_DB_NAME` it fails
881
+ (`requireTestDatabase(process.env)`). Give `expectFlatQueries` the test's
882
+ `knex` whenever the features run on it (`startTestFeatures`,
883
+ `createTestResourceRuntime`). Without `knex` it counts through a server the
884
+ test booted (`countRuntimeQueries(work)` from
885
+ `@jskit-ai/database-runtime/server/testDatabases`). A check that measures
886
+ nothing never passes: "The path sent no statement at any size" means pass
887
+ `knex`, or the path never reached the database.
888
+
889
+ `createTestResourceRuntime({ knex })` gives a test the resource runtime its
890
+ features own their tables through: `runtime.own(featureId, [resource])`
891
+ answers the handles, and `runtime.declareSystemScope(featureId, reason)` the
892
+ `systemContext` function a test reads every owner's rows with. One more test
893
+ proves every table has an owner:
894
+ `expectEveryTableOwned({ knex, runtime, infrastructureTables, environment })`
895
+ fails, naming them, when the migrated test database holds a table no feature
896
+ owns and no package keeps as infrastructure. JSKIT's packages export their
897
+ infrastructure tables for it: `LOCAL_AUTH_INFRASTRUCTURE_TABLES`
898
+ (`@jskit-ai/auth-provider-local-db-core/server/lib/index`),
899
+ `OIDC_CLIENT_INFRASTRUCTURE_TABLES` (`@jskit-ai/auth-provider-oidc-core/server/lib/index`)
900
+ and `OIDC_ISSUER_INFRASTRUCTURE_TABLES` (`@jskit-ai/oidc-issuer-core/server`).
901
+ `runtime` knows only the features the test started, so start every owner of a
902
+ migrated table, JSKIT's own included, and pass `runtime: resources` from
903
+ `startTestFeatures`. Local sign-in's tables belong to
904
+ `AuthLocalDatabaseBackendProvider`
905
+ (`@jskit-ai/auth-provider-local-db-core/server/providers/AuthLocalDatabaseBackendProvider`,
906
+ given `inputs: { "runtime.database": { knex } }`).
907
+ `LOCAL_AUTH_INFRASTRUCTURE_TABLES` lists only the identities table it keeps
908
+ outside resources.
909
+
910
+ A test of a feature and the owner APIs it requires starts them as the server
911
+ does, without HTTP: `startTestFeatures({ knex, features, requires, inputs,
912
+ createTables })` orders the features by their requires, owns their tables
913
+ through a test resource runtime, runs every boot (so `crud.contracts` is
914
+ sealed) and answers `{ capabilities, resources, stop }`. JSKIT's owners start
915
+ beside yours: `UsersExtensionsProvider` and `UsersIdentityProvider` give
916
+ `users.profiles`, and `WorkspaceRecordsFeature`
917
+ (`@jskit-ai/workspaces-core/test/WorkspaceRecordsFeature`) gives `workspaces`
918
+ and `workspaces.memberships` without the workspace routes. Never import another
919
+ package's repositories to build its owner API by hand.
920
+
921
+ A `defineCrudFeature` also requires `crud.contracts` and `runtime.http`, which
922
+ an application's server supplies. In a test, add `CrudContractsProvider`
923
+ (`@jskit-ai/crud-core/server/crudContracts`) to `features` and give
924
+ `inputs: { "runtime.http": { router: { register() {} } } }`. The routes are
925
+ then registered nowhere, and the test calls the feature's `service` and
926
+ `repository` directly.
927
+
928
+ ```js
929
+ const { capabilities, stop } = await startTestFeatures({
930
+ knex,
931
+ features: [RosterFeature, WorkspaceRecordsFeature, UsersIdentityProvider, UsersExtensionsProvider],
932
+ requires: { roster: "roster" }
933
+ });
934
+ t.after(stop);
935
+ ```
936
+
937
+ A test of the routes themselves boots the application's server and signs
938
+ people in over `app.inject`. Keep one cookie jar per person and send the
939
+ session's `csrfToken` as `csrf-token` with every write. The session answer
940
+ carries no user id, so the test finds each person by email
941
+ (`users.profiles.findByEmail(email)`).
942
+
943
+ ```js
944
+ const app = await createServer({ runtimeEnv: { ...resolveRuntimeEnv(), AUTH_LOCAL_EMAIL_CONFIRMATION: "false" } });
945
+
946
+ async function signUp(email, password) {
947
+ const cookies = new Map();
948
+ const send = async (options) => {
949
+ const cookie = [...cookies].map(([name, value]) => `${name}=${value}`).join("; ");
950
+ const response = await app.inject({ ...options, headers: { ...options.headers, cookie } });
951
+ for (const header of [response.headers["set-cookie"] ?? []].flat()) {
952
+ const [name, ...value] = header.split(";")[0].split("=");
953
+ cookies.set(name.trim(), value.join("="));
954
+ }
955
+ return response;
956
+ };
957
+ const csrf = async () => (await send({ method: "GET", url: "/api/session" })).json().csrfToken;
958
+ await send({ method: "POST", url: "/api/register", headers: { "csrf-token": await csrf() }, payload: { email, password } });
959
+ await send({ method: "POST", url: "/api/login", headers: { "csrf-token": await csrf() }, payload: { email, password } });
960
+ return send;
961
+ }
962
+ ```
963
+
964
+ ## Hard rules
965
+
966
+ These are always wrong, and lint reports them as errors in all code but the
967
+ client's, wherever it sits; tests, migrations and seeds are outside them.
968
+ `eslint-disable` cannot turn them off.
969
+
970
+ ### Lint rules
971
+
972
+ | Rule | Reports | Section |
973
+ | --- | --- | --- |
974
+ | `jskit/no-direct-knex` | a table reached by name with Knex, a `.knex` read or `knex` destructured, a transaction opened by hand | [The resource is the only door](#the-resource-is-the-only-door) |
975
+ | `jskit/no-transaction-reader` | `options.trx`, `trx \|\| knex`, a `trx = knex` default, `trx` destructured from options | [Transactions](#transactions) |
976
+ | `jskit/no-hand-scope` | a hand-written `where` on `workspace_id` or `workspaceId` | [Layer 1: the table resource](#layer-1-the-table-resource) |
977
+ | `jskit/sql-literal` | raw SQL built from strings or templates instead of one literal with bindings | [Layer 5: reports the resource cannot express](#layer-5-reports-the-resource-cannot-express) |
978
+ | `jskit/no-hand-context` | a system context made with `createSystemContext`, a context's `scopeValues` written by hand | [Background work](#background-work) |
979
+ | `jskit/knex-reason` | a `scopedQuery()` report outside a named repository method under `// Knex: <kind> - <reason>` | [Layer 5: reports the resource cannot express](#layer-5-reports-the-resource-cannot-express) |
980
+
981
+ ### Exceptions
982
+
983
+ A module whose job is data access itself (a database runtime, a migration
984
+ runner, test-database setup, an operator's data import or cutover tool, a
985
+ database console that runs the person's own SQL) may opt out of
986
+ `jskit/no-direct-knex`, `jskit/no-transaction-reader`, `jskit/no-hand-scope`
987
+ and `jskit/sql-literal` as a whole, with one line per rule before its first
988
+ line of code:
989
+
990
+ ```js
991
+ // readable-check: allow no-direct-knex - infrastructure: the test databases' snapshot and restore
992
+ ```
993
+
994
+ In a file excused from `jskit/no-direct-knex`, raw SQL runs on Knex itself,
995
+ so `jskit/sql-literal` lets it hold a statement; it is still one literal with
996
+ its values as bindings. Only a module that runs SQL it is given by design (a
997
+ console's query runner) opts out of `jskit/sql-literal` itself, in that one
998
+ header line with its reason; never with a per-line disable. `jskit check`
999
+ lists every such file for review. `jskit/no-hand-context` and
1000
+ `jskit/knex-reason` take no exception.
1001
+
1002
+ A data import or cutover tool (loading another system's records into the app's
1003
+ tables before go-live, say) is the one application module that commonly takes
1004
+ it. Check and judge, for each file:
1005
+
1006
+ - it runs only as an operator's command; nothing a request can reach imports
1007
+ it;
1008
+ - it bypasses ownership and row policies on purpose, so every read and write
1009
+ names its workspace explicitly;
1010
+ - it writes in one transaction per batch, so a failed batch leaves nothing
1011
+ half-loaded;
1012
+ - its statements per row are still measured, as any write path's are: a batch
1013
+ of 200 rows costs about what a batch of 10 does.
1014
+
1015
+ `jskit check` also fails:
1016
+
1017
+ - a table with two table resources;
1018
+ - a system scope or `systemContext` call without a reason;
1019
+ - an infrastructure exception or a feature's system scope that the app's
1020
+ checked-in `jskit-data-access.json` does not list, so each new one is
1021
+ reviewed: `{ "infrastructure": [{ "file", "rule" }], "systemScopes":
1022
+ [{ "feature", "systemScope" }] }`;
1023
+ - a module that requires `runtime.database` or `runtime.resources` without
1024
+ the infrastructure exception: features own their tables with `owns`;
1025
+ - an `eslint-disable` that names a data-access rule, or no rule, outside
1026
+ client code.