@collective-intelligence/extensions-sdk 0.3.2 → 0.3.3

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -1,8 +1,10 @@
1
1
  # Knowledge
2
2
 
3
- Declare schemas, store typed records, query them, and read and watch them from a demo app.
3
+ Declare schemas, store typed records, query them, and read and watch them from an external app.
4
4
 
5
- Knowledge stores the business records of a demo, such as Leads and Opportunities. A schema defines the fields that a record type can have, and a record is one instance that matches that schema. Schemas live in a Knowledge domain, a named container that the Organization owns. In the template, the extension package `acme.app.knowledge` declares one domain with a Lead schema and an Opportunity schema, and the demo app reads and writes Lead and Opportunity records through the SDKs.
5
+ Knowledge holds structured business records, such as Leads and Opportunities, and relation fields link records to each other. A schema defines the fields that a record type can have, and a record is one instance that matches that schema. Schemas live in a Knowledge domain, a named group of related schemas that the Organization owns. In the template, the extension package `acme.app.knowledge` declares the Acme CRM domain with a Lead schema and an Opportunity schema, and the template's web app reads and writes Lead and Opportunity records through the SDKs.
6
+
7
+ A domain holds schemas and never records. It can set a default for record versioning that its schemas use when they set none. Access policies target schemas, so a domain does not grant or limit access. The [Data resources](index.md) page compares Knowledge with Datasets and Documents.
6
8
 
7
9
  ```mermaid
8
10
  flowchart LR
@@ -28,7 +30,7 @@ A schema combines ordinary value fields with three special kinds. Each special k
28
30
 
29
31
  ## Declare a schema in a package
30
32
 
31
- A demo declares its schemas in code, inside its extension package. This is the template's Lead schema, shortened:
33
+ A customer solution declares its schemas in code, inside its extension package. This is the template's Lead schema, shortened:
32
34
 
33
35
  ```ts
34
36
  declareKnowledgeEntitySchema({
@@ -66,9 +68,9 @@ declareKnowledgeEntitySchema({
66
68
  })
67
69
  ```
68
70
 
69
- Each declared item needs a `lifecycle`. With `deployMode: "upsert"`, every deploy reconciles the schema with the code, and `init_only` creates the schema once and then leaves it alone. With `readonly: true`, Dashboard administrators cannot edit the definition, so the code stays the only source of truth. Choose `readonly: true` when the demo code owns the schema and you redeploy to change it, and `readonly: false` when a customer administrator should keep editing the schema in the Dashboard after the deploy.
71
+ Each declared item needs a `lifecycle`. With `deployMode: "upsert"`, every deploy reconciles the schema with the code, and `init_only` creates the schema once and then leaves it alone. With `readonly: true`, an Organization admin cannot edit the definition in the Dashboard, so the code stays the only source of truth. Choose `readonly: true` when the package code owns the schema and you redeploy to change it, and `readonly: false` when the customer's Organization admin should keep editing the schema in the Dashboard after the deploy.
70
72
 
71
- The template also has a Lead entity contract in `domain/lead.ts`. That contract types the resource for the demo app (its capabilities, the fields you can filter, sort and search on, and the create and update shapes) and never reaches CI. CI receives only the schema above, so when you add a field, change both files.
73
+ The template also has a Lead entity contract in `domain/lead.ts`. That contract types the resource for the web app (its capabilities, the fields you can filter, sort and search on, and the create and update shapes) and never reaches CI. CI receives only the schema above, so when you add a field, change both files.
72
74
 
73
75
  The CLI puts the schema into an Organization in five steps: `ci package build`, `validate`, `publish`, `promote` and `install --organization <id>`. The [Extensions](../develop/extensions.md) page shows the full commands.
74
76
 
@@ -124,7 +126,7 @@ A file field stores files. Its read value is `{ id, url, filename, mimeType, siz
124
126
 
125
127
  Field filters narrow records with `{ field, operator, value }` and operators such as `eq`, `in`, `exists` and `is_empty`, and text search matches plain text. Semantic search ranks records by embeddings, so it works only on a schema with embeddings enabled, and each result carries its `match.semantic_score`.
126
128
 
127
- The schema controls embeddings. `embedEntities` turns on one combined vector per record, `fieldSemantics["/field"].includeInItemEmbedding` selects the fields that go into that vector, and `fieldSemantics["/field"].embedded` gives a long text field its own passage vectors. When you change these settings through a deploy or an admin edit, CI reprocesses the existing records automatically. You do not need a new import or an Organization reset.
129
+ The schema controls embeddings. `embedEntities` turns on one combined vector per record, `fieldSemantics["/field"].includeInItemEmbedding` selects the fields that go into that vector, and `fieldSemantics["/field"].embedded` gives a long text field its own passage vectors. When you change these settings through a deploy or an edit in the Dashboard, CI reprocesses the existing records automatically. You do not need a new import or an Organization reset.
128
130
 
129
131
  > **Warning:** `embedEntities` alone gives no semantic search over records. Select at least
130
132
  > one field with `includeInItemEmbedding`, or the schema has no record vector
@@ -156,20 +158,16 @@ PostgreSQL holds the source of truth and runs semantic search with pgvector. The
156
158
 
157
159
  ## Documents and Knowledge
158
160
 
159
- Knowledge holds typed records with fields. Documents holds files in collections and folders, with versions and permissions per path. A Document Collection is either Local, where people upload files, or Connected, where a Documents connector keeps it in sync with folders in Microsoft 365 or Google Drive.
160
-
161
- The two areas share storage and processing. A Knowledge file field stores its file through Documents, in an internal collection that Documents browse and search do not show. A reader reaches that file through the Knowledge record, and Advanced Information Retrieval returns it when the reader can read the record. Both areas also use one extraction and embedding pipeline, and CI reuses one processing result when the same bytes appear twice in an Organization.
162
-
163
- A finished sync does not mean that the new files are searchable yet, because indexing runs after the sync. A workflow Wait at level `sync` waits for the import only, while level `indexing` also waits for indexing. Use `indexing` before a demo step that searches the new content. The [Connectors](connectors.md) page compares Knowledge sync with Documents sync.
161
+ Knowledge and Documents share storage and processing. A Knowledge file field stores its file through Documents, in an internal collection that Documents browse and search do not show. A reader reaches that file through the Knowledge record, and Advanced Information Retrieval returns it when the reader can read the record. Both kinds also use one extraction and embedding pipeline, and CI reuses one processing result when the same bytes appear twice in an Organization. The [Documents](documents.md) page covers Document Collections, sync and indexing.
164
162
 
165
163
  ## Read and write from an app
166
164
 
167
- A demo app reaches records in two ways, and the template uses both.
165
+ An external app reaches records in two ways, and the template uses both.
168
166
 
169
167
  | | Browser | Server |
170
168
  | ----------- | ----------------------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------------------------------- |
171
169
  | API | Entity hooks from `defineEntity({ contract, config })` in `@collective-intelligence/external-apps-client-sdk-react/resources` | `ServerClient.records` from `@collective-intelligence/external-apps-server-sdk/effect` |
172
- | Acts as | The signed-in user | The app credential (`CI_CLIENT_ID`, `CI_CLIENT_SECRET`) |
170
+ | Acts as | The signed-in user | A Server Credential, with the Access Grants of the app's server |
173
171
  | Operations | Whatever the entity contract allows: create, get, search, update, delete, upload | `create`, `get`, `search`, `update`, `delete` |
174
172
  | Typical use | Forms, lists and tables | Server routes and the template's Read Port |
175
173
 
@@ -189,21 +187,26 @@ return { ...fields, id: record.id }
189
187
 
190
188
  `records.search` accepts a filter, text search, sort, pagination and `include`. App filtering goes through `POST /application/v1/resources/:schemaId/search`, and relation field pickers use `POST /application/v1/knowledge/autocomplete`. The SDKs do not wrap semantic search; agents use the retrieval tools for that.
191
189
 
192
- > **Warning:** The server reads with the app credential, so that credential itself needs read
193
- > access to Leads. CI accepts a user's token only when the user holds the
194
- > application gate. See [Access](../platform/access.md).
190
+ > **Warning:** Server calls need a Server Credential (`dsc_`), and CI refuses the enrollment
191
+ > client secret on server calls because that secret only signs users in. Create
192
+ > one with `ci apps credentials upsert <app> --name <name> --env-file <file>`.
193
+ > The Server Credential acts with the Access Grants of the app's server, so
194
+ > those grants need read access to Leads. CI accepts a user's token only when
195
+ > the user holds the application gate. See [Access](../platform/access.md).
195
196
 
196
197
  ## Change events
197
198
 
198
- Every write to a record produces a change event, whether the write comes from the demo app, Studio, an agent or a workflow. Database triggers write a change log entry for each changed record, with one entry per record per transaction. A sync that changes nothing writes no entry.
199
+ Every write to a record produces a change event, whether the write comes from an external app, Studio, an agent or a workflow. Database triggers write a change log entry for each changed record, with one entry per record per transaction. A sync that changes nothing writes no entry.
199
200
 
200
201
  An app subscribes to a topic per schema, for example `{ schema: "acme.app.schemas.lead" }`. `useRealtimeTopic(topic, changes => ...)` receives batches of changes that carry record IDs only, and `useEntityChanges(contract)` gives a function that refreshes the entity reads for those IDs. CI checks read access for each change before it sends it. When CI cannot decide whether a reader may see a delete, it sends that reader a `reset` instead of the ID, and the client must then refetch.
201
202
 
202
- To see this in the template, open `/crm` in two browser windows and edit a Lead in one of them as a Sales manager. The other window shows the change without a reload, and it does the same when the change comes from Studio, an agent or the convert-lead workflow.
203
+ To see this in the template, open `/crm` in two browser windows and edit a Lead in one of them as a Sales manager. The other window shows the change without a reload, and it does the same when the change comes from Studio, an agent or a workflow that writes the Lead.
203
204
 
204
205
  ## Related pages
205
206
 
207
+ - [Data resources](index.md)
206
208
  - [Datasets](datasets.md)
209
+ - [Documents](documents.md)
207
210
  - [Connectors](connectors.md)
208
211
  - [Access](../platform/access.md)
209
212
  - [SDKs](../develop/sdks.md)
@@ -2,7 +2,7 @@
2
2
 
3
3
  What a package declares, how the template wires it, how to deploy it with ci deploy, and the rules its code runs under.
4
4
 
5
- A package is a TypeScript module whose default export is `declarePackage({ manifest, components, hostedContributions })`. The `ci` CLI builds that module into a release, which is one immutable published version of the package. An administrator then installs the release into one Organization, and the installation records which release that Organization runs.
5
+ A package is a TypeScript module whose default export is `declarePackage({ manifest, components, hostedContributions })`. The `ci` CLI builds that module into a release, which is one immutable published version of the package. An Organization admin then installs the release into one Organization, and the installation records which release that Organization runs.
6
6
 
7
7
  The template's package declares the Lead and Opportunity schemas, the Sales group tree, the external application of the web app, Sales Access Profiles and grants, the Max Privileges of the application's tokens, the convert-lead workflow, the lead qualifier Agent with its Skill and tool, and the Sales desk Studio app. [SDKs](sdks.md) explains which `@collective-intelligence/*` package each part imports.
8
8
 
@@ -41,7 +41,7 @@ Executable kinds also take an `implementation` class from `@collective-intellige
41
41
 
42
42
  A component refers to another component of the same package through the handle that its `declare*` call returned. For a built-in component or one from another package, pass the FQN string instead, for example `"builtin.core.knowledge.tools.query"`. TypeScript does not check these strings. Install resolves them and rolls the package back when one does not exist, and the SDK publishes no list of built-in FQNs, so the template pins its FQNs in a `bun test` that catches a typo before deploy.
43
43
 
44
- A package that declares Agents or Skills needs an administrator to install it. A package can depend on another package by FQN only, for example `dependencies: [{ fqn: "com.acme.base" }]`, and each install picks the latest eligible release of the dependency by promotion order.
44
+ A package that declares Agents or Skills needs a person to install it; an install with no signed-in user fails. A package can depend on another package by FQN only, for example `dependencies: [{ fqn: "com.acme.base" }]`, and each install picks the latest eligible release of the dependency by promotion order.
45
45
 
46
46
  ## How the template wires the package
47
47
 
@@ -102,13 +102,13 @@ To add a component to the template:
102
102
 
103
103
  Every content component sets `lifecycle: { deployMode, readonly }`.
104
104
 
105
- | Option | Effect |
106
- | ------------------------- | -------------------------------------------------------------------------------------------- |
107
- | `deployMode: "upsert"` | Each deploy brings the definition back in line with the package. |
108
- | `deployMode: "init_only"` | The first deploy creates the definition. Later deploys leave an administrator's edits alone. |
109
- | `readonly: true` | Nobody can edit the definition by hand. Records of a schema stay editable. |
105
+ | Option | Effect |
106
+ | ------------------------- | ------------------------------------------------------------------------------------------------- |
107
+ | `deployMode: "upsert"` | Each deploy brings the definition back in line with the package. |
108
+ | `deployMode: "init_only"` | The first deploy creates the definition. Later deploys leave an Organization admin's edits alone. |
109
+ | `readonly: true` | Nobody can edit the definition by hand. Records of a schema stay editable. |
110
110
 
111
- The template uses `upsert` with `readonly: true` for almost everything. The convert-lead workflow and the Sales reps grant use `init_only` with `readonly: false`, so an administrator can change them after install and an update keeps that change.
111
+ The template uses `upsert` with `readonly: true` for almost everything. The convert-lead workflow and the Sales reps grant use `init_only` with `readonly: false`, so an Organization admin can change them after install and an update keeps that change.
112
112
 
113
113
  ## Package variables
114
114
 
@@ -131,7 +131,7 @@ const vars = packageVariables(variables)
131
131
  export default declarePackage({ components, manifest, variables })
132
132
  ```
133
133
 
134
- A variable has a type of `string`, `number`, `boolean`, `string[]`, `json` or `secret`. A variable without a default is required at deploy, and a `secret` cannot have a default. When a deploy provisions a secret without a value, the install still succeeds, but runtime use fails until an administrator sets it.
134
+ A variable has a type of `string`, `number`, `boolean`, `string[]`, `json` or `secret`. A variable without a default is required at deploy, and a `secret` cannot have a default. When a deploy provisions a secret without a value, the install still succeeds, but runtime use fails until a Tenant admin sets its value in the Organization variables. An Organization admin can read the variables but cannot change them.
135
135
 
136
136
  A handle cannot go in `fqn`, `version`, `kind`, `lifecycle`, a schema projection, or a reference to another component by FQN such as `schemaFqn` or `templateFqn`. The package FQN cannot be a variable either.
137
137
 
@@ -189,12 +189,13 @@ Every call gets an `AbortSignal` in `context.signal` and can have a deadline in
189
189
 
190
190
  ## After install
191
191
 
192
- Some parts of the demo need a person after the first deploy:
192
+ Some parts of a package need a person after the first deploy:
193
193
 
194
- 1. Rotate the client secret. A confidential application has no secret after install. An administrator with `external_apps:manage` runs `ci apps rotate-secret <application-id>` or clicks Rotate secret on the Dashboard External Apps page. CI shows the secret once, and the web app stores it as `CI_CLIENT_SECRET`.
195
- 2. Add users to the groups. The package creates groups empty and grants profiles to groups. In the template, an administrator adds users to the Sales group or its `managers` subgroup.
196
- 3. Set every secret variable that the deploy left without a value.
197
- 4. Enroll the web app with `ci apps enroll --env-file apps/web/.env.local`, which writes the `CI_*` values the server SDK reads.
194
+ 1. Write the app's sign-in values. A package-declared application refuses `ci apps enroll`, so run `ci apps env <application-id-or-fqn> --env-file apps/web/.env.local`, which writes the `CI_*` values the server SDK reads.
195
+ 2. Rotate the sign-in secret. A confidential application has no secret after install. An Organization admin runs `ci apps rotate-secret <application-id>` or clicks Rotate secret on the Dashboard External Apps page. CI shows the secret once, and the web app stores it as `CI_CLIENT_SECRET`. This secret signs users in only.
196
+ 3. Create a Server Credential for the app's own server calls with `ci apps credentials upsert <application-id-or-fqn> --name <name> --env-file <file>`. CI refuses the sign-in secret on server calls. [External apps](external-apps.md) explains both secrets.
197
+ 4. Add users to the groups. The package creates groups empty and grants profiles to groups. In the template, an Organization admin adds users to the Sales group or its `managers` subgroup.
198
+ 5. Set every secret variable that the deploy left without a value.
198
199
 
199
200
  The Dashboard Extensions page lists packages on Installed and Available tabs. Install and uninstall show a spinner until the real outcome, and leaving the page does not cancel the operation. A configured Knowledge connector instance, even a disabled one, or a saved connection blocks uninstall, and the Dashboard shows the error next to the package.
200
201
 
@@ -18,6 +18,26 @@ You need one when the demo has its own UI outside Studio and the Dashboard, or w
18
18
  | CI reaching your data or actions | A Gateway |
19
19
  | Your own logic with your own store | The app API under `/api/app` |
20
20
 
21
+ ## Put it in the app or in CI
22
+
23
+ Before you write a screen, decide where each piece of its data and logic lives. [Platform architecture](../platform/architecture.md) gives the first question: who owns the records. The same question also places rules, access decisions and UI state:
24
+
25
+ | Question | Put it in CI | Keep it in the external app |
26
+ | ------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------- |
27
+ | Do people in Studio, agents or workflows use these records? | A [Knowledge](../data/knowledge.md) schema. The app reads and writes the records through the BFF. | |
28
+ | Does the customer's database stay the source of truth? | Only what CI must read or act on, exposed through a Gateway | The records, in the app's database |
29
+ | Must agents, workflows and the app apply one business rule the same way? | A tool or a workflow in the package. The app starts it and does not write the rule again. | A rule that already runs in the customer's backend, exposed as a Gateway operation |
30
+ | Who may see or change a record? | Capabilities, Knowledge data policies and Access Grants | Only the choice to hide a control |
31
+ | Is it UI state, such as form steps, filters and drafts, or app-only data? | | State machines, query state and the app API store |
32
+
33
+ Records go to Knowledge when more than the app uses them. In the template, the Sales desk Studio app lists Leads and the lead qualifier agent reads them, so Leads live in Knowledge and the web app keeps no Lead table of its own. The template's lead score is a tool that the agent and a Skill call, and the convert-lead workflow is a package declaration that the app only starts through a [Fixed workflow route](external-apps.md#reads-and-writes-through-the-sdk).
34
+
35
+ Access stays in CI even when the app shows the screen. The BFF checks an Access Requirement before it calls CI, and CI then checks each record again. A page can use the same requirement to hide a button, but the server check makes the decision.
36
+
37
+ Data that no CI user, Studio page or agent reads stays in the app. The template's intake form keeps its requests in an in-memory store behind the app API, and you replace that store with a database. When the customer's database must stay the source of truth, the records stay there too. A Gateway then exposes only what CI needs: remote resources that an Organization admin maps onto a Knowledge schema through a [connector](../data/connectors.md), dataset queries, and business operations that agents call as tools. When agents need only a few fields, keep personal data in the app and put only the needed fields into Knowledge.
38
+
39
+ Secrets stay on the app's server. The browser gets no CI token and no client secret, and in the template `NEXT_PUBLIC_APPLICATION_ID` is the only value it sees. The sign-in secret, the session secrets and the Server Credential live in the server's environment. Deploy and management tokens never reach the web app. A secret that code inside CI needs, such as an API key for a third-party service, is a `secret` package variable that a Tenant admin sets in the Organization variables after deploy, as [Extensions](extensions.md) describes. An Organization admin can read the variables but cannot change them.
40
+
21
41
  ## Declare the app
22
42
 
23
43
  A package declares the app as an OAuth client with `declareExternalApplication`. The template's declaration, with a placeholder origin:
@@ -40,19 +60,33 @@ export const acmeApplication = declareExternalApplication({
40
60
 
41
61
  The declaration has no `scopes` field. What a user may do through the app comes from grants that hold the app gate and from the Max Privileges declared for the app's user tokens, both described on [Grants and permissions](../platform/access.md).
42
62
 
43
- `ci apps enroll --env-file apps/web/.env.local` writes the app's enrollment into an env file: `CI_ISSUER`, `CI_AUDIENCE`, `CI_APPLICATION_ID`, `CI_CLIENT_ID`, `CI_ORGANIZATION_ID`, `CI_SCOPES` and, for a confidential app, `CI_CLIENT_SECRET`. Server code reads them with `readApplicationEnrollment(process.env)` and never names them one by one. `ci apps env <application>` prints the same set later without the secret. The Application Id is the registration id (the `app_id` claim), and the Client Id is the OAuth `client_id`. They are different values, and every CLI output names both.
63
+ The CLI writes the app's connection values into an env file: `CI_ISSUER`, `CI_AUDIENCE`, `CI_APPLICATION_ID`, `CI_CLIENT_ID`, `CI_ORGANIZATION_ID`, `CI_JWKS_URL` and, when a secret is known, `CI_CLIENT_SECRET`. Server code reads them with `readApplicationEnrollment(process.env)` and never names them one by one. Two commands write these values, and the right one depends on who created the app:
64
+
65
+ | The app was created by | Command | Client secret |
66
+ | -------------------------- | ----------------------------------------------------- | ----------------------------------------------------------------- |
67
+ | A package, as the template | `ci apps env <application> --env-file <path>` | Left out; issue one with `ci apps rotate-secret <application-id>` |
68
+ | The CLI | `ci apps enroll --fqn <fqn> --redirect-uri <url> ...` | Written by the first confidential create or by `--rotate-secret` |
69
+
70
+ `ci apps env` accepts an Application Id or an FQN and changes nothing in CI. `ci apps enroll` creates or replaces an app by FQN, so it refuses an app that a package declared. The Application Id is the registration id (the `app_id` claim), and the Client Id is the OAuth `client_id`. They are different values, and every CLI output names both.
44
71
 
45
72
  Next to the enrollment, the template reads `APP_ORIGIN` (the public HTTPS origin), `CI_TENANT_URL`, `SESSION_SECRETS` (comma-separated, current secret first, so you can rotate) and `NEXT_PUBLIC_APPLICATION_ID`, the only value the browser sees.
46
73
 
47
74
  ## After install
48
75
 
49
- Installing the package registers the app. Users can sign in and see data only after the first four steps below; the fifth gives refused users a readable page.
76
+ Installing the package registers the app as a confidential client with no secret. Users can sign in and see data only after the first five steps below; the sixth gives refused users a readable page. Run the CLI steps as an Organization admin, signed in with `ci login <api-url> --organization <organization-id>`.
77
+
78
+ 1. Write the connection values with `ci apps env <application> --env-file apps/web/.env.local`. The command leaves out the client secret and prints the Application Id that the next step needs.
79
+ 2. Rotate the sign-in secret with `ci apps rotate-secret <application-id>`, or with "Rotate secret" on the Dashboard External Apps page. CI shows the secret once, so store it as `CI_CLIENT_SECRET` at once. Each rotation ends the previous secret. Run `ci apps env` first: when the env file holds a secret for a different Client Id, `ci apps env` refuses to write the file.
80
+ 3. Create a Server Credential for the app's server with `ci apps credentials upsert <application> --name <name> --env-file <server-env-file>`. Use a separate env file, because this file also stores its secret as `CI_CLIENT_SECRET`.
81
+ 4. Add users to the Group whose grant holds the app gate. In the template that is Sales or Sales managers. The package declares the Groups and leaves them empty.
82
+ 5. Give the app server read access to Leads. The template's Read Port reads with the server's own credential, and the package declares no grant for it.
83
+ 6. Set `errorPath` on the federated auth so that a refused sign-in lands on a page instead of a JSON answer. The callback reports `access_denied`, `authorization_failed`, `invalid_state`, `invalid_request`, `invalid_identity` or `user_auth_unavailable`.
84
+
85
+ The app holds two different secrets. The sign-in secret from step 2 belongs to the OAuth client and only signs users in. The Server Credential from step 3, whose secret starts with `dsc_`, authenticates the server's own calls in service mode and acts with the Access Grants given to the app's server. CI refuses the sign-in secret on a server call, so a server that reads without a user needs the Server Credential.
50
86
 
51
- 1. Rotate the client secret. Install creates a confidential app with no secret, so an Organization administrator with `external_apps:manage` rotates one from the Dashboard ("Rotate secret") or with `POST /admin/organization/external-apps/{id}/credential/rotate`. CI shows the secret once.
52
- 2. Run `ci apps enroll` to write the env file, including the new secret.
53
- 3. Add users to the Group whose grant holds the app gate. In the template that is Sales or Sales managers. The package declares the Groups and leaves them empty.
54
- 4. Give the app server read access to Leads. The template's Read Port reads with the app's own credential, and the package declares no grant for it.
55
- 5. Set `errorPath` on the federated auth so that a refused sign-in lands on a page instead of a JSON answer. The callback reports `access_denied`, `authorization_failed`, `invalid_state`, `invalid_request`, `invalid_identity` or `user_auth_unavailable`.
87
+ > **Warning:** `ci apps enroll`, `ci apps redirect-uris` and `ci apps origins` refuse an app
88
+ > that a package declared. Change its redirect URIs and origins in
89
+ > `declareExternalApplication` and deploy a new package release.
56
90
 
57
91
  ## Sessions
58
92
 
@@ -74,7 +108,7 @@ const auth = createFederatedAuth({
74
108
  })
75
109
  ```
76
110
 
77
- `createFederatedAuth` handles the PKCE login, the callback, session reads and logout under `/api/auth`. Register `<origin>/api/auth/callback` with `ci apps redirect-uris add` and the origin with `ci apps origins add`. The session cookie is `__Host-ci-federated` with `Secure`, `HttpOnly` and `SameSite=Lax`, which is why `APP_ORIGIN` must use HTTPS. On a laptop without a TLS proxy, `allowLoopbackHttp: true` lets `http://localhost:3000` work with a cookie named `ci-federated` and no `Secure` flag. Keep that option off in production.
111
+ `createFederatedAuth` handles the PKCE login, the callback, session reads and logout under `/api/auth`. Register `<origin>/api/auth/callback` as a redirect URI and the origin as an allowed origin. A package-declared app lists both in its declaration; an app created with the CLI adds them with `ci apps redirect-uris add` and `ci apps origins add`. The session cookie is `__Host-ci-federated` with `Secure`, `HttpOnly` and `SameSite=Lax`, which is why `APP_ORIGIN` must use HTTPS. On a laptop without a TLS proxy, `allowLoopbackHttp: true` lets `http://localhost:3000` work with a cookie named `ci-federated` and no `Secure` flag. Keep that option off in production.
78
112
 
79
113
  Access tokens always live 300 seconds. A confidential client that passes its `clientSecret` to `createFederatedAuth` renews them silently: when a token has 30 seconds or less left, the session lookup gets a new one first. The renewed token stays in process memory and the cookie is written once at sign-in, so a server restart only triggers another renewal. The session itself lasts `sessionTtlSeconds`, 86400 by default and at most 604800, and never longer than the user's CI session.
80
114
 
@@ -137,7 +171,7 @@ The template mounts three route files:
137
171
  | `app/api/leads/[id]/convert/route.ts` | Convert a Lead through a Fixed workflow route |
138
172
  | `app/api/app/[[...path]]/route.ts` | The app's own Effect `HttpApi`, with OpenAPI at `/api/app/openapi.json` |
139
173
 
140
- A server call to CI uses one of three identities. A delegated call, the default, uses the signed-in user's token, and the template's convert route works this way. `runAsApplication` switches one call to the app's confidential credentials and needs an audit subject. A service-mode client (`createServerClient({ authentication: { mode: "service", credentials } })`) always uses the app's own credential and gets the same answer for every user, which is what the template's Read Port does.
174
+ A server call to CI uses one of three identities. A delegated call, the default, uses the signed-in user's token, and the template's convert route works this way. `runAsApplication` switches one call to the app's confidential credentials and needs an audit subject. A service-mode client (`createServerClient({ authentication: { mode: "service", credentials } })`) always uses the app's Server Credential and gets the same answer for every user, which is what the template's Read Port does.
141
175
 
142
176
  ## Reads and writes through the SDK
143
177
 
@@ -159,7 +193,7 @@ const handle = createFixedWorkflowRoute<{ leadId: string }, { leadId: string }>(
159
193
 
160
194
  The route rejects any `workflowId` or `workflowFqn` in the browser input, checks the Access Requirement, stamps `x-organization-id` from the session and returns `{ runId }`. The template's tests prove each refusal: an injected `workflowId` gets 400, no session gets 401, a missing capability gets 403, and two clicks reuse the idempotency key `convert-lead:<id>`.
161
195
 
162
- The workflow must have an External application trigger bound to the app and an audience that includes it (`audience: { applications: [application] }`). A delegated call runs the workflow as the signed-in user. A service-mode call has no user, so the run uses the workflow's run-as service principal. CI publishes a triggered workflow only when it names one, which is why the template ships `convertLeadRunner`.
196
+ The workflow must have an External application trigger bound to the app and an audience that includes it (`audience: { applications: [application] }`). A run acts as the identity that started it. A delegated call runs the workflow as the signed-in user. A call in service mode or with `runAsApplication` has no person, so the run uses the workflow's run-as service principal. CI publishes a triggered workflow only when it names one, which is why the template ships `convertLeadRunner`.
163
197
 
164
198
  A run returns its `runId` at once. The template's `convertLeadTask` posts the convert route, follows the run with `runs.follow({ runId })` until it ends (10 minutes by default), and then invalidates the entity reads so every view reloads.
165
199
 
@@ -182,7 +216,7 @@ Tags cover all CI reads, a schema, a list, one entity and one Realtime topic. A
182
216
  ```mermaid
183
217
  flowchart LR
184
218
  W1[Write through the BFF] --> N[nextInvalidate]
185
- W2["Write elsewhere<br/>workflow, Studio, agent, other app"] --> RT[Realtime change in an open tab] --> RV[Revalidate route]
219
+ W2[Write outside the BFF] --> RT[Realtime change in an open tab] --> RV[Revalidate route]
186
220
  W3[Effect server write] --> L[CacheInvalidation.layerNext on the fiber]
187
221
  N --> T[Entity, list and topic tags expire]
188
222
  RV --> T
@@ -202,9 +236,9 @@ Realtime keeps open pages current without polling. The browser asks the BFF for
202
236
 
203
237
  A topic is one entity schema, optionally filtered by one field: `{ schema, filter?: { field, eq } }`. Events carry ids only, and the page refetches. One connection per tab carries all of its topics. A `reset` event means the client missed too much and must refetch everything; the first open of a connection also resyncs once, so expect one extra refetch per page load. Pass `onReset`, or a change can be lost. `onAccessChanged` fires when the user's access changes.
204
238
 
205
- When the BFF has `secrets`, each ticket answer includes a sealed topic grant, valid for 10 minutes and bound to the session. The revalidate route `POST /api/realtime/revalidate` accepts changes only on topics in that grant, which is how a change from Studio or a workflow expires the server cache. The client renews the ticket after 8 minutes.
239
+ When the BFF has `secrets`, each ticket answer includes a sealed topic grant, valid for 10 minutes and bound to the session. The revalidate route `POST /api/realtime/revalidate` accepts changes only on topics in that grant, which is how a change written outside the BFF expires the server cache. The client renews the ticket after 8 minutes.
206
240
 
207
- To show this in a demo, open `/crm` in two windows and edit a Lead as a Sales manager in one of them. The other window updates without a reload, also when the change comes from Studio, an agent or the convert-lead workflow.
241
+ To show this in a demo, open `/crm` in two windows and edit a Lead as a Sales manager in one of them. The other window updates without a reload.
208
242
 
209
243
  > **Warning:** Do not open the stream with a hand-written `EventSource`. It cannot send the
210
244
  > `Authorization` and `X-Organization-Id` headers that CI needs, and the failure
@@ -234,7 +268,7 @@ const gateway = defineGateway({
234
268
  })
235
269
  ```
236
270
 
237
- Adapters for Next, Web Fetch, Express and Nest serve the same gateway kernel. A gateway exposes remote resources and never creates CI schemas: an administrator connects them through a Knowledge connector and maps fields onto an Organization schema. In NestJS, `bindNestOperation` binds an existing service method without decorators, and the CI entry point does not inherit your HTTP guards, pipes or interceptors, so each entry point keeps its own checks. Start Nest with `NestFactory.create(AppModule, { bodyParser: false })` so the SDK parses bounded JSON, and keep a global prefix off the gateway's `/v1/*` routes.
271
+ Adapters for Next, Web Fetch, Express and Nest serve the same gateway kernel. A gateway exposes remote resources and never creates CI schemas: an Organization admin connects them through a Knowledge connector and maps fields onto an Organization schema. In NestJS, `bindNestOperation` binds an existing service method without decorators, and the CI entry point does not inherit your HTTP guards, pipes or interceptors, so each entry point keeps its own checks. Start Nest with `NestFactory.create(AppModule, { bodyParser: false })` so the SDK parses bounded JSON, and keep a global prefix off the gateway's `/v1/*` routes.
238
272
 
239
273
  ## Related pages
240
274
 
@@ -0,0 +1,135 @@
1
+ # Infrastructure
2
+
3
+ The resources that a CI deployment runs, what each one is for, and how the Amazon EKS blueprint maps them to AWS services.
4
+
5
+ The CI deployment that your demo connects to provides every resource on this page. You do not provision or run any of them. This page explains what runs where, so that you know what sits behind the API when your package or your web app calls CI. The [Platform architecture](../platform/architecture.md) page introduces the same parts from the builder's side.
6
+
7
+ ## Runtime resources
8
+
9
+ ```mermaid
10
+ flowchart LR
11
+ subgraph outside["Outside the CI deployment"]
12
+ browser["Browser"]
13
+ studio["Studio desktop app"]
14
+ webapp["Your web app (BFF)"]
15
+ end
16
+ subgraph deployment["CI deployment"]
17
+ gateway["Gateway"]
18
+ dashboard["Dashboard and Studio install page"]
19
+ api["Backend API"]
20
+ workers["Background and interactive workers"]
21
+ compute["Compute"]
22
+ media["Media"]
23
+ pg[("PostgreSQL with pgvector")]
24
+ broker["Message broker"]
25
+ storage[("Object storage")]
26
+ end
27
+ ai["AI model providers"]
28
+ browser --> gateway
29
+ studio --> gateway
30
+ webapp -->|"Application API"| gateway
31
+ gateway --> dashboard
32
+ gateway --> api
33
+ gateway --> media
34
+ api --> pg
35
+ workers --> pg
36
+ api -->|"wake signals"| broker
37
+ broker --> workers
38
+ broker -->|"Compute jobs"| compute
39
+ compute -->|"results"| api
40
+ compute -->|"presigned URLs"| storage
41
+ browser -->|"signed upload"| storage
42
+ media -->|"signed source links"| storage
43
+ workers --> ai
44
+ ```
45
+
46
+ | Resource | What it is for |
47
+ | ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
48
+ | Gateway | Receives all HTTP traffic and routes it to the Dashboard, the backend API and Media. |
49
+ | Dashboard | A static web app that calls the backend API for all its data. It also serves the public Studio install page of each Organization, where people download Studio and connect it with one button. |
50
+ | Backend API | Serves every HTTP route that the Dashboard, Studio, the CLI and your app call. It stores long work and returns at once; workers run that work. |
51
+ | Workers | Background workers run long jobs, and interactive workers run the agent and tool work that users wait for. |
52
+ | Compute | Runs transcription, transcoding and conversion from a queue. Each capability has its own slots, so a long transcode does not block transcription. |
53
+ | Media | Serves file bytes and image transforms from URLs that the backend signs. No file byte passes through a backend endpoint. |
54
+ | PostgreSQL | Holds the authoritative data in two databases: one for application data, Knowledge and queued work, and one for datasets. The pgvector extension stores embeddings for semantic search. |
55
+ | Message broker | Carries wake signals from the API to the workers, broadcast events between backend processes, and the job queues of Compute. A wake signal is only a hint: workers find their work in PostgreSQL. |
56
+ | Object storage | Holds every file. Studio installers and updates, and archived rows, have their own stores next to the store for application files. |
57
+ | Cache | There is no separate cache service. Each backend process keeps hot reads, such as schemas, policies and settings, in memory, and broadcast events clear stale entries in every process. |
58
+ | Email provider | Sends platform email. Every message goes through a durable queue in PostgreSQL before the provider receives it. |
59
+ | AI model providers | External model services that agents, Skills and embeddings call. A Tenant admin chooses the available models; the blueprints do not provision them. |
60
+ | Logs | Every backend process and Compute write structured logs, and the CI deployment forwards them to a log service. |
61
+
62
+ ## What this means for your code
63
+
64
+ File uploads go straight from the browser to object storage. Your BFF asks CI for an upload slot, and the browser sends the bytes with one signed PUT. A Knowledge file field accepts a file up to its `maxBytes`, and never above the limit of the CI deployment, which is 1 GiB by default and at most 2 GiB. The browser must send the URL, the method and every header that CI returns. Your BFF never streams file bytes. The [External apps](external-apps.md) page shows the flow in the template.
65
+
66
+ Event streams, such as a chat answer, stay open through a heartbeat. The gateway closes a request that sends no bytes for 120 seconds and caps every response at 10 minutes. The backend sends a comment line on every event stream each 20 seconds, so a slow model does not cut the stream. If your BFF forwards a stream, pass each chunk on as it arrives so the heartbeat reaches the browser.
67
+
68
+ Long work is asynchronous. When your app starts an agent run or a workflow, the API stores the run and returns, and a worker runs it. A slow broker delays work, because workers always read their work from PostgreSQL.
69
+
70
+ Configuration changes reach every backend process. When a schema, a policy or a setting changes, a broadcast event clears the cached copy in each process after the write commits. If that event is lost, the cached copy still expires after a short time limit.
71
+
72
+ ## The Amazon EKS blueprint
73
+
74
+ A blueprint is the infrastructure code and the deployment configuration that turn the resources above into cloud services. Running CI deployments use the Amazon Elastic Kubernetes Service (EKS) blueprint. Blueprints also exist for Azure Kubernetes Service (AKS) and Azure Container Apps (ACA).
75
+
76
+ The EKS blueprint has three layers. Infrastructure as code creates the AWS foundation: the network, the cluster, the image registry, DNS, and the managed databases and buckets of each CI deployment. Helm charts describe the workloads, and Argo CD keeps the cluster in line with them. Release pins name the exact image of each service, so a new CI release is a change of pins that Argo CD rolls out.
77
+
78
+ ```mermaid
79
+ flowchart LR
80
+ users["Browser, Studio and your web app"]
81
+ cdn["Amazon CloudFront"]
82
+ nlb["Network Load Balancer"]
83
+ subgraph eks["Amazon EKS cluster"]
84
+ gateway["Envoy Gateway"]
85
+ dashboard["Dashboard"]
86
+ api["Backend API"]
87
+ workers["Workers"]
88
+ compute["Compute"]
89
+ media["Media"]
90
+ rabbit["RabbitMQ"]
91
+ end
92
+ rds[("Amazon RDS for PostgreSQL or CloudNativePG")]
93
+ s3[("Amazon S3")]
94
+ ses["Amazon SES"]
95
+ secrets["AWS Secrets Manager"]
96
+ ecr["Amazon ECR"]
97
+ users --> nlb
98
+ users -->|"media URLs"| cdn
99
+ cdn --> nlb
100
+ nlb --> gateway
101
+ gateway --> dashboard
102
+ gateway --> api
103
+ gateway --> media
104
+ api --> rds
105
+ workers --> rds
106
+ api --> rabbit
107
+ rabbit --> workers
108
+ rabbit --> compute
109
+ compute --> s3
110
+ media --> s3
111
+ eks -->|"email"| ses
112
+ secrets -.->|"credentials"| eks
113
+ ecr -.->|"images"| eks
114
+ ```
115
+
116
+ | Resource | AWS service |
117
+ | ----------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
118
+ | Containers | Amazon EKS with a managed node group. Images come from Amazon Elastic Container Registry (ECR). |
119
+ | Gateway | Envoy Gateway behind a Network Load Balancer. Amazon Route 53 holds the DNS records, and cert-manager issues the certificates. |
120
+ | PostgreSQL | Amazon Relational Database Service (RDS) for PostgreSQL in private subnets, or CloudNativePG in the cluster on encrypted Amazon Elastic Block Store (EBS) volumes. Each CI deployment chooses one. |
121
+ | Message broker | RabbitMQ in the cluster. |
122
+ | Object storage | Amazon Simple Storage Service (S3) buckets for the files and the archive of each CI deployment, and shared buckets for Studio releases. |
123
+ | Media CDN | Amazon CloudFront in front of Media. |
124
+ | Email | Amazon Simple Email Service (SES). |
125
+ | Secrets | AWS Secrets Manager. External Secrets copies the values into the cluster. |
126
+ | Workload identity | EKS Pod Identity, so pods reach S3 and Secrets Manager without stored keys. |
127
+ | Logs and metrics | Datadog receives the logs. Amazon CloudWatch Container Insights collects the cluster metrics. |
128
+
129
+ One EKS cluster can host several CI deployments. Each CI deployment has its own Kubernetes namespace with its own backend, Compute, database, message broker and storage, and it can hold several Organizations. All CI deployments on the cluster share the gateway and Media.
130
+
131
+ ## Next pages
132
+
133
+ - [Platform architecture](../platform/architecture.md): Tenants, Organizations, surfaces and the three integration models.
134
+ - [External apps](external-apps.md): Build the web app that calls CI, including direct uploads.
135
+ - [Knowledge](../data/knowledge.md): Records, schemas and semantic search on PostgreSQL.
@@ -124,9 +124,9 @@ import { useQuery } from "@tanstack/react-query"
124
124
 
125
125
  The template's own pages use the template's shadcn/ui package, `@acme/ui`. `ui-react` appears in the Studio widget and in SDK toasts.
126
126
 
127
- ### Server SDK subpaths a demo uses
127
+ ### Server SDK subpaths an external app uses
128
128
 
129
- `external-apps-server-sdk` runs only on your server, because it holds the application's client secret. Each framework peer loads only from its own subpath: `/next` loads `next`, while `/effect` loads no `next`, `pg` or `drizzle-orm`. Administrative APIs exist only on `/admin` and never reach the browser proxy. A demo uses four subpaths:
129
+ `external-apps-server-sdk` runs only on your server, because it holds the application's client secret. Each framework peer loads only from its own subpath: `/next` loads `next`, while `/effect` loads no `next`, `pg` or `drizzle-orm`. Administrative APIs exist only on `/admin` and never reach the browser proxy. An external app uses four subpaths:
130
130
 
131
131
  | Subpath | What you get |
132
132
  | ----------------- | --------------------------------------------------------------------------------------------------------------------------- |
@@ -28,14 +28,14 @@ flowchart LR
28
28
 
29
29
  1. Generate a new repository from `next-effect-app` and rename the `@acme` packages. [The solution template](template/index.md) explains the layout you get.
30
30
  2. Choose the primitives for the scenario. [Choosing primitives](scenarios/index.md) maps common demo scenarios to the parts that fit them.
31
- 3. Declare entities, access and AI parts in the packages' `ci/` folders. The [Data](data/knowledge.md), [AI](ai/index.md) and [Grants and permissions](platform/access.md) pages describe each declaration.
31
+ 3. Declare entities, access and AI parts in the packages' `ci/` folders. The [Data resources](data/index.md), [AI](ai/index.md) and [Grants and permissions](platform/access.md) pages describe each declaration.
32
32
  4. Build the screens and server routes of the web app with the [SDKs](develop/sdks.md), following the patterns in [External apps](develop/external-apps.md).
33
33
  5. Deploy the package to an Organization with `ci deploy`, as [Extensions](develop/extensions.md) describes, and point the web app at that Organization.
34
34
 
35
35
  ## Sections
36
36
 
37
37
  - [Platform](platform/architecture.md): What CI is made of, how people and apps enter it, and how access works.
38
- - [Data](data/knowledge.md): Where demo data lives: Knowledge, datasets and connectors.
38
+ - [Data resources](data/index.md): Where data lives: the Data resources Knowledge, Datasets and Documents, and the connectors that fill them.
39
39
  - [AI](ai/index.md): Tools, Skills, Agents and workflows, and how they combine.
40
40
  - [Build](develop/sdks.md): The SDKs, extension packages and external apps you write.
41
41
  - [Template](template/index.md): The layout of `next-effect-app` and its state machine approach.
@@ -43,7 +43,7 @@ flowchart LR
43
43
 
44
44
  ## Where to begin
45
45
 
46
- If you are starting a new demo, read [Platform architecture](platform/architecture.md) first, then [The solution template](template/index.md), and use [Choosing primitives](scenarios/index.md) to plan the scenario. If you are adding a feature to a demo that already runs, go straight to the page for that part: data questions belong in Data, model and automation questions in AI, and code questions in Build.
46
+ If you are starting a new demo, read [Platform architecture](platform/architecture.md) first, then [The solution template](template/index.md), and use [Choosing primitives](scenarios/index.md) to plan the scenario. If you are adding a feature to a demo that already runs, go straight to the page for that part: data questions belong in Data resources, model and automation questions in AI, and code questions in Build.
47
47
 
48
48
  ## For agents
49
49
 
@@ -58,19 +58,23 @@ GITHUB_TOKEN=$(gh auth token) opensrc path collective-intelligence-solutions/sdk
58
58
  ## Pages
59
59
 
60
60
  - Platform
61
- - [Platform architecture](platform/architecture.md): The parts of Collective Intelligence, the three integration models, and how a demo package moves from build to install.
62
- - [Entrypoints](platform/entrypoints.md): The ways people, apps and agents enter CI at runtime, and the package subpaths that demo code imports.
61
+ - [Platform architecture](platform/architecture.md): The parts of Collective Intelligence, the three integration models, and how a package moves from build to install.
62
+ - [Entrypoints](platform/entrypoints.md): The ways people, apps and agents enter CI at runtime, and the package subpaths that your code imports.
63
63
  - [Grants and permissions](platform/access.md): How CI decides what a user, a Group, an app or an API key may do, and how to change it in the template.
64
- - Data
65
- - [Knowledge](data/knowledge.md): Declare schemas, store typed records, query them, and read and watch them from a demo app.
64
+ - Data resources
65
+ - [Data resources](data/index.md): Choose between Knowledge, Datasets and Documents, and see where each kind keeps its data.
66
+ - [Knowledge](data/knowledge.md): Declare schemas, store typed records, query them, and read and watch them from an external app.
66
67
  - [Datasets](data/datasets.md): Query typed tables of measurements, events and warehouse data, and control who can read them.
68
+ - [Documents](data/documents.md): Keep files in Document Collections, sync them from Microsoft 365 or Google Drive, control who reads them, and know when their content becomes searchable.
67
69
  - [Connectors](data/connectors.md): Connect outside systems, map their fields to Knowledge schemas, and run and watch imports.
68
70
  - AI
69
71
  - [AI primitives](ai/index.md): Tools, Skills, Agents, workflows and Studio chat, how they compose, and how audiences, approvals, models, runs and timeouts behave.
72
+ - [Studio](ai/studio.md): How Organization members work in the Studio desktop app, how a package ships Studio apps, forms and widgets, and how Studio shares one solution with your external application.
70
73
  - Build
71
74
  - [SDKs](develop/sdks.md): Which @collective-intelligence package to import, from which subpath, with which Effect setup and which versions.
72
75
  - [Extensions](develop/extensions.md): What a package declares, how the template wires it, how to deploy it with ci deploy, and the rules its code runs under.
73
76
  - [External apps](develop/external-apps.md): Sign users in with CI, read and write their data through a BFF, cache shared reads and stay live with Realtime.
77
+ - [Infrastructure](develop/infrastructure.md): The resources that a CI deployment runs, what each one is for, and how the Amazon EKS blueprint maps them to AWS services.
74
78
  - Template
75
79
  - [The solution template](template/index.md): What you get from next-effect-app, where each piece of code goes, and how to take a new demo from local dev to an installed package.
76
80
  - [State machines](template/state-machines.md): How next-effect-app models UI flows as XState machines with Effect actors, keeps server data in the QueryCache, resends uncertain writes safely, and checks hard flows with Models.