@collective-intelligence/extensions-sdk-backend 0.2.6 → 0.2.8
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/AGENTS.md +4 -0
- package/README.md +7 -0
- package/dist/effect/datasets/provider-implementations.d.ts +14 -2
- package/dist/effect.js +1 -1
- package/docs/effect.md +3 -0
- package/docs/guide/ai/index.md +349 -0
- package/docs/guide/data/connectors.md +204 -0
- package/docs/guide/data/datasets.md +156 -0
- package/docs/guide/data/knowledge.md +211 -0
- package/docs/guide/develop/extensions.md +208 -0
- package/docs/guide/develop/external-apps.md +245 -0
- package/docs/guide/develop/sdks.md +233 -0
- package/docs/guide/index.md +78 -0
- package/docs/guide/platform/access.md +244 -0
- package/docs/guide/platform/architecture.md +142 -0
- package/docs/guide/platform/entrypoints.md +236 -0
- package/docs/guide/scenarios/index.md +107 -0
- package/docs/guide/template/index.md +276 -0
- package/docs/guide/template/state-machines.md +294 -0
- package/package.json +4 -4
|
@@ -0,0 +1,276 @@
|
|
|
1
|
+
# The solution template
|
|
2
|
+
|
|
3
|
+
What you get from next-effect-app, where each piece of code goes, and how to take a new demo from local dev to an installed package.
|
|
4
|
+
|
|
5
|
+
`next-effect-app` is the starting point for every demo. It is a bun and Turborepo monorepo with one Next.js 16 App Router web app (`apps/web`) and one Extension Package declaration (`apps/extensions`). You start a demo by generating a new repository from it with **Use this template** on GitHub. Its example features are patterns to copy: a demo keeps their shape and replaces their content with the customer's domain.
|
|
6
|
+
|
|
7
|
+
## What you get
|
|
8
|
+
|
|
9
|
+
```bash
|
|
10
|
+
apps/
|
|
11
|
+
web/ # @acme/web Next app: routes, composition, Data Sources, app API and BFF instances
|
|
12
|
+
extensions/ # @acme/extensions Extension Package declaration (composition root)
|
|
13
|
+
packages/
|
|
14
|
+
organization/acme/ # @acme/organization package identity, access, groups, external application
|
|
15
|
+
features/intake/ # @acme/intake lead intake: API, store, XState flow, views
|
|
16
|
+
features/lead-conversion/ # @acme/lead-conversion convert-lead workflow and browser flow
|
|
17
|
+
features/lead-qualification/ # @acme/lead-qualification lead qualifier agent, Skill, tool, Studio widget
|
|
18
|
+
entities/lead/ # @acme/lead Lead contract, CI schema, Lead API, Read Port, Realtime, views
|
|
19
|
+
entities/opportunity/ # @acme/opportunity Opportunity contract, CI schema, views
|
|
20
|
+
modules/identity/ # @acme/identity CI federated auth, CI runtime, principal, login providers
|
|
21
|
+
modules/app-api/ # @acme/app-api typed package API client, QueryClient, ApiPrefetch
|
|
22
|
+
modules/ui/ # @acme/ui shadcn/ui primitives, AI Elements, theme, workspace-kit
|
|
23
|
+
tooling/config/ # @acme/config shared TypeScript base config
|
|
24
|
+
```
|
|
25
|
+
|
|
26
|
+
Intake is a form with an XState flow and an app API that keeps its data in an in-memory store. Lead conversion is a Fixed workflow started by a single-action task. Lead qualification is an agent with a Skill, a tool and a Studio widget. The repository has no docs wiki, no infra folder and no generated client code, because the browser derives each typed client from its package `HttpApi` at runtime. It also ships an agent harness: an `AGENTS.md` per scope, and `.agents/` with skills, rules and lint scripts.
|
|
27
|
+
|
|
28
|
+
## Tiers and the import rule
|
|
29
|
+
|
|
30
|
+
Every package lives at `packages/<tier>/<name>`, keeps a flat name (`@acme/<name>`) and has exactly one `tier:*` tag in its own `turbo.json`. The six tiers, from top to bottom, are `app` (the composition roots), `organization` (package identity, access, groups, external application and deploy order), `features` (one user-visible capability each), `entities` (one business entity each), `modules` (business-agnostic machinery) and `tooling` (the shared tsconfig base).
|
|
31
|
+
|
|
32
|
+
```mermaid
|
|
33
|
+
flowchart TD
|
|
34
|
+
web["apps/web"]
|
|
35
|
+
ext["apps/extensions"]
|
|
36
|
+
org["organization<br/>@acme/organization"]
|
|
37
|
+
subgraph features["features"]
|
|
38
|
+
f1["intake"]
|
|
39
|
+
f2["lead-conversion"]
|
|
40
|
+
f3["lead-qualification"]
|
|
41
|
+
end
|
|
42
|
+
subgraph entities["entities"]
|
|
43
|
+
e1["lead"]
|
|
44
|
+
e2["opportunity"]
|
|
45
|
+
end
|
|
46
|
+
subgraph modules["modules"]
|
|
47
|
+
m1["identity"]
|
|
48
|
+
m2["app-api"]
|
|
49
|
+
m3["ui"]
|
|
50
|
+
end
|
|
51
|
+
tooling["tooling<br/>config"]
|
|
52
|
+
web --> org
|
|
53
|
+
ext -->|"passes packageComponents to declarePackage"| org
|
|
54
|
+
org -->|"composes CI declarations in deploy order"| features
|
|
55
|
+
features --> entities
|
|
56
|
+
entities --> modules
|
|
57
|
+
modules --> tooling
|
|
58
|
+
f1 -.-|"never imports"| f2
|
|
59
|
+
e1 -.-|"never imports"| e2
|
|
60
|
+
```
|
|
61
|
+
|
|
62
|
+
Imports point down the tiers and may skip a tier. Inside one tier, only modules and tooling packages import each other. A feature never imports another feature and an entity never imports another entity. When two of them must work together, a page in `apps/web` connects them.
|
|
63
|
+
|
|
64
|
+
The arrows are real configuration. The root `turbo.json` lists, for each tag, the tags it may depend on, and `bunx turbo boundaries` fails the build on a wrong import:
|
|
65
|
+
|
|
66
|
+
```json
|
|
67
|
+
"tier:features": { "dependencies": { "allow": ["tier:entities", "tier:modules", "tier:tooling"] } },
|
|
68
|
+
"tier:entities": { "dependencies": { "allow": ["tier:modules", "tier:tooling"] } },
|
|
69
|
+
"tier:modules": { "dependencies": { "allow": ["tier:modules", "tier:tooling"] } }
|
|
70
|
+
```
|
|
71
|
+
|
|
72
|
+
Workspaces import each other by package name and exported subpath, such as `@acme/lead/views`, and never by relative path.
|
|
73
|
+
|
|
74
|
+
## Package subpaths
|
|
75
|
+
|
|
76
|
+
A feature or entity package exposes only the subpaths it needs. Each subpath is a folder under `src/` with one `index.ts` and one `exports` entry in `package.json`.
|
|
77
|
+
|
|
78
|
+
| Subpath | Holds | Imported by |
|
|
79
|
+
| ---------- | ----------------------------------------------------------------------------------------------------------- | --------------------------------------------- |
|
|
80
|
+
| `domain` | Effect Schema models, resource contracts, schema FQNs, pure rules | every other subpath and other packages |
|
|
81
|
+
| `ci` | CI declarations (extensions SDK) as factories over organization handles | `@acme/organization` |
|
|
82
|
+
| `api` | the package `HttpApi` under `/api/app`, its groups and tagged errors | `server`, `headless`, the app API composition |
|
|
83
|
+
| `server` | the route Layer of the package `HttpApi` and its services | `apps/web/src/server` only |
|
|
84
|
+
| `headless` | typed client, query options, XState machines, entity bindings, Realtime topics and hooks, simulated sources | `views` and the app |
|
|
85
|
+
| `views` | wired React views on `@acme/ui`; feature parts enter through slots | the app |
|
|
86
|
+
|
|
87
|
+
Inside a package the subpaths also have a direction. `domain` imports no other subpath, `ci` imports neither `headless` nor `views`, and `headless` and `views` never import `server`. That keeps server code out of the browser bundle and keeps CI declarations free of React.
|
|
88
|
+
|
|
89
|
+
Each template package exposes a different set:
|
|
90
|
+
|
|
91
|
+
| Package | Subpaths |
|
|
92
|
+
| -------------------------- | ------------------------------------- |
|
|
93
|
+
| `@acme/lead` | `domain ci api server headless views` |
|
|
94
|
+
| `@acme/opportunity` | `domain ci headless views` |
|
|
95
|
+
| `@acme/intake` | `domain api server headless views` |
|
|
96
|
+
| `@acme/lead-conversion` | `ci headless views` |
|
|
97
|
+
| `@acme/lead-qualification` | `ci` |
|
|
98
|
+
|
|
99
|
+
`@acme/lead-qualification` adds an agent, a Skill, a tool and a Studio widget, and all of that is CI-side behavior. Its single `./ci` subpath shows that you can ship an AI capability without writing any web code.
|
|
100
|
+
|
|
101
|
+
## Features take their audience as a parameter
|
|
102
|
+
|
|
103
|
+
A feature package must not import `@acme/organization`, because organization sits above it in the tiers. A feature that offers something to a group of users therefore takes that `audience`, and the application handle, as parameters. The organization package owns the groups and passes them in, then lists every component in deploy order, where each component follows the components it references:
|
|
104
|
+
|
|
105
|
+
```ts
|
|
106
|
+
export const packageComponents = [
|
|
107
|
+
crmDomain,
|
|
108
|
+
...leadComponents({ domain: crmDomain, version: VERSION }),
|
|
109
|
+
...opportunityComponents({ domain: crmDomain, version: VERSION }),
|
|
110
|
+
salesGroup,
|
|
111
|
+
acmeApplication,
|
|
112
|
+
crmCapabilities,
|
|
113
|
+
...crmPolicies,
|
|
114
|
+
repProfile,
|
|
115
|
+
managerProfile,
|
|
116
|
+
...salesGrants,
|
|
117
|
+
applicationTokens,
|
|
118
|
+
...convertLeadComponents({ application: acmeApplication, version: VERSION }),
|
|
119
|
+
...leadQualificationComponents({
|
|
120
|
+
application: acmeApplication,
|
|
121
|
+
audience: salesMembers,
|
|
122
|
+
version: VERSION
|
|
123
|
+
}),
|
|
124
|
+
salesDeskApp
|
|
125
|
+
]
|
|
126
|
+
```
|
|
127
|
+
|
|
128
|
+
Because the organization package owns the content, `apps/extensions/src/index.ts` only declares the package:
|
|
129
|
+
|
|
130
|
+
```ts
|
|
131
|
+
export default declarePackage({
|
|
132
|
+
components: packageComponents,
|
|
133
|
+
hostedContributions: [/* the Lead score card Studio widget */],
|
|
134
|
+
manifest: packageManifest
|
|
135
|
+
})
|
|
136
|
+
```
|
|
137
|
+
|
|
138
|
+
[Extensions](../develop/extensions.md) describes each kind of declaration, and [Access](../platform/access.md) explains the Access Profiles and Access Grants in that list.
|
|
139
|
+
|
|
140
|
+
## The web app layers
|
|
141
|
+
|
|
142
|
+
Entity and feature code lives in packages. The web app keeps only routes, Composition, the sign-in gate, Data Sources and server instances, arranged in a fixed set of layers where each layer imports only the layers below it:
|
|
143
|
+
|
|
144
|
+
| Layer in `apps/web/src` | Holds |
|
|
145
|
+
| ----------------------- | --------------------------------------------------------------------------------------------------------------------- |
|
|
146
|
+
| `app/` | route files only; each re-exports a Composition export, and `app/api/**` forwards to `server/<domain>/server` |
|
|
147
|
+
| `composition/` | `providers/` (global providers, Realtime providers) and `routing/` (access, shells, pages) |
|
|
148
|
+
| `features/` | app-local capabilities: their Data Sources (`remote/server.ts`) and `page-link/` |
|
|
149
|
+
| `entities/` | app-local entity machinery that several features need (the CI session), Data Sources of entity packages, `page-link/` |
|
|
150
|
+
| `server/` | beside the layers: one folder per domain, where `server.ts` (`server-only`) holds the process instance |
|
|
151
|
+
|
|
152
|
+
Only Composition imports a Data Source, and only Data Sources and `app/api/**` import a `server/<domain>/server.ts`. Peers still meet without importing each other: the CRM page template passes `ConvertLead` into the `selectionActions` slot of `LeadWorkspace` and `LeadAttachment` into its `editorActions` slot. Each page's owner gives its `{ label, path }` in `page-link/`, and shells and links read it, so no route path is written twice.
|
|
153
|
+
|
|
154
|
+
One Next app serves two server surfaces:
|
|
155
|
+
|
|
156
|
+
| Path | Handler | What it is |
|
|
157
|
+
| ------------------------- | ----------------------- | ------------------------------------------------------------------------------------------------------------------------ |
|
|
158
|
+
| `/api/app/*` | `server/app-api/server` | the app API: the packages' Effect `HttpApi`s merged, OpenAPI at `/api/app/openapi.json`, Scalar docs at `/api/app/docs` |
|
|
159
|
+
| `/api/*` | `server/ci/server` | the CI SDK BFF: sign-in with a sealed cookie, Lead and Opportunity resource proxy, Realtime ticket and revalidate routes |
|
|
160
|
+
| `/api/leads/[id]/convert` | `server/ci/server` | the Fixed convert-lead workflow route |
|
|
161
|
+
|
|
162
|
+
The BFF holds the CI session in a sealed cookie and proxies resource calls, so the browser never holds a CI secret. [External applications](../develop/external-apps.md) covers the BFF in depth. The intake app API keeps its data in `IntakeStore.layerMemory`, and you replace that layer when a demo needs its own database.
|
|
163
|
+
|
|
164
|
+
## Start a new demo
|
|
165
|
+
|
|
166
|
+
1. Generate a repository with **Use this template**. In the new repository, delete the README "Contributing" section and `.github/workflows/close-pull-requests.yml`, which only matter in the template repository itself.
|
|
167
|
+
2. Rename the template for the customer. Replace the `@acme/` package scopes and the `acme.app.` FQNs, replace `https://acme.localhost` in `packages/organization/acme/src/application.ts` and `.env.example`, and replace the theme tokens and the sidebar brand in `apps/web/src/composition/routing/shells/workspace/app-sidebar.tsx`. Theme tokens live once, in `packages/modules/ui/src/styles/globals.css`.
|
|
168
|
+
3. Run it locally. The Overview and Lead intake pages work with no configuration, because the intake API keeps requests in memory:
|
|
169
|
+
|
|
170
|
+
```bash
|
|
171
|
+
bun install
|
|
172
|
+
bun run dev:web # http://localhost:3000
|
|
173
|
+
```
|
|
174
|
+
|
|
175
|
+
The CRM page needs a CI tenant.
|
|
176
|
+
|
|
177
|
+
4. Deploy the package to an Organization. `ci deploy` runs the whole release path in one command, as [Extensions](../develop/extensions.md) describes. The long path runs each step by hand:
|
|
178
|
+
|
|
179
|
+
```bash
|
|
180
|
+
ci package build --declaration ./apps/extensions/src/index.ts --out apps/extensions/dist/release
|
|
181
|
+
ci package validate --archive apps/extensions/dist/release/release.tar \
|
|
182
|
+
--manifest apps/extensions/dist/release/manifest.json \
|
|
183
|
+
--package-manifest apps/extensions/dist/release/package-manifest.json
|
|
184
|
+
ci package publish --backend-url <url> \
|
|
185
|
+
--archive apps/extensions/dist/release/release.tar \
|
|
186
|
+
--manifest apps/extensions/dist/release/manifest.json \
|
|
187
|
+
--package-manifest apps/extensions/dist/release/package-manifest.json \
|
|
188
|
+
--idempotency-key <key>
|
|
189
|
+
ci package promote --backend-url <url> --release <release-id> --idempotency-key <key>
|
|
190
|
+
ci package install --backend-url <url> --organization <organization-id> \
|
|
191
|
+
--package acme.app.knowledge --release <release-id> --idempotency-key <key>
|
|
192
|
+
```
|
|
193
|
+
|
|
194
|
+
Running `package build` into a scratch folder and then `package validate` is the dry run for any change to the package.
|
|
195
|
+
|
|
196
|
+
5. Rotate the application secret. The package installs a confidential application without a secret, so an Organization administrator rotates one:
|
|
197
|
+
|
|
198
|
+
```bash
|
|
199
|
+
curl -sS -X POST \
|
|
200
|
+
-H "authorization: Bearer <administrator-token>" \
|
|
201
|
+
-H "x-organization-id: <organization-id>" \
|
|
202
|
+
<url>/admin/organization/external-apps/<application-id>/credential/rotate
|
|
203
|
+
```
|
|
204
|
+
|
|
205
|
+
Copy `apps/web/.env.example` to `apps/web/.env.local`. Store `clientSecret` as `CI_CLIENT_SECRET`, `application.clientId` as `CI_CLIENT_ID` and the application id as `CI_APPLICATION_ID`. Then set `APP_ORIGIN`, `CI_TENANT_URL`, `CI_ISSUER`, `CI_AUDIENCE`, `CI_ORGANIZATION_ID`, `SESSION_SECRETS` and `NEXT_PUBLIC_APPLICATION_ID`, which equals `CI_APPLICATION_ID` and is the only value the browser sees. `ci apps enroll --env-file apps/web/.env.local` writes the enrollment values for you. Management tokens never go to Next.
|
|
206
|
+
|
|
207
|
+
6. Add users to the Sales group. CI accepts a user's token only when the user holds the application gate, so nobody can sign in yet. An administrator adds each user to the Sales group (Sales rep profile) or to its `managers` subgroup (Sales manager profile).
|
|
208
|
+
7. Sign in over HTTPS. The sealed session cookie requires HTTPS, so serve the dev server through a local HTTPS proxy:
|
|
209
|
+
|
|
210
|
+
```bash
|
|
211
|
+
portless acme bun run dev:web # https://acme.localhost
|
|
212
|
+
```
|
|
213
|
+
|
|
214
|
+
> **Warning:** The `./` in `--declaration ./apps/extensions/src/index.ts` is required.
|
|
215
|
+
> Without it, the CLI resolves the path like a module name and fails.
|
|
216
|
+
|
|
217
|
+
> **Note:** After each npm release of the CI packages, the template gets the released SDK
|
|
218
|
+
> versions pinned and one sync commit. A repository you generated earlier does
|
|
219
|
+
> not receive these updates by itself.
|
|
220
|
+
|
|
221
|
+
## What to change and what to keep
|
|
222
|
+
|
|
223
|
+
A new demo changes the organization package, the entities, the features, the theme and the origins. It keeps `modules/identity`, `modules/app-api`, the BFF and server wiring, the tier tags, the lint configuration and the harness files, because they hold no business logic. A new environment variable goes into `apps/web/.env.example`, `packages/modules/identity/src/server/env.ts`, `turbo.json` `globalEnv` and the README tenant section together.
|
|
224
|
+
|
|
225
|
+
## Checks that keep the structure honest
|
|
226
|
+
|
|
227
|
+
`@collective-intelligence/lint` is one Oxlint plugin named `ci-solution`. Extending `recommended` turns every rule on as an error:
|
|
228
|
+
|
|
229
|
+
| Rule | Checks |
|
|
230
|
+
| -------------------------------- | ---------------------------------------------------------------------------------- |
|
|
231
|
+
| `ci-solution/package-entrypoint` | an import of another package targets an entrypoint in its `package.json` `exports` |
|
|
232
|
+
| `ci-solution/subpath-imports` | imports between subpath folders follow the allowed direction |
|
|
233
|
+
| `ci-solution/subpath-entrypoint` | an import of another subpath folder targets its index file |
|
|
234
|
+
| `ci-solution/subpath-index` | every subpath folder has `index.ts` or `index.tsx` |
|
|
235
|
+
| `ci-solution/effect-subpath` | SDK Effect APIs come from the SDK `/effect` subpath |
|
|
236
|
+
| `ci-solution/no-run-in-view` | a view module calls no `fetch` and runs no Effect |
|
|
237
|
+
| `ci-solution/effect-actors` | XState side-effect actors are `fromEffect` actors |
|
|
238
|
+
| `ci-solution/effect-writes` | no module uses TanStack Query `useMutation`; writes run in `fromEffect` actors |
|
|
239
|
+
|
|
240
|
+
Each package has an `oxlint.local.ts` that extends the preset:
|
|
241
|
+
|
|
242
|
+
```ts
|
|
243
|
+
import { recommended as ciSolution } from "@collective-intelligence/lint"
|
|
244
|
+
import { defineConfig } from "oxlint"
|
|
245
|
+
|
|
246
|
+
export default defineConfig({
|
|
247
|
+
extends: [ciSolution],
|
|
248
|
+
overrides: [
|
|
249
|
+
{
|
|
250
|
+
files: ["**/*.proof.ts"],
|
|
251
|
+
rules: { "eslint/curly": "off", "jsdoc/check-tag-names": "off" }
|
|
252
|
+
}
|
|
253
|
+
]
|
|
254
|
+
})
|
|
255
|
+
```
|
|
256
|
+
|
|
257
|
+
The web app also runs a `frontend-layers` lint for layer order, domain entrypoints, folder names and the `no-peer-feature` and `no-peer-entity` rules. For coding agents, the template ships Codebase Rules in `.agents/rules/`, such as `web-layer-imports`, `ci-sdk-boundaries` and `ci-realtime-topics`, each with a path trigger and a task trigger. The day-to-day checks are:
|
|
258
|
+
|
|
259
|
+
```bash
|
|
260
|
+
bun run check-types
|
|
261
|
+
bun run test
|
|
262
|
+
bunx turbo boundaries
|
|
263
|
+
bun run lint
|
|
264
|
+
bun run format:check
|
|
265
|
+
bun run build
|
|
266
|
+
```
|
|
267
|
+
|
|
268
|
+
When you change the Lead cache, a Lead write or Lead Realtime, also run `bun run check:models` in `packages/entities/lead`. [State machines](state-machines.md) explains what that command checks.
|
|
269
|
+
|
|
270
|
+
## Related pages
|
|
271
|
+
|
|
272
|
+
- [State machines](state-machines.md)
|
|
273
|
+
- [SDKs](../develop/sdks.md)
|
|
274
|
+
- [Extensions](../develop/extensions.md)
|
|
275
|
+
- [External applications](../develop/external-apps.md)
|
|
276
|
+
- [Access](../platform/access.md)
|
|
@@ -0,0 +1,294 @@
|
|
|
1
|
+
# State machines
|
|
2
|
+
|
|
3
|
+
How next-effect-app models UI flows as XState machines with Effect actors, keeps server data in the QueryCache, resends uncertain writes safely, and checks hard flows with Models.
|
|
4
|
+
|
|
5
|
+
In `next-effect-app`, a UI flow that carries business logic is an XState machine in the `headless/` folder of its package, at `headless/<flow>-machine.ts`. Flows that belong only to the web app live at `apps/web/src/{features,entities}/<domain>/<flow>-machine.ts`. The machine decides where the flow is: editing, reviewing, submitting, failed. Every side effect is an Effect program that the machine runs as an actor, and TanStack Query owns the remote data that those programs read and write.
|
|
6
|
+
|
|
7
|
+
The machine context holds flow state only and never an API response. Two views that show the same record therefore read one cache entry, and no machine keeps a copy that can go stale. Server Components never run machines or Effects; they await a Promise boundary, and machines exist only in client code.
|
|
8
|
+
|
|
9
|
+
## Machine, task or hook
|
|
10
|
+
|
|
11
|
+
Most of the template does not use a hand-written machine. Pick the smallest tool that covers the flow:
|
|
12
|
+
|
|
13
|
+
| Pick | When | Template examples |
|
|
14
|
+
| -------------- | ---------------------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------- |
|
|
15
|
+
| XState machine | The flow has steps the user moves through, a lifecycle, or error branches that go back to an earlier step. | Lead intake (edit, review, submit), the CI session (initialize, sign in, refresh, sign out) |
|
|
16
|
+
| `effectTask` | One button runs one Effect and shows one result. | Convert lead, attach a file, sign out |
|
|
17
|
+
| Plain hook | The SDK hooks already cover it, or nothing writes. | `useEntityList`, `useEntityRecord`, `useEntityUpdate`, `useLeadRealtime`, the Lead score card |
|
|
18
|
+
|
|
19
|
+
Plain navigation and synchronous state changes need no actor at all.
|
|
20
|
+
|
|
21
|
+
## Side effects are Effect actors
|
|
22
|
+
|
|
23
|
+
Each side effect is a `fromEffect(program)` actor from `@collective-intelligence/external-apps-client-sdk-react/effect`, declared in `setup({ actors })`. The actor runs on the `runtime` that the machine gets as input, and a view gets that runtime with `useEffectRuntime()` from the `EffectRuntimeProvider` in `composition/providers`. The intake submit actor calls the app API and then invalidates the intake list, so every view that shows the list refetches:
|
|
24
|
+
|
|
25
|
+
```ts
|
|
26
|
+
export const submitIntakeActor = (submit: SubmitIntake) =>
|
|
27
|
+
fromEffect(({ draft }: IntakeInput & Readonly<{ draft: IntakeDraft }>) =>
|
|
28
|
+
submit(draft).pipe(
|
|
29
|
+
Effect.tap(() =>
|
|
30
|
+
QueryCache.use((cache) =>
|
|
31
|
+
cache.invalidate(intakeRequestsQuery.queryKey)
|
|
32
|
+
)
|
|
33
|
+
)
|
|
34
|
+
)
|
|
35
|
+
)
|
|
36
|
+
```
|
|
37
|
+
|
|
38
|
+
Remote data moves only through the `QueryCache` service of the runtime, with `fetch`, `invalidate`, `setData`, `update` and `remove`. Three lint rules enforce this. `ci-solution/effect-actors` requires `fromEffect` actors, and `ci-solution/effect-writes` forbids TanStack `useMutation` everywhere, including hooks and helpers. `ci-solution/no-run-in-view` forbids `fetch` and running an Effect in a view module.
|
|
39
|
+
|
|
40
|
+
## A machine in the template
|
|
41
|
+
|
|
42
|
+
The intake machine has four steps and a terminal state:
|
|
43
|
+
|
|
44
|
+
- `editing` goes to `reviewing` on `intake.review`.
|
|
45
|
+
- `reviewing` goes back to `editing` on `intake.edit`, or to `submitting` on `intake.confirm`.
|
|
46
|
+
- `submitting` goes to `done` when the actor succeeds, or back to `reviewing` with an error message when it fails.
|
|
47
|
+
- `done` goes to `editing` on `intake.reset`.
|
|
48
|
+
|
|
49
|
+
The `submitting` state invokes the actor. Its `onError` receives the typed failure, so the machine branches on it with `Schema.is`. `IntakeDuplicate` keeps its `_tag` from the server `HttpApi` to the browser, which is why `isDuplicate` works on the client:
|
|
50
|
+
|
|
51
|
+
```ts
|
|
52
|
+
submitting: {
|
|
53
|
+
invoke: {
|
|
54
|
+
input: ({ context }) => ({ draft: context.draft, runtime: context.runtime }),
|
|
55
|
+
onDone: {
|
|
56
|
+
actions: { params: ({ event }) => event.output, type: "submitted" },
|
|
57
|
+
target: "done",
|
|
58
|
+
},
|
|
59
|
+
onError: {
|
|
60
|
+
actions: assign({
|
|
61
|
+
error: ({ event }) =>
|
|
62
|
+
isDuplicate(event.error)
|
|
63
|
+
? "This email already has an open intake request."
|
|
64
|
+
: "The request could not be sent. Try again.",
|
|
65
|
+
}),
|
|
66
|
+
target: "reviewing",
|
|
67
|
+
},
|
|
68
|
+
src: "submitIntake",
|
|
69
|
+
},
|
|
70
|
+
},
|
|
71
|
+
```
|
|
72
|
+
|
|
73
|
+
Leaving the invoking state interrupts the Effect fiber. The machine has no UI of its own: `submitted` is a typed action, and the view provides the toast with `machine.provide({ actions })`. The view runs `useMachine(intakeMachine.provide(...), { input: { runtime } })` and renders by `snapshot.matches("editing")`, `matches("submitting")` and `matches("done")`.
|
|
74
|
+
|
|
75
|
+
The same `provide` call swaps the data source for a demo without a backend. A simulated source lives in `headless/simulated-*.ts`, built with `simulate("<pace>", effect)` from `@acme/app-api`, and only Composition or a test selects it. Tests use the same mechanism. A test drives the machine on a `ManagedRuntime` of the SDK `makeTestQueryCache` layer, never a hand-made `QueryCache` double, and asserts on cache entries or recorded calls:
|
|
76
|
+
|
|
77
|
+
```ts
|
|
78
|
+
const cache = makeTestQueryCache({ seed: [[intakeRequestsQuery.queryKey, []]] })
|
|
79
|
+
const runtime = ManagedRuntime.make(cache.layer)
|
|
80
|
+
const actor = createActor(
|
|
81
|
+
intakeMachine.provide({
|
|
82
|
+
actors: { submitIntake: submitIntakeActor(submit) }
|
|
83
|
+
}),
|
|
84
|
+
{ input: { runtime } }
|
|
85
|
+
).start()
|
|
86
|
+
// ...
|
|
87
|
+
expect(cache.calls).toEqual([
|
|
88
|
+
{ member: "invalidate", queryKey: intakeRequestsQuery.queryKey }
|
|
89
|
+
])
|
|
90
|
+
```
|
|
91
|
+
|
|
92
|
+
Under `bun test` the simulated pace is instant.
|
|
93
|
+
|
|
94
|
+
## Single actions with effectTask
|
|
95
|
+
|
|
96
|
+
For a single action, the SDK `effectTask(run)` wraps one Effect in a small machine with fixed states:
|
|
97
|
+
|
|
98
|
+
- `idle` goes to `running` on `RUN` with an input.
|
|
99
|
+
- `running` goes to `done` on success, with the result in `context.output`, or to `failed` on failure, with the error in `context.error`.
|
|
100
|
+
- `running` goes to `failed` on `CANCEL`, with a `Cause.Interrupt` reason as the error.
|
|
101
|
+
- `done` and `failed` go to `idle` on `RESET`, or back to `running` on `RUN`.
|
|
102
|
+
|
|
103
|
+
A `RUN` or `RESET` that arrives while the task is running is ignored. The convert-lead button is an `effectTask`. Its Effect starts the workflow through the BFF, follows the run to its end, and then marks every CI entity read stale:
|
|
104
|
+
|
|
105
|
+
```ts
|
|
106
|
+
export const convertLead =
|
|
107
|
+
(start: StartConvert) =>
|
|
108
|
+
({ followRun, leadId }: ConvertLeadRequest) =>
|
|
109
|
+
Effect.tryPromise({
|
|
110
|
+
catch: () => new ConvertLeadFailed({ step: "start" }),
|
|
111
|
+
try: async () => await start(leadId)
|
|
112
|
+
}).pipe(
|
|
113
|
+
Effect.flatMap(({ runId }) =>
|
|
114
|
+
Effect.tryPromise({
|
|
115
|
+
catch: () => new ConvertLeadFailed({ step: "follow" }),
|
|
116
|
+
try: async () => {
|
|
117
|
+
await followRun(runId)
|
|
118
|
+
}
|
|
119
|
+
})
|
|
120
|
+
),
|
|
121
|
+
Effect.andThen(
|
|
122
|
+
QueryCache.use((cache) => cache.invalidate(entityReadsKey))
|
|
123
|
+
)
|
|
124
|
+
)
|
|
125
|
+
|
|
126
|
+
export const convertLeadTask = effectTask(convertLead(startConvertLive))
|
|
127
|
+
```
|
|
128
|
+
|
|
129
|
+
The failure records which step failed, and `convertFailureMessage(snapshot)` maps it to a message for the view. The view calls `useMachine(convertLeadTask, { input: { runtime: useEffectRuntime() } })`, sends `{ type: "RUN", input: { followRun, leadId } }` from the button, and disables the button while `state.matches("running")`. On the server, the BFF route checks `convertLeadRequirement` on every call and starts the run with a fixed idempotency key per Lead:
|
|
130
|
+
|
|
131
|
+
```ts
|
|
132
|
+
idempotencyKey: ({ leadId }) => `convert-lead:${leadId}`,
|
|
133
|
+
```
|
|
134
|
+
|
|
135
|
+
## Uncertain writes and idempotency keys
|
|
136
|
+
|
|
137
|
+
A write is uncertain when the request left the browser and no answer came back. CI may or may not have applied it, and only a resend of the same action settles the question. The Lead attachment flow shows how the template handles this. `attachLeadFile` is an `effectTask` whose Effect checks the file's size and type locally, asks CI for an upload slot, PUTs the bytes straight to storage (never through the BFF), writes `{ attachment: { upload } }` at the Lead's revision with an idempotency key, and then invalidates the entity reads.
|
|
138
|
+
|
|
139
|
+
```mermaid
|
|
140
|
+
stateDiagram-v2
|
|
141
|
+
[*] --> idle
|
|
142
|
+
idle --> running: RUN with a new file
|
|
143
|
+
state running {
|
|
144
|
+
[*] --> checkFile
|
|
145
|
+
checkFile: Check size and type locally
|
|
146
|
+
askSlot: Ask CI for an upload slot
|
|
147
|
+
putBytes: PUT bytes to storage
|
|
148
|
+
writeLead: Write the Lead with an idempotency key
|
|
149
|
+
invalidateReads: Invalidate entity reads
|
|
150
|
+
checkFile --> askSlot
|
|
151
|
+
askSlot --> putBytes
|
|
152
|
+
putBytes --> writeLead
|
|
153
|
+
writeLead --> invalidateReads
|
|
154
|
+
}
|
|
155
|
+
running --> done: CI confirms the write
|
|
156
|
+
running --> failed: too large, wrong type, storage, conflict, refused
|
|
157
|
+
running --> failed: no answer, reason uncertain, write kept
|
|
158
|
+
failed --> running: Resubmit with the kept write, same key
|
|
159
|
+
failed --> running: RUN with a new file
|
|
160
|
+
done --> running: RUN with a new file
|
|
161
|
+
```
|
|
162
|
+
|
|
163
|
+
The steps inside `running` are steps of the Effect program, and the machine itself sees one `running` state. A resubmit skips the upload and sends only the final write. The size and type checks use `leadAttachmentAccepts` and `leadAttachmentMaxBytes` (5 MiB) from `@acme/lead/domain`. The CI schema declares the same values on the File field, so CI refuses a larger upload too, while a file that fails the local check sends nothing.
|
|
164
|
+
|
|
165
|
+
The write that might need a resend is a value the failure carries:
|
|
166
|
+
|
|
167
|
+
```ts
|
|
168
|
+
export const PendingAttachment = Schema.Struct({
|
|
169
|
+
expectedRevision: Schema.String,
|
|
170
|
+
idempotencyKey: Schema.String,
|
|
171
|
+
upload: Schema.String
|
|
172
|
+
})
|
|
173
|
+
```
|
|
174
|
+
|
|
175
|
+
When the write ends with a rejected promise or an SDK `uncertain` outcome, the task fails with reason `uncertain` and keeps that `PendingAttachment`. Sent again with the same idempotency key, CI applies the write at most once and returns its first answer. `pendingAttachment(snapshot)` reads the kept write from a failed task, and the `LeadAttachment` view shows a "Resubmit attachment" button only when it is not null. The button sends `RUN` with `{ pending }` in place of a new file, which reuses the same key and uploads nothing again. Every failure reason maps to one message, and the uncertain one reads "No answer came from CI, so the file may be attached. Resubmit to finish."
|
|
176
|
+
|
|
177
|
+
> **Warning:** A retry after an uncertain write must resend the same action with the same
|
|
178
|
+
> idempotency key. A new key makes it a new action, and CI could apply the
|
|
179
|
+
> change twice.
|
|
180
|
+
|
|
181
|
+
The SDK entity hooks follow the same rule. Their writes report `conflict`, `rejected` and `uncertain` in `status`; `submit` creates a new action with one operation key, and `resubmit` sends the previous action exactly as it was. Queries and mutations never retry by themselves. While a write is uncertain, `EntityForm` shows every field read-only, so Resubmit sends the draft that the form shows. [SDKs](../develop/sdks.md) lists these hooks.
|
|
182
|
+
|
|
183
|
+
## Realtime refreshes the cache
|
|
184
|
+
|
|
185
|
+
Realtime reaches the views through the cache. It never sends events into a machine. `useLeadRealtime` subscribes to the Lead topic, and each batch of changes refreshes the SDK entity reads and marks the app API read of each changed id stale:
|
|
186
|
+
|
|
187
|
+
```ts
|
|
188
|
+
export const leadsTopic: RealtimeTopic = { schema: leadSchemaFqn }
|
|
189
|
+
|
|
190
|
+
export const useLeadRealtime = () => {
|
|
191
|
+
const queryClient = useQueryClient()
|
|
192
|
+
const refreshEntityReads = useEntityChanges(leadContract)
|
|
193
|
+
useRealtimeTopic(leadsTopic, (changes) => {
|
|
194
|
+
refreshEntityReads(changes)
|
|
195
|
+
void refreshLeadReads(queryClient, changes)
|
|
196
|
+
})
|
|
197
|
+
}
|
|
198
|
+
```
|
|
199
|
+
|
|
200
|
+
Changes carry ids only, and the reads reload. Every write in the template ends the same way, with the cache:
|
|
201
|
+
|
|
202
|
+
| Writer | What it refreshes |
|
|
203
|
+
| -------------------------------------------------------------- | ------------------------------------------------------------------------ |
|
|
204
|
+
| Intake actor | invalidates the intake list query |
|
|
205
|
+
| Convert and attach actors | invalidate `entityReadsKey` |
|
|
206
|
+
| SDK entity hooks | store the confirmed record and reload lists |
|
|
207
|
+
| A write the BFF confirms | expires the Next cache tags of the record |
|
|
208
|
+
| A write outside the BFF (workflow, Studio, agent, another app) | reaches the Realtime Revalidate Route, which expires the Next cache tags |
|
|
209
|
+
|
|
210
|
+
Invalidation is enough for almost every flow, but one case can go wrong. A Realtime change can arrive while this tab's optimistic write is still pending, and a naive refetch would then show the old server value and hide the user's edit. The SDK entity read prevents this by waiting for the writes of its key to settle (`writesSettled`) before it fetches. The template checks that interleaving with a Model.
|
|
211
|
+
|
|
212
|
+
## Models
|
|
213
|
+
|
|
214
|
+
A Model is a `*.proof.ts` TypeScript module that states the state, actions and invariants of one flow. `tsprover` compiles it to TLA+, and the `tla-ts` checker visits every reachable state. When a rule breaks, the checker prints a trace: the sequence of actions, with the state after each one. An XState machine describes what one user sees step by step in one tab, while a Model describes what can happen when several writers interleave, and the template uses each for its own question.
|
|
215
|
+
|
|
216
|
+
| Construct | Meaning |
|
|
217
|
+
| ---------------------------------------- | ------------------------------------------------------ |
|
|
218
|
+
| top-level `let` | a state variable |
|
|
219
|
+
| exported function that changes state | an action |
|
|
220
|
+
| exported function that returns a boolean | an invariant |
|
|
221
|
+
| guard `if (<condition>) return;` | a precondition |
|
|
222
|
+
| `/** @fair */` | this action must eventually run |
|
|
223
|
+
| `/** @leadsTo target */` | a liveness property |
|
|
224
|
+
| `/** @constraint */` | a bound, so the search ends |
|
|
225
|
+
| `ghost` prefix on a `let` | records what a consumer saw, only to state a guarantee |
|
|
226
|
+
|
|
227
|
+
The Lead Realtime Model in `packages/entities/lead` is trimmed here. The full file also has `writeConfirm`, `otherWrite`, `realtimeChange` and `neverBehindConfirmed`:
|
|
228
|
+
|
|
229
|
+
```ts
|
|
230
|
+
let server: Revision = 0
|
|
231
|
+
let cached: Revision = 0
|
|
232
|
+
let writePending = false
|
|
233
|
+
let refetchDue = false
|
|
234
|
+
let ghostConfirmed: Revision = 0
|
|
235
|
+
let ghostPredicted: Revision = 0
|
|
236
|
+
|
|
237
|
+
export const writeStart = () => {
|
|
238
|
+
if (writePending) return
|
|
239
|
+
if (server === 2) return
|
|
240
|
+
writePending = true
|
|
241
|
+
ghostPredicted = server === 0 ? 1 : 2
|
|
242
|
+
cached = ghostPredicted
|
|
243
|
+
}
|
|
244
|
+
|
|
245
|
+
/** @fair */
|
|
246
|
+
export const refetch = () => {
|
|
247
|
+
if (!refetchDue) return
|
|
248
|
+
if (writePending) return
|
|
249
|
+
cached = server
|
|
250
|
+
refetchDue = false
|
|
251
|
+
}
|
|
252
|
+
|
|
253
|
+
export const pendingWriteShown = () =>
|
|
254
|
+
!writePending || cached === ghostPredicted
|
|
255
|
+
|
|
256
|
+
/** @leadsTo isFresh */
|
|
257
|
+
export const isRefetchDue = () => refetchDue && !isFresh()
|
|
258
|
+
```
|
|
259
|
+
|
|
260
|
+
The checker visits 22 states and proves two things. A Realtime refetch never hides the tab's pending optimistic write (`pendingWriteShown`, `neverBehindConfirmed`), and after a Realtime change the views eventually reach the stored revision (`@leadsTo isFresh`). The package runs it with one script:
|
|
261
|
+
|
|
262
|
+
```bash
|
|
263
|
+
cd packages/entities/lead
|
|
264
|
+
bun run check:models
|
|
265
|
+
```
|
|
266
|
+
|
|
267
|
+
With no file arguments, `tsprover` checks every `*.proof.ts` below the package and exits with code 1 on any failure. Run it when you change the Lead cache, a Lead write or Lead Realtime.
|
|
268
|
+
|
|
269
|
+
> **Note:** Pin `tsprover` exactly, as the template does with `"tsprover": "0.0.2"`. It
|
|
270
|
+
> and its checker are 0.0.x releases, so any release can change the Model
|
|
271
|
+
> language or the checker.
|
|
272
|
+
|
|
273
|
+
### When a flow deserves a Model
|
|
274
|
+
|
|
275
|
+
Most demo flows need no Model. Add a `*.proof.ts` only when the flow has one of these:
|
|
276
|
+
|
|
277
|
+
- two or more writers on shared state
|
|
278
|
+
- a background job or retry that touches client-written state
|
|
279
|
+
- recovery or resync
|
|
280
|
+
- a delivery guarantee such as "eventually delivered"
|
|
281
|
+
|
|
282
|
+
In the template, only the optimistic write against a Realtime refetch met that bar. Write one Model per service: the state the service owns becomes a top-level `let`, and each dependency stays outside and enters only as an action. The SDKs ship their own Models for the optimistic write coordinator, the session, the Realtime machine, change-log resync and topic grant tokens, so a demo inherits those guarantees and models only its own flows.
|
|
283
|
+
|
|
284
|
+
### Sensitivity Proofs
|
|
285
|
+
|
|
286
|
+
A Model that passes may simply never reach the dangerous case. A Sensitivity Proof shows that it does. You break the Model on purpose with one named Mutant, by removing one guard for an invariant or removing `@fair` from one action for a `@leadsTo` property. The checker must report a violation on the Mutant and pass again after you revert it. If the Mutant passes with the same state count, the guarded case is unreachable, and you widen the bounds or add an actor before you prove again.
|
|
287
|
+
|
|
288
|
+
A worked example: a client save and a background hash job both write `hashed`. Without the guard `if (job !== content) return;`, the invariant `hashMatchesContent` fails on the trace `save -> save -> hashJob`, because a stale job writes the hash of old content. With the guard, 11 distinct states pass. Both guards of the Lead Realtime Model have Sensitivity Proofs.
|
|
289
|
+
|
|
290
|
+
## Related pages
|
|
291
|
+
|
|
292
|
+
- [The solution template](index.md)
|
|
293
|
+
- [SDKs](../develop/sdks.md)
|
|
294
|
+
- [External applications](../develop/external-apps.md)
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@collective-intelligence/extensions-sdk-backend",
|
|
3
|
-
"version": "0.2.
|
|
3
|
+
"version": "0.2.8",
|
|
4
4
|
"description": "Framework-neutral executable contracts for Collective Intelligence extension packages, with an Effect adapter at ./effect.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"sideEffects": false,
|
|
@@ -44,7 +44,7 @@
|
|
|
44
44
|
"access": "public"
|
|
45
45
|
},
|
|
46
46
|
"dependencies": {
|
|
47
|
-
"@collective-intelligence/extensions-sdk": "0.2.
|
|
47
|
+
"@collective-intelligence/extensions-sdk": "0.2.7"
|
|
48
48
|
},
|
|
49
49
|
"peerDependencies": {
|
|
50
50
|
"effect": "^4.1.0"
|
|
@@ -55,7 +55,7 @@
|
|
|
55
55
|
"@types/bun": "1.4.2",
|
|
56
56
|
"@typescript-eslint/utils": "8.22.0",
|
|
57
57
|
"bun-types": "^1.4.2",
|
|
58
|
-
"effect": "4.1.
|
|
58
|
+
"effect": "4.1.2",
|
|
59
59
|
"eslint": "9.39.5",
|
|
60
60
|
"oxfmt": "0.66.0",
|
|
61
61
|
"oxlint": "1.90.0",
|
|
@@ -63,5 +63,5 @@
|
|
|
63
63
|
"typescript": "7.0.2",
|
|
64
64
|
"ultracite": "7.10.7"
|
|
65
65
|
},
|
|
66
|
-
"gitHead": "
|
|
66
|
+
"gitHead": "b2a4e19aabe96ba0b9285a911f59d40d1d111a4d"
|
|
67
67
|
}
|