@zackbart/connecta 0.24.4 → 0.26.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/AGENTS.md +43 -12
- package/CHANGELOG.md +524 -0
- package/README.md +32 -15
- package/SECURITY.md +6 -5
- package/bin/connecta.mjs +5 -1
- package/dist/activity.d.ts +10 -3
- package/dist/activity.js +10 -1
- package/dist/advertised-schema.js +53 -0
- package/dist/artifacts/connector.d.ts +55 -0
- package/dist/artifacts/connector.js +724 -0
- package/dist/artifacts/document.js +76 -0
- package/dist/artifacts/guide.js +140 -0
- package/dist/artifacts/html-scan.js +329 -0
- package/dist/artifacts/json.js +68 -0
- package/dist/artifacts/kv-store.d.ts +20 -0
- package/dist/artifacts/kv-store.js +140 -0
- package/dist/artifacts/markdown.js +449 -0
- package/dist/artifacts/module.d.ts +37 -0
- package/dist/artifacts/module.js +60 -0
- package/dist/artifacts/operations.d.ts +195 -0
- package/dist/artifacts/operations.js +944 -0
- package/dist/artifacts/refresh.d.ts +53 -0
- package/dist/artifacts/refresh.js +235 -0
- package/dist/artifacts/routes.js +194 -0
- package/dist/artifacts/types.d.ts +214 -0
- package/dist/artifacts/types.js +22 -0
- package/dist/artifacts/validate.d.ts +62 -0
- package/dist/artifacts/validate.js +652 -0
- package/dist/artifacts.d.ts +6 -0
- package/dist/artifacts.js +9 -0
- package/dist/auth/downstream-oauth.js +490 -106
- package/dist/call-admission.js +139 -113
- package/dist/catalog-service.js +152 -106
- package/dist/catalog.js +297 -38
- package/dist/concurrency.js +24 -23
- package/dist/connector-access.d.ts +18 -4
- package/dist/connector-access.js +65 -20
- package/dist/connector-caller.d.ts +8 -0
- package/dist/connector-caller.js +19 -0
- package/dist/connector-scope.js +6 -37
- package/dist/connectors/api.d.ts +2 -2
- package/dist/connectors/remote-mcp.d.ts +6 -5
- package/dist/connectors/remote-mcp.js +330 -240
- package/dist/credential-contract.d.ts +9 -0
- package/dist/credential-rules.js +2 -2
- package/dist/credentials.d.ts +6 -1
- package/dist/credentials.js +45 -3
- package/dist/errors.d.ts +16 -1
- package/dist/errors.js +28 -0
- package/dist/execute.js +745 -282
- package/dist/executor-admission.d.ts +11 -1
- package/dist/executor-admission.js +145 -63
- package/dist/executors/quickjs-runtime.js +4 -2
- package/dist/executors/quickjs.js +255 -212
- package/dist/fix-prompt.js +37 -0
- package/dist/index.d.ts +72 -7
- package/dist/index.js +281 -26
- package/dist/invocation.js +371 -286
- package/dist/meta-tools.js +213 -94
- package/dist/module-contracts.d.ts +21 -1
- package/dist/oauth-callback-outcome.js +94 -0
- package/dist/oauth-sealing.js +29 -0
- package/dist/operator-ui/fix-prompts.js +87 -0
- package/dist/operator-ui/generated.js +2 -2
- package/dist/operator-ui/model.d.ts +103 -1
- package/dist/operator-ui/setup-commands.js +53 -0
- package/dist/operator-ui/view.js +128 -3
- package/dist/program-source.js +321 -0
- package/dist/providers/notion.js +2 -2
- package/dist/providers/revenuecat.js +32 -5
- package/dist/providers/stripe.d.ts +3 -2
- package/dist/providers/vercel.js +1 -1
- package/dist/registry.d.ts +43 -7
- package/dist/registry.js +434 -288
- package/dist/resumable.js +1291 -0
- package/dist/routes/activity.js +74 -154
- package/dist/routes/actor-labels.js +114 -0
- package/dist/routes/credentials.js +196 -163
- package/dist/routes/mcp.js +228 -112
- package/dist/routes/oauth-management.js +73 -71
- package/dist/routes/oauth.js +44 -16
- package/dist/routes/operator.js +83 -0
- package/dist/routes/shared.d.ts +25 -1
- package/dist/routes/shared.js +20 -2
- package/dist/routes/ui-data.js +211 -0
- package/dist/routes/ui.js +76 -35
- package/dist/run-journal.js +347 -0
- package/dist/runtime/admission.js +43 -0
- package/dist/runtime/call-admission.js +67 -0
- package/dist/runtime/connector-scope.js +72 -0
- package/dist/runtime/run.js +174 -0
- package/dist/runtime/services.js +130 -0
- package/dist/runtime/shared-read.js +106 -0
- package/dist/runtime/storage.js +66 -0
- package/dist/server.js +137 -89
- package/dist/skills.js +28 -9
- package/dist/storage/file.js +39 -2
- package/dist/storage/memory.js +28 -16
- package/dist/timeout.js +39 -43
- package/dist/tool-safety.d.ts +27 -1
- package/dist/tool-safety.js +31 -0
- package/dist/types.d.ts +45 -7
- package/dist/ui.d.ts +25 -4
- package/dist/ui.js +77 -167
- package/dist/url-safety.js +157 -0
- package/dist/version.d.ts +1 -1
- package/dist/version.js +1 -1
- package/documentation/architecture.md +367 -16
- package/documentation/auth.md +163 -2
- package/documentation/code-mode.md +274 -56
- package/documentation/meta-tools.md +200 -44
- package/ethos.md +46 -47
- package/examples/worker/AGENTS.md +6 -0
- package/examples/worker/README.md +224 -14
- package/examples/worker/src/d1-activity-row.ts +14 -2
- package/examples/worker/src/d1-activity.ts +4 -2
- package/examples/worker/src/d1-storage.ts +116 -0
- package/examples/worker/src/index.ts +42 -7
- package/examples/worker/src/r2-artifact-blobs.ts +29 -0
- package/examples/worker/wrangler.jsonc +21 -5
- package/package.json +16 -4
- package/templates/node/AGENTS.md +9 -2
- package/templates/node/README.md +31 -3
- package/templates/node/package.json +1 -1
- package/templates/node/src/index.ts +27 -0
- package/dist/auth/downstream-oauth.d.ts +0 -200
- package/dist/catalog-fingerprint.d.ts +0 -19
- package/dist/catalog-limits.d.ts +0 -9
- package/dist/catalog-service.d.ts +0 -263
- package/dist/catalog.d.ts +0 -87
- package/dist/concurrency.d.ts +0 -3
- package/dist/connectors/guarded-fetch.d.ts +0 -96
- package/dist/credential-rules.d.ts +0 -85
- package/dist/execute.d.ts +0 -154
- package/dist/executor-result.d.ts +0 -10
- package/dist/executors/quickjs-child.d.ts +0 -1
- package/dist/executors/quickjs-protocol.d.ts +0 -67
- package/dist/favicon.d.ts +0 -1
- package/dist/identity.d.ts +0 -4
- package/dist/invocation.d.ts +0 -74
- package/dist/mcp-result.d.ts +0 -8
- package/dist/meta-tools.d.ts +0 -129
- package/dist/operator-ui/generated.d.ts +0 -2
- package/dist/operator-ui/view.d.ts +0 -188
- package/dist/result-shapes.d.ts +0 -13
- package/dist/retired-toolkits.d.ts +0 -2
- package/dist/routes/activity.d.ts +0 -2
- package/dist/routes/credentials.d.ts +0 -2
- package/dist/routes/mcp.d.ts +0 -10
- package/dist/routes/oauth-management.d.ts +0 -2
- package/dist/routes/oauth.d.ts +0 -2
- package/dist/routes/ui.d.ts +0 -2
- package/dist/server.d.ts +0 -9
- package/dist/skills.d.ts +0 -44
- package/dist/timeout.d.ts +0 -21
|
@@ -43,7 +43,7 @@ channel, `ctx.waitUntil`, threaded through `fetch(request, env, ctx)` — activi
|
|
|
43
43
|
writes use it, as does a stale-window catalog refresh, which owns a fresh scope
|
|
44
44
|
and deadline rather than carrying the inbound one past the request. And a fresh
|
|
45
45
|
`McpServer` per request is what makes the deployment stateless: no sessions, no
|
|
46
|
-
server push, no resumability, scope resolved rather than remembered.
|
|
46
|
+
server push, no stream resumability, scope resolved rather than remembered.
|
|
47
47
|
|
|
48
48
|
## Request lifecycle
|
|
49
49
|
|
|
@@ -61,7 +61,7 @@ top to bottom.
|
|
|
61
61
|
| 2 | MCP preflight | Allowed `OPTIONS` on `/mcp*`: 204 without admission or auth. |
|
|
62
62
|
| 2 | Other `OPTIONS` | Auth metadata first, otherwise compatibility CORS preflight. |
|
|
63
63
|
| 3 | `/.well-known/*` | Auth metadata, or 404. |
|
|
64
|
-
| 4 | `/health` | Open and payload-free: health, executor, admission, and deployment metadata, with drift as stable short hashes. |
|
|
64
|
+
| 4 | `/health` | Open and payload-free: health, executor, whether resumable writes are on, admission, and deployment metadata, with drift as stable short hashes. |
|
|
65
65
|
| 5 | `/oauth/callback/<connectorId>` | Core downstream OAuth completion, state and personal-ownership checked, independent of the UI. |
|
|
66
66
|
| 6 | `/mcp`, `/mcp/<pool>` | Admission, then auth, then a request-local MCP server. An undeclared pool, a refusing grant, and a throwing grant are one identical 404; see [pools](./auth.md#pools). |
|
|
67
67
|
| 7 | Other paths | 404. Custom HTTP routes belong to the deployment. |
|
|
@@ -77,7 +77,8 @@ An admitted non-preflight `/mcp` request then takes five steps in
|
|
|
77
77
|
|
|
78
78
|
1. **Admit.** One permit from the deployment-wide pool, taken before auth so an
|
|
79
79
|
unauthenticated flood costs a permit rather than a Clerk lookup, and held
|
|
80
|
-
until the response *body* completes,
|
|
80
|
+
until the response *body* completes, the caller leaves, or its configured
|
|
81
|
+
lifetime ends, not until the handler returns.
|
|
81
82
|
2. **Authorize.** Each `InboundAuth` provider's `authorize` in order, bearer
|
|
82
83
|
before interactive. First `ok` admits; if all fail, the last provider's
|
|
83
84
|
challenge is returned. No providers means open — development only, and it
|
|
@@ -93,21 +94,22 @@ An admitted non-preflight `/mcp` request then takes five steps in
|
|
|
93
94
|
5. **Serve.** Refuse `?toolkit=` with a 404 — the toolkits are gone
|
|
94
95
|
([#178](https://github.com/zackbart/connecta/issues/178)) but their URLs were
|
|
95
96
|
handed out, and retiring a scoping boundary into fail-open is worse than any
|
|
96
|
-
404 — then register the
|
|
97
|
+
404 — then register the eight meta-tools on a fresh `McpServer`
|
|
97
98
|
(`test/server.test.ts`, `test/code-first-surface.test.ts`).
|
|
98
99
|
|
|
99
100
|
## Layers below the meta-tools
|
|
100
101
|
|
|
101
|
-
The meta-tool handlers are thin. The work sits in
|
|
102
|
+
The meta-tool handlers are thin. The work sits in six modules the registry owns
|
|
102
103
|
or hands out, and a change usually belongs in exactly one of them:
|
|
103
104
|
|
|
104
105
|
| Module | Owns |
|
|
105
106
|
| --- | --- |
|
|
106
107
|
| `src/registry.ts` | The connector set, identity-scoped views, personal storage partitions, address resolution, catalog TTL/persistence/completeness, refresh single-flight, connector health, per-connector call limiters, and drift. Construction-time refusals live here. |
|
|
107
|
-
| `src/catalog-service.ts` | Request-local listing, search, and describe.
|
|
108
|
+
| `src/catalog-service.ts` | Request-local listing, search, and describe. Caches catalogs inside one request, fans discovery probes out under deadlines, and opts agent reads into the runtime's deferred catalog channel when one exists. |
|
|
108
109
|
| `src/invocation.ts` | One tool call: argument validation, call admission, one-attempt timeout, provider retry hints, result unwrapping, size capping, and the activity record. |
|
|
109
|
-
| `src/catalog.ts` | Ranking, description summarizing, and the compact schema
|
|
110
|
+
| `src/catalog.ts` | Ranking, description summarizing, and the compact and TypeScript schema renderers discovery shows. |
|
|
110
111
|
| `src/result-shapes.ts` | Bounded runtime-only inference and merging for output shapes learned from successful read-only calls whose providers declared none. |
|
|
112
|
+
| `src/resumable.ts`, `src/run-journal.ts` | Resumable writes: a program's host-call numbering, the write gate that pauses it, the journal a paused run becomes, the replay `resume_execution` plays from it, and the claim and write-ahead marks that keep an approved write to at most one send. |
|
|
111
113
|
|
|
112
114
|
`src/meta-tools.ts` and `src/execute.ts` are two front doors onto the same two
|
|
113
115
|
services, `CatalogService` and `InvocationService`. That is the point: a
|
|
@@ -128,6 +130,18 @@ settings win, and connecta warns the fallback was ignored. Invalid bounds throw
|
|
|
128
130
|
at construction, because a pool that quietly became unbounded is worse than a
|
|
129
131
|
deployment that refuses to boot. The queue is global FIFO across identities: a
|
|
130
132
|
capacity boundary, not tenant fairness, and one deployment serves one tenant.
|
|
133
|
+
`admission.requests.maxDurationMs` defaults to 300,000 ms and may be set to a
|
|
134
|
+
positive whole number up to 2,147,483,647 ms. Its clock starts when admission
|
|
135
|
+
grants a permit and covers authorization, tool execution, and response delivery.
|
|
136
|
+
The owning request aborts its derived signal at that deadline, including signals
|
|
137
|
+
passed to connector calls, and closes a live response. If workerd ends the
|
|
138
|
+
request without running that timer or cancellation, the controller reclaims
|
|
139
|
+
only the expired permit's scalar record on the next request (or a queued
|
|
140
|
+
request's own timer). It never reads the old request's signal or touches its
|
|
141
|
+
stream. Expired queued waiters are pruned before any handoff; every handed-off
|
|
142
|
+
permit has its own deadline even if the recipient never resumes. Lease identities
|
|
143
|
+
make an old late release harmless to a newer permit. Set the bound above the
|
|
144
|
+
longest admitted request the deployment expects to serve.
|
|
131
145
|
|
|
132
146
|
Call admission (`src/call-admission.ts`) answers what the envelope cannot see —
|
|
133
147
|
a connector's optional policy over its own `Connector.callTool` attempts,
|
|
@@ -138,15 +152,91 @@ rule refuses would charge a call that never reached the provider, the exact
|
|
|
138
152
|
accounting error a budget exists to prevent. Both layers are pinned by
|
|
139
153
|
`test/request-admission.test.ts` and `test/call-admission.test.ts`.
|
|
140
154
|
|
|
155
|
+
### Production Worker disconnects
|
|
156
|
+
|
|
157
|
+
The original production probe for [#573](https://github.com/zackbart/connecta/issues/573)
|
|
158
|
+
ran on 2026-09-25 with compatibility date `2025-01-01`, first with no flags and
|
|
159
|
+
then with `enable_request_signal`. It used Wrangler 4.114.0 and a Node 26.9.0
|
|
160
|
+
client. A raw Worker stream established the runtime behavior; a custom auth
|
|
161
|
+
provider returning a synthetic streaming 401 exercised Connecta's real admission
|
|
162
|
+
wrapper. Each case had its own Connecta instance with one request permit and no
|
|
163
|
+
queue. Health checks and subsequent MCP requests confirmed the same isolate ID.
|
|
164
|
+
The [recorded observations](https://github.com/zackbart/connecta/blob/a0ac904513fac735582b4bc9c817a48300134328/scripts/probes/worker-disconnect-2026-09-25.json)
|
|
165
|
+
contain all twelve cases, including served flags and client-end timing.
|
|
166
|
+
|
|
167
|
+
| Response and flag | Raw stream abort / cancel | Connecta source abort / cancel | Admission after response; next request |
|
|
168
|
+
| --- | --- | --- | --- |
|
|
169
|
+
| Heartbeats, no flag | neither | neither | active 1; 503 |
|
|
170
|
+
| Idle with a pending timer, no flag | neither | neither | active 1; 503 |
|
|
171
|
+
| Heartbeats, request signal enabled | abort only | both | active 0; 200 |
|
|
172
|
+
| Idle with a pending timer, request signal enabled | abort only | both | active 0; 200 |
|
|
173
|
+
| One chunk, then no timer or I/O, either configuration | neither | neither | active 1; 503 |
|
|
174
|
+
|
|
175
|
+
The last row is a different failure. The client observed the response end
|
|
176
|
+
*before* its planned disconnect, although the source never closed itself.
|
|
177
|
+
The runtime ended the response without running JavaScript cleanup. This does
|
|
178
|
+
not show a missing abort on a still-live request. The other cases remained open
|
|
179
|
+
until the client aborted. Admission was sampled 1.5 seconds later, followed by
|
|
180
|
+
an actual MCP request; this measures retained capacity, not an infinite leak.
|
|
181
|
+
|
|
182
|
+
Enable `enable_request_signal` when deploying on Workers: it let Connecta's
|
|
183
|
+
abort listener cancel the source and release admission for live responses in
|
|
184
|
+
this experiment. It does not solve runtime-ended streams. The admitted-request
|
|
185
|
+
bound above recovers capacity on the first later request at or after its
|
|
186
|
+
deadline, even if the old request's timer never ran. This is capacity recovery,
|
|
187
|
+
not proof that workerd cancelled an old stream. A still-live request reaches
|
|
188
|
+
the same hard deadline and is aborted in its own context.
|
|
189
|
+
Cloudflare documents the flag in its
|
|
190
|
+
[Request API](https://developers.cloudflare.com/workers/runtime-apis/request/).
|
|
191
|
+
|
|
192
|
+
A second production run on the same date tested the request bound at 2,000 ms.
|
|
193
|
+
All twelve raw-stream and MCP cases passed with both flag configurations. Live
|
|
194
|
+
MCP requests still rejected a competing request with 503 before the deadline;
|
|
195
|
+
the first MCP request after the deadline returned all eight tools with 200 in
|
|
196
|
+
the original isolate. Admission then reported zero active requests. Without the
|
|
197
|
+
flag, the source still recorded no abort or cancellation, so recovery did not
|
|
198
|
+
rely on runtime cleanup. The [recovery observations](https://github.com/zackbart/connecta/blob/32d8e94/scripts/probes/worker-disconnect-recovered-2026-09-25.json)
|
|
199
|
+
also retain two inconclusive queue-handoff cases: concurrent client connections
|
|
200
|
+
reached different isolates and could not establish an abandoned queued waiter.
|
|
201
|
+
A [second trial using HTTP/2](https://github.com/zackbart/connecta/blob/b174812/scripts/probes/worker-disconnect-h2-2026-09-25.json)
|
|
202
|
+
repeated the twelve passing core cases and verified queued cancellation with
|
|
203
|
+
request signals enabled. Even on one HTTP/2 connection, the no-signal queue case
|
|
204
|
+
crossed isolates and remained inconclusive. The Node and workerd regression
|
|
205
|
+
suites cover expired waiters, orphaned handoffs, and identity-safe late release
|
|
206
|
+
deterministically.
|
|
207
|
+
|
|
208
|
+
To repeat from a repository checkout, run
|
|
209
|
+
`node scripts/probes/worker-disconnect.mjs` with an authenticated Wrangler.
|
|
210
|
+
The script is not in the npm package. It deploys a disposable Worker containing
|
|
211
|
+
only synthetic data, verifies the served configuration, saves observations under `eval/results/`,
|
|
212
|
+
and deletes the Worker in `finally`. It uses no `waitUntil`, which would change
|
|
213
|
+
the lifetime being measured. The production runtime build number was not
|
|
214
|
+
exposed by the deployment; these findings are tied to the date and flags above.
|
|
215
|
+
|
|
141
216
|
## Storage, credentials, and connectors
|
|
142
217
|
|
|
143
218
|
`KVStorage` is `get`/`set`/`delete` with optional `list(prefix)`; core uses it for
|
|
144
219
|
connector state, catalogs, and result paging — a 15-minute TTL with one
|
|
145
220
|
runtime-wide accounting of stash bytes and entries, where a full stash returns the
|
|
146
221
|
successful call's preview and a paging-unavailable notice rather than a result id.
|
|
147
|
-
|
|
148
|
-
|
|
149
|
-
|
|
222
|
+
A second optional method, `compareAndSet(key, expected, next, options?)`, is an
|
|
223
|
+
atomic claim: `null` means absent (expired counts) on the way in and delete on
|
|
224
|
+
the way out. A successful write accepts the same optional `ttlSeconds` as
|
|
225
|
+
`set`. Resumable writes require it: of two `resume_execution` calls racing
|
|
226
|
+
for one paused run exactly one may win, and every write that run sends is
|
|
227
|
+
marked on its header first, so a read and a write standing in for the claim
|
|
228
|
+
could send an approved write twice. Omitted, `execute.resumableWrites` is on
|
|
229
|
+
exactly when the store has it, and asking for it over a store without it
|
|
230
|
+
refuses to construct. Downstream OAuth uses it where the store has it, so
|
|
231
|
+
resealing legacy plaintext and discarding a refused grant cannot overwrite a
|
|
232
|
+
consent that landed in between, and falls back to a read and a write where it
|
|
233
|
+
does not.
|
|
234
|
+
Adapters: `src/storage/memory.ts` and `src/storage/file.ts` (Node) both provide
|
|
235
|
+
it, and the namespaced views core hands connectors forward it only when the
|
|
236
|
+
underlying store has it. `examples/worker/` carries two more: Cloudflare KV,
|
|
237
|
+
eventually consistent and so declaring none, and a D1 store that provides it —
|
|
238
|
+
copyable reference source beside the D1 activity store, deliberately not an
|
|
239
|
+
importable subpath. The shared cases live in `test/storage-contract.ts`.
|
|
150
240
|
|
|
151
241
|
`src/credentials.ts` is the AES-GCM vault behind the root-exported
|
|
152
242
|
`CredentialVault` contract, selected through the `vault` slot. It binds connector
|
|
@@ -173,9 +263,9 @@ change who owns its auth.
|
|
|
173
263
|
|
|
174
264
|
## Optional deployment modules
|
|
175
265
|
|
|
176
|
-
`createConnecta` takes closed typed `ui`, `vault`, and `
|
|
177
|
-
factories at `/ui`, `/credentials`,
|
|
178
|
-
`/auth/bearer`. Root exports the contracts, never the implementations, and there
|
|
266
|
+
`createConnecta` takes closed typed `ui`, `vault`, `activity`, and `artifacts`
|
|
267
|
+
slots, with factories at `/ui`, `/credentials`, `/activity`, and `/artifacts`
|
|
268
|
+
and bearer auth at `/auth/bearer`. Root exports the contracts, never the implementations, and there
|
|
179
269
|
is no module array, runtime registration, or plugin lifecycle. Core keeps
|
|
180
270
|
discovery, the executor contract, invocation, permissions, and OAuth callback
|
|
181
271
|
verification; an omitted module contributes no runtime work at all.
|
|
@@ -187,8 +277,15 @@ material code explicitly permitted; it never edits the connector set, catalog,
|
|
|
187
277
|
annotations, scopes, or permission rules. Two invariants shape it: a status read
|
|
188
278
|
never starts authorization, since OAuth begins with an explicit authorized POST,
|
|
189
279
|
and each lazy details request owns a bounded downstream scope, so one failing
|
|
190
|
-
provider leaves the other connections usable.
|
|
191
|
-
|
|
280
|
+
provider leaves the other connections usable. A connector's status message
|
|
281
|
+
never reaches the page: it can quote a downstream error body, and that body can
|
|
282
|
+
quote the secret it rejected, so the details payload carries only a classified
|
|
283
|
+
`problem` — which picks fixed on-screen copy and a fixed fix prompt — and the raw
|
|
284
|
+
text goes to the server log. The OAuth and credential Test notices keep the
|
|
285
|
+
same rule: those routes answer a downstream's failure in fixed words (a Test
|
|
286
|
+
answers only `{ ok }`), log its text, and the page picks a sentence by outcome
|
|
287
|
+
without ever rendering what the server sent. Credential handoff URLs exist only while the UI is
|
|
288
|
+
mounted; OAuth callbacks never need it.
|
|
192
289
|
|
|
193
290
|
Its appearance is one token layer. `src/operator-ui/browser.css` resolves every
|
|
194
291
|
color, radius, and font through a custom property and mixes the rest from those
|
|
@@ -199,6 +296,86 @@ and an unvalidated value would be CSS injection. The dark palette is the same
|
|
|
199
296
|
tokens under `prefers-color-scheme`; `colorScheme` pins one with a `data-scheme`
|
|
200
297
|
attribute on the page.
|
|
201
298
|
|
|
299
|
+
The artifacts module — `artifacts()` from `/artifacts`, implemented in
|
|
300
|
+
`src/artifacts/` — is the one module that contributes a connector. Core
|
|
301
|
+
appends its prebuilt `artifacts` connector to the configured set, so its reads
|
|
302
|
+
and writes take the same catalog, invocation, admission, and activity paths as
|
|
303
|
+
any other connector; the slot binds its optional refresh runner once, after the
|
|
304
|
+
registry and executor exist, and a deployment
|
|
305
|
+
without it has no connector, no guide, and no artifact storage traffic. Its
|
|
306
|
+
writes are not read-only anywhere — `call_tool` refuses them and discovery
|
|
307
|
+
lists them approval-required — but they skip approval inside programs through
|
|
308
|
+
the connector's own `approval: "never"`, which `execute.approval` switches off,
|
|
309
|
+
because every write is a new immutable version anyone can roll back. The
|
|
310
|
+
module needs `publicUrl`: the links it hands out are shared, so they must never
|
|
311
|
+
come from a request's `Host`.
|
|
312
|
+
|
|
313
|
+
An artifact can version an `execute_code` program and target data document with
|
|
314
|
+
`set_refresh`, on a manual, daily, or weekly schedule. The deployment calls
|
|
315
|
+
`artifacts.runDue()` from a Node timer or Worker `scheduled()` handler; core
|
|
316
|
+
starts no background jobs. One tick scans at most 1,000 heads and starts at
|
|
317
|
+
most 10 due runs. Every run uses the configured executor, host-call budget,
|
|
318
|
+
and watchdog, with a registry view containing only shared connectors. The
|
|
319
|
+
owner admitted to `set_refresh` is stored with the program. Every run rechecks
|
|
320
|
+
that identity's current `connectorAccess` and pool grant, including continued
|
|
321
|
+
access to `artifacts.set_refresh`, then intersects those grants with shared
|
|
322
|
+
connectors. Revoking either grant stops later runs.
|
|
323
|
+
Resumable writes and approval exemptions are off. A refused call fails the
|
|
324
|
+
entire refresh even if its program catches the error. The head CAS claims one
|
|
325
|
+
run per artifact; the data commit checks the claim ID and program version in
|
|
326
|
+
the same CAS. The claim deadline starts before executor admission and aborts
|
|
327
|
+
a queued or active play when it expires. An expired old run cannot publish over
|
|
328
|
+
a newer claim. Run history retains 50 records with bounded logs. A failure leaves the
|
|
329
|
+
last good data intact and marks the artifact stale. Library and viewer show
|
|
330
|
+
only outcome enums and times, never downstream messages or logs.
|
|
331
|
+
|
|
332
|
+
With the operator UI mounted, `/artifacts` lists the team pages and
|
|
333
|
+
`/artifacts/<id>` opens one. The shell contains no page data. It reads the
|
|
334
|
+
artifact through an authenticated JSON route with the operator UI's inbound
|
|
335
|
+
identity and connector visibility, then passes the document into a sandboxed
|
|
336
|
+
frame. Snapshot links pin the view and document versions. The frame's response
|
|
337
|
+
CSP gives scripts an opaque origin, refuses fetch, XHR, images and other
|
|
338
|
+
unlisted requests. External script, style, and font origins default to none;
|
|
339
|
+
a deployment opts in to each one it trusts with page data. The shell tightens
|
|
340
|
+
its own `frame-src` to `'none'` after the fixed bootstrap loads and before it
|
|
341
|
+
hands over the page,
|
|
342
|
+
so a page script cannot navigate even its own frame to send data in a URL.
|
|
343
|
+
This browser boundary is covered by Chromium tests, including a
|
|
344
|
+
`document.open/write` rewrite.
|
|
345
|
+
|
|
346
|
+
`artifactOrigin` optionally names a distinct HTTPS origin for these pages.
|
|
347
|
+
The main host redirects every `/artifacts` path to it; the artifact host
|
|
348
|
+
answers every other path, including `/mcp` and operator APIs, with 404. The
|
|
349
|
+
artifact shell still uses the same inbound auth providers and checks the
|
|
350
|
+
identity on each data read. The bare shell and fixed frame bootstrap carry no
|
|
351
|
+
artifact data, so a visitor can reach sign-in before an authenticated read.
|
|
352
|
+
Bearer tokens in browser localStorage belong to one origin: signing in on the
|
|
353
|
+
main host does not sign in on the artifact host. A redirected visitor enters
|
|
354
|
+
the token again there; the redirect carries no credential. Cookie or session
|
|
355
|
+
auth likewise depends on the provider admitting that separate origin.
|
|
356
|
+
|
|
357
|
+
Storage is an `ArtifactStore`, and `kvArtifactStore` is the reference: over any
|
|
358
|
+
`KVStorage` with `compareAndSet` and `list`, one head record per artifact is
|
|
359
|
+
the only key ever compared-and-set, earlier versions sit beside it immutable,
|
|
360
|
+
and bodies are content-addressed — in the key-value store, or in R2 through the
|
|
361
|
+
Worker example's `r2-artifact-blobs.ts`. Workers KV is refused at construction,
|
|
362
|
+
because a write that cannot swap its head can lose a teammate's edit. Every
|
|
363
|
+
rule lives once, in `src/artifacts/operations.ts`: a write checks its base
|
|
364
|
+
against the one stream it touches, so of two writes from the same base exactly
|
|
365
|
+
one wins and the other is a `conflict` carrying a bounded `current` map, while
|
|
366
|
+
a head that moved for an unrelated reason is re-read and retried rather than
|
|
367
|
+
reported. Patches match exactly once or change nothing, rollback is a new
|
|
368
|
+
version reusing an old body, and the limits are checked on every write.
|
|
369
|
+
Validation is a hand-written tokenizer and CommonMark subset rather than a
|
|
370
|
+
dependency, because it is a lint: the page's CSP is the boundary.
|
|
371
|
+
|
|
372
|
+
Who made a version is the one fact a built-in connector needs that
|
|
373
|
+
`ConnectorContext` deliberately does not carry. `src/connector-caller.ts`
|
|
374
|
+
attaches the admitted identity beside the context on the request's scoped
|
|
375
|
+
view, the way the OAuth sealer rides beside it — readable only by in-repo
|
|
376
|
+
code, set from the authorization and never from arguments
|
|
377
|
+
(`test/identity-scope.test.ts`, `test/artifacts-connector.test.ts`).
|
|
378
|
+
|
|
202
379
|
## Import-graph purity
|
|
203
380
|
|
|
204
381
|
Nothing reachable from `src/index.ts` may import a `node:` builtin, so the same
|
|
@@ -218,6 +395,180 @@ boundary in the published tarball. The failure mode is not theoretical: one
|
|
|
218
395
|
convenience import of `node:crypto` in a shared helper stops the whole Worker
|
|
219
396
|
shape from building, in someone else's repository rather than this one.
|
|
220
397
|
|
|
398
|
+
## Effect inside
|
|
399
|
+
|
|
400
|
+
The core runs on Effect v4; no API a deployment touches does. `createConnecta`,
|
|
401
|
+
`remoteMcp()`, `api()`, the `Connector` contract, and every shipped `.d.ts`
|
|
402
|
+
are Promise-shaped and name no Effect type, so a connector author never meets a
|
|
403
|
+
second async paradigm. The reason for the rewrite is the first section of this
|
|
404
|
+
guide: the bugs here are lifetime bugs, and Effect makes a lifetime a value —
|
|
405
|
+
a Scope that closes on every exit, an interrupt that reaches whatever a fiber
|
|
406
|
+
is waiting on, a Deferred that one party completes and any number join. The
|
|
407
|
+
conversion found and closed a dozen of them, each with a test that fails on the
|
|
408
|
+
code before it: permits and leases held after their caller left, waits with no
|
|
409
|
+
bound, and races between requests.
|
|
410
|
+
|
|
411
|
+
### Shell and core
|
|
412
|
+
|
|
413
|
+
Each converted module is a shell and a core. The shell keeps its exported class
|
|
414
|
+
or function exactly — same members, same error classes, same `.d.ts`, private
|
|
415
|
+
member names included, because TypeScript ships those too. The core is an
|
|
416
|
+
Effect program behind it. In a published class it lives in a `static` block,
|
|
417
|
+
the one place that can read private state without adding a member to the
|
|
418
|
+
declarations; elsewhere it lives in an unexported function or in
|
|
419
|
+
`src/runtime/`, which no `exports` target reaches. Errors keep their classes
|
|
420
|
+
(`ConnectorCallError`, `ExecutorAdmissionError`, `CallAdmissionError`) because
|
|
421
|
+
callers and tests check `instanceof`, `name`, and `message`.
|
|
422
|
+
|
|
423
|
+
Effect callers skip the Promise and take the program: `admit` and
|
|
424
|
+
`acquireScoped` for request admission (`src/runtime/admission.ts`), `admitCall`
|
|
425
|
+
and `acquireCallScoped` for call admission, `closeScope` and `closeScopeOnExit`
|
|
426
|
+
for a connector scope, `withDeadlineEffect` where Promise code has
|
|
427
|
+
`withDeadline`. The build prunes every declaration no `exports` target reaches,
|
|
428
|
+
and `npm run check:declarations` fails if what remains names an Effect type.
|
|
429
|
+
|
|
430
|
+
### One runner
|
|
431
|
+
|
|
432
|
+
`src/runtime/run.ts` is the only place a fiber starts, and
|
|
433
|
+
`test/purity.test.ts` fails if any other file calls `Effect.run*`, `runFork`,
|
|
434
|
+
`forkDaemon`, or `ManagedRuntime.make`. It exports two ways out:
|
|
435
|
+
|
|
436
|
+
- **`runEdge(effect, { signal, runtime? })`** runs an effect at a Promise
|
|
437
|
+
boundary. It rethrows the original failure or defect, never a wrapper. It
|
|
438
|
+
turns an interrupt into the caller's `signal.reason` rather than Effect's
|
|
439
|
+
"All fibers interrupted without error". And it runs every fiber on a
|
|
440
|
+
scheduler that yields through microtasks: Effect's default yields through
|
|
441
|
+
`setImmediate`, which vitest's fake timers freeze and Workers never had. The
|
|
442
|
+
price is that a fiber never yields to I/O, so CPU-heavy work stays out of
|
|
443
|
+
Effect loops.
|
|
444
|
+
- **`detach(effect, ctx)`** is the only fire-and-forget, and it hands the work
|
|
445
|
+
to `ctx.waitUntil` when there is one.
|
|
446
|
+
|
|
447
|
+
Beside them sit `withDeadlineEffect`, which aborts the operation's own signal
|
|
448
|
+
with the labelled timeout error *before* interrupting it, so work that honors
|
|
449
|
+
the signal sees the same reason the caller does, and `fromSignal`, which turns
|
|
450
|
+
an `AbortSignal` into a failure to race against. Nothing runs Effect at module
|
|
451
|
+
scope, where a Worker may not start work, and nothing logs through
|
|
452
|
+
`Effect.log`: logging goes through the configured `Logger`, which is what
|
|
453
|
+
honors `logger: "silent"`.
|
|
454
|
+
|
|
455
|
+
Each Connecta also gets a runtime of its own (`src/runtime/services.ts`):
|
|
456
|
+
`Storage`, `Vault`, `ActivityRecorder` (one that records nothing when the module
|
|
457
|
+
is omitted), `Logger`, and `ResolvedConfig`. Creating it builds nothing, since
|
|
458
|
+
a Worker constructs its Connecta at global scope; the first run that needs the
|
|
459
|
+
services builds them, and `close()` disposes the runtime last. Two things
|
|
460
|
+
deliberately do not run on it. The request pipeline runs on no runtime, so
|
|
461
|
+
`/health` and a closed deployment's 503 keep answering after `close()`. And a
|
|
462
|
+
registry provides its own storage and logger to its programs
|
|
463
|
+
(`runOnPartition` in `src/runtime/storage.ts`), because a personal registry's
|
|
464
|
+
storage is the root's namespaced to its principal and the runtime's `Storage`
|
|
465
|
+
would be another partition's.
|
|
466
|
+
|
|
467
|
+
### What runs on Effect
|
|
468
|
+
|
|
469
|
+
| Area | Shape |
|
|
470
|
+
| --- | --- |
|
|
471
|
+
| Requests (`src/server.ts`, `src/routes/mcp.ts`, `src/routes/oauth.ts`) | One fiber per `fetch`, tied to `request.signal`. An `/mcp` request's admission permit and every `McpServer` it builds live in a Scope the response carries out, closed when the body ends, fails, or is cancelled, when the signal aborts, when its admitted-request lifetime ends in that request, or at once if the handler fails first. If workerd runs none of those callbacks, the next request reclaims only the expired admission record; it cannot close another request's Scope. The OAuth callback is uninterruptible: a single-use code's exchange and catalog invalidation are one commitment. |
|
|
472
|
+
| Admission (`executor-admission.ts`, `call-admission.ts`) | Queued waiters are Deferreds settled by whoever removes them from the queue; a wait is one flat race of grant, Clock timeout, and signal. The uncontended path stays synchronous. Two controllers, as [#453](https://github.com/zackbart/connecta/issues/453) requires. |
|
|
473
|
+
| One tool call (`invocation.ts`) | One fiber: resolution, the read-only and schema refusals, admission, and the downstream attempt sit under a single `withDeadlineEffect`, whose expiry interrupts the call wherever it is. The permit is an `acquireRelease`; the connector call is `Effect.tryPromise` over the unchanged `Connector`. |
|
|
474
|
+
| `execute_code` (`execute.ts`) | One fiber whose Scope owns the run's signal and executor lease, so a result, a throw, the watchdog, and cancellation all release the lease and abort the signal the same way. The executor's `acquire()` and `execute()` stay Promises raced against the signal and `execute.watchdogMs`. Each guest host call is a fiber of its own. |
|
|
475
|
+
| Resumable writes (`resumable.ts`) | A play's stop is a Deferred its write gate completes, raced against the executor like the watchdog; the gate for call n awaits the decision Deferreds of every lower-numbered call; header writes are one compare-and-set at a time on a per-play turn. A paused run holds nothing request-bound — its journal is data, and the next play is a new request's. |
|
|
476
|
+
| Discovery (`catalog-service.ts`) | A request-scoped cache: one shared read per connector (`runtime/shared-read.ts`), settled by the read itself and carrying its own signal and the probe timeout whichever asker starts it. Each asker waits under its own deadline and signal, so one that times out or is cancelled fails alone, and the read is cancelled only once every asker has gone. Fan-out is `Effect.forEach` under the discovery concurrency. |
|
|
477
|
+
| Registry (`registry.ts`) | Catalog persistence and the result stash are programs over `Storage`. A refresh flight is a Deferred its publishing request completes, bounded by its owner's deadline (the default probe timeout when it has none); persisted-catalog writes take per-connector turns, each a Deferred its own request completes. Same-request loads share one read the way discovery's do. |
|
|
478
|
+
| Remote MCP (`connectors/remote-mcp.ts`) | Each request scope's state holds a Scope, each connection is a lease forked from it, and a connect in flight is a Deferred carrying the client it connected. Closing a session and the transport are each bounded to a second. |
|
|
479
|
+
| Downstream OAuth (`auth/downstream-oauth.ts`) | A refresh flight is a Deferred; the owner's redemption is a fiber its abort interrupts, and committing an answer that already exists is uninterruptible. |
|
|
480
|
+
| Operator and activity data (`routes/operator.ts`) | Each JSON route is one program run by `serveOperator` behind the Promise `handle()`. Reads run under the request's signal; writes do not, so a vault write or OAuth disconnect that started reaches its cache invalidation. |
|
|
481
|
+
| QuickJS pool (`executors/quickjs.ts`, Node only) | Each child is a scoped resource whose release sends SIGTERM, then SIGKILL after a second; crash respawn backoff is a `Schedule`. |
|
|
482
|
+
|
|
483
|
+
### What stays plain
|
|
484
|
+
|
|
485
|
+
- **Everything a deployment writes against.** The `Connector` contract,
|
|
486
|
+
`api()`, the providers, custom connectors, storage adapters, and the vault
|
|
487
|
+
are Promises. A connector is called with `Effect.tryPromise`, never asked to
|
|
488
|
+
return an Effect.
|
|
489
|
+
- **Inbound authorization.** `authorize` in `src/routes/shared.ts` ships in
|
|
490
|
+
the declarations and in the `./activity` bundle; converting it cost 32 KB
|
|
491
|
+
gzip there for nothing a caller could see.
|
|
492
|
+
- **Pure synchronous code.** Ranking, schema rendering, classification,
|
|
493
|
+
`validateToolInput`, the host-call budget, the emit collector, failure
|
|
494
|
+
framing, and result shaping. The microtask scheduler never yields to I/O, so
|
|
495
|
+
wrapping them would buy nothing and could starve a request.
|
|
496
|
+
- **The MCP edges.** `@modelcontextprotocol/server` upward and the SDK client
|
|
497
|
+
downstream; Effect's own `McpServer` is not used.
|
|
498
|
+
- **The Node and browser leaves.** `node.ts`, `fileStorage`, the QuickJS
|
|
499
|
+
child, runtime, and protocol, and the operator UI's browser app.
|
|
500
|
+
|
|
501
|
+
### Across requests
|
|
502
|
+
|
|
503
|
+
Work shared across requests meets only through a Deferred that the owning
|
|
504
|
+
request completes, never through a fiber that outlives its request. Two
|
|
505
|
+
workerd rules follow, and both have bitten.
|
|
506
|
+
|
|
507
|
+
**Waiting on a shared Deferred is its own edge.** Deferred resumes a waiting
|
|
508
|
+
fiber synchronously, inside the call that completed it, which on Workers means
|
|
509
|
+
inside the completing request's I/O context. A waiter that goes on to touch its
|
|
510
|
+
own body, transport, or storage there fails with "Cannot perform I/O on behalf
|
|
511
|
+
of a different request", or hangs until its deadline. So the wait runs through
|
|
512
|
+
`runEdge`, and the waiter does its own I/O after awaiting that promise, which
|
|
513
|
+
workerd resumes in the waiter's request as it does any promise resolved from
|
|
514
|
+
another. The same holds one level out for every queue: a queued call, a queued
|
|
515
|
+
`execute_code`, and a queued `/mcp` request are each handed their permit from
|
|
516
|
+
inside the request that released one, so each awaits the controller's
|
|
517
|
+
Promise-shaped `acquire` rather than yielding its Effect program, and does its
|
|
518
|
+
own work only once that promise resolves. Effect's `Semaphore` is not used
|
|
519
|
+
across requests for the same reason, and because a same-tick newcomer can take
|
|
520
|
+
a released permit ahead of a waiter that queued earlier.
|
|
521
|
+
|
|
522
|
+
**Never read another request's signal.** On workerd, reading `aborted` or
|
|
523
|
+
`reason`, or calling `abort()`, on a signal that belongs to another request
|
|
524
|
+
throws; adding and removing listeners does not. Code that runs from another
|
|
525
|
+
request's completion — a queue handing out a permit, `close()` — must not look
|
|
526
|
+
at a waiter's signal. A waiter whose signal aborts leaves the queue from its
|
|
527
|
+
own listener, in its own request.
|
|
528
|
+
|
|
529
|
+
One corollary is easy to break by tidying. Workerd cancels a request as hung
|
|
530
|
+
when its only pending work is waiting on another request and it has no timer or
|
|
531
|
+
I/O of its own. Connecta's deadline timers are what keep a waiting request
|
|
532
|
+
alive and bounded, so no wait may lose its timer.
|
|
533
|
+
|
|
534
|
+
**A shared flight has a lifetime of its own.** A catalog refresh flight runs in
|
|
535
|
+
its owner's request, under its owner's scope and signal, and lives only as long
|
|
536
|
+
as its owner's deadline. Past that the owner may have answered and gone, taking
|
|
537
|
+
its I/O with it on workerd, and a connector that ignores its abort signal would
|
|
538
|
+
otherwise hold every later reader on a flight nothing will ever settle. So each
|
|
539
|
+
joiner waits on its own timer, set to the flight's bound: when it fires, the
|
|
540
|
+
joiner abandons the flight and makes a fresh attempt, and a reader that arrives
|
|
541
|
+
after the bound starts one without joining. Joiners also leave when the owner
|
|
542
|
+
fails for a reason that was only its own, such as its cancellation, rather than
|
|
543
|
+
inheriting it. A flight's result enters the caches only while the flight is
|
|
544
|
+
current, checked at the same points as the catalog generation, so a stuck
|
|
545
|
+
listing that lands late is never cached or persisted and never overwrites the
|
|
546
|
+
fresh attempt's.
|
|
547
|
+
|
|
548
|
+
### Why not Schema or HttpApi
|
|
549
|
+
|
|
550
|
+
Both were measured and both lost. `effect` is imported through its barrel,
|
|
551
|
+
which defeats tree-shaking: the first conversion step measured about +20 ms
|
|
552
|
+
of Worker cold start for it, and that cost was accepted over deep imports at
|
|
553
|
+
every call site. The catch is that a module the barrel reaches is paid for in
|
|
554
|
+
full, however little of it is used. Effect Schema for config validation and the
|
|
555
|
+
meta-tool inputs came to about +62 KB gzip and +13 ms of cold start in the
|
|
556
|
+
root, for a validator no clearer than the hand-written `CONFIG_SCHEMA`; the
|
|
557
|
+
meta-tool inputs stay zod, which the MCP SDK bundles regardless. `HttpApi` for
|
|
558
|
+
the operator and activity routes cost about +100 KB gzip on `./ui` and +130 KB
|
|
559
|
+
on `./activity`, and matching the wire format meant opting out of most of what
|
|
560
|
+
it does: unowned paths fall through rather than 404, a wrong method is a JSON
|
|
561
|
+
405, and the content-type and size checks run before a body is read. The root
|
|
562
|
+
entry may import only `effect` itself (`test/purity.test.ts`); nothing in
|
|
563
|
+
`src/` uses `effect/unstable/*`, which would have to stay behind a subpath.
|
|
564
|
+
|
|
565
|
+
The core's cost is recorded rather than guessed. Against 0.24.4 the root entry
|
|
566
|
+
grew from 235,346 to 279,526 bytes gzip, the Worker example from 263,949 to
|
|
567
|
+
314,336, and each provider by about 36 KB, since each reaches Effect through
|
|
568
|
+
`remoteMcp()`; `npm run check:bundle` caps every entry against
|
|
569
|
+
`scripts/bundle-budget.json`. An install carries about 53 MB of `effect`, with
|
|
570
|
+
no dependencies of its own.
|
|
571
|
+
|
|
221
572
|
## Where else to look
|
|
222
573
|
|
|
223
574
|
Beyond the modules already named:
|
|
@@ -250,7 +601,7 @@ compiling and configuring the real thing.
|
|
|
250
601
|
`OPTIONS` opts it into CORS preflight; reordering admission after auth makes
|
|
251
602
|
the cheapest possible attack the most expensive request.
|
|
252
603
|
- **`close()` is idempotent and ordered.** Both admission pools, then the
|
|
253
|
-
connector limiters, then the executor; Node's `listen()` calls it on
|
|
604
|
+
connector limiters, then the executor, then the Connecta's runtime; Node's `listen()` calls it on
|
|
254
605
|
SIGTERM/SIGINT.
|
|
255
606
|
- **Structural mistakes throw at construction.** A duplicate connector id, an
|
|
256
607
|
invalid admission rule, the removed `accessTokens` option, a missing executor:
|