@synapsor/runner 1.0.0 → 1.1.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +37 -1
- package/README.md +45 -3
- package/THREAT_MODEL.md +35 -2
- package/dist/cli.d.ts.map +1 -1
- package/dist/local-ui.d.ts +3 -0
- package/dist/local-ui.d.ts.map +1 -1
- package/dist/runner.mjs +3935 -863
- package/docs/README.md +16 -3
- package/docs/capability-authoring.md +2 -1
- package/docs/current-scope.md +5 -1
- package/docs/dsl-reference.md +7 -0
- package/docs/http-mcp.md +4 -1
- package/docs/limitations.md +20 -6
- package/docs/migrating-to-synapsor-spec.md +21 -0
- package/docs/oss-vs-cloud.md +1 -1
- package/docs/production.md +50 -8
- package/docs/release-notes.md +19 -1
- package/docs/release-policy.md +21 -0
- package/docs/runner-config-reference.md +101 -3
- package/docs/running-a-runner-fleet.md +349 -0
- package/docs/security-boundary.md +7 -0
- package/examples/runner-fleet/Dockerfile +11 -0
- package/examples/runner-fleet/README.md +52 -0
- package/examples/runner-fleet/docker-compose.yml +76 -0
- package/examples/runner-fleet/mint-dev-token.mjs +19 -0
- package/examples/runner-fleet/seed/mysql.sql +20 -0
- package/examples/runner-fleet/seed/postgres.sql +73 -0
- package/examples/runner-fleet/synapsor.runner.json +105 -0
- package/fixtures/protocol/MANIFEST.json +1 -1
- package/package.json +3 -1
- package/schemas/change-set.v1.schema.json +2 -1
- package/schemas/synapsor.runner.schema.json +115 -7
|
@@ -0,0 +1,349 @@
|
|
|
1
|
+
# Running A Small Runner Fleet
|
|
2
|
+
|
|
3
|
+
Synapsor Runner 1.1 supports a bounded small-fleet deployment. It does not
|
|
4
|
+
turn the OSS package into Synapsor Cloud and it does not promise an unbounded,
|
|
5
|
+
multi-region, or SLA-backed control plane.
|
|
6
|
+
|
|
7
|
+
```text
|
|
8
|
+
TLS load balancer
|
|
9
|
+
-> N stateless Streamable HTTP Runners
|
|
10
|
+
-> shared Postgres runtime ledger
|
|
11
|
+
-> bounded Postgres/MySQL source pools
|
|
12
|
+
-> supervised writeback workers
|
|
13
|
+
-> restricted source writer or app-owned executor
|
|
14
|
+
```
|
|
15
|
+
|
|
16
|
+
The model still receives only reviewed semantic tools. Approval, apply,
|
|
17
|
+
operator credentials, database credentials, and shared-ledger authority remain
|
|
18
|
+
outside MCP.
|
|
19
|
+
|
|
20
|
+
## Audit Before Deployment
|
|
21
|
+
|
|
22
|
+
Audit the model-facing surface before connecting a production-like database:
|
|
23
|
+
|
|
24
|
+
```bash
|
|
25
|
+
npx -y @synapsor/runner audit --example dangerous-db-mcp
|
|
26
|
+
synapsor-runner audit ./tools-list.json
|
|
27
|
+
synapsor-runner audit https://mcp.example.com --bearer-env MCP_AUDIT_TOKEN
|
|
28
|
+
```
|
|
29
|
+
|
|
30
|
+
Treat generic SQL/query tools, model-controlled tenant/principal inputs, and
|
|
31
|
+
model-facing approval/apply tools as blockers. Audit is a static review, not a
|
|
32
|
+
security proof; follow it with `tools preview`, `doctor`, and the live fleet
|
|
33
|
+
verification.
|
|
34
|
+
|
|
35
|
+
## What 1.1 Guarantees
|
|
36
|
+
|
|
37
|
+
With `storage.shared_postgres.mode = "runtime_store"`:
|
|
38
|
+
|
|
39
|
+
- each mutation holds one schema-scoped Postgres advisory lock;
|
|
40
|
+
- concurrent schema migration is serialized under that lock;
|
|
41
|
+
- two instances cannot create two active proposals for the same canonical
|
|
42
|
+
tenant/capability/object tuple;
|
|
43
|
+
- distinct verified reviewers append decisions without losing one another;
|
|
44
|
+
- one worker claim/effect completes, conflicts, retries, or enters dead letter;
|
|
45
|
+
- direct SQL and app-owned handler retries use durable source-side receipts;
|
|
46
|
+
- fixed-window rate limits are atomic across instances and keyed from verified
|
|
47
|
+
tenant context plus reviewed capability;
|
|
48
|
+
- CLI and local UI review reads use the same shared queue as MCP serving.
|
|
49
|
+
|
|
50
|
+
The current implementation restores the shared ledger into a transient SQLite
|
|
51
|
+
store while holding the advisory lock, performs one bounded operation, then
|
|
52
|
+
syncs the result. `max_entries` defaults to 10,000 and may be set from 100 to
|
|
53
|
+
100,000. Runner fails closed with `POSTGRES_RUNTIME_STORE_CAPACITY_EXCEEDED`
|
|
54
|
+
rather than copying an unbounded ledger. This serialized design is appropriate
|
|
55
|
+
for a small self-hosted fleet, not high-throughput horizontal scale.
|
|
56
|
+
|
|
57
|
+
## Verified Local Topology
|
|
58
|
+
|
|
59
|
+
The repository includes a synthetic Compose fixture:
|
|
60
|
+
|
|
61
|
+
```bash
|
|
62
|
+
docker compose --profile fleet \
|
|
63
|
+
-f examples/runner-fleet/docker-compose.yml up --build -d --wait
|
|
64
|
+
```
|
|
65
|
+
|
|
66
|
+
The stronger verification command is:
|
|
67
|
+
|
|
68
|
+
```bash
|
|
69
|
+
corepack pnpm test:fleet
|
|
70
|
+
```
|
|
71
|
+
|
|
72
|
+
Prerequisites: Node 22, pnpm through Corepack, Docker Engine, and Docker
|
|
73
|
+
Compose. It needs no AWS, Cloud account, external identity provider, or
|
|
74
|
+
internet service after container images and dependencies are available.
|
|
75
|
+
|
|
76
|
+
Latest verified result: **2026-07-12 (America/Los_Angeles), PASS**. Assertions:
|
|
77
|
+
|
|
78
|
+
```text
|
|
79
|
+
two claim-bound Runners share one bounded Postgres ledger
|
|
80
|
+
cross-tenant reads are denied
|
|
81
|
+
one active proposal survives concurrent creation
|
|
82
|
+
rate limits hold across instances
|
|
83
|
+
two verified reviewers satisfy a 2-person quorum
|
|
84
|
+
apply is blocked at 1/2 and succeeds at 2/2
|
|
85
|
+
simultaneous reviewer processes preserve both decisions
|
|
86
|
+
two competing workers create exactly one source effect
|
|
87
|
+
worker termination before write recovers without an effect
|
|
88
|
+
worker termination during an open write transaction rolls back and recovers
|
|
89
|
+
worker termination after commit recovers as already-applied
|
|
90
|
+
source-down, read-only-ledger, and timeout readiness failures recover without restart
|
|
91
|
+
shared dead letters requeue or discard with receipts and events preserved
|
|
92
|
+
Postgres and MySQL pool queues fail fast at configured bounds
|
|
93
|
+
backup digest and clean restore match
|
|
94
|
+
retention archives before delete and preserves active proposals
|
|
95
|
+
```
|
|
96
|
+
|
|
97
|
+
All source rows, identities, keys, tokens, and credentials are synthetic. The
|
|
98
|
+
test tears down its volumes.
|
|
99
|
+
|
|
100
|
+
## Claim-Bound Sessions
|
|
101
|
+
|
|
102
|
+
Use asymmetric verification for a networked fleet:
|
|
103
|
+
|
|
104
|
+
```json
|
|
105
|
+
{
|
|
106
|
+
"trusted_context": {
|
|
107
|
+
"provider": "http_claims",
|
|
108
|
+
"values": {
|
|
109
|
+
"tenant_id_key": "tenant_id",
|
|
110
|
+
"principal_key": "sub"
|
|
111
|
+
}
|
|
112
|
+
},
|
|
113
|
+
"session_auth": {
|
|
114
|
+
"provider": "jwt_asymmetric",
|
|
115
|
+
"algorithms": ["RS256"],
|
|
116
|
+
"jwks_url_env": "SYNAPSOR_SESSION_JWKS_URL",
|
|
117
|
+
"issuer": "https://identity.example",
|
|
118
|
+
"audience": "synapsor-runner",
|
|
119
|
+
"tenant_claim": "tenant_id",
|
|
120
|
+
"principal_claim": "sub",
|
|
121
|
+
"clock_skew_seconds": 30,
|
|
122
|
+
"jwks_cache_seconds": 600,
|
|
123
|
+
"jwks_cooldown_seconds": 30,
|
|
124
|
+
"fetch_timeout_ms": 3000,
|
|
125
|
+
"max_response_bytes": 1048576
|
|
126
|
+
}
|
|
127
|
+
}
|
|
128
|
+
```
|
|
129
|
+
|
|
130
|
+
Every served capability context must bind tenant and principal from
|
|
131
|
+
`HTTP_CLAIM`. Runner rejects an environment-bound contract mixed into this
|
|
132
|
+
server with `TRUSTED_CONTEXT_PROVIDER_CONFLICT` before tools are served.
|
|
133
|
+
|
|
134
|
+
Use an HTTPS JWKS URL from an allowlisted identity system. Runner rejects
|
|
135
|
+
redirects, oversized responses, unknown algorithms, `none`, missing/unknown
|
|
136
|
+
`kid`, bad issuer/audience, expired/not-yet-valid tokens, and private JWK
|
|
137
|
+
material. Unknown `kid` triggers at most one controlled refresh. For offline
|
|
138
|
+
deployments, configure one public PEM through `public_key_env` or
|
|
139
|
+
`public_key_path`; never place a private key in Runner config.
|
|
140
|
+
|
|
141
|
+
## Ledger And Source Pools
|
|
142
|
+
|
|
143
|
+
```json
|
|
144
|
+
{
|
|
145
|
+
"storage": {
|
|
146
|
+
"shared_postgres": {
|
|
147
|
+
"mode": "runtime_store",
|
|
148
|
+
"url_env": "SYNAPSOR_LEDGER_DATABASE_URL",
|
|
149
|
+
"schema": "synapsor_runner",
|
|
150
|
+
"lock_timeout_ms": 5000,
|
|
151
|
+
"max_entries": 10000
|
|
152
|
+
}
|
|
153
|
+
},
|
|
154
|
+
"sources": {
|
|
155
|
+
"app_postgres": {
|
|
156
|
+
"engine": "postgres",
|
|
157
|
+
"read_url_env": "APP_POSTGRES_READ_URL",
|
|
158
|
+
"write_url_env": "APP_POSTGRES_WRITE_URL",
|
|
159
|
+
"statement_timeout_ms": 3000,
|
|
160
|
+
"pool": {
|
|
161
|
+
"max_connections": 4,
|
|
162
|
+
"connection_timeout_ms": 2000,
|
|
163
|
+
"idle_timeout_ms": 30000,
|
|
164
|
+
"queue_timeout_ms": 1000,
|
|
165
|
+
"queue_limit": 16
|
|
166
|
+
}
|
|
167
|
+
}
|
|
168
|
+
}
|
|
169
|
+
}
|
|
170
|
+
```
|
|
171
|
+
|
|
172
|
+
Budget database connections across every replica and worker:
|
|
173
|
+
|
|
174
|
+
```text
|
|
175
|
+
replicas * sum(source max_connections) + readiness/admin headroom
|
|
176
|
+
```
|
|
177
|
+
|
|
178
|
+
Keep that below database role and instance limits. Queue overflow fails with
|
|
179
|
+
`SOURCE_POOL_QUEUE_FULL`; acquisition timeout fails with
|
|
180
|
+
`SOURCE_POOL_TIMEOUT`. Neither waits indefinitely or creates a proposal.
|
|
181
|
+
|
|
182
|
+
Use separate roles:
|
|
183
|
+
|
|
184
|
+
- source reader: `CONNECT`, schema `USAGE`, and `SELECT` only on reviewed
|
|
185
|
+
tables/views;
|
|
186
|
+
- direct writer: reviewed column-level `UPDATE` plus source receipt-table
|
|
187
|
+
`SELECT/INSERT/UPDATE`;
|
|
188
|
+
- ledger role: only the configured ledger schema and migration rights;
|
|
189
|
+
- migration administrator: run migrations separately when Runner should not
|
|
190
|
+
own DDL authority.
|
|
191
|
+
|
|
192
|
+
## Fleet-Wide Rate Limits
|
|
193
|
+
|
|
194
|
+
```json
|
|
195
|
+
{
|
|
196
|
+
"rate_limits": {
|
|
197
|
+
"enabled": true,
|
|
198
|
+
"default": { "requests": 60, "window_seconds": 60 },
|
|
199
|
+
"capabilities": {
|
|
200
|
+
"billing.propose_refund": { "requests": 10, "window_seconds": 60 }
|
|
201
|
+
}
|
|
202
|
+
}
|
|
203
|
+
}
|
|
204
|
+
```
|
|
205
|
+
|
|
206
|
+
Semantics are fixed-window, keyed by a hash of verified tenant ID and
|
|
207
|
+
capability. There is no separate burst allowance: a tenant may consume the
|
|
208
|
+
entire configured count immediately, then must wait for the aligned window to
|
|
209
|
+
reset. In local SQLite mode limits are process-local. In `runtime_store` mode
|
|
210
|
+
they use an atomic shared Postgres bucket. Rejection returns `RATE_LIMITED` and
|
|
211
|
+
`retry_after_ms` to the reset boundary, increments a metric, and creates no
|
|
212
|
+
proposal.
|
|
213
|
+
|
|
214
|
+
## Probes And Metrics
|
|
215
|
+
|
|
216
|
+
- `GET /healthz`: unauthenticated cheap process liveness; no dependency
|
|
217
|
+
checks or configuration inventory.
|
|
218
|
+
- `GET /readyz`: 200 only when catalog, required sources, authoritative ledger,
|
|
219
|
+
and required writeback/executor dependencies are ready; otherwise 503.
|
|
220
|
+
- `GET /metrics`: disabled unless configured. A non-loopback bind requires a
|
|
221
|
+
separate metrics bearer token.
|
|
222
|
+
|
|
223
|
+
```json
|
|
224
|
+
{
|
|
225
|
+
"metrics": {
|
|
226
|
+
"enabled": true,
|
|
227
|
+
"token_env": "SYNAPSOR_METRICS_TOKEN"
|
|
228
|
+
}
|
|
229
|
+
}
|
|
230
|
+
```
|
|
231
|
+
|
|
232
|
+
Do not use the model-facing MCP token as the metrics token. Metrics include
|
|
233
|
+
controlled tenant/capability/source/component labels, never object IDs,
|
|
234
|
+
principals, database URLs, tokens, or raw errors.
|
|
235
|
+
|
|
236
|
+
## Reviewers, Quorum, And Workers
|
|
237
|
+
|
|
238
|
+
Use `signed_key` or `jwt_oidc` operator identity. `dev_env` is explicitly
|
|
239
|
+
unverified and is only for local fixtures. A capability may require multiple
|
|
240
|
+
distinct reviewers in the canonical contract:
|
|
241
|
+
|
|
242
|
+
```sql
|
|
243
|
+
APPROVAL REQUIRED ROLE billing_lead
|
|
244
|
+
REQUIRE 2 APPROVALS
|
|
245
|
+
```
|
|
246
|
+
|
|
247
|
+
The same verified subject cannot fill two slots. Policy auto-approval does not
|
|
248
|
+
satisfy a multi-human quorum. Apply and workers reject the proposal until
|
|
249
|
+
progress reaches `N/N`. A rejection is terminal and auditable.
|
|
250
|
+
|
|
251
|
+
Use the shared queue explicitly:
|
|
252
|
+
|
|
253
|
+
```bash
|
|
254
|
+
synapsor-runner proposals list --config ./synapsor.runner.json
|
|
255
|
+
synapsor-runner proposals show latest --config ./synapsor.runner.json
|
|
256
|
+
synapsor-runner ui --config ./synapsor.runner.json
|
|
257
|
+
synapsor-runner worker run --yes --config ./synapsor.runner.json
|
|
258
|
+
```
|
|
259
|
+
|
|
260
|
+
Dead-letter operations require verified operator identity:
|
|
261
|
+
|
|
262
|
+
```bash
|
|
263
|
+
synapsor-runner worker dead-letter list --config ./synapsor.runner.json
|
|
264
|
+
synapsor-runner worker dead-letter show wrp_... --config ./synapsor.runner.json
|
|
265
|
+
synapsor-runner worker dead-letter requeue wrp_... --retry-budget 3 --yes \
|
|
266
|
+
--config ./synapsor.runner.json --identity alice --identity-key /secure/alice.pem
|
|
267
|
+
synapsor-runner worker dead-letter discard wrp_... --reason "closed" --yes \
|
|
268
|
+
--config ./synapsor.runner.json --identity alice --identity-key /secure/alice.pem
|
|
269
|
+
```
|
|
270
|
+
|
|
271
|
+
Requeue preserves history and refuses if a durable applied/already-applied
|
|
272
|
+
receipt already proves the effect. Discard closes the queue item without
|
|
273
|
+
deleting proposals, receipts, or events.
|
|
274
|
+
|
|
275
|
+
## Backup, Restore, And Retention
|
|
276
|
+
|
|
277
|
+
```bash
|
|
278
|
+
synapsor-runner store shared-postgres backup \
|
|
279
|
+
--url-env SYNAPSOR_LEDGER_DATABASE_URL --schema synapsor_runner \
|
|
280
|
+
--output ./ledger-backup.json
|
|
281
|
+
synapsor-runner store shared-postgres verify-backup --input ./ledger-backup.json
|
|
282
|
+
synapsor-runner store shared-postgres restore-backup \
|
|
283
|
+
--input ./ledger-backup.json --url-env SYNAPSOR_LEDGER_DATABASE_URL \
|
|
284
|
+
--schema synapsor_runner_restore --yes
|
|
285
|
+
```
|
|
286
|
+
|
|
287
|
+
Restore only accepts an empty target schema and verifies the post-restore
|
|
288
|
+
manifest digest. Store archives as sensitive database extracts with owner-only
|
|
289
|
+
permissions.
|
|
290
|
+
|
|
291
|
+
Retention is archive-before-delete:
|
|
292
|
+
|
|
293
|
+
```bash
|
|
294
|
+
synapsor-runner store shared-postgres retention --older-than 30d \
|
|
295
|
+
--url-env SYNAPSOR_LEDGER_DATABASE_URL --schema synapsor_runner --dry-run
|
|
296
|
+
synapsor-runner store shared-postgres retention --older-than 30d \
|
|
297
|
+
--url-env SYNAPSOR_LEDGER_DATABASE_URL --schema synapsor_runner \
|
|
298
|
+
--output ./ledger-archive.json --yes
|
|
299
|
+
```
|
|
300
|
+
|
|
301
|
+
The archive is written and digest-verified before deletion. Pending review,
|
|
302
|
+
approved, pending-worker, retry, failed, and dead-letter records are retained.
|
|
303
|
+
The operation appends an immutable retention event.
|
|
304
|
+
|
|
305
|
+
Restore runbook:
|
|
306
|
+
|
|
307
|
+
1. Stop ingress and workers.
|
|
308
|
+
2. Verify the backup digest.
|
|
309
|
+
3. Restore into a clean schema.
|
|
310
|
+
4. Run `store shared-postgres status` and `/readyz` against the restore.
|
|
311
|
+
5. Inspect at least one proposal, receipt, and replay through CLI.
|
|
312
|
+
6. Point one canary Runner at the restored schema.
|
|
313
|
+
7. Resume ingress, then workers, and watch readiness/dead-letter metrics.
|
|
314
|
+
|
|
315
|
+
## Shutdown, Rotation, And Upgrades
|
|
316
|
+
|
|
317
|
+
Terminate with `SIGTERM`; Runner stops accepting work, closes HTTP sessions and
|
|
318
|
+
source pools, and releases ledger locks. Give workers enough grace for the
|
|
319
|
+
configured handler timeout. A hard kill is recoverable through leases and
|
|
320
|
+
source-side idempotency receipts, but should remain an incident signal.
|
|
321
|
+
|
|
322
|
+
Rotate JWKS keys by publishing the new `kid`, waiting at least the configured
|
|
323
|
+
cache lifetime, switching issuers, then retiring the old key. Rotate metrics,
|
|
324
|
+
handler, database, and ledger credentials independently through their named
|
|
325
|
+
environment variables.
|
|
326
|
+
|
|
327
|
+
Only homogeneous Runner 1.1 instances are verified by `test:fleet`. Mixed
|
|
328
|
+
1.0/1.1 operation is not claimed. For rolling upgrades:
|
|
329
|
+
|
|
330
|
+
1. back up and verify the ledger;
|
|
331
|
+
2. run schema/config validation with the target version;
|
|
332
|
+
3. drain one worker and remove one Runner from the load balancer;
|
|
333
|
+
4. upgrade one canary, verify `/readyz`, MCP tools, proposal inspection, and
|
|
334
|
+
metrics;
|
|
335
|
+
5. continue one instance at a time;
|
|
336
|
+
6. upgrade workers after HTTP Runners.
|
|
337
|
+
|
|
338
|
+
Rollback the binary only while contracts/config remain accepted by the prior
|
|
339
|
+
minor. If the prior version does not understand a new canonical field or
|
|
340
|
+
ledger entry, restore the verified pre-upgrade backup into a clean schema and
|
|
341
|
+
roll back the full fleet. Never mix versions merely because both are `1.x`.
|
|
342
|
+
|
|
343
|
+
## TLS Boundary
|
|
344
|
+
|
|
345
|
+
Terminate public TLS or mTLS at a trusted load balancer, or configure Runner's
|
|
346
|
+
tested TLS/mTLS options directly. Keep Runner-to-database TLS verification on,
|
|
347
|
+
use private networking where possible, restrict `/metrics`, and never forward
|
|
348
|
+
untrusted identity headers as claims. The bearer JWT itself, not a proxy-added
|
|
349
|
+
tenant header, is the session authority.
|
|
@@ -37,6 +37,13 @@ Trusted context comes from local configuration, environment bindings, or Cloud
|
|
|
37
37
|
session context in Cloud mode. Tenant, principal, and authorization scope must
|
|
38
38
|
not be accepted from the model as authority.
|
|
39
39
|
|
|
40
|
+
Proposal, evidence, and replay handles are references, not bearer credentials.
|
|
41
|
+
Before returning a local MCP resource, Runner resolves the resource's owning
|
|
42
|
+
capability context again and requires both tenant and principal to match the
|
|
43
|
+
current trusted session. Missing resources and ownership mismatches return the
|
|
44
|
+
same generic `RESOURCE_NOT_FOUND` result. Local operator CLI access remains an
|
|
45
|
+
explicit host-level administrative boundary.
|
|
46
|
+
|
|
40
47
|
Proposal tools read the current row through the read credential, store evidence
|
|
41
48
|
and an exact before/after diff, and leave the source database unchanged.
|
|
42
49
|
|
|
@@ -0,0 +1,11 @@
|
|
|
1
|
+
FROM node:22-bookworm-slim
|
|
2
|
+
|
|
3
|
+
RUN corepack enable && corepack prepare pnpm@10.14.0 --activate
|
|
4
|
+
|
|
5
|
+
WORKDIR /app
|
|
6
|
+
COPY . .
|
|
7
|
+
RUN corepack pnpm install --frozen-lockfile \
|
|
8
|
+
&& corepack pnpm build:runner-package
|
|
9
|
+
|
|
10
|
+
ENTRYPOINT ["node", "apps/runner/dist/cli.js"]
|
|
11
|
+
CMD ["mcp", "serve", "--transport", "streamable-http", "--host", "0.0.0.0", "--port", "8766", "--config", "/config/synapsor.runner.json"]
|
|
@@ -0,0 +1,52 @@
|
|
|
1
|
+
# Two-Runner Fleet Fixture
|
|
2
|
+
|
|
3
|
+
This synthetic fixture starts Postgres, MySQL, and two stateless Streamable
|
|
4
|
+
HTTP Runner services. Both Runners share one bounded Postgres runtime ledger.
|
|
5
|
+
The committed passwords and HS256 key are disposable local-demo values. Never
|
|
6
|
+
reuse them outside this fixture.
|
|
7
|
+
|
|
8
|
+
From the repository root:
|
|
9
|
+
|
|
10
|
+
```bash
|
|
11
|
+
corepack pnpm install
|
|
12
|
+
docker compose --profile fleet -f examples/runner-fleet/docker-compose.yml up --build -d --wait
|
|
13
|
+
docker compose --profile fleet -f examples/runner-fleet/docker-compose.yml ps
|
|
14
|
+
```
|
|
15
|
+
|
|
16
|
+
Check both instances:
|
|
17
|
+
|
|
18
|
+
```bash
|
|
19
|
+
curl --fail http://127.0.0.1:8871/healthz
|
|
20
|
+
curl --fail http://127.0.0.1:8871/readyz
|
|
21
|
+
curl --fail http://127.0.0.1:8872/readyz
|
|
22
|
+
```
|
|
23
|
+
|
|
24
|
+
The MCP endpoint requires a claim-bearing development JWT. Generate one for
|
|
25
|
+
the fixture only:
|
|
26
|
+
|
|
27
|
+
```bash
|
|
28
|
+
node examples/runner-fleet/mint-dev-token.mjs acme local-agent
|
|
29
|
+
```
|
|
30
|
+
|
|
31
|
+
The production path should use `jwt_asymmetric` with an explicit RS256/ES256
|
|
32
|
+
allowlist and a trusted JWKS URL or public PEM. See
|
|
33
|
+
[Running A Runner Fleet](../../docs/running-a-runner-fleet.md).
|
|
34
|
+
|
|
35
|
+
Run the stronger automated verification instead of treating a green Compose
|
|
36
|
+
status as proof:
|
|
37
|
+
|
|
38
|
+
```bash
|
|
39
|
+
corepack pnpm test:fleet
|
|
40
|
+
```
|
|
41
|
+
|
|
42
|
+
That test launches two source-tree Runners, uses asymmetric claim-bound
|
|
43
|
+
sessions and concurrent signed reviewers, starts competing workers, kills
|
|
44
|
+
workers before, during, and after writeback, and verifies readiness,
|
|
45
|
+
dead-letter recovery, idempotency, pool pressure, backup/restore, and
|
|
46
|
+
archive-before-retention. It deletes all fixture volumes when complete.
|
|
47
|
+
|
|
48
|
+
Clean up the visual Compose fixture with:
|
|
49
|
+
|
|
50
|
+
```bash
|
|
51
|
+
docker compose --profile fleet -f examples/runner-fleet/docker-compose.yml down -v --remove-orphans
|
|
52
|
+
```
|
|
@@ -0,0 +1,76 @@
|
|
|
1
|
+
services:
|
|
2
|
+
postgres:
|
|
3
|
+
image: postgres:16
|
|
4
|
+
environment:
|
|
5
|
+
POSTGRES_DB: synapsor_fleet
|
|
6
|
+
POSTGRES_USER: synapsor_admin
|
|
7
|
+
POSTGRES_PASSWORD: synapsor_admin_password
|
|
8
|
+
ports:
|
|
9
|
+
- "55439:5432"
|
|
10
|
+
healthcheck:
|
|
11
|
+
test: ["CMD-SHELL", "test \"$(psql -U synapsor_admin -d synapsor_fleet -tAc \"SELECT pg_postmaster_start_time() > initialized_at FROM public.synapsor_fixture_ready LIMIT 1\" 2>/dev/null | xargs)\" = t"]
|
|
12
|
+
interval: 1s
|
|
13
|
+
timeout: 3s
|
|
14
|
+
retries: 30
|
|
15
|
+
volumes:
|
|
16
|
+
- ./seed/postgres.sql:/docker-entrypoint-initdb.d/001-schema.sql:ro
|
|
17
|
+
|
|
18
|
+
mysql:
|
|
19
|
+
image: mysql:8
|
|
20
|
+
environment:
|
|
21
|
+
MYSQL_DATABASE: synapsor_fleet
|
|
22
|
+
MYSQL_ROOT_PASSWORD: root_password
|
|
23
|
+
ports:
|
|
24
|
+
- "53309:3306"
|
|
25
|
+
healthcheck:
|
|
26
|
+
test: ["CMD", "mysqladmin", "ping", "-h", "127.0.0.1", "-proot_password"]
|
|
27
|
+
interval: 2s
|
|
28
|
+
timeout: 5s
|
|
29
|
+
retries: 40
|
|
30
|
+
volumes:
|
|
31
|
+
- ./seed/mysql.sql:/docker-entrypoint-initdb.d/001-schema.sql:ro
|
|
32
|
+
|
|
33
|
+
runner-a:
|
|
34
|
+
profiles: ["fleet"]
|
|
35
|
+
build:
|
|
36
|
+
context: ../..
|
|
37
|
+
dockerfile: examples/runner-fleet/Dockerfile
|
|
38
|
+
command: ["mcp", "serve", "--transport", "streamable-http", "--host", "0.0.0.0", "--port", "8766", "--config", "/config/synapsor.runner.json"]
|
|
39
|
+
depends_on:
|
|
40
|
+
postgres:
|
|
41
|
+
condition: service_healthy
|
|
42
|
+
environment: &runner_environment
|
|
43
|
+
FLEET_POSTGRES_READ_URL: postgresql://synapsor_reader:synapsor_reader_password@postgres:5432/synapsor_fleet
|
|
44
|
+
FLEET_POSTGRES_WRITE_URL: postgresql://synapsor_writer:synapsor_writer_password@postgres:5432/synapsor_fleet
|
|
45
|
+
SYNAPSOR_LEDGER_DATABASE_URL: postgresql://synapsor_admin:synapsor_admin_password@postgres:5432/synapsor_fleet
|
|
46
|
+
SYNAPSOR_SESSION_JWT_SECRET: synthetic-fleet-session-secret-change-before-use-0001
|
|
47
|
+
SYNAPSOR_METRICS_TOKEN: synthetic-fleet-metrics-token-change-before-use
|
|
48
|
+
ports:
|
|
49
|
+
- "8871:8766"
|
|
50
|
+
volumes:
|
|
51
|
+
- ./synapsor.runner.json:/config/synapsor.runner.json:ro
|
|
52
|
+
healthcheck:
|
|
53
|
+
test: ["CMD", "node", "-e", "fetch('http://127.0.0.1:8766/readyz').then(r=>process.exit(r.ok?0:1)).catch(()=>process.exit(1))"]
|
|
54
|
+
interval: 2s
|
|
55
|
+
timeout: 5s
|
|
56
|
+
retries: 30
|
|
57
|
+
|
|
58
|
+
runner-b:
|
|
59
|
+
profiles: ["fleet"]
|
|
60
|
+
build:
|
|
61
|
+
context: ../..
|
|
62
|
+
dockerfile: examples/runner-fleet/Dockerfile
|
|
63
|
+
command: ["mcp", "serve", "--transport", "streamable-http", "--host", "0.0.0.0", "--port", "8766", "--config", "/config/synapsor.runner.json"]
|
|
64
|
+
depends_on:
|
|
65
|
+
postgres:
|
|
66
|
+
condition: service_healthy
|
|
67
|
+
environment: *runner_environment
|
|
68
|
+
ports:
|
|
69
|
+
- "8872:8766"
|
|
70
|
+
volumes:
|
|
71
|
+
- ./synapsor.runner.json:/config/synapsor.runner.json:ro
|
|
72
|
+
healthcheck:
|
|
73
|
+
test: ["CMD", "node", "-e", "fetch('http://127.0.0.1:8766/readyz').then(r=>process.exit(r.ok?0:1)).catch(()=>process.exit(1))"]
|
|
74
|
+
interval: 2s
|
|
75
|
+
timeout: 5s
|
|
76
|
+
retries: 30
|
|
@@ -0,0 +1,19 @@
|
|
|
1
|
+
import crypto from "node:crypto";
|
|
2
|
+
|
|
3
|
+
const tenant = process.argv[2] ?? "acme";
|
|
4
|
+
const subject = process.argv[3] ?? "fleet-demo-agent";
|
|
5
|
+
const secret = "synthetic-fleet-session-secret-change-before-use-0001";
|
|
6
|
+
const now = Math.floor(Date.now() / 1000);
|
|
7
|
+
const header = { alg: "HS256", typ: "JWT" };
|
|
8
|
+
const payload = {
|
|
9
|
+
sub: subject,
|
|
10
|
+
tenant_id: tenant,
|
|
11
|
+
iss: "https://fleet.example.invalid",
|
|
12
|
+
aud: "synapsor-runner-fleet",
|
|
13
|
+
iat: now,
|
|
14
|
+
exp: now + 600,
|
|
15
|
+
};
|
|
16
|
+
const encode = (value) => Buffer.from(JSON.stringify(value)).toString("base64url");
|
|
17
|
+
const unsigned = `${encode(header)}.${encode(payload)}`;
|
|
18
|
+
const signature = crypto.createHmac("sha256", secret).update(unsigned).digest("base64url");
|
|
19
|
+
process.stdout.write(`${unsigned}.${signature}\n`);
|
|
@@ -0,0 +1,20 @@
|
|
|
1
|
+
CREATE TABLE invoices (
|
|
2
|
+
id varchar(128) PRIMARY KEY,
|
|
3
|
+
tenant_id varchar(128) NOT NULL,
|
|
4
|
+
status varchar(64) NOT NULL,
|
|
5
|
+
late_fee_cents integer NOT NULL,
|
|
6
|
+
waiver_reason varchar(500),
|
|
7
|
+
updated_at datetime(6) NOT NULL
|
|
8
|
+
);
|
|
9
|
+
|
|
10
|
+
INSERT INTO invoices (id, tenant_id, status, late_fee_cents, waiver_reason, updated_at) VALUES
|
|
11
|
+
('MYSQL-ACME', 'acme', 'overdue', 700, NULL, '2026-07-12 12:00:00.000000');
|
|
12
|
+
|
|
13
|
+
CREATE VIEW slow_invoices AS
|
|
14
|
+
SELECT invoices.*, SLEEP(0.35) AS synthetic_delay
|
|
15
|
+
FROM invoices;
|
|
16
|
+
|
|
17
|
+
CREATE USER IF NOT EXISTS 'synapsor_reader'@'%' IDENTIFIED BY 'synapsor_reader_password';
|
|
18
|
+
GRANT SELECT ON synapsor_fleet.invoices TO 'synapsor_reader'@'%';
|
|
19
|
+
GRANT SELECT ON synapsor_fleet.slow_invoices TO 'synapsor_reader'@'%';
|
|
20
|
+
FLUSH PRIVILEGES;
|
|
@@ -0,0 +1,73 @@
|
|
|
1
|
+
CREATE TABLE public.invoices (
|
|
2
|
+
id text PRIMARY KEY,
|
|
3
|
+
tenant_id text NOT NULL,
|
|
4
|
+
status text NOT NULL,
|
|
5
|
+
late_fee_cents integer NOT NULL,
|
|
6
|
+
waiver_reason text,
|
|
7
|
+
updated_at timestamptz NOT NULL
|
|
8
|
+
);
|
|
9
|
+
|
|
10
|
+
CREATE TABLE public.synapsor_writeback_receipts (
|
|
11
|
+
idempotency_key text PRIMARY KEY,
|
|
12
|
+
job_id text UNIQUE NOT NULL,
|
|
13
|
+
proposal_id text NOT NULL,
|
|
14
|
+
status text NOT NULL,
|
|
15
|
+
result_hash text,
|
|
16
|
+
created_at timestamptz NOT NULL DEFAULT now(),
|
|
17
|
+
completed_at timestamptz
|
|
18
|
+
);
|
|
19
|
+
|
|
20
|
+
CREATE TABLE public.synthetic_handler_receipts (
|
|
21
|
+
idempotency_key text PRIMARY KEY,
|
|
22
|
+
proposal_id text NOT NULL,
|
|
23
|
+
object_id text NOT NULL,
|
|
24
|
+
status text NOT NULL,
|
|
25
|
+
created_at timestamptz NOT NULL DEFAULT now()
|
|
26
|
+
);
|
|
27
|
+
|
|
28
|
+
INSERT INTO public.invoices (id, tenant_id, status, late_fee_cents, waiver_reason, updated_at) VALUES
|
|
29
|
+
('INV-ACME', 'acme', 'overdue', 2500, NULL, '2026-07-12T12:00:00Z'),
|
|
30
|
+
('INV-GLOBEX', 'globex', 'overdue', 3100, NULL, '2026-07-12T12:00:00Z'),
|
|
31
|
+
('INV-RATECO', 'rateco', 'overdue', 900, NULL, '2026-07-12T12:00:00Z'),
|
|
32
|
+
('INV-QUORUM-RACE', 'acme', 'overdue', 1400, NULL, '2026-07-12T12:00:00Z'),
|
|
33
|
+
('INV-WORKER-RACE', 'acme', 'overdue', 1600, NULL, '2026-07-12T12:00:00Z'),
|
|
34
|
+
('INV-DEAD-REQUEUE', 'acme', 'overdue', 1700, NULL, '2026-07-12T12:00:00Z'),
|
|
35
|
+
('INV-DEAD-DISCARD', 'acme', 'overdue', 1750, NULL, '2026-07-12T12:00:00Z'),
|
|
36
|
+
('INV-KILL-BEFORE', 'acme', 'overdue', 1200, NULL, '2026-07-12T12:00:00Z'),
|
|
37
|
+
('INV-KILL-DURING', 'acme', 'overdue', 1500, NULL, '2026-07-12T12:00:00Z'),
|
|
38
|
+
('INV-KILL-AFTER', 'acme', 'overdue', 1800, NULL, '2026-07-12T12:00:00Z');
|
|
39
|
+
|
|
40
|
+
CREATE FUNCTION public.synthetic_pool_delay() RETURNS integer
|
|
41
|
+
LANGUAGE plpgsql VOLATILE AS $$
|
|
42
|
+
BEGIN
|
|
43
|
+
PERFORM pg_sleep(0.15);
|
|
44
|
+
RETURN 0;
|
|
45
|
+
END
|
|
46
|
+
$$;
|
|
47
|
+
|
|
48
|
+
CREATE VIEW public.slow_invoices AS
|
|
49
|
+
SELECT invoices.*, public.synthetic_pool_delay() AS synthetic_delay
|
|
50
|
+
FROM public.invoices;
|
|
51
|
+
|
|
52
|
+
DO $$
|
|
53
|
+
BEGIN
|
|
54
|
+
CREATE ROLE synapsor_reader LOGIN PASSWORD 'synapsor_reader_password';
|
|
55
|
+
CREATE ROLE synapsor_writer LOGIN PASSWORD 'synapsor_writer_password';
|
|
56
|
+
EXCEPTION WHEN duplicate_object THEN NULL;
|
|
57
|
+
END
|
|
58
|
+
$$;
|
|
59
|
+
|
|
60
|
+
GRANT CONNECT ON DATABASE synapsor_fleet TO synapsor_reader, synapsor_writer;
|
|
61
|
+
GRANT USAGE ON SCHEMA public TO synapsor_reader, synapsor_writer;
|
|
62
|
+
GRANT SELECT ON public.invoices TO synapsor_reader, synapsor_writer;
|
|
63
|
+
GRANT SELECT ON public.slow_invoices TO synapsor_reader;
|
|
64
|
+
GRANT UPDATE (late_fee_cents, waiver_reason, updated_at) ON public.invoices TO synapsor_writer;
|
|
65
|
+
GRANT SELECT, INSERT, UPDATE ON public.synapsor_writeback_receipts TO synapsor_writer;
|
|
66
|
+
|
|
67
|
+
-- Docker's Postgres entrypoint briefly starts a temporary postmaster while it
|
|
68
|
+
-- runs init scripts. The Compose health check compares this timestamp with the
|
|
69
|
+
-- current postmaster start time so dependents wait for the final server.
|
|
70
|
+
CREATE TABLE public.synapsor_fixture_ready (
|
|
71
|
+
initialized_at timestamptz NOT NULL
|
|
72
|
+
);
|
|
73
|
+
INSERT INTO public.synapsor_fixture_ready (initialized_at) VALUES (clock_timestamp());
|