@synapsor/runner 1.4.123 → 1.5.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +144 -19
- package/README.md +132 -135
- package/dist/authoring.d.ts +23 -0
- package/dist/authoring.d.ts.map +1 -0
- package/dist/authoring.mjs +1318 -0
- package/dist/cli.d.ts.map +1 -1
- package/dist/cli.js +4 -5
- package/dist/local-ui.d.ts +14 -0
- package/dist/local-ui.d.ts.map +1 -1
- package/dist/runner.mjs +18427 -7084
- package/dist/runtime.d.ts +50 -0
- package/dist/runtime.d.ts.map +1 -0
- package/dist/runtime.mjs +15094 -0
- package/dist/shadow.d.ts +36 -0
- package/dist/shadow.d.ts.map +1 -0
- package/dist/shadow.mjs +5262 -0
- package/docs/README.md +44 -5
- package/docs/alternatives.md +122 -0
- package/docs/capability-authoring.md +44 -1
- package/docs/client-recipes.md +218 -0
- package/docs/contract-testing.md +9 -7
- package/docs/cursor-plugin.md +78 -0
- package/docs/database-enforced-scope.md +273 -0
- package/docs/dependency-license-inventory.md +6 -2
- package/docs/doctor.md +17 -0
- package/docs/effect-regression.md +179 -0
- package/docs/fresh-developer-usability.md +110 -0
- package/docs/getting-started-own-database.md +103 -2
- package/docs/host-compatibility.md +59 -0
- package/docs/local-mode.md +12 -7
- package/docs/mcp-apps.md +188 -0
- package/docs/mcp-audit.md +220 -2
- package/docs/mcp-client-setup.md +20 -4
- package/docs/mcp-clients.md +41 -4
- package/docs/oss-vs-cloud.md +3 -0
- package/docs/production.md +8 -0
- package/docs/release-notes.md +138 -14
- package/docs/release-policy.md +9 -5
- package/docs/runner-config-reference.md +25 -0
- package/docs/schema-api-candidates.md +68 -0
- package/docs/security-boundary.md +30 -5
- package/docs/shadow-studies.md +241 -0
- package/docs/troubleshooting-first-run.md +46 -0
- package/examples/mcp-postgres-billing-app-handler/app-handler.mjs +9 -1
- package/examples/mcp-postgres-billing-app-handler/schema.sql +11 -0
- package/examples/mcp-postgres-billing-app-handler/synapsor-handler.mjs +3 -0
- package/examples/raw-sql-vs-synapsor/Makefile +2 -2
- package/examples/raw-sql-vs-synapsor/README.md +6 -4
- package/examples/support-billing-agent/Makefile +5 -1
- package/examples/support-billing-agent/README.md +57 -5
- package/examples/support-billing-agent/app/README.md +6 -0
- package/examples/support-billing-agent/app/contract.ts +28 -0
- package/examples/support-billing-agent/app/effect-adapter.mjs +23 -0
- package/examples/support-billing-agent/app/record-shadow-outcomes.mjs +44 -0
- package/examples/support-billing-agent/db/schema.sql +88 -1
- package/examples/support-billing-agent/db/seed.sql +15 -15
- package/examples/support-billing-agent/scripts/run-demo.sh +4 -0
- package/examples/support-billing-agent/scripts/run-evaluation.sh +63 -0
- package/examples/support-billing-agent/shadow-study/cases.jsonl +6 -0
- package/examples/support-billing-agent/shadow-study/outcomes.jsonl +2 -0
- package/examples/support-billing-agent/synapsor.runner.json +17 -5
- package/examples/support-plan-credit/README.md +48 -2
- package/examples/support-plan-credit/mcp-client-examples/README.md +23 -0
- package/examples/support-plan-credit/mcp-client-examples/claude-code.sh +34 -0
- package/examples/support-plan-credit/mcp-client-examples/codex.config.toml +24 -0
- package/examples/support-plan-credit/mcp-client-examples/generic-stdio.mjs +35 -0
- package/examples/support-plan-credit/mcp-client-examples/generic-streamable-http.mjs +31 -0
- package/examples/support-plan-credit/mcp-client-examples/google-adk.py +40 -0
- package/examples/support-plan-credit/mcp-client-examples/langchain.mjs +29 -0
- package/examples/support-plan-credit/mcp-client-examples/llamaindex.py +36 -0
- package/examples/support-plan-credit/mcp-client-examples/openai-agents-stdio.ts +1 -1
- package/examples/support-plan-credit/mcp-client-examples/openai-agents-streamable-http.ts +1 -1
- package/examples/support-plan-credit/mcp-client-examples/vscode.mcp.json +20 -0
- package/examples/support-plan-credit/synapsor/actions/support.propose_plan_credit.contract-tests.generated.json +221 -0
- package/examples/support-plan-credit/synapsor/actions/support.propose_plan_credit.ts +34 -0
- package/fixtures/effects/changed/eff_support_late_fee_v1.result.json +58 -0
- package/fixtures/effects/dataset.json +8 -0
- package/fixtures/effects/results/eff_support_late_fee_v1.result.json +59 -0
- package/fixtures/effects/support-late-fee.effect.json +71 -0
- package/fixtures/generators/drizzle/malicious.ts +11 -0
- package/fixtures/generators/drizzle/schema.ts +12 -0
- package/fixtures/generators/openapi/external-ref.yaml +15 -0
- package/fixtures/generators/openapi/openapi.yaml +81 -0
- package/fixtures/generators/prisma/schema.prisma +34 -0
- package/fixtures/mcp-audit/README.md +14 -0
- package/fixtures/mcp-audit/cursor-bypass-config.json +38 -0
- package/fixtures/mcp-audit/dangerous-tools-list.json +33 -0
- package/fixtures/mcp-audit/reviewed-proposal-tools-list.json +60 -0
- package/package.json +29 -3
- package/schemas/effect-dataset.schema.json +19 -0
- package/schemas/effect-fixture.schema.json +191 -0
- package/schemas/effect-result.schema.json +52 -0
- package/schemas/mcp-audit-candidates.schema.json +41 -0
- package/schemas/mcp-audit-report.schema.json +203 -0
- package/schemas/schema-candidate-review.schema.json +167 -0
- package/schemas/schema-candidates.schema.json +49 -0
- package/schemas/synapsor.contract-tests.schema.json +1 -1
- package/schemas/synapsor.runner.schema.json +51 -0
|
@@ -0,0 +1,273 @@
|
|
|
1
|
+
# Database-Enforced Tenant And Principal Scope
|
|
2
|
+
|
|
3
|
+
Runner always binds tenant and principal from trusted server-side context, not
|
|
4
|
+
from model arguments. Choose the database enforcement mode deliberately:
|
|
5
|
+
|
|
6
|
+
| Diagnostic mode | What enforces scope | Protects against | Does not protect against |
|
|
7
|
+
| --- | --- | --- | --- |
|
|
8
|
+
| `application_scope` | Runner's fixed, parameterized predicates with one least-privilege credential | A model trying to widen tenant or principal arguments | A defect in Runner's predicate construction |
|
|
9
|
+
| `postgres_rls` | Runner predicates plus PostgreSQL row-level security | Omitted/wrong Runner predicates and pooled-session context leakage | A fully compromised process that can choose trusted settings while holding a broad credential |
|
|
10
|
+
| `tenant_bound` | Runner predicates plus a credential or process that cannot access other tenants | Query mistakes and a process that never receives organization-wide database authority | Incorrect grants or isolation in the credential/deployment itself |
|
|
11
|
+
|
|
12
|
+
These modes are defense in depth, not substitutes for least-privilege roles,
|
|
13
|
+
restricted views, application authorization, or staging-first validation.
|
|
14
|
+
|
|
15
|
+
## Default: Application-Level Scope
|
|
16
|
+
|
|
17
|
+
Existing source configs remain compatible. An omitted `database_scope`, or:
|
|
18
|
+
|
|
19
|
+
```json
|
|
20
|
+
{
|
|
21
|
+
"database_scope": { "mode": "application" },
|
|
22
|
+
"credential_scope": { "mode": "shared" }
|
|
23
|
+
}
|
|
24
|
+
```
|
|
25
|
+
|
|
26
|
+
means Runner adds reviewed tenant and optional principal predicates to every
|
|
27
|
+
supported read and write using one environment-bound credential. This stops
|
|
28
|
+
the model from selecting another tenant, because scope is not a tool argument.
|
|
29
|
+
It is still application-level isolation: a defect in Runner's SQL construction
|
|
30
|
+
could cross that boundary. Keep database permissions, restricted views, and
|
|
31
|
+
RLS where available.
|
|
32
|
+
|
|
33
|
+
## Trusted-Context Provenance
|
|
34
|
+
|
|
35
|
+
The tenant value is part of the security boundary. Runner accepts it through
|
|
36
|
+
these reviewed paths:
|
|
37
|
+
|
|
38
|
+
| Deployment | Trusted-context source | Diagnostic binding |
|
|
39
|
+
| --- | --- | --- |
|
|
40
|
+
| Local stdio or one explicitly single-tenant process | Operator-controlled environment or local development binding | `process_bound` |
|
|
41
|
+
| Shared Streamable HTTP | Verified signed JWT claims configured with `http_claims` and `session_auth` | `verified_http_session` |
|
|
42
|
+
| Cloud embedding | A verified per-session binding supplied by the embedding control plane | `verified_external_session` |
|
|
43
|
+
|
|
44
|
+
Runner does not treat a tool argument, query parameter, arbitrary MCP `_meta`
|
|
45
|
+
value, `X-Tenant-*` header, or unverified forwarded header as tenant or
|
|
46
|
+
principal authority. A shared HTTP catalog cannot mix claims-bound capabilities
|
|
47
|
+
with environment-bound capabilities.
|
|
48
|
+
|
|
49
|
+
For shared production HTTP, prefer asymmetric JWT verification so Runner holds
|
|
50
|
+
only public verification material. HS256 remains useful for local development
|
|
51
|
+
and controlled deployments but gives Runner access to the signing secret.
|
|
52
|
+
|
|
53
|
+
Run `doctor --json` or `tools preview --json` to inspect the effective
|
|
54
|
+
per-source assurance mode and trusted-context binding. Server startup prints
|
|
55
|
+
the same non-secret summary. An `application_scope` source used with verified
|
|
56
|
+
HTTP sessions is deliberately reported with a warning: authentication makes
|
|
57
|
+
the tenant value trustworthy, but it does not add an independent database
|
|
58
|
+
boundary.
|
|
59
|
+
|
|
60
|
+
## PostgreSQL RLS Mode
|
|
61
|
+
|
|
62
|
+
Add fixed setting names to a PostgreSQL source:
|
|
63
|
+
|
|
64
|
+
```json
|
|
65
|
+
{
|
|
66
|
+
"sources": {
|
|
67
|
+
"billing_postgres": {
|
|
68
|
+
"engine": "postgres",
|
|
69
|
+
"read_url_env": "BILLING_POSTGRES_READ_URL",
|
|
70
|
+
"write_url_env": "BILLING_POSTGRES_WRITE_URL",
|
|
71
|
+
"database_scope": {
|
|
72
|
+
"mode": "postgres_rls",
|
|
73
|
+
"tenant_setting": "synapsor.tenant_id",
|
|
74
|
+
"principal_setting": "synapsor.principal"
|
|
75
|
+
},
|
|
76
|
+
"credential_scope": { "mode": "shared" }
|
|
77
|
+
}
|
|
78
|
+
}
|
|
79
|
+
}
|
|
80
|
+
```
|
|
81
|
+
|
|
82
|
+
For each scoped transaction, Runner binds both values with parameterized
|
|
83
|
+
transaction-local `set_config(..., true)`. Reads, evidence, aggregates,
|
|
84
|
+
bounded-set selection, guarded writes, reconciliation, compensation, and
|
|
85
|
+
verification therefore use the same trusted context. Transaction-local values
|
|
86
|
+
are cleared on commit or rollback and are not reused on the next pooled
|
|
87
|
+
checkout.
|
|
88
|
+
|
|
89
|
+
Runner refuses hardened serving or writeback when the role or target cannot be
|
|
90
|
+
attested. Required properties include:
|
|
91
|
+
|
|
92
|
+
- RLS and `FORCE ROW LEVEL SECURITY` are enabled;
|
|
93
|
+
- the effective role is not a superuser, table owner, or `BYPASSRLS` role;
|
|
94
|
+
- applicable policies are permissive only where the reviewed access requires
|
|
95
|
+
them;
|
|
96
|
+
- every applicable operation policy references both configured settings;
|
|
97
|
+
- `SELECT`/`DELETE` have a suitable `USING` expression;
|
|
98
|
+
- `INSERT` has a suitable `WITH CHECK` expression;
|
|
99
|
+
- `UPDATE` has both `USING` and `WITH CHECK`.
|
|
100
|
+
|
|
101
|
+
### Example PostgreSQL policy
|
|
102
|
+
|
|
103
|
+
Use a non-owner read role and a separate non-owner write role. Adapt names and
|
|
104
|
+
grants to your schema:
|
|
105
|
+
|
|
106
|
+
Runner does not silently create or widen these policies. Review and apply the
|
|
107
|
+
SQL as a database owner, then run doctor using the steady-state non-owner role.
|
|
108
|
+
|
|
109
|
+
```sql
|
|
110
|
+
ALTER TABLE public.invoices ENABLE ROW LEVEL SECURITY;
|
|
111
|
+
ALTER TABLE public.invoices FORCE ROW LEVEL SECURITY;
|
|
112
|
+
|
|
113
|
+
CREATE POLICY invoices_reader_scope
|
|
114
|
+
ON public.invoices
|
|
115
|
+
FOR SELECT
|
|
116
|
+
TO runner_reader
|
|
117
|
+
USING (
|
|
118
|
+
tenant_id = current_setting('synapsor.tenant_id', true)
|
|
119
|
+
AND assigned_to = current_setting('synapsor.principal', true)
|
|
120
|
+
);
|
|
121
|
+
|
|
122
|
+
CREATE POLICY invoices_writer_select_scope
|
|
123
|
+
ON public.invoices
|
|
124
|
+
FOR SELECT
|
|
125
|
+
TO runner_writer
|
|
126
|
+
USING (
|
|
127
|
+
tenant_id = current_setting('synapsor.tenant_id', true)
|
|
128
|
+
AND assigned_to = current_setting('synapsor.principal', true)
|
|
129
|
+
);
|
|
130
|
+
|
|
131
|
+
CREATE POLICY invoices_writer_insert_scope
|
|
132
|
+
ON public.invoices
|
|
133
|
+
FOR INSERT
|
|
134
|
+
TO runner_writer
|
|
135
|
+
WITH CHECK (
|
|
136
|
+
tenant_id = current_setting('synapsor.tenant_id', true)
|
|
137
|
+
AND assigned_to = current_setting('synapsor.principal', true)
|
|
138
|
+
);
|
|
139
|
+
|
|
140
|
+
CREATE POLICY invoices_writer_update_scope
|
|
141
|
+
ON public.invoices
|
|
142
|
+
FOR UPDATE
|
|
143
|
+
TO runner_writer
|
|
144
|
+
USING (
|
|
145
|
+
tenant_id = current_setting('synapsor.tenant_id', true)
|
|
146
|
+
AND assigned_to = current_setting('synapsor.principal', true)
|
|
147
|
+
)
|
|
148
|
+
WITH CHECK (
|
|
149
|
+
tenant_id = current_setting('synapsor.tenant_id', true)
|
|
150
|
+
AND assigned_to = current_setting('synapsor.principal', true)
|
|
151
|
+
);
|
|
152
|
+
|
|
153
|
+
CREATE POLICY invoices_writer_delete_scope
|
|
154
|
+
ON public.invoices
|
|
155
|
+
FOR DELETE
|
|
156
|
+
TO runner_writer
|
|
157
|
+
USING (
|
|
158
|
+
tenant_id = current_setting('synapsor.tenant_id', true)
|
|
159
|
+
AND assigned_to = current_setting('synapsor.principal', true)
|
|
160
|
+
);
|
|
161
|
+
```
|
|
162
|
+
|
|
163
|
+
Grant only the operations and columns needed by the reviewed capabilities.
|
|
164
|
+
Policy expressions are not a replacement for grants.
|
|
165
|
+
|
|
166
|
+
Run the normal metadata checks before serving:
|
|
167
|
+
|
|
168
|
+
```bash
|
|
169
|
+
synapsor-runner doctor --config ./synapsor.runner.json
|
|
170
|
+
```
|
|
171
|
+
|
|
172
|
+
On a disposable or explicitly approved live target, also prove that a known
|
|
173
|
+
row disappears under a different tenant and under a different principal:
|
|
174
|
+
|
|
175
|
+
```bash
|
|
176
|
+
synapsor-runner doctor --config ./synapsor.runner.json --check-rls
|
|
177
|
+
```
|
|
178
|
+
|
|
179
|
+
`--check-rls` reads only through the configured read role. It does not mutate
|
|
180
|
+
the source. Startup and per-connection checks still fail closed when the
|
|
181
|
+
required role or policy properties are absent.
|
|
182
|
+
|
|
183
|
+
PostgreSQL RLS catches a missing application predicate. It does not make a
|
|
184
|
+
fully compromised Runner process safe if that process can choose arbitrary
|
|
185
|
+
trusted setting values while holding a credential that spans tenants. Use a
|
|
186
|
+
tenant-bound credential or isolated deployment for that threat model.
|
|
187
|
+
|
|
188
|
+
## Tenant-Bound Credentials
|
|
189
|
+
|
|
190
|
+
Contracts contain a resolver identifier, never a connection string:
|
|
191
|
+
|
|
192
|
+
```json
|
|
193
|
+
{
|
|
194
|
+
"credential_scope": {
|
|
195
|
+
"mode": "tenant_resolver",
|
|
196
|
+
"resolver": "production_tenant_credentials"
|
|
197
|
+
}
|
|
198
|
+
}
|
|
199
|
+
```
|
|
200
|
+
|
|
201
|
+
An embedding application supplies the matching resolver:
|
|
202
|
+
|
|
203
|
+
```ts
|
|
204
|
+
import {
|
|
205
|
+
startRunnerStreamableHttp,
|
|
206
|
+
type TenantCredentialResolver
|
|
207
|
+
} from "@synapsor/runner/runtime";
|
|
208
|
+
|
|
209
|
+
const credentialResolver: TenantCredentialResolver = {
|
|
210
|
+
id: "production_tenant_credentials",
|
|
211
|
+
async resolve({ source_name, access, tenant_id, principal }) {
|
|
212
|
+
const lease = await credentialBroker.issue({
|
|
213
|
+
source: source_name,
|
|
214
|
+
access,
|
|
215
|
+
tenant: tenant_id,
|
|
216
|
+
principal
|
|
217
|
+
});
|
|
218
|
+
|
|
219
|
+
return {
|
|
220
|
+
connection_url: lease.connectionUrl,
|
|
221
|
+
credential_id: lease.id,
|
|
222
|
+
expires_at: lease.expiresAt
|
|
223
|
+
};
|
|
224
|
+
}
|
|
225
|
+
};
|
|
226
|
+
|
|
227
|
+
await startRunnerStreamableHttp({
|
|
228
|
+
configPath: "./synapsor.runner.json",
|
|
229
|
+
credentialResolver
|
|
230
|
+
});
|
|
231
|
+
```
|
|
232
|
+
|
|
233
|
+
The resolver receives only verified trusted context. Runner partitions pools
|
|
234
|
+
by source, access type, tenant, principal, and credential identity; an expired
|
|
235
|
+
or rotated lease is closed before reuse. Resolution failures are fail-closed,
|
|
236
|
+
and connection URLs are not logged or stored in contracts.
|
|
237
|
+
|
|
238
|
+
The stock CLI does not load arbitrary executable resolver modules. For direct
|
|
239
|
+
CLI operation, run one Runner process per tenant with a tenant-bound credential
|
|
240
|
+
in the normal `read_url_env`/`write_url_env` variables and keep
|
|
241
|
+
`credential_scope.mode` as `shared`. This gives the process no cross-tenant
|
|
242
|
+
credential to misuse.
|
|
243
|
+
|
|
244
|
+
## MySQL
|
|
245
|
+
|
|
246
|
+
MySQL does not provide a native equivalent to PostgreSQL RLS. Runner therefore
|
|
247
|
+
does not label session variables or application predicates as independent
|
|
248
|
+
database enforcement.
|
|
249
|
+
|
|
250
|
+
Use one or more of:
|
|
251
|
+
|
|
252
|
+
- tenant-bound read/write credentials;
|
|
253
|
+
- restricted tenant-aware views or stored procedures owned and reviewed by
|
|
254
|
+
your application;
|
|
255
|
+
- separate schemas or databases;
|
|
256
|
+
- one isolated Runner deployment per tenant.
|
|
257
|
+
|
|
258
|
+
Runner's trusted-context predicates still narrow model behavior, but a shared
|
|
259
|
+
MySQL credential remains application-level scope unless the database grants,
|
|
260
|
+
views, or deployment physically prevent cross-tenant access.
|
|
261
|
+
|
|
262
|
+
## Verification
|
|
263
|
+
|
|
264
|
+
The repository's disposable PostgreSQL proof covers:
|
|
265
|
+
|
|
266
|
+
```bash
|
|
267
|
+
corepack pnpm test:database-scope
|
|
268
|
+
```
|
|
269
|
+
|
|
270
|
+
It verifies an intentionally unscoped query is still denied by RLS, correct
|
|
271
|
+
tenant/principal access succeeds, cross-scope access fails, `WITH CHECK`
|
|
272
|
+
prevents row moves, pooled scope does not leak, guarded writeback and
|
|
273
|
+
compensation remain scoped, and unsafe roles/policies fail doctor.
|
|
@@ -10,11 +10,15 @@ Summary from the current dependency set:
|
|
|
10
10
|
|
|
11
11
|
| License | Package entries |
|
|
12
12
|
| --- | ---: |
|
|
13
|
-
| MIT |
|
|
13
|
+
| MIT | 200 |
|
|
14
14
|
| Apache-2.0 | 4 |
|
|
15
15
|
| BSD-3-Clause | 3 |
|
|
16
16
|
| BSD-2-Clause | 1 |
|
|
17
|
-
| ISC |
|
|
17
|
+
| ISC | 15 |
|
|
18
|
+
| `(BSD-3-Clause OR GPL-2.0)` | 1 |
|
|
19
|
+
| `(MIT AND Zlib)` | 1 |
|
|
20
|
+
| `0BSD` | 1 |
|
|
21
|
+
| `(MIT OR CC0-1.0)` | 1 |
|
|
18
22
|
|
|
19
23
|
Apache-2.0 dependency entries reported by pnpm:
|
|
20
24
|
|
package/docs/doctor.md
CHANGED
|
@@ -20,6 +20,23 @@ The default check validates:
|
|
|
20
20
|
- MCP tool boundary, including absence of raw SQL and commit tools;
|
|
21
21
|
- local store stats.
|
|
22
22
|
|
|
23
|
+
For a source configured with `database_scope.mode = "postgres_rls"`, doctor
|
|
24
|
+
also verifies RLS/FORCE status, effective-role bypass risk, applicable
|
|
25
|
+
operation policies, `USING`/`WITH CHECK`, and both configured trusted-setting
|
|
26
|
+
names. Hardened mode fails rather than silently falling back to
|
|
27
|
+
application-only predicates.
|
|
28
|
+
|
|
29
|
+
On a disposable or explicitly approved live target, add:
|
|
30
|
+
|
|
31
|
+
```bash
|
|
32
|
+
synapsor-runner doctor --config synapsor.runner.json --check-rls
|
|
33
|
+
```
|
|
34
|
+
|
|
35
|
+
This performs a read-only cross-tenant and cross-principal canary and checks
|
|
36
|
+
that transaction-local values do not survive the pooled transaction. See
|
|
37
|
+
[Database-Enforced Tenant And Principal
|
|
38
|
+
Scope](database-enforced-scope.md).
|
|
39
|
+
|
|
23
40
|
## App-Owned Handler Checks
|
|
24
41
|
|
|
25
42
|
For `http_handler` executors, add `--check-handlers`:
|
|
@@ -0,0 +1,179 @@
|
|
|
1
|
+
# Agent Effect Regression
|
|
2
|
+
|
|
3
|
+
Contract tests answer whether a reviewed capability boundary is still valid.
|
|
4
|
+
Effect regression answers a different question:
|
|
5
|
+
|
|
6
|
+
> Did a model, prompt, policy, capability, or application change alter the
|
|
7
|
+
> business effect the agent proposes?
|
|
8
|
+
|
|
9
|
+
Runner effect evaluation is provider-neutral and offline. Runner snapshots
|
|
10
|
+
existing replay evidence into a versioned fixture, then compares a result
|
|
11
|
+
exported by your agent harness. It does not invoke a model, query a source
|
|
12
|
+
database, approve a proposal, or apply a write.
|
|
13
|
+
|
|
14
|
+
## Create A Baseline
|
|
15
|
+
|
|
16
|
+
Start from an existing replay, proposal, or shadow-study case:
|
|
17
|
+
|
|
18
|
+
```bash
|
|
19
|
+
synapsor-runner effect fixture create \
|
|
20
|
+
--from-replay replay_wrp_... \
|
|
21
|
+
--request "Waive the late fee on invoice INV-3001" \
|
|
22
|
+
--contract ./synapsor.contract.json \
|
|
23
|
+
--capability-call billing.inspect_invoice \
|
|
24
|
+
--capability-call billing.propose_late_fee_waiver \
|
|
25
|
+
--store ./.synapsor/local.db \
|
|
26
|
+
--out ./effects/late-fee.json
|
|
27
|
+
```
|
|
28
|
+
|
|
29
|
+
Use `--from-proposal wrp_...` or `--from-shadow-case shc_...` for the other
|
|
30
|
+
sources. The contract is required so Runner records its version and the
|
|
31
|
+
reviewed kept-out fields. Every `--capability-call` must exist in that
|
|
32
|
+
contract.
|
|
33
|
+
|
|
34
|
+
The fixture contains:
|
|
35
|
+
|
|
36
|
+
- the business request;
|
|
37
|
+
- trusted tenant and principal context from the stored proposal;
|
|
38
|
+
- bounded evidence and query-fingerprint snapshots already in the ledger;
|
|
39
|
+
- the permitted capability calls;
|
|
40
|
+
- expected target, business diff, policy result, conflict code, and outcome;
|
|
41
|
+
- the contract version and fields that must remain hidden;
|
|
42
|
+
- a digest that catches unreviewed fixture edits.
|
|
43
|
+
|
|
44
|
+
It contains no database URL, provider credential, or live source connection.
|
|
45
|
+
The secret and hidden-field checks fail closed while creating or loading it.
|
|
46
|
+
|
|
47
|
+
## Import A Provider-Neutral Result
|
|
48
|
+
|
|
49
|
+
Create a template:
|
|
50
|
+
|
|
51
|
+
```bash
|
|
52
|
+
synapsor-runner effect result init \
|
|
53
|
+
--fixture ./effects/late-fee.json \
|
|
54
|
+
--out ./effects/late-fee.result.json
|
|
55
|
+
```
|
|
56
|
+
|
|
57
|
+
Have your agent test harness populate this JSON after a propose-only run. The
|
|
58
|
+
format records:
|
|
59
|
+
|
|
60
|
+
- capability names and model-controlled arguments;
|
|
61
|
+
- trusted context supplied by the application outside those arguments;
|
|
62
|
+
- target, proposed diff, policy, conflict, and outcome category;
|
|
63
|
+
- fields observed by the harness;
|
|
64
|
+
- whether the harness made any new source read;
|
|
65
|
+
- whether the source database changed.
|
|
66
|
+
|
|
67
|
+
The adapter may use any model provider or application framework. Runner imports
|
|
68
|
+
the result; it does not require a proprietary LLM API.
|
|
69
|
+
|
|
70
|
+
The evaluation harness must disable apply/writeback. A result reporting a
|
|
71
|
+
source mutation always fails. A new source read also fails by default. Pass
|
|
72
|
+
`--allow-live-read` only when the test environment is explicitly disposable;
|
|
73
|
+
that flag never permits a write.
|
|
74
|
+
|
|
75
|
+
## Compare In CI
|
|
76
|
+
|
|
77
|
+
```bash
|
|
78
|
+
synapsor-runner effect run \
|
|
79
|
+
--fixture ./effects/late-fee.json \
|
|
80
|
+
--result ./effects/late-fee.result.json
|
|
81
|
+
```
|
|
82
|
+
|
|
83
|
+
`effect compare` is an alias. Both return a nonzero status when any reviewed
|
|
84
|
+
effect changes. Reports cover:
|
|
85
|
+
|
|
86
|
+
- capability calls and capability-surface expansion;
|
|
87
|
+
- trusted-context drift or model arguments selecting tenant/principal;
|
|
88
|
+
- target-object and business-diff changes;
|
|
89
|
+
- policy, conflict, outcome-category, and contract-version changes;
|
|
90
|
+
- kept-out field exposure;
|
|
91
|
+
- source mutation or an unapproved new read.
|
|
92
|
+
|
|
93
|
+
Use `--format json` or `--format junit` for CI:
|
|
94
|
+
|
|
95
|
+
```bash
|
|
96
|
+
synapsor-runner effect run \
|
|
97
|
+
--dataset ./effects/dataset.json \
|
|
98
|
+
--results-dir ./effects/results \
|
|
99
|
+
--format junit \
|
|
100
|
+
--out ./artifacts/effect-results.xml
|
|
101
|
+
```
|
|
102
|
+
|
|
103
|
+
Dataset fixture paths are relative and cannot escape the dataset directory.
|
|
104
|
+
Each result is named `<fixture_id>.result.json`.
|
|
105
|
+
|
|
106
|
+
### Run an application-owned adapter
|
|
107
|
+
|
|
108
|
+
For a real application harness, Runner can launch one provider-neutral command
|
|
109
|
+
directly, without a shell:
|
|
110
|
+
|
|
111
|
+
```bash
|
|
112
|
+
synapsor-runner effect run \
|
|
113
|
+
--dataset ./effects/dataset.json \
|
|
114
|
+
--adapter node \
|
|
115
|
+
--adapter-arg ./app/effect-adapter.mjs \
|
|
116
|
+
--result-origin deterministic-application \
|
|
117
|
+
--format junit \
|
|
118
|
+
--out ./artifacts/effect-results.xml
|
|
119
|
+
```
|
|
120
|
+
|
|
121
|
+
Use `--result-origin external-model` when the adapter actually calls a model.
|
|
122
|
+
The origin is included in text, JSON, and JUnit provenance so deterministic
|
|
123
|
+
application results are not presented as model evidence.
|
|
124
|
+
|
|
125
|
+
Runner starts the command with `shell: false`, a bounded argument list,
|
|
126
|
+
timeout/output limits, and a minimal environment that omits ambient database
|
|
127
|
+
URLs and tokens. It sets `SYNAPSOR_EFFECT_MODE=propose_only`,
|
|
128
|
+
`SYNAPSOR_EFFECT_FIXTURE_PATH`, and
|
|
129
|
+
`SYNAPSOR_EFFECT_SOURCE_DATABASE_CHANGED=false`. The command must emit exactly
|
|
130
|
+
one canonical result JSON document to stdout.
|
|
131
|
+
|
|
132
|
+
This is a process boundary, not a sandbox. Adopter-owned code can deliberately
|
|
133
|
+
load its own credentials, so keep the adapter propose-only and point it at
|
|
134
|
+
fixtures or an explicitly disposable source. Runner still rejects every result
|
|
135
|
+
that reports source mutation.
|
|
136
|
+
|
|
137
|
+
The repository's working example is
|
|
138
|
+
[`examples/support-billing-agent/app/effect-adapter.mjs`](../examples/support-billing-agent/app/effect-adapter.mjs),
|
|
139
|
+
and [`.github/workflows/effect-regression.yml`](https://github.com/Synapsor/Synapsor-Runner/blob/main/.github/workflows/effect-regression.yml)
|
|
140
|
+
runs it and uploads JUnit output on relevant pull requests.
|
|
141
|
+
|
|
142
|
+
## Accept An Intentional Change
|
|
143
|
+
|
|
144
|
+
Runner never updates a baseline during `run` or `compare`. After reviewing an
|
|
145
|
+
intentional business change, accept it explicitly:
|
|
146
|
+
|
|
147
|
+
```bash
|
|
148
|
+
synapsor-runner effect accept \
|
|
149
|
+
--fixture ./effects/late-fee.json \
|
|
150
|
+
--result ./effects/late-fee.result.json \
|
|
151
|
+
--actor release-engineer@example.com \
|
|
152
|
+
--reason "Reviewed billing policy v2" \
|
|
153
|
+
--in-place \
|
|
154
|
+
--yes
|
|
155
|
+
```
|
|
156
|
+
|
|
157
|
+
Use `--out ./effects/late-fee.v2.json` instead of `--in-place` to preserve the
|
|
158
|
+
old fixture. Acceptance appends actor, reason, timestamp, and before/after
|
|
159
|
+
baseline digests. It cannot waive:
|
|
160
|
+
|
|
161
|
+
- a mismatched fixture identity;
|
|
162
|
+
- source mutation;
|
|
163
|
+
- a new source read;
|
|
164
|
+
- trusted-context drift or model-controlled tenant/principal;
|
|
165
|
+
- hidden-field exposure.
|
|
166
|
+
|
|
167
|
+
Commit the accepted fixture and review it like application code.
|
|
168
|
+
|
|
169
|
+
## Boundaries
|
|
170
|
+
|
|
171
|
+
- This is not a general agent workflow engine.
|
|
172
|
+
- Offline `--result`/`--results-dir` mode executes no provider command or
|
|
173
|
+
application code. `--adapter` explicitly executes the adopter-owned command
|
|
174
|
+
under the bounded process contract above.
|
|
175
|
+
- This does not prove model quality outside the imported observations.
|
|
176
|
+
- This complements [`contract test`](contract-testing.md); a contract can
|
|
177
|
+
conform while an agent's proposed business effect still regresses.
|
|
178
|
+
- Replay remains read-only. Creating an effect fixture does not replay or
|
|
179
|
+
reapply the original write.
|
|
@@ -0,0 +1,110 @@
|
|
|
1
|
+
# Fresh-Developer Usability Protocol
|
|
2
|
+
|
|
3
|
+
Use this protocol with five developers who have not previously used Synapsor.
|
|
4
|
+
It is a launch gate, not an automated test. A maintainer may rehearse the
|
|
5
|
+
protocol, but must not report a participant result unless that person actually
|
|
6
|
+
completed the session.
|
|
7
|
+
|
|
8
|
+
## Goal
|
|
9
|
+
|
|
10
|
+
Measure whether a developer can connect an existing supported staging project,
|
|
11
|
+
draft one disabled Safe Action with a coding agent, receive an exact Data PR,
|
|
12
|
+
approve outside the agent, and find the guarded receipt without learning the
|
|
13
|
+
canonical JSON or SQL-like DSL first.
|
|
14
|
+
|
|
15
|
+
Measure product time separately from the initial package download. Start the
|
|
16
|
+
product timer only after `synapsor-runner --version` succeeds.
|
|
17
|
+
|
|
18
|
+
## Participant Setup
|
|
19
|
+
|
|
20
|
+
Provide each participant with:
|
|
21
|
+
|
|
22
|
+
- a fresh clone of their application or the support-plan-credit reference app;
|
|
23
|
+
- Node 22.13 or newer;
|
|
24
|
+
- a disposable Postgres staging database and least-privilege read/write roles;
|
|
25
|
+
- exported database URLs and trusted tenant/principal values;
|
|
26
|
+
- current stable Cursor, with no existing Synapsor configuration;
|
|
27
|
+
- the README only. Do not give verbal product-architecture instruction.
|
|
28
|
+
|
|
29
|
+
Never use production credentials or data. Record the OS, Node version, Cursor
|
|
30
|
+
version, package-install method, database engine, and whether the package was
|
|
31
|
+
already cached.
|
|
32
|
+
|
|
33
|
+
## Tasks
|
|
34
|
+
|
|
35
|
+
Ask the participant to complete these tasks without a maintainer taking control:
|
|
36
|
+
|
|
37
|
+
1. Run the four-second synthetic proof and identify whether the model can call
|
|
38
|
+
`approve`, `apply`, `commit`, or raw SQL.
|
|
39
|
+
2. Connect the staging project and confirm the trusted organization scope.
|
|
40
|
+
3. Tell Cursor one business action to make safe and use the generated
|
|
41
|
+
`/synapsor-protect` path or exact fallback prompt.
|
|
42
|
+
4. Find the disabled action file and explain why editing it does not activate a
|
|
43
|
+
tool.
|
|
44
|
+
5. Open the Workbench effect preview and confirm the source row is unchanged.
|
|
45
|
+
6. Explicitly activate the reviewed digest through the operator-controlled UI.
|
|
46
|
+
7. Reconnect Cursor if instructed and create one proposal through the semantic
|
|
47
|
+
tool.
|
|
48
|
+
8. Review the exact before/after Data PR, approve outside Cursor, and apply it.
|
|
49
|
+
9. Locate the receipt/replay record and explain what happens on retry.
|
|
50
|
+
10. Draft a second disabled action without changing the active tool surface.
|
|
51
|
+
|
|
52
|
+
Do not rescue a participant until they have been blocked for two minutes. Record
|
|
53
|
+
the exact screen, command, wording, or missing prerequisite that blocked them.
|
|
54
|
+
|
|
55
|
+
## Measurements
|
|
56
|
+
|
|
57
|
+
For each participant, record:
|
|
58
|
+
|
|
59
|
+
| Measure | Target |
|
|
60
|
+
| --- | --- |
|
|
61
|
+
| Installed CLI to first own-data Data PR | under 5 minutes |
|
|
62
|
+
| Installed CLI to first guarded receipt | under 8 minutes |
|
|
63
|
+
| Second disabled action draft after setup | under 2 minutes |
|
|
64
|
+
| Second action ready for explicit activation | under 5 minutes |
|
|
65
|
+
| Handwritten canonical JSON or DSL | none on the happy path |
|
|
66
|
+
| Extra CLI command after Cursor proposal call | none |
|
|
67
|
+
| Unsafe authority exposed to Cursor | none |
|
|
68
|
+
|
|
69
|
+
Also record every wrong turn, maintainer intervention, misunderstood trust
|
|
70
|
+
boundary, and request for a model-provider key or Synapsor account.
|
|
71
|
+
|
|
72
|
+
## Pass Criteria
|
|
73
|
+
|
|
74
|
+
The launch gate passes only when all five participants:
|
|
75
|
+
|
|
76
|
+
- complete the first Data PR without handwritten JSON or DSL;
|
|
77
|
+
- correctly identify that activation, approval, and apply are outside the
|
|
78
|
+
model-facing MCP surface;
|
|
79
|
+
- observe unchanged source data before approval;
|
|
80
|
+
- produce one guarded receipt; and
|
|
81
|
+
- expose only the intended semantic read/proposal tools in Cursor.
|
|
82
|
+
|
|
83
|
+
At least four of five must meet every timing target. Any cross-tenant result,
|
|
84
|
+
hidden-field exposure, pre-approval mutation, automatic activation, raw SQL
|
|
85
|
+
tool, or model-visible commit authority is a release blocker regardless of
|
|
86
|
+
timing.
|
|
87
|
+
|
|
88
|
+
## Reporting Template
|
|
89
|
+
|
|
90
|
+
Record one row per real participant:
|
|
91
|
+
|
|
92
|
+
```text
|
|
93
|
+
Participant:
|
|
94
|
+
Date:
|
|
95
|
+
Environment:
|
|
96
|
+
Cold package download:
|
|
97
|
+
Product timer start:
|
|
98
|
+
First Data PR:
|
|
99
|
+
First receipt:
|
|
100
|
+
Second draft:
|
|
101
|
+
Second activation-ready action:
|
|
102
|
+
Interventions:
|
|
103
|
+
Tool surface observed:
|
|
104
|
+
Safety-boundary explanation:
|
|
105
|
+
Outcome: pass | fail
|
|
106
|
+
```
|
|
107
|
+
|
|
108
|
+
Until five real sessions are recorded, report this gate as **owner verification
|
|
109
|
+
pending**. Do not infer usability from unit tests, protocol tests, or a
|
|
110
|
+
maintainer-run demo.
|