@sponsio/sdk 0.1.0 → 0.2.0-alpha.8
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/contracts/benchmark/agentdojo.yaml +551 -0
- package/contracts/benchmark/odcv_bench.yaml +393 -0
- package/contracts/benchmark/redcode_exec.yaml +488 -0
- package/contracts/benchmark/swebench.yaml +176 -0
- package/contracts/benchmark/tau2_bench.yaml +984 -0
- package/contracts/capability/destructive.yaml +213 -0
- package/contracts/capability/factual-claims.yaml +79 -0
- package/contracts/capability/host-config-integrity.yaml +42 -9
- package/contracts/capability/self-modify.yaml +32 -1
- package/contracts/capability/shell.yaml +19 -10
- package/contracts/incident/claude-code-secret-bypass.yaml +48 -16
- package/contracts/incident/cursor-railway-wipe.yaml +60 -4
- package/contracts/incident/openclaw.yaml +9 -62
- package/contracts/premium/.gitkeep +5 -6
- package/dist/cli/cli.js +32 -0
- package/dist/cli/cli.js.map +1 -1
- package/dist/cli/packs.d.ts.map +1 -1
- package/dist/cli/packs.js +14 -8
- package/dist/cli/packs.js.map +1 -1
- package/dist/cli/patterns.d.ts +5 -4
- package/dist/cli/patterns.d.ts.map +1 -1
- package/dist/cli/patterns.js +11 -10
- package/dist/cli/patterns.js.map +1 -1
- package/dist/cli/prompt.js +5 -5
- package/dist/cli/prompt.js.map +1 -1
- package/dist/cli/scan.js +2 -2
- package/dist/cli/skill.d.ts +1 -1
- package/dist/cli/skill.js +1 -1
- package/dist/cli/validate.js +5 -5
- package/dist/cli/validate.js.map +1 -1
- package/dist/cloud/client.d.ts +76 -0
- package/dist/cloud/client.d.ts.map +1 -0
- package/dist/cloud/client.js +198 -0
- package/dist/cloud/client.js.map +1 -0
- package/dist/cloud/evidence.d.ts +108 -0
- package/dist/cloud/evidence.d.ts.map +1 -0
- package/dist/cloud/evidence.js +184 -0
- package/dist/cloud/evidence.js.map +1 -0
- package/dist/cloud/privacy.d.ts +67 -0
- package/dist/cloud/privacy.d.ts.map +1 -0
- package/dist/cloud/privacy.js +170 -0
- package/dist/cloud/privacy.js.map +1 -0
- package/dist/cloud/view-model.d.ts +49 -0
- package/dist/cloud/view-model.d.ts.map +1 -0
- package/dist/cloud/view-model.js +172 -0
- package/dist/cloud/view-model.js.map +1 -0
- package/dist/core/config-loader.d.ts.map +1 -1
- package/dist/core/config-loader.js +81 -9
- package/dist/core/config-loader.js.map +1 -1
- package/dist/core/evaluator.d.ts +11 -1
- package/dist/core/evaluator.d.ts.map +1 -1
- package/dist/core/evaluator.js +275 -13
- package/dist/core/evaluator.js.map +1 -1
- package/dist/core/formula.d.ts +76 -17
- package/dist/core/formula.d.ts.map +1 -1
- package/dist/core/formula.js +83 -1
- package/dist/core/formula.js.map +1 -1
- package/dist/core/grounding.d.ts +44 -8
- package/dist/core/grounding.d.ts.map +1 -1
- package/dist/core/grounding.js +197 -26
- package/dist/core/grounding.js.map +1 -1
- package/dist/core/nl-parser.d.ts +2 -2
- package/dist/core/nl-parser.d.ts.map +1 -1
- package/dist/core/nl-parser.js +617 -41
- package/dist/core/nl-parser.js.map +1 -1
- package/dist/core/pack-loader.d.ts.map +1 -1
- package/dist/core/pack-loader.js +14 -2
- package/dist/core/pack-loader.js.map +1 -1
- package/dist/core/parser.d.ts.map +1 -1
- package/dist/core/parser.js +30 -1
- package/dist/core/parser.js.map +1 -1
- package/dist/core/pattern-factory.d.ts.map +1 -1
- package/dist/core/pattern-factory.js +14 -1
- package/dist/core/pattern-factory.js.map +1 -1
- package/dist/core/patterns.d.ts +118 -1
- package/dist/core/patterns.d.ts.map +1 -1
- package/dist/core/patterns.js +189 -10
- package/dist/core/patterns.js.map +1 -1
- package/dist/core/session-log.d.ts.map +1 -1
- package/dist/core/session-log.js +20 -1
- package/dist/core/session-log.js.map +1 -1
- package/dist/core/sto.d.ts +21 -27
- package/dist/core/sto.d.ts.map +1 -1
- package/dist/core/sto.js +30 -33
- package/dist/core/sto.js.map +1 -1
- package/dist/core/tool-names.d.ts +59 -0
- package/dist/core/tool-names.d.ts.map +1 -0
- package/dist/core/tool-names.js +128 -0
- package/dist/core/tool-names.js.map +1 -0
- package/dist/index.d.ts +181 -32
- package/dist/index.d.ts.map +1 -1
- package/dist/index.js +415 -67
- package/dist/index.js.map +1 -1
- package/dist/integrations/claude-agent.d.ts +5 -0
- package/dist/integrations/claude-agent.d.ts.map +1 -1
- package/dist/integrations/claude-agent.js +42 -13
- package/dist/integrations/claude-agent.js.map +1 -1
- package/dist/integrations/google-adk.d.ts +5 -3
- package/dist/integrations/google-adk.d.ts.map +1 -1
- package/dist/integrations/google-adk.js +9 -5
- package/dist/integrations/google-adk.js.map +1 -1
- package/dist/integrations/langchain.d.ts +7 -0
- package/dist/integrations/langchain.d.ts.map +1 -1
- package/dist/integrations/langchain.js +21 -3
- package/dist/integrations/langchain.js.map +1 -1
- package/dist/integrations/openai-agents.d.ts +22 -10
- package/dist/integrations/openai-agents.d.ts.map +1 -1
- package/dist/integrations/openai-agents.js +55 -15
- package/dist/integrations/openai-agents.js.map +1 -1
- package/dist/integrations/openai.d.ts +11 -0
- package/dist/integrations/openai.d.ts.map +1 -1
- package/dist/integrations/openai.js +130 -48
- package/dist/integrations/openai.js.map +1 -1
- package/dist/integrations/vercel-ai.d.ts +20 -9
- package/dist/integrations/vercel-ai.d.ts.map +1 -1
- package/dist/integrations/vercel-ai.js +131 -32
- package/dist/integrations/vercel-ai.js.map +1 -1
- package/package.json +17 -9
- package/prompts/onboard.md +26 -17
- package/prompts/scan.md +3 -5
- package/skills/SKILL.md +13 -83
- package/contracts/core/llm_safety.yaml +0 -140
- package/prompts/refresh.md +0 -118
package/prompts/scan.md
CHANGED
|
@@ -105,11 +105,9 @@ to enforce and they wedge the agent. For each tool in
|
|
|
105
105
|
here — take them seriously. A policy doc saying "no destructive
|
|
106
106
|
Railway calls without confirmation" should produce a concrete
|
|
107
107
|
arg_blacklist + irreversible_once pair.
|
|
108
|
-
-
|
|
109
|
-
|
|
110
|
-
|
|
111
|
-
(`arg_blacklist`, `rate_limit`, `irreversible_once`) over
|
|
112
|
-
trace-aware ones.
|
|
108
|
+
- Prefer single-event patterns (`arg_blacklist`, `rate_limit`,
|
|
109
|
+
`irreversible_once`) unless the code clearly establishes a required
|
|
110
|
+
ordering; only then reach for sequence-shaped rules.
|
|
113
111
|
- **`existing_yaml`**: don't duplicate rules already there. Read its
|
|
114
112
|
contracts, only emit gaps.
|
|
115
113
|
|
package/skills/SKILL.md
CHANGED
|
@@ -1,11 +1,11 @@
|
|
|
1
1
|
---
|
|
2
2
|
name: sponsio
|
|
3
|
-
description: Install, observe, tune,
|
|
3
|
+
description: Install, observe, tune, and enforce Sponsio: a runtime contract layer for LLM agents that blocks unsafe tool calls and scores output quality against declared rules. Use when the user wants to set up / add / install Sponsio, add guardrails or runtime safety to an LLM agent, generate or refine a sponsio.yaml, audit tool configurations for risks (data leaks, unguarded writes, missing confirmations), explain or review existing contracts, check what Sponsio would have blocked (`sponsio report`), move from observe to enforce mode, or debug why a contract is (or isn't) firing. Triggers on phrases like "set up sponsio", "add sponsio", "install sponsio", "add guardrails", "monitor my agent", "harden my agent", "audit my agent", "generate contracts", "explain my sponsio.yaml", "sponsio report", "flip to enforce", "false positive", "why is this rule firing".
|
|
4
4
|
---
|
|
5
5
|
|
|
6
6
|
# Sponsio — Agent Safety Lifecycle Companion
|
|
7
7
|
|
|
8
|
-
Sponsio is a Python/TypeScript runtime safety layer for LLM agents: it evaluates deterministic contracts against each tool call and can block (enforce) or just log (observe) violations.
|
|
8
|
+
Sponsio is a Python/TypeScript runtime safety layer for LLM agents: it evaluates deterministic contracts against each tool call and can block (enforce) or just log (observe) violations. The engine is deterministic-only. This skill covers the full lifecycle — first-time setup, contract authoring/review, observe-mode tuning, and flipping to enforce — by orchestrating Sponsio's CLI and explaining its output in plain language.
|
|
9
9
|
|
|
10
10
|
This skill does NOT reimplement Sponsio's logic; it calls the CLI and interprets results.
|
|
11
11
|
|
|
@@ -18,7 +18,6 @@ Dispatch by what the user is trying to do. Pick ONE workflow and follow it; do n
|
|
|
18
18
|
| Setting up Sponsio for the first time in a project ("add sponsio", "install sponsio", "add guardrails") | **W1 — Initial setup** |
|
|
19
19
|
| Handing you a codebase and asking "what could go wrong?" / wants a fresh contract file from scratch / has a policy doc to encode | **W2 — Audit & refine** |
|
|
20
20
|
| Has Sponsio running in observe mode and wants to review violations, tune thresholds, silence false positives | **W3 — Tune in observe** |
|
|
21
|
-
| Wants to re-mine contracts from accumulated production traces / periodically maintain the library | **W3b — Refresh from traces** |
|
|
22
21
|
| Ready to ship — wants to move from observe to enforce, needs regression confidence | **W4 — Flip to enforce** |
|
|
23
22
|
| Sponsio errored, a rule isn't firing when it should, a rule is firing when it shouldn't | **W5 — Troubleshoot** |
|
|
24
23
|
|
|
@@ -64,7 +63,7 @@ Goal: from "project has no Sponsio" to "agent runs under observe mode with a san
|
|
|
64
63
|
|
|
65
64
|
| Pack | Auto-included when… | Notes |
|
|
66
65
|
|---|---|---|
|
|
67
|
-
| `sponsio:core/universal` | Always | Empty stub
|
|
66
|
+
| `sponsio:core/universal` | Always | Empty stub, kept so existing `include:` lines don't error. |
|
|
68
67
|
| `sponsio:core/runaway` | Framework runs a multi-step loop (langgraph/crewai/…) | token budget, delegation depth, loop detection; no LLM calls |
|
|
69
68
|
| `sponsio:capability/shell` | A tool name matches `{bash, shell, exec, execute, execute_command, run_command, run_shell, run_bash, terminal, subprocess}` | Auto-fills `tool_rename:` if the user's tool name isn't the canonical `exec` |
|
|
70
69
|
| `sponsio:capability/filesystem` | A tool name matches `{read, read_file, open_file, write, write_file, edit, edit_file, apply_patch, patch_file, ...}` | Auto-fills `tool_rename:` and `workspace:` |
|
|
@@ -92,17 +91,16 @@ Sponsio contracts come from four sources, mixable in one yaml:
|
|
|
92
91
|
| 1 | **Shipped packs** | Pre-built, parameterized rule sets (`sponsio:core/universal`, `sponsio:capability/shell`, …) | Hand-add `include: [sponsio:<spec>]` — or W1's `onboard` does it automatically |
|
|
93
92
|
| 2 | **Extraction** | AST + optional LLM inference from your code / policy docs / execution traces | `sponsio scan <paths> [--llm] [--policy <doc>] [-t <trace-glob>]` |
|
|
94
93
|
| 3 | **User input** | An NL sentence or a structured dict the user writes | Hand-edit `sponsio.yaml`; validate a single NL string with `sponsio validate "<NL>"` |
|
|
95
|
-
| 4 | **Pattern library** | Deterministic parameterized templates (`rate_limit`, `must_precede`, `arg_blacklist`, …) — full list via `sponsio patterns
|
|
94
|
+
| 4 | **Pattern library** | Deterministic parameterized templates (`rate_limit`, `must_precede`, `arg_blacklist`, …) — full list via `sponsio patterns` | `sponsio patterns` to browse; hand-write the YAML entry |
|
|
96
95
|
|
|
97
96
|
Match the user's input to the source(s):
|
|
98
97
|
|
|
99
98
|
- "Explain / review my `sponsio.yaml`" → source 1 and/or others already applied; jump to "Explain contracts" below.
|
|
100
99
|
- "Scan my agent code" → source 2, code-only.
|
|
101
100
|
- "We have a security policy document" → source 2, add `--policy <doc> --llm`.
|
|
102
|
-
- "We already have session logs / OTLP traces" → source 2, add `-t '<glob>'` (trace mining; no LLM needed).
|
|
103
101
|
- "I know the pattern I want but not the syntax" → source 4, then show them the yaml entry.
|
|
104
102
|
|
|
105
|
-
If ambiguous: ask ONE question — "(a) scan your code, (b) extract from a policy document
|
|
103
|
+
If ambiguous: ask ONE question — "(a) scan your code, or (b) extract from a policy document?"
|
|
106
104
|
|
|
107
105
|
### Run scan (when extraction is needed)
|
|
108
106
|
|
|
@@ -115,12 +113,9 @@ sponsio scan <PATHS> --agent <AGENT> --llm -o ./sponsio.yaml
|
|
|
115
113
|
|
|
116
114
|
# + policy doc:
|
|
117
115
|
sponsio scan <PATHS> --policy <POLICY.md> --llm -o ./sponsio.yaml
|
|
118
|
-
|
|
119
|
-
# + trace mining (works on OTLP/JSON, OTLP JSONL, native, Sponsio session logs):
|
|
120
|
-
sponsio scan <PATHS> -t '~/.sponsio/sessions/<agent>/*.jsonl' -o ./sponsio.yaml
|
|
121
116
|
```
|
|
122
117
|
|
|
123
|
-
Scan auto-validates before writing; only contracts that parse cleanly are saved. Source-tagged with `source: scan` / `source: policy
|
|
118
|
+
Scan auto-validates before writing; only contracts that parse cleanly are saved. Source-tagged with `source: scan` / `source: policy`.
|
|
124
119
|
|
|
125
120
|
### Validate existing yaml (explain-only path)
|
|
126
121
|
|
|
@@ -128,7 +123,7 @@ Scan auto-validates before writing; only contracts that parse cleanly are saved.
|
|
|
128
123
|
sponsio validate --config ./sponsio.yaml --json
|
|
129
124
|
```
|
|
130
125
|
|
|
131
|
-
JSON shape per entry: `{nl, ok, type: "det"|"
|
|
126
|
+
JSON shape per entry: `{nl, ok, type: "det"|"unknown", pattern, formula, agent, section}`. `ok: false` means the NL didn't match any pattern — surface those as "ambiguous — needs refinement".
|
|
132
127
|
|
|
133
128
|
### Explain contracts (use this as the report template)
|
|
134
129
|
|
|
@@ -138,7 +133,7 @@ JSON shape per entry: `{nl, ok, type: "det"|"sto"|"unknown", pattern, formula, a
|
|
|
138
133
|
**Setup**: <N> contracts total, <sources breakdown>. Mode: <observe|enforce>.
|
|
139
134
|
|
|
140
135
|
### Active packs
|
|
141
|
-
- `sponsio:core/universal` (empty stub
|
|
136
|
+
- `sponsio:core/universal` (empty stub, kept so existing `include:` lines don't error)
|
|
142
137
|
- `sponsio:capability/shell` (11 det) — guards on <tool name after rename>
|
|
143
138
|
<...>
|
|
144
139
|
|
|
@@ -211,66 +206,6 @@ With the assumption, the rule stays silent until the integration actually emits
|
|
|
211
206
|
|
|
212
207
|
---
|
|
213
208
|
|
|
214
|
-
## W3b — Refresh from traces
|
|
215
|
-
|
|
216
|
-
Goal: treat `sponsio.yaml` as a **living library**, not a one-shot output. As real session logs accumulate, re-mine them to discover new patterns and retire stale ones, without clobbering anything the user wrote or tuned.
|
|
217
|
-
|
|
218
|
-
### When to run
|
|
219
|
-
|
|
220
|
-
- Weekly / sprint boundary — recommended cadence for an agent seeing live traffic.
|
|
221
|
-
- After a material behavior change (new tools, new workflow, new integration).
|
|
222
|
-
- Before flipping to enforce (W4) — catches late-arriving trace-sourced rules that weren't present in the original scan.
|
|
223
|
-
|
|
224
|
-
### Steps
|
|
225
|
-
|
|
226
|
-
1. Dry-run first. Always.
|
|
227
|
-
|
|
228
|
-
```bash
|
|
229
|
-
sponsio refresh --since 7d
|
|
230
|
-
```
|
|
231
|
-
|
|
232
|
-
Prints a structured diff per agent:
|
|
233
|
-
|
|
234
|
-
```
|
|
235
|
-
Agent: support_bot
|
|
236
|
-
+ new must_precede(validate_payment, charge_card)
|
|
237
|
-
~ drifted rate_limit(send_email, 5) → args [send_email, 12]
|
|
238
|
-
- stale idempotent(list_users) (not re-observed)
|
|
239
|
-
= 8 unchanged (source: trace, re-observed)
|
|
240
|
-
= 12 preserved (user / scan / policy / overrides — not touched)
|
|
241
|
-
```
|
|
242
|
-
|
|
243
|
-
2. Review with the user. For each bucket:
|
|
244
|
-
- `+ new` — "the agent started doing X that your current yaml doesn't cover." User usually wants this.
|
|
245
|
-
- `~ drifted` — threshold moved. If new value is bigger, the rule was too tight; if smaller, production tightened up. User decides.
|
|
246
|
-
- `- stale` — rule hasn't fired in this window. **Not necessarily dead** — could just be rare. Conservative default is `--mode add-only` (never remove), which we recommend for the first few refreshes until the user trusts the window size.
|
|
247
|
-
- `= preserved` — these include every user rule, `source: scan`, `source: policy`, and anything under `overrides:`. The count being non-zero is the load-bearing invariant: refresh **only** ever changes `source: trace` entries.
|
|
248
|
-
|
|
249
|
-
3. Apply.
|
|
250
|
-
|
|
251
|
-
```bash
|
|
252
|
-
sponsio refresh --since 7d --apply # default mode: replace-trace
|
|
253
|
-
sponsio refresh --since 7d --apply --mode add-only # conservative: never remove
|
|
254
|
-
```
|
|
255
|
-
|
|
256
|
-
Writes `sponsio.yaml` and backs the old file up to `sponsio.yaml.sponsio.bak`. **Comments in the YAML are not preserved** — the backup is how users recover any prose annotations they'd inlined. Warn them of this before running.
|
|
257
|
-
|
|
258
|
-
### Mode selection
|
|
259
|
-
|
|
260
|
-
| Situation | Mode |
|
|
261
|
-
|---|---|
|
|
262
|
-
| First time running refresh, or small trace window | `add-only` (never removes) |
|
|
263
|
-
| Healthy trace volume, stable agent behavior | `replace-trace` (default — recent traces are authoritative) |
|
|
264
|
-
| Window just covers a launch / migration / incident (abnormal traffic) | Skip — artifact traces mislead the miner |
|
|
265
|
-
|
|
266
|
-
### Do NOT
|
|
267
|
-
|
|
268
|
-
- Do NOT run refresh on traces from an agent still in early iteration. Wait until the workflow is stable; otherwise every sprint invalidates the library.
|
|
269
|
-
- Do NOT use `--apply` without first showing the dry-run diff.
|
|
270
|
-
- Do NOT remove `source: scan` / `source: policy` entries in the yaml by hand just because refresh doesn't touch them — they represent knowledge refresh can't reconstruct (code AST, policy docs). If you think one is wrong, edit `overrides:`, don't delete.
|
|
271
|
-
|
|
272
|
-
---
|
|
273
|
-
|
|
274
209
|
## W4 — Flip to enforce
|
|
275
210
|
|
|
276
211
|
Goal: move from observe to enforce with regression confidence — no more logging, actual blocking. **This is a production change**; don't skip the checks.
|
|
@@ -332,7 +267,6 @@ Check in this order:
|
|
|
332
267
|
1. `sponsio validate --config sponsio.yaml --json` — does the contract parse? `ok: false` means it was silently dropped.
|
|
333
268
|
2. The tool name in the contract matches the actual tool name the agent calls (LangGraph `node_id` vs the `@tool` Python name is a common trap; see `tool_rename:`).
|
|
334
269
|
3. The rule has an `A:` (assumption) that isn't holding. Look at the trace — is the precondition ever true?
|
|
335
|
-
4. The rule is a `sto` pattern (`injection_free`, `tone_*`, `semantic_pii_free`, …). The OSS engine ships no evaluator for those — `pip install sponsio[cloud]` to activate the judge pipeline. Without Cloud, OSS rejects the contract loudly at config load.
|
|
336
270
|
|
|
337
271
|
### "A rule is firing that shouldn't."
|
|
338
272
|
|
|
@@ -393,10 +327,6 @@ agents:
|
|
|
393
327
|
runtime:
|
|
394
328
|
mode: observe # "observe" | "enforce"
|
|
395
329
|
dashboard: http://localhost:8000 # URL | true | false | null
|
|
396
|
-
|
|
397
|
-
judge: # Sponsio Cloud only — required when any include uses sto patterns
|
|
398
|
-
provider: openai # openai | anthropic | gemini
|
|
399
|
-
model: gpt-4o-mini
|
|
400
330
|
```
|
|
401
331
|
|
|
402
332
|
Both `A` and `G` accept: scalar NL string, list (AND), or structured dict `{pattern, args, source?}`.
|
|
@@ -407,7 +337,7 @@ Placeholders rewritten at include-time: `<workspace>/` (from agent's `workspace:
|
|
|
407
337
|
|
|
408
338
|
## Pattern reference
|
|
409
339
|
|
|
410
|
-
Full list: `sponsio patterns` (deterministic templates only).
|
|
340
|
+
Full list: `sponsio patterns` (deterministic templates only).
|
|
411
341
|
|
|
412
342
|
### Core Temporal (14 det)
|
|
413
343
|
| Pattern | Meaning |
|
|
@@ -436,8 +366,8 @@ Full list: `sponsio patterns` (deterministic templates only). LLM-judged stochas
|
|
|
436
366
|
### Resource & Delegation (3 det)
|
|
437
367
|
`token_budget(max_tokens)` · `arg_value_range(tool, field, min, max)` · `delegation_depth_limit(max_depth)`
|
|
438
368
|
|
|
439
|
-
###
|
|
440
|
-
|
|
369
|
+
### Response-quality patterns (3 det)
|
|
370
|
+
Deterministic response-quality patterns (`no_pii` regex, `max_length`, `no_keywords`) check `llm_said` and need no judge.
|
|
441
371
|
|
|
442
372
|
---
|
|
443
373
|
|
|
@@ -492,8 +422,8 @@ If asked for something out of scope (e.g., "also check my DB schema"), say so an
|
|
|
492
422
|
|
|
493
423
|
This skill only uses these. Internal refactors are safe as long as these stay stable.
|
|
494
424
|
|
|
495
|
-
1. **CLI**: `sponsio onboard [--apply]`, `sponsio scan PATHS [--agent N] [--llm] [--policy P] [-
|
|
496
|
-
2. **YAML**: top-level `agents:` as dict; each agent has optional `include:` / `tool_rename:` / `overrides:` / `workspace:` and required `contracts:`; top-level `runtime
|
|
425
|
+
1. **CLI**: `sponsio onboard [--apply]`, `sponsio scan PATHS [--agent N] [--llm] [--policy P] [-o FILE] [--append]`, `sponsio validate [--config FILE | "NL string"] [--json]`, `sponsio check --trace FILE --config FILE --agent ID`, `sponsio report --agent ID --since DUR`, `sponsio doctor`, `sponsio patterns`, `sponsio packs`, `sponsio skill install [--tool cursor|claude|codex|both|auto]`. Exit 0 on success.
|
|
426
|
+
2. **YAML**: top-level `agents:` as dict; each agent has optional `include:` / `tool_rename:` / `overrides:` / `workspace:` and required `contracts:`; top-level `runtime:`.
|
|
497
427
|
3. **Patterns**: names in the table above keep their semantics. Renaming is a breaking change for this skill.
|
|
498
428
|
4. **`validate --json` shape**: per-contract `ok` / `type` / `pattern` / `formula` / `agent`.
|
|
499
429
|
5. **`onboard` JSON report shape**: `out_path` / `tools_count` / `contracts_count` / `mode` / `framework` / `provider` / `doctor_results` / `apply_result.diff`.
|
|
@@ -1,140 +0,0 @@
|
|
|
1
|
-
# =============================================================================
|
|
2
|
-
# sponsio/contracts/core/llm_safety.yaml
|
|
3
|
-
#
|
|
4
|
-
# !! Sponsio Cloud only !! — this pack is composed entirely of LLM-judged
|
|
5
|
-
# stochastic atoms (`injection_free`, `jailbreak_free`, `harmful`,
|
|
6
|
-
# `toxic_free`, `semantic_pii_free`). The OSS engine ships no evaluator
|
|
7
|
-
# for them and will refuse to load this pack with a clear pointer to
|
|
8
|
-
# `pip install sponsio[cloud]`.
|
|
9
|
-
#
|
|
10
|
-
# It still ships in the OSS package so Cloud installs see the same on-disk
|
|
11
|
-
# layout — Cloud's StoEvaluator (auto-discovered via the
|
|
12
|
-
# `sponsio.evaluators` entry-point group) is what actually evaluates
|
|
13
|
-
# these contracts.
|
|
14
|
-
#
|
|
15
|
-
# Once Cloud is installed, opt in from your config:
|
|
16
|
-
#
|
|
17
|
-
# include:
|
|
18
|
-
# - sponsio:core/llm_safety
|
|
19
|
-
#
|
|
20
|
-
# Why opt-in even on Cloud: every contract here triggers a judge LLM call
|
|
21
|
-
# on each evaluation. That's the right default for response-quality work
|
|
22
|
-
# but the wrong default for tool-call-only agents (no LLM responses to
|
|
23
|
-
# score → judge calls add latency without value). Splitting out of
|
|
24
|
-
# `core/universal` was the user-driven fix for "why is the SRE tool-call
|
|
25
|
-
# demo running prompt-injection scoring".
|
|
26
|
-
#
|
|
27
|
-
# Source tag convention for this file:
|
|
28
|
-
# library:tier0.<category> where category ∈
|
|
29
|
-
# adversarial | harm | privacy | scope
|
|
30
|
-
# =============================================================================
|
|
31
|
-
|
|
32
|
-
version: "1"
|
|
33
|
-
|
|
34
|
-
# NOTE: no `tools:` section here because these contracts are response-scoped.
|
|
35
|
-
# They evaluate against llm_response events, not tool calls. Agents that
|
|
36
|
-
# include this file will add their own `tools:` inventory separately.
|
|
37
|
-
|
|
38
|
-
agents:
|
|
39
|
-
# Template agent id — replaced at include time by the including config.
|
|
40
|
-
# The include mechanism substitutes this section under the real agent id.
|
|
41
|
-
"*":
|
|
42
|
-
contracts:
|
|
43
|
-
|
|
44
|
-
# -- Adversarial resistance ----------------------------------------
|
|
45
|
-
#
|
|
46
|
-
# `beta` weights the cost of a *missed* violation in the contract
|
|
47
|
-
# score. Higher beta = a missed violation hurts the score more,
|
|
48
|
-
# so the judge has to be near-certain the response is *clean*
|
|
49
|
-
# before allowing it. We ship the values below as defaults
|
|
50
|
-
# calibrated for the threat profile (slip-through cost) of each
|
|
51
|
-
# category. Override per-agent in your config if your workload
|
|
52
|
-
# has different stakes.
|
|
53
|
-
|
|
54
|
-
# 0.95 — prompt-injection slip-throughs cause downstream tool
|
|
55
|
-
# misuse with full agent privileges. High slip-through cost.
|
|
56
|
-
- desc: "Every LLM response must be free of prompt-injection compliance"
|
|
57
|
-
G:
|
|
58
|
-
pattern: injection_free
|
|
59
|
-
context_scope: full_trace
|
|
60
|
-
source: "library:tier0.adversarial"
|
|
61
|
-
beta: 0.95
|
|
62
|
-
|
|
63
|
-
# 0.90 — jailbreak compliance is bad but mostly self-contained
|
|
64
|
-
# (response-level harm, not tool-level harm).
|
|
65
|
-
- desc: "LLM must not comply with jailbreak attempts (DAN, roleplay, hypothetical bypass)"
|
|
66
|
-
G:
|
|
67
|
-
pattern: jailbreak_free
|
|
68
|
-
source: "library:tier0.adversarial"
|
|
69
|
-
beta: 0.9
|
|
70
|
-
|
|
71
|
-
# -- Harm avoidance -----------------------------------------------
|
|
72
|
-
|
|
73
|
-
# 0.95 — real-world-harm enablement (CSAM, weapons, self-harm
|
|
74
|
-
# methods) carries the same high slip-through cost as injection.
|
|
75
|
-
- desc: "Response must not encourage or enable real-world harm"
|
|
76
|
-
G:
|
|
77
|
-
pattern: harmful
|
|
78
|
-
source: "library:tier0.harm"
|
|
79
|
-
beta: 0.95
|
|
80
|
-
|
|
81
|
-
# 0.90 — toxic language is corrosive but not safety-critical;
|
|
82
|
-
# match the jailbreak weight rather than the harm weight.
|
|
83
|
-
- desc: "Response free of toxic / harassing language"
|
|
84
|
-
G:
|
|
85
|
-
pattern: toxic_free
|
|
86
|
-
source: "library:tier0.harm"
|
|
87
|
-
beta: 0.9
|
|
88
|
-
|
|
89
|
-
# -- Privacy ------------------------------------------------------
|
|
90
|
-
|
|
91
|
-
# 0.95 — PII slip-throughs are regulatory issues (GDPR / CCPA)
|
|
92
|
-
# not just trust issues. High slip-through cost.
|
|
93
|
-
- desc: "Response free of contextual PII (names tied to conditions, inferable identity)"
|
|
94
|
-
G:
|
|
95
|
-
pattern: semantic_pii_free
|
|
96
|
-
source: "library:tier0.privacy"
|
|
97
|
-
beta: 0.95
|
|
98
|
-
|
|
99
|
-
# -- Scope --------------------------------------------------------
|
|
100
|
-
#
|
|
101
|
-
# `scope_respect` is deliberately NOT shipped on by default. It
|
|
102
|
-
# needs a concrete, agent-specific scope string to be useful —
|
|
103
|
-
# something like "respond only to loan-origination questions for
|
|
104
|
-
# the approved product catalogue". A generic default like "do
|
|
105
|
-
# not act beyond the task" is irredeemably abstract: the judge
|
|
106
|
-
# has nothing to anchor on and will either flag everything or
|
|
107
|
-
# pass everything, depending on model temperament. Shipping
|
|
108
|
-
# such a rule as default-on produces noise, not signal.
|
|
109
|
-
#
|
|
110
|
-
# Add it to your own agent config once you have a concrete scope
|
|
111
|
-
# statement, e.g.::
|
|
112
|
-
#
|
|
113
|
-
# agents:
|
|
114
|
-
# support_bot:
|
|
115
|
-
# contracts:
|
|
116
|
-
# - desc: "Stays within customer-support remit"
|
|
117
|
-
# E:
|
|
118
|
-
# pattern: scope_respect
|
|
119
|
-
# args: ["answer questions about order status,
|
|
120
|
-
# refunds, and shipping — no legal advice,
|
|
121
|
-
# no medical advice, no code execution"]
|
|
122
|
-
# beta: 0.85
|
|
123
|
-
#
|
|
124
|
-
# The β=0.85 default acknowledges that "out of scope" is fuzzier
|
|
125
|
-
# than "contains PII"; a higher beta flags too aggressively on
|
|
126
|
-
# responses that drift only slightly.
|
|
127
|
-
|
|
128
|
-
# -----------------------------------------------------------------------------
|
|
129
|
-
# What this file deliberately does NOT include
|
|
130
|
-
# -----------------------------------------------------------------------------
|
|
131
|
-
#
|
|
132
|
-
# - Token / loop budgets — those live in core/budget.yaml and core/runaway.yaml
|
|
133
|
-
# respectively, so an agent with a custom budget can include only that.
|
|
134
|
-
#
|
|
135
|
-
# - Tool-specific protections — any constraint that mentions a specific tool
|
|
136
|
-
# name (exec, web_fetch, ...) belongs in capability/*.yaml.
|
|
137
|
-
#
|
|
138
|
-
# - Hallucination / faithfulness atoms — those require a retrieval source
|
|
139
|
-
# argument and are not universally applicable. See docs/sto-atoms.md §1
|
|
140
|
-
# for when to add them per-agent.
|
package/prompts/refresh.md
DELETED
|
@@ -1,118 +0,0 @@
|
|
|
1
|
-
# Contract refresh prompt — sponsio refresh from traces
|
|
2
|
-
|
|
3
|
-
You are tuning an existing `sponsio.yaml` against accumulated
|
|
4
|
-
session traces. The library already exists; you're proposing
|
|
5
|
-
deltas — added contracts, retired stale ones, tightened thresholds
|
|
6
|
-
— based on what the agent actually did.
|
|
7
|
-
|
|
8
|
-
This is the **self-evolve loop**: each refresh round looks at recent
|
|
9
|
-
near-misses (would-have-blocked in observe mode) and confirmed
|
|
10
|
-
patterns (rules that fire frequently and correctly), and proposes
|
|
11
|
-
targeted edits.
|
|
12
|
-
|
|
13
|
-
## Input
|
|
14
|
-
|
|
15
|
-
A JSON object from `sponsio refresh --emit-traces`:
|
|
16
|
-
|
|
17
|
-
```json
|
|
18
|
-
{
|
|
19
|
-
"agent": "...",
|
|
20
|
-
"since": "7d",
|
|
21
|
-
"existing_contracts": [
|
|
22
|
-
{"desc": "...", "pattern": "...", "args": [...], "source": "..."}
|
|
23
|
-
],
|
|
24
|
-
"trace_summary": {
|
|
25
|
-
"total_events": 1247,
|
|
26
|
-
"would_have_blocked": [
|
|
27
|
-
{
|
|
28
|
-
"tool": "send_email",
|
|
29
|
-
"rule_desc": "rate_limit 5",
|
|
30
|
-
"fire_count": 12,
|
|
31
|
-
"sample_calls": [
|
|
32
|
-
{"args": {...}, "ts": "...", "agent_outcome": "succeeded"}
|
|
33
|
-
]
|
|
34
|
-
}
|
|
35
|
-
],
|
|
36
|
-
"blocked_actual": [...],
|
|
37
|
-
"uncovered_patterns": [
|
|
38
|
-
{
|
|
39
|
-
"tool": "transfer_funds",
|
|
40
|
-
"call_count": 3,
|
|
41
|
-
"sample_args": [{...}],
|
|
42
|
-
"note": "tool was called 3x; no contract covers it"
|
|
43
|
-
}
|
|
44
|
-
]
|
|
45
|
-
}
|
|
46
|
-
}
|
|
47
|
-
```
|
|
48
|
-
|
|
49
|
-
## What you produce
|
|
50
|
-
|
|
51
|
-
A YAML diff with three sections:
|
|
52
|
-
|
|
53
|
-
```yaml
|
|
54
|
-
proposed_changes:
|
|
55
|
-
add:
|
|
56
|
-
- desc: "..."
|
|
57
|
-
G: {pattern: ..., args: [...]}
|
|
58
|
-
source: agent-extracted-from-traces
|
|
59
|
-
retire:
|
|
60
|
-
- match: { desc: "<exact desc of existing contract>" }
|
|
61
|
-
reason: "fired 0 times in 7d window"
|
|
62
|
-
tighten:
|
|
63
|
-
- match: { desc: "<exact desc>" }
|
|
64
|
-
from: {pattern: rate_limit, args: [send_email, 5]}
|
|
65
|
-
to: {pattern: rate_limit, args: [send_email, 10]}
|
|
66
|
-
reason: "false-positive rate 12/30 over last 7d"
|
|
67
|
-
```
|
|
68
|
-
|
|
69
|
-
## Rules of thumb
|
|
70
|
-
|
|
71
|
-
### Adding contracts
|
|
72
|
-
|
|
73
|
-
* `uncovered_patterns` with `call_count` ≥ 3 and clear semantics →
|
|
74
|
-
add a contract.
|
|
75
|
-
* Don't add for tools that fired 1–2x — too sparse.
|
|
76
|
-
* Match the source-tag convention: new agent-derived contracts get
|
|
77
|
-
`source: agent-extracted-from-traces`.
|
|
78
|
-
|
|
79
|
-
### Retiring (only `source: trace` or `source: agent-extracted-*`)
|
|
80
|
-
|
|
81
|
-
* Existing contract that fired 0 times in the window → candidate for
|
|
82
|
-
retirement, but ONLY if it's `source: trace` or
|
|
83
|
-
`source: agent-extracted-*`.
|
|
84
|
-
* **Never propose retiring** `source: scan`, `source: policy`, or
|
|
85
|
-
user-written (no source / `source: user`) contracts.
|
|
86
|
-
* If retiring, include a one-line `reason`.
|
|
87
|
-
|
|
88
|
-
### Tightening / loosening
|
|
89
|
-
|
|
90
|
-
* High false-positive rate (`would_have_blocked` clusters with
|
|
91
|
-
`agent_outcome: succeeded` — the call was actually fine) →
|
|
92
|
-
loosen the cap or add `arg_blacklist` carve-outs.
|
|
93
|
-
* High true-positive rate (would-have-blocked + agent_outcome
|
|
94
|
-
shows the agent was caught doing something problematic) →
|
|
95
|
-
tighten.
|
|
96
|
-
* Always cite the trace counts in `reason` so the user can verify.
|
|
97
|
-
|
|
98
|
-
## Pattern vocabulary
|
|
99
|
-
|
|
100
|
-
Same as plugin scan / onboard — `arg_blacklist`, `rate_limit`,
|
|
101
|
-
`loop_detection`, `irreversible_once`, `must_precede`,
|
|
102
|
-
`arg_value_range`, `arg_length_limit`.
|
|
103
|
-
|
|
104
|
-
## What you must not do
|
|
105
|
-
|
|
106
|
-
* **Don't** retire user-written or pack-derived contracts.
|
|
107
|
-
* **Don't** propose changes without trace evidence — every proposal
|
|
108
|
-
needs a count + window in `reason`.
|
|
109
|
-
* **Don't** merge proposals into the YAML directly — output the
|
|
110
|
-
diff for the user to review and apply. Apply via
|
|
111
|
-
`sponsio refresh --apply` (which respects the same source-tag
|
|
112
|
-
protections automatically) or by hand-editing for ad-hoc changes.
|
|
113
|
-
|
|
114
|
-
## Output format
|
|
115
|
-
|
|
116
|
-
ONLY the `proposed_changes:` YAML block above. No prose, no
|
|
117
|
-
markdown wrapping. The host driver will show it to the user
|
|
118
|
-
verbatim.
|