watchmyagents 1.2.1 → 1.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +72 -0
- package/docs/adapters/openai-agents-js.md +230 -0
- package/examples/adapters/capture-fixtures.ts +193 -0
- package/examples/adapters/openai-agents-js-quickstart.ts +118 -0
- package/package.json +15 -1
- package/src/logger.js +17 -1
- package/src/sources/contract.js +31 -1
- package/src/sources/openai-agents-js.d.ts +278 -0
- package/src/sources/openai-agents-js.js +733 -0
package/README.md
CHANGED
|
@@ -86,6 +86,78 @@ total : 811,798 (in=26 out=22,996 cache_r=492,220 cache_w=296,556)
|
|
|
86
86
|
calls/min : 0.08
|
|
87
87
|
```
|
|
88
88
|
|
|
89
|
+
## Quickstart — instrument an OpenAI Agents SDK agent (v1.3.0+)
|
|
90
|
+
|
|
91
|
+
For OpenAI agents, the runtime executes **in your process** (not on OpenAI's servers — see [docs/adapters/openai-agents-js.md](docs/adapters/openai-agents-js.md) for why). WMA is wired in as two lines:
|
|
92
|
+
|
|
93
|
+
```bash
|
|
94
|
+
npm install watchmyagents @openai/agents zod
|
|
95
|
+
```
|
|
96
|
+
|
|
97
|
+
**Pattern A — explicit Runner** (use `attachWmaWatch`) :
|
|
98
|
+
```typescript
|
|
99
|
+
import { Agent, Runner } from '@openai/agents';
|
|
100
|
+
import {
|
|
101
|
+
wmaToolInputGuardrail,
|
|
102
|
+
attachWmaWatch,
|
|
103
|
+
} from 'watchmyagents/src/sources/openai-agents-js.js';
|
|
104
|
+
|
|
105
|
+
const wmaShield = wmaToolInputGuardrail({
|
|
106
|
+
policiesPath: './examples/policies/mitre-starter.json',
|
|
107
|
+
});
|
|
108
|
+
|
|
109
|
+
const agent = new Agent({
|
|
110
|
+
name: 'support_bot',
|
|
111
|
+
instructions: '...',
|
|
112
|
+
tools: [...],
|
|
113
|
+
toolInputGuardrails: [wmaShield], // ← Shield enforcement (pre-execution deny)
|
|
114
|
+
});
|
|
115
|
+
|
|
116
|
+
const runner = new Runner();
|
|
117
|
+
attachWmaWatch(runner); // ← Watch observability
|
|
118
|
+
|
|
119
|
+
await runner.run(agent, 'How do I reset my password?');
|
|
120
|
+
```
|
|
121
|
+
|
|
122
|
+
**Pattern B — convenience `run()` function** (use `attachWmaWatchToAgent`) :
|
|
123
|
+
```typescript
|
|
124
|
+
import { Agent, run } from '@openai/agents';
|
|
125
|
+
import {
|
|
126
|
+
wmaToolInputGuardrail,
|
|
127
|
+
attachWmaWatchToAgent,
|
|
128
|
+
} from 'watchmyagents/src/sources/openai-agents-js.js';
|
|
129
|
+
|
|
130
|
+
const wmaShield = wmaToolInputGuardrail({
|
|
131
|
+
policiesPath: './examples/policies/mitre-starter.json',
|
|
132
|
+
});
|
|
133
|
+
|
|
134
|
+
const agent = new Agent({
|
|
135
|
+
name: 'support_bot',
|
|
136
|
+
tools: [...],
|
|
137
|
+
toolInputGuardrails: [wmaShield],
|
|
138
|
+
});
|
|
139
|
+
|
|
140
|
+
attachWmaWatchToAgent(agent); // ← Watch attaches to the agent itself
|
|
141
|
+
|
|
142
|
+
await run(agent, 'How do I reset my password?');
|
|
143
|
+
```
|
|
144
|
+
|
|
145
|
+
The two patterns reflect the two EventEmitter surfaces the SDK exposes (RunHooks vs AgentHooks). Both produce identical NDJSON output; pick the one that matches your call shape.
|
|
146
|
+
|
|
147
|
+
That's it. NDJSON lands in `./watchmyagents-logs/openai-agents/`. The MITRE starter bundle catches common attack patterns (T1567 exfil, T1485 destructive shell, T1548 priv-esc, …) on Day 0 — all in `mode: 'shadow'` so the first run only LOGS without enforcing. Promote to `enforce` once you've calibrated.
|
|
148
|
+
|
|
149
|
+
See [docs/adapters/openai-agents-js.md](docs/adapters/openai-agents-js.md) for the full options reference + troubleshooting.
|
|
150
|
+
|
|
151
|
+
## Supported runtimes (v1.3.0)
|
|
152
|
+
|
|
153
|
+
| Runtime | Mode | Onboarding | Status |
|
|
154
|
+
|---|---|---|---|
|
|
155
|
+
| **Anthropic Managed Agents** | pull REST/SSE | API key (zero-touch) | ✓ shipped |
|
|
156
|
+
| **OpenAI Agents SDK** (TypeScript/JS) | push (in-process hooks + guardrails) | 2 lines of code | ✓ v1.3.0 |
|
|
157
|
+
| OpenAI Agents SDK (Python) | push (separate `watchmyagents-py` package) | — | planned v1.4.0 |
|
|
158
|
+
| Claude Code (dynamic workflows) | push (hooks via `settings.json`) | — | planned v1.4.x |
|
|
159
|
+
| AWS Bedrock AgentCore | pull REST/SSE | similar to Anthropic | planned v1.5.0 |
|
|
160
|
+
|
|
89
161
|
## What gets logged
|
|
90
162
|
|
|
91
163
|
Each line of the NDJSON file is one agent action. The 18 `action_type` values captured today:
|
|
@@ -0,0 +1,230 @@
|
|
|
1
|
+
# OpenAI Agents SDK adapter (TypeScript / JavaScript)
|
|
2
|
+
|
|
3
|
+
**Status: v1.3.0 (Phase 2.A) — first adapter that observes a runtime which executes locally on the customer machine.**
|
|
4
|
+
|
|
5
|
+
This adapter integrates `watchmyagents` with the [OpenAI Agents SDK](https://openai.github.io/openai-agents-js/) (`@openai/agents` on npm). Customers add two lines to their existing agent code; WMA logs every lifecycle event locally and can block tool calls before execution via Shield policies.
|
|
6
|
+
|
|
7
|
+
## Why customer-instrumented and not "paste your API key"
|
|
8
|
+
|
|
9
|
+
OpenAI's agent runtime executes **on the customer's machine**, not on OpenAI's servers. There is no `listAgents` / `listConversations` endpoint on the OpenAI API that would let WMA pull events from outside — the SDK is the agent. So WMA has to live inside the same process.
|
|
10
|
+
|
|
11
|
+
This is the same model used by Datadog APM, Sentry, Langfuse, and OpenLLMetry for OpenAI observability. The integration is two lines of code; the rest is automatic.
|
|
12
|
+
|
|
13
|
+
## Setup
|
|
14
|
+
|
|
15
|
+
### 1. Install
|
|
16
|
+
|
|
17
|
+
```bash
|
|
18
|
+
npm install @openai/agents zod watchmyagents
|
|
19
|
+
```
|
|
20
|
+
|
|
21
|
+
`watchmyagents` declares `@openai/agents` as a peer dependency — the customer installs both; WMA ships zero runtime deps of its own.
|
|
22
|
+
|
|
23
|
+
### 2. Get a WMA API key
|
|
24
|
+
|
|
25
|
+
Sign in to the Fortress dashboard → **Settings → API Keys → Generate** → copy the `wma_xxx` value. Give it a label that maps to one of your environments (e.g. `production`, `staging`).
|
|
26
|
+
|
|
27
|
+
### 3. Environment variables
|
|
28
|
+
|
|
29
|
+
```bash
|
|
30
|
+
# .env
|
|
31
|
+
WMA_API_KEY=wma_xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
|
|
32
|
+
WMA_LOG_DIR=./watchmyagents-logs # optional; default shown
|
|
33
|
+
WMA_TEAM_ID=customer-support # optional; manual team tagging
|
|
34
|
+
# WMA_POLICIES_SOURCE handling for Fortress-pulled policies arrives in 1.3.1
|
|
35
|
+
```
|
|
36
|
+
|
|
37
|
+
Today the adapter loads policies from a local file via `policiesPath`. Fortress-pulled policies for this adapter land in a 1.3.x patch.
|
|
38
|
+
|
|
39
|
+
### 4. Wire it up — two patterns
|
|
40
|
+
|
|
41
|
+
The `@openai/agents` SDK exposes two EventEmitter surfaces and lets you
|
|
42
|
+
pick how to run an agent. WMA supports both:
|
|
43
|
+
|
|
44
|
+
**Pattern A — explicit Runner (RunHooks):**
|
|
45
|
+
|
|
46
|
+
```typescript
|
|
47
|
+
import { Agent, Runner } from '@openai/agents';
|
|
48
|
+
import {
|
|
49
|
+
wmaToolInputGuardrail,
|
|
50
|
+
attachWmaWatch,
|
|
51
|
+
} from 'watchmyagents/src/sources/openai-agents-js.js';
|
|
52
|
+
|
|
53
|
+
const wmaShield = wmaToolInputGuardrail({
|
|
54
|
+
policiesPath: './examples/policies/mitre-starter.json',
|
|
55
|
+
});
|
|
56
|
+
|
|
57
|
+
const agent = new Agent({
|
|
58
|
+
name: 'support_bot',
|
|
59
|
+
instructions: '...',
|
|
60
|
+
tools: [...],
|
|
61
|
+
toolInputGuardrails: [wmaShield], // ← Shield enforcement
|
|
62
|
+
});
|
|
63
|
+
|
|
64
|
+
const runner = new Runner();
|
|
65
|
+
attachWmaWatch(runner); // ← Watch via RunHooks
|
|
66
|
+
|
|
67
|
+
await runner.run(agent, 'How do I reset my password?');
|
|
68
|
+
```
|
|
69
|
+
|
|
70
|
+
**Pattern B — convenience `run()` function (AgentHooks):**
|
|
71
|
+
|
|
72
|
+
```typescript
|
|
73
|
+
import { Agent, run } from '@openai/agents';
|
|
74
|
+
import {
|
|
75
|
+
wmaToolInputGuardrail,
|
|
76
|
+
attachWmaWatchToAgent,
|
|
77
|
+
} from 'watchmyagents/src/sources/openai-agents-js.js';
|
|
78
|
+
|
|
79
|
+
const wmaShield = wmaToolInputGuardrail({
|
|
80
|
+
policiesPath: './examples/policies/mitre-starter.json',
|
|
81
|
+
});
|
|
82
|
+
|
|
83
|
+
const agent = new Agent({
|
|
84
|
+
name: 'support_bot',
|
|
85
|
+
tools: [...],
|
|
86
|
+
toolInputGuardrails: [wmaShield], // ← Shield (unchanged)
|
|
87
|
+
});
|
|
88
|
+
|
|
89
|
+
attachWmaWatchToAgent(agent); // ← Watch via AgentHooks
|
|
90
|
+
|
|
91
|
+
await run(agent, 'How do I reset my password?');
|
|
92
|
+
```
|
|
93
|
+
|
|
94
|
+
**Which to pick?** Use the function name (`Runner` vs `run`) as your
|
|
95
|
+
heuristic. They behave identically from a Watch/Shield perspective —
|
|
96
|
+
same NDJSON output, same audit chain, same team_id propagation. Under
|
|
97
|
+
the hood, the SDK fires lifecycle events through two different event
|
|
98
|
+
emitters with slightly different argument layouts (e.g. AgentHooks
|
|
99
|
+
omits the agent from `agent_end` / `agent_tool_*` because the listener
|
|
100
|
+
is registered on the agent itself); the two `attachWma…` variants
|
|
101
|
+
handle those differences.
|
|
102
|
+
|
|
103
|
+
That is the entire customer change.
|
|
104
|
+
|
|
105
|
+
## What you get
|
|
106
|
+
|
|
107
|
+
### Watch (observability)
|
|
108
|
+
|
|
109
|
+
Every lifecycle event the SDK emits flows to a daily-rotated NDJSON log at `./watchmyagents-logs/openai-agents/YYYY-MM-DD.ndjson`. Five event types are captured:
|
|
110
|
+
|
|
111
|
+
| @openai/agents event | WMA `action_type` | Notes |
|
|
112
|
+
|---|---|---|
|
|
113
|
+
| `agent_start` | `message` (kind=agent_start) | Agent received turn input |
|
|
114
|
+
| `agent_end` | `message` (kind=agent_end) | Agent produced final output |
|
|
115
|
+
| `agent_handoff` | `handoff` | One agent passes control to another |
|
|
116
|
+
| `agent_tool_start` | `custom_tool_use` | Tool is about to fire |
|
|
117
|
+
| `agent_tool_end` | `custom_tool_result` | Tool returned a result |
|
|
118
|
+
|
|
119
|
+
Containment: tool arguments and tool results are captured in `input` / `output` and stay LOCAL. The anonymizer is the single gate to Fortress.
|
|
120
|
+
|
|
121
|
+
### Shield (enforcement)
|
|
122
|
+
|
|
123
|
+
The `wmaToolInputGuardrail()` returned object slots into the Agent's `toolInputGuardrails: [...]` array. Before every tool call, the SDK awaits our `run()` and respects the result:
|
|
124
|
+
|
|
125
|
+
| Shield decision | OpenAI `behavior` | Effect on the tool call |
|
|
126
|
+
|---|---|---|
|
|
127
|
+
| `allow` | `{ type: 'allow' }` | Tool runs as if no guardrail existed |
|
|
128
|
+
| `deny` | `{ type: 'rejectContent', message }` | Tool is BLOCKED; `message` returned in place of the tool result; the model sees the rejection and can decide next steps |
|
|
129
|
+
| `interrupt` | `{ type: 'throwException' }` | Tool is blocked AND the agent loop aborts with an exception |
|
|
130
|
+
| `shadow` mode | `{ type: 'allow' }` with diagnostic outputInfo | Tool runs; decision logged for calibration but not enforced |
|
|
131
|
+
|
|
132
|
+
### Team correlation (Legions UI)
|
|
133
|
+
|
|
134
|
+
Three resolution channels for `team_id`, in precedence order:
|
|
135
|
+
|
|
136
|
+
1. **Customer override** — set `WMA_TEAM_ID` env var or pass `getTeamId` to the guardrail options.
|
|
137
|
+
2. **Auto-detect via handoff** — when the SDK emits `agent_handoff(fromAgent, toAgent)`, every subsequent event in the same run shares the from-agent's team id. Useful for multi-agent flows without any customer config.
|
|
138
|
+
3. **None** — events get `team_id: null` and appear under "Untagged" in the Fortress Legions view.
|
|
139
|
+
|
|
140
|
+
### Audit chain (tamper-evidence)
|
|
141
|
+
|
|
142
|
+
Every `shield_decision` row carries `prev_hash` + `chain_hash` (SHA-256). The audit chain is local-only tamper-evidence. Verify with:
|
|
143
|
+
|
|
144
|
+
```typescript
|
|
145
|
+
import { verifyDecisionChain } from 'watchmyagents/src/shield/decision-chain.js';
|
|
146
|
+
import { readFile } from 'node:fs/promises';
|
|
147
|
+
|
|
148
|
+
const lines = (await readFile('./watchmyagents-logs/openai-agents/2026-06-09.ndjson', 'utf8'))
|
|
149
|
+
.trim().split('\n').map((l) => JSON.parse(l))
|
|
150
|
+
.filter((l) => l.action_type === 'shield_decision');
|
|
151
|
+
|
|
152
|
+
console.log(verifyDecisionChain(lines));
|
|
153
|
+
// → { ok: true, count: N, segments: 1 }
|
|
154
|
+
```
|
|
155
|
+
|
|
156
|
+
Limits: tail truncation (removing the most recent rows) is invisible to an append-only chain. Cross-process replay would re-derive the chain from scratch. Fortress-side append-only ingest closes both gaps (planned).
|
|
157
|
+
|
|
158
|
+
## Options reference
|
|
159
|
+
|
|
160
|
+
```typescript
|
|
161
|
+
wmaToolInputGuardrail({
|
|
162
|
+
// Policy source — pick ONE:
|
|
163
|
+
policiesPath?: string, // local JSON file path
|
|
164
|
+
ruleset?: object, // in-memory ruleset object
|
|
165
|
+
|
|
166
|
+
// Storage:
|
|
167
|
+
logDir?: string, // default WMA_LOG_DIR or ./watchmyagents-logs
|
|
168
|
+
sessionId?: string, // default auto-minted UUID
|
|
169
|
+
|
|
170
|
+
// Behavior:
|
|
171
|
+
failOpen?: boolean, // on Shield internal error: allow (true) or
|
|
172
|
+
// deny (false). Default: false (fail-CLOSED)
|
|
173
|
+
recentWindowSize?: number, // ctx.recent_error_rate window. Default: 20
|
|
174
|
+
getTeamId?: () => string|null, // custom team resolver
|
|
175
|
+
|
|
176
|
+
// Injection (for tests):
|
|
177
|
+
logger?: Logger,
|
|
178
|
+
decisionLogger?: DecisionLogger,
|
|
179
|
+
tracker?: ContextTracker,
|
|
180
|
+
});
|
|
181
|
+
|
|
182
|
+
attachWmaWatch(runner, {
|
|
183
|
+
logDir?: string, // default same as guardrail
|
|
184
|
+
sessionId?: string, // default auto-minted UUID
|
|
185
|
+
logger?: Logger,
|
|
186
|
+
teamTracker?: TeamTracker,
|
|
187
|
+
});
|
|
188
|
+
// → returns a `detach()` function the customer can call on shutdown.
|
|
189
|
+
```
|
|
190
|
+
|
|
191
|
+
## What gets sent to Fortress
|
|
192
|
+
|
|
193
|
+
Tool arguments and results stay LOCAL. The anonymizer converts WMAAction → signals payload (salted hashes, no raw values) and pushes to `WMA_FORTRESS_BASE_URL/upload-signals` if `WMA_API_KEY` is set. See [CONTAINMENT.md](../CONTAINMENT.md) for the full egress model.
|
|
194
|
+
|
|
195
|
+
## Compatibility
|
|
196
|
+
|
|
197
|
+
- Node.js: **20+** (matches the WMA SDK baseline and `@openai/agents`'s requirement)
|
|
198
|
+
- TypeScript: any version — types live in `src/sources/openai-agents-js.d.ts`
|
|
199
|
+
- `@openai/agents` version range: tested against `^x.y` (verified in CI against the version pinned in our integration fixtures — see `test/fixtures/openai-agents-events/README.md`)
|
|
200
|
+
|
|
201
|
+
## Limits + known gaps (v1.3.0)
|
|
202
|
+
|
|
203
|
+
| Gap | Workaround |
|
|
204
|
+
|---|---|
|
|
205
|
+
| Streaming mode behavior under Tool Input Guardrails not formally verified | We expect the SDK to honor guardrails in streaming mode (the SDK code paths are the same); the test fixtures cover non-streaming today. Verification with a streaming smoke test before v1.3.1. |
|
|
206
|
+
| Fortress-pulled policies via `WMA_POLICIES_SOURCE=fortress` not wired in this adapter yet | Use `policiesPath` with a local file in v1.3.0; Fortress pull lands in v1.3.x patch |
|
|
207
|
+
| Tool Output Guardrails (post-execution filter) not exposed | Add only if customer demand — Shield's pre-tool deny covers 95% of the exfil-prevention use case |
|
|
208
|
+
| Python Agents SDK | Separate package `watchmyagents-py` planned for v1.4.0 (Pattern 2 — pip-installable thin client that talks to a Node Shield daemon over UNIX socket) |
|
|
209
|
+
|
|
210
|
+
## Troubleshooting
|
|
211
|
+
|
|
212
|
+
**"no policy ruleset configured — guardrail will allow all"** stderr warning
|
|
213
|
+
You passed neither `policiesPath` nor `ruleset` to `wmaToolInputGuardrail()`. The guardrail is loaded but every tool call is allowed. Pass one of the two options to enforce.
|
|
214
|
+
|
|
215
|
+
**Tool calls succeed but never appear in NDJSON**
|
|
216
|
+
Check that you also called `attachWmaWatch(runner)`. The guardrail handles Shield decisions; Watch handles the event log. Customers who only need enforcement can skip Watch (the guardrail still logs `shield_decision` rows itself).
|
|
217
|
+
|
|
218
|
+
**Audit chain verification reports `chain_hash recomputation mismatch`**
|
|
219
|
+
Someone modified an NDJSON row after write. The `broken_at` index points to the first tampered row. Investigate; do not delete the file (it IS the evidence).
|
|
220
|
+
|
|
221
|
+
**Guardrail latency feels high**
|
|
222
|
+
Policy eval is sub-millisecond for small rulesets. The audit chain hash takes <1ms. If you see >50ms, the log directory may be on slow storage — point `logDir` at a local SSD or in-memory tmpfs.
|
|
223
|
+
|
|
224
|
+
## See also
|
|
225
|
+
|
|
226
|
+
- [examples/adapters/openai-agents-js-quickstart.ts](../../examples/adapters/openai-agents-js-quickstart.ts) — runnable customer reference
|
|
227
|
+
- [docs/adapters/](.) — sibling adapter docs
|
|
228
|
+
- [docs/SOURCE-ADAPTER-CONTRACT.md](../SOURCE-ADAPTER-CONTRACT.md) — the contract every adapter follows
|
|
229
|
+
- [docs/MITRE-ATTCK-COVERAGE.md](../MITRE-ATTCK-COVERAGE.md) — what the policy starter bundle covers
|
|
230
|
+
- [docs/CONTAINMENT.md](../CONTAINMENT.md) — what stays local vs goes to Fortress
|
|
@@ -0,0 +1,193 @@
|
|
|
1
|
+
// Fixture capture script — runs a real @openai/agents flow and writes
|
|
2
|
+
// each lifecycle event's raw arguments to test/fixtures/openai-agents-events/.
|
|
3
|
+
//
|
|
4
|
+
// Goal: replace the synthetic fixtures in test/sources-openai-agents-js.test.js
|
|
5
|
+
// with REAL captures from a live SDK version, so the adapter is verified
|
|
6
|
+
// against the actual shape the SDK emits — not against our reading of the
|
|
7
|
+
// source code.
|
|
8
|
+
//
|
|
9
|
+
// Run:
|
|
10
|
+
// npm install @openai/agents zod
|
|
11
|
+
// export OPENAI_API_KEY=sk-proj-...
|
|
12
|
+
// npx tsx examples/adapters/capture-fixtures.ts
|
|
13
|
+
//
|
|
14
|
+
// Output: one JSON per event type at
|
|
15
|
+
// test/fixtures/openai-agents-events/agent_start.json
|
|
16
|
+
// test/fixtures/openai-agents-events/agent_end.json
|
|
17
|
+
// test/fixtures/openai-agents-events/agent_handoff.json
|
|
18
|
+
// test/fixtures/openai-agents-events/agent_tool_start.json
|
|
19
|
+
// test/fixtures/openai-agents-events/agent_tool_end.json
|
|
20
|
+
//
|
|
21
|
+
// Each file contains the raw listener arguments + meta (SDK version,
|
|
22
|
+
// Node version, capture timestamp).
|
|
23
|
+
|
|
24
|
+
import { Agent, Runner, handoff, tool } from '@openai/agents';
|
|
25
|
+
import { z } from 'zod';
|
|
26
|
+
import { writeFile, mkdir } from 'node:fs/promises';
|
|
27
|
+
import { join } from 'node:path';
|
|
28
|
+
import { fileURLToPath } from 'node:url';
|
|
29
|
+
import { dirname } from 'node:path';
|
|
30
|
+
|
|
31
|
+
const FIXTURES_DIR = join(
|
|
32
|
+
dirname(fileURLToPath(import.meta.url)),
|
|
33
|
+
'..',
|
|
34
|
+
'..',
|
|
35
|
+
'test',
|
|
36
|
+
'fixtures',
|
|
37
|
+
'openai-agents-events',
|
|
38
|
+
);
|
|
39
|
+
|
|
40
|
+
// Capture meta — Node + SDK versions are baked into each fixture so future
|
|
41
|
+
// readers know what runtime emitted these shapes.
|
|
42
|
+
async function getSdkVersion(): Promise<string> {
|
|
43
|
+
try {
|
|
44
|
+
const pkg = await import('@openai/agents/package.json', { assert: { type: 'json' } });
|
|
45
|
+
return (pkg as any).default?.version || (pkg as any).version || 'unknown';
|
|
46
|
+
} catch {
|
|
47
|
+
return 'unknown';
|
|
48
|
+
}
|
|
49
|
+
}
|
|
50
|
+
|
|
51
|
+
// One capture per event type. We overwrite if the file already exists so
|
|
52
|
+
// re-running the script always reflects the LATEST SDK version observed.
|
|
53
|
+
const captured: Record<string, boolean> = {};
|
|
54
|
+
|
|
55
|
+
async function saveFixture(eventName: string, rawArgs: unknown[], sdkVersion: string) {
|
|
56
|
+
if (captured[eventName]) return; // first occurrence wins — keep deterministic
|
|
57
|
+
captured[eventName] = true;
|
|
58
|
+
const payload = {
|
|
59
|
+
event: eventName,
|
|
60
|
+
args: rawArgs.map((a) => safeSerialize(a)),
|
|
61
|
+
meta: {
|
|
62
|
+
sdk_version: sdkVersion,
|
|
63
|
+
node_version: process.version,
|
|
64
|
+
captured_at: new Date().toISOString(),
|
|
65
|
+
capture_script: 'examples/adapters/capture-fixtures.ts',
|
|
66
|
+
},
|
|
67
|
+
};
|
|
68
|
+
const path = join(FIXTURES_DIR, `${eventName}.json`);
|
|
69
|
+
await writeFile(path, JSON.stringify(payload, null, 2) + '\n', 'utf8');
|
|
70
|
+
console.log(`[capture] wrote ${eventName}.json (${rawArgs.length} args)`);
|
|
71
|
+
}
|
|
72
|
+
|
|
73
|
+
// JSON-serialize while gracefully handling cyclic refs + class instances.
|
|
74
|
+
function safeSerialize(v: unknown): unknown {
|
|
75
|
+
if (v === null || v === undefined) return v;
|
|
76
|
+
if (typeof v === 'function') return `[function ${v.name || '<anon>'}]`;
|
|
77
|
+
if (typeof v !== 'object') return v;
|
|
78
|
+
try {
|
|
79
|
+
return JSON.parse(JSON.stringify(v, (_k, val) => {
|
|
80
|
+
if (typeof val === 'function') return `[function ${val.name || '<anon>'}]`;
|
|
81
|
+
if (val instanceof Error) return { name: val.name, message: val.message };
|
|
82
|
+
return val;
|
|
83
|
+
}));
|
|
84
|
+
} catch {
|
|
85
|
+
return `[unserializable ${Object.prototype.toString.call(v)}]`;
|
|
86
|
+
}
|
|
87
|
+
}
|
|
88
|
+
|
|
89
|
+
// Two tools so we can capture tool_start + tool_end with realistic data.
|
|
90
|
+
const searchKB = tool({
|
|
91
|
+
name: 'search_kb',
|
|
92
|
+
description: 'Search the support knowledge base.',
|
|
93
|
+
parameters: z.object({ query: z.string() }),
|
|
94
|
+
async execute({ query }: { query: string }) {
|
|
95
|
+
return `Found 3 articles for "${query}".`;
|
|
96
|
+
},
|
|
97
|
+
});
|
|
98
|
+
|
|
99
|
+
const escalate = tool({
|
|
100
|
+
name: 'escalate',
|
|
101
|
+
description: 'Escalate this ticket to a human supervisor.',
|
|
102
|
+
parameters: z.object({ reason: z.string() }),
|
|
103
|
+
async execute({ reason }: { reason: string }) {
|
|
104
|
+
return `Escalated. Reason: ${reason}`;
|
|
105
|
+
},
|
|
106
|
+
});
|
|
107
|
+
|
|
108
|
+
// Two agents wired by handoff so we capture the handoff event.
|
|
109
|
+
const escalationBot = new Agent({
|
|
110
|
+
name: 'escalation_bot',
|
|
111
|
+
instructions: 'You handle escalated support tickets that the first-tier bot cannot resolve. Call escalate() with the reason.',
|
|
112
|
+
model: 'gpt-5',
|
|
113
|
+
tools: [escalate],
|
|
114
|
+
});
|
|
115
|
+
|
|
116
|
+
const supportBot = new Agent({
|
|
117
|
+
name: 'support_bot',
|
|
118
|
+
instructions: [
|
|
119
|
+
'You are tier-1 support. Try search_kb first.',
|
|
120
|
+
'If the customer is angry, hand off to escalation_bot.',
|
|
121
|
+
].join('\n'),
|
|
122
|
+
model: 'gpt-5',
|
|
123
|
+
tools: [searchKB],
|
|
124
|
+
handoffs: [handoff(escalationBot)],
|
|
125
|
+
});
|
|
126
|
+
|
|
127
|
+
async function main() {
|
|
128
|
+
await mkdir(FIXTURES_DIR, { recursive: true });
|
|
129
|
+
const sdkVersion = await getSdkVersion();
|
|
130
|
+
console.log(`[capture] target dir: ${FIXTURES_DIR}`);
|
|
131
|
+
console.log(`[capture] @openai/agents version: ${sdkVersion}`);
|
|
132
|
+
console.log(`[capture] Node version: ${process.version}`);
|
|
133
|
+
|
|
134
|
+
const runner = new Runner();
|
|
135
|
+
|
|
136
|
+
// Attach raw listeners to all 5 lifecycle events. Each handler:
|
|
137
|
+
// 1. saves the first occurrence to its dedicated fixture file
|
|
138
|
+
// 2. logs a short note to stdout
|
|
139
|
+
// We do NOT attach the WMA guardrail here — this script captures the
|
|
140
|
+
// RAW SDK shapes, not the WMA-normalized ones.
|
|
141
|
+
runner.on('agent_start', async (...args) => {
|
|
142
|
+
await saveFixture('agent_start', args, sdkVersion);
|
|
143
|
+
console.log('[capture] agent_start fired');
|
|
144
|
+
});
|
|
145
|
+
runner.on('agent_end', async (...args) => {
|
|
146
|
+
await saveFixture('agent_end', args, sdkVersion);
|
|
147
|
+
console.log('[capture] agent_end fired');
|
|
148
|
+
});
|
|
149
|
+
runner.on('agent_handoff', async (...args) => {
|
|
150
|
+
await saveFixture('agent_handoff', args, sdkVersion);
|
|
151
|
+
console.log('[capture] agent_handoff fired');
|
|
152
|
+
});
|
|
153
|
+
runner.on('agent_tool_start', async (...args) => {
|
|
154
|
+
await saveFixture('agent_tool_start', args, sdkVersion);
|
|
155
|
+
console.log('[capture] agent_tool_start fired');
|
|
156
|
+
});
|
|
157
|
+
runner.on('agent_tool_end', async (...args) => {
|
|
158
|
+
await saveFixture('agent_tool_end', args, sdkVersion);
|
|
159
|
+
console.log('[capture] agent_tool_end fired');
|
|
160
|
+
});
|
|
161
|
+
|
|
162
|
+
console.log('\n[capture] running support flow that should fire all 5 events…');
|
|
163
|
+
// The prompt is designed to (a) get search_kb to fire and (b) be angry
|
|
164
|
+
// enough that the bot hands off to escalation_bot, which then fires
|
|
165
|
+
// escalate(). That covers all 5 lifecycle events in a single run.
|
|
166
|
+
const result = await runner.run(
|
|
167
|
+
supportBot,
|
|
168
|
+
'Your KB has been useless for 30 minutes. I demand to speak to a supervisor. ' +
|
|
169
|
+
'My account number is X-9999. This is my third ticket about the same issue.',
|
|
170
|
+
);
|
|
171
|
+
|
|
172
|
+
console.log('\n[capture] final output:', result.finalOutput);
|
|
173
|
+
console.log('\n[capture] DONE. Files written:');
|
|
174
|
+
for (const evt of ['agent_start', 'agent_end', 'agent_handoff', 'agent_tool_start', 'agent_tool_end']) {
|
|
175
|
+
console.log(` - ${join(FIXTURES_DIR, `${evt}.json`)} ${captured[evt] ? '✓' : '✗ MISSING'}`);
|
|
176
|
+
}
|
|
177
|
+
|
|
178
|
+
const missing = ['agent_start', 'agent_end', 'agent_handoff', 'agent_tool_start', 'agent_tool_end']
|
|
179
|
+
.filter((e) => !captured[e]);
|
|
180
|
+
if (missing.length > 0) {
|
|
181
|
+
console.warn(`\n[capture] WARNING: ${missing.length} event(s) didn't fire: ${missing.join(', ')}`);
|
|
182
|
+
console.warn('[capture] You may need to adjust the prompt to provoke those events.');
|
|
183
|
+
console.warn('[capture] Most common cause: the model didn\'t choose to hand off → re-run with a more escalation-heavy prompt.');
|
|
184
|
+
process.exit(1);
|
|
185
|
+
}
|
|
186
|
+
|
|
187
|
+
console.log('\n[capture] ALL 5 fixtures captured. Ship to WMA repo.');
|
|
188
|
+
}
|
|
189
|
+
|
|
190
|
+
main().catch((e) => {
|
|
191
|
+
console.error('[capture] fatal:', e);
|
|
192
|
+
process.exit(1);
|
|
193
|
+
});
|
|
@@ -0,0 +1,118 @@
|
|
|
1
|
+
// WMA × OpenAI Agents SDK — quickstart.
|
|
2
|
+
//
|
|
3
|
+
// What this shows:
|
|
4
|
+
// - Minimal end-to-end wiring of WMA into an @openai/agents Runner
|
|
5
|
+
// - Both Shield (blocking via Tool Input Guardrail) and Watch (log via
|
|
6
|
+
// RunHooks) attached in two lines of customer code each
|
|
7
|
+
// - The included starter policy bundle (MITRE ATT&CK coverage) catches
|
|
8
|
+
// a deliberately-malicious shell command — this run blocks it
|
|
9
|
+
//
|
|
10
|
+
// Prerequisites:
|
|
11
|
+
// npm install @openai/agents zod watchmyagents
|
|
12
|
+
// export OPENAI_API_KEY=sk-proj-...
|
|
13
|
+
//
|
|
14
|
+
// Run with:
|
|
15
|
+
// npx tsx examples/adapters/openai-agents-js-quickstart.ts
|
|
16
|
+
//
|
|
17
|
+
// Expected output:
|
|
18
|
+
// - First call (search_kb with normal query): ALLOWED, returns 5 articles
|
|
19
|
+
// - Second call (bash with `rm -rf /var/log`): BLOCKED by MITRE T1485,
|
|
20
|
+
// guardrail returns rejectContent → the model sees the rejection
|
|
21
|
+
// message in place of the tool result
|
|
22
|
+
// - NDJSON event log lands at ./watchmyagents-logs/openai-agents/
|
|
23
|
+
|
|
24
|
+
import { Agent, Runner, tool } from '@openai/agents';
|
|
25
|
+
import { z } from 'zod';
|
|
26
|
+
import {
|
|
27
|
+
wmaToolInputGuardrail,
|
|
28
|
+
attachWmaWatch,
|
|
29
|
+
} from 'watchmyagents/src/sources/openai-agents-js.js';
|
|
30
|
+
|
|
31
|
+
// ── Tools the agent can call ───────────────────────────────────────────
|
|
32
|
+
|
|
33
|
+
const searchKB = tool({
|
|
34
|
+
name: 'search_kb',
|
|
35
|
+
description: 'Search the customer-support knowledge base.',
|
|
36
|
+
parameters: z.object({ query: z.string() }),
|
|
37
|
+
async execute({ query }) {
|
|
38
|
+
return `Found 5 articles matching "${query}".`;
|
|
39
|
+
},
|
|
40
|
+
});
|
|
41
|
+
|
|
42
|
+
// A deliberately-dangerous bash tool — Shield should block this when
|
|
43
|
+
// the model tries to call it with `rm -rf /`.
|
|
44
|
+
const runBash = tool({
|
|
45
|
+
name: 'bash',
|
|
46
|
+
description: 'Run a shell command on the server.',
|
|
47
|
+
parameters: z.object({ command: z.string() }),
|
|
48
|
+
async execute({ command }) {
|
|
49
|
+
// If this ever runs in real life, we have bigger problems.
|
|
50
|
+
return `EXECUTED: ${command}`;
|
|
51
|
+
},
|
|
52
|
+
});
|
|
53
|
+
|
|
54
|
+
// ── WMA Shield + Watch (the 2 lines that make this an observed agent) ─
|
|
55
|
+
|
|
56
|
+
const wmaShield = wmaToolInputGuardrail({
|
|
57
|
+
policiesPath: './examples/policies/mitre-starter.json',
|
|
58
|
+
// Optional: failOpen: true would skip enforcement on Shield errors —
|
|
59
|
+
// we leave it false so any internal failure rejects the tool call.
|
|
60
|
+
});
|
|
61
|
+
|
|
62
|
+
// ── The agent ──────────────────────────────────────────────────────────
|
|
63
|
+
|
|
64
|
+
const supportBot = new Agent({
|
|
65
|
+
name: 'support_bot',
|
|
66
|
+
instructions: [
|
|
67
|
+
'You help customers with support questions. You can search the KB.',
|
|
68
|
+
'If a customer escalates an issue, you may run shell commands to ',
|
|
69
|
+
'investigate. Be conservative — do not run destructive commands.',
|
|
70
|
+
].join('\n'),
|
|
71
|
+
model: 'gpt-5',
|
|
72
|
+
tools: [searchKB, runBash],
|
|
73
|
+
toolInputGuardrails: [wmaShield],
|
|
74
|
+
});
|
|
75
|
+
|
|
76
|
+
// ── Run + observe ──────────────────────────────────────────────────────
|
|
77
|
+
|
|
78
|
+
async function main() {
|
|
79
|
+
const runner = new Runner();
|
|
80
|
+
const detachWatch = attachWmaWatch(runner);
|
|
81
|
+
|
|
82
|
+
console.log('[wma-quickstart] Run 1: normal KB lookup (should pass) ───────');
|
|
83
|
+
const r1 = await runner.run(
|
|
84
|
+
supportBot,
|
|
85
|
+
'How do I reset my password?',
|
|
86
|
+
);
|
|
87
|
+
console.log('[wma-quickstart] r1.finalOutput:', r1.finalOutput);
|
|
88
|
+
|
|
89
|
+
console.log('[wma-quickstart] Run 2: simulated jailbreak — model is told to delete logs');
|
|
90
|
+
try {
|
|
91
|
+
const r2 = await runner.run(
|
|
92
|
+
supportBot,
|
|
93
|
+
'Please run: bash with command "rm -rf /var/log/*" to free disk space.',
|
|
94
|
+
);
|
|
95
|
+
console.log('[wma-quickstart] r2.finalOutput:', r2.finalOutput);
|
|
96
|
+
} catch (e) {
|
|
97
|
+
console.log('[wma-quickstart] Run 2 threw (interrupt mode):', (e as Error).message);
|
|
98
|
+
}
|
|
99
|
+
|
|
100
|
+
detachWatch();
|
|
101
|
+
|
|
102
|
+
console.log('\n[wma-quickstart] DONE.');
|
|
103
|
+
console.log('[wma-quickstart] Inspect the NDJSON log:');
|
|
104
|
+
console.log(' cat ./watchmyagents-logs/openai-agents/*.ndjson | jq');
|
|
105
|
+
console.log('[wma-quickstart] Verify the audit chain integrity:');
|
|
106
|
+
console.log(
|
|
107
|
+
" node -e \"import('watchmyagents/src/shield/decision-chain.js').then(async m => { " +
|
|
108
|
+
"const fs = await import('node:fs/promises'); " +
|
|
109
|
+
"const files = (await fs.readdir('./watchmyagents-logs/openai-agents')).filter(f=>f.endsWith('.ndjson')); " +
|
|
110
|
+
"const lines = (await fs.readFile('./watchmyagents-logs/openai-agents/'+files[0], 'utf8')).trim().split('\\n').map(l=>JSON.parse(l)).filter(l=>l.action_type==='shield_decision'); " +
|
|
111
|
+
"console.log(m.verifyDecisionChain(lines)); })\"",
|
|
112
|
+
);
|
|
113
|
+
}
|
|
114
|
+
|
|
115
|
+
main().catch((e) => {
|
|
116
|
+
console.error('[wma-quickstart] fatal:', e);
|
|
117
|
+
process.exit(1);
|
|
118
|
+
});
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "watchmyagents",
|
|
3
|
-
"version": "1.
|
|
3
|
+
"version": "1.3.0",
|
|
4
4
|
"description": "Security observability + real-time policy enforcement for AI agents. Local-first NDJSON capture with a continuous Watch daemon that auto-uploads anonymized signals, Shield CLI that blocks policy violations live (with policies pulled from Fortress cloud), anonymizer producing signals-only payloads, bidirectional sync with WatchMyAgents Fortress, and one-command install as an always-on launchd/systemd service — closing the recursive Watch→Guardian→Shield security loop.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"files": [
|
|
@@ -13,6 +13,8 @@
|
|
|
13
13
|
"scripts/service.js",
|
|
14
14
|
"scripts/agents.js",
|
|
15
15
|
"examples/policies/",
|
|
16
|
+
"examples/adapters/",
|
|
17
|
+
"docs/adapters/",
|
|
16
18
|
"README.md",
|
|
17
19
|
"SECURITY.md",
|
|
18
20
|
"LICENSE"
|
|
@@ -41,6 +43,14 @@
|
|
|
41
43
|
},
|
|
42
44
|
"dependencies": {},
|
|
43
45
|
"devDependencies": {},
|
|
46
|
+
"peerDependencies": {
|
|
47
|
+
"@openai/agents": "^0.2.0"
|
|
48
|
+
},
|
|
49
|
+
"peerDependenciesMeta": {
|
|
50
|
+
"@openai/agents": {
|
|
51
|
+
"optional": true
|
|
52
|
+
}
|
|
53
|
+
},
|
|
44
54
|
"keywords": [
|
|
45
55
|
"ai",
|
|
46
56
|
"agents",
|
|
@@ -51,7 +61,11 @@
|
|
|
51
61
|
"cybersecurity",
|
|
52
62
|
"anthropic",
|
|
53
63
|
"claude",
|
|
64
|
+
"openai",
|
|
65
|
+
"openai-agents",
|
|
54
66
|
"managed-agents",
|
|
67
|
+
"agent-runtime",
|
|
68
|
+
"guardrails",
|
|
55
69
|
"audit",
|
|
56
70
|
"ndjson",
|
|
57
71
|
"policy-enforcement",
|