@nyancorn/agentests-security 0.1.1 → 1.0.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md ADDED
@@ -0,0 +1,43 @@
1
+ # Changelog
2
+
3
+ All notable changes are listed here. The format follows
4
+ [Keep a Changelog](https://keepachangelog.com), and the project follows
5
+ [semantic versioning](https://semver.org) as described in [STABILITY.md](STABILITY.md).
6
+
7
+ ## [1.0.0] - 2026-11-18
8
+
9
+ First stable release.
10
+
11
+ ### Added
12
+
13
+ - **Stability promise:** [STABILITY.md](STABILITY.md) covers the version 1 policy format,
14
+ the CLI and its exit codes, the JSON report, and the library exports. Contract tests
15
+ enforce it.
16
+ - **JSON Schema:** `schema/policy.schema.json` describes the policy file. Point your
17
+ editor at it for autocompletion and validation.
18
+ - **Security policy:** [SECURITY.md](SECURITY.md) explains private vulnerability
19
+ reporting, what's in scope, and authorized use.
20
+ - **Release workflow:** releases are published from GitHub Actions through npm
21
+ trusted publishing, with provenance and no stored npm token.
22
+ - **CI:** every push and pull request runs the type check, the tests, and a pack check
23
+ on Node.js 22 and 24.
24
+
25
+ ## [0.1.1] - 2026-11-18
26
+
27
+ ### Fixed
28
+
29
+ - **Package links:** the repository, homepage, and issue links now point to
30
+ github.com/mlgxiii/agentests-security.
31
+
32
+ ## [0.1.0] - 2026-11-18
33
+
34
+ ### Added
35
+
36
+ - **First release:** the `run`, `validate`, and `mock` commands; the
37
+ `sustained-query-volume`, `systematic-query-pattern`, and `quota-exhaustion`
38
+ scenarios; the rate limit, quota, and detection expectations; and the safety
39
+ rules (an allowed-host list, a double opt-in for production, and a request budget).
40
+
41
+ [1.0.0]: https://www.npmjs.com/package/@nyancorn/agentests-security/v/1.0.0
42
+ [0.1.1]: https://www.npmjs.com/package/@nyancorn/agentests-security/v/0.1.1
43
+ [0.1.0]: https://www.npmjs.com/package/@nyancorn/agentests-security/v/0.1.0
package/README.md CHANGED
@@ -1,5 +1,10 @@
1
1
  # agentests-security
2
2
 
3
+ [![npm](https://img.shields.io/npm/v/@nyancorn/agentests-security?logo=npm)](https://www.npmjs.com/package/@nyancorn/agentests-security)
4
+ [![CI](https://github.com/mlgxiii/agentests-security/actions/workflows/ci.yml/badge.svg)](https://github.com/mlgxiii/agentests-security/actions/workflows/ci.yml)
5
+ [![provenance](https://img.shields.io/badge/npm-provenance-blue)](SECURITY.md#supply-chain)
6
+ [![MIT license](https://img.shields.io/badge/license-MIT-blue)](LICENSE)
7
+
3
8
  Test your AI API's defenses against model extraction in CI. A security policy sends
4
9
  bounded, deterministic attack traffic to a local or explicitly authorized endpoint,
5
10
  then checks that the rate limits, quotas, and detection events still fire. If one of
@@ -32,9 +37,10 @@ policy again to see that defense's test fail.
32
37
 
33
38
  ## Point it at your API
34
39
 
35
- Write `agentests.security.yaml`:
40
+ Write `agentests.security.yaml`. The first line turns on autocompletion and validation in editors that use the YAML language server, such as VS Code with the YAML extension:
36
41
 
37
42
  ```yaml
43
+ # yaml-language-server: $schema=https://unpkg.com/@nyancorn/agentests-security@1/schema/policy.schema.json
38
44
  version: 1
39
45
  name: model-extraction-protection
40
46
  target:
@@ -81,6 +87,16 @@ endpoint, and the safety limits.
81
87
 
82
88
  Only test APIs you own or are authorized to test.
83
89
 
90
+ ## Stability and support
91
+
92
+ From 1.0, the policy format (`version: 1`), the CLI and its exit codes, the `--json`
93
+ report, and the library exports are covered by a semantic versioning promise; see
94
+ [STABILITY.md](STABILITY.md). Contract tests enforce it, and the [changelog](CHANGELOG.md)
95
+ announces every change.
96
+
97
+ Releases are published from GitHub Actions with npm provenance. Report vulnerabilities
98
+ privately as described in [SECURITY.md](SECURITY.md).
99
+
84
100
  ## Library
85
101
 
86
102
  ```ts
package/SECURITY.md ADDED
@@ -0,0 +1,49 @@
1
+ # Security policy
2
+
3
+ ## Reporting a vulnerability
4
+
5
+ Report vulnerabilities privately through GitHub:
6
+ **[Report a vulnerability](https://github.com/mlgxiii/agentests-security/security/advisories/new)**.
7
+ Please don't open a public issue.
8
+
9
+ Include the version, a description, and the steps or a policy file that reproduce it.
10
+ You can expect an acknowledgement within 3 business days and a fix or mitigation
11
+ plan within 14 days. Fixes are announced in a GitHub security advisory and the
12
+ [changelog](CHANGELOG.md), with credit if you want it.
13
+
14
+ ## Supported versions
15
+
16
+ | Version | Supported |
17
+ | --- | --- |
18
+ | 1.x (latest minor) | Yes |
19
+ | 0.x | No. Upgrade to 1.x. |
20
+
21
+ ## In scope
22
+
23
+ Ways to make the runner do something its safety rules should prevent:
24
+
25
+ - sending a request to a host that the policy doesn't authorize
26
+ - running against a production target without both opt-ins
27
+ - sending more requests than `safety.max_requests`
28
+ - leaking a `${VAR}` value into output, the report, or a host it wasn't meant for
29
+ - a malicious policy file executing code or reading files beyond the policy itself
30
+
31
+ ## Authorized use
32
+
33
+ This tool generates traffic that resembles a model extraction attack. Only run it
34
+ against APIs you own or have written permission to test. The safety rules guard
35
+ against mistakes; they don't make unauthorized testing acceptable.
36
+
37
+ ## Supply chain
38
+
39
+ Every release from 1.0.0 on is published from [GitHub Actions](.github/workflows/release.yml)
40
+ through npm trusted publishing, with [npm provenance](https://docs.npmjs.com/generating-provenance-statements)
41
+ and no long-lived npm token. (The 0.x releases were published by hand.)
42
+ You can check that a release was built from this repository:
43
+
44
+ ```sh
45
+ npm audit signatures
46
+ ```
47
+
48
+ The package has three runtime dependencies (`zod`, `yaml`, and `clinguini`) and no
49
+ install scripts.
package/STABILITY.md ADDED
@@ -0,0 +1,33 @@
1
+ # Stability
2
+
3
+ `@nyancorn/agentests-security` follows [semantic versioning](https://semver.org).
4
+ From 1.0.0, everything below is a public contract. It changes in a breaking way only
5
+ in a new major version, announced in the [changelog](CHANGELOG.md) at least one minor
6
+ release ahead, with a deprecation warning wherever one is possible.
7
+
8
+ ## Covered by the 1.x promise
9
+
10
+ | Surface | Promise |
11
+ | --- | --- |
12
+ | Policy file, `version: 1` | Every policy valid on 1.0 stays valid and keeps its meaning on every 1.x release. New optional fields may be added. A breaking format change would introduce `version: 2`, and 1.x would keep reading `version: 1`. |
13
+ | [`schema/policy.schema.json`](schema/policy.schema.json) | Matches the format above, and is published at `https://unpkg.com/@nyancorn/agentests-security@1/schema/policy.schema.json`. |
14
+ | CLI | The commands `run`, `validate`, and `mock`, their flags, and the exit codes: 0 means every test passed, 1 means a regression, 2 means an error (an invalid policy, or an unreachable or unauthorized target). |
15
+ | `run --json` report | Existing fields keep their names, types, and meanings. New fields may be added. |
16
+ | Request headers | `x-agentests-run` and `x-agentests-test` are sent on every request. |
17
+ | Safety rules | Defaults never loosen in 1.x: only local hosts unless listed, production only with both opt-ins, and a hard request budget. |
18
+ | Library exports | The names exported from the package root, and their documented signatures. |
19
+
20
+ ## Not covered
21
+
22
+ - **Human-readable output:** the wording of `run`, `validate`, and error messages. Parse `--json`, not text.
23
+ - **Prompt wording:** the exact prompts each scenario sends. They stay deterministic within a release, but may change in a minor release to better resemble real extraction traffic. A minor release may also tighten a scenario's traffic shape, so check the changelog when an expectation's result changes after an upgrade.
24
+ - **The mock API's internals:** its default limits, response bodies, and event format, beyond what the bundled example policy needs to pass.
25
+ - **Undocumented modules:** anything imported from a path inside `dist/`.
26
+
27
+ ## Supported versions
28
+
29
+ The latest 1.x minor release gets fixes. When 2.0 ships, the last 1.x release gets
30
+ security fixes for 6 months.
31
+
32
+ These promises are enforced by `tests/contract.test.ts`: a frozen version 1 policy,
33
+ the shipped schema, the export list, and the report fields.
package/package.json CHANGED
@@ -1,6 +1,6 @@
1
1
  {
2
2
  "name": "@nyancorn/agentests-security",
3
- "version": "0.1.1",
3
+ "version": "1.0.0",
4
4
  "description": "Test an AI API's defenses against model extraction in CI: rate limits, quotas, and detection, as bounded, deterministic security policies.",
5
5
  "keywords": [
6
6
  "ai-security",
@@ -30,12 +30,17 @@
30
30
  ".": {
31
31
  "types": "./dist/index.d.ts",
32
32
  "default": "./dist/index.js"
33
- }
33
+ },
34
+ "./schema.json": "./schema/policy.schema.json"
34
35
  },
35
36
  "files": [
36
37
  "dist",
38
+ "schema",
37
39
  "README.md",
38
- "LICENSE"
40
+ "LICENSE",
41
+ "STABILITY.md",
42
+ "SECURITY.md",
43
+ "CHANGELOG.md"
39
44
  ],
40
45
  "engines": {
41
46
  "node": ">=22"
@@ -44,7 +49,8 @@
44
49
  "build": "rm -rf dist && tsc -p tsconfig.json",
45
50
  "check": "tsc -p tsconfig.json --noEmit",
46
51
  "test": "npm run build && bun test ./tests",
47
- "prepublishOnly": "npm test"
52
+ "prepublishOnly": "npm test",
53
+ "schema": "bun scripts/schema.ts"
48
54
  },
49
55
  "dependencies": {
50
56
  "clinguini": "^1.0.1",
@@ -54,5 +60,9 @@
54
60
  "devDependencies": {
55
61
  "@types/node": "^22",
56
62
  "typescript": "^6.0.3"
63
+ },
64
+ "publishConfig": {
65
+ "access": "public",
66
+ "provenance": true
57
67
  }
58
68
  }
@@ -0,0 +1,268 @@
1
+ {
2
+ "$schema": "https://json-schema.org/draft/2020-12/schema",
3
+ "$id": "https://unpkg.com/@nyancorn/agentests-security@1/schema/policy.schema.json",
4
+ "title": "agentests-security policy, version 1",
5
+ "type": "object",
6
+ "properties": {
7
+ "version": {
8
+ "type": "number",
9
+ "const": 1
10
+ },
11
+ "name": {
12
+ "type": "string",
13
+ "pattern": "^[a-z0-9]+(-[a-z0-9]+)*$"
14
+ },
15
+ "target": {
16
+ "type": "object",
17
+ "properties": {
18
+ "url": {
19
+ "type": "string",
20
+ "format": "uri"
21
+ },
22
+ "environment": {
23
+ "type": "string",
24
+ "enum": [
25
+ "local",
26
+ "test",
27
+ "staging",
28
+ "production"
29
+ ]
30
+ },
31
+ "headers": {
32
+ "default": {},
33
+ "type": "object",
34
+ "propertyNames": {
35
+ "type": "string",
36
+ "minLength": 1
37
+ },
38
+ "additionalProperties": {
39
+ "type": "string"
40
+ }
41
+ },
42
+ "body": {
43
+ "default": {
44
+ "prompt": "{{prompt}}"
45
+ }
46
+ },
47
+ "timeout_ms": {
48
+ "default": 10000,
49
+ "type": "integer",
50
+ "minimum": 100,
51
+ "maximum": 60000
52
+ },
53
+ "signals": {
54
+ "default": {
55
+ "rate_limit": {
56
+ "status": [
57
+ 429
58
+ ]
59
+ },
60
+ "quota": {
61
+ "status": [
62
+ 402,
63
+ 403,
64
+ 429
65
+ ],
66
+ "body": "quota"
67
+ }
68
+ },
69
+ "type": "object",
70
+ "properties": {
71
+ "rate_limit": {
72
+ "default": {
73
+ "status": [
74
+ 429
75
+ ]
76
+ },
77
+ "type": "object",
78
+ "properties": {
79
+ "status": {
80
+ "minItems": 1,
81
+ "type": "array",
82
+ "items": {
83
+ "type": "integer",
84
+ "minimum": 100,
85
+ "maximum": 599
86
+ }
87
+ },
88
+ "body": {
89
+ "type": "string"
90
+ }
91
+ },
92
+ "required": [
93
+ "status"
94
+ ],
95
+ "additionalProperties": false
96
+ },
97
+ "quota": {
98
+ "default": {
99
+ "status": [
100
+ 402,
101
+ 403,
102
+ 429
103
+ ],
104
+ "body": "quota"
105
+ },
106
+ "type": "object",
107
+ "properties": {
108
+ "status": {
109
+ "minItems": 1,
110
+ "type": "array",
111
+ "items": {
112
+ "type": "integer",
113
+ "minimum": 100,
114
+ "maximum": 599
115
+ }
116
+ },
117
+ "body": {
118
+ "type": "string"
119
+ }
120
+ },
121
+ "required": [
122
+ "status"
123
+ ],
124
+ "additionalProperties": false
125
+ }
126
+ },
127
+ "additionalProperties": false
128
+ },
129
+ "detection": {
130
+ "type": "object",
131
+ "properties": {
132
+ "url": {
133
+ "type": "string",
134
+ "format": "uri"
135
+ },
136
+ "headers": {
137
+ "default": {},
138
+ "type": "object",
139
+ "propertyNames": {
140
+ "type": "string",
141
+ "minLength": 1
142
+ },
143
+ "additionalProperties": {
144
+ "type": "string"
145
+ }
146
+ },
147
+ "wait_ms": {
148
+ "default": 0,
149
+ "type": "integer",
150
+ "minimum": 0,
151
+ "maximum": 60000
152
+ }
153
+ },
154
+ "required": [
155
+ "url"
156
+ ],
157
+ "additionalProperties": false
158
+ }
159
+ },
160
+ "required": [
161
+ "url",
162
+ "environment"
163
+ ],
164
+ "additionalProperties": false
165
+ },
166
+ "tests": {
167
+ "minItems": 1,
168
+ "type": "array",
169
+ "items": {
170
+ "type": "object",
171
+ "properties": {
172
+ "id": {
173
+ "type": "string",
174
+ "pattern": "^[a-z0-9]+(-[a-z0-9]+)*$"
175
+ },
176
+ "description": {
177
+ "type": "string"
178
+ },
179
+ "scenario": {
180
+ "type": "string",
181
+ "enum": [
182
+ "sustained-query-volume",
183
+ "systematic-query-pattern",
184
+ "quota-exhaustion"
185
+ ]
186
+ },
187
+ "requests": {
188
+ "type": "integer",
189
+ "minimum": 1,
190
+ "maximum": 10000
191
+ },
192
+ "duration_seconds": {
193
+ "default": 0,
194
+ "type": "number",
195
+ "minimum": 0,
196
+ "maximum": 3600
197
+ },
198
+ "headers": {
199
+ "default": {},
200
+ "type": "object",
201
+ "propertyNames": {
202
+ "type": "string",
203
+ "minLength": 1
204
+ },
205
+ "additionalProperties": {
206
+ "type": "string"
207
+ }
208
+ },
209
+ "expect": {
210
+ "type": "object",
211
+ "properties": {
212
+ "rate_limit_enforced": {
213
+ "type": "boolean"
214
+ },
215
+ "quota_enforced": {
216
+ "type": "boolean"
217
+ },
218
+ "detection_event": {
219
+ "type": "boolean"
220
+ }
221
+ },
222
+ "additionalProperties": false
223
+ }
224
+ },
225
+ "required": [
226
+ "id",
227
+ "scenario",
228
+ "requests",
229
+ "expect"
230
+ ],
231
+ "additionalProperties": false
232
+ }
233
+ },
234
+ "safety": {
235
+ "type": "object",
236
+ "properties": {
237
+ "max_requests": {
238
+ "type": "integer",
239
+ "minimum": 1,
240
+ "maximum": 10000
241
+ },
242
+ "allow_production": {
243
+ "default": false,
244
+ "type": "boolean"
245
+ },
246
+ "allowed_hosts": {
247
+ "default": [],
248
+ "type": "array",
249
+ "items": {
250
+ "type": "string"
251
+ }
252
+ }
253
+ },
254
+ "required": [
255
+ "max_requests"
256
+ ],
257
+ "additionalProperties": false
258
+ }
259
+ },
260
+ "required": [
261
+ "version",
262
+ "name",
263
+ "target",
264
+ "tests",
265
+ "safety"
266
+ ],
267
+ "additionalProperties": false
268
+ }