@nyancorn/agentests-security 0.1.1 → 1.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +43 -0
- package/README.md +17 -1
- package/SECURITY.md +49 -0
- package/STABILITY.md +33 -0
- package/package.json +14 -4
- package/schema/policy.schema.json +268 -0
package/CHANGELOG.md
ADDED
|
@@ -0,0 +1,43 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
All notable changes are listed here. The format follows
|
|
4
|
+
[Keep a Changelog](https://keepachangelog.com), and the project follows
|
|
5
|
+
[semantic versioning](https://semver.org) as described in [STABILITY.md](STABILITY.md).
|
|
6
|
+
|
|
7
|
+
## [1.0.0] - 2026-11-18
|
|
8
|
+
|
|
9
|
+
First stable release.
|
|
10
|
+
|
|
11
|
+
### Added
|
|
12
|
+
|
|
13
|
+
- **Stability promise:** [STABILITY.md](STABILITY.md) covers the version 1 policy format,
|
|
14
|
+
the CLI and its exit codes, the JSON report, and the library exports. Contract tests
|
|
15
|
+
enforce it.
|
|
16
|
+
- **JSON Schema:** `schema/policy.schema.json` describes the policy file. Point your
|
|
17
|
+
editor at it for autocompletion and validation.
|
|
18
|
+
- **Security policy:** [SECURITY.md](SECURITY.md) explains private vulnerability
|
|
19
|
+
reporting, what's in scope, and authorized use.
|
|
20
|
+
- **Release workflow:** releases are published from GitHub Actions through npm
|
|
21
|
+
trusted publishing, with provenance and no stored npm token.
|
|
22
|
+
- **CI:** every push and pull request runs the type check, the tests, and a pack check
|
|
23
|
+
on Node.js 22 and 24.
|
|
24
|
+
|
|
25
|
+
## [0.1.1] - 2026-11-18
|
|
26
|
+
|
|
27
|
+
### Fixed
|
|
28
|
+
|
|
29
|
+
- **Package links:** the repository, homepage, and issue links now point to
|
|
30
|
+
github.com/mlgxiii/agentests-security.
|
|
31
|
+
|
|
32
|
+
## [0.1.0] - 2026-11-18
|
|
33
|
+
|
|
34
|
+
### Added
|
|
35
|
+
|
|
36
|
+
- **First release:** the `run`, `validate`, and `mock` commands; the
|
|
37
|
+
`sustained-query-volume`, `systematic-query-pattern`, and `quota-exhaustion`
|
|
38
|
+
scenarios; the rate limit, quota, and detection expectations; and the safety
|
|
39
|
+
rules (an allowed-host list, a double opt-in for production, and a request budget).
|
|
40
|
+
|
|
41
|
+
[1.0.0]: https://www.npmjs.com/package/@nyancorn/agentests-security/v/1.0.0
|
|
42
|
+
[0.1.1]: https://www.npmjs.com/package/@nyancorn/agentests-security/v/0.1.1
|
|
43
|
+
[0.1.0]: https://www.npmjs.com/package/@nyancorn/agentests-security/v/0.1.0
|
package/README.md
CHANGED
|
@@ -1,5 +1,10 @@
|
|
|
1
1
|
# agentests-security
|
|
2
2
|
|
|
3
|
+
[](https://www.npmjs.com/package/@nyancorn/agentests-security)
|
|
4
|
+
[](https://github.com/mlgxiii/agentests-security/actions/workflows/ci.yml)
|
|
5
|
+
[](SECURITY.md#supply-chain)
|
|
6
|
+
[](LICENSE)
|
|
7
|
+
|
|
3
8
|
Test your AI API's defenses against model extraction in CI. A security policy sends
|
|
4
9
|
bounded, deterministic attack traffic to a local or explicitly authorized endpoint,
|
|
5
10
|
then checks that the rate limits, quotas, and detection events still fire. If one of
|
|
@@ -32,9 +37,10 @@ policy again to see that defense's test fail.
|
|
|
32
37
|
|
|
33
38
|
## Point it at your API
|
|
34
39
|
|
|
35
|
-
Write `agentests.security.yaml
|
|
40
|
+
Write `agentests.security.yaml`. The first line turns on autocompletion and validation in editors that use the YAML language server, such as VS Code with the YAML extension:
|
|
36
41
|
|
|
37
42
|
```yaml
|
|
43
|
+
# yaml-language-server: $schema=https://unpkg.com/@nyancorn/agentests-security@1/schema/policy.schema.json
|
|
38
44
|
version: 1
|
|
39
45
|
name: model-extraction-protection
|
|
40
46
|
target:
|
|
@@ -81,6 +87,16 @@ endpoint, and the safety limits.
|
|
|
81
87
|
|
|
82
88
|
Only test APIs you own or are authorized to test.
|
|
83
89
|
|
|
90
|
+
## Stability and support
|
|
91
|
+
|
|
92
|
+
From 1.0, the policy format (`version: 1`), the CLI and its exit codes, the `--json`
|
|
93
|
+
report, and the library exports are covered by a semantic versioning promise; see
|
|
94
|
+
[STABILITY.md](STABILITY.md). Contract tests enforce it, and the [changelog](CHANGELOG.md)
|
|
95
|
+
announces every change.
|
|
96
|
+
|
|
97
|
+
Releases are published from GitHub Actions with npm provenance. Report vulnerabilities
|
|
98
|
+
privately as described in [SECURITY.md](SECURITY.md).
|
|
99
|
+
|
|
84
100
|
## Library
|
|
85
101
|
|
|
86
102
|
```ts
|
package/SECURITY.md
ADDED
|
@@ -0,0 +1,49 @@
|
|
|
1
|
+
# Security policy
|
|
2
|
+
|
|
3
|
+
## Reporting a vulnerability
|
|
4
|
+
|
|
5
|
+
Report vulnerabilities privately through GitHub:
|
|
6
|
+
**[Report a vulnerability](https://github.com/mlgxiii/agentests-security/security/advisories/new)**.
|
|
7
|
+
Please don't open a public issue.
|
|
8
|
+
|
|
9
|
+
Include the version, a description, and the steps or a policy file that reproduce it.
|
|
10
|
+
You can expect an acknowledgement within 3 business days and a fix or mitigation
|
|
11
|
+
plan within 14 days. Fixes are announced in a GitHub security advisory and the
|
|
12
|
+
[changelog](CHANGELOG.md), with credit if you want it.
|
|
13
|
+
|
|
14
|
+
## Supported versions
|
|
15
|
+
|
|
16
|
+
| Version | Supported |
|
|
17
|
+
| --- | --- |
|
|
18
|
+
| 1.x (latest minor) | Yes |
|
|
19
|
+
| 0.x | No. Upgrade to 1.x. |
|
|
20
|
+
|
|
21
|
+
## In scope
|
|
22
|
+
|
|
23
|
+
Ways to make the runner do something its safety rules should prevent:
|
|
24
|
+
|
|
25
|
+
- sending a request to a host that the policy doesn't authorize
|
|
26
|
+
- running against a production target without both opt-ins
|
|
27
|
+
- sending more requests than `safety.max_requests`
|
|
28
|
+
- leaking a `${VAR}` value into output, the report, or a host it wasn't meant for
|
|
29
|
+
- a malicious policy file executing code or reading files beyond the policy itself
|
|
30
|
+
|
|
31
|
+
## Authorized use
|
|
32
|
+
|
|
33
|
+
This tool generates traffic that resembles a model extraction attack. Only run it
|
|
34
|
+
against APIs you own or have written permission to test. The safety rules guard
|
|
35
|
+
against mistakes; they don't make unauthorized testing acceptable.
|
|
36
|
+
|
|
37
|
+
## Supply chain
|
|
38
|
+
|
|
39
|
+
Every release from 1.0.0 on is published from [GitHub Actions](.github/workflows/release.yml)
|
|
40
|
+
through npm trusted publishing, with [npm provenance](https://docs.npmjs.com/generating-provenance-statements)
|
|
41
|
+
and no long-lived npm token. (The 0.x releases were published by hand.)
|
|
42
|
+
You can check that a release was built from this repository:
|
|
43
|
+
|
|
44
|
+
```sh
|
|
45
|
+
npm audit signatures
|
|
46
|
+
```
|
|
47
|
+
|
|
48
|
+
The package has three runtime dependencies (`zod`, `yaml`, and `clinguini`) and no
|
|
49
|
+
install scripts.
|
package/STABILITY.md
ADDED
|
@@ -0,0 +1,33 @@
|
|
|
1
|
+
# Stability
|
|
2
|
+
|
|
3
|
+
`@nyancorn/agentests-security` follows [semantic versioning](https://semver.org).
|
|
4
|
+
From 1.0.0, everything below is a public contract. It changes in a breaking way only
|
|
5
|
+
in a new major version, announced in the [changelog](CHANGELOG.md) at least one minor
|
|
6
|
+
release ahead, with a deprecation warning wherever one is possible.
|
|
7
|
+
|
|
8
|
+
## Covered by the 1.x promise
|
|
9
|
+
|
|
10
|
+
| Surface | Promise |
|
|
11
|
+
| --- | --- |
|
|
12
|
+
| Policy file, `version: 1` | Every policy valid on 1.0 stays valid and keeps its meaning on every 1.x release. New optional fields may be added. A breaking format change would introduce `version: 2`, and 1.x would keep reading `version: 1`. |
|
|
13
|
+
| [`schema/policy.schema.json`](schema/policy.schema.json) | Matches the format above, and is published at `https://unpkg.com/@nyancorn/agentests-security@1/schema/policy.schema.json`. |
|
|
14
|
+
| CLI | The commands `run`, `validate`, and `mock`, their flags, and the exit codes: 0 means every test passed, 1 means a regression, 2 means an error (an invalid policy, or an unreachable or unauthorized target). |
|
|
15
|
+
| `run --json` report | Existing fields keep their names, types, and meanings. New fields may be added. |
|
|
16
|
+
| Request headers | `x-agentests-run` and `x-agentests-test` are sent on every request. |
|
|
17
|
+
| Safety rules | Defaults never loosen in 1.x: only local hosts unless listed, production only with both opt-ins, and a hard request budget. |
|
|
18
|
+
| Library exports | The names exported from the package root, and their documented signatures. |
|
|
19
|
+
|
|
20
|
+
## Not covered
|
|
21
|
+
|
|
22
|
+
- **Human-readable output:** the wording of `run`, `validate`, and error messages. Parse `--json`, not text.
|
|
23
|
+
- **Prompt wording:** the exact prompts each scenario sends. They stay deterministic within a release, but may change in a minor release to better resemble real extraction traffic. A minor release may also tighten a scenario's traffic shape, so check the changelog when an expectation's result changes after an upgrade.
|
|
24
|
+
- **The mock API's internals:** its default limits, response bodies, and event format, beyond what the bundled example policy needs to pass.
|
|
25
|
+
- **Undocumented modules:** anything imported from a path inside `dist/`.
|
|
26
|
+
|
|
27
|
+
## Supported versions
|
|
28
|
+
|
|
29
|
+
The latest 1.x minor release gets fixes. When 2.0 ships, the last 1.x release gets
|
|
30
|
+
security fixes for 6 months.
|
|
31
|
+
|
|
32
|
+
These promises are enforced by `tests/contract.test.ts`: a frozen version 1 policy,
|
|
33
|
+
the shipped schema, the export list, and the report fields.
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@nyancorn/agentests-security",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "1.0.0",
|
|
4
4
|
"description": "Test an AI API's defenses against model extraction in CI: rate limits, quotas, and detection, as bounded, deterministic security policies.",
|
|
5
5
|
"keywords": [
|
|
6
6
|
"ai-security",
|
|
@@ -30,12 +30,17 @@
|
|
|
30
30
|
".": {
|
|
31
31
|
"types": "./dist/index.d.ts",
|
|
32
32
|
"default": "./dist/index.js"
|
|
33
|
-
}
|
|
33
|
+
},
|
|
34
|
+
"./schema.json": "./schema/policy.schema.json"
|
|
34
35
|
},
|
|
35
36
|
"files": [
|
|
36
37
|
"dist",
|
|
38
|
+
"schema",
|
|
37
39
|
"README.md",
|
|
38
|
-
"LICENSE"
|
|
40
|
+
"LICENSE",
|
|
41
|
+
"STABILITY.md",
|
|
42
|
+
"SECURITY.md",
|
|
43
|
+
"CHANGELOG.md"
|
|
39
44
|
],
|
|
40
45
|
"engines": {
|
|
41
46
|
"node": ">=22"
|
|
@@ -44,7 +49,8 @@
|
|
|
44
49
|
"build": "rm -rf dist && tsc -p tsconfig.json",
|
|
45
50
|
"check": "tsc -p tsconfig.json --noEmit",
|
|
46
51
|
"test": "npm run build && bun test ./tests",
|
|
47
|
-
"prepublishOnly": "npm test"
|
|
52
|
+
"prepublishOnly": "npm test",
|
|
53
|
+
"schema": "bun scripts/schema.ts"
|
|
48
54
|
},
|
|
49
55
|
"dependencies": {
|
|
50
56
|
"clinguini": "^1.0.1",
|
|
@@ -54,5 +60,9 @@
|
|
|
54
60
|
"devDependencies": {
|
|
55
61
|
"@types/node": "^22",
|
|
56
62
|
"typescript": "^6.0.3"
|
|
63
|
+
},
|
|
64
|
+
"publishConfig": {
|
|
65
|
+
"access": "public",
|
|
66
|
+
"provenance": true
|
|
57
67
|
}
|
|
58
68
|
}
|
|
@@ -0,0 +1,268 @@
|
|
|
1
|
+
{
|
|
2
|
+
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
|
3
|
+
"$id": "https://unpkg.com/@nyancorn/agentests-security@1/schema/policy.schema.json",
|
|
4
|
+
"title": "agentests-security policy, version 1",
|
|
5
|
+
"type": "object",
|
|
6
|
+
"properties": {
|
|
7
|
+
"version": {
|
|
8
|
+
"type": "number",
|
|
9
|
+
"const": 1
|
|
10
|
+
},
|
|
11
|
+
"name": {
|
|
12
|
+
"type": "string",
|
|
13
|
+
"pattern": "^[a-z0-9]+(-[a-z0-9]+)*$"
|
|
14
|
+
},
|
|
15
|
+
"target": {
|
|
16
|
+
"type": "object",
|
|
17
|
+
"properties": {
|
|
18
|
+
"url": {
|
|
19
|
+
"type": "string",
|
|
20
|
+
"format": "uri"
|
|
21
|
+
},
|
|
22
|
+
"environment": {
|
|
23
|
+
"type": "string",
|
|
24
|
+
"enum": [
|
|
25
|
+
"local",
|
|
26
|
+
"test",
|
|
27
|
+
"staging",
|
|
28
|
+
"production"
|
|
29
|
+
]
|
|
30
|
+
},
|
|
31
|
+
"headers": {
|
|
32
|
+
"default": {},
|
|
33
|
+
"type": "object",
|
|
34
|
+
"propertyNames": {
|
|
35
|
+
"type": "string",
|
|
36
|
+
"minLength": 1
|
|
37
|
+
},
|
|
38
|
+
"additionalProperties": {
|
|
39
|
+
"type": "string"
|
|
40
|
+
}
|
|
41
|
+
},
|
|
42
|
+
"body": {
|
|
43
|
+
"default": {
|
|
44
|
+
"prompt": "{{prompt}}"
|
|
45
|
+
}
|
|
46
|
+
},
|
|
47
|
+
"timeout_ms": {
|
|
48
|
+
"default": 10000,
|
|
49
|
+
"type": "integer",
|
|
50
|
+
"minimum": 100,
|
|
51
|
+
"maximum": 60000
|
|
52
|
+
},
|
|
53
|
+
"signals": {
|
|
54
|
+
"default": {
|
|
55
|
+
"rate_limit": {
|
|
56
|
+
"status": [
|
|
57
|
+
429
|
|
58
|
+
]
|
|
59
|
+
},
|
|
60
|
+
"quota": {
|
|
61
|
+
"status": [
|
|
62
|
+
402,
|
|
63
|
+
403,
|
|
64
|
+
429
|
|
65
|
+
],
|
|
66
|
+
"body": "quota"
|
|
67
|
+
}
|
|
68
|
+
},
|
|
69
|
+
"type": "object",
|
|
70
|
+
"properties": {
|
|
71
|
+
"rate_limit": {
|
|
72
|
+
"default": {
|
|
73
|
+
"status": [
|
|
74
|
+
429
|
|
75
|
+
]
|
|
76
|
+
},
|
|
77
|
+
"type": "object",
|
|
78
|
+
"properties": {
|
|
79
|
+
"status": {
|
|
80
|
+
"minItems": 1,
|
|
81
|
+
"type": "array",
|
|
82
|
+
"items": {
|
|
83
|
+
"type": "integer",
|
|
84
|
+
"minimum": 100,
|
|
85
|
+
"maximum": 599
|
|
86
|
+
}
|
|
87
|
+
},
|
|
88
|
+
"body": {
|
|
89
|
+
"type": "string"
|
|
90
|
+
}
|
|
91
|
+
},
|
|
92
|
+
"required": [
|
|
93
|
+
"status"
|
|
94
|
+
],
|
|
95
|
+
"additionalProperties": false
|
|
96
|
+
},
|
|
97
|
+
"quota": {
|
|
98
|
+
"default": {
|
|
99
|
+
"status": [
|
|
100
|
+
402,
|
|
101
|
+
403,
|
|
102
|
+
429
|
|
103
|
+
],
|
|
104
|
+
"body": "quota"
|
|
105
|
+
},
|
|
106
|
+
"type": "object",
|
|
107
|
+
"properties": {
|
|
108
|
+
"status": {
|
|
109
|
+
"minItems": 1,
|
|
110
|
+
"type": "array",
|
|
111
|
+
"items": {
|
|
112
|
+
"type": "integer",
|
|
113
|
+
"minimum": 100,
|
|
114
|
+
"maximum": 599
|
|
115
|
+
}
|
|
116
|
+
},
|
|
117
|
+
"body": {
|
|
118
|
+
"type": "string"
|
|
119
|
+
}
|
|
120
|
+
},
|
|
121
|
+
"required": [
|
|
122
|
+
"status"
|
|
123
|
+
],
|
|
124
|
+
"additionalProperties": false
|
|
125
|
+
}
|
|
126
|
+
},
|
|
127
|
+
"additionalProperties": false
|
|
128
|
+
},
|
|
129
|
+
"detection": {
|
|
130
|
+
"type": "object",
|
|
131
|
+
"properties": {
|
|
132
|
+
"url": {
|
|
133
|
+
"type": "string",
|
|
134
|
+
"format": "uri"
|
|
135
|
+
},
|
|
136
|
+
"headers": {
|
|
137
|
+
"default": {},
|
|
138
|
+
"type": "object",
|
|
139
|
+
"propertyNames": {
|
|
140
|
+
"type": "string",
|
|
141
|
+
"minLength": 1
|
|
142
|
+
},
|
|
143
|
+
"additionalProperties": {
|
|
144
|
+
"type": "string"
|
|
145
|
+
}
|
|
146
|
+
},
|
|
147
|
+
"wait_ms": {
|
|
148
|
+
"default": 0,
|
|
149
|
+
"type": "integer",
|
|
150
|
+
"minimum": 0,
|
|
151
|
+
"maximum": 60000
|
|
152
|
+
}
|
|
153
|
+
},
|
|
154
|
+
"required": [
|
|
155
|
+
"url"
|
|
156
|
+
],
|
|
157
|
+
"additionalProperties": false
|
|
158
|
+
}
|
|
159
|
+
},
|
|
160
|
+
"required": [
|
|
161
|
+
"url",
|
|
162
|
+
"environment"
|
|
163
|
+
],
|
|
164
|
+
"additionalProperties": false
|
|
165
|
+
},
|
|
166
|
+
"tests": {
|
|
167
|
+
"minItems": 1,
|
|
168
|
+
"type": "array",
|
|
169
|
+
"items": {
|
|
170
|
+
"type": "object",
|
|
171
|
+
"properties": {
|
|
172
|
+
"id": {
|
|
173
|
+
"type": "string",
|
|
174
|
+
"pattern": "^[a-z0-9]+(-[a-z0-9]+)*$"
|
|
175
|
+
},
|
|
176
|
+
"description": {
|
|
177
|
+
"type": "string"
|
|
178
|
+
},
|
|
179
|
+
"scenario": {
|
|
180
|
+
"type": "string",
|
|
181
|
+
"enum": [
|
|
182
|
+
"sustained-query-volume",
|
|
183
|
+
"systematic-query-pattern",
|
|
184
|
+
"quota-exhaustion"
|
|
185
|
+
]
|
|
186
|
+
},
|
|
187
|
+
"requests": {
|
|
188
|
+
"type": "integer",
|
|
189
|
+
"minimum": 1,
|
|
190
|
+
"maximum": 10000
|
|
191
|
+
},
|
|
192
|
+
"duration_seconds": {
|
|
193
|
+
"default": 0,
|
|
194
|
+
"type": "number",
|
|
195
|
+
"minimum": 0,
|
|
196
|
+
"maximum": 3600
|
|
197
|
+
},
|
|
198
|
+
"headers": {
|
|
199
|
+
"default": {},
|
|
200
|
+
"type": "object",
|
|
201
|
+
"propertyNames": {
|
|
202
|
+
"type": "string",
|
|
203
|
+
"minLength": 1
|
|
204
|
+
},
|
|
205
|
+
"additionalProperties": {
|
|
206
|
+
"type": "string"
|
|
207
|
+
}
|
|
208
|
+
},
|
|
209
|
+
"expect": {
|
|
210
|
+
"type": "object",
|
|
211
|
+
"properties": {
|
|
212
|
+
"rate_limit_enforced": {
|
|
213
|
+
"type": "boolean"
|
|
214
|
+
},
|
|
215
|
+
"quota_enforced": {
|
|
216
|
+
"type": "boolean"
|
|
217
|
+
},
|
|
218
|
+
"detection_event": {
|
|
219
|
+
"type": "boolean"
|
|
220
|
+
}
|
|
221
|
+
},
|
|
222
|
+
"additionalProperties": false
|
|
223
|
+
}
|
|
224
|
+
},
|
|
225
|
+
"required": [
|
|
226
|
+
"id",
|
|
227
|
+
"scenario",
|
|
228
|
+
"requests",
|
|
229
|
+
"expect"
|
|
230
|
+
],
|
|
231
|
+
"additionalProperties": false
|
|
232
|
+
}
|
|
233
|
+
},
|
|
234
|
+
"safety": {
|
|
235
|
+
"type": "object",
|
|
236
|
+
"properties": {
|
|
237
|
+
"max_requests": {
|
|
238
|
+
"type": "integer",
|
|
239
|
+
"minimum": 1,
|
|
240
|
+
"maximum": 10000
|
|
241
|
+
},
|
|
242
|
+
"allow_production": {
|
|
243
|
+
"default": false,
|
|
244
|
+
"type": "boolean"
|
|
245
|
+
},
|
|
246
|
+
"allowed_hosts": {
|
|
247
|
+
"default": [],
|
|
248
|
+
"type": "array",
|
|
249
|
+
"items": {
|
|
250
|
+
"type": "string"
|
|
251
|
+
}
|
|
252
|
+
}
|
|
253
|
+
},
|
|
254
|
+
"required": [
|
|
255
|
+
"max_requests"
|
|
256
|
+
],
|
|
257
|
+
"additionalProperties": false
|
|
258
|
+
}
|
|
259
|
+
},
|
|
260
|
+
"required": [
|
|
261
|
+
"version",
|
|
262
|
+
"name",
|
|
263
|
+
"target",
|
|
264
|
+
"tests",
|
|
265
|
+
"safety"
|
|
266
|
+
],
|
|
267
|
+
"additionalProperties": false
|
|
268
|
+
}
|