@nyancorn/agentests-security 0.1.1 → 1.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +77 -0
- package/README.md +34 -1
- package/SECURITY.md +49 -0
- package/STABILITY.md +33 -0
- package/dist/cli.js +16 -4
- package/dist/index.d.ts +1 -1
- package/dist/index.js +1 -1
- package/dist/policy/types.d.ts +5 -0
- package/dist/policy/validate.d.ts +5 -0
- package/dist/policy/validate.js +38 -0
- package/dist/probe/http.d.ts +14 -2
- package/dist/probe/http.js +63 -12
- package/dist/probe/index.d.ts +4 -1
- package/dist/probe/index.js +52 -13
- package/dist/probe/redact.d.ts +12 -0
- package/dist/probe/redact.js +66 -0
- package/dist/probe/types.d.ts +18 -2
- package/dist/probe/types.js +17 -5
- package/dist/probe/verdict.d.ts +3 -1
- package/dist/probe/verdict.js +37 -9
- package/package.json +14 -4
- package/schema/policy.schema.json +291 -0
package/CHANGELOG.md
ADDED
|
@@ -0,0 +1,77 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
All notable changes are listed here. The format follows
|
|
4
|
+
[Keep a Changelog](https://keepachangelog.com), and the project follows
|
|
5
|
+
[semantic versioning](https://semver.org) as described in [STABILITY.md](STABILITY.md).
|
|
6
|
+
|
|
7
|
+
## [1.1.0] - 2026-11-18
|
|
8
|
+
|
|
9
|
+
Hardening of the runner. Every 1.0 policy is still valid.
|
|
10
|
+
|
|
11
|
+
### Added
|
|
12
|
+
|
|
13
|
+
- **`expect.rate_limit_within` and `expect.quota_within`:** require a defense to fire by a
|
|
14
|
+
given request, not just at some point in the test.
|
|
15
|
+
- **`target.detection.run_field`:** count only the detection events tagged with this run's
|
|
16
|
+
`x-agentests-run` id, so other traffic can't make a test pass. Detection headers accept
|
|
17
|
+
`{{run_id}}`. The report's evidence says whether detection was correlated
|
|
18
|
+
(`detectionCorrelated`). The mock API tags its events with `run`.
|
|
19
|
+
- **`max_error_rate` per test (default 0.05):** a test whose responses are mostly errors
|
|
20
|
+
matching no signal fails as inconclusive, with the reason in `result.inconclusive`. The
|
|
21
|
+
evidence counts these responses in `errors`.
|
|
22
|
+
- **`safety.allow_insecure_http`:** accept plain HTTP to a remote host without a warning.
|
|
23
|
+
- **`warnings` in the report**, and `Warning:` lines on stderr from `run` and `validate`.
|
|
24
|
+
- **`securityPolicyWarnings(policy)`** in the library.
|
|
25
|
+
|
|
26
|
+
### Changed
|
|
27
|
+
|
|
28
|
+
- **Errors no longer pass:** a test that expects a defense to be absent
|
|
29
|
+
(`…: false`) used to pass when the API answered with errors. It now fails as inconclusive
|
|
30
|
+
when more than 5% of responses are errors matching no signal. Raise `max_error_rate` to
|
|
31
|
+
keep the old behaviour.
|
|
32
|
+
- **Responses are read with a cap:** 64 KiB per completion response, with the rest of a
|
|
33
|
+
stream cancelled, and 4 MiB per detection response, beyond which the run stops with an error.
|
|
34
|
+
- **Secrets are redacted** from the report and from error messages.
|
|
35
|
+
|
|
36
|
+
### Deprecated
|
|
37
|
+
|
|
38
|
+
- **Plain HTTP to a remote host:** it is a warning in 1.x and will be an error in 2.0
|
|
39
|
+
unless `safety.allow_insecure_http: true` is set.
|
|
40
|
+
|
|
41
|
+
## [1.0.0] - 2026-11-18
|
|
42
|
+
|
|
43
|
+
First stable release.
|
|
44
|
+
|
|
45
|
+
### Added
|
|
46
|
+
|
|
47
|
+
- **Stability promise:** [STABILITY.md](STABILITY.md) covers the version 1 policy format,
|
|
48
|
+
the CLI and its exit codes, the JSON report, and the library exports. Contract tests
|
|
49
|
+
enforce it.
|
|
50
|
+
- **JSON Schema:** `schema/policy.schema.json` describes the policy file. Point your
|
|
51
|
+
editor at it for autocompletion and validation.
|
|
52
|
+
- **Security policy:** [SECURITY.md](SECURITY.md) explains private vulnerability
|
|
53
|
+
reporting, what's in scope, and authorized use.
|
|
54
|
+
- **Release workflow:** releases are published from GitHub Actions through npm
|
|
55
|
+
trusted publishing, with provenance and no stored npm token.
|
|
56
|
+
- **CI:** every push and pull request runs the type check, the tests, and a pack check
|
|
57
|
+
on Node.js 22 and 24.
|
|
58
|
+
|
|
59
|
+
## [0.1.1] - 2026-11-18
|
|
60
|
+
|
|
61
|
+
### Fixed
|
|
62
|
+
|
|
63
|
+
- **Package links:** the repository, homepage, and issue links now point to
|
|
64
|
+
github.com/mlgxiii/agentests-security.
|
|
65
|
+
|
|
66
|
+
## [0.1.0] - 2026-11-18
|
|
67
|
+
|
|
68
|
+
### Added
|
|
69
|
+
|
|
70
|
+
- **First release:** the `run`, `validate`, and `mock` commands; the
|
|
71
|
+
`sustained-query-volume`, `systematic-query-pattern`, and `quota-exhaustion`
|
|
72
|
+
scenarios; the rate limit, quota, and detection expectations; and the safety
|
|
73
|
+
rules (an allowed-host list, a double opt-in for production, and a request budget).
|
|
74
|
+
|
|
75
|
+
[1.0.0]: https://www.npmjs.com/package/@nyancorn/agentests-security/v/1.0.0
|
|
76
|
+
[0.1.1]: https://www.npmjs.com/package/@nyancorn/agentests-security/v/0.1.1
|
|
77
|
+
[0.1.0]: https://www.npmjs.com/package/@nyancorn/agentests-security/v/0.1.0
|
package/README.md
CHANGED
|
@@ -1,5 +1,10 @@
|
|
|
1
1
|
# agentests-security
|
|
2
2
|
|
|
3
|
+
[](https://www.npmjs.com/package/@nyancorn/agentests-security)
|
|
4
|
+
[](https://github.com/mlgxiii/agentests-security/actions/workflows/ci.yml)
|
|
5
|
+
[](SECURITY.md#supply-chain)
|
|
6
|
+
[](LICENSE)
|
|
7
|
+
|
|
3
8
|
Test your AI API's defenses against model extraction in CI. A security policy sends
|
|
4
9
|
bounded, deterministic attack traffic to a local or explicitly authorized endpoint,
|
|
5
10
|
then checks that the rate limits, quotas, and detection events still fire. If one of
|
|
@@ -32,9 +37,10 @@ policy again to see that defense's test fail.
|
|
|
32
37
|
|
|
33
38
|
## Point it at your API
|
|
34
39
|
|
|
35
|
-
Write `agentests.security.yaml
|
|
40
|
+
Write `agentests.security.yaml`. The first line turns on autocompletion and validation in editors that use the YAML language server, such as VS Code with the YAML extension:
|
|
36
41
|
|
|
37
42
|
```yaml
|
|
43
|
+
# yaml-language-server: $schema=https://unpkg.com/@nyancorn/agentests-security@1/schema/policy.schema.json
|
|
38
44
|
version: 1
|
|
39
45
|
name: model-extraction-protection
|
|
40
46
|
target:
|
|
@@ -62,6 +68,15 @@ npx @nyancorn/agentests-security run [id] # exit 0 pass, 1 regression, 2 e
|
|
|
62
68
|
npx @nyancorn/agentests-security run --json # full report with evidence, for CI artifacts
|
|
63
69
|
```
|
|
64
70
|
|
|
71
|
+
### Stricter checks (1.1)
|
|
72
|
+
|
|
73
|
+
| Field | What it does |
|
|
74
|
+
| --- | --- |
|
|
75
|
+
| `expect.rate_limit_within: 25` | The rate limit must fire by request 25, not just eventually. Needs `rate_limit_enforced: true`. |
|
|
76
|
+
| `expect.quota_within: 60` | The same for the quota. Needs `quota_enforced: true`. |
|
|
77
|
+
| `max_error_rate: 0.05` (per test, default) | If more than 5% of responses are errors that match no signal, such as 500s, the test fails as **inconclusive**. A broken API no longer passes `rate_limit_enforced: false`. |
|
|
78
|
+
| `target.detection.run_field: run` | Count only detection events whose `run` field equals this run's `x-agentests-run` id. Without it, any new event counts, even one caused by other traffic. Detection headers can also use `{{run_id}}`, for APIs that filter on the server side. |
|
|
79
|
+
|
|
65
80
|
The [policy reference](https://github.com/mlgxiii/agentests/tree/main/examples/security#the-policy)
|
|
66
81
|
covers every field: the three scenarios, the response signals, the detection
|
|
67
82
|
endpoint, and the safety limits.
|
|
@@ -76,11 +91,29 @@ endpoint, and the safety limits.
|
|
|
76
91
|
as a hard stop during the run.
|
|
77
92
|
- **Credentials:** URLs that contain credentials are rejected. Put secrets in headers as
|
|
78
93
|
`${VAR}` references instead.
|
|
94
|
+
- **TLS:** plain `http://` to a host other than localhost prints a warning, because it sends
|
|
95
|
+
your API key unencrypted. From 2.0 it will be an error. Set `safety.allow_insecure_http: true`
|
|
96
|
+
to accept it, for example on a private network.
|
|
97
|
+
- **Redaction:** the values of `${VAR}` secrets, literal values of headers such as
|
|
98
|
+
`authorization` or `x-api-key`, and URL parameters such as `api_key` or `token` are replaced
|
|
99
|
+
with `[REDACTED]` in the report and in error messages.
|
|
100
|
+
- **Bounded responses:** at most 64 KiB of each response is read, and the rest of a streaming
|
|
101
|
+
answer is cancelled. A detection endpoint returning more than 4 MiB is an error.
|
|
79
102
|
- **Traceability:** every request carries `x-agentests-run` and `x-agentests-test` headers,
|
|
80
103
|
so your logs can tell test traffic apart.
|
|
81
104
|
|
|
82
105
|
Only test APIs you own or are authorized to test.
|
|
83
106
|
|
|
107
|
+
## Stability and support
|
|
108
|
+
|
|
109
|
+
From 1.0, the policy format (`version: 1`), the CLI and its exit codes, the `--json`
|
|
110
|
+
report, and the library exports are covered by a semantic versioning promise; see
|
|
111
|
+
[STABILITY.md](STABILITY.md). Contract tests enforce it, and the [changelog](CHANGELOG.md)
|
|
112
|
+
announces every change.
|
|
113
|
+
|
|
114
|
+
Releases are published from GitHub Actions with npm provenance. Report vulnerabilities
|
|
115
|
+
privately as described in [SECURITY.md](SECURITY.md).
|
|
116
|
+
|
|
84
117
|
## Library
|
|
85
118
|
|
|
86
119
|
```ts
|
package/SECURITY.md
ADDED
|
@@ -0,0 +1,49 @@
|
|
|
1
|
+
# Security policy
|
|
2
|
+
|
|
3
|
+
## Reporting a vulnerability
|
|
4
|
+
|
|
5
|
+
Report vulnerabilities privately through GitHub:
|
|
6
|
+
**[Report a vulnerability](https://github.com/mlgxiii/agentests-security/security/advisories/new)**.
|
|
7
|
+
Please don't open a public issue.
|
|
8
|
+
|
|
9
|
+
Include the version, a description, and the steps or a policy file that reproduce it.
|
|
10
|
+
You can expect an acknowledgement within 3 business days and a fix or mitigation
|
|
11
|
+
plan within 14 days. Fixes are announced in a GitHub security advisory and the
|
|
12
|
+
[changelog](CHANGELOG.md), with credit if you want it.
|
|
13
|
+
|
|
14
|
+
## Supported versions
|
|
15
|
+
|
|
16
|
+
| Version | Supported |
|
|
17
|
+
| --- | --- |
|
|
18
|
+
| 1.x (latest minor) | Yes |
|
|
19
|
+
| 0.x | No. Upgrade to 1.x. |
|
|
20
|
+
|
|
21
|
+
## In scope
|
|
22
|
+
|
|
23
|
+
Ways to make the runner do something its safety rules should prevent:
|
|
24
|
+
|
|
25
|
+
- sending a request to a host that the policy doesn't authorize
|
|
26
|
+
- running against a production target without both opt-ins
|
|
27
|
+
- sending more requests than `safety.max_requests`
|
|
28
|
+
- leaking a `${VAR}` value into output, the report, or a host it wasn't meant for
|
|
29
|
+
- a malicious policy file executing code or reading files beyond the policy itself
|
|
30
|
+
|
|
31
|
+
## Authorized use
|
|
32
|
+
|
|
33
|
+
This tool generates traffic that resembles a model extraction attack. Only run it
|
|
34
|
+
against APIs you own or have written permission to test. The safety rules guard
|
|
35
|
+
against mistakes; they don't make unauthorized testing acceptable.
|
|
36
|
+
|
|
37
|
+
## Supply chain
|
|
38
|
+
|
|
39
|
+
Every release from 1.0.0 on is published from [GitHub Actions](.github/workflows/release.yml)
|
|
40
|
+
through npm trusted publishing, with [npm provenance](https://docs.npmjs.com/generating-provenance-statements)
|
|
41
|
+
and no long-lived npm token. (The 0.x releases were published by hand.)
|
|
42
|
+
You can check that a release was built from this repository:
|
|
43
|
+
|
|
44
|
+
```sh
|
|
45
|
+
npm audit signatures
|
|
46
|
+
```
|
|
47
|
+
|
|
48
|
+
The package has three runtime dependencies (`zod`, `yaml`, and `clinguini`) and no
|
|
49
|
+
install scripts.
|
package/STABILITY.md
ADDED
|
@@ -0,0 +1,33 @@
|
|
|
1
|
+
# Stability
|
|
2
|
+
|
|
3
|
+
`@nyancorn/agentests-security` follows [semantic versioning](https://semver.org).
|
|
4
|
+
From 1.0.0, everything below is a public contract. It changes in a breaking way only
|
|
5
|
+
in a new major version, announced in the [changelog](CHANGELOG.md) at least one minor
|
|
6
|
+
release ahead, with a deprecation warning wherever one is possible.
|
|
7
|
+
|
|
8
|
+
## Covered by the 1.x promise
|
|
9
|
+
|
|
10
|
+
| Surface | Promise |
|
|
11
|
+
| --- | --- |
|
|
12
|
+
| Policy file, `version: 1` | Every policy valid on 1.0 stays valid and keeps its meaning on every 1.x release. New optional fields may be added. A breaking format change would introduce `version: 2`, and 1.x would keep reading `version: 1`. |
|
|
13
|
+
| [`schema/policy.schema.json`](schema/policy.schema.json) | Matches the format above, and is published at `https://unpkg.com/@nyancorn/agentests-security@1/schema/policy.schema.json`. |
|
|
14
|
+
| CLI | The commands `run`, `validate`, and `mock`, their flags, and the exit codes: 0 means every test passed, 1 means a regression, 2 means an error (an invalid policy, or an unreachable or unauthorized target). |
|
|
15
|
+
| `run --json` report | Existing fields keep their names, types, and meanings. New fields may be added. |
|
|
16
|
+
| Request headers | `x-agentests-run` and `x-agentests-test` are sent on every request. |
|
|
17
|
+
| Safety rules | Defaults never loosen in 1.x: only local hosts unless listed, production only with both opt-ins, and a hard request budget. |
|
|
18
|
+
| Library exports | The names exported from the package root, and their documented signatures. |
|
|
19
|
+
|
|
20
|
+
## Not covered
|
|
21
|
+
|
|
22
|
+
- **Human-readable output:** the wording of `run`, `validate`, and error messages. Parse `--json`, not text.
|
|
23
|
+
- **Prompt wording:** the exact prompts each scenario sends. They stay deterministic within a release, but may change in a minor release to better resemble real extraction traffic. A minor release may also tighten a scenario's traffic shape, so check the changelog when an expectation's result changes after an upgrade.
|
|
24
|
+
- **The mock API's internals:** its default limits, response bodies, and event format, beyond what the bundled example policy needs to pass.
|
|
25
|
+
- **Undocumented modules:** anything imported from a path inside `dist/`.
|
|
26
|
+
|
|
27
|
+
## Supported versions
|
|
28
|
+
|
|
29
|
+
The latest 1.x minor release gets fixes. When 2.0 ships, the last 1.x release gets
|
|
30
|
+
security fixes for 6 months.
|
|
31
|
+
|
|
32
|
+
These promises are enforced by `tests/contract.test.ts`: a frozen version 1 policy,
|
|
33
|
+
the shipped schema, the export list, and the report fields.
|
package/dist/cli.js
CHANGED
|
@@ -4,8 +4,10 @@ import { clinguini, command } from 'clinguini';
|
|
|
4
4
|
import { loadSecurityPolicy, SECURITY_PATH } from "./policy/load.js";
|
|
5
5
|
import { PROTECTED_DEFENSES, startMockApi } from "./mock.js";
|
|
6
6
|
import { ProbeOrchestrator } from "./probe/index.js";
|
|
7
|
+
import { redactUrl } from "./probe/redact.js";
|
|
8
|
+
import { securityPolicyWarnings } from "./policy/validate.js";
|
|
7
9
|
/** The text of a thrown value. */
|
|
8
|
-
const message = (error) =>
|
|
10
|
+
const message = (error) => error instanceof Error ? error.message : String(error);
|
|
9
11
|
const policyFlag = {
|
|
10
12
|
name: 'policy',
|
|
11
13
|
type: 'string',
|
|
@@ -13,8 +15,16 @@ const policyFlag = {
|
|
|
13
15
|
};
|
|
14
16
|
/** The policy path a flag value names, else the default. */
|
|
15
17
|
const toPath = (value) => (typeof value === 'string' ? value : SECURITY_PATH);
|
|
16
|
-
/** `PASS|FAIL <test> <expectation>: expected <x>, got <y> (<detail>)`, one line per check. */
|
|
17
|
-
const toLines = ({ test, checks }) =>
|
|
18
|
+
/** `PASS|FAIL <test> <expectation>: expected <x>, got <y> (<detail>)`, one line per check, then a FAIL line if inconclusive. */
|
|
19
|
+
const toLines = ({ test, checks, inconclusive }) => [
|
|
20
|
+
...checks.map((check) => `${check.passed ? 'PASS' : 'FAIL'} ${test.id} ${check.expectation}: expected ${check.expected}, got ${check.actual} (${check.detail})`),
|
|
21
|
+
...(inconclusive === undefined ? [] : [`FAIL ${test.id} inconclusive: ${inconclusive}`])
|
|
22
|
+
];
|
|
23
|
+
/** Print each policy warning to stderr, so `--json` output stays parseable. */
|
|
24
|
+
const warn = (policy) => {
|
|
25
|
+
for (const warning of securityPolicyWarnings(policy))
|
|
26
|
+
console.error(`Warning: ${warning}`);
|
|
27
|
+
};
|
|
18
28
|
const runSubcommand = command({
|
|
19
29
|
name: 'run',
|
|
20
30
|
description: 'Run the security policy against its target.',
|
|
@@ -34,8 +44,9 @@ const runSubcommand = command({
|
|
|
34
44
|
allowProduction: flags['allow-production'] === true
|
|
35
45
|
});
|
|
36
46
|
const asJson = flags.json === true;
|
|
47
|
+
warn(policy);
|
|
37
48
|
if (!asJson)
|
|
38
|
-
console.log(`Security evaluation: ${policy.name} → ${policy.target.url} (${policy.target.environment})`);
|
|
49
|
+
console.log(`Security evaluation: ${policy.name} → ${redactUrl(policy.target.url)} (${policy.target.environment})`);
|
|
39
50
|
const report = await new ProbeOrchestrator(policy, process.env, {
|
|
40
51
|
onResult: (result) => {
|
|
41
52
|
if (!asJson)
|
|
@@ -72,6 +83,7 @@ const validateSubcommand = command({
|
|
|
72
83
|
allowProduction: flags['allow-production'] === true
|
|
73
84
|
});
|
|
74
85
|
const total = policy.tests.reduce((sum, test) => sum + test.requests, 0);
|
|
86
|
+
warn(policy);
|
|
75
87
|
console.log(`Policy ${policy.name} valid: ${policy.tests.length} security test(s), ${total} of ${policy.safety.max_requests} request(s).`);
|
|
76
88
|
}
|
|
77
89
|
catch (error) {
|
package/dist/index.d.ts
CHANGED
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
export { LOCAL_HOSTS, validateSecurityPolicy } from './policy/validate.ts';
|
|
1
|
+
export { LOCAL_HOSTS, securityPolicyWarnings, validateSecurityPolicy } from './policy/validate.ts';
|
|
2
2
|
export type { AgentTestSecurityOptions } from './policy/validate.ts';
|
|
3
3
|
export { parseSecurityPolicy } from './policy/parse.ts';
|
|
4
4
|
export { loadSecurityPolicy, SECURITY_PATH } from './policy/load.ts';
|
package/dist/index.js
CHANGED
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
export { LOCAL_HOSTS, validateSecurityPolicy } from "./policy/validate.js";
|
|
1
|
+
export { LOCAL_HOSTS, securityPolicyWarnings, validateSecurityPolicy } from "./policy/validate.js";
|
|
2
2
|
export { parseSecurityPolicy } from "./policy/parse.js";
|
|
3
3
|
export { loadSecurityPolicy, SECURITY_PATH } from "./policy/load.js";
|
|
4
4
|
export { AgentTestSecurityPolicySchema } from "./policy/types.js";
|
package/dist/policy/types.d.ts
CHANGED
|
@@ -28,6 +28,7 @@ export declare const AgentTestSecurityPolicySchema: z.ZodObject<{
|
|
|
28
28
|
url: z.ZodURL;
|
|
29
29
|
headers: z.ZodDefault<z.ZodRecord<z.ZodString, z.ZodString>>;
|
|
30
30
|
wait_ms: z.ZodDefault<z.ZodNumber>;
|
|
31
|
+
run_field: z.ZodOptional<z.ZodString>;
|
|
31
32
|
}, z.core.$strict>>;
|
|
32
33
|
}, z.core.$strict>;
|
|
33
34
|
tests: z.ZodArray<z.ZodObject<{
|
|
@@ -41,16 +42,20 @@ export declare const AgentTestSecurityPolicySchema: z.ZodObject<{
|
|
|
41
42
|
requests: z.ZodNumber;
|
|
42
43
|
duration_seconds: z.ZodDefault<z.ZodNumber>;
|
|
43
44
|
headers: z.ZodDefault<z.ZodRecord<z.ZodString, z.ZodString>>;
|
|
45
|
+
max_error_rate: z.ZodDefault<z.ZodNumber>;
|
|
44
46
|
expect: z.ZodObject<{
|
|
45
47
|
rate_limit_enforced: z.ZodOptional<z.ZodBoolean>;
|
|
46
48
|
quota_enforced: z.ZodOptional<z.ZodBoolean>;
|
|
47
49
|
detection_event: z.ZodOptional<z.ZodBoolean>;
|
|
50
|
+
rate_limit_within: z.ZodOptional<z.ZodNumber>;
|
|
51
|
+
quota_within: z.ZodOptional<z.ZodNumber>;
|
|
48
52
|
}, z.core.$strict>;
|
|
49
53
|
}, z.core.$strict>>;
|
|
50
54
|
safety: z.ZodObject<{
|
|
51
55
|
max_requests: z.ZodNumber;
|
|
52
56
|
allow_production: z.ZodDefault<z.ZodBoolean>;
|
|
53
57
|
allowed_hosts: z.ZodDefault<z.ZodArray<z.ZodString>>;
|
|
58
|
+
allow_insecure_http: z.ZodDefault<z.ZodBoolean>;
|
|
54
59
|
}, z.core.$strict>;
|
|
55
60
|
}, z.core.$strict>;
|
|
56
61
|
export type AgentTestSecurityPolicy = z.infer<typeof AgentTestSecurityPolicySchema>;
|
|
@@ -8,3 +8,8 @@ export interface AgentTestSecurityOptions {
|
|
|
8
8
|
export declare const LOCAL_HOSTS: string[];
|
|
9
9
|
/** Validate a parsed value as a security policy, or throw `path: message` lines: every schema problem; else, once the schema passes, every rule and safety problem. */
|
|
10
10
|
export declare const validateSecurityPolicy: (value: unknown, options?: AgentTestSecurityOptions) => AgentTestSecurityPolicy;
|
|
11
|
+
/**
|
|
12
|
+
* Problems that are not errors in 1.x but will be in 2.0: a plain `http://` URL to a host outside `LOCAL_HOSTS`,
|
|
13
|
+
* which sends the API key unencrypted, unless `safety.allow_insecure_http` accepts it.
|
|
14
|
+
*/
|
|
15
|
+
export declare const securityPolicyWarnings: (policy: AgentTestSecurityPolicy) => string[];
|
package/dist/policy/validate.js
CHANGED
|
@@ -47,6 +47,21 @@ const policyProblems = (policy, { allowProduction = false }) => {
|
|
|
47
47
|
...tests.flatMap(({ expect }, index) => expect.detection_event !== undefined && !target.detection
|
|
48
48
|
? [`tests.${index}.expect.detection_event: Requires target.detection`]
|
|
49
49
|
: []),
|
|
50
|
+
...tests.flatMap(({ expect, requests }, index) => [
|
|
51
|
+
['rate_limit_within', 'rate_limit_enforced'],
|
|
52
|
+
['quota_within', 'quota_enforced']
|
|
53
|
+
].flatMap(([within, enforced]) => {
|
|
54
|
+
const limit = expect[within];
|
|
55
|
+
if (limit === undefined)
|
|
56
|
+
return [];
|
|
57
|
+
if (expect[enforced] !== true)
|
|
58
|
+
return [`tests.${index}.expect.${within}: Requires ${enforced}: true`];
|
|
59
|
+
return limit > requests
|
|
60
|
+
? [
|
|
61
|
+
`tests.${index}.expect.${within}: ${limit} is more than the Test's ${requests} requests`
|
|
62
|
+
]
|
|
63
|
+
: [];
|
|
64
|
+
})),
|
|
50
65
|
...urls.flatMap(([path, value]) => {
|
|
51
66
|
const url = new URL(value);
|
|
52
67
|
if (url.username || url.password)
|
|
@@ -63,6 +78,29 @@ const policyProblems = (policy, { allowProduction = false }) => {
|
|
|
63
78
|
: [])
|
|
64
79
|
];
|
|
65
80
|
};
|
|
81
|
+
/**
|
|
82
|
+
* Problems that are not errors in 1.x but will be in 2.0: a plain `http://` URL to a host outside `LOCAL_HOSTS`,
|
|
83
|
+
* which sends the API key unencrypted, unless `safety.allow_insecure_http` accepts it.
|
|
84
|
+
*/
|
|
85
|
+
export const securityPolicyWarnings = (policy) => {
|
|
86
|
+
const { target, safety } = policy;
|
|
87
|
+
if (safety.allow_insecure_http)
|
|
88
|
+
return [];
|
|
89
|
+
const urls = [
|
|
90
|
+
['target.url', target.url],
|
|
91
|
+
...(target.detection
|
|
92
|
+
? [['target.detection.url', target.detection.url]]
|
|
93
|
+
: [])
|
|
94
|
+
];
|
|
95
|
+
return urls.flatMap(([path, value]) => {
|
|
96
|
+
const url = new URL(value);
|
|
97
|
+
return url.protocol === 'http:' && !LOCAL_HOSTS.includes(url.hostname.toLowerCase())
|
|
98
|
+
? [
|
|
99
|
+
`${path}: Plain HTTP to ${url.hostname} sends credentials unencrypted; use https, or set safety.allow_insecure_http: true. This becomes an error in 2.0.`
|
|
100
|
+
]
|
|
101
|
+
: [];
|
|
102
|
+
});
|
|
103
|
+
};
|
|
66
104
|
const compiles = (pattern) => {
|
|
67
105
|
try {
|
|
68
106
|
new RegExp(pattern, 'i');
|
package/dist/probe/http.d.ts
CHANGED
|
@@ -11,7 +11,19 @@ export interface ProbeResponse {
|
|
|
11
11
|
status: number;
|
|
12
12
|
body: string;
|
|
13
13
|
}
|
|
14
|
+
/** The most of a completion response the runner reads; the rest of a long or endless stream is cancelled. */
|
|
15
|
+
export declare const MAX_BODY: number;
|
|
16
|
+
/** The most of a detection events response the runner reads; a larger list should be filtered by the API. */
|
|
17
|
+
export declare const MAX_EVENTS_BODY: number;
|
|
18
|
+
/**
|
|
19
|
+
* Read at most `limit` bytes of a body and cancel the rest, so a streaming or oversized response cannot hold
|
|
20
|
+
* the run or its memory. Returns the text and whether it was cut.
|
|
21
|
+
*/
|
|
22
|
+
export declare const readCapped: (response: Response, limit: number) => Promise<{
|
|
23
|
+
text: string;
|
|
24
|
+
truncated: boolean;
|
|
25
|
+
}>;
|
|
14
26
|
/** POST one JSON request; a network failure or timeout throws, so an unreachable target is an error, never a pass or a fail. */
|
|
15
27
|
export declare const send: (fetcher: typeof fetch, url: string, headers: Record<string, string>, body: unknown, timeoutMs: number) => Promise<ProbeResponse>;
|
|
16
|
-
/** The
|
|
17
|
-
export declare const
|
|
28
|
+
/** The detection events the endpoint reports: a JSON array, or an object's `events` array. */
|
|
29
|
+
export declare const readEvents: (fetcher: typeof fetch, url: string, headers: Record<string, string>, timeoutMs: number) => Promise<unknown[]>;
|
package/dist/probe/http.js
CHANGED
|
@@ -20,7 +20,50 @@ export const expandValue = (value, env, placeholders) => {
|
|
|
20
20
|
};
|
|
21
21
|
/** `expand` applied to each header value. */
|
|
22
22
|
export const expandHeaders = (headers, env, placeholders = {}) => Object.fromEntries(Object.entries(headers).map(([name, value]) => [name, expand(value, env, placeholders)]));
|
|
23
|
-
|
|
23
|
+
/** The most of a completion response the runner reads; the rest of a long or endless stream is cancelled. */
|
|
24
|
+
export const MAX_BODY = 64 * 1024;
|
|
25
|
+
/** The most of a detection events response the runner reads; a larger list should be filtered by the API. */
|
|
26
|
+
export const MAX_EVENTS_BODY = 4 * 1024 * 1024;
|
|
27
|
+
/**
|
|
28
|
+
* Read at most `limit` bytes of a body and cancel the rest, so a streaming or oversized response cannot hold
|
|
29
|
+
* the run or its memory. Returns the text and whether it was cut.
|
|
30
|
+
*/
|
|
31
|
+
export const readCapped = async (response, limit) => {
|
|
32
|
+
if (!response.body)
|
|
33
|
+
return { text: '', truncated: false };
|
|
34
|
+
const reader = response.body.getReader();
|
|
35
|
+
const chunks = [];
|
|
36
|
+
let size = 0;
|
|
37
|
+
let truncated = false;
|
|
38
|
+
try {
|
|
39
|
+
while (true) {
|
|
40
|
+
const { done, value } = await reader.read();
|
|
41
|
+
if (done)
|
|
42
|
+
break;
|
|
43
|
+
const room = limit - size;
|
|
44
|
+
if (value.byteLength >= room) {
|
|
45
|
+
chunks.push(value.subarray(0, room));
|
|
46
|
+
size += room;
|
|
47
|
+
truncated = value.byteLength > room || !(await reader.read()).done;
|
|
48
|
+
break;
|
|
49
|
+
}
|
|
50
|
+
chunks.push(value);
|
|
51
|
+
size += value.byteLength;
|
|
52
|
+
}
|
|
53
|
+
}
|
|
54
|
+
finally {
|
|
55
|
+
if (truncated)
|
|
56
|
+
await reader.cancel().catch(() => undefined);
|
|
57
|
+
reader.releaseLock();
|
|
58
|
+
}
|
|
59
|
+
const bytes = new Uint8Array(size);
|
|
60
|
+
let offset = 0;
|
|
61
|
+
for (const chunk of chunks) {
|
|
62
|
+
bytes.set(chunk, offset);
|
|
63
|
+
offset += chunk.byteLength;
|
|
64
|
+
}
|
|
65
|
+
return { text: new TextDecoder().decode(bytes), truncated };
|
|
66
|
+
};
|
|
24
67
|
/** POST one JSON request; a network failure or timeout throws, so an unreachable target is an error, never a pass or a fail. */
|
|
25
68
|
export const send = async (fetcher, url, headers, body, timeoutMs) => {
|
|
26
69
|
try {
|
|
@@ -31,38 +74,46 @@ export const send = async (fetcher, url, headers, body, timeoutMs) => {
|
|
|
31
74
|
signal: AbortSignal.timeout(timeoutMs),
|
|
32
75
|
redirect: 'manual'
|
|
33
76
|
});
|
|
34
|
-
const text = await response
|
|
35
|
-
return { status: response.status, body: text
|
|
77
|
+
const { text } = await readCapped(response, MAX_BODY);
|
|
78
|
+
return { status: response.status, body: text };
|
|
36
79
|
}
|
|
37
80
|
catch (error) {
|
|
38
81
|
throw new Error(`${url} unreachable: ${message(error)}`, { cause: error });
|
|
39
82
|
}
|
|
40
83
|
};
|
|
41
|
-
/** The
|
|
42
|
-
export const
|
|
43
|
-
let
|
|
84
|
+
/** The detection events the endpoint reports: a JSON array, or an object's `events` array. */
|
|
85
|
+
export const readEvents = async (fetcher, url, headers, timeoutMs) => {
|
|
86
|
+
let text;
|
|
44
87
|
try {
|
|
45
|
-
response = await fetcher(url, {
|
|
88
|
+
const response = await fetcher(url, {
|
|
46
89
|
headers: { accept: 'application/json', ...headers },
|
|
47
90
|
signal: AbortSignal.timeout(timeoutMs),
|
|
48
91
|
redirect: 'manual'
|
|
49
92
|
});
|
|
93
|
+
if (!response.ok) {
|
|
94
|
+
await response.body?.cancel().catch(() => undefined);
|
|
95
|
+
throw new Error(`${url} returned ${response.status}`);
|
|
96
|
+
}
|
|
97
|
+
const read = await readCapped(response, MAX_EVENTS_BODY);
|
|
98
|
+
if (read.truncated)
|
|
99
|
+
throw new Error(`${url} returned more than ${MAX_EVENTS_BODY / 1024 / 1024} MiB; filter events by run or time on the API side`);
|
|
100
|
+
text = read.text;
|
|
50
101
|
}
|
|
51
102
|
catch (error) {
|
|
103
|
+
if (error instanceof Error && error.message.startsWith(url))
|
|
104
|
+
throw error;
|
|
52
105
|
throw new Error(`${url} unreachable: ${message(error)}`, { cause: error });
|
|
53
106
|
}
|
|
54
|
-
if (!response.ok)
|
|
55
|
-
throw new Error(`${url} returned ${response.status}`);
|
|
56
107
|
let value;
|
|
57
108
|
try {
|
|
58
|
-
value =
|
|
109
|
+
value = JSON.parse(text);
|
|
59
110
|
}
|
|
60
111
|
catch {
|
|
61
112
|
throw new Error(`${url} did not return JSON`);
|
|
62
113
|
}
|
|
63
114
|
if (Array.isArray(value))
|
|
64
|
-
return value
|
|
115
|
+
return value;
|
|
65
116
|
if (value && typeof value === 'object' && 'events' in value && Array.isArray(value.events))
|
|
66
|
-
return value.events
|
|
117
|
+
return value.events;
|
|
67
118
|
throw new Error(`${url} returned neither an array nor an object with an events array`);
|
|
68
119
|
};
|
package/dist/probe/index.d.ts
CHANGED
|
@@ -19,6 +19,9 @@ export declare const TEST_HEADER = "x-agentests-test";
|
|
|
19
19
|
export declare class ProbeOrchestrator {
|
|
20
20
|
#private;
|
|
21
21
|
constructor(policy: AgentTestSecurityPolicy, env: Record<string, string | undefined>, options?: ProbeOptions);
|
|
22
|
-
/**
|
|
22
|
+
/**
|
|
23
|
+
* Run every Test in policy order, or only the one with `id`; throws on an unknown id or an unreachable target.
|
|
24
|
+
* Secrets the policy sends never appear in the report or in a thrown message.
|
|
25
|
+
*/
|
|
23
26
|
readonly run: (id?: string) => Promise<AgentTestSecurityReport>;
|
|
24
27
|
}
|
package/dist/probe/index.js
CHANGED
|
@@ -1,8 +1,10 @@
|
|
|
1
1
|
import { randomUUID } from 'node:crypto';
|
|
2
|
-
import {
|
|
2
|
+
import { expandHeaders, expandValue, readEvents, send } from "./http.js";
|
|
3
3
|
import { scenarioPrompt } from "./scenarios.js";
|
|
4
|
-
import {
|
|
5
|
-
import {
|
|
4
|
+
import { policySecrets, redactTest, redactText, redactUrl } from "./redact.js";
|
|
5
|
+
import { message, sleep } from "./utils.js";
|
|
6
|
+
import { securityPolicyWarnings } from "../policy/validate.js";
|
|
7
|
+
import { inconclusive, judge } from "./verdict.js";
|
|
6
8
|
/** Headers sent on every request, so the target's logs can tell test traffic apart. */
|
|
7
9
|
export const RUN_HEADER = 'x-agentests-run';
|
|
8
10
|
export const TEST_HEADER = 'x-agentests-test';
|
|
@@ -32,8 +34,20 @@ export class ProbeOrchestrator {
|
|
|
32
34
|
this.#runId = options.runId ?? randomUUID();
|
|
33
35
|
this.#onResult = options.onResult;
|
|
34
36
|
}
|
|
35
|
-
/**
|
|
37
|
+
/**
|
|
38
|
+
* Run every Test in policy order, or only the one with `id`; throws on an unknown id or an unreachable target.
|
|
39
|
+
* Secrets the policy sends never appear in the report or in a thrown message.
|
|
40
|
+
*/
|
|
36
41
|
run = async (id) => {
|
|
42
|
+
const secrets = policySecrets(this.#policy, this.#env);
|
|
43
|
+
try {
|
|
44
|
+
return await this.#run(id);
|
|
45
|
+
}
|
|
46
|
+
catch (error) {
|
|
47
|
+
throw new Error(redactText(message(error), secrets));
|
|
48
|
+
}
|
|
49
|
+
};
|
|
50
|
+
#run = async (id) => {
|
|
37
51
|
const { name, target, tests } = this.#policy;
|
|
38
52
|
const selected = id === undefined ? tests : tests.filter((test) => test.id === id);
|
|
39
53
|
if (!selected.length)
|
|
@@ -43,27 +57,48 @@ export class ProbeOrchestrator {
|
|
|
43
57
|
for (const test of selected) {
|
|
44
58
|
const evidence = await this.#probe(test);
|
|
45
59
|
const checks = judge(test.expect, evidence);
|
|
46
|
-
const
|
|
60
|
+
const reason = inconclusive(test, evidence);
|
|
61
|
+
const result = {
|
|
62
|
+
test: redactTest(test),
|
|
63
|
+
passed: reason === undefined && checks.every((check) => check.passed),
|
|
64
|
+
...(reason === undefined ? {} : { inconclusive: reason }),
|
|
65
|
+
checks,
|
|
66
|
+
evidence
|
|
67
|
+
};
|
|
47
68
|
results.push(result);
|
|
48
69
|
this.#onResult?.(result);
|
|
49
70
|
}
|
|
50
71
|
return {
|
|
51
72
|
policy: name,
|
|
52
73
|
runId: this.#runId,
|
|
53
|
-
target: { url: target.url, environment: target.environment },
|
|
74
|
+
target: { url: redactUrl(target.url), environment: target.environment },
|
|
54
75
|
passed: results.filter((result) => result.passed).length,
|
|
55
76
|
total: results.length,
|
|
56
|
-
results
|
|
77
|
+
results,
|
|
78
|
+
warnings: securityPolicyWarnings(this.#policy)
|
|
57
79
|
};
|
|
58
80
|
};
|
|
81
|
+
/** This run's share of the detection events: all of them, or with `run_field` only those carrying this run's id. */
|
|
82
|
+
#countEvents = async (headers) => {
|
|
83
|
+
const { detection, timeout_ms } = this.#policy.target;
|
|
84
|
+
if (!detection)
|
|
85
|
+
return 0;
|
|
86
|
+
const events = await readEvents(this.#fetch, detection.url, headers, timeout_ms);
|
|
87
|
+
const field = detection.run_field;
|
|
88
|
+
if (field === undefined)
|
|
89
|
+
return events.length;
|
|
90
|
+
return events.filter((event) => !!event &&
|
|
91
|
+
typeof event === 'object' &&
|
|
92
|
+
String(event[field]) === this.#runId).length;
|
|
93
|
+
};
|
|
59
94
|
/** Send one Test's scenario and collect its evidence. */
|
|
60
95
|
#probe = async (test) => {
|
|
61
96
|
const { target } = this.#policy;
|
|
62
97
|
const detection = test.expect.detection_event !== undefined ? target.detection : undefined;
|
|
63
|
-
const detectionHeaders = detection
|
|
64
|
-
|
|
65
|
-
|
|
66
|
-
|
|
98
|
+
const detectionHeaders = detection
|
|
99
|
+
? expandHeaders(detection.headers, this.#env, { run_id: this.#runId })
|
|
100
|
+
: {};
|
|
101
|
+
const before = detection ? await this.#countEvents(detectionHeaders) : 0;
|
|
67
102
|
const interval = (test.duration_seconds * 1000) / test.requests;
|
|
68
103
|
const evidence = {
|
|
69
104
|
sent: 0,
|
|
@@ -72,6 +107,7 @@ export class ProbeOrchestrator {
|
|
|
72
107
|
rateLimited: 0,
|
|
73
108
|
quotaRejected: 0,
|
|
74
109
|
succeededAfterQuota: 0,
|
|
110
|
+
errors: 0,
|
|
75
111
|
elapsedMs: 0
|
|
76
112
|
};
|
|
77
113
|
const start = this.#now();
|
|
@@ -100,12 +136,13 @@ export class ProbeOrchestrator {
|
|
|
100
136
|
}
|
|
101
137
|
if (detection) {
|
|
102
138
|
const deadline = this.#now() + detection.wait_ms;
|
|
103
|
-
let after = await countEvents(
|
|
139
|
+
let after = await this.#countEvents(detectionHeaders);
|
|
104
140
|
while (after <= before && this.#now() < deadline) {
|
|
105
141
|
await this.#sleep(Math.min(250, deadline - this.#now()));
|
|
106
|
-
after = await countEvents(
|
|
142
|
+
after = await this.#countEvents(detectionHeaders);
|
|
107
143
|
}
|
|
108
144
|
evidence.detectionEvents = Math.max(0, after - before);
|
|
145
|
+
evidence.detectionCorrelated = detection.run_field !== undefined;
|
|
109
146
|
}
|
|
110
147
|
evidence.elapsedMs = this.#now() - start;
|
|
111
148
|
return evidence;
|
|
@@ -128,5 +165,7 @@ export class ProbeOrchestrator {
|
|
|
128
165
|
if (evidence.firstQuota !== undefined)
|
|
129
166
|
evidence.succeededAfterQuota++;
|
|
130
167
|
}
|
|
168
|
+
else
|
|
169
|
+
evidence.errors++;
|
|
131
170
|
};
|
|
132
171
|
}
|
|
@@ -0,0 +1,12 @@
|
|
|
1
|
+
import type { AgentTestSecurityPolicy } from '../policy/types.ts';
|
|
2
|
+
import type { AgentTestSecurityTest } from './types.ts';
|
|
3
|
+
/** What every redacted value becomes. */
|
|
4
|
+
export declare const REDACTED = "[REDACTED]";
|
|
5
|
+
/** The environment values a policy sends, plus literal values of its sensitive headers: what must never be printed. */
|
|
6
|
+
export declare const policySecrets: (policy: AgentTestSecurityPolicy, env: Record<string, string | undefined>) => string[];
|
|
7
|
+
/** A URL with the values of its sensitive query parameters redacted; anything unparsable is returned as is. */
|
|
8
|
+
export declare const redactUrl: (value: string) => string;
|
|
9
|
+
/** `text` with every secret, and every sensitive query parameter of every URL in it, redacted. */
|
|
10
|
+
export declare const redactText: (text: string, secrets: string[]) => string;
|
|
11
|
+
/** A Test as the report shows it: sensitive header values redacted unless they are only `${VAR}` references. */
|
|
12
|
+
export declare const redactTest: (test: AgentTestSecurityTest) => AgentTestSecurityTest;
|
|
@@ -0,0 +1,66 @@
|
|
|
1
|
+
/** What every redacted value becomes. */
|
|
2
|
+
export const REDACTED = '[REDACTED]';
|
|
3
|
+
/** Header and query parameter names whose values are treated as secrets. */
|
|
4
|
+
const SENSITIVE = /auth|key|token|secret|password|cookie|session|signature|^sig$/i;
|
|
5
|
+
/** Shorter values are left alone: redacting `1` or `abc` would mangle every message. */
|
|
6
|
+
const MIN_SECRET = 6;
|
|
7
|
+
/** The names of every `${VAR}` a value refers to. */
|
|
8
|
+
const references = (value) => {
|
|
9
|
+
if (typeof value === 'string')
|
|
10
|
+
return [...value.matchAll(/\$\{([A-Za-z_][A-Za-z0-9_]*)\}/g)].map((match) => match[1]);
|
|
11
|
+
if (Array.isArray(value))
|
|
12
|
+
return value.flatMap(references);
|
|
13
|
+
if (value && typeof value === 'object')
|
|
14
|
+
return Object.values(value).flatMap(references);
|
|
15
|
+
return [];
|
|
16
|
+
};
|
|
17
|
+
/** The environment values a policy sends, plus literal values of its sensitive headers: what must never be printed. */
|
|
18
|
+
export const policySecrets = (policy, env) => {
|
|
19
|
+
const { target, tests } = policy;
|
|
20
|
+
const headerSets = [
|
|
21
|
+
target.headers,
|
|
22
|
+
target.detection?.headers ?? {},
|
|
23
|
+
...tests.map((t) => t.headers)
|
|
24
|
+
];
|
|
25
|
+
const names = references([target.body, ...headerSets]);
|
|
26
|
+
const literals = headerSets.flatMap((headers) => Object.entries(headers)
|
|
27
|
+
.filter(([name, value]) => SENSITIVE.test(name) && !references(value).length)
|
|
28
|
+
.map(([, value]) => value.replace(/^(Bearer|Basic|Token)\s+/i, '')));
|
|
29
|
+
const values = [...names.map((name) => env[name] ?? ''), ...literals];
|
|
30
|
+
return [...new Set(values.filter((value) => value.length >= MIN_SECRET))].sort((a, b) => b.length - a.length);
|
|
31
|
+
};
|
|
32
|
+
/** A URL with the values of its sensitive query parameters redacted; anything unparsable is returned as is. */
|
|
33
|
+
export const redactUrl = (value) => {
|
|
34
|
+
let url;
|
|
35
|
+
try {
|
|
36
|
+
url = new URL(value);
|
|
37
|
+
}
|
|
38
|
+
catch {
|
|
39
|
+
return value;
|
|
40
|
+
}
|
|
41
|
+
let changed = false;
|
|
42
|
+
for (const name of [...url.searchParams.keys()])
|
|
43
|
+
if (SENSITIVE.test(name)) {
|
|
44
|
+
url.searchParams.set(name, REDACTED);
|
|
45
|
+
changed = true;
|
|
46
|
+
}
|
|
47
|
+
return changed ? url.toString() : value;
|
|
48
|
+
};
|
|
49
|
+
/** `text` with every secret, and every sensitive query parameter of every URL in it, redacted. */
|
|
50
|
+
export const redactText = (text, secrets) => secrets
|
|
51
|
+
.reduce((result, secret) => result.split(secret).join(REDACTED), text)
|
|
52
|
+
.replace(/https?:\/\/[^\s"'<>]+/g, redactUrl);
|
|
53
|
+
/** A Test as the report shows it: sensitive header values redacted unless they are only `${VAR}` references. */
|
|
54
|
+
export const redactTest = (test) => ({
|
|
55
|
+
...test,
|
|
56
|
+
headers: Object.fromEntries(Object.entries(test.headers).map(([name, value]) => [
|
|
57
|
+
name,
|
|
58
|
+
SENSITIVE.test(name) &&
|
|
59
|
+
value
|
|
60
|
+
.replace(/\$\{[A-Za-z_][A-Za-z0-9_]*\}/g, '')
|
|
61
|
+
.replace(/^(Bearer|Basic|Token)\s*/i, '')
|
|
62
|
+
.trim()
|
|
63
|
+
? REDACTED
|
|
64
|
+
: value
|
|
65
|
+
]))
|
|
66
|
+
});
|
package/dist/probe/types.d.ts
CHANGED
|
@@ -39,6 +39,7 @@ export declare const DetectionSchema: z.ZodObject<{
|
|
|
39
39
|
url: z.ZodURL;
|
|
40
40
|
headers: z.ZodDefault<z.ZodRecord<z.ZodString, z.ZodString>>;
|
|
41
41
|
wait_ms: z.ZodDefault<z.ZodNumber>;
|
|
42
|
+
run_field: z.ZodOptional<z.ZodString>;
|
|
42
43
|
}, z.core.$strict>;
|
|
43
44
|
export type AgentTestDetection = z.infer<typeof DetectionSchema>;
|
|
44
45
|
/** The API under test. Requests are POSTs of `body`, a JSON template whose strings may hold `{{prompt}}`, `{{index}}`, `{{scenario}}`, and `{{run_id}}`. */
|
|
@@ -67,17 +68,20 @@ export declare const TargetSchema: z.ZodObject<{
|
|
|
67
68
|
url: z.ZodURL;
|
|
68
69
|
headers: z.ZodDefault<z.ZodRecord<z.ZodString, z.ZodString>>;
|
|
69
70
|
wait_ms: z.ZodDefault<z.ZodNumber>;
|
|
71
|
+
run_field: z.ZodOptional<z.ZodString>;
|
|
70
72
|
}, z.core.$strict>>;
|
|
71
73
|
}, z.core.$strict>;
|
|
72
74
|
export type AgentTestTarget = z.infer<typeof TargetSchema>;
|
|
73
|
-
/** What a security Test expects of the defenses; at least one
|
|
75
|
+
/** What a security Test expects of the defenses; at least one of the three expectations. */
|
|
74
76
|
export declare const ExpectSchema: z.ZodObject<{
|
|
75
77
|
rate_limit_enforced: z.ZodOptional<z.ZodBoolean>;
|
|
76
78
|
quota_enforced: z.ZodOptional<z.ZodBoolean>;
|
|
77
79
|
detection_event: z.ZodOptional<z.ZodBoolean>;
|
|
80
|
+
rate_limit_within: z.ZodOptional<z.ZodNumber>;
|
|
81
|
+
quota_within: z.ZodOptional<z.ZodNumber>;
|
|
78
82
|
}, z.core.$strict>;
|
|
79
83
|
export type AgentTestExpect = z.infer<typeof ExpectSchema>;
|
|
80
|
-
export type AgentTestExpectation =
|
|
84
|
+
export type AgentTestExpectation = 'rate_limit_enforced' | 'quota_enforced' | 'detection_event';
|
|
81
85
|
/** One bounded attack simulation and its expectations; `duration_seconds` spreads the requests evenly, 0 sends them back to back. */
|
|
82
86
|
export declare const SecurityTestSchema: z.ZodObject<{
|
|
83
87
|
id: z.ZodString;
|
|
@@ -90,10 +94,13 @@ export declare const SecurityTestSchema: z.ZodObject<{
|
|
|
90
94
|
requests: z.ZodNumber;
|
|
91
95
|
duration_seconds: z.ZodDefault<z.ZodNumber>;
|
|
92
96
|
headers: z.ZodDefault<z.ZodRecord<z.ZodString, z.ZodString>>;
|
|
97
|
+
max_error_rate: z.ZodDefault<z.ZodNumber>;
|
|
93
98
|
expect: z.ZodObject<{
|
|
94
99
|
rate_limit_enforced: z.ZodOptional<z.ZodBoolean>;
|
|
95
100
|
quota_enforced: z.ZodOptional<z.ZodBoolean>;
|
|
96
101
|
detection_event: z.ZodOptional<z.ZodBoolean>;
|
|
102
|
+
rate_limit_within: z.ZodOptional<z.ZodNumber>;
|
|
103
|
+
quota_within: z.ZodOptional<z.ZodNumber>;
|
|
97
104
|
}, z.core.$strict>;
|
|
98
105
|
}, z.core.$strict>;
|
|
99
106
|
export type AgentTestSecurityTest = z.infer<typeof SecurityTestSchema>;
|
|
@@ -102,6 +109,7 @@ export declare const SafetySchema: z.ZodObject<{
|
|
|
102
109
|
max_requests: z.ZodNumber;
|
|
103
110
|
allow_production: z.ZodDefault<z.ZodBoolean>;
|
|
104
111
|
allowed_hosts: z.ZodDefault<z.ZodArray<z.ZodString>>;
|
|
112
|
+
allow_insecure_http: z.ZodDefault<z.ZodBoolean>;
|
|
105
113
|
}, z.core.$strict>;
|
|
106
114
|
export type AgentTestSafety = z.infer<typeof SafetySchema>;
|
|
107
115
|
/** The result of one expectation: what was expected, what the evidence shows, and why. */
|
|
@@ -125,14 +133,20 @@ export interface AgentTestSecurityEvidence {
|
|
|
125
133
|
firstQuota?: number;
|
|
126
134
|
/** Successful responses after the first quota response: a quota that lets traffic through again is not enforced. */
|
|
127
135
|
succeededAfterQuota: number;
|
|
136
|
+
/** Non-2xx responses that match no signal, such as 500s: evidence about the target, not its defenses. */
|
|
137
|
+
errors: number;
|
|
128
138
|
/** New detection events seen during the scenario; absent without `target.detection`. */
|
|
129
139
|
detectionEvents?: number;
|
|
140
|
+
/** Whether `detectionEvents` counts only this run's events (`target.detection.run_field`); absent without detection. */
|
|
141
|
+
detectionCorrelated?: boolean;
|
|
130
142
|
elapsedMs: number;
|
|
131
143
|
}
|
|
132
144
|
/** One security Test with its checks, evidence, and overall pass. */
|
|
133
145
|
export interface AgentTestSecurityResult {
|
|
134
146
|
test: AgentTestSecurityTest;
|
|
135
147
|
passed: boolean;
|
|
148
|
+
/** Why the evidence cannot be trusted, such as too many errors; set only then, and the Test fails. */
|
|
149
|
+
inconclusive?: string;
|
|
136
150
|
checks: AgentTestSecurityCheck[];
|
|
137
151
|
evidence: AgentTestSecurityEvidence;
|
|
138
152
|
}
|
|
@@ -147,4 +161,6 @@ export interface AgentTestSecurityReport {
|
|
|
147
161
|
passed: number;
|
|
148
162
|
total: number;
|
|
149
163
|
results: AgentTestSecurityResult[];
|
|
164
|
+
/** Policy problems that are not errors yet, such as plain HTTP to a remote host. */
|
|
165
|
+
warnings: string[];
|
|
150
166
|
}
|
package/dist/probe/types.js
CHANGED
|
@@ -25,7 +25,9 @@ export const DetectionSchema = z.strictObject({
|
|
|
25
25
|
url: z.url({ protocol: /^https?$/ }),
|
|
26
26
|
headers: HeadersSchema.default({}),
|
|
27
27
|
/** How long to keep polling for a new event after the scenario ends. */
|
|
28
|
-
wait_ms: z.number().int().min(0).max(60_000).default(0)
|
|
28
|
+
wait_ms: z.number().int().min(0).max(60_000).default(0),
|
|
29
|
+
/** The event field holding the run id the API saw in `x-agentests-run`; when set, only this run's events count. */
|
|
30
|
+
run_field: TextSchema.optional()
|
|
29
31
|
});
|
|
30
32
|
/** The API under test. Requests are POSTs of `body`, a JSON template whose strings may hold `{{prompt}}`, `{{index}}`, `{{scenario}}`, and `{{run_id}}`. */
|
|
31
33
|
export const TargetSchema = z.strictObject({
|
|
@@ -40,14 +42,20 @@ export const TargetSchema = z.strictObject({
|
|
|
40
42
|
}),
|
|
41
43
|
detection: DetectionSchema.optional()
|
|
42
44
|
});
|
|
43
|
-
/** What a security Test expects of the defenses; at least one
|
|
45
|
+
/** What a security Test expects of the defenses; at least one of the three expectations. */
|
|
44
46
|
export const ExpectSchema = z
|
|
45
47
|
.strictObject({
|
|
46
48
|
rate_limit_enforced: z.boolean().optional(),
|
|
47
49
|
quota_enforced: z.boolean().optional(),
|
|
48
|
-
detection_event: z.boolean().optional()
|
|
50
|
+
detection_event: z.boolean().optional(),
|
|
51
|
+
/** With `rate_limit_enforced: true`: the first rate-limited response must come by this request. */
|
|
52
|
+
rate_limit_within: z.number().int().min(1).optional(),
|
|
53
|
+
/** With `quota_enforced: true`: the first quota response must come by this request. */
|
|
54
|
+
quota_within: z.number().int().min(1).optional()
|
|
49
55
|
})
|
|
50
|
-
.refine((value) =>
|
|
56
|
+
.refine((value) => value.rate_limit_enforced !== undefined ||
|
|
57
|
+
value.quota_enforced !== undefined ||
|
|
58
|
+
value.detection_event !== undefined, 'Expected at least one expectation');
|
|
51
59
|
/** One bounded attack simulation and its expectations; `duration_seconds` spreads the requests evenly, 0 sends them back to back. */
|
|
52
60
|
export const SecurityTestSchema = z.strictObject({
|
|
53
61
|
id: IdSchema,
|
|
@@ -57,11 +65,15 @@ export const SecurityTestSchema = z.strictObject({
|
|
|
57
65
|
duration_seconds: z.number().min(0).max(3_600).default(0),
|
|
58
66
|
/** Merged over the target's headers for this Test only, e.g. its own API key. */
|
|
59
67
|
headers: HeadersSchema.default({}),
|
|
68
|
+
/** The largest share of responses that may be errors matching no signal before the Test is inconclusive, and fails. */
|
|
69
|
+
max_error_rate: z.number().min(0).max(1).default(0.05),
|
|
60
70
|
expect: ExpectSchema
|
|
61
71
|
});
|
|
62
72
|
/** Hard limits the runner never exceeds. Hosts other than localhost need listing; production needs `allow_production` and the `--allow-production` flag. */
|
|
63
73
|
export const SafetySchema = z.strictObject({
|
|
64
74
|
max_requests: z.number().int().min(1).max(10_000),
|
|
65
75
|
allow_production: z.boolean().default(false),
|
|
66
|
-
allowed_hosts: z.array(TextSchema).default([])
|
|
76
|
+
allowed_hosts: z.array(TextSchema).default([]),
|
|
77
|
+
/** Accept plain `http://` to a non-local host. Without it, such a URL is a warning in 1.x and an error from 2.0. */
|
|
78
|
+
allow_insecure_http: z.boolean().default(false)
|
|
67
79
|
});
|
package/dist/probe/verdict.d.ts
CHANGED
|
@@ -1,3 +1,5 @@
|
|
|
1
|
-
import type { AgentTestExpect, AgentTestSecurityCheck, AgentTestSecurityEvidence } from './types.ts';
|
|
1
|
+
import type { AgentTestSecurityTest, AgentTestExpect, AgentTestSecurityCheck, AgentTestSecurityEvidence } from './types.ts';
|
|
2
2
|
/** One check per expectation the Test sets, in a fixed order. */
|
|
3
3
|
export declare const judge: (expect: AgentTestExpect, evidence: AgentTestSecurityEvidence) => AgentTestSecurityCheck[];
|
|
4
|
+
/** Why a Test's evidence says more about the target's errors than its defenses, or undefined when it is sound. */
|
|
5
|
+
export declare const inconclusive: ({ max_error_rate }: AgentTestSecurityTest, { sent, errors, statuses }: AgentTestSecurityEvidence) => string | undefined;
|
package/dist/probe/verdict.js
CHANGED
|
@@ -1,17 +1,29 @@
|
|
|
1
1
|
/** What the evidence shows for one expectation, and the sentence explaining it. */
|
|
2
|
-
const observe = (expectation, evidence) => {
|
|
2
|
+
const observe = (expectation, evidence, within) => {
|
|
3
3
|
const { sent } = evidence;
|
|
4
4
|
switch (expectation) {
|
|
5
|
-
case 'rate_limit_enforced':
|
|
6
|
-
|
|
7
|
-
|
|
8
|
-
: {
|
|
9
|
-
|
|
10
|
-
|
|
5
|
+
case 'rate_limit_enforced': {
|
|
6
|
+
const first = evidence.firstRateLimit;
|
|
7
|
+
if (first === undefined)
|
|
8
|
+
return { actual: false, detail: `no rate-limited response in ${sent} requests` };
|
|
9
|
+
if (within !== undefined && first > within)
|
|
10
|
+
return {
|
|
11
|
+
actual: false,
|
|
12
|
+
detail: `first rate-limited at request ${first}, after the limit of ${within}`
|
|
11
13
|
};
|
|
14
|
+
return {
|
|
15
|
+
actual: true,
|
|
16
|
+
detail: `first rate-limited at request ${first} of ${sent}; ${evidence.rateLimited} rejected`
|
|
17
|
+
};
|
|
18
|
+
}
|
|
12
19
|
case 'quota_enforced':
|
|
13
20
|
if (evidence.firstQuota === undefined)
|
|
14
21
|
return { actual: false, detail: `no quota response in ${sent} requests` };
|
|
22
|
+
if (within !== undefined && evidence.firstQuota > within)
|
|
23
|
+
return {
|
|
24
|
+
actual: false,
|
|
25
|
+
detail: `quota hit at request ${evidence.firstQuota}, after the limit of ${within}`
|
|
26
|
+
};
|
|
15
27
|
if (evidence.succeededAfterQuota > 0)
|
|
16
28
|
return {
|
|
17
29
|
actual: false,
|
|
@@ -23,7 +35,8 @@ const observe = (expectation, evidence) => {
|
|
|
23
35
|
};
|
|
24
36
|
case 'detection_event': {
|
|
25
37
|
const events = evidence.detectionEvents ?? 0;
|
|
26
|
-
|
|
38
|
+
const scope = evidence.detectionCorrelated ? ' for this run' : '';
|
|
39
|
+
return { actual: events > 0, detail: `${events} new detection event(s)${scope}` };
|
|
27
40
|
}
|
|
28
41
|
}
|
|
29
42
|
};
|
|
@@ -32,6 +45,21 @@ export const judge = (expect, evidence) => ['rate_limit_enforced', 'quota_enforc
|
|
|
32
45
|
const expected = expect[expectation];
|
|
33
46
|
if (expected === undefined)
|
|
34
47
|
return [];
|
|
35
|
-
const
|
|
48
|
+
const within = expectation === 'rate_limit_enforced'
|
|
49
|
+
? expect.rate_limit_within
|
|
50
|
+
: expectation === 'quota_enforced'
|
|
51
|
+
? expect.quota_within
|
|
52
|
+
: undefined;
|
|
53
|
+
const { actual, detail } = observe(expectation, evidence, within);
|
|
36
54
|
return [{ expectation, expected, actual, passed: actual === expected, detail }];
|
|
37
55
|
});
|
|
56
|
+
/** Why a Test's evidence says more about the target's errors than its defenses, or undefined when it is sound. */
|
|
57
|
+
export const inconclusive = ({ max_error_rate }, { sent, errors, statuses }) => {
|
|
58
|
+
if (!sent || errors / sent <= max_error_rate)
|
|
59
|
+
return undefined;
|
|
60
|
+
const codes = Object.entries(statuses)
|
|
61
|
+
.filter(([status]) => !/^2/.test(status))
|
|
62
|
+
.map(([status, count]) => `${status}×${count}`)
|
|
63
|
+
.join(', ');
|
|
64
|
+
return `${errors} of ${sent} responses were errors matching no signal (non-2xx: ${codes}), over max_error_rate ${max_error_rate}`;
|
|
65
|
+
};
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@nyancorn/agentests-security",
|
|
3
|
-
"version": "
|
|
3
|
+
"version": "1.1.0",
|
|
4
4
|
"description": "Test an AI API's defenses against model extraction in CI: rate limits, quotas, and detection, as bounded, deterministic security policies.",
|
|
5
5
|
"keywords": [
|
|
6
6
|
"ai-security",
|
|
@@ -30,12 +30,17 @@
|
|
|
30
30
|
".": {
|
|
31
31
|
"types": "./dist/index.d.ts",
|
|
32
32
|
"default": "./dist/index.js"
|
|
33
|
-
}
|
|
33
|
+
},
|
|
34
|
+
"./schema.json": "./schema/policy.schema.json"
|
|
34
35
|
},
|
|
35
36
|
"files": [
|
|
36
37
|
"dist",
|
|
38
|
+
"schema",
|
|
37
39
|
"README.md",
|
|
38
|
-
"LICENSE"
|
|
40
|
+
"LICENSE",
|
|
41
|
+
"STABILITY.md",
|
|
42
|
+
"SECURITY.md",
|
|
43
|
+
"CHANGELOG.md"
|
|
39
44
|
],
|
|
40
45
|
"engines": {
|
|
41
46
|
"node": ">=22"
|
|
@@ -44,7 +49,8 @@
|
|
|
44
49
|
"build": "rm -rf dist && tsc -p tsconfig.json",
|
|
45
50
|
"check": "tsc -p tsconfig.json --noEmit",
|
|
46
51
|
"test": "npm run build && bun test ./tests",
|
|
47
|
-
"prepublishOnly": "npm test"
|
|
52
|
+
"prepublishOnly": "npm test",
|
|
53
|
+
"schema": "bun scripts/schema.ts"
|
|
48
54
|
},
|
|
49
55
|
"dependencies": {
|
|
50
56
|
"clinguini": "^1.0.1",
|
|
@@ -54,5 +60,9 @@
|
|
|
54
60
|
"devDependencies": {
|
|
55
61
|
"@types/node": "^22",
|
|
56
62
|
"typescript": "^6.0.3"
|
|
63
|
+
},
|
|
64
|
+
"publishConfig": {
|
|
65
|
+
"access": "public",
|
|
66
|
+
"provenance": true
|
|
57
67
|
}
|
|
58
68
|
}
|
|
@@ -0,0 +1,291 @@
|
|
|
1
|
+
{
|
|
2
|
+
"$schema": "https://json-schema.org/draft/2020-12/schema",
|
|
3
|
+
"$id": "https://unpkg.com/@nyancorn/agentests-security@1/schema/policy.schema.json",
|
|
4
|
+
"title": "agentests-security policy, version 1",
|
|
5
|
+
"type": "object",
|
|
6
|
+
"properties": {
|
|
7
|
+
"version": {
|
|
8
|
+
"type": "number",
|
|
9
|
+
"const": 1
|
|
10
|
+
},
|
|
11
|
+
"name": {
|
|
12
|
+
"type": "string",
|
|
13
|
+
"pattern": "^[a-z0-9]+(-[a-z0-9]+)*$"
|
|
14
|
+
},
|
|
15
|
+
"target": {
|
|
16
|
+
"type": "object",
|
|
17
|
+
"properties": {
|
|
18
|
+
"url": {
|
|
19
|
+
"type": "string",
|
|
20
|
+
"format": "uri"
|
|
21
|
+
},
|
|
22
|
+
"environment": {
|
|
23
|
+
"type": "string",
|
|
24
|
+
"enum": [
|
|
25
|
+
"local",
|
|
26
|
+
"test",
|
|
27
|
+
"staging",
|
|
28
|
+
"production"
|
|
29
|
+
]
|
|
30
|
+
},
|
|
31
|
+
"headers": {
|
|
32
|
+
"default": {},
|
|
33
|
+
"type": "object",
|
|
34
|
+
"propertyNames": {
|
|
35
|
+
"type": "string",
|
|
36
|
+
"minLength": 1
|
|
37
|
+
},
|
|
38
|
+
"additionalProperties": {
|
|
39
|
+
"type": "string"
|
|
40
|
+
}
|
|
41
|
+
},
|
|
42
|
+
"body": {
|
|
43
|
+
"default": {
|
|
44
|
+
"prompt": "{{prompt}}"
|
|
45
|
+
}
|
|
46
|
+
},
|
|
47
|
+
"timeout_ms": {
|
|
48
|
+
"default": 10000,
|
|
49
|
+
"type": "integer",
|
|
50
|
+
"minimum": 100,
|
|
51
|
+
"maximum": 60000
|
|
52
|
+
},
|
|
53
|
+
"signals": {
|
|
54
|
+
"default": {
|
|
55
|
+
"rate_limit": {
|
|
56
|
+
"status": [
|
|
57
|
+
429
|
|
58
|
+
]
|
|
59
|
+
},
|
|
60
|
+
"quota": {
|
|
61
|
+
"status": [
|
|
62
|
+
402,
|
|
63
|
+
403,
|
|
64
|
+
429
|
|
65
|
+
],
|
|
66
|
+
"body": "quota"
|
|
67
|
+
}
|
|
68
|
+
},
|
|
69
|
+
"type": "object",
|
|
70
|
+
"properties": {
|
|
71
|
+
"rate_limit": {
|
|
72
|
+
"default": {
|
|
73
|
+
"status": [
|
|
74
|
+
429
|
|
75
|
+
]
|
|
76
|
+
},
|
|
77
|
+
"type": "object",
|
|
78
|
+
"properties": {
|
|
79
|
+
"status": {
|
|
80
|
+
"minItems": 1,
|
|
81
|
+
"type": "array",
|
|
82
|
+
"items": {
|
|
83
|
+
"type": "integer",
|
|
84
|
+
"minimum": 100,
|
|
85
|
+
"maximum": 599
|
|
86
|
+
}
|
|
87
|
+
},
|
|
88
|
+
"body": {
|
|
89
|
+
"type": "string"
|
|
90
|
+
}
|
|
91
|
+
},
|
|
92
|
+
"required": [
|
|
93
|
+
"status"
|
|
94
|
+
],
|
|
95
|
+
"additionalProperties": false
|
|
96
|
+
},
|
|
97
|
+
"quota": {
|
|
98
|
+
"default": {
|
|
99
|
+
"status": [
|
|
100
|
+
402,
|
|
101
|
+
403,
|
|
102
|
+
429
|
|
103
|
+
],
|
|
104
|
+
"body": "quota"
|
|
105
|
+
},
|
|
106
|
+
"type": "object",
|
|
107
|
+
"properties": {
|
|
108
|
+
"status": {
|
|
109
|
+
"minItems": 1,
|
|
110
|
+
"type": "array",
|
|
111
|
+
"items": {
|
|
112
|
+
"type": "integer",
|
|
113
|
+
"minimum": 100,
|
|
114
|
+
"maximum": 599
|
|
115
|
+
}
|
|
116
|
+
},
|
|
117
|
+
"body": {
|
|
118
|
+
"type": "string"
|
|
119
|
+
}
|
|
120
|
+
},
|
|
121
|
+
"required": [
|
|
122
|
+
"status"
|
|
123
|
+
],
|
|
124
|
+
"additionalProperties": false
|
|
125
|
+
}
|
|
126
|
+
},
|
|
127
|
+
"additionalProperties": false
|
|
128
|
+
},
|
|
129
|
+
"detection": {
|
|
130
|
+
"type": "object",
|
|
131
|
+
"properties": {
|
|
132
|
+
"url": {
|
|
133
|
+
"type": "string",
|
|
134
|
+
"format": "uri"
|
|
135
|
+
},
|
|
136
|
+
"headers": {
|
|
137
|
+
"default": {},
|
|
138
|
+
"type": "object",
|
|
139
|
+
"propertyNames": {
|
|
140
|
+
"type": "string",
|
|
141
|
+
"minLength": 1
|
|
142
|
+
},
|
|
143
|
+
"additionalProperties": {
|
|
144
|
+
"type": "string"
|
|
145
|
+
}
|
|
146
|
+
},
|
|
147
|
+
"wait_ms": {
|
|
148
|
+
"default": 0,
|
|
149
|
+
"type": "integer",
|
|
150
|
+
"minimum": 0,
|
|
151
|
+
"maximum": 60000
|
|
152
|
+
},
|
|
153
|
+
"run_field": {
|
|
154
|
+
"type": "string"
|
|
155
|
+
}
|
|
156
|
+
},
|
|
157
|
+
"required": [
|
|
158
|
+
"url"
|
|
159
|
+
],
|
|
160
|
+
"additionalProperties": false
|
|
161
|
+
}
|
|
162
|
+
},
|
|
163
|
+
"required": [
|
|
164
|
+
"url",
|
|
165
|
+
"environment"
|
|
166
|
+
],
|
|
167
|
+
"additionalProperties": false
|
|
168
|
+
},
|
|
169
|
+
"tests": {
|
|
170
|
+
"minItems": 1,
|
|
171
|
+
"type": "array",
|
|
172
|
+
"items": {
|
|
173
|
+
"type": "object",
|
|
174
|
+
"properties": {
|
|
175
|
+
"id": {
|
|
176
|
+
"type": "string",
|
|
177
|
+
"pattern": "^[a-z0-9]+(-[a-z0-9]+)*$"
|
|
178
|
+
},
|
|
179
|
+
"description": {
|
|
180
|
+
"type": "string"
|
|
181
|
+
},
|
|
182
|
+
"scenario": {
|
|
183
|
+
"type": "string",
|
|
184
|
+
"enum": [
|
|
185
|
+
"sustained-query-volume",
|
|
186
|
+
"systematic-query-pattern",
|
|
187
|
+
"quota-exhaustion"
|
|
188
|
+
]
|
|
189
|
+
},
|
|
190
|
+
"requests": {
|
|
191
|
+
"type": "integer",
|
|
192
|
+
"minimum": 1,
|
|
193
|
+
"maximum": 10000
|
|
194
|
+
},
|
|
195
|
+
"duration_seconds": {
|
|
196
|
+
"default": 0,
|
|
197
|
+
"type": "number",
|
|
198
|
+
"minimum": 0,
|
|
199
|
+
"maximum": 3600
|
|
200
|
+
},
|
|
201
|
+
"headers": {
|
|
202
|
+
"default": {},
|
|
203
|
+
"type": "object",
|
|
204
|
+
"propertyNames": {
|
|
205
|
+
"type": "string",
|
|
206
|
+
"minLength": 1
|
|
207
|
+
},
|
|
208
|
+
"additionalProperties": {
|
|
209
|
+
"type": "string"
|
|
210
|
+
}
|
|
211
|
+
},
|
|
212
|
+
"max_error_rate": {
|
|
213
|
+
"default": 0.05,
|
|
214
|
+
"type": "number",
|
|
215
|
+
"minimum": 0,
|
|
216
|
+
"maximum": 1
|
|
217
|
+
},
|
|
218
|
+
"expect": {
|
|
219
|
+
"type": "object",
|
|
220
|
+
"properties": {
|
|
221
|
+
"rate_limit_enforced": {
|
|
222
|
+
"type": "boolean"
|
|
223
|
+
},
|
|
224
|
+
"quota_enforced": {
|
|
225
|
+
"type": "boolean"
|
|
226
|
+
},
|
|
227
|
+
"detection_event": {
|
|
228
|
+
"type": "boolean"
|
|
229
|
+
},
|
|
230
|
+
"rate_limit_within": {
|
|
231
|
+
"type": "integer",
|
|
232
|
+
"minimum": 1,
|
|
233
|
+
"maximum": 9007199254740991
|
|
234
|
+
},
|
|
235
|
+
"quota_within": {
|
|
236
|
+
"type": "integer",
|
|
237
|
+
"minimum": 1,
|
|
238
|
+
"maximum": 9007199254740991
|
|
239
|
+
}
|
|
240
|
+
},
|
|
241
|
+
"additionalProperties": false
|
|
242
|
+
}
|
|
243
|
+
},
|
|
244
|
+
"required": [
|
|
245
|
+
"id",
|
|
246
|
+
"scenario",
|
|
247
|
+
"requests",
|
|
248
|
+
"expect"
|
|
249
|
+
],
|
|
250
|
+
"additionalProperties": false
|
|
251
|
+
}
|
|
252
|
+
},
|
|
253
|
+
"safety": {
|
|
254
|
+
"type": "object",
|
|
255
|
+
"properties": {
|
|
256
|
+
"max_requests": {
|
|
257
|
+
"type": "integer",
|
|
258
|
+
"minimum": 1,
|
|
259
|
+
"maximum": 10000
|
|
260
|
+
},
|
|
261
|
+
"allow_production": {
|
|
262
|
+
"default": false,
|
|
263
|
+
"type": "boolean"
|
|
264
|
+
},
|
|
265
|
+
"allowed_hosts": {
|
|
266
|
+
"default": [],
|
|
267
|
+
"type": "array",
|
|
268
|
+
"items": {
|
|
269
|
+
"type": "string"
|
|
270
|
+
}
|
|
271
|
+
},
|
|
272
|
+
"allow_insecure_http": {
|
|
273
|
+
"default": false,
|
|
274
|
+
"type": "boolean"
|
|
275
|
+
}
|
|
276
|
+
},
|
|
277
|
+
"required": [
|
|
278
|
+
"max_requests"
|
|
279
|
+
],
|
|
280
|
+
"additionalProperties": false
|
|
281
|
+
}
|
|
282
|
+
},
|
|
283
|
+
"required": [
|
|
284
|
+
"version",
|
|
285
|
+
"name",
|
|
286
|
+
"target",
|
|
287
|
+
"tests",
|
|
288
|
+
"safety"
|
|
289
|
+
],
|
|
290
|
+
"additionalProperties": false
|
|
291
|
+
}
|