xitok 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +19 -0
- package/LICENSE +93 -0
- package/README.md +432 -0
- package/THIRD_PARTY_NOTICES.md +574 -0
- package/dist/chunk-7F2CNUMK.js +38206 -0
- package/dist/dist-PPO7F54A.js +389 -0
- package/dist/index.js +9569 -0
- package/dist/ui/assets/index-BiLMJFg0.js +242 -0
- package/dist/ui/assets/index-jMrZCA-l.css +2 -0
- package/dist/ui/favicon.svg +1 -0
- package/dist/ui/index.html +21 -0
- package/package.json +64 -0
- package/sbom.cdx.json +248 -0
package/CHANGELOG.md
ADDED
|
@@ -0,0 +1,19 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
Versions follow [Semantic Versioning](https://semver.org). Releases are cut by release-please from
|
|
4
|
+
[Conventional Commits](https://www.conventionalcommits.org); see `docs/RELEASING.md`.
|
|
5
|
+
|
|
6
|
+
## 0.1.0 (unreleased)
|
|
7
|
+
|
|
8
|
+
First release.
|
|
9
|
+
|
|
10
|
+
- `xitok init`, `discover`, `add`: set up a project from what an agent would find — OpenAPI, docs
|
|
11
|
+
platform, llms.txt, MCP, SDKs — with a library of ready-made journeys.
|
|
12
|
+
- `xitok run`: real agents (Claude Code; Codex, Gemini, OpenCode and Cursor experimental) in clean
|
|
13
|
+
rooms, several trials each, every outcome verified by running it; Wilson intervals; a signed
|
|
14
|
+
report.
|
|
15
|
+
- `xitok ci` and the GitHub Action: change-aware smoke runs on pull requests, pooled baselines,
|
|
16
|
+
Fisher's exact test for regressions, one sticky PR comment, annotations, JUnit.
|
|
17
|
+
- Docker sandbox with an egress allowlist.
|
|
18
|
+
- Model matrix, flakiness detection, an optional LLM judge that never counts, AXIS import.
|
|
19
|
+
- `xitok ui`: a local dashboard.
|
package/LICENSE
ADDED
|
@@ -0,0 +1,93 @@
|
|
|
1
|
+
Elastic License 2.0 (ELv2)
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Lynt-X Global
|
|
4
|
+
|
|
5
|
+
## Acceptance
|
|
6
|
+
|
|
7
|
+
By using the software, you agree to all of the terms and conditions below.
|
|
8
|
+
|
|
9
|
+
## Copyright License
|
|
10
|
+
|
|
11
|
+
The licensor grants you a non-exclusive, royalty-free, worldwide,
|
|
12
|
+
non-sublicensable, non-transferable license to use, copy, distribute, make
|
|
13
|
+
available, and prepare derivative works of the software, in each case subject to
|
|
14
|
+
the limitations and conditions below.
|
|
15
|
+
|
|
16
|
+
## Limitations
|
|
17
|
+
|
|
18
|
+
You may not provide the software to third parties as a hosted or managed
|
|
19
|
+
service, where the service provides users with access to any substantial set of
|
|
20
|
+
the features or functionality of the software.
|
|
21
|
+
|
|
22
|
+
You may not move, change, disable, or circumvent the license key functionality
|
|
23
|
+
in the software, and you may not remove or obscure any functionality in the
|
|
24
|
+
software that is protected by the license key.
|
|
25
|
+
|
|
26
|
+
You may not alter, remove, or obscure any licensing, copyright, or other notices
|
|
27
|
+
of the licensor in the software. Any use of the licensor's trademarks is subject
|
|
28
|
+
to applicable law.
|
|
29
|
+
|
|
30
|
+
## Patents
|
|
31
|
+
|
|
32
|
+
The licensor grants you a license, under any patent claims the licensor can
|
|
33
|
+
license, or becomes able to license, to make, have made, use, sell, offer for
|
|
34
|
+
sale, import and have imported the software, in each case subject to the
|
|
35
|
+
limitations and conditions in this license. This license does not cover any
|
|
36
|
+
patent claims that you cause to be infringed by modifications or additions to
|
|
37
|
+
the software. If you or your company make any written claim that the software
|
|
38
|
+
infringes or contributes to infringement of any patent, your patent license for
|
|
39
|
+
the software granted under these terms ends immediately. If your company makes
|
|
40
|
+
such a claim, your patent license ends immediately for work on behalf of your
|
|
41
|
+
company.
|
|
42
|
+
|
|
43
|
+
## Notices
|
|
44
|
+
|
|
45
|
+
You must ensure that anyone who gets a copy of any part of the software from you
|
|
46
|
+
also gets a copy of these terms.
|
|
47
|
+
|
|
48
|
+
If you modify the software, you must include in any modified copies of the
|
|
49
|
+
software prominent notices stating that you have modified the software.
|
|
50
|
+
|
|
51
|
+
## No Other Rights
|
|
52
|
+
|
|
53
|
+
These terms do not imply any licenses other than those expressly granted in
|
|
54
|
+
these terms.
|
|
55
|
+
|
|
56
|
+
## Termination
|
|
57
|
+
|
|
58
|
+
If you use the software in violation of these terms, such use is not licensed,
|
|
59
|
+
and your licenses will automatically terminate. If the licensor provides you with
|
|
60
|
+
a notice of your violation, and you cease all violation of this license no later
|
|
61
|
+
than 30 days after you receive that notice, your licenses will be reinstated
|
|
62
|
+
retroactively. However, if you violate these terms after such reinstatement, any
|
|
63
|
+
additional violation of these terms will cause your licenses to terminate
|
|
64
|
+
automatically and permanently.
|
|
65
|
+
|
|
66
|
+
## No Liability
|
|
67
|
+
|
|
68
|
+
*As far as the law allows, the software comes as is, without any warranty or
|
|
69
|
+
condition, and the licensor will not be liable to you for any damages arising out
|
|
70
|
+
of these terms or the use or nature of the software, under any kind of legal
|
|
71
|
+
claim.*
|
|
72
|
+
|
|
73
|
+
## Definitions
|
|
74
|
+
|
|
75
|
+
The **licensor** is the entity offering these terms, and the **software** is the
|
|
76
|
+
software the licensor makes available under these terms, including any portion of
|
|
77
|
+
it.
|
|
78
|
+
|
|
79
|
+
**you** refers to the individual or entity agreeing to these terms.
|
|
80
|
+
|
|
81
|
+
**your company** is any legal entity, sole proprietorship, or other kind of
|
|
82
|
+
organization that you work for, plus all organizations that have control over,
|
|
83
|
+
are under the control of, or are under common control with that organization.
|
|
84
|
+
**control** means ownership of substantially all the assets of an entity, or the
|
|
85
|
+
power to direct its management and policies by vote, contract, or otherwise.
|
|
86
|
+
Control can be direct or indirect.
|
|
87
|
+
|
|
88
|
+
**your licenses** are all the licenses granted to you for the software under
|
|
89
|
+
these terms.
|
|
90
|
+
|
|
91
|
+
**use** means anything you do with the software requiring one of your licenses.
|
|
92
|
+
|
|
93
|
+
**trademark** means trademarks, service marks, and similar rights.
|
package/README.md
ADDED
|
@@ -0,0 +1,432 @@
|
|
|
1
|
+
# XitOK
|
|
2
|
+
|
|
3
|
+
**XitOK** (pronounced *exit OK*) tests your API with real AI coding agents — Claude Code, Codex,
|
|
4
|
+
Gemini, OpenCode, Cursor — the way a brand-new developer's agent would meet it: in a clean room,
|
|
5
|
+
many times, with every outcome **verified by running it**, and a signed report you can prove.
|
|
6
|
+
Part of the [Lyntway](https://lyntway.com) family.
|
|
7
|
+
|
|
8
|
+
```
|
|
9
|
+
XitOK score 57.5 / 100 grade D
|
|
10
|
+
────────────────────────────────────────────────────────────────────
|
|
11
|
+
Journey claude-code codex
|
|
12
|
+
Make a first authenticated call 100% 4/4 [51–100] 25% 1/4 [5–70]
|
|
13
|
+
Verify a webhook signature 100% 4/4 [51–100] 0% 0/4 [0–49]
|
|
14
|
+
```
|
|
15
|
+
|
|
16
|
+
## Quick start
|
|
17
|
+
|
|
18
|
+
```bash
|
|
19
|
+
npx xitok init # discovers your API, writes xitok.yaml and a first journey
|
|
20
|
+
npx xitok doctor # checks agents and keys
|
|
21
|
+
npx xitok run # runs journeys × agents × trials and scores the result
|
|
22
|
+
```
|
|
23
|
+
|
|
24
|
+
Try it with no API keys: `cd examples/demo && npx xitok run` (two scripted agents stand in for real ones).
|
|
25
|
+
|
|
26
|
+
## Commands
|
|
27
|
+
|
|
28
|
+
| Command | What it does |
|
|
29
|
+
|---|---|
|
|
30
|
+
| `xitok init` | Discovers your API (OpenAPI, docs platform, llms.txt, MCP, SDKs), writes `xitok.yaml`, a first journey, and a webhook-security journey when your spec has webhooks |
|
|
31
|
+
| `xitok check <docs-url>` | Free preview of how ready a public docs site is for agents: readable without JavaScript, llms.txt, Markdown, OpenAPI, robots.txt, page weight… with what to fix first. Static checks, no agents, no signup |
|
|
32
|
+
| `xitok discover` | Shows what an agent would find in your repo — and, with `--docs-url`, every public-docs check from `xitok check` |
|
|
33
|
+
| `xitok add [template]` | Adds a ready-made journey: first call, no hardcoded secrets, secret key kept server-side, webhook signatures, current SDK version, idempotent create, rate-limit backoff (tested against a real 429), outcome on the live API, MCP tool use (`--list`) |
|
|
34
|
+
| `xitok run` | Runs every journey with every agent, several trials each, in clean rooms, with a live progress matrix; `--models` for a model matrix |
|
|
35
|
+
| `xitok watch` | Re-runs the journeys a change affects every time you save (1 trial by default, a per-session spend cap, runs never overlap) |
|
|
36
|
+
| `xitok setup-agent <claude-code\|cursor\|codex>` | Teaches your coding agent to run XitOK and fix what it finds (a skill, a Cursor rule, or an AGENTS.md section). Claude Code plugin: `/plugin marketplace add xitokhq/xitok` then `/plugin install xitok@xitok` for `/xitok:run`, `/xitok:fix`, `/xitok:journey` |
|
|
37
|
+
| `xitok setup-ci <provider>` | Writes the XitOK pipeline for GitHub Actions (pull requests, nightly baseline, new models), GitLab CI, CircleCI or Buildkite |
|
|
38
|
+
| `xitok ci` | CI mode: runs only the journeys a change affects (or `--mode full`), compares with the baseline, annotates the PR, posts one sticky comment, writes the job summary |
|
|
39
|
+
| `xitok draft` | Claude proposes new journeys from your spec and docs; only ones with a check that runs are kept, written as drafts you review and rename before they count |
|
|
40
|
+
| `xitok import axis [path]` | Imports Netlify AXIS scenarios: the rubric becomes a judge check, checks that run are derived where the rubric states something checkable, and the rest become drafts that say what to add |
|
|
41
|
+
| `xitok flaky` | Journey × agent pairs that pass only some of the time across recent runs, with the most common cause |
|
|
42
|
+
| `xitok generate <llms-txt\|agents-md\|skill>` | Drafts `llms.txt`, an `AGENTS.md` for your users' repos, or a Claude Code skill — from your OpenAPI spec, the rules your journeys enforce, and the mistakes agents made in your latest run |
|
|
43
|
+
| `xitok docs init <mintlify\|scalar\|fern\|readme\|in-repo>` | Writes a docs site into your repository for an API that has none — drafted from your spec, the rules your journeys enforce and what agents got wrong. No account, nothing published |
|
|
44
|
+
| `xitok spec draft` | Reads your code and drafts an OpenAPI document from the routes it finds, marked a draft in three places because nobody has checked it yet |
|
|
45
|
+
| `xitok upgrade --from <old spec>` | Compares two versions of your API and writes the migration as a skill agents follow: what breaks, what to change, and what it could not see. `--fail-on-breaking` stops a release that breaks callers |
|
|
46
|
+
| `xitok badge` | An SVG badge for your README: the score and grade, or `--agent` for one agent's pass rate; says "custom" unless the standard panel ran. `xitok ci` writes one too |
|
|
47
|
+
| `xitok models check` | What's new at Anthropic, OpenAI and Google since the last check; `--run` tests each new model side by side with the one you pin (daily workflow: `docs/ci/xitok-new-models.yml`) |
|
|
48
|
+
| `xitok compare <baseline> [current]` | Which journey × agent pairs regressed or improved, with Fisher's exact test so noise isn't called a regression |
|
|
49
|
+
| `xitok baseline set / show` | Promotes a run to the baseline that `xitok ci` compares against |
|
|
50
|
+
| `xitok ui` | Opens the dashboard for this project's runs: score trend, journeys × agents matrix, every trial's checks and full transcript — and a journey editor with live validation, a preview of what will run, and one-click promotion of reviewed drafts (`--read-only` to turn editing off). Local only (127.0.0.1) |
|
|
51
|
+
| `xitok report [path]` | Shows a past run (terminal, `--format md` for PR comments, `--format json`, or `--format junit`) |
|
|
52
|
+
| `xitok verify <report>` | Verifies a signed report offline; `--trust <keyId>` to also check who signed it |
|
|
53
|
+
| `xitok doctor` | Checks Node, config, agents, keys, the signing key, and Docker when the sandbox is on |
|
|
54
|
+
| `xitok schema` | Writes JSON Schemas so editors autocomplete `xitok.yaml` and journeys |
|
|
55
|
+
| `xitok completion <shell>` | Shell completions for bash, zsh and fish — commands, options, and journey ids from your project |
|
|
56
|
+
|
|
57
|
+
Every command accepts `--json`.
|
|
58
|
+
|
|
59
|
+
**Exit codes** — `0` ok · `1` score below `--fail-under`, a significant regression, or signature invalid · `2` invalid input or config · `3` nothing could run / interrupted.
|
|
60
|
+
|
|
61
|
+
## In CI
|
|
62
|
+
|
|
63
|
+
```yaml
|
|
64
|
+
# .github/workflows/xitok.yml — full examples in docs/ci/
|
|
65
|
+
- uses: xitokhq/xitok/action@v0
|
|
66
|
+
with:
|
|
67
|
+
mode: changed # only journeys whose `watch` globs match the diff
|
|
68
|
+
fail-under: 70
|
|
69
|
+
env:
|
|
70
|
+
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
|
|
71
|
+
```
|
|
72
|
+
|
|
73
|
+
`xitok ci` recognises GitHub Actions, GitLab CI (a note on the merge request with
|
|
74
|
+
`XITOK_GITLAB_TOKEN`, results in the merge request's test report), Buildkite (the summary as the
|
|
75
|
+
build annotation) and CircleCI; `xitok setup-ci <provider>` writes the pipeline.
|
|
76
|
+
|
|
77
|
+
On a pull request XitOK runs a small smoke suite for the journeys the change touches, compares
|
|
78
|
+
with the last few `main` baselines pooled, and posts one comment that it updates on every push. A
|
|
79
|
+
drop only fails the build when it is statistically significant (`ci.regressionAlpha`, default
|
|
80
|
+
0.1); smaller drops are reported as *dropped* without failing. Pairs that pass only some of the
|
|
81
|
+
time are labeled *flaky*: pooling gives them a baseline that is that mix, so they don't raise false
|
|
82
|
+
alarms. Nightly, `mode: full` runs everything and adds itself to the baseline history.
|
|
83
|
+
|
|
84
|
+
## Panels: which agents, which exact models
|
|
85
|
+
|
|
86
|
+
A score only means something if you know exactly what produced it. A **panel** is a versioned
|
|
87
|
+
lineup with pinned model versions:
|
|
88
|
+
|
|
89
|
+
```yaml
|
|
90
|
+
panel: standard # XitOK Standard Panel 2026.09: Claude Code + claude-sonnet-5,
|
|
91
|
+
# Codex + gpt-5.3-codex, Gemini CLI + gemini-3.5-flash
|
|
92
|
+
```
|
|
93
|
+
|
|
94
|
+
`panel: frontier` runs the strongest models instead; `--panel` works on `run`, `ci` and `watch`.
|
|
95
|
+
Public scores and badges use the standard panel. Choosing your own `agents` makes a *custom
|
|
96
|
+
lineup*, and reports say so. Every report records the model each agent actually ran (including a
|
|
97
|
+
switch mid-run), and `xitok doctor` warns about model aliases that can change silently and about
|
|
98
|
+
models whose origin your security team may need to review ([ADR 0003](docs/adr/0003-model-panels-and-origin-policy.md)).
|
|
99
|
+
|
|
100
|
+
## Model matrix
|
|
101
|
+
|
|
102
|
+
Which model does your API work best with? List several and each becomes its own column, with a
|
|
103
|
+
head-to-head that says which differences are real:
|
|
104
|
+
|
|
105
|
+
```yaml
|
|
106
|
+
agents:
|
|
107
|
+
- id: claude-code
|
|
108
|
+
models: [claude-opus-5, claude-sonnet-5, claude-haiku-4-5]
|
|
109
|
+
```
|
|
110
|
+
|
|
111
|
+
Or ad hoc: `xitok run --models claude-code:claude-opus-5,claude-code:claude-haiku-4-5`. Pull
|
|
112
|
+
request smoke runs use only the first model.
|
|
113
|
+
|
|
114
|
+
## Agent-to-agent (A2A)
|
|
115
|
+
|
|
116
|
+
If you ship an agent, test whether other agents can use it. An `a2a` journey gives the coding agent
|
|
117
|
+
a bridge to your agent over the Agent2Agent protocol (1.0 and 0.3), and XitOK verifies from its own
|
|
118
|
+
record of the traffic — not the coding agent's claim — that your agent took the task to the state
|
|
119
|
+
you expect:
|
|
120
|
+
|
|
121
|
+
```yaml
|
|
122
|
+
context:
|
|
123
|
+
policy: a2a
|
|
124
|
+
agentCardUrl: https://agent.acme.com # or the card's URL
|
|
125
|
+
agentHeaders: { authorization: "Bearer ${ACME_AGENT_TOKEN}" }
|
|
126
|
+
verify:
|
|
127
|
+
- type: a2a
|
|
128
|
+
expectState: completed
|
|
129
|
+
replyContains: confirmed
|
|
130
|
+
```
|
|
131
|
+
|
|
132
|
+
`xitok add a2a-task` starts one; `xitok check` reports whether your site publishes an agent card.
|
|
133
|
+
|
|
134
|
+
## LLM judge (optional, never counted)
|
|
135
|
+
|
|
136
|
+
Some qualities can't be checked by running code — is it idiomatic, is the explanation right? Add a
|
|
137
|
+
judge check and Claude grades the work against your rubric:
|
|
138
|
+
|
|
139
|
+
```yaml
|
|
140
|
+
verify:
|
|
141
|
+
- type: command # verified: decides pass or fail
|
|
142
|
+
run: node send.js
|
|
143
|
+
- type: judge # judged: recorded, never counted
|
|
144
|
+
rubric: Uses the official Acme SDK, not raw HTTP, and reads the key from the environment.
|
|
145
|
+
files: ["**/*.js"] # optional; default is every text file in the workspace
|
|
146
|
+
```
|
|
147
|
+
|
|
148
|
+
The judge uses `XITOK_JUDGE_API_KEY` or `ANTHROPIC_API_KEY` and `judge.model` (default
|
|
149
|
+
`claude-sonnet-5`); without a key, judged checks are skipped. Everything it is shown is redacted
|
|
150
|
+
first, and agent output is marked as data so the agent can't instruct its own grader. Every report
|
|
151
|
+
that used a judge says how often it agreed with the execution-verified outcome — the reason XitOK
|
|
152
|
+
scores on execution, not opinion. Judge tokens are recorded in the check's evidence; their cost is
|
|
153
|
+
not part of the agent spend.
|
|
154
|
+
|
|
155
|
+
## Sandbox
|
|
156
|
+
|
|
157
|
+
By default trials run as local processes in a clean room. For untrusted code, or when you want a
|
|
158
|
+
hard network boundary, run every agent and every check in a container:
|
|
159
|
+
|
|
160
|
+
```yaml
|
|
161
|
+
sandbox:
|
|
162
|
+
kind: docker
|
|
163
|
+
image: xitokhq/runner:latest # docker/runner/Dockerfile
|
|
164
|
+
network: allowlist # allowlist | none | open
|
|
165
|
+
allow: [api.acme.com] # added to the agent's model API, package registries and your docs
|
|
166
|
+
inspect: [api.acme.com] # optional: record which endpoints agents actually called
|
|
167
|
+
```
|
|
168
|
+
|
|
169
|
+
With `allowlist`, the container sits on an internal Docker network whose only way out is an egress
|
|
170
|
+
proxy that admits listed hosts on ports 80/443 and refuses everything else; direct connections have
|
|
171
|
+
no route. Every refused host is recorded in the report. Containers run as your user, with all Linux
|
|
172
|
+
capabilities dropped, `no-new-privileges`, and CPU, memory and process limits; the project is
|
|
173
|
+
mounted read-only. Secrets reach the container through a private env file deleted after each
|
|
174
|
+
command (or, for multi-line values, the docker CLI's own environment) and never appear in process
|
|
175
|
+
arguments.
|
|
176
|
+
|
|
177
|
+
### Shared journey libraries
|
|
178
|
+
|
|
179
|
+
One project's journeys can be inherited by others in the same organization — a security suite
|
|
180
|
+
every team must pass, kept private to you rather than published:
|
|
181
|
+
|
|
182
|
+
```yaml
|
|
183
|
+
libraries:
|
|
184
|
+
- project: prj_security # a project in your org, marked as a library
|
|
185
|
+
journeys: [webhook-*] # optional; omit to inherit all of them
|
|
186
|
+
```
|
|
187
|
+
|
|
188
|
+
Your own journeys win on a clash, and so does an earlier library: a shared suite can add checks
|
|
189
|
+
but never silently replace one of yours. A reference that matches no journey is an error rather
|
|
190
|
+
than a quiet no-op, because a run that skipped a security suite still reports as passed.
|
|
191
|
+
|
|
192
|
+
Libraries are resolved by XitOK Cloud, which is the only place that can see another project. A
|
|
193
|
+
local `xitok run` says so instead of testing less than the file asks for.
|
|
194
|
+
|
|
195
|
+
### Already have a Postman collection?
|
|
196
|
+
|
|
197
|
+
```bash
|
|
198
|
+
npx xitok import-postman ./acme.postman_collection.json
|
|
199
|
+
```
|
|
200
|
+
|
|
201
|
+
Each folder becomes a journey, and every request becomes a line of what that journey has to
|
|
202
|
+
achieve. Most teams have a collection and no OpenAPI spec, so this is usually the fastest way in.
|
|
203
|
+
|
|
204
|
+
**The collection never reaches the agent.** It tells XitOK what the API is and which endpoints
|
|
205
|
+
exist; the agent still has to work out the endpoints, the auth and the payloads from your
|
|
206
|
+
documentation. A prompt containing the method and path would measure replay, not discovery.
|
|
207
|
+
|
|
208
|
+
Those recorded endpoints are useful twice: they become the ground truth the
|
|
209
|
+
[endpoint ledger](#the-endpoint-ledger) is compared against, so an agent calling something that
|
|
210
|
+
does not exist is a finding you can prove rather than infer.
|
|
211
|
+
|
|
212
|
+
Collections routinely hold live keys. Variable **names** carry over as journey secrets; values
|
|
213
|
+
are never read.
|
|
214
|
+
|
|
215
|
+
Journeys arrive as `.yaml.draft` with a placeholder verifier you must replace — only you know what
|
|
216
|
+
success means for your API, and a check that passes without confirming anything proves nothing.
|
|
217
|
+
|
|
218
|
+
### Hand someone the evidence
|
|
219
|
+
|
|
220
|
+
```bash
|
|
221
|
+
npx xitok pack .xitok/runs/<run>/report.json
|
|
222
|
+
```
|
|
223
|
+
|
|
224
|
+
Writes a folder that answers the four questions a sceptical reader asks, in the order they ask
|
|
225
|
+
them — **what was tested**, **what happened**, **where data went**, **who vouches** — plus the
|
|
226
|
+
signed report and every transcript.
|
|
227
|
+
|
|
228
|
+
Nothing in it needs XitOK. The signature verifies offline from inside the folder, and each page
|
|
229
|
+
states its own limits: a local run says plainly that no egress was recorded, an uninspected run
|
|
230
|
+
says which endpoints are unknown *and that this is a gap in the pack rather than a finding about
|
|
231
|
+
your product*, and a rate is never shown without the interval the trial count actually supports.
|
|
232
|
+
|
|
233
|
+
It is for the people who cannot check your work any other way: an auditor, a customer's security
|
|
234
|
+
reviewer, a buyer in diligence.
|
|
235
|
+
|
|
236
|
+
### Get a second party to vouch for a result
|
|
237
|
+
|
|
238
|
+
A XitOK report is signed by whoever ran it, which shows the content has not changed and nothing
|
|
239
|
+
more. `xitok attest` has [Lyntway](https://lyntway.com) counter-sign the same claim:
|
|
240
|
+
|
|
241
|
+
```bash
|
|
242
|
+
LYNTWAY_API_KEY=... npx xitok attest .xitok/runs/<run>/report.json
|
|
243
|
+
```
|
|
244
|
+
|
|
245
|
+
```
|
|
246
|
+
✓ Attested by Lyntway
|
|
247
|
+
Receipt rcpt_b93bfbb12d602dfdd079ad0a
|
|
248
|
+
Covers sha256:a1d3f2191924ab74…
|
|
249
|
+
Evidence attested by Lyntway
|
|
250
|
+
```
|
|
251
|
+
|
|
252
|
+
The receipt names the report's content digest, so it is bound to exactly one report — and writing
|
|
253
|
+
it back does not disturb the signature, because neither the signature nor the attestation is part
|
|
254
|
+
of what was signed.
|
|
255
|
+
|
|
256
|
+
**What a receipt proves, stated plainly:** Lyntway received this report, naming this content, at
|
|
257
|
+
this time. It is not a claim that Lyntway checked the run. That is still worth having — a
|
|
258
|
+
timestamp and a signature from a party with no stake in the result — and overstating it would cost
|
|
259
|
+
more than it is worth.
|
|
260
|
+
|
|
261
|
+
### Ask XitOK from inside your agent
|
|
262
|
+
|
|
263
|
+
`xitok mcp` serves XitOK's read-only tools over MCP, so an agent can ask what it is about to build
|
|
264
|
+
on before it writes a line:
|
|
265
|
+
|
|
266
|
+
```json
|
|
267
|
+
{ "mcpServers": { "xitok": { "command": "npx", "args": ["-y", "xitok", "mcp"] } } }
|
|
268
|
+
```
|
|
269
|
+
|
|
270
|
+
The agent gets one tool, `xitok_check_docs`: give it a documentation URL and it returns what
|
|
271
|
+
passed, what will trip an agent up, and what to fix first. The answer carries its own limits —
|
|
272
|
+
*"no agents were run, so this is not a XitOK score"* — in the payload, not in documentation the
|
|
273
|
+
agent will never read.
|
|
274
|
+
|
|
275
|
+
The surface is read-only on purpose. A tool that could start billable runs is a tool an agent
|
|
276
|
+
could spend your money with, unattended.
|
|
277
|
+
|
|
278
|
+
### Running a build you already have
|
|
279
|
+
|
|
280
|
+
The Action fetches XitOK from npm by default. Point `command` at a build instead — to pin one you
|
|
281
|
+
made yourself, or to exercise the Action against a checkout:
|
|
282
|
+
|
|
283
|
+
```yaml
|
|
284
|
+
- uses: xitokhq/xitok/action@v0
|
|
285
|
+
with:
|
|
286
|
+
command: node ./apps/cli/dist/index.js
|
|
287
|
+
```
|
|
288
|
+
|
|
289
|
+
XitOK's own CI uses this to run the Action on every pull request, so the Action is tested before a
|
|
290
|
+
release exists rather than after the first customer finds out.
|
|
291
|
+
|
|
292
|
+
### Running on a cloud you already pay for
|
|
293
|
+
|
|
294
|
+
Point XitOK at Microsoft Foundry or Amazon Bedrock and the model spend draws down against
|
|
295
|
+
commitments you already have, under procurement you have already done:
|
|
296
|
+
|
|
297
|
+
```yaml
|
|
298
|
+
provider:
|
|
299
|
+
kind: foundry # foundry | bedrock | direct
|
|
300
|
+
location: acme-ai # Azure resource name, or an AWS region
|
|
301
|
+
models:
|
|
302
|
+
claude-code: acme-opus-prod # required — see below
|
|
303
|
+
```
|
|
304
|
+
|
|
305
|
+
The credential is a single environment variable — `ANTHROPIC_FOUNDRY_API_KEY` or
|
|
306
|
+
`AWS_BEARER_TOKEN_BEDROCK` — which is what a clean room with an isolated home directory can
|
|
307
|
+
carry. The sandbox allowlist follows, so agents reach your cloud and not the vendor.
|
|
308
|
+
|
|
309
|
+
**Pinning a model is required, not optional.** An unpinned alias resolves to the agent CLI's own
|
|
310
|
+
default, and Bedrock quietly falls back to an older or smaller model when that default is not
|
|
311
|
+
enabled in your account. A signed score against a model that never ran is the one result XitOK
|
|
312
|
+
must never produce, so it refuses to start instead.
|
|
313
|
+
|
|
314
|
+
Two differences are recorded in the report rather than smoothed over, because they change what a
|
|
315
|
+
score means: agents on Bedrock have no web search, and a Foundry deployment name is asserted by
|
|
316
|
+
your Azure configuration rather than observed by XitOK. A score is comparable to another only
|
|
317
|
+
when these match.
|
|
318
|
+
|
|
319
|
+
### Model traffic through the Lyntway gateway
|
|
320
|
+
|
|
321
|
+
XitOK does not run its own model gateway. Point it at [Lyntway](https://lyntway.com) and every
|
|
322
|
+
model call an agent makes is metered, capped and recorded with a signed receipt:
|
|
323
|
+
|
|
324
|
+
```yaml
|
|
325
|
+
gateway:
|
|
326
|
+
url: https://gateway.lyntway.com # unset = agents call their providers directly
|
|
327
|
+
tokenEnv: LYNTWAY_API_KEY # the variable holding the key, never the key
|
|
328
|
+
requirePassthrough: true # refuse a trial the gateway altered
|
|
329
|
+
```
|
|
330
|
+
|
|
331
|
+
Each agent's own client is repointed at `<url>/gw/<provider>` and given a bearer token; its
|
|
332
|
+
provider key is dropped from the clean room, and the sandbox allowlist stops admitting the
|
|
333
|
+
provider's host, so there is no route around the thing doing the metering.
|
|
334
|
+
|
|
335
|
+
Two things are worth understanding before you turn it on.
|
|
336
|
+
|
|
337
|
+
**Only model traffic goes this way.** Calls to the API under test stay on XitOK's own egress
|
|
338
|
+
proxy. XitOK measures a stranger's experience of your product, and a governing hop in between
|
|
339
|
+
would make the grade mean something else.
|
|
340
|
+
|
|
341
|
+
**The gateway tenant must be pinned to log-only.** Journeys carry synthetic API keys and example
|
|
342
|
+
addresses on purpose, and Lyntway's default policy blocks a credential and tokenises personal
|
|
343
|
+
data — so a default-configured tenant refuses a normal XitOK prompt outright, and a differently
|
|
344
|
+
configured one could rewrite it and have XitOK grade a run the agent never saw. With
|
|
345
|
+
`requirePassthrough` (the default), XitOK reads Lyntway's own `X-Lyntway-Decision` and
|
|
346
|
+
`X-Lyntway-Mode` on every call and refuses to treat an altered one as a measurement.
|
|
347
|
+
|
|
348
|
+
### The endpoint ledger
|
|
349
|
+
|
|
350
|
+
Name your own API under `inspect` and the report gains an **endpoint ledger**: every call the
|
|
351
|
+
agents actually made, as method, path and status.
|
|
352
|
+
|
|
353
|
+
```
|
|
354
|
+
Endpoints 2 distinct on api.acme.com
|
|
355
|
+
7× POST /v1/send → 201
|
|
356
|
+
3× GET /v1/messages?limit → 404
|
|
357
|
+
```
|
|
358
|
+
|
|
359
|
+
That is observation, not the agent's own account of itself — which is what makes it evidence that
|
|
360
|
+
an endpoint was hallucinated, or that a journey never reached your API at all.
|
|
361
|
+
|
|
362
|
+
XitOK terminates TLS inside the sandbox to see this, so it is **off until you opt a host in**, and
|
|
363
|
+
bounded deliberately:
|
|
364
|
+
|
|
365
|
+
- Only hosts you name. Everything else stays an opaque tunnel.
|
|
366
|
+
- **Model providers can never be inspected**, whatever you put in `inspect`. Your model keys and
|
|
367
|
+
the agent's reasoning are unreadable to XitOK by construction, not by promise.
|
|
368
|
+
- **Method, path and status only.** Headers and bodies are piped through unread, query-string
|
|
369
|
+
values are dropped (the keys are kept), and your secrets are redacted from paths.
|
|
370
|
+
- The certificate authority is minted per run, never written to disk outside the clean room, and
|
|
371
|
+
expires with the run. The proxy only ever holds the leaf certificates for the hosts you named.
|
|
372
|
+
|
|
373
|
+
The ledger always lists the hosts it did *not* inspect, so an empty ledger can never be misread as
|
|
374
|
+
"the agent called nothing". See [ADR 0006](docs/adr/0006-endpoint-ledger-and-tls-inspection.md).
|
|
375
|
+
|
|
376
|
+
## When something fails
|
|
377
|
+
|
|
378
|
+
Every failed trial gets a cause (authentication, wrong endpoint, outdated SDK, security, crash,
|
|
379
|
+
gave up…) from deterministic rules over its evidence, and every run ends with **how to fix** it:
|
|
380
|
+
what to change in your docs, SDK or error messages, quoting the evidence — the endpoint agents
|
|
381
|
+
guessed, the SDK version they installed, the error they hit. Test-setup problems (a missing key)
|
|
382
|
+
are kept apart, so nobody edits their docs to fix a CI secret.
|
|
383
|
+
|
|
384
|
+
If you edit a journey, its results start fresh: comparisons only pool runs of the same journey
|
|
385
|
+
version, so a harder test is never reported as a product regression.
|
|
386
|
+
|
|
387
|
+
## How results stay honest
|
|
388
|
+
|
|
389
|
+
- **Clean room.** Each trial gets a fresh workspace and a fresh `HOME`. No personal agent memory,
|
|
390
|
+
config, MCP servers or credentials are visible. Claude Code runs in `--bare` mode.
|
|
391
|
+
- **Verified, not judged.** A trial passes only when execution-based checks pass (commands, files,
|
|
392
|
+
HTTP, transcript facts). An LLM judge, when enabled, is labeled *judged* and never counts.
|
|
393
|
+
- **Errored ≠ failed.** Missing keys, missing agents, sandbox problems, or an agent that fails
|
|
394
|
+
before making a single tool call (a bad model name, its provider down) are excluded from the
|
|
395
|
+
score and reported separately — they say nothing about your product.
|
|
396
|
+
- **Statistics.** Pass rates come with 95% Wilson confidence intervals.
|
|
397
|
+
- **Secrets never leave.** Keys are redacted at the source, before anything is written.
|
|
398
|
+
- **Signed.** Reports are signed with Ed25519 over canonical JSON; `xitok verify` works offline.
|
|
399
|
+
|
|
400
|
+
The complete, versioned methodology — outcomes, statistics, the score formula, panels, comparisons,
|
|
401
|
+
flakiness and evidence: [docs/METHODOLOGY.md](docs/METHODOLOGY.md). Design: [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md). Every planned
|
|
402
|
+
feature: [docs/FEATURES.md](docs/FEATURES.md). Phases: [docs/ROADMAP.md](docs/ROADMAP.md).
|
|
403
|
+
|
|
404
|
+
## Repository
|
|
405
|
+
|
|
406
|
+
```
|
|
407
|
+
packages/spec schemas for config, journeys and reports (zod → JSON Schema)
|
|
408
|
+
packages/agents headless adapters for each AI coding agent
|
|
409
|
+
packages/core discovery, clean rooms, runner, verifiers, scoring, reports, signing
|
|
410
|
+
apps/cli the `xitok` command (bundles the dashboard)
|
|
411
|
+
apps/web the dashboard (React), served by `xitok ui`
|
|
412
|
+
examples/demo a project that runs with scripted agents and no keys
|
|
413
|
+
```
|
|
414
|
+
|
|
415
|
+
## Development
|
|
416
|
+
|
|
417
|
+
```bash
|
|
418
|
+
pnpm install
|
|
419
|
+
pnpm check # lint + build + typecheck + all tests
|
|
420
|
+
pnpm pack:smoke # the published package, installed and used on its own
|
|
421
|
+
ANTHROPIC_API_KEY=... pnpm --filter @xitok/agents test:integration # real Claude Code, a few cents
|
|
422
|
+
docker build --build-arg AGENT_PACKAGES= -t xitok/runner:test docker/runner
|
|
423
|
+
pnpm --filter @xitok/core exec vitest run test/docker.int.test.ts # real Docker sandbox
|
|
424
|
+
```
|
|
425
|
+
|
|
426
|
+
Requires Node 22+ and pnpm 10, on macOS or Linux (on Windows, use WSL 2). Contributing: [CONTRIBUTING.md](CONTRIBUTING.md). Releases:
|
|
427
|
+
[docs/RELEASING.md](docs/RELEASING.md).
|
|
428
|
+
|
|
429
|
+
## License
|
|
430
|
+
|
|
431
|
+
[Elastic License 2.0](LICENSE). Use it, modify it, redistribute it. The one thing it forbids is
|
|
432
|
+
offering XitOK to third parties as a hosted or managed service.
|