xitok 0.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/CHANGELOG.md ADDED
@@ -0,0 +1,19 @@
1
+ # Changelog
2
+
3
+ Versions follow [Semantic Versioning](https://semver.org). Releases are cut by release-please from
4
+ [Conventional Commits](https://www.conventionalcommits.org); see `docs/RELEASING.md`.
5
+
6
+ ## 0.1.0 (unreleased)
7
+
8
+ First release.
9
+
10
+ - `xitok init`, `discover`, `add`: set up a project from what an agent would find — OpenAPI, docs
11
+ platform, llms.txt, MCP, SDKs — with a library of ready-made journeys.
12
+ - `xitok run`: real agents (Claude Code; Codex, Gemini, OpenCode and Cursor experimental) in clean
13
+ rooms, several trials each, every outcome verified by running it; Wilson intervals; a signed
14
+ report.
15
+ - `xitok ci` and the GitHub Action: change-aware smoke runs on pull requests, pooled baselines,
16
+ Fisher's exact test for regressions, one sticky PR comment, annotations, JUnit.
17
+ - Docker sandbox with an egress allowlist.
18
+ - Model matrix, flakiness detection, an optional LLM judge that never counts, AXIS import.
19
+ - `xitok ui`: a local dashboard.
package/LICENSE ADDED
@@ -0,0 +1,93 @@
1
+ Elastic License 2.0 (ELv2)
2
+
3
+ Copyright (c) 2026 Lynt-X Global
4
+
5
+ ## Acceptance
6
+
7
+ By using the software, you agree to all of the terms and conditions below.
8
+
9
+ ## Copyright License
10
+
11
+ The licensor grants you a non-exclusive, royalty-free, worldwide,
12
+ non-sublicensable, non-transferable license to use, copy, distribute, make
13
+ available, and prepare derivative works of the software, in each case subject to
14
+ the limitations and conditions below.
15
+
16
+ ## Limitations
17
+
18
+ You may not provide the software to third parties as a hosted or managed
19
+ service, where the service provides users with access to any substantial set of
20
+ the features or functionality of the software.
21
+
22
+ You may not move, change, disable, or circumvent the license key functionality
23
+ in the software, and you may not remove or obscure any functionality in the
24
+ software that is protected by the license key.
25
+
26
+ You may not alter, remove, or obscure any licensing, copyright, or other notices
27
+ of the licensor in the software. Any use of the licensor's trademarks is subject
28
+ to applicable law.
29
+
30
+ ## Patents
31
+
32
+ The licensor grants you a license, under any patent claims the licensor can
33
+ license, or becomes able to license, to make, have made, use, sell, offer for
34
+ sale, import and have imported the software, in each case subject to the
35
+ limitations and conditions in this license. This license does not cover any
36
+ patent claims that you cause to be infringed by modifications or additions to
37
+ the software. If you or your company make any written claim that the software
38
+ infringes or contributes to infringement of any patent, your patent license for
39
+ the software granted under these terms ends immediately. If your company makes
40
+ such a claim, your patent license ends immediately for work on behalf of your
41
+ company.
42
+
43
+ ## Notices
44
+
45
+ You must ensure that anyone who gets a copy of any part of the software from you
46
+ also gets a copy of these terms.
47
+
48
+ If you modify the software, you must include in any modified copies of the
49
+ software prominent notices stating that you have modified the software.
50
+
51
+ ## No Other Rights
52
+
53
+ These terms do not imply any licenses other than those expressly granted in
54
+ these terms.
55
+
56
+ ## Termination
57
+
58
+ If you use the software in violation of these terms, such use is not licensed,
59
+ and your licenses will automatically terminate. If the licensor provides you with
60
+ a notice of your violation, and you cease all violation of this license no later
61
+ than 30 days after you receive that notice, your licenses will be reinstated
62
+ retroactively. However, if you violate these terms after such reinstatement, any
63
+ additional violation of these terms will cause your licenses to terminate
64
+ automatically and permanently.
65
+
66
+ ## No Liability
67
+
68
+ *As far as the law allows, the software comes as is, without any warranty or
69
+ condition, and the licensor will not be liable to you for any damages arising out
70
+ of these terms or the use or nature of the software, under any kind of legal
71
+ claim.*
72
+
73
+ ## Definitions
74
+
75
+ The **licensor** is the entity offering these terms, and the **software** is the
76
+ software the licensor makes available under these terms, including any portion of
77
+ it.
78
+
79
+ **you** refers to the individual or entity agreeing to these terms.
80
+
81
+ **your company** is any legal entity, sole proprietorship, or other kind of
82
+ organization that you work for, plus all organizations that have control over,
83
+ are under the control of, or are under common control with that organization.
84
+ **control** means ownership of substantially all the assets of an entity, or the
85
+ power to direct its management and policies by vote, contract, or otherwise.
86
+ Control can be direct or indirect.
87
+
88
+ **your licenses** are all the licenses granted to you for the software under
89
+ these terms.
90
+
91
+ **use** means anything you do with the software requiring one of your licenses.
92
+
93
+ **trademark** means trademarks, service marks, and similar rights.
package/README.md ADDED
@@ -0,0 +1,432 @@
1
+ # XitOK
2
+
3
+ **XitOK** (pronounced *exit OK*) tests your API with real AI coding agents — Claude Code, Codex,
4
+ Gemini, OpenCode, Cursor — the way a brand-new developer's agent would meet it: in a clean room,
5
+ many times, with every outcome **verified by running it**, and a signed report you can prove.
6
+ Part of the [Lyntway](https://lyntway.com) family.
7
+
8
+ ```
9
+ XitOK score 57.5 / 100 grade D
10
+ ────────────────────────────────────────────────────────────────────
11
+ Journey claude-code codex
12
+ Make a first authenticated call 100% 4/4 [51–100] 25% 1/4 [5–70]
13
+ Verify a webhook signature 100% 4/4 [51–100] 0% 0/4 [0–49]
14
+ ```
15
+
16
+ ## Quick start
17
+
18
+ ```bash
19
+ npx xitok init # discovers your API, writes xitok.yaml and a first journey
20
+ npx xitok doctor # checks agents and keys
21
+ npx xitok run # runs journeys × agents × trials and scores the result
22
+ ```
23
+
24
+ Try it with no API keys: `cd examples/demo && npx xitok run` (two scripted agents stand in for real ones).
25
+
26
+ ## Commands
27
+
28
+ | Command | What it does |
29
+ |---|---|
30
+ | `xitok init` | Discovers your API (OpenAPI, docs platform, llms.txt, MCP, SDKs), writes `xitok.yaml`, a first journey, and a webhook-security journey when your spec has webhooks |
31
+ | `xitok check <docs-url>` | Free preview of how ready a public docs site is for agents: readable without JavaScript, llms.txt, Markdown, OpenAPI, robots.txt, page weight… with what to fix first. Static checks, no agents, no signup |
32
+ | `xitok discover` | Shows what an agent would find in your repo — and, with `--docs-url`, every public-docs check from `xitok check` |
33
+ | `xitok add [template]` | Adds a ready-made journey: first call, no hardcoded secrets, secret key kept server-side, webhook signatures, current SDK version, idempotent create, rate-limit backoff (tested against a real 429), outcome on the live API, MCP tool use (`--list`) |
34
+ | `xitok run` | Runs every journey with every agent, several trials each, in clean rooms, with a live progress matrix; `--models` for a model matrix |
35
+ | `xitok watch` | Re-runs the journeys a change affects every time you save (1 trial by default, a per-session spend cap, runs never overlap) |
36
+ | `xitok setup-agent <claude-code\|cursor\|codex>` | Teaches your coding agent to run XitOK and fix what it finds (a skill, a Cursor rule, or an AGENTS.md section). Claude Code plugin: `/plugin marketplace add xitokhq/xitok` then `/plugin install xitok@xitok` for `/xitok:run`, `/xitok:fix`, `/xitok:journey` |
37
+ | `xitok setup-ci <provider>` | Writes the XitOK pipeline for GitHub Actions (pull requests, nightly baseline, new models), GitLab CI, CircleCI or Buildkite |
38
+ | `xitok ci` | CI mode: runs only the journeys a change affects (or `--mode full`), compares with the baseline, annotates the PR, posts one sticky comment, writes the job summary |
39
+ | `xitok draft` | Claude proposes new journeys from your spec and docs; only ones with a check that runs are kept, written as drafts you review and rename before they count |
40
+ | `xitok import axis [path]` | Imports Netlify AXIS scenarios: the rubric becomes a judge check, checks that run are derived where the rubric states something checkable, and the rest become drafts that say what to add |
41
+ | `xitok flaky` | Journey × agent pairs that pass only some of the time across recent runs, with the most common cause |
42
+ | `xitok generate <llms-txt\|agents-md\|skill>` | Drafts `llms.txt`, an `AGENTS.md` for your users' repos, or a Claude Code skill — from your OpenAPI spec, the rules your journeys enforce, and the mistakes agents made in your latest run |
43
+ | `xitok docs init <mintlify\|scalar\|fern\|readme\|in-repo>` | Writes a docs site into your repository for an API that has none — drafted from your spec, the rules your journeys enforce and what agents got wrong. No account, nothing published |
44
+ | `xitok spec draft` | Reads your code and drafts an OpenAPI document from the routes it finds, marked a draft in three places because nobody has checked it yet |
45
+ | `xitok upgrade --from <old spec>` | Compares two versions of your API and writes the migration as a skill agents follow: what breaks, what to change, and what it could not see. `--fail-on-breaking` stops a release that breaks callers |
46
+ | `xitok badge` | An SVG badge for your README: the score and grade, or `--agent` for one agent's pass rate; says "custom" unless the standard panel ran. `xitok ci` writes one too |
47
+ | `xitok models check` | What's new at Anthropic, OpenAI and Google since the last check; `--run` tests each new model side by side with the one you pin (daily workflow: `docs/ci/xitok-new-models.yml`) |
48
+ | `xitok compare <baseline> [current]` | Which journey × agent pairs regressed or improved, with Fisher's exact test so noise isn't called a regression |
49
+ | `xitok baseline set / show` | Promotes a run to the baseline that `xitok ci` compares against |
50
+ | `xitok ui` | Opens the dashboard for this project's runs: score trend, journeys × agents matrix, every trial's checks and full transcript — and a journey editor with live validation, a preview of what will run, and one-click promotion of reviewed drafts (`--read-only` to turn editing off). Local only (127.0.0.1) |
51
+ | `xitok report [path]` | Shows a past run (terminal, `--format md` for PR comments, `--format json`, or `--format junit`) |
52
+ | `xitok verify <report>` | Verifies a signed report offline; `--trust <keyId>` to also check who signed it |
53
+ | `xitok doctor` | Checks Node, config, agents, keys, the signing key, and Docker when the sandbox is on |
54
+ | `xitok schema` | Writes JSON Schemas so editors autocomplete `xitok.yaml` and journeys |
55
+ | `xitok completion <shell>` | Shell completions for bash, zsh and fish — commands, options, and journey ids from your project |
56
+
57
+ Every command accepts `--json`.
58
+
59
+ **Exit codes** — `0` ok · `1` score below `--fail-under`, a significant regression, or signature invalid · `2` invalid input or config · `3` nothing could run / interrupted.
60
+
61
+ ## In CI
62
+
63
+ ```yaml
64
+ # .github/workflows/xitok.yml — full examples in docs/ci/
65
+ - uses: xitokhq/xitok/action@v0
66
+ with:
67
+ mode: changed # only journeys whose `watch` globs match the diff
68
+ fail-under: 70
69
+ env:
70
+ ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
71
+ ```
72
+
73
+ `xitok ci` recognises GitHub Actions, GitLab CI (a note on the merge request with
74
+ `XITOK_GITLAB_TOKEN`, results in the merge request's test report), Buildkite (the summary as the
75
+ build annotation) and CircleCI; `xitok setup-ci <provider>` writes the pipeline.
76
+
77
+ On a pull request XitOK runs a small smoke suite for the journeys the change touches, compares
78
+ with the last few `main` baselines pooled, and posts one comment that it updates on every push. A
79
+ drop only fails the build when it is statistically significant (`ci.regressionAlpha`, default
80
+ 0.1); smaller drops are reported as *dropped* without failing. Pairs that pass only some of the
81
+ time are labeled *flaky*: pooling gives them a baseline that is that mix, so they don't raise false
82
+ alarms. Nightly, `mode: full` runs everything and adds itself to the baseline history.
83
+
84
+ ## Panels: which agents, which exact models
85
+
86
+ A score only means something if you know exactly what produced it. A **panel** is a versioned
87
+ lineup with pinned model versions:
88
+
89
+ ```yaml
90
+ panel: standard # XitOK Standard Panel 2026.09: Claude Code + claude-sonnet-5,
91
+ # Codex + gpt-5.3-codex, Gemini CLI + gemini-3.5-flash
92
+ ```
93
+
94
+ `panel: frontier` runs the strongest models instead; `--panel` works on `run`, `ci` and `watch`.
95
+ Public scores and badges use the standard panel. Choosing your own `agents` makes a *custom
96
+ lineup*, and reports say so. Every report records the model each agent actually ran (including a
97
+ switch mid-run), and `xitok doctor` warns about model aliases that can change silently and about
98
+ models whose origin your security team may need to review ([ADR 0003](docs/adr/0003-model-panels-and-origin-policy.md)).
99
+
100
+ ## Model matrix
101
+
102
+ Which model does your API work best with? List several and each becomes its own column, with a
103
+ head-to-head that says which differences are real:
104
+
105
+ ```yaml
106
+ agents:
107
+ - id: claude-code
108
+ models: [claude-opus-5, claude-sonnet-5, claude-haiku-4-5]
109
+ ```
110
+
111
+ Or ad hoc: `xitok run --models claude-code:claude-opus-5,claude-code:claude-haiku-4-5`. Pull
112
+ request smoke runs use only the first model.
113
+
114
+ ## Agent-to-agent (A2A)
115
+
116
+ If you ship an agent, test whether other agents can use it. An `a2a` journey gives the coding agent
117
+ a bridge to your agent over the Agent2Agent protocol (1.0 and 0.3), and XitOK verifies from its own
118
+ record of the traffic — not the coding agent's claim — that your agent took the task to the state
119
+ you expect:
120
+
121
+ ```yaml
122
+ context:
123
+ policy: a2a
124
+ agentCardUrl: https://agent.acme.com # or the card's URL
125
+ agentHeaders: { authorization: "Bearer ${ACME_AGENT_TOKEN}" }
126
+ verify:
127
+ - type: a2a
128
+ expectState: completed
129
+ replyContains: confirmed
130
+ ```
131
+
132
+ `xitok add a2a-task` starts one; `xitok check` reports whether your site publishes an agent card.
133
+
134
+ ## LLM judge (optional, never counted)
135
+
136
+ Some qualities can't be checked by running code — is it idiomatic, is the explanation right? Add a
137
+ judge check and Claude grades the work against your rubric:
138
+
139
+ ```yaml
140
+ verify:
141
+ - type: command # verified: decides pass or fail
142
+ run: node send.js
143
+ - type: judge # judged: recorded, never counted
144
+ rubric: Uses the official Acme SDK, not raw HTTP, and reads the key from the environment.
145
+ files: ["**/*.js"] # optional; default is every text file in the workspace
146
+ ```
147
+
148
+ The judge uses `XITOK_JUDGE_API_KEY` or `ANTHROPIC_API_KEY` and `judge.model` (default
149
+ `claude-sonnet-5`); without a key, judged checks are skipped. Everything it is shown is redacted
150
+ first, and agent output is marked as data so the agent can't instruct its own grader. Every report
151
+ that used a judge says how often it agreed with the execution-verified outcome — the reason XitOK
152
+ scores on execution, not opinion. Judge tokens are recorded in the check's evidence; their cost is
153
+ not part of the agent spend.
154
+
155
+ ## Sandbox
156
+
157
+ By default trials run as local processes in a clean room. For untrusted code, or when you want a
158
+ hard network boundary, run every agent and every check in a container:
159
+
160
+ ```yaml
161
+ sandbox:
162
+ kind: docker
163
+ image: xitokhq/runner:latest # docker/runner/Dockerfile
164
+ network: allowlist # allowlist | none | open
165
+ allow: [api.acme.com] # added to the agent's model API, package registries and your docs
166
+ inspect: [api.acme.com] # optional: record which endpoints agents actually called
167
+ ```
168
+
169
+ With `allowlist`, the container sits on an internal Docker network whose only way out is an egress
170
+ proxy that admits listed hosts on ports 80/443 and refuses everything else; direct connections have
171
+ no route. Every refused host is recorded in the report. Containers run as your user, with all Linux
172
+ capabilities dropped, `no-new-privileges`, and CPU, memory and process limits; the project is
173
+ mounted read-only. Secrets reach the container through a private env file deleted after each
174
+ command (or, for multi-line values, the docker CLI's own environment) and never appear in process
175
+ arguments.
176
+
177
+ ### Shared journey libraries
178
+
179
+ One project's journeys can be inherited by others in the same organization — a security suite
180
+ every team must pass, kept private to you rather than published:
181
+
182
+ ```yaml
183
+ libraries:
184
+ - project: prj_security # a project in your org, marked as a library
185
+ journeys: [webhook-*] # optional; omit to inherit all of them
186
+ ```
187
+
188
+ Your own journeys win on a clash, and so does an earlier library: a shared suite can add checks
189
+ but never silently replace one of yours. A reference that matches no journey is an error rather
190
+ than a quiet no-op, because a run that skipped a security suite still reports as passed.
191
+
192
+ Libraries are resolved by XitOK Cloud, which is the only place that can see another project. A
193
+ local `xitok run` says so instead of testing less than the file asks for.
194
+
195
+ ### Already have a Postman collection?
196
+
197
+ ```bash
198
+ npx xitok import-postman ./acme.postman_collection.json
199
+ ```
200
+
201
+ Each folder becomes a journey, and every request becomes a line of what that journey has to
202
+ achieve. Most teams have a collection and no OpenAPI spec, so this is usually the fastest way in.
203
+
204
+ **The collection never reaches the agent.** It tells XitOK what the API is and which endpoints
205
+ exist; the agent still has to work out the endpoints, the auth and the payloads from your
206
+ documentation. A prompt containing the method and path would measure replay, not discovery.
207
+
208
+ Those recorded endpoints are useful twice: they become the ground truth the
209
+ [endpoint ledger](#the-endpoint-ledger) is compared against, so an agent calling something that
210
+ does not exist is a finding you can prove rather than infer.
211
+
212
+ Collections routinely hold live keys. Variable **names** carry over as journey secrets; values
213
+ are never read.
214
+
215
+ Journeys arrive as `.yaml.draft` with a placeholder verifier you must replace — only you know what
216
+ success means for your API, and a check that passes without confirming anything proves nothing.
217
+
218
+ ### Hand someone the evidence
219
+
220
+ ```bash
221
+ npx xitok pack .xitok/runs/<run>/report.json
222
+ ```
223
+
224
+ Writes a folder that answers the four questions a sceptical reader asks, in the order they ask
225
+ them — **what was tested**, **what happened**, **where data went**, **who vouches** — plus the
226
+ signed report and every transcript.
227
+
228
+ Nothing in it needs XitOK. The signature verifies offline from inside the folder, and each page
229
+ states its own limits: a local run says plainly that no egress was recorded, an uninspected run
230
+ says which endpoints are unknown *and that this is a gap in the pack rather than a finding about
231
+ your product*, and a rate is never shown without the interval the trial count actually supports.
232
+
233
+ It is for the people who cannot check your work any other way: an auditor, a customer's security
234
+ reviewer, a buyer in diligence.
235
+
236
+ ### Get a second party to vouch for a result
237
+
238
+ A XitOK report is signed by whoever ran it, which shows the content has not changed and nothing
239
+ more. `xitok attest` has [Lyntway](https://lyntway.com) counter-sign the same claim:
240
+
241
+ ```bash
242
+ LYNTWAY_API_KEY=... npx xitok attest .xitok/runs/<run>/report.json
243
+ ```
244
+
245
+ ```
246
+ ✓ Attested by Lyntway
247
+ Receipt rcpt_b93bfbb12d602dfdd079ad0a
248
+ Covers sha256:a1d3f2191924ab74…
249
+ Evidence attested by Lyntway
250
+ ```
251
+
252
+ The receipt names the report's content digest, so it is bound to exactly one report — and writing
253
+ it back does not disturb the signature, because neither the signature nor the attestation is part
254
+ of what was signed.
255
+
256
+ **What a receipt proves, stated plainly:** Lyntway received this report, naming this content, at
257
+ this time. It is not a claim that Lyntway checked the run. That is still worth having — a
258
+ timestamp and a signature from a party with no stake in the result — and overstating it would cost
259
+ more than it is worth.
260
+
261
+ ### Ask XitOK from inside your agent
262
+
263
+ `xitok mcp` serves XitOK's read-only tools over MCP, so an agent can ask what it is about to build
264
+ on before it writes a line:
265
+
266
+ ```json
267
+ { "mcpServers": { "xitok": { "command": "npx", "args": ["-y", "xitok", "mcp"] } } }
268
+ ```
269
+
270
+ The agent gets one tool, `xitok_check_docs`: give it a documentation URL and it returns what
271
+ passed, what will trip an agent up, and what to fix first. The answer carries its own limits —
272
+ *"no agents were run, so this is not a XitOK score"* — in the payload, not in documentation the
273
+ agent will never read.
274
+
275
+ The surface is read-only on purpose. A tool that could start billable runs is a tool an agent
276
+ could spend your money with, unattended.
277
+
278
+ ### Running a build you already have
279
+
280
+ The Action fetches XitOK from npm by default. Point `command` at a build instead — to pin one you
281
+ made yourself, or to exercise the Action against a checkout:
282
+
283
+ ```yaml
284
+ - uses: xitokhq/xitok/action@v0
285
+ with:
286
+ command: node ./apps/cli/dist/index.js
287
+ ```
288
+
289
+ XitOK's own CI uses this to run the Action on every pull request, so the Action is tested before a
290
+ release exists rather than after the first customer finds out.
291
+
292
+ ### Running on a cloud you already pay for
293
+
294
+ Point XitOK at Microsoft Foundry or Amazon Bedrock and the model spend draws down against
295
+ commitments you already have, under procurement you have already done:
296
+
297
+ ```yaml
298
+ provider:
299
+ kind: foundry # foundry | bedrock | direct
300
+ location: acme-ai # Azure resource name, or an AWS region
301
+ models:
302
+ claude-code: acme-opus-prod # required — see below
303
+ ```
304
+
305
+ The credential is a single environment variable — `ANTHROPIC_FOUNDRY_API_KEY` or
306
+ `AWS_BEARER_TOKEN_BEDROCK` — which is what a clean room with an isolated home directory can
307
+ carry. The sandbox allowlist follows, so agents reach your cloud and not the vendor.
308
+
309
+ **Pinning a model is required, not optional.** An unpinned alias resolves to the agent CLI's own
310
+ default, and Bedrock quietly falls back to an older or smaller model when that default is not
311
+ enabled in your account. A signed score against a model that never ran is the one result XitOK
312
+ must never produce, so it refuses to start instead.
313
+
314
+ Two differences are recorded in the report rather than smoothed over, because they change what a
315
+ score means: agents on Bedrock have no web search, and a Foundry deployment name is asserted by
316
+ your Azure configuration rather than observed by XitOK. A score is comparable to another only
317
+ when these match.
318
+
319
+ ### Model traffic through the Lyntway gateway
320
+
321
+ XitOK does not run its own model gateway. Point it at [Lyntway](https://lyntway.com) and every
322
+ model call an agent makes is metered, capped and recorded with a signed receipt:
323
+
324
+ ```yaml
325
+ gateway:
326
+ url: https://gateway.lyntway.com # unset = agents call their providers directly
327
+ tokenEnv: LYNTWAY_API_KEY # the variable holding the key, never the key
328
+ requirePassthrough: true # refuse a trial the gateway altered
329
+ ```
330
+
331
+ Each agent's own client is repointed at `<url>/gw/<provider>` and given a bearer token; its
332
+ provider key is dropped from the clean room, and the sandbox allowlist stops admitting the
333
+ provider's host, so there is no route around the thing doing the metering.
334
+
335
+ Two things are worth understanding before you turn it on.
336
+
337
+ **Only model traffic goes this way.** Calls to the API under test stay on XitOK's own egress
338
+ proxy. XitOK measures a stranger's experience of your product, and a governing hop in between
339
+ would make the grade mean something else.
340
+
341
+ **The gateway tenant must be pinned to log-only.** Journeys carry synthetic API keys and example
342
+ addresses on purpose, and Lyntway's default policy blocks a credential and tokenises personal
343
+ data — so a default-configured tenant refuses a normal XitOK prompt outright, and a differently
344
+ configured one could rewrite it and have XitOK grade a run the agent never saw. With
345
+ `requirePassthrough` (the default), XitOK reads Lyntway's own `X-Lyntway-Decision` and
346
+ `X-Lyntway-Mode` on every call and refuses to treat an altered one as a measurement.
347
+
348
+ ### The endpoint ledger
349
+
350
+ Name your own API under `inspect` and the report gains an **endpoint ledger**: every call the
351
+ agents actually made, as method, path and status.
352
+
353
+ ```
354
+ Endpoints 2 distinct on api.acme.com
355
+ 7× POST /v1/send → 201
356
+ 3× GET /v1/messages?limit → 404
357
+ ```
358
+
359
+ That is observation, not the agent's own account of itself — which is what makes it evidence that
360
+ an endpoint was hallucinated, or that a journey never reached your API at all.
361
+
362
+ XitOK terminates TLS inside the sandbox to see this, so it is **off until you opt a host in**, and
363
+ bounded deliberately:
364
+
365
+ - Only hosts you name. Everything else stays an opaque tunnel.
366
+ - **Model providers can never be inspected**, whatever you put in `inspect`. Your model keys and
367
+ the agent's reasoning are unreadable to XitOK by construction, not by promise.
368
+ - **Method, path and status only.** Headers and bodies are piped through unread, query-string
369
+ values are dropped (the keys are kept), and your secrets are redacted from paths.
370
+ - The certificate authority is minted per run, never written to disk outside the clean room, and
371
+ expires with the run. The proxy only ever holds the leaf certificates for the hosts you named.
372
+
373
+ The ledger always lists the hosts it did *not* inspect, so an empty ledger can never be misread as
374
+ "the agent called nothing". See [ADR 0006](docs/adr/0006-endpoint-ledger-and-tls-inspection.md).
375
+
376
+ ## When something fails
377
+
378
+ Every failed trial gets a cause (authentication, wrong endpoint, outdated SDK, security, crash,
379
+ gave up…) from deterministic rules over its evidence, and every run ends with **how to fix** it:
380
+ what to change in your docs, SDK or error messages, quoting the evidence — the endpoint agents
381
+ guessed, the SDK version they installed, the error they hit. Test-setup problems (a missing key)
382
+ are kept apart, so nobody edits their docs to fix a CI secret.
383
+
384
+ If you edit a journey, its results start fresh: comparisons only pool runs of the same journey
385
+ version, so a harder test is never reported as a product regression.
386
+
387
+ ## How results stay honest
388
+
389
+ - **Clean room.** Each trial gets a fresh workspace and a fresh `HOME`. No personal agent memory,
390
+ config, MCP servers or credentials are visible. Claude Code runs in `--bare` mode.
391
+ - **Verified, not judged.** A trial passes only when execution-based checks pass (commands, files,
392
+ HTTP, transcript facts). An LLM judge, when enabled, is labeled *judged* and never counts.
393
+ - **Errored ≠ failed.** Missing keys, missing agents, sandbox problems, or an agent that fails
394
+ before making a single tool call (a bad model name, its provider down) are excluded from the
395
+ score and reported separately — they say nothing about your product.
396
+ - **Statistics.** Pass rates come with 95% Wilson confidence intervals.
397
+ - **Secrets never leave.** Keys are redacted at the source, before anything is written.
398
+ - **Signed.** Reports are signed with Ed25519 over canonical JSON; `xitok verify` works offline.
399
+
400
+ The complete, versioned methodology — outcomes, statistics, the score formula, panels, comparisons,
401
+ flakiness and evidence: [docs/METHODOLOGY.md](docs/METHODOLOGY.md). Design: [docs/ARCHITECTURE.md](docs/ARCHITECTURE.md). Every planned
402
+ feature: [docs/FEATURES.md](docs/FEATURES.md). Phases: [docs/ROADMAP.md](docs/ROADMAP.md).
403
+
404
+ ## Repository
405
+
406
+ ```
407
+ packages/spec schemas for config, journeys and reports (zod → JSON Schema)
408
+ packages/agents headless adapters for each AI coding agent
409
+ packages/core discovery, clean rooms, runner, verifiers, scoring, reports, signing
410
+ apps/cli the `xitok` command (bundles the dashboard)
411
+ apps/web the dashboard (React), served by `xitok ui`
412
+ examples/demo a project that runs with scripted agents and no keys
413
+ ```
414
+
415
+ ## Development
416
+
417
+ ```bash
418
+ pnpm install
419
+ pnpm check # lint + build + typecheck + all tests
420
+ pnpm pack:smoke # the published package, installed and used on its own
421
+ ANTHROPIC_API_KEY=... pnpm --filter @xitok/agents test:integration # real Claude Code, a few cents
422
+ docker build --build-arg AGENT_PACKAGES= -t xitok/runner:test docker/runner
423
+ pnpm --filter @xitok/core exec vitest run test/docker.int.test.ts # real Docker sandbox
424
+ ```
425
+
426
+ Requires Node 22+ and pnpm 10, on macOS or Linux (on Windows, use WSL 2). Contributing: [CONTRIBUTING.md](CONTRIBUTING.md). Releases:
427
+ [docs/RELEASING.md](docs/RELEASING.md).
428
+
429
+ ## License
430
+
431
+ [Elastic License 2.0](LICENSE). Use it, modify it, redistribute it. The one thing it forbids is
432
+ offering XitOK to third parties as a hosted or managed service.