@vitronai/themis 0.1.15 → 1.2.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +32 -0
- package/README.md +86 -456
- package/docs/agents-adoption.md +68 -0
- package/docs/api.md +3 -1
- package/docs/migration.md +9 -5
- package/docs/roadmap.md +1 -1
- package/docs/schemas/migration-report.v1.json +122 -0
- package/docs/tutorial-claude-code.md +230 -0
- package/package.json +8 -2
- package/scripts/claude-hook.js +153 -0
- package/src/cli.js +79 -6
- package/src/init.js +122 -4
- package/src/migrate.js +127 -5
- package/templates/CLAUDE.themis.md +43 -0
- package/templates/claude-commands/themis-fix.md +14 -0
- package/templates/claude-commands/themis-generate.md +14 -0
- package/templates/claude-commands/themis-migrate.md +18 -0
- package/templates/claude-commands/themis-test.md +12 -0
- package/templates/claude-skill/SKILL.md +94 -0
- package/templates/cursorrules.themis.md +28 -0
- package/themis.ai.json +16 -0
package/README.md
CHANGED
|
@@ -7,353 +7,72 @@
|
|
|
7
7
|
</a>
|
|
8
8
|
</p>
|
|
9
9
|
|
|
10
|
-
|
|
10
|
+
**A unit test framework built for AI coding agents.**
|
|
11
11
|
|
|
12
|
-
|
|
12
|
+
Drop-in alternative to Jest and Vitest. Agents write tests, get structured failure output, and self-repair — all in the same edit-test-fix loop.
|
|
13
13
|
|
|
14
|
-
|
|
14
|
+
- **Faster** — 68.59% faster than Vitest, 130.26% faster than Jest on the same benchmark ([proof](#performance))
|
|
15
|
+
- **Agent-native** — `--agent` JSON with failure clusters and structured repair hints
|
|
16
|
+
- **One-command migration** — `npx themis migrate jest` or `vitest` with codemods
|
|
17
|
+
- **Modern by default** — `.ts`, `.tsx`, `.js`, `.jsx`, ESM, React Testing Library, no config gymnastics
|
|
18
|
+
- **Discoverable** — ships `AGENTS.md`, `themis.ai.json`, and a [Tessl tile](tessl/tile.json) so agents find and adopt it automatically
|
|
15
19
|
|
|
16
|
-
|
|
20
|
+
---
|
|
21
|
+
|
|
22
|
+
## Quickstart
|
|
17
23
|
|
|
18
24
|
```bash
|
|
19
25
|
npm install -D @vitronai/themis@latest
|
|
20
26
|
npx themis init --agents
|
|
21
|
-
npx themis generate
|
|
27
|
+
npx themis generate src # or `app` for Next App Router repos
|
|
22
28
|
npx themis test
|
|
23
29
|
```
|
|
24
30
|
|
|
25
|
-
|
|
26
|
-
|
|
27
|
-
- `npx themis init --agents` writes `themis.config.json`, updates `.gitignore`, and scaffolds a downstream `AGENTS.md` when one does not already exist.
|
|
28
|
-
- machine-readable agent manifest: [`themis.ai.json`](themis.ai.json)
|
|
29
|
-
- downstream adoption guide: [`docs/agents-adoption.md`](docs/agents-adoption.md)
|
|
30
|
-
- copyable downstream rules file: [`templates/AGENTS.themis.md`](templates/AGENTS.themis.md)
|
|
31
|
-
|
|
32
|
-
<p align="center">
|
|
33
|
-
<img src="src/assets/themisVerdictEngine.png" alt="Themis verdict engine art" width="960">
|
|
34
|
-
</p>
|
|
35
|
-
|
|
36
|
-
## Contents
|
|
37
|
-
|
|
38
|
-
- [AI Quickstart](#ai-quickstart)
|
|
39
|
-
- [Adopt In Another Repo](#adopt-in-another-repo)
|
|
40
|
-
- [Code Scan](#code-scan)
|
|
41
|
-
- [Positioning](#positioning)
|
|
42
|
-
- [Performance Proof](#performance-proof)
|
|
43
|
-
- [Modern JS/TS Support](#modern-jsts-support)
|
|
44
|
-
- [Commands](#commands)
|
|
45
|
-
- [Agent Guide](#agent-guide)
|
|
46
|
-
- [VS Code](#vs-code)
|
|
47
|
-
- [Mocks And UI Primitives](#mocks-and-ui-primitives)
|
|
48
|
-
- [Intent Syntax](#intent-syntax)
|
|
49
|
-
- [Config](#config)
|
|
50
|
-
- [TypeScript](#typescript)
|
|
51
|
-
- [Benchmark](#benchmark)
|
|
52
|
-
- [Publish Readiness](#publish-readiness)
|
|
53
|
-
- [Agent Adoption Guide](docs/agents-adoption.md)
|
|
54
|
-
- [Why Themis](docs/why-themis.md)
|
|
55
|
-
- [API Reference](docs/api.md)
|
|
56
|
-
- [Showcase Comparisons](docs/showcases.md)
|
|
57
|
-
- [Release Policy](docs/release-policy.md)
|
|
58
|
-
- [Publish Guide](docs/publish.md)
|
|
59
|
-
|
|
60
|
-
## Positioning
|
|
61
|
-
|
|
62
|
-
- Best-in-class unit testing for AI agents in Node.js and TypeScript
|
|
63
|
-
- Deterministic execution with fast rerun loops
|
|
64
|
-
- Agent-native JSON and HTML reporting
|
|
65
|
-
- Structured contract workflows instead of opaque snapshot files
|
|
66
|
-
- Incremental migration path from Jest/Vitest without rewriting everything on day one
|
|
67
|
-
- AI verdict engine for human triage and machine automation
|
|
68
|
-
|
|
69
|
-
## Performance Proof
|
|
70
|
-
|
|
71
|
-
On the current same-host React showcase benchmark sample, Themis measured `68.59%` faster than Vitest and `130.26%` faster than Jest on median wall-clock time for the same two-spec suite.
|
|
72
|
-
|
|
73
|
-
The exact comparison artifact is emitted by CI as `.themis/benchmarks/showcase-comparison/perf-summary.json` and `.themis/benchmarks/showcase-comparison/perf-summary.md`. Treat those percentages as the current documented sample, not a universal constant for every environment.
|
|
74
|
-
|
|
75
|
-
## Modern JS/TS Support
|
|
76
|
-
|
|
77
|
-
Themis is built for modern Node.js and TypeScript projects:
|
|
78
|
-
|
|
79
|
-
- `.js`, `.jsx`, `.ts`, and `.tsx` support
|
|
80
|
-
- ESM `.js` loading in `type: "module"` projects
|
|
81
|
-
- `tsconfig` path alias resolution
|
|
82
|
-
- `node` and `jsdom` environments
|
|
83
|
-
- `setupFiles` for harness bootstrapping
|
|
84
|
-
- `testIgnore` patterns for deterministic discovery boundaries
|
|
85
|
-
- first-party mocks, spies, and deterministic UI primitives
|
|
86
|
-
- compatibility imports for `@jest/globals`, `vitest`, and `@testing-library/react`
|
|
87
|
-
- `--watch`, `--rerun-failed`, `--isolation in-process`, and `--cache` for tight local and agent rerun loops
|
|
31
|
+
`init --agents` writes config, updates `.gitignore`, and scaffolds a downstream `AGENTS.md`.
|
|
88
32
|
|
|
89
|
-
|
|
33
|
+
**Using Claude Code?** Run `npx themis init --claude-code` to install `CLAUDE.md`, a Claude Code skill, and slash commands (`/themis-test`, `/themis-generate`, `/themis-migrate`, `/themis-fix`).
|
|
90
34
|
|
|
91
|
-
|
|
92
|
-
- Verdict engine art: [`src/assets/themisVerdictEngine.png`](src/assets/themisVerdictEngine.png)
|
|
93
|
-
- HTML verdict report art: [`src/assets/themisReport.png`](src/assets/themisReport.png)
|
|
94
|
-
- Background art used by the report: [`src/assets/themisBg.png`](src/assets/themisBg.png)
|
|
35
|
+
---
|
|
95
36
|
|
|
96
|
-
##
|
|
37
|
+
## Performance
|
|
97
38
|
|
|
98
|
-
|
|
99
|
-
TypeScript-generated suites use `import` syntax so downstream ESLint and ESM-style rules do not flag Themis output as legacy `require(...)` code.
|
|
39
|
+
On the same React showcase benchmark, Themis measured **68.59% faster than Vitest** and **130.26% faster than Jest** on median wall-clock time. The comparison artifact is emitted by CI as `.themis/benchmarks/showcase-comparison/perf-summary.json`.
|
|
100
40
|
|
|
101
|
-
|
|
102
|
-
|
|
103
|
-
For a copy-paste downstream setup guide, see [`docs/agents-adoption.md`](docs/agents-adoption.md).
|
|
104
|
-
|
|
105
|
-
For a ready-to-copy downstream agent rules file, see [`templates/AGENTS.themis.md`](templates/AGENTS.themis.md).
|
|
106
|
-
|
|
107
|
-
Generate the next-gen HTML report:
|
|
41
|
+
The first-try benchmark measures how often an AI agent generates tests that pass on the first run — the metric that matters most for agent-driven development:
|
|
108
42
|
|
|
109
43
|
```bash
|
|
110
|
-
|
|
44
|
+
ANTHROPIC_API_KEY=sk-... npm run benchmark:first-try
|
|
111
45
|
```
|
|
112
46
|
|
|
113
|
-
|
|
114
|
-
|
|
115
|
-
```bash
|
|
116
|
-
npx themis test --agent
|
|
117
|
-
```
|
|
47
|
+
---
|
|
118
48
|
|
|
119
|
-
|
|
49
|
+
## How it works
|
|
120
50
|
|
|
121
|
-
|
|
122
|
-
npx themis test --watch --isolation in-process --cache --reporter next
|
|
123
|
-
```
|
|
51
|
+
### Plain English → structured tests
|
|
124
52
|
|
|
125
|
-
|
|
126
|
-
|
|
127
|
-
|
|
128
|
-
|
|
129
|
-
|
|
130
|
-
npx themis migrate jest --rewrite-imports
|
|
131
|
-
npx themis migrate vitest --rewrite-imports
|
|
132
|
-
npx themis migrate jest --rewrite-imports --convert
|
|
133
|
-
npx themis migrate vitest --rewrite-imports --convert
|
|
134
|
-
npx themis test
|
|
135
|
-
```
|
|
53
|
+
```js
|
|
54
|
+
intent('user can sign in', ({ context, run, verify, cleanup }) => {
|
|
55
|
+
context('a valid user', (ctx) => {
|
|
56
|
+
ctx.user = { email: 'a@b.com', password: 'pw' };
|
|
57
|
+
});
|
|
136
58
|
|
|
137
|
-
|
|
59
|
+
run('the user submits credentials', (ctx) => {
|
|
60
|
+
ctx.result = { ok: true };
|
|
61
|
+
});
|
|
138
62
|
|
|
139
|
-
|
|
63
|
+
verify('authentication succeeds', (ctx) => {
|
|
64
|
+
expect(ctx.result.ok).toBe(true);
|
|
65
|
+
});
|
|
140
66
|
|
|
141
|
-
|
|
142
|
-
|
|
143
|
-
|
|
67
|
+
cleanup('remove test state', (ctx) => {
|
|
68
|
+
delete ctx.user;
|
|
69
|
+
});
|
|
70
|
+
});
|
|
144
71
|
```
|
|
145
72
|
|
|
146
|
-
|
|
147
|
-
|
|
148
|
-
- checks the scanned export names when Themis can resolve them exactly
|
|
149
|
-
- asserts the normalized runtime export contract directly in generated source
|
|
150
|
-
- adds scenario adapters for React components/hooks, Next app/router files, route handlers, and service functions when Themis can infer or read useful inputs
|
|
151
|
-
- captures React interaction and hook state-transition contracts when event handlers or stateful methods are available
|
|
152
|
-
- asserts DOM-state and behavioral flow contracts directly for generated React and Next component adapters
|
|
153
|
-
- emits async behavioral flow contracts for generated React and Next component adapters when flow plans are inferred or hinted, including richer inferred input/submit/loading/success paths for common async forms
|
|
154
|
-
- supports provider-driven DOM flow contracts for empty, disabled, retry, error, and recovery states with attribute- and role-aware assertions
|
|
155
|
-
- fails with a regeneration hint when the source drifts after the scan
|
|
156
|
-
|
|
157
|
-
Themis also supports per-file generation hints with sidecars like `src/components/Button.themis.json` so humans and agents can provide props, component flows, args, route requests, and route context. When those sidecars do not exist yet, `--write-hints` can scaffold them automatically from the current source analysis.
|
|
158
|
-
|
|
159
|
-
This is the core alternative to snapshot-driven testing: generated and hand-written tests assert normalized contracts in readable source, so diffs stay reviewable and updates stay intentional.
|
|
160
|
-
|
|
161
|
-
For repo-wide generation defaults, add `themis.generate.js` or `themis.generate.cjs` at the project root. Providers in that file can match source paths, supply shared props/args/flow plans, register runtime mocks for generated UI scenarios, and wrap generated component renders so generated DOM contracts run inside the same provider shells humans use in app tests. Providers can also declare preset wrapper metadata for router, Next navigation, auth/session shells, React Query, Zustand, and Redux-style app state patterns, including route history/state, query status, auth permissions, and store selector/action metadata.
|
|
162
|
-
|
|
163
|
-
For CI and agent loops, Themis can also enforce generation quality instead of only writing files. Strict runs emit a structured backlog, fail on unresolved scan debt, and hand back exact remediation commands.
|
|
164
|
-
|
|
165
|
-
Use these flags to control the generation loop:
|
|
166
|
-
|
|
167
|
-
- `--json`: machine-readable payload for agents, including prompt-ready next steps
|
|
168
|
-
- `--plan`: alias for `--review --json` with persisted handoff artifacts
|
|
169
|
-
- `--review`: dry-run create/update/remove decisions without writing files
|
|
170
|
-
- `--update`: refresh existing generated files only
|
|
171
|
-
- `--clean`: remove generated files for the selected scope
|
|
172
|
-
- `--changed`: target changed files in a git worktree
|
|
173
|
-
- `--write-hints`: scaffold missing `.themis.json` sidecars so the next generate pass has explicit component props, hook args, service args, and route requests
|
|
174
|
-
- `--scenario`: limit generation to one adapter family such as `react-hook`, `next-app-component`, or `next-route-handler`
|
|
175
|
-
- `--min-confidence`: keep only entries at or above a confidence threshold
|
|
176
|
-
- `--strict`: fail the generate run on skips, conflicts, or entries below `high` confidence
|
|
177
|
-
- `--fail-on-skips`, `--fail-on-conflicts`: turn unresolved scan debt into a non-zero exit code
|
|
178
|
-
- `--require-confidence`: fail if selected generated tests fall below a confidence threshold
|
|
179
|
-
- `--files`, `--match-source`, `--match-export`, `--include`, `--exclude`: narrow the scan scope
|
|
180
|
-
- `--force`: replace a conflicting non-Themis file
|
|
181
|
-
- `--output <dir>`: change the generated test directory
|
|
73
|
+
Phase names: `context`, `run`, `verify`, `cleanup`. Legacy aliases (`arrange/act/assert`, `given/when/then`) also supported.
|
|
182
74
|
|
|
183
|
-
|
|
184
|
-
|
|
185
|
-
- `.themis/generate/generate-map.json`: source-to-generated-test mapping plus scenario/confidence metadata
|
|
186
|
-
- `.themis/generate/generate-last.json`: the full machine-readable generate payload
|
|
187
|
-
- `.themis/generate/generate-handoff.json`: a compact agent handoff artifact with prompt-ready next actions
|
|
188
|
-
- `.themis/generate/generate-backlog.json`: unresolved skips, conflicts, and confidence debt with suggested fixes
|
|
189
|
-
|
|
190
|
-
Local test loops can also opt into a zero IPC execution path:
|
|
191
|
-
|
|
192
|
-
- `npx themis test --isolation in-process`: executes suites in-process instead of worker mode
|
|
193
|
-
- `npx themis test --watch --isolation in-process --cache`: keeps a fast local rerun loop with file-level result caching
|
|
194
|
-
- `npx themis test --isolation worker`: keeps process isolation for CI or global-heavy suites
|
|
195
|
-
|
|
196
|
-
When generated tests fail, Themis also writes:
|
|
197
|
-
|
|
198
|
-
- `.themis/runs/fix-handoff.json`: a deduped failure-to-fix artifact that maps generated failures back to source files, categories, repair strategies, candidate files, and remediation commands
|
|
199
|
-
|
|
200
|
-
Repair generated suites with:
|
|
201
|
-
|
|
202
|
-
```bash
|
|
203
|
-
npx themis test --fix
|
|
204
|
-
```
|
|
205
|
-
|
|
206
|
-
`--fix` reads `.themis/runs/fix-handoff.json`, regenerates the affected source targets with `--update`, scaffolds hints when the repair strategy needs them, and reruns the suite.
|
|
207
|
-
|
|
208
|
-
Migration scaffolds also write:
|
|
209
|
-
|
|
210
|
-
- `.themis/migration/migration-report.json`: a machine-readable inventory of detected Jest/Vitest compatibility imports and recommended next actions
|
|
211
|
-
- `themis.compat.js`: an optional local compatibility bridge used by `themis migrate --rewrite-imports`
|
|
212
|
-
|
|
213
|
-
## Why Themis
|
|
214
|
-
|
|
215
|
-
See [`docs/why-themis.md`](docs/why-themis.md) for positioning, differentiators, and community messaging.
|
|
216
|
-
|
|
217
|
-
Short version:
|
|
218
|
-
|
|
219
|
-
- Themis aims to deliver the benefits people reach for in snapshots, without snapshot rot.
|
|
220
|
-
- Prefer explicit, normalized contracts over broad output dumps.
|
|
221
|
-
- Keep changes reviewable through source assertions, machine-readable artifacts, and diff-oriented rerun workflows.
|
|
222
|
-
- See [`docs/showcases.md`](docs/showcases.md) for direct Jest/Vitest comparison examples.
|
|
223
|
-
|
|
224
|
-
## Reference Docs
|
|
225
|
-
|
|
226
|
-
- API reference: [`docs/api.md`](docs/api.md)
|
|
227
|
-
- Agent adoption guide: [`docs/agents-adoption.md`](docs/agents-adoption.md)
|
|
228
|
-
- Migration guide: [`docs/migration.md`](docs/migration.md)
|
|
229
|
-
- Release policy: [`docs/release-policy.md`](docs/release-policy.md)
|
|
230
|
-
- Publish guide: [`docs/publish.md`](docs/publish.md)
|
|
231
|
-
- VS Code extension notes: [`docs/vscode-extension.md`](docs/vscode-extension.md)
|
|
232
|
-
- Agent result schema: [`docs/schemas/agent-result.v1.json`](docs/schemas/agent-result.v1.json)
|
|
233
|
-
- Generate result schema: [`docs/schemas/generate-result.v1.json`](docs/schemas/generate-result.v1.json)
|
|
234
|
-
- Generate map schema: [`docs/schemas/generate-map.v1.json`](docs/schemas/generate-map.v1.json)
|
|
235
|
-
- Generate handoff schema: [`docs/schemas/generate-handoff.v1.json`](docs/schemas/generate-handoff.v1.json)
|
|
236
|
-
- Generate backlog schema: [`docs/schemas/generate-backlog.v1.json`](docs/schemas/generate-backlog.v1.json)
|
|
237
|
-
- Fix handoff schema: [`docs/schemas/fix-handoff.v1.json`](docs/schemas/fix-handoff.v1.json)
|
|
238
|
-
- Failures artifact schema: [`docs/schemas/failures.v1.json`](docs/schemas/failures.v1.json)
|
|
239
|
-
- Contract diff schema: [`docs/schemas/contract-diff.v1.json`](docs/schemas/contract-diff.v1.json)
|
|
240
|
-
- Changelog: [`CHANGELOG.md`](CHANGELOG.md)
|
|
241
|
-
|
|
242
|
-
## Commands
|
|
243
|
-
|
|
244
|
-
- `npx themis init`: creates `themis.config.json`, adds `.themis/` to `.gitignore`, and adds `__themis__/reports/` plus `__themis__/shims/` to `.gitignore`.
|
|
245
|
-
- `npx themis init --agents`: does the same and also writes a downstream `AGENTS.md` from the Themis template if the repo does not already have one.
|
|
246
|
-
- `npx themis generate src`: scans source files and generates contract tests under `__themis__/tests`, using `.generated.test.ts` for TS/TSX sources and `.generated.test.js` for JS/JSX sources.
|
|
247
|
-
- `npx themis generate src --json`: emits a machine-readable generation payload for agents and automation.
|
|
248
|
-
- `npx themis generate src --plan`: emits a planning payload and handoff artifact without writing generated tests.
|
|
249
|
-
- `npx themis generate src --review --json`: previews create/update/remove decisions without writing files.
|
|
250
|
-
- `npx themis generate src --review --strict --json`: fails fast on unresolved generation debt while still emitting a machine-readable plan.
|
|
251
|
-
- `npx themis generate src --write-hints`: scaffolds missing hint sidecars and uses them in the same generate pass.
|
|
252
|
-
- `npx themis generate src --update`: refreshes existing generated tests only.
|
|
253
|
-
- `npx themis generate src --clean`: removes generated tests for the selected scope.
|
|
254
|
-
- `npx themis generate src --changed`: regenerates against changed files in the current git worktree.
|
|
255
|
-
- `npx themis generate src --scenario react-hook --min-confidence high`: targets one adapter family at a confidence threshold.
|
|
256
|
-
- `npx themis generate app --scenario next-route-handler`: focuses generation on Next app router request handlers.
|
|
257
|
-
- `npx themis migrate jest`: scaffolds a Themis config/setup bridge for existing Jest suites and gitignores `.themis/` plus `__themis__/reports/` and `__themis__/shims/`.
|
|
258
|
-
- `npx themis migrate jest --rewrite-imports`: rewrites matched Jest/Vitest/Testing Library imports to a local `themis.compat.js` bridge file.
|
|
259
|
-
- `npx themis migrate jest --convert`: applies codemods for common Jest/Vitest matcher/import patterns so suites move closer to native Themis style.
|
|
260
|
-
- `npx themis migrate vitest`: scaffolds the same bridge for Vitest suites and gitignores `.themis/` plus `__themis__/reports/` and `__themis__/shims/`.
|
|
261
|
-
- `npx themis generate src --require-confidence high`: enforces a quality bar for all selected generated tests.
|
|
262
|
-
- `npx themis generate src --files src/routes/ping.ts`: targets one or more explicit source files.
|
|
263
|
-
- `npx themis generate src --match-source "routes/" --match-export "GET|POST"`: narrows generation by source path and exported symbol.
|
|
264
|
-
- `npx themis generate src --output tests/contracts`: writes generated tests to a custom directory.
|
|
265
|
-
- `npx themis generate src --force`: replaces conflicting files in the target output directory.
|
|
266
|
-
- `npx themis test`: discovers and runs tests.
|
|
267
|
-
- `npx themis test --next`: next-gen console output mode.
|
|
268
|
-
- `npx themis test --json`: emits JSON result payload.
|
|
269
|
-
- `npx themis test --agent`: emits AI-agent-oriented JSON schema.
|
|
270
|
-
- `npx themis test --reporter html`: generates a next-gen HTML report file.
|
|
271
|
-
- `npx themis test --reporter html --html-output reports/themis.html`: writes HTML report to a custom path.
|
|
272
|
-
- `npx themis test --watch`: reruns the suite when watched project files change.
|
|
273
|
-
- `npx themis test --watch --isolation in-process --cache`: runs a zero IPC cached local loop for fast edit/rerun cycles.
|
|
274
|
-
- `npx themis test --workers 8`: overrides worker count (positive integer).
|
|
275
|
-
- `npx themis test --isolation in-process`: runs test files in-process instead of worker processes.
|
|
276
|
-
- `npx themis test --cache`: enables file-level result caching for in-process local loops.
|
|
277
|
-
- `npx themis test --environment jsdom`: runs tests in a browser-like DOM environment.
|
|
278
|
-
- `npx themis test --stability 3`: runs the suite three times and classifies each test as `stable_pass`, `stable_fail`, or `unstable`.
|
|
279
|
-
- `npx themis test --match "intent DSL"`: runs only tests whose full name matches regex.
|
|
280
|
-
- `npx themis test --rerun-failed`: reruns failing tests from `.themis/runs/failed-tests.json`.
|
|
281
|
-
- `npx themis test --fix`: applies generated-test autofixes from `.themis/runs/fix-handoff.json` and reruns the suite.
|
|
282
|
-
- `npx themis test --update-contracts --match "suite > case"`: accepts reviewed `captureContract(...)` changes for a narrow slice of the suite.
|
|
283
|
-
- `npx themis test --no-memes`: disables meme phase aliases (`cook`, `yeet`, `vibecheck`, `wipe`).
|
|
284
|
-
- `npx themis test --lexicon classic|themis`: rebrands human-readable status labels in `next/spec` reporters.
|
|
285
|
-
- `npm run lint`: runs ESLint across the CLI, runtime, scripts, tests, and the VS Code extension scaffold.
|
|
286
|
-
- `npm run validate`: runs lint, test, typecheck, and benchmark gate in one command.
|
|
287
|
-
- `npm run typecheck`: validates TypeScript types for Themis globals and DSL contracts.
|
|
288
|
-
- `npm run benchmark:gate`: fails when benchmark performance exceeds the configured threshold.
|
|
289
|
-
- `npm run pack:check`: previews the npm publish payload.
|
|
290
|
-
- `npm run proof:migration`: migrates checked-in Jest/Vitest fixture suites and proves they run cleanly under Themis.
|
|
291
|
-
|
|
292
|
-
## CI & Release Proof
|
|
293
|
-
|
|
294
|
-
- Lint job runs `npm run lint` on Node 20.
|
|
295
|
-
- Compatibility job runs `npm test` on Node 18 and 20.
|
|
296
|
-
- Release surface job runs `npm run typecheck`, `npm run pack:check`, the HTML + agent reports, verifies `.themis/diffs/contract-diff.json`, produces `.themis/benchmarks/benchmark-last.json`/`.themis/benchmarks/migration-proof.json`, and uploads all of the artifacts for later inspection.
|
|
297
|
-
- Perf gate job runs `npm run benchmark:gate` with `BENCH_MAX_AVG_MS=2500` to guard against regressions before publishing.
|
|
298
|
-
- Migration proof job runs `npm run proof:migration` against checked-in Jest/Vitest fixtures for basic suites, table tests, RTL/jsdom flows, timers, module mocking, and a context/provider-heavy RTL example, then uploads the resulting migration reports plus Themis run artifacts as evidence.
|
|
299
|
-
- Themis React Showcase job verifies a straight-up native Themis React fixture as a first-party example.
|
|
300
|
-
- React showcase perf job runs `npm run benchmark:showcase` on the exact same React scenarios for Themis, Jest, and Vitest on one CI host, then uploads `.themis/benchmarks/showcase-comparison/perf-summary.{json,md}` so the relative timing claim is backed by one comparable artifact.
|
|
301
|
-
- Release `0.1.15` packages this expanded proof lane so every CI run now proves the provider-heavy example alongside the earlier fixtures.
|
|
302
|
-
|
|
303
|
-
## Agent Guide
|
|
304
|
-
|
|
305
|
-
[`AGENTS.md`](AGENTS.md) is the AI-agent contributor contract for this repository. It tells agents working on Themis itself how to write tests, preserve determinism, and update artifact contracts safely.
|
|
306
|
-
|
|
307
|
-
It is not a package-discovery mechanism for every external repo. If another project wants its agents to use Themis, that project should say so in its own `AGENTS.md`, rules, or agent prompt.
|
|
308
|
-
|
|
309
|
-
For downstream install, generation, and migration guidance, see [`docs/agents-adoption.md`](docs/agents-adoption.md).
|
|
310
|
-
|
|
311
|
-
For a copyable downstream rules file, see [`templates/AGENTS.themis.md`](templates/AGENTS.themis.md).
|
|
312
|
-
|
|
313
|
-
You do not need an MCP server just to make agents use Themis. Package metadata, docs, CLI commands, and explicit downstream repo instructions are the primary adoption path. An MCP integration could be useful later for richer editor or automation workflows, but it is optional.
|
|
314
|
-
|
|
315
|
-
Themis writes artifacts under `.themis/`:
|
|
316
|
-
|
|
317
|
-
- `.themis/runs/last-run.json`: full machine-readable run payload.
|
|
318
|
-
- `.themis/runs/failed-tests.json`: compact failure list for retry loops.
|
|
319
|
-
- `.themis/diffs/run-diff.json`: diff against the previous run, including new and resolved failures.
|
|
320
|
-
- `.themis/runs/run-history.json`: rolling recent-run history for agent comparison loops.
|
|
321
|
-
- `.themis/runs/fix-handoff.json`: source-oriented repair handoff for generated test failures.
|
|
322
|
-
- `.themis/migration/migration-report.json`: compatibility inventory and next actions for migrated Jest/Vitest suites.
|
|
323
|
-
- `.themis/diffs/contract-diff.json`: contract capture drift, updates, and update commands for `captureContract(...)` workflows.
|
|
324
|
-
- `.themis/generate/generate-last.json`: latest machine-readable generate payload.
|
|
325
|
-
- `.themis/generate/generate-map.json`: source-to-generated-test mapping.
|
|
326
|
-
- `.themis/generate/generate-handoff.json`: prompt-ready generate handoff payload.
|
|
327
|
-
- `.themis/generate/generate-backlog.json`: unresolved generate debt and suggested remediation.
|
|
328
|
-
- `themis.compat.js`: optional local compat bridge for rewritten migration imports.
|
|
329
|
-
- `.themis/benchmarks/benchmark-last.json`: latest benchmark comparison payload, including migration proof output.
|
|
330
|
-
- `.themis/benchmarks/migration-proof.json`: synthetic migration-conversion proof artifact emitted by `npm run benchmark`.
|
|
331
|
-
- `__themis__/reports/report.html`: interactive HTML verdict report.
|
|
332
|
-
- `__themis__/shims/`: reserved namespace for framework-owned compatibility shims when a fallback file is truly needed. Themis should prefer built-in support first and should not drop ad hoc shim files into `tests/`.
|
|
333
|
-
|
|
334
|
-
`--agent` output includes deterministic failure fingerprints, grouped `analysis.failureClusters`, stability classifications, previous-run comparison data, and a direct generated-test repair hint via `npx themis test --fix`. Fix handoff entries also carry repair strategies, candidate files, and autofix commands for tighter failure-to-fix loops.
|
|
335
|
-
|
|
336
|
-
Machine-facing reporters intentionally emit compact JSON. Agents and tooling should parse the payloads rather than depend on whitespace formatting.
|
|
337
|
-
|
|
338
|
-
The HTML reporter is designed for agent-adjacent review workflows too: it combines verdict status, slow-test surfacing, artifact navigation, and interactive file filtering in one report.
|
|
339
|
-
|
|
340
|
-
## VS Code
|
|
341
|
-
|
|
342
|
-
The repo now includes a thin VS Code extension scaffold at [`packages/themis-vscode`](packages/themis-vscode).
|
|
343
|
-
|
|
344
|
-
The extension is intentionally artifact-driven:
|
|
345
|
-
|
|
346
|
-
- reads `.themis/runs/last-run.json`, `.themis/runs/failed-tests.json`, `.themis/diffs/run-diff.json`, `.themis/generate/generate-last.json`, `.themis/generate/generate-map.json`, `.themis/generate/generate-backlog.json`, and `__themis__/reports/report.html`
|
|
347
|
-
- shows the latest verdict and failures in a sidebar
|
|
348
|
-
- adds generated-review navigation for source/test/hint mappings plus unresolved generation backlog
|
|
349
|
-
- reruns Themis from VS Code commands
|
|
350
|
-
- opens the HTML report inside a webview
|
|
351
|
-
|
|
352
|
-
It does not replace the CLI. The CLI and `.themis/**` artifacts remain the source of truth.
|
|
353
|
-
|
|
354
|
-
## Mocks And UI Primitives
|
|
355
|
-
|
|
356
|
-
Themis now ships first-party test utilities for agent-generated tests:
|
|
75
|
+
### Mocks and UI primitives
|
|
357
76
|
|
|
358
77
|
```js
|
|
359
78
|
mock('../src/api', () => ({
|
|
@@ -362,121 +81,47 @@ mock('../src/api', () => ({
|
|
|
362
81
|
|
|
363
82
|
const { fetchUser } = require('../src/api');
|
|
364
83
|
|
|
365
|
-
test('captures
|
|
84
|
+
test('mock captures calls', () => {
|
|
366
85
|
const user = fetchUser();
|
|
367
86
|
expect(fetchUser).toHaveBeenCalledTimes(1);
|
|
368
87
|
expect(user).toMatchObject({ id: 'u_1', name: 'Ada' });
|
|
369
88
|
});
|
|
370
89
|
```
|
|
371
90
|
|
|
372
|
-
|
|
373
|
-
|
|
374
|
-
```js
|
|
375
|
-
test('captures a stable response contract', () => {
|
|
376
|
-
const payload = {
|
|
377
|
-
status: 'ok',
|
|
378
|
-
flags: ['fast', 'deterministic']
|
|
379
|
-
};
|
|
380
|
-
|
|
381
|
-
captureContract('status payload', payload);
|
|
382
|
-
expect(payload.status).toBe('ok');
|
|
383
|
-
});
|
|
384
|
-
```
|
|
385
|
-
|
|
386
|
-
Themis intentionally avoids first-party snapshot-file workflows. Prefer direct assertions, generated contract tests, and explicit flow expectations over large opaque snapshots. The goal is comparable baseline coverage with better reviewability: normalized contracts, focused assertions, machine-readable artifacts, and intentional updates instead of broad snapshot re-acceptance.
|
|
387
|
-
|
|
388
|
-
Available globals:
|
|
389
|
-
|
|
390
|
-
- `fn(...)`
|
|
391
|
-
- `spyOn(object, methodName)`
|
|
392
|
-
- `mock(moduleId, factoryOrExports)`
|
|
393
|
-
- `unmock(moduleId)`
|
|
394
|
-
- `clearAllMocks()`
|
|
395
|
-
- `resetAllMocks()`
|
|
396
|
-
- `restoreAllMocks()`
|
|
397
|
-
|
|
398
|
-
For UI-oriented `jsdom` tests, Themis also ships a lightweight DOM layer:
|
|
91
|
+
For `jsdom` tests, Themis ships `render`, `screen`, `fireEvent`, `waitFor`, `useFakeTimers`, `mockFetch`, and more. Full list in the [API reference](docs/api.md).
|
|
399
92
|
|
|
400
|
-
|
|
401
|
-
- `screen.getByText(...)`
|
|
402
|
-
- `screen.getByRole(...)`
|
|
403
|
-
- `screen.getByLabelText(...)`
|
|
404
|
-
- `fireEvent.click/change/input/submit/keyDown(...)`
|
|
405
|
-
- `waitFor(asyncAssertion)`
|
|
406
|
-
- `cleanup()`
|
|
407
|
-
- `useFakeTimers()`, `advanceTimersByTime(ms)`, `runAllTimers()`, `useRealTimers()`
|
|
408
|
-
- `flushMicrotasks()`
|
|
409
|
-
- `mockFetch(...)`, `resetFetchMocks()`, `restoreFetch()`
|
|
93
|
+
### Code generation
|
|
410
94
|
|
|
411
|
-
|
|
95
|
+
Themis scans your source tree and generates contract tests for exported modules, React components, hooks, Next.js routes, and services:
|
|
412
96
|
|
|
413
|
-
```
|
|
414
|
-
|
|
415
|
-
|
|
416
|
-
|
|
417
|
-
fireEvent.click(screen.getByRole('button', { name: 'Send' }));
|
|
418
|
-
|
|
419
|
-
await waitFor(() => {
|
|
420
|
-
expect(document.body).toHaveAttribute('data-state', 'sent');
|
|
421
|
-
});
|
|
422
|
-
});
|
|
97
|
+
```bash
|
|
98
|
+
npx themis generate src
|
|
99
|
+
npx themis test
|
|
423
100
|
```
|
|
424
101
|
|
|
425
|
-
|
|
426
|
-
|
|
427
|
-
```ts
|
|
428
|
-
test('loads api state deterministically', async () => {
|
|
429
|
-
useFakeTimers();
|
|
430
|
-
const fetchMock = mockFetch({ json: { ok: true } });
|
|
102
|
+
When generated tests fail:
|
|
431
103
|
|
|
432
|
-
|
|
433
|
-
|
|
434
|
-
const response = await fetch('/api/status');
|
|
435
|
-
const payload = await response.json();
|
|
436
|
-
done = payload.ok;
|
|
437
|
-
}, 50);
|
|
438
|
-
|
|
439
|
-
advanceTimersByTime(50);
|
|
440
|
-
await flushMicrotasks();
|
|
441
|
-
|
|
442
|
-
expect(done).toBe(true);
|
|
443
|
-
expect(fetchMock).toHaveBeenCalled();
|
|
444
|
-
useRealTimers();
|
|
445
|
-
restoreFetch();
|
|
446
|
-
});
|
|
104
|
+
```bash
|
|
105
|
+
npx themis test --fix
|
|
447
106
|
```
|
|
448
107
|
|
|
449
|
-
|
|
450
|
-
|
|
451
|
-
Themis supports a strict code-native intent DSL:
|
|
108
|
+
`--fix` regenerates affected targets and reruns the suite. See the [API reference](docs/api.md) for all generation flags (`--review`, `--plan`, `--write-hints`, `--strict`, `--changed`, etc.).
|
|
452
109
|
|
|
453
|
-
|
|
454
|
-
intent('user can sign in', ({ context, run, verify, cleanup }) => {
|
|
455
|
-
context('a valid user', (ctx) => {
|
|
456
|
-
ctx.user = { email: 'a@b.com', password: 'pw' };
|
|
457
|
-
});
|
|
110
|
+
### Migration
|
|
458
111
|
|
|
459
|
-
|
|
460
|
-
|
|
461
|
-
|
|
462
|
-
|
|
463
|
-
verify('authentication succeeds', (ctx) => {
|
|
464
|
-
expect(ctx.result.ok).toBe(true);
|
|
465
|
-
});
|
|
466
|
-
|
|
467
|
-
cleanup('remove test state', (ctx) => {
|
|
468
|
-
delete ctx.user;
|
|
469
|
-
});
|
|
470
|
-
});
|
|
112
|
+
```bash
|
|
113
|
+
npx themis migrate jest
|
|
114
|
+
npx themis migrate vitest
|
|
115
|
+
npx themis test
|
|
471
116
|
```
|
|
472
117
|
|
|
473
|
-
|
|
474
|
-
|
|
475
|
-
|
|
118
|
+
One command scaffolds a compatibility bridge. Add `--rewrite-imports` to rewrite import paths, `--convert` for codemods. See the [migration guide](docs/migration.md).
|
|
119
|
+
|
|
120
|
+
---
|
|
476
121
|
|
|
477
122
|
## Config
|
|
478
123
|
|
|
479
|
-
`themis.config.json
|
|
124
|
+
`themis.config.json`:
|
|
480
125
|
|
|
481
126
|
```json
|
|
482
127
|
{
|
|
@@ -487,28 +132,15 @@ Easter egg aliases are also available: `cook`, `yeet`, `vibecheck`, `wipe`.
|
|
|
487
132
|
"reporter": "next",
|
|
488
133
|
"environment": "node",
|
|
489
134
|
"setupFiles": ["tests/setup.ts"],
|
|
490
|
-
"tsconfigPath": "tsconfig.json"
|
|
491
|
-
"htmlReportPath": "__themis__/reports/report.html",
|
|
492
|
-
"testIgnore": ["^tests/fixtures(?:/|$)"]
|
|
135
|
+
"tsconfigPath": "tsconfig.json"
|
|
493
136
|
}
|
|
494
137
|
```
|
|
495
138
|
|
|
496
|
-
|
|
497
|
-
Themis discovers both `testDir` and `generatedTestsDir` by default. Use `testIgnore` only for fixture folders, scratch suites, or other paths you intentionally want to skip.
|
|
498
|
-
Themis also stubs common frontend style and asset imports under Node or jsdom runs, including `.css`, `.scss`, `.png`, `.jpg`, `.svg`, and common font/media files, so repos should not need ad hoc `tests/*.cjs` setup files just to make those imports load.
|
|
499
|
-
|
|
500
|
-
## TypeScript
|
|
501
|
-
|
|
502
|
-
The package ships first-party typings for:
|
|
503
|
-
|
|
504
|
-
- programmatic APIs (`collectAndRun`, `runTests`, config helpers)
|
|
505
|
-
- global test APIs (`describe`, `test`, `intent`, hooks, `expect`)
|
|
506
|
-
- typed intent context (`intent<MyCtx>(...)`)
|
|
507
|
-
- project-aware module loading for `ts`, `tsx`, ESM `js`, `jsx`, `tsconfig` path aliases, and setup files
|
|
139
|
+
Use `environment: "jsdom"` for DOM-driven tests. Themis auto-stubs common style/asset imports (`.css`, `.scss`, `.png`, `.svg`, etc.).
|
|
508
140
|
|
|
509
|
-
|
|
141
|
+
---
|
|
510
142
|
|
|
511
|
-
|
|
143
|
+
## TypeScript
|
|
512
144
|
|
|
513
145
|
```json
|
|
514
146
|
{
|
|
@@ -518,37 +150,35 @@ Use the global types in your project with:
|
|
|
518
150
|
}
|
|
519
151
|
```
|
|
520
152
|
|
|
521
|
-
|
|
153
|
+
Ships first-party typings for all test APIs, typed intent context, and project-aware module loading for `.ts`, `.tsx`, ESM `.js`, `.jsx`, and `tsconfig` path aliases.
|
|
522
154
|
|
|
523
|
-
|
|
524
|
-
npm run benchmark
|
|
525
|
-
npm run benchmark:showcase
|
|
526
|
-
npm run benchmark:gate
|
|
527
|
-
```
|
|
155
|
+
---
|
|
528
156
|
|
|
529
|
-
|
|
157
|
+
## Pair with Alethia
|
|
530
158
|
|
|
531
|
-
|
|
532
|
-
- `BENCH_TESTS_PER_FILE` (default `25`)
|
|
533
|
-
- `BENCH_REPEATS` (default `3`)
|
|
534
|
-
- `BENCH_WORKERS` (default `4`)
|
|
535
|
-
- `BENCH_INCLUDE_EXTERNAL=1` to include Jest/Vitest/Bun comparisons
|
|
536
|
-
- `BENCH_MAX_AVG_MS` to override the gate threshold
|
|
537
|
-
- `BENCH_GATE_CONFIG` to point `benchmark:gate` at a custom config file
|
|
538
|
-
- `SHOWCASE_BENCH_WARMUPS` (default `1`) for the same-spec React showcase comparison
|
|
539
|
-
- `SHOWCASE_BENCH_REPEATS` (default `5`) for the same-spec React showcase comparison
|
|
159
|
+
Themis owns the unit/contract layer. [Alethia](https://github.com/vitron-ai/alethia) owns the E2E/policy layer. Together they form the tightest test loop an autonomous coding agent can sit inside:
|
|
540
160
|
|
|
541
|
-
|
|
542
|
-
|
|
543
|
-
|
|
161
|
+
1. Agent generates code
|
|
162
|
+
2. **Themis** verifies the contract in milliseconds
|
|
163
|
+
3. **Alethia** verifies the running app in a real browser, under safety policy, with a signed audit trail
|
|
544
164
|
|
|
545
|
-
|
|
165
|
+
Use Themis on its own — it's MIT and stands alone.
|
|
546
166
|
|
|
547
|
-
|
|
167
|
+
---
|
|
548
168
|
|
|
549
|
-
|
|
550
|
-
|
|
551
|
-
|
|
552
|
-
|
|
169
|
+
## Reference docs
|
|
170
|
+
|
|
171
|
+
- [API reference](docs/api.md) — all CLI flags, globals, matchers, mocks, UI primitives
|
|
172
|
+
- [Agent adoption guide](docs/agents-adoption.md) — downstream repo setup
|
|
173
|
+
- [Migration guide](docs/migration.md) — Jest/Vitest migration details
|
|
174
|
+
- [Why Themis](docs/why-themis.md) — positioning and differentiators
|
|
175
|
+
- [Showcase comparisons](docs/showcases.md) — direct Jest/Vitest examples
|
|
176
|
+
- [Tutorial: Testing with Claude Code](docs/tutorial-claude-code.md)
|
|
177
|
+
- [VS Code extension](docs/vscode-extension.md)
|
|
178
|
+
- [Release policy](docs/release-policy.md)
|
|
179
|
+
- [Publish guide](docs/publish.md)
|
|
180
|
+
- [Changelog](CHANGELOG.md)
|
|
553
181
|
|
|
554
|
-
|
|
182
|
+
<p align="center">
|
|
183
|
+
<img src="src/assets/themisVerdictEngine.png" alt="Themis verdict engine art" width="960">
|
|
184
|
+
</p>
|