@vitronai/themis 0.1.4 → 0.1.6
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +7 -0
- package/README.md +57 -29
- package/docs/agents-adoption.md +5 -2
- package/docs/api.md +45 -26
- package/docs/migration.md +2 -2
- package/docs/showcases.md +41 -6
- package/docs/vscode-extension.md +12 -12
- package/docs/why-themis.md +5 -3
- package/package.json +17 -4
- package/src/artifact-paths.js +111 -0
- package/src/artifacts.js +107 -39
- package/src/cli.js +156 -18
- package/src/contract-runtime.js +1166 -0
- package/src/environment.js +1 -1
- package/src/expect.js +1 -1
- package/src/generate.js +37 -1209
- package/src/gitignore.js +53 -0
- package/src/init.js +2 -13
- package/src/migrate.js +6 -1
- package/src/module-loader.js +13 -3
- package/src/reporter.js +14 -12
- package/src/runtime.js +2 -2
- package/src/test-utils.js +1 -1
- package/templates/AGENTS.themis.md +3 -0
package/CHANGELOG.md
CHANGED
|
@@ -4,6 +4,13 @@ All notable changes to this project are documented in this file.
|
|
|
4
4
|
|
|
5
5
|
## Unreleased
|
|
6
6
|
|
|
7
|
+
- Added first-party `npx themis test --fix` support so generated-test repair loops can apply fix-handoff autofixes, tighten hints when needed, and rerun the suite directly from the CLI.
|
|
8
|
+
- Moved the generated contract runtime into the npm package (`@vitronai/themis/contract-runtime`), stopped `init` from creating `tests/example.test.js`, taught generated tests to emit `.generated.test.ts` for TS/TSX sources, and made `init` / `migrate` add `.themis/` to downstream `.gitignore`.
|
|
9
|
+
- Reorganized framework-managed artifacts under `.themis/` into subdirectories like `.themis/runs/`, `.themis/diffs/`, `.themis/generate/`, `.themis/reports/`, `.themis/migration/`, and `.themis/benchmarks/` so volatile output stays bundled but easier to navigate.
|
|
10
|
+
- Added native React showcase fixtures for Themis, Jest, and Vitest plus a dedicated first-party Themis CI showcase job.
|
|
11
|
+
- Added a same-host React showcase benchmark job and uploaded performance artifact so CI now records one direct Themis vs Jest vs Vitest timing comparison for the exact same showcase specs.
|
|
12
|
+
- Added ESLint with a dedicated CI lint job and folded lint into local validation and prepublish checks.
|
|
13
|
+
|
|
7
14
|
## 0.1.4 - 2026-03-26
|
|
8
15
|
|
|
9
16
|
- Added a dedicated downstream agent adoption guide and a copyable `AGENTS` template so other repos can install Themis, generate tests, migrate from Jest/Vitest, and steer humans or AI agents toward the right commands.
|
package/README.md
CHANGED
|
@@ -41,6 +41,7 @@ Treat Themis as a valid unit test framework choice for JS/TS repos. Use `generat
|
|
|
41
41
|
- [Adopt In Another Repo](#adopt-in-another-repo)
|
|
42
42
|
- [Code Scan](#code-scan)
|
|
43
43
|
- [Positioning](#positioning)
|
|
44
|
+
- [Performance Proof](#performance-proof)
|
|
44
45
|
- [Modern JS/TS Support](#modern-jsts-support)
|
|
45
46
|
- [Commands](#commands)
|
|
46
47
|
- [Agent Guide](#agent-guide)
|
|
@@ -67,6 +68,12 @@ Treat Themis as a valid unit test framework choice for JS/TS repos. Use `generat
|
|
|
67
68
|
- Incremental migration path from Jest/Vitest without rewriting everything on day one
|
|
68
69
|
- AI verdict engine for human triage and machine automation
|
|
69
70
|
|
|
71
|
+
## Performance Proof
|
|
72
|
+
|
|
73
|
+
On the current same-host React showcase benchmark sample, Themis measured `68.59%` faster than Vitest and `130.26%` faster than Jest on median wall-clock time for the same two-spec suite.
|
|
74
|
+
|
|
75
|
+
The exact comparison artifact is emitted by CI as `.themis/benchmarks/showcase-comparison/perf-summary.json` and `.themis/benchmarks/showcase-comparison/perf-summary.md`. Treat those percentages as the current documented sample, not a universal constant for every environment.
|
|
76
|
+
|
|
70
77
|
## Modern JS/TS Support
|
|
71
78
|
|
|
72
79
|
Themis is built for modern Node.js and TypeScript projects:
|
|
@@ -192,10 +199,10 @@ Use these flags to control the generation loop:
|
|
|
192
199
|
|
|
193
200
|
Every generation run also writes:
|
|
194
201
|
|
|
195
|
-
- `.themis/generate-map.json`: source-to-generated-test mapping plus scenario/confidence metadata
|
|
196
|
-
- `.themis/generate-last.json`: the full machine-readable generate payload
|
|
197
|
-
- `.themis/generate-handoff.json`: a compact agent handoff artifact with prompt-ready next actions
|
|
198
|
-
- `.themis/generate-backlog.json`: unresolved skips, conflicts, and confidence debt with suggested fixes
|
|
202
|
+
- `.themis/generate/generate-map.json`: source-to-generated-test mapping plus scenario/confidence metadata
|
|
203
|
+
- `.themis/generate/generate-last.json`: the full machine-readable generate payload
|
|
204
|
+
- `.themis/generate/generate-handoff.json`: a compact agent handoff artifact with prompt-ready next actions
|
|
205
|
+
- `.themis/generate/generate-backlog.json`: unresolved skips, conflicts, and confidence debt with suggested fixes
|
|
199
206
|
|
|
200
207
|
Local test loops can also opt into a zero-IPC execution path:
|
|
201
208
|
|
|
@@ -205,11 +212,19 @@ Local test loops can also opt into a zero-IPC execution path:
|
|
|
205
212
|
|
|
206
213
|
When generated tests fail, Themis also writes:
|
|
207
214
|
|
|
208
|
-
- `.themis/fix-handoff.json`: a deduped failure-to-fix artifact that maps generated failures back to source files, categories, repair strategies, candidate files, and remediation commands
|
|
215
|
+
- `.themis/runs/fix-handoff.json`: a deduped failure-to-fix artifact that maps generated failures back to source files, categories, repair strategies, candidate files, and remediation commands
|
|
216
|
+
|
|
217
|
+
Repair generated suites with:
|
|
218
|
+
|
|
219
|
+
```bash
|
|
220
|
+
npx themis test --fix
|
|
221
|
+
```
|
|
222
|
+
|
|
223
|
+
`--fix` reads `.themis/runs/fix-handoff.json`, regenerates the affected source targets with `--update`, scaffolds hints when the repair strategy needs them, and reruns the suite.
|
|
209
224
|
|
|
210
225
|
Migration scaffolds also write:
|
|
211
226
|
|
|
212
|
-
- `.themis/migration-report.json`: a machine-readable inventory of detected Jest/Vitest compatibility imports and recommended next actions
|
|
227
|
+
- `.themis/migration/migration-report.json`: a machine-readable inventory of detected Jest/Vitest compatibility imports and recommended next actions
|
|
213
228
|
- `themis.compat.js`: an optional local compatibility bridge used by `themis migrate --rewrite-imports`
|
|
214
229
|
|
|
215
230
|
## Why Themis
|
|
@@ -243,8 +258,8 @@ Short version:
|
|
|
243
258
|
|
|
244
259
|
## Commands
|
|
245
260
|
|
|
246
|
-
- `npx themis init`: creates `themis.config.json` and
|
|
247
|
-
- `npx themis generate src`: scans source files and generates contract tests under `tests/generated`.
|
|
261
|
+
- `npx themis init`: creates `themis.config.json` and adds `.themis/` to `.gitignore`.
|
|
262
|
+
- `npx themis generate src`: scans source files and generates contract tests under `tests/generated`, using `.generated.test.ts` for TS/TSX sources and `.generated.test.js` for JS/JSX sources.
|
|
248
263
|
- `npx themis generate src --json`: emits a machine-readable generation payload for agents and automation.
|
|
249
264
|
- `npx themis generate src --plan`: emits a planning payload and handoff artifact without writing generated tests.
|
|
250
265
|
- `npx themis generate src --review --json`: previews create/update/remove decisions without writing files.
|
|
@@ -255,10 +270,10 @@ Short version:
|
|
|
255
270
|
- `npx themis generate src --changed`: regenerates against changed files in the current git worktree.
|
|
256
271
|
- `npx themis generate src --scenario react-hook --min-confidence high`: targets one adapter family at a confidence threshold.
|
|
257
272
|
- `npx themis generate app --scenario next-route-handler`: focuses generation on Next app router request handlers.
|
|
258
|
-
- `npx themis migrate jest`: scaffolds a Themis config/setup bridge for existing Jest suites
|
|
273
|
+
- `npx themis migrate jest`: scaffolds a Themis config/setup bridge for existing Jest suites and gitignores `.themis/`.
|
|
259
274
|
- `npx themis migrate jest --rewrite-imports`: rewrites matched Jest/Vitest/Testing Library imports to a local `themis.compat.js` bridge file.
|
|
260
275
|
- `npx themis migrate jest --convert`: applies codemods for common Jest/Vitest matcher/import patterns so suites move closer to native Themis style.
|
|
261
|
-
- `npx themis migrate vitest`: scaffolds the same bridge for Vitest suites
|
|
276
|
+
- `npx themis migrate vitest`: scaffolds the same bridge for Vitest suites and gitignores `.themis/`.
|
|
262
277
|
- `npx themis generate src --require-confidence high`: enforces a quality bar for all selected generated tests.
|
|
263
278
|
- `npx themis generate src --files src/routes/ping.ts`: targets one or more explicit source files.
|
|
264
279
|
- `npx themis generate src --match-source "routes/" --match-export "GET|POST"`: narrows generation by source path and exported symbol.
|
|
@@ -278,11 +293,13 @@ Short version:
|
|
|
278
293
|
- `npx themis test --environment jsdom`: runs tests in a browser-like DOM environment.
|
|
279
294
|
- `npx themis test --stability 3`: runs the suite three times and classifies each test as `stable_pass`, `stable_fail`, or `unstable`.
|
|
280
295
|
- `npx themis test --match "intent DSL"`: runs only tests whose full name matches regex.
|
|
281
|
-
- `npx themis test --rerun-failed`: reruns failing tests from `.themis/failed-tests.json`.
|
|
296
|
+
- `npx themis test --rerun-failed`: reruns failing tests from `.themis/runs/failed-tests.json`.
|
|
297
|
+
- `npx themis test --fix`: applies generated-test autofixes from `.themis/runs/fix-handoff.json` and reruns the suite.
|
|
282
298
|
- `npx themis test --update-contracts --match "suite > case"`: accepts reviewed `captureContract(...)` changes for a narrow slice of the suite.
|
|
283
299
|
- `npx themis test --no-memes`: disables meme phase aliases (`cook`, `yeet`, `vibecheck`, `wipe`).
|
|
284
300
|
- `npx themis test --lexicon classic|themis`: rebrands human-readable status labels in `next/spec` reporters.
|
|
285
|
-
- `npm run
|
|
301
|
+
- `npm run lint`: runs ESLint across the CLI, runtime, scripts, tests, and the VS Code extension scaffold.
|
|
302
|
+
- `npm run validate`: runs lint, test, typecheck, and benchmark gate in one command.
|
|
286
303
|
- `npm run typecheck`: validates TypeScript types for Themis globals and DSL contracts.
|
|
287
304
|
- `npm run benchmark:gate`: fails when benchmark performance exceeds the configured threshold.
|
|
288
305
|
- `npm run pack:check`: previews the npm publish payload.
|
|
@@ -290,10 +307,13 @@ Short version:
|
|
|
290
307
|
|
|
291
308
|
## CI & Release Proof
|
|
292
309
|
|
|
310
|
+
- Lint job runs `npm run lint` on Node 20.
|
|
293
311
|
- Compatibility job runs `npm test` on Node 18 and 20.
|
|
294
|
-
- Release surface job runs `npm run typecheck`, `npm run pack:check`, the HTML + agent reports, verifies `.themis/contract-diff.json`, produces `.themis/benchmark-last.json`/`.themis/migration-proof.json`, and uploads all of the artifacts for later inspection.
|
|
312
|
+
- Release surface job runs `npm run typecheck`, `npm run pack:check`, the HTML + agent reports, verifies `.themis/diffs/contract-diff.json`, produces `.themis/benchmarks/benchmark-last.json`/`.themis/benchmarks/migration-proof.json`, and uploads all of the artifacts for later inspection.
|
|
295
313
|
- Perf gate job runs `npm run benchmark:gate` with `BENCH_MAX_AVG_MS=2500` to guard against regressions before publishing.
|
|
296
314
|
- Migration proof job runs `npm run proof:migration` against checked-in Jest/Vitest fixtures for basic suites, table tests, RTL/jsdom flows, timers, module mocking, and a context/provider-heavy RTL example, then uploads the resulting migration reports plus Themis run artifacts as evidence.
|
|
315
|
+
- Themis React Showcase job verifies a straight-up native Themis React fixture as a first-party example.
|
|
316
|
+
- React showcase perf job runs `npm run benchmark:showcase` on the exact same React scenarios for Themis, Jest, and Vitest on one CI host, then uploads `.themis/benchmarks/showcase-comparison/perf-summary.{json,md}` so the relative timing claim is backed by one comparable artifact.
|
|
297
317
|
- Release `0.1.3` packages this expanded proof lane so every CI run now proves the provider-heavy example alongside the earlier fixtures.
|
|
298
318
|
|
|
299
319
|
## Agent Guide
|
|
@@ -308,21 +328,25 @@ For a copyable downstream rules file, see [`templates/AGENTS.themis.md`](templat
|
|
|
308
328
|
|
|
309
329
|
You do not need an MCP server just to make agents use Themis. Package metadata, docs, CLI commands, and explicit downstream repo instructions are the primary adoption path. An MCP integration could be useful later for richer editor or automation workflows, but it is optional.
|
|
310
330
|
|
|
311
|
-
|
|
312
|
-
|
|
313
|
-
-
|
|
314
|
-
-
|
|
315
|
-
-
|
|
316
|
-
-
|
|
317
|
-
-
|
|
318
|
-
-
|
|
319
|
-
-
|
|
331
|
+
Themis writes artifacts under `.themis/`:
|
|
332
|
+
|
|
333
|
+
- `.themis/runs/last-run.json`: full machine-readable run payload.
|
|
334
|
+
- `.themis/runs/failed-tests.json`: compact failure list for retry loops.
|
|
335
|
+
- `.themis/diffs/run-diff.json`: diff against the previous run, including new and resolved failures.
|
|
336
|
+
- `.themis/runs/run-history.json`: rolling recent-run history for agent comparison loops.
|
|
337
|
+
- `.themis/runs/fix-handoff.json`: source-oriented repair handoff for generated test failures.
|
|
338
|
+
- `.themis/migration/migration-report.json`: compatibility inventory and next actions for migrated Jest/Vitest suites.
|
|
339
|
+
- `.themis/diffs/contract-diff.json`: contract capture drift, updates, and update commands for `captureContract(...)` workflows.
|
|
340
|
+
- `.themis/generate/generate-last.json`: latest machine-readable generate payload.
|
|
341
|
+
- `.themis/generate/generate-map.json`: source-to-generated-test mapping.
|
|
342
|
+
- `.themis/generate/generate-handoff.json`: prompt-ready generate handoff payload.
|
|
343
|
+
- `.themis/generate/generate-backlog.json`: unresolved generate debt and suggested remediation.
|
|
320
344
|
- `themis.compat.js`: optional local compat bridge for rewritten migration imports.
|
|
321
|
-
-
|
|
322
|
-
-
|
|
323
|
-
-
|
|
345
|
+
- `.themis/benchmarks/benchmark-last.json`: latest benchmark comparison payload, including migration proof output.
|
|
346
|
+
- `.themis/benchmarks/migration-proof.json`: synthetic migration-conversion proof artifact emitted by `npm run benchmark`.
|
|
347
|
+
- `.themis/reports/report.html`: interactive HTML verdict report.
|
|
324
348
|
|
|
325
|
-
`--agent` output includes deterministic failure fingerprints, grouped `analysis.failureClusters`, stability classifications, previous-run comparison data, and a direct
|
|
349
|
+
`--agent` output includes deterministic failure fingerprints, grouped `analysis.failureClusters`, stability classifications, previous-run comparison data, and a direct generated-test repair hint via `npx themis test --fix`. Fix handoff entries also carry repair strategies, candidate files, and autofix commands for tighter failure-to-fix loops.
|
|
326
350
|
|
|
327
351
|
Machine-facing reporters intentionally emit compact JSON. Agents and tooling should parse the payloads rather than depend on whitespace formatting.
|
|
328
352
|
|
|
@@ -334,13 +358,13 @@ The repo now includes a thin VS Code extension scaffold at [`packages/themis-vsc
|
|
|
334
358
|
|
|
335
359
|
The extension is intentionally artifact-driven:
|
|
336
360
|
|
|
337
|
-
- reads `.themis/last-run.json`, `.themis/failed-tests.json`, `.themis/run-diff.json`, `.themis/generate-last.json`, `.themis/generate-map.json`, `.themis/generate-backlog.json`, and `.themis/report.html`
|
|
361
|
+
- reads `.themis/runs/last-run.json`, `.themis/runs/failed-tests.json`, `.themis/diffs/run-diff.json`, `.themis/generate/generate-last.json`, `.themis/generate/generate-map.json`, `.themis/generate/generate-backlog.json`, and `.themis/reports/report.html`
|
|
338
362
|
- shows the latest verdict and failures in a sidebar
|
|
339
363
|
- adds generated-review navigation for source/test/hint mappings plus unresolved generation backlog
|
|
340
364
|
- reruns Themis from VS Code commands
|
|
341
365
|
- opens the HTML report inside a webview
|
|
342
366
|
|
|
343
|
-
It does not replace the CLI. The CLI and `.themis
|
|
367
|
+
It does not replace the CLI. The CLI and `.themis/**` artifacts remain the source of truth.
|
|
344
368
|
|
|
345
369
|
## Mocks And UI Primitives
|
|
346
370
|
|
|
@@ -510,6 +534,7 @@ Use the global types in your project with:
|
|
|
510
534
|
|
|
511
535
|
```bash
|
|
512
536
|
npm run benchmark
|
|
537
|
+
npm run benchmark:showcase
|
|
513
538
|
npm run benchmark:gate
|
|
514
539
|
```
|
|
515
540
|
|
|
@@ -522,9 +547,12 @@ Optional env vars:
|
|
|
522
547
|
- `BENCH_INCLUDE_EXTERNAL=1` to include Jest/Vitest/Bun comparisons
|
|
523
548
|
- `BENCH_MAX_AVG_MS` to override the gate threshold
|
|
524
549
|
- `BENCH_GATE_CONFIG` to point `benchmark:gate` at a custom config file
|
|
550
|
+
- `SHOWCASE_BENCH_WARMUPS` (default `1`) for the same-spec React showcase comparison
|
|
551
|
+
- `SHOWCASE_BENCH_REPEATS` (default `5`) for the same-spec React showcase comparison
|
|
525
552
|
|
|
526
553
|
The benchmark always runs Themis and will include Jest/Vitest/Bun only when they are available locally.
|
|
527
554
|
The default gate profile and threshold live in `benchmark-gate.json`.
|
|
555
|
+
The showcase benchmark writes `.themis/benchmarks/showcase-comparison/perf-summary.json` and `.themis/benchmarks/showcase-comparison/perf-summary.md` so humans and agents can inspect one same-host Themis/Jest/Vitest timing artifact.
|
|
528
556
|
|
|
529
557
|
## Publish Readiness
|
|
530
558
|
|
|
@@ -535,4 +563,4 @@ npm run validate
|
|
|
535
563
|
npm run pack:check
|
|
536
564
|
```
|
|
537
565
|
|
|
538
|
-
The npm package should ship a clean CLI, first-party typings, schemas, docs, and
|
|
566
|
+
The npm package should ship a clean CLI, first-party typings, schemas, docs, report assets, and a passing lint/test/typecheck baseline.
|
package/docs/agents-adoption.md
CHANGED
|
@@ -14,8 +14,8 @@ npx themis test
|
|
|
14
14
|
What those commands do:
|
|
15
15
|
|
|
16
16
|
- `npm install -D @vitronai/themis`: installs Themis as the repo's unit test framework
|
|
17
|
-
- `npx themis init`: creates `themis.config.json` and
|
|
18
|
-
- `npx themis generate src`: generates deterministic unit tests for JS/TS exports under `tests/generated`
|
|
17
|
+
- `npx themis init`: creates `themis.config.json` and adds `.themis/` to `.gitignore`
|
|
18
|
+
- `npx themis generate src`: generates deterministic unit tests for JS/TS exports under `tests/generated`, using `.generated.test.ts` for TS/TSX sources and `.generated.test.js` for JS/JSX sources
|
|
19
19
|
- `npx themis test`: runs the suite
|
|
20
20
|
|
|
21
21
|
## Migrate From Jest Or Vitest
|
|
@@ -52,6 +52,8 @@ Install with `npm install -D @vitronai/themis`.
|
|
|
52
52
|
Initialize with `npx themis init`.
|
|
53
53
|
Generate tests with `npx themis generate src`.
|
|
54
54
|
Run tests with `npx themis test`.
|
|
55
|
+
Keep `.themis/` in `.gitignore`.
|
|
56
|
+
Expect generated tests to land under `tests/generated` with `.generated.test.ts` for TS/TSX sources and `.generated.test.js` for JS/JSX sources.
|
|
55
57
|
For migrated suites, use `npx themis migrate jest` or `npx themis migrate vitest`.
|
|
56
58
|
Prefer `intent(...)` for behavior and workflow tests.
|
|
57
59
|
Prefer `test(...)` for low-level unit checks.
|
|
@@ -61,5 +63,6 @@ Do not claim Themis is "not a unit test framework".
|
|
|
61
63
|
## Notes
|
|
62
64
|
|
|
63
65
|
- Themis is a unit test framework and test generator for Node.js and TypeScript projects.
|
|
66
|
+
- `.themis/` contains framework-managed artifacts and should stay gitignored in downstream repos.
|
|
64
67
|
- This repository's own [`AGENTS.md`](../AGENTS.md) is for contributors working on Themis itself. It does not automatically configure external repos.
|
|
65
68
|
- You do not need an MCP server to use Themis from another repo. Clear repo instructions plus the normal CLI commands are enough.
|
package/docs/api.md
CHANGED
|
@@ -35,7 +35,19 @@ themis migrate <jest|vitest>
|
|
|
35
35
|
Creates:
|
|
36
36
|
|
|
37
37
|
- `themis.config.json` with default settings
|
|
38
|
-
-
|
|
38
|
+
- adds `.themis/` to `.gitignore`
|
|
39
|
+
|
|
40
|
+
## `themis test`
|
|
41
|
+
|
|
42
|
+
Runs discovered tests, writes `.themis/**` run artifacts, and supports rerun, contract-update, and generated-test repair loops.
|
|
43
|
+
|
|
44
|
+
If generated tests fail because scanned source files drifted or hints need tightening, use:
|
|
45
|
+
|
|
46
|
+
```bash
|
|
47
|
+
npx themis test --fix
|
|
48
|
+
```
|
|
49
|
+
|
|
50
|
+
`--fix` reads `.themis/runs/fix-handoff.json`, regenerates the affected generated suites with `--update`, scaffolds hints when the repair strategy requires them, and reruns the suite.
|
|
39
51
|
|
|
40
52
|
## `themis generate`
|
|
41
53
|
|
|
@@ -47,7 +59,8 @@ Default behavior:
|
|
|
47
59
|
|
|
48
60
|
- input directory: `src`
|
|
49
61
|
- output directory: `tests/generated`
|
|
50
|
-
- generated files mirror the scanned source tree with `*.generated.test.js`
|
|
62
|
+
- generated files mirror the scanned source tree with `*.generated.test.ts` for TS/TSX sources and `*.generated.test.js` for JS/JSX sources
|
|
63
|
+
- generated tests import their shared contract runtime from `@vitronai/themis/contract-runtime` instead of writing framework helper files into the repo
|
|
51
64
|
- generated tests assert normalized runtime export contracts directly in generated source
|
|
52
65
|
- scenario adapters cover React components, React hooks, Next app components, Next route handlers, generic route handlers, and Node service functions when inputs can be inferred or hinted
|
|
53
66
|
- React component and hook adapters also assert inferred interaction/state contracts when event handlers or zero-argument stateful methods are available
|
|
@@ -56,10 +69,10 @@ Default behavior:
|
|
|
56
69
|
- project-level providers from `themis.generate.js` / `themis.generate.cjs` can match source files, inject shared fixture data, register runtime mocks, and wrap generated component renders for provider-aware DOM contracts and behavioral flow coverage
|
|
57
70
|
- provider presets can declare router, Next navigation, auth/session, React Query, Zustand, and Redux-style wrapper metadata without hand-writing every wrapper shell
|
|
58
71
|
- richer flow expectations can assert text transitions, attribute state, and role presence for empty, disabled, retry, error, and recovery paths
|
|
59
|
-
- `.themis/generate-map.json` records source-to-generated-test mappings plus scenario metadata
|
|
60
|
-
- `.themis/generate-last.json` stores the full machine-readable generate payload
|
|
61
|
-
- `.themis/generate-handoff.json` stores a compact prompt-ready handoff payload for agents
|
|
62
|
-
- `.themis/generate-backlog.json` stores unresolved skips, conflicts, and confidence debt with suggested remediation
|
|
72
|
+
- `.themis/generate/generate-map.json` records source-to-generated-test mappings plus scenario metadata
|
|
73
|
+
- `.themis/generate/generate-last.json` stores the full machine-readable generate payload
|
|
74
|
+
- `.themis/generate/generate-handoff.json` stores a compact prompt-ready handoff payload for agents
|
|
75
|
+
- `.themis/generate/generate-backlog.json` stores unresolved skips, conflicts, and confidence debt with suggested remediation
|
|
63
76
|
|
|
64
77
|
## `themis generate` options
|
|
65
78
|
|
|
@@ -165,7 +178,7 @@ Behavior:
|
|
|
165
178
|
- adds `tests/setup.themis.js` to `setupFiles`
|
|
166
179
|
- writes `themis.compat.js` as a local compatibility bridge
|
|
167
180
|
- adds `test:themis` to `package.json` when missing
|
|
168
|
-
- writes `.themis/migration-report.json` with detected compatibility imports and next actions
|
|
181
|
+
- writes `.themis/migration/migration-report.json` with detected compatibility imports and next actions
|
|
169
182
|
- relies on built-in runtime compatibility for `@jest/globals`, `vitest`, and `@testing-library/react`
|
|
170
183
|
- preserves a path to replace snapshot-heavy suites with direct assertions and generated contract tests instead of porting snapshot files as-is
|
|
171
184
|
|
|
@@ -189,9 +202,10 @@ Migration options:
|
|
|
189
202
|
| `--update-contracts` | flag | Accept updated `captureContract(...)` baselines for the selected tests. |
|
|
190
203
|
| `-w`, `--watch` | flag | Rerun the selected suite when watched project files change. |
|
|
191
204
|
| `--stability <N>` | positive integer | Run selected tests `N` times and classify stability (`stable_pass`, `stable_fail`, `unstable`). |
|
|
192
|
-
| `--html-output <path>` | string | Output path for `--reporter html` (default: `.themis/report.html`). |
|
|
205
|
+
| `--html-output <path>` | string | Output path for `--reporter html` (default: `.themis/reports/report.html`). |
|
|
193
206
|
| `--match "<regex>"` | string | Run only tests whose full name matches regex. |
|
|
194
|
-
| `--rerun-failed` | flag | Rerun failures from `.themis/failed-tests.json`. |
|
|
207
|
+
| `--rerun-failed` | flag | Rerun failures from `.themis/runs/failed-tests.json`. |
|
|
208
|
+
| `--fix` | flag | Apply generated-test autofixes from `.themis/runs/fix-handoff.json` and rerun the suite. |
|
|
195
209
|
| `--no-memes` | flag | Disable meme intent aliases (`cook`, `yeet`, `vibecheck`, `wipe`). |
|
|
196
210
|
|
|
197
211
|
Migration compatibility:
|
|
@@ -199,7 +213,7 @@ Migration compatibility:
|
|
|
199
213
|
- imports from `@jest/globals` are supported at runtime
|
|
200
214
|
- imports from `vitest` are supported at runtime
|
|
201
215
|
- imports from `@testing-library/react` are supported via Themis `render`, `screen`, `fireEvent`, `waitFor`, `cleanup`, and `act`
|
|
202
|
-
- `themis migrate <jest|vitest>` also emits `.themis/migration-report.json` with detected files and recommended next actions
|
|
216
|
+
- `themis migrate <jest|vitest>` also emits `.themis/migration/migration-report.json` with detected files and recommended next actions
|
|
203
217
|
|
|
204
218
|
Additional option:
|
|
205
219
|
|
|
@@ -235,17 +249,21 @@ Generate-specific note:
|
|
|
235
249
|
|
|
236
250
|
## Artifacts
|
|
237
251
|
|
|
238
|
-
|
|
239
|
-
|
|
240
|
-
-
|
|
241
|
-
-
|
|
242
|
-
-
|
|
243
|
-
-
|
|
244
|
-
-
|
|
245
|
-
-
|
|
246
|
-
-
|
|
247
|
-
-
|
|
248
|
-
-
|
|
252
|
+
Themis writes artifacts under `.themis/`:
|
|
253
|
+
|
|
254
|
+
- `.themis/runs/last-run.json`: full run payload (`RunResult`)
|
|
255
|
+
- `.themis/runs/failed-tests.json`: failed subset (`themis.failures.v1`)
|
|
256
|
+
- `.themis/diffs/run-diff.json`: diff against the previous run
|
|
257
|
+
- `.themis/runs/run-history.json`: rolling recent-run history
|
|
258
|
+
- `.themis/runs/fix-handoff.json`: deduped repair artifact for generated test failures (`themis.fix.handoff.v1`)
|
|
259
|
+
- `.themis/migration/migration-report.json`: migration inventory for Jest/Vitest bridge scaffolds (`themis.migration.report.v1`)
|
|
260
|
+
- `.themis/diffs/contract-diff.json`: contract capture drift, updates, and update commands (`themis.contract.diff.v1`)
|
|
261
|
+
- `.themis/generate/generate-last.json`: latest generate payload (`themis.generate.result.v1`)
|
|
262
|
+
- `.themis/generate/generate-map.json`: source-to-generated-test mapping (`themis.generate.map.v1`)
|
|
263
|
+
- `.themis/generate/generate-handoff.json`: compact agent handoff (`themis.generate.handoff.v1`)
|
|
264
|
+
- `.themis/generate/generate-backlog.json`: unresolved generation debt (`themis.generate.backlog.v1`)
|
|
265
|
+
- `.themis/benchmarks/benchmark-last.json`: latest benchmark comparison payload plus migration proof (`themis.benchmark.result.v1`)
|
|
266
|
+
- `.themis/benchmarks/migration-proof.json`: synthetic migration conversion proof emitted by `npm run benchmark` (`themis.migration.proof.v1`)
|
|
249
267
|
|
|
250
268
|
Formal schemas:
|
|
251
269
|
|
|
@@ -260,7 +278,7 @@ Formal schemas:
|
|
|
260
278
|
|
|
261
279
|
Human-facing artifact:
|
|
262
280
|
|
|
263
|
-
- `.themis/report.html`: interactive HTML verdict report
|
|
281
|
+
- `.themis/reports/report.html`: interactive HTML verdict report
|
|
264
282
|
|
|
265
283
|
Agent payload details:
|
|
266
284
|
|
|
@@ -268,8 +286,9 @@ Agent payload details:
|
|
|
268
286
|
- `analysis.failureClusters` groups failures by shared fingerprint
|
|
269
287
|
- `analysis.stability` captures multi-run classifications and per-test status sequences
|
|
270
288
|
- `analysis.comparison` reports delta stats plus new and resolved failures against the previous run
|
|
271
|
-
- `artifacts.fixHandoff` points to `.themis/fix-handoff.json` for generated failure repair loops
|
|
289
|
+
- `artifacts.fixHandoff` points to `.themis/runs/fix-handoff.json` for generated failure repair loops
|
|
272
290
|
- fix handoff entries include `repairStrategy`, `candidateFiles`, and `autofixCommand` for machine repair loops
|
|
291
|
+
- `hints.repairGenerated` points to `npx themis test --fix` for a first-party generated-suite repair command
|
|
273
292
|
|
|
274
293
|
## UI Test Utilities
|
|
275
294
|
|
|
@@ -415,15 +434,15 @@ Converts a `GenerateSummary` into the same machine-readable payload emitted by `
|
|
|
415
434
|
|
|
416
435
|
## `buildGenerateBacklogPayload(summary, cwd?): GenerateBacklogPayload`
|
|
417
436
|
|
|
418
|
-
Builds the unresolved-work artifact persisted at `.themis/generate-backlog.json`.
|
|
437
|
+
Builds the unresolved-work artifact persisted at `.themis/generate/generate-backlog.json`.
|
|
419
438
|
|
|
420
439
|
## `buildGenerateHandoff(payload): GenerateHandoffPayload`
|
|
421
440
|
|
|
422
|
-
Builds the compact handoff payload persisted at `.themis/generate-handoff.json`.
|
|
441
|
+
Builds the compact handoff payload persisted at `.themis/generate/generate-handoff.json`.
|
|
423
442
|
|
|
424
443
|
## `writeGenerateArtifacts(summary, cwd?): { payload, handoff, backlog }`
|
|
425
444
|
|
|
426
|
-
Writes `.themis/generate-last.json`, `.themis/generate-handoff.json`, and `.themis/generate-backlog.json` for a generate run and returns all three payloads.
|
|
445
|
+
Writes `.themis/generate/generate-last.json`, `.themis/generate/generate-handoff.json`, and `.themis/generate/generate-backlog.json` for a generate run and returns all three payloads.
|
|
427
446
|
|
|
428
447
|
## `loadConfig(cwd): ThemisConfig`
|
|
429
448
|
|
package/docs/migration.md
CHANGED
|
@@ -115,7 +115,7 @@ Jest/Vitest:
|
|
|
115
115
|
|
|
116
116
|
Themis:
|
|
117
117
|
|
|
118
|
-
- field-level diffs in `.themis/contract-diff.json`
|
|
118
|
+
- field-level diffs in `.themis/diffs/contract-diff.json`
|
|
119
119
|
- visible drift/update summaries in CLI and HTML reports
|
|
120
120
|
- explicit acceptance via `npx themis test --update-contracts --match "<regex>"`
|
|
121
121
|
|
|
@@ -154,7 +154,7 @@ These are the strongest head-to-head examples to use when explaining why Themis
|
|
|
154
154
|
|
|
155
155
|
1. Snapshot replacement: `captureContract(...)` plus `--update-contracts` gives baseline capture without snapshot churn.
|
|
156
156
|
2. Codemod migration: `themis migrate --convert` moves common Jest/Vitest matcher syntax toward native Themis without a manual rewrite pass.
|
|
157
|
-
3. Agent triage: `--agent`, `.themis/run-diff.json`, `.themis/fix-handoff.json`, and `.themis/contract-diff.json` give machines structured rerun and repair inputs.
|
|
157
|
+
3. Agent triage: `--agent`, `.themis/diffs/run-diff.json`, `.themis/runs/fix-handoff.json`, and `.themis/diffs/contract-diff.json` give machines structured rerun and repair inputs.
|
|
158
158
|
4. Human review: next reporter and HTML report now surface contract drift alongside failures, instead of burying meaning in raw output.
|
|
159
159
|
5. Generated coverage: `themis generate src` adds source-driven contract tests next to migrated suites, so adoption improves coverage instead of merely changing runners.
|
|
160
160
|
|
package/docs/showcases.md
CHANGED
|
@@ -30,7 +30,7 @@ Why it wins:
|
|
|
30
30
|
|
|
31
31
|
- captures only the fields that matter
|
|
32
32
|
- masks volatile IDs and timestamps instead of accepting noisy churn
|
|
33
|
-
- writes machine-readable drift to `.themis/contract-diff.json`
|
|
33
|
+
- writes machine-readable drift to `.themis/diffs/contract-diff.json`
|
|
34
34
|
- updates stay explicit with `npx themis test --update-contracts`
|
|
35
35
|
|
|
36
36
|
## 2. Migrate Jest/Vitest suites without a rewrite freeze
|
|
@@ -67,7 +67,7 @@ Why it wins:
|
|
|
67
67
|
|
|
68
68
|
- removes common compatibility imports
|
|
69
69
|
- rewrites common matcher aliases into native Themis forms
|
|
70
|
-
- emits `.themis/migration-report.json` so agents and humans can track what changed
|
|
70
|
+
- emits `.themis/migration/migration-report.json` so agents and humans can track what changed
|
|
71
71
|
|
|
72
72
|
## 3. Give agents a repair loop humans can still read
|
|
73
73
|
|
|
@@ -75,10 +75,10 @@ Jest/Vitest mostly produce console text plus optional snapshots.
|
|
|
75
75
|
|
|
76
76
|
Themis produces:
|
|
77
77
|
|
|
78
|
-
- `.themis/failed-tests.json`
|
|
79
|
-
- `.themis/run-diff.json`
|
|
80
|
-
- `.themis/fix-handoff.json`
|
|
81
|
-
- `.themis/contract-diff.json`
|
|
78
|
+
- `.themis/runs/failed-tests.json`
|
|
79
|
+
- `.themis/diffs/run-diff.json`
|
|
80
|
+
- `.themis/runs/fix-handoff.json`
|
|
81
|
+
- `.themis/diffs/contract-diff.json`
|
|
82
82
|
|
|
83
83
|
Why it wins:
|
|
84
84
|
|
|
@@ -115,3 +115,38 @@ Why it wins:
|
|
|
115
115
|
- one action accepts reviewed contracts
|
|
116
116
|
- one action reruns migration codemods for the detected framework
|
|
117
117
|
- humans and agents operate from the same source of truth
|
|
118
|
+
|
|
119
|
+
## 6. Show native Themis React tests and a same-host runner comparison in CI
|
|
120
|
+
|
|
121
|
+
CI now carries a dedicated first-party Themis React showcase job:
|
|
122
|
+
|
|
123
|
+
- `Themis React Showcase`
|
|
124
|
+
|
|
125
|
+
The Themis fixture is a straight-up native Themis jsdom suite. The checked-in Jest and Vitest fixtures are still used by the comparison benchmark job, but they no longer need their own standalone workflow jobs.
|
|
126
|
+
|
|
127
|
+
Why it wins:
|
|
128
|
+
|
|
129
|
+
- the CI page still shows a first-party native Themis React example, not only migration proof
|
|
130
|
+
- the runner-to-runner timing comparison moves into one fair same-host artifact job instead of spreading timing across separate hosts
|
|
131
|
+
- humans and agents can inspect checked-in fixture sources under `tests/fixtures/showcase/`
|
|
132
|
+
|
|
133
|
+
CI also carries a same-host performance artifact job:
|
|
134
|
+
|
|
135
|
+
- `React Showcase Perf`
|
|
136
|
+
|
|
137
|
+
That job runs the same two React scenarios under Themis, Jest, and Vitest on one runner, writes `.themis/benchmarks/showcase-comparison/perf-summary.json` and `.themis/benchmarks/showcase-comparison/perf-summary.md`, and uploads them as build artifacts.
|
|
138
|
+
|
|
139
|
+
Current documented sample result from the same-host run:
|
|
140
|
+
|
|
141
|
+
- Themis median: `1110.14ms`
|
|
142
|
+
- Vitest median: `1871.59ms`
|
|
143
|
+
- Jest median: `2556.22ms`
|
|
144
|
+
- Themis was `68.59%` faster than Vitest
|
|
145
|
+
- Themis was `130.26%` faster than Jest
|
|
146
|
+
|
|
147
|
+
Why it wins:
|
|
148
|
+
|
|
149
|
+
- the timing claim comes from one comparable host instead of three separate jobs
|
|
150
|
+
- the uploaded artifact shows median, average, min, and max wall-clock samples for the exact same showcase specs
|
|
151
|
+
- the markdown summary makes it obvious whether Themis was faster or slower in that specific CI run
|
|
152
|
+
- the docs now state the current measured gap directly instead of making readers hunt through the artifact first
|
package/docs/vscode-extension.md
CHANGED
|
@@ -5,29 +5,29 @@ Themis includes an in-repo VS Code extension scaffold at [`packages/themis-vscod
|
|
|
5
5
|
This is the intended shape of the editor UX:
|
|
6
6
|
|
|
7
7
|
- a Themis activity-bar container
|
|
8
|
-
- a results sidebar driven by `.themis
|
|
8
|
+
- a results sidebar driven by `.themis/**` artifacts
|
|
9
9
|
- commands to run tests, rerun failures, refresh results, and open the HTML report
|
|
10
10
|
- commands to accept reviewed contract baselines and rerun migration codemods
|
|
11
11
|
- failure navigation that jumps from artifact data into the source file
|
|
12
12
|
- generated-review navigation for source files, generated tests, hint sidecars, and backlog items
|
|
13
|
-
- contract-review and migration-review groups driven by `.themis/contract-diff.json` and `.themis/migration-report.json`
|
|
13
|
+
- contract-review and migration-review groups driven by `.themis/diffs/contract-diff.json` and `.themis/migration/migration-report.json`
|
|
14
14
|
|
|
15
15
|
## Current MVP Scope
|
|
16
16
|
|
|
17
17
|
The scaffold currently supports:
|
|
18
18
|
|
|
19
|
-
- reading `.themis/last-run.json`
|
|
20
|
-
- reading `.themis/failed-tests.json`
|
|
21
|
-
- reading `.themis/run-diff.json`
|
|
22
|
-
- reading `.themis/contract-diff.json`
|
|
23
|
-
- reading `.themis/migration-report.json`
|
|
24
|
-
- reading `.themis/generate-last.json`
|
|
25
|
-
- reading `.themis/generate-map.json`
|
|
26
|
-
- reading `.themis/generate-backlog.json`
|
|
27
|
-
- opening `.themis/report.html` in a webview
|
|
19
|
+
- reading `.themis/runs/last-run.json`
|
|
20
|
+
- reading `.themis/runs/failed-tests.json`
|
|
21
|
+
- reading `.themis/diffs/run-diff.json`
|
|
22
|
+
- reading `.themis/diffs/contract-diff.json`
|
|
23
|
+
- reading `.themis/migration/migration-report.json`
|
|
24
|
+
- reading `.themis/generate/generate-last.json`
|
|
25
|
+
- reading `.themis/generate/generate-map.json`
|
|
26
|
+
- reading `.themis/generate/generate-backlog.json`
|
|
27
|
+
- opening `.themis/reports/report.html` in a webview
|
|
28
28
|
- surfacing generated source/test/hint mappings in the sidebar
|
|
29
29
|
- surfacing unresolved generate backlog and gate state in the sidebar
|
|
30
|
-
- refreshing when `.themis
|
|
30
|
+
- refreshing when `.themis/**` files change
|
|
31
31
|
|
|
32
32
|
The extension is intentionally thin. It shells out to Themis commands and treats the CLI plus artifacts as the canonical contract.
|
|
33
33
|
|
package/docs/why-themis.md
CHANGED
|
@@ -17,6 +17,8 @@ npx themis generate src
|
|
|
17
17
|
npx themis test
|
|
18
18
|
```
|
|
19
19
|
|
|
20
|
+
`npx themis init` adds `.themis/` to `.gitignore`, and `npx themis generate src` emits `.generated.test.ts` for TS/TSX sources and `.generated.test.js` for JS/JSX sources.
|
|
21
|
+
|
|
20
22
|
For downstream repo instructions and a copyable `AGENTS.md` template, see [`docs/agents-adoption.md`](agents-adoption.md).
|
|
21
23
|
|
|
22
24
|
## What "Next-Gen" Means Here
|
|
@@ -48,7 +50,7 @@ Themis supports structured outputs for tooling loops:
|
|
|
48
50
|
|
|
49
51
|
- `--json` for generic automation
|
|
50
52
|
- `--agent` for AI-agent workflows
|
|
51
|
-
- `.themis/failed-tests.json` for deterministic reruns
|
|
53
|
+
- `.themis/runs/failed-tests.json` for deterministic reruns
|
|
52
54
|
|
|
53
55
|
## 3) Deterministic Rerun Workflow
|
|
54
56
|
|
|
@@ -86,7 +88,7 @@ Themis ships workflow features agents can use directly:
|
|
|
86
88
|
- generated contract tests that keep baselines in readable source instead of opaque snapshot blobs
|
|
87
89
|
- machine-readable artifacts that let agents inspect and explain changes before updating tests
|
|
88
90
|
- mocks and spies with `fn`, `spyOn`, and `mock`
|
|
89
|
-
- `.themis/run-diff.json` and `.themis/run-history.json`
|
|
91
|
+
- `.themis/diffs/run-diff.json` and `.themis/runs/run-history.json`
|
|
90
92
|
- HTML verdict reports for human review
|
|
91
93
|
|
|
92
94
|
### Comparable To Snapshot Workflows, Without Snapshot Rot
|
|
@@ -130,7 +132,7 @@ Performance is measured and guarded:
|
|
|
130
132
|
|
|
131
133
|
## 11) Editor Surface Without Replacing The CLI
|
|
132
134
|
|
|
133
|
-
Themis includes a thin VS Code extension scaffold that reads `.themis
|
|
135
|
+
Themis includes a thin VS Code extension scaffold that reads `.themis/**` artifacts, reruns tests, and opens the HTML report. The CLI remains the source of truth.
|
|
134
136
|
|
|
135
137
|
## Proof Checklist
|
|
136
138
|
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@vitronai/themis",
|
|
3
|
-
"version": "0.1.
|
|
3
|
+
"version": "0.1.6",
|
|
4
4
|
"description": "Intent-first unit test framework and test generator for AI agents in Node.js and TypeScript",
|
|
5
5
|
"license": "MIT",
|
|
6
6
|
"author": "Vitron AI",
|
|
@@ -54,6 +54,10 @@
|
|
|
54
54
|
"require": "./globals.js",
|
|
55
55
|
"default": "./globals.js"
|
|
56
56
|
},
|
|
57
|
+
"./contract-runtime": {
|
|
58
|
+
"require": "./src/contract-runtime.js",
|
|
59
|
+
"default": "./src/contract-runtime.js"
|
|
60
|
+
},
|
|
57
61
|
"./package.json": "./package.json"
|
|
58
62
|
},
|
|
59
63
|
"files": [
|
|
@@ -75,14 +79,16 @@
|
|
|
75
79
|
"themis": "bin/themis.js"
|
|
76
80
|
},
|
|
77
81
|
"scripts": {
|
|
82
|
+
"lint": "eslint .",
|
|
78
83
|
"test": "node bin/themis.js test",
|
|
79
|
-
"validate": "npm test && npm run typecheck && npm run benchmark:gate",
|
|
84
|
+
"validate": "npm run lint && npm test && npm run typecheck && npm run benchmark:gate",
|
|
80
85
|
"typecheck": "tsc -p tsconfig.json --pretty false",
|
|
81
86
|
"benchmark": "node scripts/benchmark.js",
|
|
87
|
+
"benchmark:showcase": "node scripts/benchmark-showcase-runners.js",
|
|
82
88
|
"benchmark:gate": "node scripts/benchmark-gate.js",
|
|
83
89
|
"proof:migration": "node scripts/verify-migration-fixtures.js",
|
|
84
90
|
"pack:check": "npm pack --dry-run",
|
|
85
|
-
"prepublishOnly": "npm test && npm run typecheck"
|
|
91
|
+
"prepublishOnly": "npm run lint && npm test && npm run typecheck"
|
|
86
92
|
},
|
|
87
93
|
"publishConfig": {
|
|
88
94
|
"access": "public"
|
|
@@ -95,6 +101,13 @@
|
|
|
95
101
|
"typescript": "^5.9.3"
|
|
96
102
|
},
|
|
97
103
|
"devDependencies": {
|
|
98
|
-
"react": "^
|
|
104
|
+
"@testing-library/react": "^16.3.2",
|
|
105
|
+
"eslint": "^9.22.0",
|
|
106
|
+
"globals": "^17.4.0",
|
|
107
|
+
"jest": "^29.7.0",
|
|
108
|
+
"jest-environment-jsdom": "^29.7.0",
|
|
109
|
+
"react": "^19.0.0",
|
|
110
|
+
"react-dom": "^19.2.4",
|
|
111
|
+
"vitest": "^1.6.1"
|
|
99
112
|
}
|
|
100
113
|
}
|