@vitronai/themis 0.1.0-beta.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +41 -0
- package/LICENSE +21 -0
- package/README.md +285 -0
- package/benchmark-gate.json +10 -0
- package/bin/themis.js +8 -0
- package/docs/api.md +210 -0
- package/docs/publish.md +55 -0
- package/docs/release-policy.md +54 -0
- package/docs/schemas/agent-result.v1.json +277 -0
- package/docs/schemas/failures.v1.json +78 -0
- package/docs/vscode-extension.md +40 -0
- package/docs/why-themis.md +111 -0
- package/globals.d.ts +22 -0
- package/globals.js +1 -0
- package/index.d.ts +190 -0
- package/index.js +17 -0
- package/package.json +90 -0
- package/src/artifacts.js +207 -0
- package/src/assets/themisBg.png +0 -0
- package/src/assets/themisLogo.png +0 -0
- package/src/assets/themisReport.png +0 -0
- package/src/cli.js +395 -0
- package/src/config.js +52 -0
- package/src/discovery.js +34 -0
- package/src/environment.js +108 -0
- package/src/expect.js +175 -0
- package/src/init.js +22 -0
- package/src/module-loader.js +489 -0
- package/src/reporter.js +2141 -0
- package/src/runner.js +168 -0
- package/src/runtime.js +472 -0
- package/src/snapshots.js +90 -0
- package/src/stability.js +98 -0
- package/src/test-utils.js +201 -0
- package/src/verdict.js +71 -0
- package/src/watch.js +154 -0
- package/src/worker.js +26 -0
package/CHANGELOG.md
ADDED
|
@@ -0,0 +1,41 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
All notable changes to this project are documented in this file.
|
|
4
|
+
|
|
5
|
+
## Unreleased
|
|
6
|
+
|
|
7
|
+
### Changed
|
|
8
|
+
|
|
9
|
+
- Moved the package line from alpha to beta and defined beta compatibility expectations for CLI, artifacts, and the JS/TS package surface.
|
|
10
|
+
- Tightened npm/package positioning around Themis as an intent-first unit test framework for AI agents in Node.js and TypeScript.
|
|
11
|
+
- Updated publish-facing docs to consistently frame Themis as an AI verdict engine for human and agent review loops.
|
|
12
|
+
- Added `npm run pack:check` for npm publish payload validation.
|
|
13
|
+
- Expanded the HTML report with quick actions, failure-first triage, richer file cards, and test-level drilldown.
|
|
14
|
+
- Added embedded brand/report asset visuals to the README.
|
|
15
|
+
- Added project-aware JS/TS runtime support for TSX, ESM `.js`, `tsconfig` path aliases, `setupFiles`, and `jsdom` environments.
|
|
16
|
+
- Added first-party snapshots, mocks, run-diff artifacts, and watch mode for tighter agent rerun loops.
|
|
17
|
+
- Added an in-repo VS Code extension scaffold for artifact-driven result viewing, reruns, and HTML report opening.
|
|
18
|
+
- Refreshed README, AGENTS, and supporting docs to match the current package scope, JS/TS feature set, artifact contracts, and extension surface.
|
|
19
|
+
|
|
20
|
+
## 0.1.0-alpha.1 - 2026-02-13
|
|
21
|
+
|
|
22
|
+
### Added
|
|
23
|
+
|
|
24
|
+
- Intent integration test suite with deterministic fixture coverage.
|
|
25
|
+
- Async phase, ordering, cleanup, and failure-priority intent tests.
|
|
26
|
+
- TypeScript definitions for package API, globals, and intent DSL.
|
|
27
|
+
- Typecheck lane (`npm run typecheck`).
|
|
28
|
+
- Benchmark gate command (`npm run benchmark:gate`) and config (`benchmark-gate.json`).
|
|
29
|
+
- Branded CLI banner for human-readable reporters.
|
|
30
|
+
- Snapshot-based CLI output tests with normalization for timing/path/version variance.
|
|
31
|
+
- Agent test-authoring contract (`AGENTS.md`).
|
|
32
|
+
- Community positioning doc (`docs/why-themis.md`).
|
|
33
|
+
- Full API reference (`docs/api.md`) and release policy (`docs/release-policy.md`).
|
|
34
|
+
- Formal JSON schema docs for agent and failures payloads.
|
|
35
|
+
|
|
36
|
+
### Changed
|
|
37
|
+
|
|
38
|
+
- Preferred intent phases are now `context/run/verify/cleanup`.
|
|
39
|
+
- Legacy phase aliases remain supported for compatibility.
|
|
40
|
+
- Added optional meme aliases (`cook/yeet/vibecheck/wipe`) and strict disable flag (`--no-memes`).
|
|
41
|
+
- Hardened npm package metadata, exports, files whitelist, and publish config.
|
package/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 vitron.ai
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
package/README.md
ADDED
|
@@ -0,0 +1,285 @@
|
|
|
1
|
+
# Themis
|
|
2
|
+
|
|
3
|
+
<p align="center">
|
|
4
|
+
<img src="src/assets/themisLogo.png" alt="Themis logo mark" width="42" valign="middle">
|
|
5
|
+
<a href="https://github.com/vitron-ai/themis/actions/workflows/ci.yml">
|
|
6
|
+
<img src="https://img.shields.io/github/actions/workflow/status/vitron-ai/themis/ci.yml?branch=main&style=for-the-badge&label=THEMIS%20VERDICT%20PIPELINE&labelColor=111827&color=16a34a" alt="Themis verdict pipeline status" valign="middle">
|
|
7
|
+
</a>
|
|
8
|
+
</p>
|
|
9
|
+
|
|
10
|
+
Themis is an intent-first unit test framework for AI agents in Node.js and TypeScript.
|
|
11
|
+
|
|
12
|
+
It is built to be the best test loop for agent workflows: deterministic reruns, machine-readable outputs, strict phase semantics, and a branded AI verdict engine for humans.
|
|
13
|
+
|
|
14
|
+
<p align="center">
|
|
15
|
+
<img src="src/assets/themisReport.png" alt="Themis HTML verdict report" width="960">
|
|
16
|
+
</p>
|
|
17
|
+
|
|
18
|
+
## Contents
|
|
19
|
+
|
|
20
|
+
- [Quickstart](#quickstart)
|
|
21
|
+
- [Positioning](#positioning)
|
|
22
|
+
- [Modern JS/TS Support](#modern-jsts-support)
|
|
23
|
+
- [Commands](#commands)
|
|
24
|
+
- [Agent Guide](#agent-guide)
|
|
25
|
+
- [VS Code](#vs-code)
|
|
26
|
+
- [Snapshots And Mocks](#snapshots-and-mocks)
|
|
27
|
+
- [Intent Syntax](#intent-syntax)
|
|
28
|
+
- [Config](#config)
|
|
29
|
+
- [TypeScript](#typescript)
|
|
30
|
+
- [Benchmark](#benchmark)
|
|
31
|
+
- [Publish Readiness](#publish-readiness)
|
|
32
|
+
- [Why Themis](docs/why-themis.md)
|
|
33
|
+
- [API Reference](docs/api.md)
|
|
34
|
+
- [Release Policy](docs/release-policy.md)
|
|
35
|
+
- [Publish Guide](docs/publish.md)
|
|
36
|
+
- [Contributing](CONTRIBUTING.md)
|
|
37
|
+
- [Security](SECURITY.md)
|
|
38
|
+
|
|
39
|
+
## Positioning
|
|
40
|
+
|
|
41
|
+
- Best-in-class unit testing for AI agents in Node.js and TypeScript
|
|
42
|
+
- Deterministic execution with fast rerun loops
|
|
43
|
+
- Agent-native JSON and HTML reporting
|
|
44
|
+
- AI verdict engine for human triage and machine automation
|
|
45
|
+
|
|
46
|
+
## Modern JS/TS Support
|
|
47
|
+
|
|
48
|
+
Themis is built for modern Node.js and TypeScript projects:
|
|
49
|
+
|
|
50
|
+
- `.js`, `.jsx`, `.ts`, and `.tsx` support
|
|
51
|
+
- ESM `.js` loading in `type: "module"` projects
|
|
52
|
+
- `tsconfig` path alias resolution
|
|
53
|
+
- `node` and `jsdom` environments
|
|
54
|
+
- `setupFiles` for harness bootstrapping
|
|
55
|
+
- first-party snapshots, mocks, and spies
|
|
56
|
+
- `--watch` and `--rerun-failed` for tight local and agent rerun loops
|
|
57
|
+
|
|
58
|
+
## Visuals
|
|
59
|
+
|
|
60
|
+
- Real Themis brand mark: [`src/assets/themisLogo.png`](src/assets/themisLogo.png)
|
|
61
|
+
- HTML verdict report art: [`src/assets/themisReport.png`](src/assets/themisReport.png)
|
|
62
|
+
- Background art used by the report: [`src/assets/themisBg.png`](src/assets/themisBg.png)
|
|
63
|
+
|
|
64
|
+
## Quickstart
|
|
65
|
+
|
|
66
|
+
```bash
|
|
67
|
+
npm install -D @vitronai/themis
|
|
68
|
+
npx themis init
|
|
69
|
+
npx themis test
|
|
70
|
+
```
|
|
71
|
+
|
|
72
|
+
Generate the next-gen HTML report:
|
|
73
|
+
|
|
74
|
+
```bash
|
|
75
|
+
npx themis test --reporter html
|
|
76
|
+
```
|
|
77
|
+
|
|
78
|
+
Use the AI-agent payload:
|
|
79
|
+
|
|
80
|
+
```bash
|
|
81
|
+
npx themis test --agent
|
|
82
|
+
```
|
|
83
|
+
|
|
84
|
+
Stay in a local rerun loop while editing:
|
|
85
|
+
|
|
86
|
+
```bash
|
|
87
|
+
npx themis test --watch --reporter next
|
|
88
|
+
```
|
|
89
|
+
|
|
90
|
+
## Why Themis
|
|
91
|
+
|
|
92
|
+
See [`docs/why-themis.md`](docs/why-themis.md) for positioning, differentiators, and community messaging.
|
|
93
|
+
|
|
94
|
+
## Reference Docs
|
|
95
|
+
|
|
96
|
+
- API reference: [`docs/api.md`](docs/api.md)
|
|
97
|
+
- Release policy: [`docs/release-policy.md`](docs/release-policy.md)
|
|
98
|
+
- Publish guide: [`docs/publish.md`](docs/publish.md)
|
|
99
|
+
- VS Code extension notes: [`docs/vscode-extension.md`](docs/vscode-extension.md)
|
|
100
|
+
- Agent result schema: [`docs/schemas/agent-result.v1.json`](docs/schemas/agent-result.v1.json)
|
|
101
|
+
- Failures artifact schema: [`docs/schemas/failures.v1.json`](docs/schemas/failures.v1.json)
|
|
102
|
+
- Changelog: [`CHANGELOG.md`](CHANGELOG.md)
|
|
103
|
+
- Contributing: [`CONTRIBUTING.md`](CONTRIBUTING.md)
|
|
104
|
+
- Security: [`SECURITY.md`](SECURITY.md)
|
|
105
|
+
|
|
106
|
+
## Commands
|
|
107
|
+
|
|
108
|
+
- `npx themis init`: creates `themis.config.json` and a sample test.
|
|
109
|
+
- `npx themis test`: discovers and runs tests.
|
|
110
|
+
- `npx themis test --next`: next-gen console output mode.
|
|
111
|
+
- `npx themis test --json`: emits JSON result payload.
|
|
112
|
+
- `npx themis test --agent`: emits AI-agent-oriented JSON schema.
|
|
113
|
+
- `npx themis test --reporter html`: generates a next-gen HTML report file.
|
|
114
|
+
- `npx themis test --reporter html --html-output reports/themis.html`: writes HTML report to a custom path.
|
|
115
|
+
- `npx themis test --watch`: reruns the suite when watched project files change.
|
|
116
|
+
- `npx themis test --workers 8`: overrides worker count (positive integer).
|
|
117
|
+
- `npx themis test --environment jsdom`: runs tests in a browser-like DOM environment.
|
|
118
|
+
- `npx themis test -u`: updates stored snapshots.
|
|
119
|
+
- `npx themis test --stability 3`: runs the suite three times and classifies each test as `stable_pass`, `stable_fail`, or `unstable`.
|
|
120
|
+
- `npx themis test --match "intent DSL"`: runs only tests whose full name matches regex.
|
|
121
|
+
- `npx themis test --rerun-failed`: reruns failing tests from `.themis/failed-tests.json`.
|
|
122
|
+
- `npx themis test --no-memes`: disables meme phase aliases (`cook`, `yeet`, `vibecheck`, `wipe`).
|
|
123
|
+
- `npx themis test --lexicon classic|themis`: rebrands human-readable status labels in `next/spec` reporters.
|
|
124
|
+
- `npm run validate`: runs test, typecheck, and benchmark gate in one command.
|
|
125
|
+
- `npm run typecheck`: validates TypeScript types for Themis globals and DSL contracts.
|
|
126
|
+
- `npm run benchmark:gate`: fails when benchmark performance exceeds the configured threshold.
|
|
127
|
+
- `npm run pack:check`: previews the npm publish payload.
|
|
128
|
+
|
|
129
|
+
## Agent Guide
|
|
130
|
+
|
|
131
|
+
See [`AGENTS.md`](AGENTS.md) for the AI-agent test authoring contract used in this repository.
|
|
132
|
+
|
|
133
|
+
Each run writes artifacts to `.themis/`:
|
|
134
|
+
|
|
135
|
+
- `last-run.json`: full machine-readable run payload.
|
|
136
|
+
- `failed-tests.json`: compact failure list for retry loops.
|
|
137
|
+
- `run-diff.json`: diff against the previous run, including new and resolved failures.
|
|
138
|
+
- `run-history.json`: rolling recent-run history for agent comparison loops.
|
|
139
|
+
- `report.html`: interactive HTML verdict report.
|
|
140
|
+
|
|
141
|
+
`--agent` output includes deterministic failure fingerprints, grouped `analysis.failureClusters`, stability classifications, and previous-run comparison data to help AI agents deduplicate and prioritize failures.
|
|
142
|
+
|
|
143
|
+
Machine-facing reporters intentionally emit compact JSON. Agents and tooling should parse the payloads rather than depend on whitespace formatting.
|
|
144
|
+
|
|
145
|
+
The HTML reporter is designed for agent-adjacent review workflows too: it combines verdict status, slow-test surfacing, artifact navigation, and interactive file filtering in one report.
|
|
146
|
+
|
|
147
|
+
## VS Code
|
|
148
|
+
|
|
149
|
+
The repo now includes a thin VS Code extension scaffold at [`packages/themis-vscode`](packages/themis-vscode).
|
|
150
|
+
|
|
151
|
+
The extension is intentionally artifact-driven:
|
|
152
|
+
|
|
153
|
+
- reads `.themis/last-run.json`, `.themis/failed-tests.json`, `.themis/run-diff.json`, and `.themis/report.html`
|
|
154
|
+
- shows the latest verdict and failures in a sidebar
|
|
155
|
+
- reruns Themis from VS Code commands
|
|
156
|
+
- opens the HTML report inside a webview
|
|
157
|
+
|
|
158
|
+
It does not replace the CLI. The CLI and `.themis/*` artifacts remain the source of truth.
|
|
159
|
+
|
|
160
|
+
## Snapshots And Mocks
|
|
161
|
+
|
|
162
|
+
Themis now ships first-party test utilities for agent-generated tests:
|
|
163
|
+
|
|
164
|
+
```js
|
|
165
|
+
mock('../src/api', () => ({
|
|
166
|
+
fetchUser: fn(() => ({ id: 'u_1', name: 'Ada' }))
|
|
167
|
+
}));
|
|
168
|
+
|
|
169
|
+
const { fetchUser } = require('../src/api');
|
|
170
|
+
|
|
171
|
+
test('captures a stable UI contract', () => {
|
|
172
|
+
const user = fetchUser();
|
|
173
|
+
expect(fetchUser).toHaveBeenCalledTimes(1);
|
|
174
|
+
expect(user).toMatchSnapshot();
|
|
175
|
+
});
|
|
176
|
+
```
|
|
177
|
+
|
|
178
|
+
Available globals:
|
|
179
|
+
|
|
180
|
+
- `fn(...)`
|
|
181
|
+
- `spyOn(object, methodName)`
|
|
182
|
+
- `mock(moduleId, factoryOrExports)`
|
|
183
|
+
- `unmock(moduleId)`
|
|
184
|
+
- `clearAllMocks()`
|
|
185
|
+
- `resetAllMocks()`
|
|
186
|
+
- `restoreAllMocks()`
|
|
187
|
+
- `expect(value).toMatchSnapshot()`
|
|
188
|
+
|
|
189
|
+
## Intent Syntax
|
|
190
|
+
|
|
191
|
+
Themis supports a strict code-native intent DSL:
|
|
192
|
+
|
|
193
|
+
```js
|
|
194
|
+
intent('user can sign in', ({ context, run, verify, cleanup }) => {
|
|
195
|
+
context('a valid user', (ctx) => {
|
|
196
|
+
ctx.user = { email: 'a@b.com', password: 'pw' };
|
|
197
|
+
});
|
|
198
|
+
|
|
199
|
+
run('the user submits credentials', (ctx) => {
|
|
200
|
+
ctx.result = { ok: true };
|
|
201
|
+
});
|
|
202
|
+
|
|
203
|
+
verify('authentication succeeds', (ctx) => {
|
|
204
|
+
expect(ctx.result.ok).toBe(true);
|
|
205
|
+
});
|
|
206
|
+
|
|
207
|
+
cleanup('remove test state', (ctx) => {
|
|
208
|
+
delete ctx.user;
|
|
209
|
+
});
|
|
210
|
+
});
|
|
211
|
+
```
|
|
212
|
+
|
|
213
|
+
Preferred phase names are `context`, `run`, `verify`, `cleanup`.
|
|
214
|
+
Legacy aliases are still supported (`arrange/act/assert`, `given/when/then`, `setup/infer/teardown`).
|
|
215
|
+
Easter egg aliases are also available: `cook`, `yeet`, `vibecheck`, `wipe`.
|
|
216
|
+
|
|
217
|
+
## Config
|
|
218
|
+
|
|
219
|
+
`themis.config.json`
|
|
220
|
+
|
|
221
|
+
```json
|
|
222
|
+
{
|
|
223
|
+
"testDir": "tests",
|
|
224
|
+
"testRegex": "\\.(test|spec)\\.(js|jsx|ts|tsx)$",
|
|
225
|
+
"maxWorkers": 7,
|
|
226
|
+
"reporter": "next",
|
|
227
|
+
"environment": "node",
|
|
228
|
+
"setupFiles": ["tests/setup.ts"],
|
|
229
|
+
"tsconfigPath": "tsconfig.json"
|
|
230
|
+
}
|
|
231
|
+
```
|
|
232
|
+
|
|
233
|
+
Modern JS/TS projects can opt into `environment: "jsdom"` for DOM-driven tests and `setupFiles` for hooks, polyfills, or harness bootstrapping.
|
|
234
|
+
|
|
235
|
+
## TypeScript
|
|
236
|
+
|
|
237
|
+
The package ships first-party typings for:
|
|
238
|
+
|
|
239
|
+
- programmatic APIs (`collectAndRun`, `runTests`, config helpers)
|
|
240
|
+
- global test APIs (`describe`, `test`, `intent`, hooks, `expect`)
|
|
241
|
+
- typed intent context (`intent<MyCtx>(...)`)
|
|
242
|
+
- project-aware module loading for `ts`, `tsx`, ESM `js`, `jsx`, `tsconfig` path aliases, and setup files
|
|
243
|
+
|
|
244
|
+
Themis is designed to feel native in modern TypeScript projects without requiring a separate Babel or ts-node setup just to run tests.
|
|
245
|
+
|
|
246
|
+
Use the global types in your project with:
|
|
247
|
+
|
|
248
|
+
```json
|
|
249
|
+
{
|
|
250
|
+
"compilerOptions": {
|
|
251
|
+
"types": ["@vitronai/themis/globals"]
|
|
252
|
+
}
|
|
253
|
+
}
|
|
254
|
+
```
|
|
255
|
+
|
|
256
|
+
## Benchmark
|
|
257
|
+
|
|
258
|
+
```bash
|
|
259
|
+
npm run benchmark
|
|
260
|
+
npm run benchmark:gate
|
|
261
|
+
```
|
|
262
|
+
|
|
263
|
+
Optional env vars:
|
|
264
|
+
|
|
265
|
+
- `BENCH_FILES` (default `40`)
|
|
266
|
+
- `BENCH_TESTS_PER_FILE` (default `25`)
|
|
267
|
+
- `BENCH_REPEATS` (default `3`)
|
|
268
|
+
- `BENCH_WORKERS` (default `4`)
|
|
269
|
+
- `BENCH_INCLUDE_EXTERNAL=1` to include Jest/Vitest/Bun comparisons
|
|
270
|
+
- `BENCH_MAX_AVG_MS` to override the gate threshold
|
|
271
|
+
- `BENCH_GATE_CONFIG` to point `benchmark:gate` at a custom config file
|
|
272
|
+
|
|
273
|
+
The benchmark always runs Themis and will include Jest/Vitest/Bun only when they are available locally.
|
|
274
|
+
The default gate profile and threshold live in `benchmark-gate.json`.
|
|
275
|
+
|
|
276
|
+
## Publish Readiness
|
|
277
|
+
|
|
278
|
+
Before publishing a release:
|
|
279
|
+
|
|
280
|
+
```bash
|
|
281
|
+
npm run validate
|
|
282
|
+
npm run pack:check
|
|
283
|
+
```
|
|
284
|
+
|
|
285
|
+
The npm package should ship a clean CLI, first-party typings, schemas, docs, and report assets.
|
package/bin/themis.js
ADDED
package/docs/api.md
ADDED
|
@@ -0,0 +1,210 @@
|
|
|
1
|
+
# Themis API Reference
|
|
2
|
+
|
|
3
|
+
This document defines the public API surface for Themis `0.1.0-beta.0`.
|
|
4
|
+
|
|
5
|
+
## CLI
|
|
6
|
+
|
|
7
|
+
## Command
|
|
8
|
+
|
|
9
|
+
```bash
|
|
10
|
+
themis test [options]
|
|
11
|
+
themis init
|
|
12
|
+
```
|
|
13
|
+
|
|
14
|
+
## `themis init`
|
|
15
|
+
|
|
16
|
+
Creates:
|
|
17
|
+
|
|
18
|
+
- `themis.config.json` with default settings
|
|
19
|
+
- `tests/example.test.js` sample test (if missing)
|
|
20
|
+
|
|
21
|
+
## `themis test` options
|
|
22
|
+
|
|
23
|
+
| Option | Type | Description |
|
|
24
|
+
| --- | --- | --- |
|
|
25
|
+
| `--json` | flag | Print full run payload JSON (`RunResult`). |
|
|
26
|
+
| `--agent` | flag | Print AI-agent-oriented JSON payload (`themis.agent.result.v1`) with failure fingerprints and cluster analysis. |
|
|
27
|
+
| `--next` | flag | Use next-gen human reporter. |
|
|
28
|
+
| `--reporter spec\|next\|json\|agent\|html` | string | Explicit reporter override. |
|
|
29
|
+
| `--workers <N>` | positive integer | Override worker count. Invalid values fail fast. |
|
|
30
|
+
| `--environment node\|jsdom` | string | Override the configured test environment. |
|
|
31
|
+
| `-w`, `--watch` | flag | Rerun the selected suite when watched project files change. |
|
|
32
|
+
| `-u`, `--update-snapshots` | flag | Update snapshot files when `toMatchSnapshot()` values change. |
|
|
33
|
+
| `--stability <N>` | positive integer | Run selected tests `N` times and classify stability (`stable_pass`, `stable_fail`, `unstable`). |
|
|
34
|
+
| `--html-output <path>` | string | Output path for `--reporter html` (default: `.themis/report.html`). |
|
|
35
|
+
| `--match "<regex>"` | string | Run only tests whose full name matches regex. |
|
|
36
|
+
| `--rerun-failed` | flag | Rerun failures from `.themis/failed-tests.json`. |
|
|
37
|
+
| `--no-memes` | flag | Disable meme intent aliases (`cook`, `yeet`, `vibecheck`, `wipe`). |
|
|
38
|
+
| `--lexicon classic\|themis` | string | Human reporter terminology mode for `next/spec`. |
|
|
39
|
+
|
|
40
|
+
## Exit behavior
|
|
41
|
+
|
|
42
|
+
- Exit code `0`: all selected tests passed or were skipped.
|
|
43
|
+
- Exit code `1`: any selected test failed, invalid usage, or runtime error.
|
|
44
|
+
|
|
45
|
+
`--lexicon` does not affect machine payload contracts (`--json`, `--agent`, or artifacts).
|
|
46
|
+
|
|
47
|
+
## Artifacts
|
|
48
|
+
|
|
49
|
+
Each run writes to `.themis/`:
|
|
50
|
+
|
|
51
|
+
- `last-run.json`: full run payload (`RunResult`)
|
|
52
|
+
- `failed-tests.json`: failed subset (`themis.failures.v1`)
|
|
53
|
+
- `run-diff.json`: diff against the previous run
|
|
54
|
+
- `run-history.json`: rolling recent-run history
|
|
55
|
+
|
|
56
|
+
Formal schemas:
|
|
57
|
+
|
|
58
|
+
- `docs/schemas/agent-result.v1.json`
|
|
59
|
+
- `docs/schemas/failures.v1.json`
|
|
60
|
+
|
|
61
|
+
Human-facing artifact:
|
|
62
|
+
|
|
63
|
+
- `.themis/report.html`: interactive HTML verdict report
|
|
64
|
+
|
|
65
|
+
Agent payload details:
|
|
66
|
+
|
|
67
|
+
- each `failures[]` entry includes a deterministic `fingerprint`
|
|
68
|
+
- `analysis.failureClusters` groups failures by shared fingerprint
|
|
69
|
+
- `analysis.stability` captures multi-run classifications and per-test status sequences
|
|
70
|
+
- `analysis.comparison` reports delta stats plus new and resolved failures against the previous run
|
|
71
|
+
|
|
72
|
+
## Config File (`themis.config.json`)
|
|
73
|
+
|
|
74
|
+
| Field | Type | Default | Notes |
|
|
75
|
+
| --- | --- | --- | --- |
|
|
76
|
+
| `testDir` | string | `"tests"` | Root directory for discovery. |
|
|
77
|
+
| `testRegex` | string | `"\\.(test\|spec)\\.(js\|jsx\|ts\|tsx)$"` | Regex for filenames. |
|
|
78
|
+
| `maxWorkers` | integer | `max(1, cpuCount - 1)` | Parallel worker limit. |
|
|
79
|
+
| `reporter` | `spec\|next\|json\|agent\|html` | `"next"` | Default reporter when no CLI override is set. |
|
|
80
|
+
| `environment` | `node\|jsdom` | `"node"` | Test runtime environment. |
|
|
81
|
+
| `setupFiles` | `string[]` | `[]` | Files loaded before each test file. |
|
|
82
|
+
| `tsconfigPath` | `string \| null` | `"tsconfig.json"` | Project tsconfig used for TSX and alias-aware transpilation. |
|
|
83
|
+
|
|
84
|
+
## Programmatic API
|
|
85
|
+
|
|
86
|
+
Import:
|
|
87
|
+
|
|
88
|
+
```js
|
|
89
|
+
const themis = require('@vitronai/themis');
|
|
90
|
+
```
|
|
91
|
+
|
|
92
|
+
## `main(argv: string[]): Promise<void>`
|
|
93
|
+
|
|
94
|
+
Programmatic entrypoint for CLI behavior.
|
|
95
|
+
|
|
96
|
+
## `collectAndRun(filePath, options?): Promise<FileResult>`
|
|
97
|
+
|
|
98
|
+
Runs tests in one file in-process with global API installation.
|
|
99
|
+
|
|
100
|
+
Options:
|
|
101
|
+
|
|
102
|
+
- `match?: string | null`
|
|
103
|
+
- `allowedFullNames?: string[] | null`
|
|
104
|
+
- `noMemes?: boolean`
|
|
105
|
+
- `cwd?: string`
|
|
106
|
+
- `environment?: "node" | "jsdom"`
|
|
107
|
+
- `setupFiles?: string[]`
|
|
108
|
+
- `tsconfigPath?: string | null`
|
|
109
|
+
- `updateSnapshots?: boolean`
|
|
110
|
+
|
|
111
|
+
## `runTests(files, options?): Promise<RunResult>`
|
|
112
|
+
|
|
113
|
+
Runs multiple files in worker threads.
|
|
114
|
+
|
|
115
|
+
Options:
|
|
116
|
+
|
|
117
|
+
- `maxWorkers?: number`
|
|
118
|
+
- `match?: string | null`
|
|
119
|
+
- `allowedFullNames?: string[] | null`
|
|
120
|
+
- `noMemes?: boolean`
|
|
121
|
+
- `cwd?: string`
|
|
122
|
+
- `environment?: "node" | "jsdom"`
|
|
123
|
+
- `setupFiles?: string[]`
|
|
124
|
+
- `tsconfigPath?: string | null`
|
|
125
|
+
- `updateSnapshots?: boolean`
|
|
126
|
+
|
|
127
|
+
## `discoverTests(cwd, config): string[]`
|
|
128
|
+
|
|
129
|
+
Discovers test files from config.
|
|
130
|
+
|
|
131
|
+
## `loadConfig(cwd): ThemisConfig`
|
|
132
|
+
|
|
133
|
+
Loads `themis.config.json` and merges with defaults.
|
|
134
|
+
|
|
135
|
+
## `initConfig(cwd): void`
|
|
136
|
+
|
|
137
|
+
Creates `themis.config.json` if missing.
|
|
138
|
+
|
|
139
|
+
## `DEFAULT_CONFIG: ThemisConfig`
|
|
140
|
+
|
|
141
|
+
Default configuration object used by `loadConfig` and `initConfig`.
|
|
142
|
+
|
|
143
|
+
## `expect(received)`
|
|
144
|
+
|
|
145
|
+
Built-in matcher API:
|
|
146
|
+
|
|
147
|
+
- `toBe(expected)`
|
|
148
|
+
- `toEqual(expected)`
|
|
149
|
+
- `toMatchObject(expected)`
|
|
150
|
+
- `toBeTruthy()`
|
|
151
|
+
- `toBeFalsy()`
|
|
152
|
+
- `toBeDefined()`
|
|
153
|
+
- `toBeUndefined()`
|
|
154
|
+
- `toBeNull()`
|
|
155
|
+
- `toHaveLength(expected)`
|
|
156
|
+
- `toContain(item)`
|
|
157
|
+
- `toThrow(match?)`
|
|
158
|
+
- `toHaveBeenCalled()`
|
|
159
|
+
- `toHaveBeenCalledTimes(expected)`
|
|
160
|
+
- `toHaveBeenCalledWith(...args)`
|
|
161
|
+
- `toMatchSnapshot(name?)`
|
|
162
|
+
|
|
163
|
+
Machine-facing note:
|
|
164
|
+
|
|
165
|
+
- `--json` and `--agent` emit compact JSON by design. Tooling should parse the payload rather than rely on pretty-printed formatting.
|
|
166
|
+
|
|
167
|
+
## Global Test API
|
|
168
|
+
|
|
169
|
+
When test files run under Themis, these globals are available:
|
|
170
|
+
|
|
171
|
+
- `describe(name, fn)`
|
|
172
|
+
- `test(name, fn)` and `it(name, fn)`
|
|
173
|
+
- `beforeAll(fn)`, `beforeEach(fn)`, `afterEach(fn)`, `afterAll(fn)`
|
|
174
|
+
- `expect(...)`
|
|
175
|
+
- `fn(implementation?)`
|
|
176
|
+
- `spyOn(object, methodName)`
|
|
177
|
+
- `mock(moduleId, factoryOrExports?)`
|
|
178
|
+
- `unmock(moduleId)`
|
|
179
|
+
- `clearAllMocks()`
|
|
180
|
+
- `resetAllMocks()`
|
|
181
|
+
- `restoreAllMocks()`
|
|
182
|
+
- `intent(name, define)`
|
|
183
|
+
|
|
184
|
+
## Intent DSL
|
|
185
|
+
|
|
186
|
+
Preferred phases:
|
|
187
|
+
|
|
188
|
+
- `context`
|
|
189
|
+
- `run`
|
|
190
|
+
- `verify`
|
|
191
|
+
- `cleanup`
|
|
192
|
+
|
|
193
|
+
Compatibility aliases:
|
|
194
|
+
|
|
195
|
+
- `arrange`, `act`, `assert`
|
|
196
|
+
- `given`, `when`, `then`
|
|
197
|
+
- `setup`, `infer`, `teardown`, `finally`
|
|
198
|
+
|
|
199
|
+
Meme aliases (default enabled):
|
|
200
|
+
|
|
201
|
+
- `cook`, `yeet`, `vibecheck`, `wipe`
|
|
202
|
+
|
|
203
|
+
Disable meme aliases with `--no-memes` or programmatic `noMemes: true`.
|
|
204
|
+
|
|
205
|
+
Ordering guarantees:
|
|
206
|
+
|
|
207
|
+
- Arrange-phase aliases must run before Act/Assert.
|
|
208
|
+
- Act-phase aliases must run before Assert.
|
|
209
|
+
- At least one Assert-phase alias is required.
|
|
210
|
+
- Cleanup phases always run after run phases.
|
package/docs/publish.md
ADDED
|
@@ -0,0 +1,55 @@
|
|
|
1
|
+
# Publish Guide
|
|
2
|
+
|
|
3
|
+
This guide covers publishing Themis to npm.
|
|
4
|
+
|
|
5
|
+
## Preconditions
|
|
6
|
+
|
|
7
|
+
- npm account has publish access for scope `@vitronai`
|
|
8
|
+
- you are logged in (`npm login`)
|
|
9
|
+
- working tree is clean
|
|
10
|
+
- quality checks pass
|
|
11
|
+
|
|
12
|
+
## Release Checklist
|
|
13
|
+
|
|
14
|
+
1. Update version in `package.json` (`npm version ...`).
|
|
15
|
+
2. Update `CHANGELOG.md`.
|
|
16
|
+
3. Confirm the public messaging still matches the project:
|
|
17
|
+
- best-in-class unit test framework for AI agents in Node.js and TypeScript
|
|
18
|
+
- AI verdict engine for human and agent review loops
|
|
19
|
+
4. Run:
|
|
20
|
+
|
|
21
|
+
```bash
|
|
22
|
+
npm run validate
|
|
23
|
+
npm run pack:check
|
|
24
|
+
```
|
|
25
|
+
|
|
26
|
+
5. Review publish payload:
|
|
27
|
+
|
|
28
|
+
```bash
|
|
29
|
+
npm pack --dry-run
|
|
30
|
+
tar -tf vitronai-themis-*.tgz
|
|
31
|
+
```
|
|
32
|
+
|
|
33
|
+
6. Publish:
|
|
34
|
+
|
|
35
|
+
```bash
|
|
36
|
+
npm publish --access public
|
|
37
|
+
```
|
|
38
|
+
|
|
39
|
+
## Post Publish
|
|
40
|
+
|
|
41
|
+
Verify installation:
|
|
42
|
+
|
|
43
|
+
```bash
|
|
44
|
+
npx @vitronai/themis test --help
|
|
45
|
+
```
|
|
46
|
+
|
|
47
|
+
If needed, deprecate a release:
|
|
48
|
+
|
|
49
|
+
```bash
|
|
50
|
+
npm deprecate @vitronai/themis@<version> "<message>"
|
|
51
|
+
```
|
|
52
|
+
|
|
53
|
+
## VS Code Extension
|
|
54
|
+
|
|
55
|
+
The scaffold under `packages/themis-vscode` is a separate release surface. It does not ship in the npm tarball and can be published later on its own schedule.
|
|
@@ -0,0 +1,54 @@
|
|
|
1
|
+
# Release Policy
|
|
2
|
+
|
|
3
|
+
This document defines release expectations for Themis.
|
|
4
|
+
|
|
5
|
+
## Current Stage
|
|
6
|
+
|
|
7
|
+
Current line: `0.1.0-beta.x`
|
|
8
|
+
|
|
9
|
+
Beta means:
|
|
10
|
+
|
|
11
|
+
- the core CLI, artifact, and JS/TS package shape should be considered mostly stable
|
|
12
|
+
- compatibility remains a deliberate goal across beta builds
|
|
13
|
+
- breaking changes are still possible, but they should be uncommon and called out clearly
|
|
14
|
+
|
|
15
|
+
## Versioning Model
|
|
16
|
+
|
|
17
|
+
Themis follows semantic versioning semantics with pre-release tags:
|
|
18
|
+
|
|
19
|
+
- `0.1.0-alpha.N`: unstable early pre-release
|
|
20
|
+
- `0.1.0-beta.N`: pre-release with mostly stable product shape
|
|
21
|
+
- `0.1.0`: first stable contract baseline
|
|
22
|
+
- `0.1.x`: backward-compatible fixes and minor enhancements
|
|
23
|
+
- `0.2.0+`: intentional breaking changes with migration notes
|
|
24
|
+
|
|
25
|
+
## Compatibility Rules During Beta
|
|
26
|
+
|
|
27
|
+
- CLI flags should remain stable unless a clear product issue requires change.
|
|
28
|
+
- Programmatic API should prefer additive changes over renames or removals.
|
|
29
|
+
- Payload schemas keep explicit `schema` identifiers (`themis.agent.result.v1`, `themis.failures.v1`).
|
|
30
|
+
- If schema behavior changes in a breaking way, schema ID must bump.
|
|
31
|
+
|
|
32
|
+
## Deprecation Policy
|
|
33
|
+
|
|
34
|
+
- New deprecations are documented in `CHANGELOG.md`.
|
|
35
|
+
- Deprecated APIs should keep at least one pre-release cycle before removal when practical.
|
|
36
|
+
- Meme aliases can be disabled any time with `--no-memes`; strict environments should use this flag.
|
|
37
|
+
|
|
38
|
+
## Quality Gates
|
|
39
|
+
|
|
40
|
+
Before publishing:
|
|
41
|
+
|
|
42
|
+
- `npm test`
|
|
43
|
+
- `npm run typecheck`
|
|
44
|
+
- `npm run benchmark:gate`
|
|
45
|
+
|
|
46
|
+
## Publishing Policy
|
|
47
|
+
|
|
48
|
+
- Publish target: npm public package `@vitronai/themis`.
|
|
49
|
+
- Required metadata: license, repository, bugs URL, homepage.
|
|
50
|
+
- Recommended command:
|
|
51
|
+
|
|
52
|
+
```bash
|
|
53
|
+
npm publish --access public
|
|
54
|
+
```
|