@artemiskit/cli 0.2.2 → 0.2.4
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +85 -0
- package/dist/index.js +1800 -777
- package/dist/src/cli.d.ts.map +1 -1
- package/dist/src/commands/history.d.ts.map +1 -1
- package/dist/src/commands/redteam.d.ts.map +1 -1
- package/dist/src/commands/run.d.ts.map +1 -1
- package/dist/src/commands/stress.d.ts.map +1 -1
- package/dist/src/commands/validate.d.ts +6 -0
- package/dist/src/commands/validate.d.ts.map +1 -0
- package/package.json +6 -6
- package/src/cli.ts +2 -0
- package/src/commands/history.ts +58 -9
- package/src/commands/redteam.ts +28 -1
- package/src/commands/run.ts +121 -3
- package/src/commands/stress.ts +28 -0
- package/src/commands/validate.ts +254 -0
package/CHANGELOG.md
CHANGED
|
@@ -1,5 +1,90 @@
|
|
|
1
1
|
# @artemiskit/cli
|
|
2
2
|
|
|
3
|
+
## 0.2.4
|
|
4
|
+
|
|
5
|
+
### Patch Changes
|
|
6
|
+
|
|
7
|
+
- 16604a6: ## New Features
|
|
8
|
+
|
|
9
|
+
### Validate Command
|
|
10
|
+
|
|
11
|
+
New `artemiskit validate` command for validating scenario files without running them:
|
|
12
|
+
|
|
13
|
+
- **YAML syntax validation** - Catches formatting errors
|
|
14
|
+
- **Schema validation** - Validates against ArtemisKit schema using Zod
|
|
15
|
+
- **Semantic validation** - Detects duplicate case IDs, undefined variables
|
|
16
|
+
- **Warnings** - Identifies deprecated fields, missing descriptions, performance hints
|
|
17
|
+
|
|
18
|
+
Options:
|
|
19
|
+
|
|
20
|
+
- `--json` - Output results as JSON
|
|
21
|
+
- `--strict` - Treat warnings as errors
|
|
22
|
+
- `--quiet` - Only show errors
|
|
23
|
+
- `--export junit` - Export to JUnit XML for CI integration
|
|
24
|
+
|
|
25
|
+
### JUnit XML Export
|
|
26
|
+
|
|
27
|
+
Added JUnit XML export support for CI/CD integration with Jenkins, GitHub Actions, GitLab CI, and other systems:
|
|
28
|
+
|
|
29
|
+
- `akit run scenarios/ --export junit` - Export run results
|
|
30
|
+
- `akit redteam scenarios/chatbot.yaml --export junit` - Export security test results
|
|
31
|
+
- `akit validate scenarios/ --export junit` - Export validation results
|
|
32
|
+
|
|
33
|
+
JUnit reports include:
|
|
34
|
+
|
|
35
|
+
- Test suite metadata (run ID, provider, model, success rate)
|
|
36
|
+
- Individual test cases with pass/fail status
|
|
37
|
+
- Failure details with matcher type and expected values
|
|
38
|
+
- Timing information for each test
|
|
39
|
+
|
|
40
|
+
- Updated dependencies [16604a6]
|
|
41
|
+
- @artemiskit/core@0.2.4
|
|
42
|
+
- @artemiskit/reports@0.2.4
|
|
43
|
+
- @artemiskit/adapter-openai@0.1.11
|
|
44
|
+
- @artemiskit/adapter-vercel-ai@0.1.11
|
|
45
|
+
- @artemiskit/redteam@0.2.4
|
|
46
|
+
|
|
47
|
+
## 0.2.3
|
|
48
|
+
|
|
49
|
+
### Patch Changes
|
|
50
|
+
|
|
51
|
+
- 37403aa: ## v0.2.3 - Cost Tracking & Compliance Features
|
|
52
|
+
|
|
53
|
+
### Cost Tracking
|
|
54
|
+
|
|
55
|
+
- **Automatic cost estimation**: Run results now include estimated API costs based on token usage and model pricing data
|
|
56
|
+
- **Cost display in output**: Summary output shows total tokens and estimated cost for each run
|
|
57
|
+
- **`--budget` flag**: Set a maximum budget in USD for `run`, `redteam`, and `stress` commands - the command fails (exit code 1) if the estimated cost exceeds the budget
|
|
58
|
+
|
|
59
|
+
### History Enhancements
|
|
60
|
+
|
|
61
|
+
- **`--show-cost` flag**: Display cost column and total in `history` command output
|
|
62
|
+
- Cost data is stored with each run for historical tracking
|
|
63
|
+
|
|
64
|
+
### Markdown Export
|
|
65
|
+
|
|
66
|
+
- **`--export markdown` flag**: Export run and redteam results to compliance-ready markdown format
|
|
67
|
+
- **`--export-output` flag**: Specify custom output directory for exports (default: `./artemis-exports`)
|
|
68
|
+
- Markdown reports include:
|
|
69
|
+
- Summary table with pass/fail rates, latency, token usage, and cost metrics
|
|
70
|
+
- Detailed results for failed test cases (run) or vulnerabilities found (redteam)
|
|
71
|
+
- Configuration used for the run
|
|
72
|
+
- Redaction summary (if enabled)
|
|
73
|
+
- Recommendations for remediation (redteam)
|
|
74
|
+
|
|
75
|
+
### CI/CD Integration
|
|
76
|
+
|
|
77
|
+
- Budget enforcement in pipelines: `akit run scenarios/ --ci --budget 5.00`
|
|
78
|
+
- Cost tracking in CI summary output with `ARTEMISKIT_COST_USD` variable
|
|
79
|
+
- Automatic markdown report generation for compliance documentation
|
|
80
|
+
|
|
81
|
+
- Updated dependencies [37403aa]
|
|
82
|
+
- @artemiskit/core@0.2.3
|
|
83
|
+
- @artemiskit/reports@0.2.3
|
|
84
|
+
- @artemiskit/adapter-openai@0.1.10
|
|
85
|
+
- @artemiskit/adapter-vercel-ai@0.1.10
|
|
86
|
+
- @artemiskit/redteam@0.2.3
|
|
87
|
+
|
|
3
88
|
## 0.2.2
|
|
4
89
|
|
|
5
90
|
### Patch Changes
|