@artemiskit/core 0.1.2 → 0.1.3
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/CHANGELOG.md +6 -0
- package/README.md +117 -0
- package/package.json +1 -1
package/CHANGELOG.md
CHANGED
package/README.md
ADDED
|
@@ -0,0 +1,117 @@
|
|
|
1
|
+
# @artemiskit/core
|
|
2
|
+
|
|
3
|
+
Core runtime, evaluators, and storage for the ArtemisKit LLM evaluation toolkit.
|
|
4
|
+
|
|
5
|
+
## Installation
|
|
6
|
+
|
|
7
|
+
```bash
|
|
8
|
+
npm install @artemiskit/core
|
|
9
|
+
# or
|
|
10
|
+
bun add @artemiskit/core
|
|
11
|
+
```
|
|
12
|
+
|
|
13
|
+
## Overview
|
|
14
|
+
|
|
15
|
+
This package provides the foundational components for ArtemisKit:
|
|
16
|
+
|
|
17
|
+
- **Runner** - Execute test scenarios against LLM providers
|
|
18
|
+
- **Evaluators** - Built-in expectation matchers (contains, exact, regex, fuzzy, llm_grader, json_schema)
|
|
19
|
+
- **Storage** - Local filesystem and Supabase storage backends
|
|
20
|
+
- **Redaction** - PII/sensitive data filtering with built-in patterns
|
|
21
|
+
- **Schema** - YAML scenario parsing and validation
|
|
22
|
+
|
|
23
|
+
## Usage
|
|
24
|
+
|
|
25
|
+
Most users should use the [`@artemiskit/cli`](https://www.npmjs.com/package/@artemiskit/cli) package for running evaluations. This package is primarily for programmatic usage or building custom integrations.
|
|
26
|
+
|
|
27
|
+
```typescript
|
|
28
|
+
import {
|
|
29
|
+
parseScenarioFile,
|
|
30
|
+
runScenario,
|
|
31
|
+
createAdapter
|
|
32
|
+
} from '@artemiskit/core';
|
|
33
|
+
|
|
34
|
+
// Parse a scenario file
|
|
35
|
+
const scenario = await parseScenarioFile('./my-test.yaml');
|
|
36
|
+
|
|
37
|
+
// Create an adapter (requires adapter package to be registered)
|
|
38
|
+
const client = await createAdapter({
|
|
39
|
+
provider: 'openai',
|
|
40
|
+
apiKey: process.env.OPENAI_API_KEY,
|
|
41
|
+
});
|
|
42
|
+
|
|
43
|
+
// Run the scenario
|
|
44
|
+
const result = await runScenario({
|
|
45
|
+
scenario,
|
|
46
|
+
client,
|
|
47
|
+
project: 'my-project',
|
|
48
|
+
onCaseComplete: (caseResult, index, total) => {
|
|
49
|
+
console.log(`Completed ${index + 1}/${total}: ${caseResult.passed ? 'PASS' : 'FAIL'}`);
|
|
50
|
+
},
|
|
51
|
+
});
|
|
52
|
+
|
|
53
|
+
console.log(`Success: ${result.success}`);
|
|
54
|
+
console.log(`Passed: ${result.manifest.metrics.passed_cases}/${result.manifest.metrics.total_cases}`);
|
|
55
|
+
```
|
|
56
|
+
|
|
57
|
+
**Note:** The `createAdapter` function requires adapter packages (like `@artemiskit/adapter-openai`) to be installed and registered. For standalone usage, import adapters directly:
|
|
58
|
+
|
|
59
|
+
```typescript
|
|
60
|
+
import { OpenAIAdapter } from '@artemiskit/adapter-openai';
|
|
61
|
+
|
|
62
|
+
const client = new OpenAIAdapter({
|
|
63
|
+
provider: 'openai',
|
|
64
|
+
apiKey: process.env.OPENAI_API_KEY,
|
|
65
|
+
});
|
|
66
|
+
```
|
|
67
|
+
|
|
68
|
+
## Evaluators
|
|
69
|
+
|
|
70
|
+
Built-in expectation types:
|
|
71
|
+
|
|
72
|
+
| Type | Description |
|
|
73
|
+
|------|-------------|
|
|
74
|
+
| `contains` | Check if response contains text |
|
|
75
|
+
| `exact` | Exact string match |
|
|
76
|
+
| `regex` | Regular expression match |
|
|
77
|
+
| `fuzzy` | Fuzzy string similarity (Levenshtein) |
|
|
78
|
+
| `llm_grader` | LLM-based response grading |
|
|
79
|
+
| `json_schema` | Validate JSON structure |
|
|
80
|
+
|
|
81
|
+
## Redaction
|
|
82
|
+
|
|
83
|
+
Built-in patterns for sensitive data:
|
|
84
|
+
|
|
85
|
+
- `email` - Email addresses
|
|
86
|
+
- `phone` - Phone numbers
|
|
87
|
+
- `ssn` - Social Security Numbers
|
|
88
|
+
- `credit_card` - Credit card numbers
|
|
89
|
+
- `api_key` - API keys (sk_, pk_, etc.)
|
|
90
|
+
- `jwt` - JWT tokens
|
|
91
|
+
- `aws_key` - AWS access keys
|
|
92
|
+
- `ipv4` - IPv4 addresses
|
|
93
|
+
- `secrets` - Password/secret assignments
|
|
94
|
+
|
|
95
|
+
```typescript
|
|
96
|
+
import { Redactor, BUILTIN_PATTERNS } from '@artemiskit/core';
|
|
97
|
+
|
|
98
|
+
const redactor = new Redactor({
|
|
99
|
+
enabled: true,
|
|
100
|
+
patterns: [BUILTIN_PATTERNS.EMAIL, BUILTIN_PATTERNS.PHONE],
|
|
101
|
+
});
|
|
102
|
+
|
|
103
|
+
const result = redactor.redact('Contact: user@example.com');
|
|
104
|
+
// result.text = 'Contact: [REDACTED]'
|
|
105
|
+
```
|
|
106
|
+
|
|
107
|
+
## Related Packages
|
|
108
|
+
|
|
109
|
+
- [`@artemiskit/cli`](https://www.npmjs.com/package/@artemiskit/cli) - Command-line interface
|
|
110
|
+
- [`@artemiskit/adapter-openai`](https://www.npmjs.com/package/@artemiskit/adapter-openai) - OpenAI/Azure adapter
|
|
111
|
+
- [`@artemiskit/adapter-anthropic`](https://www.npmjs.com/package/@artemiskit/adapter-anthropic) - Anthropic Claude adapter
|
|
112
|
+
- [`@artemiskit/redteam`](https://www.npmjs.com/package/@artemiskit/redteam) - Security testing
|
|
113
|
+
- [`@artemiskit/reports`](https://www.npmjs.com/package/@artemiskit/reports) - HTML report generation
|
|
114
|
+
|
|
115
|
+
## License
|
|
116
|
+
|
|
117
|
+
Apache-2.0
|