testflow-ai 0.1.0 โ 0.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +352 -132
- package/dist/ai.d.ts +20 -6
- package/dist/ai.d.ts.map +1 -1
- package/dist/ai.js +102 -10
- package/dist/ai.js.map +1 -1
- package/dist/cli.js +11 -4
- package/dist/cli.js.map +1 -1
- package/dist/parser.d.ts.map +1 -1
- package/dist/parser.js +7 -2
- package/dist/parser.js.map +1 -1
- package/dist/types.d.ts +15 -7
- package/dist/types.d.ts.map +1 -1
- package/package.json +4 -4
package/README.md
CHANGED
|
@@ -1,65 +1,65 @@
|
|
|
1
|
-
|
|
1
|
+
<div align="center">
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
# ๐งช testflow-ai
|
|
4
4
|
|
|
5
|
-
|
|
6
|
-
[](https://opensource.org/licenses/MIT)
|
|
7
|
-
[](https://nodejs.org)
|
|
5
|
+
**Declarative API testing powered by YAML flows**
|
|
8
6
|
|
|
9
|
-
|
|
7
|
+
*Version-controlled โข Human-readable โข AI-friendly*
|
|
10
8
|
|
|
11
|
-
|
|
9
|
+
[](https://www.npmjs.com/package/testflow-ai)
|
|
10
|
+
[](https://www.npmjs.com/package/testflow-ai)
|
|
11
|
+
[](https://opensource.org/licenses/MIT)
|
|
12
|
+
[](https://nodejs.org)
|
|
12
13
|
|
|
13
|
-
|
|
14
|
+
[๐ Documentation](#-documentation) โข [๐ Quick Start](#-quick-start) โข [๐ป Examples](#-real-world-example) โข [๐ค AI Providers](#-ai-powered-evaluation)
|
|
14
15
|
|
|
15
|
-
|
|
16
|
+
</div>
|
|
16
17
|
|
|
17
|
-
|
|
18
|
-
- **IDE AI assistants** worked for one-off requests but burned through tokens, lost context, and couldn't maintain complex multi-step flows.
|
|
19
|
-
- **MCP servers and tooling** required significant setup and ongoing maintenance.
|
|
18
|
+
---
|
|
20
19
|
|
|
21
|
-
|
|
20
|
+
## ๐ฏ What is testflow-ai?
|
|
22
21
|
|
|
23
|
-
|
|
24
|
-
2. Defines multi-step flows declaratively
|
|
25
|
-
3. Captures variables between steps automatically
|
|
26
|
-
4. Supports REST, GraphQL, and async operations
|
|
27
|
-
5. Can leverage a **local AI model** for intelligent assertions
|
|
28
|
-
6. Runs in CI/CD with zero cloud dependencies
|
|
22
|
+
**testflow-ai** lets you define API tests in YAML files, run them from the command line or as a library, and use AI models (local or cloud) for intelligent assertions. No GUI, no vendor lock-in, works with any HTTP/GraphQL API.
|
|
29
23
|
|
|
30
|
-
|
|
24
|
+
> **๐ก Born from real-world frustration:** After months of testing APIs with Postman and burning tokens with ChatGPT, I built this to centralize tests in version-controlled YAML files with local AI support.
|
|
31
25
|
|
|
32
|
-
|
|
26
|
+
### โจ Key Features
|
|
33
27
|
|
|
34
|
-
|
|
28
|
+
<div align="center">
|
|
35
29
|
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
|
|
40
|
-
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
|
|
44
|
-
|
|
45
|
-
|
|
30
|
+
| ๐จ Feature | ๐ Description |
|
|
31
|
+
|:----------:|:-------------:|
|
|
32
|
+
| **๐ YAML Flows** | Define test sequences declaratively โ version-controlled and human-readable |
|
|
33
|
+
| **๐ Variable Capture** | Extract values from responses, reuse in later steps automatically |
|
|
34
|
+
| **โ
Rich Assertions** | 10+ operators: equals, contains, exists, greaterThan, matches, and more |
|
|
35
|
+
| **๐ GraphQL Native** | First-class support for queries and mutations |
|
|
36
|
+
| **โณ Async Polling** | `waitUntil` for operations that take time (background jobs, processing) |
|
|
37
|
+
| **๐ค AI Evaluation** | Assert with natural language using Ollama, OpenAI, or Anthropic |
|
|
38
|
+
| **๐ Context Files** | Define base URLs, endpoints, and rules in Markdown |
|
|
39
|
+
| **๐ Multiple Formats** | Console (colored), JSON (CI/CD), or Markdown reports |
|
|
40
|
+
| **๐ฏ Tag Filtering** | Run subsets of your test suite (`--tags smoke,e2e`) |
|
|
41
|
+
| **๐ฅ๏ธ CLI + API** | Use from terminal (`npx testflow`) or import as a library |
|
|
42
|
+
|
|
43
|
+
</div>
|
|
46
44
|
|
|
47
45
|
---
|
|
48
46
|
|
|
49
|
-
## Quick Start
|
|
47
|
+
## ๐ Quick Start
|
|
50
48
|
|
|
51
|
-
### 1
|
|
49
|
+
### 1๏ธโฃ Install
|
|
52
50
|
|
|
53
51
|
```bash
|
|
54
52
|
npm install testflow-ai
|
|
55
53
|
```
|
|
56
54
|
|
|
57
|
-
### 2
|
|
55
|
+
### 2๏ธโฃ Create a test flow
|
|
56
|
+
|
|
57
|
+
Create `tests/health.yaml`:
|
|
58
58
|
|
|
59
59
|
```yaml
|
|
60
|
-
# tests/health.yaml
|
|
61
60
|
name: Health Check
|
|
62
61
|
tags: [smoke]
|
|
62
|
+
|
|
63
63
|
steps:
|
|
64
64
|
- name: Ping API
|
|
65
65
|
request:
|
|
@@ -71,17 +71,39 @@ steps:
|
|
|
71
71
|
value: 200
|
|
72
72
|
```
|
|
73
73
|
|
|
74
|
-
### 3
|
|
74
|
+
### 3๏ธโฃ Run
|
|
75
75
|
|
|
76
76
|
```bash
|
|
77
77
|
npx testflow tests/health.yaml
|
|
78
78
|
```
|
|
79
79
|
|
|
80
|
-
That's it
|
|
80
|
+
**That's it.** No config files, no GUI, no account.
|
|
81
|
+
|
|
82
|
+
---
|
|
83
|
+
|
|
84
|
+
## ๐ก Why testflow-ai?
|
|
85
|
+
|
|
86
|
+
I was building a backend that started as a simple API and grew into a system with GraphQL, async workers, state machines, and AI-powered evaluations.
|
|
87
|
+
|
|
88
|
+
Testing started simple โ a few requests in Postman. Then the project scaled:
|
|
89
|
+
|
|
90
|
+
<div align="center">
|
|
91
|
+
|
|
92
|
+
| โ Problem | โ
Solution |
|
|
93
|
+
|:----------:|:-----------:|
|
|
94
|
+
| **Postman / Insomnia** became unmanageable | YAML files in version control |
|
|
95
|
+
| **IDE AI assistants** burned tokens, lost context | Local AI via Ollama (free, private) |
|
|
96
|
+
| **MCP servers** required complex setup | Zero dependencies beyond Node.js |
|
|
97
|
+
| **Manual token copying** between requests | Automatic variable capture |
|
|
98
|
+
| **No CI/CD integration** | JSON output, exit codes, GitHub Actions ready |
|
|
99
|
+
|
|
100
|
+
</div>
|
|
101
|
+
|
|
102
|
+
**testflow-ai** solves all of this.
|
|
81
103
|
|
|
82
104
|
---
|
|
83
105
|
|
|
84
|
-
## Installation
|
|
106
|
+
## ๐ฆ Installation
|
|
85
107
|
|
|
86
108
|
```bash
|
|
87
109
|
npm install testflow-ai
|
|
@@ -93,37 +115,40 @@ yarn add testflow-ai
|
|
|
93
115
|
|
|
94
116
|
---
|
|
95
117
|
|
|
96
|
-
## CLI Usage
|
|
118
|
+
## ๐ฅ๏ธ CLI Usage
|
|
97
119
|
|
|
98
120
|
```bash
|
|
99
121
|
# Run specific files
|
|
100
|
-
testflow flow1.yaml flow2.yaml
|
|
122
|
+
npx testflow flow1.yaml flow2.yaml
|
|
101
123
|
|
|
102
124
|
# Run all YAML files in a directory
|
|
103
|
-
testflow --dir ./tests
|
|
125
|
+
npx testflow --dir ./tests
|
|
104
126
|
|
|
105
127
|
# Use a context file for base URLs
|
|
106
|
-
testflow --dir ./tests --context ./context.md
|
|
128
|
+
npx testflow --dir ./tests --context ./context.md
|
|
107
129
|
|
|
108
|
-
# Filter by tags
|
|
109
|
-
testflow --dir ./tests --tags smoke
|
|
130
|
+
# Filter by tags (run only smoke tests)
|
|
131
|
+
npx testflow --dir ./tests --tags smoke
|
|
110
132
|
|
|
111
133
|
# JSON output (for CI/CD)
|
|
112
|
-
testflow --dir ./tests --format json
|
|
134
|
+
npx testflow --dir ./tests --format json
|
|
113
135
|
|
|
114
136
|
# Markdown output (for reports)
|
|
115
|
-
testflow --dir ./tests --format markdown
|
|
137
|
+
npx testflow --dir ./tests --format markdown
|
|
116
138
|
|
|
117
|
-
# Verbose mode
|
|
118
|
-
testflow --dir ./tests -v
|
|
139
|
+
# Verbose mode (see step-by-step execution)
|
|
140
|
+
npx testflow --dir ./tests -v
|
|
119
141
|
|
|
120
|
-
# With AI evaluation
|
|
121
|
-
testflow --dir ./tests --ai-model llama3.2:3b
|
|
142
|
+
# With AI evaluation
|
|
143
|
+
npx testflow --dir ./tests --ai-provider ollama --ai-model llama3.2:3b
|
|
144
|
+
npx testflow --dir ./tests --ai-provider openai --ai-key $OPENAI_API_KEY --ai-model gpt-4o-mini
|
|
122
145
|
```
|
|
123
146
|
|
|
124
147
|
---
|
|
125
148
|
|
|
126
|
-
## Programmatic API
|
|
149
|
+
## ๐ป Programmatic API
|
|
150
|
+
|
|
151
|
+
### Simple usage
|
|
127
152
|
|
|
128
153
|
```typescript
|
|
129
154
|
import { runTests } from 'testflow-ai';
|
|
@@ -140,6 +165,97 @@ console.log(`${report.passedFlows}/${report.totalFlows} passed`);
|
|
|
140
165
|
process.exit(report.failedFlows > 0 ? 1 : 0);
|
|
141
166
|
```
|
|
142
167
|
|
|
168
|
+
---
|
|
169
|
+
|
|
170
|
+
## ๐ป Real-World Example
|
|
171
|
+
|
|
172
|
+
Here's how we use it in production at [educational-rewards](https://github.com/carbajalmarcos/educational-rewards):
|
|
173
|
+
|
|
174
|
+
### Project Structure
|
|
175
|
+
|
|
176
|
+
```
|
|
177
|
+
tests/declarative/
|
|
178
|
+
โโโ index.ts # Test runner
|
|
179
|
+
โโโ context.md # API context
|
|
180
|
+
โโโ flows/
|
|
181
|
+
โโโ health-check.yaml
|
|
182
|
+
โโโ complete-reward-flow.yaml
|
|
183
|
+
โโโ submission-attempts.yaml
|
|
184
|
+
```
|
|
185
|
+
|
|
186
|
+
### Test Runner (`index.ts`)
|
|
187
|
+
|
|
188
|
+
```typescript
|
|
189
|
+
import { runTests, type RunnerOptions } from 'testflow-ai';
|
|
190
|
+
import * as path from 'path';
|
|
191
|
+
|
|
192
|
+
async function main() {
|
|
193
|
+
const options: RunnerOptions = {
|
|
194
|
+
contextFile: path.join(__dirname, 'context.md'),
|
|
195
|
+
testDir: path.join(__dirname, 'flows'),
|
|
196
|
+
tags: process.argv.includes('--tags=smoke') ? ['smoke'] : undefined,
|
|
197
|
+
format: 'console',
|
|
198
|
+
verbose: false,
|
|
199
|
+
};
|
|
200
|
+
|
|
201
|
+
const report = await runTests(options);
|
|
202
|
+
process.exit(report.failedFlows > 0 ? 1 : 0);
|
|
203
|
+
}
|
|
204
|
+
|
|
205
|
+
main();
|
|
206
|
+
```
|
|
207
|
+
|
|
208
|
+
### Context File (`context.md`)
|
|
209
|
+
|
|
210
|
+
```markdown
|
|
211
|
+
# Mambita API Context
|
|
212
|
+
|
|
213
|
+
## Base URLs
|
|
214
|
+
- graphql: http://localhost:3000/graphql
|
|
215
|
+
- tasks: http://localhost:8000
|
|
216
|
+
|
|
217
|
+
## Endpoints
|
|
218
|
+
- POST /graphql - GraphQL endpoint
|
|
219
|
+
- POST /api/v1/pool/seed - Seed task pool
|
|
220
|
+
```
|
|
221
|
+
|
|
222
|
+
### Test Flow (`flows/complete-reward-flow.yaml`)
|
|
223
|
+
|
|
224
|
+
```yaml
|
|
225
|
+
name: Complete Reward Flow (E2E)
|
|
226
|
+
tags: [e2e, reward]
|
|
227
|
+
|
|
228
|
+
steps:
|
|
229
|
+
- name: Start reward
|
|
230
|
+
request:
|
|
231
|
+
method: POST
|
|
232
|
+
url: "{graphql}"
|
|
233
|
+
graphql:
|
|
234
|
+
query: |
|
|
235
|
+
mutation StartReward($input: StartRewardInput!) {
|
|
236
|
+
startReward(input: $input) {
|
|
237
|
+
id
|
|
238
|
+
state
|
|
239
|
+
taskInstances { id state }
|
|
240
|
+
}
|
|
241
|
+
}
|
|
242
|
+
variables:
|
|
243
|
+
input:
|
|
244
|
+
childId: "${childId}"
|
|
245
|
+
catalogRewardId: "${catalogItemId}"
|
|
246
|
+
capture:
|
|
247
|
+
- name: rewardId
|
|
248
|
+
path: data.startReward.id
|
|
249
|
+
```
|
|
250
|
+
|
|
251
|
+
### Running Tests
|
|
252
|
+
|
|
253
|
+
```bash
|
|
254
|
+
pnpm testflow:run # All tests
|
|
255
|
+
pnpm testflow:smoke # Smoke tests only
|
|
256
|
+
pnpm testflow submission-attempts # Specific flow
|
|
257
|
+
```
|
|
258
|
+
|
|
143
259
|
### Advanced usage
|
|
144
260
|
|
|
145
261
|
```typescript
|
|
@@ -149,7 +265,7 @@ import { TestRunner, FlowExecutor, parseYamlFile, parseContextFile } from 'testf
|
|
|
149
265
|
const runner = new TestRunner({
|
|
150
266
|
contextFile: './context.md',
|
|
151
267
|
testFiles: ['./tests/critical.yaml'],
|
|
152
|
-
ai: { model: 'mistral:7b' },
|
|
268
|
+
ai: { provider: 'ollama', model: 'mistral:7b' },
|
|
153
269
|
});
|
|
154
270
|
const report = await runner.run();
|
|
155
271
|
|
|
@@ -162,56 +278,53 @@ const result = await executor.executeFlow(flow);
|
|
|
162
278
|
|
|
163
279
|
---
|
|
164
280
|
|
|
165
|
-
## Test Flow Reference
|
|
281
|
+
## ๐ Test Flow Reference
|
|
166
282
|
|
|
167
283
|
### Basic structure
|
|
168
284
|
|
|
169
285
|
```yaml
|
|
170
|
-
name: Flow
|
|
171
|
-
description:
|
|
286
|
+
name: User Registration Flow
|
|
287
|
+
description: Create and verify a new user
|
|
172
288
|
tags:
|
|
289
|
+
- users
|
|
173
290
|
- smoke
|
|
174
|
-
- e2e
|
|
175
291
|
|
|
176
292
|
steps:
|
|
177
|
-
- name:
|
|
293
|
+
- name: Create user
|
|
178
294
|
request:
|
|
179
295
|
method: POST
|
|
180
|
-
url: "{api}/
|
|
296
|
+
url: "{api}/users"
|
|
181
297
|
headers:
|
|
182
298
|
Content-Type: application/json
|
|
183
299
|
body:
|
|
184
|
-
|
|
300
|
+
email: alice@example.com
|
|
301
|
+
name: Alice
|
|
185
302
|
capture:
|
|
186
|
-
- name:
|
|
187
|
-
path: data.
|
|
303
|
+
- name: userId
|
|
304
|
+
path: data.id
|
|
188
305
|
assertions:
|
|
189
306
|
- path: status
|
|
190
307
|
operator: equals
|
|
191
308
|
value: 201
|
|
192
|
-
|
|
193
|
-
|
|
194
|
-
|
|
309
|
+
- path: data.email
|
|
310
|
+
operator: equals
|
|
311
|
+
value: alice@example.com
|
|
195
312
|
|
|
196
|
-
|
|
197
|
-
steps:
|
|
198
|
-
- name: Create resource
|
|
313
|
+
- name: Verify user
|
|
199
314
|
request:
|
|
200
|
-
method:
|
|
201
|
-
url: "{api}/
|
|
202
|
-
|
|
203
|
-
|
|
204
|
-
|
|
205
|
-
|
|
206
|
-
title: New Resource
|
|
207
|
-
active: true
|
|
315
|
+
method: GET
|
|
316
|
+
url: "{api}/users/${userId}"
|
|
317
|
+
assertions:
|
|
318
|
+
- path: data.id
|
|
319
|
+
operator: equals
|
|
320
|
+
value: "${userId}"
|
|
208
321
|
```
|
|
209
322
|
|
|
210
323
|
### GraphQL requests
|
|
211
324
|
|
|
212
325
|
```yaml
|
|
213
326
|
steps:
|
|
214
|
-
- name: Query
|
|
327
|
+
- name: Query user
|
|
215
328
|
request:
|
|
216
329
|
method: POST
|
|
217
330
|
url: "{graphql}"
|
|
@@ -227,7 +340,7 @@ steps:
|
|
|
227
340
|
variables:
|
|
228
341
|
id: "${userId}"
|
|
229
342
|
capture:
|
|
230
|
-
- name:
|
|
343
|
+
- name: userEmail
|
|
231
344
|
path: data.user.email
|
|
232
345
|
```
|
|
233
346
|
|
|
@@ -258,18 +371,19 @@ steps:
|
|
|
258
371
|
Authorization: "Bearer ${token}"
|
|
259
372
|
```
|
|
260
373
|
|
|
261
|
-
Supported
|
|
374
|
+
**Supported patterns:**
|
|
375
|
+
|
|
262
376
|
- `${variable}` โ simple variable
|
|
263
377
|
- `${data.nested.field}` โ nested path
|
|
264
378
|
- `${items[0].id}` โ array access
|
|
265
379
|
|
|
266
|
-
###
|
|
380
|
+
### Async polling (waitUntil)
|
|
267
381
|
|
|
268
|
-
For
|
|
382
|
+
For operations that take time โ polls until condition is met or timeout:
|
|
269
383
|
|
|
270
384
|
```yaml
|
|
271
385
|
steps:
|
|
272
|
-
- name: Wait for
|
|
386
|
+
- name: Wait for job completion
|
|
273
387
|
request:
|
|
274
388
|
method: GET
|
|
275
389
|
url: "{api}/jobs/${jobId}"
|
|
@@ -287,22 +401,26 @@ steps:
|
|
|
287
401
|
|
|
288
402
|
---
|
|
289
403
|
|
|
290
|
-
## Assertions
|
|
404
|
+
## โ
Assertions
|
|
291
405
|
|
|
292
|
-
|
|
293
|
-
|
|
294
|
-
|
|
|
295
|
-
|
|
296
|
-
| `
|
|
297
|
-
| `
|
|
406
|
+
<div align="center">
|
|
407
|
+
|
|
408
|
+
| Operator | Description | Example |
|
|
409
|
+
|:--------:|:-----------:|:-------:|
|
|
410
|
+
| `equals` | Exact match (deep equality) | `value: 200` |
|
|
411
|
+
| `notEquals` | Not equal | `value: null` |
|
|
412
|
+
| `contains` | String/array contains | `value: "success"` |
|
|
413
|
+
| `notContains` | Does not contain | `value: "error"` |
|
|
298
414
|
| `exists` | Not null/undefined | โ |
|
|
299
415
|
| `notExists` | Is null/undefined | โ |
|
|
300
|
-
| `greaterThan` | Number comparison | `0` |
|
|
301
|
-
| `lessThan` | Number comparison | `100` |
|
|
302
|
-
| `matches` | Regex match | `"^[a-z]+$"` |
|
|
303
|
-
| `ai-evaluate` | AI-powered evaluation | `"Is this
|
|
416
|
+
| `greaterThan` | Number comparison | `value: 0` |
|
|
417
|
+
| `lessThan` | Number comparison | `value: 100` |
|
|
418
|
+
| `matches` | Regex match | `value: "^[a-z]+$"` |
|
|
419
|
+
| `ai-evaluate` | AI-powered evaluation | `value: "Is this valid?"` |
|
|
420
|
+
|
|
421
|
+
</div>
|
|
304
422
|
|
|
305
|
-
|
|
423
|
+
**Special paths:**
|
|
306
424
|
|
|
307
425
|
- `status` โ HTTP status code (when value is a number)
|
|
308
426
|
- `httpStatus` โ always the HTTP status code
|
|
@@ -311,11 +429,23 @@ steps:
|
|
|
311
429
|
|
|
312
430
|
---
|
|
313
431
|
|
|
314
|
-
## AI-Powered Evaluation
|
|
432
|
+
## ๐ค AI-Powered Evaluation
|
|
433
|
+
|
|
434
|
+
Use AI to assert things that are hard to express with traditional operators. **testflow-ai** supports multiple providers:
|
|
435
|
+
|
|
436
|
+
<div align="center">
|
|
437
|
+
|
|
438
|
+
| Provider | Type | Setup | Best For |
|
|
439
|
+
|:--------:|:----:|:-----:|:--------:|
|
|
440
|
+
| **๐ฆ Ollama** | Local | Free, no API key | Privacy, offline, cost-effective |
|
|
441
|
+
| **๐ค OpenAI** | Cloud | API key required | High accuracy, GPT-4 |
|
|
442
|
+
| **๐ง Anthropic** | Cloud | API key required | Claude models, safety-focused |
|
|
315
443
|
|
|
316
|
-
|
|
444
|
+
</div>
|
|
317
445
|
|
|
318
|
-
###
|
|
446
|
+
### ๐ฆ Ollama (Local, Recommended)
|
|
447
|
+
|
|
448
|
+
**No cloud API keys, no data leaves your machine.**
|
|
319
449
|
|
|
320
450
|
1. **Install Ollama** โ [ollama.com/download](https://ollama.com/download)
|
|
321
451
|
|
|
@@ -332,17 +462,82 @@ ollama pull llama3.2:1b
|
|
|
332
462
|
ollama pull mistral:7b
|
|
333
463
|
```
|
|
334
464
|
|
|
335
|
-
|
|
465
|
+
1. **Start Ollama** (runs on `http://localhost:11434` by default):
|
|
336
466
|
|
|
337
467
|
```bash
|
|
338
468
|
ollama serve
|
|
339
469
|
```
|
|
340
470
|
|
|
471
|
+
**Usage:**
|
|
472
|
+
|
|
473
|
+
```bash
|
|
474
|
+
# CLI
|
|
475
|
+
npx testflow --dir ./tests --ai-provider ollama --ai-model llama3.2:3b
|
|
476
|
+
|
|
477
|
+
# Programmatic
|
|
478
|
+
const report = await runTests({
|
|
479
|
+
testDir: './tests',
|
|
480
|
+
ai: {
|
|
481
|
+
provider: 'ollama',
|
|
482
|
+
url: 'http://localhost:11434',
|
|
483
|
+
model: 'llama3.2:3b',
|
|
484
|
+
},
|
|
485
|
+
});
|
|
486
|
+
```
|
|
487
|
+
|
|
488
|
+
### ๐ค OpenAI (Cloud)
|
|
489
|
+
|
|
490
|
+
**Requires API key from [platform.openai.com](https://platform.openai.com/api-keys)**
|
|
491
|
+
|
|
492
|
+
```bash
|
|
493
|
+
# CLI
|
|
494
|
+
npx testflow --dir ./tests \
|
|
495
|
+
--ai-provider openai \
|
|
496
|
+
--ai-key $OPENAI_API_KEY \
|
|
497
|
+
--ai-model gpt-4o-mini
|
|
498
|
+
|
|
499
|
+
# Programmatic
|
|
500
|
+
const report = await runTests({
|
|
501
|
+
testDir: './tests',
|
|
502
|
+
ai: {
|
|
503
|
+
provider: 'openai',
|
|
504
|
+
apiKey: process.env.OPENAI_API_KEY,
|
|
505
|
+
model: 'gpt-4o-mini',
|
|
506
|
+
},
|
|
507
|
+
});
|
|
508
|
+
```
|
|
509
|
+
|
|
510
|
+
**Supported models:** `gpt-4o`, `gpt-4o-mini`, `gpt-4-turbo`, `gpt-3.5-turbo`
|
|
511
|
+
|
|
512
|
+
### ๐ง Anthropic (Cloud)
|
|
513
|
+
|
|
514
|
+
**Requires API key from [console.anthropic.com](https://console.anthropic.com/)**
|
|
515
|
+
|
|
516
|
+
```bash
|
|
517
|
+
# CLI
|
|
518
|
+
npx testflow --dir ./tests \
|
|
519
|
+
--ai-provider anthropic \
|
|
520
|
+
--ai-key $ANTHROPIC_API_KEY \
|
|
521
|
+
--ai-model claude-3-haiku-20240307
|
|
522
|
+
|
|
523
|
+
# Programmatic
|
|
524
|
+
const report = await runTests({
|
|
525
|
+
testDir: './tests',
|
|
526
|
+
ai: {
|
|
527
|
+
provider: 'anthropic',
|
|
528
|
+
apiKey: process.env.ANTHROPIC_API_KEY,
|
|
529
|
+
model: 'claude-3-haiku-20240307',
|
|
530
|
+
},
|
|
531
|
+
});
|
|
532
|
+
```
|
|
533
|
+
|
|
534
|
+
**Supported models:** `claude-3-5-sonnet-20241022`, `claude-3-opus-20240229`, `claude-3-haiku-20240307`
|
|
535
|
+
|
|
341
536
|
### Using AI assertions
|
|
342
537
|
|
|
343
538
|
```yaml
|
|
344
539
|
steps:
|
|
345
|
-
- name: Check
|
|
540
|
+
- name: Check article quality
|
|
346
541
|
request:
|
|
347
542
|
method: GET
|
|
348
543
|
url: "{api}/articles/1"
|
|
@@ -351,45 +546,31 @@ steps:
|
|
|
351
546
|
- path: status
|
|
352
547
|
operator: equals
|
|
353
548
|
value: 200
|
|
354
|
-
# AI-powered assertion
|
|
549
|
+
# AI-powered assertion (works with any provider)
|
|
355
550
|
- path: data.content
|
|
356
551
|
operator: ai-evaluate
|
|
357
552
|
value: "Does this article contain a coherent explanation with at least two paragraphs?"
|
|
358
553
|
```
|
|
359
554
|
|
|
360
|
-
### CLI with AI
|
|
361
|
-
|
|
362
|
-
```bash
|
|
363
|
-
testflow --dir ./tests --ai-model llama3.2:3b
|
|
364
|
-
testflow --dir ./tests --ai-url http://192.168.1.10:11434 --ai-model mistral:7b
|
|
365
|
-
```
|
|
366
|
-
|
|
367
|
-
### Programmatic with AI
|
|
368
|
-
|
|
369
|
-
```typescript
|
|
370
|
-
const report = await runTests({
|
|
371
|
-
testDir: './tests',
|
|
372
|
-
ai: {
|
|
373
|
-
url: 'http://localhost:11434',
|
|
374
|
-
model: 'llama3.2:3b',
|
|
375
|
-
timeout: 30000,
|
|
376
|
-
},
|
|
377
|
-
});
|
|
378
|
-
```
|
|
379
|
-
|
|
380
555
|
### Context file AI config
|
|
381
556
|
|
|
382
557
|
```markdown
|
|
383
558
|
## AI Configuration
|
|
559
|
+
- provider: ollama
|
|
384
560
|
- url: http://localhost:11434
|
|
385
561
|
- model: llama3.2:3b
|
|
562
|
+
|
|
563
|
+
# Or for cloud providers:
|
|
564
|
+
# provider: openai
|
|
565
|
+
# apiKey: ${OPENAI_API_KEY}
|
|
566
|
+
# model: gpt-4o-mini
|
|
386
567
|
```
|
|
387
568
|
|
|
388
|
-
>
|
|
569
|
+
> **๐ Privacy note:** Ollama runs entirely locally. OpenAI and Anthropic send data to their APIs. Choose based on your privacy requirements.
|
|
389
570
|
|
|
390
571
|
---
|
|
391
572
|
|
|
392
|
-
## Context Files
|
|
573
|
+
## ๐ Context Files
|
|
393
574
|
|
|
394
575
|
Define your project context in Markdown. The runner uses it to resolve `{baseUrlKey}` references in your YAML flows.
|
|
395
576
|
|
|
@@ -413,13 +594,14 @@ Brief description of your API.
|
|
|
413
594
|
- Authentication required for /users
|
|
414
595
|
|
|
415
596
|
## AI Configuration
|
|
597
|
+
- provider: ollama
|
|
416
598
|
- url: http://localhost:11434
|
|
417
599
|
- model: llama3.2:3b
|
|
418
600
|
```
|
|
419
601
|
|
|
420
602
|
---
|
|
421
603
|
|
|
422
|
-
## CI/CD Integration
|
|
604
|
+
## ๐ CI/CD Integration
|
|
423
605
|
|
|
424
606
|
### GitHub Actions
|
|
425
607
|
|
|
@@ -441,16 +623,16 @@ jobs:
|
|
|
441
623
|
path: results.json
|
|
442
624
|
```
|
|
443
625
|
|
|
444
|
-
|
|
626
|
+
**Exit codes:**
|
|
445
627
|
|
|
446
628
|
- `0` โ all flows passed
|
|
447
629
|
- `1` โ one or more flows failed
|
|
448
630
|
|
|
449
631
|
---
|
|
450
632
|
|
|
451
|
-
## Output Examples
|
|
633
|
+
## ๐ Output Examples
|
|
452
634
|
|
|
453
|
-
### Console
|
|
635
|
+
### Console Output
|
|
454
636
|
|
|
455
637
|
```
|
|
456
638
|
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
|
|
@@ -458,10 +640,10 @@ jobs:
|
|
|
458
640
|
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ
|
|
459
641
|
|
|
460
642
|
Summary:
|
|
461
|
-
Total:
|
|
462
|
-
Passed:
|
|
463
|
-
Failed:
|
|
464
|
-
Duration:
|
|
643
|
+
Total: 5 flows
|
|
644
|
+
Passed: 5
|
|
645
|
+
Failed: 0
|
|
646
|
+
Duration: 2450ms
|
|
465
647
|
|
|
466
648
|
Narrative:
|
|
467
649
|
|
|
@@ -485,7 +667,7 @@ Narrative:
|
|
|
485
667
|
|
|
486
668
|
---
|
|
487
669
|
|
|
488
|
-
## Roadmap
|
|
670
|
+
## ๐บ๏ธ Roadmap
|
|
489
671
|
|
|
490
672
|
- [ ] Database assertions (verify records directly via SQL)
|
|
491
673
|
- [ ] gRPC / RPC support
|
|
@@ -497,6 +679,44 @@ Narrative:
|
|
|
497
679
|
|
|
498
680
|
---
|
|
499
681
|
|
|
500
|
-
##
|
|
682
|
+
## ๐ Examples
|
|
683
|
+
|
|
684
|
+
See the [`examples/`](./examples) directory for:
|
|
685
|
+
|
|
686
|
+
- REST CRUD flows
|
|
687
|
+
- GraphQL queries and mutations
|
|
688
|
+
- Authentication flows
|
|
689
|
+
- Context file templates
|
|
690
|
+
|
|
691
|
+
---
|
|
692
|
+
|
|
693
|
+
## ๐ License
|
|
501
694
|
|
|
502
695
|
MIT
|
|
696
|
+
|
|
697
|
+
---
|
|
698
|
+
|
|
699
|
+
<div align="center">
|
|
700
|
+
|
|
701
|
+
**Made with โค๏ธ by [Marcos Carbajal](https://github.com/carbajalmarcos)**
|
|
702
|
+
|
|
703
|
+
[โญ Star on GitHub](https://github.com/carbajalmarcos/testflow-ai) โข [๐ฆ npm](https://www.npmjs.com/package/testflow-ai) โข [๐ Report Bug](https://github.com/carbajalmarcos/testflow-ai/issues) โข [๐ฌ Discussions](https://github.com/carbajalmarcos/testflow-ai/discussions)
|
|
704
|
+
|
|
705
|
+
---
|
|
706
|
+
|
|
707
|
+
### โ Support this project
|
|
708
|
+
|
|
709
|
+
If you find **testflow-ai** useful, consider supporting its development:
|
|
710
|
+
|
|
711
|
+
[](https://buymeacoffee.com/carbajalmarcos)
|
|
712
|
+
[](https://github.com/sponsors/carbajalmarcos)
|
|
713
|
+
[](https://ko-fi.com/carbajalmarcos)
|
|
714
|
+
|
|
715
|
+
**Crypto donations:**
|
|
716
|
+
|
|
717
|
+
- **Bitcoin (BTC):** `bc1qv0ddjg3wcgujk9ad66v9msz8manu5tanhvq0fn`
|
|
718
|
+
- **ERC-20 USDT:** `0x79F57C9D45d2D40420EF071DDAaA27057618E7C8`
|
|
719
|
+
|
|
720
|
+
*Every contribution helps make this project better!*
|
|
721
|
+
|
|
722
|
+
</div>
|
package/dist/ai.d.ts
CHANGED
|
@@ -1,19 +1,33 @@
|
|
|
1
1
|
/**
|
|
2
|
-
* AI evaluation
|
|
2
|
+
* AI evaluation with support for multiple providers.
|
|
3
3
|
*
|
|
4
|
-
*
|
|
5
|
-
*
|
|
4
|
+
* Supports:
|
|
5
|
+
* - Ollama (local, default)
|
|
6
|
+
* - OpenAI (cloud)
|
|
7
|
+
* - Anthropic (cloud)
|
|
8
|
+
* - Custom providers via adapter pattern
|
|
6
9
|
*/
|
|
7
10
|
import type { AiConfig, AiEvaluation } from './types.js';
|
|
8
11
|
/** Merge partial user config with defaults. */
|
|
9
12
|
export declare function resolveAiConfig(partial?: Partial<AiConfig>): AiConfig;
|
|
10
13
|
/**
|
|
11
|
-
* Evaluate a value against a natural-language prompt using
|
|
14
|
+
* Evaluate a value against a natural-language prompt using AI.
|
|
12
15
|
*
|
|
13
16
|
* @example
|
|
14
17
|
* ```ts
|
|
15
|
-
*
|
|
16
|
-
*
|
|
18
|
+
* // Ollama (local)
|
|
19
|
+
* const result = await evaluateWithAi(
|
|
20
|
+
* { provider: 'ollama', model: 'llama3.2:3b' },
|
|
21
|
+
* responseBody,
|
|
22
|
+
* 'Does this contain a valid user object?'
|
|
23
|
+
* );
|
|
24
|
+
*
|
|
25
|
+
* // OpenAI (cloud)
|
|
26
|
+
* const result = await evaluateWithAi(
|
|
27
|
+
* { provider: 'openai', apiKey: process.env.OPENAI_API_KEY, model: 'gpt-4o-mini' },
|
|
28
|
+
* responseBody,
|
|
29
|
+
* 'Is this response well-formatted?'
|
|
30
|
+
* );
|
|
17
31
|
* ```
|
|
18
32
|
*/
|
|
19
33
|
export declare function evaluateWithAi(config: AiConfig, actual: unknown, prompt: string): Promise<AiEvaluation>;
|
package/dist/ai.d.ts.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"ai.d.ts","sourceRoot":"","sources":["../src/ai.ts"],"names":[],"mappings":"AAAA
|
|
1
|
+
{"version":3,"file":"ai.d.ts","sourceRoot":"","sources":["../src/ai.ts"],"names":[],"mappings":"AAAA;;;;;;;;GAQG;AAGH,OAAO,KAAK,EAAE,QAAQ,EAAE,YAAY,EAAE,MAAM,YAAY,CAAC;AASzD,+CAA+C;AAC/C,wBAAgB,eAAe,CAAC,OAAO,CAAC,EAAE,OAAO,CAAC,QAAQ,CAAC,GAAG,QAAQ,CAErE;AAED;;;;;;;;;;;;;;;;;;;GAmBG;AACH,wBAAsB,cAAc,CAChC,MAAM,EAAE,QAAQ,EAChB,MAAM,EAAE,OAAO,EACf,MAAM,EAAE,MAAM,GACf,OAAO,CAAC,YAAY,CAAC,CAkBvB"}
|
package/dist/ai.js
CHANGED
|
@@ -1,9 +1,12 @@
|
|
|
1
1
|
"use strict";
|
|
2
2
|
/**
|
|
3
|
-
* AI evaluation
|
|
3
|
+
* AI evaluation with support for multiple providers.
|
|
4
4
|
*
|
|
5
|
-
*
|
|
6
|
-
*
|
|
5
|
+
* Supports:
|
|
6
|
+
* - Ollama (local, default)
|
|
7
|
+
* - OpenAI (cloud)
|
|
8
|
+
* - Anthropic (cloud)
|
|
9
|
+
* - Custom providers via adapter pattern
|
|
7
10
|
*/
|
|
8
11
|
var __importDefault = (this && this.__importDefault) || function (mod) {
|
|
9
12
|
return (mod && mod.__esModule) ? mod : { "default": mod };
|
|
@@ -13,6 +16,7 @@ exports.resolveAiConfig = resolveAiConfig;
|
|
|
13
16
|
exports.evaluateWithAi = evaluateWithAi;
|
|
14
17
|
const axios_1 = __importDefault(require("axios"));
|
|
15
18
|
const DEFAULT_AI_CONFIG = {
|
|
19
|
+
provider: 'ollama',
|
|
16
20
|
url: 'http://localhost:11434',
|
|
17
21
|
model: 'llama3.2:3b',
|
|
18
22
|
timeout: 30_000,
|
|
@@ -22,12 +26,23 @@ function resolveAiConfig(partial) {
|
|
|
22
26
|
return { ...DEFAULT_AI_CONFIG, ...partial };
|
|
23
27
|
}
|
|
24
28
|
/**
|
|
25
|
-
* Evaluate a value against a natural-language prompt using
|
|
29
|
+
* Evaluate a value against a natural-language prompt using AI.
|
|
26
30
|
*
|
|
27
31
|
* @example
|
|
28
32
|
* ```ts
|
|
29
|
-
*
|
|
30
|
-
*
|
|
33
|
+
* // Ollama (local)
|
|
34
|
+
* const result = await evaluateWithAi(
|
|
35
|
+
* { provider: 'ollama', model: 'llama3.2:3b' },
|
|
36
|
+
* responseBody,
|
|
37
|
+
* 'Does this contain a valid user object?'
|
|
38
|
+
* );
|
|
39
|
+
*
|
|
40
|
+
* // OpenAI (cloud)
|
|
41
|
+
* const result = await evaluateWithAi(
|
|
42
|
+
* { provider: 'openai', apiKey: process.env.OPENAI_API_KEY, model: 'gpt-4o-mini' },
|
|
43
|
+
* responseBody,
|
|
44
|
+
* 'Is this response well-formatted?'
|
|
45
|
+
* );
|
|
31
46
|
* ```
|
|
32
47
|
*/
|
|
33
48
|
async function evaluateWithAi(config, actual, prompt) {
|
|
@@ -36,14 +51,27 @@ async function evaluateWithAi(config, actual, prompt) {
|
|
|
36
51
|
'Respond ONLY with valid JSON: {"pass": true/false, "confidence": 0.0-1.0, "reason": "brief explanation"}',
|
|
37
52
|
].join(' ');
|
|
38
53
|
const userPrompt = `Criteria: ${prompt}\n\nData:\n${JSON.stringify(actual, null, 2)}`;
|
|
54
|
+
switch (config.provider) {
|
|
55
|
+
case 'ollama':
|
|
56
|
+
return evaluateWithOllama(config, systemPrompt, userPrompt);
|
|
57
|
+
case 'openai':
|
|
58
|
+
return evaluateWithOpenAI(config, systemPrompt, userPrompt);
|
|
59
|
+
case 'anthropic':
|
|
60
|
+
return evaluateWithAnthropic(config, systemPrompt, userPrompt);
|
|
61
|
+
default:
|
|
62
|
+
return { pass: false, confidence: 0, reason: `Unknown AI provider: ${config.provider}` };
|
|
63
|
+
}
|
|
64
|
+
}
|
|
65
|
+
/** Ollama (local) evaluation. */
|
|
66
|
+
async function evaluateWithOllama(config, systemPrompt, userPrompt) {
|
|
39
67
|
try {
|
|
40
|
-
const { data } = await axios_1.default.post(`${config.url}/api/generate`, {
|
|
41
|
-
model: config.model,
|
|
68
|
+
const { data } = await axios_1.default.post(`${config.url || 'http://localhost:11434'}/api/generate`, {
|
|
69
|
+
model: config.model || 'llama3.2:3b',
|
|
42
70
|
prompt: userPrompt,
|
|
43
71
|
system: systemPrompt,
|
|
44
72
|
stream: false,
|
|
45
73
|
format: 'json',
|
|
46
|
-
}, { timeout: config.timeout });
|
|
74
|
+
}, { timeout: config.timeout || 30_000 });
|
|
47
75
|
const parsed = JSON.parse(data.response);
|
|
48
76
|
return {
|
|
49
77
|
pass: Boolean(parsed.pass),
|
|
@@ -53,7 +81,71 @@ async function evaluateWithAi(config, actual, prompt) {
|
|
|
53
81
|
}
|
|
54
82
|
catch (err) {
|
|
55
83
|
const message = err instanceof Error ? err.message : String(err);
|
|
56
|
-
return { pass: false, confidence: 0, reason: `
|
|
84
|
+
return { pass: false, confidence: 0, reason: `Ollama evaluation failed: ${message}` };
|
|
85
|
+
}
|
|
86
|
+
}
|
|
87
|
+
/** OpenAI (cloud) evaluation. */
|
|
88
|
+
async function evaluateWithOpenAI(config, systemPrompt, userPrompt) {
|
|
89
|
+
if (!config.apiKey) {
|
|
90
|
+
return { pass: false, confidence: 0, reason: 'OpenAI API key is required' };
|
|
91
|
+
}
|
|
92
|
+
try {
|
|
93
|
+
const { data } = await axios_1.default.post('https://api.openai.com/v1/chat/completions', {
|
|
94
|
+
model: config.model || 'gpt-4o-mini',
|
|
95
|
+
messages: [
|
|
96
|
+
{ role: 'system', content: systemPrompt },
|
|
97
|
+
{ role: 'user', content: userPrompt },
|
|
98
|
+
],
|
|
99
|
+
response_format: { type: 'json_object' },
|
|
100
|
+
temperature: 0.1,
|
|
101
|
+
}, {
|
|
102
|
+
headers: {
|
|
103
|
+
'Authorization': `Bearer ${config.apiKey}`,
|
|
104
|
+
'Content-Type': 'application/json',
|
|
105
|
+
},
|
|
106
|
+
timeout: config.timeout || 30_000,
|
|
107
|
+
});
|
|
108
|
+
const parsed = JSON.parse(data.choices[0].message.content);
|
|
109
|
+
return {
|
|
110
|
+
pass: Boolean(parsed.pass),
|
|
111
|
+
confidence: Number(parsed.confidence) || 0,
|
|
112
|
+
reason: String(parsed.reason || ''),
|
|
113
|
+
};
|
|
114
|
+
}
|
|
115
|
+
catch (err) {
|
|
116
|
+
const message = err instanceof Error ? err.message : String(err);
|
|
117
|
+
return { pass: false, confidence: 0, reason: `OpenAI evaluation failed: ${message}` };
|
|
118
|
+
}
|
|
119
|
+
}
|
|
120
|
+
/** Anthropic (cloud) evaluation. */
|
|
121
|
+
async function evaluateWithAnthropic(config, systemPrompt, userPrompt) {
|
|
122
|
+
if (!config.apiKey) {
|
|
123
|
+
return { pass: false, confidence: 0, reason: 'Anthropic API key is required' };
|
|
124
|
+
}
|
|
125
|
+
try {
|
|
126
|
+
const { data } = await axios_1.default.post('https://api.anthropic.com/v1/messages', {
|
|
127
|
+
model: config.model || 'claude-3-haiku-20240307',
|
|
128
|
+
max_tokens: 1024,
|
|
129
|
+
system: systemPrompt,
|
|
130
|
+
messages: [{ role: 'user', content: userPrompt }],
|
|
131
|
+
}, {
|
|
132
|
+
headers: {
|
|
133
|
+
'x-api-key': config.apiKey,
|
|
134
|
+
'anthropic-version': '2023-06-01',
|
|
135
|
+
'Content-Type': 'application/json',
|
|
136
|
+
},
|
|
137
|
+
timeout: config.timeout || 30_000,
|
|
138
|
+
});
|
|
139
|
+
const parsed = JSON.parse(data.content[0].text);
|
|
140
|
+
return {
|
|
141
|
+
pass: Boolean(parsed.pass),
|
|
142
|
+
confidence: Number(parsed.confidence) || 0,
|
|
143
|
+
reason: String(parsed.reason || ''),
|
|
144
|
+
};
|
|
145
|
+
}
|
|
146
|
+
catch (err) {
|
|
147
|
+
const message = err instanceof Error ? err.message : String(err);
|
|
148
|
+
return { pass: false, confidence: 0, reason: `Anthropic evaluation failed: ${message}` };
|
|
57
149
|
}
|
|
58
150
|
}
|
|
59
151
|
//# sourceMappingURL=ai.js.map
|
package/dist/ai.js.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"ai.js","sourceRoot":"","sources":["../src/ai.ts"],"names":[],"mappings":";AAAA;;;;;
|
|
1
|
+
{"version":3,"file":"ai.js","sourceRoot":"","sources":["../src/ai.ts"],"names":[],"mappings":";AAAA;;;;;;;;GAQG;;;;;AAaH,0CAEC;AAsBD,wCAsBC;AAzDD,kDAA0B;AAG1B,MAAM,iBAAiB,GAAa;IAChC,QAAQ,EAAE,QAAQ;IAClB,GAAG,EAAE,wBAAwB;IAC7B,KAAK,EAAE,aAAa;IACpB,OAAO,EAAE,MAAM;CAClB,CAAC;AAEF,+CAA+C;AAC/C,SAAgB,eAAe,CAAC,OAA2B;IACvD,OAAO,EAAE,GAAG,iBAAiB,EAAE,GAAG,OAAO,EAAE,CAAC;AAChD,CAAC;AAED;;;;;;;;;;;;;;;;;;;GAmBG;AACI,KAAK,UAAU,cAAc,CAChC,MAAgB,EAChB,MAAe,EACf,MAAc;IAEd,MAAM,YAAY,GAAG;QACjB,iFAAiF;QACjF,0GAA0G;KAC7G,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC;IAEZ,MAAM,UAAU,GAAG,aAAa,MAAM,cAAc,IAAI,CAAC,SAAS,CAAC,MAAM,EAAE,IAAI,EAAE,CAAC,CAAC,EAAE,CAAC;IAEtF,QAAQ,MAAM,CAAC,QAAQ,EAAE,CAAC;QACtB,KAAK,QAAQ;YACT,OAAO,kBAAkB,CAAC,MAAM,EAAE,YAAY,EAAE,UAAU,CAAC,CAAC;QAChE,KAAK,QAAQ;YACT,OAAO,kBAAkB,CAAC,MAAM,EAAE,YAAY,EAAE,UAAU,CAAC,CAAC;QAChE,KAAK,WAAW;YACZ,OAAO,qBAAqB,CAAC,MAAM,EAAE,YAAY,EAAE,UAAU,CAAC,CAAC;QACnE;YACI,OAAO,EAAE,IAAI,EAAE,KAAK,EAAE,UAAU,EAAE,CAAC,EAAE,MAAM,EAAE,wBAAwB,MAAM,CAAC,QAAQ,EAAE,EAAE,CAAC;IACjG,CAAC;AACL,CAAC;AAED,iCAAiC;AACjC,KAAK,UAAU,kBAAkB,CAC7B,MAAgB,EAChB,YAAoB,EACpB,UAAkB;IAElB,IAAI,CAAC;QACD,MAAM,EAAE,IAAI,EAAE,GAAG,MAAM,eAAK,CAAC,IAAI,CAC7B,GAAG,MAAM,CAAC,GAAG,IAAI,wBAAwB,eAAe,EACxD;YACI,KAAK,EAAE,MAAM,CAAC,KAAK,IAAI,aAAa;YACpC,MAAM,EAAE,UAAU;YAClB,MAAM,EAAE,YAAY;YACpB,MAAM,EAAE,KAAK;YACb,MAAM,EAAE,MAAM;SACjB,EACD,EAAE,OAAO,EAAE,MAAM,CAAC,OAAO,IAAI,MAAM,EAAE,CACxC,CAAC;QAEF,MAAM,MAAM,GAAG,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,QAAQ,CAAC,CAAC;QACzC,OAAO;YACH,IAAI,EAAE,OAAO,CAAC,MAAM,CAAC,IAAI,CAAC;YAC1B,UAAU,EAAE,MAAM,CAAC,MAAM,CAAC,UAAU,CAAC,IAAI,CAAC;YAC1C,MAAM,EAAE,MAAM,CAAC,MAAM,CAAC,MAAM,IAAI,EAAE,CAAC;SACtC,CAAC;IACN,CAAC;IAAC,OAAO,GAAG,EAAE,CAAC;QACX,MAAM,OAAO,GAAG,GAAG,YAAY,KAAK,CAAC,CAAC,CAAC,GAAG,CAAC,OAAO,CAAC,CAAC,CAAC,MAAM,CAAC,GAAG,CAAC,CAAC;QACjE,OAAO,EAAE,IAAI,EAAE,KAAK,EAAE,UAAU,EAAE,CAAC,EAAE,MAAM,EAAE,6BAA6B,OAAO,EAAE,EAAE,CAAC;IAC1F,CAAC;AACL,CAAC;AAED,iCAAiC;AACjC,KAAK,UAAU,kBAAkB,CAC7B,MAAgB,EAChB,YAAoB,EACpB,UAAkB;IAElB,IAAI,CAAC,MAAM,CAAC,MAAM,EAAE,CAAC;QACjB,OAAO,EAAE,IAAI,EAAE,KAAK,EAAE,UAAU,EAAE,CAAC,EAAE,MAAM,EAAE,4BAA4B,EAAE,CAAC;IAChF,CAAC;IAED,IAAI,CAAC;QACD,MAAM,EAAE,IAAI,EAAE,GAAG,MAAM,eAAK,CAAC,IAAI,CAC7B,4CAA4C,EAC5C;YACI,KAAK,EAAE,MAAM,CAAC,KAAK,IAAI,aAAa;YACpC,QAAQ,EAAE;gBACN,EAAE,IAAI,EAAE,QAAQ,EAAE,OAAO,EAAE,YAAY,EAAE;gBACzC,EAAE,IAAI,EAAE,MAAM,EAAE,OAAO,EAAE,UAAU,EAAE;aACxC;YACD,eAAe,EAAE,EAAE,IAAI,EAAE,aAAa,EAAE;YACxC,WAAW,EAAE,GAAG;SACnB,EACD;YACI,OAAO,EAAE;gBACL,eAAe,EAAE,UAAU,MAAM,CAAC,MAAM,EAAE;gBAC1C,cAAc,EAAE,kBAAkB;aACrC;YACD,OAAO,EAAE,MAAM,CAAC,OAAO,IAAI,MAAM;SACpC,CACJ,CAAC;QAEF,MAAM,MAAM,GAAG,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,OAAO,CAAC,CAAC,CAAC,CAAC,OAAO,CAAC,OAAO,CAAC,CAAC;QAC3D,OAAO;YACH,IAAI,EAAE,OAAO,CAAC,MAAM,CAAC,IAAI,CAAC;YAC1B,UAAU,EAAE,MAAM,CAAC,MAAM,CAAC,UAAU,CAAC,IAAI,CAAC;YAC1C,MAAM,EAAE,MAAM,CAAC,MAAM,CAAC,MAAM,IAAI,EAAE,CAAC;SACtC,CAAC;IACN,CAAC;IAAC,OAAO,GAAG,EAAE,CAAC;QACX,MAAM,OAAO,GAAG,GAAG,YAAY,KAAK,CAAC,CAAC,CAAC,GAAG,CAAC,OAAO,CAAC,CAAC,CAAC,MAAM,CAAC,GAAG,CAAC,CAAC;QACjE,OAAO,EAAE,IAAI,EAAE,KAAK,EAAE,UAAU,EAAE,CAAC,EAAE,MAAM,EAAE,6BAA6B,OAAO,EAAE,EAAE,CAAC;IAC1F,CAAC;AACL,CAAC;AAED,oCAAoC;AACpC,KAAK,UAAU,qBAAqB,CAChC,MAAgB,EAChB,YAAoB,EACpB,UAAkB;IAElB,IAAI,CAAC,MAAM,CAAC,MAAM,EAAE,CAAC;QACjB,OAAO,EAAE,IAAI,EAAE,KAAK,EAAE,UAAU,EAAE,CAAC,EAAE,MAAM,EAAE,+BAA+B,EAAE,CAAC;IACnF,CAAC;IAED,IAAI,CAAC;QACD,MAAM,EAAE,IAAI,EAAE,GAAG,MAAM,eAAK,CAAC,IAAI,CAC7B,uCAAuC,EACvC;YACI,KAAK,EAAE,MAAM,CAAC,KAAK,IAAI,yBAAyB;YAChD,UAAU,EAAE,IAAI;YAChB,MAAM,EAAE,YAAY;YACpB,QAAQ,EAAE,CAAC,EAAE,IAAI,EAAE,MAAM,EAAE,OAAO,EAAE,UAAU,EAAE,CAAC;SACpD,EACD;YACI,OAAO,EAAE;gBACL,WAAW,EAAE,MAAM,CAAC,MAAM;gBAC1B,mBAAmB,EAAE,YAAY;gBACjC,cAAc,EAAE,kBAAkB;aACrC;YACD,OAAO,EAAE,MAAM,CAAC,OAAO,IAAI,MAAM;SACpC,CACJ,CAAC;QAEF,MAAM,MAAM,GAAG,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,OAAO,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC;QAChD,OAAO;YACH,IAAI,EAAE,OAAO,CAAC,MAAM,CAAC,IAAI,CAAC;YAC1B,UAAU,EAAE,MAAM,CAAC,MAAM,CAAC,UAAU,CAAC,IAAI,CAAC;YAC1C,MAAM,EAAE,MAAM,CAAC,MAAM,CAAC,MAAM,IAAI,EAAE,CAAC;SACtC,CAAC;IACN,CAAC;IAAC,OAAO,GAAG,EAAE,CAAC;QACX,MAAM,OAAO,GAAG,GAAG,YAAY,KAAK,CAAC,CAAC,CAAC,GAAG,CAAC,OAAO,CAAC,CAAC,CAAC,MAAM,CAAC,GAAG,CAAC,CAAC;QACjE,OAAO,EAAE,IAAI,EAAE,KAAK,EAAE,UAAU,EAAE,CAAC,EAAE,MAAM,EAAE,gCAAgC,OAAO,EAAE,EAAE,CAAC;IAC7F,CAAC;AACL,CAAC"}
|
package/dist/cli.js
CHANGED
|
@@ -23,8 +23,10 @@ program
|
|
|
23
23
|
.option('-t, --tags <tags>', 'Comma-separated tags to filter')
|
|
24
24
|
.option('-f, --format <format>', 'Output: console, json, markdown', 'console')
|
|
25
25
|
.option('-v, --verbose', 'Verbose output')
|
|
26
|
-
.option('--ai-
|
|
27
|
-
.option('--ai-
|
|
26
|
+
.option('--ai-provider <provider>', 'AI provider: ollama, openai, anthropic (default: ollama)')
|
|
27
|
+
.option('--ai-url <url>', 'AI API URL (for Ollama, default: http://localhost:11434)')
|
|
28
|
+
.option('--ai-model <model>', 'AI model name (varies by provider)')
|
|
29
|
+
.option('--ai-key <key>', 'API key (required for OpenAI/Anthropic)')
|
|
28
30
|
.action(async (files, opts) => {
|
|
29
31
|
try {
|
|
30
32
|
const report = await (0, runner_js_1.runTests)({
|
|
@@ -34,8 +36,13 @@ program
|
|
|
34
36
|
tags: opts.tags?.split(','),
|
|
35
37
|
format: opts.format,
|
|
36
38
|
verbose: opts.verbose,
|
|
37
|
-
ai: opts.aiUrl || opts.aiModel
|
|
38
|
-
? {
|
|
39
|
+
ai: opts.aiProvider || opts.aiUrl || opts.aiModel || opts.aiKey
|
|
40
|
+
? {
|
|
41
|
+
provider: opts.aiProvider || 'ollama',
|
|
42
|
+
url: opts.aiUrl,
|
|
43
|
+
model: opts.aiModel,
|
|
44
|
+
apiKey: opts.aiKey,
|
|
45
|
+
}
|
|
39
46
|
: undefined,
|
|
40
47
|
});
|
|
41
48
|
process.exit(report.failedFlows > 0 ? 1 : 0);
|
package/dist/cli.js.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"cli.js","sourceRoot":"","sources":["../src/cli.ts"],"names":[],"mappings":";;AAEA;;;;;;;GAOG;;AAEH,yCAAoC;AACpC,2CAAuC;AAEvC,MAAM,OAAO,GAAG,IAAI,mBAAO,EAAE,CAAC;AAE9B,OAAO;KACF,IAAI,CAAC,UAAU,CAAC;KAChB,WAAW,CAAC,+CAA+C,CAAC;KAC5D,OAAO,CAAC,OAAO,CAAC,CAAC;AAEtB,OAAO;KACF,QAAQ,CAAC,YAAY,EAAE,wBAAwB,CAAC;KAChD,MAAM,CAAC,sBAAsB,EAAE,+BAA+B,CAAC;KAC/D,MAAM,CAAC,uBAAuB,EAAE,sCAAsC,CAAC;KACvE,MAAM,CAAC,mBAAmB,EAAE,gCAAgC,CAAC;KAC7D,MAAM,CAAC,uBAAuB,EAAE,iCAAiC,EAAE,SAAS,CAAC;KAC7E,MAAM,CAAC,eAAe,EAAE,gBAAgB,CAAC;KACzC,MAAM,CAAC,gBAAgB,EAAE,
|
|
1
|
+
{"version":3,"file":"cli.js","sourceRoot":"","sources":["../src/cli.ts"],"names":[],"mappings":";;AAEA;;;;;;;GAOG;;AAEH,yCAAoC;AACpC,2CAAuC;AAEvC,MAAM,OAAO,GAAG,IAAI,mBAAO,EAAE,CAAC;AAE9B,OAAO;KACF,IAAI,CAAC,UAAU,CAAC;KAChB,WAAW,CAAC,+CAA+C,CAAC;KAC5D,OAAO,CAAC,OAAO,CAAC,CAAC;AAEtB,OAAO;KACF,QAAQ,CAAC,YAAY,EAAE,wBAAwB,CAAC;KAChD,MAAM,CAAC,sBAAsB,EAAE,+BAA+B,CAAC;KAC/D,MAAM,CAAC,uBAAuB,EAAE,sCAAsC,CAAC;KACvE,MAAM,CAAC,mBAAmB,EAAE,gCAAgC,CAAC;KAC7D,MAAM,CAAC,uBAAuB,EAAE,iCAAiC,EAAE,SAAS,CAAC;KAC7E,MAAM,CAAC,eAAe,EAAE,gBAAgB,CAAC;KACzC,MAAM,CAAC,0BAA0B,EAAE,0DAA0D,CAAC;KAC9F,MAAM,CAAC,gBAAgB,EAAE,0DAA0D,CAAC;KACpF,MAAM,CAAC,oBAAoB,EAAE,oCAAoC,CAAC;KAClE,MAAM,CAAC,gBAAgB,EAAE,yCAAyC,CAAC;KACnE,MAAM,CAAC,KAAK,EAAE,KAAe,EAAE,IAAI,EAAE,EAAE;IACpC,IAAI,CAAC;QACD,MAAM,MAAM,GAAG,MAAM,IAAA,oBAAQ,EAAC;YAC1B,SAAS,EAAE,KAAK,CAAC,MAAM,GAAG,CAAC,CAAC,CAAC,CAAC,KAAK,CAAC,CAAC,CAAC,SAAS;YAC/C,WAAW,EAAE,IAAI,CAAC,OAAO;YACzB,OAAO,EAAE,IAAI,CAAC,GAAG;YACjB,IAAI,EAAE,IAAI,CAAC,IAAI,EAAE,KAAK,CAAC,GAAG,CAAC;YAC3B,MAAM,EAAE,IAAI,CAAC,MAAM;YACnB,OAAO,EAAE,IAAI,CAAC,OAAO;YACrB,EAAE,EAAE,IAAI,CAAC,UAAU,IAAI,IAAI,CAAC,KAAK,IAAI,IAAI,CAAC,OAAO,IAAI,IAAI,CAAC,KAAK;gBAC3D,CAAC,CAAC;oBACE,QAAQ,EAAE,IAAI,CAAC,UAAU,IAAI,QAAQ;oBACrC,GAAG,EAAE,IAAI,CAAC,KAAK;oBACf,KAAK,EAAE,IAAI,CAAC,OAAO;oBACnB,MAAM,EAAE,IAAI,CAAC,KAAK;iBACrB;gBACD,CAAC,CAAC,SAAS;SAClB,CAAC,CAAC;QAEH,OAAO,CAAC,IAAI,CAAC,MAAM,CAAC,WAAW,GAAG,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC;IACjD,CAAC;IAAC,OAAO,GAAG,EAAE,CAAC;QACX,OAAO,CAAC,KAAK,CAAC,OAAO,GAAG,YAAY,KAAK,CAAC,CAAC,CAAC,GAAG,CAAC,OAAO,CAAC,CAAC,CAAC,GAAG,EAAE,CAAC,CAAC;QACjE,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC;IACpB,CAAC;AACL,CAAC,CAAC,CAAC;AAEP,OAAO,CAAC,KAAK,EAAE,CAAC"}
|
package/dist/parser.d.ts.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"parser.d.ts","sourceRoot":"","sources":["../src/parser.ts"],"names":[],"mappings":"AAAA;;GAEG;AAKH,OAAO,KAAK,EAAE,QAAQ,EAAY,cAAc,EAAE,MAAM,YAAY,CAAC;AAIrE,2CAA2C;AAC3C,wBAAgB,SAAS,CAAC,OAAO,EAAE,MAAM,GAAG,QAAQ,CAGnD;AAED,uCAAuC;AACvC,wBAAsB,aAAa,CAAC,QAAQ,EAAE,MAAM,GAAG,OAAO,CAAC,QAAQ,CAAC,CAGvE;AA4CD,mDAAmD;AACnD,wBAAgB,YAAY,CAAC,OAAO,EAAE,MAAM,EAAE,YAAY,SAAQ,GAAG,cAAc,CAWlF;AAED,8CAA8C;AAC9C,wBAAsB,gBAAgB,CAAC,QAAQ,EAAE,MAAM,GAAG,OAAO,CAAC,cAAc,CAAC,CAGhF;
|
|
1
|
+
{"version":3,"file":"parser.d.ts","sourceRoot":"","sources":["../src/parser.ts"],"names":[],"mappings":"AAAA;;GAEG;AAKH,OAAO,KAAK,EAAE,QAAQ,EAAY,cAAc,EAAE,MAAM,YAAY,CAAC;AAIrE,2CAA2C;AAC3C,wBAAgB,SAAS,CAAC,OAAO,EAAE,MAAM,GAAG,QAAQ,CAGnD;AAED,uCAAuC;AACvC,wBAAsB,aAAa,CAAC,QAAQ,EAAE,MAAM,GAAG,OAAO,CAAC,QAAQ,CAAC,CAGvE;AA4CD,mDAAmD;AACnD,wBAAgB,YAAY,CAAC,OAAO,EAAE,MAAM,EAAE,YAAY,SAAQ,GAAG,cAAc,CAWlF;AAED,8CAA8C;AAC9C,wBAAsB,gBAAgB,CAAC,QAAQ,EAAE,MAAM,GAAG,OAAO,CAAC,cAAc,CAAC,CAGhF;AAkFD,uEAAuE;AACvE,wBAAsB,iBAAiB,CAAC,GAAG,EAAE,MAAM,GAAG,OAAO,CAAC,MAAM,EAAE,CAAC,CAGtE"}
|
package/dist/parser.js
CHANGED
|
@@ -173,12 +173,17 @@ function parseAiConfig(content) {
|
|
|
173
173
|
if (!content)
|
|
174
174
|
return undefined;
|
|
175
175
|
const kv = parseKeyValues(content);
|
|
176
|
-
if (!kv['url'] && !kv['model'])
|
|
176
|
+
if (!kv['url'] && !kv['model'] && !kv['provider'])
|
|
177
177
|
return undefined;
|
|
178
|
-
|
|
178
|
+
const config = {
|
|
179
|
+
provider: kv['provider'] || 'ollama',
|
|
179
180
|
url: kv['url'] || 'http://localhost:11434',
|
|
180
181
|
model: kv['model'] || 'llama3.2:3b',
|
|
181
182
|
};
|
|
183
|
+
if (kv['apiKey'] || kv['api_key']) {
|
|
184
|
+
config.apiKey = kv['apiKey'] || kv['api_key'];
|
|
185
|
+
}
|
|
186
|
+
return config;
|
|
182
187
|
}
|
|
183
188
|
// -- File discovery --
|
|
184
189
|
/** Recursively find all `.yaml` / `.yml` test files in a directory. */
|
package/dist/parser.js.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"parser.js","sourceRoot":"","sources":["../src/parser.ts"],"names":[],"mappings":";AAAA;;GAEG;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;AAUH,8BAGC;AAGD,sCAGC;AA6CD,oCAWC;AAGD,4CAGC;
|
|
1
|
+
{"version":3,"file":"parser.js","sourceRoot":"","sources":["../src/parser.ts"],"names":[],"mappings":";AAAA;;GAEG;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;AAUH,8BAGC;AAGD,sCAGC;AA6CD,oCAWC;AAGD,4CAGC;AAmFD,8CAGC;AArKD,+BAAgD;AAChD,gDAAkC;AAClC,2CAA6B;AAG7B,qBAAqB;AAErB,2CAA2C;AAC3C,SAAgB,SAAS,CAAC,OAAe;IACvC,MAAM,MAAM,GAAG,IAAA,YAAe,EAAC,OAAO,CAAC,CAAC;IACxC,OAAO,aAAa,CAAC,MAAM,CAAC,CAAC;AAC/B,CAAC;AAED,uCAAuC;AAChC,KAAK,UAAU,aAAa,CAAC,QAAgB;IAClD,MAAM,OAAO,GAAG,MAAM,EAAE,CAAC,QAAQ,CAAC,QAAQ,EAAE,OAAO,CAAC,CAAC;IACrD,OAAO,SAAS,CAAC,OAAO,CAAC,CAAC;AAC5B,CAAC;AAED,SAAS,aAAa,CAAC,IAAa;IAClC,MAAM,GAAG,GAAG,IAA+B,CAAC;IAE5C,IAAI,CAAC,GAAG,CAAC,IAAI,IAAI,CAAC,GAAG,CAAC,KAAK,EAAE,CAAC;QAC5B,MAAM,IAAI,KAAK,CAAC,+CAA+C,CAAC,CAAC;IACnE,CAAC;IAED,OAAO;QACL,IAAI,EAAE,GAAG,CAAC,IAAc;QACxB,WAAW,EAAE,GAAG,CAAC,WAAiC;QAClD,IAAI,EAAE,GAAG,CAAC,IAA4B;QACtC,KAAK,EAAG,GAAG,CAAC,KAAmB,CAAC,GAAG,CAAC,aAAa,CAAC;KACnD,CAAC;AACJ,CAAC;AAED,SAAS,aAAa,CAAC,IAAa;IAClC,MAAM,GAAG,GAAG,IAA+B,CAAC;IAE5C,IAAI,CAAC,GAAG,CAAC,IAAI,IAAI,CAAC,GAAG,CAAC,OAAO,EAAE,CAAC;QAC9B,MAAM,IAAI,KAAK,CAAC,4CAA4C,CAAC,CAAC;IAChE,CAAC;IAED,MAAM,GAAG,GAAG,GAAG,CAAC,OAAkC,CAAC;IAEnD,OAAO;QACL,IAAI,EAAE,GAAG,CAAC,IAAc;QACxB,WAAW,EAAE,GAAG,CAAC,WAAiC;QAClD,OAAO,EAAE;YACP,MAAM,EAAE,CAAE,GAAG,CAAC,MAAiB,IAAI,KAAK,CAAC,CAAC,WAAW,EAAmC;YACxF,GAAG,EAAE,GAAG,CAAC,GAAa;YACtB,OAAO,EAAE,GAAG,CAAC,OAA6C;YAC1D,IAAI,EAAE,GAAG,CAAC,IAAI;YACd,OAAO,EAAE,GAAG,CAAC,OAAqD;SACnE;QACD,OAAO,EAAE,GAAG,CAAC,OAA0C;QACvD,UAAU,EAAE,GAAG,CAAC,UAAgD;QAChE,SAAS,EAAE,GAAG,CAAC,SAA8C;KAC9D,CAAC;AACJ,CAAC;AAED,wBAAwB;AAExB,mDAAmD;AACnD,SAAgB,YAAY,CAAC,OAAe,EAAE,YAAY,GAAG,KAAK;IAChE,MAAM,QAAQ,GAAG,eAAe,CAAC,OAAO,CAAC,CAAC;IAE1C,OAAO;QACL,IAAI,EAAE,QAAQ,CAAC,MAAM,CAAC,IAAI,YAAY;QACtC,WAAW,EAAE,QAAQ,CAAC,aAAa,CAAC,IAAI,EAAE;QAC1C,QAAQ,EAAE,cAAc,CAAC,QAAQ,CAAC,WAAW,CAAC,IAAI,QAAQ,CAAC,MAAM,CAAC,IAAI,EAAE,CAAC;QACzE,SAAS,EAAE,cAAc,CAAC,QAAQ,CAAC,WAAW,CAAC,IAAI,EAAE,CAAC;QACtD,KAAK,EAAE,SAAS,CAAC,QAAQ,CAAC,OAAO,CAAC,IAAI,QAAQ,CAAC,gBAAgB,CAAC,IAAI,EAAE,CAAC;QACvE,EAAE,EAAE,aAAa,CAAC,QAAQ,CAAC,kBAAkB,CAAC,IAAI,QAAQ,CAAC,IAAI,CAAC,IAAI,EAAE,CAAC;KACxE,CAAC;AACJ,CAAC;AAED,8CAA8C;AACvC,KAAK,UAAU,gBAAgB,CAAC,QAAgB;IACrD,MAAM,OAAO,GAAG,MAAM,EAAE,CAAC,QAAQ,CAAC,QAAQ,EAAE,OAAO,CAAC,CAAC;IACrD,OAAO,YAAY,CAAC,OAAO,EAAE,IAAI,CAAC,QAAQ,CAAC,QAAQ,EAAE,KAAK,CAAC,CAAC,CAAC;AAC/D,CAAC;AAED,SAAS,eAAe,CAAC,OAAe;IACtC,MAAM,QAAQ,GAA2B,EAAE,CAAC;IAC5C,MAAM,KAAK,GAAG,OAAO,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC;IAClC,IAAI,cAAc,GAAG,aAAa,CAAC;IACnC,IAAI,MAAM,GAAa,EAAE,CAAC;IAE1B,KAAK,MAAM,IAAI,IAAI,KAAK,EAAE,CAAC;QACzB,MAAM,EAAE,GAAG,IAAI,CAAC,KAAK,CAAC,YAAY,CAAC,CAAC;QACpC,MAAM,OAAO,GAAG,IAAI,CAAC,KAAK,CAAC,gBAAgB,CAAC,CAAC;QAE7C,IAAI,EAAE,IAAI,CAAC,QAAQ,CAAC,MAAM,CAAC,EAAE,CAAC;YAC5B,yCAAyC;YACzC,IAAI,MAAM,CAAC,MAAM;gBAAE,QAAQ,CAAC,cAAc,CAAC,WAAW,EAAE,CAAC,GAAG,MAAM,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC,IAAI,EAAE,CAAC;YACrF,QAAQ,CAAC,MAAM,CAAC,GAAG,EAAE,CAAC,CAAC,CAAC,CAAC,IAAI,EAAE,CAAC;YAChC,cAAc,GAAG,aAAa,CAAC;YAC/B,MAAM,GAAG,EAAE,CAAC;QACd,CAAC;aAAM,IAAI,OAAO,EAAE,CAAC;YACnB,IAAI,MAAM,CAAC,MAAM;gBAAE,QAAQ,CAAC,cAAc,CAAC,WAAW,EAAE,CAAC,GAAG,MAAM,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC,IAAI,EAAE,CAAC;YACrF,cAAc,GAAG,OAAO,CAAC,CAAC,CAAC,CAAC;YAC5B,MAAM,GAAG,EAAE,CAAC;QACd,CAAC;aAAM,CAAC;YACN,MAAM,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;QACpB,CAAC;IACH,CAAC;IACD,IAAI,MAAM,CAAC,MAAM;QAAE,QAAQ,CAAC,cAAc,CAAC,WAAW,EAAE,CAAC,GAAG,MAAM,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC,IAAI,EAAE,CAAC;IAErF,OAAO,QAAQ,CAAC;AAClB,CAAC;AAED,SAAS,cAAc,CAAC,OAAe;IACrC,MAAM,MAAM,GAA2B,EAAE,CAAC;IAC1C,KAAK,MAAM,IAAI,IAAI,OAAO,CAAC,KAAK,CAAC,IAAI,CAAC,EAAE,CAAC;QACvC,MAAM,CAAC,GAAG,IAAI,CAAC,KAAK,CAAC,sBAAsB,CAAC,CAAC;QAC7C,IAAI,CAAC;YAAE,MAAM,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,CAAC,IAAI,EAAE,CAAC;IACpC,CAAC;IACD,OAAO,MAAM,CAAC;AAChB,CAAC;AAED,SAAS,cAAc,CAAC,OAAe;IACrC,MAAM,SAAS,GAAgC,EAAE,CAAC;IAClD,KAAK,MAAM,IAAI,IAAI,OAAO,CAAC,KAAK,CAAC,IAAI,CAAC,EAAE,CAAC;QACvC,MAAM,CAAC,GAAG,IAAI,CAAC,KAAK,CAAC,0EAA0E,CAAC,CAAC;QACjG,IAAI,CAAC,EAAE,CAAC;YACN,SAAS,CAAC,IAAI,CAAC;gBACb,IAAI,EAAE,CAAC,CAAC,CAAC,CAAC,IAAI,GAAG,CAAC,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC,EAAE;gBAC/B,MAAM,EAAE,CAAC,CAAC,CAAC,CAAC,CAAC,WAAW,EAA8C;gBACtE,IAAI,EAAE,CAAC,CAAC,CAAC,CAAC;gBACV,WAAW,EAAE,CAAC,CAAC,CAAC,CAAC;aAClB,CAAC,CAAC;QACL,CAAC;IACH,CAAC;IACD,OAAO,SAAS,CAAC;AACnB,CAAC;AAED,SAAS,SAAS,CAAC,OAAe;IAChC,MAAM,KAAK,GAAa,EAAE,CAAC;IAC3B,KAAK,MAAM,IAAI,IAAI,OAAO,CAAC,KAAK,CAAC,IAAI,CAAC,EAAE,CAAC;QACvC,MAAM,CAAC,GAAG,IAAI,CAAC,KAAK,CAAC,YAAY,CAAC,CAAC;QACnC,IAAI,CAAC;YAAE,KAAK,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,IAAI,EAAE,CAAC,CAAC;IACjC,CAAC;IACD,OAAO,KAAK,CAAC;AACf,CAAC;AAED,SAAS,aAAa,CAAC,OAAe;IACpC,IAAI,CAAC,OAAO;QAAE,OAAO,SAAS,CAAC;IAC/B,MAAM,EAAE,GAAG,cAAc,CAAC,OAAO,CAAC,CAAC;IACnC,IAAI,CAAC,EAAE,CAAC,KAAK,CAAC,IAAI,CAAC,EAAE,CAAC,OAAO,CAAC,IAAI,CAAC,EAAE,CAAC,UAAU,CAAC;QAAE,OAAO,SAAS,CAAC;IACpE,MAAM,MAAM,GAAsC;QAChD,QAAQ,EAAG,EAAE,CAAC,UAAU,CAAuC,IAAI,QAAQ;QAC3E,GAAG,EAAE,EAAE,CAAC,KAAK,CAAC,IAAI,wBAAwB;QAC1C,KAAK,EAAE,EAAE,CAAC,OAAO,CAAC,IAAI,aAAa;KACpC,CAAC;IACF,IAAI,EAAE,CAAC,QAAQ,CAAC,IAAI,EAAE,CAAC,SAAS,CAAC,EAAE,CAAC;QAClC,MAAM,CAAC,MAAM,GAAG,EAAE,CAAC,QAAQ,CAAC,IAAI,EAAE,CAAC,SAAS,CAAC,CAAC;IAChD,CAAC;IACD,OAAO,MAAM,CAAC;AAChB,CAAC;AAED,uBAAuB;AAEvB,uEAAuE;AAChE,KAAK,UAAU,iBAAiB,CAAC,GAAW;IACjD,MAAM,EAAE,IAAI,EAAE,GAAG,MAAM,MAAM,CAAC,MAAM,CAAC,CAAC;IACtC,OAAO,IAAI,CAAC,IAAI,CAAC,IAAI,CAAC,GAAG,EAAE,iBAAiB,CAAC,CAAC,CAAC;AACjD,CAAC"}
|
package/dist/types.d.ts
CHANGED
|
@@ -19,8 +19,10 @@ export interface EndpointDefinition {
|
|
|
19
19
|
}
|
|
20
20
|
/** AI configuration from the context file. */
|
|
21
21
|
export interface AiContextConfig {
|
|
22
|
-
|
|
23
|
-
|
|
22
|
+
provider?: AiProvider;
|
|
23
|
+
url?: string;
|
|
24
|
+
apiKey?: string;
|
|
25
|
+
model?: string;
|
|
24
26
|
}
|
|
25
27
|
/** A complete test flow: a named sequence of steps. */
|
|
26
28
|
export interface TestFlow {
|
|
@@ -123,14 +125,20 @@ export interface TestReport {
|
|
|
123
125
|
flows: FlowResult[];
|
|
124
126
|
narrative: string;
|
|
125
127
|
}
|
|
128
|
+
/** AI provider type. */
|
|
129
|
+
export type AiProvider = 'ollama' | 'openai' | 'anthropic';
|
|
126
130
|
/** AI evaluator configuration. */
|
|
127
131
|
export interface AiConfig {
|
|
128
|
-
/**
|
|
129
|
-
|
|
130
|
-
/**
|
|
131
|
-
|
|
132
|
+
/** AI provider (default: `ollama`). */
|
|
133
|
+
provider?: AiProvider;
|
|
134
|
+
/** API URL (for Ollama, default: `http://localhost:11434`). */
|
|
135
|
+
url?: string;
|
|
136
|
+
/** API key (required for OpenAI and Anthropic). */
|
|
137
|
+
apiKey?: string;
|
|
138
|
+
/** Model name (default varies by provider). */
|
|
139
|
+
model?: string;
|
|
132
140
|
/** Request timeout in ms (default: 30 000). */
|
|
133
|
-
timeout
|
|
141
|
+
timeout?: number;
|
|
134
142
|
}
|
|
135
143
|
/** Result returned by the AI evaluator. */
|
|
136
144
|
export interface AiEvaluation {
|
package/dist/types.d.ts.map
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
{"version":3,"file":"types.d.ts","sourceRoot":"","sources":["../src/types.ts"],"names":[],"mappings":"AAAA;;GAEG;AAIH,mDAAmD;AACnD,MAAM,WAAW,cAAc;IAC7B,IAAI,EAAE,MAAM,CAAC;IACb,WAAW,EAAE,MAAM,CAAC;IACpB,QAAQ,EAAE,MAAM,CAAC,MAAM,EAAE,MAAM,CAAC,CAAC;IACjC,SAAS,EAAE,kBAAkB,EAAE,CAAC;IAChC,KAAK,EAAE,MAAM,EAAE,CAAC;IAChB,EAAE,CAAC,EAAE,eAAe,CAAC;CACtB;AAED,+DAA+D;AAC/D,MAAM,WAAW,kBAAkB;IACjC,IAAI,EAAE,MAAM,CAAC;IACb,MAAM,EAAE,KAAK,GAAG,MAAM,GAAG,KAAK,GAAG,QAAQ,GAAG,OAAO,CAAC;IACpD,IAAI,EAAE,MAAM,CAAC;IACb,WAAW,CAAC,EAAE,MAAM,CAAC;CACtB;AAED,8CAA8C;AAC9C,MAAM,WAAW,eAAe;IAC9B,GAAG,EAAE,MAAM,CAAC;
|
|
1
|
+
{"version":3,"file":"types.d.ts","sourceRoot":"","sources":["../src/types.ts"],"names":[],"mappings":"AAAA;;GAEG;AAIH,mDAAmD;AACnD,MAAM,WAAW,cAAc;IAC7B,IAAI,EAAE,MAAM,CAAC;IACb,WAAW,EAAE,MAAM,CAAC;IACpB,QAAQ,EAAE,MAAM,CAAC,MAAM,EAAE,MAAM,CAAC,CAAC;IACjC,SAAS,EAAE,kBAAkB,EAAE,CAAC;IAChC,KAAK,EAAE,MAAM,EAAE,CAAC;IAChB,EAAE,CAAC,EAAE,eAAe,CAAC;CACtB;AAED,+DAA+D;AAC/D,MAAM,WAAW,kBAAkB;IACjC,IAAI,EAAE,MAAM,CAAC;IACb,MAAM,EAAE,KAAK,GAAG,MAAM,GAAG,KAAK,GAAG,QAAQ,GAAG,OAAO,CAAC;IACpD,IAAI,EAAE,MAAM,CAAC;IACb,WAAW,CAAC,EAAE,MAAM,CAAC;CACtB;AAED,8CAA8C;AAC9C,MAAM,WAAW,eAAe;IAC9B,QAAQ,CAAC,EAAE,UAAU,CAAC;IACtB,GAAG,CAAC,EAAE,MAAM,CAAC;IACb,MAAM,CAAC,EAAE,MAAM,CAAC;IAChB,KAAK,CAAC,EAAE,MAAM,CAAC;CAChB;AAID,uDAAuD;AACvD,MAAM,WAAW,QAAQ;IACvB,IAAI,EAAE,MAAM,CAAC;IACb,WAAW,CAAC,EAAE,MAAM,CAAC;IACrB,IAAI,CAAC,EAAE,MAAM,EAAE,CAAC;IAChB,KAAK,EAAE,QAAQ,EAAE,CAAC;CACnB;AAED,wCAAwC;AACxC,MAAM,WAAW,QAAQ;IACvB,IAAI,EAAE,MAAM,CAAC;IACb,WAAW,CAAC,EAAE,MAAM,CAAC;IACrB,OAAO,EAAE,iBAAiB,CAAC;IAC3B,OAAO,CAAC,EAAE,iBAAiB,EAAE,CAAC;IAC9B,UAAU,CAAC,EAAE,mBAAmB,EAAE,CAAC;IACnC,mEAAmE;IACnE,SAAS,CAAC,EAAE,eAAe,CAAC;CAC7B;AAED,+BAA+B;AAC/B,MAAM,WAAW,iBAAiB;IAChC,MAAM,EAAE,KAAK,GAAG,MAAM,GAAG,KAAK,GAAG,QAAQ,GAAG,OAAO,CAAC;IACpD,GAAG,EAAE,MAAM,CAAC;IACZ,OAAO,CAAC,EAAE,MAAM,CAAC,MAAM,EAAE,MAAM,CAAC,CAAC;IACjC,IAAI,CAAC,EAAE,OAAO,CAAC;IACf,OAAO,CAAC,EAAE,cAAc,CAAC;CAC1B;AAED,uCAAuC;AACvC,MAAM,WAAW,cAAc;IAC7B,KAAK,EAAE,MAAM,CAAC;IACd,SAAS,CAAC,EAAE,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,CAAC;IACpC,aAAa,CAAC,EAAE,MAAM,CAAC;CACxB;AAED,+DAA+D;AAC/D,MAAM,WAAW,iBAAiB;IAChC,IAAI,EAAE,MAAM,CAAC;IACb,iFAAiF;IACjF,IAAI,EAAE,MAAM,CAAC;CACd;AAED,0CAA0C;AAC1C,MAAM,WAAW,mBAAmB;IAClC,+EAA+E;IAC/E,IAAI,EAAE,MAAM,CAAC;IACb,QAAQ,EACJ,QAAQ,GACR,WAAW,GACX,UAAU,GACV,aAAa,GACb,QAAQ,GACR,WAAW,GACX,aAAa,GACb,UAAU,GACV,SAAS,GACT,aAAa,CAAC;IAClB,8DAA8D;IAC9D,KAAK,CAAC,EAAE,OAAO,CAAC;IAChB,uCAAuC;IACvC,OAAO,CAAC,EAAE,MAAM,CAAC;CAClB;AAED,kDAAkD;AAClD,MAAM,WAAW,eAAe;IAC9B,IAAI,EAAE,MAAM,CAAC;IACb,QAAQ,EAAE,QAAQ,GAAG,WAAW,GAAG,QAAQ,GAAG,WAAW,CAAC;IAC1D,KAAK,CAAC,EAAE,OAAO,CAAC;IAChB,6CAA6C;IAC7C,OAAO,CAAC,EAAE,MAAM,CAAC;IACjB,4CAA4C;IAC5C,QAAQ,CAAC,EAAE,MAAM,CAAC;CACnB;AAID,2CAA2C;AAC3C,MAAM,WAAW,UAAU;IACzB,IAAI,EAAE,QAAQ,CAAC;IACf,OAAO,EAAE,OAAO,CAAC;IACjB,QAAQ,EAAE,MAAM,CAAC;IACjB,KAAK,EAAE,UAAU,EAAE,CAAC;IACpB,SAAS,EAAE,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,CAAC;CACpC;AAED,yCAAyC;AACzC,MAAM,WAAW,UAAU;IACzB,IAAI,EAAE,QAAQ,CAAC;IACf,OAAO,EAAE,OAAO,CAAC;IACjB,QAAQ,EAAE,MAAM,CAAC;IACjB,OAAO,EAAE;QAAE,MAAM,EAAE,MAAM,CAAC;QAAC,GAAG,EAAE,MAAM,CAAC;QAAC,IAAI,CAAC,EAAE,OAAO,CAAA;KAAE,CAAC;IACzD,QAAQ,CAAC,EAAE;QAAE,MAAM,EAAE,MAAM,CAAC;QAAC,OAAO,EAAE,MAAM,CAAC,MAAM,EAAE,MAAM,CAAC,CAAC;QAAC,IAAI,EAAE,OAAO,CAAA;KAAE,CAAC;IAC9E,QAAQ,EAAE,MAAM,CAAC,MAAM,EAAE,OAAO,CAAC,CAAC;IAClC,UAAU,EAAE,eAAe,EAAE,CAAC;IAC9B,KAAK,CAAC,EAAE,MAAM,CAAC;CAChB;AAED,+CAA+C;AAC/C,MAAM,WAAW,eAAe;IAC9B,SAAS,EAAE,mBAAmB,CAAC;IAC/B,OAAO,EAAE,OAAO,CAAC;IACjB,MAAM,CAAC,EAAE,OAAO,CAAC;IACjB,OAAO,EAAE,MAAM,CAAC;CACjB;AAID,8BAA8B;AAC9B,MAAM,WAAW,UAAU;IACzB,SAAS,EAAE,IAAI,CAAC;IAChB,QAAQ,EAAE,MAAM,CAAC;IACjB,UAAU,EAAE,MAAM,CAAC;IACnB,WAAW,EAAE,MAAM,CAAC;IACpB,WAAW,EAAE,MAAM,CAAC;IACpB,KAAK,EAAE,UAAU,EAAE,CAAC;IACpB,SAAS,EAAE,MAAM,CAAC;CACnB;AAID,wBAAwB;AACxB,MAAM,MAAM,UAAU,GAAG,QAAQ,GAAG,QAAQ,GAAG,WAAW,CAAC;AAE3D,kCAAkC;AAClC,MAAM,WAAW,QAAQ;IACvB,uCAAuC;IACvC,QAAQ,CAAC,EAAE,UAAU,CAAC;IACtB,+DAA+D;IAC/D,GAAG,CAAC,EAAE,MAAM,CAAC;IACb,mDAAmD;IACnD,MAAM,CAAC,EAAE,MAAM,CAAC;IAChB,+CAA+C;IAC/C,KAAK,CAAC,EAAE,MAAM,CAAC;IACf,+CAA+C;IAC/C,OAAO,CAAC,EAAE,MAAM,CAAC;CAClB;AAED,2CAA2C;AAC3C,MAAM,WAAW,YAAY;IAC3B,IAAI,EAAE,OAAO,CAAC;IACd,UAAU,EAAE,MAAM,CAAC;IACnB,MAAM,EAAE,MAAM,CAAC;CAChB"}
|
package/package.json
CHANGED
|
@@ -1,16 +1,16 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "testflow-ai",
|
|
3
|
-
"version": "0.
|
|
3
|
+
"version": "0.3.0",
|
|
4
4
|
"description": "Declarative API testing powered by YAML flows. Replace Postman with version-controlled, AI-friendly test definitions.",
|
|
5
5
|
"author": "Marcos Carbajal",
|
|
6
6
|
"license": "MIT",
|
|
7
7
|
"repository": {
|
|
8
8
|
"type": "git",
|
|
9
|
-
"url": "https://github.com/
|
|
9
|
+
"url": "https://github.com/carbajalmarcos/testflow-ai"
|
|
10
10
|
},
|
|
11
|
-
"homepage": "https://github.com/
|
|
11
|
+
"homepage": "https://github.com/carbajalmarcos/testflow-ai#readme",
|
|
12
12
|
"bugs": {
|
|
13
|
-
"url": "https://github.com/
|
|
13
|
+
"url": "https://github.com/carbajalmarcos/testflow-ai/issues"
|
|
14
14
|
},
|
|
15
15
|
"main": "./dist/index.js",
|
|
16
16
|
"types": "./dist/index.d.ts",
|