testflow-ai 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/LICENSE +21 -0
- package/README.md +502 -0
- package/dist/ai.d.ts +20 -0
- package/dist/ai.d.ts.map +1 -0
- package/dist/ai.js +59 -0
- package/dist/ai.js.map +1 -0
- package/dist/cli.d.ts +11 -0
- package/dist/cli.d.ts.map +1 -0
- package/dist/cli.js +49 -0
- package/dist/cli.js.map +1 -0
- package/dist/executor.d.ts +28 -0
- package/dist/executor.d.ts.map +1 -0
- package/dist/executor.js +390 -0
- package/dist/executor.js.map +1 -0
- package/dist/index.d.ts +28 -0
- package/dist/index.d.ts.map +1 -0
- package/dist/index.js +53 -0
- package/dist/index.js.map +1 -0
- package/dist/parser.d.ts +15 -0
- package/dist/parser.d.ts.map +1 -0
- package/dist/parser.js +189 -0
- package/dist/parser.js.map +1 -0
- package/dist/reporter.d.ts +15 -0
- package/dist/reporter.d.ts.map +1 -0
- package/dist/reporter.js +152 -0
- package/dist/reporter.js.map +1 -0
- package/dist/runner.d.ts +30 -0
- package/dist/runner.d.ts.map +1 -0
- package/dist/runner.js +81 -0
- package/dist/runner.js.map +1 -0
- package/dist/types.d.ts +141 -0
- package/dist/types.d.ts.map +1 -0
- package/dist/types.js +6 -0
- package/dist/types.js.map +1 -0
- package/package.json +70 -0
package/LICENSE
ADDED
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2025 Marcos Carbajal
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
package/README.md
ADDED
|
@@ -0,0 +1,502 @@
|
|
|
1
|
+
# testflow-ai
|
|
2
|
+
|
|
3
|
+
> Declarative API testing powered by YAML flows. Version-controlled, human-readable, AI-friendly.
|
|
4
|
+
|
|
5
|
+
[](https://www.npmjs.com/package/testflow-ai)
|
|
6
|
+
[](https://opensource.org/licenses/MIT)
|
|
7
|
+
[](https://nodejs.org)
|
|
8
|
+
|
|
9
|
+
---
|
|
10
|
+
|
|
11
|
+
## Why?
|
|
12
|
+
|
|
13
|
+
I was building a backend that started as a simple API and grew into a system with GraphQL, async workers, state machines, and AI-powered evaluations.
|
|
14
|
+
|
|
15
|
+
Testing started simple ā a few requests in Postman. Then the project scaled:
|
|
16
|
+
|
|
17
|
+
- **Postman / Insomnia** became unmanageable. Dozens of collections, manual token copying, no version control.
|
|
18
|
+
- **IDE AI assistants** worked for one-off requests but burned through tokens, lost context, and couldn't maintain complex multi-step flows.
|
|
19
|
+
- **MCP servers and tooling** required significant setup and ongoing maintenance.
|
|
20
|
+
|
|
21
|
+
I needed something that:
|
|
22
|
+
|
|
23
|
+
1. Lives in the repo alongside my code
|
|
24
|
+
2. Defines multi-step flows declaratively
|
|
25
|
+
3. Captures variables between steps automatically
|
|
26
|
+
4. Supports REST, GraphQL, and async operations
|
|
27
|
+
5. Can leverage a **local AI model** for intelligent assertions
|
|
28
|
+
6. Runs in CI/CD with zero cloud dependencies
|
|
29
|
+
|
|
30
|
+
**testflow-ai** is the result.
|
|
31
|
+
|
|
32
|
+
---
|
|
33
|
+
|
|
34
|
+
## Features
|
|
35
|
+
|
|
36
|
+
- š **YAML test flows** ā define test sequences declaratively
|
|
37
|
+
- š **Variable capture** ā extract values from responses, reuse in later steps
|
|
38
|
+
- ā
**Rich assertions** ā equals, contains, exists, greaterThan, matches, and more
|
|
39
|
+
- š **GraphQL native** ā first-class support for queries and mutations
|
|
40
|
+
- ā³ **Async polling** ā `waitUntil` for operations that take time
|
|
41
|
+
- š¤ **AI evaluation** ā assert with natural language via a local Ollama model
|
|
42
|
+
- š **Context files** ā define base URLs, endpoints, and rules in Markdown
|
|
43
|
+
- š **Multiple formats** ā console, JSON, or Markdown reports
|
|
44
|
+
- šÆ **Tag filtering** ā run subsets of your test suite
|
|
45
|
+
- š„ļø **CLI + API** ā use from the terminal or import as a library
|
|
46
|
+
|
|
47
|
+
---
|
|
48
|
+
|
|
49
|
+
## Quick Start
|
|
50
|
+
|
|
51
|
+
### 1. Install
|
|
52
|
+
|
|
53
|
+
```bash
|
|
54
|
+
npm install testflow-ai
|
|
55
|
+
```
|
|
56
|
+
|
|
57
|
+
### 2. Create a test flow
|
|
58
|
+
|
|
59
|
+
```yaml
|
|
60
|
+
# tests/health.yaml
|
|
61
|
+
name: Health Check
|
|
62
|
+
tags: [smoke]
|
|
63
|
+
steps:
|
|
64
|
+
- name: Ping API
|
|
65
|
+
request:
|
|
66
|
+
method: GET
|
|
67
|
+
url: http://localhost:3000/health
|
|
68
|
+
assertions:
|
|
69
|
+
- path: status
|
|
70
|
+
operator: equals
|
|
71
|
+
value: 200
|
|
72
|
+
```
|
|
73
|
+
|
|
74
|
+
### 3. Run
|
|
75
|
+
|
|
76
|
+
```bash
|
|
77
|
+
npx testflow tests/health.yaml
|
|
78
|
+
```
|
|
79
|
+
|
|
80
|
+
That's it. No config files, no GUI, no account.
|
|
81
|
+
|
|
82
|
+
---
|
|
83
|
+
|
|
84
|
+
## Installation
|
|
85
|
+
|
|
86
|
+
```bash
|
|
87
|
+
npm install testflow-ai
|
|
88
|
+
# or
|
|
89
|
+
pnpm add testflow-ai
|
|
90
|
+
# or
|
|
91
|
+
yarn add testflow-ai
|
|
92
|
+
```
|
|
93
|
+
|
|
94
|
+
---
|
|
95
|
+
|
|
96
|
+
## CLI Usage
|
|
97
|
+
|
|
98
|
+
```bash
|
|
99
|
+
# Run specific files
|
|
100
|
+
testflow flow1.yaml flow2.yaml
|
|
101
|
+
|
|
102
|
+
# Run all YAML files in a directory
|
|
103
|
+
testflow --dir ./tests
|
|
104
|
+
|
|
105
|
+
# Use a context file for base URLs
|
|
106
|
+
testflow --dir ./tests --context ./context.md
|
|
107
|
+
|
|
108
|
+
# Filter by tags
|
|
109
|
+
testflow --dir ./tests --tags smoke,auth
|
|
110
|
+
|
|
111
|
+
# JSON output (for CI/CD)
|
|
112
|
+
testflow --dir ./tests --format json
|
|
113
|
+
|
|
114
|
+
# Markdown output (for reports)
|
|
115
|
+
testflow --dir ./tests --format markdown
|
|
116
|
+
|
|
117
|
+
# Verbose mode
|
|
118
|
+
testflow --dir ./tests -v
|
|
119
|
+
|
|
120
|
+
# With AI evaluation (Ollama)
|
|
121
|
+
testflow --dir ./tests --ai-model llama3.2:3b
|
|
122
|
+
```
|
|
123
|
+
|
|
124
|
+
---
|
|
125
|
+
|
|
126
|
+
## Programmatic API
|
|
127
|
+
|
|
128
|
+
```typescript
|
|
129
|
+
import { runTests } from 'testflow-ai';
|
|
130
|
+
|
|
131
|
+
const report = await runTests({
|
|
132
|
+
contextFile: './context.md',
|
|
133
|
+
testDir: './tests',
|
|
134
|
+
tags: ['smoke'],
|
|
135
|
+
format: 'console',
|
|
136
|
+
verbose: true,
|
|
137
|
+
});
|
|
138
|
+
|
|
139
|
+
console.log(`${report.passedFlows}/${report.totalFlows} passed`);
|
|
140
|
+
process.exit(report.failedFlows > 0 ? 1 : 0);
|
|
141
|
+
```
|
|
142
|
+
|
|
143
|
+
### Advanced usage
|
|
144
|
+
|
|
145
|
+
```typescript
|
|
146
|
+
import { TestRunner, FlowExecutor, parseYamlFile, parseContextFile } from 'testflow-ai';
|
|
147
|
+
|
|
148
|
+
// Runner with full control
|
|
149
|
+
const runner = new TestRunner({
|
|
150
|
+
contextFile: './context.md',
|
|
151
|
+
testFiles: ['./tests/critical.yaml'],
|
|
152
|
+
ai: { model: 'mistral:7b' },
|
|
153
|
+
});
|
|
154
|
+
const report = await runner.run();
|
|
155
|
+
|
|
156
|
+
// Manual execution
|
|
157
|
+
const context = await parseContextFile('./context.md');
|
|
158
|
+
const flow = await parseYamlFile('./tests/flow.yaml');
|
|
159
|
+
const executor = new FlowExecutor(context, true);
|
|
160
|
+
const result = await executor.executeFlow(flow);
|
|
161
|
+
```
|
|
162
|
+
|
|
163
|
+
---
|
|
164
|
+
|
|
165
|
+
## Test Flow Reference
|
|
166
|
+
|
|
167
|
+
### Basic structure
|
|
168
|
+
|
|
169
|
+
```yaml
|
|
170
|
+
name: Flow Name
|
|
171
|
+
description: What this flow tests
|
|
172
|
+
tags:
|
|
173
|
+
- smoke
|
|
174
|
+
- e2e
|
|
175
|
+
|
|
176
|
+
steps:
|
|
177
|
+
- name: Step Name
|
|
178
|
+
request:
|
|
179
|
+
method: POST
|
|
180
|
+
url: "{api}/endpoint"
|
|
181
|
+
headers:
|
|
182
|
+
Content-Type: application/json
|
|
183
|
+
body:
|
|
184
|
+
key: value
|
|
185
|
+
capture:
|
|
186
|
+
- name: variableName
|
|
187
|
+
path: data.field
|
|
188
|
+
assertions:
|
|
189
|
+
- path: status
|
|
190
|
+
operator: equals
|
|
191
|
+
value: 201
|
|
192
|
+
```
|
|
193
|
+
|
|
194
|
+
### REST requests
|
|
195
|
+
|
|
196
|
+
```yaml
|
|
197
|
+
steps:
|
|
198
|
+
- name: Create resource
|
|
199
|
+
request:
|
|
200
|
+
method: POST
|
|
201
|
+
url: "{api}/resources"
|
|
202
|
+
headers:
|
|
203
|
+
Content-Type: application/json
|
|
204
|
+
Authorization: "Bearer ${token}"
|
|
205
|
+
body:
|
|
206
|
+
title: New Resource
|
|
207
|
+
active: true
|
|
208
|
+
```
|
|
209
|
+
|
|
210
|
+
### GraphQL requests
|
|
211
|
+
|
|
212
|
+
```yaml
|
|
213
|
+
steps:
|
|
214
|
+
- name: Query users
|
|
215
|
+
request:
|
|
216
|
+
method: POST
|
|
217
|
+
url: "{graphql}"
|
|
218
|
+
graphql:
|
|
219
|
+
query: |
|
|
220
|
+
query GetUser($id: ID!) {
|
|
221
|
+
user(id: $id) {
|
|
222
|
+
id
|
|
223
|
+
email
|
|
224
|
+
name
|
|
225
|
+
}
|
|
226
|
+
}
|
|
227
|
+
variables:
|
|
228
|
+
id: "${userId}"
|
|
229
|
+
capture:
|
|
230
|
+
- name: email
|
|
231
|
+
path: data.user.email
|
|
232
|
+
```
|
|
233
|
+
|
|
234
|
+
### Variable capture and interpolation
|
|
235
|
+
|
|
236
|
+
Variables captured in one step are available in all subsequent steps:
|
|
237
|
+
|
|
238
|
+
```yaml
|
|
239
|
+
steps:
|
|
240
|
+
- name: Login
|
|
241
|
+
request:
|
|
242
|
+
method: POST
|
|
243
|
+
url: "{api}/auth/login"
|
|
244
|
+
body:
|
|
245
|
+
email: admin@example.com
|
|
246
|
+
password: secret
|
|
247
|
+
capture:
|
|
248
|
+
- name: token
|
|
249
|
+
path: data.accessToken
|
|
250
|
+
- name: userId
|
|
251
|
+
path: data.user.id
|
|
252
|
+
|
|
253
|
+
- name: Get profile
|
|
254
|
+
request:
|
|
255
|
+
method: GET
|
|
256
|
+
url: "{api}/users/${userId}"
|
|
257
|
+
headers:
|
|
258
|
+
Authorization: "Bearer ${token}"
|
|
259
|
+
```
|
|
260
|
+
|
|
261
|
+
Supported interpolation patterns:
|
|
262
|
+
- `${variable}` ā simple variable
|
|
263
|
+
- `${data.nested.field}` ā nested path
|
|
264
|
+
- `${items[0].id}` ā array access
|
|
265
|
+
|
|
266
|
+
### Polling (waitUntil)
|
|
267
|
+
|
|
268
|
+
For async operations ā polls until a condition is met or timeout:
|
|
269
|
+
|
|
270
|
+
```yaml
|
|
271
|
+
steps:
|
|
272
|
+
- name: Wait for processing
|
|
273
|
+
request:
|
|
274
|
+
method: GET
|
|
275
|
+
url: "{api}/jobs/${jobId}"
|
|
276
|
+
waitUntil:
|
|
277
|
+
path: data.status
|
|
278
|
+
operator: equals
|
|
279
|
+
value: "COMPLETED"
|
|
280
|
+
timeout: 30000 # max wait (ms)
|
|
281
|
+
interval: 2000 # poll every (ms)
|
|
282
|
+
assertions:
|
|
283
|
+
- path: data.status
|
|
284
|
+
operator: equals
|
|
285
|
+
value: "COMPLETED"
|
|
286
|
+
```
|
|
287
|
+
|
|
288
|
+
---
|
|
289
|
+
|
|
290
|
+
## Assertions
|
|
291
|
+
|
|
292
|
+
| Operator | Description | Example value |
|
|
293
|
+
|----------|-------------|---------------|
|
|
294
|
+
| `equals` | Exact match (deep equality) | `200` |
|
|
295
|
+
| `notEquals` | Not equal | `null` |
|
|
296
|
+
| `contains` | String/array contains | `"success"` |
|
|
297
|
+
| `notContains` | Does not contain | `"error"` |
|
|
298
|
+
| `exists` | Not null/undefined | ā |
|
|
299
|
+
| `notExists` | Is null/undefined | ā |
|
|
300
|
+
| `greaterThan` | Number comparison | `0` |
|
|
301
|
+
| `lessThan` | Number comparison | `100` |
|
|
302
|
+
| `matches` | Regex match | `"^[a-z]+$"` |
|
|
303
|
+
| `ai-evaluate` | AI-powered evaluation | `"Is this a valid user?"` |
|
|
304
|
+
|
|
305
|
+
### Special paths
|
|
306
|
+
|
|
307
|
+
- `status` ā HTTP status code (when value is a number)
|
|
308
|
+
- `httpStatus` ā always the HTTP status code
|
|
309
|
+
- `data.field` ā response body field
|
|
310
|
+
- `data.items[0].id` ā array access
|
|
311
|
+
|
|
312
|
+
---
|
|
313
|
+
|
|
314
|
+
## AI-Powered Evaluation
|
|
315
|
+
|
|
316
|
+
Use a local LLM to assert things that are hard to express with traditional operators.
|
|
317
|
+
|
|
318
|
+
### Setup Ollama
|
|
319
|
+
|
|
320
|
+
1. **Install Ollama** ā [ollama.com/download](https://ollama.com/download)
|
|
321
|
+
|
|
322
|
+
2. **Pull a model:**
|
|
323
|
+
|
|
324
|
+
```bash
|
|
325
|
+
# Recommended ā good balance of speed and quality
|
|
326
|
+
ollama pull llama3.2:3b
|
|
327
|
+
|
|
328
|
+
# Faster, lighter (for limited hardware)
|
|
329
|
+
ollama pull llama3.2:1b
|
|
330
|
+
|
|
331
|
+
# More accurate (needs ~8GB RAM)
|
|
332
|
+
ollama pull mistral:7b
|
|
333
|
+
```
|
|
334
|
+
|
|
335
|
+
3. **Start Ollama** (runs on `http://localhost:11434` by default):
|
|
336
|
+
|
|
337
|
+
```bash
|
|
338
|
+
ollama serve
|
|
339
|
+
```
|
|
340
|
+
|
|
341
|
+
### Using AI assertions
|
|
342
|
+
|
|
343
|
+
```yaml
|
|
344
|
+
steps:
|
|
345
|
+
- name: Check response quality
|
|
346
|
+
request:
|
|
347
|
+
method: GET
|
|
348
|
+
url: "{api}/articles/1"
|
|
349
|
+
assertions:
|
|
350
|
+
# Traditional assertion
|
|
351
|
+
- path: status
|
|
352
|
+
operator: equals
|
|
353
|
+
value: 200
|
|
354
|
+
# AI-powered assertion
|
|
355
|
+
- path: data.content
|
|
356
|
+
operator: ai-evaluate
|
|
357
|
+
value: "Does this article contain a coherent explanation with at least two paragraphs?"
|
|
358
|
+
```
|
|
359
|
+
|
|
360
|
+
### CLI with AI
|
|
361
|
+
|
|
362
|
+
```bash
|
|
363
|
+
testflow --dir ./tests --ai-model llama3.2:3b
|
|
364
|
+
testflow --dir ./tests --ai-url http://192.168.1.10:11434 --ai-model mistral:7b
|
|
365
|
+
```
|
|
366
|
+
|
|
367
|
+
### Programmatic with AI
|
|
368
|
+
|
|
369
|
+
```typescript
|
|
370
|
+
const report = await runTests({
|
|
371
|
+
testDir: './tests',
|
|
372
|
+
ai: {
|
|
373
|
+
url: 'http://localhost:11434',
|
|
374
|
+
model: 'llama3.2:3b',
|
|
375
|
+
timeout: 30000,
|
|
376
|
+
},
|
|
377
|
+
});
|
|
378
|
+
```
|
|
379
|
+
|
|
380
|
+
### Context file AI config
|
|
381
|
+
|
|
382
|
+
```markdown
|
|
383
|
+
## AI Configuration
|
|
384
|
+
- url: http://localhost:11434
|
|
385
|
+
- model: llama3.2:3b
|
|
386
|
+
```
|
|
387
|
+
|
|
388
|
+
> AI evaluation requires Ollama running locally. No cloud API keys, no data leaves your machine.
|
|
389
|
+
|
|
390
|
+
---
|
|
391
|
+
|
|
392
|
+
## Context Files
|
|
393
|
+
|
|
394
|
+
Define your project context in Markdown. The runner uses it to resolve `{baseUrlKey}` references in your YAML flows.
|
|
395
|
+
|
|
396
|
+
```markdown
|
|
397
|
+
# My API
|
|
398
|
+
|
|
399
|
+
## Description
|
|
400
|
+
Brief description of your API.
|
|
401
|
+
|
|
402
|
+
## Base URLs
|
|
403
|
+
- api: http://localhost:3000
|
|
404
|
+
- graphql: http://localhost:3000/graphql
|
|
405
|
+
|
|
406
|
+
## Endpoints
|
|
407
|
+
- POST /users - Create user
|
|
408
|
+
- GET /users/:id - Get user
|
|
409
|
+
- POST /graphql - GraphQL endpoint
|
|
410
|
+
|
|
411
|
+
## Rules
|
|
412
|
+
- All endpoints return JSON
|
|
413
|
+
- Authentication required for /users
|
|
414
|
+
|
|
415
|
+
## AI Configuration
|
|
416
|
+
- url: http://localhost:11434
|
|
417
|
+
- model: llama3.2:3b
|
|
418
|
+
```
|
|
419
|
+
|
|
420
|
+
---
|
|
421
|
+
|
|
422
|
+
## CI/CD Integration
|
|
423
|
+
|
|
424
|
+
### GitHub Actions
|
|
425
|
+
|
|
426
|
+
```yaml
|
|
427
|
+
jobs:
|
|
428
|
+
api-tests:
|
|
429
|
+
runs-on: ubuntu-latest
|
|
430
|
+
steps:
|
|
431
|
+
- uses: actions/checkout@v4
|
|
432
|
+
- uses: actions/setup-node@v4
|
|
433
|
+
with:
|
|
434
|
+
node-version: '20'
|
|
435
|
+
- run: npm ci
|
|
436
|
+
- run: npm run start:server &
|
|
437
|
+
- run: npx testflow --dir ./tests --context ./context.md --format json > results.json
|
|
438
|
+
- uses: actions/upload-artifact@v4
|
|
439
|
+
with:
|
|
440
|
+
name: test-results
|
|
441
|
+
path: results.json
|
|
442
|
+
```
|
|
443
|
+
|
|
444
|
+
### Exit codes
|
|
445
|
+
|
|
446
|
+
- `0` ā all flows passed
|
|
447
|
+
- `1` ā one or more flows failed
|
|
448
|
+
|
|
449
|
+
---
|
|
450
|
+
|
|
451
|
+
## Output Examples
|
|
452
|
+
|
|
453
|
+
### Console
|
|
454
|
+
|
|
455
|
+
```
|
|
456
|
+
āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā
|
|
457
|
+
TESTFLOW AI ā RESULTS
|
|
458
|
+
āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā
|
|
459
|
+
|
|
460
|
+
Summary:
|
|
461
|
+
Total: 3 flows
|
|
462
|
+
Passed: 2
|
|
463
|
+
Failed: 1
|
|
464
|
+
Duration: 542ms
|
|
465
|
+
|
|
466
|
+
Narrative:
|
|
467
|
+
|
|
468
|
+
ā
**User CRUD**
|
|
469
|
+
ā Create user
|
|
470
|
+
š¦ userId: abc-123
|
|
471
|
+
ā Read user
|
|
472
|
+
ā Update user
|
|
473
|
+
|
|
474
|
+
ā
**Auth Flow**
|
|
475
|
+
ā Login
|
|
476
|
+
š¦ token: eyJhbGā¦
|
|
477
|
+
ā Access protected route
|
|
478
|
+
|
|
479
|
+
ā **Payment Flow**
|
|
480
|
+
ā Create payment
|
|
481
|
+
ā ļø Expected status to equal 200, got 500
|
|
482
|
+
|
|
483
|
+
āāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāāā
|
|
484
|
+
```
|
|
485
|
+
|
|
486
|
+
---
|
|
487
|
+
|
|
488
|
+
## Roadmap
|
|
489
|
+
|
|
490
|
+
- [ ] Database assertions (verify records directly via SQL)
|
|
491
|
+
- [ ] gRPC / RPC support
|
|
492
|
+
- [ ] OpenAPI spec ā auto-generate test flows
|
|
493
|
+
- [ ] Watch mode (re-run on file change)
|
|
494
|
+
- [ ] Parallel flow execution
|
|
495
|
+
- [ ] HTML report output
|
|
496
|
+
- [ ] `testflow init` wizard
|
|
497
|
+
|
|
498
|
+
---
|
|
499
|
+
|
|
500
|
+
## License
|
|
501
|
+
|
|
502
|
+
MIT
|
package/dist/ai.d.ts
ADDED
|
@@ -0,0 +1,20 @@
|
|
|
1
|
+
/**
|
|
2
|
+
* AI evaluation via Ollama (local LLM).
|
|
3
|
+
*
|
|
4
|
+
* Sends a prompt + data to a locally running Ollama instance and returns
|
|
5
|
+
* a structured pass/fail evaluation. No cloud API keys required.
|
|
6
|
+
*/
|
|
7
|
+
import type { AiConfig, AiEvaluation } from './types.js';
|
|
8
|
+
/** Merge partial user config with defaults. */
|
|
9
|
+
export declare function resolveAiConfig(partial?: Partial<AiConfig>): AiConfig;
|
|
10
|
+
/**
|
|
11
|
+
* Evaluate a value against a natural-language prompt using a local LLM.
|
|
12
|
+
*
|
|
13
|
+
* @example
|
|
14
|
+
* ```ts
|
|
15
|
+
* const result = await evaluateWithAi(config, responseBody, 'Does this contain a valid user object?');
|
|
16
|
+
* // { pass: true, confidence: 0.95, reason: 'Response contains id, email, and name fields.' }
|
|
17
|
+
* ```
|
|
18
|
+
*/
|
|
19
|
+
export declare function evaluateWithAi(config: AiConfig, actual: unknown, prompt: string): Promise<AiEvaluation>;
|
|
20
|
+
//# sourceMappingURL=ai.d.ts.map
|
package/dist/ai.d.ts.map
ADDED
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"ai.d.ts","sourceRoot":"","sources":["../src/ai.ts"],"names":[],"mappings":"AAAA;;;;;GAKG;AAGH,OAAO,KAAK,EAAE,QAAQ,EAAE,YAAY,EAAE,MAAM,YAAY,CAAC;AAQzD,+CAA+C;AAC/C,wBAAgB,eAAe,CAAC,OAAO,CAAC,EAAE,OAAO,CAAC,QAAQ,CAAC,GAAG,QAAQ,CAErE;AAED;;;;;;;;GAQG;AACH,wBAAsB,cAAc,CAClC,MAAM,EAAE,QAAQ,EAChB,MAAM,EAAE,OAAO,EACf,MAAM,EAAE,MAAM,GACb,OAAO,CAAC,YAAY,CAAC,CA+BvB"}
|
package/dist/ai.js
ADDED
|
@@ -0,0 +1,59 @@
|
|
|
1
|
+
"use strict";
|
|
2
|
+
/**
|
|
3
|
+
* AI evaluation via Ollama (local LLM).
|
|
4
|
+
*
|
|
5
|
+
* Sends a prompt + data to a locally running Ollama instance and returns
|
|
6
|
+
* a structured pass/fail evaluation. No cloud API keys required.
|
|
7
|
+
*/
|
|
8
|
+
var __importDefault = (this && this.__importDefault) || function (mod) {
|
|
9
|
+
return (mod && mod.__esModule) ? mod : { "default": mod };
|
|
10
|
+
};
|
|
11
|
+
Object.defineProperty(exports, "__esModule", { value: true });
|
|
12
|
+
exports.resolveAiConfig = resolveAiConfig;
|
|
13
|
+
exports.evaluateWithAi = evaluateWithAi;
|
|
14
|
+
const axios_1 = __importDefault(require("axios"));
|
|
15
|
+
const DEFAULT_AI_CONFIG = {
|
|
16
|
+
url: 'http://localhost:11434',
|
|
17
|
+
model: 'llama3.2:3b',
|
|
18
|
+
timeout: 30_000,
|
|
19
|
+
};
|
|
20
|
+
/** Merge partial user config with defaults. */
|
|
21
|
+
function resolveAiConfig(partial) {
|
|
22
|
+
return { ...DEFAULT_AI_CONFIG, ...partial };
|
|
23
|
+
}
|
|
24
|
+
/**
|
|
25
|
+
* Evaluate a value against a natural-language prompt using a local LLM.
|
|
26
|
+
*
|
|
27
|
+
* @example
|
|
28
|
+
* ```ts
|
|
29
|
+
* const result = await evaluateWithAi(config, responseBody, 'Does this contain a valid user object?');
|
|
30
|
+
* // { pass: true, confidence: 0.95, reason: 'Response contains id, email, and name fields.' }
|
|
31
|
+
* ```
|
|
32
|
+
*/
|
|
33
|
+
async function evaluateWithAi(config, actual, prompt) {
|
|
34
|
+
const systemPrompt = [
|
|
35
|
+
'You are a test evaluator. Analyze the provided data against the given criteria.',
|
|
36
|
+
'Respond ONLY with valid JSON: {"pass": true/false, "confidence": 0.0-1.0, "reason": "brief explanation"}',
|
|
37
|
+
].join(' ');
|
|
38
|
+
const userPrompt = `Criteria: ${prompt}\n\nData:\n${JSON.stringify(actual, null, 2)}`;
|
|
39
|
+
try {
|
|
40
|
+
const { data } = await axios_1.default.post(`${config.url}/api/generate`, {
|
|
41
|
+
model: config.model,
|
|
42
|
+
prompt: userPrompt,
|
|
43
|
+
system: systemPrompt,
|
|
44
|
+
stream: false,
|
|
45
|
+
format: 'json',
|
|
46
|
+
}, { timeout: config.timeout });
|
|
47
|
+
const parsed = JSON.parse(data.response);
|
|
48
|
+
return {
|
|
49
|
+
pass: Boolean(parsed.pass),
|
|
50
|
+
confidence: Number(parsed.confidence) || 0,
|
|
51
|
+
reason: String(parsed.reason || ''),
|
|
52
|
+
};
|
|
53
|
+
}
|
|
54
|
+
catch (err) {
|
|
55
|
+
const message = err instanceof Error ? err.message : String(err);
|
|
56
|
+
return { pass: false, confidence: 0, reason: `AI evaluation failed: ${message}` };
|
|
57
|
+
}
|
|
58
|
+
}
|
|
59
|
+
//# sourceMappingURL=ai.js.map
|
package/dist/ai.js.map
ADDED
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"ai.js","sourceRoot":"","sources":["../src/ai.ts"],"names":[],"mappings":";AAAA;;;;;GAKG;;;;;AAYH,0CAEC;AAWD,wCAmCC;AA1DD,kDAA0B;AAG1B,MAAM,iBAAiB,GAAa;IAClC,GAAG,EAAE,wBAAwB;IAC7B,KAAK,EAAE,aAAa;IACpB,OAAO,EAAE,MAAM;CAChB,CAAC;AAEF,+CAA+C;AAC/C,SAAgB,eAAe,CAAC,OAA2B;IACzD,OAAO,EAAE,GAAG,iBAAiB,EAAE,GAAG,OAAO,EAAE,CAAC;AAC9C,CAAC;AAED;;;;;;;;GAQG;AACI,KAAK,UAAU,cAAc,CAClC,MAAgB,EAChB,MAAe,EACf,MAAc;IAEd,MAAM,YAAY,GAAG;QACnB,iFAAiF;QACjF,0GAA0G;KAC3G,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC;IAEZ,MAAM,UAAU,GAAG,aAAa,MAAM,cAAc,IAAI,CAAC,SAAS,CAAC,MAAM,EAAE,IAAI,EAAE,CAAC,CAAC,EAAE,CAAC;IAEtF,IAAI,CAAC;QACH,MAAM,EAAE,IAAI,EAAE,GAAG,MAAM,eAAK,CAAC,IAAI,CAC/B,GAAG,MAAM,CAAC,GAAG,eAAe,EAC5B;YACE,KAAK,EAAE,MAAM,CAAC,KAAK;YACnB,MAAM,EAAE,UAAU;YAClB,MAAM,EAAE,YAAY;YACpB,MAAM,EAAE,KAAK;YACb,MAAM,EAAE,MAAM;SACf,EACD,EAAE,OAAO,EAAE,MAAM,CAAC,OAAO,EAAE,CAC5B,CAAC;QAEF,MAAM,MAAM,GAAG,IAAI,CAAC,KAAK,CAAC,IAAI,CAAC,QAAQ,CAAC,CAAC;QACzC,OAAO;YACL,IAAI,EAAE,OAAO,CAAC,MAAM,CAAC,IAAI,CAAC;YAC1B,UAAU,EAAE,MAAM,CAAC,MAAM,CAAC,UAAU,CAAC,IAAI,CAAC;YAC1C,MAAM,EAAE,MAAM,CAAC,MAAM,CAAC,MAAM,IAAI,EAAE,CAAC;SACpC,CAAC;IACJ,CAAC;IAAC,OAAO,GAAG,EAAE,CAAC;QACb,MAAM,OAAO,GAAG,GAAG,YAAY,KAAK,CAAC,CAAC,CAAC,GAAG,CAAC,OAAO,CAAC,CAAC,CAAC,MAAM,CAAC,GAAG,CAAC,CAAC;QACjE,OAAO,EAAE,IAAI,EAAE,KAAK,EAAE,UAAU,EAAE,CAAC,EAAE,MAAM,EAAE,yBAAyB,OAAO,EAAE,EAAE,CAAC;IACpF,CAAC;AACH,CAAC"}
|
package/dist/cli.d.ts
ADDED
|
@@ -0,0 +1,11 @@
|
|
|
1
|
+
#!/usr/bin/env node
|
|
2
|
+
/**
|
|
3
|
+
* testflow-ai CLI entry point.
|
|
4
|
+
*
|
|
5
|
+
* Usage:
|
|
6
|
+
* testflow --dir ./tests --context ./context.md
|
|
7
|
+
* testflow flow1.yaml flow2.yaml -v
|
|
8
|
+
* testflow --dir ./tests --tags smoke --format json
|
|
9
|
+
*/
|
|
10
|
+
export {};
|
|
11
|
+
//# sourceMappingURL=cli.d.ts.map
|
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"cli.d.ts","sourceRoot":"","sources":["../src/cli.ts"],"names":[],"mappings":";AAEA;;;;;;;GAOG"}
|
package/dist/cli.js
ADDED
|
@@ -0,0 +1,49 @@
|
|
|
1
|
+
#!/usr/bin/env node
|
|
2
|
+
"use strict";
|
|
3
|
+
/**
|
|
4
|
+
* testflow-ai CLI entry point.
|
|
5
|
+
*
|
|
6
|
+
* Usage:
|
|
7
|
+
* testflow --dir ./tests --context ./context.md
|
|
8
|
+
* testflow flow1.yaml flow2.yaml -v
|
|
9
|
+
* testflow --dir ./tests --tags smoke --format json
|
|
10
|
+
*/
|
|
11
|
+
Object.defineProperty(exports, "__esModule", { value: true });
|
|
12
|
+
const commander_1 = require("commander");
|
|
13
|
+
const runner_js_1 = require("./runner.js");
|
|
14
|
+
const program = new commander_1.Command();
|
|
15
|
+
program
|
|
16
|
+
.name('testflow')
|
|
17
|
+
.description('Declarative API testing powered by YAML flows')
|
|
18
|
+
.version('0.1.0');
|
|
19
|
+
program
|
|
20
|
+
.argument('[files...]', 'YAML test files to run')
|
|
21
|
+
.option('-c, --context <file>', 'Path to context markdown file')
|
|
22
|
+
.option('-d, --dir <directory>', 'Directory containing YAML test files')
|
|
23
|
+
.option('-t, --tags <tags>', 'Comma-separated tags to filter')
|
|
24
|
+
.option('-f, --format <format>', 'Output: console, json, markdown', 'console')
|
|
25
|
+
.option('-v, --verbose', 'Verbose output')
|
|
26
|
+
.option('--ai-url <url>', 'Ollama URL (default: http://localhost:11434)')
|
|
27
|
+
.option('--ai-model <model>', 'Ollama model (default: llama3.2:3b)')
|
|
28
|
+
.action(async (files, opts) => {
|
|
29
|
+
try {
|
|
30
|
+
const report = await (0, runner_js_1.runTests)({
|
|
31
|
+
testFiles: files.length > 0 ? files : undefined,
|
|
32
|
+
contextFile: opts.context,
|
|
33
|
+
testDir: opts.dir,
|
|
34
|
+
tags: opts.tags?.split(','),
|
|
35
|
+
format: opts.format,
|
|
36
|
+
verbose: opts.verbose,
|
|
37
|
+
ai: opts.aiUrl || opts.aiModel
|
|
38
|
+
? { url: opts.aiUrl, model: opts.aiModel }
|
|
39
|
+
: undefined,
|
|
40
|
+
});
|
|
41
|
+
process.exit(report.failedFlows > 0 ? 1 : 0);
|
|
42
|
+
}
|
|
43
|
+
catch (err) {
|
|
44
|
+
console.error(`\nā ${err instanceof Error ? err.message : err}`);
|
|
45
|
+
process.exit(1);
|
|
46
|
+
}
|
|
47
|
+
});
|
|
48
|
+
program.parse();
|
|
49
|
+
//# sourceMappingURL=cli.js.map
|
package/dist/cli.js.map
ADDED
|
@@ -0,0 +1 @@
|
|
|
1
|
+
{"version":3,"file":"cli.js","sourceRoot":"","sources":["../src/cli.ts"],"names":[],"mappings":";;AAEA;;;;;;;GAOG;;AAEH,yCAAoC;AACpC,2CAAuC;AAEvC,MAAM,OAAO,GAAG,IAAI,mBAAO,EAAE,CAAC;AAE9B,OAAO;KACF,IAAI,CAAC,UAAU,CAAC;KAChB,WAAW,CAAC,+CAA+C,CAAC;KAC5D,OAAO,CAAC,OAAO,CAAC,CAAC;AAEtB,OAAO;KACF,QAAQ,CAAC,YAAY,EAAE,wBAAwB,CAAC;KAChD,MAAM,CAAC,sBAAsB,EAAE,+BAA+B,CAAC;KAC/D,MAAM,CAAC,uBAAuB,EAAE,sCAAsC,CAAC;KACvE,MAAM,CAAC,mBAAmB,EAAE,gCAAgC,CAAC;KAC7D,MAAM,CAAC,uBAAuB,EAAE,iCAAiC,EAAE,SAAS,CAAC;KAC7E,MAAM,CAAC,eAAe,EAAE,gBAAgB,CAAC;KACzC,MAAM,CAAC,gBAAgB,EAAE,8CAA8C,CAAC;KACxE,MAAM,CAAC,oBAAoB,EAAE,qCAAqC,CAAC;KACnE,MAAM,CAAC,KAAK,EAAE,KAAe,EAAE,IAAI,EAAE,EAAE;IACpC,IAAI,CAAC;QACD,MAAM,MAAM,GAAG,MAAM,IAAA,oBAAQ,EAAC;YAC1B,SAAS,EAAE,KAAK,CAAC,MAAM,GAAG,CAAC,CAAC,CAAC,CAAC,KAAK,CAAC,CAAC,CAAC,SAAS;YAC/C,WAAW,EAAE,IAAI,CAAC,OAAO;YACzB,OAAO,EAAE,IAAI,CAAC,GAAG;YACjB,IAAI,EAAE,IAAI,CAAC,IAAI,EAAE,KAAK,CAAC,GAAG,CAAC;YAC3B,MAAM,EAAE,IAAI,CAAC,MAAM;YACnB,OAAO,EAAE,IAAI,CAAC,OAAO;YACrB,EAAE,EAAE,IAAI,CAAC,KAAK,IAAI,IAAI,CAAC,OAAO;gBAC1B,CAAC,CAAC,EAAE,GAAG,EAAE,IAAI,CAAC,KAAK,EAAE,KAAK,EAAE,IAAI,CAAC,OAAO,EAAE;gBAC1C,CAAC,CAAC,SAAS;SAClB,CAAC,CAAC;QAEH,OAAO,CAAC,IAAI,CAAC,MAAM,CAAC,WAAW,GAAG,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC;IACjD,CAAC;IAAC,OAAO,GAAG,EAAE,CAAC;QACX,OAAO,CAAC,KAAK,CAAC,OAAO,GAAG,YAAY,KAAK,CAAC,CAAC,CAAC,GAAG,CAAC,OAAO,CAAC,CAAC,CAAC,GAAG,EAAE,CAAC,CAAC;QACjE,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC;IACpB,CAAC;AACL,CAAC,CAAC,CAAC;AAEP,OAAO,CAAC,KAAK,EAAE,CAAC"}
|