testflow-ai 0.1.1 β 0.4.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +553 -143
- package/dist/ai.d.ts +20 -6
- package/dist/ai.d.ts.map +1 -1
- package/dist/ai.js +102 -10
- package/dist/ai.js.map +1 -1
- package/dist/cli.js +11 -4
- package/dist/cli.js.map +1 -1
- package/dist/parser.d.ts.map +1 -1
- package/dist/parser.js +7 -2
- package/dist/parser.js.map +1 -1
- package/dist/types.d.ts +15 -7
- package/dist/types.d.ts.map +1 -1
- package/examples/auth-flow.yaml +52 -0
- package/examples/context.md +25 -0
- package/examples/graphql-flow.yaml +58 -0
- package/examples/rest-crud.yaml +76 -0
- package/examples/todo-crud.yaml +97 -0
- package/examples/todo-graphql.yaml +59 -0
- package/examples/todo-list-context.md +27 -0
- package/package.json +68 -67
package/README.md
CHANGED
|
@@ -1,87 +1,163 @@
|
|
|
1
|
-
|
|
1
|
+
<div align="center">
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
# π§ͺ testflow-ai
|
|
4
4
|
|
|
5
|
-
|
|
6
|
-
|
|
7
|
-
|
|
5
|
+
**YAML API flows + optional LLM assertions (local Ollama or cloud)**
|
|
6
|
+
|
|
7
|
+
*Version-controlled β’ CI/CD-ready β’ Human-readable*
|
|
8
|
+
|
|
9
|
+
[](https://www.npmjs.com/package/testflow-ai)
|
|
10
|
+
[](https://www.npmjs.com/package/testflow-ai)
|
|
11
|
+
[](https://opensource.org/licenses/MIT)
|
|
12
|
+
[](https://nodejs.org)
|
|
13
|
+
|
|
14
|
+
β
**Multi-step flows** (create β capture β reuse β assert)
|
|
15
|
+
π€ **Assert "hard" responses with AI** (privacy-first via Ollama)
|
|
16
|
+
π **Keep API context in Markdown** (great for humans + AI agents)
|
|
17
|
+
|
|
18
|
+
[π Documentation](#-documentation) β’ [π Quick Start](#-quick-start) β’ [π» Examples](#-real-world-example) β’ [π€ AI Providers](#-ai-powered-evaluation)
|
|
19
|
+
|
|
20
|
+
</div>
|
|
8
21
|
|
|
9
22
|
---
|
|
10
23
|
|
|
11
|
-
##
|
|
24
|
+
## π― What is testflow-ai?
|
|
25
|
+
|
|
26
|
+
**testflow-ai** lets you describe API scenarios in YAML files, run them from the command line or as a library, and (optionally) ask an AI model to judge complex responses. No GUI, no vendor lockβin, and it works with any HTTP/GraphQL API.
|
|
12
27
|
|
|
13
|
-
|
|
28
|
+
> **π‘ Born from real-world frustration:**
|
|
29
|
+
> After days of testing APIs with Postman and burning tokens with ChatGPT, I built this to centralize tests in version-controlled YAML files with local AI support.
|
|
30
|
+
> I wanted something that felt more like a **test agent**: a tool that could **create data, mutate it, delete it, and walk full flows endβtoβend**, but defined in plain files, close to the code, and easy to run in CI.
|
|
31
|
+
> **testflow-ai** is that tool: a thin engine that turns YAML flows into real HTTP calls, variable captures, assertions, and (if you want) AIβpowered checks.
|
|
14
32
|
|
|
15
|
-
|
|
33
|
+
---
|
|
16
34
|
|
|
17
|
-
|
|
18
|
-
- **IDE AI assistants** worked for one-off requests but burned through tokens, lost context, and couldn't maintain complex multi-step flows.
|
|
19
|
-
- **MCP servers and tooling** required significant setup and ongoing maintenance.
|
|
35
|
+
## β¨ Why it's different
|
|
20
36
|
|
|
21
|
-
|
|
37
|
+
Most API testing tools are either **GUI-first** (collections) or **code-first** (JS/TS test code).
|
|
38
|
+
**testflow-ai** is **flow-first**: readable YAML that runs in CI β with an optional AI judge when classic assertions aren't enough.
|
|
22
39
|
|
|
23
|
-
|
|
24
|
-
2. Defines multi-step flows declaratively
|
|
25
|
-
3. Captures variables between steps automatically
|
|
26
|
-
4. Supports REST, GraphQL, and async operations
|
|
27
|
-
5. Can leverage a **local AI model** for intelligent assertions
|
|
28
|
-
6. Runs in CI/CD with zero cloud dependencies
|
|
40
|
+
**What you get:**
|
|
29
41
|
|
|
30
|
-
|
|
42
|
+
- **Flow engine**: multi-step scenarios with capture + interpolation (CRUD, auth, webhooks, background jobs)
|
|
43
|
+
- **AI assertions**: validate complex text/structured responses with natural language checks (Ollama/OpenAI/Anthropic)
|
|
44
|
+
- **Context-as-docs**: a Markdown file that explains base URLs, endpoints, and rules β perfect input for AI agents too
|
|
31
45
|
|
|
32
46
|
---
|
|
33
47
|
|
|
34
|
-
##
|
|
48
|
+
## β
When to use testflow-ai
|
|
49
|
+
|
|
50
|
+
- You want **version-controlled API E2E flows** (not a GUI collection)
|
|
51
|
+
- You need **multi-step chaining** (create β capture id β update β verify)
|
|
52
|
+
- You want **CI/CD-ready output** (console/json/markdown + exit codes + no external deps)
|
|
53
|
+
- You sometimes need an **AI judge** for fuzzy checks (content quality, summaries, "is this coherent?")
|
|
54
|
+
|
|
55
|
+
## π« When NOT to use it
|
|
56
|
+
|
|
57
|
+
- You only need **schema/property-based fuzzing** from OpenAPI
|
|
58
|
+
- You prefer **writing tests in code** (Jest/Vitest) with full programmatic control
|
|
59
|
+
- You need **browser/UI testing** (Playwright/Cypress territory)
|
|
60
|
+
|
|
61
|
+
---
|
|
62
|
+
|
|
63
|
+
## π What testflow-ai optimizes for
|
|
64
|
+
|
|
65
|
+
<div align="center">
|
|
66
|
+
|
|
67
|
+
| Goal | testflow-ai |
|
|
68
|
+
|:----:|:-----------:|
|
|
69
|
+
| Human-readable flows in Git | β
|
|
|
70
|
+
| Multi-step chaining + captures | β
|
|
|
71
|
+
| CI/CD-ready (exit codes, JSON) | β
|
|
|
72
|
+
| Optional AI-based assertions | β
|
|
|
73
|
+
| GUI collections | β (not a goal) |
|
|
74
|
+
| Full code-based test suites | β (use your test framework) |
|
|
35
75
|
|
|
36
|
-
|
|
37
|
-
- π **Variable capture** β extract values from responses, reuse in later steps
|
|
38
|
-
- β
**Rich assertions** β equals, contains, exists, greaterThan, matches, and more
|
|
39
|
-
- π **GraphQL native** β first-class support for queries and mutations
|
|
40
|
-
- β³ **Async polling** β `waitUntil` for operations that take time
|
|
41
|
-
- π€ **AI evaluation** β assert with natural language via a local Ollama model
|
|
42
|
-
- π **Context files** β define base URLs, endpoints, and rules in Markdown
|
|
43
|
-
- π **Multiple formats** β console, JSON, or Markdown reports
|
|
44
|
-
- π― **Tag filtering** β run subsets of your test suite
|
|
45
|
-
- π₯οΈ **CLI + API** β use from the terminal or import as a library
|
|
76
|
+
</div>
|
|
46
77
|
|
|
47
78
|
---
|
|
48
79
|
|
|
49
|
-
|
|
80
|
+
### β¨ Key Features
|
|
50
81
|
|
|
51
|
-
|
|
82
|
+
<div align="center">
|
|
83
|
+
|
|
84
|
+
| π¨ Feature | π Description |
|
|
85
|
+
|:----------:|:-------------:|
|
|
86
|
+
| **π YAML Flows** | Define test sequences declaratively β version-controlled and human-readable |
|
|
87
|
+
| **π Variable Capture** | Extract values from responses, reuse in later steps automatically |
|
|
88
|
+
| **β
Rich Assertions** | 10+ operators: equals, contains, exists, greaterThan, matches, and more |
|
|
89
|
+
| **π GraphQL Native** | First-class support for queries and mutations |
|
|
90
|
+
| **β³ Async Polling** | `waitUntil` for operations that take time (background jobs, processing) |
|
|
91
|
+
| **π€ AI Evaluation** | Assert with natural language using Ollama, OpenAI, or Anthropic |
|
|
92
|
+
| **π Context Files** | Define base URLs, endpoints, and rules in Markdown |
|
|
93
|
+
| **π Multiple Formats** | Console (colored), JSON (CI/CD), or Markdown reports |
|
|
94
|
+
| **π― Tag Filtering** | Run subsets of your test suite (`--tags smoke,e2e`) |
|
|
95
|
+
| **π₯οΈ CLI + API** | Use from terminal (`npx testflow`) or import as a library |
|
|
96
|
+
|
|
97
|
+
</div>
|
|
98
|
+
|
|
99
|
+
---
|
|
100
|
+
|
|
101
|
+
## π Quick Start
|
|
52
102
|
|
|
53
103
|
```bash
|
|
54
|
-
npm
|
|
104
|
+
npm i -D testflow-ai
|
|
55
105
|
```
|
|
56
106
|
|
|
57
|
-
|
|
107
|
+
**Create `context.md`:**
|
|
108
|
+
|
|
109
|
+
```markdown
|
|
110
|
+
# My API
|
|
111
|
+
|
|
112
|
+
## Base URLs
|
|
113
|
+
- api: http://localhost:3000
|
|
114
|
+
```
|
|
115
|
+
|
|
116
|
+
**Create `tests/todo.yaml`:**
|
|
58
117
|
|
|
59
118
|
```yaml
|
|
60
|
-
|
|
61
|
-
name: Health Check
|
|
119
|
+
name: Todo flow
|
|
62
120
|
tags: [smoke]
|
|
121
|
+
|
|
63
122
|
steps:
|
|
64
|
-
- name:
|
|
123
|
+
- name: Create todo
|
|
65
124
|
request:
|
|
66
|
-
method:
|
|
67
|
-
url:
|
|
125
|
+
method: POST
|
|
126
|
+
url: "{api}/todos"
|
|
127
|
+
headers:
|
|
128
|
+
Content-Type: application/json
|
|
129
|
+
body:
|
|
130
|
+
title: "Buy milk"
|
|
131
|
+
completed: false
|
|
132
|
+
capture:
|
|
133
|
+
- name: todoId
|
|
134
|
+
path: data.id
|
|
68
135
|
assertions:
|
|
69
136
|
- path: status
|
|
70
137
|
operator: equals
|
|
71
|
-
value:
|
|
138
|
+
value: 201
|
|
139
|
+
|
|
140
|
+
- name: Fetch todo
|
|
141
|
+
request:
|
|
142
|
+
method: GET
|
|
143
|
+
url: "{api}/todos/${todoId}"
|
|
144
|
+
assertions:
|
|
145
|
+
- path: data.title
|
|
146
|
+
operator: equals
|
|
147
|
+
value: "Buy milk"
|
|
72
148
|
```
|
|
73
149
|
|
|
74
|
-
|
|
150
|
+
**Run:**
|
|
75
151
|
|
|
76
152
|
```bash
|
|
77
|
-
npx testflow tests/
|
|
153
|
+
npx testflow --context ./context.md tests/todo.yaml
|
|
78
154
|
```
|
|
79
155
|
|
|
80
|
-
That's it
|
|
156
|
+
**That's it.** No config files, no GUI, no account.
|
|
81
157
|
|
|
82
158
|
---
|
|
83
159
|
|
|
84
|
-
## Installation
|
|
160
|
+
## π¦ Installation
|
|
85
161
|
|
|
86
162
|
```bash
|
|
87
163
|
npm install testflow-ai
|
|
@@ -93,37 +169,40 @@ yarn add testflow-ai
|
|
|
93
169
|
|
|
94
170
|
---
|
|
95
171
|
|
|
96
|
-
## CLI Usage
|
|
172
|
+
## π₯οΈ CLI Usage
|
|
97
173
|
|
|
98
174
|
```bash
|
|
99
175
|
# Run specific files
|
|
100
|
-
testflow flow1.yaml flow2.yaml
|
|
176
|
+
npx testflow flow1.yaml flow2.yaml
|
|
101
177
|
|
|
102
178
|
# Run all YAML files in a directory
|
|
103
|
-
testflow --dir ./tests
|
|
179
|
+
npx testflow --dir ./tests
|
|
104
180
|
|
|
105
181
|
# Use a context file for base URLs
|
|
106
|
-
testflow --dir ./tests --context ./context.md
|
|
182
|
+
npx testflow --dir ./tests --context ./context.md
|
|
107
183
|
|
|
108
|
-
# Filter by tags
|
|
109
|
-
testflow --dir ./tests --tags smoke
|
|
184
|
+
# Filter by tags (run only smoke tests)
|
|
185
|
+
npx testflow --dir ./tests --tags smoke
|
|
110
186
|
|
|
111
187
|
# JSON output (for CI/CD)
|
|
112
|
-
testflow --dir ./tests --format json
|
|
188
|
+
npx testflow --dir ./tests --format json
|
|
113
189
|
|
|
114
190
|
# Markdown output (for reports)
|
|
115
|
-
testflow --dir ./tests --format markdown
|
|
191
|
+
npx testflow --dir ./tests --format markdown
|
|
116
192
|
|
|
117
|
-
# Verbose mode
|
|
118
|
-
testflow --dir ./tests -v
|
|
193
|
+
# Verbose mode (see step-by-step execution)
|
|
194
|
+
npx testflow --dir ./tests -v
|
|
119
195
|
|
|
120
|
-
# With AI evaluation
|
|
121
|
-
testflow --dir ./tests --ai-model llama3.2:3b
|
|
196
|
+
# With AI evaluation
|
|
197
|
+
npx testflow --dir ./tests --ai-provider ollama --ai-model llama3.2:3b
|
|
198
|
+
npx testflow --dir ./tests --ai-provider openai --ai-key $OPENAI_API_KEY --ai-model gpt-4o-mini
|
|
122
199
|
```
|
|
123
200
|
|
|
124
201
|
---
|
|
125
202
|
|
|
126
|
-
## Programmatic API
|
|
203
|
+
## π» Programmatic API
|
|
204
|
+
|
|
205
|
+
### Simple usage
|
|
127
206
|
|
|
128
207
|
```typescript
|
|
129
208
|
import { runTests } from 'testflow-ai';
|
|
@@ -140,6 +219,131 @@ console.log(`${report.passedFlows}/${report.totalFlows} passed`);
|
|
|
140
219
|
process.exit(report.failedFlows > 0 ? 1 : 0);
|
|
141
220
|
```
|
|
142
221
|
|
|
222
|
+
---
|
|
223
|
+
|
|
224
|
+
<details>
|
|
225
|
+
<summary><b>π» Real-World Example</b> (click to expand)</summary>
|
|
226
|
+
|
|
227
|
+
Here's a complete example using a Todo List API:
|
|
228
|
+
|
|
229
|
+
### Project Structure
|
|
230
|
+
|
|
231
|
+
```
|
|
232
|
+
my-api/
|
|
233
|
+
βββ tests/
|
|
234
|
+
β βββ index.ts # Test runner
|
|
235
|
+
β βββ context.md # API context
|
|
236
|
+
β βββ flows/
|
|
237
|
+
β βββ health-check.yaml
|
|
238
|
+
β βββ todo-crud.yaml
|
|
239
|
+
β βββ todo-complete-flow.yaml
|
|
240
|
+
βββ package.json
|
|
241
|
+
```
|
|
242
|
+
|
|
243
|
+
### Test Runner (`tests/index.ts`)
|
|
244
|
+
|
|
245
|
+
```typescript
|
|
246
|
+
import { runTests, type RunnerOptions } from 'testflow-ai';
|
|
247
|
+
import * as path from 'path';
|
|
248
|
+
|
|
249
|
+
async function main() {
|
|
250
|
+
const options: RunnerOptions = {
|
|
251
|
+
contextFile: path.join(__dirname, 'context.md'),
|
|
252
|
+
testDir: path.join(__dirname, 'flows'),
|
|
253
|
+
tags: process.argv.includes('--tags=smoke') ? ['smoke'] : undefined,
|
|
254
|
+
format: 'console',
|
|
255
|
+
verbose: false,
|
|
256
|
+
};
|
|
257
|
+
|
|
258
|
+
const report = await runTests(options);
|
|
259
|
+
process.exit(report.failedFlows > 0 ? 1 : 0);
|
|
260
|
+
}
|
|
261
|
+
|
|
262
|
+
main();
|
|
263
|
+
```
|
|
264
|
+
|
|
265
|
+
### Context File (`tests/context.md`)
|
|
266
|
+
|
|
267
|
+
```markdown
|
|
268
|
+
# Todo List API
|
|
269
|
+
|
|
270
|
+
## Description
|
|
271
|
+
A simple REST API for managing todo items.
|
|
272
|
+
|
|
273
|
+
## Base URLs
|
|
274
|
+
- api: http://localhost:3000
|
|
275
|
+
- graphql: http://localhost:3000/graphql
|
|
276
|
+
|
|
277
|
+
## Endpoints
|
|
278
|
+
- POST /todos - Create a new todo
|
|
279
|
+
- GET /todos/:id - Get todo by ID
|
|
280
|
+
- PUT /todos/:id - Update todo
|
|
281
|
+
- DELETE /todos/:id - Delete todo
|
|
282
|
+
- POST /graphql - GraphQL endpoint
|
|
283
|
+
```
|
|
284
|
+
|
|
285
|
+
### Test Flow (`tests/flows/todo-crud.yaml`)
|
|
286
|
+
|
|
287
|
+
```yaml
|
|
288
|
+
name: Todo CRUD Flow
|
|
289
|
+
tags: [todos, crud, smoke]
|
|
290
|
+
|
|
291
|
+
steps:
|
|
292
|
+
- name: Create todo
|
|
293
|
+
request:
|
|
294
|
+
method: POST
|
|
295
|
+
url: "{api}/todos"
|
|
296
|
+
headers:
|
|
297
|
+
Content-Type: application/json
|
|
298
|
+
body:
|
|
299
|
+
title: "Buy groceries"
|
|
300
|
+
completed: false
|
|
301
|
+
capture:
|
|
302
|
+
- name: todoId
|
|
303
|
+
path: data.id
|
|
304
|
+
assertions:
|
|
305
|
+
- path: status
|
|
306
|
+
operator: equals
|
|
307
|
+
value: 201
|
|
308
|
+
- path: data.title
|
|
309
|
+
operator: equals
|
|
310
|
+
value: "Buy groceries"
|
|
311
|
+
|
|
312
|
+
- name: Get todo
|
|
313
|
+
request:
|
|
314
|
+
method: GET
|
|
315
|
+
url: "{api}/todos/${todoId}"
|
|
316
|
+
assertions:
|
|
317
|
+
- path: data.id
|
|
318
|
+
operator: equals
|
|
319
|
+
value: "${todoId}"
|
|
320
|
+
|
|
321
|
+
- name: Update todo
|
|
322
|
+
request:
|
|
323
|
+
method: PUT
|
|
324
|
+
url: "{api}/todos/${todoId}"
|
|
325
|
+
headers:
|
|
326
|
+
Content-Type: application/json
|
|
327
|
+
body:
|
|
328
|
+
completed: true
|
|
329
|
+
assertions:
|
|
330
|
+
- path: data.completed
|
|
331
|
+
operator: equals
|
|
332
|
+
value: true
|
|
333
|
+
```
|
|
334
|
+
|
|
335
|
+
### Running Tests
|
|
336
|
+
|
|
337
|
+
```bash
|
|
338
|
+
# Add to package.json scripts:
|
|
339
|
+
"test:e2e": "ts-node tests/index.ts"
|
|
340
|
+
"test:smoke": "ts-node tests/index.ts --tags=smoke"
|
|
341
|
+
|
|
342
|
+
# Then run:
|
|
343
|
+
npm run test:e2e
|
|
344
|
+
npm run test:smoke
|
|
345
|
+
```
|
|
346
|
+
|
|
143
347
|
### Advanced usage
|
|
144
348
|
|
|
145
349
|
```typescript
|
|
@@ -149,7 +353,7 @@ import { TestRunner, FlowExecutor, parseYamlFile, parseContextFile } from 'testf
|
|
|
149
353
|
const runner = new TestRunner({
|
|
150
354
|
contextFile: './context.md',
|
|
151
355
|
testFiles: ['./tests/critical.yaml'],
|
|
152
|
-
ai: { model: 'mistral:7b' },
|
|
356
|
+
ai: { provider: 'ollama', model: 'mistral:7b' },
|
|
153
357
|
});
|
|
154
358
|
const report = await runner.run();
|
|
155
359
|
|
|
@@ -160,58 +364,58 @@ const executor = new FlowExecutor(context, true);
|
|
|
160
364
|
const result = await executor.executeFlow(flow);
|
|
161
365
|
```
|
|
162
366
|
|
|
367
|
+
</details>
|
|
368
|
+
|
|
163
369
|
---
|
|
164
370
|
|
|
165
|
-
|
|
371
|
+
<details>
|
|
372
|
+
<summary><b>π Test Flow Reference</b> (click to expand)</summary>
|
|
166
373
|
|
|
167
374
|
### Basic structure
|
|
168
375
|
|
|
169
376
|
```yaml
|
|
170
|
-
name: Flow
|
|
171
|
-
description:
|
|
377
|
+
name: User Registration Flow
|
|
378
|
+
description: Create and verify a new user
|
|
172
379
|
tags:
|
|
380
|
+
- users
|
|
173
381
|
- smoke
|
|
174
|
-
- e2e
|
|
175
382
|
|
|
176
383
|
steps:
|
|
177
|
-
- name:
|
|
384
|
+
- name: Create user
|
|
178
385
|
request:
|
|
179
386
|
method: POST
|
|
180
|
-
url: "{api}/
|
|
387
|
+
url: "{api}/users"
|
|
181
388
|
headers:
|
|
182
389
|
Content-Type: application/json
|
|
183
390
|
body:
|
|
184
|
-
|
|
391
|
+
email: alice@example.com
|
|
392
|
+
name: Alice
|
|
185
393
|
capture:
|
|
186
|
-
- name:
|
|
187
|
-
path: data.
|
|
394
|
+
- name: userId
|
|
395
|
+
path: data.id
|
|
188
396
|
assertions:
|
|
189
397
|
- path: status
|
|
190
398
|
operator: equals
|
|
191
399
|
value: 201
|
|
192
|
-
|
|
193
|
-
|
|
194
|
-
|
|
400
|
+
- path: data.email
|
|
401
|
+
operator: equals
|
|
402
|
+
value: alice@example.com
|
|
195
403
|
|
|
196
|
-
|
|
197
|
-
steps:
|
|
198
|
-
- name: Create resource
|
|
404
|
+
- name: Verify user
|
|
199
405
|
request:
|
|
200
|
-
method:
|
|
201
|
-
url: "{api}/
|
|
202
|
-
|
|
203
|
-
|
|
204
|
-
|
|
205
|
-
|
|
206
|
-
title: New Resource
|
|
207
|
-
active: true
|
|
406
|
+
method: GET
|
|
407
|
+
url: "{api}/users/${userId}"
|
|
408
|
+
assertions:
|
|
409
|
+
- path: data.id
|
|
410
|
+
operator: equals
|
|
411
|
+
value: "${userId}"
|
|
208
412
|
```
|
|
209
413
|
|
|
210
414
|
### GraphQL requests
|
|
211
415
|
|
|
212
416
|
```yaml
|
|
213
417
|
steps:
|
|
214
|
-
- name: Query
|
|
418
|
+
- name: Query user
|
|
215
419
|
request:
|
|
216
420
|
method: POST
|
|
217
421
|
url: "{graphql}"
|
|
@@ -227,7 +431,7 @@ steps:
|
|
|
227
431
|
variables:
|
|
228
432
|
id: "${userId}"
|
|
229
433
|
capture:
|
|
230
|
-
- name:
|
|
434
|
+
- name: userEmail
|
|
231
435
|
path: data.user.email
|
|
232
436
|
```
|
|
233
437
|
|
|
@@ -258,18 +462,19 @@ steps:
|
|
|
258
462
|
Authorization: "Bearer ${token}"
|
|
259
463
|
```
|
|
260
464
|
|
|
261
|
-
Supported
|
|
465
|
+
**Supported patterns:**
|
|
466
|
+
|
|
262
467
|
- `${variable}` β simple variable
|
|
263
468
|
- `${data.nested.field}` β nested path
|
|
264
469
|
- `${items[0].id}` β array access
|
|
265
470
|
|
|
266
|
-
###
|
|
471
|
+
### Async polling (waitUntil)
|
|
267
472
|
|
|
268
|
-
For
|
|
473
|
+
For operations that take time β polls until condition is met or timeout:
|
|
269
474
|
|
|
270
475
|
```yaml
|
|
271
476
|
steps:
|
|
272
|
-
- name: Wait for
|
|
477
|
+
- name: Wait for job completion
|
|
273
478
|
request:
|
|
274
479
|
method: GET
|
|
275
480
|
url: "{api}/jobs/${jobId}"
|
|
@@ -285,37 +490,58 @@ steps:
|
|
|
285
490
|
value: "COMPLETED"
|
|
286
491
|
```
|
|
287
492
|
|
|
493
|
+
</details>
|
|
494
|
+
|
|
288
495
|
---
|
|
289
496
|
|
|
290
|
-
|
|
497
|
+
<details>
|
|
498
|
+
<summary><b>β
Assertions</b> (click to expand)</summary>
|
|
499
|
+
|
|
500
|
+
<div align="center">
|
|
291
501
|
|
|
292
|
-
| Operator | Description | Example
|
|
293
|
-
|
|
294
|
-
| `equals` | Exact match (deep equality) | `200` |
|
|
295
|
-
| `notEquals` | Not equal | `null` |
|
|
296
|
-
| `contains` | String/array contains | `"success"` |
|
|
297
|
-
| `notContains` | Does not contain | `"error"` |
|
|
502
|
+
| Operator | Description | Example |
|
|
503
|
+
|:--------:|:-----------:|:-------:|
|
|
504
|
+
| `equals` | Exact match (deep equality) | `value: 200` |
|
|
505
|
+
| `notEquals` | Not equal | `value: null` |
|
|
506
|
+
| `contains` | String/array contains | `value: "success"` |
|
|
507
|
+
| `notContains` | Does not contain | `value: "error"` |
|
|
298
508
|
| `exists` | Not null/undefined | β |
|
|
299
509
|
| `notExists` | Is null/undefined | β |
|
|
300
|
-
| `greaterThan` | Number comparison | `0` |
|
|
301
|
-
| `lessThan` | Number comparison | `100` |
|
|
302
|
-
| `matches` | Regex match | `"^[a-z]+$"` |
|
|
303
|
-
| `ai-evaluate` | AI-powered evaluation | `"Is this
|
|
510
|
+
| `greaterThan` | Number comparison | `value: 0` |
|
|
511
|
+
| `lessThan` | Number comparison | `value: 100` |
|
|
512
|
+
| `matches` | Regex match | `value: "^[a-z]+$"` |
|
|
513
|
+
| `ai-evaluate` | AI-powered evaluation | `value: "Is this valid?"` |
|
|
304
514
|
|
|
305
|
-
|
|
515
|
+
</div>
|
|
516
|
+
|
|
517
|
+
**Special paths:**
|
|
306
518
|
|
|
307
519
|
- `status` β HTTP status code (when value is a number)
|
|
308
520
|
- `httpStatus` β always the HTTP status code
|
|
309
521
|
- `data.field` β response body field
|
|
310
522
|
- `data.items[0].id` β array access
|
|
311
523
|
|
|
524
|
+
</details>
|
|
525
|
+
|
|
312
526
|
---
|
|
313
527
|
|
|
314
|
-
## AI-Powered Evaluation
|
|
528
|
+
## π€ AI-Powered Evaluation
|
|
529
|
+
|
|
530
|
+
Use AI to assert things that are hard to express with traditional operators. **testflow-ai** supports multiple providers:
|
|
531
|
+
|
|
532
|
+
<div align="center">
|
|
315
533
|
|
|
316
|
-
|
|
534
|
+
| Provider | Type | Setup | Best For |
|
|
535
|
+
|:--------:|:----:|:-----:|:--------:|
|
|
536
|
+
| **π¦ Ollama** | Local | Free, no API key | Privacy, offline, cost-effective |
|
|
537
|
+
| **π€ OpenAI** | Cloud | API key required | High accuracy, GPT-4 |
|
|
538
|
+
| **π§ Anthropic** | Cloud | API key required | Claude models, safety-focused |
|
|
317
539
|
|
|
318
|
-
|
|
540
|
+
</div>
|
|
541
|
+
|
|
542
|
+
### π¦ Ollama (Local, Recommended)
|
|
543
|
+
|
|
544
|
+
**No cloud API keys, no data leaves your machine.**
|
|
319
545
|
|
|
320
546
|
1. **Install Ollama** β [ollama.com/download](https://ollama.com/download)
|
|
321
547
|
|
|
@@ -332,17 +558,82 @@ ollama pull llama3.2:1b
|
|
|
332
558
|
ollama pull mistral:7b
|
|
333
559
|
```
|
|
334
560
|
|
|
335
|
-
|
|
561
|
+
1. **Start Ollama** (runs on `http://localhost:11434` by default):
|
|
336
562
|
|
|
337
563
|
```bash
|
|
338
564
|
ollama serve
|
|
339
565
|
```
|
|
340
566
|
|
|
567
|
+
**Usage:**
|
|
568
|
+
|
|
569
|
+
```bash
|
|
570
|
+
# CLI
|
|
571
|
+
npx testflow --dir ./tests --ai-provider ollama --ai-model llama3.2:3b
|
|
572
|
+
|
|
573
|
+
# Programmatic
|
|
574
|
+
const report = await runTests({
|
|
575
|
+
testDir: './tests',
|
|
576
|
+
ai: {
|
|
577
|
+
provider: 'ollama',
|
|
578
|
+
url: 'http://localhost:11434',
|
|
579
|
+
model: 'llama3.2:3b',
|
|
580
|
+
},
|
|
581
|
+
});
|
|
582
|
+
```
|
|
583
|
+
|
|
584
|
+
### π€ OpenAI (Cloud)
|
|
585
|
+
|
|
586
|
+
**Requires API key from [platform.openai.com](https://platform.openai.com/api-keys)**
|
|
587
|
+
|
|
588
|
+
```bash
|
|
589
|
+
# CLI
|
|
590
|
+
npx testflow --dir ./tests \
|
|
591
|
+
--ai-provider openai \
|
|
592
|
+
--ai-key $OPENAI_API_KEY \
|
|
593
|
+
--ai-model gpt-4o-mini
|
|
594
|
+
|
|
595
|
+
# Programmatic
|
|
596
|
+
const report = await runTests({
|
|
597
|
+
testDir: './tests',
|
|
598
|
+
ai: {
|
|
599
|
+
provider: 'openai',
|
|
600
|
+
apiKey: process.env.OPENAI_API_KEY,
|
|
601
|
+
model: 'gpt-4o-mini',
|
|
602
|
+
},
|
|
603
|
+
});
|
|
604
|
+
```
|
|
605
|
+
|
|
606
|
+
**Supported models:** `gpt-4o`, `gpt-4o-mini`, `gpt-4-turbo`, `gpt-3.5-turbo`
|
|
607
|
+
|
|
608
|
+
### π§ Anthropic (Cloud)
|
|
609
|
+
|
|
610
|
+
**Requires API key from [console.anthropic.com](https://console.anthropic.com/)**
|
|
611
|
+
|
|
612
|
+
```bash
|
|
613
|
+
# CLI
|
|
614
|
+
npx testflow --dir ./tests \
|
|
615
|
+
--ai-provider anthropic \
|
|
616
|
+
--ai-key $ANTHROPIC_API_KEY \
|
|
617
|
+
--ai-model claude-3-haiku-20240307
|
|
618
|
+
|
|
619
|
+
# Programmatic
|
|
620
|
+
const report = await runTests({
|
|
621
|
+
testDir: './tests',
|
|
622
|
+
ai: {
|
|
623
|
+
provider: 'anthropic',
|
|
624
|
+
apiKey: process.env.ANTHROPIC_API_KEY,
|
|
625
|
+
model: 'claude-3-haiku-20240307',
|
|
626
|
+
},
|
|
627
|
+
});
|
|
628
|
+
```
|
|
629
|
+
|
|
630
|
+
**Supported models:** `claude-3-5-sonnet-20241022`, `claude-3-opus-20240229`, `claude-3-haiku-20240307`
|
|
631
|
+
|
|
341
632
|
### Using AI assertions
|
|
342
633
|
|
|
343
634
|
```yaml
|
|
344
635
|
steps:
|
|
345
|
-
- name: Check
|
|
636
|
+
- name: Check article quality
|
|
346
637
|
request:
|
|
347
638
|
method: GET
|
|
348
639
|
url: "{api}/articles/1"
|
|
@@ -351,45 +642,104 @@ steps:
|
|
|
351
642
|
- path: status
|
|
352
643
|
operator: equals
|
|
353
644
|
value: 200
|
|
354
|
-
# AI-powered assertion
|
|
645
|
+
# AI-powered assertion (works with any provider)
|
|
355
646
|
- path: data.content
|
|
356
647
|
operator: ai-evaluate
|
|
357
648
|
value: "Does this article contain a coherent explanation with at least two paragraphs?"
|
|
358
649
|
```
|
|
359
650
|
|
|
360
|
-
###
|
|
651
|
+
### Context file AI config
|
|
361
652
|
|
|
362
|
-
```
|
|
363
|
-
|
|
364
|
-
|
|
653
|
+
```markdown
|
|
654
|
+
## AI Configuration
|
|
655
|
+
- provider: ollama
|
|
656
|
+
- url: http://localhost:11434
|
|
657
|
+
- model: llama3.2:3b
|
|
658
|
+
|
|
659
|
+
# Or for cloud providers:
|
|
660
|
+
# provider: openai
|
|
661
|
+
# apiKey: ${OPENAI_API_KEY}
|
|
662
|
+
# model: gpt-4o-mini
|
|
365
663
|
```
|
|
366
664
|
|
|
367
|
-
|
|
665
|
+
> **π Privacy note:** Ollama runs entirely locally. OpenAI and Anthropic send data to their APIs. Choose based on your privacy requirements.
|
|
666
|
+
|
|
667
|
+
### π€ AI assertions in CI (recommended settings)
|
|
668
|
+
|
|
669
|
+
AI checks can be non-deterministic. For CI, prefer:
|
|
670
|
+
|
|
671
|
+
- **Deterministic settings** (e.g. `temperature: 0` for OpenAI/Anthropic)
|
|
672
|
+
- **Short, specific prompts** (avoid vague questions)
|
|
673
|
+
- **Stable models** (avoid preview/beta models)
|
|
674
|
+
|
|
675
|
+
Example:
|
|
368
676
|
|
|
369
677
|
```typescript
|
|
370
|
-
|
|
371
|
-
|
|
372
|
-
|
|
373
|
-
|
|
374
|
-
|
|
375
|
-
timeout: 30000,
|
|
376
|
-
},
|
|
377
|
-
});
|
|
678
|
+
ai: {
|
|
679
|
+
provider: 'openai',
|
|
680
|
+
model: 'gpt-4o-mini',
|
|
681
|
+
// OpenAI doesn't expose temperature in our API yet, but use stable models
|
|
682
|
+
}
|
|
378
683
|
```
|
|
379
684
|
|
|
380
|
-
|
|
685
|
+
---
|
|
381
686
|
|
|
382
|
-
|
|
383
|
-
|
|
384
|
-
|
|
385
|
-
-
|
|
687
|
+
<details>
|
|
688
|
+
<summary><b>π Security & secrets</b> (click to expand)</summary>
|
|
689
|
+
|
|
690
|
+
- **Avoid committing API keys.** Use environment variables (e.g. `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`).
|
|
691
|
+
- The runner **redacts** common secret fields in logs (Authorization headers, tokens, cookies) when verbose mode is enabled.
|
|
692
|
+
- Keep sensitive data out of YAML files β use environment variable interpolation or context files with `.gitignore`.
|
|
693
|
+
|
|
694
|
+
**Example:**
|
|
695
|
+
|
|
696
|
+
```yaml
|
|
697
|
+
headers:
|
|
698
|
+
Authorization: "Bearer ${API_TOKEN}" # Use env vars
|
|
386
699
|
```
|
|
387
700
|
|
|
388
|
-
|
|
701
|
+
**Best practices:**
|
|
702
|
+
|
|
703
|
+
- Store secrets in `.env` files (add to `.gitignore`)
|
|
704
|
+
- Use context files for non-sensitive config (base URLs, endpoints)
|
|
705
|
+
- Never commit API keys or tokens in YAML files
|
|
706
|
+
|
|
707
|
+
</details>
|
|
708
|
+
|
|
709
|
+
---
|
|
710
|
+
|
|
711
|
+
<details>
|
|
712
|
+
<summary><b>π§© YAML schema & autocomplete (VSCode)</b> (click to expand)</summary>
|
|
713
|
+
|
|
714
|
+
We provide a JSON Schema for `*.yaml` test flows so you get autocomplete + validation in editors.
|
|
715
|
+
|
|
716
|
+
**VSCode setup** (`.vscode/settings.json`):
|
|
717
|
+
|
|
718
|
+
```json
|
|
719
|
+
{
|
|
720
|
+
"yaml.schemas": {
|
|
721
|
+
"https://raw.githubusercontent.com/carbajalmarcos/testflow-ai/main/schemas/testflow.schema.json": [
|
|
722
|
+
"tests/**/*.yaml",
|
|
723
|
+
"**/*.testflow.yaml"
|
|
724
|
+
]
|
|
725
|
+
}
|
|
726
|
+
}
|
|
727
|
+
```
|
|
728
|
+
|
|
729
|
+
This gives you:
|
|
730
|
+
|
|
731
|
+
- β
Autocomplete for `name`, `steps`, `request`, `assertions`, etc.
|
|
732
|
+
- β
Validation for required fields and types
|
|
733
|
+
- β
Hover documentation for operators and options
|
|
734
|
+
|
|
735
|
+
> **Note:** JSON Schema coming in a future release. For now, TypeScript types provide autocomplete via `import type { TestFlow } from 'testflow-ai'`.
|
|
736
|
+
|
|
737
|
+
</details>
|
|
389
738
|
|
|
390
739
|
---
|
|
391
740
|
|
|
392
|
-
|
|
741
|
+
<details>
|
|
742
|
+
<summary><b>π Context Files</b> (click to expand)</summary>
|
|
393
743
|
|
|
394
744
|
Define your project context in Markdown. The runner uses it to resolve `{baseUrlKey}` references in your YAML flows.
|
|
395
745
|
|
|
@@ -413,19 +763,29 @@ Brief description of your API.
|
|
|
413
763
|
- Authentication required for /users
|
|
414
764
|
|
|
415
765
|
## AI Configuration
|
|
766
|
+
- provider: ollama
|
|
416
767
|
- url: http://localhost:11434
|
|
417
768
|
- model: llama3.2:3b
|
|
418
769
|
```
|
|
419
770
|
|
|
771
|
+
</details>
|
|
772
|
+
|
|
420
773
|
---
|
|
421
774
|
|
|
422
|
-
|
|
775
|
+
<details>
|
|
776
|
+
<summary><b>π CI/CD Integration</b> (click to expand)</summary>
|
|
777
|
+
|
|
778
|
+
**testflow-ai** works in any CI/CD pipeline:
|
|
779
|
+
|
|
780
|
+
- Exit code `0` = success, `1` = failure (CI will fail automatically)
|
|
781
|
+
- JSON output: `--format json` for parsing results
|
|
782
|
+
- Tag filtering: `--tags smoke` for faster runs
|
|
423
783
|
|
|
424
784
|
### GitHub Actions
|
|
425
785
|
|
|
426
786
|
```yaml
|
|
427
787
|
jobs:
|
|
428
|
-
|
|
788
|
+
test:
|
|
429
789
|
runs-on: ubuntu-latest
|
|
430
790
|
steps:
|
|
431
791
|
- uses: actions/checkout@v4
|
|
@@ -433,24 +793,21 @@ jobs:
|
|
|
433
793
|
with:
|
|
434
794
|
node-version: '20'
|
|
435
795
|
- run: npm ci
|
|
796
|
+
- run: npm install -D testflow-ai
|
|
436
797
|
- run: npm run start:server &
|
|
437
|
-
- run: npx testflow --dir ./tests --context ./context.md
|
|
438
|
-
- uses: actions/upload-artifact@v4
|
|
439
|
-
with:
|
|
440
|
-
name: test-results
|
|
441
|
-
path: results.json
|
|
798
|
+
- run: npx testflow --dir ./tests --context ./context.md
|
|
442
799
|
```
|
|
443
800
|
|
|
444
|
-
|
|
801
|
+
That's it. If tests fail, the job fails automatically (exit code 1).
|
|
445
802
|
|
|
446
|
-
|
|
447
|
-
- `1` β one or more flows failed
|
|
803
|
+
</details>
|
|
448
804
|
|
|
449
805
|
---
|
|
450
806
|
|
|
451
|
-
|
|
807
|
+
<details>
|
|
808
|
+
<summary><b>π Output Examples</b> (click to expand)</summary>
|
|
452
809
|
|
|
453
|
-
### Console
|
|
810
|
+
### Console Output
|
|
454
811
|
|
|
455
812
|
```
|
|
456
813
|
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
|
|
@@ -458,10 +815,10 @@ jobs:
|
|
|
458
815
|
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
|
|
459
816
|
|
|
460
817
|
Summary:
|
|
461
|
-
Total:
|
|
462
|
-
Passed:
|
|
463
|
-
Failed:
|
|
464
|
-
Duration:
|
|
818
|
+
Total: 5 flows
|
|
819
|
+
Passed: 5
|
|
820
|
+
Failed: 0
|
|
821
|
+
Duration: 2450ms
|
|
465
822
|
|
|
466
823
|
Narrative:
|
|
467
824
|
|
|
@@ -483,9 +840,12 @@ Narrative:
|
|
|
483
840
|
ββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββ
|
|
484
841
|
```
|
|
485
842
|
|
|
843
|
+
</details>
|
|
844
|
+
|
|
486
845
|
---
|
|
487
846
|
|
|
488
|
-
|
|
847
|
+
<details>
|
|
848
|
+
<summary><b>πΊοΈ Roadmap</b> (click to expand)</summary>
|
|
489
849
|
|
|
490
850
|
- [ ] Database assertions (verify records directly via SQL)
|
|
491
851
|
- [ ] gRPC / RPC support
|
|
@@ -495,8 +855,58 @@ Narrative:
|
|
|
495
855
|
- [ ] HTML report output
|
|
496
856
|
- [ ] `testflow init` wizard
|
|
497
857
|
|
|
858
|
+
</details>
|
|
859
|
+
|
|
860
|
+
---
|
|
861
|
+
|
|
862
|
+
## π Examples
|
|
863
|
+
|
|
864
|
+
See the [`examples/`](./examples) directory for:
|
|
865
|
+
|
|
866
|
+
- **Todo List CRUD** (`todo-crud.yaml`) - Complete REST CRUD flow
|
|
867
|
+
- **Todo GraphQL** (`todo-graphql.yaml`) - GraphQL mutations and queries
|
|
868
|
+
- **REST CRUD** (`rest-crud.yaml`) - User management example
|
|
869
|
+
- **GraphQL Flow** (`graphql-flow.yaml`) - GraphQL with variable capture
|
|
870
|
+
- **Auth Flow** (`auth-flow.yaml`) - Authentication and protected routes
|
|
871
|
+
- **Context Files** (`context.md`, `todo-list-context.md`) - API context templates
|
|
872
|
+
|
|
873
|
+
**Quick start with examples:**
|
|
874
|
+
|
|
875
|
+
```bash
|
|
876
|
+
# Run specific example
|
|
877
|
+
npx testflow --context ./examples/todo-list-context.md ./examples/todo-crud.yaml
|
|
878
|
+
|
|
879
|
+
# Run all examples in directory
|
|
880
|
+
npx testflow --dir ./examples --context ./examples/context.md
|
|
881
|
+
```
|
|
882
|
+
|
|
498
883
|
---
|
|
499
884
|
|
|
500
|
-
## License
|
|
885
|
+
## π License
|
|
501
886
|
|
|
502
887
|
MIT
|
|
888
|
+
|
|
889
|
+
---
|
|
890
|
+
|
|
891
|
+
<div align="center">
|
|
892
|
+
|
|
893
|
+
**Made with β€οΈ by [Marcos Carbajal](https://github.com/carbajalmarcos)**
|
|
894
|
+
|
|
895
|
+
[β Star on GitHub](https://github.com/carbajalmarcos/testflow-ai) β’ [π¦ npm](https://www.npmjs.com/package/testflow-ai) β’ [π Report Bug](https://github.com/carbajalmarcos/testflow-ai/issues) β’ [π¬ Discussions](https://github.com/carbajalmarcos/testflow-ai/discussions)
|
|
896
|
+
|
|
897
|
+
---
|
|
898
|
+
|
|
899
|
+
### β Support this project
|
|
900
|
+
|
|
901
|
+
If you find **testflow-ai** useful, consider supporting its development:
|
|
902
|
+
|
|
903
|
+
[](https://buymeacoffee.com/carbajalmarcos)
|
|
904
|
+
|
|
905
|
+
**Crypto donations:**
|
|
906
|
+
|
|
907
|
+
- **Bitcoin (BTC):** `bc1qv0ddjg3wcgujk9ad66v9msz8manu5tanhvq0fn`
|
|
908
|
+
- **ERC-20 USDT:** `0x79F57C9D45d2D40420EF071DDAaA27057618E7C8`
|
|
909
|
+
|
|
910
|
+
*Every contribution helps make this project better!*
|
|
911
|
+
|
|
912
|
+
</div>
|