testflow-ai 0.1.1 β†’ 0.4.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -1,87 +1,163 @@
1
- # testflow-ai
1
+ <div align="center">
2
2
 
3
- > Declarative API testing powered by YAML flows. Version-controlled, human-readable, AI-friendly.
3
+ # πŸ§ͺ testflow-ai
4
4
 
5
- [![npm version](https://img.shields.io/npm/v/testflow-ai.svg)](https://www.npmjs.com/package/testflow-ai)
6
- [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
7
- [![Node.js](https://img.shields.io/badge/node-%3E%3D18-green.svg)](https://nodejs.org)
5
+ **YAML API flows + optional LLM assertions (local Ollama or cloud)**
6
+
7
+ *Version-controlled β€’ CI/CD-ready β€’ Human-readable*
8
+
9
+ [![npm version](https://img.shields.io/npm/v/testflow-ai.svg?style=for-the-badge&color=blue)](https://www.npmjs.com/package/testflow-ai)
10
+ [![npm downloads](https://img.shields.io/npm/dm/testflow-ai.svg?style=for-the-badge&color=green)](https://www.npmjs.com/package/testflow-ai)
11
+ [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg?style=for-the-badge)](https://opensource.org/licenses/MIT)
12
+ [![Node.js](https://img.shields.io/badge/node-%3E%3D18-green.svg?style=for-the-badge)](https://nodejs.org)
13
+
14
+ βœ… **Multi-step flows** (create β†’ capture β†’ reuse β†’ assert)
15
+ πŸ€– **Assert "hard" responses with AI** (privacy-first via Ollama)
16
+ πŸ“„ **Keep API context in Markdown** (great for humans + AI agents)
17
+
18
+ [πŸ“– Documentation](#-documentation) β€’ [πŸš€ Quick Start](#-quick-start) β€’ [πŸ’» Examples](#-real-world-example) β€’ [πŸ€– AI Providers](#-ai-powered-evaluation)
19
+
20
+ </div>
8
21
 
9
22
  ---
10
23
 
11
- ## Why?
24
+ ## 🎯 What is testflow-ai?
25
+
26
+ **testflow-ai** lets you describe API scenarios in YAML files, run them from the command line or as a library, and (optionally) ask an AI model to judge complex responses. No GUI, no vendor lock‑in, and it works with any HTTP/GraphQL API.
12
27
 
13
- I was building a backend that started as a simple API and grew into a system with GraphQL, async workers, state machines, and AI-powered evaluations.
28
+ > **πŸ’‘ Born from real-world frustration:**
29
+ > After days of testing APIs with Postman and burning tokens with ChatGPT, I built this to centralize tests in version-controlled YAML files with local AI support.
30
+ > I wanted something that felt more like a **test agent**: a tool that could **create data, mutate it, delete it, and walk full flows end‑to‑end**, but defined in plain files, close to the code, and easy to run in CI.
31
+ > **testflow-ai** is that tool: a thin engine that turns YAML flows into real HTTP calls, variable captures, assertions, and (if you want) AI‑powered checks.
14
32
 
15
- Testing started simple β€” a few requests in Postman. Then the project scaled:
33
+ ---
16
34
 
17
- - **Postman / Insomnia** became unmanageable. Dozens of collections, manual token copying, no version control.
18
- - **IDE AI assistants** worked for one-off requests but burned through tokens, lost context, and couldn't maintain complex multi-step flows.
19
- - **MCP servers and tooling** required significant setup and ongoing maintenance.
35
+ ## ✨ Why it's different
20
36
 
21
- I needed something that:
37
+ Most API testing tools are either **GUI-first** (collections) or **code-first** (JS/TS test code).
38
+ **testflow-ai** is **flow-first**: readable YAML that runs in CI β€” with an optional AI judge when classic assertions aren't enough.
22
39
 
23
- 1. Lives in the repo alongside my code
24
- 2. Defines multi-step flows declaratively
25
- 3. Captures variables between steps automatically
26
- 4. Supports REST, GraphQL, and async operations
27
- 5. Can leverage a **local AI model** for intelligent assertions
28
- 6. Runs in CI/CD with zero cloud dependencies
40
+ **What you get:**
29
41
 
30
- **testflow-ai** is the result.
42
+ - **Flow engine**: multi-step scenarios with capture + interpolation (CRUD, auth, webhooks, background jobs)
43
+ - **AI assertions**: validate complex text/structured responses with natural language checks (Ollama/OpenAI/Anthropic)
44
+ - **Context-as-docs**: a Markdown file that explains base URLs, endpoints, and rules β€” perfect input for AI agents too
31
45
 
32
46
  ---
33
47
 
34
- ## Features
48
+ ## βœ… When to use testflow-ai
49
+
50
+ - You want **version-controlled API E2E flows** (not a GUI collection)
51
+ - You need **multi-step chaining** (create β†’ capture id β†’ update β†’ verify)
52
+ - You want **CI/CD-ready output** (console/json/markdown + exit codes + no external deps)
53
+ - You sometimes need an **AI judge** for fuzzy checks (content quality, summaries, "is this coherent?")
54
+
55
+ ## 🚫 When NOT to use it
56
+
57
+ - You only need **schema/property-based fuzzing** from OpenAPI
58
+ - You prefer **writing tests in code** (Jest/Vitest) with full programmatic control
59
+ - You need **browser/UI testing** (Playwright/Cypress territory)
60
+
61
+ ---
62
+
63
+ ## πŸ†š What testflow-ai optimizes for
64
+
65
+ <div align="center">
66
+
67
+ | Goal | testflow-ai |
68
+ |:----:|:-----------:|
69
+ | Human-readable flows in Git | βœ… |
70
+ | Multi-step chaining + captures | βœ… |
71
+ | CI/CD-ready (exit codes, JSON) | βœ… |
72
+ | Optional AI-based assertions | βœ… |
73
+ | GUI collections | ❌ (not a goal) |
74
+ | Full code-based test suites | ❌ (use your test framework) |
35
75
 
36
- - πŸ“ **YAML test flows** β€” define test sequences declaratively
37
- - πŸ”— **Variable capture** β€” extract values from responses, reuse in later steps
38
- - βœ… **Rich assertions** β€” equals, contains, exists, greaterThan, matches, and more
39
- - πŸ”„ **GraphQL native** β€” first-class support for queries and mutations
40
- - ⏳ **Async polling** β€” `waitUntil` for operations that take time
41
- - πŸ€– **AI evaluation** β€” assert with natural language via a local Ollama model
42
- - πŸ“„ **Context files** β€” define base URLs, endpoints, and rules in Markdown
43
- - πŸ“Š **Multiple formats** β€” console, JSON, or Markdown reports
44
- - 🎯 **Tag filtering** β€” run subsets of your test suite
45
- - πŸ–₯️ **CLI + API** β€” use from the terminal or import as a library
76
+ </div>
46
77
 
47
78
  ---
48
79
 
49
- ## Quick Start
80
+ ### ✨ Key Features
50
81
 
51
- ### 1. Install
82
+ <div align="center">
83
+
84
+ | 🎨 Feature | πŸ“ Description |
85
+ |:----------:|:-------------:|
86
+ | **πŸ“ YAML Flows** | Define test sequences declaratively β€” version-controlled and human-readable |
87
+ | **πŸ”— Variable Capture** | Extract values from responses, reuse in later steps automatically |
88
+ | **βœ… Rich Assertions** | 10+ operators: equals, contains, exists, greaterThan, matches, and more |
89
+ | **πŸ”„ GraphQL Native** | First-class support for queries and mutations |
90
+ | **⏳ Async Polling** | `waitUntil` for operations that take time (background jobs, processing) |
91
+ | **πŸ€– AI Evaluation** | Assert with natural language using Ollama, OpenAI, or Anthropic |
92
+ | **πŸ“„ Context Files** | Define base URLs, endpoints, and rules in Markdown |
93
+ | **πŸ“Š Multiple Formats** | Console (colored), JSON (CI/CD), or Markdown reports |
94
+ | **🎯 Tag Filtering** | Run subsets of your test suite (`--tags smoke,e2e`) |
95
+ | **πŸ–₯️ CLI + API** | Use from terminal (`npx testflow`) or import as a library |
96
+
97
+ </div>
98
+
99
+ ---
100
+
101
+ ## πŸš€ Quick Start
52
102
 
53
103
  ```bash
54
- npm install testflow-ai
104
+ npm i -D testflow-ai
55
105
  ```
56
106
 
57
- ### 2. Create a test flow
107
+ **Create `context.md`:**
108
+
109
+ ```markdown
110
+ # My API
111
+
112
+ ## Base URLs
113
+ - api: http://localhost:3000
114
+ ```
115
+
116
+ **Create `tests/todo.yaml`:**
58
117
 
59
118
  ```yaml
60
- # tests/health.yaml
61
- name: Health Check
119
+ name: Todo flow
62
120
  tags: [smoke]
121
+
63
122
  steps:
64
- - name: Ping API
123
+ - name: Create todo
65
124
  request:
66
- method: GET
67
- url: http://localhost:3000/health
125
+ method: POST
126
+ url: "{api}/todos"
127
+ headers:
128
+ Content-Type: application/json
129
+ body:
130
+ title: "Buy milk"
131
+ completed: false
132
+ capture:
133
+ - name: todoId
134
+ path: data.id
68
135
  assertions:
69
136
  - path: status
70
137
  operator: equals
71
- value: 200
138
+ value: 201
139
+
140
+ - name: Fetch todo
141
+ request:
142
+ method: GET
143
+ url: "{api}/todos/${todoId}"
144
+ assertions:
145
+ - path: data.title
146
+ operator: equals
147
+ value: "Buy milk"
72
148
  ```
73
149
 
74
- ### 3. Run
150
+ **Run:**
75
151
 
76
152
  ```bash
77
- npx testflow tests/health.yaml
153
+ npx testflow --context ./context.md tests/todo.yaml
78
154
  ```
79
155
 
80
- That's it. No config files, no GUI, no account.
156
+ **That's it.** No config files, no GUI, no account.
81
157
 
82
158
  ---
83
159
 
84
- ## Installation
160
+ ## πŸ“¦ Installation
85
161
 
86
162
  ```bash
87
163
  npm install testflow-ai
@@ -93,37 +169,40 @@ yarn add testflow-ai
93
169
 
94
170
  ---
95
171
 
96
- ## CLI Usage
172
+ ## πŸ–₯️ CLI Usage
97
173
 
98
174
  ```bash
99
175
  # Run specific files
100
- testflow flow1.yaml flow2.yaml
176
+ npx testflow flow1.yaml flow2.yaml
101
177
 
102
178
  # Run all YAML files in a directory
103
- testflow --dir ./tests
179
+ npx testflow --dir ./tests
104
180
 
105
181
  # Use a context file for base URLs
106
- testflow --dir ./tests --context ./context.md
182
+ npx testflow --dir ./tests --context ./context.md
107
183
 
108
- # Filter by tags
109
- testflow --dir ./tests --tags smoke,auth
184
+ # Filter by tags (run only smoke tests)
185
+ npx testflow --dir ./tests --tags smoke
110
186
 
111
187
  # JSON output (for CI/CD)
112
- testflow --dir ./tests --format json
188
+ npx testflow --dir ./tests --format json
113
189
 
114
190
  # Markdown output (for reports)
115
- testflow --dir ./tests --format markdown
191
+ npx testflow --dir ./tests --format markdown
116
192
 
117
- # Verbose mode
118
- testflow --dir ./tests -v
193
+ # Verbose mode (see step-by-step execution)
194
+ npx testflow --dir ./tests -v
119
195
 
120
- # With AI evaluation (Ollama)
121
- testflow --dir ./tests --ai-model llama3.2:3b
196
+ # With AI evaluation
197
+ npx testflow --dir ./tests --ai-provider ollama --ai-model llama3.2:3b
198
+ npx testflow --dir ./tests --ai-provider openai --ai-key $OPENAI_API_KEY --ai-model gpt-4o-mini
122
199
  ```
123
200
 
124
201
  ---
125
202
 
126
- ## Programmatic API
203
+ ## πŸ’» Programmatic API
204
+
205
+ ### Simple usage
127
206
 
128
207
  ```typescript
129
208
  import { runTests } from 'testflow-ai';
@@ -140,6 +219,131 @@ console.log(`${report.passedFlows}/${report.totalFlows} passed`);
140
219
  process.exit(report.failedFlows > 0 ? 1 : 0);
141
220
  ```
142
221
 
222
+ ---
223
+
224
+ <details>
225
+ <summary><b>πŸ’» Real-World Example</b> (click to expand)</summary>
226
+
227
+ Here's a complete example using a Todo List API:
228
+
229
+ ### Project Structure
230
+
231
+ ```
232
+ my-api/
233
+ β”œβ”€β”€ tests/
234
+ β”‚ β”œβ”€β”€ index.ts # Test runner
235
+ β”‚ β”œβ”€β”€ context.md # API context
236
+ β”‚ └── flows/
237
+ β”‚ β”œβ”€β”€ health-check.yaml
238
+ β”‚ β”œβ”€β”€ todo-crud.yaml
239
+ β”‚ └── todo-complete-flow.yaml
240
+ └── package.json
241
+ ```
242
+
243
+ ### Test Runner (`tests/index.ts`)
244
+
245
+ ```typescript
246
+ import { runTests, type RunnerOptions } from 'testflow-ai';
247
+ import * as path from 'path';
248
+
249
+ async function main() {
250
+ const options: RunnerOptions = {
251
+ contextFile: path.join(__dirname, 'context.md'),
252
+ testDir: path.join(__dirname, 'flows'),
253
+ tags: process.argv.includes('--tags=smoke') ? ['smoke'] : undefined,
254
+ format: 'console',
255
+ verbose: false,
256
+ };
257
+
258
+ const report = await runTests(options);
259
+ process.exit(report.failedFlows > 0 ? 1 : 0);
260
+ }
261
+
262
+ main();
263
+ ```
264
+
265
+ ### Context File (`tests/context.md`)
266
+
267
+ ```markdown
268
+ # Todo List API
269
+
270
+ ## Description
271
+ A simple REST API for managing todo items.
272
+
273
+ ## Base URLs
274
+ - api: http://localhost:3000
275
+ - graphql: http://localhost:3000/graphql
276
+
277
+ ## Endpoints
278
+ - POST /todos - Create a new todo
279
+ - GET /todos/:id - Get todo by ID
280
+ - PUT /todos/:id - Update todo
281
+ - DELETE /todos/:id - Delete todo
282
+ - POST /graphql - GraphQL endpoint
283
+ ```
284
+
285
+ ### Test Flow (`tests/flows/todo-crud.yaml`)
286
+
287
+ ```yaml
288
+ name: Todo CRUD Flow
289
+ tags: [todos, crud, smoke]
290
+
291
+ steps:
292
+ - name: Create todo
293
+ request:
294
+ method: POST
295
+ url: "{api}/todos"
296
+ headers:
297
+ Content-Type: application/json
298
+ body:
299
+ title: "Buy groceries"
300
+ completed: false
301
+ capture:
302
+ - name: todoId
303
+ path: data.id
304
+ assertions:
305
+ - path: status
306
+ operator: equals
307
+ value: 201
308
+ - path: data.title
309
+ operator: equals
310
+ value: "Buy groceries"
311
+
312
+ - name: Get todo
313
+ request:
314
+ method: GET
315
+ url: "{api}/todos/${todoId}"
316
+ assertions:
317
+ - path: data.id
318
+ operator: equals
319
+ value: "${todoId}"
320
+
321
+ - name: Update todo
322
+ request:
323
+ method: PUT
324
+ url: "{api}/todos/${todoId}"
325
+ headers:
326
+ Content-Type: application/json
327
+ body:
328
+ completed: true
329
+ assertions:
330
+ - path: data.completed
331
+ operator: equals
332
+ value: true
333
+ ```
334
+
335
+ ### Running Tests
336
+
337
+ ```bash
338
+ # Add to package.json scripts:
339
+ "test:e2e": "ts-node tests/index.ts"
340
+ "test:smoke": "ts-node tests/index.ts --tags=smoke"
341
+
342
+ # Then run:
343
+ npm run test:e2e
344
+ npm run test:smoke
345
+ ```
346
+
143
347
  ### Advanced usage
144
348
 
145
349
  ```typescript
@@ -149,7 +353,7 @@ import { TestRunner, FlowExecutor, parseYamlFile, parseContextFile } from 'testf
149
353
  const runner = new TestRunner({
150
354
  contextFile: './context.md',
151
355
  testFiles: ['./tests/critical.yaml'],
152
- ai: { model: 'mistral:7b' },
356
+ ai: { provider: 'ollama', model: 'mistral:7b' },
153
357
  });
154
358
  const report = await runner.run();
155
359
 
@@ -160,58 +364,58 @@ const executor = new FlowExecutor(context, true);
160
364
  const result = await executor.executeFlow(flow);
161
365
  ```
162
366
 
367
+ </details>
368
+
163
369
  ---
164
370
 
165
- ## Test Flow Reference
371
+ <details>
372
+ <summary><b>πŸ“ Test Flow Reference</b> (click to expand)</summary>
166
373
 
167
374
  ### Basic structure
168
375
 
169
376
  ```yaml
170
- name: Flow Name
171
- description: What this flow tests
377
+ name: User Registration Flow
378
+ description: Create and verify a new user
172
379
  tags:
380
+ - users
173
381
  - smoke
174
- - e2e
175
382
 
176
383
  steps:
177
- - name: Step Name
384
+ - name: Create user
178
385
  request:
179
386
  method: POST
180
- url: "{api}/endpoint"
387
+ url: "{api}/users"
181
388
  headers:
182
389
  Content-Type: application/json
183
390
  body:
184
- key: value
391
+ email: alice@example.com
392
+ name: Alice
185
393
  capture:
186
- - name: variableName
187
- path: data.field
394
+ - name: userId
395
+ path: data.id
188
396
  assertions:
189
397
  - path: status
190
398
  operator: equals
191
399
  value: 201
192
- ```
193
-
194
- ### REST requests
400
+ - path: data.email
401
+ operator: equals
402
+ value: alice@example.com
195
403
 
196
- ```yaml
197
- steps:
198
- - name: Create resource
404
+ - name: Verify user
199
405
  request:
200
- method: POST
201
- url: "{api}/resources"
202
- headers:
203
- Content-Type: application/json
204
- Authorization: "Bearer ${token}"
205
- body:
206
- title: New Resource
207
- active: true
406
+ method: GET
407
+ url: "{api}/users/${userId}"
408
+ assertions:
409
+ - path: data.id
410
+ operator: equals
411
+ value: "${userId}"
208
412
  ```
209
413
 
210
414
  ### GraphQL requests
211
415
 
212
416
  ```yaml
213
417
  steps:
214
- - name: Query users
418
+ - name: Query user
215
419
  request:
216
420
  method: POST
217
421
  url: "{graphql}"
@@ -227,7 +431,7 @@ steps:
227
431
  variables:
228
432
  id: "${userId}"
229
433
  capture:
230
- - name: email
434
+ - name: userEmail
231
435
  path: data.user.email
232
436
  ```
233
437
 
@@ -258,18 +462,19 @@ steps:
258
462
  Authorization: "Bearer ${token}"
259
463
  ```
260
464
 
261
- Supported interpolation patterns:
465
+ **Supported patterns:**
466
+
262
467
  - `${variable}` β€” simple variable
263
468
  - `${data.nested.field}` β€” nested path
264
469
  - `${items[0].id}` β€” array access
265
470
 
266
- ### Polling (waitUntil)
471
+ ### Async polling (waitUntil)
267
472
 
268
- For async operations β€” polls until a condition is met or timeout:
473
+ For operations that take time β€” polls until condition is met or timeout:
269
474
 
270
475
  ```yaml
271
476
  steps:
272
- - name: Wait for processing
477
+ - name: Wait for job completion
273
478
  request:
274
479
  method: GET
275
480
  url: "{api}/jobs/${jobId}"
@@ -285,37 +490,58 @@ steps:
285
490
  value: "COMPLETED"
286
491
  ```
287
492
 
493
+ </details>
494
+
288
495
  ---
289
496
 
290
- ## Assertions
497
+ <details>
498
+ <summary><b>βœ… Assertions</b> (click to expand)</summary>
499
+
500
+ <div align="center">
291
501
 
292
- | Operator | Description | Example value |
293
- |----------|-------------|---------------|
294
- | `equals` | Exact match (deep equality) | `200` |
295
- | `notEquals` | Not equal | `null` |
296
- | `contains` | String/array contains | `"success"` |
297
- | `notContains` | Does not contain | `"error"` |
502
+ | Operator | Description | Example |
503
+ |:--------:|:-----------:|:-------:|
504
+ | `equals` | Exact match (deep equality) | `value: 200` |
505
+ | `notEquals` | Not equal | `value: null` |
506
+ | `contains` | String/array contains | `value: "success"` |
507
+ | `notContains` | Does not contain | `value: "error"` |
298
508
  | `exists` | Not null/undefined | β€” |
299
509
  | `notExists` | Is null/undefined | β€” |
300
- | `greaterThan` | Number comparison | `0` |
301
- | `lessThan` | Number comparison | `100` |
302
- | `matches` | Regex match | `"^[a-z]+$"` |
303
- | `ai-evaluate` | AI-powered evaluation | `"Is this a valid user?"` |
510
+ | `greaterThan` | Number comparison | `value: 0` |
511
+ | `lessThan` | Number comparison | `value: 100` |
512
+ | `matches` | Regex match | `value: "^[a-z]+$"` |
513
+ | `ai-evaluate` | AI-powered evaluation | `value: "Is this valid?"` |
304
514
 
305
- ### Special paths
515
+ </div>
516
+
517
+ **Special paths:**
306
518
 
307
519
  - `status` β€” HTTP status code (when value is a number)
308
520
  - `httpStatus` β€” always the HTTP status code
309
521
  - `data.field` β€” response body field
310
522
  - `data.items[0].id` β€” array access
311
523
 
524
+ </details>
525
+
312
526
  ---
313
527
 
314
- ## AI-Powered Evaluation
528
+ ## πŸ€– AI-Powered Evaluation
529
+
530
+ Use AI to assert things that are hard to express with traditional operators. **testflow-ai** supports multiple providers:
531
+
532
+ <div align="center">
315
533
 
316
- Use a local LLM to assert things that are hard to express with traditional operators.
534
+ | Provider | Type | Setup | Best For |
535
+ |:--------:|:----:|:-----:|:--------:|
536
+ | **πŸ¦™ Ollama** | Local | Free, no API key | Privacy, offline, cost-effective |
537
+ | **πŸ€– OpenAI** | Cloud | API key required | High accuracy, GPT-4 |
538
+ | **🧠 Anthropic** | Cloud | API key required | Claude models, safety-focused |
317
539
 
318
- ### Setup Ollama
540
+ </div>
541
+
542
+ ### πŸ¦™ Ollama (Local, Recommended)
543
+
544
+ **No cloud API keys, no data leaves your machine.**
319
545
 
320
546
  1. **Install Ollama** β€” [ollama.com/download](https://ollama.com/download)
321
547
 
@@ -332,17 +558,82 @@ ollama pull llama3.2:1b
332
558
  ollama pull mistral:7b
333
559
  ```
334
560
 
335
- 3. **Start Ollama** (runs on `http://localhost:11434` by default):
561
+ 1. **Start Ollama** (runs on `http://localhost:11434` by default):
336
562
 
337
563
  ```bash
338
564
  ollama serve
339
565
  ```
340
566
 
567
+ **Usage:**
568
+
569
+ ```bash
570
+ # CLI
571
+ npx testflow --dir ./tests --ai-provider ollama --ai-model llama3.2:3b
572
+
573
+ # Programmatic
574
+ const report = await runTests({
575
+ testDir: './tests',
576
+ ai: {
577
+ provider: 'ollama',
578
+ url: 'http://localhost:11434',
579
+ model: 'llama3.2:3b',
580
+ },
581
+ });
582
+ ```
583
+
584
+ ### πŸ€– OpenAI (Cloud)
585
+
586
+ **Requires API key from [platform.openai.com](https://platform.openai.com/api-keys)**
587
+
588
+ ```bash
589
+ # CLI
590
+ npx testflow --dir ./tests \
591
+ --ai-provider openai \
592
+ --ai-key $OPENAI_API_KEY \
593
+ --ai-model gpt-4o-mini
594
+
595
+ # Programmatic
596
+ const report = await runTests({
597
+ testDir: './tests',
598
+ ai: {
599
+ provider: 'openai',
600
+ apiKey: process.env.OPENAI_API_KEY,
601
+ model: 'gpt-4o-mini',
602
+ },
603
+ });
604
+ ```
605
+
606
+ **Supported models:** `gpt-4o`, `gpt-4o-mini`, `gpt-4-turbo`, `gpt-3.5-turbo`
607
+
608
+ ### 🧠 Anthropic (Cloud)
609
+
610
+ **Requires API key from [console.anthropic.com](https://console.anthropic.com/)**
611
+
612
+ ```bash
613
+ # CLI
614
+ npx testflow --dir ./tests \
615
+ --ai-provider anthropic \
616
+ --ai-key $ANTHROPIC_API_KEY \
617
+ --ai-model claude-3-haiku-20240307
618
+
619
+ # Programmatic
620
+ const report = await runTests({
621
+ testDir: './tests',
622
+ ai: {
623
+ provider: 'anthropic',
624
+ apiKey: process.env.ANTHROPIC_API_KEY,
625
+ model: 'claude-3-haiku-20240307',
626
+ },
627
+ });
628
+ ```
629
+
630
+ **Supported models:** `claude-3-5-sonnet-20241022`, `claude-3-opus-20240229`, `claude-3-haiku-20240307`
631
+
341
632
  ### Using AI assertions
342
633
 
343
634
  ```yaml
344
635
  steps:
345
- - name: Check response quality
636
+ - name: Check article quality
346
637
  request:
347
638
  method: GET
348
639
  url: "{api}/articles/1"
@@ -351,45 +642,104 @@ steps:
351
642
  - path: status
352
643
  operator: equals
353
644
  value: 200
354
- # AI-powered assertion
645
+ # AI-powered assertion (works with any provider)
355
646
  - path: data.content
356
647
  operator: ai-evaluate
357
648
  value: "Does this article contain a coherent explanation with at least two paragraphs?"
358
649
  ```
359
650
 
360
- ### CLI with AI
651
+ ### Context file AI config
361
652
 
362
- ```bash
363
- testflow --dir ./tests --ai-model llama3.2:3b
364
- testflow --dir ./tests --ai-url http://192.168.1.10:11434 --ai-model mistral:7b
653
+ ```markdown
654
+ ## AI Configuration
655
+ - provider: ollama
656
+ - url: http://localhost:11434
657
+ - model: llama3.2:3b
658
+
659
+ # Or for cloud providers:
660
+ # provider: openai
661
+ # apiKey: ${OPENAI_API_KEY}
662
+ # model: gpt-4o-mini
365
663
  ```
366
664
 
367
- ### Programmatic with AI
665
+ > **πŸ”’ Privacy note:** Ollama runs entirely locally. OpenAI and Anthropic send data to their APIs. Choose based on your privacy requirements.
666
+
667
+ ### πŸ€– AI assertions in CI (recommended settings)
668
+
669
+ AI checks can be non-deterministic. For CI, prefer:
670
+
671
+ - **Deterministic settings** (e.g. `temperature: 0` for OpenAI/Anthropic)
672
+ - **Short, specific prompts** (avoid vague questions)
673
+ - **Stable models** (avoid preview/beta models)
674
+
675
+ Example:
368
676
 
369
677
  ```typescript
370
- const report = await runTests({
371
- testDir: './tests',
372
- ai: {
373
- url: 'http://localhost:11434',
374
- model: 'llama3.2:3b',
375
- timeout: 30000,
376
- },
377
- });
678
+ ai: {
679
+ provider: 'openai',
680
+ model: 'gpt-4o-mini',
681
+ // OpenAI doesn't expose temperature in our API yet, but use stable models
682
+ }
378
683
  ```
379
684
 
380
- ### Context file AI config
685
+ ---
381
686
 
382
- ```markdown
383
- ## AI Configuration
384
- - url: http://localhost:11434
385
- - model: llama3.2:3b
687
+ <details>
688
+ <summary><b>πŸ”’ Security & secrets</b> (click to expand)</summary>
689
+
690
+ - **Avoid committing API keys.** Use environment variables (e.g. `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`).
691
+ - The runner **redacts** common secret fields in logs (Authorization headers, tokens, cookies) when verbose mode is enabled.
692
+ - Keep sensitive data out of YAML files β€” use environment variable interpolation or context files with `.gitignore`.
693
+
694
+ **Example:**
695
+
696
+ ```yaml
697
+ headers:
698
+ Authorization: "Bearer ${API_TOKEN}" # Use env vars
386
699
  ```
387
700
 
388
- > AI evaluation requires Ollama running locally. No cloud API keys, no data leaves your machine.
701
+ **Best practices:**
702
+
703
+ - Store secrets in `.env` files (add to `.gitignore`)
704
+ - Use context files for non-sensitive config (base URLs, endpoints)
705
+ - Never commit API keys or tokens in YAML files
706
+
707
+ </details>
708
+
709
+ ---
710
+
711
+ <details>
712
+ <summary><b>🧩 YAML schema & autocomplete (VSCode)</b> (click to expand)</summary>
713
+
714
+ We provide a JSON Schema for `*.yaml` test flows so you get autocomplete + validation in editors.
715
+
716
+ **VSCode setup** (`.vscode/settings.json`):
717
+
718
+ ```json
719
+ {
720
+ "yaml.schemas": {
721
+ "https://raw.githubusercontent.com/carbajalmarcos/testflow-ai/main/schemas/testflow.schema.json": [
722
+ "tests/**/*.yaml",
723
+ "**/*.testflow.yaml"
724
+ ]
725
+ }
726
+ }
727
+ ```
728
+
729
+ This gives you:
730
+
731
+ - βœ… Autocomplete for `name`, `steps`, `request`, `assertions`, etc.
732
+ - βœ… Validation for required fields and types
733
+ - βœ… Hover documentation for operators and options
734
+
735
+ > **Note:** JSON Schema coming in a future release. For now, TypeScript types provide autocomplete via `import type { TestFlow } from 'testflow-ai'`.
736
+
737
+ </details>
389
738
 
390
739
  ---
391
740
 
392
- ## Context Files
741
+ <details>
742
+ <summary><b>πŸ“„ Context Files</b> (click to expand)</summary>
393
743
 
394
744
  Define your project context in Markdown. The runner uses it to resolve `{baseUrlKey}` references in your YAML flows.
395
745
 
@@ -413,19 +763,29 @@ Brief description of your API.
413
763
  - Authentication required for /users
414
764
 
415
765
  ## AI Configuration
766
+ - provider: ollama
416
767
  - url: http://localhost:11434
417
768
  - model: llama3.2:3b
418
769
  ```
419
770
 
771
+ </details>
772
+
420
773
  ---
421
774
 
422
- ## CI/CD Integration
775
+ <details>
776
+ <summary><b>πŸ”„ CI/CD Integration</b> (click to expand)</summary>
777
+
778
+ **testflow-ai** works in any CI/CD pipeline:
779
+
780
+ - Exit code `0` = success, `1` = failure (CI will fail automatically)
781
+ - JSON output: `--format json` for parsing results
782
+ - Tag filtering: `--tags smoke` for faster runs
423
783
 
424
784
  ### GitHub Actions
425
785
 
426
786
  ```yaml
427
787
  jobs:
428
- api-tests:
788
+ test:
429
789
  runs-on: ubuntu-latest
430
790
  steps:
431
791
  - uses: actions/checkout@v4
@@ -433,24 +793,21 @@ jobs:
433
793
  with:
434
794
  node-version: '20'
435
795
  - run: npm ci
796
+ - run: npm install -D testflow-ai
436
797
  - run: npm run start:server &
437
- - run: npx testflow --dir ./tests --context ./context.md --format json > results.json
438
- - uses: actions/upload-artifact@v4
439
- with:
440
- name: test-results
441
- path: results.json
798
+ - run: npx testflow --dir ./tests --context ./context.md
442
799
  ```
443
800
 
444
- ### Exit codes
801
+ That's it. If tests fail, the job fails automatically (exit code 1).
445
802
 
446
- - `0` β€” all flows passed
447
- - `1` β€” one or more flows failed
803
+ </details>
448
804
 
449
805
  ---
450
806
 
451
- ## Output Examples
807
+ <details>
808
+ <summary><b>πŸ“Š Output Examples</b> (click to expand)</summary>
452
809
 
453
- ### Console
810
+ ### Console Output
454
811
 
455
812
  ```
456
813
  ══════════════════════════════════════════════════════════════
@@ -458,10 +815,10 @@ jobs:
458
815
  ══════════════════════════════════════════════════════════════
459
816
 
460
817
  Summary:
461
- Total: 3 flows
462
- Passed: 2
463
- Failed: 1
464
- Duration: 542ms
818
+ Total: 5 flows
819
+ Passed: 5
820
+ Failed: 0
821
+ Duration: 2450ms
465
822
 
466
823
  Narrative:
467
824
 
@@ -483,9 +840,12 @@ Narrative:
483
840
  ══════════════════════════════════════════════════════════════
484
841
  ```
485
842
 
843
+ </details>
844
+
486
845
  ---
487
846
 
488
- ## Roadmap
847
+ <details>
848
+ <summary><b>πŸ—ΊοΈ Roadmap</b> (click to expand)</summary>
489
849
 
490
850
  - [ ] Database assertions (verify records directly via SQL)
491
851
  - [ ] gRPC / RPC support
@@ -495,8 +855,58 @@ Narrative:
495
855
  - [ ] HTML report output
496
856
  - [ ] `testflow init` wizard
497
857
 
858
+ </details>
859
+
860
+ ---
861
+
862
+ ## πŸ“š Examples
863
+
864
+ See the [`examples/`](./examples) directory for:
865
+
866
+ - **Todo List CRUD** (`todo-crud.yaml`) - Complete REST CRUD flow
867
+ - **Todo GraphQL** (`todo-graphql.yaml`) - GraphQL mutations and queries
868
+ - **REST CRUD** (`rest-crud.yaml`) - User management example
869
+ - **GraphQL Flow** (`graphql-flow.yaml`) - GraphQL with variable capture
870
+ - **Auth Flow** (`auth-flow.yaml`) - Authentication and protected routes
871
+ - **Context Files** (`context.md`, `todo-list-context.md`) - API context templates
872
+
873
+ **Quick start with examples:**
874
+
875
+ ```bash
876
+ # Run specific example
877
+ npx testflow --context ./examples/todo-list-context.md ./examples/todo-crud.yaml
878
+
879
+ # Run all examples in directory
880
+ npx testflow --dir ./examples --context ./examples/context.md
881
+ ```
882
+
498
883
  ---
499
884
 
500
- ## License
885
+ ## πŸ“„ License
501
886
 
502
887
  MIT
888
+
889
+ ---
890
+
891
+ <div align="center">
892
+
893
+ **Made with ❀️ by [Marcos Carbajal](https://github.com/carbajalmarcos)**
894
+
895
+ [⭐ Star on GitHub](https://github.com/carbajalmarcos/testflow-ai) β€’ [πŸ“¦ npm](https://www.npmjs.com/package/testflow-ai) β€’ [πŸ› Report Bug](https://github.com/carbajalmarcos/testflow-ai/issues) β€’ [πŸ’¬ Discussions](https://github.com/carbajalmarcos/testflow-ai/discussions)
896
+
897
+ ---
898
+
899
+ ### β˜• Support this project
900
+
901
+ If you find **testflow-ai** useful, consider supporting its development:
902
+
903
+ [![Buy Me A Coffee](https://img.shields.io/badge/Buy%20Me%20A%20Coffee-ffdd00?style=for-the-badge&logo=buy-me-a-coffee&logoColor=black)](https://buymeacoffee.com/carbajalmarcos)
904
+
905
+ **Crypto donations:**
906
+
907
+ - **Bitcoin (BTC):** `bc1qv0ddjg3wcgujk9ad66v9msz8manu5tanhvq0fn`
908
+ - **ERC-20 USDT:** `0x79F57C9D45d2D40420EF071DDAaA27057618E7C8`
909
+
910
+ *Every contribution helps make this project better!*
911
+
912
+ </div>