testflow-ai 0.3.0 β†’ 0.4.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -2,15 +2,19 @@
2
2
 
3
3
  # πŸ§ͺ testflow-ai
4
4
 
5
- **Declarative API testing powered by YAML flows**
5
+ **YAML API flows + optional LLM assertions (local Ollama or cloud)**
6
6
 
7
- *Version-controlled β€’ Human-readable β€’ AI-friendly*
7
+ *Version-controlled β€’ CI/CD-ready β€’ Human-readable*
8
8
 
9
9
  [![npm version](https://img.shields.io/npm/v/testflow-ai.svg?style=for-the-badge&color=blue)](https://www.npmjs.com/package/testflow-ai)
10
10
  [![npm downloads](https://img.shields.io/npm/dm/testflow-ai.svg?style=for-the-badge&color=green)](https://www.npmjs.com/package/testflow-ai)
11
11
  [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg?style=for-the-badge)](https://opensource.org/licenses/MIT)
12
12
  [![Node.js](https://img.shields.io/badge/node-%3E%3D18-green.svg?style=for-the-badge)](https://nodejs.org)
13
13
 
14
+ βœ… **Multi-step flows** (create β†’ capture β†’ reuse β†’ assert)
15
+ πŸ€– **Assert "hard" responses with AI** (privacy-first via Ollama)
16
+ πŸ“„ **Keep API context in Markdown** (great for humans + AI agents)
17
+
14
18
  [πŸ“– Documentation](#-documentation) β€’ [πŸš€ Quick Start](#-quick-start) β€’ [πŸ’» Examples](#-real-world-example) β€’ [πŸ€– AI Providers](#-ai-powered-evaluation)
15
19
 
16
20
  </div>
@@ -19,9 +23,59 @@
19
23
 
20
24
  ## 🎯 What is testflow-ai?
21
25
 
22
- **testflow-ai** lets you define API tests in YAML files, run them from the command line or as a library, and use AI models (local or cloud) for intelligent assertions. No GUI, no vendor lock-in, works with any HTTP/GraphQL API.
26
+ **testflow-ai** lets you describe API scenarios in YAML files, run them from the command line or as a library, and (optionally) ask an AI model to judge complex responses. No GUI, no vendor lock‑in, and it works with any HTTP/GraphQL API.
27
+
28
+ > **πŸ’‘ Born from real-world frustration:**
29
+ > After days of testing APIs with Postman and burning tokens with ChatGPT, I built this to centralize tests in version-controlled YAML files with local AI support.
30
+ > I wanted something that felt more like a **test agent**: a tool that could **create data, mutate it, delete it, and walk full flows end‑to‑end**, but defined in plain files, close to the code, and easy to run in CI.
31
+ > **testflow-ai** is that tool: a thin engine that turns YAML flows into real HTTP calls, variable captures, assertions, and (if you want) AI‑powered checks.
32
+
33
+ ---
34
+
35
+ ## ✨ Why it's different
36
+
37
+ Most API testing tools are either **GUI-first** (collections) or **code-first** (JS/TS test code).
38
+ **testflow-ai** is **flow-first**: readable YAML that runs in CI β€” with an optional AI judge when classic assertions aren't enough.
39
+
40
+ **What you get:**
41
+
42
+ - **Flow engine**: multi-step scenarios with capture + interpolation (CRUD, auth, webhooks, background jobs)
43
+ - **AI assertions**: validate complex text/structured responses with natural language checks (Ollama/OpenAI/Anthropic)
44
+ - **Context-as-docs**: a Markdown file that explains base URLs, endpoints, and rules β€” perfect input for AI agents too
45
+
46
+ ---
47
+
48
+ ## βœ… When to use testflow-ai
49
+
50
+ - You want **version-controlled API E2E flows** (not a GUI collection)
51
+ - You need **multi-step chaining** (create β†’ capture id β†’ update β†’ verify)
52
+ - You want **CI/CD-ready output** (console/json/markdown + exit codes + no external deps)
53
+ - You sometimes need an **AI judge** for fuzzy checks (content quality, summaries, "is this coherent?")
54
+
55
+ ## 🚫 When NOT to use it
56
+
57
+ - You only need **schema/property-based fuzzing** from OpenAPI
58
+ - You prefer **writing tests in code** (Jest/Vitest) with full programmatic control
59
+ - You need **browser/UI testing** (Playwright/Cypress territory)
60
+
61
+ ---
62
+
63
+ ## πŸ†š What testflow-ai optimizes for
23
64
 
24
- > **πŸ’‘ Born from real-world frustration:** After months of testing APIs with Postman and burning tokens with ChatGPT, I built this to centralize tests in version-controlled YAML files with local AI support.
65
+ <div align="center">
66
+
67
+ | Goal | testflow-ai |
68
+ |:----:|:-----------:|
69
+ | Human-readable flows in Git | βœ… |
70
+ | Multi-step chaining + captures | βœ… |
71
+ | CI/CD-ready (exit codes, JSON) | βœ… |
72
+ | Optional AI-based assertions | βœ… |
73
+ | GUI collections | ❌ (not a goal) |
74
+ | Full code-based test suites | ❌ (use your test framework) |
75
+
76
+ </div>
77
+
78
+ ---
25
79
 
26
80
  ### ✨ Key Features
27
81
 
@@ -46,63 +100,63 @@
46
100
 
47
101
  ## πŸš€ Quick Start
48
102
 
49
- ### 1️⃣ Install
50
-
51
103
  ```bash
52
- npm install testflow-ai
104
+ npm i -D testflow-ai
53
105
  ```
54
106
 
55
- ### 2️⃣ Create a test flow
107
+ **Create `context.md`:**
56
108
 
57
- Create `tests/health.yaml`:
109
+ ```markdown
110
+ # My API
111
+
112
+ ## Base URLs
113
+ - api: http://localhost:3000
114
+ ```
115
+
116
+ **Create `tests/todo.yaml`:**
58
117
 
59
118
  ```yaml
60
- name: Health Check
119
+ name: Todo flow
61
120
  tags: [smoke]
62
121
 
63
122
  steps:
64
- - name: Ping API
123
+ - name: Create todo
65
124
  request:
66
- method: GET
67
- url: http://localhost:3000/health
125
+ method: POST
126
+ url: "{api}/todos"
127
+ headers:
128
+ Content-Type: application/json
129
+ body:
130
+ title: "Buy milk"
131
+ completed: false
132
+ capture:
133
+ - name: todoId
134
+ path: data.id
68
135
  assertions:
69
136
  - path: status
70
137
  operator: equals
71
- value: 200
138
+ value: 201
139
+
140
+ - name: Fetch todo
141
+ request:
142
+ method: GET
143
+ url: "{api}/todos/${todoId}"
144
+ assertions:
145
+ - path: data.title
146
+ operator: equals
147
+ value: "Buy milk"
72
148
  ```
73
149
 
74
- ### 3️⃣ Run
150
+ **Run:**
75
151
 
76
152
  ```bash
77
- npx testflow tests/health.yaml
153
+ npx testflow --context ./context.md tests/todo.yaml
78
154
  ```
79
155
 
80
156
  **That's it.** No config files, no GUI, no account.
81
157
 
82
158
  ---
83
159
 
84
- ## πŸ’‘ Why testflow-ai?
85
-
86
- I was building a backend that started as a simple API and grew into a system with GraphQL, async workers, state machines, and AI-powered evaluations.
87
-
88
- Testing started simple β€” a few requests in Postman. Then the project scaled:
89
-
90
- <div align="center">
91
-
92
- | ❌ Problem | βœ… Solution |
93
- |:----------:|:-----------:|
94
- | **Postman / Insomnia** became unmanageable | YAML files in version control |
95
- | **IDE AI assistants** burned tokens, lost context | Local AI via Ollama (free, private) |
96
- | **MCP servers** required complex setup | Zero dependencies beyond Node.js |
97
- | **Manual token copying** between requests | Automatic variable capture |
98
- | **No CI/CD integration** | JSON output, exit codes, GitHub Actions ready |
99
-
100
- </div>
101
-
102
- **testflow-ai** solves all of this.
103
-
104
- ---
105
-
106
160
  ## πŸ“¦ Installation
107
161
 
108
162
  ```bash
@@ -167,23 +221,26 @@ process.exit(report.failedFlows > 0 ? 1 : 0);
167
221
 
168
222
  ---
169
223
 
170
- ## πŸ’» Real-World Example
224
+ <details>
225
+ <summary><b>πŸ’» Real-World Example</b> (click to expand)</summary>
171
226
 
172
- Here's how we use it in production at [educational-rewards](https://github.com/carbajalmarcos/educational-rewards):
227
+ Here's a complete example using a Todo List API:
173
228
 
174
229
  ### Project Structure
175
230
 
176
231
  ```
177
- tests/declarative/
178
- β”œβ”€β”€ index.ts # Test runner
179
- β”œβ”€β”€ context.md # API context
180
- └── flows/
181
- β”œβ”€β”€ health-check.yaml
182
- β”œβ”€β”€ complete-reward-flow.yaml
183
- └── submission-attempts.yaml
232
+ my-api/
233
+ β”œβ”€β”€ tests/
234
+ β”‚ β”œβ”€β”€ index.ts # Test runner
235
+ β”‚ β”œβ”€β”€ context.md # API context
236
+ β”‚ └── flows/
237
+ β”‚ β”œβ”€β”€ health-check.yaml
238
+ β”‚ β”œβ”€β”€ todo-crud.yaml
239
+ β”‚ └── todo-complete-flow.yaml
240
+ └── package.json
184
241
  ```
185
242
 
186
- ### Test Runner (`index.ts`)
243
+ ### Test Runner (`tests/index.ts`)
187
244
 
188
245
  ```typescript
189
246
  import { runTests, type RunnerOptions } from 'testflow-ai';
@@ -205,55 +262,86 @@ async function main() {
205
262
  main();
206
263
  ```
207
264
 
208
- ### Context File (`context.md`)
265
+ ### Context File (`tests/context.md`)
209
266
 
210
267
  ```markdown
211
- # Mambita API Context
268
+ # Todo List API
269
+
270
+ ## Description
271
+ A simple REST API for managing todo items.
212
272
 
213
273
  ## Base URLs
274
+ - api: http://localhost:3000
214
275
  - graphql: http://localhost:3000/graphql
215
- - tasks: http://localhost:8000
216
276
 
217
277
  ## Endpoints
278
+ - POST /todos - Create a new todo
279
+ - GET /todos/:id - Get todo by ID
280
+ - PUT /todos/:id - Update todo
281
+ - DELETE /todos/:id - Delete todo
218
282
  - POST /graphql - GraphQL endpoint
219
- - POST /api/v1/pool/seed - Seed task pool
220
283
  ```
221
284
 
222
- ### Test Flow (`flows/complete-reward-flow.yaml`)
285
+ ### Test Flow (`tests/flows/todo-crud.yaml`)
223
286
 
224
287
  ```yaml
225
- name: Complete Reward Flow (E2E)
226
- tags: [e2e, reward]
288
+ name: Todo CRUD Flow
289
+ tags: [todos, crud, smoke]
227
290
 
228
291
  steps:
229
- - name: Start reward
292
+ - name: Create todo
230
293
  request:
231
294
  method: POST
232
- url: "{graphql}"
233
- graphql:
234
- query: |
235
- mutation StartReward($input: StartRewardInput!) {
236
- startReward(input: $input) {
237
- id
238
- state
239
- taskInstances { id state }
240
- }
241
- }
242
- variables:
243
- input:
244
- childId: "${childId}"
245
- catalogRewardId: "${catalogItemId}"
295
+ url: "{api}/todos"
296
+ headers:
297
+ Content-Type: application/json
298
+ body:
299
+ title: "Buy groceries"
300
+ completed: false
246
301
  capture:
247
- - name: rewardId
248
- path: data.startReward.id
302
+ - name: todoId
303
+ path: data.id
304
+ assertions:
305
+ - path: status
306
+ operator: equals
307
+ value: 201
308
+ - path: data.title
309
+ operator: equals
310
+ value: "Buy groceries"
311
+
312
+ - name: Get todo
313
+ request:
314
+ method: GET
315
+ url: "{api}/todos/${todoId}"
316
+ assertions:
317
+ - path: data.id
318
+ operator: equals
319
+ value: "${todoId}"
320
+
321
+ - name: Update todo
322
+ request:
323
+ method: PUT
324
+ url: "{api}/todos/${todoId}"
325
+ headers:
326
+ Content-Type: application/json
327
+ body:
328
+ completed: true
329
+ assertions:
330
+ - path: data.completed
331
+ operator: equals
332
+ value: true
249
333
  ```
250
334
 
251
335
  ### Running Tests
252
336
 
253
337
  ```bash
254
- pnpm testflow:run # All tests
255
- pnpm testflow:smoke # Smoke tests only
256
- pnpm testflow submission-attempts # Specific flow
338
+ # Add to package.json scripts:
339
+ "test:e2e": "ts-node tests/index.ts"
340
+ "test:smoke": "ts-node tests/index.ts --tags=smoke"
341
+
342
+ # Then run:
343
+ npm run test:e2e
344
+ npm run test:smoke
257
345
  ```
258
346
 
259
347
  ### Advanced usage
@@ -276,9 +364,12 @@ const executor = new FlowExecutor(context, true);
276
364
  const result = await executor.executeFlow(flow);
277
365
  ```
278
366
 
367
+ </details>
368
+
279
369
  ---
280
370
 
281
- ## πŸ“ Test Flow Reference
371
+ <details>
372
+ <summary><b>πŸ“ Test Flow Reference</b> (click to expand)</summary>
282
373
 
283
374
  ### Basic structure
284
375
 
@@ -399,9 +490,12 @@ steps:
399
490
  value: "COMPLETED"
400
491
  ```
401
492
 
493
+ </details>
494
+
402
495
  ---
403
496
 
404
- ## βœ… Assertions
497
+ <details>
498
+ <summary><b>βœ… Assertions</b> (click to expand)</summary>
405
499
 
406
500
  <div align="center">
407
501
 
@@ -427,6 +521,8 @@ steps:
427
521
  - `data.field` β€” response body field
428
522
  - `data.items[0].id` β€” array access
429
523
 
524
+ </details>
525
+
430
526
  ---
431
527
 
432
528
  ## πŸ€– AI-Powered Evaluation
@@ -568,9 +664,82 @@ steps:
568
664
 
569
665
  > **πŸ”’ Privacy note:** Ollama runs entirely locally. OpenAI and Anthropic send data to their APIs. Choose based on your privacy requirements.
570
666
 
667
+ ### πŸ€– AI assertions in CI (recommended settings)
668
+
669
+ AI checks can be non-deterministic. For CI, prefer:
670
+
671
+ - **Deterministic settings** (e.g. `temperature: 0` for OpenAI/Anthropic)
672
+ - **Short, specific prompts** (avoid vague questions)
673
+ - **Stable models** (avoid preview/beta models)
674
+
675
+ Example:
676
+
677
+ ```typescript
678
+ ai: {
679
+ provider: 'openai',
680
+ model: 'gpt-4o-mini',
681
+ // OpenAI doesn't expose temperature in our API yet, but use stable models
682
+ }
683
+ ```
684
+
571
685
  ---
572
686
 
573
- ## πŸ“„ Context Files
687
+ <details>
688
+ <summary><b>πŸ”’ Security & secrets</b> (click to expand)</summary>
689
+
690
+ - **Avoid committing API keys.** Use environment variables (e.g. `OPENAI_API_KEY`, `ANTHROPIC_API_KEY`).
691
+ - The runner **redacts** common secret fields in logs (Authorization headers, tokens, cookies) when verbose mode is enabled.
692
+ - Keep sensitive data out of YAML files β€” use environment variable interpolation or context files with `.gitignore`.
693
+
694
+ **Example:**
695
+
696
+ ```yaml
697
+ headers:
698
+ Authorization: "Bearer ${API_TOKEN}" # Use env vars
699
+ ```
700
+
701
+ **Best practices:**
702
+
703
+ - Store secrets in `.env` files (add to `.gitignore`)
704
+ - Use context files for non-sensitive config (base URLs, endpoints)
705
+ - Never commit API keys or tokens in YAML files
706
+
707
+ </details>
708
+
709
+ ---
710
+
711
+ <details>
712
+ <summary><b>🧩 YAML schema & autocomplete (VSCode)</b> (click to expand)</summary>
713
+
714
+ We provide a JSON Schema for `*.yaml` test flows so you get autocomplete + validation in editors.
715
+
716
+ **VSCode setup** (`.vscode/settings.json`):
717
+
718
+ ```json
719
+ {
720
+ "yaml.schemas": {
721
+ "https://raw.githubusercontent.com/carbajalmarcos/testflow-ai/main/schemas/testflow.schema.json": [
722
+ "tests/**/*.yaml",
723
+ "**/*.testflow.yaml"
724
+ ]
725
+ }
726
+ }
727
+ ```
728
+
729
+ This gives you:
730
+
731
+ - βœ… Autocomplete for `name`, `steps`, `request`, `assertions`, etc.
732
+ - βœ… Validation for required fields and types
733
+ - βœ… Hover documentation for operators and options
734
+
735
+ > **Note:** JSON Schema coming in a future release. For now, TypeScript types provide autocomplete via `import type { TestFlow } from 'testflow-ai'`.
736
+
737
+ </details>
738
+
739
+ ---
740
+
741
+ <details>
742
+ <summary><b>πŸ“„ Context Files</b> (click to expand)</summary>
574
743
 
575
744
  Define your project context in Markdown. The runner uses it to resolve `{baseUrlKey}` references in your YAML flows.
576
745
 
@@ -599,15 +768,24 @@ Brief description of your API.
599
768
  - model: llama3.2:3b
600
769
  ```
601
770
 
771
+ </details>
772
+
602
773
  ---
603
774
 
604
- ## πŸ”„ CI/CD Integration
775
+ <details>
776
+ <summary><b>πŸ”„ CI/CD Integration</b> (click to expand)</summary>
777
+
778
+ **testflow-ai** works in any CI/CD pipeline:
779
+
780
+ - Exit code `0` = success, `1` = failure (CI will fail automatically)
781
+ - JSON output: `--format json` for parsing results
782
+ - Tag filtering: `--tags smoke` for faster runs
605
783
 
606
784
  ### GitHub Actions
607
785
 
608
786
  ```yaml
609
787
  jobs:
610
- api-tests:
788
+ test:
611
789
  runs-on: ubuntu-latest
612
790
  steps:
613
791
  - uses: actions/checkout@v4
@@ -615,22 +793,19 @@ jobs:
615
793
  with:
616
794
  node-version: '20'
617
795
  - run: npm ci
796
+ - run: npm install -D testflow-ai
618
797
  - run: npm run start:server &
619
- - run: npx testflow --dir ./tests --context ./context.md --format json > results.json
620
- - uses: actions/upload-artifact@v4
621
- with:
622
- name: test-results
623
- path: results.json
798
+ - run: npx testflow --dir ./tests --context ./context.md
624
799
  ```
625
800
 
626
- **Exit codes:**
801
+ That's it. If tests fail, the job fails automatically (exit code 1).
627
802
 
628
- - `0` β€” all flows passed
629
- - `1` β€” one or more flows failed
803
+ </details>
630
804
 
631
805
  ---
632
806
 
633
- ## πŸ“Š Output Examples
807
+ <details>
808
+ <summary><b>πŸ“Š Output Examples</b> (click to expand)</summary>
634
809
 
635
810
  ### Console Output
636
811
 
@@ -665,9 +840,12 @@ Narrative:
665
840
  ══════════════════════════════════════════════════════════════
666
841
  ```
667
842
 
843
+ </details>
844
+
668
845
  ---
669
846
 
670
- ## πŸ—ΊοΈ Roadmap
847
+ <details>
848
+ <summary><b>πŸ—ΊοΈ Roadmap</b> (click to expand)</summary>
671
849
 
672
850
  - [ ] Database assertions (verify records directly via SQL)
673
851
  - [ ] gRPC / RPC support
@@ -677,16 +855,30 @@ Narrative:
677
855
  - [ ] HTML report output
678
856
  - [ ] `testflow init` wizard
679
857
 
858
+ </details>
859
+
680
860
  ---
681
861
 
682
862
  ## πŸ“š Examples
683
863
 
684
864
  See the [`examples/`](./examples) directory for:
685
865
 
686
- - REST CRUD flows
687
- - GraphQL queries and mutations
688
- - Authentication flows
689
- - Context file templates
866
+ - **Todo List CRUD** (`todo-crud.yaml`) - Complete REST CRUD flow
867
+ - **Todo GraphQL** (`todo-graphql.yaml`) - GraphQL mutations and queries
868
+ - **REST CRUD** (`rest-crud.yaml`) - User management example
869
+ - **GraphQL Flow** (`graphql-flow.yaml`) - GraphQL with variable capture
870
+ - **Auth Flow** (`auth-flow.yaml`) - Authentication and protected routes
871
+ - **Context Files** (`context.md`, `todo-list-context.md`) - API context templates
872
+
873
+ **Quick start with examples:**
874
+
875
+ ```bash
876
+ # Run specific example
877
+ npx testflow --context ./examples/todo-list-context.md ./examples/todo-crud.yaml
878
+
879
+ # Run all examples in directory
880
+ npx testflow --dir ./examples --context ./examples/context.md
881
+ ```
690
882
 
691
883
  ---
692
884
 
@@ -709,8 +901,6 @@ MIT
709
901
  If you find **testflow-ai** useful, consider supporting its development:
710
902
 
711
903
  [![Buy Me A Coffee](https://img.shields.io/badge/Buy%20Me%20A%20Coffee-ffdd00?style=for-the-badge&logo=buy-me-a-coffee&logoColor=black)](https://buymeacoffee.com/carbajalmarcos)
712
- [![GitHub Sponsors](https://img.shields.io/badge/GitHub%20Sponsors-ea4aaa?style=for-the-badge&logo=github&logoColor=white)](https://github.com/sponsors/carbajalmarcos)
713
- [![Ko-fi](https://img.shields.io/badge/Ko--fi-F16061?style=for-the-badge&logo=ko-fi&logoColor=white)](https://ko-fi.com/carbajalmarcos)
714
904
 
715
905
  **Crypto donations:**
716
906
 
@@ -0,0 +1,52 @@
1
+ # Auth flow β€” Login, use token, access protected resource.
2
+
3
+ name: Authentication Flow
4
+ description: Login and access a protected endpoint using the returned token
5
+ tags:
6
+ - auth
7
+ - smoke
8
+
9
+ steps:
10
+ - name: Login
11
+ request:
12
+ method: POST
13
+ url: "{api}/auth/login"
14
+ headers:
15
+ Content-Type: application/json
16
+ body:
17
+ email: admin@example.com
18
+ password: secret
19
+ capture:
20
+ - name: token
21
+ path: data.accessToken
22
+ - name: userId
23
+ path: data.user.id
24
+ assertions:
25
+ - path: status
26
+ operator: equals
27
+ value: 200
28
+ - path: data.accessToken
29
+ operator: exists
30
+
31
+ - name: Access protected route
32
+ request:
33
+ method: GET
34
+ url: "{api}/users/${userId}"
35
+ headers:
36
+ Authorization: "Bearer ${token}"
37
+ assertions:
38
+ - path: status
39
+ operator: equals
40
+ value: 200
41
+ - path: data.id
42
+ operator: equals
43
+ value: "${userId}"
44
+
45
+ - name: Reject without token
46
+ request:
47
+ method: GET
48
+ url: "{api}/users/${userId}"
49
+ assertions:
50
+ - path: status
51
+ operator: equals
52
+ value: 401
@@ -0,0 +1,25 @@
1
+ # My API
2
+
3
+ ## Description
4
+ Example API context for testflow-ai.
5
+
6
+ ## Base URLs
7
+ - api: http://localhost:3000
8
+ - graphql: http://localhost:3000/graphql
9
+
10
+ ## Endpoints
11
+ - POST /users - Create a new user
12
+ - GET /users/:id - Get user by ID
13
+ - PUT /users/:id - Update user
14
+ - DELETE /users/:id - Delete user
15
+ - POST /auth/login - Authenticate and get token
16
+ - POST /graphql - GraphQL endpoint
17
+
18
+ ## Rules
19
+ - All endpoints return JSON
20
+ - Authentication required for /users endpoints
21
+ - Rate limit: 100 requests per minute
22
+
23
+ ## AI Configuration
24
+ - url: http://localhost:11434
25
+ - model: llama3.2:3b
@@ -0,0 +1,58 @@
1
+ # GraphQL β€” Mutation + query with variable capture.
2
+
3
+ name: GraphQL User Flow
4
+ description: Create a user via mutation, then query it back
5
+ tags:
6
+ - graphql
7
+ - users
8
+
9
+ steps:
10
+ - name: Create user (mutation)
11
+ request:
12
+ method: POST
13
+ url: "{graphql}"
14
+ graphql:
15
+ query: |
16
+ mutation CreateUser($input: CreateUserInput!) {
17
+ createUser(input: $input) {
18
+ id
19
+ email
20
+ name
21
+ }
22
+ }
23
+ variables:
24
+ input:
25
+ email: bob@example.com
26
+ name: Bob
27
+ capture:
28
+ - name: userId
29
+ path: data.createUser.id
30
+ assertions:
31
+ - path: data.createUser.email
32
+ operator: equals
33
+ value: bob@example.com
34
+ - path: data.createUser.id
35
+ operator: exists
36
+
37
+ - name: Query user
38
+ request:
39
+ method: POST
40
+ url: "{graphql}"
41
+ graphql:
42
+ query: |
43
+ query GetUser($id: ID!) {
44
+ user(id: $id) {
45
+ id
46
+ email
47
+ name
48
+ }
49
+ }
50
+ variables:
51
+ id: "${userId}"
52
+ assertions:
53
+ - path: data.user.id
54
+ operator: equals
55
+ value: "${userId}"
56
+ - path: data.user.email
57
+ operator: equals
58
+ value: bob@example.com
@@ -0,0 +1,76 @@
1
+ # REST CRUD β€” Create, read, update, verify a user.
2
+
3
+ name: User CRUD
4
+ description: Full lifecycle test for a REST user resource
5
+ tags:
6
+ - users
7
+ - crud
8
+ - smoke
9
+
10
+ steps:
11
+ - name: Create user
12
+ request:
13
+ method: POST
14
+ url: "{api}/users"
15
+ headers:
16
+ Content-Type: application/json
17
+ body:
18
+ email: alice@example.com
19
+ name: Alice
20
+ capture:
21
+ - name: userId
22
+ path: data.id
23
+ - name: userEmail
24
+ path: data.email
25
+ assertions:
26
+ - path: status
27
+ operator: equals
28
+ value: 201
29
+ - path: data.email
30
+ operator: equals
31
+ value: alice@example.com
32
+ - path: data.id
33
+ operator: exists
34
+
35
+ - name: Read user
36
+ request:
37
+ method: GET
38
+ url: "{api}/users/${userId}"
39
+ assertions:
40
+ - path: status
41
+ operator: equals
42
+ value: 200
43
+ - path: data.id
44
+ operator: equals
45
+ value: "${userId}"
46
+ - path: data.email
47
+ operator: equals
48
+ value: "${userEmail}"
49
+
50
+ - name: Update user
51
+ request:
52
+ method: PUT
53
+ url: "{api}/users/${userId}"
54
+ headers:
55
+ Content-Type: application/json
56
+ body:
57
+ name: Alice Updated
58
+ assertions:
59
+ - path: status
60
+ operator: equals
61
+ value: 200
62
+ - path: data.name
63
+ operator: equals
64
+ value: Alice Updated
65
+
66
+ - name: Verify update
67
+ request:
68
+ method: GET
69
+ url: "{api}/users/${userId}"
70
+ assertions:
71
+ - path: data.name
72
+ operator: equals
73
+ value: Alice Updated
74
+ - path: data.email
75
+ operator: equals
76
+ value: "${userEmail}"
@@ -0,0 +1,97 @@
1
+ # Todo List CRUD β€” Complete lifecycle test
2
+
3
+ name: Todo CRUD Flow
4
+ description: Create, read, update, and delete a todo item
5
+ tags:
6
+ - todos
7
+ - crud
8
+ - smoke
9
+
10
+ steps:
11
+ - name: Create todo
12
+ request:
13
+ method: POST
14
+ url: "{api}/todos"
15
+ headers:
16
+ Content-Type: application/json
17
+ body:
18
+ title: "Buy groceries"
19
+ completed: false
20
+ capture:
21
+ - name: todoId
22
+ path: data.id
23
+ - name: todoTitle
24
+ path: data.title
25
+ assertions:
26
+ - path: status
27
+ operator: equals
28
+ value: 201
29
+ - path: data.title
30
+ operator: equals
31
+ value: "Buy groceries"
32
+ - path: data.completed
33
+ operator: equals
34
+ value: false
35
+ - path: data.id
36
+ operator: exists
37
+
38
+ - name: Read todo
39
+ request:
40
+ method: GET
41
+ url: "{api}/todos/${todoId}"
42
+ assertions:
43
+ - path: status
44
+ operator: equals
45
+ value: 200
46
+ - path: data.id
47
+ operator: equals
48
+ value: "${todoId}"
49
+ - path: data.title
50
+ operator: equals
51
+ value: "${todoTitle}"
52
+
53
+ - name: Update todo
54
+ request:
55
+ method: PUT
56
+ url: "{api}/todos/${todoId}"
57
+ headers:
58
+ Content-Type: application/json
59
+ body:
60
+ completed: true
61
+ assertions:
62
+ - path: status
63
+ operator: equals
64
+ value: 200
65
+ - path: data.completed
66
+ operator: equals
67
+ value: true
68
+
69
+ - name: Verify update
70
+ request:
71
+ method: GET
72
+ url: "{api}/todos/${todoId}"
73
+ assertions:
74
+ - path: data.completed
75
+ operator: equals
76
+ value: true
77
+ - path: data.title
78
+ operator: equals
79
+ value: "${todoTitle}"
80
+
81
+ - name: Delete todo
82
+ request:
83
+ method: DELETE
84
+ url: "{api}/todos/${todoId}"
85
+ assertions:
86
+ - path: status
87
+ operator: equals
88
+ value: 200
89
+
90
+ - name: Verify deletion
91
+ request:
92
+ method: GET
93
+ url: "{api}/todos/${todoId}"
94
+ assertions:
95
+ - path: status
96
+ operator: equals
97
+ value: 404
@@ -0,0 +1,59 @@
1
+ # GraphQL Todo Flow β€” Mutation + query with variable capture
2
+
3
+ name: GraphQL Todo Flow
4
+ description: Create a todo via mutation, then query it back
5
+ tags:
6
+ - graphql
7
+ - todos
8
+
9
+ steps:
10
+ - name: Create todo (mutation)
11
+ request:
12
+ method: POST
13
+ url: "{graphql}"
14
+ graphql:
15
+ query: |
16
+ mutation CreateTodo($input: CreateTodoInput!) {
17
+ createTodo(input: $input) {
18
+ id
19
+ title
20
+ completed
21
+ createdAt
22
+ }
23
+ }
24
+ variables:
25
+ input:
26
+ title: "Learn testflow-ai"
27
+ completed: false
28
+ capture:
29
+ - name: todoId
30
+ path: data.createTodo.id
31
+ assertions:
32
+ - path: data.createTodo.title
33
+ operator: equals
34
+ value: "Learn testflow-ai"
35
+ - path: data.createTodo.id
36
+ operator: exists
37
+
38
+ - name: Query todo
39
+ request:
40
+ method: POST
41
+ url: "{graphql}"
42
+ graphql:
43
+ query: |
44
+ query GetTodo($id: ID!) {
45
+ todo(id: $id) {
46
+ id
47
+ title
48
+ completed
49
+ }
50
+ }
51
+ variables:
52
+ id: "${todoId}"
53
+ assertions:
54
+ - path: data.todo.id
55
+ operator: equals
56
+ value: "${todoId}"
57
+ - path: data.todo.title
58
+ operator: equals
59
+ value: "Learn testflow-ai"
@@ -0,0 +1,27 @@
1
+ # Todo List API
2
+
3
+ ## Description
4
+ A simple REST API for managing todo items with authentication.
5
+
6
+ ## Base URLs
7
+ - api: http://localhost:3000
8
+ - graphql: http://localhost:3000/graphql
9
+
10
+ ## Endpoints
11
+ - POST /todos - Create a new todo
12
+ - GET /todos/:id - Get todo by ID
13
+ - GET /todos - List all todos
14
+ - PUT /todos/:id - Update todo
15
+ - DELETE /todos/:id - Delete todo
16
+ - POST /auth/login - Authenticate and get token
17
+ - POST /graphql - GraphQL endpoint
18
+
19
+ ## Rules
20
+ - All endpoints return JSON
21
+ - Authentication required for /todos endpoints
22
+ - Todos have: id, title, completed, createdAt
23
+
24
+ ## AI Configuration
25
+ - provider: ollama
26
+ - url: http://localhost:11434
27
+ - model: llama3.2:3b
package/package.json CHANGED
@@ -1,70 +1,71 @@
1
1
  {
2
- "name": "testflow-ai",
3
- "version": "0.3.0",
4
- "description": "Declarative API testing powered by YAML flows. Replace Postman with version-controlled, AI-friendly test definitions.",
5
- "author": "Marcos Carbajal",
6
- "license": "MIT",
7
- "repository": {
8
- "type": "git",
9
- "url": "https://github.com/carbajalmarcos/testflow-ai"
10
- },
11
- "homepage": "https://github.com/carbajalmarcos/testflow-ai#readme",
12
- "bugs": {
13
- "url": "https://github.com/carbajalmarcos/testflow-ai/issues"
14
- },
15
- "main": "./dist/index.js",
16
- "types": "./dist/index.d.ts",
17
- "exports": {
18
- ".": {
19
- "types": "./dist/index.d.ts",
20
- "default": "./dist/index.js"
2
+ "name": "testflow-ai",
3
+ "version": "0.4.0",
4
+ "description": "Declarative API testing powered by YAML flows. Replace Postman with version-controlled, AI-friendly test definitions.",
5
+ "author": "Marcos Carbajal",
6
+ "license": "MIT",
7
+ "repository": {
8
+ "type": "git",
9
+ "url": "https://github.com/carbajalmarcos/testflow-ai"
10
+ },
11
+ "homepage": "https://github.com/carbajalmarcos/testflow-ai#readme",
12
+ "bugs": {
13
+ "url": "https://github.com/carbajalmarcos/testflow-ai/issues"
14
+ },
15
+ "main": "./dist/index.js",
16
+ "types": "./dist/index.d.ts",
17
+ "exports": {
18
+ ".": {
19
+ "types": "./dist/index.d.ts",
20
+ "default": "./dist/index.js"
21
+ }
22
+ },
23
+ "bin": {
24
+ "testflow": "./dist/cli.js",
25
+ "tflow": "./dist/cli.js"
26
+ },
27
+ "files": [
28
+ "dist",
29
+ "README.md",
30
+ "LICENSE",
31
+ "examples"
32
+ ],
33
+ "scripts": {
34
+ "build": "tsc",
35
+ "test": "vitest run",
36
+ "test:watch": "vitest",
37
+ "lint": "tsc --noEmit",
38
+ "clean": "rm -rf dist",
39
+ "prepublishOnly": "npm run build"
40
+ },
41
+ "keywords": [
42
+ "testing",
43
+ "api-testing",
44
+ "declarative",
45
+ "yaml",
46
+ "e2e",
47
+ "integration",
48
+ "graphql",
49
+ "rest",
50
+ "http",
51
+ "ai",
52
+ "ollama",
53
+ "flow",
54
+ "ci-cd",
55
+ "test-automation"
56
+ ],
57
+ "dependencies": {
58
+ "axios": "^1.7.0",
59
+ "commander": "^12.1.0",
60
+ "glob": "^11.0.0",
61
+ "yaml": "^2.4.0"
62
+ },
63
+ "devDependencies": {
64
+ "@types/node": "^22.0.0",
65
+ "typescript": "^5.7.0",
66
+ "vitest": "^2.0.0"
67
+ },
68
+ "engines": {
69
+ "node": ">=18.0.0"
21
70
  }
22
- },
23
- "bin": {
24
- "testflow": "./dist/cli.js",
25
- "tflow": "./dist/cli.js"
26
- },
27
- "files": [
28
- "dist",
29
- "README.md",
30
- "LICENSE"
31
- ],
32
- "scripts": {
33
- "build": "tsc",
34
- "test": "vitest run",
35
- "test:watch": "vitest",
36
- "lint": "tsc --noEmit",
37
- "clean": "rm -rf dist",
38
- "prepublishOnly": "npm run build"
39
- },
40
- "keywords": [
41
- "testing",
42
- "api-testing",
43
- "declarative",
44
- "yaml",
45
- "e2e",
46
- "integration",
47
- "graphql",
48
- "rest",
49
- "http",
50
- "ai",
51
- "ollama",
52
- "flow",
53
- "ci-cd",
54
- "test-automation"
55
- ],
56
- "dependencies": {
57
- "axios": "^1.7.0",
58
- "commander": "^12.1.0",
59
- "glob": "^11.0.0",
60
- "yaml": "^2.4.0"
61
- },
62
- "devDependencies": {
63
- "@types/node": "^22.0.0",
64
- "typescript": "^5.7.0",
65
- "vitest": "^2.0.0"
66
- },
67
- "engines": {
68
- "node": ">=18.0.0"
69
- }
70
71
  }