pytest-httpchain 0.2.4__tar.gz → 0.4.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- {pytest_httpchain-0.2.4 → pytest_httpchain-0.4.0}/PKG-INFO +35 -21
- {pytest_httpchain-0.2.4 → pytest_httpchain-0.4.0}/README.md +33 -19
- {pytest_httpchain-0.2.4 → pytest_httpchain-0.4.0}/pyproject.toml +2 -5
- {pytest_httpchain-0.2.4 → pytest_httpchain-0.4.0}/src/pytest_httpchain/carrier.py +21 -25
- pytest_httpchain-0.4.0/src/pytest_httpchain/cli.py +83 -0
- {pytest_httpchain-0.2.4 → pytest_httpchain-0.4.0}/src/pytest_httpchain/har_writer.py +1 -2
- {pytest_httpchain-0.2.4 → pytest_httpchain-0.4.0}/src/pytest_httpchain/plugin.py +20 -2
- {pytest_httpchain-0.2.4 → pytest_httpchain-0.4.0}/src/pytest_httpchain/skill.md +40 -10
- {pytest_httpchain-0.2.4 → pytest_httpchain-0.4.0}/src/pytest_httpchain/utils.py +3 -3
- pytest_httpchain-0.4.0/src/pytest_httpchain/validation.py +814 -0
- pytest_httpchain-0.2.4/src/pytest_httpchain/cli.py +0 -67
- {pytest_httpchain-0.2.4 → pytest_httpchain-0.4.0}/LICENSE +0 -0
- {pytest_httpchain-0.2.4 → pytest_httpchain-0.4.0}/src/pytest_httpchain/__init__.py +0 -0
- {pytest_httpchain-0.2.4 → pytest_httpchain-0.4.0}/src/pytest_httpchain/constants.py +0 -0
- {pytest_httpchain-0.2.4 → pytest_httpchain-0.4.0}/src/pytest_httpchain/exceptions.py +0 -0
- {pytest_httpchain-0.2.4 → pytest_httpchain-0.4.0}/src/pytest_httpchain/report_formatter.py +0 -0
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
Metadata-Version: 2.4
|
|
2
2
|
Name: pytest-httpchain
|
|
3
|
-
Version: 0.
|
|
3
|
+
Version: 0.4.0
|
|
4
4
|
Summary: pytest plugin for HTTP testing using JSON files
|
|
5
5
|
Keywords: testing,pytest,requests
|
|
6
6
|
Author: Alexander Eresov
|
|
@@ -18,8 +18,8 @@ Requires-Dist: httpx[http2]>=0.27.0
|
|
|
18
18
|
Requires-Dist: pydantic>=2.11.7
|
|
19
19
|
Requires-Dist: pyrate-limiter>=3.0.0
|
|
20
20
|
Requires-Dist: pytest-httpchain-jsonref
|
|
21
|
-
Requires-Dist: pytest-httpchain-mcp
|
|
22
21
|
Requires-Dist: pytest-httpchain-models
|
|
22
|
+
Requires-Dist: pytest-httpchain-templates
|
|
23
23
|
Requires-Dist: pytest-order>=1.3.0
|
|
24
24
|
Requires-Dist: typer>=0.16.0
|
|
25
25
|
Requires-Python: >=3.13, <4.0
|
|
@@ -196,37 +196,51 @@ pip install 'git+https://github.com/aeresov/pytest-httpchain@main'
|
|
|
196
196
|
- Template expressions support list/dict comprehensions. You can limit the maximum comprehension length using `max_comprehension_length` ini option, default value is **50000**.
|
|
197
197
|
- Parallel stage iterations (repeat/foreach) have a safety limit configurable via `max_parallel_iterations` ini option, default value is **10000**.
|
|
198
198
|
|
|
199
|
-
##
|
|
199
|
+
## AI agent support
|
|
200
200
|
|
|
201
|
-
`pytest-httpchain`
|
|
201
|
+
`pytest-httpchain` ships tooling to help AI coding agents (and humans) author and check test scenarios.
|
|
202
202
|
|
|
203
|
-
###
|
|
203
|
+
### Claude Code skill
|
|
204
204
|
|
|
205
|
-
Install the
|
|
205
|
+
Install the authoring skill into your project (or `--global` for personal scope):
|
|
206
206
|
|
|
207
207
|
```bash
|
|
208
|
-
uvx pytest-httpchain install
|
|
208
|
+
uvx pytest-httpchain install
|
|
209
209
|
```
|
|
210
210
|
|
|
211
|
-
|
|
211
|
+
This writes `.claude/skills/pytest-httpchain/SKILL.md` with guidance for writing scenarios.
|
|
212
212
|
|
|
213
|
-
|
|
214
|
-
|
|
215
|
-
|
|
216
|
-
|
|
217
|
-
|
|
218
|
-
|
|
219
|
-
}
|
|
220
|
-
}
|
|
221
|
-
}
|
|
213
|
+
### Scenario validation
|
|
214
|
+
|
|
215
|
+
Validate scenario files for structure and common problems — undefined variables, variables referenced before they are saved (data-flow ordering), duplicate stage names, fixture/variable conflicts, no-op `verify` steps, and contradictory body checks:
|
|
216
|
+
|
|
217
|
+
```bash
|
|
218
|
+
uvx pytest-httpchain validate tests/test_login.http.json
|
|
222
219
|
```
|
|
223
220
|
|
|
224
|
-
|
|
221
|
+
Each finding carries a stable diagnostic code (`HTTPCHAINxxx`) and a severity. It exits non-zero when any file is invalid, so it doubles as a CI gate. Use `--format json` for machine-readable output (editor/CI integration):
|
|
222
|
+
|
|
223
|
+
```bash
|
|
224
|
+
uvx pytest-httpchain validate --format json tests/test_login.http.json
|
|
225
|
+
```
|
|
225
226
|
|
|
226
|
-
The
|
|
227
|
+
The same checks also run automatically at **pytest collection time** — semantic errors fail collection and warnings are reported — so `pytest --collect-only` validates every scenario in your suite.
|
|
227
228
|
|
|
228
|
-
|
|
229
|
-
|
|
229
|
+
For deeper, opt-in checks, add `--deep`: it imports your `module:func` references to confirm they resolve, checks their call signatures (including the injected `response` for save/verify functions), and verifies referenced files and schemas exist. Because it imports your code it is never run at collection time; pair it with `--strict` to fail CI on any warning, and `--syspath` to add import roots:
|
|
230
|
+
|
|
231
|
+
```bash
|
|
232
|
+
uvx pytest-httpchain validate --deep --strict tests/test_login.http.json
|
|
233
|
+
```
|
|
234
|
+
|
|
235
|
+
### Editor schema
|
|
236
|
+
|
|
237
|
+
A JSON Schema is published for as-you-type validation and autocomplete. Reference it from your test files:
|
|
238
|
+
|
|
239
|
+
```json
|
|
240
|
+
{
|
|
241
|
+
"$schema": "https://aeresov.github.io/pytest-httpchain/schema/scenario.schema.json"
|
|
242
|
+
}
|
|
243
|
+
```
|
|
230
244
|
|
|
231
245
|
## Documentation
|
|
232
246
|
|
|
@@ -169,37 +169,51 @@ pip install 'git+https://github.com/aeresov/pytest-httpchain@main'
|
|
|
169
169
|
- Template expressions support list/dict comprehensions. You can limit the maximum comprehension length using `max_comprehension_length` ini option, default value is **50000**.
|
|
170
170
|
- Parallel stage iterations (repeat/foreach) have a safety limit configurable via `max_parallel_iterations` ini option, default value is **10000**.
|
|
171
171
|
|
|
172
|
-
##
|
|
172
|
+
## AI agent support
|
|
173
173
|
|
|
174
|
-
`pytest-httpchain`
|
|
174
|
+
`pytest-httpchain` ships tooling to help AI coding agents (and humans) author and check test scenarios.
|
|
175
175
|
|
|
176
|
-
###
|
|
176
|
+
### Claude Code skill
|
|
177
177
|
|
|
178
|
-
Install the
|
|
178
|
+
Install the authoring skill into your project (or `--global` for personal scope):
|
|
179
179
|
|
|
180
180
|
```bash
|
|
181
|
-
uvx pytest-httpchain install
|
|
181
|
+
uvx pytest-httpchain install
|
|
182
182
|
```
|
|
183
183
|
|
|
184
|
-
|
|
184
|
+
This writes `.claude/skills/pytest-httpchain/SKILL.md` with guidance for writing scenarios.
|
|
185
185
|
|
|
186
|
-
|
|
187
|
-
|
|
188
|
-
|
|
189
|
-
|
|
190
|
-
|
|
191
|
-
|
|
192
|
-
}
|
|
193
|
-
}
|
|
194
|
-
}
|
|
186
|
+
### Scenario validation
|
|
187
|
+
|
|
188
|
+
Validate scenario files for structure and common problems — undefined variables, variables referenced before they are saved (data-flow ordering), duplicate stage names, fixture/variable conflicts, no-op `verify` steps, and contradictory body checks:
|
|
189
|
+
|
|
190
|
+
```bash
|
|
191
|
+
uvx pytest-httpchain validate tests/test_login.http.json
|
|
195
192
|
```
|
|
196
193
|
|
|
197
|
-
|
|
194
|
+
Each finding carries a stable diagnostic code (`HTTPCHAINxxx`) and a severity. It exits non-zero when any file is invalid, so it doubles as a CI gate. Use `--format json` for machine-readable output (editor/CI integration):
|
|
195
|
+
|
|
196
|
+
```bash
|
|
197
|
+
uvx pytest-httpchain validate --format json tests/test_login.http.json
|
|
198
|
+
```
|
|
198
199
|
|
|
199
|
-
The
|
|
200
|
+
The same checks also run automatically at **pytest collection time** — semantic errors fail collection and warnings are reported — so `pytest --collect-only` validates every scenario in your suite.
|
|
200
201
|
|
|
201
|
-
|
|
202
|
-
|
|
202
|
+
For deeper, opt-in checks, add `--deep`: it imports your `module:func` references to confirm they resolve, checks their call signatures (including the injected `response` for save/verify functions), and verifies referenced files and schemas exist. Because it imports your code it is never run at collection time; pair it with `--strict` to fail CI on any warning, and `--syspath` to add import roots:
|
|
203
|
+
|
|
204
|
+
```bash
|
|
205
|
+
uvx pytest-httpchain validate --deep --strict tests/test_login.http.json
|
|
206
|
+
```
|
|
207
|
+
|
|
208
|
+
### Editor schema
|
|
209
|
+
|
|
210
|
+
A JSON Schema is published for as-you-type validation and autocomplete. Reference it from your test files:
|
|
211
|
+
|
|
212
|
+
```json
|
|
213
|
+
{
|
|
214
|
+
"$schema": "https://aeresov.github.io/pytest-httpchain/schema/scenario.schema.json"
|
|
215
|
+
}
|
|
216
|
+
```
|
|
203
217
|
|
|
204
218
|
## Documentation
|
|
205
219
|
|
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
[project]
|
|
2
2
|
name = "pytest-httpchain"
|
|
3
|
-
version = "0.
|
|
3
|
+
version = "0.4.0"
|
|
4
4
|
description = "pytest plugin for HTTP testing using JSON files"
|
|
5
5
|
readme = "README.md"
|
|
6
6
|
requires-python = ">=3.13,<4.0"
|
|
@@ -10,8 +10,8 @@ dependencies = [
|
|
|
10
10
|
"pydantic>=2.11.7",
|
|
11
11
|
"pyrate-limiter>=3.0.0",
|
|
12
12
|
"pytest-httpchain-jsonref",
|
|
13
|
-
"pytest-httpchain-mcp",
|
|
14
13
|
"pytest-httpchain-models",
|
|
14
|
+
"pytest-httpchain-templates",
|
|
15
15
|
"pytest-order>=1.3.0",
|
|
16
16
|
"typer>=0.16.0",
|
|
17
17
|
]
|
|
@@ -54,7 +54,6 @@ members = ["packages/*"]
|
|
|
54
54
|
pytest-httpchain-core = { workspace = true }
|
|
55
55
|
pytest-httpchain-jsonref = { workspace = true }
|
|
56
56
|
pytest-httpchain-templates = { workspace = true }
|
|
57
|
-
pytest-httpchain-mcp = { workspace = true }
|
|
58
57
|
pytest-httpchain-models = { workspace = true }
|
|
59
58
|
pytest-httpchain-userfunc = { workspace = true }
|
|
60
59
|
|
|
@@ -103,7 +102,6 @@ pythonpath = [
|
|
|
103
102
|
"packages/pytest-httpchain-core/src",
|
|
104
103
|
"packages/pytest-httpchain-jsonref/src",
|
|
105
104
|
"packages/pytest-httpchain-templates/src",
|
|
106
|
-
"packages/pytest-httpchain-mcp/src",
|
|
107
105
|
"packages/pytest-httpchain-models/src",
|
|
108
106
|
"packages/pytest-httpchain-userfunc/src",
|
|
109
107
|
]
|
|
@@ -124,7 +122,6 @@ source = [
|
|
|
124
122
|
"packages/pytest-httpchain-core/src",
|
|
125
123
|
"packages/pytest-httpchain-jsonref/src",
|
|
126
124
|
"packages/pytest-httpchain-templates/src",
|
|
127
|
-
"packages/pytest-httpchain-mcp/src",
|
|
128
125
|
"packages/pytest-httpchain-models/src",
|
|
129
126
|
"packages/pytest-httpchain-userfunc/src",
|
|
130
127
|
]
|
|
@@ -180,7 +180,7 @@ class Carrier:
|
|
|
180
180
|
cls.last_request = exc.request
|
|
181
181
|
if exc.response is not None:
|
|
182
182
|
cls.last_response = exc.response
|
|
183
|
-
raise StageExecutionError(f"Parallel execution failed at iteration {idx}: {exc}")
|
|
183
|
+
raise StageExecutionError(f"Parallel execution failed at iteration {idx}: {exc}") from exc
|
|
184
184
|
|
|
185
185
|
except (
|
|
186
186
|
TemplatesError,
|
|
@@ -199,7 +199,7 @@ class Carrier:
|
|
|
199
199
|
"method": request_model.method,
|
|
200
200
|
"url": str(request_model.url),
|
|
201
201
|
"headers": request_model.headers,
|
|
202
|
-
"params": request_model.params,
|
|
202
|
+
"params": request_model.params or None,
|
|
203
203
|
"timeout": request_model.timeout,
|
|
204
204
|
"follow_redirects": request_model.allow_redirects,
|
|
205
205
|
}
|
|
@@ -209,7 +209,7 @@ class Carrier:
|
|
|
209
209
|
auth_result = call_user_function(request_model.auth)
|
|
210
210
|
request_kwargs["auth"] = auth_result
|
|
211
211
|
except UserFunctionError as e:
|
|
212
|
-
raise RequestError(f"Failed to configure authentication: {
|
|
212
|
+
raise RequestError(f"Failed to configure authentication: {e}") from None
|
|
213
213
|
|
|
214
214
|
match request_model.body:
|
|
215
215
|
case None:
|
|
@@ -233,19 +233,16 @@ class Carrier:
|
|
|
233
233
|
|
|
234
234
|
case BinaryBody(binary=file_path):
|
|
235
235
|
try:
|
|
236
|
-
|
|
237
|
-
binary_data = f.read()
|
|
238
|
-
request_kwargs["content"] = binary_data
|
|
236
|
+
request_kwargs["content"] = Path(file_path).read_bytes()
|
|
239
237
|
except FileNotFoundError:
|
|
240
238
|
raise RequestError(f"Binary file not found: {file_path}") from None
|
|
241
239
|
|
|
242
240
|
case FilesBody(files=file_paths):
|
|
243
241
|
files_list = []
|
|
244
242
|
for field_name, file_path in file_paths.items():
|
|
243
|
+
path = Path(file_path)
|
|
245
244
|
try:
|
|
246
|
-
|
|
247
|
-
file_content = f.read()
|
|
248
|
-
files_list.append((field_name, (Path(file_path).name, file_content)))
|
|
245
|
+
files_list.append((field_name, (path.name, path.read_bytes())))
|
|
249
246
|
except FileNotFoundError:
|
|
250
247
|
raise RequestError(f"File not found for upload: {file_path}") from None
|
|
251
248
|
request_kwargs["files"] = files_list
|
|
@@ -257,13 +254,13 @@ class Carrier:
|
|
|
257
254
|
try:
|
|
258
255
|
return cls.client.request(**request_kwargs)
|
|
259
256
|
except httpx.TimeoutException as e:
|
|
260
|
-
raise RequestError(f"HTTP request timed out: {
|
|
257
|
+
raise RequestError(f"HTTP request timed out: {e}") from None
|
|
261
258
|
except httpx.ConnectError as e:
|
|
262
|
-
raise RequestError(f"HTTP connection error: {
|
|
259
|
+
raise RequestError(f"HTTP connection error: {e}") from None
|
|
263
260
|
except httpx.HTTPError as e:
|
|
264
|
-
raise RequestError(f"HTTP request failed: {
|
|
261
|
+
raise RequestError(f"HTTP request failed: {e}") from None
|
|
265
262
|
except Exception as e:
|
|
266
|
-
raise RequestError(f"Unexpected error: {
|
|
263
|
+
raise RequestError(f"Unexpected error: {e}") from None
|
|
267
264
|
|
|
268
265
|
@staticmethod
|
|
269
266
|
def _process_save_step(save_model: Save, response: httpx.Response, context: ChainMap[str, Any]) -> dict[str, Any]:
|
|
@@ -274,21 +271,21 @@ class Carrier:
|
|
|
274
271
|
try:
|
|
275
272
|
response_json = response.json()
|
|
276
273
|
except (json.JSONDecodeError, UnicodeDecodeError) as e:
|
|
277
|
-
raise SaveError(f"Cannot extract variables, response is not valid JSON: {
|
|
274
|
+
raise SaveError(f"Cannot extract variables, response is not valid JSON: {e}") from None
|
|
278
275
|
|
|
279
276
|
for var_name, jmespath_expr in save_model.jmespath.items():
|
|
280
277
|
try:
|
|
281
278
|
saved_value = jmespath.search(jmespath_expr, response_json)
|
|
282
279
|
step_saved[var_name] = saved_value
|
|
283
280
|
except jmespath.exceptions.JMESPathError as e:
|
|
284
|
-
raise SaveError(f"Error saving variable {var_name}: {
|
|
281
|
+
raise SaveError(f"Error saving variable {var_name}: {e}") from None
|
|
285
282
|
|
|
286
283
|
case SubstitutionsSave():
|
|
287
284
|
try:
|
|
288
285
|
substitution_result = process_substitutions(save_model.substitutions, context)
|
|
289
286
|
step_saved.update(substitution_result)
|
|
290
287
|
except TemplatesError as e:
|
|
291
|
-
raise SaveError(f"Error processing substitutions: {
|
|
288
|
+
raise SaveError(f"Error processing substitutions: {e}") from None
|
|
292
289
|
|
|
293
290
|
case UserFunctionsSave():
|
|
294
291
|
for func_item in save_model.user_functions:
|
|
@@ -298,12 +295,11 @@ class Carrier:
|
|
|
298
295
|
if not isinstance(func_result, dict):
|
|
299
296
|
raise SaveError(f"Save function must return dict, got {type(func_result).__name__}")
|
|
300
297
|
|
|
301
|
-
|
|
302
|
-
step_saved.update(result_dict)
|
|
298
|
+
step_saved.update(func_result)
|
|
303
299
|
except SaveError:
|
|
304
300
|
raise
|
|
305
301
|
except UserFunctionError as e:
|
|
306
|
-
raise SaveError(f"Error calling user function '{func_item}': {
|
|
302
|
+
raise SaveError(f"Error calling user function '{func_item}': {e}") from None
|
|
307
303
|
|
|
308
304
|
return step_saved
|
|
309
305
|
|
|
@@ -333,7 +329,7 @@ class Carrier:
|
|
|
333
329
|
except VerificationError:
|
|
334
330
|
raise
|
|
335
331
|
except UserFunctionError as e:
|
|
336
|
-
raise VerificationError(f"Error calling user function '{func_item}': {
|
|
332
|
+
raise VerificationError(f"Error calling user function '{func_item}': {e}") from None
|
|
337
333
|
|
|
338
334
|
if verify_model.body.schema:
|
|
339
335
|
schema = verify_model.body.schema
|
|
@@ -343,21 +339,21 @@ class Carrier:
|
|
|
343
339
|
schema = json.loads(schema_path.read_text())
|
|
344
340
|
check_json_schema(schema)
|
|
345
341
|
except (OSError, json.JSONDecodeError) as e:
|
|
346
|
-
raise VerificationError(f"Error reading body schema file '{schema_path}': {
|
|
342
|
+
raise VerificationError(f"Error reading body schema file '{schema_path}': {e}") from None
|
|
347
343
|
except jsonschema.SchemaError as e:
|
|
348
344
|
raise VerificationError(f"Invalid JSON Schema in file '{schema_path}': {e}") from None
|
|
349
345
|
|
|
350
346
|
try:
|
|
351
347
|
response_json = response.json()
|
|
352
348
|
except (json.JSONDecodeError, UnicodeDecodeError) as e:
|
|
353
|
-
raise VerificationError(f"Cannot validate schema, response is not valid JSON: {
|
|
349
|
+
raise VerificationError(f"Cannot validate schema, response is not valid JSON: {e}") from None
|
|
354
350
|
|
|
355
351
|
try:
|
|
356
352
|
jsonschema.validate(instance=response_json, schema=schema)
|
|
357
353
|
except jsonschema.ValidationError as e:
|
|
358
|
-
raise VerificationError(f"Body schema validation failed: {
|
|
354
|
+
raise VerificationError(f"Body schema validation failed: {e}") from None
|
|
359
355
|
except jsonschema.SchemaError as e:
|
|
360
|
-
raise VerificationError(f"Invalid body validation schema: {
|
|
356
|
+
raise VerificationError(f"Invalid body validation schema: {e}") from None
|
|
361
357
|
|
|
362
358
|
for substring in verify_model.body.contains:
|
|
363
359
|
if substring not in response.text:
|
|
@@ -438,7 +434,7 @@ class Carrier:
|
|
|
438
434
|
try:
|
|
439
435
|
ctx.__exit__(None, None, None)
|
|
440
436
|
except Exception as e:
|
|
441
|
-
logger.error(f"Error while cleaning up context manager fixture: {
|
|
437
|
+
logger.error(f"Error while cleaning up context manager fixture: {e}")
|
|
442
438
|
|
|
443
439
|
if cls.client is not None:
|
|
444
440
|
cls.client.close()
|
|
@@ -0,0 +1,83 @@
|
|
|
1
|
+
import enum
|
|
2
|
+
import json
|
|
3
|
+
from pathlib import Path
|
|
4
|
+
from typing import Annotated
|
|
5
|
+
|
|
6
|
+
import typer
|
|
7
|
+
|
|
8
|
+
app = typer.Typer()
|
|
9
|
+
|
|
10
|
+
SKILL_FILE = Path(__file__).parent / "skill.md"
|
|
11
|
+
|
|
12
|
+
|
|
13
|
+
class OutputFormat(enum.StrEnum):
|
|
14
|
+
text = "text"
|
|
15
|
+
json = "json"
|
|
16
|
+
|
|
17
|
+
|
|
18
|
+
@app.command()
|
|
19
|
+
def validate(
|
|
20
|
+
paths: Annotated[list[Path], typer.Argument(help="Scenario JSON file(s) to validate.")],
|
|
21
|
+
ref_parent_traversal_depth: Annotated[int, typer.Option(help="Maximum $ref parent directory traversal depth.")] = 3,
|
|
22
|
+
output_format: Annotated[OutputFormat, typer.Option("--format", help="Output format: human-readable text or machine-readable JSON.")] = OutputFormat.text,
|
|
23
|
+
deep: Annotated[bool, typer.Option("--deep", help="Run deep checks: resolve user-function imports/signatures and referenced files. Imports user modules.")] = False,
|
|
24
|
+
syspath: Annotated[list[Path] | None, typer.Option("--syspath", help="Extra directories to add to sys.path for --deep import resolution (repeatable).")] = None,
|
|
25
|
+
strict: Annotated[bool, typer.Option("--strict", help="Treat warnings as failures for the exit code.")] = False,
|
|
26
|
+
) -> None:
|
|
27
|
+
"""Validate pytest-httpchain scenario file(s).
|
|
28
|
+
|
|
29
|
+
Reports errors and warnings (each with a stable HTTPCHAINxxx diagnostic code)
|
|
30
|
+
per file and exits non-zero if any file is invalid (or, with --strict, has any
|
|
31
|
+
warnings).
|
|
32
|
+
"""
|
|
33
|
+
from pytest_httpchain.validation import validate_scenario
|
|
34
|
+
|
|
35
|
+
results = [(path, validate_scenario(path, ref_parent_traversal_depth=ref_parent_traversal_depth, deep=deep, syspaths=list(syspath or []))) for path in paths]
|
|
36
|
+
|
|
37
|
+
def passed(result) -> bool:
|
|
38
|
+
return result.valid and not (strict and result.warnings)
|
|
39
|
+
|
|
40
|
+
all_passed = all(passed(result) for _, result in results)
|
|
41
|
+
|
|
42
|
+
if output_format is OutputFormat.json:
|
|
43
|
+
payload = {
|
|
44
|
+
"valid": all_passed,
|
|
45
|
+
"files": [{"path": str(path), "result": result.model_dump()} for path, result in results],
|
|
46
|
+
}
|
|
47
|
+
typer.echo(json.dumps(payload, indent=2, default=str))
|
|
48
|
+
else:
|
|
49
|
+
for path, result in results:
|
|
50
|
+
if not result.valid:
|
|
51
|
+
status = "INVALID"
|
|
52
|
+
elif result.warnings:
|
|
53
|
+
status = "FAILED (warnings)" if strict else "OK with warnings"
|
|
54
|
+
else:
|
|
55
|
+
status = "OK"
|
|
56
|
+
typer.echo(f"{path}: {status}")
|
|
57
|
+
for diagnostic in result.diagnostics:
|
|
58
|
+
typer.echo(f" {diagnostic.severity} [{diagnostic.code}]: {diagnostic.message}")
|
|
59
|
+
|
|
60
|
+
raise typer.Exit(0 if all_passed else 1)
|
|
61
|
+
|
|
62
|
+
|
|
63
|
+
@app.command()
|
|
64
|
+
def install(
|
|
65
|
+
global_: Annotated[bool, typer.Option("--global", "-g", help="Install to ~/.claude (personal scope) instead of project")] = False,
|
|
66
|
+
project_dir: Annotated[Path, typer.Option(help="Project directory (ignored with --global)")] = Path("."),
|
|
67
|
+
) -> None:
|
|
68
|
+
"""Install the Claude Code skill for authoring test scenarios."""
|
|
69
|
+
if global_:
|
|
70
|
+
_install_skill(Path.home() / ".claude" / "skills" / "pytest-httpchain")
|
|
71
|
+
else:
|
|
72
|
+
_install_skill(project_dir.resolve() / ".claude" / "skills" / "pytest-httpchain")
|
|
73
|
+
|
|
74
|
+
|
|
75
|
+
def _install_skill(skill_dir: Path) -> None:
|
|
76
|
+
skill_dir.mkdir(parents=True, exist_ok=True)
|
|
77
|
+
dest = skill_dir / "SKILL.md"
|
|
78
|
+
dest.write_text(SKILL_FILE.read_text())
|
|
79
|
+
typer.echo(f"Installed skill to {dest}")
|
|
80
|
+
|
|
81
|
+
|
|
82
|
+
if __name__ == "__main__":
|
|
83
|
+
app()
|
|
@@ -238,7 +238,6 @@ def write_har_file(
|
|
|
238
238
|
entry = request_response_to_har_entry(request, response, started_datetime, elapsed_ms)
|
|
239
239
|
har = create_har_log([entry], comment=f"Test: {test_name}")
|
|
240
240
|
|
|
241
|
-
|
|
242
|
-
json.dump(har, f, indent=2, ensure_ascii=False)
|
|
241
|
+
filepath.write_text(json.dumps(har, indent=2, ensure_ascii=False), encoding="utf-8")
|
|
243
242
|
|
|
244
243
|
return filepath
|
|
@@ -1,6 +1,7 @@
|
|
|
1
1
|
import logging
|
|
2
2
|
import re
|
|
3
3
|
import types
|
|
4
|
+
import warnings
|
|
4
5
|
from collections.abc import Iterable
|
|
5
6
|
from pathlib import Path
|
|
6
7
|
from typing import Any
|
|
@@ -19,10 +20,15 @@ from .carrier import Carrier, create_test_class
|
|
|
19
20
|
from .har_writer import write_har_file
|
|
20
21
|
from .report_formatter import format_request, format_response
|
|
21
22
|
from .utils import make_marker
|
|
23
|
+
from .validation import check_scenario
|
|
22
24
|
|
|
23
25
|
logger = logging.getLogger(__name__)
|
|
24
26
|
|
|
25
27
|
|
|
28
|
+
class ScenarioValidationWarning(pytest.PytestWarning):
|
|
29
|
+
"""A collected scenario has a non-fatal validation issue (e.g. an undefined variable)."""
|
|
30
|
+
|
|
31
|
+
|
|
26
32
|
class JsonModule(python.Module):
|
|
27
33
|
"""JSON test module that collects and executes HTTP chain tests.
|
|
28
34
|
|
|
@@ -42,9 +48,9 @@ class JsonModule(python.Module):
|
|
|
42
48
|
root_path=root_path,
|
|
43
49
|
)
|
|
44
50
|
except pytest_httpchain_jsonref.ReferenceResolverError as e:
|
|
45
|
-
raise nodes.Collector.CollectError(f"Cannot load JSON file {self.path}: {
|
|
51
|
+
raise nodes.Collector.CollectError(f"Cannot load JSON file {self.path}: {e}") from None
|
|
46
52
|
except Exception as e:
|
|
47
|
-
raise nodes.Collector.CollectError(f"Failed to parse JSON file {self.path}: {
|
|
53
|
+
raise nodes.Collector.CollectError(f"Failed to parse JSON file {self.path}: {e}") from None
|
|
48
54
|
|
|
49
55
|
# validate general scenario structure
|
|
50
56
|
try:
|
|
@@ -59,6 +65,18 @@ class JsonModule(python.Module):
|
|
|
59
65
|
full_error_msg = f"Cannot parse test scenario in {self.path}:\n" + "\n".join(error_details)
|
|
60
66
|
raise nodes.Collector.CollectError(full_error_msg) from None
|
|
61
67
|
|
|
68
|
+
# semantic validation: cross-cutting checks the schema cannot express
|
|
69
|
+
# (duplicate stage names, fixture/variable conflicts, undefined/forward-referenced
|
|
70
|
+
# variables, no-op verify, contradictory body checks, ...)
|
|
71
|
+
diagnostics, _ = check_scenario(scenario, test_data)
|
|
72
|
+
for diagnostic in diagnostics:
|
|
73
|
+
if diagnostic.severity == "warning":
|
|
74
|
+
warnings.warn(ScenarioValidationWarning(f"{self.path}: [{diagnostic.code}] {diagnostic.message}"), stacklevel=2)
|
|
75
|
+
error_diagnostics = [d for d in diagnostics if d.severity == "error"]
|
|
76
|
+
if error_diagnostics:
|
|
77
|
+
detail = "\n".join(f" - [{d.code}] {d.message}" for d in error_diagnostics)
|
|
78
|
+
raise nodes.Collector.CollectError(f"Invalid test scenario in {self.path}:\n{detail}")
|
|
79
|
+
|
|
62
80
|
# generate python test class
|
|
63
81
|
max_parallel_iterations = int(self.config.getini(ConfigOptions.MAX_PARALLEL_ITERATIONS))
|
|
64
82
|
CarrierClass = create_test_class(scenario, self.name, max_parallel_iterations=max_parallel_iterations)
|
|
@@ -82,10 +82,6 @@ Response is a list of verify and save steps, executed in order:
|
|
|
82
82
|
"verify": {
|
|
83
83
|
"status": 200,
|
|
84
84
|
"headers": { "content-type": "application/json" },
|
|
85
|
-
"expressions": [
|
|
86
|
-
"{{ user_count > 0 }}",
|
|
87
|
-
"{{ 'error' not in body }}"
|
|
88
|
-
],
|
|
89
85
|
"body": {
|
|
90
86
|
"schema": { "type": "object", "required": ["id"] },
|
|
91
87
|
"contains": ["expected text"],
|
|
@@ -103,10 +99,22 @@ Response is a list of verify and save steps, executed in order:
|
|
|
103
99
|
"total": "length(items)"
|
|
104
100
|
}
|
|
105
101
|
}
|
|
102
|
+
},
|
|
103
|
+
{
|
|
104
|
+
"verify": {
|
|
105
|
+
"expressions": [
|
|
106
|
+
"{{ total > 0 }}",
|
|
107
|
+
"{{ user_name != '' }}"
|
|
108
|
+
]
|
|
109
|
+
}
|
|
106
110
|
}
|
|
107
111
|
]
|
|
108
112
|
```
|
|
109
113
|
|
|
114
|
+
**Important:** `verify.expressions` are `{{ }}` templates evaluated against the **context** (saved variables, fixtures, substitutions). The HTTP response is **not** ambient in templates — there is no `response`/`status_code`/`body`/`json` variable. To assert on response data, either:
|
|
115
|
+
- use `verify.status`, `verify.headers`, `verify.body` (these check the response directly), or
|
|
116
|
+
- `save` the value first (e.g. via `jmespath`) and reference the saved variable in a later `expressions` step (as shown above).
|
|
117
|
+
|
|
110
118
|
**Save types:**
|
|
111
119
|
- `{"jmespath": {...}}` - extract values from JSON response via JMESPath
|
|
112
120
|
- `{"substitutions": [...]}` - compute values using template expressions
|
|
@@ -229,12 +237,9 @@ Or iterate over parameter sets in parallel:
|
|
|
229
237
|
"url": "{{ base }}/users/{{ user_id }}"
|
|
230
238
|
},
|
|
231
239
|
"response": [
|
|
232
|
-
{
|
|
233
|
-
|
|
234
|
-
|
|
235
|
-
"expressions": ["{{ body.name == 'Alice' }}"]
|
|
236
|
-
}
|
|
237
|
-
}
|
|
240
|
+
{ "verify": { "status": 200 } },
|
|
241
|
+
{ "save": { "jmespath": { "name": "name" } } },
|
|
242
|
+
{ "verify": { "expressions": ["{{ name == 'Alice' }}"] } }
|
|
238
243
|
]
|
|
239
244
|
},
|
|
240
245
|
{
|
|
@@ -250,3 +255,28 @@ Or iterate over parameter sets in parallel:
|
|
|
250
255
|
]
|
|
251
256
|
}
|
|
252
257
|
```
|
|
258
|
+
|
|
259
|
+
## Validate your scenario
|
|
260
|
+
|
|
261
|
+
After writing a scenario, validate it (no server or network needed):
|
|
262
|
+
|
|
263
|
+
```bash
|
|
264
|
+
pytest-httpchain validate test_<name>.http.json
|
|
265
|
+
```
|
|
266
|
+
|
|
267
|
+
It checks structure plus semantics a JSON Schema cannot, each with a stable `HTTPCHAINxxx` code:
|
|
268
|
+
|
|
269
|
+
- `HTTPCHAIN003` — a `{{ var }}` that is never defined, saved, or provided as a fixture (likely a typo).
|
|
270
|
+
- `HTTPCHAIN004` — a variable used **before** the stage that saves it, or used in a stage's request when it is only saved in that same stage's response. Remember: a value `save`d in a stage's response is available to *later* response steps and *later* stages, never to the request that produced it.
|
|
271
|
+
- `HTTPCHAIN006` — a `verify` step that asserts nothing.
|
|
272
|
+
- `HTTPCHAIN007` / `HTTPCHAIN008` — body `contains`/`not_contains` (or `matches`/`not_matches`) that list the same value, which can never pass.
|
|
273
|
+
|
|
274
|
+
Add `--format json` for machine-readable output. The same checks run automatically during `pytest --collect-only`.
|
|
275
|
+
|
|
276
|
+
For a deeper check that imports your `module:func` references (confirming they resolve and their signatures match — including the injected `response` for save/verify functions) and verifies referenced files/schemas exist, add `--deep` (optionally `--syspath <dir>` for import roots, `--strict` to fail on warnings):
|
|
277
|
+
|
|
278
|
+
```bash
|
|
279
|
+
pytest-httpchain validate --deep test_<name>.http.json
|
|
280
|
+
```
|
|
281
|
+
|
|
282
|
+
Note: the HTTP response is **not** ambient in `{{ }}` templates — `save` what you need from a response first, then reference the saved variable.
|
|
@@ -31,11 +31,11 @@ def make_marker(mark_str: str) -> pytest.MarkDecorator:
|
|
|
31
31
|
|
|
32
32
|
def process_substitutions(
|
|
33
33
|
substitutions: Sequence[Substitution],
|
|
34
|
-
context: Mapping[str, Any] =
|
|
34
|
+
context: Mapping[str, Any] | None = None,
|
|
35
35
|
) -> dict[str, Any]:
|
|
36
|
-
result = {}
|
|
36
|
+
result: dict[str, Any] = {}
|
|
37
37
|
for step in substitutions:
|
|
38
|
-
current_context = {**context, **result}
|
|
38
|
+
current_context = {**(context or {}), **result}
|
|
39
39
|
match step:
|
|
40
40
|
case FunctionsSubstitution():
|
|
41
41
|
for alias, func_def in step.functions.items():
|