schemathesis 4.0.7__tar.gz → 4.0.9__tar.gz

{schemathesis-4.0.7 → schemathesis-4.0.9}/.github/workflows/release.yml

@@ -32,6 +32,22 @@ jobs:
           fi
         shell: bash
 
+
+      - name: Generate stable tag
+        id: stable-tag
+        run: |
+          if [ "${{ env.is_prerelease }}" == "false" ]; then
+            if [ "${{ matrix.tag-suffix }}" == "" ]; then
+              echo "stable_tag=type=raw,value=stable" >> $GITHUB_OUTPUT
+            elif [ "${{ matrix.tag-suffix }}" == "-bookworm" ]; then
+              echo "stable_tag=type=raw,value=stable-bookworm" >> $GITHUB_OUTPUT
+            else
+              echo "stable_tag=" >> $GITHUB_OUTPUT
+            fi
+          else
+            echo "stable_tag=" >> $GITHUB_OUTPUT
+          fi
+
       - name: Docker meta
         id: meta
         uses: docker/metadata-action@v5
@@ -44,7 +60,13 @@ jobs:
           tags: |
             type=semver,pattern={{raw}}${{ matrix.tag-suffix }}
             type=semver,pattern={{version}}${{ matrix.tag-suffix }}
-            ${{ env.is_prerelease == 'false' && 'type=raw,value=stable${{ matrix.tag-suffix }}' || '' }}
+            ${{ steps.stable-tag.outputs.stable_tag }}
+
+      - name: Build metadata
+        run: |
+          echo "Tags: ${{ steps.meta.outputs.tags }}"
+          echo "Is prerelease: ${{ env.is_prerelease }}"
+          echo "Tag suffix: '${{ matrix.tag-suffix }}'"
 
       - name: Set up QEMU
         uses: docker/setup-qemu-action@v3

{schemathesis-4.0.7 → schemathesis-4.0.9}/CHANGELOG.md

@@ -1,6 +1,27 @@
 # Changelog
 
-## [Unreleased](https://github.com/schemathesis/schemathesis/compare/v4.0.7...HEAD) - TBD
+## [Unreleased](https://github.com/schemathesis/schemathesis/compare/v4.0.9...HEAD) - TBD
+
+## [4.0.9](https://github.com/schemathesis/schemathesis/compare/v4.0.8...v4.0.9) - 2025-07-13
+
+### :bug: Fixed
+
+- Do not generate negative test cases during the coverage phase if the original schema accepts any value.
+
+### :wrench: Changed
+
+- Do not display API probing result in the test phases summary.
+- Do not display API probing errors in CLI output. Treat errors as a missing capability.
+- Display cURL code samples on more network errors. [#2940](https://github.com/schemathesis/schemathesis/issues/2940) 
+
+## [4.0.8](https://github.com/schemathesis/schemathesis/compare/v4.0.7...v4.0.8) - 2025-07-12
+
+### :bug: Fixed
+
+- Use `utf-8` encoding when generating JUnit reports.
+- Properly display Schemathesis version in CLI.
+- Test case metadata containing incorrect info about generated components causing false positive failures in `negative_data_rejection` and `positive_data_acceptance` checks. [#2900](https://github.com/schemathesis/schemathesis/issues/2900)
+- Negative test cases with invalid query parameters mistakenly becoming positive test cases.
 
 ## [4.0.7](https://github.com/schemathesis/schemathesis/compare/v4.0.6...v4.0.7) - 2025-07-08
 
@@ -30,7 +51,7 @@
 
 ### :bug: Fixed
 
-- Ignoring some string formats during positive test case generation in the coverage phase. [#2930](https://github.com/schemathesis/schemathesis/issues/2930) 
+- Ignoring some string formats during positive test case generation in the coverage phase. [#2930](https://github.com/schemathesis/schemathesis/issues/2930)
 
 ## [4.0.4](https://github.com/schemathesis/schemathesis/compare/v4.0.3...v4.0.4) - 2025-07-02
 

{schemathesis-4.0.7 → schemathesis-4.0.9}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: schemathesis
-Version: 4.0.7
+Version: 4.0.9
 Summary: Property-based testing framework for Open API and GraphQL based apps
 Project-URL: Documentation, https://schemathesis.readthedocs.io/en/stable/
 Project-URL: Changelog, https://github.com/schemathesis/schemathesis/blob/master/CHANGELOG.md

{schemathesis-4.0.7 → schemathesis-4.0.9}/docs/faq.md

@@ -5,10 +5,10 @@
 Schemathesis generates three types of data:
 
 - **Schema examples** from your API documentation
-- **Valid test data** that follows schema constraints  
+- **Valid test data** that follows schema constraints
 - **Invalid test data** that deliberately breaks constraints
 
-The data covers all JSON Schema types for OpenAPI and valid queries for GraphQL. 
+The data covers all JSON Schema types for OpenAPI and valid queries for GraphQL.
 
 Note, that some generated data may be rejected by your API if the validation rules are not expressed in your schema.
 
@@ -16,22 +16,27 @@ Note, that some generated data may be rejected by your API if the validation rul
 
 Schemathesis identifies problems in three main categories:
 
-**API Contract Violations**
+**Schema Compliance Issues**
 
-- Responses not matching documented schemas
+- Response bodies not matching schemas
 - Undocumented status codes
-- Missing or incorrect content types
+- Missing required headers
+- Wrong content types
 
 **Implementation Flaws**
 
-- Server errors (5xx responses)
-- Data validation issues (accepting invalid data or rejecting valid data)
+- Server crashes (5xx responses)
+- Accepting invalid data
+- Rejecting valid data
+- Allowing missing required headers
 - Authentication bypasses
 
 **Stateful Behavior Issues**
 
-- Resources accessible after deletion
-- Resources unavailable after creation
+- Deleted resources still accessible
+- Created resources not available
+
+See more details in the [Checks reference](reference/checks.md).
 
 ## How should I run Schemathesis?
 

schemathesis-4.0.9/docs/reference/checks.md

@@ -0,0 +1,229 @@
+# Checks
+
+Schemathesis automatically validates API responses using various checks. Each check targets specific aspects of API behavior and its compliance to the specification.
+
+## Core Response Validation
+
+These checks validate that API responses conform to your API schema.
+
+### `not_a_server_error`
+
+Catches server-side errors and implementation issues
+
+**Examples** 
+
+- `500 Internal Server Error`
+- `502 Bad Gateway` 
+- GraphQL response with `"errors": [{...}]`
+
+---
+
+### `status_code_conformance`
+
+Verifies response status codes match schema documentation
+
+**Examples**
+
+- Schema documents `200, 404` but API returns `403`
+- Schema documents `201` but API returns `200`
+
+---
+
+### `content_type_conformance`
+
+Validates `Content-Type` header matches schema
+
+**Examples**
+
+- Schema specifies `application/json` but API returns `text/html`
+- Schema specifies `application/xml` but API returns `application/json`
+
+---
+
+### `response_headers_conformance`
+
+Ensures required response headers are present and valid
+
+**Example**
+
+Schema specifies response must include `X-RateLimit-Remaining` header:
+
+```http
+GET /api/data
+
+Response: 200 OK
+Content-Type: application/json
+# Missing required X-RateLimit-Remaining header ❌
+
+{"data": [...]}
+```
+
+---
+
+### `response_schema_conformance`
+
+Validates response body against JSON Schema
+
+**Example**
+
+Schema defines user object as:
+
+```yaml
+type: object
+required: [id, name, email]
+properties:
+  id: {type: integer}
+  name: {type: string}
+  email: {type: string, format: email}
+```
+
+But API returns:
+
+```http
+GET /users/789
+
+Response: 200 OK
+{
+  "id": "789",          ❌ Should be integer, not string
+  "name": "Eve"         ❌ Missing required 'email' field
+}
+```
+
+## Input Handling Validation
+
+These checks verify how your API processes different types of request data.
+
+### `negative_data_rejection`
+
+Verifies API rejects invalid data with appropriate error responses
+
+**Example**
+
+Schema specifies `age` must be an integer:
+
+```http
+POST /users
+Content-Type: application/json
+
+{"name": "Charlie", "age": "twenty-five"}
+
+Response: 200 OK  ❌ Should return 400 Bad Request
+{"id": 456, "name": "Charlie", "age": null}
+```
+
+---
+
+### `positive_data_acceptance`
+
+Verifies API accepts valid request data
+
+**Example**
+
+Request perfectly matches schema:
+
+```http
+POST /users
+Content-Type: application/json
+
+{"name": "Diana", "age": 25, "email": "diana@example.com"}
+
+Response: 400 Bad Request  ❌ Should accept valid data
+{"error": "Invalid request"}
+```
+
+---
+
+### `missing_required_header`
+
+Verifies APIs return 4xx for missing required headers
+
+**Example**
+
+Schema requires `X-API-Key` header, but:
+
+```http
+GET /protected-resource
+# Missing required X-API-Key header
+
+Response: 200 OK  ❌ Should return 400 Bad Request or 401 Unauthorized
+```
+
+---
+
+### `unsupported_method`
+
+Verifies APIs return 405 for undocumented HTTP methods
+
+**Example**
+
+Schema only documents `GET /users/{id}` and `PUT /users/{id}`, but:
+
+```http
+PATCH /users/123
+Content-Type: application/json
+
+{"name": "Bob"}
+
+Response: 200 OK  ❌ Should return 405 Method Not Allowed
+```
+
+## Stateful Behavior
+
+These checks test API behavior across sequences of operations.
+
+### `use_after_free`
+
+Detects when deleted resources remain accessible
+
+**Example**
+
+```http
+DELETE /users/123
+
+Response: 204 No Content
+```
+
+Then:
+
+```http
+GET /users/123
+
+Response: 200 OK  ❌ Should return 404 Not Found
+```
+
+---
+
+### `ensure_resource_availability`
+
+Verifies created resources are immediately accessible
+
+**Example**
+
+```http
+POST /users
+Content-Type: application/json
+
+{"name": "Alice", "email": "alice@example.com"}
+
+Response: 201 Created
+Location: /users/123
+```
+
+Then immediately:
+
+```http
+GET /users/123
+
+Response: 404 Not Found  ❌ Should return the created user
+```
+
+## Security
+
+### `ignored_auth`
+
+Verifies authentication is properly enforced
+
+**Example**
+
+- Endpoint requires authentication but accepts requests without auth headers
+- Returns `200 OK` instead of `401 Unauthorized`

schemathesis-4.0.9/docs/reference/hooks.md

@@ -0,0 +1,80 @@
+# Hooks Reference
+
+## Schema-Level Hooks
+
+Hooks that execute during schema loading and operation initialization.
+
+| Hook | Signature | Execution Order | Modifies |
+|------|-----------|-----------------|----------|
+| **before_load_schema** | `(ctx, raw_schema: dict) -> None` | Before schema parsing | `raw_schema` dict |
+| **after_load_schema** | `(ctx, schema: BaseSchema) -> None` | After schema instance created | `schema` object |
+| **before_process_path** | `(ctx, path: str, methods: dict) -> None` | For each API path | `methods` dict |
+| **before_init_operation** | `(ctx, operation: APIOperation) -> None` | For each operation | `APIOperation` object |
+| **before_add_examples** | `(ctx, examples: list[Case]) -> None` | Before scheduling all existing examples for testing | `Case` list |
+
+!!! note
+
+    All schema-level hooks return `None` and modify their arguments in-place.
+
+## Data-Generation Hooks
+
+Control how test data is generated for each request component.
+
+| Hook Pattern | Signature | Purpose | Return |
+|--------------|-----------|---------|---------|
+| **filter_{component}** | `(ctx, value: Any) -> bool` | Exclude generated values | `True` to keep, `False` to reject |
+| **map_{component}** | `(ctx, value: Any) -> Any` | Transform generated values | Modified value |
+| **flatmap_{component}** | `(ctx, value: Any) -> SearchStrategy` | Generate dependent values | New strategy based on input |
+
+Where `{component}` is one of: `path_parameters`, `headers`, `cookies`, `query`, `body`, `case`
+
+## Request-Lifecycle Hooks
+
+Hooks that execute during test case execution.
+
+| Hook | Signature | Execution Stage | Purpose |
+|------|-----------|-----------------|---------|
+| **before_call** | `(ctx, case: Case) -> None` | Before HTTP request | Modify test case (headers, body, etc.) |
+| **process_call_kwargs** | `(ctx, case: Case, kwargs: dict) -> dict` | Before `case.call()` | Modify request library arguments (timeout, verify, etc.) |
+| **after_call** | `(ctx, case: Case, response: Response) -> None` | After HTTP response | Inspect/modify response before checks |
+
+**Flow:** `before_call` → `process_call_kwargs` → HTTP Request → `after_call` → Checks
+
+## Hook Registration
+
+| Scope | Registration Method | Available Hooks |
+|-------|-------------------|-----------------|
+| **Global** | `@schemathesis.hook` or `@schemathesis.hook("hook_name")` | All hooks |
+| **Schema** | `@schema.hooks.hook` or `@schema.hooks.hook("hook_name")` | All hooks except `before_load_schema`, `after_load_schema` |
+| **Test** | `@schema.hooks.apply(hook_func)` | All hooks except `before_load_schema`, `after_load_schema` |
+
+**Hook Name Detection:** If not specified, uses function name (must match hook name exactly).
+
+## Conditional Application
+
+Apply hooks selectively using filters.
+
+### Filter Methods
+
+| Method | Logic | Description |
+|--------|-------|-------------|
+| `.apply_to(...)` | AND | Apply only when ALL conditions match |
+| `.skip_for(...)` | OR | Skip when ANY condition matches |
+
+### Available Filters
+
+| Filter | Type | Example | Description |
+|--------|------|---------|-------------|
+| `path` | `str | list[str]` | `"/users"` | Exact path match |
+| `path_regex` | `str` | `r"^/users/\d+$"` | Path pattern match |
+| `method` | `str | list[str]` | `["GET", "POST"]` | HTTP method |
+| `method_regex` | `str` | `"GET|POST"` | HTTP method pattern match |
+| `name` | `str | list[str]` | `"GET /users"` | Operation name |
+| `name_regex` | `str` | `"(GET|POST) /(users|orders)"` | Operation name pattern match |
+| `tag` | `str | list[str]` | `["users", "admin"]` | OpenAPI tags |
+| `tag_regex` | `str` | `"users|payments"` | OpenAPI tags pattern match |
+| `operation_id` | `str | list[str]` | `"get_user"` | operationId |
+| `operation_id_regex` | `str` | `"(get|list)_user"` | operationId pattern match |
+
+!!! note
+    Filters are not available for `before_process_path`, `before_load_schema`, and `after_load_schema` hooks. 

{schemathesis-4.0.7 → schemathesis-4.0.9}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "schemathesis"
-version = "4.0.7"
+version = "4.0.9"
 description = "Property-based testing framework for Open API and GraphQL based apps"
 keywords = ["pytest", "hypothesis", "openapi", "graphql", "testing"]
 authors = [{ name = "Dmitry Dygalo", email = "dmitry@dygalo.dev" }]

{schemathesis-4.0.7 → schemathesis-4.0.9}/src/schemathesis/cli/commands/run/handlers/junitxml.py

@@ -40,7 +40,7 @@ class JunitXMLHandler(EventHandler):
             test_suites = [
                 TestSuite("schemathesis", test_cases=list(self.test_cases.values()), hostname=platform.node())
             ]
-            with open(self.path, "w") as fd:
+            with open(self.path, "w", encoding="utf-8") as fd:
                 to_xml_report_file(file_descriptor=fd, test_suites=test_suites, prettyprint=True, encoding="utf-8")
 
     def get_or_create_test_case(self, label: str) -> TestCase:

{schemathesis-4.0.7 → schemathesis-4.0.9}/src/schemathesis/cli/commands/run/handlers/output.py

@@ -20,10 +20,9 @@ from schemathesis.config import ProjectConfig, ReportFormat, SchemathesisWarning
 from schemathesis.core.errors import LoaderError, LoaderErrorKind, format_exception, split_traceback
 from schemathesis.core.failures import MessageBlock, Severity, format_failures
 from schemathesis.core.output import prepare_response_payload
-from schemathesis.core.result import Err, Ok
+from schemathesis.core.result import Ok
 from schemathesis.core.version import SCHEMATHESIS_VERSION
 from schemathesis.engine import Status, events
-from schemathesis.engine.errors import EngineErrorInfo
 from schemathesis.engine.phases import PhaseName, PhaseSkipReason
 from schemathesis.engine.phases.probes import ProbeOutcome
 from schemathesis.engine.recorder import Interaction, ScenarioRecorder
@@ -960,7 +959,9 @@ class OutputHandler(EventHandler):
                     table.add_row(f"{probe_run.probe.name}:", Text(icon, style=style))
 
                 message = Padding(table, BLOCK_PADDING)
-            elif event.status == Status.SKIP:
+            else:
+                assert event.status == Status.SKIP
+                assert isinstance(event.payload, Ok)
                 message = Padding(
                     Text.assemble(
                         ("⏭   ", ""),
@@ -968,17 +969,6 @@ class OutputHandler(EventHandler):
                     ),
                     BLOCK_PADDING,
                 )
-            else:
-                assert event.status == Status.ERROR
-                assert isinstance(event.payload, Err)
-                error = EngineErrorInfo(event.payload.err())
-                message = Padding(
-                    Text.assemble(
-                        ("🚫  ", ""),
-                        (f"API probing failed: {error.message}", Style(color="red")),
-                    ),
-                    BLOCK_PADDING,
-                )
             self.console.print(message)
             self.console.print()
         elif phase.name == PhaseName.STATEFUL_TESTING and phase.is_enabled and self.stateful_tests_manager is not None:
@@ -1337,6 +1327,9 @@ class OutputHandler(EventHandler):
         click.echo(_style("Test Phases:", bold=True))
 
         for phase in PhaseName:
+            if phase == PhaseName.PROBING:
+                # It is not a test phase
+                continue
             status, skip_reason = self.phases[phase]
 
             if status == Status.SKIP:

{schemathesis-4.0.7 → schemathesis-4.0.9}/src/schemathesis/core/version.py

@@ -1,7 +1,7 @@
 from importlib import metadata
 
 try:
-    SCHEMATHESIS_VERSION = metadata.version(__package__)
+    SCHEMATHESIS_VERSION = metadata.version("schemathesis")
 except metadata.PackageNotFoundError:
     # Local run without installation
     SCHEMATHESIS_VERSION = "dev"

{schemathesis-4.0.7 → schemathesis-4.0.9}/src/schemathesis/engine/errors.py

@@ -426,7 +426,15 @@ def is_unrecoverable_network_error(exc: Exception) -> bool:
 
     def has_connection_reset(inner: BaseException) -> bool:
         exc_str = str(inner)
-        if any(pattern in exc_str for pattern in ["Connection reset by peer", "[Errno 104]", "ECONNRESET"]):
+        if any(
+            pattern in exc_str
+            for pattern in [
+                "Connection reset by peer",
+                "[Errno 104]",
+                "ECONNRESET",
+                "An established connection was aborted",
+            ]
+        ):
             return True
 
         if inner.__context__ is not None:
@@ -434,7 +442,7 @@ def is_unrecoverable_network_error(exc: Exception) -> bool:
 
         return False
 
-    if isinstance(exc, requests.Timeout):
+    if isinstance(exc, (requests.Timeout, requests.exceptions.ChunkedEncodingError)):
         return True
     if isinstance(exc.__context__, ProtocolError):
         if len(exc.__context__.args) == 2 and isinstance(exc.__context__.args[1], RemoteDisconnected):

{schemathesis-4.0.7 → schemathesis-4.0.9}/src/schemathesis/engine/phases/probes.py

@@ -13,7 +13,7 @@ import warnings
 from dataclasses import dataclass
 from typing import TYPE_CHECKING
 
-from schemathesis.core.result import Err, Ok, Result
+from schemathesis.core.result import Ok, Result
 from schemathesis.core.transport import USER_AGENT
 from schemathesis.engine import Status, events
 from schemathesis.transport.prepare import get_default_headers
@@ -41,16 +41,11 @@ def execute(ctx: EngineContext, phase: Phase) -> EventGenerator:
     payload: Result[ProbePayload, Exception] | None = None
     for result in probes:
         if isinstance(result.probe, NullByteInHeader) and result.is_failure:
-            from ...specs.openapi import formats
-            from ...specs.openapi.formats import HEADER_FORMAT, header_values
+            from schemathesis.specs.openapi import formats
+            from schemathesis.specs.openapi.formats import HEADER_FORMAT, header_values
 
             formats.register(HEADER_FORMAT, header_values(exclude_characters="\n\r\x00"))
-        if result.error is not None:
-            status = Status.ERROR
-            payload = Err(result.error)
-        else:
-            status = Status.SUCCESS
-            payload = Ok(ProbePayload(probes=probes))
+        payload = Ok(ProbePayload(probes=probes))
     yield events.PhaseFinished(phase=phase, status=status, payload=payload)
 
 

{schemathesis-4.0.7 → schemathesis-4.0.9}/src/schemathesis/generation/coverage.py

@@ -177,6 +177,18 @@ class CoverageContext:
                 return bool(value)
         return True
 
+    def can_be_negated(self, schema: dict[str, Any]) -> bool:
+        # Path, query, header, and cookie parameters will be stringified anyway
+        # If there are no constraints, then anything will match the original schema after serialization
+        if self.location in ("query", "path", "header", "cookie"):
+            cleaned = {
+                k: v
+                for k, v in schema.items()
+                if not k.startswith("x-") and k not in ["description", "example", "examples"]
+            }
+            return cleaned != {}
+        return True
+
     def generate_from(self, strategy: st.SearchStrategy) -> Any:
         return cached_draw(strategy)
 
@@ -389,6 +401,8 @@ def cover_schema_iter(
             yield from _cover_positive_for_type(ctx, schema, ty)
     if GenerationMode.NEGATIVE in ctx.generation_modes:
         template = None
+        if not ctx.can_be_negated(schema):
+            return
         for key, value in schema.items():
             with _ignore_unfixable(), ctx.at(key):
                 if key == "enum":

{schemathesis-4.0.7 → schemathesis-4.0.9}/src/schemathesis/generation/hypothesis/builder.py

@@ -624,7 +624,9 @@ def _iter_coverage_cases(
                 ),
             )
         # Generate duplicate query parameters
-        if generate_duplicate_query_parameters and operation.query:
+        # NOTE: if the query schema has no constraints, then we may have no negative test cases at all
+        # as they all will match the original schema and therefore will be considered as positive ones
+        if generate_duplicate_query_parameters and operation.query and "query" in template:
             container = template["query"]
             for parameter in operation.query:
                 instant = Instant()
@@ -656,24 +658,26 @@ def _iter_coverage_cases(
                 name = parameter.name
                 location = parameter.location
                 container_name = LOCATION_TO_CONTAINER[location]
-                container = template[container_name]
-                data = template.with_container(
-                    container_name=container_name,
-                    value={k: v for k, v in container.items() if k != name},
-                    generation_mode=GenerationMode.NEGATIVE,
-                )
-                yield operation.Case(
-                    **data.kwargs,
-                    _meta=CaseMetadata(
-                        generation=GenerationInfo(time=instant.elapsed, mode=GenerationMode.NEGATIVE),
-                        components=data.components,
-                        phase=PhaseInfo.coverage(
-                            description=f"Missing `{name}` at {location}",
-                            parameter=name,
-                            parameter_location=location,
+                # NOTE: if the schema is overly permissive we may not have any negative test cases
+                if container_name in template:
+                    container = template[container_name]
+                    data = template.with_container(
+                        container_name=container_name,
+                        value={k: v for k, v in container.items() if k != name},
+                        generation_mode=GenerationMode.NEGATIVE,
+                    )
+                    yield operation.Case(
+                        **data.kwargs,
+                        _meta=CaseMetadata(
+                            generation=GenerationInfo(time=instant.elapsed, mode=GenerationMode.NEGATIVE),
+                            components=data.components,
+                            phase=PhaseInfo.coverage(
+                                description=f"Missing `{name}` at {location}",
+                                parameter=name,
+                                parameter_location=location,
+                            ),
                         ),
-                    ),
-                )
+                    )
     # Generate combinations for each location
     for location, parameter_set in [
         ("query", operation.query),