schemathesis 4.0.6__tar.gz → 4.0.8__tar.gz

{schemathesis-4.0.6 → schemathesis-4.0.8}/.github/workflows/release.yml

@@ -32,6 +32,22 @@ jobs:
           fi
         shell: bash
 
+
+      - name: Generate stable tag
+        id: stable-tag
+        run: |
+          if [ "${{ env.is_prerelease }}" == "false" ]; then
+            if [ "${{ matrix.tag-suffix }}" == "" ]; then
+              echo "stable_tag=type=raw,value=stable" >> $GITHUB_OUTPUT
+            elif [ "${{ matrix.tag-suffix }}" == "-bookworm" ]; then
+              echo "stable_tag=type=raw,value=stable-bookworm" >> $GITHUB_OUTPUT
+            else
+              echo "stable_tag=" >> $GITHUB_OUTPUT
+            fi
+          else
+            echo "stable_tag=" >> $GITHUB_OUTPUT
+          fi
+
       - name: Docker meta
         id: meta
         uses: docker/metadata-action@v5
@@ -44,7 +60,13 @@ jobs:
           tags: |
             type=semver,pattern={{raw}}${{ matrix.tag-suffix }}
             type=semver,pattern={{version}}${{ matrix.tag-suffix }}
-            ${{ env.is_prerelease == 'false' && 'type=raw,value=stable${{ matrix.tag-suffix }}' || '' }}
+            ${{ steps.stable-tag.outputs.stable_tag }}
+
+      - name: Build metadata
+        run: |
+          echo "Tags: ${{ steps.meta.outputs.tags }}"
+          echo "Is prerelease: ${{ env.is_prerelease }}"
+          echo "Tag suffix: '${{ matrix.tag-suffix }}'"
 
       - name: Set up QEMU
         uses: docker/setup-qemu-action@v3

{schemathesis-4.0.6 → schemathesis-4.0.8}/CHANGELOG.md

@@ -1,6 +1,25 @@
 # Changelog
 
-## [Unreleased](https://github.com/schemathesis/schemathesis/compare/v4.0.6...HEAD) - TBD
+## [Unreleased](https://github.com/schemathesis/schemathesis/compare/v4.0.8...HEAD) - TBD
+
+## [4.0.8](https://github.com/schemathesis/schemathesis/compare/v4.0.7...v4.0.8) - 2025-07-12
+
+### :bug: Fixed
+
+- Use `utf-8` encoding when generating JUnit reports.
+- Properly display Schemathesis version in CLI.
+- Test case metadata containing incorrect info about generated components causing false positive failures in `negative_data_rejection` and `positive_data_acceptance` checks. [#2900](https://github.com/schemathesis/schemathesis/issues/2900)
+- Negative test cases with invalid query parameters mistakenly becoming positive test cases.
+
+## [4.0.7](https://github.com/schemathesis/schemathesis/compare/v4.0.6...v4.0.7) - 2025-07-08
+
+### :rocket: Added
+
+- Support for `application/yaml` media type.
+
+### :wrench: Changed
+
+- Validate schema of YAML response.
 
 ## [4.0.6](https://github.com/schemathesis/schemathesis/compare/v4.0.5...v4.0.6) - 2025-07-07
 
@@ -10,7 +29,7 @@
 
 ### :wrench: Changed
 
-- Consider connection error as a failed API probe. [#2935](https://github.com/schemathesis/schemathesis/issues/2935)  
+- Consider connection error as a failed API probe. [#2935](https://github.com/schemathesis/schemathesis/issues/2935)
 
 ## [4.0.5](https://github.com/schemathesis/schemathesis/compare/v4.0.4...v4.0.5) - 2025-07-04
 

{schemathesis-4.0.6 → schemathesis-4.0.8}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: schemathesis
-Version: 4.0.6
+Version: 4.0.8
 Summary: Property-based testing framework for Open API and GraphQL based apps
 Project-URL: Documentation, https://schemathesis.readthedocs.io/en/stable/
 Project-URL: Changelog, https://github.com/schemathesis/schemathesis/blob/master/CHANGELOG.md

{schemathesis-4.0.6 → schemathesis-4.0.8}/docs/faq.md

@@ -5,10 +5,10 @@
 Schemathesis generates three types of data:
 
 - **Schema examples** from your API documentation
-- **Valid test data** that follows schema constraints  
+- **Valid test data** that follows schema constraints
 - **Invalid test data** that deliberately breaks constraints
 
-The data covers all JSON Schema types for OpenAPI and valid queries for GraphQL. 
+The data covers all JSON Schema types for OpenAPI and valid queries for GraphQL.
 
 Note, that some generated data may be rejected by your API if the validation rules are not expressed in your schema.
 
@@ -16,22 +16,27 @@ Note, that some generated data may be rejected by your API if the validation rul
 
 Schemathesis identifies problems in three main categories:
 
-**API Contract Violations**
+**Schema Compliance Issues**
 
-- Responses not matching documented schemas
+- Response bodies not matching schemas
 - Undocumented status codes
-- Missing or incorrect content types
+- Missing required headers
+- Wrong content types
 
 **Implementation Flaws**
 
-- Server errors (5xx responses)
-- Data validation issues (accepting invalid data or rejecting valid data)
+- Server crashes (5xx responses)
+- Accepting invalid data
+- Rejecting valid data
+- Allowing missing required headers
 - Authentication bypasses
 
 **Stateful Behavior Issues**
 
-- Resources accessible after deletion
-- Resources unavailable after creation
+- Deleted resources still accessible
+- Created resources not available
+
+See more details in the [Checks reference](reference/checks.md).
 
 ## How should I run Schemathesis?
 

schemathesis-4.0.8/docs/reference/checks.md

@@ -0,0 +1,229 @@
+# Checks
+
+Schemathesis automatically validates API responses using various checks. Each check targets specific aspects of API behavior and its compliance to the specification.
+
+## Core Response Validation
+
+These checks validate that API responses conform to your API schema.
+
+### `not_a_server_error`
+
+Catches server-side errors and implementation issues
+
+**Examples** 
+
+- `500 Internal Server Error`
+- `502 Bad Gateway` 
+- GraphQL response with `"errors": [{...}]`
+
+---
+
+### `status_code_conformance`
+
+Verifies response status codes match schema documentation
+
+**Examples**
+
+- Schema documents `200, 404` but API returns `403`
+- Schema documents `201` but API returns `200`
+
+---
+
+### `content_type_conformance`
+
+Validates `Content-Type` header matches schema
+
+**Examples**
+
+- Schema specifies `application/json` but API returns `text/html`
+- Schema specifies `application/xml` but API returns `application/json`
+
+---
+
+### `response_headers_conformance`
+
+Ensures required response headers are present and valid
+
+**Example**
+
+Schema specifies response must include `X-RateLimit-Remaining` header:
+
+```http
+GET /api/data
+
+Response: 200 OK
+Content-Type: application/json
+# Missing required X-RateLimit-Remaining header ❌
+
+{"data": [...]}
+```
+
+---
+
+### `response_schema_conformance`
+
+Validates response body against JSON Schema
+
+**Example**
+
+Schema defines user object as:
+
+```yaml
+type: object
+required: [id, name, email]
+properties:
+  id: {type: integer}
+  name: {type: string}
+  email: {type: string, format: email}
+```
+
+But API returns:
+
+```http
+GET /users/789
+
+Response: 200 OK
+{
+  "id": "789",          ❌ Should be integer, not string
+  "name": "Eve"         ❌ Missing required 'email' field
+}
+```
+
+## Input Handling Validation
+
+These checks verify how your API processes different types of request data.
+
+### `negative_data_rejection`
+
+Verifies API rejects invalid data with appropriate error responses
+
+**Example**
+
+Schema specifies `age` must be an integer:
+
+```http
+POST /users
+Content-Type: application/json
+
+{"name": "Charlie", "age": "twenty-five"}
+
+Response: 200 OK  ❌ Should return 400 Bad Request
+{"id": 456, "name": "Charlie", "age": null}
+```
+
+---
+
+### `positive_data_acceptance`
+
+Verifies API accepts valid request data
+
+**Example**
+
+Request perfectly matches schema:
+
+```http
+POST /users
+Content-Type: application/json
+
+{"name": "Diana", "age": 25, "email": "diana@example.com"}
+
+Response: 400 Bad Request  ❌ Should accept valid data
+{"error": "Invalid request"}
+```
+
+---
+
+### `missing_required_header`
+
+Verifies APIs return 4xx for missing required headers
+
+**Example**
+
+Schema requires `X-API-Key` header, but:
+
+```http
+GET /protected-resource
+# Missing required X-API-Key header
+
+Response: 200 OK  ❌ Should return 400 Bad Request or 401 Unauthorized
+```
+
+---
+
+### `unsupported_method`
+
+Verifies APIs return 405 for undocumented HTTP methods
+
+**Example**
+
+Schema only documents `GET /users/{id}` and `PUT /users/{id}`, but:
+
+```http
+PATCH /users/123
+Content-Type: application/json
+
+{"name": "Bob"}
+
+Response: 200 OK  ❌ Should return 405 Method Not Allowed
+```
+
+## Stateful Behavior
+
+These checks test API behavior across sequences of operations.
+
+### `use_after_free`
+
+Detects when deleted resources remain accessible
+
+**Example**
+
+```http
+DELETE /users/123
+
+Response: 204 No Content
+```
+
+Then:
+
+```http
+GET /users/123
+
+Response: 200 OK  ❌ Should return 404 Not Found
+```
+
+---
+
+### `ensure_resource_availability`
+
+Verifies created resources are immediately accessible
+
+**Example**
+
+```http
+POST /users
+Content-Type: application/json
+
+{"name": "Alice", "email": "alice@example.com"}
+
+Response: 201 Created
+Location: /users/123
+```
+
+Then immediately:
+
+```http
+GET /users/123
+
+Response: 404 Not Found  ❌ Should return the created user
+```
+
+## Security
+
+### `ignored_auth`
+
+Verifies authentication is properly enforced
+
+**Example**
+
+- Endpoint requires authentication but accepts requests without auth headers
+- Returns `200 OK` instead of `401 Unauthorized`

{schemathesis-4.0.6 → schemathesis-4.0.8}/docs/reference/hooks.md

@@ -10,8 +10,6 @@
 | **before\_init\_operation** | `(context, operation: APIOperation) -> None`  | Adjust an `APIOperation` before building strategies (e.g., restrict enum choices, set defaults). |
 | **before\_add\_examples**   | `(context, examples: list[Case]) -> None`     | Append explicit `Case` instances before the `examples` phase.                              |
 
-## Data-Generation Hooks
-
 ## Request-Lifecycle Hooks
 
 Mutate or inspect the test case and HTTP request flow.

{schemathesis-4.0.6 → schemathesis-4.0.8}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "schemathesis"
-version = "4.0.6"
+version = "4.0.8"
 description = "Property-based testing framework for Open API and GraphQL based apps"
 keywords = ["pytest", "hypothesis", "openapi", "graphql", "testing"]
 authors = [{ name = "Dmitry Dygalo", email = "dmitry@dygalo.dev" }]

{schemathesis-4.0.6 → schemathesis-4.0.8}/src/schemathesis/cli/commands/run/handlers/junitxml.py

@@ -40,7 +40,7 @@ class JunitXMLHandler(EventHandler):
             test_suites = [
                 TestSuite("schemathesis", test_cases=list(self.test_cases.values()), hostname=platform.node())
             ]
-            with open(self.path, "w") as fd:
+            with open(self.path, "w", encoding="utf-8") as fd:
                 to_xml_report_file(file_descriptor=fd, test_suites=test_suites, prettyprint=True, encoding="utf-8")
 
     def get_or_create_test_case(self, label: str) -> TestCase:

{schemathesis-4.0.6 → schemathesis-4.0.8}/src/schemathesis/core/deserialization.py

@@ -4,6 +4,9 @@ import re
 from functools import lru_cache
 from typing import TYPE_CHECKING, Any, BinaryIO, TextIO
 
+from schemathesis.core import media_types
+from schemathesis.core.transport import Response
+
 if TYPE_CHECKING:
     import yaml
 
@@ -63,3 +66,14 @@ def deserialize_yaml(stream: str | bytes | TextIO | BinaryIO) -> Any:
     import yaml
 
     return yaml.load(stream, get_yaml_loader())
+
+
+def deserialize_response(response: Response, content_type: str) -> Any:
+    if media_types.is_yaml(content_type):
+        encoding = response.encoding or "utf-8"
+        return deserialize_yaml(response.content.decode(encoding))
+    if media_types.is_json(content_type):
+        return response.json()
+    raise NotImplementedError(
+        f"Unsupported Content-Type: {content_type!r}. Supported types are: application/yaml, application/json."
+    )

{schemathesis-4.0.6 → schemathesis-4.0.8}/src/schemathesis/core/media_types.py

@@ -55,7 +55,7 @@ def is_json(value: str) -> bool:
 
 def is_yaml(value: str) -> bool:
     """Detect whether the content type is YAML-compatible."""
-    return value in ("text/yaml", "text/x-yaml", "application/x-yaml", "text/vnd.yaml")
+    return value in ("text/yaml", "text/x-yaml", "application/x-yaml", "text/vnd.yaml", "application/yaml")
 
 
 def is_plain_text(value: str) -> bool:

{schemathesis-4.0.6 → schemathesis-4.0.8}/src/schemathesis/core/version.py

@@ -1,7 +1,7 @@
 from importlib import metadata
 
 try:
-    SCHEMATHESIS_VERSION = metadata.version(__package__)
+    SCHEMATHESIS_VERSION = metadata.version("schemathesis")
 except metadata.PackageNotFoundError:
     # Local run without installation
     SCHEMATHESIS_VERSION = "dev"

{schemathesis-4.0.6 → schemathesis-4.0.8}/src/schemathesis/specs/openapi/_hypothesis.py

@@ -14,7 +14,7 @@ from hypothesis_jsonschema import from_schema
 from requests.structures import CaseInsensitiveDict
 
 from schemathesis.config import GenerationConfig
-from schemathesis.core import NOT_SET, NotSet, media_types
+from schemathesis.core import NOT_SET, media_types
 from schemathesis.core.control import SkipTest
 from schemathesis.core.errors import SERIALIZERS_SUGGESTION_MESSAGE, SerializationNotPossible
 from schemathesis.core.transforms import deepclone
@@ -57,10 +57,10 @@ def openapi_cases(
     hooks: HookDispatcher | None = None,
     auth_storage: auths.AuthStorage | None = None,
     generation_mode: GenerationMode = GenerationMode.POSITIVE,
-    path_parameters: NotSet | dict[str, Any] = NOT_SET,
-    headers: NotSet | dict[str, Any] = NOT_SET,
-    cookies: NotSet | dict[str, Any] = NOT_SET,
-    query: NotSet | dict[str, Any] = NOT_SET,
+    path_parameters: dict[str, Any] | None = None,
+    headers: dict[str, Any] | None = None,
+    cookies: dict[str, Any] | None = None,
+    query: dict[str, Any] | None = None,
     body: Any = NOT_SET,
     media_type: str | None = None,
     phase: TestPhase = TestPhase.FUZZING,
@@ -220,7 +220,7 @@ def _get_body_strategy(
 
 
 def get_parameters_value(
-    value: NotSet | dict[str, Any],
+    value: dict[str, Any] | None,
     location: str,
     draw: Callable,
     operation: APIOperation,
@@ -234,7 +234,7 @@ def get_parameters_value(
     If the value is not set, then generate it from the relevant strategy. Otherwise, check what is missing in it and
     generate those parts.
     """
-    if isinstance(value, NotSet) or not value:
+    if value is None:
         strategy = get_parameters_strategy(operation, strategy_factory, location, generation_config)
         strategy = apply_hooks(operation, ctx, hooks, strategy, location)
         return draw(strategy)
@@ -274,7 +274,7 @@ def any_negated_values(values: list[ValueContainer]) -> bool:
 
 def generate_parameter(
     location: str,
-    explicit: NotSet | dict[str, Any],
+    explicit: dict[str, Any] | None,
     operation: APIOperation,
     draw: Callable,
     ctx: HookContext,

{schemathesis-4.0.6 → schemathesis-4.0.8}/src/schemathesis/specs/openapi/negative/mutations.py

@@ -223,6 +223,7 @@ def change_type(context: MutationContext, draw: Draw, schema: Schema) -> Mutatio
     if len(candidates) == 1:
         new_type = candidates.pop()
         schema["type"] = new_type
+        _ensure_query_serializes_to_non_empty(context, schema)
         prevent_unsatisfiable_schema(schema, new_type)
         return MutationResult.SUCCESS
     # Choose one type that will be present in the final candidates list
@@ -235,10 +236,18 @@ def change_type(context: MutationContext, draw: Draw, schema: Schema) -> Mutatio
     ]
     new_type = draw(st.sampled_from(remaining_candidates))
     schema["type"] = new_type
+    _ensure_query_serializes_to_non_empty(context, schema)
     prevent_unsatisfiable_schema(schema, new_type)
     return MutationResult.SUCCESS
 
 
+def _ensure_query_serializes_to_non_empty(context: MutationContext, schema: Schema) -> None:
+    if context.is_query_location and schema.get("type") == "array":
+        # Query parameters with empty arrays or arrays of `None` or empty arrays / objects will not appear in the final URL
+        schema["minItems"] = schema.get("minItems") or 1
+        schema.setdefault("items", {}).update({"not": {"enum": [None, [], {}]}})
+
+
 def _get_type_candidates(context: MutationContext, schema: Schema) -> set[str]:
     types = set(get_type(schema))
     if context.is_path_location:

{schemathesis-4.0.6 → schemathesis-4.0.8}/src/schemathesis/specs/openapi/schemas.py

@@ -29,7 +29,7 @@ from requests.exceptions import InvalidHeader
 from requests.structures import CaseInsensitiveDict
 from requests.utils import check_header_validity
 
-from schemathesis.core import NOT_SET, NotSet, Specification, media_types
+from schemathesis.core import NOT_SET, NotSet, Specification, deserialization, media_types
 from schemathesis.core.compat import RefResolutionError
 from schemathesis.core.errors import InternalError, InvalidSchema, LoaderError, LoaderErrorKind, OperationNotFound
 from schemathesis.core.failures import Failure, FailureGroup, MalformedJson
@@ -617,17 +617,30 @@ class BaseOpenAPISchema(BaseSchema):
             formatted_content_types = [f"\n- `{content_type}`" for content_type in all_media_types]
             message = f"The following media types are documented in the schema:{''.join(formatted_content_types)}"
             failures.append(MissingContentType(operation=operation.label, message=message, media_types=all_media_types))
-            content_type = None
+            # Default content type
+            content_type = "application/json"
         else:
             content_type = content_types[0]
-        if content_type and not media_types.is_json(content_type):
-            _maybe_raise_one_or_more(failures)
-            return None
         try:
-            data = response.json()
+            data = deserialization.deserialize_response(response, content_type)
         except JSONDecodeError as exc:
             failures.append(MalformedJson.from_exception(operation=operation.label, exc=exc))
             _maybe_raise_one_or_more(failures)
+            return None
+        except NotImplementedError:
+            # If the content type is not supported, we cannot validate it
+            _maybe_raise_one_or_more(failures)
+            return None
+        except Exception as exc:
+            failures.append(
+                Failure(
+                    operation=operation.label,
+                    title="Content deserialization error",
+                    message=f"Failed to deserialize response content:\n\n  {exc}",
+                )
+            )
+            _maybe_raise_one_or_more(failures)
+            return None
         with self._validating_response(scopes) as resolver:
             try:
                 jsonschema.validate(

{schemathesis-4.0.6 → schemathesis-4.0.8}/test/cli/__snapshots__/test_checks/test_format_parameter_csv_response.raw

@@ -24,11 +24,11 @@ Schemathesis dev
 __________________________________ GET /data ___________________________________
 1. Test Case ID: <PLACEHOLDER>
 
-- JSON deserialization error
+- Content deserialization error
 
-    Response must be valid JSON with 'Content-Type: application/json' header:
+    Failed to deserialize response content:
 
-      Expecting value: line 1 column 1 (char 0)
+      Malformed media type: ``
 
 [200] OK:
 
@@ -53,7 +53,7 @@ Test Phases:
   ⏭  Stateful (disabled)
 
 Failures:
-  ❌ JSON deserialization error: 1
+  ❌ Content deserialization error: 1
 
 Test cases:
   N generated, N found N unique failures

{schemathesis-4.0.6 → schemathesis-4.0.8}/test/conftest.py

@@ -32,8 +32,10 @@ from werkzeug.test import TestResponse
 import schemathesis.cli
 from schemathesis import auths, hooks
 from schemathesis.cli.commands.run.executor import CUSTOM_HANDLERS
+from schemathesis.cli.commands.run.handlers import output
 from schemathesis.core.hooks import HOOKS_MODULE_ENV_VAR
 from schemathesis.core.transport import Response
+from schemathesis.core.version import SCHEMATHESIS_VERSION
 from schemathesis.specs.openapi import media_types
 
 from .apps import _graphql as graphql
@@ -59,6 +61,8 @@ logging.getLogger("pyrate_limiter").setLevel(logging.CRITICAL)
 # `pytest test -m hypothesis --hypothesis-profile <profile-name>`
 settings.register_profile("CI", max_examples=2000)
 
+output.SCHEMATHESIS_VERSION = "dev"
+
 
 @pytest.fixture(autouse=True)
 def reset_hooks():
@@ -357,6 +361,9 @@ class CliSnapshotConfig:
             data,
             flags=re.MULTILINE,
         )
+        version_line = "Schemathesis dev"
+        data = data.replace(f"Schemathesis v{SCHEMATHESIS_VERSION}", version_line)
+        data = re.sub("━+", "━" * len(version_line), data)
         data = data.replace(str(SITE_PACKAGES), site_packages)
         data = re.sub(", line [0-9]+,", ", line XXX,", data)
         data = re.sub(r"Scenarios:.*\d+", r"Scenarios:    N", data)

{schemathesis-4.0.6 → schemathesis-4.0.8}/test/engine/test_checks.py

@@ -381,6 +381,23 @@ def test_response_schema_conformance_openapi(openapi_30, content, definition, re
     assert case.operation.is_valid_response(response)
 
 
+def test_response_schema_conformance_yaml(openapi_30, response_factory):
+    content = b"success: true"
+    definition = {
+        "responses": {"default": {"description": "text", "content": {"application/yaml": {"schema": SUCCESS_SCHEMA}}}}
+    }
+    response = Response.from_requests(
+        response_factory.requests(
+            content=content,
+            content_type="application/yaml",
+        ),
+        True,
+    )
+    case = make_case(openapi_30, definition)
+    assert response_schema_conformance(CTX, response, case) is None
+    assert case.operation.is_valid_response(response)
+
+
 def test_response_schema_conformance_openapi_31_boolean(openapi_30, response_factory):
     response = Response.from_requests(response_factory.requests(content=b'{"success": true}'), True)
     case = make_case(

{schemathesis-4.0.6 → schemathesis-4.0.8}/test/python/test_models.py

@@ -14,6 +14,7 @@ from schemathesis.core.failures import Failure, FailureGroup
 from schemathesis.core.transforms import merge_at
 from schemathesis.core.transport import USER_AGENT, Response
 from schemathesis.generation import GenerationMode
+from schemathesis.generation.meta import ComponentKind
 from schemathesis.schemas import APIOperation
 from schemathesis.specs.openapi.checks import content_type_conformance, response_schema_conformance
 from schemathesis.transport.prepare import get_default_headers
@@ -255,6 +256,25 @@ def test_call_and_validate(openapi3_schema_url, kwargs):
     test()
 
 
+@pytest.mark.operations("custom_format")
+def test_metadata_has_only_relevant_components(openapi3_schema_url):
+    api_schema = schemathesis.openapi.from_url(openapi3_schema_url)
+
+    operation = api_schema["/custom_format"]["GET"]
+
+    @given(
+        case=operation.as_strategy(generation_mode=GenerationMode.POSITIVE)
+        | operation.as_strategy(generation_mode=GenerationMode.NEGATIVE)
+    )
+    @settings(max_examples=10, deadline=None)
+    def test(case):
+        # Metadata should only contain components relevant to the API operation
+        assert len(case.meta.components) == 1
+        assert ComponentKind.QUERY in case.meta.components
+
+    test()
+
+
 @pytest.mark.operations("success")
 def test_call_and_validate_for_asgi(fastapi_app):
     api_schema = schemathesis.openapi.from_dict(fastapi_app.openapi())