PyPI - schemathesis - Versions diffs - 4.0.0a4__tar.gz → 4.0.0a5__tar.gz - Mend

schemathesis 4.0.0a4tar.gz → 4.0.0a5tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Files changed (605) hide show

{schemathesis-4.0.0a4 → schemathesis-4.0.0a5}/.github/workflows/codeql-analysis.yml RENAMED Viewed

@@ -38,7 +38,7 @@ jobs:
       # Initializes the CodeQL tools for scanning.
       - name: Initialize CodeQL
-        uses: github/codeql-action/init@v3.28.5
+        uses: github/codeql-action/init@v3.28.9
         with:
           languages: ${{ matrix.language }}
           # If you wish to specify custom queries, you can do so here or in a config file.
@@ -49,7 +49,7 @@ jobs:
       # Autobuild attempts to build any compiled languages  (C/C++, C#, or Java).
       # If this step fails, then you should remove it and run the build manually (see below)
       - name: Autobuild
-        uses: github/codeql-action/autobuild@v3.28.5
+        uses: github/codeql-action/autobuild@v3.28.9
       # ℹ️ Command-line programs to run using the OS shell.
       # 📚 https://git.io/JvXDl
@@ -63,4 +63,4 @@ jobs:
       #   make release
       - name: Perform CodeQL Analysis
-        uses: github/codeql-action/analyze@v3.28.5
+        uses: github/codeql-action/analyze@v3.28.9

{schemathesis-4.0.0a4 → schemathesis-4.0.0a5}/.pre-commit-config.yaml RENAMED Viewed

@@ -25,7 +25,7 @@ repos:
       - id: yamllint
   - repo: https://github.com/pre-commit/mirrors-mypy
-    rev: v1.14.1
+    rev: v1.15.0
     hooks:
       - id: mypy
         exclude: ^(docs/|test/|test-corpus/|example/|benches/).*$
@@ -33,12 +33,12 @@ repos:
         additional_dependencies: [types-requests, types-PyYAML]
   - repo: https://github.com/astral-sh/ruff-pre-commit
-    rev: v0.9.4
+    rev: v0.9.7
     hooks:
       - id: ruff-format
   - repo: https://github.com/astral-sh/ruff-pre-commit
-    rev: v0.9.4
+    rev: v0.9.7
     hooks:
       - id: ruff

{schemathesis-4.0.0a4 → schemathesis-4.0.0a5}/PKG-INFO RENAMED Viewed

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: schemathesis
-Version: 4.0.0a4
+Version: 4.0.0a5
 Summary: Property-based testing framework for Open API and GraphQL based apps
 Project-URL: Documentation, https://schemathesis.readthedocs.io/en/stable/
 Project-URL: Changelog, https://schemathesis.readthedocs.io/en/stable/changelog.html
@@ -137,36 +137,17 @@ Schemathesis is an API testing tool that automatically finds crashes and validat
 ### Highlights
-🎯 **Catches Hard-to-Find Bugs**
+- 🎯 **Catches Hard-to-Find Bugs**: Automatically uncover crashes and spec violations that manual testing might miss.
-- Uncover hidden crashes and edge cases that manual testing might miss
-- Identify spec violations and ensure your API adheres to its contract
+- ⚡ **Accelerates Testing**: Generate a wide range of test cases directly from your API schema.
-⚡ **Accelerates Testing Cycles**
+- 🧩 **Integrates Seamlessly**: Works with popular API formats such as OpenAPI and GraphQL, and easily integrates into your existing CI/CD workflows.
-- Automatically generate a wide range of test cases based on your API schema
-- Save time by reducing the need for manual test case creation
+- 🔧 **Customizable and Extendable**: Leverage Python extensions to configure and extend your test generation.
-🧩 **Integrates Seamlessly**
+- 🐞 **Simplifies Debugging**: Detailed reports and reproducible test cases with cURL commands streamline troubleshooting.
-- Works with popular API formats such as OpenAPI, GraphQL.
-- Easily integrate into your existing CI/CD workflows.
-🔧 **Customizable and Extendable**
-- Tune the testing process using Python extensions.
-- Adjust the testing flow to suit your needs with rich configuration options.
-🐞 **Simplifies Debugging**
-- Get detailed reports to identify and fix issues quickly.
-- Reproduce failing test cases with cURL commands.
-🔬 **Proven by Research**
-- Validated through academic studies on API testing automation
-- Featured in [ICSE 2022 paper](https://ieeexplore.ieee.org/document/9793781) on semantics-aware fuzzing
-- Recognized in [ACM survey](https://dl.acm.org/doi/10.1145/3617175) as state-of-the-art RESTful API testing tool
+- 🔬 **Proven by Research**: Validated through academic studies on API testing automation, featured in the [ICSE 2022 paper](https://ieeexplore.ieee.org/document/9793781) on semantics-aware fuzzing, and recognized in an [ACM survey](https://dl.acm.org/doi/10.1145/3617175) as a state-of-the-art RESTful API testing tool.
 ## Installation

{schemathesis-4.0.0a4 → schemathesis-4.0.0a5}/README.md RENAMED Viewed

@@ -41,36 +41,17 @@ Schemathesis is an API testing tool that automatically finds crashes and validat
 ### Highlights
-🎯 **Catches Hard-to-Find Bugs**
+- 🎯 **Catches Hard-to-Find Bugs**: Automatically uncover crashes and spec violations that manual testing might miss.
-- Uncover hidden crashes and edge cases that manual testing might miss
-- Identify spec violations and ensure your API adheres to its contract
+- ⚡ **Accelerates Testing**: Generate a wide range of test cases directly from your API schema.
-⚡ **Accelerates Testing Cycles**
+- 🧩 **Integrates Seamlessly**: Works with popular API formats such as OpenAPI and GraphQL, and easily integrates into your existing CI/CD workflows.
-- Automatically generate a wide range of test cases based on your API schema
-- Save time by reducing the need for manual test case creation
+- 🔧 **Customizable and Extendable**: Leverage Python extensions to configure and extend your test generation.
-🧩 **Integrates Seamlessly**
+- 🐞 **Simplifies Debugging**: Detailed reports and reproducible test cases with cURL commands streamline troubleshooting.
-- Works with popular API formats such as OpenAPI, GraphQL.
-- Easily integrate into your existing CI/CD workflows.
-🔧 **Customizable and Extendable**
-- Tune the testing process using Python extensions.
-- Adjust the testing flow to suit your needs with rich configuration options.
-🐞 **Simplifies Debugging**
-- Get detailed reports to identify and fix issues quickly.
-- Reproduce failing test cases with cURL commands.
-🔬 **Proven by Research**
-- Validated through academic studies on API testing automation
-- Featured in [ICSE 2022 paper](https://ieeexplore.ieee.org/document/9793781) on semantics-aware fuzzing
-- Recognized in [ACM survey](https://dl.acm.org/doi/10.1145/3617175) as state-of-the-art RESTful API testing tool
+- 🔬 **Proven by Research**: Validated through academic studies on API testing automation, featured in the [ICSE 2022 paper](https://ieeexplore.ieee.org/document/9793781) on semantics-aware fuzzing, and recognized in an [ACM survey](https://dl.acm.org/doi/10.1145/3617175) as a state-of-the-art RESTful API testing tool.
 ## Installation

{schemathesis-4.0.0a4 → schemathesis-4.0.0a5}/docs/changelog.rst RENAMED Viewed

@@ -1,9 +1,53 @@
 Changelog
 =========
-:version:`Unreleased <v4.0.0-alpha.4...HEAD>` - TBD
+:version:`Unreleased <v4.0.0-alpha.5...HEAD>` - TBD
 ---------------------------------------------------
+.. _v4.0.0-alpha.5:
+:version:`4.0.0-alpha.5 <v4.0.0-alpha.4...v4.0.0-alpha.5>` - 2025-02-23
+-----------------------------------------------------------------------
+This release introduces a new phase management system for CLI that simplifies test execution control and separates unit testing into different stages.
+Phase configuration changes:
+- ``examples`` (formerly ``explicit``): Runs examples specified in the API schema
+- ``fuzzing`` (formerly ``generate``): Testing with randomly generated test cases
+- ``coverage``: Deterministic testing of schema constraints and boundary values
+- ``reuse`` and ``shrink`` remain enabled by default. Disable via ``--generation-database=none`` and ``--no-shrink``.
+- ``target`` phase available via ``--generation-maximize=<METRIC>``
+**NOTE**: Pytest integration does not currently have a way to disable the coverage phase. Python API support is planned for future releases.
+**Added**
+- Warning for 4xx-only operations during unit tests to help identify configuration issues.
+**Changed**
+- Separate ``coverage`` and ``examples`` into independent testing phases.
+- Replace ``--hypothesis-phases`` with ``--phases``.
+- Do not report ``unsupported_method`` failure if the API returned HTTP 200 on OPTIONS request.
+- Add HTTP 406 status to the list of status codes that are expected for negative test cases.
+- The experimental ``--experimental-no-failfast`` option has been stabilized as ``--continue-on-failure``.
+  This option ensures all test cases within a scenario are executed, even if failures occur.
+**Fixed**
+- **Coverage phase**: Missing test case metadata leading to some failures not being detected.
+- **Coverage phase**: Missing parameter overrides.
+- **Coverage phase**: Custom auth implementation not applied to test cases.
+- **Coverage phase**: Not applying parameter serialization rules during test generation.
+- Incorrect quantifiers merging for some patterns. :issue:`2732`
+- Showing ``1 errors`` instead of ``1 error`` in CLI output.
+**Removed**
+- ``--hypothesis-no-phases``.
+- ``--exitfirst``. Use ``--max-failures=1`` instead.
 .. _v4.0.0-alpha.4:
 :version:`4.0.0-alpha.4 <v4.0.0-alpha.3...v4.0.0-alpha.4>` - 2025-02-03

{schemathesis-4.0.0a4 → schemathesis-4.0.0a5}/docs/contrib.rst RENAMED Viewed

@@ -7,7 +7,7 @@ Unique data generation
 ~~~~~~~~~~~~~~~~~~~~~~
 By default, Schemathesis may generate the same test cases as all data is randomized. If this behavior does not match your expectations, or
-your test budges, you can force Schemathesis to generate unique test cases.
+your test budgets, you can force Schemathesis to generate unique test cases.
 In CLI:

{schemathesis-4.0.0a4 → schemathesis-4.0.0a5}/docs/experimental.rst RENAMED Viewed

@@ -68,38 +68,6 @@ This will enable the OpenAPI 3.1 experimental feature for any Schemathesis runs
 Current Experimental Features
 -----------------------------
-.. _coverage-phase:
-Coverage phase
-~~~~~~~~~~~~~~
-Schemathesis primarily generates randomized tests; therefore, the outcomes can vary between test runs.
-During the coverage phase, Schemathesis deterministically generates test cases to cover common "positive" and "negative" testing scenarios depending on the configuration.
-It covers common testing patterns such as:
-- Boundary values: Minimum, maximum, and near-boundary values for numeric types and string lengths.
-- Data types: Generating values for different data types specified in the schema.
-- Constraints: Testing against schema constraints like `required` fields, `pattern`, `format`, and `uniqueItems`.
-Additionally, it uses examples in the API schema as the source of "positive" values.
-In CLI
-~~~~~~
-.. code-block:: bash
-    st run https://example.schemathesis.io/openapi.json --experimental=coverage-phase
-Using Environment Variables
-~~~~~~~~~~~~~~~~~~~~~~~~~~~
-.. code-block:: bash
-    export SCHEMATHESIS_EXPERIMENTAL_COVERAGE_PHASE=true
-For more details, join the `GitHub Discussion #2418 <https://github.com/schemathesis/schemathesis/discussions/2418>`_.
 Positive Data Acceptance
 ~~~~~~~~~~~~~~~~~~~~~~~~
@@ -168,34 +136,6 @@ Configure status codes:
     export SCHEMATHESIS_EXPERIMENTAL_NEGATIVE_DATA_REJECTION_ALLOWED_STATUSES=4XX
-Schema Analysis
-~~~~~~~~~~~~~~~
-Enables integration with Schemathesis.io to optimize test case generation for better bug detection.
-This feature infers appropriate data generation strategies, supports uncommon media types, adjusts schemas for faster data generation, and detects the web server to generate more targeted test data.
-.. _schema-analysis-cli:
-In CLI
-~~~~~~
-.. code-block:: bash
-  st run https://example.schemathesis.io/openapi.json --experimental=schema-analysis
-.. _schema-analysis-env-vars:
-Using Environment Variables
-~~~~~~~~~~~~~~~~~~~~~~~~~~~
-To enable schema analysis support via environment variables, you can set:
-.. code-block:: bash
-    export SCHEMATHESIS_EXPERIMENTAL_SCHEMA_ANALYSIS=true
-For more details, join the `GitHub Discussion #2056 <https://github.com/schemathesis/schemathesis/discussions/2056>`_.
 Stabilization of Experimental Features
 --------------------------------------

{schemathesis-4.0.0a4 → schemathesis-4.0.0a5}/pyproject.toml RENAMED Viewed

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 [project]
 name = "schemathesis"
-version = "4.0.0-alpha.4"
+version = "4.0.0-alpha.5"
 description = "Property-based testing framework for Open API and GraphQL based apps"
 keywords = ["pytest", "hypothesis", "openapi", "graphql", "testing"]
 authors = [{ name = "Dmitry Dygalo", email = "dmitry@dygalo.dev" }]

{schemathesis-4.0.0a4 → schemathesis-4.0.0a5}/src/schemathesis/cli/commands/run/__init__.py RENAMED Viewed

@@ -15,7 +15,6 @@ from schemathesis.cli.commands.run.filters import FilterArguments, with_filters
 from schemathesis.cli.commands.run.hypothesis import (
     HYPOTHESIS_IN_MEMORY_DATABASE_IDENTIFIER,
     HealthCheck,
-    Phase,
     prepare_health_checks,
     prepare_phases,
     prepare_settings,
@@ -44,7 +43,7 @@ from schemathesis.specs.openapi.checks import *  # noqa: F401, F403
 COLOR_OPTIONS_INVALID_USAGE_MESSAGE = "Can't use `--no-color` and `--force-color` simultaneously"
-DEFAULT_PHASES = ("unit", "stateful")
+DEFAULT_PHASES = ("examples", "coverage", "fuzzing", "stateful")
 @click.argument("schema", type=str)  # type: ignore[misc]
@@ -76,7 +75,7 @@ DEFAULT_PHASES = ("unit", "stateful")
 @grouped_option(
     "--phases",
     help="A comma-separated list of test phases to run",
-    type=CsvChoice(["unit", "stateful"]),
+    type=CsvChoice(DEFAULT_PHASES),
     default=",".join(DEFAULT_PHASES),
     metavar="",
 )
@@ -117,15 +116,6 @@ DEFAULT_PHASES = ("unit", "stateful")
     show_default=True,
     metavar="",
 )
-@grouped_option(
-    "-x",
-    "--exitfirst",
-    "exit_first",
-    is_flag=True,
-    default=False,
-    help="Terminate the test suite immediately upon the first failure or error encountered",
-    show_default=True,
-)
 @grouped_option(
     "--max-failures",
     "max_failures",
@@ -133,6 +123,14 @@ DEFAULT_PHASES = ("unit", "stateful")
     help="Terminate the test suite after reaching a specified number of failures or errors",
     show_default=True,
 )
+@grouped_option(
+    "--continue-on-failure",
+    "continue_on_failure",
+    help="Continue executing all test cases within a scenario, even after encountering failures",
+    is_flag=True,
+    default=False,
+    metavar="",
+)
 @grouped_option(
     "--max-response-time",
     help="Maximum allowed API response time in seconds",
@@ -307,15 +305,6 @@ DEFAULT_PHASES = ("unit", "stateful")
     multiple=True,
     metavar="FEATURES",
 )
-@grouped_option(
-    "--experimental-no-failfast",
-    "no_failfast",
-    help="Continue testing an API operation after a failure is found",
-    is_flag=True,
-    default=False,
-    metavar="",
-    envvar="SCHEMATHESIS_EXPERIMENTAL_NO_FAILFAST",
-)
 @grouped_option(
     "--experimental-missing-required-header-allowed-statuses",
     "missing_required_header_allowed_statuses",
@@ -407,7 +396,7 @@ DEFAULT_PHASES = ("unit", "stateful")
     default=None,
     callback=validation.reduce_list,
     show_default=True,
-    metavar="TARGET",
+    metavar="METRIC",
 )
 @grouped_option(
     "--generation-with-security-parameters",
@@ -486,19 +475,6 @@ DEFAULT_PHASES = ("unit", "stateful")
     type=str,
     callback=validation.validate_set_path,
 )
-@group("Hypothesis engine options")
-@grouped_option(
-    "--hypothesis-phases",
-    help="Testing phases to execute",
-    type=CsvEnumChoice(Phase),
-    metavar="",
-)
-@grouped_option(
-    "--hypothesis-no-phases",
-    help="Testing phases to exclude from execution",
-    type=CsvEnumChoice(Phase),
-    metavar="",
-)
 @group("Global options")
 @grouped_option("--no-color", help="Disable ANSI color escape codes", type=bool, is_flag=True)
 @grouped_option("--force-color", help="Explicitly tells to enable ANSI color escape codes", type=bool, is_flag=True)
@@ -513,7 +489,6 @@ def run(
     set_cookie: dict[str, str],
     set_path: dict[str, str],
     experiments: list,
-    no_failfast: bool,
     missing_required_header_allowed_statuses: list[str],
     positive_data_acceptance_allowed_statuses: list[str],
     negative_data_rejection_allowed_statuses: list[str],
@@ -521,8 +496,8 @@ def run(
     excluded_check_names: Sequence[str],
     max_response_time: float | None = None,
     phases: Sequence[str] = DEFAULT_PHASES,
-    exit_first: bool = False,
     max_failures: int | None = None,
+    continue_on_failure: bool = False,
     include_path: Sequence[str] = (),
     include_path_regex: str | None = None,
     include_method: Sequence[str] = (),
@@ -565,8 +540,6 @@ def run(
     output_sanitize: bool = True,
     output_truncate: bool = True,
     contrib_openapi_fill_missing_examples: bool = False,
-    hypothesis_phases: list[Phase] | None = None,
-    hypothesis_no_phases: list[Phase] | None = None,
     generation_modes: tuple[GenerationMode, ...] = DEFAULT_GENERATOR_MODES,
     generation_seed: int | None = None,
     generation_max_examples: int | None = None,
@@ -593,7 +566,7 @@ def run(
     validation.validate_schema(schema, base_url)
-    _hypothesis_phases = prepare_phases(hypothesis_phases, hypothesis_no_phases, generation_no_shrink)
+    _hypothesis_phases = prepare_phases(generation_no_shrink)
     _hypothesis_suppress_health_check = prepare_health_checks(suppress_health_check)
     for experiment in experiments:
@@ -640,9 +613,6 @@ def run(
         max_response_time=max_response_time,
     ).into()
-    if exit_first and max_failures is None:
-        max_failures = 1
     report_config = None
     if report_formats or report_junit_path or report_vcr_path or report_har_path:
         report_config = ReportConfig(
@@ -687,7 +657,7 @@ def run(
                     with_security_parameters=generation_with_security_parameters,
                 ),
                 max_failures=max_failures,
-                no_failfast=no_failfast,
+                continue_on_failure=continue_on_failure,
                 unique_inputs=generation_unique_inputs,
                 seed=seed,
                 workers_num=workers_num,

{schemathesis-4.0.0a4 → schemathesis-4.0.0a5}/src/schemathesis/cli/commands/run/checks.py RENAMED Viewed

@@ -69,10 +69,9 @@ class CheckArguments:
             checks_config[_max_response_time] = MaxResponseTimeConfig(self.max_response_time)
             selected_checks.append(_max_response_time)
-        if experimental.COVERAGE_PHASE.is_enabled:
-            from schemathesis.specs.openapi.checks import unsupported_method
+        from schemathesis.specs.openapi.checks import unsupported_method
-            selected_checks.append(unsupported_method)
+        selected_checks.append(unsupported_method)
         # Exclude checks based on their names
         selected_checks = [check for check in selected_checks if check.__name__ not in self.excluded_check_names]

{schemathesis-4.0.0a4 → schemathesis-4.0.0a5}/src/schemathesis/cli/commands/run/context.py RENAMED Viewed

@@ -20,8 +20,9 @@ if TYPE_CHECKING:
 class Statistic:
     """Running statistics about test execution."""
-    outcomes: dict[Status, int]
     failures: dict[str, dict[str, GroupedFailures]]
+    # Track first case_id where each unique failure was found
+    unique_failures_map: dict[Failure, str]
     extraction_failures: set[ExtractionFailure]
@@ -32,8 +33,8 @@ class Statistic:
     cases_without_checks: int
     __slots__ = (
-        "outcomes",
         "failures",
+        "unique_failures_map",
         "extraction_failures",
         "tested_operations",
         "total_cases",
@@ -42,15 +43,15 @@ class Statistic:
     )
     def __init__(self) -> None:
-        self.outcomes = {}
         self.failures = {}
+        self.unique_failures_map = {}
         self.extraction_failures = set()
         self.tested_operations = set()
         self.total_cases = 0
         self.cases_with_failures = 0
         self.cases_without_checks = 0
-    def record_checks(self, recorder: ScenarioRecorder) -> None:
+    def on_scenario_finished(self, recorder: ScenarioRecorder) -> None:
         """Update statistics and store failures from a new batch of checks."""
         from schemathesis.generation.stateful.state_machine import ExtractionFailure
@@ -80,20 +81,32 @@ class Statistic:
             self.tested_operations.add(case.value.operation.label)
             has_failures = False
-            for check in checks:
-                response = recorder.interactions[case_id].response
+            current_case_failures = []
+            last_failure_info = None
-                # Collect failures
+            for check in checks:
                 if check.failure_info is not None:
-                    has_failures = True
-                    if case_id not in failures:
-                        failures[case_id] = GroupedFailures(
-                            case_id=case_id,
-                            code_sample=check.failure_info.code_sample,
-                            failures=[],
-                            response=response,
-                        )
-                    failures[case_id].failures.append(check.failure_info.failure)
+                    failure = check.failure_info.failure
+                    # Check if this is a new unique failure
+                    if failure not in self.unique_failures_map:
+                        last_failure_info = check.failure_info
+                        self.unique_failures_map[failure] = case_id
+                        current_case_failures.append(failure)
+                        has_failures = True
+                    else:
+                        # This failure was already seen - skip it
+                        continue
+            if current_case_failures:
+                assert last_failure_info is not None
+                failures[case_id] = GroupedFailures(
+                    case_id=case_id,
+                    code_sample=last_failure_info.code_sample,
+                    failures=current_case_failures,
+                    response=recorder.interactions[case_id].response,
+                )
             if has_failures:
                 self.cases_with_failures += 1
@@ -178,7 +191,7 @@ class ExecutionContext:
     def on_event(self, event: events.EngineEvent) -> None:
         if isinstance(event, events.ScenarioFinished):
-            self.statistic.record_checks(event.recorder)
+            self.statistic.on_scenario_finished(event.recorder)
         elif isinstance(event, events.NonFatalError) or (
             isinstance(event, events.PhaseFinished)
             and event.phase.is_enabled

{schemathesis-4.0.0a4 → schemathesis-4.0.0a5}/src/schemathesis/cli/commands/run/executor.py RENAMED Viewed

@@ -125,6 +125,7 @@ def initialize_handlers(config: RunConfig) -> list[EventHandler]:
             seed=config.engine.execution.seed,
             rate_limit=config.rate_limit,
             wait_for_schema=config.wait_for_schema,
+            engine_config=config.engine,
             report_config=config.report,
         )
     )

schemathesis 4.0.0a4__tar.gz → 4.0.0a5__tar.gz

schemathesis 4.0.0a4tar.gz → 4.0.0a5tar.gz