pytest-json-ctrf 0.5.3__tar.gz → 0.6.0__tar.gz

pytest_json_ctrf-0.6.0/MANIFEST.in

@@ -0,0 +1,2 @@
+exclude .env.example
+prune tests

{pytest_json_ctrf-0.5.3 → pytest_json_ctrf-0.6.0}/PKG-INFO

@@ -1,9 +1,9 @@
 Metadata-Version: 2.4
 Name: pytest-json-ctrf
-Version: 0.5.3
+Version: 0.6.0
 Summary: Pytest plugin to generate json report in CTRF (Common Test Report Format)
 Author: Oleksii Ostapov
-Project-URL: Infopulse, https://infopulse.com/
+Project-URL: QAMania, https://qamania.org
 Project-URL: Homepage, https://github.com/infopulse/pytest-common-test-report-json
 Classifier: Framework :: Pytest
 Classifier: Programming Language :: Python :: 3
@@ -24,7 +24,9 @@ Do not worry if report in GitHub does not appear immediately. It takes some time
 - Generates JSON report
 - Tested to work correctly with and without [pytest-xdist](https://pypi.org/project/pytest-xdist/)
 - Tested to get browser name from [pytest-playwright](https://pypi.org/project/pytest-playwright/)
-- Parametrized tests are supported and reported as 1 test with multiple retries
+- Reruns from [pytest-rerunfailures](https://pypi.org/project/pytest-rerunfailures/) are reported as `retries`, `retryAttempts` and `flaky`
+- Each test gets a `suite` hierarchy: file, then classes
+- Parametrized tests are supported: each case is reported as a separate test named by its full pytest node id (e.g. `test_file.py::test_param[1]`), with its values in `parameters`
 
 ## Installation
 
@@ -41,13 +43,20 @@ pytest --ctrf report.json
 ```
 Environment Variables may be used to specify the required Environment Object
 fields when using the CTRF [slack-test-reporter][ctrf-slack-test-reporter-url].
+All of them are optional: a variable that is not set is left out of the report, see [.env.example](./.env.example).
 
 ```bash
 CTRF_BUILD_NAME="Pytest JSON CTRF Report"
-CTRF_BUILD_NUMBER="000"
-CTRF_BUILD_URL="https://ctrf.io]"
+CTRF_BUILD_NUMBER=123
+CTRF_BUILD_URL="https://ctrf.io"
+CTRF_TEST_ENVIRONMENT="staging"
+CTRF_TEST_TYPE="e2e"
 ```
 
+`CTRF_BUILD_NUMBER` must be an integer. Otherwise a warning is issued and `buildNumber` is omitted from the report.  
+`osPlatform`, `osRelease` and `osVersion` are detected automatically.  
+`CTRF_TEST_TYPE` is written to every test's `type` field.
+
 ## JSON example
 
 More info here: https://ctrf.io/docs/schema/examples
@@ -55,11 +64,12 @@ More info here: https://ctrf.io/docs/schema/examples
 ```json
 {
   "reportFormat": "CTRF",
-  "specVersion": "0.0.0",
-  "generatedBy": "pytest",  
+  "specVersion": "1.0.0",
+  "generatedBy": "pytest",
   "results": {
     "tool": {
-      "name": "jest"
+      "name": "pytest",
+      "version": "9.0.3"
     },
     "summary": {
       "tests": 3,
@@ -73,8 +83,12 @@ More info here: https://ctrf.io/docs/schema/examples
     },
     "environment": {
         "buildName": "Pytest JSON CTRF Report",
-        "buildNumber": "000",
-        "buildUrl": "https://ctrf.io"
+        "buildUrl": "https://ctrf.io",
+        "osPlatform": "linux",
+        "osRelease": "6.8.0-45-generic",
+        "osVersion": "#45-Ubuntu SMP PREEMPT_DYNAMIC",
+        "testEnvironment": "staging",
+        "buildNumber": 123
     },
     "tests": [
       {
@@ -114,6 +128,13 @@ The `pytest_runtest_logreport` hook in the controller node is used to collect th
 - https://github.com/infopulse/Playwright-course-python -> The report will be used in the demo project as +1 report option
 
 ## Changelog
+- 0.6.0 - fixed issue 12 - report now validates against the CTRF schema. Thanks to [@jamesarosen](https://github.com/jamesarosen) for the detailed report!
+  - `filePath`, `rawStatus` and integer `buildNumber`; environment fields are added only when their variables are set (no defaults), plus `osPlatform`, `osRelease`, `osVersion` and `type` (`CTRF_TEST_TYPE`)
+  - parametrized tests are named by the full node id and get `parameters`
+  - `message` is the real failure message; skip and xfail reasons are reported in `message`
+  - reruns (pytest-rerunfailures) are reported as `retries`, `retryAttempts` and `flaky`
+  - new `suite` field and summary `other`, `flaky`, `suites`, `duration`
+  - CI validates reports against the CTRF schema
 - 0.5.3 - fixed issue 6 - @pytest.mark.parametrize will no longer be added to tags to prevent exhausting memory
 - 0.5.1 - added mandatory root object fields: reportFormat, specVersion, generatedBy
 - 0.5.0 - Changed logic of handling parametrized tests. Previously they were marked as retries, now they are reported as separate tests

{pytest_json_ctrf-0.5.3 → pytest_json_ctrf-0.6.0}/README.md

@@ -9,7 +9,9 @@ Do not worry if report in GitHub does not appear immediately. It takes some time
 - Generates JSON report
 - Tested to work correctly with and without [pytest-xdist](https://pypi.org/project/pytest-xdist/)
 - Tested to get browser name from [pytest-playwright](https://pypi.org/project/pytest-playwright/)
-- Parametrized tests are supported and reported as 1 test with multiple retries
+- Reruns from [pytest-rerunfailures](https://pypi.org/project/pytest-rerunfailures/) are reported as `retries`, `retryAttempts` and `flaky`
+- Each test gets a `suite` hierarchy: file, then classes
+- Parametrized tests are supported: each case is reported as a separate test named by its full pytest node id (e.g. `test_file.py::test_param[1]`), with its values in `parameters`
 
 ## Installation
 
@@ -26,13 +28,20 @@ pytest --ctrf report.json
 ```
 Environment Variables may be used to specify the required Environment Object
 fields when using the CTRF [slack-test-reporter][ctrf-slack-test-reporter-url].
+All of them are optional: a variable that is not set is left out of the report, see [.env.example](./.env.example).
 
 ```bash
 CTRF_BUILD_NAME="Pytest JSON CTRF Report"
-CTRF_BUILD_NUMBER="000"
-CTRF_BUILD_URL="https://ctrf.io]"
+CTRF_BUILD_NUMBER=123
+CTRF_BUILD_URL="https://ctrf.io"
+CTRF_TEST_ENVIRONMENT="staging"
+CTRF_TEST_TYPE="e2e"
 ```
 
+`CTRF_BUILD_NUMBER` must be an integer. Otherwise a warning is issued and `buildNumber` is omitted from the report.  
+`osPlatform`, `osRelease` and `osVersion` are detected automatically.  
+`CTRF_TEST_TYPE` is written to every test's `type` field.
+
 ## JSON example
 
 More info here: https://ctrf.io/docs/schema/examples
@@ -40,11 +49,12 @@ More info here: https://ctrf.io/docs/schema/examples
 ```json
 {
   "reportFormat": "CTRF",
-  "specVersion": "0.0.0",
-  "generatedBy": "pytest",  
+  "specVersion": "1.0.0",
+  "generatedBy": "pytest",
   "results": {
     "tool": {
-      "name": "jest"
+      "name": "pytest",
+      "version": "9.0.3"
     },
     "summary": {
       "tests": 3,
@@ -58,8 +68,12 @@ More info here: https://ctrf.io/docs/schema/examples
     },
     "environment": {
         "buildName": "Pytest JSON CTRF Report",
-        "buildNumber": "000",
-        "buildUrl": "https://ctrf.io"
+        "buildUrl": "https://ctrf.io",
+        "osPlatform": "linux",
+        "osRelease": "6.8.0-45-generic",
+        "osVersion": "#45-Ubuntu SMP PREEMPT_DYNAMIC",
+        "testEnvironment": "staging",
+        "buildNumber": 123
     },
     "tests": [
       {
@@ -99,6 +113,13 @@ The `pytest_runtest_logreport` hook in the controller node is used to collect th
 - https://github.com/infopulse/Playwright-course-python -> The report will be used in the demo project as +1 report option
 
 ## Changelog
+- 0.6.0 - fixed issue 12 - report now validates against the CTRF schema. Thanks to [@jamesarosen](https://github.com/jamesarosen) for the detailed report!
+  - `filePath`, `rawStatus` and integer `buildNumber`; environment fields are added only when their variables are set (no defaults), plus `osPlatform`, `osRelease`, `osVersion` and `type` (`CTRF_TEST_TYPE`)
+  - parametrized tests are named by the full node id and get `parameters`
+  - `message` is the real failure message; skip and xfail reasons are reported in `message`
+  - reruns (pytest-rerunfailures) are reported as `retries`, `retryAttempts` and `flaky`
+  - new `suite` field and summary `other`, `flaky`, `suites`, `duration`
+  - CI validates reports against the CTRF schema
 - 0.5.3 - fixed issue 6 - @pytest.mark.parametrize will no longer be added to tags to prevent exhausting memory
 - 0.5.1 - added mandatory root object fields: reportFormat, specVersion, generatedBy
 - 0.5.0 - Changed logic of handling parametrized tests. Previously they were marked as retries, now they are reported as separate tests

{pytest_json_ctrf-0.5.3 → pytest_json_ctrf-0.6.0}/ctrf/Report.py

@@ -1,8 +1,11 @@
 import os
+import platform
+import sys
 import pytest
 import time
 import json
-from typing import Optional
+import warnings
+from uuid import uuid4
 from datetime import datetime, timezone
 from pytest import TestReport
 from collections import OrderedDict
@@ -12,7 +15,6 @@ from .TestObject import TestObject, TestStatus
 class Report:
     def __init__(self):
         self.test_items = OrderedDict()
-        self.prepared_tests = dict()
         self.start_time = None
         self.stop_time = None
 
@@ -30,27 +32,47 @@ class Report:
         }
 
     def list_tests_by_status(self, status: TestStatus) -> list:
-        return [test for test in self.prepared_tests.values() if test.status == status]
+        return [test for test in self.test_items.values() if test.status == status]
 
     def _get_summary(self) -> dict:
-        return {
-            'tests': len(self.prepared_tests),
+        tests = list(self.test_items.values())
+        summary = {
+            'tests': len(tests),
             'passed': len(self.list_tests_by_status(TestStatus.PASSED)),
             'failed': len(self.list_tests_by_status(TestStatus.FAILED)),
             'skipped': len(self.list_tests_by_status(TestStatus.SKIPPED)),
             'pending': len(self.list_tests_by_status(TestStatus.PENDING)),
-            'other': 0,
+            'other': len(self.list_tests_by_status(TestStatus.OTHER)),
+            'flaky': len([test for test in tests if test.flaky]),
+            # a suite is a file or class that directly contains tests
+            'suites': len({tuple(test.suite) for test in tests if test.suite}),
             'start': self.start_time,
-            'stop': self.stop_time
+            'stop': self.stop_time,
         }
+        if self.start_time is not None and self.stop_time is not None:
+            summary['duration'] = self.stop_time - self.start_time
+        return summary
 
     @staticmethod
     def _get_environment() -> dict:
-        return {
-            "buildName": os.getenv("CTRF_BUILD_NAME", "Pytest JSON CTRF Report"),
-            "buildNumber": os.getenv("CTRF_BUILD_NUMBER", "000"),
-            "buildUrl": os.getenv("CTRF_BUILD_URL", "https://ctrf.io"),
+        env = {
+            "osPlatform": sys.platform,
+            "osRelease": platform.release(),
+            "osVersion": platform.version(),
         }
+        for field, variable in (("buildName", "CTRF_BUILD_NAME"),
+                                ("buildUrl", "CTRF_BUILD_URL"),
+                                ("testEnvironment", "CTRF_TEST_ENVIRONMENT")):
+            value = os.getenv(variable)
+            if value:
+                env[field] = value
+        raw = os.getenv("CTRF_BUILD_NUMBER")
+        if raw:
+            try:
+                env["buildNumber"] = int(raw)
+            except ValueError:
+                warnings.warn(f"CTRF_BUILD_NUMBER={raw!r} is not an integer; buildNumber omitted from the report")
+        return env
 
     def collect(self, report: TestReport) -> None:
         if report.nodeid not in self.test_items.keys():
@@ -61,34 +83,19 @@ class Report:
         test.update(report)
         self.test_items[report.nodeid] = test
 
-    def process_retries(self) -> None:
-        for test_id, test_data in self.test_items.items():
-            name = str(test_id)
-            test: Optional[TestObject] = self.prepared_tests.get(name)
-            if test:
-                test.status = test_data.status
-                test.raw_status = test_data.raw_status
-                if test_data.status != TestStatus.PASSED:
-                    test.status = test_data.status
-                    test.raw_status = test_data.raw_status
-                    test.message = test_data.message
-                    test.trace = test_data.trace
-                test.retries += 1
-            else:
-                self.prepared_tests[name] = test_data
-
     def get_report(self) -> dict:
-        self.process_retries()
+        test_type = os.getenv("CTRF_TEST_TYPE") or None
         return {
             'reportFormat': 'CTRF',
-            'specVersion': '0.0.0',
+            'specVersion': '1.0.0',
+            'reportId': str(uuid4()),
             'timestamp': datetime.now(timezone.utc).isoformat(),
             'generatedBy': 'pytest',
             'results': {
                 "tool": self._get_tool(),
                 "summary": self._get_summary(),
                 "environment": self._get_environment(),
-                "tests": [test.serialize() for test in self.prepared_tests.values()]
+                "tests": [test.serialize(test_type) for test in self.test_items.values()]
             }
         }
 

pytest_json_ctrf-0.6.0/ctrf/TestObject.py

@@ -0,0 +1,166 @@
+from enum import Enum
+from typing import Any, Optional, List, Dict
+
+from pytest import TestReport
+
+
+class TestStatus(Enum):
+    PASSED = "passed"
+    FAILED = "failed"
+    SKIPPED = "skipped"
+    PENDING = "pending"
+    OTHER = "other"
+
+
+class TestObject:
+    name: str
+    _status: TestStatus
+    raw_status: Optional[str]
+    start: int
+    stop: int
+    retries: int
+    message: Optional[str]
+
+    worker_id: Optional[str]
+    file_path: str
+    suite: List[str]
+    tags: List[str]
+    browser: Optional[str]
+    parameters: Optional[Dict[str, Any]]
+    trace: Optional[str]
+    retry_attempts: List[Dict[str, Any]]
+
+    def __init__(self, report: TestReport, worker_id: Optional[str] = None):
+        self.name = report.nodeid
+        self.worker_id = worker_id
+        self.file_path = report.location[0]
+        self.suite = self._suite(report.nodeid)
+        self.tags: List[str] = []
+        self.browser = None
+        self.parameters = None
+        self.retry_attempts = []
+        self._reset_attempt()
+
+    def _reset_attempt(self) -> None:
+        self._status = TestStatus.PENDING
+        self.raw_status = None
+        self.message = None
+        self.trace = None
+        self.start = 0
+        self.stop = 0
+        self._setup_seen = False
+
+    @staticmethod
+    def _suite(nodeid: str) -> List[str]:
+        # "dir/test_file.py::TestClass::test_name[param]" -> ["dir/test_file.py", "TestClass"]
+        # parameters are cut off first, they may contain "::"
+        return nodeid.split('[', 1)[0].split('::')[:-1]
+
+    @property
+    def status(self) -> TestStatus:
+        return self._status
+
+    @property
+    def retries(self) -> int:
+        return len(self.retry_attempts)
+
+    @property
+    def flaky(self) -> bool:
+        return self._status == TestStatus.PASSED and any(
+            attempt['status'] == TestStatus.FAILED.value for attempt in self.retry_attempts)
+
+    def _finish_attempt(self) -> None:
+        attempt = {
+            'attempt': len(self.retry_attempts) + 1,
+            'status': self._status.value,
+            'duration': max(self.stop - self.start, 0),
+            'start': self.start,
+            'stop': self.stop,
+            'message': self.message,
+            'trace': self.trace,
+        }
+        self.retry_attempts.append({key: value for key, value in attempt.items() if value is not None})
+        self._reset_attempt()
+
+    def set_status(self, report: TestReport) -> TestStatus:
+        # the first non-passed outcome wins; message and trace are taken from that same report
+        if self._status in (TestStatus.SKIPPED, TestStatus.FAILED, TestStatus.OTHER):
+            return self._status
+        elif report.skipped:
+            self._status = TestStatus.SKIPPED
+            if hasattr(report, 'wasxfail'):
+                self.raw_status = 'xfailed'
+                self.message = report.wasxfail or None
+                self.trace = report.longreprtext or None
+            else:
+                self.message = self._skip_reason(report)
+        elif report.failed or report.outcome == 'rerun':
+            # "rerun" is a failed attempt that pytest-rerunfailures will run again
+            self._status = TestStatus.FAILED
+            self.raw_status = f"{report.when}_{report.outcome}"
+            self.message = self._failure_message(report)
+            self.trace = report.longreprtext or None
+        elif report.passed:
+            self._status = TestStatus.PASSED
+        else:
+            self._status = TestStatus.OTHER
+        return self._status
+
+    @staticmethod
+    def _failure_message(report: TestReport) -> Optional[str]:
+        # same text pytest shows in the short test summary, e.g. "AssertionError: assert 1 == 2"
+        crash = getattr(report.longrepr, 'reprcrash', None)
+        if crash is not None and crash.message:
+            return crash.message
+        # plain-text longrepr (some plugins, xdist worker crashes): the last line is usually the error
+        lines = [line for line in report.longreprtext.splitlines() if line.strip()]
+        return lines[-1].strip() if lines else None
+
+    @staticmethod
+    def _skip_reason(report: TestReport) -> Optional[str]:
+        # for skips pytest stores longrepr as a (path, lineno, reason) tuple
+        if isinstance(report.longrepr, tuple) and len(report.longrepr) == 3:
+            reason = str(report.longrepr[2])
+            if reason.startswith('Skipped: '):
+                reason = reason[len('Skipped: '):]
+            return reason or None
+        return report.longreprtext or None
+
+    def update(self, report: TestReport) -> None:
+        if report.when == "setup":
+            # every attempt starts with setup, a second one means the test is being rerun
+            if self._setup_seen:
+                self._finish_attempt()
+            self._setup_seen = True
+            self.start = int(report.start * 1000)
+        self.set_status(report)
+        if report.when == "teardown":
+            self.stop = int(report.stop * 1000)
+        if hasattr(report, '_ctrf_metadata'):
+            self.tags = report._ctrf_metadata.get('tags')
+            self.browser = report._ctrf_metadata.get('browser')
+            self.parameters = report._ctrf_metadata.get('parameters')
+
+    def serialize(self, test_type: Optional[str] = None) -> dict:
+        result: Dict[str, Any] = {
+            'name': self.name,
+            'status': self._status.value,
+            'rawStatus': self.raw_status,
+            'duration': self.stop - self.start,
+            'start': self.start,
+            'stop': self.stop,
+            'suite': self.suite,
+            'type': test_type,
+            'retries': self.retries,
+            'retryAttempts': self.retry_attempts,
+            'flaky': True if self.flaky else None,
+            'filePath': self.file_path,
+            'tags': self.tags,
+            'browser': self.browser,
+            'parameters': self.parameters,
+            'trace': self.trace,
+            'message': self.message
+        }
+        if self.worker_id:
+            result['extra'] = {'worker': self.worker_id}
+        return {key: value for key, value in result.items() if value not in [None, '', []]}

{pytest_json_ctrf-0.5.3 → pytest_json_ctrf-0.6.0}/ctrf/main.py

@@ -48,7 +48,23 @@ def ctrf_json_metadata(request: FixtureRequest):
         tags.append(tag)
     request.node._ctrf_metadata.setdefault('tags', tags)
     if hasattr(request.node, 'callspec'):
-        browser = request.node.callspec.params.get('browser_name')
+        params = request.node.callspec.params
+        browser = params.get('browser_name')
         if browser:
             request.node._ctrf_metadata.setdefault('browser', browser)
+        request.node._ctrf_metadata.setdefault(
+            'parameters', {key: _serializable_param(value) for key, value in params.items()})
     return request.node._ctrf_metadata
+
+
+MAX_PARAM_LENGTH = 200
+
+
+def _serializable_param(value):
+    # keep only primitives: report metadata is sent between xdist workers and dumped to json;
+    # long values are truncated to keep the report small (see issue 6)
+    if not isinstance(value, (str, int, float, bool, type(None))):
+        value = repr(value)
+    if isinstance(value, str) and len(value) > MAX_PARAM_LENGTH:
+        value = value[:MAX_PARAM_LENGTH] + '...'
+    return value

{pytest_json_ctrf-0.5.3 → pytest_json_ctrf-0.6.0}/pyproject.toml

@@ -6,11 +6,11 @@ build-backend = "setuptools.build_meta"
 exclude = [
     ".github", "tests", "build", "dist", ".venv",
     "pytest_json_ctrf.egg-info", ".env", ".gitignore",
-    "CHANGELOG.md", "assets", "test-results"]
+    "CHANGELOG.md", "assets", "test-results", "schema_validator"]
 
 [project]
 name = "pytest-json-ctrf"
-version = "0.5.3"
+version = "0.6.0"
 
 dependencies = [
     "pytest>6.0.0",
@@ -30,7 +30,7 @@ classifiers = [
 ]
 
 [project.urls]
-"Infopulse" = "https://infopulse.com/"
+"QAMania" = "https://qamania.org"
 "Homepage" = "https://github.com/infopulse/pytest-common-test-report-json"
 
 [tool.pytest.ini_options]
@@ -41,3 +41,10 @@ markers = [
 
 [project.entry-points.pytest11]
 ctrf = "ctrf.main"
+
+[dependency-groups]
+dev = [
+    "jsonschema>=4.23.0",
+    "pytest-rerunfailures>=14.0",
+    "pytest-xdist>=3.6.1",
+]

{pytest_json_ctrf-0.5.3 → pytest_json_ctrf-0.6.0}/pytest_json_ctrf.egg-info/PKG-INFO

@@ -1,9 +1,9 @@
 Metadata-Version: 2.4
 Name: pytest-json-ctrf
-Version: 0.5.3
+Version: 0.6.0
 Summary: Pytest plugin to generate json report in CTRF (Common Test Report Format)
 Author: Oleksii Ostapov
-Project-URL: Infopulse, https://infopulse.com/
+Project-URL: QAMania, https://qamania.org
 Project-URL: Homepage, https://github.com/infopulse/pytest-common-test-report-json
 Classifier: Framework :: Pytest
 Classifier: Programming Language :: Python :: 3
@@ -24,7 +24,9 @@ Do not worry if report in GitHub does not appear immediately. It takes some time
 - Generates JSON report
 - Tested to work correctly with and without [pytest-xdist](https://pypi.org/project/pytest-xdist/)
 - Tested to get browser name from [pytest-playwright](https://pypi.org/project/pytest-playwright/)
-- Parametrized tests are supported and reported as 1 test with multiple retries
+- Reruns from [pytest-rerunfailures](https://pypi.org/project/pytest-rerunfailures/) are reported as `retries`, `retryAttempts` and `flaky`
+- Each test gets a `suite` hierarchy: file, then classes
+- Parametrized tests are supported: each case is reported as a separate test named by its full pytest node id (e.g. `test_file.py::test_param[1]`), with its values in `parameters`
 
 ## Installation
 
@@ -41,13 +43,20 @@ pytest --ctrf report.json
 ```
 Environment Variables may be used to specify the required Environment Object
 fields when using the CTRF [slack-test-reporter][ctrf-slack-test-reporter-url].
+All of them are optional: a variable that is not set is left out of the report, see [.env.example](./.env.example).
 
 ```bash
 CTRF_BUILD_NAME="Pytest JSON CTRF Report"
-CTRF_BUILD_NUMBER="000"
-CTRF_BUILD_URL="https://ctrf.io]"
+CTRF_BUILD_NUMBER=123
+CTRF_BUILD_URL="https://ctrf.io"
+CTRF_TEST_ENVIRONMENT="staging"
+CTRF_TEST_TYPE="e2e"
 ```
 
+`CTRF_BUILD_NUMBER` must be an integer. Otherwise a warning is issued and `buildNumber` is omitted from the report.  
+`osPlatform`, `osRelease` and `osVersion` are detected automatically.  
+`CTRF_TEST_TYPE` is written to every test's `type` field.
+
 ## JSON example
 
 More info here: https://ctrf.io/docs/schema/examples
@@ -55,11 +64,12 @@ More info here: https://ctrf.io/docs/schema/examples
 ```json
 {
   "reportFormat": "CTRF",
-  "specVersion": "0.0.0",
-  "generatedBy": "pytest",  
+  "specVersion": "1.0.0",
+  "generatedBy": "pytest",
   "results": {
     "tool": {
-      "name": "jest"
+      "name": "pytest",
+      "version": "9.0.3"
     },
     "summary": {
       "tests": 3,
@@ -73,8 +83,12 @@ More info here: https://ctrf.io/docs/schema/examples
     },
     "environment": {
         "buildName": "Pytest JSON CTRF Report",
-        "buildNumber": "000",
-        "buildUrl": "https://ctrf.io"
+        "buildUrl": "https://ctrf.io",
+        "osPlatform": "linux",
+        "osRelease": "6.8.0-45-generic",
+        "osVersion": "#45-Ubuntu SMP PREEMPT_DYNAMIC",
+        "testEnvironment": "staging",
+        "buildNumber": 123
     },
     "tests": [
       {
@@ -114,6 +128,13 @@ The `pytest_runtest_logreport` hook in the controller node is used to collect th
 - https://github.com/infopulse/Playwright-course-python -> The report will be used in the demo project as +1 report option
 
 ## Changelog
+- 0.6.0 - fixed issue 12 - report now validates against the CTRF schema. Thanks to [@jamesarosen](https://github.com/jamesarosen) for the detailed report!
+  - `filePath`, `rawStatus` and integer `buildNumber`; environment fields are added only when their variables are set (no defaults), plus `osPlatform`, `osRelease`, `osVersion` and `type` (`CTRF_TEST_TYPE`)
+  - parametrized tests are named by the full node id and get `parameters`
+  - `message` is the real failure message; skip and xfail reasons are reported in `message`
+  - reruns (pytest-rerunfailures) are reported as `retries`, `retryAttempts` and `flaky`
+  - new `suite` field and summary `other`, `flaky`, `suites`, `duration`
+  - CI validates reports against the CTRF schema
 - 0.5.3 - fixed issue 6 - @pytest.mark.parametrize will no longer be added to tags to prevent exhausting memory
 - 0.5.1 - added mandatory root object fields: reportFormat, specVersion, generatedBy
 - 0.5.0 - Changed logic of handling parametrized tests. Previously they were marked as retries, now they are reported as separate tests

{pytest_json_ctrf-0.5.3 → pytest_json_ctrf-0.6.0}/pytest_json_ctrf.egg-info/SOURCES.txt

@@ -1,3 +1,4 @@
+MANIFEST.in
 README.md
 pyproject.toml
 ctrf/BaseMetadataReport.py
@@ -11,6 +12,4 @@ pytest_json_ctrf.egg-info/SOURCES.txt
 pytest_json_ctrf.egg-info/dependency_links.txt
 pytest_json_ctrf.egg-info/entry_points.txt
 pytest_json_ctrf.egg-info/requires.txt
-pytest_json_ctrf.egg-info/top_level.txt
-tests/test_example.py
-tests/test_plugin.py
+pytest_json_ctrf.egg-info/top_level.txt

pytest_json_ctrf-0.6.0/pytest_json_ctrf.egg-info/top_level.txt

@@ -0,0 +1 @@
+ctrf

pytest_json_ctrf-0.5.3/ctrf/TestObject.py

@@ -1,107 +0,0 @@
-from enum import Enum
-from typing import Any, Optional, List, Dict
-
-from pytest import TestReport
-
-
-class TestStatus(Enum):
-    PASSED = "passed"
-    FAILED = "failed"
-    SKIPPED = "skipped"
-    PENDING = "pending"
-    OTHER = "other"
-
-
-class TestObject:
-    name: str
-    _status: TestStatus
-    raw_status: Optional[str]
-    start: int
-    stop: int
-    retries: int
-    message: Optional[str]
-
-    worker_id: Optional[str]
-    file_path: str
-    tags: List[str]
-    browser: Optional[str]
-    trace: Optional[str]
-
-    def __init__(self, report: TestReport, worker_id: Optional[str] = None):
-        self.name = report.nodeid.split('[')[0]
-        self._status = TestStatus.PENDING
-        self._status = self.set_status(report)
-        self.raw_status = None
-        self.start = 0
-        self.stop = 0
-        self.retries = 0
-        self.message = None
-
-        self.worker_id = worker_id
-        self.file_path = report.location[0]
-        self.tags: List[str] = []
-        self.browser = None
-        self.trace = None
-
-    @property
-    def status(self) -> TestStatus:
-        return self._status
-
-    @status.setter
-    def status(self, value: TestStatus) -> None:
-        if self._status in (TestStatus.PENDING, TestStatus.PASSED):
-            self._status = value
-            return
-        elif self._status in (TestStatus.SKIPPED, TestStatus.FAILED):
-            return
-
-    def set_status(self, report: TestReport) -> TestStatus:
-        if self._status in (TestStatus.SKIPPED, TestStatus.FAILED):
-            return self._status
-        elif report.skipped:
-            self._status = TestStatus.SKIPPED
-        elif report.failed:
-            self._status = TestStatus.FAILED
-            self.raw_status = f"{report.when}_{report.outcome}"
-            self.message = f"The test failed in the {report.when} phase"
-            if hasattr(report, 'longreprtext'):
-                if "AssertionError" in report.longreprtext:
-                    self.message += " due to an assertion error"
-                elif "Exception" in report.longreprtext:
-                    self.message += " due to an exception"
-        elif report.passed:
-            self._status = TestStatus.PASSED
-        else:
-            self._status = TestStatus.OTHER
-        return self._status
-
-    def update(self, report: TestReport) -> None:
-        self.set_status(report)
-        if report.when == "setup" and self.start == 0:
-            self.start = int(report.start * 1000)
-        if report.when == "teardown" and self.stop == 0:
-            self.stop = int(report.stop * 1000)
-        if report.longrepr and len(report.longreprtext) > 0:
-            self.trace = report.longreprtext
-        if hasattr(report, '_ctrf_metadata'):
-            self.tags = report._ctrf_metadata.get('tags')
-            self.browser = report._ctrf_metadata.get('browser')
-
-    def serialize(self) -> dict:
-        result: Dict[str, Any] = {
-            'name': self.name,
-            'status': self._status.value,
-            'raw_status': self.raw_status,
-            'duration': self.stop - self.start,
-            'start': self.start,
-            'stop': self.stop,
-            'retries': None if self.retries == 1 else self.retries,
-            'file_path': self.file_path,
-            'tags': self.tags,
-            'browser': self.browser,
-            'trace': self.trace,
-            'message': self.message
-        }
-        if self.worker_id:
-            result['extra'] = {'worker': self.worker_id}
-        return {key: value for key, value in result.items() if value not in [None, '', []]}

pytest_json_ctrf-0.5.3/pytest_json_ctrf.egg-info/top_level.txt

@@ -1,2 +0,0 @@
-build
-ctrf

pytest_json_ctrf-0.5.3/tests/test_example.py

@@ -1,65 +0,0 @@
-from pytest import mark, fixture
-
-
-@fixture
-def pre_fail():
-    raise Exception("This is a test exception")
-
-
-@fixture
-def post_fail():
-    yield
-    raise Exception("This is a test exception")
-
-
-def test_do_calculations_1_pass():
-    assert 2 + 2 == 4
-
-
-@mark.regression
-def test_do_calculations_2_pass():
-    assert 3 + 3 == 6
-
-
-@mark.skip
-def test_do_calculations_3_skip():
-    assert 4 + 4 == 8
-
-
-@mark.smoke
-def test_do_calculations_4_smoke():
-    assert 5 + 5 == 10
-
-
-def test_do_calculations_5_fail():
-    assert 6 + 6 == 13
-
-
-def test_do_calculations_6_error():
-    raise Exception("This is a test exception")
-
-
-def test_do_calculations_7_pre_error(pre_fail):
-    assert 7 + 7 == 14
-
-
-def test_do_calculations_8_post_error(post_fail):
-    assert 8 + 8 == 16
-
-
-@mark.parametrize('param', [1, 2, 3, 4, 5])
-def test_parametrized(param):
-    assert param < 10
-
-
-@mark.parametrize('param', [6, 7, 8, 9, 10])
-def test_parametrized_fail(param):
-    assert param != 8
-
-@mark.test_id(1234, hello='world')
-def test_with_test_id():
-    assert 1 == 1
-
-@mark.test_id(1235)
-def test_with_test_id_2():
-    assert 1 == 1

pytest_json_ctrf-0.5.3/tests/test_plugin.py

@@ -1,108 +0,0 @@
-import json
-from pytest import Pytester, fixture
-
-pytest_plugins = ["pytester", "xdist", "ctrf"]
-test_file = "tests/test_example.py"
-basic_test_args = ["-k", "test_example"]
-
-
-@fixture
-def ctrf_report_sync(pytester: Pytester):
-    pytester.copy_example(test_file)
-    pytester.runpytest(*[*basic_test_args, "--ctrf", "report.json"])
-    with open(pytester.path / "report.json") as file:
-        yield json.load(file)
-
-
-@fixture
-def ctrf_report_xdist(pytester: Pytester):
-    pytester.copy_example(test_file)
-    pytester.runpytest(*[*basic_test_args, "--ctrf", "report.json", "-n", "3"])
-    with open(pytester.path / "report.json") as file:
-        yield json.load(file)
-
-
-def test_without_plugin_no_xdist(pytester: Pytester):
-    pytester.copy_example(test_file)
-    result = pytester.runpytest(*basic_test_args)
-    result.assert_outcomes(passed=15, skipped=1, failed=3, errors=2)
-
-
-def test_without_plugin_with_xdist(pytester: Pytester):
-    pytester.copy_example(test_file)
-    result = pytester.runpytest(*[*basic_test_args, "-n", "3"])
-    result.assert_outcomes(passed=15, skipped=1, failed=3, errors=2)
-    assert "created: 3/3 workers" in result.stdout.str()
-
-
-def test_with_markers(pytester: Pytester):
-    pytester.copy_example(test_file)
-    args = [*basic_test_args, "--ctrf", "report.json", "-m smoke"]
-    pytester.runpytest(*args)
-    with open(pytester.path / "report.json") as file:
-        report = json.load(file)
-    assert report["results"]["summary"]["tests"] == 1
-    assert report["results"]["summary"]["passed"] == 1
-    assert report["results"]["summary"]["failed"] == 0
-    assert report["results"]["summary"]["skipped"] == 0
-
-
-def test_with_plugin_no_xdist(ctrf_report_sync):
-    assert ctrf_report_sync["results"]["summary"]["tests"] == 20
-    assert ctrf_report_sync["results"]["summary"]["passed"] == 14
-    assert ctrf_report_sync["results"]["summary"]["failed"] == 5
-    assert ctrf_report_sync["results"]["summary"]["skipped"] == 1
-
-
-def test_with_plugin_failed_details(ctrf_report_sync):
-    for test in ctrf_report_sync["results"]["tests"]:
-        if test["status"] == "failed" and test.get("raw_status") == "call_failed":
-            assert test.get('trace') is not None
-            assert test.get("message") is not None
-            assert test.get("raw_status") is not None
-
-
-def test_any_test_has_timestamps(ctrf_report_sync):
-    for test in ctrf_report_sync["results"]["tests"]:
-        assert test.get("start") is not None
-        assert test.get("stop") is not None
-        assert test.get("duration") is not None
-
-
-def test_with_plugin_with_xdist(ctrf_report_xdist):
-    assert ctrf_report_xdist["results"]["summary"]["tests"] == 20
-    assert ctrf_report_xdist["results"]["summary"]["passed"] == 14
-    assert ctrf_report_xdist["results"]["summary"]["failed"] == 5
-    assert ctrf_report_xdist["results"]["summary"]["skipped"] == 1
-
-
-def test_with_plugin_failed_details_xdist(ctrf_report_xdist):
-    for test in ctrf_report_xdist["results"]["tests"]:
-        if test["status"] == "failed" and test.get("raw_status") == "call_failed":
-            assert test.get('trace') is not None
-            assert test.get("message") is not None
-            assert test.get("raw_status") is not None
-
-
-def test_any_test_has_timestamps_xdist(ctrf_report_xdist):
-    for test in ctrf_report_xdist["results"]["tests"]:
-        assert test.get("start") is not None
-        assert test.get("stop") is not None
-        assert test.get("duration") is not None
-
-def test_parametrize_marker_not_in_tags(pytester: Pytester):
-    pytester.makepyfile("""
-        import pytest
-
-        @pytest.mark.parametrize('param', [{'key': 'value', 'data': 'x' * 1000}, {'key': 'other', 'data': 'y' * 1000}], ids=['case1', 'case2'])
-        def test_parametrized(param):
-            assert True
-    """)
-    pytester.runpytest("--ctrf", "report.json")
-    with open(pytester.path / "report.json") as file:
-        report = json.load(file)
-    for test in report["results"]["tests"]:
-        tags = test.get("tags", [])
-        assert all("parametrize" not in tag for tag in tags), (
-            f"parametrize marker should not appear in tags, but got: {tags}"
-        )