spl2-testing-framework 1.6.0__tar.gz → 1.7.1__tar.gz

{spl2_testing_framework-1.6.0 → spl2_testing_framework-1.7.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: spl2-testing-framework
-Version: 1.6.0
+Version: 1.7.1
 Summary: SPL2 Testing Framework
 Home-page: https://github.com/splunk/spl2-testing-framework
 Keywords: SPL2 testing framework
@@ -131,6 +131,8 @@ It is possible to pass more options to the command, which works also with pytest
 
 Additionally, the following options are supported:
 
+* `--test_dir` - directory where test files (`.test.json`, `.test.spl2`, `module.json`) are located. Defaults to the current working directory. Discovery is recursive.
+* `--code_dir` - directory where SPL2 code modules (`.spl2`) are located. Defaults to the current working directory. Falls back to `--test_dir` if the module is not found. See [Separate code and test directories](#separate-code-and-test-directories) for details.
 * `--ignore_empty_strings` - to ignore empty strings in the results
 * `--ignore_additional_fields_in_actual` - to ignore fields present in actual results but not in expected results (useful when actual results contain extra fields that should not affect comparison)
 * `--create_comparison_sheet` - to create a comparison sheet in `comparison_box_test` folder using actual and expected outputs (works only when running box tests)
@@ -152,10 +154,53 @@ Sample configuration which may be used:
     - Parameters:
       `--type [cli | splunk | cloud] --test_dir /tests/resources -o log_cli=true --log-cli-level=INFO --verbose`
     - If test dir is not specified, current working directory will be used
+    - Use `--code_dir` if SPL2 code modules are in a separate directory from tests
     - If necessary another pytest options can be added
 
 Note: It's necessary to set "pytest" as default test runner in PyCharm settings
 
+### Separate code and test directories
+
+By default, the framework expects SPL2 code modules (`.spl2` files) to live alongside their test files. The `--code_dir` option allows you to keep code and tests in separate directory trees.
+
+#### How it works
+
+| What is searched | Primary directory | Fallback directory |
+|------------------|-------------------|--------------------|
+| **Code modules** (`.spl2`) | `--code_dir` | `--test_dir` |
+| **Test files** (`.test.json`, `.test.spl2`) | `--test_dir` | `--code_dir` |
+
+Both searches are recursive. If the primary directory has no results, the framework falls back to the other directory and logs the action.
+
+When multiple files with the same name are found within a directory, a warning is logged and the lexicographically first match is used.
+
+#### Example project layout
+
+```
+my-project/
+├── src/                          # --code_dir
+│   ├── network_traffic.spl2
+│   └── dns_lookup.spl2
+└── tests/                        # --test_dir
+    ├── network_traffic.test.json
+    ├── dns_lookup.test.json
+    └── dns_lookup.test.spl2
+```
+
+Run with:
+
+```bash
+spl2_tests_run cli --test_dir tests --code_dir src
+```
+
+#### Backward compatibility
+
+When `--code_dir` is not specified (or both flags point to the same directory), the framework behaves exactly as before — code modules are found next to their test files via the same recursive search.
+
+#### Applies to all test types
+
+The `--code_dir` option works for box tests, unit tests, and single SPL2 file execution.
+
 ## Executing a spl2 file
 
 This framework also supports executing a single spl2 file and prints the results in command line as well as a log file.
@@ -179,6 +224,7 @@ It is possible to pass more options to the command, which works also with pytest
 
 * `--test_dir` - Path in which the template file and module.json are available. If not provided, it will look for the
   current directory for the template file and module.json file
+* `--code_dir` - Path where SPL2 code modules are located, if different from `--test_dir`. Falls back to `--test_dir` when not set. See [Separate code and test directories](#separate-code-and-test-directories).
 * `--template_file` - The spl2 template file to execute
 * `--sample_file` - A file containing all the samples required for the template file. If not provided, it will look for
   the samples in module.json file of the corresponding template file
@@ -214,8 +260,8 @@ When set (`--check_splunk_results=<option>`), box tests (after running the SPL2
 
 | Value | Behavior |
 |-------|----------|
-| **CIM** | For each output event, push to Splunk, query by host, then validate that the returned event has the expected CIM fields (from `expected_cim_fields.cim_fields` in the box test). Uses `validate_compatibility()`. Skips the test if the box test has no `cim_fields`. |
-| **TA** | For each output event, push to Splunk, query by host, then assert that the returned event equals the expected output event for that index. |
+| **CIM** | For each output event, ingest via HEC, load back from Splunk, then check CIM fields from `expected_cim_fields.cim_fields` with `validate_compatibility(actual, expected)`. Skips if the box test has no `cim_fields`. |
+| **TA** | For each output event, ingest via HEC, load back from Splunk, then assert equality with the expected row for that index. |
 
 For **TA** only, you can name top-level fields to drop on **both** the queried event and each `expected_destination_result` row by adding **`ignore_fields_in_splunk_check`**: an object mapping **field name → short reason**. Use the same key under `test` in each entry in `*.test.json` (same file as `expected_destination_result`):
 

{spl2_testing_framework-1.6.0 → spl2_testing_framework-1.7.1}/README.md

@@ -98,6 +98,8 @@ It is possible to pass more options to the command, which works also with pytest
 
 Additionally, the following options are supported:
 
+* `--test_dir` - directory where test files (`.test.json`, `.test.spl2`, `module.json`) are located. Defaults to the current working directory. Discovery is recursive.
+* `--code_dir` - directory where SPL2 code modules (`.spl2`) are located. Defaults to the current working directory. Falls back to `--test_dir` if the module is not found. See [Separate code and test directories](#separate-code-and-test-directories) for details.
 * `--ignore_empty_strings` - to ignore empty strings in the results
 * `--ignore_additional_fields_in_actual` - to ignore fields present in actual results but not in expected results (useful when actual results contain extra fields that should not affect comparison)
 * `--create_comparison_sheet` - to create a comparison sheet in `comparison_box_test` folder using actual and expected outputs (works only when running box tests)
@@ -119,10 +121,53 @@ Sample configuration which may be used:
     - Parameters:
       `--type [cli | splunk | cloud] --test_dir /tests/resources -o log_cli=true --log-cli-level=INFO --verbose`
     - If test dir is not specified, current working directory will be used
+    - Use `--code_dir` if SPL2 code modules are in a separate directory from tests
     - If necessary another pytest options can be added
 
 Note: It's necessary to set "pytest" as default test runner in PyCharm settings
 
+### Separate code and test directories
+
+By default, the framework expects SPL2 code modules (`.spl2` files) to live alongside their test files. The `--code_dir` option allows you to keep code and tests in separate directory trees.
+
+#### How it works
+
+| What is searched | Primary directory | Fallback directory |
+|------------------|-------------------|--------------------|
+| **Code modules** (`.spl2`) | `--code_dir` | `--test_dir` |
+| **Test files** (`.test.json`, `.test.spl2`) | `--test_dir` | `--code_dir` |
+
+Both searches are recursive. If the primary directory has no results, the framework falls back to the other directory and logs the action.
+
+When multiple files with the same name are found within a directory, a warning is logged and the lexicographically first match is used.
+
+#### Example project layout
+
+```
+my-project/
+├── src/                          # --code_dir
+│   ├── network_traffic.spl2
+│   └── dns_lookup.spl2
+└── tests/                        # --test_dir
+    ├── network_traffic.test.json
+    ├── dns_lookup.test.json
+    └── dns_lookup.test.spl2
+```
+
+Run with:
+
+```bash
+spl2_tests_run cli --test_dir tests --code_dir src
+```
+
+#### Backward compatibility
+
+When `--code_dir` is not specified (or both flags point to the same directory), the framework behaves exactly as before — code modules are found next to their test files via the same recursive search.
+
+#### Applies to all test types
+
+The `--code_dir` option works for box tests, unit tests, and single SPL2 file execution.
+
 ## Executing a spl2 file
 
 This framework also supports executing a single spl2 file and prints the results in command line as well as a log file.
@@ -146,6 +191,7 @@ It is possible to pass more options to the command, which works also with pytest
 
 * `--test_dir` - Path in which the template file and module.json are available. If not provided, it will look for the
   current directory for the template file and module.json file
+* `--code_dir` - Path where SPL2 code modules are located, if different from `--test_dir`. Falls back to `--test_dir` when not set. See [Separate code and test directories](#separate-code-and-test-directories).
 * `--template_file` - The spl2 template file to execute
 * `--sample_file` - A file containing all the samples required for the template file. If not provided, it will look for
   the samples in module.json file of the corresponding template file
@@ -181,8 +227,8 @@ When set (`--check_splunk_results=<option>`), box tests (after running the SPL2
 
 | Value | Behavior |
 |-------|----------|
-| **CIM** | For each output event, push to Splunk, query by host, then validate that the returned event has the expected CIM fields (from `expected_cim_fields.cim_fields` in the box test). Uses `validate_compatibility()`. Skips the test if the box test has no `cim_fields`. |
-| **TA** | For each output event, push to Splunk, query by host, then assert that the returned event equals the expected output event for that index. |
+| **CIM** | For each output event, ingest via HEC, load back from Splunk, then check CIM fields from `expected_cim_fields.cim_fields` with `validate_compatibility(actual, expected)`. Skips if the box test has no `cim_fields`. |
+| **TA** | For each output event, ingest via HEC, load back from Splunk, then assert equality with the expected row for that index. |
 
 For **TA** only, you can name top-level fields to drop on **both** the queried event and each `expected_destination_result` row by adding **`ignore_fields_in_splunk_check`**: an object mapping **field name → short reason**. Use the same key under `test` in each entry in `*.test.json` (same file as `expected_destination_result`):
 

{spl2_testing_framework-1.6.0 → spl2_testing_framework-1.7.1}/pyproject.toml

@@ -15,7 +15,7 @@
 
 [tool.poetry]
 name = "spl2-testing-framework"
-version = "1.6.0"
+version = "1.7.1"
 description = "SPL2 Testing Framework"
 authors = ["mgazda-splunk <mgazda@splunk.com>"]
 readme = "README.md"

{spl2_testing_framework-1.6.0 → spl2_testing_framework-1.7.1}/spl2_testing_framework/conftest.py

@@ -55,6 +55,7 @@ CHECK_SPLUNK_RESULTS = "check_splunk_results"
 PERFORMANCE_CHECK = "performance_check"
 TEST_TYPE = "type"
 TEST_DIR = "test_dir"
+CODE_DIR = "code_dir"
 LIMIT_TESTS = "limit_tests"
 CLI_BENCH = "cli_bench"
 
@@ -70,6 +71,7 @@ _LOGGER = logging.getLogger(__name__)
 def pytest_addoption(parser):
     parser.addoption(f"--{TEST_TYPE}", action="store", default="cli")
     parser.addoption(f"--{TEST_DIR}", action="store", default=".")
+    parser.addoption(f"--{CODE_DIR}", action="store", default=".")
     parser.addoption(
         f"--{PERFORMANCE_CHECK}",
         action="store",
@@ -93,13 +95,16 @@ def pytest_generate_tests(metafunc):
     """This function is used for dynamic generation of tests based on the test files in the directory specified by the
     --test_dir option"""
     test_dir = metafunc.config.getoption(TEST_DIR)
+    code_dir = metafunc.config.getoption(CODE_DIR)
     performance_check = metafunc.config.getoption(PERFORMANCE_CHECK)
     limit_tests = metafunc.config.getoption(LIMIT_TESTS)
     _set_results_options(metafunc)
 
     if BOX_TEST in metafunc.fixturenames:
         _LOGGER.info("Collecting Box Tests")
-        module_tests = BoxTestDiscovery(test_dir, performance_check, limit_tests)
+        module_tests = BoxTestDiscovery(
+            test_dir, performance_check, limit_tests, code_dir=code_dir
+        )
         module_tests.discover_tests()
         metafunc.parametrize(
             "%s" % BOX_TEST, module_tests.tests, ids=module_tests.tests.get_ids()
@@ -119,6 +124,7 @@ def pytest_generate_tests(metafunc):
             sample_delimiter,
             performance_check=performance_check,
             limit_tests=limit_tests,
+            code_dir=code_dir,
         )
         try:
             spl2_pipelines.discover_tests()
@@ -132,7 +138,7 @@ def pytest_generate_tests(metafunc):
 
     if UNIT_TEST in metafunc.fixturenames:
         _LOGGER.info("Collecting Unit Tests")
-        unit_tests = UTDiscovery(test_dir, limit_tests)
+        unit_tests = UTDiscovery(test_dir, limit_tests, code_dir=code_dir)
         unit_tests.discover_tests()
         metafunc.parametrize(
             UNIT_TEST, unit_tests.tests, ids=unit_tests.tests.get_ids()

{spl2_testing_framework-1.6.0 → spl2_testing_framework-1.7.1}/spl2_testing_framework/tools/spl2test_runner.py

@@ -23,6 +23,9 @@ from spl2_testing_framework.tools.performance import PerformanceCheck
 from spl2_testing_framework.tools.search_clients.cli_search_client import (
     CLISearchClient,
 )
+from spl2_testing_framework.tools.splunk_integration.splunk_handler import (
+    SplunkEventNotFoundError,
+)
 from spl2_testing_framework.tools.test_types import UnitTest, BoxTest, SingleSPL2
 from spl2_testing_framework.tools.utils import (
     _make_functions_visible_for_testing,
@@ -227,6 +230,15 @@ class SPL2TestRunner:
             return row
         return {k: v for k, v in row.items() if k not in ignore_keys}
 
+    def _splunk_process_event(self, box_test: BoxTest, event):
+        try:
+            return self.splunk_handler.process_event(event)
+        except SplunkEventNotFoundError as e:
+            raise SplunkEventNotFoundError(
+                e.event_uuid,
+                context=f"box_test={box_test.name!r}",
+            ) from e
+
     def _check_splunk_results(self, box_test, job):
         output_result = job.result.get("destination", [])
         if self.check_splunk_results == "CIM":
@@ -234,13 +246,13 @@ class SPL2TestRunner:
                 pytest.skip("CIM output not available for this test")
 
             for event in output_result:
-                processed_event = self.splunk_handler.process_event(event)
-                validate_compatibility(box_test.cim_fields, processed_event)
+                processed_event = self._splunk_process_event(box_test, event)
+                validate_compatibility(processed_event, box_test.cim_fields)
         elif self.check_splunk_results == "TA":
             ignore_spec = box_test.ignore_fields_in_splunk_check
             ignore_keys = frozenset(ignore_spec.keys())
             for event, expected in zip(output_result, box_test.output):
-                processed_event = self.splunk_handler.process_event(event)
+                processed_event = self._splunk_process_event(box_test, event)
                 actual = self._strip_ignored_fields_for_splunk_compare(
                     processed_event, ignore_keys
                 )

{spl2_testing_framework-1.6.0 → spl2_testing_framework-1.7.1}/spl2_testing_framework/tools/splunk_integration/splunk_handler.py

@@ -14,6 +14,7 @@
 
 import logging
 import uuid
+from typing import Optional
 
 import requests
 import splunklib.client as client
@@ -26,6 +27,21 @@ from requests.auth import HTTPBasicAuth
 _LOGGER = logging.getLogger(__name__)
 
 
+SPL2_TF_EVENT_UUID_FIELD = "spl2_testing_framework_event_uuid"
+SPL2_TF_DEFAULT_HEC_HOST = "spl2-testing-framework"
+
+
+class SplunkEventNotFoundError(Exception):
+    """Raised when HEC ingest succeeded but search returned no matching event."""
+
+    def __init__(self, event_uuid: str, *, context: Optional[str] = None):
+        self.event_uuid = event_uuid
+        msg = f"No event found after HEC submit (event_uuid={event_uuid!r})"
+        if context:
+            msg = f"{context}: {msg}"
+        super().__init__(msg)
+
+
 class SplunkHandler:
     def __init__(self, config, auth):
         self._config = config
@@ -36,18 +52,45 @@ class SplunkHandler:
         self._create_index()
 
     def process_event(self, event):
-        host = f"test_{uuid.uuid4()}"
-
-        self.push_to_hec(event, host)
-        res = self.get_events_by_host_and_index(host)
+        event_uuid = str(uuid.uuid4())
+        metadata_host = event.get("host")
+        if metadata_host is not None and str(metadata_host).strip() != "":
+            hec_host = str(metadata_host).strip()
+        else:
+            hec_host = SPL2_TF_DEFAULT_HEC_HOST
+
+        self.push_to_hec(event, hec_host, event_uuid)
+        res = self.get_events_by_event_uuid(event_uuid)
         if not res:
-            raise Exception("No event found.")
+            raise SplunkEventNotFoundError(event_uuid)
 
         if len(res) > 1:
             _LOGGER.error("Found more than one event, the result may not be correct")
-        return res[0]
-
-    def push_to_hec(self, data, host):
+        out = dict(res[0])
+        out.pop(SPL2_TF_EVENT_UUID_FIELD, None)
+        return out
+
+    @staticmethod
+    def _hec_event_url(raw_url: str) -> str:
+        if "/services/collector/raw" in raw_url:
+            return raw_url.replace(
+                "/services/collector/raw", "/services/collector/event"
+            )
+        return raw_url
+
+    @staticmethod
+    def _raw_as_event_string(raw):
+        """Normalize pipeline `_raw` for HEC JSON `event` (must be a string).
+
+        ``str`` is passed through ``str()``; ``bytes`` are decoded as UTF-8 with
+        replacement on invalid sequences (avoids :class:`UnicodeDecodeError` for
+        odd binary payloads; prefer valid UTF-8 log text in normal use).
+        """
+        if isinstance(raw, bytes):
+            return raw.decode("utf-8", errors="replace")
+        return str(raw)
+
+    def push_to_hec(self, data, host, event_uuid):
         sourcetype = data.get("sourcetype", "UNKNOWN")
 
         if sourcetype == "UNKNOWN":
@@ -55,29 +98,31 @@ class SplunkHandler:
 
         source = data.get("source", None)
 
-        params = {
+        payload = {
+            "event": SplunkHandler._raw_as_event_string(data["_raw"]),
             "sourcetype": sourcetype,
             "index": self.splunk_config["index"],
             "host": host,
+            "fields": {SPL2_TF_EVENT_UUID_FIELD: event_uuid},
         }
-
         if source is not None:
-            params["source"] = source
-
-        event_data = data["_raw"].encode("utf-8")
+            payload["source"] = source
 
         response = requests.post(
-            url=self.splunk_config["url"],
-            data=event_data,
-            params=params,
+            url=self._hec_event_url(self.splunk_config["url"]),
+            json=payload,
             auth=self._auth,
             verify=False,
         )
 
         response.raise_for_status()
 
-    def get_events_by_host_and_index(self, host):
-        SPL_TEMPLATE = "| search index={index} host={host} | fields *"
+    @staticmethod
+    def _splunk_search_term(value: str) -> str:
+        return '"' + str(value).replace("\\", "\\\\").replace('"', '\\"') + '"'
+
+    def get_events_by_event_uuid(self, event_uuid):
+        SPL_TEMPLATE = "| search index={index} {uuid_field}={uuid} | fields *"
         # Connect to Splunk
         service = client.connect(
             host=self.splunk_config["ip"],
@@ -86,7 +131,11 @@ class SplunkHandler:
             password=self.splunk_config["password"],
             app="search",
         )
-        SPL = SPL_TEMPLATE.format(index=self.splunk_config["index"], host=host)
+        SPL = SPL_TEMPLATE.format(
+            index=self.splunk_config["index"],
+            uuid_field=SPL2_TF_EVENT_UUID_FIELD,
+            uuid=self._splunk_search_term(event_uuid),
+        )
 
         results_all = []
         retry = 0

{spl2_testing_framework-1.6.0 → spl2_testing_framework-1.7.1}/spl2_testing_framework/tools/test_discovery.py

@@ -19,6 +19,8 @@ import logging
 import os
 import pathlib
 import re
+from collections import Counter
+from typing import Optional
 
 from spl2_testing_framework.tools.results import Metrics, Results
 from spl2_testing_framework.tools.test_types import (
@@ -34,33 +36,151 @@ _LOGGER = logging.getLogger(__name__)
 class SamplesNotFoundException(Exception):
     """An Exception to be raised if samples are not found"""
 
-    pass
+
+class CodeModuleNotFoundError(Exception):
+    """Raised when a code module cannot be found in any search directory"""
 
 
 class TestDiscovery(abc.ABC):
     """Base class for test discovery classes"""
 
-    def __init__(self, path: str, performance_check: str, limit_tests: int = None):
+    def __init__(
+        self,
+        path: str,
+        performance_check: str,
+        limit_tests: Optional[int] = None,
+        code_dir: Optional[str] = None,
+    ):
         self.path = pathlib.Path(path)
+        self.code_dir = pathlib.Path(code_dir) if code_dir else self.path
         self._performance_check = performance_check
         self.limit_tests = limit_tests
         self.tests = TestSuite()
 
+    @property
+    def _dirs_differ(self) -> bool:
+        try:
+            return self._dirs_differ_cached
+        except AttributeError:
+            self._dirs_differ_cached = self.code_dir.resolve() != self.path.resolve()
+            return self._dirs_differ_cached
+
+    @staticmethod
+    def _warn_duplicate_basenames(
+        paths: list, directory: pathlib.Path, label: str
+    ) -> None:
+        names = [p.name for p in paths]
+        dupes = [name for name, count in Counter(names).items() if count > 1]
+        if dupes:
+            _LOGGER.warning(
+                "Multiple %s files with the same name found in '%s': %s",
+                label,
+                directory,
+                dupes,
+            )
+
     def _read_tests(self, test_name: str) -> None:
-        tests_paths = self.path.rglob(test_name)
+        _LOGGER.info(
+            "Searching for test files '%s' in test directory: %s",
+            test_name,
+            self.path,
+        )
+        tests_paths = list(self.path.rglob(test_name))
+        if tests_paths:
+            self._warn_duplicate_basenames(tests_paths, self.path, "test")
+            _LOGGER.info(
+                "Found %d test file(s) in test directory: %s",
+                len(tests_paths),
+                [str(p) for p in tests_paths],
+            )
+        elif self._dirs_differ:
+            _LOGGER.info(
+                "No test files '%s' found in test directory '%s', "
+                "falling back to code directory: %s",
+                test_name,
+                self.path,
+                self.code_dir,
+            )
+            tests_paths = list(self.code_dir.rglob(test_name))
+            if tests_paths:
+                self._warn_duplicate_basenames(tests_paths, self.code_dir, "test")
+                _LOGGER.info(
+                    "Found %d test file(s) in code directory: %s",
+                    len(tests_paths),
+                    [str(p) for p in tests_paths],
+                )
+            else:
+                _LOGGER.warning(
+                    "No test files '%s' found in code directory either.", test_name
+                )
+        else:
+            _LOGGER.warning("No test files '%s' found in test directory.", test_name)
+
         self._raw_tests = {
             x: pathlib.Path(x).read_text(encoding="utf-8") for x in tests_paths
         }
 
+    def _search_directory(
+        self, filename: str, directory: pathlib.Path, label: str
+    ) -> Optional[pathlib.Path]:
+        """Search a directory recursively for *filename*.
+
+        Returns the first match (sorted lexicographically) or ``None``.
+        Warns when multiple matches exist.
+        """
+        matches = sorted(directory.rglob(filename))
+        if not matches:
+            return None
+        if len(matches) > 1:
+            _LOGGER.warning(
+                "Multiple matches for code module '%s' in %s '%s': %s. "
+                "Using first match.",
+                filename,
+                label,
+                directory,
+                [str(m) for m in matches],
+            )
+        _LOGGER.info("Resolved code module (%s): %s", label, matches[0])
+        return matches[0]
+
+    def _resolve_code_module(self, filename: str) -> pathlib.Path:
+        """Resolve a code module by filename.
+
+        Searches code_dir first (recursive), falls back to test_dir.
+        Warns when multiple files share the same name in a directory.
+        """
+        _LOGGER.info(
+            "Searching for code module '%s' in code directory: %s",
+            filename,
+            self.code_dir,
+        )
+        result = self._search_directory(filename, self.code_dir, "code directory")
+        if result is not None:
+            return result
+
+        if self._dirs_differ:
+            _LOGGER.info(
+                "Code module '%s' not found in code directory '%s', "
+                "falling back to test directory: %s",
+                filename,
+                self.code_dir,
+                self.path,
+            )
+            result = self._search_directory(filename, self.path, "test directory")
+            if result is not None:
+                return result
+
+        raise CodeModuleNotFoundError(
+            f"Code module '{filename}' not found in code directory '{self.code_dir}' "
+            f"or test directory '{self.path}'"
+        )
+
 
 class BoxTestDiscovery(TestDiscovery):
     """Discovers tests for box tests, looking into module.test.json files"""
 
     _IGNORE_FIELDS_IN_SPLUNK_CHECK_KEY = "ignore_fields_in_splunk_check"
 
-    def __init__(self, path: str, performance_check: str, limit_tests: int = None):
-        super().__init__(path, performance_check, limit_tests)
-
     @classmethod
     def _parse_ignore_fields_in_splunk_check(cls, test_block: dict) -> dict:
         raw = test_block.get(cls._IGNORE_FIELDS_IN_SPLUNK_CHECK_KEY)
@@ -110,7 +230,6 @@ class BoxTestDiscovery(TestDiscovery):
         }
 
         for module_path, content in self._parsed_modules.items():
-            path = module_path.parent
             tests_added = 0
             for entry in content:
                 if self.limit_tests is not None and tests_added >= self.limit_tests:
@@ -121,7 +240,7 @@ class BoxTestDiscovery(TestDiscovery):
                 test_block = entry["test"]
                 name = entry["filename"].replace(".spl2", "")
                 test_input = test_block["source"]
-                code_module_path = path / entry["filename"]
+                code_module_path = self._resolve_code_module(entry["filename"])
                 output = [
                     Results(x)
                     for x in test_block.get("expected_destination_result", [])
@@ -161,9 +280,10 @@ class SingleSPL2Discovery(TestDiscovery):
         sample_file=None,
         sample_delimiter="\n",
         performance_check="no",
-        limit_tests: int = None,
+        limit_tests: Optional[int] = None,
+        code_dir: Optional[str] = None,
     ):
-        super().__init__(path, performance_check, limit_tests)
+        super().__init__(path, performance_check, limit_tests, code_dir=code_dir)
         self.template_file = template_file
         self.sample_file = sample_file
         self.sample_delimiter = sample_delimiter
@@ -180,7 +300,9 @@ class SingleSPL2Discovery(TestDiscovery):
             _LOGGER.info("Reading samples from provided sample file.")
             events = self._read_samples_from_file(delimiter=self.sample_delimiter)
             name = self.template_file.replace(".spl2", "")
-            code_module_path = next(self.path.rglob(self.template_file), None)
+            code_module_path = self._resolve_code_module(
+                os.path.basename(self.template_file)
+            )
         else:
             _LOGGER.info("Reading samples from corresponding module.json file")
             self._parsed_modules = {
@@ -188,12 +310,11 @@ class SingleSPL2Discovery(TestDiscovery):
             }
             events = None
             for module_path, content in self._parsed_modules.items():
-                path = module_path.parent
                 break_outer = False
                 for test in content:
                     if test["filename"] == os.path.basename(self.template_file):
                         name = test["filename"].replace(".spl2", "")
-                        code_module_path = path / test["filename"]
+                        code_module_path = self._resolve_code_module(test["filename"])
                         events = test["context"]["events"]
                         if isinstance(events, str):
                             events = events.split("\r\n")
@@ -246,8 +367,15 @@ class SingleSPL2Discovery(TestDiscovery):
 class UTDiscovery(TestDiscovery):
     """Discovers tests for unit tests, looking into .test.spl2 files"""
 
-    def __init__(self, path: str, limit_tests: int = None):
-        super().__init__(path, performance_check="no", limit_tests=limit_tests)
+    def __init__(
+        self,
+        path: str,
+        limit_tests: Optional[int] = None,
+        code_dir: Optional[str] = None,
+    ):
+        super().__init__(
+            path, performance_check="no", limit_tests=limit_tests, code_dir=code_dir
+        )
 
     def discover_tests(self) -> None:
         self._read_tests("*.test.spl2")
@@ -264,7 +392,7 @@ class UTDiscovery(TestDiscovery):
             file_name = path.name
             content = path.read_text(encoding="utf-8")
             code_module_name = file_name.replace(".test.", ".")
-            code_module_path = path.parent / code_module_name
+            code_module_path = self._resolve_code_module(code_module_name)
 
             tests_added = 0
             for test in tests:

{spl2_testing_framework-1.6.0 → spl2_testing_framework-1.7.1}/spl2_testing_framework/tools/utils.py

@@ -67,13 +67,18 @@ def calculate_reduction_rate(original_event, reduced_event):
     return reduction_rates
 
 
-def validate_compatibility(expected, parsed_output):
-    """Validate if all fields from parsed_output are present in expected and have the same value."""
+def validate_compatibility(actual, expected):
+    """Check that every field in ``expected`` exists in ``actual`` with the same value.
+
+    Order matches the usual test convention ``assert actual == expected`` (observed
+    value first, reference second). For CIM checks, ``actual`` is the Splunk-processed
+    event and ``expected`` is ``box_test.cim_fields``.
+    """
     extracted_but_wrong = {}
     not_extracted = {}
     for field, value in expected.items():
-        if field in parsed_output:
-            if parsed_output[field] == value:
+        if field in actual:
+            if actual[field] == value:
                 _LOGGER.info(f"FIELD {field} in reduced event")
                 continue
             else: