spl2-testing-framework 1.7.3__tar.gz → 1.8.0__tar.gz

{spl2_testing_framework-1.7.3 → spl2_testing_framework-1.8.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: spl2-testing-framework
-Version: 1.7.3
+Version: 1.8.0
 Summary: SPL2 Testing Framework
 Home-page: https://github.com/splunk/spl2-testing-framework
 Keywords: SPL2 testing framework
@@ -35,7 +35,7 @@ Description-Content-Type: text/markdown
 
 ## Overview
 
-The **SPL2 Testing Framework** enables running SPL2 tests both locally ( or on any Splunk instance with SPL2
+The **SPL2 Testing Framework** enables running SPL2 tests both locally (or on any Splunk instance with SPL2
 orchestrator), remotely (using external cloud environments) or using cli.
 
 - **For Cloud**: It uses
@@ -138,6 +138,9 @@ Additionally, the following options are supported:
 * `--create_comparison_sheet` - to create a comparison sheet in `comparison_box_test` folder using actual and expected outputs (works only when running box tests)
 * `--cli_bench` - when running tests with CLI, this option enables benchmarking mode by adding `-b` and `-n` flags to the spl2-processor-cli command. The value specifies the number of events to test (e.g., `--cli_bench=1000` will add `-b -n 1000` to the CLI command). **In bench mode, all tests always succeed and CLI output is printed to logs** - this is useful for performance testing and benchmarking. This is only applicable when using `--type cli`.
 * `--check_splunk_results` - option to validate box test output by sending events to a Splunk instance (using HEC) and querying them back, with support for **CIM** (Common Information Model) and **TA** (Technology Add-on) checks.
+* `--reduction_stats` - when set, **box tests** record a reduction rate per run: input `_raw` vs output `_raw` (after pairing input/output events). Rates are grouped by **pipeline** (the `.spl2` code module stem) and **sourcetype** (from the input event if set, else the output event, else `UNKNOWN`). Off by default.
+* `--reduction_stats_file` - path to a JSON file written at the **end of the pytest session** with collected rates (default `reduction_stats.json`). Only used when `--reduction_stats` is set.
+* `--reduction_stats_full` - when set together with `--reduction_stats`, each pipeline’s JSON payload also includes a pandas **`describe()`** summary (count, mean, std, min, quartiles, max) per sourcetype column. Only used when `--reduction_stats` is set.
 
 * Note: The `pytest.ini.sample` file allows you to define command parameters. Just update the configurations, rename the
 file by removing the `.sample` extension, and execute the command.
@@ -153,6 +156,7 @@ Sample configuration which may be used:
     - Module: `spl2_testing_framework.test_runner`
     - Parameters:
       `--type [cli | splunk | cloud] --test_dir /tests/resources -o log_cli=true --log-cli-level=INFO --verbose`
+    - Optional: append `--reduction_stats` (and `--reduction_stats_file`, `--reduction_stats_full` if needed) to collect box-test reduction JSON as described above.
     - If test dir is not specified, current working directory will be used
     - Use `--code_dir` if SPL2 code modules are in a separate directory from tests
     - If necessary another pytest options can be added
@@ -252,7 +256,31 @@ command ("|")
 Content of this file will also be printed to stdout.
 
 This checks can be applied only to box tests, as assertions which are used in unit tests may impact spl2 pipeline
-performance. 
+performance.
+
+### Reduction rate statistics (box tests)
+
+Optional telemetry for **how much** each pipeline shrinks event payloads (length of input `_raw` vs matched output `_raw`), broken down by **pipeline** and **sourcetype**.
+
+**Enable**
+
+- Pass `--reduction_stats` to pytest (or add it in `pytest.ini` / your IDE run configuration).
+
+**Output**
+
+- During the session, each box test appends one or more floating-point rates into an in-memory structure: `pipeline_name → sourcetype → [rates…]`.
+- After all tests finish, if `--reduction_stats` was set, the framework logs a short **pandas** summary per pipeline and writes **`--reduction_stats_file`** (default `reduction_stats.json`).
+- With **`--reduction_stats_full`**, the JSON file also embeds per-pipeline **`describe()`** statistics for each sourcetype column (handy for dashboards or quick distribution checks).
+
+**How rates are computed (high level)**
+
+- Input events are parsed from the box test `source` string; output rows come from the job’s destination results.
+- Input/output rows are **paired** using similarity on `_raw` (with a threshold and a same-length positional fallback). Only **matched** pairs contribute rates.
+- For each pair, lengths use a **canonical JSON length** when `_raw` parses as JSON or a Python literal; otherwise the raw string length is used, so identical logical content can yield a **0** reduction rate even if string formatting differs.
+
+**Scope**
+
+- Applies to **box tests** only. Single-SPL2 runs and unit tests do not populate reduction stats.
 
 ### Check splunk results
 
@@ -263,13 +291,13 @@ When set (`--check_splunk_results=<option>`), box tests (after running the SPL2
 | **CIM** | For each output event, ingest via HEC, load back from Splunk, then check CIM fields from `expected_cim_fields.cim_fields` with `validate_compatibility(actual, expected)`. Skips if the box test has no `cim_fields`. |
 | **TA** | For each output event, ingest via HEC, load back from Splunk, then assert equality with the expected row for that index. |
 
-For **TA** only, you can name top-level fields to drop on **both** the queried event and each `expected_destination_result` row by adding **`ignore_fields_in_splunk_check`**: an object mapping **field name → short reason**. Use the same key under `test` in each entry in `*.test.json` (same file as `expected_destination_result`):
+You can name top-level fields to drop on **both** the pipeline/actual row and each matching `expected_destination_result` row before they are compared by adding **`ignore_fields_in_destination_compare`**: an object mapping **field name → short reason**. Use the same key under `test` in each entry in `*.test.json` (same file as `expected_destination_result`):
 
 ```json
 "test": {
   "source": "...",
   "expected_destination_result": [ ... ],
-  "ignore_fields_in_splunk_check": {
+  "ignore_fields_in_destination_compare": {
     "_raw": "Raw event differs after Splunk indexing",
     "_time": "Timestamp may vary due to ingestion latency"
   }
@@ -277,8 +305,7 @@ For **TA** only, you can name top-level fields to drop on **both** the queried e
 ```
 
 
-If the key is omitted, no fields are ignored. Used only when `--check_splunk_results=TA` (ignored for other modes and for the default 
-in-memory box test assert). If the file or key is missing, no fields are ignored.
+If the key is omitted, no fields are ignored. It applies to the **default in-memory** box test assert (no `--check_splunk_results`) and to **`--check_splunk_results=TA`**. It does **not** apply to **`--check_splunk_results=CIM`**. If the file or key is missing, no fields are ignored.
 
 #### Configuration:
 

{spl2_testing_framework-1.7.3 → spl2_testing_framework-1.8.0}/README.md

@@ -2,7 +2,7 @@
 
 ## Overview
 
-The **SPL2 Testing Framework** enables running SPL2 tests both locally ( or on any Splunk instance with SPL2
+The **SPL2 Testing Framework** enables running SPL2 tests both locally (or on any Splunk instance with SPL2
 orchestrator), remotely (using external cloud environments) or using cli.
 
 - **For Cloud**: It uses
@@ -105,6 +105,9 @@ Additionally, the following options are supported:
 * `--create_comparison_sheet` - to create a comparison sheet in `comparison_box_test` folder using actual and expected outputs (works only when running box tests)
 * `--cli_bench` - when running tests with CLI, this option enables benchmarking mode by adding `-b` and `-n` flags to the spl2-processor-cli command. The value specifies the number of events to test (e.g., `--cli_bench=1000` will add `-b -n 1000` to the CLI command). **In bench mode, all tests always succeed and CLI output is printed to logs** - this is useful for performance testing and benchmarking. This is only applicable when using `--type cli`.
 * `--check_splunk_results` - option to validate box test output by sending events to a Splunk instance (using HEC) and querying them back, with support for **CIM** (Common Information Model) and **TA** (Technology Add-on) checks.
+* `--reduction_stats` - when set, **box tests** record a reduction rate per run: input `_raw` vs output `_raw` (after pairing input/output events). Rates are grouped by **pipeline** (the `.spl2` code module stem) and **sourcetype** (from the input event if set, else the output event, else `UNKNOWN`). Off by default.
+* `--reduction_stats_file` - path to a JSON file written at the **end of the pytest session** with collected rates (default `reduction_stats.json`). Only used when `--reduction_stats` is set.
+* `--reduction_stats_full` - when set together with `--reduction_stats`, each pipeline’s JSON payload also includes a pandas **`describe()`** summary (count, mean, std, min, quartiles, max) per sourcetype column. Only used when `--reduction_stats` is set.
 
 * Note: The `pytest.ini.sample` file allows you to define command parameters. Just update the configurations, rename the
 file by removing the `.sample` extension, and execute the command.
@@ -120,6 +123,7 @@ Sample configuration which may be used:
     - Module: `spl2_testing_framework.test_runner`
     - Parameters:
       `--type [cli | splunk | cloud] --test_dir /tests/resources -o log_cli=true --log-cli-level=INFO --verbose`
+    - Optional: append `--reduction_stats` (and `--reduction_stats_file`, `--reduction_stats_full` if needed) to collect box-test reduction JSON as described above.
     - If test dir is not specified, current working directory will be used
     - Use `--code_dir` if SPL2 code modules are in a separate directory from tests
     - If necessary another pytest options can be added
@@ -219,7 +223,31 @@ command ("|")
 Content of this file will also be printed to stdout.
 
 This checks can be applied only to box tests, as assertions which are used in unit tests may impact spl2 pipeline
-performance. 
+performance.
+
+### Reduction rate statistics (box tests)
+
+Optional telemetry for **how much** each pipeline shrinks event payloads (length of input `_raw` vs matched output `_raw`), broken down by **pipeline** and **sourcetype**.
+
+**Enable**
+
+- Pass `--reduction_stats` to pytest (or add it in `pytest.ini` / your IDE run configuration).
+
+**Output**
+
+- During the session, each box test appends one or more floating-point rates into an in-memory structure: `pipeline_name → sourcetype → [rates…]`.
+- After all tests finish, if `--reduction_stats` was set, the framework logs a short **pandas** summary per pipeline and writes **`--reduction_stats_file`** (default `reduction_stats.json`).
+- With **`--reduction_stats_full`**, the JSON file also embeds per-pipeline **`describe()`** statistics for each sourcetype column (handy for dashboards or quick distribution checks).
+
+**How rates are computed (high level)**
+
+- Input events are parsed from the box test `source` string; output rows come from the job’s destination results.
+- Input/output rows are **paired** using similarity on `_raw` (with a threshold and a same-length positional fallback). Only **matched** pairs contribute rates.
+- For each pair, lengths use a **canonical JSON length** when `_raw` parses as JSON or a Python literal; otherwise the raw string length is used, so identical logical content can yield a **0** reduction rate even if string formatting differs.
+
+**Scope**
+
+- Applies to **box tests** only. Single-SPL2 runs and unit tests do not populate reduction stats.
 
 ### Check splunk results
 
@@ -230,13 +258,13 @@ When set (`--check_splunk_results=<option>`), box tests (after running the SPL2
 | **CIM** | For each output event, ingest via HEC, load back from Splunk, then check CIM fields from `expected_cim_fields.cim_fields` with `validate_compatibility(actual, expected)`. Skips if the box test has no `cim_fields`. |
 | **TA** | For each output event, ingest via HEC, load back from Splunk, then assert equality with the expected row for that index. |
 
-For **TA** only, you can name top-level fields to drop on **both** the queried event and each `expected_destination_result` row by adding **`ignore_fields_in_splunk_check`**: an object mapping **field name → short reason**. Use the same key under `test` in each entry in `*.test.json` (same file as `expected_destination_result`):
+You can name top-level fields to drop on **both** the pipeline/actual row and each matching `expected_destination_result` row before they are compared by adding **`ignore_fields_in_destination_compare`**: an object mapping **field name → short reason**. Use the same key under `test` in each entry in `*.test.json` (same file as `expected_destination_result`):
 
 ```json
 "test": {
   "source": "...",
   "expected_destination_result": [ ... ],
-  "ignore_fields_in_splunk_check": {
+  "ignore_fields_in_destination_compare": {
     "_raw": "Raw event differs after Splunk indexing",
     "_time": "Timestamp may vary due to ingestion latency"
   }
@@ -244,8 +272,7 @@ For **TA** only, you can name top-level fields to drop on **both** the queried e
 ```
 
 
-If the key is omitted, no fields are ignored. Used only when `--check_splunk_results=TA` (ignored for other modes and for the default 
-in-memory box test assert). If the file or key is missing, no fields are ignored.
+If the key is omitted, no fields are ignored. It applies to the **default in-memory** box test assert (no `--check_splunk_results`) and to **`--check_splunk_results=TA`**. It does **not** apply to **`--check_splunk_results=CIM`**. If the file or key is missing, no fields are ignored.
 
 #### Configuration:
 

{spl2_testing_framework-1.7.3 → spl2_testing_framework-1.8.0}/pyproject.toml

@@ -15,7 +15,7 @@
 
 [tool.poetry]
 name = "spl2-testing-framework"
-version = "1.7.3"
+version = "1.8.0"
 description = "SPL2 Testing Framework"
 authors = ["mgazda-splunk <mgazda@splunk.com>"]
 readme = "README.md"

{spl2_testing_framework-1.7.3 → spl2_testing_framework-1.8.0}/spl2_testing_framework/conftest.py

@@ -14,19 +14,21 @@
 
 
 import json
+import logging
+import math
 import os
 from collections import defaultdict
 
-import pandas as pd
 import pytest
 
-from .tools.results import Results
-from .tools.splunk_integration.splunk_handler import SplunkHandler
-
 pytest.register_assert_rewrite("spl2_testing_framework")
 
 from requests.auth import AuthBase, HTTPBasicAuth
 
+from .logger_manager import setup_logging
+from .tools.results import Results
+from .tools.splunk_integration.splunk_handler import SplunkHandler
+
 from spl2_testing_framework.tools.search_clients.cli_search_client import (
     CLISearchClient,
 )
@@ -58,10 +60,7 @@ TEST_DIR = "test_dir"
 CODE_DIR = "code_dir"
 LIMIT_TESTS = "limit_tests"
 CLI_BENCH = "cli_bench"
-
-import logging
-
-from .logger_manager import setup_logging
+REDUCTION_STATS = "reduction_stats"
 
 setup_logging()
 
@@ -89,6 +88,24 @@ def pytest_addoption(parser):
     parser.addoption(f"--{IGNORE_ADDITIONAL_FIELDS_IN_ACTUAL}", action="store_true")
     parser.addoption(f"--{LIMIT_TESTS}", action="store", type=int, default=None)
     parser.addoption(f"--{CLI_BENCH}", action="store", type=int, default=None)
+    parser.addoption(
+        f"--{REDUCTION_STATS}",
+        action="store_true",
+        default=False,
+        help="Collect reduction rates per pipeline/sourcetype and save to file (see --reduction_stats_file, --reduction_stats_full). By default disabled.",
+    )
+    parser.addoption(
+        "--reduction_stats_file",
+        action="store",
+        default="reduction_stats.json",
+        help="Path to save reduction rates (JSON). Used only when --reduction_stats is set.",
+    )
+    parser.addoption(
+        "--reduction_stats_full",
+        action="store_true",
+        default=False,
+        help="Include full dataframe stats (count, mean, std, min, 25%%, 50%%, 75%%, max) in the saved file. Used only when --reduction_stats is set.",
+    )
 
 
 def pytest_generate_tests(metafunc):
@@ -169,7 +186,7 @@ def spl2_test_runner(
         url = f"https://{config['tenant']}.api.{config['cloud_instance']}/{config['tenant']}"
         search_client = CloudSearchClient(url=url, auth=bearer_token_authentication)
     else:
-        raise Exception(
+        raise ValueError(
             "Unknown test type: {}. Supported tests types: 'splunk', 'cloud', 'cli'".format(
                 test_type
             )
@@ -188,6 +205,9 @@ def run_configurations(request):
         ),
         CLI_BENCH: request.config.getoption(f"--{CLI_BENCH}"),
         CHECK_SPLUNK_RESULTS: request.config.getoption(f"--{CHECK_SPLUNK_RESULTS}"),
+        REDUCTION_STATS: request.config.getoption(
+            f"--{REDUCTION_STATS}", default=False
+        ),
     }
 
 
@@ -328,17 +348,71 @@ def splunk_handler(config, splunk_token_authentication):
 
 
 @pytest.fixture(scope="session")
-def stats():
+def stats(request):
     s = defaultdict(lambda: defaultdict(list))
     yield s
 
+    collect_reduction = request.config.getoption(f"--{REDUCTION_STATS}", default=False)
+    if not collect_reduction:
+        return
+
+    _LOGGER.info("Reduction stats: reporting (--reduction_stats was set)")
+    if not s:
+        _LOGGER.info(
+            "Reduction stats: no data (no box tests ran or no rates collected)"
+        )
+        return
+
+    import pandas as pd  # lazy import — only needed when --reduction_stats is set
+
+    def _pad_to_equal_length(rates_dict):
+        """Pad lists to same length with NaN so pd.DataFrame accepts different-length columns."""
+        if not rates_dict:
+            return rates_dict
+        max_len = max(len(lst) for lst in rates_dict.values())
+        return {
+            k: list(lst) + [float("nan")] * (max_len - len(lst))
+            for k, lst in rates_dict.items()
+        }
+
     for template, v in s.items():
-        x = pd.DataFrame(v)
+        v_padded = _pad_to_equal_length(dict(v))
+        x = pd.DataFrame(v_padded)
         _LOGGER.info(" *** *** ***")
         _LOGGER.info("Collected statistics: ")
         _LOGGER.info("Template name: %s ", template)
         _LOGGER.info(x.describe())
 
+    out_path = request.config.getoption(
+        "--reduction_stats_file", default="reduction_stats.json"
+    )
+    include_full = request.config.getoption("--reduction_stats_full", default=False)
+    if out_path:
+        data = {}
+        for pipeline, sourcetype_rates in s.items():
+            rates_dict = dict(sourcetype_rates)
+            if include_full:
+                rates_padded = _pad_to_equal_length(rates_dict)
+                df = pd.DataFrame(rates_padded)
+                describe = df.describe().to_dict()
+                # NaN is not JSON-serializable; convert to None
+                describe_serializable = {
+                    k: {
+                        kk: (None if isinstance(vv, float) and math.isnan(vv) else vv)
+                        for kk, vv in v.items()
+                    }
+                    for k, v in describe.items()
+                }
+                data[pipeline] = {
+                    "rates": rates_dict,
+                    "describe": describe_serializable,
+                }
+            else:
+                data[pipeline] = rates_dict
+        with open(out_path, "w", encoding="utf-8") as f:
+            json.dump(data, f, indent=2)
+        _LOGGER.info("Reduction rates saved to %s", out_path)
+
 
 def _set_results_options(metafunc):
     ignore_empty_strings = metafunc.config.getoption(IGNORE_EMPTY_STRINGS)

spl2_testing_framework-1.8.0/spl2_testing_framework/spl2_test_config.json

@@ -0,0 +1,18 @@
+{
+  "host": "",
+  "cloud_instance": "",
+  "tenant": "",
+  "bearer_token": "",
+  "port": "",
+  "user": "",
+  "password": "",
+  "SPLUNK_INSTANCE": {
+    "ip": "localhost",
+    "port": 8088,
+    "api_port": 8089,
+    "username": "admin",
+    "password": "Chang3d!",
+    "index": "testx",
+    "hec_token": "4a8a737d-5452-426c-a6f7-106dca4e813f"
+  }
+}

{spl2_testing_framework-1.7.3 → spl2_testing_framework-1.8.0}/spl2_testing_framework/tools/performance.py

@@ -249,7 +249,14 @@ class PerformanceCheck:
 
     @staticmethod
     def _preserve_timestamps_through_fields(code_module: str) -> str:
-        """Replace '| fields' keep-mode with a no-op so _timestamp_perf* fields are not stripped."""
+        """Replace '| fields' keep-mode with a no-op so _timestamp_perf* fields are not stripped.
+
+        During performance instrumentation the pipeline runs with a slightly different shape:
+        keep-mode ``| fields`` commands are removed to prevent them from stripping the injected
+        ``_timestamp_perf*`` variables.  This means perf numbers reflect a pipeline with all
+        fields passing through, which is slightly heavier than production but avoids silently
+        dropping the instrumentation fields and producing zeroed measurements.
+        """
         return re.sub(
             r"\|\s*fields\s+(?!-)(.+)",
             r"| eval _perf_fields_preserved=1",
@@ -297,11 +304,13 @@ class PerformanceCheck:
             flags=re.M,
         )
 
-        func_timestamps = re.findall(rf"_time_{f['f_name']}_placeholder", new_function)
+        func_timestamps = re.findall(
+            rf"_time_{re.escape(f['f_name'])}_placeholder", new_function
+        )
 
         for n, timestamp in enumerate(func_timestamps):
             new_function = re.sub(
-                timestamp,
+                re.escape(timestamp),
                 rf"_timestamp_perf{f['f_name']}_{n:03d}",
                 new_function,
                 count=1,
@@ -318,7 +327,7 @@ class PerformanceCheck:
         )
 
         func_timestamps = re.findall(
-            rf"\$_time_{f['f_name']}_placeholder", new_function
+            rf"\$_time_{re.escape(f['f_name'])}_placeholder", new_function
         )
 
         for n, timestamp in enumerate(func_timestamps):

{spl2_testing_framework-1.7.3 → spl2_testing_framework-1.8.0}/spl2_testing_framework/tools/results.py

@@ -175,14 +175,12 @@ class Results(UserDict):
                                         actual_data[key][i], expected_data[key][i]
                                     )
 
-        return actual_data
-
     def remove_fields_not_in_expected(self, expected):
         """Remove fields from this Results object that are not present in the expected Results object.
         This is the public method to be called when comparing results.
         """
         if self._ignore_additional_fields_in_actual and isinstance(expected, Results):
-            self.data = self._remove_extra_fields(self.data, expected.data)
+            self._remove_extra_fields(self.data, expected.data)
         return self
 
     def __lt__(self, other):

{spl2_testing_framework-1.7.3 → spl2_testing_framework-1.8.0}/spl2_testing_framework/tools/spl2test_runner.py

@@ -33,9 +33,7 @@ from spl2_testing_framework.tools.test_types import UnitTest, BoxTest, SingleSPL
 from spl2_testing_framework.tools.utils import (
     _make_functions_visible_for_testing,
     validate_compatibility,
-    calculate_reduction_rate,
-    parse_raw_from_input,
-    parse_raw_from_reduced_events,
+    calculate_reduction_rate_and_update_stats,
 )
 
 _LOGGER = logging.getLogger(__name__)
@@ -44,13 +42,21 @@ _LOGGER = logging.getLogger(__name__)
 class SPL2TestRunner:
     """Main class responsible for running tests"""
 
-    def __init__(self, search_client, run_configurations, splunk_handler, stats):
+    def __init__(
+        self, search_client, run_configurations, splunk_handler=None, stats=None
+    ):
         self._search_client = search_client
         self.create_comparison_sheet = run_configurations.get("create_comparison_sheet")
         self.check_splunk_results = run_configurations.get("check_splunk_results")
+        self.collect_reduction_stats = run_configurations.get("reduction_stats", False)
         self.comp_obj = ComparisonSheet()
         self.reduction_stats = stats
         if self.check_splunk_results:
+            if splunk_handler is None:
+                raise ValueError(
+                    f"check_splunk_results={self.check_splunk_results!r} requires a "
+                    "splunk_handler to be provided, but got None."
+                )
             splunk_handler.setup()
             self.splunk_handler = splunk_handler
 
@@ -181,26 +187,48 @@ class SPL2TestRunner:
 
         perf.check_performance(output_result)
 
-        reduction_rate = calculate_reduction_rate(
-            parse_raw_from_input(box_test.input),
-            parse_raw_from_reduced_events(output_result),
-        )
-        self.reduction_stats[box_test.name]["reduction_rate"].extend(reduction_rate)
-
-        _LOGGER.info("Reduction rate for this event: %s", reduction_rate)
+        if self.collect_reduction_stats:
+            reduction_rate = calculate_reduction_rate_and_update_stats(
+                box_test.code_module_path.stem,
+                box_test.input,
+                output_result,
+                self.reduction_stats,
+            )
+            _LOGGER.info("Reduction rate for this event: %s", reduction_rate)
 
         if self.check_splunk_results:
             self._check_splunk_results(box_test, output_result)
         else:
             if box_test.output:
-                # Remove fields from actual results that are not in expected results if configured
+                ignore_keys = frozenset(
+                    box_test.ignore_fields_in_destination_compare.keys()
+                )
                 for i, actual_result in enumerate(output_result):
                     if i < len(box_test.output):
                         actual_result.remove_fields_not_in_expected(box_test.output[i])
 
                 if self.create_comparison_sheet:
                     self.comp_obj.create_comparison_sheet(output_result, box_test)
-                assert sorted(output_result) == sorted(box_test.output)
+                if ignore_keys:
+                    stripped_actual = [
+                        Results(
+                            self._strip_ignored_fields_for_destination_compare(
+                                row.data, ignore_keys
+                            )
+                        )
+                        for row in output_result
+                    ]
+                    stripped_expected = [
+                        Results(
+                            self._strip_ignored_fields_for_destination_compare(
+                                row.data, ignore_keys
+                            )
+                        )
+                        for row in box_test.output
+                    ]
+                    assert sorted(stripped_actual) == sorted(stripped_expected)
+                else:
+                    assert sorted(output_result) == sorted(box_test.output)
                 _LOGGER.info(f"Output check passed: {box_test.name}")
                 _LOGGER.debug("Received: \n%s", str(output_result))
                 _LOGGER.debug("Expected: \n%s", str(box_test.output))
@@ -243,7 +271,7 @@ class SPL2TestRunner:
             return copy.deepcopy(d)
 
     @staticmethod
-    def _strip_ignored_fields_for_splunk_compare(
+    def _strip_ignored_fields_for_destination_compare(
         row: dict, ignore_keys: frozenset
     ) -> dict:
         if not ignore_keys:
@@ -260,7 +288,12 @@ class SPL2TestRunner:
             ) from e
 
     def _check_splunk_results(self, box_test, destination_rows):
-        """Use the same destination rows as ``run_box_test`` (from ``get_job_results``), not ``job.result``."""
+        """Validate HEC round-trip rows against the box test (CIM or TA).
+
+        ``destination_rows`` must be the same list returned from
+        ``get_job_results(job)["destination"]`` as used for reduction stats and
+        non-Splunk output assertions — not ``job.result``.
+        """
         if self.check_splunk_results == "CIM":
             if not box_test.cim_fields:
                 pytest.skip("CIM output not available for this test")
@@ -279,8 +312,13 @@ class SPL2TestRunner:
             if not box_test.output:
                 pytest.skip("TA expected output not available for this test")
 
-            ignore_spec = box_test.ignore_fields_in_splunk_check
+            ignore_spec = box_test.ignore_fields_in_destination_compare
             ignore_keys = frozenset(ignore_spec.keys())
+            assert len(destination_rows) == len(box_test.output), (
+                f"TA Splunk check for {box_test.name!r}: "
+                f"got {len(destination_rows)} destination row(s) but "
+                f"{len(box_test.output)} expected output(s); zip would hide a mismatch."
+            )
             for event, expected in zip(destination_rows, box_test.output):
                 processed_event = self._splunk_process_event(
                     box_test, self._row_as_dict(event)
@@ -290,15 +328,19 @@ class SPL2TestRunner:
                 # dropping keys not present on the expected row before asserting.
                 actual_dict = dict(processed_event)
                 Results._remove_extra_fields(actual_dict, expected_data)
-                actual_cmp = self._strip_ignored_fields_for_splunk_compare(
+                actual_cmp = self._strip_ignored_fields_for_destination_compare(
                     actual_dict, ignore_keys
                 )
-                expected_cmp = self._strip_ignored_fields_for_splunk_compare(
+                expected_cmp = self._strip_ignored_fields_for_destination_compare(
                     expected_data, ignore_keys
                 )
                 assert actual_cmp == expected_cmp, (
-                    f"TA Splunk compare mismatch for {box_test.name}; "
-                    f"ignored_fields={ignore_spec!r}"
+                    f"TA destination compare mismatch for {box_test.name}; "
+                    f"ignore_fields_in_destination_compare={ignore_spec!r}"
                 )
         else:
-            return
+            _LOGGER.warning(
+                "Unknown check_splunk_results value %r — expected 'CIM' or 'TA'. "
+                "No Splunk validation performed.",
+                self.check_splunk_results,
+            )

{spl2_testing_framework-1.7.3 → spl2_testing_framework-1.8.0}/spl2_testing_framework/tools/splunk_integration/splunk_handler.py

@@ -13,16 +13,18 @@
 # limitations under the License.
 
 import logging
+import time
 import uuid
 from typing import Optional
 
 import requests
+import urllib3
 import splunklib.client as client
 import splunklib.results as results
-
-import time
-
 from requests.auth import HTTPBasicAuth
+import json
+
+urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
 
 _LOGGER = logging.getLogger(__name__)
 
@@ -82,10 +84,12 @@ class SplunkHandler:
     def _raw_as_event_string(raw):
         """Normalize pipeline `_raw` for HEC JSON `event` (must be a string).
 
-        ``str`` is passed through ``str()``; ``bytes`` are decoded as UTF-8 with
-        replacement on invalid sequences (avoids :class:`UnicodeDecodeError` for
-        odd binary payloads; prefer valid UTF-8 log text in normal use).
+        ``dict`` is serialized as valid JSON (double-quoted) so that Splunk's
+        field extractions (e.g. GCP TA KV_MODE=json) can parse it correctly.
+        ``bytes`` are decoded as UTF-8. Everything else uses ``str()``.
         """
+        if isinstance(raw, dict):
+            return json.dumps(raw)  # {"data": {...}} - valid JSON
         if isinstance(raw, bytes):
             return raw.decode("utf-8", errors="replace")
         return str(raw)
@@ -165,4 +169,11 @@ class SplunkHandler:
             ),
             verify=False,
         )
+        if response.status_code == 409:
+            _LOGGER.info(
+                "Index '%s' already exists — skipping creation.",
+                self.splunk_config["index"],
+            )
+            return
+        response.raise_for_status()
         return response.json()

{spl2_testing_framework-1.7.3 → spl2_testing_framework-1.8.0}/spl2_testing_framework/tools/test_discovery.py

@@ -179,11 +179,11 @@ class TestDiscovery(abc.ABC):
 class BoxTestDiscovery(TestDiscovery):
     """Discovers tests for box tests, looking into module.test.json files"""
 
-    _IGNORE_FIELDS_IN_SPLUNK_CHECK_KEY = "ignore_fields_in_splunk_check"
+    _IGNORE_FIELDS_IN_DESTINATION_COMPARE_KEY = "ignore_fields_in_destination_compare"
 
     @classmethod
-    def _parse_ignore_fields_in_splunk_check(cls, test_block: dict) -> dict:
-        raw = test_block.get(cls._IGNORE_FIELDS_IN_SPLUNK_CHECK_KEY)
+    def _parse_ignore_fields_in_destination_compare(cls, test_block: dict) -> dict:
+        raw = test_block.get(cls._IGNORE_FIELDS_IN_DESTINATION_COMPARE_KEY)
         if raw is None:
             return {}
         if isinstance(raw, list):
@@ -194,14 +194,14 @@ class BoxTestDiscovery(TestDiscovery):
                 else:
                     _LOGGER.warning(
                         "%s list entries must be strings; skipping %r",
-                        cls._IGNORE_FIELDS_IN_SPLUNK_CHECK_KEY,
+                        cls._IGNORE_FIELDS_IN_DESTINATION_COMPARE_KEY,
                         k,
                     )
             return out
         if not isinstance(raw, dict):
             _LOGGER.warning(
                 "%s must be an object (field -> reason) or a list of field names; ignoring %r",
-                cls._IGNORE_FIELDS_IN_SPLUNK_CHECK_KEY,
+                cls._IGNORE_FIELDS_IN_DESTINATION_COMPARE_KEY,
                 raw,
             )
             return {}
@@ -210,7 +210,7 @@ class BoxTestDiscovery(TestDiscovery):
             if not isinstance(key, str):
                 _LOGGER.warning(
                     "%s keys must be strings; skipping %r",
-                    cls._IGNORE_FIELDS_IN_SPLUNK_CHECK_KEY,
+                    cls._IGNORE_FIELDS_IN_DESTINATION_COMPARE_KEY,
                     key,
                 )
                 continue
@@ -252,8 +252,8 @@ class BoxTestDiscovery(TestDiscovery):
                     Metrics(x)
                     for x in test_block.get("expected_metrics_destination_result", [])
                 ]
-                ignore_fields_in_splunk_check = (
-                    self._parse_ignore_fields_in_splunk_check(test_block)
+                ignore_fields_in_destination_compare = (
+                    self._parse_ignore_fields_in_destination_compare(test_block)
                 )
                 box_test = BoxTest(
                     name=name,
@@ -263,7 +263,7 @@ class BoxTestDiscovery(TestDiscovery):
                     module_path=module_path,
                     code_module_path=code_module_path,
                     cim_fields=cim_fields,
-                    ignore_fields_in_splunk_check=ignore_fields_in_splunk_check,
+                    ignore_fields_in_destination_compare=ignore_fields_in_destination_compare,
                 )
                 box_test.meta["performance"] = self._performance_check
                 self.tests.add(box_test)

{spl2_testing_framework-1.7.3 → spl2_testing_framework-1.8.0}/spl2_testing_framework/tools/test_types.py

@@ -46,8 +46,8 @@ class BoxTest(Test):
     code_module_path: Path
     metrics: list = field(default_factory=list)
     cim_fields: dict = field(default_factory=dict)
-    # TA Splunk check: field name -> why it is ignored (comparison-only; keys stripped from both sides)
-    ignore_fields_in_splunk_check: dict = field(default_factory=dict)
+    # Field name -> reason; stripped on both sides before destination asserts (not used for CIM).
+    ignore_fields_in_destination_compare: dict = field(default_factory=dict)
 
 
 @dataclass

spl2_testing_framework-1.8.0/spl2_testing_framework/tools/utils.py

@@ -0,0 +1,331 @@
+# Copyright 2026 Splunk Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+import ast
+import json
+import logging
+import pathlib
+import re
+from difflib import SequenceMatcher
+from functools import lru_cache
+
+_LOGGER = logging.getLogger(__name__)
+
+# Minimum similarity (0-1) for an output event to be paired with an input event.
+# SequenceMatcher.ratio() is used; below this threshold the output is left unpaired.
+MATCH_SIMILARITY_THRESHOLD = 0.3
+
+
+def get_framework_root() -> pathlib.Path:
+    framework_root = pathlib.Path(__file__).parents[1]
+    return framework_root
+
+
+def _make_functions_visible_for_testing(code_module_path):
+    code_module = code_module_path.read_text(encoding="utf-8")
+    functions = re.findall(r"^function\s+([^(]+)\(", code_module, flags=re.M)
+    for function in functions:
+        if not re.search(r"export\s" + function, code_module):
+            code_module = re.sub(
+                r"^function\s+" + function,
+                f"export function {function}",
+                code_module,
+                flags=re.M,
+            )
+    return code_module
+
+
+@lru_cache(None)
+def read_assertions_module():
+    root_folder = get_framework_root()
+    assertions_file = root_folder / "spl2_utils" / "assertions.spl2"
+    return assertions_file.read_text(encoding="utf-8")
+
+
+@lru_cache(None)
+def read_commands_modules():
+    root_folder = get_framework_root()
+    commands_module = root_folder / "spl2_utils" / "logs_to_metrics.spl2"
+    return commands_module.read_text(encoding="utf-8")
+
+
+def _mapping_from_event_row(ev):
+    """Return a dict-like mapping for a destination row (plain dict or ``Results``-style ``.data``)."""
+    if isinstance(ev, dict):
+        return ev
+    inner = getattr(ev, "data", None)
+    if isinstance(inner, dict):
+        return inner
+    return ev
+
+
+def _raw_from_event(ev):
+    m = _mapping_from_event_row(ev)
+    return (
+        str(m.get("_raw", "")) if isinstance(m, dict) else str(getattr(m, "_raw", ""))
+    )
+
+
+def _similarity(a: str, b: str) -> float:
+    """Return similarity ratio in [0, 1] using SequenceMatcher."""
+    if not a and not b:
+        return 1.0
+    if not a or not b:
+        return 0.0
+    return SequenceMatcher(None, a, b).ratio()
+
+
+def match_input_output_events(input_events, output_events, min_similarity=None):
+    """Pair input events with output events by content similarity using difflib.SequenceMatcher.
+
+    For each output event, the unused input event with the highest similarity ratio is chosen.
+    Pairs are only formed when the ratio is at least min_similarity (default MATCH_SIMILARITY_THRESHOLD).
+    Each input is used at most once. Handles reordering and dropped events.
+
+    Fallback: when similarity matching yields no pairs, input/output lengths are equal, and
+    there are at least two events on each side, pair by position (input[i], output[i]).
+    This covers pipelines (e.g. JSON/CrowdStrike) where _raw is heavily transformed so
+    similarity is low but order is preserved. Single-event cases are not paired here so
+    unrelated one-to-one runs do not get a false match.
+    """
+    if not output_events or not input_events:
+        return []
+    threshold = (
+        min_similarity if min_similarity is not None else MATCH_SIMILARITY_THRESHOLD
+    )
+    used_input_indices = set()
+    pairs = []
+
+    for out_ev in output_events:
+        out_r = _raw_from_event(out_ev)
+        best_in_idx = None
+        best_ratio = -1.0
+
+        for i, in_ev in enumerate(input_events):
+            if i in used_input_indices:
+                continue
+            # Input first: SequenceMatcher.ratio() is not symmetric when autojunk is on;
+            # input-then-output matches historical behavior and keeps unrelated pairs below threshold.
+            ratio = _similarity(_raw_from_event(in_ev), out_r)
+            if ratio > best_ratio and ratio >= threshold:
+                best_ratio = ratio
+                best_in_idx = i
+
+        if best_in_idx is not None:
+            used_input_indices.add(best_in_idx)
+            pairs.append((input_events[best_in_idx], out_ev))
+        else:
+            _LOGGER.debug(
+                "No input matched output _raw (len=%s, threshold=%.2f)",
+                len(out_r),
+                threshold,
+            )
+
+    if not pairs and len(input_events) == len(output_events) and len(input_events) >= 2:
+        pairs = list(zip(input_events, output_events))
+        _LOGGER.debug(
+            "Similarity matching produced no pairs; using position-based pairing "
+            "(equal lengths %s, >=2)",
+            len(input_events),
+        )
+
+    return pairs
+
+
+def _content_length_for_rate(s):
+    """Length used for reduction rate: canonical JSON length if parseable, else raw string length.
+
+    So identical JSON/dict content (e.g. original as JSON string, reduced as dict repr) gives
+    the same length and rate 0.
+    """
+    raw = str(s)
+    for parse in (json.loads, ast.literal_eval):
+        try:
+            obj = parse(raw)
+            return len(json.dumps(obj, sort_keys=True, separators=(",", ":")))
+        except (json.JSONDecodeError, ValueError, SyntaxError, TypeError):
+            continue
+    return len(raw)
+
+
+def calculate_reduction_rate(original_event, reduced_event):
+    """Calculate how much each event was reduced. Both lists must be the same length (matched pairs).
+
+    When both values are parseable as JSON or Python dict, lengths are compared in canonical form
+    so identical content (e.g. JSON string vs dict repr) yields rate 0.
+    """
+    if len(original_event) != len(reduced_event):
+        raise ValueError(
+            "original_event and reduced_event must have the same length (use match_input_output_events first)"
+        )
+    reduction_rates = []
+    for original, reduced in zip(original_event, reduced_event):
+        original_event_len = _content_length_for_rate(original)
+        reduced_event_len = _content_length_for_rate(reduced)
+        if original_event_len == 0:
+            reduction_rates.append(0.0)
+        else:
+            reduction_rates.append(
+                (original_event_len - reduced_event_len) / original_event_len
+            )
+    return reduction_rates
+
+
+def _sourcetype_for_pair(input_ev, output_ev, default="UNKNOWN"):
+    """Sourcetype for a matched pair: from input if set, else from output, else default."""
+    input_ev = _mapping_from_event_row(input_ev)
+    output_ev = _mapping_from_event_row(output_ev)
+    return input_ev.get("sourcetype") or output_ev.get("sourcetype") or default
+
+
+def calculate_reduction_rate_and_update_stats(
+    pipeline, input_str, output_events, stats
+):
+    """Compute reduction rates for input vs output events and update stats by pipeline and sourcetype.
+
+    Pairs input and output events via match_input_output_events (SequenceMatcher similarity).
+    Appends each rate to stats[pipeline][sourcetype]. Sourcetype is taken from the input event
+    if present, otherwise from the output event, otherwise \"UNKNOWN\". Returns the list of
+    reduction rates for matched pairs only.
+    """
+    input_events = parse_events_from_input(input_str)
+    pairs = match_input_output_events(input_events, output_events)
+    if not pairs:
+        _LOGGER.debug(
+            "No input-output pairs for reduction rate (pipeline=%s)", pipeline
+        )
+        return []
+
+    original_raw = [_raw_from_event(in_ev) for in_ev, _ in pairs]
+    reduced_raw = [_raw_from_event(out_ev) for _, out_ev in pairs]
+    reduction_rates = calculate_reduction_rate(original_raw, reduced_raw)
+    sourcetypes = [_sourcetype_for_pair(in_ev, out_ev) for in_ev, out_ev in pairs]
+    for sourcetype, rate in zip(sourcetypes, reduction_rates):
+        stats[pipeline][sourcetype].append(rate)
+    return reduction_rates
+
+
+def validate_compatibility(actual, expected):
+    """Check that every field in *expected* exists in *actual* with the same value.
+
+    Order matches the usual test convention ``assert actual == expected`` (observed
+    value first, reference second). For Splunk CIM checks, the runner passes both
+    sides through :class:`~spl2_testing_framework.tools.results.Results` first so
+    numeric and other types match pipeline/CLI normalization.
+    Extra fields in *actual* that are not in *expected* are silently ignored,
+    so callers only need to specify the subset of fields they care about.
+
+    Raises ``AssertionError`` with a human-readable summary listing every
+    mismatched and every missing field in a single message.
+    """
+    wrong_value = {}
+    missing = {}
+
+    for field, expected_val in expected.items():
+        if field not in actual:
+            missing[field] = expected_val
+            _LOGGER.error("FIELD %s is missing from actual event", field)
+            continue
+
+        actual_val = actual[field]
+        if actual_val == expected_val:
+            _LOGGER.info("FIELD %s OK", field)
+        else:
+            wrong_value[field] = {"actual": actual_val, "expected": expected_val}
+            _LOGGER.error(
+                "FIELD %s mismatch: actual=%r, expected=%r",
+                field,
+                actual_val,
+                expected_val,
+            )
+
+    message = _format_compatibility_errors(wrong_value, missing)
+    assert message is None, message
+
+
+def _format_compatibility_errors(wrong_value, missing):
+    """Return a formatted error string, or ``None`` when there are no errors."""
+    sections = []
+    if wrong_value:
+        lines = [
+            f"  {field}: actual={d['actual']!r}, expected={d['expected']!r}"
+            for field, d in wrong_value.items()
+        ]
+        sections.append("Wrong values:\n" + "\n".join(lines))
+    if missing:
+        lines = [f"  {field}: expected={val!r}" for field, val in missing.items()]
+        sections.append("Missing fields:\n" + "\n".join(lines))
+    if not sections:
+        return None
+    return "\n" + "\n".join(sections)
+
+
+_QUOTED_STRING_OR_BARE_KEY = re.compile(
+    r'"(?:[^"\\]|\\.)*"' r"|" r"([{,])\s*([A-Za-z_]+)\s*(:)"
+)
+
+
+def _quote_unquoted_keys(input_str):
+    """Wrap bare (unquoted) object keys in double quotes to produce valid JSON,
+    without corrupting content inside JSON string values.
+
+    Uses regex alternation: the first branch consumes quoted strings (so their
+    content is never touched), the second branch captures bare keys to quote.
+    """
+
+    def _replacer(m):
+        if m.group(1) is None:
+            return m.group(0)
+        return f'{m.group(1)}"{m.group(2)}":'
+
+    return _QUOTED_STRING_OR_BARE_KEY.sub(_replacer, input_str)
+
+
+def parse_events_from_input(input_str):
+    """Parse SPL2-style or JSON array of events and return list of event dicts.
+    Each dict has at least '_raw'; may also have 'source', 'sourcetype', 'host'.
+    Handles unquoted keys (e.g. {_raw: "x", sourcetype: "st"}) by converting to JSON.
+    """
+    normalized = _quote_unquoted_keys(input_str)
+    try:
+        parsed = json.loads(normalized)
+    except json.JSONDecodeError:
+        parsed = ast.literal_eval(normalized)
+    return list(parsed)
+
+
+def parse_raw_from_input(input_str):
+    """Parse SPL2-style or JSON array of events and return list of _raw field values.
+    Handles unquoted keys (e.g. {_raw: "x"}) by converting them to JSON-quoted keys.
+    """
+    return [item["_raw"] for item in parse_events_from_input(input_str)]
+
+
+def get_sourcetypes_from_input(input_str, default="UNKNOWN"):
+    """Return sourcetype for each event in the box test input.
+    Use this to associate each input event with a sourcetype for reduction-rate
+    calculation per pipeline and sourcetype.
+    """
+    events = parse_events_from_input(input_str)
+    return [e.get("sourcetype", default) for e in events]
+
+
+def get_sourcetypes_from_events(events, default="UNKNOWN"):
+    """Return sourcetype for each event in a list of event dicts (e.g. pipeline output).
+    Use with run_box_test output_result to get sourcetype per reduced event.
+    """
+    return [e.get("sourcetype", default) for e in events]
+
+
+def parse_raw_from_reduced_events(events):
+    return [str(x["_raw"]) for x in events]

spl2_testing_framework-1.7.3/spl2_testing_framework/spl2_test_config.json

@@ -1,18 +0,0 @@
-{
-  "host": "",
-  "cloud_instance": "",
-  "tenant": "",
-  "bearer_token": "",
-  "port": "",
-  "user": "",
-  "password": "",
-  "SPLUNK_INSTANCE": {
-    "ip": "",
-    "port": "",
-    "api_port": "",
-    "username": "",
-    "password": "",
-    "index": "",
-    "hec_token": ""
-  }
-}

spl2_testing_framework-1.7.3/spl2_testing_framework/tools/utils.py

@@ -1,159 +0,0 @@
-# Copyright 2026 Splunk Inc.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-import ast
-import json
-import logging
-import pathlib
-import re
-from functools import lru_cache
-
-_LOGGER = logging.getLogger(__name__)
-
-
-def get_framework_root() -> pathlib.Path:
-    framework_root = pathlib.Path(__file__).parents[1]
-    return framework_root
-
-
-def _make_functions_visible_for_testing(code_module_path):
-    code_module = code_module_path.read_text(encoding="utf-8")
-    functions = re.findall(r"^function\s+([^(]+)\(", code_module, flags=re.M)
-    for function in functions:
-        if not re.search(r"export\s" + function, code_module):
-            code_module = re.sub(
-                r"^function\s+" + function,
-                f"export function {function}",
-                code_module,
-                flags=re.M,
-            )
-    return code_module
-
-
-@lru_cache(None)
-def read_assertions_module():
-    root_folder = get_framework_root()
-    assertions_file = root_folder / "spl2_utils" / "assertions.spl2"
-    return assertions_file.read_text(encoding="utf-8")
-
-
-@lru_cache(None)
-def read_commands_modules():
-    root_folder = get_framework_root()
-    commands_module = root_folder / "spl2_utils" / "logs_to_metrics.spl2"
-    return commands_module.read_text(encoding="utf-8")
-
-
-def calculate_reduction_rate(original_event, reduced_event):
-    """Calculate how much the event was reduced."""
-    reduction_rates = []
-    for original, reduced in zip(original_event, reduced_event):
-        original_event_len = len(str(original))
-        reduced_event_len = len(str(reduced))
-
-        reduction_rates.append(
-            (original_event_len - reduced_event_len) / original_event_len
-        )
-    return reduction_rates
-
-
-def validate_compatibility(actual, expected):
-    """Check that every field in *expected* exists in *actual* with the same value.
-
-    Order matches the usual test convention ``assert actual == expected`` (observed
-    value first, reference second). For Splunk CIM checks, the runner passes both
-    sides through :class:`~spl2_testing_framework.tools.results.Results` first so
-    numeric and other types match pipeline/CLI normalization.
-    Extra fields in *actual* that are not in *expected* are silently ignored,
-    so callers only need to specify the subset of fields they care about.
-
-    Raises ``AssertionError`` with a human-readable summary listing every
-    mismatched and every missing field in a single message.
-    """
-    wrong_value = {}
-    missing = {}
-
-    for field, expected_val in expected.items():
-        if field not in actual:
-            missing[field] = expected_val
-            _LOGGER.error("FIELD %s is missing from actual event", field)
-            continue
-
-        actual_val = actual[field]
-        if actual_val == expected_val:
-            _LOGGER.info("FIELD %s OK", field)
-        else:
-            wrong_value[field] = {"actual": actual_val, "expected": expected_val}
-            _LOGGER.error(
-                "FIELD %s mismatch: actual=%r, expected=%r",
-                field,
-                actual_val,
-                expected_val,
-            )
-
-    message = _format_compatibility_errors(wrong_value, missing)
-    assert message is None, message
-
-
-def _format_compatibility_errors(wrong_value, missing):
-    """Return a formatted error string, or ``None`` when there are no errors."""
-    sections = []
-    if wrong_value:
-        lines = [
-            f"  {field}: actual={d['actual']!r}, expected={d['expected']!r}"
-            for field, d in wrong_value.items()
-        ]
-        sections.append("Wrong values:\n" + "\n".join(lines))
-    if missing:
-        lines = [f"  {field}: expected={val!r}" for field, val in missing.items()]
-        sections.append("Missing fields:\n" + "\n".join(lines))
-    if not sections:
-        return None
-    return "\n" + "\n".join(sections)
-
-
-_QUOTED_STRING_OR_BARE_KEY = re.compile(
-    r'"(?:[^"\\]|\\.)*"' r"|" r"([{,])\s*([A-Za-z_]+)\s*(:)"
-)
-
-
-def _quote_unquoted_keys(input_str):
-    """Wrap bare (unquoted) object keys in double quotes to produce valid JSON,
-    without corrupting content inside JSON string values.
-
-    Uses regex alternation: the first branch consumes quoted strings (so their
-    content is never touched), the second branch captures bare keys to quote.
-    """
-
-    def _replacer(m):
-        if m.group(1) is None:
-            return m.group(0)
-        return f'{m.group(1)}"{m.group(2)}":'
-
-    return _QUOTED_STRING_OR_BARE_KEY.sub(_replacer, input_str)
-
-
-def parse_raw_from_input(input_str):
-    """Parse SPL2-style or JSON array of events and return list of _raw field values.
-    Handles unquoted keys (e.g. {_raw: "x"}) by converting them to JSON-quoted keys.
-    """
-    normalized = _quote_unquoted_keys(input_str)
-    try:
-        parsed = json.loads(normalized)
-    except json.JSONDecodeError:
-        parsed = ast.literal_eval(normalized)
-    return [item["_raw"] for item in parsed]
-
-
-def parse_raw_from_reduced_events(events):
-    return [str(x["_raw"]) for x in events]