spl2-testing-framework 1.7.3__tar.gz → 1.8.1__tar.gz

{spl2_testing_framework-1.7.3 → spl2_testing_framework-1.8.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: spl2-testing-framework
-Version: 1.7.3
+Version: 1.8.1
 Summary: SPL2 Testing Framework
 Home-page: https://github.com/splunk/spl2-testing-framework
 Keywords: SPL2 testing framework
@@ -35,7 +35,7 @@ Description-Content-Type: text/markdown
 
 ## Overview
 
-The **SPL2 Testing Framework** enables running SPL2 tests both locally ( or on any Splunk instance with SPL2
+The **SPL2 Testing Framework** enables running SPL2 tests both locally (or on any Splunk instance with SPL2
 orchestrator), remotely (using external cloud environments) or using cli.
 
 - **For Cloud**: It uses
@@ -138,6 +138,9 @@ Additionally, the following options are supported:
 * `--create_comparison_sheet` - to create a comparison sheet in `comparison_box_test` folder using actual and expected outputs (works only when running box tests)
 * `--cli_bench` - when running tests with CLI, this option enables benchmarking mode by adding `-b` and `-n` flags to the spl2-processor-cli command. The value specifies the number of events to test (e.g., `--cli_bench=1000` will add `-b -n 1000` to the CLI command). **In bench mode, all tests always succeed and CLI output is printed to logs** - this is useful for performance testing and benchmarking. This is only applicable when using `--type cli`.
 * `--check_splunk_results` - option to validate box test output by sending events to a Splunk instance (using HEC) and querying them back, with support for **CIM** (Common Information Model) and **TA** (Technology Add-on) checks.
+* `--reduction_stats` - when set, **box tests** record a reduction rate per run: input `_raw` vs output `_raw` (after pairing input/output events). Rates are grouped by **pipeline** (the `.spl2` code module stem) and **sourcetype** (from the input event if set, else the output event, else `UNKNOWN`). Off by default.
+* `--reduction_stats_file` - path to a JSON file written at the **end of the pytest session** with collected rates (default `reduction_stats.json`). Only used when `--reduction_stats` is set.
+* `--reduction_stats_full` - when set together with `--reduction_stats`, each pipeline’s JSON payload also includes a pandas **`describe()`** summary (count, mean, std, min, quartiles, max) per sourcetype column. Only used when `--reduction_stats` is set.
 
 * Note: The `pytest.ini.sample` file allows you to define command parameters. Just update the configurations, rename the
 file by removing the `.sample` extension, and execute the command.
@@ -153,6 +156,7 @@ Sample configuration which may be used:
     - Module: `spl2_testing_framework.test_runner`
     - Parameters:
       `--type [cli | splunk | cloud] --test_dir /tests/resources -o log_cli=true --log-cli-level=INFO --verbose`
+    - Optional: append `--reduction_stats` (and `--reduction_stats_file`, `--reduction_stats_full` if needed) to collect box-test reduction JSON as described above.
     - If test dir is not specified, current working directory will be used
     - Use `--code_dir` if SPL2 code modules are in a separate directory from tests
     - If necessary another pytest options can be added
@@ -252,7 +256,31 @@ command ("|")
 Content of this file will also be printed to stdout.
 
 This checks can be applied only to box tests, as assertions which are used in unit tests may impact spl2 pipeline
-performance. 
+performance.
+
+### Reduction rate statistics (box tests)
+
+Optional telemetry for **how much** each pipeline shrinks event payloads (length of input `_raw` vs matched output `_raw`), broken down by **pipeline** and **sourcetype**.
+
+**Enable**
+
+- Pass `--reduction_stats` to pytest (or add it in `pytest.ini` / your IDE run configuration).
+
+**Output**
+
+- During the session, each box test appends one or more floating-point rates into an in-memory structure: `pipeline_name → sourcetype → [rates…]`.
+- After all tests finish, if `--reduction_stats` was set, the framework logs a short **pandas** summary per pipeline and writes **`--reduction_stats_file`** (default `reduction_stats.json`).
+- With **`--reduction_stats_full`**, the JSON file also embeds per-pipeline **`describe()`** statistics for each sourcetype column (handy for dashboards or quick distribution checks).
+
+**How rates are computed (high level)**
+
+- Input events are parsed from the box test `source` string; output rows come from the job’s destination results.
+- Input/output rows are **paired** using similarity on `_raw` (with a threshold and a same-length positional fallback). Only **matched** pairs contribute rates.
+- For each pair, lengths use a **canonical JSON length** when `_raw` parses as JSON or a Python literal; otherwise the raw string length is used, so identical logical content can yield a **0** reduction rate even if string formatting differs.
+
+**Scope**
+
+- Applies to **box tests** only. Single-SPL2 runs and unit tests do not populate reduction stats.
 
 ### Check splunk results
 
@@ -263,13 +291,13 @@ When set (`--check_splunk_results=<option>`), box tests (after running the SPL2
 | **CIM** | For each output event, ingest via HEC, load back from Splunk, then check CIM fields from `expected_cim_fields.cim_fields` with `validate_compatibility(actual, expected)`. Skips if the box test has no `cim_fields`. |
 | **TA** | For each output event, ingest via HEC, load back from Splunk, then assert equality with the expected row for that index. |
 
-For **TA** only, you can name top-level fields to drop on **both** the queried event and each `expected_destination_result` row by adding **`ignore_fields_in_splunk_check`**: an object mapping **field name → short reason**. Use the same key under `test` in each entry in `*.test.json` (same file as `expected_destination_result`):
+You can name top-level fields to drop on **both** the pipeline/actual row and each matching `expected_destination_result` row before they are compared by adding **`ignore_fields_in_destination_compare`**: an object mapping **field name → short reason**. Use the same key under `test` in each entry in `*.test.json` (same file as `expected_destination_result`):
 
 ```json
 "test": {
   "source": "...",
   "expected_destination_result": [ ... ],
-  "ignore_fields_in_splunk_check": {
+  "ignore_fields_in_destination_compare": {
     "_raw": "Raw event differs after Splunk indexing",
     "_time": "Timestamp may vary due to ingestion latency"
   }
@@ -277,8 +305,7 @@ For **TA** only, you can name top-level fields to drop on **both** the queried e
 ```
 
 
-If the key is omitted, no fields are ignored. Used only when `--check_splunk_results=TA` (ignored for other modes and for the default 
-in-memory box test assert). If the file or key is missing, no fields are ignored.
+If the key is omitted, no fields are ignored. It applies to the **default in-memory** box test assert (no `--check_splunk_results`) and to **`--check_splunk_results=TA`**. It does **not** apply to **`--check_splunk_results=CIM`**. If the file or key is missing, no fields are ignored.
 
 #### Configuration:
 

{spl2_testing_framework-1.7.3 → spl2_testing_framework-1.8.1}/README.md

@@ -2,7 +2,7 @@
 
 ## Overview
 
-The **SPL2 Testing Framework** enables running SPL2 tests both locally ( or on any Splunk instance with SPL2
+The **SPL2 Testing Framework** enables running SPL2 tests both locally (or on any Splunk instance with SPL2
 orchestrator), remotely (using external cloud environments) or using cli.
 
 - **For Cloud**: It uses
@@ -105,6 +105,9 @@ Additionally, the following options are supported:
 * `--create_comparison_sheet` - to create a comparison sheet in `comparison_box_test` folder using actual and expected outputs (works only when running box tests)
 * `--cli_bench` - when running tests with CLI, this option enables benchmarking mode by adding `-b` and `-n` flags to the spl2-processor-cli command. The value specifies the number of events to test (e.g., `--cli_bench=1000` will add `-b -n 1000` to the CLI command). **In bench mode, all tests always succeed and CLI output is printed to logs** - this is useful for performance testing and benchmarking. This is only applicable when using `--type cli`.
 * `--check_splunk_results` - option to validate box test output by sending events to a Splunk instance (using HEC) and querying them back, with support for **CIM** (Common Information Model) and **TA** (Technology Add-on) checks.
+* `--reduction_stats` - when set, **box tests** record a reduction rate per run: input `_raw` vs output `_raw` (after pairing input/output events). Rates are grouped by **pipeline** (the `.spl2` code module stem) and **sourcetype** (from the input event if set, else the output event, else `UNKNOWN`). Off by default.
+* `--reduction_stats_file` - path to a JSON file written at the **end of the pytest session** with collected rates (default `reduction_stats.json`). Only used when `--reduction_stats` is set.
+* `--reduction_stats_full` - when set together with `--reduction_stats`, each pipeline’s JSON payload also includes a pandas **`describe()`** summary (count, mean, std, min, quartiles, max) per sourcetype column. Only used when `--reduction_stats` is set.
 
 * Note: The `pytest.ini.sample` file allows you to define command parameters. Just update the configurations, rename the
 file by removing the `.sample` extension, and execute the command.
@@ -120,6 +123,7 @@ Sample configuration which may be used:
     - Module: `spl2_testing_framework.test_runner`
     - Parameters:
       `--type [cli | splunk | cloud] --test_dir /tests/resources -o log_cli=true --log-cli-level=INFO --verbose`
+    - Optional: append `--reduction_stats` (and `--reduction_stats_file`, `--reduction_stats_full` if needed) to collect box-test reduction JSON as described above.
     - If test dir is not specified, current working directory will be used
     - Use `--code_dir` if SPL2 code modules are in a separate directory from tests
     - If necessary another pytest options can be added
@@ -219,7 +223,31 @@ command ("|")
 Content of this file will also be printed to stdout.
 
 This checks can be applied only to box tests, as assertions which are used in unit tests may impact spl2 pipeline
-performance. 
+performance.
+
+### Reduction rate statistics (box tests)
+
+Optional telemetry for **how much** each pipeline shrinks event payloads (length of input `_raw` vs matched output `_raw`), broken down by **pipeline** and **sourcetype**.
+
+**Enable**
+
+- Pass `--reduction_stats` to pytest (or add it in `pytest.ini` / your IDE run configuration).
+
+**Output**
+
+- During the session, each box test appends one or more floating-point rates into an in-memory structure: `pipeline_name → sourcetype → [rates…]`.
+- After all tests finish, if `--reduction_stats` was set, the framework logs a short **pandas** summary per pipeline and writes **`--reduction_stats_file`** (default `reduction_stats.json`).
+- With **`--reduction_stats_full`**, the JSON file also embeds per-pipeline **`describe()`** statistics for each sourcetype column (handy for dashboards or quick distribution checks).
+
+**How rates are computed (high level)**
+
+- Input events are parsed from the box test `source` string; output rows come from the job’s destination results.
+- Input/output rows are **paired** using similarity on `_raw` (with a threshold and a same-length positional fallback). Only **matched** pairs contribute rates.
+- For each pair, lengths use a **canonical JSON length** when `_raw` parses as JSON or a Python literal; otherwise the raw string length is used, so identical logical content can yield a **0** reduction rate even if string formatting differs.
+
+**Scope**
+
+- Applies to **box tests** only. Single-SPL2 runs and unit tests do not populate reduction stats.
 
 ### Check splunk results
 
@@ -230,13 +258,13 @@ When set (`--check_splunk_results=<option>`), box tests (after running the SPL2
 | **CIM** | For each output event, ingest via HEC, load back from Splunk, then check CIM fields from `expected_cim_fields.cim_fields` with `validate_compatibility(actual, expected)`. Skips if the box test has no `cim_fields`. |
 | **TA** | For each output event, ingest via HEC, load back from Splunk, then assert equality with the expected row for that index. |
 
-For **TA** only, you can name top-level fields to drop on **both** the queried event and each `expected_destination_result` row by adding **`ignore_fields_in_splunk_check`**: an object mapping **field name → short reason**. Use the same key under `test` in each entry in `*.test.json` (same file as `expected_destination_result`):
+You can name top-level fields to drop on **both** the pipeline/actual row and each matching `expected_destination_result` row before they are compared by adding **`ignore_fields_in_destination_compare`**: an object mapping **field name → short reason**. Use the same key under `test` in each entry in `*.test.json` (same file as `expected_destination_result`):
 
 ```json
 "test": {
   "source": "...",
   "expected_destination_result": [ ... ],
-  "ignore_fields_in_splunk_check": {
+  "ignore_fields_in_destination_compare": {
     "_raw": "Raw event differs after Splunk indexing",
     "_time": "Timestamp may vary due to ingestion latency"
   }
@@ -244,8 +272,7 @@ For **TA** only, you can name top-level fields to drop on **both** the queried e
 ```
 
 
-If the key is omitted, no fields are ignored. Used only when `--check_splunk_results=TA` (ignored for other modes and for the default 
-in-memory box test assert). If the file or key is missing, no fields are ignored.
+If the key is omitted, no fields are ignored. It applies to the **default in-memory** box test assert (no `--check_splunk_results`) and to **`--check_splunk_results=TA`**. It does **not** apply to **`--check_splunk_results=CIM`**. If the file or key is missing, no fields are ignored.
 
 #### Configuration:
 

{spl2_testing_framework-1.7.3 → spl2_testing_framework-1.8.1}/pyproject.toml

@@ -15,7 +15,7 @@
 
 [tool.poetry]
 name = "spl2-testing-framework"
-version = "1.7.3"
+version = "1.8.1"
 description = "SPL2 Testing Framework"
 authors = ["mgazda-splunk <mgazda@splunk.com>"]
 readme = "README.md"

{spl2_testing_framework-1.7.3 → spl2_testing_framework-1.8.1}/spl2_testing_framework/conftest.py

@@ -14,19 +14,21 @@
 
 
 import json
+import logging
+import math
 import os
 from collections import defaultdict
 
-import pandas as pd
 import pytest
 
-from .tools.results import Results
-from .tools.splunk_integration.splunk_handler import SplunkHandler
-
 pytest.register_assert_rewrite("spl2_testing_framework")
 
 from requests.auth import AuthBase, HTTPBasicAuth
 
+from .logger_manager import setup_logging
+from .tools.results import Results
+from .tools.splunk_integration.splunk_handler import SplunkHandler
+
 from spl2_testing_framework.tools.search_clients.cli_search_client import (
     CLISearchClient,
 )
@@ -58,10 +60,7 @@ TEST_DIR = "test_dir"
 CODE_DIR = "code_dir"
 LIMIT_TESTS = "limit_tests"
 CLI_BENCH = "cli_bench"
-
-import logging
-
-from .logger_manager import setup_logging
+REDUCTION_STATS = "reduction_stats"
 
 setup_logging()
 
@@ -89,6 +88,24 @@ def pytest_addoption(parser):
     parser.addoption(f"--{IGNORE_ADDITIONAL_FIELDS_IN_ACTUAL}", action="store_true")
     parser.addoption(f"--{LIMIT_TESTS}", action="store", type=int, default=None)
     parser.addoption(f"--{CLI_BENCH}", action="store", type=int, default=None)
+    parser.addoption(
+        f"--{REDUCTION_STATS}",
+        action="store_true",
+        default=False,
+        help="Collect reduction rates per pipeline/sourcetype and save to file (see --reduction_stats_file, --reduction_stats_full). By default disabled.",
+    )
+    parser.addoption(
+        "--reduction_stats_file",
+        action="store",
+        default="reduction_stats.json",
+        help="Path to save reduction rates (JSON). Used only when --reduction_stats is set.",
+    )
+    parser.addoption(
+        "--reduction_stats_full",
+        action="store_true",
+        default=False,
+        help="Include full dataframe stats (count, mean, std, min, 25%%, 50%%, 75%%, max) in the saved file. Used only when --reduction_stats is set.",
+    )
 
 
 def pytest_generate_tests(metafunc):
@@ -169,7 +186,7 @@ def spl2_test_runner(
         url = f"https://{config['tenant']}.api.{config['cloud_instance']}/{config['tenant']}"
         search_client = CloudSearchClient(url=url, auth=bearer_token_authentication)
     else:
-        raise Exception(
+        raise ValueError(
             "Unknown test type: {}. Supported tests types: 'splunk', 'cloud', 'cli'".format(
                 test_type
             )
@@ -188,6 +205,9 @@ def run_configurations(request):
         ),
         CLI_BENCH: request.config.getoption(f"--{CLI_BENCH}"),
         CHECK_SPLUNK_RESULTS: request.config.getoption(f"--{CHECK_SPLUNK_RESULTS}"),
+        REDUCTION_STATS: request.config.getoption(
+            f"--{REDUCTION_STATS}", default=False
+        ),
     }
 
 
@@ -328,17 +348,71 @@ def splunk_handler(config, splunk_token_authentication):
 
 
 @pytest.fixture(scope="session")
-def stats():
+def stats(request):
     s = defaultdict(lambda: defaultdict(list))
     yield s
 
+    collect_reduction = request.config.getoption(f"--{REDUCTION_STATS}", default=False)
+    if not collect_reduction:
+        return
+
+    _LOGGER.info("Reduction stats: reporting (--reduction_stats was set)")
+    if not s:
+        _LOGGER.info(
+            "Reduction stats: no data (no box tests ran or no rates collected)"
+        )
+        return
+
+    import pandas as pd  # lazy import — only needed when --reduction_stats is set
+
+    def _pad_to_equal_length(rates_dict):
+        """Pad lists to same length with NaN so pd.DataFrame accepts different-length columns."""
+        if not rates_dict:
+            return rates_dict
+        max_len = max(len(lst) for lst in rates_dict.values())
+        return {
+            k: list(lst) + [float("nan")] * (max_len - len(lst))
+            for k, lst in rates_dict.items()
+        }
+
     for template, v in s.items():
-        x = pd.DataFrame(v)
+        v_padded = _pad_to_equal_length(dict(v))
+        x = pd.DataFrame(v_padded)
         _LOGGER.info(" *** *** ***")
         _LOGGER.info("Collected statistics: ")
         _LOGGER.info("Template name: %s ", template)
         _LOGGER.info(x.describe())
 
+    out_path = request.config.getoption(
+        "--reduction_stats_file", default="reduction_stats.json"
+    )
+    include_full = request.config.getoption("--reduction_stats_full", default=False)
+    if out_path:
+        data = {}
+        for pipeline, sourcetype_rates in s.items():
+            rates_dict = dict(sourcetype_rates)
+            if include_full:
+                rates_padded = _pad_to_equal_length(rates_dict)
+                df = pd.DataFrame(rates_padded)
+                describe = df.describe().to_dict()
+                # NaN is not JSON-serializable; convert to None
+                describe_serializable = {
+                    k: {
+                        kk: (None if isinstance(vv, float) and math.isnan(vv) else vv)
+                        for kk, vv in v.items()
+                    }
+                    for k, v in describe.items()
+                }
+                data[pipeline] = {
+                    "rates": rates_dict,
+                    "describe": describe_serializable,
+                }
+            else:
+                data[pipeline] = rates_dict
+        with open(out_path, "w", encoding="utf-8") as f:
+            json.dump(data, f, indent=2)
+        _LOGGER.info("Reduction rates saved to %s", out_path)
+
 
 def _set_results_options(metafunc):
     ignore_empty_strings = metafunc.config.getoption(IGNORE_EMPTY_STRINGS)

spl2_testing_framework-1.8.1/spl2_testing_framework/spl2_test_config.json

@@ -0,0 +1,18 @@
+{
+  "host": "",
+  "cloud_instance": "",
+  "tenant": "",
+  "bearer_token": "",
+  "port": "",
+  "user": "",
+  "password": "",
+  "SPLUNK_INSTANCE": {
+    "ip": "localhost",
+    "port": 8088,
+    "api_port": 8089,
+    "username": "admin",
+    "password": "Chang3d!",
+    "index": "testx",
+    "hec_token": "4a8a737d-5452-426c-a6f7-106dca4e813f"
+  }
+}

{spl2_testing_framework-1.7.3 → spl2_testing_framework-1.8.1}/spl2_testing_framework/tools/performance.py

@@ -249,7 +249,14 @@ class PerformanceCheck:
 
     @staticmethod
     def _preserve_timestamps_through_fields(code_module: str) -> str:
-        """Replace '| fields' keep-mode with a no-op so _timestamp_perf* fields are not stripped."""
+        """Replace '| fields' keep-mode with a no-op so _timestamp_perf* fields are not stripped.
+
+        During performance instrumentation the pipeline runs with a slightly different shape:
+        keep-mode ``| fields`` commands are removed to prevent them from stripping the injected
+        ``_timestamp_perf*`` variables.  This means perf numbers reflect a pipeline with all
+        fields passing through, which is slightly heavier than production but avoids silently
+        dropping the instrumentation fields and producing zeroed measurements.
+        """
         return re.sub(
             r"\|\s*fields\s+(?!-)(.+)",
             r"| eval _perf_fields_preserved=1",
@@ -297,11 +304,13 @@ class PerformanceCheck:
             flags=re.M,
         )
 
-        func_timestamps = re.findall(rf"_time_{f['f_name']}_placeholder", new_function)
+        func_timestamps = re.findall(
+            rf"_time_{re.escape(f['f_name'])}_placeholder", new_function
+        )
 
         for n, timestamp in enumerate(func_timestamps):
             new_function = re.sub(
-                timestamp,
+                re.escape(timestamp),
                 rf"_timestamp_perf{f['f_name']}_{n:03d}",
                 new_function,
                 count=1,
@@ -318,7 +327,7 @@ class PerformanceCheck:
         )
 
         func_timestamps = re.findall(
-            rf"\$_time_{f['f_name']}_placeholder", new_function
+            rf"\$_time_{re.escape(f['f_name'])}_placeholder", new_function
         )
 
         for n, timestamp in enumerate(func_timestamps):

{spl2_testing_framework-1.7.3 → spl2_testing_framework-1.8.1}/spl2_testing_framework/tools/results.py

@@ -175,14 +175,12 @@ class Results(UserDict):
                                         actual_data[key][i], expected_data[key][i]
                                     )
 
-        return actual_data
-
     def remove_fields_not_in_expected(self, expected):
         """Remove fields from this Results object that are not present in the expected Results object.
         This is the public method to be called when comparing results.
         """
         if self._ignore_additional_fields_in_actual and isinstance(expected, Results):
-            self.data = self._remove_extra_fields(self.data, expected.data)
+            self._remove_extra_fields(self.data, expected.data)
         return self
 
     def __lt__(self, other):

{spl2_testing_framework-1.7.3 → spl2_testing_framework-1.8.1}/spl2_testing_framework/tools/spl2test_runner.py

@@ -33,9 +33,7 @@ from spl2_testing_framework.tools.test_types import UnitTest, BoxTest, SingleSPL
 from spl2_testing_framework.tools.utils import (
     _make_functions_visible_for_testing,
     validate_compatibility,
-    calculate_reduction_rate,
-    parse_raw_from_input,
-    parse_raw_from_reduced_events,
+    calculate_reduction_rate_and_update_stats,
 )
 
 _LOGGER = logging.getLogger(__name__)
@@ -44,13 +42,21 @@ _LOGGER = logging.getLogger(__name__)
 class SPL2TestRunner:
     """Main class responsible for running tests"""
 
-    def __init__(self, search_client, run_configurations, splunk_handler, stats):
+    def __init__(
+        self, search_client, run_configurations, splunk_handler=None, stats=None
+    ):
         self._search_client = search_client
         self.create_comparison_sheet = run_configurations.get("create_comparison_sheet")
         self.check_splunk_results = run_configurations.get("check_splunk_results")
+        self.collect_reduction_stats = run_configurations.get("reduction_stats", False)
         self.comp_obj = ComparisonSheet()
         self.reduction_stats = stats
         if self.check_splunk_results:
+            if splunk_handler is None:
+                raise ValueError(
+                    f"check_splunk_results={self.check_splunk_results!r} requires a "
+                    "splunk_handler to be provided, but got None."
+                )
             splunk_handler.setup()
             self.splunk_handler = splunk_handler
 
@@ -181,26 +187,48 @@ class SPL2TestRunner:
 
         perf.check_performance(output_result)
 
-        reduction_rate = calculate_reduction_rate(
-            parse_raw_from_input(box_test.input),
-            parse_raw_from_reduced_events(output_result),
-        )
-        self.reduction_stats[box_test.name]["reduction_rate"].extend(reduction_rate)
-
-        _LOGGER.info("Reduction rate for this event: %s", reduction_rate)
+        if self.collect_reduction_stats:
+            reduction_rate = calculate_reduction_rate_and_update_stats(
+                box_test.code_module_path.stem,
+                box_test.input,
+                output_result,
+                self.reduction_stats,
+            )
+            _LOGGER.info("Reduction rate for this event: %s", reduction_rate)
 
         if self.check_splunk_results:
             self._check_splunk_results(box_test, output_result)
         else:
             if box_test.output:
-                # Remove fields from actual results that are not in expected results if configured
+                ignore_keys = frozenset(
+                    box_test.ignore_fields_in_destination_compare.keys()
+                )
                 for i, actual_result in enumerate(output_result):
                     if i < len(box_test.output):
                         actual_result.remove_fields_not_in_expected(box_test.output[i])
 
                 if self.create_comparison_sheet:
                     self.comp_obj.create_comparison_sheet(output_result, box_test)
-                assert sorted(output_result) == sorted(box_test.output)
+                if ignore_keys:
+                    stripped_actual = [
+                        Results(
+                            self._strip_ignored_fields_for_destination_compare(
+                                row.data, ignore_keys
+                            )
+                        )
+                        for row in output_result
+                    ]
+                    stripped_expected = [
+                        Results(
+                            self._strip_ignored_fields_for_destination_compare(
+                                row.data, ignore_keys
+                            )
+                        )
+                        for row in box_test.output
+                    ]
+                    assert sorted(stripped_actual) == sorted(stripped_expected)
+                else:
+                    assert sorted(output_result) == sorted(box_test.output)
                 _LOGGER.info(f"Output check passed: {box_test.name}")
                 _LOGGER.debug("Received: \n%s", str(output_result))
                 _LOGGER.debug("Expected: \n%s", str(box_test.output))
@@ -243,7 +271,7 @@ class SPL2TestRunner:
             return copy.deepcopy(d)
 
     @staticmethod
-    def _strip_ignored_fields_for_splunk_compare(
+    def _strip_ignored_fields_for_destination_compare(
         row: dict, ignore_keys: frozenset
     ) -> dict:
         if not ignore_keys:
@@ -260,7 +288,12 @@ class SPL2TestRunner:
             ) from e
 
     def _check_splunk_results(self, box_test, destination_rows):
-        """Use the same destination rows as ``run_box_test`` (from ``get_job_results``), not ``job.result``."""
+        """Validate HEC round-trip rows against the box test (CIM or TA).
+
+        ``destination_rows`` must be the same list returned from
+        ``get_job_results(job)["destination"]`` as used for reduction stats and
+        non-Splunk output assertions — not ``job.result``.
+        """
         if self.check_splunk_results == "CIM":
             if not box_test.cim_fields:
                 pytest.skip("CIM output not available for this test")
@@ -279,9 +312,27 @@ class SPL2TestRunner:
             if not box_test.output:
                 pytest.skip("TA expected output not available for this test")
 
-            ignore_spec = box_test.ignore_fields_in_splunk_check
+            ignore_spec = box_test.ignore_fields_in_destination_compare
             ignore_keys = frozenset(ignore_spec.keys())
-            for event, expected in zip(destination_rows, box_test.output):
+            assert len(destination_rows) == len(box_test.output), (
+                f"TA Splunk check for {box_test.name!r}: "
+                f"got {len(destination_rows)} destination row(s) but "
+                f"{len(box_test.output)} expected output(s); zip would hide a mismatch."
+            )
+
+            def _raw_sort_key(row):
+                raw = row.data.get("_raw") if hasattr(row, "data") else row.get("_raw")
+                if isinstance(raw, str):
+                    try:
+                        raw = json.loads(raw)
+                    except (ValueError, TypeError):
+                        pass
+                return json.dumps(raw, sort_keys=True, default=str)
+
+            for event, expected in zip(
+                sorted(destination_rows, key=_raw_sort_key),
+                sorted(box_test.output, key=_raw_sort_key),
+            ):
                 processed_event = self._splunk_process_event(
                     box_test, self._row_as_dict(event)
                 )
@@ -290,15 +341,19 @@ class SPL2TestRunner:
                 # dropping keys not present on the expected row before asserting.
                 actual_dict = dict(processed_event)
                 Results._remove_extra_fields(actual_dict, expected_data)
-                actual_cmp = self._strip_ignored_fields_for_splunk_compare(
+                actual_cmp = self._strip_ignored_fields_for_destination_compare(
                     actual_dict, ignore_keys
                 )
-                expected_cmp = self._strip_ignored_fields_for_splunk_compare(
+                expected_cmp = self._strip_ignored_fields_for_destination_compare(
                     expected_data, ignore_keys
                 )
                 assert actual_cmp == expected_cmp, (
-                    f"TA Splunk compare mismatch for {box_test.name}; "
-                    f"ignored_fields={ignore_spec!r}"
+                    f"TA destination compare mismatch for {box_test.name}; "
+                    f"ignore_fields_in_destination_compare={ignore_spec!r}"
                 )
         else:
-            return
+            _LOGGER.warning(
+                "Unknown check_splunk_results value %r — expected 'CIM' or 'TA'. "
+                "No Splunk validation performed.",
+                self.check_splunk_results,
+            )

{spl2_testing_framework-1.7.3 → spl2_testing_framework-1.8.1}/spl2_testing_framework/tools/splunk_integration/splunk_handler.py

@@ -13,16 +13,18 @@
 # limitations under the License.
 
 import logging
+import time
 import uuid
 from typing import Optional
 
 import requests
+import urllib3
 import splunklib.client as client
 import splunklib.results as results
-
-import time
-
 from requests.auth import HTTPBasicAuth
+import json
+
+urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)
 
 _LOGGER = logging.getLogger(__name__)
 
@@ -82,10 +84,12 @@ class SplunkHandler:
     def _raw_as_event_string(raw):
         """Normalize pipeline `_raw` for HEC JSON `event` (must be a string).
 
-        ``str`` is passed through ``str()``; ``bytes`` are decoded as UTF-8 with
-        replacement on invalid sequences (avoids :class:`UnicodeDecodeError` for
-        odd binary payloads; prefer valid UTF-8 log text in normal use).
+        ``dict`` is serialized as valid JSON (double-quoted) so that Splunk's
+        field extractions (e.g. GCP TA KV_MODE=json) can parse it correctly.
+        ``bytes`` are decoded as UTF-8. Everything else uses ``str()``.
         """
+        if isinstance(raw, dict):
+            return json.dumps(raw)  # {"data": {...}} - valid JSON
         if isinstance(raw, bytes):
             return raw.decode("utf-8", errors="replace")
         return str(raw)
@@ -165,4 +169,11 @@ class SplunkHandler:
             ),
             verify=False,
         )
+        if response.status_code == 409:
+            _LOGGER.info(
+                "Index '%s' already exists — skipping creation.",
+                self.splunk_config["index"],
+            )
+            return
+        response.raise_for_status()
         return response.json()