spl2-testing-framework 1.7.2__tar.gz → 1.8.0__tar.gz

{spl2_testing_framework-1.7.2 → spl2_testing_framework-1.8.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: spl2-testing-framework
-Version: 1.7.2
+Version: 1.8.0
 Summary: SPL2 Testing Framework
 Home-page: https://github.com/splunk/spl2-testing-framework
 Keywords: SPL2 testing framework
@@ -35,7 +35,7 @@ Description-Content-Type: text/markdown
 
 ## Overview
 
-The **SPL2 Testing Framework** enables running SPL2 tests both locally ( or on any Splunk instance with SPL2
+The **SPL2 Testing Framework** enables running SPL2 tests both locally (or on any Splunk instance with SPL2
 orchestrator), remotely (using external cloud environments) or using cli.
 
 - **For Cloud**: It uses
@@ -138,6 +138,9 @@ Additionally, the following options are supported:
 * `--create_comparison_sheet` - to create a comparison sheet in `comparison_box_test` folder using actual and expected outputs (works only when running box tests)
 * `--cli_bench` - when running tests with CLI, this option enables benchmarking mode by adding `-b` and `-n` flags to the spl2-processor-cli command. The value specifies the number of events to test (e.g., `--cli_bench=1000` will add `-b -n 1000` to the CLI command). **In bench mode, all tests always succeed and CLI output is printed to logs** - this is useful for performance testing and benchmarking. This is only applicable when using `--type cli`.
 * `--check_splunk_results` - option to validate box test output by sending events to a Splunk instance (using HEC) and querying them back, with support for **CIM** (Common Information Model) and **TA** (Technology Add-on) checks.
+* `--reduction_stats` - when set, **box tests** record a reduction rate per run: input `_raw` vs output `_raw` (after pairing input/output events). Rates are grouped by **pipeline** (the `.spl2` code module stem) and **sourcetype** (from the input event if set, else the output event, else `UNKNOWN`). Off by default.
+* `--reduction_stats_file` - path to a JSON file written at the **end of the pytest session** with collected rates (default `reduction_stats.json`). Only used when `--reduction_stats` is set.
+* `--reduction_stats_full` - when set together with `--reduction_stats`, each pipeline’s JSON payload also includes a pandas **`describe()`** summary (count, mean, std, min, quartiles, max) per sourcetype column. Only used when `--reduction_stats` is set.
 
 * Note: The `pytest.ini.sample` file allows you to define command parameters. Just update the configurations, rename the
 file by removing the `.sample` extension, and execute the command.
@@ -153,6 +156,7 @@ Sample configuration which may be used:
     - Module: `spl2_testing_framework.test_runner`
     - Parameters:
       `--type [cli | splunk | cloud] --test_dir /tests/resources -o log_cli=true --log-cli-level=INFO --verbose`
+    - Optional: append `--reduction_stats` (and `--reduction_stats_file`, `--reduction_stats_full` if needed) to collect box-test reduction JSON as described above.
     - If test dir is not specified, current working directory will be used
     - Use `--code_dir` if SPL2 code modules are in a separate directory from tests
     - If necessary another pytest options can be added
@@ -252,7 +256,31 @@ command ("|")
 Content of this file will also be printed to stdout.
 
 This checks can be applied only to box tests, as assertions which are used in unit tests may impact spl2 pipeline
-performance. 
+performance.
+
+### Reduction rate statistics (box tests)
+
+Optional telemetry for **how much** each pipeline shrinks event payloads (length of input `_raw` vs matched output `_raw`), broken down by **pipeline** and **sourcetype**.
+
+**Enable**
+
+- Pass `--reduction_stats` to pytest (or add it in `pytest.ini` / your IDE run configuration).
+
+**Output**
+
+- During the session, each box test appends one or more floating-point rates into an in-memory structure: `pipeline_name → sourcetype → [rates…]`.
+- After all tests finish, if `--reduction_stats` was set, the framework logs a short **pandas** summary per pipeline and writes **`--reduction_stats_file`** (default `reduction_stats.json`).
+- With **`--reduction_stats_full`**, the JSON file also embeds per-pipeline **`describe()`** statistics for each sourcetype column (handy for dashboards or quick distribution checks).
+
+**How rates are computed (high level)**
+
+- Input events are parsed from the box test `source` string; output rows come from the job’s destination results.
+- Input/output rows are **paired** using similarity on `_raw` (with a threshold and a same-length positional fallback). Only **matched** pairs contribute rates.
+- For each pair, lengths use a **canonical JSON length** when `_raw` parses as JSON or a Python literal; otherwise the raw string length is used, so identical logical content can yield a **0** reduction rate even if string formatting differs.
+
+**Scope**
+
+- Applies to **box tests** only. Single-SPL2 runs and unit tests do not populate reduction stats.
 
 ### Check splunk results
 
@@ -263,13 +291,13 @@ When set (`--check_splunk_results=<option>`), box tests (after running the SPL2
 | **CIM** | For each output event, ingest via HEC, load back from Splunk, then check CIM fields from `expected_cim_fields.cim_fields` with `validate_compatibility(actual, expected)`. Skips if the box test has no `cim_fields`. |
 | **TA** | For each output event, ingest via HEC, load back from Splunk, then assert equality with the expected row for that index. |
 
-For **TA** only, you can name top-level fields to drop on **both** the queried event and each `expected_destination_result` row by adding **`ignore_fields_in_splunk_check`**: an object mapping **field name → short reason**. Use the same key under `test` in each entry in `*.test.json` (same file as `expected_destination_result`):
+You can name top-level fields to drop on **both** the pipeline/actual row and each matching `expected_destination_result` row before they are compared by adding **`ignore_fields_in_destination_compare`**: an object mapping **field name → short reason**. Use the same key under `test` in each entry in `*.test.json` (same file as `expected_destination_result`):
 
 ```json
 "test": {
   "source": "...",
   "expected_destination_result": [ ... ],
-  "ignore_fields_in_splunk_check": {
+  "ignore_fields_in_destination_compare": {
     "_raw": "Raw event differs after Splunk indexing",
     "_time": "Timestamp may vary due to ingestion latency"
   }
@@ -277,8 +305,7 @@ For **TA** only, you can name top-level fields to drop on **both** the queried e
 ```
 
 
-If the key is omitted, no fields are ignored. Used only when `--check_splunk_results=TA` (ignored for other modes and for the default 
-in-memory box test assert). If the file or key is missing, no fields are ignored.
+If the key is omitted, no fields are ignored. It applies to the **default in-memory** box test assert (no `--check_splunk_results`) and to **`--check_splunk_results=TA`**. It does **not** apply to **`--check_splunk_results=CIM`**. If the file or key is missing, no fields are ignored.
 
 #### Configuration:
 

{spl2_testing_framework-1.7.2 → spl2_testing_framework-1.8.0}/README.md

@@ -2,7 +2,7 @@
 
 ## Overview
 
-The **SPL2 Testing Framework** enables running SPL2 tests both locally ( or on any Splunk instance with SPL2
+The **SPL2 Testing Framework** enables running SPL2 tests both locally (or on any Splunk instance with SPL2
 orchestrator), remotely (using external cloud environments) or using cli.
 
 - **For Cloud**: It uses
@@ -105,6 +105,9 @@ Additionally, the following options are supported:
 * `--create_comparison_sheet` - to create a comparison sheet in `comparison_box_test` folder using actual and expected outputs (works only when running box tests)
 * `--cli_bench` - when running tests with CLI, this option enables benchmarking mode by adding `-b` and `-n` flags to the spl2-processor-cli command. The value specifies the number of events to test (e.g., `--cli_bench=1000` will add `-b -n 1000` to the CLI command). **In bench mode, all tests always succeed and CLI output is printed to logs** - this is useful for performance testing and benchmarking. This is only applicable when using `--type cli`.
 * `--check_splunk_results` - option to validate box test output by sending events to a Splunk instance (using HEC) and querying them back, with support for **CIM** (Common Information Model) and **TA** (Technology Add-on) checks.
+* `--reduction_stats` - when set, **box tests** record a reduction rate per run: input `_raw` vs output `_raw` (after pairing input/output events). Rates are grouped by **pipeline** (the `.spl2` code module stem) and **sourcetype** (from the input event if set, else the output event, else `UNKNOWN`). Off by default.
+* `--reduction_stats_file` - path to a JSON file written at the **end of the pytest session** with collected rates (default `reduction_stats.json`). Only used when `--reduction_stats` is set.
+* `--reduction_stats_full` - when set together with `--reduction_stats`, each pipeline’s JSON payload also includes a pandas **`describe()`** summary (count, mean, std, min, quartiles, max) per sourcetype column. Only used when `--reduction_stats` is set.
 
 * Note: The `pytest.ini.sample` file allows you to define command parameters. Just update the configurations, rename the
 file by removing the `.sample` extension, and execute the command.
@@ -120,6 +123,7 @@ Sample configuration which may be used:
     - Module: `spl2_testing_framework.test_runner`
     - Parameters:
       `--type [cli | splunk | cloud] --test_dir /tests/resources -o log_cli=true --log-cli-level=INFO --verbose`
+    - Optional: append `--reduction_stats` (and `--reduction_stats_file`, `--reduction_stats_full` if needed) to collect box-test reduction JSON as described above.
     - If test dir is not specified, current working directory will be used
     - Use `--code_dir` if SPL2 code modules are in a separate directory from tests
     - If necessary another pytest options can be added
@@ -219,7 +223,31 @@ command ("|")
 Content of this file will also be printed to stdout.
 
 This checks can be applied only to box tests, as assertions which are used in unit tests may impact spl2 pipeline
-performance. 
+performance.
+
+### Reduction rate statistics (box tests)
+
+Optional telemetry for **how much** each pipeline shrinks event payloads (length of input `_raw` vs matched output `_raw`), broken down by **pipeline** and **sourcetype**.
+
+**Enable**
+
+- Pass `--reduction_stats` to pytest (or add it in `pytest.ini` / your IDE run configuration).
+
+**Output**
+
+- During the session, each box test appends one or more floating-point rates into an in-memory structure: `pipeline_name → sourcetype → [rates…]`.
+- After all tests finish, if `--reduction_stats` was set, the framework logs a short **pandas** summary per pipeline and writes **`--reduction_stats_file`** (default `reduction_stats.json`).
+- With **`--reduction_stats_full`**, the JSON file also embeds per-pipeline **`describe()`** statistics for each sourcetype column (handy for dashboards or quick distribution checks).
+
+**How rates are computed (high level)**
+
+- Input events are parsed from the box test `source` string; output rows come from the job’s destination results.
+- Input/output rows are **paired** using similarity on `_raw` (with a threshold and a same-length positional fallback). Only **matched** pairs contribute rates.
+- For each pair, lengths use a **canonical JSON length** when `_raw` parses as JSON or a Python literal; otherwise the raw string length is used, so identical logical content can yield a **0** reduction rate even if string formatting differs.
+
+**Scope**
+
+- Applies to **box tests** only. Single-SPL2 runs and unit tests do not populate reduction stats.
 
 ### Check splunk results
 
@@ -230,13 +258,13 @@ When set (`--check_splunk_results=<option>`), box tests (after running the SPL2
 | **CIM** | For each output event, ingest via HEC, load back from Splunk, then check CIM fields from `expected_cim_fields.cim_fields` with `validate_compatibility(actual, expected)`. Skips if the box test has no `cim_fields`. |
 | **TA** | For each output event, ingest via HEC, load back from Splunk, then assert equality with the expected row for that index. |
 
-For **TA** only, you can name top-level fields to drop on **both** the queried event and each `expected_destination_result` row by adding **`ignore_fields_in_splunk_check`**: an object mapping **field name → short reason**. Use the same key under `test` in each entry in `*.test.json` (same file as `expected_destination_result`):
+You can name top-level fields to drop on **both** the pipeline/actual row and each matching `expected_destination_result` row before they are compared by adding **`ignore_fields_in_destination_compare`**: an object mapping **field name → short reason**. Use the same key under `test` in each entry in `*.test.json` (same file as `expected_destination_result`):
 
 ```json
 "test": {
   "source": "...",
   "expected_destination_result": [ ... ],
-  "ignore_fields_in_splunk_check": {
+  "ignore_fields_in_destination_compare": {
     "_raw": "Raw event differs after Splunk indexing",
     "_time": "Timestamp may vary due to ingestion latency"
   }
@@ -244,8 +272,7 @@ For **TA** only, you can name top-level fields to drop on **both** the queried e
 ```
 
 
-If the key is omitted, no fields are ignored. Used only when `--check_splunk_results=TA` (ignored for other modes and for the default 
-in-memory box test assert). If the file or key is missing, no fields are ignored.
+If the key is omitted, no fields are ignored. It applies to the **default in-memory** box test assert (no `--check_splunk_results`) and to **`--check_splunk_results=TA`**. It does **not** apply to **`--check_splunk_results=CIM`**. If the file or key is missing, no fields are ignored.
 
 #### Configuration:
 

{spl2_testing_framework-1.7.2 → spl2_testing_framework-1.8.0}/pyproject.toml

@@ -15,7 +15,7 @@
 
 [tool.poetry]
 name = "spl2-testing-framework"
-version = "1.7.2"
+version = "1.8.0"
 description = "SPL2 Testing Framework"
 authors = ["mgazda-splunk <mgazda@splunk.com>"]
 readme = "README.md"

{spl2_testing_framework-1.7.2 → spl2_testing_framework-1.8.0}/spl2_testing_framework/conftest.py

@@ -14,19 +14,21 @@
 
 
 import json
+import logging
+import math
 import os
 from collections import defaultdict
 
-import pandas as pd
 import pytest
 
-from .tools.results import Results
-from .tools.splunk_integration.splunk_handler import SplunkHandler
-
 pytest.register_assert_rewrite("spl2_testing_framework")
 
 from requests.auth import AuthBase, HTTPBasicAuth
 
+from .logger_manager import setup_logging
+from .tools.results import Results
+from .tools.splunk_integration.splunk_handler import SplunkHandler
+
 from spl2_testing_framework.tools.search_clients.cli_search_client import (
     CLISearchClient,
 )
@@ -58,10 +60,7 @@ TEST_DIR = "test_dir"
 CODE_DIR = "code_dir"
 LIMIT_TESTS = "limit_tests"
 CLI_BENCH = "cli_bench"
-
-import logging
-
-from .logger_manager import setup_logging
+REDUCTION_STATS = "reduction_stats"
 
 setup_logging()
 
@@ -89,6 +88,24 @@ def pytest_addoption(parser):
     parser.addoption(f"--{IGNORE_ADDITIONAL_FIELDS_IN_ACTUAL}", action="store_true")
     parser.addoption(f"--{LIMIT_TESTS}", action="store", type=int, default=None)
     parser.addoption(f"--{CLI_BENCH}", action="store", type=int, default=None)
+    parser.addoption(
+        f"--{REDUCTION_STATS}",
+        action="store_true",
+        default=False,
+        help="Collect reduction rates per pipeline/sourcetype and save to file (see --reduction_stats_file, --reduction_stats_full). By default disabled.",
+    )
+    parser.addoption(
+        "--reduction_stats_file",
+        action="store",
+        default="reduction_stats.json",
+        help="Path to save reduction rates (JSON). Used only when --reduction_stats is set.",
+    )
+    parser.addoption(
+        "--reduction_stats_full",
+        action="store_true",
+        default=False,
+        help="Include full dataframe stats (count, mean, std, min, 25%%, 50%%, 75%%, max) in the saved file. Used only when --reduction_stats is set.",
+    )
 
 
 def pytest_generate_tests(metafunc):
@@ -169,7 +186,7 @@ def spl2_test_runner(
         url = f"https://{config['tenant']}.api.{config['cloud_instance']}/{config['tenant']}"
         search_client = CloudSearchClient(url=url, auth=bearer_token_authentication)
     else:
-        raise Exception(
+        raise ValueError(
             "Unknown test type: {}. Supported tests types: 'splunk', 'cloud', 'cli'".format(
                 test_type
             )
@@ -188,6 +205,9 @@ def run_configurations(request):
         ),
         CLI_BENCH: request.config.getoption(f"--{CLI_BENCH}"),
         CHECK_SPLUNK_RESULTS: request.config.getoption(f"--{CHECK_SPLUNK_RESULTS}"),
+        REDUCTION_STATS: request.config.getoption(
+            f"--{REDUCTION_STATS}", default=False
+        ),
     }
 
 
@@ -328,17 +348,71 @@ def splunk_handler(config, splunk_token_authentication):
 
 
 @pytest.fixture(scope="session")
-def stats():
+def stats(request):
     s = defaultdict(lambda: defaultdict(list))
     yield s
 
+    collect_reduction = request.config.getoption(f"--{REDUCTION_STATS}", default=False)
+    if not collect_reduction:
+        return
+
+    _LOGGER.info("Reduction stats: reporting (--reduction_stats was set)")
+    if not s:
+        _LOGGER.info(
+            "Reduction stats: no data (no box tests ran or no rates collected)"
+        )
+        return
+
+    import pandas as pd  # lazy import — only needed when --reduction_stats is set
+
+    def _pad_to_equal_length(rates_dict):
+        """Pad lists to same length with NaN so pd.DataFrame accepts different-length columns."""
+        if not rates_dict:
+            return rates_dict
+        max_len = max(len(lst) for lst in rates_dict.values())
+        return {
+            k: list(lst) + [float("nan")] * (max_len - len(lst))
+            for k, lst in rates_dict.items()
+        }
+
     for template, v in s.items():
-        x = pd.DataFrame(v)
+        v_padded = _pad_to_equal_length(dict(v))
+        x = pd.DataFrame(v_padded)
         _LOGGER.info(" *** *** ***")
         _LOGGER.info("Collected statistics: ")
         _LOGGER.info("Template name: %s ", template)
         _LOGGER.info(x.describe())
 
+    out_path = request.config.getoption(
+        "--reduction_stats_file", default="reduction_stats.json"
+    )
+    include_full = request.config.getoption("--reduction_stats_full", default=False)
+    if out_path:
+        data = {}
+        for pipeline, sourcetype_rates in s.items():
+            rates_dict = dict(sourcetype_rates)
+            if include_full:
+                rates_padded = _pad_to_equal_length(rates_dict)
+                df = pd.DataFrame(rates_padded)
+                describe = df.describe().to_dict()
+                # NaN is not JSON-serializable; convert to None
+                describe_serializable = {
+                    k: {
+                        kk: (None if isinstance(vv, float) and math.isnan(vv) else vv)
+                        for kk, vv in v.items()
+                    }
+                    for k, v in describe.items()
+                }
+                data[pipeline] = {
+                    "rates": rates_dict,
+                    "describe": describe_serializable,
+                }
+            else:
+                data[pipeline] = rates_dict
+        with open(out_path, "w", encoding="utf-8") as f:
+            json.dump(data, f, indent=2)
+        _LOGGER.info("Reduction rates saved to %s", out_path)
+
 
 def _set_results_options(metafunc):
     ignore_empty_strings = metafunc.config.getoption(IGNORE_EMPTY_STRINGS)

spl2_testing_framework-1.8.0/spl2_testing_framework/spl2_test_config.json

@@ -0,0 +1,18 @@
+{
+  "host": "",
+  "cloud_instance": "",
+  "tenant": "",
+  "bearer_token": "",
+  "port": "",
+  "user": "",
+  "password": "",
+  "SPLUNK_INSTANCE": {
+    "ip": "localhost",
+    "port": 8088,
+    "api_port": 8089,
+    "username": "admin",
+    "password": "Chang3d!",
+    "index": "testx",
+    "hec_token": "4a8a737d-5452-426c-a6f7-106dca4e813f"
+  }
+}

{spl2_testing_framework-1.7.2 → spl2_testing_framework-1.8.0}/spl2_testing_framework/tools/performance.py

@@ -249,7 +249,14 @@ class PerformanceCheck:
 
     @staticmethod
     def _preserve_timestamps_through_fields(code_module: str) -> str:
-        """Replace '| fields' keep-mode with a no-op so _timestamp_perf* fields are not stripped."""
+        """Replace '| fields' keep-mode with a no-op so _timestamp_perf* fields are not stripped.
+
+        During performance instrumentation the pipeline runs with a slightly different shape:
+        keep-mode ``| fields`` commands are removed to prevent them from stripping the injected
+        ``_timestamp_perf*`` variables.  This means perf numbers reflect a pipeline with all
+        fields passing through, which is slightly heavier than production but avoids silently
+        dropping the instrumentation fields and producing zeroed measurements.
+        """
         return re.sub(
             r"\|\s*fields\s+(?!-)(.+)",
             r"| eval _perf_fields_preserved=1",
@@ -297,11 +304,13 @@ class PerformanceCheck:
             flags=re.M,
         )
 
-        func_timestamps = re.findall(rf"_time_{f['f_name']}_placeholder", new_function)
+        func_timestamps = re.findall(
+            rf"_time_{re.escape(f['f_name'])}_placeholder", new_function
+        )
 
         for n, timestamp in enumerate(func_timestamps):
             new_function = re.sub(
-                timestamp,
+                re.escape(timestamp),
                 rf"_timestamp_perf{f['f_name']}_{n:03d}",
                 new_function,
                 count=1,
@@ -318,7 +327,7 @@ class PerformanceCheck:
         )
 
         func_timestamps = re.findall(
-            rf"\$_time_{f['f_name']}_placeholder", new_function
+            rf"\$_time_{re.escape(f['f_name'])}_placeholder", new_function
         )
 
         for n, timestamp in enumerate(func_timestamps):

{spl2_testing_framework-1.7.2 → spl2_testing_framework-1.8.0}/spl2_testing_framework/tools/results.py

@@ -13,12 +13,11 @@
 # limitations under the License.
 
 
+import ast
 import json
 from collections import UserDict
 from datetime import datetime, timezone
-from os import remove
 from typing import Any
-import pytest
 
 
 class Results(UserDict):
@@ -38,16 +37,34 @@ class Results(UserDict):
         self._remove_timestamps_if_necessary()
         self._remove_all_empty_strings_from_dicts_if_necessary()
 
+    @staticmethod
+    def _coerce_string_if_json_or_python_literal(value: str) -> Any:
+        """If ``value`` looks like JSON or a Python ``list``/``dict`` literal, parse it.
+
+        Test JSON and Splunk sometimes embed ``list``/``dict`` as strings; JSON only
+        accepts double quotes, so single-quoted Python reprs need ``literal_eval``.
+        """
+        candidate = value.strip()
+        if not (candidate.startswith("[") or candidate.startswith("{")):
+            return value
+        try:
+            return json.loads(candidate)
+        except Exception:
+            try:
+                parsed = ast.literal_eval(candidate)
+            except (ValueError, SyntaxError):
+                return value
+            if isinstance(parsed, (list, dict)):
+                return parsed
+            return value
+
     @staticmethod
     def _cast_types(data: Any):
         """Casts everything into dicts / lists / strings.
         This is necessary because Splunk, IP and CLI often return different types of data.
         """
-        if isinstance(data, str) and (data.startswith("[") or data.startswith("{")):
-            try:
-                data = json.loads(data)
-            except Exception:
-                pass  # If it fails, it's not a json, so it's just a normal string
+        if isinstance(data, str):
+            data = Results._coerce_string_if_json_or_python_literal(data)
 
         if isinstance(data, list):
             data = [Results._cast_types(item) for item in data]
@@ -55,7 +72,7 @@ class Results(UserDict):
             for key, value in data.items():
                 data[key] = Results._cast_types(value)
                 if key == "_time":
-                    data[key] = Results._convert_timestamp(value)
+                    data[key] = Results._convert_timestamp(data[key])
         if isinstance(data, (int, float)):
             data = str(data)
 
@@ -66,7 +83,7 @@ class Results(UserDict):
         """We need to convert timestamp, as there are different formats returned by cloud / CLI / splunk"""
         try:
             ts = datetime.fromisoformat(timestamp)
-        except ValueError:
+        except (ValueError, TypeError):
             ts = datetime.fromtimestamp(float(timestamp), timezone.utc)
 
         return ts.isoformat()
@@ -158,14 +175,12 @@ class Results(UserDict):
                                         actual_data[key][i], expected_data[key][i]
                                     )
 
-        return actual_data
-
     def remove_fields_not_in_expected(self, expected):
         """Remove fields from this Results object that are not present in the expected Results object.
         This is the public method to be called when comparing results.
         """
         if self._ignore_additional_fields_in_actual and isinstance(expected, Results):
-            self.data = self._remove_extra_fields(self.data, expected.data)
+            self._remove_extra_fields(self.data, expected.data)
         return self
 
     def __lt__(self, other):