spl2-testing-framework 1.5.0__tar.gz → 1.6.0__tar.gz

{spl2_testing_framework-1.5.0 → spl2_testing_framework-1.6.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: spl2-testing-framework
-Version: 1.5.0
+Version: 1.6.0
 Summary: SPL2 Testing Framework
 Home-page: https://github.com/splunk/spl2-testing-framework
 Keywords: SPL2 testing framework
@@ -25,6 +25,7 @@ Requires-Dist: pytest (==7.4.4)
 Requires-Dist: pytest-subtests (==0.13.1)
 Requires-Dist: pytest-xdist (==3.5.0)
 Requires-Dist: requests (>=2.31.0,<3.0.0)
+Requires-Dist: splunk-sdk (>=2.1.1,<3.0.0)
 Requires-Dist: tabulate (==0.9.0)
 Requires-Dist: urllib3 (>=1.26.19,<2.0.0)
 Project-URL: Repository, https://github.com/splunk/spl2-testing-framework
@@ -134,8 +135,9 @@ Additionally, the following options are supported:
 * `--ignore_additional_fields_in_actual` - to ignore fields present in actual results but not in expected results (useful when actual results contain extra fields that should not affect comparison)
 * `--create_comparison_sheet` - to create a comparison sheet in `comparison_box_test` folder using actual and expected outputs (works only when running box tests)
 * `--cli_bench` - when running tests with CLI, this option enables benchmarking mode by adding `-b` and `-n` flags to the spl2-processor-cli command. The value specifies the number of events to test (e.g., `--cli_bench=1000` will add `-b -n 1000` to the CLI command). **In bench mode, all tests always succeed and CLI output is printed to logs** - this is useful for performance testing and benchmarking. This is only applicable when using `--type cli`.
+* `--check_splunk_results` - option to validate box test output by sending events to a Splunk instance (using HEC) and querying them back, with support for **CIM** (Common Information Model) and **TA** (Technology Add-on) checks.
 
-Note: The `pytest.ini.sample` file allows you to define command parameters. Just update the configurations, rename the
+* Note: The `pytest.ini.sample` file allows you to define command parameters. Just update the configurations, rename the
 file by removing the `.sample` extension, and execute the command.
 
 ### Run tests in IDE [PyCharm]
@@ -206,9 +208,80 @@ Content of this file will also be printed to stdout.
 This checks can be applied only to box tests, as assertions which are used in unit tests may impact spl2 pipeline
 performance. 
 
+### Check splunk results
+
+When set (`--check_splunk_results=<option>`), box tests (after running the SPL2 pipeline) send pipeline output to the configured Splunk instance and validate results there instead of comparing only in-memory expected vs actual.
+
+| Value | Behavior |
+|-------|----------|
+| **CIM** | For each output event, push to Splunk, query by host, then validate that the returned event has the expected CIM fields (from `expected_cim_fields.cim_fields` in the box test). Uses `validate_compatibility()`. Skips the test if the box test has no `cim_fields`. |
+| **TA** | For each output event, push to Splunk, query by host, then assert that the returned event equals the expected output event for that index. |
+
+For **TA** only, you can name top-level fields to drop on **both** the queried event and each `expected_destination_result` row by adding **`ignore_fields_in_splunk_check`**: an object mapping **field name → short reason**. Use the same key under `test` in each entry in `*.test.json` (same file as `expected_destination_result`):
+
+```json
+"test": {
+  "source": "...",
+  "expected_destination_result": [ ... ],
+  "ignore_fields_in_splunk_check": {
+    "_raw": "Raw event differs after Splunk indexing",
+    "_time": "Timestamp may vary due to ingestion latency"
+  }
+}
+```
+
+
+If the key is omitted, no fields are ignored. Used only when `--check_splunk_results=TA` (ignored for other modes and for the default 
+in-memory box test assert). If the file or key is missing, no fields are ignored.
+
+#### Configuration:
+
+`SPLUNK_INSTANCE` in `spl2_test_config.json`
+
+New top-level key for the Splunk instance used by `--check_splunk_results`:
+
+```json
+{
+  "SPLUNK_INSTANCE": {
+    "ip": "<hostname-or-ip>",
+    "port": 8088,
+    "api_port": 8089,
+    "username": "<splunk-user>",
+    "password": "<splunk-password>",
+    "index": "<index-name>",
+    "hec_token": "<hec-token>"
+  }
+}
+```
+
+- **ip** – Splunk server host (e.g. `localhost` or hostname).  
+- **port** – HEC port (e.g. 8088).  
+- **api_port** – Management/API port (e.g. 8089) for Search API and index creation.  
+- **username** / **password** – Used for Search API and index creation (HTTP Basic).  
+- **index** – Index where HEC events are sent and then searched.  
+- **hec_token** – Token for HEC (`Authorization: Splunk <token>`).
+
+Do not commit real credentials. Prefer environment variables (below) or a local config that is not in version control.
+
+#### Environment variables
+
+Any `SPLUNK_INSTANCE` field can be set via the environment; config file values override these only when non-empty.
+
+| Variable | Maps to |
+|----------|--------|
+| `SPL2_TF_SPLUNK_INSTANCE_IP` | `ip` |
+| `SPL2_TF_SPLUNK_INSTANCE_PORT` | `port` |
+| `SPL2_TF_SPLUNK_INSTANCE_API_PORT` | `api_port` |
+| `SPL2_TF_SPLUNK_INSTANCE_USERNAME` | `username` |
+| `SPL2_TF_SPLUNK_INSTANCE_PASSWORD` | `password` |
+| `SPL2_TF_SPLUNK_INSTANCE_INDEX` | `index` |
+| `SPL2_TF_SPLUNK_INSTANCE_HEC_TOKEN` | `hec_token` |
+
+
 ## Format all files
-To format all files, run:
+To format all files run:
 
 ```bash
 black spl2_testing_framework tests
 ```
+

{spl2_testing_framework-1.5.0 → spl2_testing_framework-1.6.0}/README.md

@@ -102,8 +102,9 @@ Additionally, the following options are supported:
 * `--ignore_additional_fields_in_actual` - to ignore fields present in actual results but not in expected results (useful when actual results contain extra fields that should not affect comparison)
 * `--create_comparison_sheet` - to create a comparison sheet in `comparison_box_test` folder using actual and expected outputs (works only when running box tests)
 * `--cli_bench` - when running tests with CLI, this option enables benchmarking mode by adding `-b` and `-n` flags to the spl2-processor-cli command. The value specifies the number of events to test (e.g., `--cli_bench=1000` will add `-b -n 1000` to the CLI command). **In bench mode, all tests always succeed and CLI output is printed to logs** - this is useful for performance testing and benchmarking. This is only applicable when using `--type cli`.
+* `--check_splunk_results` - option to validate box test output by sending events to a Splunk instance (using HEC) and querying them back, with support for **CIM** (Common Information Model) and **TA** (Technology Add-on) checks.
 
-Note: The `pytest.ini.sample` file allows you to define command parameters. Just update the configurations, rename the
+* Note: The `pytest.ini.sample` file allows you to define command parameters. Just update the configurations, rename the
 file by removing the `.sample` extension, and execute the command.
 
 ### Run tests in IDE [PyCharm]
@@ -174,9 +175,79 @@ Content of this file will also be printed to stdout.
 This checks can be applied only to box tests, as assertions which are used in unit tests may impact spl2 pipeline
 performance. 
 
+### Check splunk results
+
+When set (`--check_splunk_results=<option>`), box tests (after running the SPL2 pipeline) send pipeline output to the configured Splunk instance and validate results there instead of comparing only in-memory expected vs actual.
+
+| Value | Behavior |
+|-------|----------|
+| **CIM** | For each output event, push to Splunk, query by host, then validate that the returned event has the expected CIM fields (from `expected_cim_fields.cim_fields` in the box test). Uses `validate_compatibility()`. Skips the test if the box test has no `cim_fields`. |
+| **TA** | For each output event, push to Splunk, query by host, then assert that the returned event equals the expected output event for that index. |
+
+For **TA** only, you can name top-level fields to drop on **both** the queried event and each `expected_destination_result` row by adding **`ignore_fields_in_splunk_check`**: an object mapping **field name → short reason**. Use the same key under `test` in each entry in `*.test.json` (same file as `expected_destination_result`):
+
+```json
+"test": {
+  "source": "...",
+  "expected_destination_result": [ ... ],
+  "ignore_fields_in_splunk_check": {
+    "_raw": "Raw event differs after Splunk indexing",
+    "_time": "Timestamp may vary due to ingestion latency"
+  }
+}
+```
+
+
+If the key is omitted, no fields are ignored. Used only when `--check_splunk_results=TA` (ignored for other modes and for the default 
+in-memory box test assert). If the file or key is missing, no fields are ignored.
+
+#### Configuration:
+
+`SPLUNK_INSTANCE` in `spl2_test_config.json`
+
+New top-level key for the Splunk instance used by `--check_splunk_results`:
+
+```json
+{
+  "SPLUNK_INSTANCE": {
+    "ip": "<hostname-or-ip>",
+    "port": 8088,
+    "api_port": 8089,
+    "username": "<splunk-user>",
+    "password": "<splunk-password>",
+    "index": "<index-name>",
+    "hec_token": "<hec-token>"
+  }
+}
+```
+
+- **ip** – Splunk server host (e.g. `localhost` or hostname).  
+- **port** – HEC port (e.g. 8088).  
+- **api_port** – Management/API port (e.g. 8089) for Search API and index creation.  
+- **username** / **password** – Used for Search API and index creation (HTTP Basic).  
+- **index** – Index where HEC events are sent and then searched.  
+- **hec_token** – Token for HEC (`Authorization: Splunk <token>`).
+
+Do not commit real credentials. Prefer environment variables (below) or a local config that is not in version control.
+
+#### Environment variables
+
+Any `SPLUNK_INSTANCE` field can be set via the environment; config file values override these only when non-empty.
+
+| Variable | Maps to |
+|----------|--------|
+| `SPL2_TF_SPLUNK_INSTANCE_IP` | `ip` |
+| `SPL2_TF_SPLUNK_INSTANCE_PORT` | `port` |
+| `SPL2_TF_SPLUNK_INSTANCE_API_PORT` | `api_port` |
+| `SPL2_TF_SPLUNK_INSTANCE_USERNAME` | `username` |
+| `SPL2_TF_SPLUNK_INSTANCE_PASSWORD` | `password` |
+| `SPL2_TF_SPLUNK_INSTANCE_INDEX` | `index` |
+| `SPL2_TF_SPLUNK_INSTANCE_HEC_TOKEN` | `hec_token` |
+
+
 ## Format all files
-To format all files, run:
+To format all files run:
 
 ```bash
 black spl2_testing_framework tests
-```
+```

{spl2_testing_framework-1.5.0 → spl2_testing_framework-1.6.0}/pyproject.toml

@@ -15,7 +15,7 @@
 
 [tool.poetry]
 name = "spl2-testing-framework"
-version = "1.5.0"
+version = "1.6.0"
 description = "SPL2 Testing Framework"
 authors = ["mgazda-splunk <mgazda@splunk.com>"]
 readme = "README.md"
@@ -52,6 +52,7 @@ numpy = [
     { version = ">=1.26,<2.0", python = ">=3.13" }
 ]
 openpyxl = "^3.1.2"
+splunk-sdk = "^2.1.1"
 
 [tool.poetry.group.dev.dependencies]
 pre-commit = "*"

{spl2_testing_framework-1.5.0 → spl2_testing_framework-1.6.0}/spl2_testing_framework/conftest.py

@@ -15,10 +15,13 @@
 
 import json
 import os
+from collections import defaultdict
 
+import pandas as pd
 import pytest
 
 from .tools.results import Results
+from .tools.splunk_integration.splunk_handler import SplunkHandler
 
 pytest.register_assert_rewrite("spl2_testing_framework")
 
@@ -48,6 +51,7 @@ SINGLE_SPL2_FILE = "single_spl2_file"
 IGNORE_EMPTY_STRINGS = "ignore_empty_strings"
 IGNORE_ADDITIONAL_FIELDS_IN_ACTUAL = "ignore_additional_fields_in_actual"
 CREATE_COMPARISON_SHEET = "create_comparison_sheet"
+CHECK_SPLUNK_RESULTS = "check_splunk_results"
 PERFORMANCE_CHECK = "performance_check"
 TEST_TYPE = "type"
 TEST_DIR = "test_dir"
@@ -75,6 +79,9 @@ def pytest_addoption(parser):
     parser.addoption("--template_file", action="store", default=None)
     parser.addoption("--sample_file", action="store", default=None)
     parser.addoption("--sample_delimiter", action="store", default="\n")
+    parser.addoption(
+        f"--{CHECK_SPLUNK_RESULTS}", action="store", type=str, default=None
+    )
     parser.addoption(f"--{CREATE_COMPARISON_SHEET}", action="store_true", default=False)
     parser.addoption(f"--{IGNORE_EMPTY_STRINGS}", action="store_true")
     parser.addoption(f"--{IGNORE_ADDITIONAL_FIELDS_IN_ACTUAL}", action="store_true")
@@ -139,6 +146,8 @@ def spl2_test_runner(
     basic_authentication,
     bearer_token_authentication,
     run_configurations,
+    splunk_handler,
+    stats,
 ):
     """This fixture is used to create a test runner object based on the test type specified by the --type option
     3 types of tests are supported: 'splunk', 'cloud', 'cli'
@@ -160,7 +169,7 @@ def spl2_test_runner(
             )
         )
 
-    x = SPL2TestRunner(search_client, run_configurations)
+    x = SPL2TestRunner(search_client, run_configurations, splunk_handler, stats)
     return x
 
 
@@ -172,11 +181,35 @@ def run_configurations(request):
             f"--{CREATE_COMPARISON_SHEET}"
         ),
         CLI_BENCH: request.config.getoption(f"--{CLI_BENCH}"),
+        CHECK_SPLUNK_RESULTS: request.config.getoption(f"--{CHECK_SPLUNK_RESULTS}"),
     }
 
 
 @pytest.fixture(scope="session")
-def config():
+def splunk_instance_config():
+    splunk_instance_ip = os.environ.get("SPL2_TF_SPLUNK_INSTANCE_IP", None)
+    splunk_instance_port = os.environ.get("SPL2_TF_SPLUNK_INSTANCE_PORT", None)
+    splunk_instance_api_port = os.environ.get("SPL2_TF_SPLUNK_INSTANCE_API_PORT", None)
+    splunk_instance_username = os.environ.get("SPL2_TF_SPLUNK_INSTANCE_USERNAME", None)
+    splunk_instance_password = os.environ.get("SPL2_TF_SPLUNK_INSTANCE_PASSWORD", None)
+    splunk_instance_index = os.environ.get("SPL2_TF_SPLUNK_INSTANCE_INDEX", None)
+    splunk_instance_hec_token = os.environ.get(
+        "SPL2_TF_SPLUNK_INSTANCE_HEC_TOKEN", None
+    )
+
+    return {
+        "ip": splunk_instance_ip,
+        "port": splunk_instance_port,
+        "api_port": splunk_instance_api_port,
+        "username": splunk_instance_username,
+        "password": splunk_instance_password,
+        "index": splunk_instance_index,
+        "hec_token": splunk_instance_hec_token,
+    }
+
+
+@pytest.fixture(scope="session")
+def config(splunk_instance_config):
     """This fixture is used to read the configuration from the environment variables or from the spl2_test_config.json
     Configuration set using environment variables will be overwritten by settings from spl2_test_config.json file,
     however empty values in spl2_test_config.json will be ignored.
@@ -219,6 +252,12 @@ def config():
     if conf_file.get("cloud_instance", None):
         cloud_instance = conf_file["cloud_instance"]
 
+    for k, v in splunk_instance_config.items():
+        if v is None:
+            splunk_instance_config[k] = conf_file.get("SPLUNK_INSTANCE", {}).get(
+                k, None
+            )
+
     return {
         "host": host,
         "port": port,
@@ -227,6 +266,7 @@ def config():
         "bearer_token": bearer_token,
         "tenant": tenant,
         "cloud_instance": cloud_instance,
+        "SPLUNK_INSTANCE": splunk_instance_config,
     }
 
 
@@ -251,6 +291,49 @@ def bearer_token_authentication(config):
     return HTTPBearerTokenAuth(token)
 
 
+@pytest.fixture(scope="session")
+def splunk_token_authentication(config):
+    class SplunkTokenAuth(AuthBase):
+        """Attaches HTTP Basic Authentication to the given Request object."""
+
+        def __init__(self, token):
+            self.token = token
+
+        def __call__(self, r):
+            r.headers["Authorization"] = f"Splunk {self.token}"
+            return r
+
+    token = config["SPLUNK_INSTANCE"].get("hec_token", "")
+    splunk_auth = SplunkTokenAuth(token)
+    return splunk_auth
+
+
+@pytest.fixture(scope="session")
+def splunk_handler(config, splunk_token_authentication):
+    config["SPLUNK_INSTANCE"][
+        "url"
+    ] = f"https://{config['SPLUNK_INSTANCE']['ip']}:{config['SPLUNK_INSTANCE']['port']}/services/collector/raw"
+    config["SPLUNK_INSTANCE"][
+        "index_url"
+    ] = f"https://{config['SPLUNK_INSTANCE']['ip']}:{config['SPLUNK_INSTANCE']['api_port']}/services/data/indexes"
+
+    sh = SplunkHandler(config, splunk_token_authentication)
+    return sh
+
+
+@pytest.fixture(scope="session")
+def stats():
+    s = defaultdict(lambda: defaultdict(list))
+    yield s
+
+    for template, v in s.items():
+        x = pd.DataFrame(v)
+        _LOGGER.info(" *** *** ***")
+        _LOGGER.info("Collected statistics: ")
+        _LOGGER.info("Template name: %s ", template)
+        _LOGGER.info(x.describe())
+
+
 def _set_results_options(metafunc):
     ignore_empty_strings = metafunc.config.getoption(IGNORE_EMPTY_STRINGS)
     Results._ignore_empty_strings = ignore_empty_strings

spl2_testing_framework-1.6.0/spl2_testing_framework/spl2_test_config.json

@@ -0,0 +1,18 @@
+{
+  "host": "",
+  "cloud_instance": "",
+  "tenant": "",
+  "bearer_token": "",
+  "port": "",
+  "user": "",
+  "password": "",
+  "SPLUNK_INSTANCE": {
+    "ip": "",
+    "port": "",
+    "api_port": "",
+    "username": "",
+    "password": "",
+    "index": "",
+    "hec_token": ""
+  }
+}

{spl2_testing_framework-1.5.0 → spl2_testing_framework-1.6.0}/spl2_testing_framework/tools/performance.py

@@ -38,7 +38,7 @@ class PerformanceCheck:
     PIPELINE_PATTERN = re.compile(
         r"(?P<fullmatch>"
         r"^\s*?\$pipeline\s*?=\s*?\|?\s*?from\s*?\$source\s*?"  # first line of pipeline
-        r"(.|\n)*?"  # anything in between
+        r"(.|\n)*"  # greedy: match everything up to the LAST into $destination
         r"(\|\s*?into\s*?\$destination)"  # last line of pipeline definition
         r")",
         flags=re.M | re.VERBOSE,
@@ -46,8 +46,8 @@ class PerformanceCheck:
 
     FUNCTION_PATTERN = re.compile(
         r"^(?P<fullmatch>"
-        r"(export\s+)?function\s+(?P<f_name>\S+)\((?P<source>\S+):"  # function header
-        r".*{\n\s+return\s+(\|\s+)?from\s+(?P=source)\n*"  # return statement
+        r"(export\s+)?function\s+(?P<f_name>\S+)\((?P<source>\$\w+)"  # function header (source may lack type)
+        r"[^{]*{\n\s+return\s+(\|\s+)?(?:from|FROM)\s+(?P=source)\n*"  # return statement
         r"(?P<statements>(.|\n)*?)"  # anything before closing bracket
         r"(?P<closing_bracket>^\s*})"  # closing bracket [ASSUMED IT'S IN THE NEW LINE]
         r")",
@@ -56,6 +56,20 @@ class PerformanceCheck:
 
     STATEMENT_PATTERN = re.compile(r"^(?P<lx>\s*?\|.*)$", re.M)
 
+    SCALAR_FUNCTION_PATTERN = re.compile(
+        r"^(?P<fullmatch>"
+        r"(?:export\s+)?function\s+(?P<f_name>\S+)\("
+        r"[^{]*{\n"
+        r"(?P<statements>[\s\S]*?)"
+        r"(?P<closing_bracket>^\s*})"
+        r")",
+        flags=re.M | re.VERBOSE,
+    )
+
+    SCALAR_STATEMENT_PATTERN = re.compile(
+        r"^(?P<lx>\s*(?:\$\w+\s*=(?!=)|return\s).+)$", re.M
+    )
+
     def __init__(self, test: Test):
         self._test = test
         self._performance_check_type = test.meta.get("performance", None)
@@ -135,13 +149,26 @@ class PerformanceCheck:
             output = copy(self.code_module)
 
             for timestamp_name, value, relative in calculated_diffs:
+                marker = f">>> {(value * 1000):.6f}ms [{relative:.2%}] <<< "
                 output = re.sub(
                     rf"\| eval {timestamp_name}=time\(\)",
-                    f">>> {(value * 1000):.6f}ms [{relative:.2%}] <<< ",
+                    marker,
+                    output,
+                )
+                output = re.sub(
+                    rf"^\${timestamp_name}\s*=\s*time\(\)\s*\n",
+                    marker,
                     output,
+                    flags=re.M,
                 )
 
             output = re.sub(r"\| eval _timestamp_\w+_\d{3}=time\(\)", r"", output)
+            output = re.sub(
+                r"^\$_timestamp_\w+_\d{3}\s*=\s*time\(\)\s*\n?",
+                r"",
+                output,
+                flags=re.M,
+            )
 
             _LOGGER.info(output)
 
@@ -154,7 +181,10 @@ class PerformanceCheck:
     def _save_output(self, output: str, event_number: int) -> None:
         """Save output to text file"""
         os.makedirs(self.OUTPUT_DIR, exist_ok=True)
-        filename = f"{self.OUTPUT_DIR}/performance_{self._test.name}_{event_number}_{datetime.now()}.txt"
+        safe_name = (
+            self._test.name.replace(os.sep, "_").replace("/", "_").replace("\\", "_")
+        )
+        filename = f"{self.OUTPUT_DIR}/performance_{safe_name}_{event_number}_{datetime.now()}.txt"
 
         with open(filename, "w") as f:
             f.write(output)
@@ -181,13 +211,18 @@ class PerformanceCheck:
             flags=re.M,
         )
 
-        code_module = re.sub(
-            r"(\|\s*?into\s*?\$destination)",
-            r"| eval _timestamp_perf_stop = time()"
-            r"| eval _timestamp_perf_exec = _timestamp_perf_stop - _timestamp_perf_start \1",
-            code_module,
-            flags=re.M,
-        )
+        pattern = re.compile(r"(\|\s*?into\s*?\$destination)", re.M)
+        matches = list(pattern.finditer(code_module))
+        if matches:
+            last = matches[-1]
+            replacement = (
+                "| eval _timestamp_perf_stop = time()"
+                "| eval _timestamp_perf_exec = _timestamp_perf_stop - _timestamp_perf_start "
+                + last.group(1)
+            )
+            code_module = (
+                code_module[: last.start()] + replacement + code_module[last.end() :]
+            )
 
         return code_module
 
@@ -196,6 +231,7 @@ class PerformanceCheck:
 
         functions = self.__parse_functions(code_module)
         pipeline = self.__parse_pipeline(code_module)
+        scalar_functions = self.__parse_scalar_functions(code_module)
 
         functions.append(pipeline)
 
@@ -203,14 +239,37 @@ class PerformanceCheck:
             new_function = self.__add_timestamps_to_function(f)
             code_module = code_module.replace(f["fullmatch"], new_function)
 
+        for f in scalar_functions:
+            new_function = self.__add_timestamps_to_scalar_function(f)
+            code_module = code_module.replace(f["fullmatch"], new_function)
+
+        code_module = self._preserve_timestamps_through_fields(code_module)
+
         return code_module
 
+    @staticmethod
+    def _preserve_timestamps_through_fields(code_module: str) -> str:
+        """Replace '| fields' keep-mode with a no-op so _timestamp_perf* fields are not stripped."""
+        return re.sub(
+            r"\|\s*fields\s+(?!-)(.+)",
+            r"| eval _perf_fields_preserved=1",
+            code_module,
+        )
+
     def __parse_functions(self, code_module: str) -> List[Dict]:
         """Parse all functions from code module. The finding contains groups necessary for parsing it
         and adding timestamps"""
         functions = [m.groupdict() for m in self.FUNCTION_PATTERN.finditer(code_module)]
         return functions
 
+    def __parse_scalar_functions(self, code_module: str) -> List[Dict]:
+        """Parse scalar functions (no FROM $source) that use $variable = expr assignments."""
+        dataset_fn_names = {f["f_name"] for f in self.__parse_functions(code_module)}
+        all_fns = [
+            m.groupdict() for m in self.SCALAR_FUNCTION_PATTERN.finditer(code_module)
+        ]
+        return [f for f in all_fns if f["f_name"] not in dataset_fn_names]
+
     def __parse_pipeline(self, code_module: str) -> Dict:
         """Parse pipeline definition from code module. The finding contains groups necessary for parsing it
         and adding timestamps"""
@@ -249,3 +308,25 @@ class PerformanceCheck:
             )
 
         return new_function
+
+    def __add_timestamps_to_scalar_function(self, f: Dict) -> str:
+        """Add timestamps to scalar functions that use $variable = expr assignments.
+        Injects $-prefixed variable assignments instead of | eval commands."""
+        new_function = self.SCALAR_STATEMENT_PATTERN.sub(
+            rf"$_time_{f['f_name']}_placeholder = time()\n\g<lx>",
+            f["fullmatch"],
+        )
+
+        func_timestamps = re.findall(
+            rf"\$_time_{f['f_name']}_placeholder", new_function
+        )
+
+        for n, timestamp in enumerate(func_timestamps):
+            new_function = re.sub(
+                re.escape(timestamp),
+                rf"$_timestamp_perf{f['f_name']}_{n:03d}",
+                new_function,
+                count=1,
+            )
+
+        return new_function

{spl2_testing_framework-1.5.0 → spl2_testing_framework-1.6.0}/spl2_testing_framework/tools/search_clients/cli_search_client.py

@@ -75,8 +75,13 @@ class CLISearchClient(SearchClient):
         # Normal mode - raise errors as before
         if result.stderr:
             raise RuntimeError(result.stderr)
-
-        job.result = json.loads(result.stdout)
+        try:
+            job.result = json.loads(result.stdout)
+        except json.decoder.JSONDecodeError:
+            _LOGGER.error(
+                "Can't decode output for %s. Details: %s ", job.test_name, result
+            )
+            raise
         return
 
     def _print_bench_results(self, job, result):

{spl2_testing_framework-1.5.0 → spl2_testing_framework-1.6.0}/spl2_testing_framework/tools/spl2test_runner.py

@@ -14,6 +14,7 @@
 
 
 import logging
+import pytest
 
 from tabulate import tabulate
 
@@ -23,7 +24,13 @@ from spl2_testing_framework.tools.search_clients.cli_search_client import (
     CLISearchClient,
 )
 from spl2_testing_framework.tools.test_types import UnitTest, BoxTest, SingleSPL2
-from spl2_testing_framework.tools.utils import _make_functions_visible_for_testing
+from spl2_testing_framework.tools.utils import (
+    _make_functions_visible_for_testing,
+    validate_compatibility,
+    calculate_reduction_rate,
+    parse_raw_from_input,
+    parse_raw_from_reduced_events,
+)
 
 _LOGGER = logging.getLogger(__name__)
 
@@ -31,10 +38,15 @@ _LOGGER = logging.getLogger(__name__)
 class SPL2TestRunner:
     """Main class responsible for running tests"""
 
-    def __init__(self, search_client, run_configurations):
+    def __init__(self, search_client, run_configurations, splunk_handler, stats):
         self._search_client = search_client
         self.create_comparison_sheet = run_configurations.get("create_comparison_sheet")
+        self.check_splunk_results = run_configurations.get("check_splunk_results")
         self.comp_obj = ComparisonSheet()
+        self.reduction_stats = stats
+        if self.check_splunk_results:
+            splunk_handler.setup()
+            self.splunk_handler = splunk_handler
 
     def run_single_spl2_file(self, single_spl2_file: SingleSPL2) -> None:
         """Run a single spl2 file and print the output"""
@@ -163,30 +175,81 @@ class SPL2TestRunner:
 
         perf.check_performance(output_result)
 
-        if box_test.output:
-            # Remove fields from actual results that are not in expected results if configured
-            for i, actual_result in enumerate(output_result):
-                if i < len(box_test.output):
-                    actual_result.remove_fields_not_in_expected(box_test.output[i])
-
-            if self.create_comparison_sheet:
-                self.comp_obj.create_comparison_sheet(output_result, box_test)
-            assert sorted(output_result) == sorted(box_test.output)
-            _LOGGER.info(f"Output check passed: {box_test.name}")
-            _LOGGER.debug("Received: \n%s", str(output_result))
-            _LOGGER.debug("Expected: \n%s", str(box_test.output))
+        reduction_rate = calculate_reduction_rate(
+            parse_raw_from_input(box_test.input),
+            parse_raw_from_reduced_events(output_result),
+        )
+        self.reduction_stats[box_test.name]["reduction_rate"].extend(reduction_rate)
+
+        _LOGGER.info("Reduction rate for this event: %s", reduction_rate)
+
+        if self.check_splunk_results:
+            self._check_splunk_results(box_test, job)
         else:
-            _LOGGER.info(f"Output check skipped (expected empty): {box_test.name}")
-
-        if box_test.metrics:
-            # Remove fields from actual metrics that are not in expected metrics if configured
-            for i, actual_metric in enumerate(metrics_result):
-                if i < len(box_test.metrics):
-                    actual_metric.remove_fields_not_in_expected(box_test.metrics[i])
-
-            assert sorted(metrics_result) == sorted(box_test.metrics)
-            _LOGGER.info(f"Metric check passed: {box_test.name}")
-            _LOGGER.debug("Received: \n%s", str(metrics_result))
-            _LOGGER.debug("Expected: \n%s", str(box_test.metrics))
+            if box_test.output:
+                # Remove fields from actual results that are not in expected results if configured
+                for i, actual_result in enumerate(output_result):
+                    if i < len(box_test.output):
+                        actual_result.remove_fields_not_in_expected(box_test.output[i])
+
+                if self.create_comparison_sheet:
+                    self.comp_obj.create_comparison_sheet(output_result, box_test)
+                assert sorted(output_result) == sorted(box_test.output)
+                _LOGGER.info(f"Output check passed: {box_test.name}")
+                _LOGGER.debug("Received: \n%s", str(output_result))
+                _LOGGER.debug("Expected: \n%s", str(box_test.output))
+            else:
+                _LOGGER.info(f"Output check skipped (expected empty): {box_test.name}")
+                _LOGGER.debug("Actual results received: \n%s", str(output_result))
+
+            if box_test.metrics:
+                # Remove fields from actual metrics that are not in expected metrics if configured
+                for i, actual_metric in enumerate(metrics_result):
+                    if i < len(box_test.metrics):
+                        actual_metric.remove_fields_not_in_expected(box_test.metrics[i])
+
+                assert sorted(metrics_result) == sorted(box_test.metrics)
+                _LOGGER.info(f"Metric check passed: {box_test.name}")
+                _LOGGER.debug("Received: \n%s", str(metrics_result))
+                _LOGGER.debug("Expected: \n%s", str(box_test.metrics))
+            else:
+                _LOGGER.info(f"Metric check skipped (expected empty): {box_test.name}")
+
+    @staticmethod
+    def _row_as_dict(row):
+        return row.data if hasattr(row, "data") else dict(row)
+
+    @staticmethod
+    def _strip_ignored_fields_for_splunk_compare(
+        row: dict, ignore_keys: frozenset
+    ) -> dict:
+        if not ignore_keys:
+            return row
+        return {k: v for k, v in row.items() if k not in ignore_keys}
+
+    def _check_splunk_results(self, box_test, job):
+        output_result = job.result.get("destination", [])
+        if self.check_splunk_results == "CIM":
+            if not box_test.cim_fields:
+                pytest.skip("CIM output not available for this test")
+
+            for event in output_result:
+                processed_event = self.splunk_handler.process_event(event)
+                validate_compatibility(box_test.cim_fields, processed_event)
+        elif self.check_splunk_results == "TA":
+            ignore_spec = box_test.ignore_fields_in_splunk_check
+            ignore_keys = frozenset(ignore_spec.keys())
+            for event, expected in zip(output_result, box_test.output):
+                processed_event = self.splunk_handler.process_event(event)
+                actual = self._strip_ignored_fields_for_splunk_compare(
+                    processed_event, ignore_keys
+                )
+                expected_cmp = self._strip_ignored_fields_for_splunk_compare(
+                    self._row_as_dict(expected), ignore_keys
+                )
+                assert actual == expected_cmp, (
+                    f"TA Splunk compare mismatch for {box_test.name}; "
+                    f"ignored_fields={ignore_spec!r}"
+                )
         else:
-            _LOGGER.info(f"Metric check skipped (expected empty): {box_test.name}")
+            return

spl2_testing_framework-1.6.0/spl2_testing_framework/tools/splunk_integration/__init__.py

@@ -0,0 +1,13 @@
+# Copyright 2026 Splunk Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.

spl2_testing_framework-1.6.0/spl2_testing_framework/tools/splunk_integration/splunk_handler.py

@@ -0,0 +1,119 @@
+# Copyright 2026 Splunk Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import logging
+import uuid
+
+import requests
+import splunklib.client as client
+import splunklib.results as results
+
+import time
+
+from requests.auth import HTTPBasicAuth
+
+_LOGGER = logging.getLogger(__name__)
+
+
+class SplunkHandler:
+    def __init__(self, config, auth):
+        self._config = config
+        self.splunk_config = config.get("SPLUNK_INSTANCE", {})
+        self._auth = auth
+
+    def setup(self):
+        self._create_index()
+
+    def process_event(self, event):
+        host = f"test_{uuid.uuid4()}"
+
+        self.push_to_hec(event, host)
+        res = self.get_events_by_host_and_index(host)
+        if not res:
+            raise Exception("No event found.")
+
+        if len(res) > 1:
+            _LOGGER.error("Found more than one event, the result may not be correct")
+        return res[0]
+
+    def push_to_hec(self, data, host):
+        sourcetype = data.get("sourcetype", "UNKNOWN")
+
+        if sourcetype == "UNKNOWN":
+            _LOGGER.warning("Source type is UNKNOWN. The result may be not correct")
+
+        source = data.get("source", None)
+
+        params = {
+            "sourcetype": sourcetype,
+            "index": self.splunk_config["index"],
+            "host": host,
+        }
+
+        if source is not None:
+            params["source"] = source
+
+        event_data = data["_raw"].encode("utf-8")
+
+        response = requests.post(
+            url=self.splunk_config["url"],
+            data=event_data,
+            params=params,
+            auth=self._auth,
+            verify=False,
+        )
+
+        response.raise_for_status()
+
+    def get_events_by_host_and_index(self, host):
+        SPL_TEMPLATE = "| search index={index} host={host} | fields *"
+        # Connect to Splunk
+        service = client.connect(
+            host=self.splunk_config["ip"],
+            port=self.splunk_config["api_port"],
+            username=self.splunk_config["username"],
+            password=self.splunk_config["password"],
+            app="search",
+        )
+        SPL = SPL_TEMPLATE.format(index=self.splunk_config["index"], host=host)
+
+        results_all = []
+        retry = 0
+        while not results_all and retry < 20:
+            kwargs = {"adhoc_search_level": "verbose"}
+            job = service.jobs.create(SPL, **kwargs)
+            while not job.is_done():
+                time.sleep(0.25)
+            results_splunk = job.results(output_mode="json")
+            rr = results.JSONResultsReader(results_splunk)
+            for result in rr:
+                if isinstance(result, results.Message):
+                    _LOGGER.info("%s: %s", result.type, result.message)
+                elif isinstance(result, dict):
+                    # Normal events are returned as dicts
+                    results_all.append(result)
+            retry = retry + 1
+            time.sleep(0.25)
+        return results_all
+
+    def _create_index(self):
+        response = requests.post(
+            url=self.splunk_config["index_url"],
+            data={"name": self.splunk_config["index"], "output_mode": "json"},
+            auth=HTTPBasicAuth(
+                self.splunk_config["username"], self.splunk_config["password"]
+            ),
+            verify=False,
+        )
+        return response.json()

{spl2_testing_framework-1.5.0 → spl2_testing_framework-1.6.0}/spl2_testing_framework/tools/test_discovery.py

@@ -56,11 +56,52 @@ class TestDiscovery(abc.ABC):
 class BoxTestDiscovery(TestDiscovery):
     """Discovers tests for box tests, looking into module.test.json files"""
 
+    _IGNORE_FIELDS_IN_SPLUNK_CHECK_KEY = "ignore_fields_in_splunk_check"
+
     def __init__(self, path: str, performance_check: str, limit_tests: int = None):
         super().__init__(path, performance_check, limit_tests)
 
+    @classmethod
+    def _parse_ignore_fields_in_splunk_check(cls, test_block: dict) -> dict:
+        raw = test_block.get(cls._IGNORE_FIELDS_IN_SPLUNK_CHECK_KEY)
+        if raw is None:
+            return {}
+        if isinstance(raw, list):
+            out = {}
+            for k in raw:
+                if isinstance(k, str):
+                    out[k] = ""
+                else:
+                    _LOGGER.warning(
+                        "%s list entries must be strings; skipping %r",
+                        cls._IGNORE_FIELDS_IN_SPLUNK_CHECK_KEY,
+                        k,
+                    )
+            return out
+        if not isinstance(raw, dict):
+            _LOGGER.warning(
+                "%s must be an object (field -> reason) or a list of field names; ignoring %r",
+                cls._IGNORE_FIELDS_IN_SPLUNK_CHECK_KEY,
+                raw,
+            )
+            return {}
+        out = {}
+        for key, reason in raw.items():
+            if not isinstance(key, str):
+                _LOGGER.warning(
+                    "%s keys must be strings; skipping %r",
+                    cls._IGNORE_FIELDS_IN_SPLUNK_CHECK_KEY,
+                    key,
+                )
+                continue
+            if isinstance(reason, str):
+                out[key] = reason
+            else:
+                out[key] = str(reason)
+        return out
+
     def discover_tests(self) -> None:
-        self._read_tests("module.test.json")
+        self._read_tests("*.test.json")
         self._parse_tests()
 
     def _parse_tests(self) -> None:
@@ -71,33 +112,42 @@ class BoxTestDiscovery(TestDiscovery):
         for module_path, content in self._parsed_modules.items():
             path = module_path.parent
             tests_added = 0
-            for test in content:
+            for entry in content:
                 if self.limit_tests is not None and tests_added >= self.limit_tests:
                     _LOGGER.info(
                         f"Limit of {self.limit_tests} tests reached for {module_path}. Stopping."
                     )
                     break
-                name = test["filename"].replace(".spl2", "")
-                test_input = test["test"]["source"]
-                code_module_path = path / test["filename"]
+                test_block = entry["test"]
+                name = entry["filename"].replace(".spl2", "")
+                test_input = test_block["source"]
+                code_module_path = path / entry["filename"]
                 output = [
                     Results(x)
-                    for x in test["test"].get("expected_destination_result", [])
+                    for x in test_block.get("expected_destination_result", [])
                 ]
+                cim_fields = test_block.get("expected_cim_fields", {}).get(
+                    "cim_fields", {}
+                )
                 metrics = [
                     Metrics(x)
-                    for x in test["test"].get("expected_metrics_destination_result", [])
+                    for x in test_block.get("expected_metrics_destination_result", [])
                 ]
-                test = BoxTest(
+                ignore_fields_in_splunk_check = (
+                    self._parse_ignore_fields_in_splunk_check(test_block)
+                )
+                box_test = BoxTest(
                     name=name,
                     input=test_input,
                     output=output,
                     metrics=metrics,
                     module_path=module_path,
                     code_module_path=code_module_path,
+                    cim_fields=cim_fields,
+                    ignore_fields_in_splunk_check=ignore_fields_in_splunk_check,
                 )
-                test.meta["performance"] = self._performance_check
-                self.tests.add(test)
+                box_test.meta["performance"] = self._performance_check
+                self.tests.add(box_test)
                 tests_added += 1
 
 

{spl2_testing_framework-1.5.0 → spl2_testing_framework-1.6.0}/spl2_testing_framework/tools/test_types.py

@@ -45,6 +45,9 @@ class BoxTest(Test):
     module_path: Path
     code_module_path: Path
     metrics: list = field(default_factory=list)
+    cim_fields: dict = field(default_factory=dict)
+    # TA Splunk check: field name -> why it is ignored (comparison-only; keys stripped from both sides)
+    ignore_fields_in_splunk_check: dict = field(default_factory=dict)
 
 
 @dataclass

spl2_testing_framework-1.6.0/spl2_testing_framework/tools/utils.py

@@ -0,0 +1,114 @@
+# Copyright 2026 Splunk Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+import ast
+import json
+import logging
+import pathlib
+import re
+from functools import lru_cache
+
+_LOGGER = logging.getLogger(__name__)
+
+
+def get_framework_root() -> pathlib.Path:
+    framework_root = pathlib.Path(__file__).parents[1]
+    return framework_root
+
+
+def _make_functions_visible_for_testing(code_module_path):
+    code_module = code_module_path.read_text(encoding="utf-8")
+    functions = re.findall(r"^function\s+([^(]+)\(", code_module, flags=re.M)
+    for function in functions:
+        if not re.search(r"export\s" + function, code_module):
+            code_module = re.sub(
+                r"^function\s+" + function,
+                f"export function {function}",
+                code_module,
+                flags=re.M,
+            )
+    return code_module
+
+
+@lru_cache(None)
+def read_assertions_module():
+    root_folder = get_framework_root()
+    assertions_file = root_folder / "spl2_utils" / "assertions.spl2"
+    return assertions_file.read_text(encoding="utf-8")
+
+
+@lru_cache(None)
+def read_commands_modules():
+    root_folder = get_framework_root()
+    commands_module = root_folder / "spl2_utils" / "logs_to_metrics.spl2"
+    return commands_module.read_text(encoding="utf-8")
+
+
+def calculate_reduction_rate(original_event, reduced_event):
+    """Calculate how much the event was reduced."""
+    reduction_rates = []
+    for original, reduced in zip(original_event, reduced_event):
+        original_event_len = len(str(original))
+        reduced_event_len = len(str(reduced))
+
+        reduction_rates.append(
+            (original_event_len - reduced_event_len) / original_event_len
+        )
+    return reduction_rates
+
+
+def validate_compatibility(expected, parsed_output):
+    """Validate if all fields from parsed_output are present in expected and have the same value."""
+    extracted_but_wrong = {}
+    not_extracted = {}
+    for field, value in expected.items():
+        if field in parsed_output:
+            if parsed_output[field] == value:
+                _LOGGER.info(f"FIELD {field} in reduced event")
+                continue
+            else:
+                extracted_but_wrong[field] = value
+                _LOGGER.error(f"FIELD {field} in reduced event has wrong value")
+
+        else:
+            not_extracted[field] = value
+            _LOGGER.error(f"FIELD {field} is not present in the reduced event")
+
+    _LOGGER.info(f"Checking wrong extractions")
+    assert extracted_but_wrong == {}
+    _LOGGER.info(f"Checking missing extractions")
+    assert not_extracted == {}
+
+
+def parse_raw_from_input(input_str):
+    """Parse SPL2-style or JSON array of events and return list of _raw field values.
+    Handles unquoted keys (e.g. {_raw: "x"}) by converting them to JSON-quoted keys.
+    """
+    # Only replace unquoted keys: key must be preceded by { or , (and optional space)
+    key_pattern = re.compile(
+        r"(\{|,)\s*(_raw|source|sourcetype|host)\s*:", re.MULTILINE
+    )
+
+    def quote_key(match):
+        return f'{match.group(1)}"{match.group(2)}":'
+
+    normalized = key_pattern.sub(quote_key, input_str)
+    try:
+        parsed = json.loads(normalized)
+    except json.JSONDecodeError:
+        parsed = ast.literal_eval(normalized)
+    return [item["_raw"] for item in parsed]
+
+
+def parse_raw_from_reduced_events(events):
+    return [str(x["_raw"]) for x in events]

spl2_testing_framework-1.5.0/spl2_testing_framework/spl2_test_config.json

@@ -1,9 +0,0 @@
-{
-  "host": "",
-  "cloud_instance": "",
-  "tenant": "",
-  "bearer_token": "",
-  "port": "",
-  "user": "",
-  "password": ""
-}

spl2_testing_framework-1.5.0/spl2_testing_framework/tools/utils.py

@@ -1,51 +0,0 @@
-# Copyright 2026 Splunk Inc.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-
-import pathlib
-import re
-from functools import lru_cache
-
-
-def get_framework_root() -> pathlib.Path:
-    framework_root = pathlib.Path(__file__).parents[1]
-    return framework_root
-
-
-def _make_functions_visible_for_testing(code_module_path):
-    code_module = code_module_path.read_text(encoding="utf-8")
-    functions = re.findall(r"^function\s+([^(]+)\(", code_module, flags=re.M)
-    for function in functions:
-        if not re.search(r"export\s" + function, code_module):
-            code_module = re.sub(
-                r"^function\s+" + function,
-                f"export function {function}",
-                code_module,
-                flags=re.M,
-            )
-    return code_module
-
-
-@lru_cache(None)
-def read_assertions_module():
-    root_folder = get_framework_root()
-    assertions_file = root_folder / "spl2_utils" / "assertions.spl2"
-    return assertions_file.read_text(encoding="utf-8")
-
-
-@lru_cache(None)
-def read_commands_modules():
-    root_folder = get_framework_root()
-    commands_module = root_folder / "spl2_utils" / "logs_to_metrics.spl2"
-    return commands_module.read_text(encoding="utf-8")