spl2-testing-framework 1.5.1__tar.gz → 1.7.0__tar.gz

{spl2_testing_framework-1.5.1 → spl2_testing_framework-1.7.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: spl2-testing-framework
-Version: 1.5.1
+Version: 1.7.0
 Summary: SPL2 Testing Framework
 Home-page: https://github.com/splunk/spl2-testing-framework
 Keywords: SPL2 testing framework
@@ -25,6 +25,7 @@ Requires-Dist: pytest (==7.4.4)
 Requires-Dist: pytest-subtests (==0.13.1)
 Requires-Dist: pytest-xdist (==3.5.0)
 Requires-Dist: requests (>=2.31.0,<3.0.0)
+Requires-Dist: splunk-sdk (>=2.1.1,<3.0.0)
 Requires-Dist: tabulate (==0.9.0)
 Requires-Dist: urllib3 (>=1.26.19,<2.0.0)
 Project-URL: Repository, https://github.com/splunk/spl2-testing-framework
@@ -130,12 +131,15 @@ It is possible to pass more options to the command, which works also with pytest
 
 Additionally, the following options are supported:
 
+* `--test_dir` - directory where test files (`.test.json`, `.test.spl2`, `module.json`) are located. Defaults to the current working directory. Discovery is recursive.
+* `--code_dir` - directory where SPL2 code modules (`.spl2`) are located. Defaults to the current working directory. Falls back to `--test_dir` if the module is not found. See [Separate code and test directories](#separate-code-and-test-directories) for details.
 * `--ignore_empty_strings` - to ignore empty strings in the results
 * `--ignore_additional_fields_in_actual` - to ignore fields present in actual results but not in expected results (useful when actual results contain extra fields that should not affect comparison)
 * `--create_comparison_sheet` - to create a comparison sheet in `comparison_box_test` folder using actual and expected outputs (works only when running box tests)
 * `--cli_bench` - when running tests with CLI, this option enables benchmarking mode by adding `-b` and `-n` flags to the spl2-processor-cli command. The value specifies the number of events to test (e.g., `--cli_bench=1000` will add `-b -n 1000` to the CLI command). **In bench mode, all tests always succeed and CLI output is printed to logs** - this is useful for performance testing and benchmarking. This is only applicable when using `--type cli`.
+* `--check_splunk_results` - option to validate box test output by sending events to a Splunk instance (using HEC) and querying them back, with support for **CIM** (Common Information Model) and **TA** (Technology Add-on) checks.
 
-Note: The `pytest.ini.sample` file allows you to define command parameters. Just update the configurations, rename the
+* Note: The `pytest.ini.sample` file allows you to define command parameters. Just update the configurations, rename the
 file by removing the `.sample` extension, and execute the command.
 
 ### Run tests in IDE [PyCharm]
@@ -150,10 +154,53 @@ Sample configuration which may be used:
     - Parameters:
       `--type [cli | splunk | cloud] --test_dir /tests/resources -o log_cli=true --log-cli-level=INFO --verbose`
     - If test dir is not specified, current working directory will be used
+    - Use `--code_dir` if SPL2 code modules are in a separate directory from tests
     - If necessary another pytest options can be added
 
 Note: It's necessary to set "pytest" as default test runner in PyCharm settings
 
+### Separate code and test directories
+
+By default, the framework expects SPL2 code modules (`.spl2` files) to live alongside their test files. The `--code_dir` option allows you to keep code and tests in separate directory trees.
+
+#### How it works
+
+| What is searched | Primary directory | Fallback directory |
+|------------------|-------------------|--------------------|
+| **Code modules** (`.spl2`) | `--code_dir` | `--test_dir` |
+| **Test files** (`.test.json`, `.test.spl2`) | `--test_dir` | `--code_dir` |
+
+Both searches are recursive. If the primary directory has no results, the framework falls back to the other directory and logs the action.
+
+When multiple files with the same name are found within a directory, a warning is logged and the lexicographically first match is used.
+
+#### Example project layout
+
+```
+my-project/
+├── src/                          # --code_dir
+│   ├── network_traffic.spl2
+│   └── dns_lookup.spl2
+└── tests/                        # --test_dir
+    ├── network_traffic.test.json
+    ├── dns_lookup.test.json
+    └── dns_lookup.test.spl2
+```
+
+Run with:
+
+```bash
+spl2_tests_run cli --test_dir tests --code_dir src
+```
+
+#### Backward compatibility
+
+When `--code_dir` is not specified (or both flags point to the same directory), the framework behaves exactly as before — code modules are found next to their test files via the same recursive search.
+
+#### Applies to all test types
+
+The `--code_dir` option works for box tests, unit tests, and single SPL2 file execution.
+
 ## Executing a spl2 file
 
 This framework also supports executing a single spl2 file and prints the results in command line as well as a log file.
@@ -177,6 +224,7 @@ It is possible to pass more options to the command, which works also with pytest
 
 * `--test_dir` - Path in which the template file and module.json are available. If not provided, it will look for the
   current directory for the template file and module.json file
+* `--code_dir` - Path where SPL2 code modules are located, if different from `--test_dir`. Falls back to `--test_dir` when not set. See [Separate code and test directories](#separate-code-and-test-directories).
 * `--template_file` - The spl2 template file to execute
 * `--sample_file` - A file containing all the samples required for the template file. If not provided, it will look for
   the samples in module.json file of the corresponding template file
@@ -206,6 +254,76 @@ Content of this file will also be printed to stdout.
 This checks can be applied only to box tests, as assertions which are used in unit tests may impact spl2 pipeline
 performance. 
 
+### Check splunk results
+
+When set (`--check_splunk_results=<option>`), box tests (after running the SPL2 pipeline) send pipeline output to the configured Splunk instance and validate results there instead of comparing only in-memory expected vs actual.
+
+| Value | Behavior |
+|-------|----------|
+| **CIM** | For each output event, push to Splunk, query by host, then validate that the returned event has the expected CIM fields (from `expected_cim_fields.cim_fields` in the box test). Uses `validate_compatibility()`. Skips the test if the box test has no `cim_fields`. |
+| **TA** | For each output event, push to Splunk, query by host, then assert that the returned event equals the expected output event for that index. |
+
+For **TA** only, you can name top-level fields to drop on **both** the queried event and each `expected_destination_result` row by adding **`ignore_fields_in_splunk_check`**: an object mapping **field name → short reason**. Use the same key under `test` in each entry in `*.test.json` (same file as `expected_destination_result`):
+
+```json
+"test": {
+  "source": "...",
+  "expected_destination_result": [ ... ],
+  "ignore_fields_in_splunk_check": {
+    "_raw": "Raw event differs after Splunk indexing",
+    "_time": "Timestamp may vary due to ingestion latency"
+  }
+}
+```
+
+
+If the key is omitted, no fields are ignored. Used only when `--check_splunk_results=TA` (ignored for other modes and for the default 
+in-memory box test assert). If the file or key is missing, no fields are ignored.
+
+#### Configuration:
+
+`SPLUNK_INSTANCE` in `spl2_test_config.json`
+
+New top-level key for the Splunk instance used by `--check_splunk_results`:
+
+```json
+{
+  "SPLUNK_INSTANCE": {
+    "ip": "<hostname-or-ip>",
+    "port": 8088,
+    "api_port": 8089,
+    "username": "<splunk-user>",
+    "password": "<splunk-password>",
+    "index": "<index-name>",
+    "hec_token": "<hec-token>"
+  }
+}
+```
+
+- **ip** – Splunk server host (e.g. `localhost` or hostname).  
+- **port** – HEC port (e.g. 8088).  
+- **api_port** – Management/API port (e.g. 8089) for Search API and index creation.  
+- **username** / **password** – Used for Search API and index creation (HTTP Basic).  
+- **index** – Index where HEC events are sent and then searched.  
+- **hec_token** – Token for HEC (`Authorization: Splunk <token>`).
+
+Do not commit real credentials. Prefer environment variables (below) or a local config that is not in version control.
+
+#### Environment variables
+
+Any `SPLUNK_INSTANCE` field can be set via the environment; config file values override these only when non-empty.
+
+| Variable | Maps to |
+|----------|--------|
+| `SPL2_TF_SPLUNK_INSTANCE_IP` | `ip` |
+| `SPL2_TF_SPLUNK_INSTANCE_PORT` | `port` |
+| `SPL2_TF_SPLUNK_INSTANCE_API_PORT` | `api_port` |
+| `SPL2_TF_SPLUNK_INSTANCE_USERNAME` | `username` |
+| `SPL2_TF_SPLUNK_INSTANCE_PASSWORD` | `password` |
+| `SPL2_TF_SPLUNK_INSTANCE_INDEX` | `index` |
+| `SPL2_TF_SPLUNK_INSTANCE_HEC_TOKEN` | `hec_token` |
+
+
 ## Format all files
 To format all files run:
 

{spl2_testing_framework-1.5.1 → spl2_testing_framework-1.7.0}/README.md

@@ -98,12 +98,15 @@ It is possible to pass more options to the command, which works also with pytest
 
 Additionally, the following options are supported:
 
+* `--test_dir` - directory where test files (`.test.json`, `.test.spl2`, `module.json`) are located. Defaults to the current working directory. Discovery is recursive.
+* `--code_dir` - directory where SPL2 code modules (`.spl2`) are located. Defaults to the current working directory. Falls back to `--test_dir` if the module is not found. See [Separate code and test directories](#separate-code-and-test-directories) for details.
 * `--ignore_empty_strings` - to ignore empty strings in the results
 * `--ignore_additional_fields_in_actual` - to ignore fields present in actual results but not in expected results (useful when actual results contain extra fields that should not affect comparison)
 * `--create_comparison_sheet` - to create a comparison sheet in `comparison_box_test` folder using actual and expected outputs (works only when running box tests)
 * `--cli_bench` - when running tests with CLI, this option enables benchmarking mode by adding `-b` and `-n` flags to the spl2-processor-cli command. The value specifies the number of events to test (e.g., `--cli_bench=1000` will add `-b -n 1000` to the CLI command). **In bench mode, all tests always succeed and CLI output is printed to logs** - this is useful for performance testing and benchmarking. This is only applicable when using `--type cli`.
+* `--check_splunk_results` - option to validate box test output by sending events to a Splunk instance (using HEC) and querying them back, with support for **CIM** (Common Information Model) and **TA** (Technology Add-on) checks.
 
-Note: The `pytest.ini.sample` file allows you to define command parameters. Just update the configurations, rename the
+* Note: The `pytest.ini.sample` file allows you to define command parameters. Just update the configurations, rename the
 file by removing the `.sample` extension, and execute the command.
 
 ### Run tests in IDE [PyCharm]
@@ -118,10 +121,53 @@ Sample configuration which may be used:
     - Parameters:
       `--type [cli | splunk | cloud] --test_dir /tests/resources -o log_cli=true --log-cli-level=INFO --verbose`
     - If test dir is not specified, current working directory will be used
+    - Use `--code_dir` if SPL2 code modules are in a separate directory from tests
     - If necessary another pytest options can be added
 
 Note: It's necessary to set "pytest" as default test runner in PyCharm settings
 
+### Separate code and test directories
+
+By default, the framework expects SPL2 code modules (`.spl2` files) to live alongside their test files. The `--code_dir` option allows you to keep code and tests in separate directory trees.
+
+#### How it works
+
+| What is searched | Primary directory | Fallback directory |
+|------------------|-------------------|--------------------|
+| **Code modules** (`.spl2`) | `--code_dir` | `--test_dir` |
+| **Test files** (`.test.json`, `.test.spl2`) | `--test_dir` | `--code_dir` |
+
+Both searches are recursive. If the primary directory has no results, the framework falls back to the other directory and logs the action.
+
+When multiple files with the same name are found within a directory, a warning is logged and the lexicographically first match is used.
+
+#### Example project layout
+
+```
+my-project/
+├── src/                          # --code_dir
+│   ├── network_traffic.spl2
+│   └── dns_lookup.spl2
+└── tests/                        # --test_dir
+    ├── network_traffic.test.json
+    ├── dns_lookup.test.json
+    └── dns_lookup.test.spl2
+```
+
+Run with:
+
+```bash
+spl2_tests_run cli --test_dir tests --code_dir src
+```
+
+#### Backward compatibility
+
+When `--code_dir` is not specified (or both flags point to the same directory), the framework behaves exactly as before — code modules are found next to their test files via the same recursive search.
+
+#### Applies to all test types
+
+The `--code_dir` option works for box tests, unit tests, and single SPL2 file execution.
+
 ## Executing a spl2 file
 
 This framework also supports executing a single spl2 file and prints the results in command line as well as a log file.
@@ -145,6 +191,7 @@ It is possible to pass more options to the command, which works also with pytest
 
 * `--test_dir` - Path in which the template file and module.json are available. If not provided, it will look for the
   current directory for the template file and module.json file
+* `--code_dir` - Path where SPL2 code modules are located, if different from `--test_dir`. Falls back to `--test_dir` when not set. See [Separate code and test directories](#separate-code-and-test-directories).
 * `--template_file` - The spl2 template file to execute
 * `--sample_file` - A file containing all the samples required for the template file. If not provided, it will look for
   the samples in module.json file of the corresponding template file
@@ -174,6 +221,76 @@ Content of this file will also be printed to stdout.
 This checks can be applied only to box tests, as assertions which are used in unit tests may impact spl2 pipeline
 performance. 
 
+### Check splunk results
+
+When set (`--check_splunk_results=<option>`), box tests (after running the SPL2 pipeline) send pipeline output to the configured Splunk instance and validate results there instead of comparing only in-memory expected vs actual.
+
+| Value | Behavior |
+|-------|----------|
+| **CIM** | For each output event, push to Splunk, query by host, then validate that the returned event has the expected CIM fields (from `expected_cim_fields.cim_fields` in the box test). Uses `validate_compatibility()`. Skips the test if the box test has no `cim_fields`. |
+| **TA** | For each output event, push to Splunk, query by host, then assert that the returned event equals the expected output event for that index. |
+
+For **TA** only, you can name top-level fields to drop on **both** the queried event and each `expected_destination_result` row by adding **`ignore_fields_in_splunk_check`**: an object mapping **field name → short reason**. Use the same key under `test` in each entry in `*.test.json` (same file as `expected_destination_result`):
+
+```json
+"test": {
+  "source": "...",
+  "expected_destination_result": [ ... ],
+  "ignore_fields_in_splunk_check": {
+    "_raw": "Raw event differs after Splunk indexing",
+    "_time": "Timestamp may vary due to ingestion latency"
+  }
+}
+```
+
+
+If the key is omitted, no fields are ignored. Used only when `--check_splunk_results=TA` (ignored for other modes and for the default 
+in-memory box test assert). If the file or key is missing, no fields are ignored.
+
+#### Configuration:
+
+`SPLUNK_INSTANCE` in `spl2_test_config.json`
+
+New top-level key for the Splunk instance used by `--check_splunk_results`:
+
+```json
+{
+  "SPLUNK_INSTANCE": {
+    "ip": "<hostname-or-ip>",
+    "port": 8088,
+    "api_port": 8089,
+    "username": "<splunk-user>",
+    "password": "<splunk-password>",
+    "index": "<index-name>",
+    "hec_token": "<hec-token>"
+  }
+}
+```
+
+- **ip** – Splunk server host (e.g. `localhost` or hostname).  
+- **port** – HEC port (e.g. 8088).  
+- **api_port** – Management/API port (e.g. 8089) for Search API and index creation.  
+- **username** / **password** – Used for Search API and index creation (HTTP Basic).  
+- **index** – Index where HEC events are sent and then searched.  
+- **hec_token** – Token for HEC (`Authorization: Splunk <token>`).
+
+Do not commit real credentials. Prefer environment variables (below) or a local config that is not in version control.
+
+#### Environment variables
+
+Any `SPLUNK_INSTANCE` field can be set via the environment; config file values override these only when non-empty.
+
+| Variable | Maps to |
+|----------|--------|
+| `SPL2_TF_SPLUNK_INSTANCE_IP` | `ip` |
+| `SPL2_TF_SPLUNK_INSTANCE_PORT` | `port` |
+| `SPL2_TF_SPLUNK_INSTANCE_API_PORT` | `api_port` |
+| `SPL2_TF_SPLUNK_INSTANCE_USERNAME` | `username` |
+| `SPL2_TF_SPLUNK_INSTANCE_PASSWORD` | `password` |
+| `SPL2_TF_SPLUNK_INSTANCE_INDEX` | `index` |
+| `SPL2_TF_SPLUNK_INSTANCE_HEC_TOKEN` | `hec_token` |
+
+
 ## Format all files
 To format all files run:
 

{spl2_testing_framework-1.5.1 → spl2_testing_framework-1.7.0}/pyproject.toml

@@ -15,7 +15,7 @@
 
 [tool.poetry]
 name = "spl2-testing-framework"
-version = "1.5.1"
+version = "1.7.0"
 description = "SPL2 Testing Framework"
 authors = ["mgazda-splunk <mgazda@splunk.com>"]
 readme = "README.md"
@@ -52,6 +52,7 @@ numpy = [
     { version = ">=1.26,<2.0", python = ">=3.13" }
 ]
 openpyxl = "^3.1.2"
+splunk-sdk = "^2.1.1"
 
 [tool.poetry.group.dev.dependencies]
 pre-commit = "*"

{spl2_testing_framework-1.5.1 → spl2_testing_framework-1.7.0}/spl2_testing_framework/conftest.py

@@ -15,10 +15,13 @@
 
 import json
 import os
+from collections import defaultdict
 
+import pandas as pd
 import pytest
 
 from .tools.results import Results
+from .tools.splunk_integration.splunk_handler import SplunkHandler
 
 pytest.register_assert_rewrite("spl2_testing_framework")
 
@@ -48,9 +51,11 @@ SINGLE_SPL2_FILE = "single_spl2_file"
 IGNORE_EMPTY_STRINGS = "ignore_empty_strings"
 IGNORE_ADDITIONAL_FIELDS_IN_ACTUAL = "ignore_additional_fields_in_actual"
 CREATE_COMPARISON_SHEET = "create_comparison_sheet"
+CHECK_SPLUNK_RESULTS = "check_splunk_results"
 PERFORMANCE_CHECK = "performance_check"
 TEST_TYPE = "type"
 TEST_DIR = "test_dir"
+CODE_DIR = "code_dir"
 LIMIT_TESTS = "limit_tests"
 CLI_BENCH = "cli_bench"
 
@@ -66,6 +71,7 @@ _LOGGER = logging.getLogger(__name__)
 def pytest_addoption(parser):
     parser.addoption(f"--{TEST_TYPE}", action="store", default="cli")
     parser.addoption(f"--{TEST_DIR}", action="store", default=".")
+    parser.addoption(f"--{CODE_DIR}", action="store", default=".")
     parser.addoption(
         f"--{PERFORMANCE_CHECK}",
         action="store",
@@ -75,6 +81,9 @@ def pytest_addoption(parser):
     parser.addoption("--template_file", action="store", default=None)
     parser.addoption("--sample_file", action="store", default=None)
     parser.addoption("--sample_delimiter", action="store", default="\n")
+    parser.addoption(
+        f"--{CHECK_SPLUNK_RESULTS}", action="store", type=str, default=None
+    )
     parser.addoption(f"--{CREATE_COMPARISON_SHEET}", action="store_true", default=False)
     parser.addoption(f"--{IGNORE_EMPTY_STRINGS}", action="store_true")
     parser.addoption(f"--{IGNORE_ADDITIONAL_FIELDS_IN_ACTUAL}", action="store_true")
@@ -86,13 +95,16 @@ def pytest_generate_tests(metafunc):
     """This function is used for dynamic generation of tests based on the test files in the directory specified by the
     --test_dir option"""
     test_dir = metafunc.config.getoption(TEST_DIR)
+    code_dir = metafunc.config.getoption(CODE_DIR)
     performance_check = metafunc.config.getoption(PERFORMANCE_CHECK)
     limit_tests = metafunc.config.getoption(LIMIT_TESTS)
     _set_results_options(metafunc)
 
     if BOX_TEST in metafunc.fixturenames:
         _LOGGER.info("Collecting Box Tests")
-        module_tests = BoxTestDiscovery(test_dir, performance_check, limit_tests)
+        module_tests = BoxTestDiscovery(
+            test_dir, performance_check, limit_tests, code_dir=code_dir
+        )
         module_tests.discover_tests()
         metafunc.parametrize(
             "%s" % BOX_TEST, module_tests.tests, ids=module_tests.tests.get_ids()
@@ -112,6 +124,7 @@ def pytest_generate_tests(metafunc):
             sample_delimiter,
             performance_check=performance_check,
             limit_tests=limit_tests,
+            code_dir=code_dir,
         )
         try:
             spl2_pipelines.discover_tests()
@@ -125,7 +138,7 @@ def pytest_generate_tests(metafunc):
 
     if UNIT_TEST in metafunc.fixturenames:
         _LOGGER.info("Collecting Unit Tests")
-        unit_tests = UTDiscovery(test_dir, limit_tests)
+        unit_tests = UTDiscovery(test_dir, limit_tests, code_dir=code_dir)
         unit_tests.discover_tests()
         metafunc.parametrize(
             UNIT_TEST, unit_tests.tests, ids=unit_tests.tests.get_ids()
@@ -139,6 +152,8 @@ def spl2_test_runner(
     basic_authentication,
     bearer_token_authentication,
     run_configurations,
+    splunk_handler,
+    stats,
 ):
     """This fixture is used to create a test runner object based on the test type specified by the --type option
     3 types of tests are supported: 'splunk', 'cloud', 'cli'
@@ -160,7 +175,7 @@ def spl2_test_runner(
             )
         )
 
-    x = SPL2TestRunner(search_client, run_configurations)
+    x = SPL2TestRunner(search_client, run_configurations, splunk_handler, stats)
     return x
 
 
@@ -172,11 +187,35 @@ def run_configurations(request):
             f"--{CREATE_COMPARISON_SHEET}"
         ),
         CLI_BENCH: request.config.getoption(f"--{CLI_BENCH}"),
+        CHECK_SPLUNK_RESULTS: request.config.getoption(f"--{CHECK_SPLUNK_RESULTS}"),
     }
 
 
 @pytest.fixture(scope="session")
-def config():
+def splunk_instance_config():
+    splunk_instance_ip = os.environ.get("SPL2_TF_SPLUNK_INSTANCE_IP", None)
+    splunk_instance_port = os.environ.get("SPL2_TF_SPLUNK_INSTANCE_PORT", None)
+    splunk_instance_api_port = os.environ.get("SPL2_TF_SPLUNK_INSTANCE_API_PORT", None)
+    splunk_instance_username = os.environ.get("SPL2_TF_SPLUNK_INSTANCE_USERNAME", None)
+    splunk_instance_password = os.environ.get("SPL2_TF_SPLUNK_INSTANCE_PASSWORD", None)
+    splunk_instance_index = os.environ.get("SPL2_TF_SPLUNK_INSTANCE_INDEX", None)
+    splunk_instance_hec_token = os.environ.get(
+        "SPL2_TF_SPLUNK_INSTANCE_HEC_TOKEN", None
+    )
+
+    return {
+        "ip": splunk_instance_ip,
+        "port": splunk_instance_port,
+        "api_port": splunk_instance_api_port,
+        "username": splunk_instance_username,
+        "password": splunk_instance_password,
+        "index": splunk_instance_index,
+        "hec_token": splunk_instance_hec_token,
+    }
+
+
+@pytest.fixture(scope="session")
+def config(splunk_instance_config):
     """This fixture is used to read the configuration from the environment variables or from the spl2_test_config.json
     Configuration set using environment variables will be overwritten by settings from spl2_test_config.json file,
     however empty values in spl2_test_config.json will be ignored.
@@ -219,6 +258,12 @@ def config():
     if conf_file.get("cloud_instance", None):
         cloud_instance = conf_file["cloud_instance"]
 
+    for k, v in splunk_instance_config.items():
+        if v is None:
+            splunk_instance_config[k] = conf_file.get("SPLUNK_INSTANCE", {}).get(
+                k, None
+            )
+
     return {
         "host": host,
         "port": port,
@@ -227,6 +272,7 @@ def config():
         "bearer_token": bearer_token,
         "tenant": tenant,
         "cloud_instance": cloud_instance,
+        "SPLUNK_INSTANCE": splunk_instance_config,
     }
 
 
@@ -251,6 +297,49 @@ def bearer_token_authentication(config):
     return HTTPBearerTokenAuth(token)
 
 
+@pytest.fixture(scope="session")
+def splunk_token_authentication(config):
+    class SplunkTokenAuth(AuthBase):
+        """Attaches HTTP Basic Authentication to the given Request object."""
+
+        def __init__(self, token):
+            self.token = token
+
+        def __call__(self, r):
+            r.headers["Authorization"] = f"Splunk {self.token}"
+            return r
+
+    token = config["SPLUNK_INSTANCE"].get("hec_token", "")
+    splunk_auth = SplunkTokenAuth(token)
+    return splunk_auth
+
+
+@pytest.fixture(scope="session")
+def splunk_handler(config, splunk_token_authentication):
+    config["SPLUNK_INSTANCE"][
+        "url"
+    ] = f"https://{config['SPLUNK_INSTANCE']['ip']}:{config['SPLUNK_INSTANCE']['port']}/services/collector/raw"
+    config["SPLUNK_INSTANCE"][
+        "index_url"
+    ] = f"https://{config['SPLUNK_INSTANCE']['ip']}:{config['SPLUNK_INSTANCE']['api_port']}/services/data/indexes"
+
+    sh = SplunkHandler(config, splunk_token_authentication)
+    return sh
+
+
+@pytest.fixture(scope="session")
+def stats():
+    s = defaultdict(lambda: defaultdict(list))
+    yield s
+
+    for template, v in s.items():
+        x = pd.DataFrame(v)
+        _LOGGER.info(" *** *** ***")
+        _LOGGER.info("Collected statistics: ")
+        _LOGGER.info("Template name: %s ", template)
+        _LOGGER.info(x.describe())
+
+
 def _set_results_options(metafunc):
     ignore_empty_strings = metafunc.config.getoption(IGNORE_EMPTY_STRINGS)
     Results._ignore_empty_strings = ignore_empty_strings

spl2_testing_framework-1.7.0/spl2_testing_framework/spl2_test_config.json

@@ -0,0 +1,18 @@
+{
+  "host": "",
+  "cloud_instance": "",
+  "tenant": "",
+  "bearer_token": "",
+  "port": "",
+  "user": "",
+  "password": "",
+  "SPLUNK_INSTANCE": {
+    "ip": "",
+    "port": "",
+    "api_port": "",
+    "username": "",
+    "password": "",
+    "index": "",
+    "hec_token": ""
+  }
+}

{spl2_testing_framework-1.5.1 → spl2_testing_framework-1.7.0}/spl2_testing_framework/tools/search_clients/cli_search_client.py

@@ -75,8 +75,13 @@ class CLISearchClient(SearchClient):
         # Normal mode - raise errors as before
         if result.stderr:
             raise RuntimeError(result.stderr)
-
-        job.result = json.loads(result.stdout)
+        try:
+            job.result = json.loads(result.stdout)
+        except json.decoder.JSONDecodeError:
+            _LOGGER.error(
+                "Can't decode output for %s. Details: %s ", job.test_name, result
+            )
+            raise
         return
 
     def _print_bench_results(self, job, result):