spl2-testing-framework 1.8.3__tar.gz → 1.8.4__tar.gz

{spl2_testing_framework-1.8.3 → spl2_testing_framework-1.8.4}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: spl2-testing-framework
-Version: 1.8.3
+Version: 1.8.4
 Summary: SPL2 Testing Framework
 Home-page: https://github.com/splunk/spl2-testing-framework
 Keywords: SPL2 testing framework
@@ -137,6 +137,7 @@ Additionally, the following options are supported:
 * `--ignore_additional_fields_in_actual` - to ignore fields present in actual results but not in expected results (useful when actual results contain extra fields that should not affect comparison)
 * `--create_comparison_sheet` - to create a comparison sheet in `comparison_box_test` folder using actual and expected outputs (works only when running box tests)
 * `--cli_bench` - when running tests with CLI, this option enables benchmarking mode by adding `-b` and `-n` flags to the spl2-processor-cli command. The value specifies the number of events to test (e.g., `--cli_bench=1000` will add `-b -n 1000` to the CLI command). **In bench mode, all tests always succeed and CLI output is printed to logs** - this is useful for performance testing and benchmarking. This is only applicable when using `--type cli`.
+* `--lookups_root_dir` - root directory passed to `spl2-processor-cli` as `--lookupsRootDir` for resolving lookup `transforms.conf`. Defaults to `.` (current working directory). For a standard TA layout use `--lookups_root_dir=package/default`. Only applicable when using `--type cli`.
 * `--check_splunk_results` - option to validate box test output by sending events to a Splunk instance (using HEC) and querying them back, with support for **CIM** (Common Information Model) and **TA** (Technology Add-on) checks.
 * `--reduction_stats` - when set, **box tests** record a reduction rate per run: input `_raw` vs output `_raw` (after pairing input/output events). Rates are grouped by **pipeline** (the `.spl2` code module stem) and **sourcetype** (from the input event if set, else the output event, else `UNKNOWN`). Off by default.
 * `--reduction_stats_file` - path to a JSON file written at the **end of the pytest session** with collected rates (default `reduction_stats.json`). Only used when `--reduction_stats` is set.
@@ -307,6 +308,26 @@ You can name top-level fields to drop on **both** the pipeline/actual row and ea
 
 If the key is omitted, no fields are ignored. It applies to the **default in-memory** box test assert (no `--check_splunk_results`) and to **`--check_splunk_results=TA`**. It does **not** apply to **`--check_splunk_results=CIM`**. If the file or key is missing, no fields are ignored.
 
+#### Skipping a test for specific run modes
+
+Use **`skip_on`** in the `test` block to skip an individual test entry depending on how the framework is invoked:
+
+```json
+"test": {
+  "source": "...",
+  "expected_destination_result": [ ... ],
+  "skip_on": ["splunk_ta", "splunk_cim"]
+}
+```
+
+| Value | Skipped when |
+|-------|-------------|
+| `"standard"` | Running without `--check_splunk_results` (default in-memory compare) |
+| `"splunk_ta"` | Running with `--check_splunk_results=TA` |
+| `"splunk_cim"` | Running with `--check_splunk_results=CIM` |
+
+A single value can be given as a plain string instead of a list. Unrecognised values are ignored with a warning.
+
 #### Configuration:
 
 `SPLUNK_INSTANCE` in `spl2_test_config.json`
@@ -351,6 +372,45 @@ Any `SPLUNK_INSTANCE` field can be set via the environment; config file values o
 | `SPL2_TF_SPLUNK_INSTANCE_HEC_TOKEN` | `hec_token` |
 
 
+## Starting a local Splunk instance for TA testing
+
+`spl2_splunk_start` spins up a `splunk/splunk` Docker container with the TA mounted as an installed app and writes a ready-to-use `spl2_test_config.json`.
+
+```bash
+spl2_splunk_start --ta-path /path/to/ta
+```
+
+The command:
+1. Detects the TA app directory — prefers `<ta-path>/package/` (standard TA layout), falls back to `<ta-path>` itself.
+2. Reads `app.manifest` to derive the Splunk app name.
+3. Starts the container with the app directory mounted as a volume.
+4. Waits for Ansible provisioning (`PLAY RECAP`) to complete, then polls the HEC health endpoint until Splunk is ready.
+5. Writes `spl2_test_config.json` in the current directory with all connection details pre-filled.
+
+**Options**
+
+| Option | Default | Description |
+|--------|---------|-------------|
+| `--ta-path` | `.` | Path to the TA root directory. |
+| `--splunk-version` | `10.4` | Splunk Docker image tag. |
+| `--no-config` | off | Skip writing `spl2_test_config.json`. |
+| `--config-dest` | `spl2_test_config.json` | Where to write the config file. |
+
+Once the container is up, run your tests with:
+
+```bash
+spl2_tests_run splunk --check_splunk_results=TA
+```
+
+**Requirements:** Docker must be running. The container exposes ports `8088` (HEC) and `8089` (API). If a container named `spl2-splunk` already exists with a matching app mount and Splunk version it is reused (started if stopped); if it exists with different parameters it is removed and recreated.
+
+Override the default credentials via environment variables if needed:
+
+| Variable | Default | Purpose |
+|----------|---------|---------|
+| `SPL2_TF_LOCAL_SPLUNK_PASSWORD` | `Chang3d!` | Splunk admin password |
+| `SPL2_TF_LOCAL_SPLUNK_HEC_TOKEN` | *(built-in token)* | HEC token |
+
 ## Format all files
 To format all files run:
 

{spl2_testing_framework-1.8.3 → spl2_testing_framework-1.8.4}/README.md

@@ -104,6 +104,7 @@ Additionally, the following options are supported:
 * `--ignore_additional_fields_in_actual` - to ignore fields present in actual results but not in expected results (useful when actual results contain extra fields that should not affect comparison)
 * `--create_comparison_sheet` - to create a comparison sheet in `comparison_box_test` folder using actual and expected outputs (works only when running box tests)
 * `--cli_bench` - when running tests with CLI, this option enables benchmarking mode by adding `-b` and `-n` flags to the spl2-processor-cli command. The value specifies the number of events to test (e.g., `--cli_bench=1000` will add `-b -n 1000` to the CLI command). **In bench mode, all tests always succeed and CLI output is printed to logs** - this is useful for performance testing and benchmarking. This is only applicable when using `--type cli`.
+* `--lookups_root_dir` - root directory passed to `spl2-processor-cli` as `--lookupsRootDir` for resolving lookup `transforms.conf`. Defaults to `.` (current working directory). For a standard TA layout use `--lookups_root_dir=package/default`. Only applicable when using `--type cli`.
 * `--check_splunk_results` - option to validate box test output by sending events to a Splunk instance (using HEC) and querying them back, with support for **CIM** (Common Information Model) and **TA** (Technology Add-on) checks.
 * `--reduction_stats` - when set, **box tests** record a reduction rate per run: input `_raw` vs output `_raw` (after pairing input/output events). Rates are grouped by **pipeline** (the `.spl2` code module stem) and **sourcetype** (from the input event if set, else the output event, else `UNKNOWN`). Off by default.
 * `--reduction_stats_file` - path to a JSON file written at the **end of the pytest session** with collected rates (default `reduction_stats.json`). Only used when `--reduction_stats` is set.
@@ -274,6 +275,26 @@ You can name top-level fields to drop on **both** the pipeline/actual row and ea
 
 If the key is omitted, no fields are ignored. It applies to the **default in-memory** box test assert (no `--check_splunk_results`) and to **`--check_splunk_results=TA`**. It does **not** apply to **`--check_splunk_results=CIM`**. If the file or key is missing, no fields are ignored.
 
+#### Skipping a test for specific run modes
+
+Use **`skip_on`** in the `test` block to skip an individual test entry depending on how the framework is invoked:
+
+```json
+"test": {
+  "source": "...",
+  "expected_destination_result": [ ... ],
+  "skip_on": ["splunk_ta", "splunk_cim"]
+}
+```
+
+| Value | Skipped when |
+|-------|-------------|
+| `"standard"` | Running without `--check_splunk_results` (default in-memory compare) |
+| `"splunk_ta"` | Running with `--check_splunk_results=TA` |
+| `"splunk_cim"` | Running with `--check_splunk_results=CIM` |
+
+A single value can be given as a plain string instead of a list. Unrecognised values are ignored with a warning.
+
 #### Configuration:
 
 `SPLUNK_INSTANCE` in `spl2_test_config.json`
@@ -318,6 +339,45 @@ Any `SPLUNK_INSTANCE` field can be set via the environment; config file values o
 | `SPL2_TF_SPLUNK_INSTANCE_HEC_TOKEN` | `hec_token` |
 
 
+## Starting a local Splunk instance for TA testing
+
+`spl2_splunk_start` spins up a `splunk/splunk` Docker container with the TA mounted as an installed app and writes a ready-to-use `spl2_test_config.json`.
+
+```bash
+spl2_splunk_start --ta-path /path/to/ta
+```
+
+The command:
+1. Detects the TA app directory — prefers `<ta-path>/package/` (standard TA layout), falls back to `<ta-path>` itself.
+2. Reads `app.manifest` to derive the Splunk app name.
+3. Starts the container with the app directory mounted as a volume.
+4. Waits for Ansible provisioning (`PLAY RECAP`) to complete, then polls the HEC health endpoint until Splunk is ready.
+5. Writes `spl2_test_config.json` in the current directory with all connection details pre-filled.
+
+**Options**
+
+| Option | Default | Description |
+|--------|---------|-------------|
+| `--ta-path` | `.` | Path to the TA root directory. |
+| `--splunk-version` | `10.4` | Splunk Docker image tag. |
+| `--no-config` | off | Skip writing `spl2_test_config.json`. |
+| `--config-dest` | `spl2_test_config.json` | Where to write the config file. |
+
+Once the container is up, run your tests with:
+
+```bash
+spl2_tests_run splunk --check_splunk_results=TA
+```
+
+**Requirements:** Docker must be running. The container exposes ports `8088` (HEC) and `8089` (API). If a container named `spl2-splunk` already exists with a matching app mount and Splunk version it is reused (started if stopped); if it exists with different parameters it is removed and recreated.
+
+Override the default credentials via environment variables if needed:
+
+| Variable | Default | Purpose |
+|----------|---------|---------|
+| `SPL2_TF_LOCAL_SPLUNK_PASSWORD` | `Chang3d!` | Splunk admin password |
+| `SPL2_TF_LOCAL_SPLUNK_HEC_TOKEN` | *(built-in token)* | HEC token |
+
 ## Format all files
 To format all files run:
 

{spl2_testing_framework-1.8.3 → spl2_testing_framework-1.8.4}/pyproject.toml

@@ -15,7 +15,7 @@
 
 [tool.poetry]
 name = "spl2-testing-framework"
-version = "1.8.3"
+version = "1.8.4"
 description = "SPL2 Testing Framework"
 authors = ["mgazda-splunk <mgazda@splunk.com>"]
 readme = "README.md"
@@ -30,6 +30,7 @@ include = "spl2_testing_framework"
 [tool.poetry.scripts]
 spl2_tests_run = "spl2_testing_framework.test_runner:run"
 single_spl2_file_run = "spl2_testing_framework.single_spl2_file_runner:run"
+spl2_splunk_start = "spl2_testing_framework.splunk_setup:run"
 
 [tool.poetry.dependencies]
 python = "^3.7.1"

{spl2_testing_framework-1.8.3 → spl2_testing_framework-1.8.4}/spl2_testing_framework/conftest.py

@@ -60,6 +60,7 @@ TEST_DIR = "test_dir"
 CODE_DIR = "code_dir"
 LIMIT_TESTS = "limit_tests"
 CLI_BENCH = "cli_bench"
+LOOKUPS_ROOT_DIR = "lookups_root_dir"
 REDUCTION_STATS = "reduction_stats"
 
 setup_logging()
@@ -88,6 +89,13 @@ def pytest_addoption(parser):
     parser.addoption(f"--{IGNORE_ADDITIONAL_FIELDS_IN_ACTUAL}", action="store_true")
     parser.addoption(f"--{LIMIT_TESTS}", action="store", type=int, default=None)
     parser.addoption(f"--{CLI_BENCH}", action="store", type=int, default=None)
+    parser.addoption(
+        f"--{LOOKUPS_ROOT_DIR}",
+        action="store",
+        type=str,
+        default=".",
+        help="Root directory for lookups transforms.conf passed to spl2-processor-cli as --lookupsRootDir. Defaults to '.' (current directory).",
+    )
     parser.addoption(
         f"--{REDUCTION_STATS}",
         action="store_true",
@@ -181,7 +189,10 @@ def spl2_test_runner(
         search_client = SplunkSearchClient(url=url, auth=basic_authentication)
     elif test_type == "cli":
         cli_bench = run_configurations.get(CLI_BENCH)
-        search_client = CLISearchClient(cli_bench=cli_bench)
+        lookups_root_dir = run_configurations.get(LOOKUPS_ROOT_DIR)
+        search_client = CLISearchClient(
+            cli_bench=cli_bench, lookups_root_dir=lookups_root_dir
+        )
     elif test_type == "cloud":
         url = f"https://{config['tenant']}.api.{config['cloud_instance']}/{config['tenant']}"
         search_client = CloudSearchClient(url=url, auth=bearer_token_authentication)
@@ -204,6 +215,7 @@ def run_configurations(request):
             f"--{CREATE_COMPARISON_SHEET}"
         ),
         CLI_BENCH: request.config.getoption(f"--{CLI_BENCH}"),
+        LOOKUPS_ROOT_DIR: request.config.getoption(f"--{LOOKUPS_ROOT_DIR}"),
         CHECK_SPLUNK_RESULTS: request.config.getoption(f"--{CHECK_SPLUNK_RESULTS}"),
         REDUCTION_STATS: request.config.getoption(
             f"--{REDUCTION_STATS}", default=False

{spl2_testing_framework-1.8.3 → spl2_testing_framework-1.8.4}/spl2_testing_framework/spl2_test_config.json

@@ -12,7 +12,7 @@
     "api_port": 8089,
     "username": "admin",
     "password": "Chang3d!",
-    "index": "testx",
+    "index": "spl2tf",
     "hec_token": "4a8a737d-5452-426c-a6f7-106dca4e813f"
   }
 }

spl2_testing_framework-1.8.4/spl2_testing_framework/splunk_setup.py

@@ -0,0 +1,356 @@
+# Copyright 2026 Splunk Inc.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+"""
+spl2_splunk_start — spin up a local Splunk container with a TA mounted and
+dump a ready-to-use spl2_test_config.json.
+
+Usage:
+    spl2_splunk_start [--ta-path /path/to/ta] [--splunk-version 10.4] [--no-config]
+
+The TA's ``package/`` directory is mounted into the container as an installed
+Splunk app.  Splunk is pre-configured with a HEC token so you can immediately
+run:
+
+    spl2_tests_run splunk --check_splunk_results=TA
+"""
+
+import argparse
+import json
+import os
+import select
+import subprocess
+import sys
+import time
+from pathlib import Path
+
+_SPLUNK_PASSWORD = os.environ.get("SPL2_TF_LOCAL_SPLUNK_PASSWORD", "Chang3d!")
+_HEC_TOKEN = os.environ.get(
+    "SPL2_TF_LOCAL_SPLUNK_HEC_TOKEN", "4a8a737d-5452-426c-a6f7-106dca4e813f"
+)
+_INDEX = "spl2tf"
+_CONTAINER_NAME = "spl2-splunk"
+_HEC_PORT = 8088
+_API_PORT = 8089
+_CONFIG_FILENAME = "spl2_test_config.json"
+
+
+def _find_ta_app_dir(ta_path: Path) -> Path:
+    """Return the directory to mount as a Splunk app.
+
+    Prefers ``<ta_path>/package`` (standard TA layout).  Falls back to
+    ``ta_path`` itself so the command also works when pointed directly at a
+    built/extracted app directory.
+    """
+    candidate = ta_path / "package"
+    if candidate.is_dir():
+        return candidate
+    return ta_path
+
+
+def _resolve_app_name(app_dir: Path) -> str:
+    """Derive a Splunk app name from app.manifest or the directory name."""
+    manifest = app_dir / "app.manifest"
+    if manifest.exists():
+        try:
+            data = json.loads(manifest.read_text())
+            name = data.get("info", {}).get("id", {}).get("name") or data.get("app_id")
+            if name:
+                return name
+        except (json.JSONDecodeError, AttributeError):
+            pass
+    return app_dir.name
+
+
+def _docker(*args, check=True, capture=False):
+    cmd = ["docker", *args]
+    return subprocess.run(
+        cmd,
+        check=check,
+        capture_output=capture,
+        text=True,
+    )
+
+
+def _container_exists(name: str) -> bool:
+    result = _docker(
+        "ps",
+        "-a",
+        "--filter",
+        f"name=^{name}$",
+        "--format",
+        "{{.Names}}",
+        capture=True,
+    )
+    return name in result.stdout.strip().splitlines()
+
+
+def _container_inspect(name: str) -> dict:
+    result = _docker("inspect", name, capture=True, check=False)
+    if result.returncode != 0:
+        return {}
+    try:
+        return json.loads(result.stdout)[0]
+    except (json.JSONDecodeError, IndexError):
+        return {}
+
+
+def _container_matches(name: str, app_dir_abs: str, splunk_version: str) -> bool:
+    """Return True if the existing container was started with the same app mount and image."""
+    info = _container_inspect(name)
+    if not info:
+        return False
+
+    image = info.get("Config", {}).get("Image", "")
+    if f"splunk/splunk:{splunk_version}" not in image:
+        return False
+
+    mounts = info.get("Mounts", [])
+    for mount in mounts:
+        if mount.get("Source") == app_dir_abs:
+            return True
+    return False
+
+
+def _write_config(dest: Path) -> None:
+    config = {
+        "host": "https://localhost",
+        "cloud_instance": "",
+        "tenant": "",
+        "bearer_token": "",
+        "port": str(_API_PORT),
+        "user": "admin",
+        "password": _SPLUNK_PASSWORD,
+        "SPLUNK_INSTANCE": {
+            "ip": "localhost",
+            "port": _HEC_PORT,
+            "api_port": _API_PORT,
+            "username": "admin",
+            "password": _SPLUNK_PASSWORD,
+            "index": _INDEX,
+            "hec_token": _HEC_TOKEN,
+        },
+    }
+    dest.write_text(json.dumps(config, indent=2))
+    print(f"Config written to {dest}")
+
+
+def _wait_for_ansible(timeout: int = 300) -> bool:
+    """Stream container logs until Ansible finishes (PLAY RECAP line appears).
+
+    Returns True on success, False on timeout or container stop.
+    Uses a polling read loop so the timeout is always enforced.
+    """
+    print("Waiting for Splunk Ansible provisioning to finish", end="", flush=True)
+    deadline = time.time() + timeout
+    proc = subprocess.Popen(
+        ["docker", "logs", "-f", _CONTAINER_NAME],
+        stdout=subprocess.PIPE,
+        stderr=subprocess.STDOUT,
+    )
+    try:
+        buf = b""
+        while time.time() < deadline:
+            remaining = deadline - time.time()
+            ready, _, _ = select.select([proc.stdout], [], [], min(remaining, 2.0))
+            if ready:
+                chunk = proc.stdout.read(4096)
+                if not chunk:
+                    break
+                buf += chunk
+                lines = buf.split(b"\n")
+                buf = lines[-1]
+                for line in lines[:-1]:
+                    text = line.decode("utf-8", errors="replace")
+                    if "PLAY RECAP" in text:
+                        print(" done.")
+                        return True
+                    print(".", end="", flush=True)
+            else:
+                if proc.poll() is not None:
+                    break
+    finally:
+        try:
+            proc.kill()
+        except OSError:
+            pass
+        proc.wait()
+
+    print()
+    print(
+        "ERROR: Ansible provisioning did not complete within the timeout. "
+        "Check container logs with: docker logs spl2-splunk",
+        file=sys.stderr,
+    )
+    return False
+
+
+def _wait_for_hec(timeout: int = 60) -> bool:
+    """Poll the HEC health endpoint — returns True once splunkd is fully up."""
+    import urllib.request
+    import ssl
+
+    hec_health = f"https://localhost:{_HEC_PORT}/services/collector/health"
+    ctx = ssl.create_default_context()
+    ctx.check_hostname = False
+    ctx.verify_mode = ssl.CERT_NONE
+
+    print("Waiting for Splunk HEC to become ready", end="", flush=True)
+    deadline = time.time() + timeout
+    while time.time() < deadline:
+        try:
+            with urllib.request.urlopen(hec_health, context=ctx, timeout=3) as resp:
+                if resp.status == 200:
+                    print(" ready.")
+                    return True
+        except Exception:
+            pass
+        print(".", end="", flush=True)
+        time.sleep(3)
+
+    print()
+    print(
+        "ERROR: HEC did not become ready within the timeout. "
+        "Check container logs with: docker logs spl2-splunk",
+        file=sys.stderr,
+    )
+    return False
+
+
+def start(
+    ta_path: Path, splunk_version: str, write_config: bool, config_dest: Path
+) -> None:
+    app_dir = _find_ta_app_dir(ta_path)
+    if not app_dir.is_dir():
+        print(f"ERROR: TA app directory not found at {app_dir}", file=sys.stderr)
+        sys.exit(1)
+
+    app_name = _resolve_app_name(app_dir)
+    app_dir_abs = str(app_dir.resolve())
+    container_app_path = f"/opt/splunk/etc/apps/{app_name}"
+
+    print(f"TA app dir : {app_dir_abs}")
+    print(f"App name   : {app_name}")
+    print(f"Container  : {_CONTAINER_NAME}")
+    print(f"Splunk     : splunk/splunk:{splunk_version}")
+
+    already_running = False
+    if _container_exists(_CONTAINER_NAME):
+        info = _container_inspect(_CONTAINER_NAME)
+        if _container_matches(_CONTAINER_NAME, app_dir_abs, splunk_version):
+            if info.get("State", {}).get("Running"):
+                print(
+                    f"Container '{_CONTAINER_NAME}' already running with matching app and version."
+                )
+                already_running = True
+            else:
+                print(
+                    f"Container '{_CONTAINER_NAME}' exists with matching config — starting it."
+                )
+                _docker("start", _CONTAINER_NAME)
+        else:
+            print(
+                f"Container '{_CONTAINER_NAME}' exists but has a different app or Splunk version — removing it."
+            )
+            _docker("rm", "-f", _CONTAINER_NAME)
+            _launch(app_dir_abs, container_app_path, splunk_version)
+    else:
+        _launch(app_dir_abs, container_app_path, splunk_version)
+
+    if not already_running and not _wait_for_ansible():
+        sys.exit(1)
+
+    if not _wait_for_hec():
+        sys.exit(1)
+
+    if write_config:
+        _write_config(config_dest)
+
+    print()
+    print("Splunk is up. Run your tests with:")
+    print("  spl2_tests_run splunk --check_splunk_results=TA")
+
+
+def _launch(app_dir_abs: str, container_app_path: str, splunk_version: str) -> None:
+    _docker(
+        "run",
+        "-d",
+        "--name",
+        _CONTAINER_NAME,
+        "--platform",
+        "linux/amd64",
+        "-p",
+        f"{_HEC_PORT}:{_HEC_PORT}",
+        "-p",
+        f"{_API_PORT}:{_API_PORT}",
+        "-e",
+        f"SPLUNK_PASSWORD={_SPLUNK_PASSWORD}",
+        "-e",
+        "SPLUNK_START_ARGS=--accept-license",
+        "-e",
+        "SPLUNK_GENERAL_TERMS=--accept-sgt-current-at-splunk-com",
+        "-e",
+        f"SPLUNK_HEC_TOKEN={_HEC_TOKEN}",
+        "-v",
+        f"{app_dir_abs}:{container_app_path}",
+        f"splunk/splunk:{splunk_version}",
+    )
+    print(f"Container '{_CONTAINER_NAME}' started.")
+
+
+def run() -> None:
+    """Entry point for the ``spl2_splunk_start`` console script."""
+    parser = argparse.ArgumentParser(
+        description=(
+            "Start a local Splunk container with a TA mounted and write "
+            "spl2_test_config.json ready for --check_splunk_results."
+        )
+    )
+    parser.add_argument(
+        "--ta-path",
+        default=".",
+        help=(
+            "Path to the TA root directory (must contain a 'package/' subdirectory "
+            "or be the app directory itself). Defaults to the current directory."
+        ),
+    )
+    parser.add_argument(
+        "--splunk-version",
+        default="10.4",
+        help="Splunk Docker image tag (default: 10.4).",
+    )
+    parser.add_argument(
+        "--no-config",
+        action="store_true",
+        default=False,
+        help="Skip writing spl2_test_config.json.",
+    )
+    parser.add_argument(
+        "--config-dest",
+        default=_CONFIG_FILENAME,
+        help=f"Where to write the config file (default: {_CONFIG_FILENAME} in CWD).",
+    )
+
+    args = parser.parse_args()
+
+    start(
+        ta_path=Path(args.ta_path).resolve(),
+        splunk_version=args.splunk_version,
+        write_config=not args.no_config,
+        config_dest=Path(args.config_dest),
+    )
+
+
+if __name__ == "__main__":
+    run()

{spl2_testing_framework-1.8.3 → spl2_testing_framework-1.8.4}/spl2_testing_framework/tools/jobs/cli_job.py

@@ -19,6 +19,10 @@ from .job import Job
 class CLIAssertionJob(Job):
     """A job that is executed on CLI in unit tests."""
 
+    def __init__(self, *args, lookups_root_dir=None, **kwargs):
+        super().__init__(*args, **kwargs)
+        self.lookups_root_dir = lookups_root_dir
+
     def create(self) -> "CLIAssertionJob":
         """Creates a job. To run the job use run_job() method."""
 
@@ -30,6 +34,9 @@ class CLIAssertionJob(Job):
             self.test_name,
         ]
 
+        if self.lookups_root_dir is not None:
+            command.extend(["--lookupsRootDir", self.lookups_root_dir])
+
         input_data = "\n\n".join(
             [
                 (  # for assertion tests we don't add assertions module
@@ -56,10 +63,12 @@ class CLISimpleJob(Job):
         code_module_content,
         test_name,
         cli_bench=None,
+        lookups_root_dir=None,
     ):
         code_module_content += f"\n$source = from {str(source)};"
         super().__init__("", code_module_name, code_module_content, test_name)
         self.cli_bench = cli_bench
+        self.lookups_root_dir = lookups_root_dir
 
     def create(self) -> "CLISimpleJob":
         """Creates a job. To run the job use run_job() method."""
@@ -71,6 +80,9 @@ class CLISimpleJob(Job):
         if self.cli_bench is not None:
             command.extend(["-b", "-n", str(self.cli_bench)])
 
+        if self.lookups_root_dir is not None:
+            command.extend(["--lookupsRootDir", self.lookups_root_dir])
+
         input_data = "\n\n".join(
             [
                 self.commands_module,

{spl2_testing_framework-1.8.3 → spl2_testing_framework-1.8.4}/spl2_testing_framework/tools/search_clients/cli_search_client.py

@@ -29,21 +29,30 @@ class CLISearchClient(SearchClient):
 
     jobs_dict = {}
 
-    def __init__(self, cli_bench=None):
-        """Initialize CLISearchClient with optional bench parameter.
+    def __init__(self, cli_bench=None, lookups_root_dir=None):
+        """Initialize CLISearchClient with optional bench and lookups parameters.
 
         Args:
             cli_bench: Number of events to use for benchmarking (adds -b and -n flags)
+            lookups_root_dir: Root directory for lookups transforms.conf (adds --lookupsRootDir)
         """
         self.cli_bench = cli_bench
+        self.lookups_root_dir = lookups_root_dir
 
-    @staticmethod
     def create_job(
-        test_module: str, code_file_name: str, code_to_test: str, test_statement: str
+        self,
+        test_module: str,
+        code_file_name: str,
+        code_to_test: str,
+        test_statement: str,
     ) -> Job:
         """Creates a job for unit test run."""
         return CLIAssertionJob(
-            test_module, code_file_name, code_to_test, test_statement
+            test_module,
+            code_file_name,
+            code_to_test,
+            test_statement,
+            lookups_root_dir=self.lookups_root_dir,
         ).create()
 
     def create_simple_job(
@@ -52,7 +61,12 @@ class CLISearchClient(SearchClient):
         """Creates a job for box test run."""
 
         return CLISimpleJob(
-            source, code_module_name, code_module_content, test_name, self.cli_bench
+            source,
+            code_module_name,
+            code_module_content,
+            test_name,
+            self.cli_bench,
+            self.lookups_root_dir,
         ).create()
 
     def run_job(self, job):

{spl2_testing_framework-1.8.3 → spl2_testing_framework-1.8.4}/spl2_testing_framework/tools/spl2test_runner.py

@@ -159,6 +159,16 @@ class SPL2TestRunner:
 
     def run_box_test(self, box_test: BoxTest) -> None:
         """Run a box test"""
+        if box_test.skip_on:
+            if self.check_splunk_results == "TA" and "splunk_ta" in box_test.skip_on:
+                pytest.skip("Skipped for splunk_ta (skip_on)")
+            elif (
+                self.check_splunk_results == "CIM" and "splunk_cim" in box_test.skip_on
+            ):
+                pytest.skip("Skipped for splunk_cim (skip_on)")
+            elif not self.check_splunk_results and "standard" in box_test.skip_on:
+                pytest.skip("Skipped for standard (skip_on)")
+
         code_to_test = _make_functions_visible_for_testing(box_test.code_module_path)
 
         perf = PerformanceCheck(box_test)
@@ -203,9 +213,10 @@ class SPL2TestRunner:
                 ignore_keys = frozenset(
                     box_test.ignore_fields_in_destination_compare.keys()
                 )
-                for i, actual_result in enumerate(output_result):
-                    if i < len(box_test.output):
-                        actual_result.remove_fields_not_in_expected(box_test.output[i])
+                for actual_result, expected_result in zip(
+                    sorted(output_result), sorted(box_test.output)
+                ):
+                    actual_result.remove_fields_not_in_expected(expected_result)
 
                 if self.create_comparison_sheet:
                     self.comp_obj.create_comparison_sheet(output_result, box_test)
@@ -238,9 +249,10 @@ class SPL2TestRunner:
 
             if box_test.metrics:
                 # Remove fields from actual metrics that are not in expected metrics if configured
-                for i, actual_metric in enumerate(metrics_result):
-                    if i < len(box_test.metrics):
-                        actual_metric.remove_fields_not_in_expected(box_test.metrics[i])
+                for actual_metric, expected_metric in zip(
+                    sorted(metrics_result), sorted(box_test.metrics)
+                ):
+                    actual_metric.remove_fields_not_in_expected(expected_metric)
 
                 assert sorted(metrics_result) == sorted(box_test.metrics)
                 _LOGGER.info(f"Metric check passed: {box_test.name}")
@@ -259,14 +271,13 @@ class SPL2TestRunner:
 
         ``Results._cast_types`` updates dicts in place. Splunk-shaped rows are
         JSON-like; cloning via :func:`json.dumps` / :func:`json.loads` is
-        typically faster than :func:`copy.deepcopy`. Non-JSON values use
-        ``default=str`` (same idea as serializing for HEC). Falls back to
-        deepcopy only if JSON cloning fails.
+        typically faster than :func:`copy.deepcopy`. Falls back to deepcopy for
+        any row that contains non-JSON-serializable values.
         """
         if not d:
             return {}
         try:
-            return json.loads(json.dumps(d, default=str))
+            return json.loads(json.dumps(d))
         except (TypeError, ValueError):
             return copy.deepcopy(d)
 
@@ -339,7 +350,11 @@ class SPL2TestRunner:
                 expected_data = self._row_as_dict(expected)
                 # Splunk adds many search-time fields; mirror in-memory compare by
                 # dropping keys not present on the expected row before asserting.
-                actual_dict = dict(processed_event)
+                # Apply the same Results normalization (e.g. "True"->"true") that
+                # the non-Splunk path gets automatically via Results(_cast_types).
+                actual_dict = Results._cast_types(
+                    self._tree_copy_for_results(dict(processed_event))
+                )
                 Results._remove_extra_fields(actual_dict, expected_data)
                 actual_cmp = self._strip_ignored_fields_for_destination_compare(
                     actual_dict, ignore_keys

{spl2_testing_framework-1.8.3 → spl2_testing_framework-1.8.4}/spl2_testing_framework/tools/test_discovery.py

@@ -180,6 +180,34 @@ class BoxTestDiscovery(TestDiscovery):
     """Discovers tests for box tests, looking into module.test.json files"""
 
     _IGNORE_FIELDS_IN_DESTINATION_COMPARE_KEY = "ignore_fields_in_destination_compare"
+    _VALID_SKIP_ON_VALUES = frozenset({"standard", "splunk_ta", "splunk_cim"})
+
+    @classmethod
+    def _parse_skip_on(cls, test_block: dict) -> frozenset:
+        raw = test_block.get("skip_on")
+        if raw is None:
+            return frozenset()
+        if isinstance(raw, str):
+            raw = [raw]
+        if not isinstance(raw, list):
+            _LOGGER.warning(
+                "skip_on must be a string or list of strings; ignoring %r", raw
+            )
+            return frozenset()
+        result = set()
+        for v in raw:
+            if not isinstance(v, str):
+                _LOGGER.warning("skip_on entries must be strings; skipping %r", v)
+                continue
+            if v not in cls._VALID_SKIP_ON_VALUES:
+                _LOGGER.warning(
+                    "Unknown skip_on value %r; valid values: %s",
+                    v,
+                    sorted(cls._VALID_SKIP_ON_VALUES),
+                )
+                continue
+            result.add(v)
+        return frozenset(result)
 
     @classmethod
     def _parse_ignore_fields_in_destination_compare(cls, test_block: dict) -> dict:
@@ -255,6 +283,7 @@ class BoxTestDiscovery(TestDiscovery):
                 ignore_fields_in_destination_compare = (
                     self._parse_ignore_fields_in_destination_compare(test_block)
                 )
+                skip_on = self._parse_skip_on(test_block)
                 box_test = BoxTest(
                     name=name,
                     input=test_input,
@@ -264,6 +293,7 @@ class BoxTestDiscovery(TestDiscovery):
                     code_module_path=code_module_path,
                     cim_fields=cim_fields,
                     ignore_fields_in_destination_compare=ignore_fields_in_destination_compare,
+                    skip_on=skip_on,
                 )
                 box_test.meta["performance"] = self._performance_check
                 self.tests.add(box_test)

{spl2_testing_framework-1.8.3 → spl2_testing_framework-1.8.4}/spl2_testing_framework/tools/test_types.py

@@ -48,6 +48,8 @@ class BoxTest(Test):
     cim_fields: dict = field(default_factory=dict)
     # Field name -> reason; stripped on both sides before destination asserts (not used for CIM).
     ignore_fields_in_destination_compare: dict = field(default_factory=dict)
+    # Set of test modes to skip: "standard", "splunk_ta", "splunk_cim"
+    skip_on: frozenset = field(default_factory=frozenset)
 
 
 @dataclass