spl2-testing-framework 1.8.5__tar.gz → 1.8.7__tar.gz

{spl2_testing_framework-1.8.5 → spl2_testing_framework-1.8.7}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: spl2-testing-framework
-Version: 1.8.5
+Version: 1.8.7
 Summary: SPL2 Testing Framework
 Home-page: https://github.com/splunk/spl2-testing-framework
 Keywords: SPL2 testing framework
@@ -135,6 +135,7 @@ Additionally, the following options are supported:
 * `--code_dir` - directory where SPL2 code modules (`.spl2`) are located. Defaults to the current working directory. Falls back to `--test_dir` if the module is not found. See [Separate code and test directories](#separate-code-and-test-directories) for details.
 * `--ignore_empty_strings` - to ignore empty strings in the results
 * `--ignore_additional_fields_in_actual` - to ignore fields present in actual results but not in expected results (useful when actual results contain extra fields that should not affect comparison)
+* `--keep_tag_fields` - by default, fields whose name starts with `tag::` (e.g. `tag::metric_name`) are stripped from **both** the actual and expected rows before the destination compare. Pass `--keep_tag_fields` to disable this behaviour and include `tag::` fields in the comparison.
 * `--create_comparison_sheet` - to create a comparison sheet in `comparison_box_test` folder using actual and expected outputs (works only when running box tests)
 * `--cli_bench` - when running tests with CLI, this option enables benchmarking mode by adding `-b` and `-n` flags to the spl2-processor-cli command. The value specifies the number of events to test (e.g., `--cli_bench=1000` will add `-b -n 1000` to the CLI command). **In bench mode, all tests always succeed and CLI output is printed to logs** - this is useful for performance testing and benchmarking. This is only applicable when using `--type cli`.
 * `--lookups_root_dir` - root directory passed to `spl2-processor-cli` as `--lookupsRootDir` for resolving lookup `transforms.conf`. Defaults to `.` (current working directory). For a standard TA layout use `--lookups_root_dir=package/default`. Only applicable when using `--type cli`.
@@ -142,6 +143,7 @@ Additionally, the following options are supported:
 * `--reduction_stats` - when set, **box tests** record a reduction rate per run: input `_raw` vs output `_raw` (after pairing input/output events). Rates are grouped by **pipeline** (the `.spl2` code module stem) and **sourcetype** (from the input event if set, else the output event, else `UNKNOWN`). Off by default.
 * `--reduction_stats_file` - path to a JSON file written at the **end of the pytest session** with collected rates (default `reduction_stats.json`). Only used when `--reduction_stats` is set.
 * `--reduction_stats_full` - when set together with `--reduction_stats`, each pipeline’s JSON payload also includes a pandas **`describe()`** summary (count, mean, std, min, quartiles, max) per sourcetype column. Only used when `--reduction_stats` is set.
+* `--include_placeholder_tests` - include tests whose source contains unresolved `##name##` placeholders. By default such tests are skipped with a warning.
 
 * Note: The `pytest.ini.sample` file allows you to define command parameters. Just update the configurations, rename the
 file by removing the `.sample` extension, and execute the command.

{spl2_testing_framework-1.8.5 → spl2_testing_framework-1.8.7}/README.md

@@ -102,6 +102,7 @@ Additionally, the following options are supported:
 * `--code_dir` - directory where SPL2 code modules (`.spl2`) are located. Defaults to the current working directory. Falls back to `--test_dir` if the module is not found. See [Separate code and test directories](#separate-code-and-test-directories) for details.
 * `--ignore_empty_strings` - to ignore empty strings in the results
 * `--ignore_additional_fields_in_actual` - to ignore fields present in actual results but not in expected results (useful when actual results contain extra fields that should not affect comparison)
+* `--keep_tag_fields` - by default, fields whose name starts with `tag::` (e.g. `tag::metric_name`) are stripped from **both** the actual and expected rows before the destination compare. Pass `--keep_tag_fields` to disable this behaviour and include `tag::` fields in the comparison.
 * `--create_comparison_sheet` - to create a comparison sheet in `comparison_box_test` folder using actual and expected outputs (works only when running box tests)
 * `--cli_bench` - when running tests with CLI, this option enables benchmarking mode by adding `-b` and `-n` flags to the spl2-processor-cli command. The value specifies the number of events to test (e.g., `--cli_bench=1000` will add `-b -n 1000` to the CLI command). **In bench mode, all tests always succeed and CLI output is printed to logs** - this is useful for performance testing and benchmarking. This is only applicable when using `--type cli`.
 * `--lookups_root_dir` - root directory passed to `spl2-processor-cli` as `--lookupsRootDir` for resolving lookup `transforms.conf`. Defaults to `.` (current working directory). For a standard TA layout use `--lookups_root_dir=package/default`. Only applicable when using `--type cli`.
@@ -109,6 +110,7 @@ Additionally, the following options are supported:
 * `--reduction_stats` - when set, **box tests** record a reduction rate per run: input `_raw` vs output `_raw` (after pairing input/output events). Rates are grouped by **pipeline** (the `.spl2` code module stem) and **sourcetype** (from the input event if set, else the output event, else `UNKNOWN`). Off by default.
 * `--reduction_stats_file` - path to a JSON file written at the **end of the pytest session** with collected rates (default `reduction_stats.json`). Only used when `--reduction_stats` is set.
 * `--reduction_stats_full` - when set together with `--reduction_stats`, each pipeline’s JSON payload also includes a pandas **`describe()`** summary (count, mean, std, min, quartiles, max) per sourcetype column. Only used when `--reduction_stats` is set.
+* `--include_placeholder_tests` - include tests whose source contains unresolved `##name##` placeholders. By default such tests are skipped with a warning.
 
 * Note: The `pytest.ini.sample` file allows you to define command parameters. Just update the configurations, rename the
 file by removing the `.sample` extension, and execute the command.

{spl2_testing_framework-1.8.5 → spl2_testing_framework-1.8.7}/pyproject.toml

@@ -15,7 +15,7 @@
 
 [tool.poetry]
 name = "spl2-testing-framework"
-version = "1.8.5"
+version = "1.8.7"
 description = "SPL2 Testing Framework"
 authors = ["mgazda-splunk <mgazda@splunk.com>"]
 readme = "README.md"

{spl2_testing_framework-1.8.5 → spl2_testing_framework-1.8.7}/spl2_testing_framework/conftest.py

@@ -52,6 +52,7 @@ SINGLE_SPL2_FILE = "single_spl2_file"
 
 IGNORE_EMPTY_STRINGS = "ignore_empty_strings"
 IGNORE_ADDITIONAL_FIELDS_IN_ACTUAL = "ignore_additional_fields_in_actual"
+KEEP_TAG_FIELDS = "keep_tag_fields"
 CREATE_COMPARISON_SHEET = "create_comparison_sheet"
 CHECK_SPLUNK_RESULTS = "check_splunk_results"
 PERFORMANCE_CHECK = "performance_check"
@@ -62,6 +63,7 @@ LIMIT_TESTS = "limit_tests"
 CLI_BENCH = "cli_bench"
 LOOKUPS_ROOT_DIR = "lookups_root_dir"
 REDUCTION_STATS = "reduction_stats"
+INCLUDE_PLACEHOLDER_TESTS = "include_placeholder_tests"
 
 setup_logging()
 
@@ -87,6 +89,12 @@ def pytest_addoption(parser):
     parser.addoption(f"--{CREATE_COMPARISON_SHEET}", action="store_true", default=False)
     parser.addoption(f"--{IGNORE_EMPTY_STRINGS}", action="store_true")
     parser.addoption(f"--{IGNORE_ADDITIONAL_FIELDS_IN_ACTUAL}", action="store_true")
+    parser.addoption(
+        f"--{KEEP_TAG_FIELDS}",
+        action="store_true",
+        default=False,
+        help="Do not strip tag:: fields from destination compare. By default, fields whose name starts with 'tag::' are excluded from the comparison.",
+    )
     parser.addoption(f"--{LIMIT_TESTS}", action="store", type=int, default=None)
     parser.addoption(f"--{CLI_BENCH}", action="store", type=int, default=None)
     parser.addoption(
@@ -114,6 +122,12 @@ def pytest_addoption(parser):
         default=False,
         help="Include full dataframe stats (count, mean, std, min, 25%%, 50%%, 75%%, max) in the saved file. Used only when --reduction_stats is set.",
     )
+    parser.addoption(
+        f"--{INCLUDE_PLACEHOLDER_TESTS}",
+        action="store_true",
+        default=False,
+        help="Include tests whose source contains unresolved ##name## placeholders. By default such tests are skipped with a warning.",
+    )
 
 
 def pytest_generate_tests(metafunc):
@@ -123,12 +137,17 @@ def pytest_generate_tests(metafunc):
     code_dir = metafunc.config.getoption(CODE_DIR)
     performance_check = metafunc.config.getoption(PERFORMANCE_CHECK)
     limit_tests = metafunc.config.getoption(LIMIT_TESTS)
+    include_placeholder_tests = metafunc.config.getoption(INCLUDE_PLACEHOLDER_TESTS)
     _set_results_options(metafunc)
 
     if BOX_TEST in metafunc.fixturenames:
         _LOGGER.info("Collecting Box Tests")
         module_tests = BoxTestDiscovery(
-            test_dir, performance_check, limit_tests, code_dir=code_dir
+            test_dir,
+            performance_check,
+            limit_tests,
+            code_dir=code_dir,
+            include_placeholder_tests=include_placeholder_tests,
         )
         module_tests.discover_tests()
         metafunc.parametrize(
@@ -433,3 +452,5 @@ def _set_results_options(metafunc):
         IGNORE_ADDITIONAL_FIELDS_IN_ACTUAL
     )
     Results._ignore_additional_fields_in_actual = ignore_fields_in_actual
+    keep_tag_fields = metafunc.config.getoption(KEEP_TAG_FIELDS)
+    Results._ignore_tag_fields = not keep_tag_fields

{spl2_testing_framework-1.8.5 → spl2_testing_framework-1.8.7}/spl2_testing_framework/tools/results.py

@@ -26,6 +26,7 @@ _WHOLE_FLOAT_STR_RE = re.compile(r"^(-?\d+)\.0+$")
 class Results(UserDict):
     _ignore_empty_strings = False
     _ignore_additional_fields_in_actual = False
+    _ignore_tag_fields = True
     """Class responsible for parsing and storing results from Splunk, IP and CLI.
     It's necessary to parse nested jsons and other python types to strings in order to compare them
     """
@@ -201,6 +202,15 @@ class Results(UserDict):
             self._remove_extra_fields(self.data, expected.data)
         return self
 
+    @staticmethod
+    def is_ignored_in_destination_compare(key: str) -> bool:
+        """Return True if ``key`` should be dropped from destination comparison regardless of
+        the per-test ``ignore_fields_in_destination_compare`` list.  Currently this covers
+        ``tag::*`` fields, which are stripped by default and only kept when
+        ``--keep_tag_fields`` is passed.
+        """
+        return Results._ignore_tag_fields and key.startswith("tag::")
+
     def __lt__(self, other):
         if "_raw" in self.data:
             s = self.data.get("_raw")

{spl2_testing_framework-1.8.5 → spl2_testing_framework-1.8.7}/spl2_testing_framework/tools/spl2test_runner.py

@@ -220,7 +220,7 @@ class SPL2TestRunner:
 
                 if self.create_comparison_sheet:
                     self.comp_obj.create_comparison_sheet(output_result, box_test)
-                if ignore_keys:
+                if ignore_keys or Results.is_ignored_in_destination_compare("tag::"):
                     stripped_actual = [
                         Results(
                             self._strip_ignored_fields_for_destination_compare(
@@ -285,9 +285,11 @@ class SPL2TestRunner:
     def _strip_ignored_fields_for_destination_compare(
         row: dict, ignore_keys: frozenset
     ) -> dict:
-        if not ignore_keys:
-            return row
-        return {k: v for k, v in row.items() if k not in ignore_keys}
+        return {
+            k: v
+            for k, v in row.items()
+            if k not in ignore_keys and not Results.is_ignored_in_destination_compare(k)
+        }
 
     def _splunk_process_event(self, box_test: BoxTest, event):
         try:

{spl2_testing_framework-1.8.5 → spl2_testing_framework-1.8.7}/spl2_testing_framework/tools/test_discovery.py

@@ -50,11 +50,13 @@ class TestDiscovery(abc.ABC):
         performance_check: str,
         limit_tests: Optional[int] = None,
         code_dir: Optional[str] = None,
+        include_placeholder_tests: bool = False,
     ):
         self.path = pathlib.Path(path)
         self.code_dir = pathlib.Path(code_dir) if code_dir else self.path
         self._performance_check = performance_check
         self.limit_tests = limit_tests
+        self.include_placeholder_tests = include_placeholder_tests
         self.tests = TestSuite()
 
     @property
@@ -181,6 +183,7 @@ class BoxTestDiscovery(TestDiscovery):
 
     _IGNORE_FIELDS_IN_DESTINATION_COMPARE_KEY = "ignore_fields_in_destination_compare"
     _VALID_SKIP_ON_VALUES = frozenset({"standard", "splunk_ta", "splunk_cim"})
+    _PLACEHOLDER_PATTERN = re.compile(r"##\w+##")
 
     @classmethod
     def _parse_skip_on(cls, test_block: dict) -> frozenset:
@@ -268,6 +271,18 @@ class BoxTestDiscovery(TestDiscovery):
                 test_block = entry["test"]
                 name = entry["filename"].replace(".spl2", "")
                 test_input = test_block["source"]
+                if (
+                    isinstance(test_input, str)
+                    and self._PLACEHOLDER_PATTERN.search(test_input)
+                    and not self.include_placeholder_tests
+                ):
+                    _LOGGER.warning(
+                        "Skipping test '%s' in '%s': source contains unresolved placeholder(s) (##name##). "
+                        "Use --include_placeholder_tests to force-include these tests.",
+                        name,
+                        module_path,
+                    )
+                    continue
                 code_module_path = self._resolve_code_module(entry["filename"])
                 output = [
                     Results(x)