spl2-testing-framework 1.6.0__tar.gz → 1.7.0__tar.gz

{spl2_testing_framework-1.6.0 → spl2_testing_framework-1.7.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.1
 Name: spl2-testing-framework
-Version: 1.6.0
+Version: 1.7.0
 Summary: SPL2 Testing Framework
 Home-page: https://github.com/splunk/spl2-testing-framework
 Keywords: SPL2 testing framework
@@ -131,6 +131,8 @@ It is possible to pass more options to the command, which works also with pytest
 
 Additionally, the following options are supported:
 
+* `--test_dir` - directory where test files (`.test.json`, `.test.spl2`, `module.json`) are located. Defaults to the current working directory. Discovery is recursive.
+* `--code_dir` - directory where SPL2 code modules (`.spl2`) are located. Defaults to the current working directory. Falls back to `--test_dir` if the module is not found. See [Separate code and test directories](#separate-code-and-test-directories) for details.
 * `--ignore_empty_strings` - to ignore empty strings in the results
 * `--ignore_additional_fields_in_actual` - to ignore fields present in actual results but not in expected results (useful when actual results contain extra fields that should not affect comparison)
 * `--create_comparison_sheet` - to create a comparison sheet in `comparison_box_test` folder using actual and expected outputs (works only when running box tests)
@@ -152,10 +154,53 @@ Sample configuration which may be used:
     - Parameters:
       `--type [cli | splunk | cloud] --test_dir /tests/resources -o log_cli=true --log-cli-level=INFO --verbose`
     - If test dir is not specified, current working directory will be used
+    - Use `--code_dir` if SPL2 code modules are in a separate directory from tests
     - If necessary another pytest options can be added
 
 Note: It's necessary to set "pytest" as default test runner in PyCharm settings
 
+### Separate code and test directories
+
+By default, the framework expects SPL2 code modules (`.spl2` files) to live alongside their test files. The `--code_dir` option allows you to keep code and tests in separate directory trees.
+
+#### How it works
+
+| What is searched | Primary directory | Fallback directory |
+|------------------|-------------------|--------------------|
+| **Code modules** (`.spl2`) | `--code_dir` | `--test_dir` |
+| **Test files** (`.test.json`, `.test.spl2`) | `--test_dir` | `--code_dir` |
+
+Both searches are recursive. If the primary directory has no results, the framework falls back to the other directory and logs the action.
+
+When multiple files with the same name are found within a directory, a warning is logged and the lexicographically first match is used.
+
+#### Example project layout
+
+```
+my-project/
+├── src/                          # --code_dir
+│   ├── network_traffic.spl2
+│   └── dns_lookup.spl2
+└── tests/                        # --test_dir
+    ├── network_traffic.test.json
+    ├── dns_lookup.test.json
+    └── dns_lookup.test.spl2
+```
+
+Run with:
+
+```bash
+spl2_tests_run cli --test_dir tests --code_dir src
+```
+
+#### Backward compatibility
+
+When `--code_dir` is not specified (or both flags point to the same directory), the framework behaves exactly as before — code modules are found next to their test files via the same recursive search.
+
+#### Applies to all test types
+
+The `--code_dir` option works for box tests, unit tests, and single SPL2 file execution.
+
 ## Executing a spl2 file
 
 This framework also supports executing a single spl2 file and prints the results in command line as well as a log file.
@@ -179,6 +224,7 @@ It is possible to pass more options to the command, which works also with pytest
 
 * `--test_dir` - Path in which the template file and module.json are available. If not provided, it will look for the
   current directory for the template file and module.json file
+* `--code_dir` - Path where SPL2 code modules are located, if different from `--test_dir`. Falls back to `--test_dir` when not set. See [Separate code and test directories](#separate-code-and-test-directories).
 * `--template_file` - The spl2 template file to execute
 * `--sample_file` - A file containing all the samples required for the template file. If not provided, it will look for
   the samples in module.json file of the corresponding template file

{spl2_testing_framework-1.6.0 → spl2_testing_framework-1.7.0}/README.md

@@ -98,6 +98,8 @@ It is possible to pass more options to the command, which works also with pytest
 
 Additionally, the following options are supported:
 
+* `--test_dir` - directory where test files (`.test.json`, `.test.spl2`, `module.json`) are located. Defaults to the current working directory. Discovery is recursive.
+* `--code_dir` - directory where SPL2 code modules (`.spl2`) are located. Defaults to the current working directory. Falls back to `--test_dir` if the module is not found. See [Separate code and test directories](#separate-code-and-test-directories) for details.
 * `--ignore_empty_strings` - to ignore empty strings in the results
 * `--ignore_additional_fields_in_actual` - to ignore fields present in actual results but not in expected results (useful when actual results contain extra fields that should not affect comparison)
 * `--create_comparison_sheet` - to create a comparison sheet in `comparison_box_test` folder using actual and expected outputs (works only when running box tests)
@@ -119,10 +121,53 @@ Sample configuration which may be used:
     - Parameters:
       `--type [cli | splunk | cloud] --test_dir /tests/resources -o log_cli=true --log-cli-level=INFO --verbose`
     - If test dir is not specified, current working directory will be used
+    - Use `--code_dir` if SPL2 code modules are in a separate directory from tests
     - If necessary another pytest options can be added
 
 Note: It's necessary to set "pytest" as default test runner in PyCharm settings
 
+### Separate code and test directories
+
+By default, the framework expects SPL2 code modules (`.spl2` files) to live alongside their test files. The `--code_dir` option allows you to keep code and tests in separate directory trees.
+
+#### How it works
+
+| What is searched | Primary directory | Fallback directory |
+|------------------|-------------------|--------------------|
+| **Code modules** (`.spl2`) | `--code_dir` | `--test_dir` |
+| **Test files** (`.test.json`, `.test.spl2`) | `--test_dir` | `--code_dir` |
+
+Both searches are recursive. If the primary directory has no results, the framework falls back to the other directory and logs the action.
+
+When multiple files with the same name are found within a directory, a warning is logged and the lexicographically first match is used.
+
+#### Example project layout
+
+```
+my-project/
+├── src/                          # --code_dir
+│   ├── network_traffic.spl2
+│   └── dns_lookup.spl2
+└── tests/                        # --test_dir
+    ├── network_traffic.test.json
+    ├── dns_lookup.test.json
+    └── dns_lookup.test.spl2
+```
+
+Run with:
+
+```bash
+spl2_tests_run cli --test_dir tests --code_dir src
+```
+
+#### Backward compatibility
+
+When `--code_dir` is not specified (or both flags point to the same directory), the framework behaves exactly as before — code modules are found next to their test files via the same recursive search.
+
+#### Applies to all test types
+
+The `--code_dir` option works for box tests, unit tests, and single SPL2 file execution.
+
 ## Executing a spl2 file
 
 This framework also supports executing a single spl2 file and prints the results in command line as well as a log file.
@@ -146,6 +191,7 @@ It is possible to pass more options to the command, which works also with pytest
 
 * `--test_dir` - Path in which the template file and module.json are available. If not provided, it will look for the
   current directory for the template file and module.json file
+* `--code_dir` - Path where SPL2 code modules are located, if different from `--test_dir`. Falls back to `--test_dir` when not set. See [Separate code and test directories](#separate-code-and-test-directories).
 * `--template_file` - The spl2 template file to execute
 * `--sample_file` - A file containing all the samples required for the template file. If not provided, it will look for
   the samples in module.json file of the corresponding template file

{spl2_testing_framework-1.6.0 → spl2_testing_framework-1.7.0}/pyproject.toml

@@ -15,7 +15,7 @@
 
 [tool.poetry]
 name = "spl2-testing-framework"
-version = "1.6.0"
+version = "1.7.0"
 description = "SPL2 Testing Framework"
 authors = ["mgazda-splunk <mgazda@splunk.com>"]
 readme = "README.md"

{spl2_testing_framework-1.6.0 → spl2_testing_framework-1.7.0}/spl2_testing_framework/conftest.py

@@ -55,6 +55,7 @@ CHECK_SPLUNK_RESULTS = "check_splunk_results"
 PERFORMANCE_CHECK = "performance_check"
 TEST_TYPE = "type"
 TEST_DIR = "test_dir"
+CODE_DIR = "code_dir"
 LIMIT_TESTS = "limit_tests"
 CLI_BENCH = "cli_bench"
 
@@ -70,6 +71,7 @@ _LOGGER = logging.getLogger(__name__)
 def pytest_addoption(parser):
     parser.addoption(f"--{TEST_TYPE}", action="store", default="cli")
     parser.addoption(f"--{TEST_DIR}", action="store", default=".")
+    parser.addoption(f"--{CODE_DIR}", action="store", default=".")
     parser.addoption(
         f"--{PERFORMANCE_CHECK}",
         action="store",
@@ -93,13 +95,16 @@ def pytest_generate_tests(metafunc):
     """This function is used for dynamic generation of tests based on the test files in the directory specified by the
     --test_dir option"""
     test_dir = metafunc.config.getoption(TEST_DIR)
+    code_dir = metafunc.config.getoption(CODE_DIR)
     performance_check = metafunc.config.getoption(PERFORMANCE_CHECK)
     limit_tests = metafunc.config.getoption(LIMIT_TESTS)
     _set_results_options(metafunc)
 
     if BOX_TEST in metafunc.fixturenames:
         _LOGGER.info("Collecting Box Tests")
-        module_tests = BoxTestDiscovery(test_dir, performance_check, limit_tests)
+        module_tests = BoxTestDiscovery(
+            test_dir, performance_check, limit_tests, code_dir=code_dir
+        )
         module_tests.discover_tests()
         metafunc.parametrize(
             "%s" % BOX_TEST, module_tests.tests, ids=module_tests.tests.get_ids()
@@ -119,6 +124,7 @@ def pytest_generate_tests(metafunc):
             sample_delimiter,
             performance_check=performance_check,
             limit_tests=limit_tests,
+            code_dir=code_dir,
         )
         try:
             spl2_pipelines.discover_tests()
@@ -132,7 +138,7 @@ def pytest_generate_tests(metafunc):
 
     if UNIT_TEST in metafunc.fixturenames:
         _LOGGER.info("Collecting Unit Tests")
-        unit_tests = UTDiscovery(test_dir, limit_tests)
+        unit_tests = UTDiscovery(test_dir, limit_tests, code_dir=code_dir)
         unit_tests.discover_tests()
         metafunc.parametrize(
             UNIT_TEST, unit_tests.tests, ids=unit_tests.tests.get_ids()

{spl2_testing_framework-1.6.0 → spl2_testing_framework-1.7.0}/spl2_testing_framework/tools/test_discovery.py

@@ -19,6 +19,8 @@ import logging
 import os
 import pathlib
 import re
+from collections import Counter
+from typing import Optional
 
 from spl2_testing_framework.tools.results import Metrics, Results
 from spl2_testing_framework.tools.test_types import (
@@ -34,33 +36,151 @@ _LOGGER = logging.getLogger(__name__)
 class SamplesNotFoundException(Exception):
     """An Exception to be raised if samples are not found"""
 
-    pass
+
+class CodeModuleNotFoundError(Exception):
+    """Raised when a code module cannot be found in any search directory"""
 
 
 class TestDiscovery(abc.ABC):
     """Base class for test discovery classes"""
 
-    def __init__(self, path: str, performance_check: str, limit_tests: int = None):
+    def __init__(
+        self,
+        path: str,
+        performance_check: str,
+        limit_tests: Optional[int] = None,
+        code_dir: Optional[str] = None,
+    ):
         self.path = pathlib.Path(path)
+        self.code_dir = pathlib.Path(code_dir) if code_dir else self.path
         self._performance_check = performance_check
         self.limit_tests = limit_tests
         self.tests = TestSuite()
 
+    @property
+    def _dirs_differ(self) -> bool:
+        try:
+            return self._dirs_differ_cached
+        except AttributeError:
+            self._dirs_differ_cached = self.code_dir.resolve() != self.path.resolve()
+            return self._dirs_differ_cached
+
+    @staticmethod
+    def _warn_duplicate_basenames(
+        paths: list, directory: pathlib.Path, label: str
+    ) -> None:
+        names = [p.name for p in paths]
+        dupes = [name for name, count in Counter(names).items() if count > 1]
+        if dupes:
+            _LOGGER.warning(
+                "Multiple %s files with the same name found in '%s': %s",
+                label,
+                directory,
+                dupes,
+            )
+
     def _read_tests(self, test_name: str) -> None:
-        tests_paths = self.path.rglob(test_name)
+        _LOGGER.info(
+            "Searching for test files '%s' in test directory: %s",
+            test_name,
+            self.path,
+        )
+        tests_paths = list(self.path.rglob(test_name))
+        if tests_paths:
+            self._warn_duplicate_basenames(tests_paths, self.path, "test")
+            _LOGGER.info(
+                "Found %d test file(s) in test directory: %s",
+                len(tests_paths),
+                [str(p) for p in tests_paths],
+            )
+        elif self._dirs_differ:
+            _LOGGER.info(
+                "No test files '%s' found in test directory '%s', "
+                "falling back to code directory: %s",
+                test_name,
+                self.path,
+                self.code_dir,
+            )
+            tests_paths = list(self.code_dir.rglob(test_name))
+            if tests_paths:
+                self._warn_duplicate_basenames(tests_paths, self.code_dir, "test")
+                _LOGGER.info(
+                    "Found %d test file(s) in code directory: %s",
+                    len(tests_paths),
+                    [str(p) for p in tests_paths],
+                )
+            else:
+                _LOGGER.warning(
+                    "No test files '%s' found in code directory either.", test_name
+                )
+        else:
+            _LOGGER.warning("No test files '%s' found in test directory.", test_name)
+
         self._raw_tests = {
             x: pathlib.Path(x).read_text(encoding="utf-8") for x in tests_paths
         }
 
+    def _search_directory(
+        self, filename: str, directory: pathlib.Path, label: str
+    ) -> Optional[pathlib.Path]:
+        """Search a directory recursively for *filename*.
+
+        Returns the first match (sorted lexicographically) or ``None``.
+        Warns when multiple matches exist.
+        """
+        matches = sorted(directory.rglob(filename))
+        if not matches:
+            return None
+        if len(matches) > 1:
+            _LOGGER.warning(
+                "Multiple matches for code module '%s' in %s '%s': %s. "
+                "Using first match.",
+                filename,
+                label,
+                directory,
+                [str(m) for m in matches],
+            )
+        _LOGGER.info("Resolved code module (%s): %s", label, matches[0])
+        return matches[0]
+
+    def _resolve_code_module(self, filename: str) -> pathlib.Path:
+        """Resolve a code module by filename.
+
+        Searches code_dir first (recursive), falls back to test_dir.
+        Warns when multiple files share the same name in a directory.
+        """
+        _LOGGER.info(
+            "Searching for code module '%s' in code directory: %s",
+            filename,
+            self.code_dir,
+        )
+        result = self._search_directory(filename, self.code_dir, "code directory")
+        if result is not None:
+            return result
+
+        if self._dirs_differ:
+            _LOGGER.info(
+                "Code module '%s' not found in code directory '%s', "
+                "falling back to test directory: %s",
+                filename,
+                self.code_dir,
+                self.path,
+            )
+            result = self._search_directory(filename, self.path, "test directory")
+            if result is not None:
+                return result
+
+        raise CodeModuleNotFoundError(
+            f"Code module '{filename}' not found in code directory '{self.code_dir}' "
+            f"or test directory '{self.path}'"
+        )
+
 
 class BoxTestDiscovery(TestDiscovery):
     """Discovers tests for box tests, looking into module.test.json files"""
 
     _IGNORE_FIELDS_IN_SPLUNK_CHECK_KEY = "ignore_fields_in_splunk_check"
 
-    def __init__(self, path: str, performance_check: str, limit_tests: int = None):
-        super().__init__(path, performance_check, limit_tests)
-
     @classmethod
     def _parse_ignore_fields_in_splunk_check(cls, test_block: dict) -> dict:
         raw = test_block.get(cls._IGNORE_FIELDS_IN_SPLUNK_CHECK_KEY)
@@ -110,7 +230,6 @@ class BoxTestDiscovery(TestDiscovery):
         }
 
         for module_path, content in self._parsed_modules.items():
-            path = module_path.parent
             tests_added = 0
             for entry in content:
                 if self.limit_tests is not None and tests_added >= self.limit_tests:
@@ -121,7 +240,7 @@ class BoxTestDiscovery(TestDiscovery):
                 test_block = entry["test"]
                 name = entry["filename"].replace(".spl2", "")
                 test_input = test_block["source"]
-                code_module_path = path / entry["filename"]
+                code_module_path = self._resolve_code_module(entry["filename"])
                 output = [
                     Results(x)
                     for x in test_block.get("expected_destination_result", [])
@@ -161,9 +280,10 @@ class SingleSPL2Discovery(TestDiscovery):
         sample_file=None,
         sample_delimiter="\n",
         performance_check="no",
-        limit_tests: int = None,
+        limit_tests: Optional[int] = None,
+        code_dir: Optional[str] = None,
     ):
-        super().__init__(path, performance_check, limit_tests)
+        super().__init__(path, performance_check, limit_tests, code_dir=code_dir)
         self.template_file = template_file
         self.sample_file = sample_file
         self.sample_delimiter = sample_delimiter
@@ -180,7 +300,9 @@ class SingleSPL2Discovery(TestDiscovery):
             _LOGGER.info("Reading samples from provided sample file.")
             events = self._read_samples_from_file(delimiter=self.sample_delimiter)
             name = self.template_file.replace(".spl2", "")
-            code_module_path = next(self.path.rglob(self.template_file), None)
+            code_module_path = self._resolve_code_module(
+                os.path.basename(self.template_file)
+            )
         else:
             _LOGGER.info("Reading samples from corresponding module.json file")
             self._parsed_modules = {
@@ -188,12 +310,11 @@ class SingleSPL2Discovery(TestDiscovery):
             }
             events = None
             for module_path, content in self._parsed_modules.items():
-                path = module_path.parent
                 break_outer = False
                 for test in content:
                     if test["filename"] == os.path.basename(self.template_file):
                         name = test["filename"].replace(".spl2", "")
-                        code_module_path = path / test["filename"]
+                        code_module_path = self._resolve_code_module(test["filename"])
                         events = test["context"]["events"]
                         if isinstance(events, str):
                             events = events.split("\r\n")
@@ -246,8 +367,15 @@ class SingleSPL2Discovery(TestDiscovery):
 class UTDiscovery(TestDiscovery):
     """Discovers tests for unit tests, looking into .test.spl2 files"""
 
-    def __init__(self, path: str, limit_tests: int = None):
-        super().__init__(path, performance_check="no", limit_tests=limit_tests)
+    def __init__(
+        self,
+        path: str,
+        limit_tests: Optional[int] = None,
+        code_dir: Optional[str] = None,
+    ):
+        super().__init__(
+            path, performance_check="no", limit_tests=limit_tests, code_dir=code_dir
+        )
 
     def discover_tests(self) -> None:
         self._read_tests("*.test.spl2")
@@ -264,7 +392,7 @@ class UTDiscovery(TestDiscovery):
             file_name = path.name
             content = path.read_text(encoding="utf-8")
             code_module_name = file_name.replace(".test.", ".")
-            code_module_path = path.parent / code_module_name
+            code_module_path = self._resolve_code_module(code_module_name)
 
             tests_added = 0
             for test in tests: