cli-test-framework 0.5.3__tar.gz → 0.8.0__tar.gz

cli_test_framework-0.8.0/PKG-INFO

@@ -0,0 +1,232 @@
+Metadata-Version: 2.4
+Name: cli-test-framework
+Version: 0.8.0
+Summary: A powerful command line testing framework in Python with setup modules, parallel execution, and file comparison capabilities.
+Home-page: https://github.com/ozil111/cli-test-framework
+Author: Xiaotong Wang
+Author-email: xiaotongwang98@gmail.com
+Project-URL: Documentation, https://github.com/ozil111/cli-test-framework/blob/main/docs/user_manual.md
+Project-URL: Source, https://github.com/ozil111/cli-test-framework
+Project-URL: Tracker, https://github.com/ozil111/cli-test-framework/issues
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Topic :: Software Development :: Testing
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+Requires-Dist: dukpy==0.5.0
+Requires-Dist: h5py<4.0.0,>=3.11.0
+Requires-Dist: numpy>=1.21.0; python_version < "3.12"
+Requires-Dist: numpy>=1.26.0; python_version >= "3.12"
+Requires-Dist: setuptools>=75.8.0
+Requires-Dist: wheel>=0.45.1
+Dynamic: author
+Dynamic: author-email
+Dynamic: classifier
+Dynamic: description
+Dynamic: description-content-type
+Dynamic: home-page
+Dynamic: project-url
+Dynamic: requires-dist
+Dynamic: requires-python
+Dynamic: summary
+
+# CLI Test Framework
+
+A lightweight automated testing framework for command-line applications. Define test cases in JSON/YAML, run all validations with a single command.
+
+Particularly suited for scientific computing — deep HDF5 support with regex table matching, data filtering, and tolerance-based comparison, making simulation result verification effortless.
+
+## Why this exists
+
+This project was created from a real need: regression testing finite-element software.
+
+In solver-style projects, checking the exit code is not enough. A test often needs to run multiple commands, generate numerical result files, compare HDF5/CSV outputs with tolerances, and make sure old cases do not become slower over time.
+
+CLI Test Framework was built for that workflow.
+
+## Highlights
+
+- **Golden File Assertion** — `compare_files` embedded in test `expected`, compares output files against baselines with tolerance
+- **Parallel Execution** — Multi-thread / multi-process, 3-5x speedup
+- **Resource-Aware Scheduling** — Automatic CPU core management, prevents solver thread runaway
+- **Sequence Steps** — Multi-step execution within a single test case, fail-fast
+- **Setup Module** — Auto-configure environment variables before tests, auto-cleanup after
+- **File Comparison** — Text / JSON / CSV / XML / HDF5 / Binary, with CLI and embedded assertion support
+- **Filtered Execution** — Run specific test cases by name
+- **JUnit XML Output** — `--junit-xml` for GitLab CI / Jenkins / CircleCI test report panels
+
+## Quick Start
+
+```bash
+pip install cli-test-framework
+```
+
+### 30-Second Setup
+
+1. Create `test_cases.json`:
+
+```json
+{
+    "test_cases": [
+        {
+            "name": "hello",
+            "command": "echo",
+            "args": ["Hello World"],
+            "expected": {
+                "return_code": 0,
+                "output_contains": ["Hello World"]
+            }
+        }
+    ]
+}
+```
+
+2. Run:
+
+```bash
+cli-test run test_cases.json
+```
+
+### Golden File Comparison in Tests
+
+Run a simulation, then compare its output file against a reference:
+
+```json
+{
+    "test_cases": [
+        {
+            "name": "FEA displacement check",
+            "command": "my_solver",
+            "args": ["--input", "case1.dat", "--output", "out.h5"],
+            "expected": {
+                "return_code": 0,
+                "compare_files": [
+                    {
+                        "actual": "out.h5",
+                        "baseline": "ref/golden.h5",
+                        "rtol": 1e-5,
+                        "atol": 1e-8,
+                        "tables": ["NASTRAN/RESULT/NODAL/DISPLACEMENT"]
+                    }
+                ]
+            }
+        }
+    ]
+}
+```
+
+- `actual` – file produced by the command
+- `baseline` – reference file to compare against
+- `type` – comparator type (auto-detected from extension if omitted: `.h5`→h5, `.json`→json, `.csv`→csv, `.xml`→xml, `.txt`→text)
+- all other keys are forwarded as comparator parameters (`rtol`, `atol`, `tables`, `table_regex`, `data_filter`, `encoding`, `structure_only`, `delimiter`, `compare_mode`, `key_field`, etc.)
+
+Multiple files and mixed assertion types coexist naturally:
+
+```json
+{
+    "expected": {
+        "return_code": 0,
+        "output_contains": ["simulation finished"],
+        "compare_files": [
+            {"actual": "out.h5", "baseline": "ref/disp.h5", "rtol": 1e-5},
+            {"actual": "report.csv", "baseline": "ref/expected.csv", "rtol": 1e-6}
+        ]
+    }
+}
+```
+
+### Parallel Execution
+
+```bash
+cli-test run test_cases.json --parallel --workers 4
+```
+
+### JUnit XML (CI Integration)
+
+```bash
+cli-test run test_cases.json --junit-xml report.xml
+```
+
+In GitLab CI:
+
+```yaml
+# .gitlab-ci.yml
+artifacts:
+  reports:
+    junit: report.xml
+```
+
+### Python API
+
+```python
+from cli_test_framework.runners import JSONRunner, ParallelJSONRunner
+
+# Sequential
+runner = JSONRunner(config_file="test_cases.json")
+success = runner.run_tests()
+
+# Parallel
+runner = ParallelJSONRunner(config_file="test_cases.json", max_workers=4, execution_mode="thread")
+success = runner.run_tests()
+```
+
+### File Comparison (Standalone CLI)
+
+```bash
+compare-files result1.h5 result2.h5 --h5-table-regex "output_.*" --h5-rtol 1e-5
+```
+
+📖 **Full Documentation**: [docs/user_manual.md](docs/user_manual_en.md)
+
+## Changelog
+
+### 0.8.0
+
+- **JUnit XML output**: New `--junit-xml <filepath>` CLI option writes JUnit-format XML reports that integrate directly with GitLab CI, Jenkins, CircleCI, and other CI tools. Supports `passed` / `failed` / `timeout` / `error` status mapping. Zero extra dependencies — built with `xml.etree.ElementTree`.
+
+### 0.7.0
+
+- **Unified logging system**: All diagnostic output (executor, runner, scheduler, setup) now uses Python's standard `logging` module under the `cli_test_framework` namespace. Library users can suppress output entirely via `logging.getLogger("cli_test_framework").setLevel(logging.WARNING)`. Removes the previous ad-hoc `print()` + `_print_lock` pattern — `logging` is inherently thread-safe.
+- **Default handler**: A `StreamHandler` at INFO level is installed on first import, so CLI behavior is unchanged. Use `--verbose` / `--debug` to enable DEBUG-level output.
+- **Public API**: `get_logger(name)` exposed via `cli_test_framework.get_logger` for consistent logger creation in extensions.
+
+### 0.6.0
+
+- **Golden file assertion**: `compare_files` is now a first-class assertion in test `expected` — compare output files against baselines directly in your test definitions, with full tolerance and parameter support. The `file_comparator` subsystem is now integrated into the assertion pipeline, completing the closed loop from command execution to result verification.
+
+### 0.5.2
+
+- Runtime history tracking (`--history-dir`): persist per-case execution time in `.symtest`, enable smart scheduling & regression detection
+- Regression warning: alert when a case runs slower than historical average × threshold (`--regression-threshold`, default 1.5)
+- Smart scheduling: parallel runner prioritizes historical `avg_duration` over config `estimated_time` for task ordering
+- Per-case duration now shown in test result output
+
+### 0.5.1
+- Run specific test cases by name (`-t` / `test_case_filter`)
+
+### 0.5.0
+- Steps feature: sequential multi-command execution within a single test case, fail-fast
+
+### 0.4.2
+- Resource-aware scheduling: auto-detect CPU cores, semaphore-based core allocation
+- Auto-inject `OMP_NUM_THREADS` / `MKL_NUM_THREADS` / `NPROC` to prevent solver thread runaway
+- Per-test `timeout` support to prevent hanging
+
+### 0.4.1
+- Multi-thread / multi-process parallel execution, 3-5x speedup
+
+## Contributing
+
+Before submitting a PR, please make sure all tests pass:
+
+```bash
+python tests\run_all.py
+```
+
+## License
+
+MIT

cli_test_framework-0.8.0/README.md

@@ -0,0 +1,196 @@
+# CLI Test Framework
+
+A lightweight automated testing framework for command-line applications. Define test cases in JSON/YAML, run all validations with a single command.
+
+Particularly suited for scientific computing — deep HDF5 support with regex table matching, data filtering, and tolerance-based comparison, making simulation result verification effortless.
+
+## Why this exists
+
+This project was created from a real need: regression testing finite-element software.
+
+In solver-style projects, checking the exit code is not enough. A test often needs to run multiple commands, generate numerical result files, compare HDF5/CSV outputs with tolerances, and make sure old cases do not become slower over time.
+
+CLI Test Framework was built for that workflow.
+
+## Highlights
+
+- **Golden File Assertion** — `compare_files` embedded in test `expected`, compares output files against baselines with tolerance
+- **Parallel Execution** — Multi-thread / multi-process, 3-5x speedup
+- **Resource-Aware Scheduling** — Automatic CPU core management, prevents solver thread runaway
+- **Sequence Steps** — Multi-step execution within a single test case, fail-fast
+- **Setup Module** — Auto-configure environment variables before tests, auto-cleanup after
+- **File Comparison** — Text / JSON / CSV / XML / HDF5 / Binary, with CLI and embedded assertion support
+- **Filtered Execution** — Run specific test cases by name
+- **JUnit XML Output** — `--junit-xml` for GitLab CI / Jenkins / CircleCI test report panels
+
+## Quick Start
+
+```bash
+pip install cli-test-framework
+```
+
+### 30-Second Setup
+
+1. Create `test_cases.json`:
+
+```json
+{
+    "test_cases": [
+        {
+            "name": "hello",
+            "command": "echo",
+            "args": ["Hello World"],
+            "expected": {
+                "return_code": 0,
+                "output_contains": ["Hello World"]
+            }
+        }
+    ]
+}
+```
+
+2. Run:
+
+```bash
+cli-test run test_cases.json
+```
+
+### Golden File Comparison in Tests
+
+Run a simulation, then compare its output file against a reference:
+
+```json
+{
+    "test_cases": [
+        {
+            "name": "FEA displacement check",
+            "command": "my_solver",
+            "args": ["--input", "case1.dat", "--output", "out.h5"],
+            "expected": {
+                "return_code": 0,
+                "compare_files": [
+                    {
+                        "actual": "out.h5",
+                        "baseline": "ref/golden.h5",
+                        "rtol": 1e-5,
+                        "atol": 1e-8,
+                        "tables": ["NASTRAN/RESULT/NODAL/DISPLACEMENT"]
+                    }
+                ]
+            }
+        }
+    ]
+}
+```
+
+- `actual` – file produced by the command
+- `baseline` – reference file to compare against
+- `type` – comparator type (auto-detected from extension if omitted: `.h5`→h5, `.json`→json, `.csv`→csv, `.xml`→xml, `.txt`→text)
+- all other keys are forwarded as comparator parameters (`rtol`, `atol`, `tables`, `table_regex`, `data_filter`, `encoding`, `structure_only`, `delimiter`, `compare_mode`, `key_field`, etc.)
+
+Multiple files and mixed assertion types coexist naturally:
+
+```json
+{
+    "expected": {
+        "return_code": 0,
+        "output_contains": ["simulation finished"],
+        "compare_files": [
+            {"actual": "out.h5", "baseline": "ref/disp.h5", "rtol": 1e-5},
+            {"actual": "report.csv", "baseline": "ref/expected.csv", "rtol": 1e-6}
+        ]
+    }
+}
+```
+
+### Parallel Execution
+
+```bash
+cli-test run test_cases.json --parallel --workers 4
+```
+
+### JUnit XML (CI Integration)
+
+```bash
+cli-test run test_cases.json --junit-xml report.xml
+```
+
+In GitLab CI:
+
+```yaml
+# .gitlab-ci.yml
+artifacts:
+  reports:
+    junit: report.xml
+```
+
+### Python API
+
+```python
+from cli_test_framework.runners import JSONRunner, ParallelJSONRunner
+
+# Sequential
+runner = JSONRunner(config_file="test_cases.json")
+success = runner.run_tests()
+
+# Parallel
+runner = ParallelJSONRunner(config_file="test_cases.json", max_workers=4, execution_mode="thread")
+success = runner.run_tests()
+```
+
+### File Comparison (Standalone CLI)
+
+```bash
+compare-files result1.h5 result2.h5 --h5-table-regex "output_.*" --h5-rtol 1e-5
+```
+
+📖 **Full Documentation**: [docs/user_manual.md](docs/user_manual_en.md)
+
+## Changelog
+
+### 0.8.0
+
+- **JUnit XML output**: New `--junit-xml <filepath>` CLI option writes JUnit-format XML reports that integrate directly with GitLab CI, Jenkins, CircleCI, and other CI tools. Supports `passed` / `failed` / `timeout` / `error` status mapping. Zero extra dependencies — built with `xml.etree.ElementTree`.
+
+### 0.7.0
+
+- **Unified logging system**: All diagnostic output (executor, runner, scheduler, setup) now uses Python's standard `logging` module under the `cli_test_framework` namespace. Library users can suppress output entirely via `logging.getLogger("cli_test_framework").setLevel(logging.WARNING)`. Removes the previous ad-hoc `print()` + `_print_lock` pattern — `logging` is inherently thread-safe.
+- **Default handler**: A `StreamHandler` at INFO level is installed on first import, so CLI behavior is unchanged. Use `--verbose` / `--debug` to enable DEBUG-level output.
+- **Public API**: `get_logger(name)` exposed via `cli_test_framework.get_logger` for consistent logger creation in extensions.
+
+### 0.6.0
+
+- **Golden file assertion**: `compare_files` is now a first-class assertion in test `expected` — compare output files against baselines directly in your test definitions, with full tolerance and parameter support. The `file_comparator` subsystem is now integrated into the assertion pipeline, completing the closed loop from command execution to result verification.
+
+### 0.5.2
+
+- Runtime history tracking (`--history-dir`): persist per-case execution time in `.symtest`, enable smart scheduling & regression detection
+- Regression warning: alert when a case runs slower than historical average × threshold (`--regression-threshold`, default 1.5)
+- Smart scheduling: parallel runner prioritizes historical `avg_duration` over config `estimated_time` for task ordering
+- Per-case duration now shown in test result output
+
+### 0.5.1
+- Run specific test cases by name (`-t` / `test_case_filter`)
+
+### 0.5.0
+- Steps feature: sequential multi-command execution within a single test case, fail-fast
+
+### 0.4.2
+- Resource-aware scheduling: auto-detect CPU cores, semaphore-based core allocation
+- Auto-inject `OMP_NUM_THREADS` / `MKL_NUM_THREADS` / `NPROC` to prevent solver thread runaway
+- Per-test `timeout` support to prevent hanging
+
+### 0.4.1
+- Multi-thread / multi-process parallel execution, 3-5x speedup
+
+## Contributing
+
+Before submitting a PR, please make sure all tests pass:
+
+```bash
+python tests\run_all.py
+```
+
+## License
+
+MIT

{cli_test_framework-0.5.3 → cli_test_framework-0.8.0}/docs/user_manual_en.md

@@ -117,6 +117,9 @@ cli-test run test_cases.json --verbose
 # Debug mode
 cli-test run test_cases.json --debug
 
+# JUnit XML output for CI
+cli-test run test_cases.json --junit-xml report.xml
+
 # Output format
 cli-test run test_cases.json --output-format json|html|text
 ```
@@ -162,6 +165,80 @@ for detail in runner.results["details"]:
     print(detail["name"], detail["status"], detail.get("message", ""))
 ```
 
+### JUnit XML Output (CI Integration)
+
+Generate JUnit-format XML reports for CI tools like GitLab CI, Jenkins, or CircleCI:
+
+```bash
+cli-test run test_cases.json --junit-xml report.xml
+```
+
+This produces a standard JUnit XML file that can be consumed by CI report panels:
+
+```xml
+<?xml version="1.0" encoding="UTF-8"?>
+<testsuite name="test_cases" tests="3" failures="1" errors="1" time="12.345">
+  <testcase name="test_ok" classname="test_cases" time="0.123" />
+  <testcase name="test_fail" classname="test_cases" time="0.456">
+    <failure message="expected 'hello'" type="AssertionError">...</failure>
+  </testcase>
+  <testcase name="test_timeout" classname="test_cases" time="10.0">
+    <error message="Timeout reached" type="TimeoutExpired">...</error>
+  </testcase>
+</testsuite>
+```
+
+**CI configuration examples:**
+
+**GitLab CI** (`.gitlab-ci.yml`):
+```yaml
+test:
+  script:
+    - cli-test run test_cases.json --junit-xml report.xml
+  artifacts:
+    reports:
+      junit: report.xml
+```
+
+**Jenkins Pipeline**:
+```groovy
+stage('Test') {
+    steps {
+        sh 'cli-test run test_cases.json --junit-xml report.xml'
+        junit 'report.xml'
+    }
+}
+```
+
+**GitHub Actions**:
+```yaml
+- name: Run tests
+  run: cli-test run test_cases.json --junit-xml report.xml
+- name: Publish test results
+  uses: dorny/test-reporter@v1
+  with:
+    name: CLI Tests
+    path: report.xml
+    reporter: java-junit
+```
+
+**Python API**:
+```python
+from cli_test_framework import write_junit_xml
+
+runner = JSONRunner(config_file="test_cases.json")
+runner.run_tests()
+write_junit_xml(runner.results, "report.xml", suite_name="my_tests")
+```
+
+Status mapping:
+| Test status | JUnit element | CI interpretation |
+|---|---|---|
+| `passed` | (no child) | Passed |
+| `failed` (assertion) | `<failure>` | Failure |
+| `failed` (execution error) | `<error>` | Error |
+| `timeout` | `<error>` | Error |
+
 ## Setup Module
 
 The Setup module performs initialization before tests and cleanup after tests.
@@ -434,6 +511,26 @@ result.differences # list
 
 ## Extension Development
 
+### Controlling Log Output
+
+The framework uses Python's standard `logging` module. By default, a console handler at `INFO` level is installed. You can suppress or redirect output:
+
+```python
+import logging
+
+# Suppress all framework output when used as a library
+logging.getLogger("cli_test_framework").setLevel(logging.WARNING)
+
+# Or remove the handler entirely
+logging.getLogger("cli_test_framework").handlers.clear()
+
+# Redirect to a file
+file_handler = logging.FileHandler("test.log")
+file_handler.setLevel(logging.DEBUG)
+file_handler.setFormatter(logging.Formatter("%(asctime)s [%(levelname)s] %(name)s: %(message)s"))
+logging.getLogger("cli_test_framework").addHandler(file_handler)
+```
+
 ### Custom Runner
 
 ```python
@@ -451,16 +548,21 @@ class CustomRunner(BaseRunner):
 
 ### Custom Setup Plugin
 
+Use `get_logger` for consistent logging in your extensions:
+
 ```python
-from cli_test_framework import BaseSetup
+from cli_test_framework import BaseSetup, get_logger
 
 class MySetup(BaseSetup):
+    def __init__(self, config=None):
+        super().__init__(config)
+        self.logger = get_logger(__name__)
+
     def setup(self):
-        # self.config provides access to the passed configuration dict
-        pass
+        self.logger.info("Running MySetup...")
 
     def teardown(self):
-        pass
+        self.logger.info("Tearing down MySetup...")
 ```
 
 ### Custom Assertions

{cli_test_framework-0.5.3 → cli_test_framework-0.8.0}/setup.py

@@ -8,7 +8,7 @@ with open(os.path.join(this_directory, 'README.md'), encoding='utf-8') as f:
 
 setup(
     name="cli-test-framework",
-    version="0.5.3",
+    version="0.8.0",
     author="Xiaotong Wang",
     author_email="xiaotongwang98@gmail.com",
     description="A powerful command line testing framework in Python with setup modules, parallel execution, and file comparison capabilities.",
@@ -19,10 +19,9 @@ setup(
     package_dir={"": "src"},
     install_requires=[
         "dukpy==0.5.0",
-        "h5py>=3.8.0,<4.0.0; python_version<'3.12'",
-        "h5py>=3.10.0,<4.0.0; python_version>='3.12'",
-        "numpy>=1.21.0,<2.0.0; python_version<'3.12'",
-        "numpy>=1.26.0,<2.0.0; python_version>='3.12'",
+        "h5py>=3.11.0,<4.0.0",
+        "numpy>=1.21.0; python_version<'3.12'",
+        "numpy>=1.26.0; python_version>='3.12'",
         "setuptools>=75.8.0",
         "wheel>=0.45.1"
     ],

cli_test_framework-0.8.0/src/cli_test_framework/__init__.py

@@ -0,0 +1,45 @@
+"""
+CLI Test Framework - A powerful command-line testing framework
+
+This package provides tools for testing command-line applications and scripts
+with support for parallel execution and advanced file comparison capabilities.
+
+Logging
+-------
+All diagnostic and status messages go through Python's standard ``logging``
+module under the ``cli_test_framework`` namespace.  By default only a
+``NullHandler`` is attached — no output is produced on import.  The CLI
+entry point calls ``setup_console_logging()`` to enable stderr output at
+runtime.  Library users can enable console logging explicitly::
+
+    from cli_test_framework.logging_config import setup_console_logging
+    setup_console_logging(level=logging.DEBUG)
+"""
+
+__version__ = "0.8.0"
+__author__ = "Xiaotong Wang"
+__email__ = "xiaotongwang98@gmail.com"
+
+# Import main classes for convenient access
+from .runners.json_runner import JSONRunner
+from .runners.parallel_json_runner import ParallelJSONRunner
+from .runners.yaml_runner import YAMLRunner
+from .core.test_case import TestCase
+from .core.assertions import Assertions
+from .core.setup import BaseSetup, EnvironmentSetup, SetupManager
+from .logging_config import get_logger, setup_console_logging
+from .utils.junit_xml_writer import write_junit_xml
+
+__all__ = [
+    'JSONRunner',
+    'ParallelJSONRunner',
+    'YAMLRunner',
+    'TestCase',
+    'Assertions',
+    'BaseSetup',
+    'EnvironmentSetup',
+    'SetupManager',
+    'get_logger',
+    'setup_console_logging',
+    'write_junit_xml',
+]