cli-test-framework 0.5.4__tar.gz → 0.8.1__tar.gz

{cli_test_framework-0.5.4/src/cli_test_framework.egg-info → cli_test_framework-0.8.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: cli-test-framework
-Version: 0.5.4
+Version: 0.8.1
 Summary: A powerful command line testing framework in Python with setup modules, parallel execution, and file comparison capabilities.
 Home-page: https://github.com/ozil111/cli-test-framework
 Author: Xiaotong Wang
@@ -50,12 +50,14 @@ CLI Test Framework was built for that workflow.
 
 ## Highlights
 
+- **Golden File Assertion** — `compare_files` embedded in test `expected`, compares output files against baselines with tolerance
 - **Parallel Execution** — Multi-thread / multi-process, 3-5x speedup
 - **Resource-Aware Scheduling** — Automatic CPU core management, prevents solver thread runaway
 - **Sequence Steps** — Multi-step execution within a single test case, fail-fast
 - **Setup Module** — Auto-configure environment variables before tests, auto-cleanup after
-- **File Comparison** — Text / JSON / HDF5 / Binary, ready to use from command line
+- **File Comparison** — Text / JSON / CSV / XML / HDF5 / Binary, with CLI and embedded assertion support
 - **Filtered Execution** — Run specific test cases by name
+- **JUnit XML Output** — `--junit-xml` for GitLab CI / Jenkins / CircleCI test report panels
 
 ## Quick Start
 
@@ -89,12 +91,75 @@ pip install cli-test-framework
 cli-test run test_cases.json
 ```
 
+### Golden File Comparison in Tests
+
+Run a simulation, then compare its output file against a reference:
+
+```json
+{
+    "test_cases": [
+        {
+            "name": "FEA displacement check",
+            "command": "my_solver",
+            "args": ["--input", "case1.dat", "--output", "out.h5"],
+            "expected": {
+                "return_code": 0,
+                "compare_files": [
+                    {
+                        "actual": "out.h5",
+                        "baseline": "ref/golden.h5",
+                        "rtol": 1e-5,
+                        "atol": 1e-8,
+                        "tables": ["NASTRAN/RESULT/NODAL/DISPLACEMENT"]
+                    }
+                ]
+            }
+        }
+    ]
+}
+```
+
+- `actual` – file produced by the command
+- `baseline` – reference file to compare against
+- `type` – comparator type (auto-detected from extension if omitted: `.h5`→h5, `.json`→json, `.csv`→csv, `.xml`→xml, `.txt`→text)
+- all other keys are forwarded as comparator parameters (`rtol`, `atol`, `tables`, `table_regex`, `data_filter`, `encoding`, `structure_only`, `delimiter`, `compare_mode`, `key_field`, etc.)
+
+Multiple files and mixed assertion types coexist naturally:
+
+```json
+{
+    "expected": {
+        "return_code": 0,
+        "output_contains": ["simulation finished"],
+        "compare_files": [
+            {"actual": "out.h5", "baseline": "ref/disp.h5", "rtol": 1e-5},
+            {"actual": "report.csv", "baseline": "ref/expected.csv", "rtol": 1e-6}
+        ]
+    }
+}
+```
+
 ### Parallel Execution
 
 ```bash
 cli-test run test_cases.json --parallel --workers 4
 ```
 
+### JUnit XML (CI Integration)
+
+```bash
+cli-test run test_cases.json --junit-xml report.xml
+```
+
+In GitLab CI:
+
+```yaml
+# .gitlab-ci.yml
+artifacts:
+  reports:
+    junit: report.xml
+```
+
 ### Python API
 
 ```python
@@ -109,7 +174,7 @@ runner = ParallelJSONRunner(config_file="test_cases.json", max_workers=4, execut
 success = runner.run_tests()
 ```
 
-### File Comparison
+### File Comparison (Standalone CLI)
 
 ```bash
 compare-files result1.h5 result2.h5 --h5-table-regex "output_.*" --h5-rtol 1e-5
@@ -119,6 +184,20 @@ compare-files result1.h5 result2.h5 --h5-table-regex "output_.*" --h5-rtol 1e-5
 
 ## Changelog
 
+### 0.8.0
+
+- **JUnit XML output**: New `--junit-xml <filepath>` CLI option writes JUnit-format XML reports that integrate directly with GitLab CI, Jenkins, CircleCI, and other CI tools. Supports `passed` / `failed` / `timeout` / `error` status mapping. Zero extra dependencies — built with `xml.etree.ElementTree`.
+
+### 0.7.0
+
+- **Unified logging system**: All diagnostic output (executor, runner, scheduler, setup) now uses Python's standard `logging` module under the `cli_test_framework` namespace. Library users can suppress output entirely via `logging.getLogger("cli_test_framework").setLevel(logging.WARNING)`. Removes the previous ad-hoc `print()` + `_print_lock` pattern — `logging` is inherently thread-safe.
+- **Default handler**: A `StreamHandler` at INFO level is installed on first import, so CLI behavior is unchanged. Use `--verbose` / `--debug` to enable DEBUG-level output.
+- **Public API**: `get_logger(name)` exposed via `cli_test_framework.get_logger` for consistent logger creation in extensions.
+
+### 0.6.0
+
+- **Golden file assertion**: `compare_files` is now a first-class assertion in test `expected` — compare output files against baselines directly in your test definitions, with full tolerance and parameter support. The `file_comparator` subsystem is now integrated into the assertion pipeline, completing the closed loop from command execution to result verification.
+
 ### 0.5.2
 
 - Runtime history tracking (`--history-dir`): persist per-case execution time in `.symtest`, enable smart scheduling & regression detection

cli_test_framework-0.8.1/README.md

@@ -0,0 +1,196 @@
+# CLI Test Framework
+
+A lightweight automated testing framework for command-line applications. Define test cases in JSON/YAML, run all validations with a single command.
+
+Particularly suited for scientific computing — deep HDF5 support with regex table matching, data filtering, and tolerance-based comparison, making simulation result verification effortless.
+
+## Why this exists
+
+This project was created from a real need: regression testing finite-element software.
+
+In solver-style projects, checking the exit code is not enough. A test often needs to run multiple commands, generate numerical result files, compare HDF5/CSV outputs with tolerances, and make sure old cases do not become slower over time.
+
+CLI Test Framework was built for that workflow.
+
+## Highlights
+
+- **Golden File Assertion** — `compare_files` embedded in test `expected`, compares output files against baselines with tolerance
+- **Parallel Execution** — Multi-thread / multi-process, 3-5x speedup
+- **Resource-Aware Scheduling** — Automatic CPU core management, prevents solver thread runaway
+- **Sequence Steps** — Multi-step execution within a single test case, fail-fast
+- **Setup Module** — Auto-configure environment variables before tests, auto-cleanup after
+- **File Comparison** — Text / JSON / CSV / XML / HDF5 / Binary, with CLI and embedded assertion support
+- **Filtered Execution** — Run specific test cases by name
+- **JUnit XML Output** — `--junit-xml` for GitLab CI / Jenkins / CircleCI test report panels
+
+## Quick Start
+
+```bash
+pip install cli-test-framework
+```
+
+### 30-Second Setup
+
+1. Create `test_cases.json`:
+
+```json
+{
+    "test_cases": [
+        {
+            "name": "hello",
+            "command": "echo",
+            "args": ["Hello World"],
+            "expected": {
+                "return_code": 0,
+                "output_contains": ["Hello World"]
+            }
+        }
+    ]
+}
+```
+
+2. Run:
+
+```bash
+cli-test run test_cases.json
+```
+
+### Golden File Comparison in Tests
+
+Run a simulation, then compare its output file against a reference:
+
+```json
+{
+    "test_cases": [
+        {
+            "name": "FEA displacement check",
+            "command": "my_solver",
+            "args": ["--input", "case1.dat", "--output", "out.h5"],
+            "expected": {
+                "return_code": 0,
+                "compare_files": [
+                    {
+                        "actual": "out.h5",
+                        "baseline": "ref/golden.h5",
+                        "rtol": 1e-5,
+                        "atol": 1e-8,
+                        "tables": ["NASTRAN/RESULT/NODAL/DISPLACEMENT"]
+                    }
+                ]
+            }
+        }
+    ]
+}
+```
+
+- `actual` – file produced by the command
+- `baseline` – reference file to compare against
+- `type` – comparator type (auto-detected from extension if omitted: `.h5`→h5, `.json`→json, `.csv`→csv, `.xml`→xml, `.txt`→text)
+- all other keys are forwarded as comparator parameters (`rtol`, `atol`, `tables`, `table_regex`, `data_filter`, `encoding`, `structure_only`, `delimiter`, `compare_mode`, `key_field`, etc.)
+
+Multiple files and mixed assertion types coexist naturally:
+
+```json
+{
+    "expected": {
+        "return_code": 0,
+        "output_contains": ["simulation finished"],
+        "compare_files": [
+            {"actual": "out.h5", "baseline": "ref/disp.h5", "rtol": 1e-5},
+            {"actual": "report.csv", "baseline": "ref/expected.csv", "rtol": 1e-6}
+        ]
+    }
+}
+```
+
+### Parallel Execution
+
+```bash
+cli-test run test_cases.json --parallel --workers 4
+```
+
+### JUnit XML (CI Integration)
+
+```bash
+cli-test run test_cases.json --junit-xml report.xml
+```
+
+In GitLab CI:
+
+```yaml
+# .gitlab-ci.yml
+artifacts:
+  reports:
+    junit: report.xml
+```
+
+### Python API
+
+```python
+from cli_test_framework.runners import JSONRunner, ParallelJSONRunner
+
+# Sequential
+runner = JSONRunner(config_file="test_cases.json")
+success = runner.run_tests()
+
+# Parallel
+runner = ParallelJSONRunner(config_file="test_cases.json", max_workers=4, execution_mode="thread")
+success = runner.run_tests()
+```
+
+### File Comparison (Standalone CLI)
+
+```bash
+compare-files result1.h5 result2.h5 --h5-table-regex "output_.*" --h5-rtol 1e-5
+```
+
+📖 **Full Documentation**: [docs/user_manual.md](docs/user_manual_en.md)
+
+## Changelog
+
+### 0.8.0
+
+- **JUnit XML output**: New `--junit-xml <filepath>` CLI option writes JUnit-format XML reports that integrate directly with GitLab CI, Jenkins, CircleCI, and other CI tools. Supports `passed` / `failed` / `timeout` / `error` status mapping. Zero extra dependencies — built with `xml.etree.ElementTree`.
+
+### 0.7.0
+
+- **Unified logging system**: All diagnostic output (executor, runner, scheduler, setup) now uses Python's standard `logging` module under the `cli_test_framework` namespace. Library users can suppress output entirely via `logging.getLogger("cli_test_framework").setLevel(logging.WARNING)`. Removes the previous ad-hoc `print()` + `_print_lock` pattern — `logging` is inherently thread-safe.
+- **Default handler**: A `StreamHandler` at INFO level is installed on first import, so CLI behavior is unchanged. Use `--verbose` / `--debug` to enable DEBUG-level output.
+- **Public API**: `get_logger(name)` exposed via `cli_test_framework.get_logger` for consistent logger creation in extensions.
+
+### 0.6.0
+
+- **Golden file assertion**: `compare_files` is now a first-class assertion in test `expected` — compare output files against baselines directly in your test definitions, with full tolerance and parameter support. The `file_comparator` subsystem is now integrated into the assertion pipeline, completing the closed loop from command execution to result verification.
+
+### 0.5.2
+
+- Runtime history tracking (`--history-dir`): persist per-case execution time in `.symtest`, enable smart scheduling & regression detection
+- Regression warning: alert when a case runs slower than historical average × threshold (`--regression-threshold`, default 1.5)
+- Smart scheduling: parallel runner prioritizes historical `avg_duration` over config `estimated_time` for task ordering
+- Per-case duration now shown in test result output
+
+### 0.5.1
+- Run specific test cases by name (`-t` / `test_case_filter`)
+
+### 0.5.0
+- Steps feature: sequential multi-command execution within a single test case, fail-fast
+
+### 0.4.2
+- Resource-aware scheduling: auto-detect CPU cores, semaphore-based core allocation
+- Auto-inject `OMP_NUM_THREADS` / `MKL_NUM_THREADS` / `NPROC` to prevent solver thread runaway
+- Per-test `timeout` support to prevent hanging
+
+### 0.4.1
+- Multi-thread / multi-process parallel execution, 3-5x speedup
+
+## Contributing
+
+Before submitting a PR, please make sure all tests pass:
+
+```bash
+python tests\run_all.py
+```
+
+## License
+
+MIT