symtest-cli 1.3.0__tar.gz

symtest_cli-1.3.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2024 CLI Test Framework Contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR OUT OF IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

symtest_cli-1.3.0/MANIFEST.in

@@ -0,0 +1,4 @@
+include README.md
+include LICENSE
+recursive-include docs *
+recursive-include tests * 

symtest_cli-1.3.0/PKG-INFO

@@ -0,0 +1,316 @@
+Metadata-Version: 2.4
+Name: symtest-cli
+Version: 1.3.0
+Summary: Regression testing for CLI applications, with multi-step workflows and numerical file comparison.
+Home-page: https://github.com/ozil111/symtest
+Author: Xiaotong Wang
+Author-email: xiaotongwang98@gmail.com
+Project-URL: Documentation, https://github.com/ozil111/symtest/blob/main/docs/user_manual.md
+Project-URL: Source, https://github.com/ozil111/symtest
+Project-URL: Tracker, https://github.com/ozil111/symtest/issues
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Topic :: Software Development :: Testing
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: h5py<4.0.0,>=3.11.0
+Requires-Dist: numpy>=1.21.0; python_version < "3.12"
+Requires-Dist: numpy>=1.26.0; python_version >= "3.12"
+Provides-Extra: yaml
+Requires-Dist: PyYAML>=6.0; extra == "yaml"
+Provides-Extra: tui
+Requires-Dist: textual>=0.40.0; extra == "tui"
+Provides-Extra: all
+Requires-Dist: PyYAML>=6.0; extra == "all"
+Requires-Dist: textual>=0.40.0; extra == "all"
+Provides-Extra: dev
+Requires-Dist: PyYAML>=6.0; extra == "dev"
+Requires-Dist: textual>=0.40.0; extra == "dev"
+Requires-Dist: pytest>=7.0; extra == "dev"
+Requires-Dist: pytest-asyncio>=0.21.0; extra == "dev"
+Requires-Dist: pytest-cov>=4.0; extra == "dev"
+Dynamic: author
+Dynamic: author-email
+Dynamic: classifier
+Dynamic: description
+Dynamic: description-content-type
+Dynamic: home-page
+Dynamic: license-file
+Dynamic: project-url
+Dynamic: provides-extra
+Dynamic: requires-dist
+Dynamic: requires-python
+Dynamic: summary
+
+# CLI Test Framework
+
+[中文](README_cn.md) | English
+
+> **Note:** This project was formerly known as `cli-test-framework`.
+> Since version 1.3.0 it is released as
+> [`symtest`](https://pypi.org/project/symtest/); the old package name is no
+> longer updated. Install with:
+>
+> ```bash
+> pip install symtest
+> ```
+
+A feature-focused automated testing framework for command-line applications.
+It is built for regression suites that need more than an exit-code check:
+multi-step commands, numerical result comparison, large configuration sets,
+parallel execution, and CI-ready reports.
+
+The project grew out of finite-element solver development, where a single test
+may run several programs, produce HDF5 or CSV results, compare them with
+tolerances, and track execution time across revisions.
+
+## What it solves
+
+CLI Test Framework keeps the execution workflow and its acceptance criteria in
+one JSON or YAML configuration:
+
+- **Execute workflows** — single commands or fail-fast step sequences, with
+  timeouts, retries, variables, tags, and expected-failure support.
+- **Verify results** — return codes, output text and regular expressions, plus
+  text, JSON, CSV, XML, HDF5, binary, and custom-script file comparisons.
+- **Manage large suites** — split configurations with `import`, reuse templates
+  with `extends`, filter by name or tag, and inspect cases across files in the
+  optional TUI.
+- **Iterate and integrate** — parallel execution, `--last-failed`, step-level
+  `--resume`, runtime history, structured reports, and JUnit XML output.
+
+The emphasis is practical: features are added to solve test workflows that
+occur in real CLI and scientific-computing projects.
+
+## Installation
+
+Python 3.9 or newer is required.
+
+```bash
+pip install symtest
+```
+
+YAML and the TUI are optional:
+
+```bash
+pip install "symtest[yaml]"
+pip install "symtest[tui]"
+pip install "symtest[all]"
+```
+
+The default installation includes HDF5 and numerical comparison support.
+
+## Quick start
+
+Create `test_cases.json`:
+
+```json
+{
+  "test_cases": [
+    {
+      "name": "hello",
+      "command": "echo",
+      "args": ["Hello World"],
+      "tags": ["smoke"],
+      "expected": {
+        "return_code": 0,
+        "output_contains": ["Hello World"]
+      }
+    }
+  ]
+}
+```
+
+Run it:
+
+```bash
+symtest run test_cases.json
+```
+
+Validate a configuration without executing it:
+
+```bash
+symtest validate test_cases.json
+```
+
+## Numerical golden-file testing
+
+File comparisons can be part of a test's acceptance criteria. Comparator
+parameters such as tolerances, table selection, filters, and encodings are
+declared next to the command:
+
+```json
+{
+  "test_cases": [
+    {
+      "name": "FEA displacement check",
+      "command": "my_solver",
+      "args": ["--input", "case1.dat", "--output", "out.h5"],
+      "expected": {
+        "return_code": 0,
+        "output_contains": ["simulation finished"],
+        "compare_files": [
+          {
+            "actual": "out.h5",
+            "baseline": "ref/golden.h5",
+            "rtol": 1e-5,
+            "atol": 1e-8,
+            "tables": ["NASTRAN/RESULT/NODAL/DISPLACEMENT"]
+          },
+          {
+            "actual": "summary.csv",
+            "baseline": "ref/summary.csv",
+            "rtol": 1e-6
+          }
+        ]
+      }
+    }
+  ]
+}
+```
+
+The comparator type is inferred from the extension when `type` is omitted.
+Built-in types are `text`, `json`, `csv`, `xml`, `h5`, `binary`, and `script`.
+Workspace comparators can be added without changing the framework.
+
+To intentionally accept changed outputs, use `--update-baseline`. Because this
+can overwrite reference files, interactive runs require typing `yes`;
+non-interactive runs must explicitly add `--yes`:
+
+```bash
+symtest run test_cases.json --update-baseline
+symtest run test_cases.json --update-baseline --yes   # automation / CI
+```
+
+Keep baselines under version control and review every update.
+
+## Multi-step and iterative workflows
+
+A case may contain an ordered `steps` list. Execution stops at the first failed
+step. For long workflows, `--resume` reuses saved state and skips steps that
+already passed:
+
+```bash
+symtest run solver_tests.json
+symtest run solver_tests.json --last-failed
+symtest run solver_tests.json -t long_case --resume
+```
+
+`--resume` deliberately trusts that workspace artifacts have not changed
+between runs.
+
+## Large test suites and the optional TUI
+
+Large suites can be divided into sub-configurations:
+
+```json
+{
+  "test_cases": [
+    {"import": "cases/text_tests.json", "tags": ["text"]},
+    {"import": "cases/h5_tests.json", "tags": ["h5", "regression"]}
+  ]
+}
+```
+
+The optional TUI provides one searchable view across imported files. It is
+intended as an aid for locating cases and reviewing scenario coverage in large
+projects, rather than a requirement for normal test execution.
+
+```bash
+symtest tui main_config.json
+```
+
+## Parallel execution and resources
+
+```bash
+symtest run test_cases.json --parallel --workers 4
+symtest run test_cases.json --parallel --execution-mode process
+```
+
+Thread mode currently supports CPU-token allocation, solver thread environment
+variables, and longest-processing-time-first scheduling using estimates or
+runtime history. Process mode provides execution isolation but does not yet use
+the resource scheduler. Memory enforcement, priority semantics, and broader
+resource scheduling remain active areas of development.
+
+## CI and reports
+
+```bash
+symtest run test_cases.json \
+  --parallel --workers 4 \
+  --junit-xml report.xml
+```
+
+The current suite contains 750 unit, integration, and end-to-end tests with 83%
+line coverage. CI exercises Windows and Linux across Python 3.9 through 3.13.
+
+## Python API
+
+```python
+from symtest.runners import JSONRunner, ParallelJSONRunner
+
+runner = ParallelJSONRunner(
+    config_file="test_cases.json",
+    max_workers=4,
+    execution_mode="thread",
+    history_dir="./hist",
+    variables={"solver": "/opt/solver/bin/solver"},
+)
+
+success = runner.run_tests()
+for detail in runner.results["details"]:
+    print(detail["name"], detail["status"], detail.get("duration"))
+```
+
+## Standalone file comparison
+
+```bash
+compare-files result1.h5 result2.h5 --h5-table-regex "output_.*" --h5-rtol 1e-5
+compare-files data1.csv data2.csv --csv-rtol 1e-4 --csv-data-filter ">1e-6"
+compare-files data1.json data2.json --json-compare-mode key-based --json-key-field id
+```
+
+## AI-assisted TDD, as a side benefit
+
+The same configuration can serve as a machine-readable acceptance contract.
+Structured validation failures, comparison details, and targeted reruns work
+well in an AI-assisted TDD loop:
+
+```text
+define acceptance criteria
+    → run the relevant cases
+    → inspect the structured failure
+    → change the implementation
+    → rerun with --last-failed
+    → run the full regression suite
+```
+
+This is a useful consequence of explicit tests and structured results, not a
+requirement for using the framework.
+
+## Documentation
+
+- [User manual](docs/user_manual_en.md)
+- [Design document](docs/design_en.md)
+- [Plugin examples](examples/plugins/README.md)
+
+## Development
+
+Install the package with all optional and test dependencies:
+
+```bash
+pip install -e ".[dev]"
+python -m pytest tests/unit tests/integration tests/e2e
+```
+
+Bug fixes, comparator plugins, documentation improvements, and reports from
+real-world test workflows are welcome.
+
+## License
+
+MIT

symtest_cli-1.3.0/README.md

@@ -0,0 +1,267 @@
+# CLI Test Framework
+
+[中文](README_cn.md) | English
+
+> **Note:** This project was formerly known as `cli-test-framework`.
+> Since version 1.3.0 it is released as
+> [`symtest`](https://pypi.org/project/symtest/); the old package name is no
+> longer updated. Install with:
+>
+> ```bash
+> pip install symtest
+> ```
+
+A feature-focused automated testing framework for command-line applications.
+It is built for regression suites that need more than an exit-code check:
+multi-step commands, numerical result comparison, large configuration sets,
+parallel execution, and CI-ready reports.
+
+The project grew out of finite-element solver development, where a single test
+may run several programs, produce HDF5 or CSV results, compare them with
+tolerances, and track execution time across revisions.
+
+## What it solves
+
+CLI Test Framework keeps the execution workflow and its acceptance criteria in
+one JSON or YAML configuration:
+
+- **Execute workflows** — single commands or fail-fast step sequences, with
+  timeouts, retries, variables, tags, and expected-failure support.
+- **Verify results** — return codes, output text and regular expressions, plus
+  text, JSON, CSV, XML, HDF5, binary, and custom-script file comparisons.
+- **Manage large suites** — split configurations with `import`, reuse templates
+  with `extends`, filter by name or tag, and inspect cases across files in the
+  optional TUI.
+- **Iterate and integrate** — parallel execution, `--last-failed`, step-level
+  `--resume`, runtime history, structured reports, and JUnit XML output.
+
+The emphasis is practical: features are added to solve test workflows that
+occur in real CLI and scientific-computing projects.
+
+## Installation
+
+Python 3.9 or newer is required.
+
+```bash
+pip install symtest
+```
+
+YAML and the TUI are optional:
+
+```bash
+pip install "symtest[yaml]"
+pip install "symtest[tui]"
+pip install "symtest[all]"
+```
+
+The default installation includes HDF5 and numerical comparison support.
+
+## Quick start
+
+Create `test_cases.json`:
+
+```json
+{
+  "test_cases": [
+    {
+      "name": "hello",
+      "command": "echo",
+      "args": ["Hello World"],
+      "tags": ["smoke"],
+      "expected": {
+        "return_code": 0,
+        "output_contains": ["Hello World"]
+      }
+    }
+  ]
+}
+```
+
+Run it:
+
+```bash
+symtest run test_cases.json
+```
+
+Validate a configuration without executing it:
+
+```bash
+symtest validate test_cases.json
+```
+
+## Numerical golden-file testing
+
+File comparisons can be part of a test's acceptance criteria. Comparator
+parameters such as tolerances, table selection, filters, and encodings are
+declared next to the command:
+
+```json
+{
+  "test_cases": [
+    {
+      "name": "FEA displacement check",
+      "command": "my_solver",
+      "args": ["--input", "case1.dat", "--output", "out.h5"],
+      "expected": {
+        "return_code": 0,
+        "output_contains": ["simulation finished"],
+        "compare_files": [
+          {
+            "actual": "out.h5",
+            "baseline": "ref/golden.h5",
+            "rtol": 1e-5,
+            "atol": 1e-8,
+            "tables": ["NASTRAN/RESULT/NODAL/DISPLACEMENT"]
+          },
+          {
+            "actual": "summary.csv",
+            "baseline": "ref/summary.csv",
+            "rtol": 1e-6
+          }
+        ]
+      }
+    }
+  ]
+}
+```
+
+The comparator type is inferred from the extension when `type` is omitted.
+Built-in types are `text`, `json`, `csv`, `xml`, `h5`, `binary`, and `script`.
+Workspace comparators can be added without changing the framework.
+
+To intentionally accept changed outputs, use `--update-baseline`. Because this
+can overwrite reference files, interactive runs require typing `yes`;
+non-interactive runs must explicitly add `--yes`:
+
+```bash
+symtest run test_cases.json --update-baseline
+symtest run test_cases.json --update-baseline --yes   # automation / CI
+```
+
+Keep baselines under version control and review every update.
+
+## Multi-step and iterative workflows
+
+A case may contain an ordered `steps` list. Execution stops at the first failed
+step. For long workflows, `--resume` reuses saved state and skips steps that
+already passed:
+
+```bash
+symtest run solver_tests.json
+symtest run solver_tests.json --last-failed
+symtest run solver_tests.json -t long_case --resume
+```
+
+`--resume` deliberately trusts that workspace artifacts have not changed
+between runs.
+
+## Large test suites and the optional TUI
+
+Large suites can be divided into sub-configurations:
+
+```json
+{
+  "test_cases": [
+    {"import": "cases/text_tests.json", "tags": ["text"]},
+    {"import": "cases/h5_tests.json", "tags": ["h5", "regression"]}
+  ]
+}
+```
+
+The optional TUI provides one searchable view across imported files. It is
+intended as an aid for locating cases and reviewing scenario coverage in large
+projects, rather than a requirement for normal test execution.
+
+```bash
+symtest tui main_config.json
+```
+
+## Parallel execution and resources
+
+```bash
+symtest run test_cases.json --parallel --workers 4
+symtest run test_cases.json --parallel --execution-mode process
+```
+
+Thread mode currently supports CPU-token allocation, solver thread environment
+variables, and longest-processing-time-first scheduling using estimates or
+runtime history. Process mode provides execution isolation but does not yet use
+the resource scheduler. Memory enforcement, priority semantics, and broader
+resource scheduling remain active areas of development.
+
+## CI and reports
+
+```bash
+symtest run test_cases.json \
+  --parallel --workers 4 \
+  --junit-xml report.xml
+```
+
+The current suite contains 750 unit, integration, and end-to-end tests with 83%
+line coverage. CI exercises Windows and Linux across Python 3.9 through 3.13.
+
+## Python API
+
+```python
+from symtest.runners import JSONRunner, ParallelJSONRunner
+
+runner = ParallelJSONRunner(
+    config_file="test_cases.json",
+    max_workers=4,
+    execution_mode="thread",
+    history_dir="./hist",
+    variables={"solver": "/opt/solver/bin/solver"},
+)
+
+success = runner.run_tests()
+for detail in runner.results["details"]:
+    print(detail["name"], detail["status"], detail.get("duration"))
+```
+
+## Standalone file comparison
+
+```bash
+compare-files result1.h5 result2.h5 --h5-table-regex "output_.*" --h5-rtol 1e-5
+compare-files data1.csv data2.csv --csv-rtol 1e-4 --csv-data-filter ">1e-6"
+compare-files data1.json data2.json --json-compare-mode key-based --json-key-field id
+```
+
+## AI-assisted TDD, as a side benefit
+
+The same configuration can serve as a machine-readable acceptance contract.
+Structured validation failures, comparison details, and targeted reruns work
+well in an AI-assisted TDD loop:
+
+```text
+define acceptance criteria
+    → run the relevant cases
+    → inspect the structured failure
+    → change the implementation
+    → rerun with --last-failed
+    → run the full regression suite
+```
+
+This is a useful consequence of explicit tests and structured results, not a
+requirement for using the framework.
+
+## Documentation
+
+- [User manual](docs/user_manual_en.md)
+- [Design document](docs/design_en.md)
+- [Plugin examples](examples/plugins/README.md)
+
+## Development
+
+Install the package with all optional and test dependencies:
+
+```bash
+pip install -e ".[dev]"
+python -m pytest tests/unit tests/integration tests/e2e
+```
+
+Bug fixes, comparator plugins, documentation improvements, and reports from
+real-world test workflows are welcome.
+
+## License
+
+MIT