cli-test-framework 0.5.0__tar.gz → 0.5.1__tar.gz

{cli_test_framework-0.5.0/src/cli_test_framework.egg-info → cli_test_framework-0.5.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: cli-test-framework
-Version: 0.5.0
+Version: 0.5.1
 Summary: A powerful command line testing framework in Python with setup modules, parallel execution, and file comparison capabilities.
 Home-page: https://github.com/ozil111/cli-test-framework
 Author: Xiaotong Wang

{cli_test_framework-0.5.0 → cli_test_framework-0.5.1}/docs/user_manual.md

@@ -7,11 +7,14 @@
 4. [Test Case Definition](#test-case-definition)
 5. [Setup Module](#setup-module)
 6. [Parallel Testing](#parallel-testing)
-7. [File Comparison](#file-comparison)
-8. [Advanced Features](#advanced-features)
-9. [Troubleshooting](#troubleshooting)
-10. [API Reference](#api-reference)
-11. [Examples](#examples)
+7. [Sequence Testing](#sequence-testing)
+8. [File Comparison](#file-comparison)
+9. [Advanced Features](#advanced-features)
+10. [Troubleshooting](#troubleshooting)
+11. [API Reference](#api-reference)
+12. [Examples](#examples)
+
+
 
 ## Introduction
 
@@ -19,6 +22,7 @@ The CLI Testing Framework is a powerful tool designed for testing command-line a
 
 ### Key Features
 - Parallel test execution with thread and process support
+- Sequence test execution with multi-step fail-fast
 - JSON/YAML test case definition
 - Advanced file comparison capabilities
 - Comprehensive reporting
@@ -74,6 +78,18 @@ runner = JSONRunner(
 success = runner.run_tests()
 ```
 
+3. Run only specified test case(s):
+```python
+from cli_test_framework.runners import JSONRunner
+
+runner = JSONRunner(
+    config_file="test_cases.json",
+    workspace="/path/to/workspace",
+    test_case_filter=["test_ls_command", "test_echo_command"]
+)
+success = runner.run_tests()
+```
+
 ### Using the Command Line
 
 ```bash
@@ -82,6 +98,12 @@ cli-test run test_cases.json
 
 # Run tests in parallel
 cli-test run test_cases.json --parallel --workers 4
+
+# Run only specified test case(s)
+cli-test run test_cases.json --test-case test_ls_command
+
+# Run multiple specified test cases
+cli-test run test_cases.json -t test_ls_command -t test_echo_command
 ```
 
 ## Test Case Definition
@@ -292,6 +314,54 @@ runner = ParallelJSONRunner(
 success = runner.run_tests()
 ```
 
+## Sequence Testing
+
+Sequence testing allows a single test case to contain multiple ordered steps that execute sequentially. If any step fails, subsequent steps are skipped (fail-fast).
+
+### JSON Format
+
+```json
+{
+    "test_cases": [
+        {
+            "name": "Multi-step Test",
+            "steps": [
+                {
+                    "command": "echo",
+                    "args": ["step1"],
+                    "expected": { "return_code": 0, "output_contains": ["step1"] }
+                },
+                {
+                    "command": "echo",
+                    "args": ["step2"],
+                    "expected": { "return_code": 0, "output_contains": ["step2"] }
+                }
+            ]
+        }
+    ]
+}
+```
+
+### YAML Format
+
+```yaml
+test_cases:
+  - name: Multi-step Test
+    steps:
+      - command: echo
+        args: ["step1"]
+        expected:
+          return_code: 0
+          output_contains: ["step1"]
+      - command: echo
+        args: ["step2"]
+        expected:
+          return_code: 0
+          output_contains: ["step2"]
+```
+
+Each step supports `command`, `args`, `expected`, and `timeout` fields. When a step fails, the overall result reports which step failed (e.g., "Failed at step 2/3: ...").
+
 ## File Comparison
 
 ### Basic File Comparison
@@ -355,6 +425,52 @@ compare-files binary1.bin binary2.bin --chunk-size 16384
 
 ## Advanced Features
 
+### Specifying Test Cases
+
+You can run only specific test cases by name using the `test_case_filter` parameter (Python API) or the `--test-case` / `-t` flag (CLI).
+
+#### CLI Usage
+
+```bash
+# Run a single test case by name
+cli-test run test_cases.json --test-case test_ls_command
+
+# Run multiple test cases (repeat the flag)
+cli-test run test_cases.json -t test_ls_command -t test_echo_command
+
+# Works with parallel mode too
+cli-test run test_cases.json --parallel --test-case test_ls_command
+```
+
+#### Python API Usage
+
+```python
+from cli_test_framework.runners import JSONRunner, ParallelJSONRunner, YAMLRunner
+
+# With JSONRunner
+runner = JSONRunner(
+    config_file="test_cases.json",
+    test_case_filter=["test_ls_command", "test_echo_command"]
+)
+
+# With YAMLRunner
+runner = YAMLRunner(
+    config_file="test_cases.yaml",
+    test_case_filter=["test_case_1"]
+)
+
+# With ParallelJSONRunner
+runner = ParallelJSONRunner(
+    config_file="test_cases.json",
+    test_case_filter=["heavy_test"],
+    max_workers=4
+)
+
+success = runner.run_tests()
+```
+
+If a specified test case name is not found, a warning will be printed. The filter matches test case names exactly.
+
 ### Custom Assertions
 ```python
 from cli_test_framework.assertions import BaseAssertion
@@ -422,24 +538,37 @@ runner = JSONRunner(
 #### JSONRunner
 ```python
 class JSONRunner:
-    def __init__(self, config_file, workspace=None, debug=False):
+    def __init__(self, config_file, workspace=None, test_case_filter=None):
         """
         Initialize JSONRunner
         :param config_file: Path to JSON test case file
         :param workspace: Working directory for test execution
-        :param debug: Enable debug mode
+        :param test_case_filter: Optional list of test case names to run (runs all if None)
+        """
+```
+
+#### YAMLRunner
+```python
+class YAMLRunner:
+    def __init__(self, config_file, workspace=None, test_case_filter=None):
+        """
+        Initialize YAMLRunner
+        :param config_file: Path to YAML test case file
+        :param workspace: Working directory for test execution
+        :param test_case_filter: Optional list of test case names to run (runs all if None)
         """
 ```
 
 #### ParallelJSONRunner
 ```python
 class ParallelJSONRunner:
-    def __init__(self, config_file, max_workers=None, execution_mode="thread"):
+    def __init__(self, config_file, max_workers=None, execution_mode="thread", test_case_filter=None):
         """
         Initialize ParallelJSONRunner
         :param config_file: Path to JSON test case file
         :param max_workers: Maximum number of parallel workers
         :param execution_mode: "thread" or "process"
+        :param test_case_filter: Optional list of test case names to run (runs all if None)
         """
 ```
 

{cli_test_framework-0.5.0 → cli_test_framework-0.5.1}/setup.py

@@ -8,7 +8,7 @@ with open(os.path.join(this_directory, 'README.md'), encoding='utf-8') as f:
 
 setup(
     name="cli-test-framework",
-    version="0.5.0",
+    version="0.5.1",
     author="Xiaotong Wang",
     author_email="xiaotongwang98@gmail.com",
     description="A powerful command line testing framework in Python with setup modules, parallel execution, and file comparison capabilities.",

{cli_test_framework-0.5.0 → cli_test_framework-0.5.1}/src/cli_test_framework/cli.py

@@ -40,6 +40,8 @@ Examples:
                            help='Parallel execution mode (default: thread)')
     run_parser.add_argument('--output-format', choices=['text', 'json', 'html'], default='text',
                            help='Output format for test results')
+    run_parser.add_argument('--test-case', '-t', action='append', default=None,
+                           help='Run only specified test case(s) by name (can be used multiple times)')
     run_parser.add_argument('--verbose', '-v', action='store_true', help='Enable verbose output')
     run_parser.add_argument('--debug', action='store_true', help='Enable debug mode')
     
@@ -64,19 +66,22 @@ def run_tests(args):
                 config_file=str(config_file),
                 workspace=args.workspace,
                 max_workers=args.workers,
-                execution_mode=args.execution_mode
+                execution_mode=args.execution_mode,
+                test_case_filter=args.test_case
             )
         else:
             # Use appropriate single-threaded runner
             if file_ext in ['.json']:
                 runner = JSONRunner(
                     config_file=str(config_file),
-                    workspace=args.workspace
+                    workspace=args.workspace,
+                    test_case_filter=args.test_case
                 )
             elif file_ext in ['.yaml', '.yml']:
                 runner = YAMLRunner(
                     config_file=str(config_file),
-                    workspace=args.workspace
+                    workspace=args.workspace,
+                    test_case_filter=args.test_case
                 )
             else:
                 print(f"Error: Unsupported configuration file format: {file_ext}")

{cli_test_framework-0.5.0 → cli_test_framework-0.5.1}/src/cli_test_framework/core/base_runner.py

@@ -7,13 +7,15 @@ from .setup import SetupManager, EnvironmentSetup
 from .execution import execute_single_test_case
 
 class BaseRunner(ABC):
-    def __init__(self, config_file: str, workspace: Optional[str] = None):
+    def __init__(self, config_file: str, workspace: Optional[str] = None,
+                 test_case_filter: Optional[List[str]] = None):
         if workspace:
             self.workspace = Path(workspace)
         else:
             self.workspace = Path(__file__).parent.parent.parent
         self.config_path = self.workspace / config_file
         self.test_cases: List[TestCase] = []
+        self.test_case_filter: Optional[List[str]] = test_case_filter
         self.results: Dict[str, Any] = {
             "total": 0,
             "passed": 0,
@@ -44,12 +46,28 @@ class BaseRunner(ABC):
         #         # 动态加载自定义setup插件
         #         pass
 
+    def _apply_test_case_filter(self) -> None:
+        """根据 test_case_filter 过滤测试用例"""
+        if self.test_case_filter:
+            original_count = len(self.test_cases)
+            self.test_cases = [tc for tc in self.test_cases if tc.name in self.test_case_filter]
+            filtered_out = original_count - len(self.test_cases)
+            if filtered_out > 0:
+                print(f"Filtered out {filtered_out} test case(s). Running {len(self.test_cases)} specified case(s).")
+            if not self.test_cases:
+                print(f"Warning: No matching test cases found for: {self.test_case_filter}")
+
     def run_tests(self) -> bool:
         """Run all test cases and return whether all tests passed"""
         try:
             self.load_test_cases()
+            self._apply_test_case_filter()
             self.results["total"] = len(self.test_cases)
             
+            if self.results["total"] == 0:
+                print("No test cases to run.")
+                return False
+            
             # 执行setup任务
             self.setup_manager.setup_all()
             

{cli_test_framework-0.5.0 → cli_test_framework-0.5.1}/src/cli_test_framework/core/parallel_runner.py

@@ -12,7 +12,8 @@ class ParallelRunner(BaseRunner):
     
     def __init__(self, config_file: str, workspace: Optional[str] = None, 
                  max_workers: Optional[int] = None, 
-                 execution_mode: str = "thread"):
+                 execution_mode: str = "thread",
+                 test_case_filter: Optional[List[str]] = None):
         """
         初始化并行运行器
         
@@ -21,8 +22,9 @@ class ParallelRunner(BaseRunner):
             workspace: 工作目录
             max_workers: 最大并发数，默认为CPU核心数
             execution_mode: 执行模式，'thread'(线程) 或 'process'(进程)
+            test_case_filter: 只运行指定名称的测试用例
         """
-        super().__init__(config_file, workspace)
+        super().__init__(config_file, workspace, test_case_filter)
         self.max_workers = max_workers
         self.execution_mode = execution_mode
         self.lock = threading.Lock()  # 用于线程安全的结果更新
@@ -31,8 +33,13 @@ class ParallelRunner(BaseRunner):
         """并行运行所有测试用例"""
         try:
             self.load_test_cases()
+            self._apply_test_case_filter()
             self.results["total"] = len(self.test_cases)
             
+            if self.results["total"] == 0:
+                print("No test cases to run.")
+                return False
+            
             # 执行setup任务
             self.setup_manager.setup_all()
             

{cli_test_framework-0.5.0 → cli_test_framework-0.5.1}/src/cli_test_framework/core/test_case.py

@@ -4,6 +4,7 @@ from typing import List, Dict, Any, Optional
 @dataclass
 class TestCaseStep:
     """A single step within a sequence test case."""
+    __test__ = False
     command: str
     args: List[str]
     expected: Dict[str, Any]
@@ -11,6 +12,7 @@ class TestCaseStep:
 
 @dataclass
 class TestCase:
+    __test__ = False
     name: str
     command: str = ""
     args: List[str] = field(default_factory=list)

{cli_test_framework-0.5.0 → cli_test_framework-0.5.1}/src/cli_test_framework/runners/json_runner.py

@@ -8,8 +8,9 @@ import json
 import sys
 
 class JSONRunner(BaseRunner):
-    def __init__(self, config_file="test_cases.json", workspace: Optional[str] = None):
-        super().__init__(config_file, workspace)
+    def __init__(self, config_file="test_cases.json", workspace: Optional[str] = None,
+                 test_case_filter: Optional[list] = None):
+        super().__init__(config_file, workspace, test_case_filter)
         # Backward-compatible attribute for tests that patch path_resolver
         self.path_resolver = PathResolver(self.workspace)
 

{cli_test_framework-0.5.0 → cli_test_framework-0.5.1}/src/cli_test_framework/runners/parallel_json_runner.py

@@ -13,7 +13,8 @@ class ParallelJSONRunner(ParallelRunner):
     """并行JSON测试运行器"""
     
     def __init__(self, config_file="test_cases.json", workspace: Optional[str] = None,
-                 max_workers: Optional[int] = None, execution_mode: str = "thread"):
+                 max_workers: Optional[int] = None, execution_mode: str = "thread",
+                 test_case_filter: Optional[list] = None):
         """
         初始化并行JSON运行器
         
@@ -22,6 +23,7 @@ class ParallelJSONRunner(ParallelRunner):
             workspace: 工作目录
             max_workers: 最大并发数
             execution_mode: 执行模式，'thread' 或 'process'
+            test_case_filter: 只运行指定名称的测试用例
         """
         # 1. 自动感知：获取物理核心数
         # 如果获取失败默认为 4，留 2 个核给系统/Python (防止 GUI 卡死)
@@ -34,7 +36,7 @@ class ParallelJSONRunner(ParallelRunner):
         if max_workers is None:
             max_workers = self.total_physical
             
-        super().__init__(config_file, workspace, max_workers, execution_mode)
+        super().__init__(config_file, workspace, max_workers, execution_mode, test_case_filter)
         # Backward-compatible attribute for potential external patches/tests
         self.path_resolver = PathResolver(self.workspace)
         self._print_lock = threading.Lock()  # 用于控制输出顺序

{cli_test_framework-0.5.0 → cli_test_framework-0.5.1}/src/cli_test_framework/runners/yaml_runner.py

@@ -7,8 +7,9 @@ from ..utils.path_resolver import PathResolver, parse_command_string, resolve_pa
 import sys
 
 class YAMLRunner(BaseRunner):
-    def __init__(self, config_file="test_cases.yaml", workspace: Optional[str] = None):
-        super().__init__(config_file, workspace)
+    def __init__(self, config_file="test_cases.yaml", workspace: Optional[str] = None,
+                 test_case_filter: Optional[list] = None):
+        super().__init__(config_file, workspace, test_case_filter)
         # Backward-compatible attribute for tests that patch path_resolver
         self.path_resolver = PathResolver(self.workspace)
 

{cli_test_framework-0.5.0 → cli_test_framework-0.5.1/src/cli_test_framework.egg-info}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: cli-test-framework
-Version: 0.5.0
+Version: 0.5.1
 Summary: A powerful command line testing framework in Python with setup modules, parallel execution, and file comparison capabilities.
 Home-page: https://github.com/ozil111/cli-test-framework
 Author: Xiaotong Wang

{cli_test_framework-0.5.0 → cli_test_framework-0.5.1}/src/cli_test_framework.egg-info/SOURCES.txt

@@ -68,6 +68,9 @@ tests/fixtures/test_cases.yaml
 tests/fixtures/test_cases1.json
 tests/fixtures/test_with_setup.json
 tests/fixtures/test_with_setup.yaml
+tests/integration/test_sequence.py
+tests/integration/__pycache__/test_sequence.cpython-312-pytest-7.4.4.pyc
+tests/integration/__pycache__/test_sequence.cpython-312-pytest-9.0.3.pyc
 tests/integration/file_compare/test_binary_compare.py
 tests/integration/file_compare/test_h5_compare.py
 tests/integration/file_compare/test_json_compare.py
@@ -97,6 +100,9 @@ tests/unit/core/__pycache__/test_setup.cpython-312-pytest-7.4.4.pyc
 tests/unit/core/__pycache__/test_setup.cpython-312-pytest-9.0.3.pyc
 tests/unit/core/__pycache__/test_setup.cpython-39-pytest-8.3.4.pyc
 tests/unit/runners/test_json_yaml_runner.py
+tests/unit/runners/test_test_case_filter.py
 tests/unit/runners/__pycache__/test_json_yaml_runner.cpython-312-pytest-7.4.4.pyc
 tests/unit/runners/__pycache__/test_json_yaml_runner.cpython-312-pytest-9.0.3.pyc
-tests/unit/runners/__pycache__/test_json_yaml_runner.cpython-39-pytest-8.3.4.pyc
+tests/unit/runners/__pycache__/test_json_yaml_runner.cpython-39-pytest-8.3.4.pyc
+tests/unit/runners/__pycache__/test_test_case_filter.cpython-312-pytest-7.4.4.pyc
+tests/unit/runners/__pycache__/test_test_case_filter.cpython-312-pytest-9.0.3.pyc