cli-test-framework 0.4.5__tar.gz → 0.5.1__tar.gz

{cli_test_framework-0.4.5/src/cli_test_framework.egg-info → cli_test_framework-0.5.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: cli-test-framework
-Version: 0.4.5
+Version: 0.5.1
 Summary: A powerful command line testing framework in Python with setup modules, parallel execution, and file comparison capabilities.
 Home-page: https://github.com/ozil111/cli-test-framework
 Author: Xiaotong Wang
@@ -18,8 +18,10 @@ Classifier: Topic :: Software Development :: Libraries :: Python Modules
 Requires-Python: >=3.9
 Description-Content-Type: text/markdown
 Requires-Dist: dukpy==0.5.0
-Requires-Dist: h5py>=3.8.0
-Requires-Dist: numpy>=1.21.0
+Requires-Dist: h5py<4.0.0,>=3.8.0; python_version < "3.12"
+Requires-Dist: h5py<4.0.0,>=3.10.0; python_version >= "3.12"
+Requires-Dist: numpy<2.0.0,>=1.21.0; python_version < "3.12"
+Requires-Dist: numpy<2.0.0,>=1.26.0; python_version >= "3.12"
 Requires-Dist: setuptools>=75.8.0
 Requires-Dist: wheel>=0.45.1
 Dynamic: author

{cli_test_framework-0.4.5 → cli_test_framework-0.5.1}/docs/user_manual.md

@@ -7,11 +7,14 @@
 4. [Test Case Definition](#test-case-definition)
 5. [Setup Module](#setup-module)
 6. [Parallel Testing](#parallel-testing)
-7. [File Comparison](#file-comparison)
-8. [Advanced Features](#advanced-features)
-9. [Troubleshooting](#troubleshooting)
-10. [API Reference](#api-reference)
-11. [Examples](#examples)
+7. [Sequence Testing](#sequence-testing)
+8. [File Comparison](#file-comparison)
+9. [Advanced Features](#advanced-features)
+10. [Troubleshooting](#troubleshooting)
+11. [API Reference](#api-reference)
+12. [Examples](#examples)
+
+
 
 ## Introduction
 
@@ -19,6 +22,7 @@ The CLI Testing Framework is a powerful tool designed for testing command-line a
 
 ### Key Features
 - Parallel test execution with thread and process support
+- Sequence test execution with multi-step fail-fast
 - JSON/YAML test case definition
 - Advanced file comparison capabilities
 - Comprehensive reporting
@@ -74,6 +78,18 @@ runner = JSONRunner(
 success = runner.run_tests()
 ```
 
+3. Run only specified test case(s):
+```python
+from cli_test_framework.runners import JSONRunner
+
+runner = JSONRunner(
+    config_file="test_cases.json",
+    workspace="/path/to/workspace",
+    test_case_filter=["test_ls_command", "test_echo_command"]
+)
+success = runner.run_tests()
+```
+
 ### Using the Command Line
 
 ```bash
@@ -82,6 +98,12 @@ cli-test run test_cases.json
 
 # Run tests in parallel
 cli-test run test_cases.json --parallel --workers 4
+
+# Run only specified test case(s)
+cli-test run test_cases.json --test-case test_ls_command
+
+# Run multiple specified test cases
+cli-test run test_cases.json -t test_ls_command -t test_echo_command
 ```
 
 ## Test Case Definition
@@ -292,6 +314,54 @@ runner = ParallelJSONRunner(
 success = runner.run_tests()
 ```
 
+## Sequence Testing
+
+Sequence testing allows a single test case to contain multiple ordered steps that execute sequentially. If any step fails, subsequent steps are skipped (fail-fast).
+
+### JSON Format
+
+```json
+{
+    "test_cases": [
+        {
+            "name": "Multi-step Test",
+            "steps": [
+                {
+                    "command": "echo",
+                    "args": ["step1"],
+                    "expected": { "return_code": 0, "output_contains": ["step1"] }
+                },
+                {
+                    "command": "echo",
+                    "args": ["step2"],
+                    "expected": { "return_code": 0, "output_contains": ["step2"] }
+                }
+            ]
+        }
+    ]
+}
+```
+
+### YAML Format
+
+```yaml
+test_cases:
+  - name: Multi-step Test
+    steps:
+      - command: echo
+        args: ["step1"]
+        expected:
+          return_code: 0
+          output_contains: ["step1"]
+      - command: echo
+        args: ["step2"]
+        expected:
+          return_code: 0
+          output_contains: ["step2"]
+```
+
+Each step supports `command`, `args`, `expected`, and `timeout` fields. When a step fails, the overall result reports which step failed (e.g., "Failed at step 2/3: ...").
+
 ## File Comparison
 
 ### Basic File Comparison
@@ -355,6 +425,52 @@ compare-files binary1.bin binary2.bin --chunk-size 16384
 
 ## Advanced Features
 
+### Specifying Test Cases
+
+You can run only specific test cases by name using the `test_case_filter` parameter (Python API) or the `--test-case` / `-t` flag (CLI).
+
+#### CLI Usage
+
+```bash
+# Run a single test case by name
+cli-test run test_cases.json --test-case test_ls_command
+
+# Run multiple test cases (repeat the flag)
+cli-test run test_cases.json -t test_ls_command -t test_echo_command
+
+# Works with parallel mode too
+cli-test run test_cases.json --parallel --test-case test_ls_command
+```
+
+#### Python API Usage
+
+```python
+from cli_test_framework.runners import JSONRunner, ParallelJSONRunner, YAMLRunner
+
+# With JSONRunner
+runner = JSONRunner(
+    config_file="test_cases.json",
+    test_case_filter=["test_ls_command", "test_echo_command"]
+)
+
+# With YAMLRunner
+runner = YAMLRunner(
+    config_file="test_cases.yaml",
+    test_case_filter=["test_case_1"]
+)
+
+# With ParallelJSONRunner
+runner = ParallelJSONRunner(
+    config_file="test_cases.json",
+    test_case_filter=["heavy_test"],
+    max_workers=4
+)
+
+success = runner.run_tests()
+```
+
+If a specified test case name is not found, a warning will be printed. The filter matches test case names exactly.
+
 ### Custom Assertions
 ```python
 from cli_test_framework.assertions import BaseAssertion
@@ -422,24 +538,37 @@ runner = JSONRunner(
 #### JSONRunner
 ```python
 class JSONRunner:
-    def __init__(self, config_file, workspace=None, debug=False):
+    def __init__(self, config_file, workspace=None, test_case_filter=None):
         """
         Initialize JSONRunner
         :param config_file: Path to JSON test case file
         :param workspace: Working directory for test execution
-        :param debug: Enable debug mode
+        :param test_case_filter: Optional list of test case names to run (runs all if None)
+        """
+```
+
+#### YAMLRunner
+```python
+class YAMLRunner:
+    def __init__(self, config_file, workspace=None, test_case_filter=None):
+        """
+        Initialize YAMLRunner
+        :param config_file: Path to YAML test case file
+        :param workspace: Working directory for test execution
+        :param test_case_filter: Optional list of test case names to run (runs all if None)
         """
 ```
 
 #### ParallelJSONRunner
 ```python
 class ParallelJSONRunner:
-    def __init__(self, config_file, max_workers=None, execution_mode="thread"):
+    def __init__(self, config_file, max_workers=None, execution_mode="thread", test_case_filter=None):
         """
         Initialize ParallelJSONRunner
         :param config_file: Path to JSON test case file
         :param max_workers: Maximum number of parallel workers
         :param execution_mode: "thread" or "process"
+        :param test_case_filter: Optional list of test case names to run (runs all if None)
         """
 ```
 

{cli_test_framework-0.4.5 → cli_test_framework-0.5.1}/setup.py

@@ -8,7 +8,7 @@ with open(os.path.join(this_directory, 'README.md'), encoding='utf-8') as f:
 
 setup(
     name="cli-test-framework",
-    version="0.4.5",
+    version="0.5.1",
     author="Xiaotong Wang",
     author_email="xiaotongwang98@gmail.com",
     description="A powerful command line testing framework in Python with setup modules, parallel execution, and file comparison capabilities.",
@@ -19,8 +19,10 @@ setup(
     package_dir={"": "src"},
     install_requires=[
         "dukpy==0.5.0",
-        "h5py>=3.8.0",
-        "numpy>=1.21.0",
+        "h5py>=3.8.0,<4.0.0; python_version<'3.12'",
+        "h5py>=3.10.0,<4.0.0; python_version>='3.12'",
+        "numpy>=1.21.0,<2.0.0; python_version<'3.12'",
+        "numpy>=1.26.0,<2.0.0; python_version>='3.12'",
         "setuptools>=75.8.0",
         "wheel>=0.45.1"
     ],
@@ -45,4 +47,4 @@ setup(
         'Source': 'https://github.com/ozil111/cli-test-framework',
         'Tracker': 'https://github.com/ozil111/cli-test-framework/issues',
     },
-)
+)

{cli_test_framework-0.4.5 → cli_test_framework-0.5.1}/src/cli_test_framework/cli.py

@@ -40,6 +40,8 @@ Examples:
                            help='Parallel execution mode (default: thread)')
     run_parser.add_argument('--output-format', choices=['text', 'json', 'html'], default='text',
                            help='Output format for test results')
+    run_parser.add_argument('--test-case', '-t', action='append', default=None,
+                           help='Run only specified test case(s) by name (can be used multiple times)')
     run_parser.add_argument('--verbose', '-v', action='store_true', help='Enable verbose output')
     run_parser.add_argument('--debug', action='store_true', help='Enable debug mode')
     
@@ -64,19 +66,22 @@ def run_tests(args):
                 config_file=str(config_file),
                 workspace=args.workspace,
                 max_workers=args.workers,
-                execution_mode=args.execution_mode
+                execution_mode=args.execution_mode,
+                test_case_filter=args.test_case
             )
         else:
             # Use appropriate single-threaded runner
             if file_ext in ['.json']:
                 runner = JSONRunner(
                     config_file=str(config_file),
-                    workspace=args.workspace
+                    workspace=args.workspace,
+                    test_case_filter=args.test_case
                 )
             elif file_ext in ['.yaml', '.yml']:
                 runner = YAMLRunner(
                     config_file=str(config_file),
-                    workspace=args.workspace
+                    workspace=args.workspace,
+                    test_case_filter=args.test_case
                 )
             else:
                 print(f"Error: Unsupported configuration file format: {file_ext}")

cli_test_framework-0.5.1/src/cli_test_framework/core/base_runner.py

@@ -0,0 +1,162 @@
+from abc import ABC, abstractmethod
+from pathlib import Path
+from typing import List, Dict, Any, Optional
+from .test_case import TestCase
+from .assertions import Assertions
+from .setup import SetupManager, EnvironmentSetup
+from .execution import execute_single_test_case
+
+class BaseRunner(ABC):
+    def __init__(self, config_file: str, workspace: Optional[str] = None,
+                 test_case_filter: Optional[List[str]] = None):
+        if workspace:
+            self.workspace = Path(workspace)
+        else:
+            self.workspace = Path(__file__).parent.parent.parent
+        self.config_path = self.workspace / config_file
+        self.test_cases: List[TestCase] = []
+        self.test_case_filter: Optional[List[str]] = test_case_filter
+        self.results: Dict[str, Any] = {
+            "total": 0,
+            "passed": 0,
+            "failed": 0,
+            "details": []
+        }
+        self.assertions = Assertions()
+        self.setup_manager = SetupManager()
+
+    @abstractmethod
+    def load_test_cases(self) -> None:
+        """Load test cases from configuration file"""
+        pass
+    
+    def load_setup_from_config(self, config: Dict[str, Any]) -> None:
+        """从配置文件加载setup配置"""
+        setup_config = config.get("setup", {})
+        
+        # 处理环境变量设置
+        if "environment_variables" in setup_config:
+            env_setup = EnvironmentSetup({"environment_variables": setup_config["environment_variables"]})
+            self.setup_manager.add_setup(env_setup)
+        
+        # 这里可以扩展支持其他类型的setup插件
+        # 例如：
+        # if "custom_setups" in setup_config:
+        #     for custom_setup_config in setup_config["custom_setups"]:
+        #         # 动态加载自定义setup插件
+        #         pass
+
+    def _apply_test_case_filter(self) -> None:
+        """根据 test_case_filter 过滤测试用例"""
+        if self.test_case_filter:
+            original_count = len(self.test_cases)
+            self.test_cases = [tc for tc in self.test_cases if tc.name in self.test_case_filter]
+            filtered_out = original_count - len(self.test_cases)
+            if filtered_out > 0:
+                print(f"Filtered out {filtered_out} test case(s). Running {len(self.test_cases)} specified case(s).")
+            if not self.test_cases:
+                print(f"Warning: No matching test cases found for: {self.test_case_filter}")
+
+    def run_tests(self) -> bool:
+        """Run all test cases and return whether all tests passed"""
+        try:
+            self.load_test_cases()
+            self._apply_test_case_filter()
+            self.results["total"] = len(self.test_cases)
+            
+            if self.results["total"] == 0:
+                print("No test cases to run.")
+                return False
+            
+            # 执行setup任务
+            self.setup_manager.setup_all()
+            
+            print(f"\nStarting test execution... Total tests: {self.results['total']}")
+            print("=" * 50)
+            
+            for i, case in enumerate(self.test_cases, 1):
+                print(f"\nRunning test {i}/{self.results['total']}: {case.name}")
+                result = self.run_single_test(case)
+                self.results["details"].append(result)
+                if result["status"] == "passed":
+                    self.results["passed"] += 1
+                    print(f"✓ Test passed: {case.name}")
+                else:
+                    self.results["failed"] += 1
+                    print(f"✗ Test failed: {case.name}")
+                    if result["message"]:
+                        print(f"  Error: {result['message']}")
+                    
+            print("\n" + "=" * 50)
+            print(f"Test execution completed. Passed: {self.results['passed']}, Failed: {self.results['failed']}")
+            return self.results["failed"] == 0
+        finally:
+            # 确保teardown总是被执行
+            self.setup_manager.teardown_all()
+
+    def _run_sequence(self, case: TestCase) -> Dict[str, Any]:
+        """Run a sequence test case with multiple steps (fail-fast)."""
+        combined_output = ""
+        total_duration = 0.0
+        all_passed = True
+        last_result = None
+        failed_step = None
+
+        for i, step in enumerate(case.steps):
+            step_name = f"{case.name} [step {i+1}/{len(case.steps)}]"
+            case_data = {
+                "name": step_name,
+                "command": step.command,
+                "args": step.args,
+                "expected": step.expected,
+                "description": None,
+                "timeout": step.timeout,
+                "resources": None,
+            }
+
+            command_preview = f"{step.command} {' '.join(step.args)}".strip()
+            print(f"  Executing step {i+1}/{len(case.steps)}: {command_preview}")
+
+            result = execute_single_test_case(
+                case_data, str(self.workspace) if self.workspace else None
+            )
+
+            if result["output"].strip():
+                print("  Command output:")
+                for line in result["output"].splitlines():
+                    print(f"    {line}")
+
+            combined_output += result["output"]
+            total_duration += result["duration"]
+            last_result = result
+
+            if result["status"] != "passed":
+                all_passed = False
+                failed_step = i + 1
+                if result.get("message"):
+                    print(f"  Error at step {i+1}: {result['message']}")
+                break
+
+        status = "passed" if all_passed else last_result["status"]
+        message = ""
+        if not all_passed:
+            message = f"Failed at step {failed_step}/{len(case.steps)}: {last_result['message']}"
+
+        command_summary = " -> ".join(
+            f"{s.command} {' '.join(s.args)}".strip() for s in case.steps
+        )
+
+        return {
+            "name": case.name,
+            "status": status,
+            "message": message,
+            "command": command_summary,
+            "output": combined_output,
+            "return_code": last_result["return_code"] if last_result else None,
+            "duration": total_duration,
+        }
+
+    @abstractmethod
+    def run_single_test(self, case: TestCase) -> Dict[str, str]:
+        """Run a single test case and return the result"""
+        pass

{cli_test_framework-0.4.5 → cli_test_framework-0.5.1}/src/cli_test_framework/core/parallel_runner.py

@@ -12,7 +12,8 @@ class ParallelRunner(BaseRunner):
     
     def __init__(self, config_file: str, workspace: Optional[str] = None, 
                  max_workers: Optional[int] = None, 
-                 execution_mode: str = "thread"):
+                 execution_mode: str = "thread",
+                 test_case_filter: Optional[List[str]] = None):
         """
         初始化并行运行器
         
@@ -21,8 +22,9 @@ class ParallelRunner(BaseRunner):
             workspace: 工作目录
             max_workers: 最大并发数，默认为CPU核心数
             execution_mode: 执行模式，'thread'(线程) 或 'process'(进程)
+            test_case_filter: 只运行指定名称的测试用例
         """
-        super().__init__(config_file, workspace)
+        super().__init__(config_file, workspace, test_case_filter)
         self.max_workers = max_workers
         self.execution_mode = execution_mode
         self.lock = threading.Lock()  # 用于线程安全的结果更新
@@ -31,8 +33,13 @@ class ParallelRunner(BaseRunner):
         """并行运行所有测试用例"""
         try:
             self.load_test_cases()
+            self._apply_test_case_filter()
             self.results["total"] = len(self.test_cases)
             
+            if self.results["total"] == 0:
+                print("No test cases to run.")
+                return False
+            
             # 执行setup任务
             self.setup_manager.setup_all()
             
@@ -61,7 +68,16 @@ class ParallelRunner(BaseRunner):
                                 "args": case.args,
                                 "expected": case.expected,
                                 "timeout": case.timeout,
-                                "resources": case.resources
+                                "resources": case.resources,
+                                "steps": [
+                                    {
+                                        "command": s.command,
+                                        "args": s.args,
+                                        "expected": s.expected,
+                                        "timeout": s.timeout,
+                                    }
+                                    for s in case.steps
+                                ] if case.steps else None,
                             },
                             str(self.workspace) if self.workspace else None
                         ): (i, case) 

cli_test_framework-0.5.1/src/cli_test_framework/core/process_worker.py

@@ -0,0 +1,111 @@
+"""
+进程工作器模块
+用于多进程并行测试执行，避免序列化问题
+"""
+
+from typing import Dict, Any, List
+from .execution import execute_single_test_case
+from .types import TestCaseData
+
+def _run_sequence_in_process(test_index: int, case_data: Dict[str, Any], workspace: str = None) -> Dict[str, Any]:
+    """Run a sequence test case with multiple steps (fail-fast) in a process worker."""
+    steps: List[Dict[str, Any]] = case_data["steps"]
+    combined_output = ""
+    total_duration = 0.0
+    all_passed = True
+    last_result = None
+    failed_step = None
+
+    for i, step in enumerate(steps):
+        step_name = f"{case_data['name']} [step {i+1}/{len(steps)}]"
+        step_case: TestCaseData = {
+            "name": step_name,
+            "command": step["command"],
+            "args": step["args"],
+            "expected": step["expected"],
+            "description": None,
+            "timeout": step.get("timeout"),
+            "resources": None,
+        }
+
+        command_preview = f"{step['command']} {' '.join(step['args'])}".strip()
+        print(f"  [Process Worker {test_index}] Executing step {i+1}/{len(steps)}: {command_preview}")
+
+        result = execute_single_test_case(step_case, workspace)
+
+        if result["output"].strip():
+            print(f"  [Process Worker {test_index}] Command output for {step_name}:")
+            for line in result["output"].splitlines():
+                print(f"    {line}")
+
+        combined_output += result["output"]
+        total_duration += result["duration"]
+        last_result = result
+
+        if result["status"] != "passed":
+            all_passed = False
+            failed_step = i + 1
+            if result.get("message"):
+                print(f"  [Process Worker {test_index}] Error at step {i+1}: {result['message']}")
+            break
+
+    status = "passed" if all_passed else last_result["status"]
+    message = ""
+    if not all_passed:
+        message = f"Failed at step {failed_step}/{len(steps)}: {last_result['message']}"
+
+    command_summary = " -> ".join(
+        f"{s['command']} {' '.join(s['args'])}".strip() for s in steps
+    )
+
+    return {
+        "name": case_data["name"],
+        "status": status,
+        "message": message,
+        "command": command_summary,
+        "output": combined_output,
+        "return_code": last_result["return_code"] if last_result else None,
+        "duration": total_duration,
+    }
+
+def run_test_in_process(test_index: int, case_data: Dict[str, Any], workspace: str = None) -> Dict[str, Any]:
+    """
+    在独立进程中运行单个测试用例
+    
+    Args:
+        test_index: 测试索引
+        case_data: 测试用例数据字典
+        workspace: 工作目录
+    
+    Returns:
+        测试结果字典
+    """
+    # Sequence mode
+    if case_data.get("steps"):
+        return _run_sequence_in_process(test_index, case_data, workspace)
+
+    # Single command mode
+    case: TestCaseData = {
+        "name": case_data["name"],
+        "command": case_data["command"],
+        "args": case_data["args"],
+        "expected": case_data["expected"],
+        "description": case_data.get("description"),
+        "timeout": case_data.get("timeout"),
+        "resources": case_data.get("resources"),
+    }
+
+    command_preview = f"{case['command']} {' '.join(case['args'])}".strip()
+    print(f"  [Process Worker {test_index}] Executing command: {command_preview}")
+
+    result = execute_single_test_case(case, workspace)
+
+    if result["output"].strip():
+        print(f"  [Process Worker {test_index}] Command output for {case['name']}:")
+        for line in result["output"].splitlines():
+            print(f"    {line}")
+
+    if result["status"] != "passed" and result.get("message"):
+        print(f"  [Process Worker {test_index}] Error for {case['name']}: {result['message']}")
+
+    return result 

cli_test_framework-0.5.1/src/cli_test_framework/core/test_case.py

@@ -0,0 +1,47 @@
+from dataclasses import dataclass, field
+from typing import List, Dict, Any, Optional
+
+@dataclass
+class TestCaseStep:
+    """A single step within a sequence test case."""
+    __test__ = False
+    command: str
+    args: List[str]
+    expected: Dict[str, Any]
+    timeout: Optional[float] = None
+
+@dataclass
+class TestCase:
+    __test__ = False
+    name: str
+    command: str = ""
+    args: List[str] = field(default_factory=list)
+    expected: Dict[str, Any] = field(default_factory=dict)
+    description: str = ""
+    timeout: Optional[float] = None
+    resources: Optional[Dict[str, Any]] = None
+    steps: Optional[List[TestCaseStep]] = None
+    
+    def to_dict(self) -> Dict[str, Any]:
+        """Convert test case to dictionary format"""
+        print("Convert test case to dictionary format")
+        print(self.command)
+        result = {
+            "name": self.name,
+            "command": self.command,
+            "args": self.args,
+            "expected": self.expected,
+            "timeout": self.timeout,
+            "resources": self.resources,
+        }
+        if self.steps is not None:
+            result["steps"] = [
+                {
+                    "command": s.command,
+                    "args": s.args,
+                    "expected": s.expected,
+                    "timeout": s.timeout,
+                }
+                for s in self.steps
+            ]
+        return result