cli-test-framework 0.2.3__tar.gz → 0.3.0__tar.gz

{cli_test_framework-0.2.3 → cli_test_framework-0.3.0}/CHANGELOG.md

@@ -1,5 +1,65 @@
 # 变更日志
 
+## [v0.3.0] - 2025-06-07
+
+### 🎉 重大新功能 - Setup模块系统
+
+#### 插件化Setup架构
+- **BaseSetup抽象基类**：定义了setup插件的标准接口，支持setup()和teardown()方法
+- **SetupManager管理器**：统一管理多个setup插件的生命周期
+- **可扩展设计**：用户可以轻松创建自定义setup插件
+
+#### 内置环境变量插件
+- **EnvironmentSetup插件**：自动设置和管理环境变量
+- **智能恢复机制**：测试完成后自动恢复原始环境状态，支持新增和覆盖的环境变量
+- **配置文件集成**：直接在JSON/YAML测试配置中定义环境变量
+
+#### 全面Runner集成
+- **JSONRunner**：完全支持setup配置的加载和执行
+- **YAMLRunner**：完全支持setup配置的加载和执行  
+- **ParallelJSONRunner**：完全支持setup配置，在并行模式下正确处理setup/teardown
+- **自动生命周期**：setup和teardown在测试前后自动执行
+
+#### 配置文件语法扩展
+```json
+{
+  "setup": {
+    "environment_variables": {
+      "TEST_ENV": "development",
+      "API_URL": "http://localhost:8080"
+    }
+  },
+  "test_cases": [...]
+}
+```
+
+### 🔧 改进和修复
+- **Python版本要求**：从Python 3.6+更新为Python 3.9+（由于依赖版本要求）
+- **错误处理增强**：改进了setup/teardown的异常处理，使用try-finally确保清理总是执行
+- **并行安全**：Setup模块在并行执行模式下的线程安全处理
+- **包导出修复**：移除了错误的FileComparator导出
+
+### 🧪 测试覆盖
+- **新增16个单元测试**：覆盖setup模块所有核心功能
+- **集成测试**：确保与所有runner的兼容性
+- **边界条件测试**：测试空配置、错误情况等边界条件
+
+### 📚 文档更新
+- **用户手册扩展**：新增完整的Setup模块章节，包含详细使用说明和最佳实践
+- **示例代码**：提供了完整的使用示例和自定义插件示例
+- **API文档**：完整的setup模块API文档和类型注解
+
+### ⚠️ 破坏性变更
+- **Python版本要求**：从3.6+更新为3.9+
+- **包结构调整**：setup模块相关类现在从cli_test_framework包导出
+
+### 🚀 迁移指南
+- **现有代码兼容**：现有的测试配置文件无需修改即可正常工作
+- **新功能使用**：在配置文件中添加setup部分即可使用环境变量设置
+- **自定义插件**：继承BaseSetup类创建自定义setup插件
+
+---
+
 ## [v2.0.0] - 2025-05-27
 
 ### 🚀 重大新功能

{cli_test_framework-0.2.3/src/cli_test_framework.egg-info → cli_test_framework-0.3.0}/PKG-INFO

@@ -1,7 +1,7 @@
 Metadata-Version: 2.4
 Name: cli-test-framework
-Version: 0.2.3
-Summary: A small command line testing framework in Python with file comparison capabilities.
+Version: 0.3.0
+Summary: A powerful command line testing framework in Python with setup modules, parallel execution, and file comparison capabilities.
 Home-page: https://github.com/yourusername/cli-test-framework
 Author: Xiaotong Wang
 Author-email: xiaotongwang98@gmail.com
@@ -15,7 +15,7 @@ Classifier: Development Status :: 4 - Beta
 Classifier: Intended Audience :: Developers
 Classifier: Topic :: Software Development :: Testing
 Classifier: Topic :: Software Development :: Libraries :: Python Modules
-Requires-Python: >=3.6
+Requires-Python: >=3.9
 Description-Content-Type: text/markdown
 Requires-Dist: dukpy==0.5.0
 Requires-Dist: h5py>=3.8.0

{cli_test_framework-0.2.3 → cli_test_framework-0.3.0}/docs/user_manual.md

@@ -5,12 +5,13 @@
 2. [Installation](#installation)
 3. [Basic Usage](#basic-usage)
 4. [Test Case Definition](#test-case-definition)
-5. [Parallel Testing](#parallel-testing)
-6. [File Comparison](#file-comparison)
-7. [Advanced Features](#advanced-features)
-8. [Troubleshooting](#troubleshooting)
-9. [API Reference](#api-reference)
-10. [Examples](#examples)
+5. [Setup Module](#setup-module)
+6. [Parallel Testing](#parallel-testing)
+7. [File Comparison](#file-comparison)
+8. [Advanced Features](#advanced-features)
+9. [Troubleshooting](#troubleshooting)
+10. [API Reference](#api-reference)
+11. [Examples](#examples)
 
 ## Introduction
 
@@ -121,6 +122,152 @@ test_cases:
         - ".*regex pattern.*"
 ```
 
+## Setup Module
+
+The Setup Module provides a plugin-based system for executing pre-test setup tasks and post-test cleanup. It's designed to be extensible and supports multiple setup plugins running in sequence.
+
+### Key Features
+- **Plugin Architecture**: Extensible design allowing custom setup plugins
+- **Built-in Environment Plugin**: Set environment variables for tests
+- **Full Runner Support**: Works with JSONRunner, YAMLRunner, and ParallelJSONRunner
+- **Lifecycle Management**: Automatic setup and teardown with proper cleanup
+
+### Environment Variable Setup
+
+#### JSON Configuration
+```json
+{
+  "setup": {
+    "environment_variables": {
+      "TEST_ENV": "development",
+      "DEBUG_MODE": "true",
+      "API_URL": "http://localhost:8080",
+      "DATABASE_URL": "sqlite:///test.db"
+    }
+  },
+  "test_cases": [
+    {
+      "name": "Test with environment variables",
+      "command": "python",
+      "args": ["-c", "import os; print(f'Env: {os.environ.get(\"TEST_ENV\")}')"],
+      "expected": {
+        "return_code": 0,
+        "output_contains": ["Env: development"]
+      }
+    }
+  ]
+}
+```
+
+#### YAML Configuration
+```yaml
+setup:
+  environment_variables:
+    TEST_ENV: "production"
+    DATABASE_URL: "postgresql://localhost:5432/test"
+    MAX_CONNECTIONS: "10"
+    TIMEOUT_SECONDS: "30"
+
+test_cases:
+  - name: "Test database environment"
+    command: "python"
+    args: 
+      - "-c"
+      - "import os; print(f'DB: {os.environ.get(\"DATABASE_URL\")}')"
+    expected:
+      return_code: 0
+      output_contains:
+        - "DB: postgresql://localhost:5432/test"
+```
+
+### Custom Setup Plugins
+
+#### Creating Custom Plugins
+```python
+from cli_test_framework import BaseSetup
+
+class DatabaseSetup(BaseSetup):
+    def setup(self):
+        """Initialize test database"""
+        print("Setting up test database...")
+        # Your database initialization code here
+        
+    def teardown(self):
+        """Clean up test database"""
+        print("Cleaning up test database...")
+        # Your database cleanup code here
+
+class ServiceSetup(BaseSetup):
+    def setup(self):
+        """Start test services"""
+        self.service_port = self.config.get('port', 8080)
+        print(f"Starting test service on port {self.service_port}")
+        # Your service startup code here
+        
+    def teardown(self):
+        """Stop test services"""
+        print("Stopping test services...")
+        # Your service shutdown code here
+```
+
+#### Using Custom Plugins
+```python
+from cli_test_framework import JSONRunner
+
+# Create runner
+runner = JSONRunner("test_cases.json")
+
+# Add custom setup plugins
+db_setup = DatabaseSetup({"connection": "test_db"})
+service_setup = ServiceSetup({"port": 9090})
+
+runner.setup_manager.add_setup(db_setup)
+runner.setup_manager.add_setup(service_setup)
+
+# Run tests (setup will be executed automatically)
+success = runner.run_tests()
+```
+
+### Execution Flow
+
+1. **Load Configuration**: Read setup configuration from test file
+2. **Setup Phase**: Execute all setup plugins in order
+   - Environment variables are set
+   - Custom setups are executed
+   - Setup status is reported
+3. **Test Execution**: Run all test cases with setup environment
+4. **Teardown Phase**: Clean up all setups in reverse order
+   - Environment variables are restored
+   - Custom cleanups are executed
+   - Cleanup is guaranteed even if tests fail
+
+### Best Practices
+
+1. **Idempotent Operations**: Make setup operations safe to run multiple times
+2. **Proper Cleanup**: Always implement teardown to avoid side effects
+3. **Error Handling**: Setup failures stop test execution, teardown failures don't
+4. **Resource Management**: Use try-finally patterns in custom plugins
+5. **Configuration Validation**: Check required configuration parameters in setup
+
+### Integration Examples
+
+#### With JSON Runner
+```bash
+cli-test test_with_setup.json
+```
+
+#### With YAML Runner
+```bash
+cli-test test_with_setup.yaml --runner yaml
+```
+
+#### With Parallel Runner
+```bash
+cli-test test_with_setup.json --runner parallel --max-workers 4
+```
+
+Note: In parallel mode, setup and teardown run in the main thread to ensure environment consistency.
+
 ## Parallel Testing
 
 ### Thread Mode

{cli_test_framework-0.2.3 → cli_test_framework-0.3.0}/setup.py

@@ -8,10 +8,10 @@ with open(os.path.join(this_directory, 'README.md'), encoding='utf-8') as f:
 
 setup(
     name="cli-test-framework",
-    version="0.2.3",
+    version="0.3.0",
     author="Xiaotong Wang",
     author_email="xiaotongwang98@gmail.com",
-    description="A small command line testing framework in Python with file comparison capabilities.",
+    description="A powerful command line testing framework in Python with setup modules, parallel execution, and file comparison capabilities.",
     long_description=long_description,
     long_description_content_type="text/markdown",
     url="https://github.com/yourusername/cli-test-framework",
@@ -39,7 +39,7 @@ setup(
         "Topic :: Software Development :: Testing",
         "Topic :: Software Development :: Libraries :: Python Modules",
     ],
-    python_requires='>=3.6',
+    python_requires='>=3.9',
     project_urls={
         'Documentation': 'https://github.com/yourusername/cli-test-framework/docs/user_manual.md',
         'Source': 'https://github.com/yourusername/cli-test-framework',

{cli_test_framework-0.2.3 → cli_test_framework-0.3.0}/src/cli_test_framework/__init__.py

@@ -5,7 +5,7 @@ This package provides tools for testing command-line applications and scripts
 with support for parallel execution and advanced file comparison capabilities.
 """
 
-__version__ = "0.2.3"
+__version__ = "0.3.0"
 __author__ = "Xiaotong Wang"
 __email__ = "xiaotongwang98@gmail.com"
 
@@ -13,9 +13,17 @@ __email__ = "xiaotongwang98@gmail.com"
 from .runners.json_runner import JSONRunner
 from .runners.parallel_json_runner import ParallelJSONRunner
 from .runners.yaml_runner import YAMLRunner
+from .core.test_case import TestCase
+from .core.assertions import Assertions
+from .core.setup import BaseSetup, EnvironmentSetup, SetupManager
 
 __all__ = [
     'JSONRunner',
     'ParallelJSONRunner', 
     'YAMLRunner',
+    'TestCase',
+    'Assertions',
+    'BaseSetup',
+    'EnvironmentSetup',
+    'SetupManager'
 ] 

cli_test_framework-0.3.0/src/cli_test_framework/core/base_runner.py

@@ -0,0 +1,81 @@
+from abc import ABC, abstractmethod
+from pathlib import Path
+from typing import List, Dict, Any, Optional
+from .test_case import TestCase
+from .assertions import Assertions
+from .setup import SetupManager, EnvironmentSetup
+
+class BaseRunner(ABC):
+    def __init__(self, config_file: str, workspace: Optional[str] = None):
+        if workspace:
+            self.workspace = Path(workspace)
+        else:
+            self.workspace = Path(__file__).parent.parent.parent
+        self.config_path = self.workspace / config_file
+        self.test_cases: List[TestCase] = []
+        self.results: Dict[str, Any] = {
+            "total": 0,
+            "passed": 0,
+            "failed": 0,
+            "details": []
+        }
+        self.assertions = Assertions()
+        self.setup_manager = SetupManager()
+
+    @abstractmethod
+    def load_test_cases(self) -> None:
+        """Load test cases from configuration file"""
+        pass
+    
+    def load_setup_from_config(self, config: Dict[str, Any]) -> None:
+        """从配置文件加载setup配置"""
+        setup_config = config.get("setup", {})
+        
+        # 处理环境变量设置
+        if "environment_variables" in setup_config:
+            env_setup = EnvironmentSetup({"environment_variables": setup_config["environment_variables"]})
+            self.setup_manager.add_setup(env_setup)
+        
+        # 这里可以扩展支持其他类型的setup插件
+        # 例如：
+        # if "custom_setups" in setup_config:
+        #     for custom_setup_config in setup_config["custom_setups"]:
+        #         # 动态加载自定义setup插件
+        #         pass
+
+    def run_tests(self) -> bool:
+        """Run all test cases and return whether all tests passed"""
+        try:
+            self.load_test_cases()
+            self.results["total"] = len(self.test_cases)
+            
+            # 执行setup任务
+            self.setup_manager.setup_all()
+            
+            print(f"\nStarting test execution... Total tests: {self.results['total']}")
+            print("=" * 50)
+            
+            for i, case in enumerate(self.test_cases, 1):
+                print(f"\nRunning test {i}/{self.results['total']}: {case.name}")
+                result = self.run_single_test(case)
+                self.results["details"].append(result)
+                if result["status"] == "passed":
+                    self.results["passed"] += 1
+                    print(f"✓ Test passed: {case.name}")
+                else:
+                    self.results["failed"] += 1
+                    print(f"✗ Test failed: {case.name}")
+                    if result["message"]:
+                        print(f"  Error: {result['message']}")
+                    
+            print("\n" + "=" * 50)
+            print(f"Test execution completed. Passed: {self.results['passed']}, Failed: {self.results['failed']}")
+            return self.results["failed"] == 0
+        finally:
+            # 确保teardown总是被执行
+            self.setup_manager.teardown_all()
+
+    @abstractmethod
+    def run_single_test(self, case: TestCase) -> Dict[str, str]:
+        """Run a single test case and return the result"""
+        pass

cli_test_framework-0.3.0/src/cli_test_framework/core/parallel_runner.py

@@ -0,0 +1,125 @@
+from abc import ABC
+from concurrent.futures import ThreadPoolExecutor, ProcessPoolExecutor, as_completed
+from typing import List, Dict, Any, Optional, Union
+import time
+import threading
+from .base_runner import BaseRunner
+from .test_case import TestCase
+from .process_worker import run_test_in_process
+
+class ParallelRunner(BaseRunner):
+    """并行测试运行器基类，支持多线程和多进程执行"""
+    
+    def __init__(self, config_file: str, workspace: Optional[str] = None, 
+                 max_workers: Optional[int] = None, 
+                 execution_mode: str = "thread"):
+        """
+        初始化并行运行器
+        
+        Args:
+            config_file: 配置文件路径
+            workspace: 工作目录
+            max_workers: 最大并发数，默认为CPU核心数
+            execution_mode: 执行模式，'thread'(线程) 或 'process'(进程)
+        """
+        super().__init__(config_file, workspace)
+        self.max_workers = max_workers
+        self.execution_mode = execution_mode
+        self.lock = threading.Lock()  # 用于线程安全的结果更新
+        
+    def run_tests(self) -> bool:
+        """并行运行所有测试用例"""
+        try:
+            self.load_test_cases()
+            self.results["total"] = len(self.test_cases)
+            
+            # 执行setup任务
+            self.setup_manager.setup_all()
+            
+            print(f"\nStarting parallel test execution... Total tests: {self.results['total']}")
+            print(f"Execution mode: {self.execution_mode}, Max workers: {self.max_workers or 'auto'}")
+            print("=" * 50)
+            
+            start_time = time.time()
+            
+            if self.execution_mode == "process":
+                executor_class = ProcessPoolExecutor
+            else:
+                executor_class = ThreadPoolExecutor
+                
+            with executor_class(max_workers=self.max_workers) as executor:
+                # 提交所有测试任务
+                if self.execution_mode == "process":
+                    # 进程模式：使用独立的工作器函数
+                    future_to_case = {
+                        executor.submit(
+                            run_test_in_process, 
+                            i, 
+                            {
+                                "name": case.name,
+                                "command": case.command,
+                                "args": case.args,
+                                "expected": case.expected
+                            },
+                            str(self.workspace) if self.workspace else None
+                        ): (i, case) 
+                        for i, case in enumerate(self.test_cases, 1)
+                    }
+                else:
+                    # 线程模式：使用实例方法
+                    future_to_case = {
+                        executor.submit(self._run_test_with_index, i, case): (i, case) 
+                        for i, case in enumerate(self.test_cases, 1)
+                    }
+                
+                # 收集结果
+                for future in as_completed(future_to_case):
+                    test_index, case = future_to_case[future]
+                    try:
+                        result = future.result()
+                        self._update_results(result, test_index, case)
+                    except Exception as exc:
+                        error_result = {
+                            "name": case.name,
+                            "status": "failed",
+                            "message": f"Test execution failed: {str(exc)}",
+                            "output": "",
+                            "command": "",
+                            "return_code": None
+                        }
+                        self._update_results(error_result, test_index, case)
+            
+            end_time = time.time()
+            execution_time = end_time - start_time
+            
+            print("\n" + "=" * 50)
+            print(f"Parallel test execution completed in {execution_time:.2f} seconds")
+            print(f"Passed: {self.results['passed']}, Failed: {self.results['failed']}")
+            return self.results["failed"] == 0
+        finally:
+            # 确保teardown总是被执行
+            self.setup_manager.teardown_all()
+    
+    def _run_test_with_index(self, test_index: int, case: TestCase) -> Dict[str, Any]:
+        """运行单个测试并返回结果（包含索引信息）"""
+        print(f"[Worker] Running test {test_index}: {case.name}")
+        result = self.run_single_test(case)
+        return result
+    
+    def _update_results(self, result: Dict[str, Any], test_index: int, case: TestCase) -> None:
+        """线程安全地更新测试结果"""
+        with self.lock:
+            self.results["details"].append(result)
+            if result["status"] == "passed":
+                self.results["passed"] += 1
+                print(f"✓ Test {test_index} passed: {case.name}")
+            else:
+                self.results["failed"] += 1
+                print(f"✗ Test {test_index} failed: {case.name}")
+                if result["message"]:
+                    print(f"  Error: {result['message']}")
+    
+    def run_tests_sequential(self) -> bool:
+        """回退到顺序执行模式"""
+        print("Falling back to sequential execution...")
+        return super().run_tests() 

cli_test_framework-0.3.0/src/cli_test_framework/core/setup.py

@@ -0,0 +1,134 @@
+from abc import ABC, abstractmethod
+from typing import Dict, Any, Optional
+import os
+
+
+class BaseSetup(ABC):
+    """测试前置任务的基类，允许用户以插件形式自定义"""
+    
+    def __init__(self, config: Optional[Dict[str, Any]] = None):
+        """
+        初始化Setup
+        
+        Args:
+            config: 配置参数字典
+        """
+        self.config = config or {}
+    
+    @abstractmethod
+    def setup(self) -> None:
+        """
+        执行前置任务设置
+        子类必须实现此方法
+        """
+        pass
+    
+    @abstractmethod
+    def teardown(self) -> None:
+        """
+        执行后置清理任务
+        子类必须实现此方法
+        """
+        pass
+    
+    def get_name(self) -> str:
+        """返回Setup的名称"""
+        return self.__class__.__name__
+
+
+class EnvironmentSetup(BaseSetup):
+    """内置的环境变量设置插件"""
+    
+    def __init__(self, config: Optional[Dict[str, Any]] = None):
+        super().__init__(config)
+        self._original_env = {}  # 存储原始环境变量值
+        self._added_env = set()  # 存储新添加的环境变量
+    
+    def setup(self) -> None:
+        """设置环境变量"""
+        env_vars = self.config.get('environment_variables', {})
+        
+        if not env_vars:
+            return
+            
+        print(f"Setting up environment variables...")
+        
+        for key, value in env_vars.items():
+            # 保存原始值
+            if key in os.environ:
+                self._original_env[key] = os.environ[key]
+            else:
+                self._added_env.add(key)
+            
+            # 设置新值
+            os.environ[key] = str(value)
+            print(f"  {key} = {value}")
+    
+    def teardown(self) -> None:
+        """恢复环境变量"""
+        if not self._original_env and not self._added_env:
+            return
+            
+        print(f"Restoring environment variables...")
+        
+        # 恢复原始值
+        for key, value in self._original_env.items():
+            os.environ[key] = value
+            print(f"  Restored {key}")
+        
+        # 删除新添加的环境变量
+        for key in self._added_env:
+            if key in os.environ:
+                del os.environ[key]
+                print(f"  Removed {key}")
+        
+        # 清空记录
+        self._original_env.clear()
+        self._added_env.clear()
+
+
+class SetupManager:
+    """Setup管理器，负责管理多个Setup插件"""
+    
+    def __init__(self):
+        self.setups = []
+    
+    def add_setup(self, setup: BaseSetup) -> None:
+        """添加Setup插件"""
+        self.setups.append(setup)
+    
+    def setup_all(self) -> None:
+        """执行所有Setup的前置任务"""
+        if not self.setups:
+            return
+            
+        print("\n" + "=" * 50)
+        print("Executing setup tasks...")
+        print("=" * 50)
+        
+        for setup in self.setups:
+            try:
+                print(f"\nRunning setup: {setup.get_name()}")
+                setup.setup()
+            except Exception as e:
+                print(f"Error in setup {setup.get_name()}: {str(e)}")
+                raise
+    
+    def teardown_all(self) -> None:
+        """执行所有Setup的后置清理任务（逆序执行）"""
+        if not self.setups:
+            return
+            
+        print("\n" + "=" * 50)
+        print("Executing teardown tasks...")
+        print("=" * 50)
+        
+        # 逆序执行teardown
+        for setup in reversed(self.setups):
+            try:
+                print(f"\nRunning teardown: {setup.get_name()}")
+                setup.teardown()
+            except Exception as e:
+                print(f"Error in teardown {setup.get_name()}: {str(e)}")
+                # teardown错误不应该阻止其他teardown的执行
+                continue 

{cli_test_framework-0.2.3 → cli_test_framework-0.3.0}/src/cli_test_framework/runners/json_runner.py

@@ -18,6 +18,9 @@ class JSONRunner(BaseRunner):
             with open(self.config_path, 'r', encoding='utf-8') as f:
                 config = json.load(f)
 
+            # 加载setup配置
+            self.load_setup_from_config(config)
+
             required_fields = ["name", "command", "args", "expected"]
             for case in config["test_cases"]:
                 if not all(field in case for field in required_fields):

{cli_test_framework-0.2.3 → cli_test_framework-0.3.0}/src/cli_test_framework/runners/parallel_json_runner.py

@@ -31,6 +31,9 @@ class ParallelJSONRunner(ParallelRunner):
             with open(self.config_path, 'r', encoding='utf-8') as f:
                 config = json.load(f)
 
+            # 加载setup配置
+            self.load_setup_from_config(config)
+
             required_fields = ["name", "command", "args", "expected"]
             for case in config["test_cases"]:
                 if not all(field in case for field in required_fields):

{cli_test_framework-0.2.3 → cli_test_framework-0.3.0}/src/cli_test_framework/runners/yaml_runner.py

@@ -17,6 +17,9 @@ class YAMLRunner(BaseRunner):
             with open(self.config_path, 'r', encoding='utf-8') as f:
                 config = yaml.safe_load(f)
                 
+            # 加载setup配置
+            self.load_setup_from_config(config)
+                
             required_fields = ["name", "command", "args", "expected"]
             for case in config["test_cases"]:
                 if not all(field in case for field in required_fields):