cli-test-framework 0.2.0__tar.gz → 0.2.1__tar.gz

cli_test_framework-0.2.1/CHANGELOG.md

@@ -0,0 +1,114 @@
+# 变更日志
+
+## [v2.0.0] - 2025-05-27
+
+### 🚀 重大新功能
+
+#### 并行测试执行
+- **多线程并行执行**：支持 I/O 密集型测试的高效并行处理
+- **多进程并行执行**：支持 CPU 密集型测试的完全隔离执行
+- **可配置并发数**：灵活设置最大工作线程/进程数
+- **性能提升**：典型场景下可获得 2-4 倍加速比
+
+#### 智能命令解析
+- **复杂命令支持**：智能处理 `"python ./script.py"` 等复杂命令格式
+- **系统命令识别**：自动识别系统命令和相对路径可执行文件
+- **路径解析增强**：更准确的路径转换和命令构建
+
+### ✨ 功能增强
+
+#### 线程安全设计
+- **结果收集锁**：确保并发环境下测试结果的正确性
+- **输出控制锁**：防止并发输出混乱
+- **异常隔离**：单个测试失败不影响其他测试
+
+#### 新增运行器
+- **ParallelJSONRunner**：支持并行执行的 JSON 测试运行器
+- **进程工作器**：独立的进程执行模块，解决序列化问题
+
+### 🔧 问题修复
+
+#### 命令解析修复
+- **修复问题**：`'D:\Document\xcode\Compare-File-Tool\python' 不是内部或外部命令`
+- **根本原因**：复杂命令字符串被整体当作命令名处理
+- **解决方案**：智能分割命令字符串，分别处理命令和参数部分
+
+#### 路径解析优化
+- **系统命令白名单**：扩展系统命令识别列表
+- **相对路径处理**：改进相对路径和绝对路径的判断逻辑
+
+### 📚 文档更新
+
+#### README 全面重写
+- **英文版 README.md**：全新的结构和内容组织
+- **中文版 README_cn.md**：同步更新，保持一致性
+- **并行测试指南**：详细的 PARALLEL_TESTING_GUIDE.md
+
+#### 新增示例
+- **parallel_example.py**：性能比较演示
+- **performance_test.py**：自动化性能测试
+- **test_parallel_runner.py**：并行功能单元测试
+
+### 🎯 性能基准
+
+| 测试场景 | 顺序执行 | 并行执行(线程) | 并行执行(进程) | 加速比 |
+|----------|----------|----------------|----------------|--------|
+| 6个简单测试 | 0.12秒 | 0.03秒 | 0.16秒 | 3.84x |
+| 10个I/O测试 | 5.2秒 | 1.4秒 | 2.1秒 | 3.7x |
+| 20个CPU测试 | 12.8秒 | 8.9秒 | 6.2秒 | 2.1x |
+
+### 🔄 向后兼容性
+
+- **完全兼容**：现有的 JSONRunner 代码无需修改
+- **渐进式升级**：可以逐步迁移到并行执行
+- **配置兼容**：现有测试用例配置文件无需更改
+
+### 📦 依赖更新
+
+```txt
+pytest
+PyYAML  # 修正了之前的 pyyaml 拼写错误
+concurrent.futures  # 标准库，无需额外安装
+```
+
+### 🚀 使用示例
+
+#### 基本并行执行
+```python
+from src.runners.parallel_json_runner import ParallelJSONRunner
+
+runner = ParallelJSONRunner(
+    config_file="test_cases.json",
+    max_workers=4,
+    execution_mode="thread"
+)
+success = runner.run_tests()
+```
+
+#### 性能比较
+```bash
+# 运行性能比较示例
+python parallel_example.py
+
+# 输出示例：
+# 顺序执行时间:     0.12 秒
+# 并行执行时间(线程): 0.03 秒 (加速比: 3.84x)
+# 并行执行时间(进程): 0.16 秒 (加速比: 0.73x)
+```
+
+### 🔮 未来计划
+
+- **分布式执行**：支持跨机器的测试执行
+- **Web UI**：提供图形化的测试管理界面
+- **更多格式支持**：XML、TOML 等配置格式
+- **性能监控**：详细的执行时间和资源使用统计
+
+---
+
+## [v1.0.0] - 2025-03-05
+
+### 初始版本
+- 基础的 JSON/YAML 测试运行器
+- 路径解析功能
+- 断言机制
+- 报告生成 

cli_test_framework-0.2.1/MANIFEST.in

@@ -0,0 +1,6 @@
+include README.md
+include LICENSE
+include CHANGELOG.md
+include CONTRIBUTING.md
+recursive-include docs *
+recursive-include tests * 

cli_test_framework-0.2.1/PKG-INFO

@@ -0,0 +1,445 @@
+Metadata-Version: 2.4
+Name: cli-test-framework
+Version: 0.2.1
+Summary: A small command line testing framework in Python with file comparison capabilities.
+Home-page: https://github.com/yourusername/cli-test-framework
+Author: Xiaotong Wang
+Author-email: xiaotongwang98@gmail.com
+Project-URL: Documentation, https://github.com/yourusername/cli-test-framework/docs/user_manual.md
+Project-URL: Source, https://github.com/yourusername/cli-test-framework
+Project-URL: Tracker, https://github.com/yourusername/cli-test-framework/issues
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Topic :: Software Development :: Testing
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Requires-Python: >=3.6
+Description-Content-Type: text/markdown
+Requires-Dist: dukpy==0.5.0
+Requires-Dist: h5py>=3.8.0
+Requires-Dist: numpy>=2.0.1
+Requires-Dist: setuptools>=75.8.0
+Requires-Dist: wheel>=0.45.1
+Dynamic: author
+Dynamic: author-email
+Dynamic: classifier
+Dynamic: description
+Dynamic: description-content-type
+Dynamic: home-page
+Dynamic: project-url
+Dynamic: requires-dist
+Dynamic: requires-python
+Dynamic: summary
+
+# CLI Testing Framework
+
+## 1. Overview
+
+This is a lightweight and extensible automated testing framework that supports defining test cases via JSON/YAML formats, providing complete test execution, result verification, and report generation capabilities. The framework is designed to provide standardized test management for command-line tools and scripts, with enterprise-grade parallel execution support and advanced file comparison features.
+
+## 2. Features
+
+- **🚀 Parallel Test Execution**: Support for multi-threading and multi-processing parallel testing with significant performance improvements
+- **🏗️ Modular Architecture**: Decoupled design of core components (runner/assertion/report)
+- **📄 Multi-Format Support**: Native support for JSON/YAML test case formats
+- **🧠 Intelligent Command Parsing**: Smart handling of complex commands like `"python ./script.py"`
+- **📁 Smart Path Resolution**: Automatic handling of relative and absolute path conversions
+- **✅ Rich Assertion Mechanism**: Return code validation, output content matching, regex verification
+- **🔌 Extensible Interfaces**: Quickly implement new test format support by inheriting BaseRunner
+- **🔒 Isolated Execution Environment**: Independent sub-process execution ensures test isolation
+- **📊 Comprehensive Reports**: Detailed pass rate statistics and failure diagnostics
+- **🔧 Thread-Safe Design**: Robust concurrent execution with proper synchronization
+- **📝 Advanced File Comparison**: Support for comparing various file types (text, binary, JSON, HDF5) with detailed diff output
+
+## 3. Quick Start
+
+### Environment Requirements
+
+```bash
+pip install cli-test-framework
+Python >= 3.6
+```
+
+### Sequential Execution
+
+```python
+from src.runners.json_runner import JSONRunner
+
+runner = JSONRunner(
+    config_file="path/to/test_cases.json",
+    workspace="/project/root"
+)
+success = runner.run_tests()
+```
+
+### Parallel Execution
+
+```python
+from src.runners.parallel_json_runner import ParallelJSONRunner
+
+# Multi-threaded execution (recommended for I/O-intensive tests)
+runner = ParallelJSONRunner(
+    config_file="path/to/test_cases.json",
+    workspace="/project/root",
+    max_workers=4,           # Maximum concurrent workers
+    execution_mode="thread"  # "thread" or "process"
+)
+success = runner.run_tests()
+```
+
+### File Comparison
+
+```bash
+# Compare two text files
+compare-files file1.txt file2.txt
+
+# Compare JSON files with key-based comparison
+compare-files data1.json data2.json --json-compare-mode key-based --json-key-field id
+
+# Compare HDF5 files with specific options
+compare-files data1.h5 data2.h5 --h5-table table1,table2 --h5-rtol 1e-6
+
+# Compare binary files with similarity check
+compare-files binary1.bin binary2.bin --similarity
+```
+
+## 4. Test Case Format
+
+### JSON Format
+
+```json
+{
+    "test_cases": [
+        {
+            "name": "File Comparison Test",
+            "command": "compare-files",
+            "args": ["file1.txt", "file2.txt", "--verbose"],
+            "expected": {
+                "return_code": 0,
+                "output_contains": ["Files are identical"],
+                "output_matches": [".*comparison completed.*"]
+            }
+        }
+    ]
+}
+```
+
+### YAML Format
+
+```yaml
+test_cases:
+  - name: Directory Scan Test
+    command: ls
+    args:
+      - -l
+      - docs/
+    expected:
+      return_code: 0
+      output_matches: ".*\\.md$"
+```
+
+## 5. File Comparison Features
+
+### Supported File Types
+
+- **Text Files**: Plain text, source code, markdown, etc.
+- **JSON Files**: With exact or key-based comparison
+- **HDF5 Files**: Structure and content comparison with numerical tolerance
+- **Binary Files**: With optional similarity index calculation
+
+### Comparison Options
+
+#### Text Comparison
+```bash
+compare-files file1.txt file2.txt \
+    --start-line 10 \
+    --end-line 20 \
+    --encoding utf-8
+```
+
+#### JSON Comparison
+```bash
+compare-files data1.json data2.json \
+    --json-compare-mode key-based \
+    --json-key-field id,name
+```
+
+#### HDF5 Comparison
+```bash
+compare-files data1.h5 data2.h5 \
+    --h5-table table1,table2 \
+    --h5-structure-only \
+    --h5-rtol 1e-5 \
+    --h5-atol 1e-8
+```
+
+#### Binary Comparison
+```bash
+compare-files binary1.bin binary2.bin \
+    --similarity \
+    --chunk-size 16384
+```
+
+### Output Formats
+
+- **Text**: Human-readable diff output
+- **JSON**: Structured comparison results
+- **HTML**: Visual diff with syntax highlighting
+
+## 6. System Architecture
+
+### Enhanced Architecture Flow
+
+```mermaid
+graph TD
+    A[Test Cases] --> B{Execution Mode}
+    B -->|Sequential| C[JSONRunner/YAMLRunner]
+    B -->|Parallel| D[ParallelRunner]
+    D --> E[ThreadPoolExecutor/ProcessPoolExecutor]
+    C --> F[Command Parser]
+    E --> F
+    F --> G[Path Resolver]
+    G --> H[Sub-process Execution]
+    H --> I[Assertion Engine]
+    I --> J[Thread-Safe Result Collection]
+    J --> K[Report Generator]
+    L[File Comparator] --> M[Text Comparator]
+    L --> N[JSON Comparator]
+    L --> O[HDF5 Comparator]
+    L --> P[Binary Comparator]
+```
+
+### Core Components
+
+#### 1. Intelligent Command Parser
+```python
+# Handles complex commands like "python ./script.py"
+command_parts = case["command"].split()
+if len(command_parts) > 1:
+    actual_command = resolve_command(command_parts[0])  # "python"
+    script_parts = resolve_paths(command_parts[1:])     # "./script.py" -> full path
+    final_command = f"{actual_command} {' '.join(script_parts)}"
+```
+
+#### 2. Enhanced Path Resolver
+```python
+def resolve_command(self, command: str) -> str:
+    system_commands = {
+        'echo', 'ping', 'python', 'node', 'java', 'docker', ...
+    }
+    if command in system_commands or Path(command).is_absolute():
+        return command
+    return str(self.workspace / command)
+```
+
+#### 3. Parallel Runner Base Class
+```python
+class ParallelRunner(BaseRunner):
+    def __init__(self, max_workers=None, execution_mode="thread"):
+        self.max_workers = max_workers or os.cpu_count()
+        self.execution_mode = execution_mode
+        self._results_lock = threading.Lock()
+        self._print_lock = threading.Lock()
+```
+
+## 7. Advanced Usage
+
+### Performance Testing
+
+```python
+# Quick performance test
+python performance_test.py
+
+# Unit tests for parallel functionality
+python -m pytest tests/test_parallel_runner.py -v
+```
+
+### Error Handling and Fallback
+
+```python
+try:
+    runner = ParallelJSONRunner(config_file="test_cases.json")
+    success = runner.run_tests()
+    
+    if not success:
+        # Check failed tests
+        for detail in runner.results["details"]:
+            if detail["status"] == "failed":
+                print(f"Failed test: {detail['name']}")
+                print(f"Error: {detail['message']}")
+                
+except Exception as e:
+    print(f"Execution error: {e}")
+    # Fallback to sequential execution
+    runner.run_tests_sequential()
+```
+
+### Best Practices
+
+1. **Choose Appropriate Concurrency**:
+   ```python
+   import os
+   
+   # For CPU-intensive tasks
+   max_workers = os.cpu_count()
+   
+   # For I/O-intensive tasks
+   max_workers = os.cpu_count() * 2
+   ```
+
+2. **Test Case Design**:
+   - ✅ Ensure test independence (no dependencies between tests)
+   - ✅ Avoid shared resource conflicts (different files/ports)
+   - ✅ Use relative paths (framework handles resolution automatically)
+
+3. **Debugging**:
+   ```python
+   # Enable verbose output for debugging
+   runner = ParallelJSONRunner(
+       config_file="test_cases.json",
+       max_workers=1,  # Set to 1 for easier debugging
+       execution_mode="thread"
+   )
+   ```
+
+## 8. Example Demonstrations
+
+### Input Example
+
+```json
+{
+    "test_cases": [
+        {
+            "name": "Python Version Check",
+            "command": "python --version",
+            "args": [],
+            "expected": {
+                "output_matches": "Python 3\\.[89]\\.",
+                "return_code": 0
+            }
+        },
+        {
+            "name": "File Processing Test",
+            "command": "python ./process_file.py",
+            "args": ["input.txt", "--output", "result.txt"],
+            "expected": {
+                "return_code": 0,
+                "output_contains": ["Processing completed"]
+            }
+        }
+    ]
+}
+```
+
+### Output Report
+
+```
+Test Results Summary:
+Total Tests: 15
+Passed: 15
+Failed: 0
+
+Performance Statistics:
+Sequential execution time: 12.45 seconds
+Parallel execution time:   3.21 seconds
+Speedup ratio:            3.88x
+
+Detailed Results:
+✓ Python Version Check
+✓ File Processing Test
+✓ JSON Comparison Test
+...
+```
+
+## 9. Troubleshooting
+
+### Common Issues
+
+1. **Process Mode Serialization Error**
+   - **Cause**: Objects contain non-serializable attributes (like locks)
+   - **Solution**: Use independent process worker functions
+
+2. **Path Resolution Error**
+   - **Cause**: System commands treated as relative paths
+   - **Solution**: Update `PathResolver` system command list
+
+3. **Performance Not Improved**
+   - **Cause**: Test cases too short, parallel overhead exceeds benefits
+   - **Solution**: Increase test case count or use more complex tests
+
+4. **Command Not Found Error**
+   - **Cause**: Complex commands like `"python ./script.py"` not parsed correctly
+   - **Solution**: Framework now automatically handles this (fixed in latest version)
+
+### Debug Tips
+
+```python
+# Enable detailed logging
+import logging
+logging.basicConfig(level=logging.DEBUG)
+
+# Check detailed results
+import json
+print(json.dumps(runner.results, indent=2, ensure_ascii=False))
+```
+
+## 10. Extension and Customization
+
+### Adding New Runners
+
+```python
+class XMLRunner(BaseRunner):
+    def load_test_cases(self):
+        import xml.etree.ElementTree as ET
+        # Parse XML structure and convert to TestCase objects
+        
+class CustomParallelRunner(ParallelRunner):
+    def custom_preprocessing(self):
+        # Add custom logic before test execution
+        pass
+```
+
+### Custom Assertions
+
+```python
+class CustomAssertions(Assertions):
+    @staticmethod
+    def performance_threshold(execution_time, max_time):
+        if execution_time > max_time:
+            raise AssertionError(f"Execution too slow: {execution_time}s > {max_time}s")
+```
+
+## 11. Version Compatibility
+
+- **Python Version**: 3.6+
+- **Dependencies**: Standard library only (no external dependencies for core functionality)
+- **Backward Compatibility**: Fully compatible with existing `JSONRunner` code
+- **Platform Support**: Windows, macOS, Linux
+
+## 12. Performance Benchmarks
+
+| Test Scenario | Sequential | Parallel (Thread) | Parallel (Process) | Speedup |
+|---------------|------------|-------------------|-------------------|---------|
+| 10 I/O tests  | 5.2s       | 1.4s              | 2.1s              | 3.7x    |
+| 20 CPU tests  | 12.8s      | 8.9s              | 6.2s              | 2.1x    |
+| Mixed tests   | 8.5s       | 2.3s              | 3.1s              | 3.7x    |
+
+## 13. Contributing
+
+1. Fork the repository
+2. Create a feature branch
+3. Add tests for new functionality
+4. Ensure all tests pass: `python -m pytest tests/ -v`
+5. Submit a pull request
+
+## 14. License
+
+This project is licensed under the MIT License - see the LICENSE file for details.
+
+---
+
+**🚀 Ready to supercharge your testing workflow with parallel execution and advanced file comparison!**
+
+For detailed parallel testing guide, see: [PARALLEL_TESTING_GUIDE.md](PARALLEL_TESTING_GUIDE.md)