cli-test-framework 0.3.2__tar.gz → 0.3.4__tar.gz

{cli_test_framework-0.3.2 → cli_test_framework-0.3.4}/CHANGELOG.md

@@ -1,6 +1,18 @@
 # 变更日志
 
-## [v0.3.2] - 2024-12-29
+## [v0.3.3] - 2025-06-09
+
+### 🔧 项目信息更新
+- **GitHub仓库更新**：更新项目仓库地址为 https://github.com/ozil111/cli-test-framework
+- **PyPI主页优化**：恢复使用README.md作为PyPI项目主页描述，更适合对外展示
+- **文档链接调整**：用户手册通过Documentation链接访问，结构更加清晰
+
+### 📚 链接修复
+- 更新所有文档中的GitHub仓库链接
+- 修正PyPI项目的Documentation链接指向
+- 改善文档访问体验和项目结构
+
+## [v0.3.2] - 2024-06-09
 
 ### 🔧 重大改进
 - **PyPI文档更新**：PyPI项目页面现在显示完整的用户手册内容（包含Setup模块文档）
@@ -13,7 +25,7 @@
 - 增加了指向完整用户手册的链接和说明
 - 改善了文档结构和可读性
 
-## [v0.3.1] - 2024-12-29
+## [v0.3.1] - 2024-06-08
 
 ### 🔧 改进
 - **文档更新**：完善了用户手册中的Setup模块文档

cli_test_framework-0.3.4/PKG-INFO

@@ -0,0 +1,522 @@
+Metadata-Version: 2.4
+Name: cli-test-framework
+Version: 0.3.4
+Summary: A powerful command line testing framework in Python with setup modules, parallel execution, and file comparison capabilities.
+Home-page: https://github.com/ozil111/cli-test-framework
+Author: Xiaotong Wang
+Author-email: xiaotongwang98@gmail.com
+Project-URL: Documentation, https://github.com/ozil111/cli-test-framework/blob/main/docs/user_manual.md
+Project-URL: Source, https://github.com/ozil111/cli-test-framework
+Project-URL: Tracker, https://github.com/ozil111/cli-test-framework/issues
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Topic :: Software Development :: Testing
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+Requires-Dist: dukpy==0.5.0
+Requires-Dist: h5py>=3.10.0
+Requires-Dist: numpy<2.0.0,>=1.24.0
+Requires-Dist: PyYAML>=6.0
+Requires-Dist: setuptools>=75.8.0
+Requires-Dist: wheel>=0.45.1
+Dynamic: author
+Dynamic: author-email
+Dynamic: classifier
+Dynamic: description
+Dynamic: description-content-type
+Dynamic: home-page
+Dynamic: project-url
+Dynamic: requires-dist
+Dynamic: requires-python
+Dynamic: summary
+
+# CLI Testing Framework
+
+## 1. Overview
+
+This is a lightweight and extensible automated testing framework that supports defining test cases via JSON/YAML formats, providing complete test execution, result verification, and report generation capabilities. The framework is designed to provide standardized test management for command-line tools and scripts, with enterprise-grade parallel execution support and advanced file comparison features.
+
+## 2. Features
+
+- **🚀 Parallel Test Execution**: Support for multi-threading and multi-processing parallel testing with significant performance improvements
+- **🔧 Setup Module System**: Plugin-based architecture for pre-test setup tasks (environment variables, database initialization, service startup)
+- **🏗️ Modular Architecture**: Decoupled design of core components (runner/assertion/report/setup)
+- **📄 Multi-Format Support**: Native support for JSON/YAML test case formats
+- **🧠 Intelligent Command Parsing**: Smart handling of complex commands like `"python ./script.py"`
+- **📁 Smart Path Resolution**: Automatic handling of relative and absolute path conversions
+- **✅ Rich Assertion Mechanism**: Return code validation, output content matching, regex verification
+- **🔌 Extensible Interfaces**: Quickly implement new test format support by inheriting BaseRunner
+- **🔒 Isolated Execution Environment**: Independent sub-process execution ensures test isolation
+- **📊 Comprehensive Reports**: Detailed pass rate statistics and failure diagnostics
+- **🔧 Thread-Safe Design**: Robust concurrent execution with proper synchronization
+- **📝 Advanced File Comparison**: Support for comparing various file types (text, binary, JSON, HDF5) with detailed diff output
+
+## 3. Quick Start
+
+### Environment Requirements
+
+```bash
+pip install cli-test-framework
+Python >= 3.9
+```
+
+### Sequential Execution
+
+```python
+from src.runners.json_runner import JSONRunner
+
+runner = JSONRunner(
+    config_file="path/to/test_cases.json",
+    workspace="/project/root"
+)
+success = runner.run_tests()
+```
+
+### Parallel Execution
+
+```python
+from src.runners.parallel_json_runner import ParallelJSONRunner
+
+# Multi-threaded execution (recommended for I/O-intensive tests)
+runner = ParallelJSONRunner(
+    config_file="path/to/test_cases.json",
+    workspace="/project/root",
+    max_workers=4,           # Maximum concurrent workers
+    execution_mode="thread"  # "thread" or "process"
+)
+success = runner.run_tests()
+```
+
+### Setup Module Usage
+
+```python
+from cli_test_framework import JSONRunner, EnvironmentSetup
+
+# Using built-in environment variable setup
+runner = JSONRunner("test_cases.json")
+env_setup = EnvironmentSetup({
+    "TEST_ENV": "development",
+    "API_URL": "http://localhost:8080"
+})
+runner.setup_manager.add_setup(env_setup)
+success = runner.run_tests()
+```
+
+### File Comparison
+
+```bash
+# Compare two text files
+compare-files file1.txt file2.txt
+
+# Compare JSON files with key-based comparison
+compare-files data1.json data2.json --json-compare-mode key-based --json-key-field id
+
+# Compare HDF5 files with specific options
+compare-files data1.h5 data2.h5 --h5-table table1,table2 --h5-rtol 1e-6
+
+# Compare binary files with similarity check
+compare-files binary1.bin binary2.bin --similarity
+```
+
+## 4. Test Case Format
+
+### JSON Format
+
+```json
+{
+    "setup": {
+        "environment_variables": {
+            "TEST_ENV": "development",
+            "API_URL": "http://localhost:8080",
+            "DEBUG_MODE": "true"
+        }
+    },
+    "test_cases": [
+        {
+            "name": "Environment Variable Test",
+            "command": "python",
+            "args": ["-c", "import os; print(f'Environment: {os.environ.get(\"TEST_ENV\")}')"],
+            "expected": {
+                "return_code": 0,
+                "output_contains": ["Environment: development"]
+            }
+        },
+        {
+            "name": "File Comparison Test", 
+            "command": "compare-files",
+            "args": ["file1.txt", "file2.txt", "--verbose"],
+            "expected": {
+                "return_code": 0,
+                "output_contains": ["Files are identical"],
+                "output_matches": [".*comparison completed.*"]
+            }
+        }
+    ]
+}
+```
+
+### YAML Format
+
+```yaml
+setup:
+  environment_variables:
+    TEST_ENV: "production"
+    DATABASE_URL: "sqlite:///test.db"
+
+test_cases:
+  - name: Environment Test
+    command: python
+    args:
+      - "-c"
+      - "import os; print(f'DB: {os.environ.get(\"DATABASE_URL\")}')"
+    expected:
+      return_code: 0
+      output_contains:
+        - "DB: sqlite:///test.db"
+  
+  - name: Directory Scan Test
+    command: ls
+    args:
+      - -l
+      - docs/
+    expected:
+      return_code: 0
+      output_matches: ".*\\.md$"
+```
+
+## 5. File Comparison Features
+
+### Supported File Types
+
+- **Text Files**: Plain text, source code, markdown, etc.
+- **JSON Files**: With exact or key-based comparison
+- **HDF5 Files**: Structure and content comparison with numerical tolerance
+- **Binary Files**: With optional similarity index calculation
+
+### Comparison Options
+
+#### Text Comparison
+```bash
+compare-files file1.txt file2.txt \
+    --start-line 10 \
+    --end-line 20 \
+    --encoding utf-8
+```
+
+#### JSON Comparison
+```bash
+compare-files data1.json data2.json \
+    --json-compare-mode key-based \
+    --json-key-field id,name
+```
+
+#### HDF5 Comparison
+
+**New Feature**: HDF5 group path expansion! By default, when you specify a group path in `--h5-table`, the comparator will automatically expand and compare all datasets and subgroups within that path.
+
+```bash
+# Compare specific tables/groups with auto-expansion (default behavior)
+compare-files data1.h5 data2.h5 \
+    --h5-table group1/subgroupA \
+    --h5-rtol 1e-5 \
+    --h5-atol 1e-8
+
+# Disable auto-expansion to compare only the specified path itself
+compare-files data1.h5 data2.h5 \
+    --h5-table group1 \
+    --h5-no-expand-path
+
+# Use regex patterns (also supports auto-expansion)
+compare-files data1.h5 data2.h5 \
+    --h5-table-regex "group1/.*" \
+    --h5-structure-only
+```
+
+#### Binary Comparison
+```bash
+compare-files binary1.bin binary2.bin \
+    --similarity \
+    --chunk-size 16384
+```
+
+### Output Formats
+
+- **Text**: Human-readable diff output
+- **JSON**: Structured comparison results
+- **HTML**: Visual diff with syntax highlighting
+
+## 6. System Architecture
+
+### Enhanced Architecture Flow
+
+```mermaid
+graph TD
+    A[Test Cases] --> B{Execution Mode}
+    B -->|Sequential| C[JSONRunner/YAMLRunner]
+    B -->|Parallel| D[ParallelRunner]
+    D --> E[ThreadPoolExecutor/ProcessPoolExecutor]
+    C --> F[Command Parser]
+    E --> F
+    F --> G[Path Resolver]
+    G --> H[Sub-process Execution]
+    H --> I[Assertion Engine]
+    I --> J[Thread-Safe Result Collection]
+    J --> K[Report Generator]
+    L[File Comparator] --> M[Text Comparator]
+    L --> N[JSON Comparator]
+    L --> O[HDF5 Comparator]
+    L --> P[Binary Comparator]
+```
+
+### Core Components
+
+#### 1. Intelligent Command Parser
+```python
+# Handles complex commands like "python ./script.py"
+command_parts = case["command"].split()
+if len(command_parts) > 1:
+    actual_command = resolve_command(command_parts[0])  # "python"
+    script_parts = resolve_paths(command_parts[1:])     # "./script.py" -> full path
+    final_command = f"{actual_command} {' '.join(script_parts)}"
+```
+
+#### 2. Enhanced Path Resolver
+```python
+def resolve_command(self, command: str) -> str:
+    system_commands = {
+        'echo', 'ping', 'python', 'node', 'java', 'docker', ...
+    }
+    if command in system_commands or Path(command).is_absolute():
+        return command
+    return str(self.workspace / command)
+```
+
+#### 3. Parallel Runner Base Class
+```python
+class ParallelRunner(BaseRunner):
+    def __init__(self, max_workers=None, execution_mode="thread"):
+        self.max_workers = max_workers or os.cpu_count()
+        self.execution_mode = execution_mode
+        self._results_lock = threading.Lock()
+        self._print_lock = threading.Lock()
+```
+
+## 7. Advanced Usage
+
+### Performance Testing
+
+```python
+# Quick performance test
+python performance_test.py
+
+# Unit tests for parallel functionality
+python -m pytest tests/test_parallel_runner.py -v
+```
+
+### Error Handling and Fallback
+
+```python
+try:
+    runner = ParallelJSONRunner(config_file="test_cases.json")
+    success = runner.run_tests()
+    
+    if not success:
+        # Check failed tests
+        for detail in runner.results["details"]:
+            if detail["status"] == "failed":
+                print(f"Failed test: {detail['name']}")
+                print(f"Error: {detail['message']}")
+                
+except Exception as e:
+    print(f"Execution error: {e}")
+    # Fallback to sequential execution
+    runner.run_tests_sequential()
+```
+
+### Best Practices
+
+1. **Choose Appropriate Concurrency**:
+   ```python
+   import os
+   
+   # For CPU-intensive tasks
+   max_workers = os.cpu_count()
+   
+   # For I/O-intensive tasks
+   max_workers = os.cpu_count() * 2
+   ```
+
+2. **Test Case Design**:
+   - ✅ Ensure test independence (no dependencies between tests)
+   - ✅ Avoid shared resource conflicts (different files/ports)
+   - ✅ Use relative paths (framework handles resolution automatically)
+
+3. **Debugging**:
+   ```python
+   # Enable verbose output for debugging
+   runner = ParallelJSONRunner(
+       config_file="test_cases.json",
+       max_workers=1,  # Set to 1 for easier debugging
+       execution_mode="thread"
+   )
+   ```
+
+## 8. Example Demonstrations
+
+### Input Example
+
+```json
+{
+    "test_cases": [
+        {
+            "name": "Python Version Check",
+            "command": "python --version",
+            "args": [],
+            "expected": {
+                "output_matches": "Python 3\\.[89]\\.",
+                "return_code": 0
+            }
+        },
+        {
+            "name": "File Processing Test",
+            "command": "python ./process_file.py",
+            "args": ["input.txt", "--output", "result.txt"],
+            "expected": {
+                "return_code": 0,
+                "output_contains": ["Processing completed"]
+            }
+        }
+    ]
+}
+```
+
+### Output Report
+
+```
+Test Results Summary:
+Total Tests: 15
+Passed: 15
+Failed: 0
+
+Performance Statistics:
+Sequential execution time: 12.45 seconds
+Parallel execution time:   3.21 seconds
+Speedup ratio:            3.88x
+
+Detailed Results:
+✓ Python Version Check
+✓ File Processing Test
+✓ JSON Comparison Test
+...
+```
+
+## 9. Troubleshooting
+
+### Common Issues
+
+1. **Process Mode Serialization Error**
+   - **Cause**: Objects contain non-serializable attributes (like locks)
+   - **Solution**: Use independent process worker functions
+
+2. **Path Resolution Error**
+   - **Cause**: System commands treated as relative paths
+   - **Solution**: Update `PathResolver` system command list
+
+3. **Performance Not Improved**
+   - **Cause**: Test cases too short, parallel overhead exceeds benefits
+   - **Solution**: Increase test case count or use more complex tests
+
+4. **Command Not Found Error**
+   - **Cause**: Complex commands like `"python ./script.py"` not parsed correctly
+   - **Solution**: Framework now automatically handles this (fixed in latest version)
+
+### Debug Tips
+
+```python
+# Enable detailed logging
+import logging
+logging.basicConfig(level=logging.DEBUG)
+
+# Check detailed results
+import json
+print(json.dumps(runner.results, indent=2, ensure_ascii=False))
+```
+
+## 10. Extension and Customization
+
+### Adding New Runners
+
+```python
+class XMLRunner(BaseRunner):
+    def load_test_cases(self):
+        import xml.etree.ElementTree as ET
+        # Parse XML structure and convert to TestCase objects
+        
+class CustomParallelRunner(ParallelRunner):
+    def custom_preprocessing(self):
+        # Add custom logic before test execution
+        pass
+```
+
+### Custom Assertions
+
+```python
+class CustomAssertions(Assertions):
+    @staticmethod
+    def performance_threshold(execution_time, max_time):
+        if execution_time > max_time:
+            raise AssertionError(f"Execution too slow: {execution_time}s > {max_time}s")
+```
+
+## 11. Version Compatibility
+
+- **Python Version**: 3.6+
+- **Dependencies**: Standard library only (no external dependencies for core functionality)
+- **Backward Compatibility**: Fully compatible with existing `JSONRunner` code
+- **Platform Support**: Windows, macOS, Linux
+
+## 12. Performance Benchmarks
+
+| Test Scenario | Sequential | Parallel (Thread) | Parallel (Process) | Speedup |
+|---------------|------------|-------------------|-------------------|---------|
+| 10 I/O tests  | 5.2s       | 1.4s              | 2.1s              | 3.7x    |
+| 20 CPU tests  | 12.8s      | 8.9s              | 6.2s              | 2.1x    |
+| Mixed tests   | 8.5s       | 2.3s              | 3.1s              | 3.7x    |
+
+## 13. Contributing
+
+1. Fork the repository
+2. Create a feature branch
+3. Add tests for new functionality
+4. Ensure all tests pass: `python -m pytest tests/ -v`
+5. Submit a pull request
+
+## 14. License
+
+This project is licensed under the MIT License - see the LICENSE file for details.
+
+---
+
+## 📚 Complete Documentation
+
+For comprehensive documentation including detailed Setup Module guide, API reference, and advanced usage examples, see:
+
+**[📖 Complete User Manual](https://github.com/ozil111/cli-test-framework/blob/main/docs/user_manual.md)**
+
+The user manual includes:
+- 🔧 **Setup Module**: Complete guide for environment variables and custom plugins
+- 🚀 **Parallel Testing**: Advanced parallel execution strategies  
+- 📁 **File Comparison**: Detailed comparison capabilities for all file types
+- 🔌 **API Reference**: Full API documentation and examples
+- 🛠️ **Troubleshooting**: Common issues and solutions
+- 📝 **Best Practices**: Recommended patterns and configurations
+
+---
+
+**🚀 Ready to supercharge your testing workflow with setup modules, parallel execution and advanced file comparison!**
+
+For detailed parallel testing guide, see: [PARALLEL_TESTING_GUIDE.md](https://github.com/ozil111/cli-test-framework/blob/main/PARALLEL_TESTING_GUIDE.md)

{cli_test_framework-0.3.2 → cli_test_framework-0.3.4}/README.md

@@ -25,7 +25,7 @@ This is a lightweight and extensible automated testing framework that supports d
 
 ```bash
 pip install cli-test-framework
-Python >= 3.6
+Python >= 3.9
 ```
 
 ### Sequential Execution
@@ -179,12 +179,25 @@ compare-files data1.json data2.json \
 ```
 
 #### HDF5 Comparison
+
+**New Feature**: HDF5 group path expansion! By default, when you specify a group path in `--h5-table`, the comparator will automatically expand and compare all datasets and subgroups within that path.
+
 ```bash
+# Compare specific tables/groups with auto-expansion (default behavior)
 compare-files data1.h5 data2.h5 \
-    --h5-table table1,table2 \
-    --h5-structure-only \
+    --h5-table group1/subgroupA \
     --h5-rtol 1e-5 \
     --h5-atol 1e-8
+
+# Disable auto-expansion to compare only the specified path itself
+compare-files data1.h5 data2.h5 \
+    --h5-table group1 \
+    --h5-no-expand-path
+
+# Use regex patterns (also supports auto-expansion)
+compare-files data1.h5 data2.h5 \
+    --h5-table-regex "group1/.*" \
+    --h5-structure-only
 ```
 
 #### Binary Comparison
@@ -456,7 +469,7 @@ This project is licensed under the MIT License - see the LICENSE file for detail
 
 For comprehensive documentation including detailed Setup Module guide, API reference, and advanced usage examples, see:
 
-**[📖 Complete User Manual](docs/user_manual.md)**
+**[📖 Complete User Manual](https://github.com/ozil111/cli-test-framework/blob/main/docs/user_manual.md)**
 
 The user manual includes:
 - 🔧 **Setup Module**: Complete guide for environment variables and custom plugins
@@ -470,4 +483,4 @@ The user manual includes:
 
 **🚀 Ready to supercharge your testing workflow with setup modules, parallel execution and advanced file comparison!**
 
-For detailed parallel testing guide, see: [PARALLEL_TESTING_GUIDE.md](PARALLEL_TESTING_GUIDE.md)
+For detailed parallel testing guide, see: [PARALLEL_TESTING_GUIDE.md](https://github.com/ozil111/cli-test-framework/blob/main/PARALLEL_TESTING_GUIDE.md)

{cli_test_framework-0.3.2 → cli_test_framework-0.3.4}/docs/user_manual.md

@@ -27,7 +27,7 @@ The CLI Testing Framework is a powerful tool designed for testing command-line a
 ## Installation
 
 ### Prerequisites
-- Python 3.6 or higher
+- Python 3.9 or higher
 - pip package manager
 
 ### Basic Installation
@@ -37,7 +37,7 @@ pip install cli-test-framework
 
 ### Development Installation
 ```bash
-git clone https://github.com/yourusername/cli-test-framework.git
+git clone https://github.com/ozil111/cli-test-framework.git
 cd cli-test-framework
 pip install -e .
 ```

{cli_test_framework-0.3.2 → cli_test_framework-0.3.4}/setup.py

@@ -1,26 +1,27 @@
 from setuptools import setup, find_packages
 import os
 
-# Read the contents of the user manual file
+# Read the contents of the README file
 this_directory = os.path.abspath(os.path.dirname(__file__))
-with open(os.path.join(this_directory, 'docs', 'user_manual.md'), encoding='utf-8') as f:
+with open(os.path.join(this_directory, 'README.md'), encoding='utf-8') as f:
     long_description = f.read()
 
 setup(
     name="cli-test-framework",
-    version="0.3.2",
+    version="0.3.4",
     author="Xiaotong Wang",
     author_email="xiaotongwang98@gmail.com",
     description="A powerful command line testing framework in Python with setup modules, parallel execution, and file comparison capabilities.",
     long_description=long_description,
     long_description_content_type="text/markdown",
-    url="https://github.com/yourusername/cli-test-framework",
+    url="https://github.com/ozil111/cli-test-framework",
     packages=find_packages(where="src"),
     package_dir={"": "src"},
     install_requires=[
         "dukpy==0.5.0",
-        "h5py>=3.8.0",
-        "numpy>=2.0.1",
+        "h5py>=3.10.0",
+        "numpy>=1.24.0,<2.0.0",
+        "PyYAML>=6.0",
         "setuptools>=75.8.0",
         "wheel>=0.45.1"
     ],
@@ -41,8 +42,8 @@ setup(
     ],
     python_requires='>=3.9',
     project_urls={
-        'Documentation': 'https://github.com/yourusername/cli-test-framework/docs/user_manual.md',
-        'Source': 'https://github.com/yourusername/cli-test-framework',
-        'Tracker': 'https://github.com/yourusername/cli-test-framework/issues',
+        'Documentation': 'https://github.com/ozil111/cli-test-framework/blob/main/docs/user_manual.md',
+        'Source': 'https://github.com/ozil111/cli-test-framework',
+        'Tracker': 'https://github.com/ozil111/cli-test-framework/issues',
     },
 )

{cli_test_framework-0.3.2 → cli_test_framework-0.3.4}/src/cli_test_framework/__init__.py

@@ -5,7 +5,7 @@ This package provides tools for testing command-line applications and scripts
 with support for parallel execution and advanced file comparison capabilities.
 """
 
-__version__ = "0.3.2"
+__version__ = "0.3.3"
 __author__ = "Xiaotong Wang"
 __email__ = "xiaotongwang98@gmail.com"
 

{cli_test_framework-0.3.2 → cli_test_framework-0.3.4}/src/cli_test_framework/commands/compare.py

@@ -67,6 +67,8 @@ def parse_arguments():
                          help="Relative tolerance for numerical comparison in HDF5 files")
     h5_group.add_argument("--h5-atol", type=float, default=1e-8,
                          help="Absolute tolerance for numerical comparison in HDF5 files")
+    h5_group.add_argument("--h5-no-expand-path", dest="h5_expand_path", action="store_false",
+                         help="Do not expand HDF5 group paths to compare all sub-items.")
     
     return parser.parse_args()
 
@@ -148,6 +150,7 @@ def main():
             comparator_kwargs["show_content_diff"] = args.h5_show_content_diff
             comparator_kwargs["rtol"] = args.h5_rtol
             comparator_kwargs["atol"] = args.h5_atol
+            comparator_kwargs["expand_path"] = args.h5_expand_path
         
         if file_type == "binary":
             comparator_kwargs["similarity"] = args.similarity