cli-test-framework 0.4.0__tar.gz → 0.4.2__tar.gz

{cli_test_framework-0.4.0/src/cli_test_framework.egg-info → cli_test_framework-0.4.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: cli-test-framework
-Version: 0.4.0
+Version: 0.4.2
 Summary: A powerful command line testing framework in Python with setup modules, parallel execution, and file comparison capabilities.
 Home-page: https://github.com/ozil111/cli-test-framework
 Author: Xiaotong Wang
@@ -53,6 +53,7 @@ This is a lightweight and extensible automated testing framework that supports d
 - **📊 Comprehensive Reports**: Detailed pass rate statistics and failure diagnostics
 - **🔧 Thread-Safe Design**: Robust concurrent execution with proper synchronization
 - **📝 Advanced File Comparison**: Support for comparing various file types (text, binary, JSON, HDF5) with detailed diff output
+- **🎛️ Resource-Aware Scheduling**: Per-test timeout and resource hints (CPU cores / estimated time / memory / priority) with automatic CPU core detection and semaphore-based scheduling to prevent resource conflicts and solver "runaway" scenarios
 
 ## 3. Quick Start
 
@@ -80,16 +81,22 @@ success = runner.run_tests()
 ```python
 from src.runners.parallel_json_runner import ParallelJSONRunner
 
-# Multi-threaded execution (recommended for I/O-intensive tests)
+# Multi-threaded execution with resource-aware scheduling
 runner = ParallelJSONRunner(
     config_file="path/to/test_cases.json",
     workspace="/project/root",
     max_workers=4,           # Maximum concurrent workers
-    execution_mode="thread"  # "thread" or "process"
+    execution_mode="thread"  # "thread" (supports CPU scheduling) or "process"
 )
 success = runner.run_tests()
 ```
 
+**Resource-Aware Scheduling**: When using `execution_mode="thread"`, the framework automatically:
+- Detects available CPU cores on your machine
+- Manages CPU resource allocation using semaphore-based scheduling
+- Prevents resource conflicts by queuing tasks that require more cores than available
+- Injects environment variables to constrain solver threads (prevents "runaway" scenarios)
+
 ### Setup Module Usage
 
 ```python
@@ -144,6 +151,21 @@ compare-files binary1.bin binary2.bin --similarity
                 "output_contains": ["Environment: development"]
             }
         },
+        {
+            "name": "Full_Car_Crash_Simulation",
+            "command": "radioss_solver",
+            "args": ["-i", "input.0000.rad"],
+            "timeout": 36000,
+            "resources": {
+                "cpu_cores": 4,
+                "estimated_time": 18000,
+                "min_memory_mb": 16000,
+                "priority": 10
+            },
+            "expected": {
+                "return_code": 0
+            }
+        },
         {
             "name": "File Comparison Test", 
             "command": "compare-files",
@@ -187,6 +209,61 @@ test_cases:
       output_matches: ".*\\.md$"
 ```
 
+### Resource-Aware Configuration
+
+For simulation and long-running tasks (CAE/FEA), you can specify resource requirements to enable intelligent scheduling with automatic CPU core management:
+
+```json
+{
+    "name": "Full_Car_Crash_Simulation",
+    "command": "radioss_solver",
+    "args": ["-i", "input.0000.rad"],
+    "timeout": 36000,
+    "resources": {
+        "cpu_cores": 4,
+        "estimated_time": 18000,
+        "min_memory_mb": 16000,
+        "priority": 10
+    },
+    "expected": {
+        "return_code": 0
+    }
+}
+```
+
+**Field Descriptions:**
+
+- **`timeout`** (optional, float): **Hard limit in seconds**. If the test exceeds this time, it will be killed. Default: 3600 seconds (1 hour). Set to `null` for unlimited (not recommended).
+  - Common values: `60` (1 min), `300` (5 min), `3600` (1 hour), `18000` (5 hours), `86400` (24 hours)
+
+- **`resources.cpu_cores`** (optional, int): **Number of CPU cores required by this task**. The framework automatically detects available CPU cores and uses semaphore-based scheduling to manage resource allocation. Tasks that require more cores than available will wait until resources are freed. Default: `1` core.
+  - **How it works**: The framework automatically detects your machine's CPU count (e.g., 16 cores), reserves 2 cores for the system, and creates a resource pool with the remaining cores (e.g., 14 cores). Tasks acquire cores from this pool before execution.
+  - **Environment injection**: When a task starts, the framework automatically sets `OMP_NUM_THREADS`, `MKL_NUM_THREADS`, and `NPROC` environment variables to constrain solver threads, preventing "runaway" scenarios where solvers ignore Python's scheduling.
+  - **Example scenarios**:
+    - Machine with 16 cores (14 available): 3 tasks requiring 4 cores each can run concurrently (3×4=12 cores used, 2 cores free)
+    - Machine with 8 cores (6 available): 1 task requiring 4 cores + 1 task requiring 2 cores can run concurrently
+  - **Recommendations**:
+    - Heavy simulations: `4-8` cores
+    - Medium tasks: `2-4` cores  
+    - Lightweight scripts: `1` core (default)
+
+- **`resources.estimated_time`** (optional, float): **Estimated duration in seconds** for LPT (Longest Processing Time) scheduling. Tasks with longer estimated times are scheduled first in parallel runs to improve throughput.
+  - Example: `18000` = 5 hours, `3600` = 1 hour, `300` = 5 minutes
+
+- **`resources.min_memory_mb`** (optional, float): **Estimated memory requirement in MB**. Used for OOM (Out Of Memory) risk warnings. Currently informational only.
+  - Example: `16000` = 16 GB, `8192` = 8 GB, `4096` = 4 GB
+
+- **`resources.priority`** (optional, int): **Task priority** (higher number = higher priority). Currently informational only. Recommended range: 0-10.
+  - `10`: Critical/blocking tasks (must run first)
+  - `7-9`: High priority (important business paths)
+  - `4-6`: Normal priority
+  - `1-3`: Low priority / exploratory tests
+  - `0` or unset: Default priority
+
+**Note:** 
+- All time values (`timeout`, `estimated_time`) are in **seconds**, not milliseconds. This matches Python's `subprocess.run(timeout=...)` API.
+- **CPU core scheduling is only active in thread mode**. Process mode will fall back to original behavior (process-level isolation provides some resource separation).
+
 ## 5. File Comparison Features
 
 ### Supported File Types
@@ -354,12 +431,35 @@ except Exception as e:
    max_workers = os.cpu_count() * 2
    ```
 
-2. **Test Case Design**:
+2. **Resource-Aware Scheduling**:
+   - **For CAE/FEA simulations**: Always specify `cpu_cores` in your test configuration to prevent resource conflicts
+   - **Mixed workloads**: Configure lightweight tasks with `cpu_cores: 1` and heavy simulations with appropriate core counts
+   - **Example mixed configuration**:
+     ```json
+     {
+         "test_cases": [
+             {
+                 "name": "Heavy Simulation",
+                 "command": "radioss_solver",
+                 "resources": { "cpu_cores": 4 }
+             },
+             {
+                 "name": "Lightweight Script",
+                 "command": "python script.py",
+                 "resources": { "cpu_cores": 1 }
+             }
+         ]
+     }
+     ```
+   - **Monitor resource usage**: The framework prints resource acquisition/release logs to help you understand scheduling behavior
+
+3. **Test Case Design**:
    - ✅ Ensure test independence (no dependencies between tests)
    - ✅ Avoid shared resource conflicts (different files/ports)
    - ✅ Use relative paths (framework handles resolution automatically)
+   - ✅ Specify `cpu_cores` for CPU-intensive tasks to enable intelligent scheduling
 
-3. **Debugging**:
+4. **Debugging**:
    ```python
    # Enable verbose output for debugging
    runner = ParallelJSONRunner(
@@ -544,7 +644,22 @@ compare-files data1.h5 data2.h5 --h5-data-filter '<=0.01'
 
 # 版本更新日志
 
-## 0.3.7 (Latest)
+## 0.4.2 (Latest)
+
+### ✨ New Features
+- **Resource-Aware CPU Scheduling**: Automatic CPU core detection and semaphore-based scheduling to prevent resource conflicts
+  - Added `cpu_cores` field in `resources` configuration to specify CPU requirements per task
+  - Automatic environment variable injection (`OMP_NUM_THREADS`, `MKL_NUM_THREADS`, `NPROC`) to constrain solver threads
+  - Prevents solver "runaway" scenarios where solvers ignore Python's scheduling
+  - Intelligent resource pool management: automatically reserves 2 cores for system use
+- **Enhanced execution engine**: Support for custom environment variables in test execution
+
+### 🔧 Improvements
+- **Better resource management**: Tasks now wait for available CPU cores instead of overwhelming the system
+- **Automatic CPU detection**: No manual configuration needed - framework detects available cores automatically
+- **Thread-safe resource allocation**: Semaphore-based scheduling ensures thread-safe resource management
+
+## 0.3.7
 
 ### 🐛 Bug Fixes
 - **Fixed H5 table regex matching**: `--h5-table-regex=table1,table2` now correctly matches both `table1` and `table2` instead of treating the entire string as a single regex pattern

{cli_test_framework-0.4.0 → cli_test_framework-0.4.2}/README.md

@@ -18,6 +18,7 @@ This is a lightweight and extensible automated testing framework that supports d
 - **📊 Comprehensive Reports**: Detailed pass rate statistics and failure diagnostics
 - **🔧 Thread-Safe Design**: Robust concurrent execution with proper synchronization
 - **📝 Advanced File Comparison**: Support for comparing various file types (text, binary, JSON, HDF5) with detailed diff output
+- **🎛️ Resource-Aware Scheduling**: Per-test timeout and resource hints (CPU cores / estimated time / memory / priority) with automatic CPU core detection and semaphore-based scheduling to prevent resource conflicts and solver "runaway" scenarios
 
 ## 3. Quick Start
 
@@ -45,16 +46,22 @@ success = runner.run_tests()
 ```python
 from src.runners.parallel_json_runner import ParallelJSONRunner
 
-# Multi-threaded execution (recommended for I/O-intensive tests)
+# Multi-threaded execution with resource-aware scheduling
 runner = ParallelJSONRunner(
     config_file="path/to/test_cases.json",
     workspace="/project/root",
     max_workers=4,           # Maximum concurrent workers
-    execution_mode="thread"  # "thread" or "process"
+    execution_mode="thread"  # "thread" (supports CPU scheduling) or "process"
 )
 success = runner.run_tests()
 ```
 
+**Resource-Aware Scheduling**: When using `execution_mode="thread"`, the framework automatically:
+- Detects available CPU cores on your machine
+- Manages CPU resource allocation using semaphore-based scheduling
+- Prevents resource conflicts by queuing tasks that require more cores than available
+- Injects environment variables to constrain solver threads (prevents "runaway" scenarios)
+
 ### Setup Module Usage
 
 ```python
@@ -109,6 +116,21 @@ compare-files binary1.bin binary2.bin --similarity
                 "output_contains": ["Environment: development"]
             }
         },
+        {
+            "name": "Full_Car_Crash_Simulation",
+            "command": "radioss_solver",
+            "args": ["-i", "input.0000.rad"],
+            "timeout": 36000,
+            "resources": {
+                "cpu_cores": 4,
+                "estimated_time": 18000,
+                "min_memory_mb": 16000,
+                "priority": 10
+            },
+            "expected": {
+                "return_code": 0
+            }
+        },
         {
             "name": "File Comparison Test", 
             "command": "compare-files",
@@ -152,6 +174,61 @@ test_cases:
       output_matches: ".*\\.md$"
 ```
 
+### Resource-Aware Configuration
+
+For simulation and long-running tasks (CAE/FEA), you can specify resource requirements to enable intelligent scheduling with automatic CPU core management:
+
+```json
+{
+    "name": "Full_Car_Crash_Simulation",
+    "command": "radioss_solver",
+    "args": ["-i", "input.0000.rad"],
+    "timeout": 36000,
+    "resources": {
+        "cpu_cores": 4,
+        "estimated_time": 18000,
+        "min_memory_mb": 16000,
+        "priority": 10
+    },
+    "expected": {
+        "return_code": 0
+    }
+}
+```
+
+**Field Descriptions:**
+
+- **`timeout`** (optional, float): **Hard limit in seconds**. If the test exceeds this time, it will be killed. Default: 3600 seconds (1 hour). Set to `null` for unlimited (not recommended).
+  - Common values: `60` (1 min), `300` (5 min), `3600` (1 hour), `18000` (5 hours), `86400` (24 hours)
+
+- **`resources.cpu_cores`** (optional, int): **Number of CPU cores required by this task**. The framework automatically detects available CPU cores and uses semaphore-based scheduling to manage resource allocation. Tasks that require more cores than available will wait until resources are freed. Default: `1` core.
+  - **How it works**: The framework automatically detects your machine's CPU count (e.g., 16 cores), reserves 2 cores for the system, and creates a resource pool with the remaining cores (e.g., 14 cores). Tasks acquire cores from this pool before execution.
+  - **Environment injection**: When a task starts, the framework automatically sets `OMP_NUM_THREADS`, `MKL_NUM_THREADS`, and `NPROC` environment variables to constrain solver threads, preventing "runaway" scenarios where solvers ignore Python's scheduling.
+  - **Example scenarios**:
+    - Machine with 16 cores (14 available): 3 tasks requiring 4 cores each can run concurrently (3×4=12 cores used, 2 cores free)
+    - Machine with 8 cores (6 available): 1 task requiring 4 cores + 1 task requiring 2 cores can run concurrently
+  - **Recommendations**:
+    - Heavy simulations: `4-8` cores
+    - Medium tasks: `2-4` cores  
+    - Lightweight scripts: `1` core (default)
+
+- **`resources.estimated_time`** (optional, float): **Estimated duration in seconds** for LPT (Longest Processing Time) scheduling. Tasks with longer estimated times are scheduled first in parallel runs to improve throughput.
+  - Example: `18000` = 5 hours, `3600` = 1 hour, `300` = 5 minutes
+
+- **`resources.min_memory_mb`** (optional, float): **Estimated memory requirement in MB**. Used for OOM (Out Of Memory) risk warnings. Currently informational only.
+  - Example: `16000` = 16 GB, `8192` = 8 GB, `4096` = 4 GB
+
+- **`resources.priority`** (optional, int): **Task priority** (higher number = higher priority). Currently informational only. Recommended range: 0-10.
+  - `10`: Critical/blocking tasks (must run first)
+  - `7-9`: High priority (important business paths)
+  - `4-6`: Normal priority
+  - `1-3`: Low priority / exploratory tests
+  - `0` or unset: Default priority
+
+**Note:** 
+- All time values (`timeout`, `estimated_time`) are in **seconds**, not milliseconds. This matches Python's `subprocess.run(timeout=...)` API.
+- **CPU core scheduling is only active in thread mode**. Process mode will fall back to original behavior (process-level isolation provides some resource separation).
+
 ## 5. File Comparison Features
 
 ### Supported File Types
@@ -319,12 +396,35 @@ except Exception as e:
    max_workers = os.cpu_count() * 2
    ```
 
-2. **Test Case Design**:
+2. **Resource-Aware Scheduling**:
+   - **For CAE/FEA simulations**: Always specify `cpu_cores` in your test configuration to prevent resource conflicts
+   - **Mixed workloads**: Configure lightweight tasks with `cpu_cores: 1` and heavy simulations with appropriate core counts
+   - **Example mixed configuration**:
+     ```json
+     {
+         "test_cases": [
+             {
+                 "name": "Heavy Simulation",
+                 "command": "radioss_solver",
+                 "resources": { "cpu_cores": 4 }
+             },
+             {
+                 "name": "Lightweight Script",
+                 "command": "python script.py",
+                 "resources": { "cpu_cores": 1 }
+             }
+         ]
+     }
+     ```
+   - **Monitor resource usage**: The framework prints resource acquisition/release logs to help you understand scheduling behavior
+
+3. **Test Case Design**:
    - ✅ Ensure test independence (no dependencies between tests)
    - ✅ Avoid shared resource conflicts (different files/ports)
    - ✅ Use relative paths (framework handles resolution automatically)
+   - ✅ Specify `cpu_cores` for CPU-intensive tasks to enable intelligent scheduling
 
-3. **Debugging**:
+4. **Debugging**:
    ```python
    # Enable verbose output for debugging
    runner = ParallelJSONRunner(
@@ -509,7 +609,22 @@ compare-files data1.h5 data2.h5 --h5-data-filter '<=0.01'
 
 # 版本更新日志
 
-## 0.3.7 (Latest)
+## 0.4.2 (Latest)
+
+### ✨ New Features
+- **Resource-Aware CPU Scheduling**: Automatic CPU core detection and semaphore-based scheduling to prevent resource conflicts
+  - Added `cpu_cores` field in `resources` configuration to specify CPU requirements per task
+  - Automatic environment variable injection (`OMP_NUM_THREADS`, `MKL_NUM_THREADS`, `NPROC`) to constrain solver threads
+  - Prevents solver "runaway" scenarios where solvers ignore Python's scheduling
+  - Intelligent resource pool management: automatically reserves 2 cores for system use
+- **Enhanced execution engine**: Support for custom environment variables in test execution
+
+### 🔧 Improvements
+- **Better resource management**: Tasks now wait for available CPU cores instead of overwhelming the system
+- **Automatic CPU detection**: No manual configuration needed - framework detects available cores automatically
+- **Thread-safe resource allocation**: Semaphore-based scheduling ensures thread-safe resource management
+
+## 0.3.7
 
 ### 🐛 Bug Fixes
 - **Fixed H5 table regex matching**: `--h5-table-regex=table1,table2` now correctly matches both `table1` and `table2` instead of treating the entire string as a single regex pattern

{cli_test_framework-0.4.0 → cli_test_framework-0.4.2}/setup.py

@@ -8,7 +8,7 @@ with open(os.path.join(this_directory, 'README.md'), encoding='utf-8') as f:
 
 setup(
     name="cli-test-framework",
-    version="0.4.0",
+    version="0.4.2",
     author="Xiaotong Wang",
     author_email="xiaotongwang98@gmail.com",
     description="A powerful command line testing framework in Python with setup modules, parallel execution, and file comparison capabilities.",

{cli_test_framework-0.4.0 → cli_test_framework-0.4.2}/src/cli_test_framework/core/execution.py

@@ -1,6 +1,7 @@
 import subprocess
 import time
-from typing import Optional
+import os
+from typing import Optional, Dict
 
 from .assertions import Assertions
 from .types import ExpectedResult, TestCaseData, TestResultData
@@ -23,12 +24,18 @@ def validate_result(expected: ExpectedResult, actual: TestResultData) -> None:
         assertions.matches(actual["output"], expected["output_matches"])
 
 
-def execute_single_test_case(case: TestCaseData, workspace: Optional[str] = None) -> TestResultData:
+def execute_single_test_case(case: TestCaseData, workspace: Optional[str] = None, env: Optional[Dict[str, str]] = None) -> TestResultData:
     """
     Stateless execution of a single test case.
+    
+    Args:
+        case: Test case data
+        workspace: Working directory for test execution
+        env: Optional environment variables to inject/override (merged with os.environ)
     """
     start_time = time.time()
     full_command = f"{case['command']} {' '.join(case['args'])}".strip()
+    timeout_limit = case.get("timeout", 3600)
 
     result: TestResultData = {
         "name": case["name"],
@@ -40,6 +47,12 @@ def execute_single_test_case(case: TestCaseData, workspace: Optional[str] = None
         "duration": 0.0,
     }
 
+    # Prepare environment variables
+    # Default to current environment, merge with provided env if any
+    current_env = os.environ.copy()
+    if env:
+        current_env.update(env)
+
     try:
         process = subprocess.run(
             full_command,
@@ -48,6 +61,8 @@ def execute_single_test_case(case: TestCaseData, workspace: Optional[str] = None
             text=True,
             check=False,
             shell=True,
+            timeout=timeout_limit if timeout_limit is not None else None,
+            env=current_env,
         )
 
         output = process.stdout + process.stderr
@@ -56,6 +71,11 @@ def execute_single_test_case(case: TestCaseData, workspace: Optional[str] = None
 
         validate_result(case["expected"], result)
         result["status"] = "passed"
+    except subprocess.TimeoutExpired as exc:
+        result["status"] = "timeout"
+        result["message"] = f"Timeout reached! Killed after {timeout_limit} seconds."
+        result["output"] = (exc.stdout or "") + (exc.stderr or "")
+        result["return_code"] = None
     except AssertionError as exc:
         result["message"] = str(exc)
     except Exception as exc:

{cli_test_framework-0.4.0 → cli_test_framework-0.4.2}/src/cli_test_framework/core/parallel_runner.py

@@ -59,7 +59,9 @@ class ParallelRunner(BaseRunner):
                                 "name": case.name,
                                 "command": case.command,
                                 "args": case.args,
-                                "expected": case.expected
+                                "expected": case.expected,
+                                "timeout": case.timeout,
+                                "resources": case.resources
                             },
                             str(self.workspace) if self.workspace else None
                         ): (i, case) 

{cli_test_framework-0.4.0 → cli_test_framework-0.4.2}/src/cli_test_framework/core/process_worker.py

@@ -25,6 +25,8 @@ def run_test_in_process(test_index: int, case_data: Dict[str, Any], workspace: s
         "args": case_data["args"],
         "expected": case_data["expected"],
         "description": case_data.get("description"),
+        "timeout": case_data.get("timeout"),
+        "resources": case_data.get("resources"),
     }
 
     command_preview = f"{case['command']} {' '.join(case['args'])}".strip()

{cli_test_framework-0.4.0 → cli_test_framework-0.4.2}/src/cli_test_framework/core/test_case.py

@@ -1,5 +1,5 @@
 from dataclasses import dataclass
-from typing import List, Dict, Any
+from typing import List, Dict, Any, Optional
 
 @dataclass
 class TestCase:
@@ -8,6 +8,8 @@ class TestCase:
     args: List[str]
     expected: Dict[str, Any]
     description: str = ""
+    timeout: Optional[float] = None
+    resources: Optional[Dict[str, Any]] = None
     
     def to_dict(self) -> Dict[str, Any]:
         """Convert test case to dictionary format"""
@@ -17,5 +19,7 @@ class TestCase:
             "name": self.name,
             "command": self.command,
             "args": self.args,
-            "expected": self.expected
+            "expected": self.expected,
+            "timeout": self.timeout,
+            "resources": self.resources,
         }

{cli_test_framework-0.4.0 → cli_test_framework-0.4.2}/src/cli_test_framework/core/types.py

@@ -9,6 +9,15 @@ class ExpectedResult(TypedDict, total=False):
     output_matches: Optional[str]
 
 
+class ResourceRequirements(TypedDict, total=False):
+    """Optional resource hints for scheduling."""
+
+    estimated_time: float  # seconds, used for ordering (LPT)
+    min_memory_mb: float  # soft hint to avoid OOM
+    priority: int  # higher value => higher priority
+    cpu_cores: int  # number of CPU cores required by this task
+
+
 class TestCaseData(TypedDict):
     """Input data shape for a test case after解析/路径处理."""
 
@@ -17,6 +26,8 @@ class TestCaseData(TypedDict):
     args: List[str]
     expected: ExpectedResult
     description: Optional[str]
+    timeout: Optional[float]
+    resources: Optional[ResourceRequirements]
 
 
 class SetupConfig(TypedDict):