smartpool-examples 0.1.0__tar.gz

smartpool_examples-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 王炳辉
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

smartpool_examples-0.1.0/PKG-INFO

@@ -0,0 +1,84 @@
+Metadata-Version: 2.4
+Name: smartpool-examples
+Version: 0.1.0
+Summary: Examples for smartpool.
+Author-email: "王炳辉 (Bing-Hui WANG)" <binghui.wang@foxmail.com>
+License: MIT
+Project-URL: Homepage, https://github.com/Time-Coder/smartpool
+Project-URL: Repository, https://github.com/Time-Coder/smartpool.git
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: pysmartpool
+Requires-Dist: matplotlib
+Requires-Dist: scikit-learn
+Requires-Dist: numpy
+Requires-Dist: rich
+Requires-Dist: click
+Requires-Dist: joblib
+Requires-Dist: datawatcher
+Dynamic: license-file
+
+# SmartPool Examples
+
+This package contains practical examples demonstrating the capabilities of SmartPool for various computational tasks.
+
+## Examples Overview
+
+### 1. Prime Number Counting (`count_prime`)
+
+Count the number of prime numbers below 10000 using smartpool.ProcessPool.
+Demonstrates basic usage of smartpool.ProcessPool.
+
+#### Running the Example
+
+```bash
+python -m smartpool_examples.count_prime
+```
+
+### 2. Cross-Validation for Deep Learning models (`cross_validation`)
+
+Demonstrates SmartPool's capabilities for machine learning workloads with GPU resource management.
+
+#### Running the Example
+
+```bash
+# Using ProcessPool  
+python -m smartpool_examples.cross_validation --pool smartpool.ProcessPool
+
+# Using ThreadPool
+python -m smartpool_examples.cross_validation --pool smartpool.ThreadPool
+
+# Using multiprocessing.Pool
+python -m smartpool_examples.cross_validation --pool multiprocessing.Pool
+
+# Using concurrent.futures.ProcessPoolExecutor
+python -m smartpool_examples.cross_validation --pool concurrent.futures.ProcessPoolExecutor
+
+# Using concurrent.futures.ThreadPoolExecutor
+python -m smartpool_examples.cross_validation --pool concurrent.futures.ThreadPoolExecutor
+
+# Using joblib.Parallel(backend='loky')
+python -m smartpool_examples.cross_validation --pool joblib.Parallel(backend='loky')
+
+# Using joblib.Parallel(backend='threading')
+python -m smartpool_examples.cross_validation --pool joblib.Parallel(backend='threading')
+
+# Using Ray
+python -m smartpool_examples.cross_validation --pool ray
+```
+
+#### What it Demonstrates
+
+- GPU memory management and core allocation
+- Automatic device selection (CPU vs GPU)
+- Cross-validation pipeline parallelization
+- Resource monitoring during training
+- Performance comparison with external frameworks
+
+
+## License
+
+MIT License - see main smartpool repository for details

smartpool_examples-0.1.0/README.md

@@ -0,0 +1,61 @@
+# SmartPool Examples
+
+This package contains practical examples demonstrating the capabilities of SmartPool for various computational tasks.
+
+## Examples Overview
+
+### 1. Prime Number Counting (`count_prime`)
+
+Count the number of prime numbers below 10000 using smartpool.ProcessPool.
+Demonstrates basic usage of smartpool.ProcessPool.
+
+#### Running the Example
+
+```bash
+python -m smartpool_examples.count_prime
+```
+
+### 2. Cross-Validation for Deep Learning models (`cross_validation`)
+
+Demonstrates SmartPool's capabilities for machine learning workloads with GPU resource management.
+
+#### Running the Example
+
+```bash
+# Using ProcessPool  
+python -m smartpool_examples.cross_validation --pool smartpool.ProcessPool
+
+# Using ThreadPool
+python -m smartpool_examples.cross_validation --pool smartpool.ThreadPool
+
+# Using multiprocessing.Pool
+python -m smartpool_examples.cross_validation --pool multiprocessing.Pool
+
+# Using concurrent.futures.ProcessPoolExecutor
+python -m smartpool_examples.cross_validation --pool concurrent.futures.ProcessPoolExecutor
+
+# Using concurrent.futures.ThreadPoolExecutor
+python -m smartpool_examples.cross_validation --pool concurrent.futures.ThreadPoolExecutor
+
+# Using joblib.Parallel(backend='loky')
+python -m smartpool_examples.cross_validation --pool joblib.Parallel(backend='loky')
+
+# Using joblib.Parallel(backend='threading')
+python -m smartpool_examples.cross_validation --pool joblib.Parallel(backend='threading')
+
+# Using Ray
+python -m smartpool_examples.cross_validation --pool ray
+```
+
+#### What it Demonstrates
+
+- GPU memory management and core allocation
+- Automatic device selection (CPU vs GPU)
+- Cross-validation pipeline parallelization
+- Resource monitoring during training
+- Performance comparison with external frameworks
+
+
+## License
+
+MIT License - see main smartpool repository for details

smartpool_examples-0.1.0/pyproject.toml

@@ -0,0 +1,32 @@
+[build-system]
+requires = ["setuptools>=45", "wheel"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "smartpool-examples"
+version = "0.1.0"
+description = "Examples for smartpool."
+readme = "README.md"
+authors = [
+    { name = "王炳辉 (Bing-Hui WANG)", email = "binghui.wang@foxmail.com" }
+]
+license = { text = "MIT" }
+classifiers = [
+    "Programming Language :: Python :: 3",
+    "License :: OSI Approved :: MIT License",
+    "Operating System :: OS Independent",
+]
+dependencies = [
+    "pysmartpool",
+    "matplotlib",
+    "scikit-learn",
+    "numpy",
+    "rich",
+    "click",
+    "joblib",
+    "datawatcher"
+]
+
+[project.urls]
+Homepage = "https://github.com/Time-Coder/smartpool"
+Repository = "https://github.com/Time-Coder/smartpool.git"

smartpool_examples-0.1.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

smartpool_examples-0.1.0/smartpool_examples/count_prime/__main__.py

@@ -0,0 +1,31 @@
+import os
+import sys
+
+self_folder = os.path.dirname(os.path.abspath(__file__)).replace("\\", "/")
+sys.path.append(self_folder)
+
+from smartpool import ProcessPool
+
+from count_prime import count_prime
+
+
+if __name__ == "__main__":
+    print("Use ProcessPool to count prime numbers lower than 10000.")
+    print(f"See source code at folder {os.path.dirname(os.path.abspath(__file__))}")
+    
+    tasks = []
+    start = 0
+    while start < 10000:
+        stop = start + 1000
+        tasks.append((start, stop))
+        start = stop
+        
+    with ProcessPool() as pool:
+        futures = []
+        for task in tasks:
+            future = pool.submit(count_prime, args=task)
+            futures.append(future)
+        
+        total_primes_count = sum(future.result() for future in futures)
+        print(total_primes_count)
+    

smartpool_examples-0.1.0/smartpool_examples/count_prime/count_prime.py

@@ -0,0 +1,18 @@
+import math
+
+
+def is_prime(num:int):
+    if num < 2:
+        return False
+    for i in range(2, int(math.sqrt(num)) + 1):
+        if num % i == 0:
+            return False
+    return True
+
+
+def count_prime(start:int, stop:int):
+    count = 0
+    for i in range(start, stop):
+        if is_prime(i):
+            count += 1
+    return count

smartpool_examples-0.1.0/smartpool_examples/cross_validation/__main__.py

@@ -0,0 +1,244 @@
+from smartpool import ProcessPool, ThreadPool, DataSize, limit_num_single_thread
+limit_num_single_thread()
+
+import click
+
+
+@click.command(help=f"Use smartpool to do 5-fold cross validatation for 7 deep learning models for handwritten digit recognition task.")
+@click.option(
+    '--pool', default='smartpool.ProcessPool', type=click.Choice([
+        'smartpool.ProcessPool',
+        'smartpool.ThreadPool',
+        'multiprocessing.Pool',
+        'concurrent.futures.ProcessPoolExecutor',
+        'concurrent.futures.ThreadPoolExecutor',
+        "joblib.Parallel(backend='loky')",
+        "joblib.Parallel(backend='threading')",
+        "ray"
+    ]),
+    help="choose process pool implementations"
+)
+@click.option(
+    '--max_workers', default=0, type=int,
+    help='max number of workers to use, 0 to use all available cores'
+)
+def main(pool:str="smart", max_workers:int=0):
+    import os
+    os.environ["RAY_ACCEL_ENV_VAR_OVERRIDE_ON_ZERO"] = "0"
+    
+    print(f"Use {pool} to do 5-fold cross validatation for 7 deep learning models for handwritten digit recognition task.")
+    print("Use `python -m smartpool_examples.cross_validation --help` to see all options.")
+    print(f"See source code at folder {os.path.dirname(os.path.abspath(__file__))}")
+    print("\npreparing data...")
+    
+    try:
+        import torch
+        import torch.nn as nn
+    except ImportError:
+        print("PyTorch is not installed. Follow https://pytorch.org/ instructions to install PyTorch.")
+        exit(1)
+
+    try:
+        import torchvision
+    except ImportError:
+        print("torchvision is not installed. Use `pip install torchvision` to install torchvision.")
+        exit(1)
+
+    import time
+    from sklearn.model_selection import KFold
+    import numpy as np
+    from rich.progress import Progress, BarColumn, TextColumn, TimeRemainingColumn
+    import multiprocessing as mp
+    import queue
+    from collections import defaultdict
+    from concurrent.futures import Future
+    from typing import Dict, Union
+
+    import os
+    import sys
+
+    self_folder = os.path.dirname(os.path.abspath(__file__)).replace("\\", "/")
+    sys.path.append(self_folder)
+
+    import models
+    from data_utils import prepare_data
+    from model_utils import train_single_fold, ErrorInfo, ProgressInfo, TrainingResult
+    from visualization import plot_results, print_results_table
+    from config import EPOCHS
+
+    if max_workers == 0:
+        max_workers = os.cpu_count()
+
+    model_classes = [
+        cls for cls in models.__dict__.values()
+        if isinstance(cls, type) and issubclass(cls, nn.Module) and cls != nn.Module
+    ]
+    
+    dataset = prepare_data()
+    kfold = KFold(n_splits=5, shuffle=True, random_state=42)
+    
+    manager = mp.Manager()
+
+    if pool != "ray":
+        progress_queue:queue.Queue[Union[ProgressInfo, ErrorInfo]] = manager.Queue()
+    else:
+        try:
+            import ray
+            import ray.util.queue
+        except ImportError:
+            print("Ray is not installed. Use `pip install ray` to install Ray.")
+            exit(1)
+        
+        progress_queue:queue.Queue[Union[ProgressInfo, ErrorInfo]] = ray.util.queue.Queue()
+
+    tasks = []
+    for fold_idx, (train_indices, val_indices) in enumerate(kfold.split(dataset)):
+        for model_class in model_classes:
+            tasks.append((fold_idx, model_class, train_indices.copy(), val_indices.copy(), dataset, progress_queue))
+    
+    task_progress_bars = {}
+    best_device = 'cuda' if torch.cuda.is_available() else 'cpu'
+    
+    start_time = time.perf_counter()
+    with Progress(
+        TextColumn("[progress.description]{task.description}"),
+        BarColumn(),
+        TextColumn("[progress.percentage]{task.percentage:>3.0f}%"),
+        TimeRemainingColumn()
+    ) as progress:
+        
+        active_tasks = {}
+
+        if pool == "smartpool.ProcessPool":
+            process_pool = ProcessPool(max_workers=max_workers, use_torch=True)
+        elif pool == "smartpool.ThreadPool":
+            process_pool = ThreadPool(max_workers=max_workers, use_torch=True)
+        elif pool == "concurrent.futures.ProcessPoolExecutor":
+            from concurrent.futures import ProcessPoolExecutor
+            process_pool = ProcessPoolExecutor(max_workers=max_workers)
+        elif pool == "concurrent.futures.ThreadPoolExecutor":
+            from concurrent.futures import ThreadPoolExecutor
+            process_pool = ThreadPoolExecutor(max_workers=max_workers)
+        elif pool == "multiprocessing.Pool":
+            import multiprocessing
+            process_pool = multiprocessing.Pool(processes=max_workers)
+        elif pool == "joblib.Parallel(backend='loky')":
+            from joblib import Parallel, delayed
+            process_pool = Parallel(n_jobs=max_workers, backend='loky', return_as="generator")
+        elif pool == "joblib.Parallel(backend='threading')":
+            from joblib import Parallel, delayed
+            process_pool = Parallel(n_jobs=max_workers, backend='threading', return_as="generator")
+        
+        print("submitting training tasks...")
+        futures_map:Dict[str, Future] = {}
+        futures = []
+        for i, task_args in enumerate(tasks):
+            if pool.startswith("smartpool."):
+                future = process_pool.submit(
+                    train_single_fold,
+                    args=task_args,
+                    need_cpu_cores=1,
+                    need_cpu_mem=1.1*DataSize.GB,
+                    need_gpu_cores=1000,
+                    need_gpu_mem=0.2*DataSize.GB
+                )
+            elif pool.startswith("concurrent.futures."):
+                future = process_pool.submit(train_single_fold, *task_args, best_device if i % max_workers < 5 else 'cpu')
+            elif pool == "multiprocessing.Pool":
+                future = process_pool.apply_async(train_single_fold, args=(*task_args, best_device if i % max_workers < 5 else 'cpu'))
+            elif pool.startswith("joblib"):
+                future = delayed(train_single_fold)(*task_args, best_device if i % max_workers < 5 else 'cpu')
+            elif pool == "ray":
+                future = ray.remote(num_cpus=1, num_gpus=(0.2 if i % max_workers < 5 else 0), memory=1.1*DataSize.GB)(train_single_fold).remote(*task_args, best_device if i % max_workers < 5 else 'cpu')
+            
+            fold_idx = task_args[0]
+            model_class = task_args[1]
+            model_name = model_class.__name__
+            task_key = f"{model_name}_fold_{fold_idx}"
+            futures_map[task_key] = future
+            futures.append(future)
+        
+        print(f"training all models in {pool} ...")
+        if pool.startswith("joblib"):
+            joblib_results = process_pool(futures)
+
+        finished_tasks = set()
+        while True:
+            progress_info:Union[ProgressInfo, ErrorInfo] = progress_queue.get()
+            if isinstance(progress_info, ErrorInfo):
+                print(progress_info.traceback)
+                break
+
+            task_key = f"{progress_info.model_name}_fold_{progress_info.fold_idx}"
+            
+            if task_key not in task_progress_bars:
+                initial_desc = f"train {progress_info.model_name} on {progress_info.device} "
+                initial_desc += f"for fold {progress_info.fold_idx+1}/5"
+                task_progress_bars[task_key] = progress.add_task(initial_desc, total=100)
+                active_tasks[task_key] = True
+            
+            if task_key in task_progress_bars:
+                epoch_progress = (progress_info.epoch - 1) / 5
+                batch_progress = progress_info.batch / progress_info.total_batches
+                total_progress = (epoch_progress + batch_progress / 5) * 100
+                
+                if progress_info.epoch == 5 and progress_info.batch == progress_info.total_batches:
+                    total_progress = 100.0
+                    finished_tasks.add(task_key)
+                
+                new_desc = f"train {progress_info.model_name} on {progress_info.device} "
+                new_desc += f"for fold {progress_info.fold_idx+1}/5 - Epoch {progress_info.epoch}/{EPOCHS} "
+                new_desc += f"Loss: {progress_info.avg_loss:.4f} "
+                new_desc += f"Val Acc: {progress_info.val_accuracy*100:.2f}%"
+                if progress_info.device.startswith("cuda"):
+                    new_desc = "[bright_cyan]" + new_desc
+                
+                progress.update(
+                    task_progress_bars[task_key], 
+                    completed=total_progress,
+                    description=new_desc
+                )
+                if total_progress >= 100.0:
+                    progress.update(task_progress_bars[task_key], visible=False)
+
+            if len(finished_tasks) == len(futures_map):
+                break
+
+    model_results = defaultdict(list)
+    if pool in ["smartpool.ProcessPool", "smartpool.ThreadPool", "concurrent.futures.ProcessPoolExecutor", "concurrent.futures.ThreadPoolExecutor", "multiprocessing.Pool"]:
+        for task_key, future in futures_map.items():
+            if pool == "multiprocessing.Pool":
+                result:TrainingResult = future.get()
+            else:
+                result:TrainingResult = future.result()
+
+            model_results[result.model_name].append(result.val_accuracy)
+    elif pool.startswith("joblib"):
+        for result in joblib_results:
+            model_results[result.model_name].append(result.val_accuracy)
+    elif pool == "ray":
+        ray_results = ray.get(futures)
+        for result in ray_results:
+            model_results[result.model_name].append(result.val_accuracy)
+    
+    stop_time = time.perf_counter()
+    print(f"train completed in {stop_time - start_time:.2f} seconds")
+
+    print("analysing results...")
+
+    stats = {}
+    for model_name, accuracies in model_results.items():
+        stats[model_name] = {
+            'mean': np.mean(accuracies),
+            'std': np.std(accuracies),
+            'min': np.min(accuracies),
+            'max': np.max(accuracies),
+            'accuracies': accuracies
+        }
+    
+    print_results_table(stats)
+    plot_results(model_results, stats)
+
+
+if __name__ == "__main__":
+    main()

smartpool_examples-0.1.0/smartpool_examples/cross_validation/config.py

@@ -0,0 +1,11 @@
+# Training parameters
+BATCH_SIZE = 128
+EPOCHS = 5
+LEARNING_RATE = 0.001
+
+# Data settings
+import os
+self_folder = os.path.dirname(os.path.abspath(__file__))
+
+DATA_ROOT = f'{self_folder}/data'
+DATASET_NAME = 'MNIST'

smartpool_examples-0.1.0/smartpool_examples/cross_validation/data_utils.py

@@ -0,0 +1,47 @@
+import os
+from torch.utils.data import DataLoader, Subset
+from torchvision import datasets, transforms
+
+from config import DATA_ROOT, BATCH_SIZE
+
+
+def prepare_data():
+    transform = transforms.Compose([
+        transforms.ToTensor(),
+        transforms.Normalize((0.1307,), (0.3081,))
+    ])
+    
+    mnist_exists = (
+        os.path.exists(os.path.join(DATA_ROOT, 'MNIST', 'raw')) and
+        os.path.exists(os.path.join(DATA_ROOT, 'MNIST', 'processed'))
+    )
+    
+    dataset = datasets.MNIST(
+        root=DATA_ROOT, 
+        train=True, 
+        download=not mnist_exists, 
+        transform=transform
+    )
+    dataset.data.share_memory_()
+    dataset.targets.share_memory_()
+    return dataset
+
+
+def create_data_loaders(dataset, train_indices, val_indices):
+    train_subset = Subset(dataset, train_indices)
+    val_subset = Subset(dataset, val_indices)
+    
+    train_loader = DataLoader(
+        train_subset, 
+        batch_size=BATCH_SIZE, 
+        shuffle=True, 
+        pin_memory=True
+    )
+    val_loader = DataLoader(
+        val_subset, 
+        batch_size=BATCH_SIZE, 
+        shuffle=False, 
+        pin_memory=True
+    )
+    
+    return train_loader, val_loader

smartpool_examples-0.1.0/smartpool_examples/cross_validation/model_utils.py

@@ -0,0 +1,173 @@
+import torch
+import torch.nn as nn
+import torch.optim as optim
+
+import traceback
+from dataclasses import dataclass
+
+from config import LEARNING_RATE, EPOCHS
+from data_utils import create_data_loaders
+
+from smartpool import move_optimizer_to, best_device
+
+
+@dataclass
+class TrainingResult:
+    fold_idx:int
+    model_name:str
+    val_accuracy:float
+
+
+@dataclass
+class ProgressInfo:
+    model_name:str
+    fold_idx:int
+    epoch:int
+    batch:int
+    total_batches:int
+    device:str
+    avg_loss:float
+    val_accuracy:float
+
+
+@dataclass
+class ErrorInfo:
+    exception:BaseException
+    traceback:str
+
+def train_single_fold(fold_idx, model_class, train_indices, val_indices, dataset, progress_queue, device=None):
+    try:
+        return _train_single_fold(fold_idx, model_class, train_indices, val_indices, dataset, progress_queue, device)
+    except BaseException as e:
+        error_info = ErrorInfo(e, traceback.format_exc())
+        progress_queue.put(error_info)
+        raise e
+
+def _train_single_fold(fold_idx, model_class, train_indices, val_indices, dataset, progress_queue, user_device):
+    train_loader, val_loader = create_data_loaders(dataset, train_indices, val_indices)
+    num_batches = len(train_loader)
+    model = model_class()
+    
+    device = user_device
+    if user_device is None:
+        device = best_device()
+
+    old_device = device
+    model.to(device, non_blocking=True)
+    
+    criterion = nn.CrossEntropyLoss()
+    optimizer = optim.Adam(model.parameters(), lr=LEARNING_RATE)
+    
+    initial_progress = ProgressInfo(
+        model_name=model_class.__name__,
+        fold_idx=fold_idx,
+        epoch=1,
+        batch=0,
+        total_batches=num_batches,
+        device=device,
+        avg_loss=0.0,
+        val_accuracy=0.0
+    )
+    progress_queue.put(initial_progress)
+
+    last_val_accuracy = 0.0
+    
+    for epoch in range(EPOCHS):
+        epoch_loss = 0.0
+        model.train()
+        for batch_idx, (data, target) in enumerate(train_loader):
+            if user_device is None:
+                device = best_device()
+
+            data = data.to(device, non_blocking=True)
+            target = target.to(device, non_blocking=True)
+                    
+            if device != old_device:
+                model.to(device, non_blocking=True)
+                move_optimizer_to(optimizer, device)
+                old_device = device
+                    
+            optimizer.zero_grad()
+            output = model(data)
+            loss = criterion(output, target)
+            loss.backward()
+            optimizer.step()
+                    
+            epoch_loss += loss.item()
+                    
+            progress_info = ProgressInfo(
+                model_name=model_class.__name__,
+                fold_idx=fold_idx,
+                epoch=epoch + 1,
+                batch=batch_idx + 1,
+                total_batches=num_batches,
+                device=device,
+                avg_loss=epoch_loss / (batch_idx + 1),
+                val_accuracy=last_val_accuracy
+            )
+            progress_queue.put(progress_info)
+        
+        model.eval()
+        correct = 0
+        total = 0
+        val_accuracy = 0.0
+        
+        with torch.no_grad():
+            for data, target in val_loader:
+                if user_device is None:
+                    device = best_device()
+
+                data = data.to(device, non_blocking=True)
+                target = target.to(device, non_blocking=True)
+                        
+                if device != old_device:
+                    model.to(device, non_blocking=True)
+                    move_optimizer_to(optimizer, device)
+                    old_device = device
+                        
+                output = model(data)
+                pred = output.argmax(dim=1, keepdim=True)
+                correct += pred.eq(target.view_as(pred)).sum().item()
+                total += target.size(0)
+        
+        val_accuracy = correct / total
+        last_val_accuracy = val_accuracy
+        model.train()
+        
+        final_progress = ProgressInfo(
+            model_name=model_class.__name__,
+            fold_idx=fold_idx,
+            epoch=epoch + 1,
+            batch=num_batches,
+            total_batches=num_batches,
+            device=device,
+            avg_loss=epoch_loss / num_batches,
+            val_accuracy=val_accuracy
+        )
+        progress_queue.put(final_progress)
+    
+    model.eval()
+    correct = 0
+    total = 0
+    
+    with torch.no_grad():
+        for data, target in val_loader:
+            if user_device is None:
+                device = best_device()
+
+            data = data.to(device, non_blocking=True)
+            target = target.to(device, non_blocking=True)
+
+            if device != old_device:
+                model.to(device, non_blocking=True)
+                move_optimizer_to(optimizer, device)
+                old_device = device
+
+            output = model(data)
+            pred = output.argmax(dim=1, keepdim=True)
+            correct += pred.eq(target.view_as(pred)).sum().item()
+            total += target.size(0)
+    
+    val_accuracy = correct / total
+    
+    return TrainingResult(fold_idx, model_class.__name__, val_accuracy)

smartpool_examples-0.1.0/smartpool_examples/cross_validation/models/LeNet5.py

@@ -0,0 +1,27 @@
+import torch.nn as nn
+
+
+class LeNet5(nn.Module):
+    
+    def __init__(self):
+        super(LeNet5, self).__init__()
+        self.conv = nn.Sequential(
+            nn.Conv2d(1, 6, 5),      # 28x28 -> 24x24
+            nn.ReLU(),
+            nn.MaxPool2d(2),          # 24x24 -> 12x12
+            nn.Conv2d(6, 16, 5),      # 12x12 -> 8x8
+            nn.ReLU(),
+            nn.MaxPool2d(2)           # 8x8 -> 4x4
+        )
+        self.fc = nn.Sequential(
+            nn.Linear(16*4*4, 120),
+            nn.ReLU(),
+            nn.Linear(120, 84),
+            nn.ReLU(),
+            nn.Linear(84, 10)
+        )
+    
+    def forward(self, x):
+        x = self.conv(x)
+        x = x.view(x.size(0), -1)
+        return self.fc(x)

smartpool_examples-0.1.0/smartpool_examples/cross_validation/models/MLP.py

@@ -0,0 +1,21 @@
+import torch.nn as nn
+
+
+class MLP(nn.Module):
+    
+    def __init__(self):
+        super(MLP, self).__init__()
+        self.flatten = nn.Flatten()
+        self.fc = nn.Sequential(
+            nn.Linear(28*28, 512),
+            nn.ReLU(),
+            nn.Dropout(0.2),
+            nn.Linear(512, 256),
+            nn.ReLU(),
+            nn.Dropout(0.2),
+            nn.Linear(256, 10)
+        )
+    
+    def forward(self, x):
+        x = self.flatten(x)
+        return self.fc(x)

smartpool_examples-0.1.0/smartpool_examples/cross_validation/models/ModernCNN.py

@@ -0,0 +1,33 @@
+import torch.nn as nn
+
+
+class ModernCNN(nn.Module):
+    
+    def __init__(self):
+        super(ModernCNN, self).__init__()
+        self.conv = nn.Sequential(
+            nn.Conv2d(1, 32, 3, padding=1),
+            nn.BatchNorm2d(32),
+            nn.ReLU(),
+            nn.Conv2d(32, 64, 3, padding=1),
+            nn.BatchNorm2d(64),
+            nn.ReLU(),
+            nn.MaxPool2d(2),          # 14x14
+            nn.Dropout(0.25),
+            nn.Conv2d(64, 128, 3, padding=1),
+            nn.BatchNorm2d(128),
+            nn.ReLU(),
+            nn.MaxPool2d(2),          # 7x7
+            nn.Dropout(0.25)
+        )
+        self.fc = nn.Sequential(
+            nn.Linear(128*7*7, 256),
+            nn.ReLU(),
+            nn.Dropout(0.5),
+            nn.Linear(256, 10)
+        )
+    
+    def forward(self, x):
+        x = self.conv(x)
+        x = x.view(x.size(0), -1)
+        return self.fc(x)

smartpool_examples-0.1.0/smartpool_examples/cross_validation/models/ResNeXt.py

@@ -0,0 +1,57 @@
+import torch.nn as nn
+
+class ResNeXtBlock(nn.Module):
+    def __init__(self, in_channels, out_channels, cardinality=32, stride=1):
+        super(ResNeXtBlock, self).__init__()
+        self.conv1 = nn.Conv2d(in_channels, out_channels, 1, bias=False)
+        self.bn1 = nn.BatchNorm2d(out_channels)
+        self.conv2 = nn.Conv2d(out_channels, out_channels, 3, stride=stride, padding=1, groups=cardinality, bias=False)
+        self.bn2 = nn.BatchNorm2d(out_channels)
+        self.conv3 = nn.Conv2d(out_channels, out_channels * 2, 1, bias=False)
+        self.bn3 = nn.BatchNorm2d(out_channels * 2)
+        self.relu = nn.ReLU(inplace=True)
+        
+        self.shortcut = nn.Sequential()
+        if stride != 1 or in_channels != out_channels * 2:
+            self.shortcut = nn.Sequential(
+                nn.Conv2d(in_channels, out_channels * 2, 1, stride=stride, bias=False),
+                nn.BatchNorm2d(out_channels * 2)
+            )
+
+    def forward(self, x):
+        identity = x
+        out = self.relu(self.bn1(self.conv1(x)))
+        out = self.relu(self.bn2(self.conv2(out)))
+        out = self.bn3(self.conv3(out))
+        out += self.shortcut(identity)
+        out = self.relu(out)
+        return out
+
+class ResNeXt(nn.Module):
+    def __init__(self):
+        super(ResNeXt, self).__init__()
+        self.conv1 = nn.Conv2d(1, 32, 3, padding=1)
+        self.bn1 = nn.BatchNorm2d(32)
+        self.relu = nn.ReLU(inplace=True)
+        
+        self.layer1 = self._make_layer(32, 64, 2, stride=2)
+        self.layer2 = self._make_layer(128, 128, 2, stride=2)
+        
+        self.avgpool = nn.AdaptiveAvgPool2d((1, 1))
+        self.fc = nn.Linear(256, 10)
+
+    def _make_layer(self, in_channels, out_channels, blocks, stride=1):
+        layers = []
+        layers.append(ResNeXtBlock(in_channels, out_channels, stride=stride))
+        for _ in range(1, blocks):
+            layers.append(ResNeXtBlock(out_channels * 2, out_channels))
+        return nn.Sequential(*layers)
+
+    def forward(self, x):
+        x = self.relu(self.bn1(self.conv1(x)))
+        x = self.layer1(x)
+        x = self.layer2(x)
+        x = self.avgpool(x)
+        x = x.view(x.size(0), -1)
+        x = self.fc(x)
+        return x

smartpool_examples-0.1.0/smartpool_examples/cross_validation/models/ResNeXtV2.py

@@ -0,0 +1,59 @@
+import torch.nn as nn
+
+class PreActResNeXtBlock(nn.Module):
+    def __init__(self, in_channels, out_channels, cardinality=32, stride=1):
+        super(PreActResNeXtBlock, self).__init__()
+        self.bn1 = nn.BatchNorm2d(in_channels)
+        self.relu = nn.ReLU(inplace=True)
+        self.conv1 = nn.Conv2d(in_channels, out_channels, 1, bias=False)
+        self.bn2 = nn.BatchNorm2d(out_channels)
+        self.conv2 = nn.Conv2d(out_channels, out_channels, 3, stride=stride, padding=1, groups=cardinality, bias=False)
+        self.bn3 = nn.BatchNorm2d(out_channels)
+        self.conv3 = nn.Conv2d(out_channels, out_channels * 2, 1, bias=False)
+        
+        self.shortcut = nn.Sequential()
+        if stride != 1 or in_channels != out_channels * 2:
+            self.shortcut = nn.Sequential(
+                nn.Conv2d(in_channels, out_channels * 2, 1, stride=stride, bias=False)
+            )
+
+    def forward(self, x):
+        identity = x
+        out = self.relu(self.bn1(x))
+        out = self.conv1(out)
+        out = self.relu(self.bn2(out))
+        out = self.conv2(out)
+        out = self.relu(self.bn3(out))
+        out = self.conv3(out)
+        out += self.shortcut(identity)
+        return out
+
+class ResNeXtV2(nn.Module):
+    def __init__(self):
+        super(ResNeXtV2, self).__init__()
+        self.conv1 = nn.Conv2d(1, 32, 3, padding=1)
+        
+        self.layer1 = self._make_layer(32, 64, 2, stride=2)
+        self.layer2 = self._make_layer(128, 128, 2, stride=2)
+        
+        self.bn = nn.BatchNorm2d(256)
+        self.relu = nn.ReLU(inplace=True)
+        self.avgpool = nn.AdaptiveAvgPool2d((1, 1))
+        self.fc = nn.Linear(256, 10)
+
+    def _make_layer(self, in_channels, out_channels, blocks, stride=1):
+        layers = []
+        layers.append(PreActResNeXtBlock(in_channels, out_channels, stride=stride))
+        for _ in range(1, blocks):
+            layers.append(PreActResNeXtBlock(out_channels * 2, out_channels))
+        return nn.Sequential(*layers)
+
+    def forward(self, x):
+        x = self.conv1(x)
+        x = self.layer1(x)
+        x = self.layer2(x)
+        x = self.relu(self.bn(x))
+        x = self.avgpool(x)
+        x = x.view(x.size(0), -1)
+        x = self.fc(x)
+        return x

smartpool_examples-0.1.0/smartpool_examples/cross_validation/models/ResNet.py

@@ -0,0 +1,54 @@
+import torch.nn as nn
+
+class ResidualBlock(nn.Module):
+    def __init__(self, in_channels, out_channels, stride=1):
+        super(ResidualBlock, self).__init__()
+        self.conv1 = nn.Conv2d(in_channels, out_channels, 3, stride=stride, padding=1, bias=False)
+        self.bn1 = nn.BatchNorm2d(out_channels)
+        self.relu = nn.ReLU(inplace=True)
+        self.conv2 = nn.Conv2d(out_channels, out_channels, 3, padding=1, bias=False)
+        self.bn2 = nn.BatchNorm2d(out_channels)
+        
+        self.shortcut = nn.Sequential()
+        if stride != 1 or in_channels != out_channels:
+            self.shortcut = nn.Sequential(
+                nn.Conv2d(in_channels, out_channels, 1, stride=stride, bias=False),
+                nn.BatchNorm2d(out_channels)
+            )
+
+    def forward(self, x):
+        identity = x
+        out = self.relu(self.bn1(self.conv1(x)))
+        out = self.bn2(self.conv2(out))
+        out += self.shortcut(identity)
+        out = self.relu(out)
+        return out
+
+class ResNet(nn.Module):
+    def __init__(self):
+        super(ResNet, self).__init__()
+        self.conv1 = nn.Conv2d(1, 32, 3, padding=1)
+        self.bn1 = nn.BatchNorm2d(32)
+        self.relu = nn.ReLU(inplace=True)
+        
+        self.layer1 = self._make_layer(32, 64, 2, stride=2)
+        self.layer2 = self._make_layer(64, 128, 2, stride=2)
+        
+        self.avgpool = nn.AdaptiveAvgPool2d((1, 1))
+        self.fc = nn.Linear(128, 10)
+
+    def _make_layer(self, in_channels, out_channels, blocks, stride=1):
+        layers = []
+        layers.append(ResidualBlock(in_channels, out_channels, stride))
+        for _ in range(1, blocks):
+            layers.append(ResidualBlock(out_channels, out_channels))
+        return nn.Sequential(*layers)
+
+    def forward(self, x):
+        x = self.relu(self.bn1(self.conv1(x)))
+        x = self.layer1(x)
+        x = self.layer2(x)
+        x = self.avgpool(x)
+        x = x.view(x.size(0), -1)
+        x = self.fc(x)
+        return x

smartpool_examples-0.1.0/smartpool_examples/cross_validation/models/ResNetV2.py

@@ -0,0 +1,55 @@
+import torch.nn as nn
+
+class PreActBlock(nn.Module):
+    def __init__(self, in_channels, out_channels, stride=1):
+        super(PreActBlock, self).__init__()
+        self.bn1 = nn.BatchNorm2d(in_channels)
+        self.relu = nn.ReLU(inplace=True)
+        self.conv1 = nn.Conv2d(in_channels, out_channels, 3, stride=stride, padding=1, bias=False)
+        self.bn2 = nn.BatchNorm2d(out_channels)
+        self.conv2 = nn.Conv2d(out_channels, out_channels, 3, padding=1, bias=False)
+        
+        self.shortcut = nn.Sequential()
+        if stride != 1 or in_channels != out_channels:
+            self.shortcut = nn.Sequential(
+                nn.Conv2d(in_channels, out_channels, 1, stride=stride, bias=False)
+            )
+
+    def forward(self, x):
+        identity = x
+        out = self.relu(self.bn1(x))
+        out = self.conv1(out)
+        out = self.relu(self.bn2(out))
+        out = self.conv2(out)
+        out += self.shortcut(identity)
+        return out
+
+class ResNetV2(nn.Module):
+    def __init__(self):
+        super(ResNetV2, self).__init__()
+        self.conv1 = nn.Conv2d(1, 32, 3, padding=1)
+        
+        self.layer1 = self._make_layer(32, 64, 2, stride=2)
+        self.layer2 = self._make_layer(64, 128, 2, stride=2)
+        
+        self.bn = nn.BatchNorm2d(128)
+        self.relu = nn.ReLU(inplace=True)
+        self.avgpool = nn.AdaptiveAvgPool2d((1, 1))
+        self.fc = nn.Linear(128, 10)
+
+    def _make_layer(self, in_channels, out_channels, blocks, stride=1):
+        layers = []
+        layers.append(PreActBlock(in_channels, out_channels, stride))
+        for _ in range(1, blocks):
+            layers.append(PreActBlock(out_channels, out_channels))
+        return nn.Sequential(*layers)
+
+    def forward(self, x):
+        x = self.conv1(x)
+        x = self.layer1(x)
+        x = self.layer2(x)
+        x = self.relu(self.bn(x))
+        x = self.avgpool(x)
+        x = x.view(x.size(0), -1)
+        x = self.fc(x)
+        return x

smartpool_examples-0.1.0/smartpool_examples/cross_validation/models/__init__.py

@@ -0,0 +1,9 @@
+from .LeNet5 import LeNet5
+from .MLP import MLP
+from .ModernCNN import ModernCNN
+from .ResNet import ResNet
+from .ResNetV2 import ResNetV2
+from .ResNeXt import ResNeXt
+from .ResNeXtV2 import ResNeXtV2
+
+__all__ = ['LeNet5', 'MLP', 'ModernCNN', 'ResNet', 'ResNetV2', 'ResNeXt', 'ResNeXtV2']

smartpool_examples-0.1.0/smartpool_examples/cross_validation/visualization.py

@@ -0,0 +1,86 @@
+import matplotlib.pyplot as plt
+import numpy as np
+import rich
+from rich.table import Table
+from typing import Dict, Any
+
+
+def plot_results(model_results:Dict[str, Any], stats:Dict[str, Any]):
+    fig, ax = plt.subplots(figsize=(12, 8))
+    
+    plt.rcParams["font.family"] = ["Microsoft YaHei", "Arial", "sans-serif"]
+    plt.rcParams["axes.unicode_minus"] = False
+    
+    n_models = len(model_results)
+    n_folds = 5
+    x_pos = np.arange(n_models)
+    
+    # 调整柱状图宽度和间距，使柱子整体居中对齐
+    total_width = 0.8  # 总宽度
+    bar_width = total_width / n_folds  # 每个柱子的宽度
+    
+    # 定义不同fold的颜色
+    colors = ['#1f77b4', '#ff7f0e', '#2ca02c', '#d62728', '#9467bd']
+    
+    for fold_idx in range(n_folds):
+        fold_accuracies = []
+        for module_result in model_results.values():
+            if fold_idx < len(module_result):
+                fold_accuracies.append(module_result[fold_idx])
+            else:
+                fold_accuracies.append(0)
+        
+        # 计算每个柱子的位置，使其整体居中
+        offset = (fold_idx - n_folds/2 + 0.5) * bar_width
+        ax.bar(x_pos + offset, fold_accuracies, 
+               bar_width, alpha=0.7, 
+               color=colors[fold_idx],
+               label=f'Fold {fold_idx+1}')
+    
+    means = [stat['mean'] for stat in stats.values()]
+    stds = [stat['std'] for stat in stats.values()]
+    
+    ax.errorbar(x_pos, means, yerr=stds, fmt='o', color='black', 
+                capsize=5, capthick=2, elinewidth=2, markersize=8,
+                label='Mean ± Std Dev')
+    
+    for i, (mean, std) in enumerate(zip(means, stds)):
+        ax.annotate(f'{mean:.4f}\n±{std:.4f}', 
+                   xy=(x_pos[i], mean), 
+                   xytext=(0, 10),
+                   textcoords='offset points',
+                   ha='center', va='bottom',
+                   fontsize=9)
+    
+    ax.set_xlabel('Model Type', fontsize=12)
+    ax.set_ylabel('Validation Accuracy', fontsize=12)
+    ax.set_title('Handwritten Digit Recognition Models: 5-Fold Cross-Validation Comparison', fontsize=14, pad=20)
+    ax.set_xticks(x_pos)
+    ax.set_xticklabels(model_results.keys())
+    ax.legend(loc='lower right')
+    ax.grid(True, alpha=0.3)
+    ax.set_ylim(0.8, 1.0)
+    
+    plt.tight_layout()
+    plt.savefig('cross_validation_results.png', dpi=300, bbox_inches='tight')
+    plt.show()
+
+def print_results_table(stats:Dict[str, Any]):
+    table = Table(title="5-Fold Cross-Validation Results Summary")
+    table.add_column("Model", style="cyan")
+    table.add_column("Mean Accuracy", style="magenta")
+    table.add_column("Std Deviation", style="green")
+    table.add_column("Min", style="yellow")
+    table.add_column("Max", style="blue")
+    
+    for model_name, stat in stats.items():
+        stat = stats[model_name]
+        table.add_row(
+            model_name,
+            f"{stat['mean']:.4f}",
+            f"{stat['std']:.4f}",
+            f"{stat['min']:.4f}",
+            f"{stat['max']:.4f}"
+        )
+    
+    rich.print(table)

smartpool_examples-0.1.0/smartpool_examples.egg-info/PKG-INFO

@@ -0,0 +1,84 @@
+Metadata-Version: 2.4
+Name: smartpool-examples
+Version: 0.1.0
+Summary: Examples for smartpool.
+Author-email: "王炳辉 (Bing-Hui WANG)" <binghui.wang@foxmail.com>
+License: MIT
+Project-URL: Homepage, https://github.com/Time-Coder/smartpool
+Project-URL: Repository, https://github.com/Time-Coder/smartpool.git
+Classifier: Programming Language :: Python :: 3
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: pysmartpool
+Requires-Dist: matplotlib
+Requires-Dist: scikit-learn
+Requires-Dist: numpy
+Requires-Dist: rich
+Requires-Dist: click
+Requires-Dist: joblib
+Requires-Dist: datawatcher
+Dynamic: license-file
+
+# SmartPool Examples
+
+This package contains practical examples demonstrating the capabilities of SmartPool for various computational tasks.
+
+## Examples Overview
+
+### 1. Prime Number Counting (`count_prime`)
+
+Count the number of prime numbers below 10000 using smartpool.ProcessPool.
+Demonstrates basic usage of smartpool.ProcessPool.
+
+#### Running the Example
+
+```bash
+python -m smartpool_examples.count_prime
+```
+
+### 2. Cross-Validation for Deep Learning models (`cross_validation`)
+
+Demonstrates SmartPool's capabilities for machine learning workloads with GPU resource management.
+
+#### Running the Example
+
+```bash
+# Using ProcessPool  
+python -m smartpool_examples.cross_validation --pool smartpool.ProcessPool
+
+# Using ThreadPool
+python -m smartpool_examples.cross_validation --pool smartpool.ThreadPool
+
+# Using multiprocessing.Pool
+python -m smartpool_examples.cross_validation --pool multiprocessing.Pool
+
+# Using concurrent.futures.ProcessPoolExecutor
+python -m smartpool_examples.cross_validation --pool concurrent.futures.ProcessPoolExecutor
+
+# Using concurrent.futures.ThreadPoolExecutor
+python -m smartpool_examples.cross_validation --pool concurrent.futures.ThreadPoolExecutor
+
+# Using joblib.Parallel(backend='loky')
+python -m smartpool_examples.cross_validation --pool joblib.Parallel(backend='loky')
+
+# Using joblib.Parallel(backend='threading')
+python -m smartpool_examples.cross_validation --pool joblib.Parallel(backend='threading')
+
+# Using Ray
+python -m smartpool_examples.cross_validation --pool ray
+```
+
+#### What it Demonstrates
+
+- GPU memory management and core allocation
+- Automatic device selection (CPU vs GPU)
+- Cross-validation pipeline parallelization
+- Resource monitoring during training
+- Performance comparison with external frameworks
+
+
+## License
+
+MIT License - see main smartpool repository for details

smartpool_examples-0.1.0/smartpool_examples.egg-info/SOURCES.txt

@@ -0,0 +1,26 @@
+LICENSE
+README.md
+pyproject.toml
+smartpool_examples/__init__.py
+smartpool_examples.egg-info/PKG-INFO
+smartpool_examples.egg-info/SOURCES.txt
+smartpool_examples.egg-info/dependency_links.txt
+smartpool_examples.egg-info/requires.txt
+smartpool_examples.egg-info/top_level.txt
+smartpool_examples/count_prime/__init__.py
+smartpool_examples/count_prime/__main__.py
+smartpool_examples/count_prime/count_prime.py
+smartpool_examples/cross_validation/__init__.py
+smartpool_examples/cross_validation/__main__.py
+smartpool_examples/cross_validation/config.py
+smartpool_examples/cross_validation/data_utils.py
+smartpool_examples/cross_validation/model_utils.py
+smartpool_examples/cross_validation/visualization.py
+smartpool_examples/cross_validation/models/LeNet5.py
+smartpool_examples/cross_validation/models/MLP.py
+smartpool_examples/cross_validation/models/ModernCNN.py
+smartpool_examples/cross_validation/models/ResNeXt.py
+smartpool_examples/cross_validation/models/ResNeXtV2.py
+smartpool_examples/cross_validation/models/ResNet.py
+smartpool_examples/cross_validation/models/ResNetV2.py
+smartpool_examples/cross_validation/models/__init__.py

smartpool_examples-0.1.0/smartpool_examples.egg-info/dependency_links.txt

@@ -0,0 +1 @@
+

smartpool_examples-0.1.0/smartpool_examples.egg-info/requires.txt

@@ -0,0 +1,8 @@
+pysmartpool
+matplotlib
+scikit-learn
+numpy
+rich
+click
+joblib
+datawatcher

smartpool_examples-0.1.0/smartpool_examples.egg-info/top_level.txt

@@ -0,0 +1 @@
+smartpool_examples