silver-run 0.1.0__tar.gz

silver_run-0.1.0/CHANGELOG.md

@@ -0,0 +1,33 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [0.1.0] - 2024-08-04
+
+### Added
+- Initial release of silver-run
+- Backend-neutral training lifecycle management
+- Event logging and inspection for training runs
+- Checkpoint management with pluggable storage backends
+- Async/await support for modern Python training workflows
+- In-memory checkpoint store for local development
+- Full training state machine (created, running, paused, stopped, cancelled, completed, failed)
+- Comprehensive test suite with async test coverage
+- Support for Python 3.8-3.12
+
+### Features
+- `TrainingRun` - Main class for managing training lifecycle
+- `TrainingBackend` - Abstract interface for training implementations
+- `CheckpointStore` - Pluggable checkpoint storage interface
+- `MemoryCheckpointStore` - In-memory checkpoint implementation
+- `TrainingContext` - Context object passed to training backends
+- State management (start, pause, resume, stop, cancel, complete, fail)
+- Event emission and logging with timestamps
+- Checkpoint creation and retrieval
+- Backend execution with automatic state management
+- Pause/resume functionality for long-running training
+
+## [Unreleased]

silver_run-0.1.0/CONTRIBUTING.md

@@ -0,0 +1,115 @@
+# Contributing to silver-run
+
+Thank you for your interest in contributing to silver-run! This document provides guidelines and instructions for contributing to the project.
+
+## Development Setup
+
+### Prerequisites
+- Python 3.8 or higher
+- Git
+- Virtual environment (recommended)
+
+### Setting Up Development Environment
+
+1. **Clone the repository**
+   ```bash
+   git clone https://github.com/adfgdartec/silver-run.git
+   cd silver-run
+   ```
+
+2. **Create a virtual environment**
+   ```bash
+   python -m venv venv
+   source venv/bin/activate  # On Windows: venv\Scripts\activate
+   ```
+
+3. **Install development dependencies**
+   ```bash
+   pip install -e ".[dev]"
+   ```
+
+4. **Run tests**
+   ```bash
+   pytest
+   ```
+
+5. **Run tests with coverage**
+   ```bash
+   pytest --cov=silver_run --cov-report=html
+   ```
+
+## Code Style
+
+We use the following tools to maintain code quality:
+
+- **flake8** for linting
+- **mypy** for type checking
+- **pytest** for testing
+
+Run all quality checks:
+```bash
+flake8 src/ tests/
+mypy src/
+pytest
+```
+
+## Making Changes
+
+1. **Create a branch**
+   ```bash
+   git checkout -b feature/your-feature-name
+   ```
+
+2. **Make your changes**
+   - Write clear, descriptive commit messages
+   - Add tests for new functionality
+   - Update documentation as needed
+
+3. **Run tests**
+   ```bash
+   pytest
+   ```
+
+4. **Submit a pull request**
+   - Describe your changes clearly
+   - Reference any related issues
+   - Ensure all tests pass
+
+## Testing
+
+We aim for high test coverage. When adding new features:
+
+- Write unit tests for new functions
+- Test async functions properly
+- Test edge cases and error conditions
+- Ensure existing tests still pass
+
+### Test Structure
+```
+tests/
+├── __init__.py
+└── test_run.py
+```
+
+## Documentation
+
+- Update docstrings for any modified functions
+- Add examples for new features
+- Update README.md if user-facing changes are made
+
+## Release Process
+
+Releases are managed by maintainers:
+
+1. Update version in `pyproject.toml`
+2. Update `CHANGELOG.md`
+3. Create a GitHub release
+4. Package will be automatically published to PyPI
+
+## Questions?
+
+Feel free to open an issue for questions or discussions about contributions.
+
+## License
+
+By contributing, you agree that your contributions will be licensed under the Apache-2.0 License.

silver_run-0.1.0/LICENSE

@@ -0,0 +1,12 @@
+Apache License
+Version 2.0, January 2004
+
+Copyright 2026 Silver Contributors
+
+Licensed under the Apache License, Version 2.0. You may obtain a copy of the
+License at https://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software distributed
+under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR
+CONDITIONS OF ANY KIND, either express or implied. See the License for the
+specific language governing permissions and limitations under the License.

silver_run-0.1.0/MANIFEST.in

@@ -0,0 +1,6 @@
+include README.md
+include LICENSE
+include CHANGELOG.md
+include CONTRIBUTING.md
+recursive-include src *.py
+recursive-include tests *.py

silver_run-0.1.0/PKG-INFO

@@ -0,0 +1,319 @@
+Metadata-Version: 2.4
+Name: silver-run
+Version: 0.1.0
+Summary: Backend-neutral ML run lifecycle, events, and checkpoints for Silver.
+License-Expression: Apache-2.0
+Project-URL: Homepage, https://github.com/adfgdartec/silver-run
+Project-URL: Repository, https://github.com/adfgdartec/silver-run
+Project-URL: Issues, https://github.com/adfgdartec/silver-run/issues
+Keywords: machine-learning,training,checkpoint,experiments,python
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.8
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.8
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Provides-Extra: dev
+Requires-Dist: pytest>=7.0.0; extra == "dev"
+Requires-Dist: pytest-asyncio>=0.21.0; extra == "dev"
+Requires-Dist: pytest-cov>=4.0.0; extra == "dev"
+Requires-Dist: flake8>=6.0.0; extra == "dev"
+Requires-Dist: mypy>=1.0.0; extra == "dev"
+Requires-Dist: build>=0.10.0; extra == "dev"
+Requires-Dist: twine>=4.0.0; extra == "dev"
+Dynamic: license-file
+
+# silver-run
+
+[![Python Version](https://img.shields.io/badge/python-3.8%2B-blue.svg)](https://www.python.org/downloads/)
+[![License](https://img.shields.io/badge/license-Apache%202.0-green.svg)](LICENSE)
+[![Tests](https://img.shields.io/badge/tests-passing-brightgreen.svg)](tests/)
+[![Code Style](https://img.shields.io/badge/code%20style-flake8-blue.svg)](https://flake8.pycqa.org/)
+
+Backend-neutral ML run lifecycle, events, and checkpoints for Silver. A Python package designed for ML researchers who need flexible training orchestration across different frameworks.
+
+## Installation
+
+```bash
+pip install silver-run
+```
+
+## Quick Start
+
+```python
+from silver_run import TrainingRun, TrainingBackend, TrainingContext
+import asyncio
+
+class MyBackend(TrainingBackend):
+    async def run(self, context: TrainingContext):
+        for epoch in range(10):
+            if context.should_stop():
+                break
+            # Your training logic here
+            context.emit({
+                "kind": "epoch",
+                "epoch": epoch,
+                "metrics": {"loss": 0.5 - epoch * 0.05}
+            })
+            await asyncio.sleep(0.1)
+
+async def main():
+    run = TrainingRun()
+    backend = MyBackend()
+    final_state = await run.execute(backend)
+    print(f"Run finished with state: {final_state.value}")
+
+asyncio.run(main())
+```
+
+## Features
+
+- **Training Lifecycle Management**: Full state machine (created, running, paused, stopped, cancelled, completed, failed)
+- **Event Logging**: Comprehensive event tracking with timestamps for training observability
+- **Checkpoint Management**: Pluggable storage backends for model checkpointing
+- **Backend-Agnostic**: Works with PyTorch, TensorFlow, JAX, or any custom training framework
+- **Async/Await Support**: Modern Python async patterns for concurrent training
+- **Pause/Resume**: Control long-running training jobs with pause and resume functionality
+- **Type Safety**: Full type hints for better IDE support and fewer bugs
+
+## Use Cases
+
+### PyTorch Training Integration
+
+```python
+from silver_run import TrainingRun, TrainingBackend, TrainingContext
+import torch
+import asyncio
+
+class PyTorchBackend(TrainingBackend):
+    def __init__(self, model, optimizer, train_loader):
+        self.model = model
+        self.optimizer = optimizer
+        self.train_loader = train_loader
+    
+    async def run(self, context: TrainingContext):
+        for epoch in range(10):
+            if context.should_stop():
+                break
+            
+            self.model.train()
+            total_loss = 0
+            
+            for batch_idx, (data, target) in enumerate(self.train_loader):
+                self.optimizer.zero_grad()
+                output = self.model(data)
+                loss = torch.nn.functional.cross_entropy(output, target)
+                loss.backward()
+                self.optimizer.step()
+                total_loss += loss.item()
+            
+            # Emit epoch completion event
+            context.emit({
+                "kind": "epoch",
+                "epoch": epoch,
+                "metrics": {"loss": total_loss / len(self.train_loader)}
+            })
+            
+            # Checkpoint every 5 epochs
+            if epoch % 5 == 0:
+                await context.checkpoint({
+                    "epoch": epoch,
+                    "model_state_dict": self.model.state_dict(),
+                    "optimizer_state_dict": self.optimizer.state_dict()
+                })
+
+async def main():
+    model = torch.nn.Linear(10, 2)
+    optimizer = torch.optim.Adam(model.parameters())
+    train_loader = [...]  # Your data loader
+    
+    run = TrainingRun()
+    backend = PyTorchBackend(model, optimizer, train_loader)
+    final_state = await run.execute(backend)
+    
+    # Review events
+    for event in run.events():
+        print(f"{event.kind}: {event.data}")
+
+asyncio.run(main())
+```
+
+### Training with Pause/Resume
+
+```python
+from silver_run import TrainingRun, TrainingBackend
+import asyncio
+
+class LongRunningBackend(TrainingBackend):
+    async def run(self, context: TrainingContext):
+        for step in range(1000):
+            if context.should_stop():
+                break
+            
+            # Simulate training step
+            await asyncio.sleep(0.01)
+            
+            # Emit progress
+            if step % 100 == 0:
+                context.emit({
+                    "kind": "progress",
+                    "step": step,
+                    "total": 1000
+                })
+
+async def main():
+    run = TrainingRun()
+    backend = LongRunningBackend()
+    
+    # Start training in background
+    training_task = asyncio.create_task(run.execute(backend))
+    
+    # Pause after some time
+    await asyncio.sleep(0.5)
+    run.pause()
+    print("Training paused")
+    
+    # Resume after some time
+    await asyncio.sleep(0.5)
+    run.resume()
+    print("Training resumed")
+    
+    # Wait for completion
+    final_state = await training_task
+    print(f"Training finished: {final_state.value}")
+
+asyncio.run(main())
+```
+
+### Custom Checkpoint Storage
+
+```python
+from silver_run import TrainingRun, CheckpointStore, Checkpoint
+import asyncio
+
+class S3CheckpointStore(CheckpointStore):
+    def __init__(self, bucket, prefix):
+        self.bucket = bucket
+        self.prefix = prefix
+        self.checkpoints = {}
+    
+    async def save(self, checkpoint: Checkpoint):
+        # Save to S3
+        key = f"{self.prefix}/{checkpoint.id}"
+        print(f"Saving checkpoint to S3: {key}")
+        self.checkpoints[checkpoint.id] = checkpoint
+    
+    async def latest(self):
+        if not self.checkpoints:
+            return None
+        return list(self.checkpoints.values())[-1]
+    
+    async def get(self, id: str):
+        return self.checkpoints.get(id)
+
+async def main():
+    store = S3CheckpointStore("my-bucket", "checkpoints")
+    run = TrainingRun(options=TrainingRunOptions(checkpoint_store=store))
+    
+    # Use custom checkpoint store
+    await run.checkpoint({"model": "state"}, "checkpoint-1")
+    latest = await run.latest_checkpoint()
+    print(f"Latest checkpoint: {latest.id}")
+
+asyncio.run(main())
+```
+
+## Advanced Usage
+
+### Event Filtering and Analysis
+
+```python
+from silver_run import TrainingRun
+
+# Filter events by type
+def get_epoch_events(run):
+    return [e for e in run.events() if e.kind == "epoch"]
+
+def get_error_events(run):
+    return [e for e in run.events() if e.kind == "error"]
+
+# Analyze training progression
+def analyze_training(run):
+    epoch_events = get_epoch_events(run)
+    losses = [e.data.get("metrics", {}).get("loss") for e in epoch_events]
+    
+    if losses:
+        print(f"Initial loss: {losses[0]}")
+        print(f"Final loss: {losses[-1]}")
+        print(f"Loss reduction: {losses[0] - losses[-1]}")
+```
+
+### Multi-Run Experiments
+
+```python
+from silver_run import TrainingRun
+import asyncio
+
+async def run_experiment(config):
+    run = TrainingRun()
+    backend = MyBackend(config)
+    return await run.execute(backend)
+
+async def main():
+    configs = [
+        {"learning_rate": 0.001},
+        {"learning_rate": 0.01},
+        {"learning_rate": 0.1}
+    ]
+    
+    results = await asyncio.gather(*[
+        run_experiment(config) for config in configs
+    ])
+    
+    for config, result in zip(configs, results):
+        print(f"LR {config['learning_rate']}: {result.value}")
+
+asyncio.run(main())
+```
+
+## Requirements
+
+- Python 3.8+
+
+## Development
+
+```bash
+# Install development dependencies
+pip install -e ".[dev]"
+
+# Run tests
+pytest
+
+# Run tests with coverage
+pytest --cov=silver_run --cov-report=html
+
+# Run linting
+flake8 src/ tests/
+mypy src/
+```
+
+## Contributing
+
+Contributions are welcome! Please see [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.
+
+## License
+
+Apache-2.0 - see [LICENSE](LICENSE) file for details.
+
+## Related Packages
+
+- [silver-data](https://github.com/adfgdartec/silver-data) - Dataset handling
+- [silver-diagnostics](https://github.com/adfgdartec/silver-diagnostics) - ML diagnostics
+- [silver-adapters](https://github.com/adfgdartec/silver-adapters) - Framework adapters