flexlock 0.8.2__py3-none-any.whl

flexlock-0.8.2.dist-info/METADATA

@@ -0,0 +1,357 @@
+Metadata-Version: 2.4
+Name: flexlock
+Version: 0.8.2
+Summary: A lightweight and painless SciOps library
+Author-email: Quentin Febvre <quentin.febvre@gmail.com>
+License-Expression: MIT
+Requires-Python: >=3.11
+Requires-Dist: cloudpickle<4,>=3.0.0
+Requires-Dist: gitpython<4,>=3.1.45
+Requires-Dist: joblib<2,>=1.5.2
+Requires-Dist: loguru<0.8,>=0.7.3
+Requires-Dist: omegaconf<3,>=2.3.0
+Requires-Dist: pandas<3,>=2.3.2
+Requires-Dist: xxhash<4,>=3.5.0
+Description-Content-Type: text/markdown
+
+# FlexLock
+
+**Reproducible, Composable, and Scalable Computational Experiments**
+
+[![Documentation](https://img.shields.io/badge/docs-latest-blue.svg)](https://quentinf00.github.io/flexlock/)
+[![PyPI version](https://badge.fury.io/py/flexlock.svg)](https://badge.fury.io/py/flexlock)
+[![License](https://img.shields.io/badge/License-MIT-yellow.svg)](https://opensource.org/licenses/MIT)
+
+FlexLock is a Python library that makes your computational experiments reproducible, composable, and efficient. It automatically tracks everything (code, config, data), detects when you're rerunning identical experiments (and skips them), and helps you build multi-stage pipelines that just work.
+
+## Why FlexLock?
+
+### The Problem
+
+You're running ML experiments and:
+- **Lost track**: "Which hyperparameters gave me that 95% accuracy?"
+- **Can't reproduce**: "This worked last week, what changed?"
+- **Wasting compute**: Running the same preprocessing 10 times
+- **Pipeline chaos**: Multi-stage workflows that break constantly
+
+### The FlexLock Solution
+
+```python
+from flexlock import flexcli
+
+@flexcli
+def train(lr=0.01, epochs=10):
+    # ... your training code ...
+    return {"accuracy": accuracy}
+```
+
+That's it! You now have:
+- ✅ **Automatic tracking**: Code, config, and data snapshots
+- ✅ **Smart caching**: Never rerun identical experiments
+- ✅ **CLI for free**: `python train.py -o lr=0.1`
+- ✅ **Parameter sweeps**: `--sweep "0.001,0.01,0.1" --n_jobs 3`
+- ✅ **Perfect reproducibility**: `flexlock diff` shows exactly what changed
+
+## Quick Start
+
+### Install
+
+```bash
+pip install flexlock
+```
+
+### Example 1: Simple Script
+
+```python
+# train.py
+from flexlock import flexcli
+
+@flexcli
+def train(lr: float = 0.01, epochs: int = 10):
+    """Train a model."""
+    print(f"Training with lr={lr} for {epochs} epochs")
+    # ... training code ...
+    return {"accuracy": 0.95}
+
+if __name__ == "__main__":
+    train()
+```
+
+Run it:
+
+```bash
+# Default run
+$ python train.py
+Training with lr=0.01 for 10 epochs
+✓ Results saved to: results/train/run_0001/
+
+# Override parameters
+$ python train.py -o lr=0.1 epochs=20
+
+# Parameter sweep (runs 3 experiments in parallel)
+$ python train.py --sweep "0.001,0.01,0.1" --sweep-target lr --n_jobs 3
+```
+
+### Example 2: Multi-Stage Pipeline
+
+```python
+# pipeline.py
+from flexlock.api import Project
+
+# Load configurations
+proj = Project(defaults='configs.defaults')
+
+# Stage 1: Preprocess
+preprocess_result = proj.submit(proj.get('preprocess'))
+
+# Stage 2: Train (uses preprocess output)
+train_cfg = proj.get('train')
+train_cfg.data_dir = preprocess_result.output_dir
+train_result = proj.submit(train_cfg)
+
+# Stage 3: Evaluate
+eval_cfg = proj.get('evaluate')
+eval_cfg.model_dir = train_result.save_dir
+eval_result = proj.submit(eval_cfg)
+
+print(f"Final accuracy: {eval_result.accuracy}")
+```
+
+Run it again? FlexLock automatically skips unchanged stages:
+
+```bash
+$ python pipeline.py
+⚡ Cache Hit! Preprocess already done
+⚡ Cache Hit! Training already done
+⚡ Cache Hit! Evaluation already done
+Final accuracy: 0.95
+```
+
+## Key Features
+
+### 🔒 Automatic Reproducibility
+
+Every run creates a snapshot with:
+- **Git tree hash**: Exact code state (not just commit)
+- **Full configuration**: All parameters used
+- **Data fingerprints**: Input data hashes
+- **Environment**: System info
+
+Compare any two runs:
+
+```bash
+$ flexlock diff run_0001 run_0002
+Differences found:
+  Config:
+    optimizer.lr: 0.001 → 0.01
+  Code:
+    Git tree: abc123 → def456 (model.py modified)
+```
+
+### ⚡ Smart Run Detection
+
+FlexLock never wastes computation. It automatically detects when you're rerunning an experiment with identical inputs and reuses the cached result:
+
+```python
+# First run: executes
+result1 = proj.submit(config)
+
+# Second run with same config: instant cache hit!
+result2 = proj.submit(config)  # ⚡ Skipped!
+```
+
+This works even:
+- Across different machines (with shared storage)
+- Weeks later
+- With different result directories (paths are normalized)
+
+### 🔄 Composable Pipelines
+
+Build complex workflows from simple stages:
+
+```python
+# Each stage is a function
+def preprocess(input_data, output_dir): ...
+def train(data_dir, model_type, lr): ...
+def evaluate(model_dir, test_data): ...
+
+# Compose them into a pipeline
+proj = Project(defaults='configs.defaults')
+
+prep = proj.submit(proj.get('preprocess'))
+train_cfg = proj.get('train')
+train_cfg.data_dir = prep.output_dir
+model = proj.submit(train_cfg)
+
+eval_cfg = proj.get('evaluate')
+eval_cfg.model_dir = model.save_dir
+results = proj.submit(eval_cfg)
+```
+
+Failed at train? Fix it and rerun - FlexLock skips the successful preprocess stage.
+
+### 🚀 Parallel Execution
+
+Run sweeps in parallel automatically:
+
+```python
+# Define sweep
+sweep = [
+    {"lr": 0.001, "batch_size": 32},
+    {"lr": 0.01, "batch_size": 32},
+    {"lr": 0.1, "batch_size": 64}
+]
+
+# Run in parallel (local multiprocessing)
+results = proj.submit(config, sweep=sweep, n_jobs=4)
+
+# Or on a cluster (Slurm, PBS)
+results = proj.submit(config, sweep=sweep, slurm_config="slurm.yaml")
+```
+
+### 📊 Configuration Management
+
+Three ways to configure, all type-safe:
+
+```python
+# Method 1: @flexcli (simplest)
+@flexcli
+def train(lr=0.01, epochs=10): ...
+
+# Method 2: Python configs (most powerful)
+from flexlock import py2cfg
+config = py2cfg(
+    train,
+    model=py2cfg(Transformer, layers=12, hidden_size=768),
+    optimizer=py2cfg(Adam, lr=0.001),
+    epochs=100
+)
+
+# Method 3: YAML (declarative)
+# config.yaml
+train:
+  model:
+    layers: 12
+    hidden_size: 768
+  optimizer:
+    type: adam
+    lr: 0.001
+  epochs: 100
+```
+
+## Philosophy
+
+FlexLock is built on three principles:
+
+1. **Configuration is Code**: Use Python's full expressiveness or YAML's simplicity
+2. **Automatic Reproducibility**: Every run is tracked, no manual logging needed
+3. **Smart Execution**: Never rerun identical experiments
+
+Read more in the [Philosophy & Design](https://quentinf00.github.io/flexlock/philosophy.html) docs.
+
+## Documentation
+
+- **[Quickstart Guide](https://quentinf00.github.io/flexlock/quickstart.html)**: Get started in 5 minutes
+- **[Tutorials](https://quentinf00.github.io/flexlock/tutorials/)**: Learn by example
+  - [01. Basics](https://quentinf00.github.io/flexlock/tutorials/01_basics.html): `@flexcli`, CLI args, sweeps
+  - [02. Reproducibility](https://quentinf00.github.io/flexlock/tutorials/02_reproducibility.html): Snapshots, diffs
+  - [03. YAML Configs](https://quentinf00.github.io/flexlock/tutorials/03_yaml_config.html): Multi-stage YAML
+  - [04. Python Configs](https://quentinf00.github.io/flexlock/tutorials/04_python_config.html): `py2cfg`, nested objects
+  - [05. Pipelines](https://quentinf00.github.io/flexlock/tutorials/05_pipeline.html): Multi-stage workflows
+- **[User Guide](https://quentinf00.github.io/flexlock/getting_started.html)**: In-depth feature docs
+- **[API Reference](https://quentinf00.github.io/flexlock/api.html)**: Complete API documentation
+
+## Installation
+
+### From PyPI
+
+```bash
+pip install flexlock
+```
+
+### From Conda
+
+```bash
+conda install -c quentinf00 flexlock
+```
+
+### From Source
+
+```bash
+git clone https://github.com/quentinf00/flexlock.git
+cd flexlock
+pip install -e .
+```
+
+## Requirements
+
+- Python 3.8+
+- OmegaConf
+- Loguru
+- PyYAML
+
+## Use Cases
+
+### ✅ Perfect For:
+
+- **ML Research**: Hyperparameter tuning, ablation studies, architecture search
+- **Data Science**: Exploratory analysis with automatic tracking
+- **Scientific Computing**: Parameter sweeps, reproducible experiments
+- **Data Pipelines**: Multi-stage ETL with dependency tracking
+- **AutoML**: Large-scale hyperparameter optimization
+
+### ⚠️ Less Suitable For:
+
+- **Production APIs**: Use Flask/FastAPI for serving models
+- **Real-time Systems**: FlexLock is for offline experiments
+- **Simple One-Off Scripts**: Adds overhead for single-run scripts
+
+## Comparison with Other Tools
+
+| Feature | FlexLock | Hydra | MLflow | DVC |
+|---------|----------|-------|--------|-----|
+| **Configuration** | Python + YAML | YAML | Code | YAML |
+| **Reproducibility** | Automatic | Manual | Partial | Partial |
+| **Smart Caching** | ✅ | ❌ | ❌ | ✅ |
+| **Multi-stage Pipelines** | ✅ | ❌ | ❌ | ✅ |
+| **Parallel Sweeps** | ✅ | ✅ | ❌ | ❌ |
+| **Learning Curve** | Low | Medium | Low | High |
+
+## Contributing
+
+We welcome contributions! See [CONTRIBUTING.md](CONTRIBUTING.md) for guidelines.
+
+## License
+
+MIT License. See [LICENSE](LICENSE) for details.
+
+## Citation
+
+If you use FlexLock in your research, please cite:
+
+```bibtex
+@software{flexlock2024,
+  title = {FlexLock: Reproducible, Composable, and Scalable Computational Experiments},
+  author = {FlexLock Team},
+  year = {2024},
+  url = {https://github.com/quentinf00/flexlock}
+}
+```
+
+## Acknowledgments
+
+FlexLock builds on ideas from:
+- **Hydra**: Configuration composition patterns
+- **MLflow**: Experiment tracking concepts
+- **DVC**: Pipeline dependency tracking
+- **Sacred**: Experiment observation patterns
+
+## Support
+
+- **Documentation**: [https://quentinf00.github.io/flexlock/](https://quentinf00.github.io/flexlock/)
+- **GitHub Issues**: [Report bugs](https://github.com/quentinf00/flexlock/issues)
+- **Discussions**: [Ask questions](https://github.com/quentinf00/flexlock/discussions)
+
+---
+
+Made with ❤️ by the FlexLock team

flexlock-0.8.2.dist-info/RECORD

@@ -0,0 +1,45 @@
+flexlock/__init__.py,sha256=9c_D3abgUdb7xrXsreDfh9NJ00UHOf7qjszSdp3AwoQ,2226
+flexlock/api.py,sha256=b-utO_O6Vwd4EzeVFtn9d_JVYl7WjluCWlgbyD4JIME,59957
+flexlock/cli.py,sha256=Qrq2o6AipAnGd5W8weq0o8h3IgY2kPmXYkJiV7wqPb4,32403
+flexlock/config.py,sha256=1hJCLJZHdNC6RrJKqhdb320yOMEFdUiXkvLMlwLolz0,4006
+flexlock/context.py,sha256=WzFhGFI-zMUMTdGWfi3i3yAN_iyA2SbxuYZfZ_VCwbw,280
+flexlock/data_hash.py,sha256=MOkt2KUgCzGTihZVZYOKuCNMmK9qiHOPPtzHLfyERKk,10433
+flexlock/debug.py,sha256=y4zxz75_ntfhvT0F7vEoH7NhnEaymu9Fm9Ie5Ccs1CI,14858
+flexlock/diff.py,sha256=CXK7zNPHSu6uoPQ5jlKYpxsCzWe537XSTxaor7kBt6k,9554
+flexlock/diff_cli.py,sha256=LxSR7Y6nO816TYlZ1GU52BU5WXJ70WFgjLvd73snucE,5750
+flexlock/exceptions.py,sha256=9ADrXSGPqyMkDdDfEt-L5fTTCNZePDn3FbUM8xxiQc4,1065
+flexlock/export.py,sha256=hEcxgCTySnfQHSysiWHusDFvTz3ondUJQRN6jK2mRcY,3956
+flexlock/fingerprint.py,sha256=PUsP1aN-ZHVD5wJl7yDbXWABDS0gNTLLa03t7a0PzAg,4538
+flexlock/flexcli.py,sha256=FggMh-yiD8s3sxdG4po5yhxtDxLzhSI0Tm6v8m9P618,5204
+flexlock/freeze.py,sha256=QdE3XlIbNV0tN8mnvyfSwz8iW1Pt7BixemM6ijWGxhU,11241
+flexlock/git_utils.py,sha256=N71kQjVCloSGGVhwpGHpCCj36DnQSzVi1k1hs3VJEtI,6045
+flexlock/index.py,sha256=XqLQoDHbkp1RvyLesuaH8OraQl5YHlrW_RzpcSfxfVQ,9651
+flexlock/load_stage.py,sha256=GrHGzPAFen13zTdmYdjI2ggrXFH0tvsVdR24ifoA8_E,2017
+flexlock/mlflow.py,sha256=RwRIw9KYMOkH7sO42CMIkfrTgELp2IvdzSuGuguhAAI,7680
+flexlock/parallel.py,sha256=NQrfHf3pNY-62cyfRjjYYAKYvU_zPS630kq3GzHI29A,16662
+flexlock/query.py,sha256=j-Q89CyQz0HYua_QzRFb-5rA85Y1hb6o7Sq_A_aVDB4,26683
+flexlock/report.py,sha256=bLcy2h9HhYo0lX6PFG-Yb2cILbv4gPD6dKwUYYGgl-E,2564
+flexlock/report_template.html,sha256=6fz7DAs6cGl1PF3P0s3WpVYj-t330L2Q-68s1Ymhaz4,10375
+flexlock/resolvers.py,sha256=bzaVIlaht0BzBnBchbitsYLTw-F_8d46QXbY-06SVCE,9100
+flexlock/run_cli.py,sha256=Z2sdNUsVQDzK3-v1cyWGCtg16HebFE50kkPZ_3q11iw,1285
+flexlock/run_record.py,sha256=cqEL7LyighrzwuZqJ8NOiJdFmqDxKzuEnHNSqMtQ3II,6710
+flexlock/runner.py,sha256=NM3PmONey6i40jKPhWyF6qOaC-h4HBFVvbiyiJQYG80,23401
+flexlock/save_dir.py,sha256=LUc204z4RsIqvYZoXsdSl4Va7gF0lUoSmBgATl5DUs8,8567
+flexlock/snapshot.py,sha256=9mPsQNc70404jx-nIUpR8rDiFjTJ7JjIV21K_i9oDIc,8887
+flexlock/status_cli.py,sha256=cFYIl1Z_8SoMQJ0fuf3ntZJh5Y_AzK-31_09qYIcGaU,10217
+flexlock/taskdb.py,sha256=BJ_OiryyumNskljDm-NQpZPMun7b66yi1FeYqYv1w_g,23276
+flexlock/utils.py,sha256=4A4MdM7CNKWWpz_nse0F7KIOHnA54L9a2sSMayooPtQ,26250
+flexlock/worker.py,sha256=phQrOpQUBHkFpsFh7tr2VdZfkytEfLQhLs7xJN5f0fM,9024
+flexlock/worker_cli.py,sha256=qEB27EqP3W1YoQASlIbqbHx4Njq_xGOVcxnZqK3IIXY,4973
+flexlock/backends/__init__.py,sha256=tHK-B_om2eRHl94H_arZqM8WFgLVmJoEIWIeXxJcrOs,205
+flexlock/backends/base.py,sha256=hJaQW9Fgpym_2rZs31nOtbCjloGO4V8hoB5vVSp9fxk,1057
+flexlock/backends/pbs.py,sha256=X-1LRFNJPKJkumJpF4lIJdxICbop0P3hKa_PqDRsRsg,7382
+flexlock/backends/slurm.py,sha256=qmG5twnGC7CQPoOBpLoRmZ9QRifWkStfXiSTn7JJLI0,12230
+flexlock/skills/flexlock-new-stage/SKILL.md,sha256=KCuurVRsGP3Tf_DF2bMlY7MEhcgDxn0LW10j-nzMmxc,2502
+flexlock/skills/flexlock-report/SKILL.md,sha256=O-m8Rglhttw3IB-s9M3K2rpq-oiOuYr8jZ4cUlv7MHI,1671
+flexlock/skills/flexlock-run-and-watch/SKILL.md,sha256=SycdkUekgHl9EFpCmtpEHgLNk0_CxU5NujjmE_m71IQ,2332
+flexlock/skills/flexlock-survey/SKILL.md,sha256=I4imIn2yomW1cOinbZn8pCaxxXFIJXxFi-g5tSP8VXM,2315
+flexlock-0.8.2.dist-info/METADATA,sha256=zGtEJoEHfY1dpopLY71_orYtVH6mcuuZQRsJQw9o5B4,9797
+flexlock-0.8.2.dist-info/WHEEL,sha256=lCkmxWfQsSc9CfIClYeavTdQeEX2toPqufh9gI35EQA,87
+flexlock-0.8.2.dist-info/entry_points.txt,sha256=JRQDXmol961tfxHc_VDFvJqpsFC-GjSI5XZ1ayGGBwc,248
+flexlock-0.8.2.dist-info/RECORD,,

flexlock-0.8.2.dist-info/WHEEL

@@ -0,0 +1,4 @@
+Wheel-Version: 1.0
+Generator: hatchling 1.31.0
+Root-Is-Purelib: true
+Tag: py3-none-any

flexlock-0.8.2.dist-info/entry_points.txt

@@ -0,0 +1,7 @@
+[console_scripts]
+flexlock = flexlock.cli:main
+flexlock-diff = flexlock.diff_cli:main
+flexlock-export = flexlock.export:main
+flexlock-run = flexlock.run_cli:main
+flexlock-status = flexlock.status_cli:main
+flexlock-worker = flexlock.worker_cli:main