speedtronic 0.1.0__tar.gz

speedtronic-0.1.0/LICENSE

@@ -0,0 +1,178 @@
+Apache License
+Version 2.0, January 2004
+http://www.apache.org/licenses/
+
+TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+1. Definitions.
+
+   "License" shall mean the terms and conditions for use, reproduction,
+   and distribution as defined by Sections 1 through 9 of this document.
+
+   "Licensor" shall mean the copyright owner or entity authorized by
+   the copyright owner that is granting the License.
+
+   "Legal Entity" shall mean the union of the acting entity and all
+   other entities that control, are controlled by, or are under common
+   control with that entity. For the purposes of this definition,
+   "control" means (i) the power, direct or indirect, to cause the
+   direction or management of such entity, whether by contract or
+   otherwise, or (ii) ownership of fifty percent (50%) or more of the
+   outstanding shares, or (iii) beneficial ownership of such entity.
+
+   "You" (or "Your") shall mean an individual or Legal Entity
+   exercising permissions granted by this License.
+
+   "Source" form shall mean the preferred form for making modifications,
+   including but not limited to software source code, documentation
+   source, and configuration files.
+
+   "Object" form shall mean any form resulting from mechanical
+   transformation or translation of a Source form, including but
+   not limited to compiled object code, generated documentation,
+   and conversions to other media types.
+
+   "Work" shall mean the work of authorship, whether in Source or
+   Object form, made available under the License, as indicated by a
+   copyright notice that is included in or attached to the work
+   (an example is provided in the Appendix below).
+
+   "Derivative Works" shall mean any work, whether in Source or Object
+   form, that is based on (or derived from) the Work and for which the
+   editorial revisions, annotations, elaborations, or other modifications
+   represent, as a whole, an original work of authorship. For the purposes
+   of this License, Derivative Works shall not include works that remain
+   separable from, or merely link (or bind by name) to the interfaces of,
+   the Work and Derivative Works thereof.
+
+   "Contribution" shall mean any work of authorship, including
+   the original version of the Work and any modifications or additions
+   to that Work or Derivative Works thereof, that is intentionally
+   submitted to Licensor for inclusion in the Work by the copyright owner
+   or by an individual or Legal Entity authorized to submit on behalf of
+   the copyright owner. For the purposes of this definition, "submitted"
+   means any form of electronic, verbal, or written communication sent
+   to the Licensor or its representatives, including but not limited to
+   communication on electronic mailing lists, source code control systems,
+   and issue tracking systems that are managed by, or on behalf of, the
+   Licensor for the purpose of discussing and improving the Work, but
+   excluding communication that is conspicuously marked or otherwise
+   designated in writing by the copyright owner as "Not a Contribution."
+
+   "Contributor" shall mean Licensor and any individual or Legal Entity
+   on behalf of whom a Contribution has been received by Licensor and
+   subsequently incorporated within the Work.
+
+2. Grant of Copyright License. Subject to the terms and conditions of
+   this License, each Contributor hereby grants to You a perpetual,
+   worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+   copyright license to reproduce, prepare Derivative Works of,
+   publicly display, publicly perform, sublicense, and distribute the
+   Work and such Derivative Works in Source or Object form.
+
+3. Grant of Patent License. Subject to the terms and conditions of
+   this License, each Contributor hereby grants to You a perpetual,
+   worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+   (except as stated in this section) patent license to make, have made,
+   use, offer to sell, sell, import, and otherwise transfer the Work,
+   where such license applies only to those patent claims licensable
+   by such Contributor that are necessarily infringed by their
+   Contribution(s) alone or by combination of their Contribution(s)
+   with the Work to which such Contribution(s) was submitted. If You
+   institute patent litigation against any entity (including a
+   cross-claim or counterclaim in a lawsuit) alleging that the Work
+   or a Contribution incorporated within the Work constitutes direct
+   or contributory patent infringement, then any patent licenses
+   granted to You under this License for that Work shall terminate
+   as of the date such litigation is filed.
+
+4. Redistribution. You may reproduce and distribute copies of the
+   Work or Derivative Works thereof in any medium, with or without
+   modifications, and in Source or Object form, provided that You
+   meet the following conditions:
+
+   (a) You must give any other recipients of the Work or
+       Derivative Works a copy of this License; and
+
+   (b) You must cause any modified files to carry prominent notices
+       stating that You changed the files; and
+
+   (c) You must retain, in the Source form of any Derivative Works
+       that You distribute, all copyright, patent, trademark, and
+       attribution notices from the Source form of the Work,
+       excluding those notices that do not pertain to any part of
+       the Derivative Works; and
+
+   (d) If the Work includes a "NOTICE" text file as part of its
+       distribution, then any Derivative Works that You distribute must
+       include a readable copy of the attribution notices contained
+       within such NOTICE file, excluding those notices that do not
+       pertain to any part of the Derivative Works, in at least one
+       of the following places: within a NOTICE text file distributed
+       as part of the Derivative Works; within the Source form or
+       documentation, if provided along with the Derivative Works; or,
+       within a display generated by the Derivative Works, if and
+       wherever such third-party notices normally appear. The contents
+       of the NOTICE file are for informational purposes only and
+       do not modify the License. You may add Your own attribution
+       notices within Derivative Works that You distribute, alongside
+       or as an addendum to the NOTICE text from the Work, provided
+       that such additional attribution notices cannot be construed
+       as modifying the License.
+
+   You may add Your own copyright statement to Your modifications and
+   may provide additional or different license terms and conditions
+   for use, reproduction, or distribution of Your modifications, or
+   for any such Derivative Works as a whole, provided Your use,
+   reproduction, and distribution of the Work otherwise complies with
+   the conditions stated in this License.
+
+5. Submission of Contributions. Unless You explicitly state otherwise,
+   any Contribution intentionally submitted for inclusion in the Work
+   by You to the Licensor shall be under the terms and conditions of
+   this License, without any additional terms or conditions.
+   Notwithstanding the above, nothing herein shall supersede or modify
+   the terms of any separate license agreement you may have executed
+   with Licensor regarding such Contributions.
+
+6. Trademarks. This License does not grant permission to use the trade
+   names, trademarks, service marks, or product names of the Licensor,
+   except as required for reasonable and customary use in describing the
+   origin of the Work and reproducing the content of the NOTICE file.
+
+7. Disclaimer of Warranty. Unless required by applicable law or
+   agreed to in writing, Licensor provides the Work (and each
+   Contributor provides its Contributions) on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+   implied, including, without limitation, any warranties or conditions
+   of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+   PARTICULAR PURPOSE. You are solely responsible for determining the
+   appropriateness of using or redistributing the Work and assume any
+   risks associated with Your exercise of permissions under this License.
+
+8. Limitation of Liability. In no event and under no legal theory,
+   whether in tort (including negligence), contract, or otherwise,
+   unless required by applicable law (such as deliberate and grossly
+   negligent acts) or agreed to in writing, shall any Contributor be
+   liable to You for damages, including any direct, indirect, special,
+   incidental, or consequential damages of any character arising as a
+   result of this License or out of the use or inability to use the
+   Work (including but not limited to damages for loss of goodwill,
+   work stoppage, computer failure or malfunction, or any and all
+   other commercial damages or losses), even if such Contributor
+   has been advised of the possibility of such damages.
+
+9. Accepting Warranty or Additional Liability. While redistributing
+   the Work or Derivative Works thereof, You may choose to offer,
+   and charge a fee for, acceptance of support, warranty, indemnity,
+   or other liability obligations and/or rights consistent with this
+   License. However, in accepting such obligations, You may act only
+   on Your own behalf and on Your sole responsibility, not on behalf
+   of any other Contributor, and only if You agree to indemnify,
+   defend, and hold each Contributor harmless for any liability
+   incurred by, or claims asserted against, such Contributor by reason
+   of your accepting any such warranty or additional liability.
+
+END OF TERMS AND CONDITIONS
+
+Copyright 2026 BenchLabs

speedtronic-0.1.0/MANIFEST.in

@@ -0,0 +1,12 @@
+include LICENSE
+include README.md
+include pyproject.toml
+recursive-include configs *.yaml
+recursive-include examples *.py *.yaml
+recursive-include tests *.py
+
+global-exclude __pycache__
+global-exclude *.py[cod]
+prune .speedtronic
+prune checkpoints
+prune runs

speedtronic-0.1.0/PKG-INFO

@@ -0,0 +1,288 @@
+Metadata-Version: 2.4
+Name: speedtronic
+Version: 0.1.0
+Summary: A fast, efficient, architecture-neutral training engine with DumbDiLoCo
+Author: BenchLabs
+License-Expression: Apache-2.0
+Project-URL: Homepage, https://github.com/bench-labs-org/Speedtronic
+Project-URL: Repository, https://github.com/bench-labs-org/Speedtronic
+Project-URL: Issues, https://github.com/bench-labs-org/Speedtronic/issues
+Keywords: artificial-intelligence,deep-learning,distributed-training,pytorch,training
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: torch>=2.1
+Requires-Dist: safetensors>=0.4
+Requires-Dist: PyYAML>=6.0
+Requires-Dist: numpy>=1.24
+Requires-Dist: huggingface-hub>=0.23
+Provides-Extra: dev
+Requires-Dist: pytest>=7.4; extra == "dev"
+Requires-Dist: ruff>=0.5; extra == "dev"
+Provides-Extra: logging
+Requires-Dist: wandb>=0.16; extra == "logging"
+Requires-Dist: tensorboard>=2.14; extra == "logging"
+Dynamic: license-file
+
+<div align="center">
+
+<img width="256" height="256" alt="Speedtronic Logo" src="https://github.com/user-attachments/assets/3389569b-430f-49f3-98b8-7c2d73dca8dc" />
+
+# Speedtronic
+
+*GPU-agnostic AI training, built for speed.*
+
+</div>
+
+[![PyPI version](https://img.shields.io/pypi/v/speedtronic.svg)](https://pypi.org/project/speedtronic/)
+
+**Be Fast, Be Efficient.**
+
+Speedtronic is an open-source PyTorch training framework. You bring your own `nn.Module`, dataset, and a single YAML config; it runs an efficient training loop on CPU, CUDA, or MPS, with optional asynchronous multi-node training (DumbDiLoCo) synced through a Hugging Face Hub repo instead of a dedicated parameter server.
+
+## What you get
+
+- Architecture-neutral training engine: works with any `nn.Module`, not just the bundled model.
+- Reference GPT-style decoder-only transformer (RoPE, GQA, SwiGLU, tied embeddings) so the repo runs out of the box.
+- Single config file per run: local and distributed runs share the same code path.
+- Efficiency defaults: auto mixed precision, gradient accumulation, streaming data loading with prefetch, fused AdamW when available, opt-in `torch.compile` and gradient checkpointing.
+- Local resumable checkpoints (model, optimizer, scheduler, RNG, step counters).
+- DumbDiLoCo: DiLoCo-style local-steps-then-sync, with the Hub as transport. One node is master, the rest are workers. No extra infrastructure to host.
+- Pluggable metrics: stdout/file/JSONL built in, W&B and TensorBoard via optional hooks.
+
+## Requirements
+
+- Python >= 3.10
+- PyTorch >= 2.1 (install the CPU or CUDA wheel appropriate for your machine first)
+- `safetensors`, `PyYAML`, `numpy`, `huggingface-hub` (installed automatically)
+
+## Install
+
+```bash
+pip install speedtronic
+# optional logging integrations
+pip install 'speedtronic[logging]'
+```
+
+For a source checkout:
+
+```bash
+pip install -e .
+# dev tools (pytest, ruff)
+pip install -e '.[dev]'
+```
+
+## Quickstart (60 seconds)
+
+Train the bundled reference model on synthetic data (CPU-safe, no downloads):
+
+```bash
+speedtronic train --config configs/smoke.yaml
+```
+
+Validate a config without training:
+
+```bash
+speedtronic validate --config configs/smoke.yaml
+```
+
+Same run from Python:
+
+```python
+from speedtronic import SpeedtronicConfig
+from speedtronic.runtime import train_from_config
+
+config = SpeedtronicConfig.from_dict({
+    "run": {"name": "demo", "max_steps": 100, "device": "cpu"},
+    "model": {
+        "name": "reference_transformer",
+        "vocab_size": 256,
+        "max_seq_len": 64,
+        "n_layer": 2,
+        "n_head": 4,
+        "n_kv_head": 2,
+        "d_model": 64,
+    },
+    "data": {"micro_batch_size": 2, "target_batch_size": 8},
+    "precision": {"mode": "fp32"},
+})
+result = train_from_config(config)
+print(result.steps, result.final_loss)
+```
+
+CLI flags override the file without editing it:
+
+```bash
+speedtronic train --config configs/smoke.yaml --max-steps 100 --output-dir runs/demo --resume
+```
+
+## Project layout
+
+```text
+src/speedtronic/
+  config.py        # SpeedtronicConfig: validation, YAML/JSON/dict loading
+  trainer.py       # Trainer: accumulation, precision, compile, checkpoint hooks
+  runtime.py       # build_runtime / train_from_config: wiring from config to Trainer
+  model.py         # reference transformer (example model, not a framework dependency)
+  registry.py      # model registry for custom architectures
+  data.py          # synthetic + streaming text datasets, dataloader builder
+  precision.py     # device and dtype capability detection
+  checkpoint.py    # atomic local checkpoints
+  profiling.py     # MetricLogger, stdout/file/JSONL output, hook fan-out
+  integrations.py  # optional W&B / TensorBoard hooks
+  cli.py           # speedtronic train | validate
+  distributed/     # DumbDiLoCo: hub.py, diloco.py, outer.py, tensors.py
+configs/
+  smoke.yaml       # minimal CPU run
+  diloco.yaml      # local template with distributed section
+examples/
+  train_reference.py
+  diloco_master.yaml
+tests/             # CPU-only, Hub fakes (no network credentials needed)
+```
+
+## Configuration
+
+One `SpeedtronicConfig` drives everything. Load from YAML, JSON, a JSON/YAML string, or a plain dict. Unknown keys are rejected so typos fail fast.
+
+| Section | Controls |
+|---|---|
+| `run` | name, seed, device (`auto`/`cpu`/`cuda`/`mps`), `max_steps`, `output_dir` |
+| `model` | registered model name + architecture params |
+| `data` | `micro_batch_size` / `target_batch_size` (accumulation derived), text path or synthetic source, workers, prefetch, pin_memory, shuffle |
+| `optimizer` | AdamW LR, betas, eps, weight decay, optional fused, grad clip |
+| `scheduler` | `cosine` (default) or `constant`, warmup steps, min LR ratio |
+| `precision` | `auto`/`bf16`/`fp16`/`fp32`; auto picks bf16 on capable CUDA, fp32 elsewhere |
+| `compile` | opt-in `torch.compile`; logs and continues uncompiled if unsupported |
+| `gradient_checkpointing` | calls the model's `set_gradient_checkpointing()` hook if present |
+| `checkpoint` | directory, interval, retention (`keep_last`), resume flag |
+| `logging` | level, file, JSONL file, step cadence, hook list |
+| `distributed` | DumbDiLoCo role, node id, Hub repo, inner/outer loop tuning (see below) |
+
+Top-level aliases (`name`, `max_steps`, `output_dir`, `seed`, `device`) are accepted as shorthand for `run.*`.
+
+## Bring your own model
+
+The engine only requires a module that maps a batch to a loss. Return a tensor, a `{"loss": ...}` dict, or a `(loss, ...)` tuple. Dict batches are passed as kwargs; tuple batches are passed as `(inputs, labels)`.
+
+```python
+import torch
+from speedtronic import register_model
+
+@register_model("my_model")
+def make_model(**kwargs):
+    return torch.nn.TransformerEncoder(
+        torch.nn.TransformerEncoderLayer(d_model=kwargs["d_model"], nhead=kwargs.get("n_head", 4)),
+        num_layers=kwargs.get("n_layer", 2),
+    )
+```
+
+```yaml
+model:
+  name: my_model
+  d_model: 256
+```
+
+For activation checkpointing, expose `set_gradient_checkpointing(enabled)` on your module; the trainer calls it and warns if it is absent.
+
+## Bring your own data
+
+- Default: deterministic synthetic token stream (good for smoke tests and benchmarking the loop).
+- `data.text_path`: streams a UTF-8 file in fixed-size token blocks without loading it into RAM.
+- `build_dataloader(..., dataset=..., tokenizer=...)`: pass any PyTorch `Dataset`/`IterableDataset` directly.
+- `micro_batch_size` is what the loader emits; `target_batch_size` is the effective batch after accumulation. The trainer handles the math and keeps mixed precision correct across accumulation steps.
+
+## Efficiency behavior
+
+- **Precision:** `auto` uses bf16 where `torch.cuda.is_bf16_supported()`, fp16 with `GradScaler` on CUDA otherwise, fp32 on CPU/MPS. Explicit modes are honored unless the hardware cannot run them, in which case the run falls back to fp32 instead of crashing.
+- **`torch.compile`:** never a hard failure. Unsupported combinations log and continue eagerly, including a runtime fallback if a compiled step throws.
+- **Fused AdamW:** used only when construction with `fused=True` succeeds on CUDA.
+- **Attention:** the reference model uses `scaled_dot_product_attention` so PyTorch picks flash / memory-efficient / math per device. No flash-attn dependency.
+- **Data:** `num_workers`, `prefetch_factor`, and `pin_memory` are configurable; iterable datasets skip sampler shuffling (implement shuffling in the stream if needed).
+
+## Checkpoints and resume
+
+Checkpoints store model weights, optimizer state, scheduler state, step/sample/token counters, RNG state, redacted config copy, and (for DumbDiLoCo) coordinator state. Files are written to temp + atomic rename; a `latest.json` pointer tracks the newest checkpoint.
+
+```bash
+speedtronic train --config configs/smoke.yaml --resume
+```
+
+`max_steps` is a global step target, so resuming with the same config continues rather than restarting. Saved Hub tokens are redacted (`<redacted>`) in stored configs; provide tokens via environment at runtime.
+
+## DumbDiLoCo (distributed without a parameter server)
+
+Each node trains locally for `inner_steps` AdamW steps, uploads a pseudo-gradient (`weights_at_last_sync - current_weights`) to its own `nodes/<node_id>/` folder in a Hub repo, and periodically pulls the newest global weights. The master aggregates whatever deltas arrived since the last round (simple mean), applies Nesterov momentum SGD, and publishes `global/latest.safetensors` + `global/step_count.json`. Nodes never share a clock and never block on stragglers.
+
+```yaml
+distributed:
+  enabled: true
+  mode: dumb_diloco
+  role: master       # worker on the other machines
+  node_id: node-a    # unique per participant, path-safe
+  repo_id: my-org/my-run
+  # token: prefer HF_TOKEN env var over YAML
+  inner_steps: 500
+  poll_interval: 60
+  outer_lr: 0.7
+  outer_momentum: 0.9
+```
+
+Repo layout on the Hub:
+
+```text
+<run>/
+  global/latest.safetensors
+  global/step_count.json
+  nodes/<node_id>/delta_<local_step>.safetensors
+```
+
+Operational notes:
+
+- The master creates the repo (private by default) and attempts `collaborators` grants where the installed Hub SDK supports it; otherwise grant write access in the repo settings UI.
+- `node_id` defaults to `HF_USERNAME` / `USER` / `local` if unset; set it explicitly per machine.
+- Transient Hub failures retry with exponential backoff and never kill local training; a failed delta upload is retried as a cumulative delta at the next boundary.
+- A corrupt delta is logged with its filename and skipped for that round.
+- A restarted worker resumes from the latest global weights (at most one partial inner loop is lost). A restarted master resumes global weights, outer step, processed-delta set, and Nesterov momentum from its local `state_dir`.
+- `distributed.reset_inner_optimizer` (default `true`) clears Adam state at each inner-loop boundary so stale moments do not leak across sync rounds.
+
+Security model: trusted collaborators only. Anyone with a valid write token for the repo can influence the global model. There is no cryptographic node identity, Byzantine fault tolerance, or staleness weighting. Do not point this at a public-write repo.
+
+## Metrics
+
+Every run logs `loss`, `lr`, `steps/sec`, `samples/sec`, `tokens/sec`, memory (when the backend exposes it), and — for DumbDiLoCo — outer step plus deltas found/included per round. Output goes to stdout by default, optionally to a file and/or JSONL (`logging.file`, `logging.json_file`). `MetricLogger(hooks=[...])` accepts `(event, payload)` callables or objects with `on_event`; W&B and TensorBoard adapters live in `integrations.py` and stay optional dependencies.
+
+## Development
+
+```bash
+pip install -e '.[dev]'
+pytest -q
+ruff check src tests examples
+ruff format --check src tests examples
+python -m compileall src
+```
+
+Tests are CPU-only. Hub tests use fakes; no network or credentials required.
+
+## Scope and non-goals
+
+- One reference model, not a model zoo.
+- No tokenizer training tooling.
+- No FSDP or pipeline parallelism.
+- No open/adversarial participation mode.
+- No hosted dashboard; CLI + config files only.
+- Mid-inner-loop worker resume is out of scope (restart from latest global).
+
+## License
+
+Apache-2.0. See `LICENSE`.