ravex 0.0.2__tar.gz

ravex-0.0.2/LICENSE

@@ -0,0 +1,201 @@
+                                 Apache License
+                           Version 2.0, January 2004
+                        http://www.apache.org/licenses/
+
+   TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
+
+   1. Definitions.
+
+      "License" shall mean the terms and conditions for use, reproduction,
+      and distribution as defined by Sections 1 through 9 of this document.
+
+      "Licensor" shall mean the copyright owner or entity authorized by
+      the copyright owner that is granting the License.
+
+      "Legal Entity" shall mean the union of the acting entity and all
+      other entities that control, are controlled by, or are under common
+      control with that entity. For the purposes of this definition,
+      "control" means (i) the power, direct or indirect, to cause the
+      direction or management of such entity, whether by contract or
+      otherwise, or (ii) ownership of fifty percent (50%) or more of the
+      outstanding shares, or (iii) beneficial ownership of such entity.
+
+      "You" (or "Your") shall mean an individual or Legal Entity
+      exercising permissions granted by this License.
+
+      "Source" form shall mean the preferred form for making modifications,
+      including but not limited to software source code, documentation
+      source, and configuration files.
+
+      "Object" form shall mean any form resulting from mechanical
+      transformation or translation of a Source form, including but
+      not limited to compiled object code, generated documentation,
+      and conversions to other media types.
+
+      "Work" shall mean the work of authorship, whether in Source or
+      Object form, made available under the License, as indicated by a
+      copyright notice that is included in or attached to the work
+      (an example is provided in the Appendix below).
+
+      "Derivative Works" shall mean any work, whether in Source or Object
+      form, that is based on (or derived from) the Work and for which the
+      editorial revisions, annotations, elaborations, or other modifications
+      represent, as a whole, an original work of authorship. For the purposes
+      of this License, Derivative Works shall not include works that remain
+      separable from, or merely link (or bind by name) to the interfaces of,
+      the Work and Derivative Works thereof.
+
+      "Contribution" shall mean any work of authorship, including
+      the original version of the Work and any modifications or additions
+      to that Work or Derivative Works thereof, that is intentionally
+      submitted to Licensor for inclusion in the Work by the copyright owner
+      or by an individual or Legal Entity authorized to submit on behalf of
+      the copyright owner. For the purposes of this definition, "submitted"
+      means any form of electronic, verbal, or written communication sent
+      to the Licensor or its representatives, including but not limited to
+      communication on electronic mailing lists, source code control systems,
+      and issue tracking systems that are managed by, or on behalf of, the
+      Licensor for the purpose of discussing and improving the Work, but
+      excluding communication that is conspicuously marked or otherwise
+      designated in writing by the copyright owner as "Not a Contribution."
+
+      "Contributor" shall mean Licensor and any individual or Legal Entity
+      on behalf of whom a Contribution has been received by Licensor and
+      subsequently incorporated within the Work.
+
+   2. Grant of Copyright License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      copyright license to reproduce, prepare Derivative Works of,
+      publicly display, publicly perform, sublicense, and distribute the
+      Work and such Derivative Works in Source or Object form.
+
+   3. Grant of Patent License. Subject to the terms and conditions of
+      this License, each Contributor hereby grants to You a perpetual,
+      worldwide, non-exclusive, no-charge, royalty-free, irrevocable
+      (except as stated in this section) patent license to make, have made,
+      use, offer to sell, sell, import, and otherwise transfer the Work,
+      where such license applies only to those patent claims licensable
+      by such Contributor that are necessarily infringed by their
+      Contribution(s) alone or by combination of their Contribution(s)
+      with the Work to which such Contribution(s) was submitted. If You
+      institute patent litigation against any entity (including a
+      cross-claim or counterclaim in a lawsuit) alleging that the Work
+      or a Contribution incorporated within the Work constitutes direct
+      or contributory patent infringement, then any patent licenses
+      granted to You under this License for that Work shall terminate
+      as of the date such litigation is filed.
+
+   4. Redistribution. You may reproduce and distribute copies of the
+      Work or Derivative Works thereof in any medium, with or without
+      modifications, and in Source or Object form, provided that You
+      meet the following conditions:
+
+      (a) You must give any other recipients of the Work or
+          Derivative Works a copy of this License; and
+
+      (b) You must cause any modified files to carry prominent notices
+          stating that You changed the files; and
+
+      (c) You must retain, in the Source form of any Derivative Works
+          that You distribute, all copyright, patent, trademark, and
+          attribution notices from the Source form of the Work,
+          excluding those notices that do not pertain to any part of
+          the Derivative Works; and
+
+      (d) If the Work includes a "NOTICE" text file as part of its
+          distribution, then any Derivative Works that You distribute must
+          include a readable copy of the attribution notices contained
+          within such NOTICE file, excluding those notices that do not
+          pertain to any part of the Derivative Works, in at least one
+          of the following places: within a NOTICE text file distributed
+          as part of the Derivative Works; within the Source form or
+          documentation, if provided along with the Derivative Works; or,
+          within a display generated by the Derivative Works, if and
+          wherever such third-party notices normally appear. The contents
+          of the NOTICE file are for informational purposes only and
+          do not modify the License. You may add Your own attribution
+          notices within Derivative Works that You distribute, alongside
+          or as an addendum to the NOTICE text from the Work, provided
+          that such additional attribution notices cannot be construed
+          as modifying the License.
+
+      You may add Your own copyright statement to Your modifications and
+      may provide additional or different license terms and conditions
+      for use, reproduction, or distribution of Your modifications, or
+      for any such Derivative Works as a whole, provided Your use,
+      reproduction, and distribution of the Work otherwise complies with
+      the conditions stated in this License.
+
+   5. Submission of Contributions. Unless You explicitly state otherwise,
+      any Contribution intentionally submitted for inclusion in the Work
+      by You to the Licensor shall be under the terms and conditions of
+      this License, without any additional terms or conditions.
+      Notwithstanding the above, nothing herein shall supersede or modify
+      the terms of any separate license agreement you may have executed
+      with Licensor regarding such Contributions.
+
+   6. Trademarks. This License does not grant permission to use the trade
+      names, trademarks, service marks, or product names of the Licensor,
+      except as required for reasonable and customary use in describing the
+      origin of the Work and reproducing the content of the NOTICE file.
+
+   7. Disclaimer of Warranty. Unless required by applicable law or
+      agreed to in writing, Licensor provides the Work (and each
+      Contributor provides its Contributions) on an "AS IS" BASIS,
+      WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or
+      implied, including, without limitation, any warranties or conditions
+      of TITLE, NON-INFRINGEMENT, MERCHANTABILITY, or FITNESS FOR A
+      PARTICULAR PURPOSE. You are solely responsible for determining the
+      appropriateness of using or redistributing the Work and assume any
+      risks associated with Your exercise of permissions under this License.
+
+   8. Limitation of Liability. In no event and under no legal theory,
+      whether in tort (including negligence), contract, or otherwise,
+      unless required by applicable law (such as deliberate and grossly
+      negligent acts) or agreed to in writing, shall any Contributor be
+      liable to You for damages, including any direct, indirect, special,
+      incidental, or consequential damages of any character arising as a
+      result of this License or out of the use or inability to use the
+      Work (including but not limited to damages for loss of goodwill,
+      work stoppage, computer failure or malfunction, or any and all
+      other commercial damages or losses), even if such Contributor
+      has been advised of the possibility of such damages.
+
+   9. Accepting Warranty or Additional Liability. While redistributing
+      the Work or Derivative Works thereof, You may choose to offer,
+      and charge a fee for, acceptance of support, warranty, indemnity,
+      or other liability obligations and/or rights consistent with this
+      License. However, in accepting such obligations, You may act only
+      on Your own behalf and on Your sole responsibility, not on behalf
+      of any other Contributor, and only if You agree to indemnify,
+      defend, and hold each Contributor harmless for any liability
+      incurred by, or claims asserted against, such Contributor by reason
+      of your accepting any such warranty or additional liability.
+
+   END OF TERMS AND CONDITIONS
+
+   APPENDIX: How to apply the Apache License to your work.
+
+      To apply the Apache License to your work, attach the following
+      boilerplate notice, with the fields enclosed by brackets "[]"
+      replaced with your own identifying information. (Don't include
+      the brackets!)  The text should be enclosed in the appropriate
+      comment syntax for the file format. We also recommend that a
+      file or class name and description of purpose be included on the
+      same "printed page" as the copyright notice for easier
+      identification within third-party archives.
+
+   Copyright [yyyy] [name of copyright owner]
+
+   Licensed under the Apache License, Version 2.0 (the "License");
+   you may not use this file except in compliance with the License.
+   You may obtain a copy of the License at
+
+       http://www.apache.org/licenses/LICENSE-2.0
+
+   Unless required by applicable law or agreed to in writing, software
+   distributed under the License is distributed on an "AS IS" BASIS,
+   WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+   See the License for the specific language governing permissions and
+   limitations under the License.

ravex-0.0.2/NOTICE

@@ -0,0 +1,7 @@
+Ravex
+Copyright 2026 Jonathan Vecchione
+
+This product includes software developed as part of the GPU Zero project
+(https://codeberg.org/JHNMACHINE).
+
+Licensed under the Apache License, Version 2.0. See LICENSE for the full text.

ravex-0.0.2/PKG-INFO

@@ -0,0 +1,253 @@
+Metadata-Version: 2.4
+Name: ravex
+Version: 0.0.2
+Summary: Transparent checkpoint and resume for PyTorch training, with no changes to your training code
+Author: JHNMACHINE
+License: Apache-2.0
+Project-URL: Homepage, https://codeberg.org/JHNMACHINE/ravex
+Project-URL: Source, https://codeberg.org/JHNMACHINE/ravex
+Project-URL: Issues, https://codeberg.org/JHNMACHINE/ravex/issues
+Project-URL: Changelog, https://codeberg.org/JHNMACHINE/ravex/src/branch/main/CHANGELOG.md
+Keywords: pytorch,checkpoint,training,resume,mlops,preemption
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: Apache Software License
+Classifier: Programming Language :: Python :: 3
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+License-File: NOTICE
+Requires-Dist: PyYAML>=5.1
+Provides-Extra: moonclip
+Requires-Dist: moonclip>=0.0.4; extra == "moonclip"
+Provides-Extra: dev
+Requires-Dist: pytest>=7; extra == "dev"
+Requires-Dist: numpy; extra == "dev"
+Dynamic: license-file
+
+# Ravex
+
+Transparent checkpoint and resume for PyTorch training. Your training script
+does not change — not one line, not one import.
+
+```bash
+pip install ravex
+ravex enable
+```
+
+Ravex itself is pure Python and installs anywhere. Its default checkpoint engine,
+[Moonclip](https://codeberg.org/JHNMACHINE/moonclip), ships wheels for Linux
+x86_64 only — so `pip install "ravex[moonclip]"` is a Linux thing, and on any
+other platform Ravex falls back to `torch_save` on its own. What changed between
+versions is in
+[CHANGELOG.md](https://codeberg.org/JHNMACHINE/ravex/src/branch/main/CHANGELOG.md).
+
+Drop a `ravex.yaml` next to your code and run what you always ran:
+
+```bash
+python train.py
+```
+
+The run now checkpoints itself. If the process dies — spot instance reclaimed,
+node rebooted, OOM killer, power cut — running the same command again picks up
+where it stopped: same weights, same optimizer moments, same LR schedule, same
+RNG state, same position in the dataset.
+
+## Why this is not just `torch.save`
+
+Saving the weights is the easy part. What makes a resumed run *continue* rather
+than merely *restart from a good place* is everything around them:
+
+| Restored | Why it matters |
+|---|---|
+| Model weights and buffers | the obvious part |
+| Optimizer state | Adam moments; without them the first steps after resume are wrong |
+| LR scheduler | resume at the wrong LR and the loss curve visibly kinks |
+| AMP `GradScaler` | its loss scale is tuned state, not a constant |
+| RNG: torch, CUDA, Python, NumPy | dropout masks and augmentations replay identically |
+| Dataset position | you continue with batch 4001, not batch 1 |
+
+Ravex's test suite asserts the strong form of this: a run killed at step 20 and
+resumed produces losses that are **bit-identical**, step by step, to the run
+that was never interrupted, and the same final weights.
+
+## Two ways to use it
+
+**Zero code changes.** `ravex enable` installs a one-line `.pth` file in
+site-packages, which Python executes at interpreter startup. From then on Ravex
+attaches itself to any training process that has a `ravex.yaml`.
+
+**One line**, when you would rather be explicit:
+
+```python
+import ravex
+ravex.activate()
+```
+
+Both do the same thing. The `.pth` route exists so that a platform can enable
+checkpointing for code it does not own.
+
+## Configuration
+
+`ravex.yaml`, anywhere at or above the working directory:
+
+```yaml
+checkpoint_every: 500        # optimizer steps between checkpoints
+backend: moonclip            # moonclip | torch_save
+storage:
+  type: local                # local | s3 | r2
+  path: ./checkpoints
+keep_last: 5
+max_steps: null              # optional hard stop, see below
+sharded_checkpoints: gather  # gather | per_rank, for FSDP — see below
+```
+
+Every option also reads from `RAVEX_*` environment variables, which win over
+the file — so a scheduler can override a config committed to the repository:
+
+```bash
+RAVEX_CHECKPOINT_EVERY=100 RAVEX_STORAGE_TYPE=r2 RAVEX_STORAGE_BUCKET=runs python train.py
+```
+
+Credentials are never read from the config file. Set `RAVEX_S3_ACCESS_KEY` /
+`RAVEX_S3_SECRET_KEY`, or the usual `AWS_*` pair.
+
+Full reference: [docs/configuration.md](docs/configuration.md).
+
+## Backends
+
+**`moonclip`** (default) — the [Moonclip](https://codeberg.org/JHNMACHINE/moonclip)
+engine: per-tensor delta tracking, so unchanged weights cost zero I/O; zstd
+compression; background writes; direct S3/R2 sync.
+
+**`torch_save`** — one `.pt` file per checkpoint, written on a background
+thread. Used automatically when Moonclip is not installed. Correct, just larger
+and slower.
+
+## How it works
+
+Ravex patches five things in PyTorch and nothing in your code:
+
+- `nn.Module.__init__` and `.train()` — to notice your models
+- `Optimizer.__init__` — to attach a step hook to every optimizer
+- `DataLoader.__init__` and `.__iter__` — to track the dataset position and to
+  find the one moment where a resume can be applied
+
+The step counter advances once per `optimizer.step()`, so gradient accumulation
+needs no special handling. Checkpoints are collected at the *top of an
+iteration*, never inside one: mid-iteration the LR scheduler has not stepped
+yet, and a checkpoint taken there resumes with a stale learning rate.
+
+Collection runs on the training thread — it has to, to be consistent with the
+step that just finished — and copies the state; the write itself happens in the
+background. What the loop pays for is the copy, not the I/O.
+
+More detail: [docs/how-it-works.md](docs/how-it-works.md).
+
+## Safety
+
+Ravex is designed to be un-noticeable when it works and harmless when it does
+not:
+
+- every hook is wrapped; if one raises, your call still returns normally
+- if a checkpoint fails, Ravex disables itself and logs it — training continues
+- nothing is ever written to stdout; logs go to `log_file`, or to stderr at
+  WARNING and above
+- installing the package changes nothing on its own. Without `ravex enable`
+  there is no `.pth`; with it, Ravex still only wakes up for projects that have
+  a `ravex.yaml` or set `RAVEX_ENABLED=1`
+- `ravex disable` removes the autoloader; `RAVEX_ENABLED=0` turns it off for a
+  single run
+
+## Status and limits
+
+Alpha. Works with plain PyTorch loops, and with anything built on them, since
+the hooks are on PyTorch itself.
+
+### With a framework driving the loop
+
+HuggingFace `Trainer` and Lightning are covered by their own tests, and the
+result deserves to be stated precisely rather than as "it works":
+
+- **State restoration is exact.** Model, optimizer, LR scheduler and step count
+  all come back. With the per-step randomness removed, a killed run resumes
+  into a loss sequence identical to the uninterrupted one.
+- **Replay is not.** With shuffling and dropout on, the resumed run continues
+  correctly from the checkpointed state but sees a different draw. Both
+  frameworks iterate the dataloader on their own schedule and consume the
+  global RNG around the loop, so the epoch-start snapshot no longer lines up.
+
+Plain loops, DDP and FSDP *are* bit-exact with randomness on. This is a
+framework-interaction limit, not a general one, and it costs you a different
+shuffle from the resume point onwards — not a wrong model.
+
+Verified: plain loops, gradient accumulation, LR schedulers, AMP loss-scale
+state, `num_workers > 0`, DDP, and FSDP. A killed `torchrun` job resumes on
+*every* rank with bit-identical losses, sharded or not, and the checkpoint it
+leaves behind loads into a plain single-process model afterwards.
+
+### Sharded models
+
+With FSDP each rank holds a slice of every parameter, so `state_dict()` returns
+a fragment. Two ways to turn that into a checkpoint, picked with
+`sharded_checkpoints`:
+
+**`gather`** (default) rebuilds the whole state on rank 0, which writes it. The
+checkpoint is then independent of the topology that produced it — eight GPUs in,
+one out — and it does not scale: rank 0 has to hold the entire model and
+optimizer in host memory, and it is the rank that then does the writing.
+
+**`per_rank`** has every rank write its own shard into its own store,
+`<storage.path>/rank_<n>`. Nothing is gathered, so nothing is bounded by one
+rank's memory, and on a 1.48B model collecting the state went from 15.6 s to
+1.5 s. What you give up is the resharding: those shards are cut for one topology,
+so the checkpoint resumes at the same world size and starts clean at any other.
+Needs FSDP2 — under FSDP1 Ravex degrades to `gather` and says so.
+
+Either way, collecting is a **collective**: every rank participates, and there is
+**no final checkpoint at exit** for a sharded model. Shutdown is where ranks stop
+being in lockstep, and a collective nobody else joins hangs. Losing the last few
+steps is bounded; a hang is not. Set `checkpoint_every` accordingly.
+
+Numbers and the FSDP1 details: [docs/configuration.md](docs/configuration.md).
+
+Known limits today:
+
+- **`IterableDataset`**: no index sampler exists, so the stream position cannot
+  be replayed. Everything else is still restored.
+- **Your loop's bounds**: a resumed script runs its own `for epoch in
+  range(N)` again from the top; it has no idea 3000 steps already happened. Set
+  `max_steps` and Ravex ends the run at the right step regardless of how many
+  times the process restarted.
+
+Under AMP, note that an overflowing gradient makes `scaler.step()` skip the
+optimizer. Ravex counts optimizer steps, not loop iterations, so a skipped
+iteration does not advance the counter — which is the right unit, since nothing
+about the model changed, but it does mean the step count and the number of
+batches you fed differ.
+
+The GPU paths — AMP with real fp16 overflow, the CUDA RNG, FSDP1, NCCL — are
+covered by `integration/test_cuda.py`, which skips without a GPU. They were
+last verified on 8× RTX 5060 Ti with torch 2.12/cu130.
+
+## Development
+
+```bash
+pip install -e ".[dev]"
+pytest
+```
+
+The unit suite runs in-process. The parts that only exist across a real process
+boundary — the `.pth` autoloader, a resume starting from an empty interpreter,
+`torchrun` — live in `integration/` and need Linux:
+
+```bash
+docker build -f integration/Dockerfile -t ravex-integration .
+docker run --rm ravex-integration
+```
+
+## Licence
+
+Apache 2.0. See [LICENSE](LICENSE).

ravex-0.0.2/README.md

@@ -0,0 +1,224 @@
+# Ravex
+
+Transparent checkpoint and resume for PyTorch training. Your training script
+does not change — not one line, not one import.
+
+```bash
+pip install ravex
+ravex enable
+```
+
+Ravex itself is pure Python and installs anywhere. Its default checkpoint engine,
+[Moonclip](https://codeberg.org/JHNMACHINE/moonclip), ships wheels for Linux
+x86_64 only — so `pip install "ravex[moonclip]"` is a Linux thing, and on any
+other platform Ravex falls back to `torch_save` on its own. What changed between
+versions is in
+[CHANGELOG.md](https://codeberg.org/JHNMACHINE/ravex/src/branch/main/CHANGELOG.md).
+
+Drop a `ravex.yaml` next to your code and run what you always ran:
+
+```bash
+python train.py
+```
+
+The run now checkpoints itself. If the process dies — spot instance reclaimed,
+node rebooted, OOM killer, power cut — running the same command again picks up
+where it stopped: same weights, same optimizer moments, same LR schedule, same
+RNG state, same position in the dataset.
+
+## Why this is not just `torch.save`
+
+Saving the weights is the easy part. What makes a resumed run *continue* rather
+than merely *restart from a good place* is everything around them:
+
+| Restored | Why it matters |
+|---|---|
+| Model weights and buffers | the obvious part |
+| Optimizer state | Adam moments; without them the first steps after resume are wrong |
+| LR scheduler | resume at the wrong LR and the loss curve visibly kinks |
+| AMP `GradScaler` | its loss scale is tuned state, not a constant |
+| RNG: torch, CUDA, Python, NumPy | dropout masks and augmentations replay identically |
+| Dataset position | you continue with batch 4001, not batch 1 |
+
+Ravex's test suite asserts the strong form of this: a run killed at step 20 and
+resumed produces losses that are **bit-identical**, step by step, to the run
+that was never interrupted, and the same final weights.
+
+## Two ways to use it
+
+**Zero code changes.** `ravex enable` installs a one-line `.pth` file in
+site-packages, which Python executes at interpreter startup. From then on Ravex
+attaches itself to any training process that has a `ravex.yaml`.
+
+**One line**, when you would rather be explicit:
+
+```python
+import ravex
+ravex.activate()
+```
+
+Both do the same thing. The `.pth` route exists so that a platform can enable
+checkpointing for code it does not own.
+
+## Configuration
+
+`ravex.yaml`, anywhere at or above the working directory:
+
+```yaml
+checkpoint_every: 500        # optimizer steps between checkpoints
+backend: moonclip            # moonclip | torch_save
+storage:
+  type: local                # local | s3 | r2
+  path: ./checkpoints
+keep_last: 5
+max_steps: null              # optional hard stop, see below
+sharded_checkpoints: gather  # gather | per_rank, for FSDP — see below
+```
+
+Every option also reads from `RAVEX_*` environment variables, which win over
+the file — so a scheduler can override a config committed to the repository:
+
+```bash
+RAVEX_CHECKPOINT_EVERY=100 RAVEX_STORAGE_TYPE=r2 RAVEX_STORAGE_BUCKET=runs python train.py
+```
+
+Credentials are never read from the config file. Set `RAVEX_S3_ACCESS_KEY` /
+`RAVEX_S3_SECRET_KEY`, or the usual `AWS_*` pair.
+
+Full reference: [docs/configuration.md](docs/configuration.md).
+
+## Backends
+
+**`moonclip`** (default) — the [Moonclip](https://codeberg.org/JHNMACHINE/moonclip)
+engine: per-tensor delta tracking, so unchanged weights cost zero I/O; zstd
+compression; background writes; direct S3/R2 sync.
+
+**`torch_save`** — one `.pt` file per checkpoint, written on a background
+thread. Used automatically when Moonclip is not installed. Correct, just larger
+and slower.
+
+## How it works
+
+Ravex patches five things in PyTorch and nothing in your code:
+
+- `nn.Module.__init__` and `.train()` — to notice your models
+- `Optimizer.__init__` — to attach a step hook to every optimizer
+- `DataLoader.__init__` and `.__iter__` — to track the dataset position and to
+  find the one moment where a resume can be applied
+
+The step counter advances once per `optimizer.step()`, so gradient accumulation
+needs no special handling. Checkpoints are collected at the *top of an
+iteration*, never inside one: mid-iteration the LR scheduler has not stepped
+yet, and a checkpoint taken there resumes with a stale learning rate.
+
+Collection runs on the training thread — it has to, to be consistent with the
+step that just finished — and copies the state; the write itself happens in the
+background. What the loop pays for is the copy, not the I/O.
+
+More detail: [docs/how-it-works.md](docs/how-it-works.md).
+
+## Safety
+
+Ravex is designed to be un-noticeable when it works and harmless when it does
+not:
+
+- every hook is wrapped; if one raises, your call still returns normally
+- if a checkpoint fails, Ravex disables itself and logs it — training continues
+- nothing is ever written to stdout; logs go to `log_file`, or to stderr at
+  WARNING and above
+- installing the package changes nothing on its own. Without `ravex enable`
+  there is no `.pth`; with it, Ravex still only wakes up for projects that have
+  a `ravex.yaml` or set `RAVEX_ENABLED=1`
+- `ravex disable` removes the autoloader; `RAVEX_ENABLED=0` turns it off for a
+  single run
+
+## Status and limits
+
+Alpha. Works with plain PyTorch loops, and with anything built on them, since
+the hooks are on PyTorch itself.
+
+### With a framework driving the loop
+
+HuggingFace `Trainer` and Lightning are covered by their own tests, and the
+result deserves to be stated precisely rather than as "it works":
+
+- **State restoration is exact.** Model, optimizer, LR scheduler and step count
+  all come back. With the per-step randomness removed, a killed run resumes
+  into a loss sequence identical to the uninterrupted one.
+- **Replay is not.** With shuffling and dropout on, the resumed run continues
+  correctly from the checkpointed state but sees a different draw. Both
+  frameworks iterate the dataloader on their own schedule and consume the
+  global RNG around the loop, so the epoch-start snapshot no longer lines up.
+
+Plain loops, DDP and FSDP *are* bit-exact with randomness on. This is a
+framework-interaction limit, not a general one, and it costs you a different
+shuffle from the resume point onwards — not a wrong model.
+
+Verified: plain loops, gradient accumulation, LR schedulers, AMP loss-scale
+state, `num_workers > 0`, DDP, and FSDP. A killed `torchrun` job resumes on
+*every* rank with bit-identical losses, sharded or not, and the checkpoint it
+leaves behind loads into a plain single-process model afterwards.
+
+### Sharded models
+
+With FSDP each rank holds a slice of every parameter, so `state_dict()` returns
+a fragment. Two ways to turn that into a checkpoint, picked with
+`sharded_checkpoints`:
+
+**`gather`** (default) rebuilds the whole state on rank 0, which writes it. The
+checkpoint is then independent of the topology that produced it — eight GPUs in,
+one out — and it does not scale: rank 0 has to hold the entire model and
+optimizer in host memory, and it is the rank that then does the writing.
+
+**`per_rank`** has every rank write its own shard into its own store,
+`<storage.path>/rank_<n>`. Nothing is gathered, so nothing is bounded by one
+rank's memory, and on a 1.48B model collecting the state went from 15.6 s to
+1.5 s. What you give up is the resharding: those shards are cut for one topology,
+so the checkpoint resumes at the same world size and starts clean at any other.
+Needs FSDP2 — under FSDP1 Ravex degrades to `gather` and says so.
+
+Either way, collecting is a **collective**: every rank participates, and there is
+**no final checkpoint at exit** for a sharded model. Shutdown is where ranks stop
+being in lockstep, and a collective nobody else joins hangs. Losing the last few
+steps is bounded; a hang is not. Set `checkpoint_every` accordingly.
+
+Numbers and the FSDP1 details: [docs/configuration.md](docs/configuration.md).
+
+Known limits today:
+
+- **`IterableDataset`**: no index sampler exists, so the stream position cannot
+  be replayed. Everything else is still restored.
+- **Your loop's bounds**: a resumed script runs its own `for epoch in
+  range(N)` again from the top; it has no idea 3000 steps already happened. Set
+  `max_steps` and Ravex ends the run at the right step regardless of how many
+  times the process restarted.
+
+Under AMP, note that an overflowing gradient makes `scaler.step()` skip the
+optimizer. Ravex counts optimizer steps, not loop iterations, so a skipped
+iteration does not advance the counter — which is the right unit, since nothing
+about the model changed, but it does mean the step count and the number of
+batches you fed differ.
+
+The GPU paths — AMP with real fp16 overflow, the CUDA RNG, FSDP1, NCCL — are
+covered by `integration/test_cuda.py`, which skips without a GPU. They were
+last verified on 8× RTX 5060 Ti with torch 2.12/cu130.
+
+## Development
+
+```bash
+pip install -e ".[dev]"
+pytest
+```
+
+The unit suite runs in-process. The parts that only exist across a real process
+boundary — the `.pth` autoloader, a resume starting from an empty interpreter,
+`torchrun` — live in `integration/` and need Linux:
+
+```bash
+docker build -f integration/Dockerfile -t ravex-integration .
+docker run --rm ravex-integration
+```
+
+## Licence
+
+Apache 2.0. See [LICENSE](LICENSE).