flashruntime 0.4.2__tar.gz → 0.5.0__tar.gz

{flashruntime-0.4.2 → flashruntime-0.5.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: flashruntime
-Version: 0.4.2
+Version: 0.5.0
 Summary: Open fault-tolerant distributed ML runtime: strategy planning, job protocol, leases, checkpointing, and recovery across heterogeneous compute.
 Author: Zolli Labs
 License-Expression: Apache-2.0
@@ -35,6 +35,8 @@ Provides-Extra: artifacts
 Requires-Dist: minio>=7.2; extra == "artifacts"
 Provides-Extra: oss
 Requires-Dist: oss2>=2.18; extra == "oss"
+Provides-Extra: schema
+Requires-Dist: jsonschema>=4.20; extra == "schema"
 Provides-Extra: monitor
 Requires-Dist: psutil>=5.9; extra == "monitor"
 Provides-Extra: service
@@ -194,25 +196,51 @@ DDP run on CPU). The strategy planner's code walkthrough lives in
 An honest benchmark suite (`python -m benchmarks run`) records every result with
 the host that produced it and reports missing comparators as skipped rows, never
 faked. A measured baseline is committed under
-[`benchmarks/results/`](benchmarks/results/). On **Apple M4 (CPU)**:
+[`benchmarks/results/`](benchmarks/results/).
+
+### What it costs to lend your machine
+
+Measured on an **Apple M4, 16 GB, macOS, CPU only**. Every sentence below is
+*generated* from the committed result file by
+[`benchmarks/headline.py`](benchmarks/headline.py), never typed here by hand: if
+the README drifts from the JSON,
+`tests/test_headline.py::test_readme_quotes_the_committed_numbers` fails.
+
+Waiting for work, the host agent:
+
+- uses 0.13% of your computer's memory while waiting for work
+  <br><sub>measured on a 16 GB machine; the megabyte figure is fixed, so the percentage is larger on a smaller machine</sub>
+- and under 0.1% of one processor core
+  <br><sub>set by --poll-seconds (1.0s); a faster cadence costs more</sub>
+
+Running someone else's job on it:
+
+- 0 of 24 programs written to break out reached the owner's files, network or disk — each one first verified to succeed when the protection was switched off
+  <br><sub>these named attacks did not get out; that is not proof the sandbox is unescapable — see SECURITY.md</sub>
+- the safety measures cost the job's owner nothing we can measure at this workload (between 6% faster and 1% slower)
+  <br><sub>negative because --network none skips network setup the bare comparison pays; workload-specific, and it is charged to the job's owner, never to the person lending the machine</sub>
+
+### What the runtime does for a job
 
 | Scenario | Measured | Read it honestly |
 |---|---|---|
-| Launch overhead | **+0.04 s** | `flash.submit` vs bare `torchrun`, paired. A one-time fixed cost, not per-step. |
-| Checkpoint cost | **−1.2 ms / checkpoint** (p90 6.2 ms) | A few-KB state dict; the median delta is *below* the run-to-run noise floor at this size (it goes slightly negative), so checkpoints are effectively free here — the p90 bounds it. A larger model surfaces a real positive cost. |
 | Auto-resume | **40 steps not recomputed** | The size-*independent* guarantee: resume never re-does work past the last valid checkpoint. |
 | Adoption | **7 lines** | To make a vanilla script framework-ready (difflib vs each project's own docs). |
-| Resilience | **5/5 faults handled · 20/20 integrity under `kill -9`** | Every fault type routed to the right typed recovery; a checkpoint survives a mid-write `SIGKILL` on 20/20 kills (naive `torch.save`: 20/20 corrupted). 16-trial storm, half crash-armed: 16/16 completed, all 8 crashed trials auto-resumed, 0 manual. Dead-worker MTTD **3.05 s**, MTTR **~3.5 ms**. |
-
-The *timing* rows are smoke-scale on one host: the models are tiny, so the
-*wall-clock* saved by recovery is at the noise floor here. The figure that
-scales is `steps_not_recomputed`, not seconds — the seconds-saved grows with the
-compute between the last checkpoint and the crash (negligible at smoke size,
-hours on a real job). The resilience counts are exact, not smoke-diluted: they
-are *counted* from real fault injection (`kill -9` in the checkpoint-write
-window, a storm of crash-armed trials, a killed lease worker), and the full
-Resilience table with per-scenario method lives in
-[the benchmarks page](docs/site/benchmarks.md). Full provenance and caveats live
+| Resilience | **5/5 faults handled · 20/20 integrity under `kill -9`** | Every fault type routed to the right typed recovery; a checkpoint survives a mid-write `SIGKILL` on 20/20 kills (naive `torch.save`: 20/20 corrupted). 16-trial storm, half crash-armed: 16/16 completed, all 8 crashed trials auto-resumed, 0 manual. |
+
+Two rows that used to sit here — a `+0.04 s` launch cost and a
+`−1.2 ms / checkpoint` saving — have been **removed rather than updated**:
+neither reproduced when re-measured, and a number that no longer reproduces is
+not a smaller number, it is not a result. The figure that scales with a real job
+is `steps_not_recomputed`, not seconds; the seconds saved grow with the compute
+between the last checkpoint and the crash (negligible at smoke size, hours on a
+real job). The resilience counts are exact, not smoke-diluted: they are
+*counted* from real fault injection (`kill -9` in the checkpoint-write window, a
+storm of crash-armed trials, a killed worker).
+
+Full statistics — every percentile, every comparator, and the technical units
+the sentences above deliberately avoid — live on
+[the benchmarks page](docs/site/benchmarks.md); full provenance and caveats live
 in the result file.
 
 ---

{flashruntime-0.4.2 → flashruntime-0.5.0}/README.md

@@ -140,25 +140,51 @@ DDP run on CPU). The strategy planner's code walkthrough lives in
 An honest benchmark suite (`python -m benchmarks run`) records every result with
 the host that produced it and reports missing comparators as skipped rows, never
 faked. A measured baseline is committed under
-[`benchmarks/results/`](benchmarks/results/). On **Apple M4 (CPU)**:
+[`benchmarks/results/`](benchmarks/results/).
+
+### What it costs to lend your machine
+
+Measured on an **Apple M4, 16 GB, macOS, CPU only**. Every sentence below is
+*generated* from the committed result file by
+[`benchmarks/headline.py`](benchmarks/headline.py), never typed here by hand: if
+the README drifts from the JSON,
+`tests/test_headline.py::test_readme_quotes_the_committed_numbers` fails.
+
+Waiting for work, the host agent:
+
+- uses 0.13% of your computer's memory while waiting for work
+  <br><sub>measured on a 16 GB machine; the megabyte figure is fixed, so the percentage is larger on a smaller machine</sub>
+- and under 0.1% of one processor core
+  <br><sub>set by --poll-seconds (1.0s); a faster cadence costs more</sub>
+
+Running someone else's job on it:
+
+- 0 of 24 programs written to break out reached the owner's files, network or disk — each one first verified to succeed when the protection was switched off
+  <br><sub>these named attacks did not get out; that is not proof the sandbox is unescapable — see SECURITY.md</sub>
+- the safety measures cost the job's owner nothing we can measure at this workload (between 6% faster and 1% slower)
+  <br><sub>negative because --network none skips network setup the bare comparison pays; workload-specific, and it is charged to the job's owner, never to the person lending the machine</sub>
+
+### What the runtime does for a job
 
 | Scenario | Measured | Read it honestly |
 |---|---|---|
-| Launch overhead | **+0.04 s** | `flash.submit` vs bare `torchrun`, paired. A one-time fixed cost, not per-step. |
-| Checkpoint cost | **−1.2 ms / checkpoint** (p90 6.2 ms) | A few-KB state dict; the median delta is *below* the run-to-run noise floor at this size (it goes slightly negative), so checkpoints are effectively free here — the p90 bounds it. A larger model surfaces a real positive cost. |
 | Auto-resume | **40 steps not recomputed** | The size-*independent* guarantee: resume never re-does work past the last valid checkpoint. |
 | Adoption | **7 lines** | To make a vanilla script framework-ready (difflib vs each project's own docs). |
-| Resilience | **5/5 faults handled · 20/20 integrity under `kill -9`** | Every fault type routed to the right typed recovery; a checkpoint survives a mid-write `SIGKILL` on 20/20 kills (naive `torch.save`: 20/20 corrupted). 16-trial storm, half crash-armed: 16/16 completed, all 8 crashed trials auto-resumed, 0 manual. Dead-worker MTTD **3.05 s**, MTTR **~3.5 ms**. |
-
-The *timing* rows are smoke-scale on one host: the models are tiny, so the
-*wall-clock* saved by recovery is at the noise floor here. The figure that
-scales is `steps_not_recomputed`, not seconds — the seconds-saved grows with the
-compute between the last checkpoint and the crash (negligible at smoke size,
-hours on a real job). The resilience counts are exact, not smoke-diluted: they
-are *counted* from real fault injection (`kill -9` in the checkpoint-write
-window, a storm of crash-armed trials, a killed lease worker), and the full
-Resilience table with per-scenario method lives in
-[the benchmarks page](docs/site/benchmarks.md). Full provenance and caveats live
+| Resilience | **5/5 faults handled · 20/20 integrity under `kill -9`** | Every fault type routed to the right typed recovery; a checkpoint survives a mid-write `SIGKILL` on 20/20 kills (naive `torch.save`: 20/20 corrupted). 16-trial storm, half crash-armed: 16/16 completed, all 8 crashed trials auto-resumed, 0 manual. |
+
+Two rows that used to sit here — a `+0.04 s` launch cost and a
+`−1.2 ms / checkpoint` saving — have been **removed rather than updated**:
+neither reproduced when re-measured, and a number that no longer reproduces is
+not a smaller number, it is not a result. The figure that scales with a real job
+is `steps_not_recomputed`, not seconds; the seconds saved grow with the compute
+between the last checkpoint and the crash (negligible at smoke size, hours on a
+real job). The resilience counts are exact, not smoke-diluted: they are
+*counted* from real fault injection (`kill -9` in the checkpoint-write window, a
+storm of crash-armed trials, a killed worker).
+
+Full statistics — every percentile, every comparator, and the technical units
+the sentences above deliberately avoid — live on
+[the benchmarks page](docs/site/benchmarks.md); full provenance and caveats live
 in the result file.
 
 ---

flashruntime-0.5.0/flashml_workloads/chunks.py

@@ -0,0 +1,137 @@
+"""Chunk arithmetic for elastic work distribution.
+
+A pass over the data is cut into `total_chunks` UNIFORM chunks. Uniformity
+is not a convenience: `fedavg_weights` clips any contribution whose norm
+exceeds 3x the round's median, and that cap is only silent on an honest
+round while every contribution covers the same amount of data. Sizing a
+machine's work to its own speed would make a fast machine's honest delta
+look like an attack. So heterogeneity is expressed as how MANY chunks a
+machine completes, never how big one is.
+
+Pure stdlib, no I/O: this module is imported both by the driver (which runs
+inside the cloud API and must not grow dependencies) and by the worker
+(which runs inside a --network none container).
+"""
+
+from __future__ import annotations
+
+from typing import Iterable
+
+__all__ = [
+    "ALLOTMENT_HEADROOM",
+    "chunk_count",
+    "chunk_sequence",
+    "coverage",
+    "rotation_start",
+    "slot_start",
+]
+
+#: How much more work a task slot is allotted than its share of the round.
+#: Slots are offers, not assignments — most rounds will have slots nobody
+#: claims — so a claimed slot must be able to carry more than its even share
+#: or a round with few machines could never reach coverage. 2.0 is a
+#: starting point to be replaced by measured throughput once round 1 has
+#: run; it is not tuned. [assumption]
+ALLOTMENT_HEADROOM: float = 2.0
+
+
+def chunk_count(dataset_size: int, target_chunk_size: int) -> int:
+    """How many chunks one pass over `dataset_size` examples becomes.
+
+    Rounds UP: a remainder smaller than `target_chunk_size` is still data,
+    and dropping it would silently train on less than the pass the caller
+    asked for.
+    """
+    if dataset_size < 1:
+        raise ValueError(f"dataset_size must be >= 1, got {dataset_size}")
+    if target_chunk_size < 1:
+        raise ValueError(f"target_chunk_size must be >= 1, got {target_chunk_size}")
+    return -(-dataset_size // target_chunk_size)
+
+
+def rotation_start(slot_index: int, total_chunks: int, slots: int) -> int:
+    """Where task slot `slot_index` begins its walk through the chunks.
+
+    Slots are spread evenly so two machines working at the same speed cover
+    disjoint data — they only collide once one has wrapped nearly a full
+    pass, by which point the round is close to complete anyway. Overlap is
+    wasteful but never wrong: `coverage` counts distinct chunks.
+    """
+    if total_chunks < 1:
+        raise ValueError(f"total_chunks must be >= 1, got {total_chunks}")
+    if slots < 1:
+        raise ValueError(f"slots must be >= 1, got {slots}")
+    return (slot_index * total_chunks // slots) % total_chunks
+
+
+def slot_start(slot_index: int, total_chunks: int, slots: int,
+               chunk_offset: int = 0) -> int:
+    """Where slot `slot_index` of a round at `chunk_offset` begins.
+
+    `rotation_start` spreads the slots of ONE round apart from each other.
+    It is a pure function of the slot index, so on its own it hands slot 0
+    the identical chunk sequence in every round of a run — and since a round
+    closes as soon as it hits its coverage target (which is below a full
+    pass whenever `sync_every < 1.0`) and leases are FIFO, the same low slot
+    indices are claimed each round and the same prefix of the data is
+    trained over and over while the tail is never trained at all. With
+    `total_chunks=100, slots=20, sync_every=0.5` and two claimants, every
+    round covers chunks 0–54 and reports a healthy `covered=0.55`; chunks
+    55–99 are never touched by any round of the run.
+
+    `chunk_offset` is what advances the whole pattern between rounds: the
+    driver sets it to the round's index times the round's coverage TARGET,
+    so round r+1 starts where round r's target ended.
+
+    That advance is MODULAR, not a one-way march. Offsets are
+    `(r * target) % total_chunks`, so they cycle: at `total_chunks=100,
+    target=50` the sequence is 0, 50, 0, 50, … The distinct offsets are
+    spaced `gcd(target, total_chunks)` apart and each round covers `target`
+    chunks, which is never smaller than that gap — so one cycle is one full
+    pass over the data, and the cycle then repeats for the next epoch.
+    Coverage is guaranteed within a pass, not across a whole run; a run of
+    many epochs revisits the same offsets by design.
+
+    This function exists so the offset is applied in exactly one place. Two
+    callers need it — `service/modea._expand_fedavg` (which stamps
+    `start_chunk` into each slot's payload) and `fedavg_driver` (which
+    rebuilds the same allowed set to verify what a slot reports). Drift
+    between them is graded, not binary: a one-chunk disagreement trims an
+    id off each end of every allotment and the round survives degraded,
+    while a large one intersects to nothing, weighs every contribution zero
+    and kills the round. `test_a_drifting_expansion_start_is_not_silently
+    _credited` exercises the second.
+    """
+    return (rotation_start(slot_index, total_chunks, slots)
+            + int(chunk_offset)) % total_chunks
+
+
+def chunk_sequence(start: int, total_chunks: int, budget: int) -> list[int]:
+    """The ordered chunk ids a task walks, at most `budget` of them.
+
+    Cycles: a machine fast enough to finish a whole pass wraps to the start
+    rather than sitting idle for the rest of the round. Repeating a chunk
+    re-trains on data someone may already have covered, which costs effort
+    and not correctness.
+    """
+    if total_chunks < 1:
+        raise ValueError(f"total_chunks must be >= 1, got {total_chunks}")
+    if budget < 0:
+        raise ValueError(f"budget must be >= 0, got {budget}")
+    return [(start + i) % total_chunks for i in range(budget)]
+
+
+def coverage(completed: Iterable[Iterable[int]], total_chunks: int) -> float:
+    """Fraction of the pass covered by the union of every machine's chunks.
+
+    Union, not sum: two machines that both did chunk 7 covered one chunk of
+    data between them. This is the number a round closes on, so counting
+    effort instead of data would let a round "complete" having trained on a
+    fraction of the pass.
+    """
+    if total_chunks < 1:
+        raise ValueError(f"total_chunks must be >= 1, got {total_chunks}")
+    distinct: set[int] = set()
+    for ids in completed:
+        distinct.update(ids)
+    return len(distinct) / total_chunks