pytscope 0.2.1__tar.gz

pytscope-0.2.1/.gitignore

@@ -0,0 +1,13 @@
+__pycache__/
+*.py[cod]
+*.egg-info/
+.eggs/
+build/
+dist/
+.pytest_cache/
+.coverage
+htmlcov/
+.venv/
+venv/
+runs/
+.DS_Store

pytscope-0.2.1/CHANGELOG.md

@@ -0,0 +1,162 @@
+# Changelog
+
+All notable changes to this project are documented here. The format is based on
+[Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and this project
+adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+
+## [0.2.1] - 2026-06-08
+
+### Changed
+- **Project renamed from `trainscope` to `pytscope`.** Both `trainscope` and
+  `tscope` were already taken on PyPI; `pytscope` was available. The import
+  path, CLI command, PyPI distribution name, and GitHub repository
+  (`Sumu004/pytscope`) are now all `pytscope`. No functional changes —
+  this release exists solely to publish under the new name (the v0.2.0 tag
+  was cut from the pre-rename tree, so its build still carried the old name).
+
+## [0.2.0] - 2026-06-08
+
+### Added
+- **Amber-LED "hardware panel" terminal reports.** `pytscope analyze`/`diff`
+  now render every section as a lit panel — a colored `●` indicator (red /
+  amber / green by what it's reporting: a stalling step-time breakdown, a
+  named straggler, a low MFU, …) in front of each heading — plus gradient
+  meter bars rendered as lit/unlit block segments (`█`/`░`), severity-coded
+  findings, and unicode sparklines for step-time and loss trends. The whole
+  grammar (LED color ↔ severity ↔ bar gradient) is consistent across every
+  vertical, and the run summary is condensed onto one heading line (steps ·
+  ms/step · throughput · median/p95/CV) with the old double-blank-line
+  padding between sections removed — same information, faster to scan.
+  Pure ANSI escapes, no new deps: auto-detects whether the terminal can
+  render color (honoring `NO_COLOR`/`FORCE_COLOR` and the new `--color
+  {auto,always,never}` flag), and degrades to byte-identical plain text when
+  piped to a file, redirected in CI, or explicitly disabled — nothing is
+  ever written to disk beyond the run itself (`tests/test_cli_report.py`).
+
+### Validated
+- **Real multi-GPU NCCL run (Kaggle, 2× T4, 2026-06-08)** — closes two of the
+  three "needs hardware" gaps in `docs/VALIDATION.md`:
+  - **Straggler attribution: exact pass.** `DIST.STRAGGLER` named the injected
+    rank correctly (z=14.1, 100% critical-path share vs 50% expected by chance,
+    27.5% wall lost to imbalance).
+  - **Exposed communication: directionally confirmed**, plus a genuine finding
+    about the hardware itself — on PCIe-only GPU pairs (no NVLink) the
+    all-reduce is link-bandwidth-bound, so absolute exposed-comm time stays
+    roughly constant across batch sizes (`DIST.EXPOSED_COMM` correctly fired
+    HIGH for both the small- and large-batch configs: 72% vs 62% exposed,
+    overlap improving in the predicted direction). Documented as an
+    interconnect-topology caveat in `docs/VALIDATION.md`.
+  - Full report, raw console captures, and analysis:
+    [`docs/validation-runs/2026-06-08-kaggle-2xT4/RESULTS.md`](docs/validation-runs/2026-06-08-kaggle-2xT4/RESULTS.md).
+
+### Fixed
+- **`examples/efficiency_mfu.py` never selected CUDA** — its device probe only
+  checked `mps`/`cpu`, so on a CUDA box it silently profiled the CPU and
+  reported a meaningless ~0% MFU anchored against an A100 peak it never
+  touched (caught by the Kaggle validation run above). Now checks `cuda` first
+  and, when on CUDA, leaves `peak_flops` unset so `AutoProfiler` looks the
+  actual device up in the hardware peak table instead of hard-coding an
+  A100 anchor.
+
+### Added
+- **`docs/validation-runs/`** — a ready-to-run Kaggle notebook
+  (`kaggle_2xT4.ipynb`) plus step-by-step instructions that execute the full
+  multi-GPU validation protocol from `docs/VALIDATION.md` (straggler
+  attribution, exposed-comm overlap, MFU sanity) on **real NCCL/CUDA, for
+  free**, using Kaggle's 2× T4 notebook tier — no paid GPU rental needed.
+
+### Fixed
+- **CLI crashed on Windows** (`UnicodeEncodeError: 'charmap' codec can't encode
+  character 'Δ'`) whenever a report containing `Δ`/`—`/`•` was printed
+  with the console in its default `cp1252` encoding — this is what every CI
+  run on `windows-latest` was hitting (`pytscope diff`'s metrics table header
+  uses `Δ`). `pytscope.cli.main` now re-points stdout/stderr at a UTF-8
+  encoder with `errors="replace"` on entry; a no-op on platforms already UTF-8.
+
+### Changed
+- **AutoProfiler is now correct on real models.** A forward re-entrancy guard
+  means gradient accumulation records one step per `optimizer.step` (not per
+  micro-batch), and activation checkpointing (forward recomputed during backward)
+  no longer corrupts the step structure. Tested with both.
+- Added `docs/VALIDATION.md` (what's validated where + a multi-GPU protocol) and
+  made `examples/ddp_gloo.py` run under `torchrun` on NCCL/CUDA as well as CPU
+  gloo, so the GPU validation is executable.
+
+### Added
+- **Training Efficiency Budget (MFU)** — a single accounting identity that
+  decomposes attributed wall time into named line items (useful compute /
+  compute overhead / data stall / communication / other) that sum **exactly** to
+  wall, anchored at the top by Model FLOPs Utilization. FLOPs are counted
+  automatically (`AutoProfiler(measure_flops=True)` via torch FlopCounterMode);
+  peak comes from a built-in GPU table (`hardware.peak_flops_for`) or
+  `--peak-tflops`. Recoverable line items are ranked by payoff; `EFFICIENCY.LOW_MFU`
+  (anchored) / `EFFICIENCY.RECOVERABLE` (no anchor) rules point at the biggest win.
+  CLI: `analyze --flops-per-step --peak-tflops`.
+- **Exposed-communication analysis** — ingest a `torch.profiler`/Kineto trace
+  (`analyze --trace`, or auto-detected `trace.json[.gz]` in the run dir) and
+  compute, via exact interval arithmetic, how much collective communication
+  overlaps compute vs is *exposed* (on the critical path). Reports overlap
+  efficiency and a per-collective breakdown; new `DIST.EXPOSED_COMM` rule. The
+  overlap math is exact (tested on synthetic traces with known answers); the
+  parser is validated against a real `torch.profiler` export.
+- **Automatic instrumentation** — `AutoProfiler(run_dir, model, optimizer)`
+  captures the full phase timeline (data / forward / backward / optimizer, plus
+  synchronous `comm`) with **zero changes to the training loop**, via PyTorch
+  forward hooks + an `optimizer.step` wrapper + collective patching. The step is
+  held open for post-step `log(loss=…)`. All hooks/patches are restored on
+  `finish()`. Assumes one forward/backward per step (use `Profiler` for gradient
+  accumulation).
+- **Distributed vertical (headline)** — multi-rank critical-path analysis for
+  data-parallel training. `Profiler(distributed=True)` records every rank to
+  `run_dir/rank{k}/`; the analyzer aligns ranks on one timeline and computes
+  critical-path wall loss, communication fraction, sync skew, and load imbalance.
+- **Statistical straggler detection** — identifies a *persistent* straggler rank
+  via a binomial-persistence test (is one rank consistently the critical path,
+  beyond chance?), not a fixed threshold. Rules: `DIST.STRAGGLER`,
+  `DIST.LOAD_IMBALANCE`, `DIST.COMM_BOUND`.
+- **Pipeline-bubble analyzer** — measures achieved bubble from a per-stage
+  schedule and compares it to the inherent GPipe minimum `(p-1)/(m+p-1)`, so it
+  flags only *excess* bubble (`DIST.PIPELINE_BUBBLE`). Reproduces the closed form
+  exactly (tested across p, m).
+- **`comm()` context manager** to attribute collective time to a `comm` phase.
+- **Real `examples/ddp_gloo.py`** — runs genuine multi-process gloo DDP (CPU, no
+  GPU needed) with an injectable straggler; `pytscope analyze` then identifies
+  it. Backed by a real multi-process integration test.
+- CLI `analyze` auto-detects multi-rank run directories.
+
+### Changed
+- Packaging: version is now single-sourced from `pytscope.__version__` (Hatch
+  dynamic version); `docs/` added to the sdist.
+
+## [0.1.0] - 2026-06-06
+
+First public beta. One telemetry backbone feeding four analysis verticals plus a
+cross-signal diagnosis engine.
+
+### Added
+- **Profiler** — live, integer-nanosecond timing core with `step()` / `mark()`
+  primitives, `iter_data()` dataloader timing, scalar logging, and optional
+  device-memory capture. ~3 µs/step overhead.
+- **Integrations** — one-line PyTorch Lightning and Hugging Face `Trainer`
+  callbacks; DDP rank-aware (non-zero ranks no-op by default).
+- **Timing vertical** — per-step attribution to data / forward / backward /
+  optimizer with median/p95 and rules for dataloader-bound, backward-heavy,
+  optimizer-heavy, and jitter.
+- **Memory vertical** — CUDA + Apple MPS capture; fragmentation and
+  leak/growth detection.
+- **Convergence vertical** — loss trend, divergence (NaN/Inf), and robust
+  local-window spike detection for loss and grad-norm.
+- **Cross-signal rule** — correlates spikes across loss / grad-norm / step-time /
+  memory on one aligned timeline (the headline diagnostic).
+- **Reproducibility vertical** — `pytscope diff A B` compares provenance,
+  config, and outcomes; diagnoses nondeterminism and finds the first divergence
+  step.
+- **CLI** — `pytscope analyze` and `pytscope diff`.
+- Pure-stdlib core; CUDA/MPS/CPU examples; 58 tests.
+
+[Unreleased]: https://github.com/Sumu004/pytscope/compare/v0.2.1...HEAD
+[0.2.1]: https://github.com/Sumu004/pytscope/releases/tag/v0.2.1
+[0.2.0]: https://github.com/Sumu004/pytscope/releases/tag/v0.2.0
+[0.1.0]: https://github.com/Sumu004/pytscope/releases/tag/v0.1.0

pytscope-0.2.1/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Sumukh Chaluvaraju
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

pytscope-0.2.1/PKG-INFO

@@ -0,0 +1,367 @@
+Metadata-Version: 2.4
+Name: pytscope
+Version: 0.2.1
+Summary: An intelligence layer for ML training: live profiling + post-training diagnosis.
+Project-URL: Homepage, https://github.com/Sumu004/pytscope
+Project-URL: Repository, https://github.com/Sumu004/pytscope
+Project-URL: Documentation, https://github.com/Sumu004/pytscope/tree/main/docs
+Project-URL: Issues, https://github.com/Sumu004/pytscope/issues
+Project-URL: Changelog, https://github.com/Sumu004/pytscope/blob/main/CHANGELOG.md
+Author-email: Sumukh Chaluvaraju <sumukhchaluvaraj@gmail.com>
+Maintainer-email: Sumukh Chaluvaraju <sumukhchaluvaraj@gmail.com>
+License: MIT
+License-File: LICENSE
+Keywords: deep-learning,machine-learning,mlops,observability,performance,profiling,pytorch,training
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: System :: Monitoring
+Classifier: Typing :: Typed
+Requires-Python: >=3.9
+Provides-Extra: dev
+Requires-Dist: build; extra == 'dev'
+Requires-Dist: pytest-cov; extra == 'dev'
+Requires-Dist: pytest>=7; extra == 'dev'
+Requires-Dist: ruff>=0.4; extra == 'dev'
+Requires-Dist: twine; extra == 'dev'
+Provides-Extra: huggingface
+Requires-Dist: transformers>=4.30; extra == 'huggingface'
+Provides-Extra: lightning
+Requires-Dist: lightning>=2.0; extra == 'lightning'
+Provides-Extra: torch
+Requires-Dist: torch>=2.0; extra == 'torch'
+Description-Content-Type: text/markdown
+
+<div align="center">
+
+# pytscope
+
+**An intelligence layer for ML training — go beyond collecting metrics to *explaining* them.**
+
+[![CI](https://github.com/Sumu004/pytscope/actions/workflows/ci.yml/badge.svg)](https://github.com/Sumu004/pytscope/actions/workflows/ci.yml)
+[![PyPI version](https://img.shields.io/pypi/v/pytscope.svg)](https://pypi.org/project/pytscope/)
+[![Python versions](https://img.shields.io/pypi/pyversions/pytscope.svg)](https://pypi.org/project/pytscope/)
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
+
+[Quickstart](#quickstart) · [Why it's different](#why-its-different) ·
+[Demos](#try-the-demos) · [Validation](#status--validation) · [Docs](#documentation)
+
+</div>
+
+Standard profilers hand you a 50 MB trace and leave the "so what do I change?"
+to you. `pytscope` captures **timing, memory, convergence signals, and
+provenance on one aligned per-step timeline**, then runs a diagnosis engine that
+turns the raw numbers into ranked, actionable findings.
+
+```
+● TIMING — 95 steps · 23.1 ms/step · 43.3 steps/s   (median 22.0 · p95 28.4 ms · CV 0.18)
+  step time  ▃▄▅▃▂▃▄▆▃▂▃▄▅▃▂  (low→high)
+  data       ████████████████░░░░░░░░░░░░░░  52.0%    12.01 ms
+  forward    █████████░░░░░░░░░░░░░░░░░░░░░  17.3%     4.00 ms
+  backward   ██████████████░░░░░░░░░░░░░░░░  26.0%     6.01 ms
+  optimizer  █░░░░░░░░░░░░░░░░░░░░░░░░░░░░░   4.3%     1.00 ms
+
+● FINDINGS (1)
+  [HIGH] Input pipeline is a bottleneck  (TIMING.DATALOADER_BOUND)
+        52% of step time is spent fetching data (12.0 ms/step). The
+        accelerator is stalling on the dataloader.
+        -> Raise DataLoader num_workers, set persistent_workers=True and
+           pin_memory=True, prefetch, or move heavy transforms off the hot path.
+```
+
+In a real terminal, each `●` is a lit indicator that's colored red, amber, or
+green by what it's reporting — the same "hardware panel" grammar carried
+through every section, gradient meter bar, and severity tag.
+
+## The headline: a Training Efficiency Budget
+
+Most profilers hand you a list of findings. pytscope also gives you a single
+**accounting identity** — every second of training, decomposed into named line
+items that provably sum to your measured wall time, anchored to hardware peak
+(**MFU**, Model FLOPs Utilization):
+
+```
+● EFFICIENCY BUDGET — wall-time decomposition
+  MFU 38.0%  ·  useful compute 38.0% of 142.00s wall
+  useful_compute    ███████████░░░░░░░░░░░░░░░░░░  38.0%   53.96s
+  compute_overhead  █████░░░░░░░░░░░░░░░░░░░░░░░░  16.0%   22.72s (recoverable)
+  data_stall        ████████░░░░░░░░░░░░░░░░░░░░░  27.0%   38.34s (recoverable)
+  communication     █████░░░░░░░░░░░░░░░░░░░░░░░░  19.0%   26.98s (recoverable)
+
+  [HIGH] MFU is 38% — 62% of wall is recoverable  (EFFICIENCY.LOW_MFU)
+        Biggest recoverable line: data_stall at 27% of wall.
+        -> Start with data_stall: raise num_workers, persistent_workers, prefetch.
+```
+
+Because the phase timeline partitions each step, the decomposition is **exact** —
+the line items sum to wall with no fudge factor, which makes the model
+falsifiable. And every recoverable line is *seconds you can win back*, so fixes
+rank themselves by payoff. FLOPs are counted automatically
+(`AutoProfiler(measure_flops=True)`); peak comes from a built-in GPU table or
+`--peak-tflops`.
+
+```bash
+python examples/efficiency_mfu.py && pytscope analyze runs/mfu
+```
+
+## Why it's different
+
+One backbone, four lenses. Every analyzer reads the same `StepRecord` timeline,
+so findings can **cross-correlate signals no single existing tool aligns**:
+
+| Vertical | Status |
+|----------|--------|
+| **Distributed** — multi-rank critical-path, straggler & comm/pipeline-bubble analysis | ✅ |
+| **Timing** — attribute step time to data / fwd / bwd / optimizer | ✅ |
+| **Convergence** — loss/grad-norm trend, divergence, spikes | ✅ |
+| **Memory** — peak attribution, fragmentation, leak/growth | ✅ |
+| **Cross-signal** — correlate spikes across all axes on one timeline | ✅ |
+| **Reproducibility** — provenance capture + run-vs-run diff & drift diagnosis | ✅ |
+
+The core is **pure-stdlib** — no heavy deps to profile your training.
+
+### The headline: a finding no single-axis tool can make
+
+```
+● FINDINGS (1)
+  [HIGH] Correlated instability at steps 70–72  (CROSS.CORRELATED_INSTABILITY)
+        At steps 70–72, 3 independent axes spike simultaneously (grad_norm,
+        loss, step_time): loss=3.579, grad_norm=45, step_time=25.6ms.
+        Co-occurrence across axes is strong evidence of a real optimization
+        event, not noise.
+        -> Inspect the LR schedule, gradient clipping, and the batch around
+           steps 70–72. A simultaneous loss + grad-norm spike usually means
+           the update blew up (LR too high / bad batch).
+```
+
+HTA sees only timing; Cockpit only gradients; W&B only logged scalars. pytscope
+sees them **on one clock** and reports the correlation. Reproduce it with
+`python examples/cross_signal.py && pytscope analyze runs/cross`.
+
+### Distributed: the straggler no single-rank profiler can name
+
+In synchronous data-parallel training every rank waits at the gradient
+all-reduce for the **slowest** rank. That idle time is pure waste, and it's
+invisible to any single-rank profiler — you only see it by putting all ranks on
+one timeline. pytscope does, and uses a **statistical persistence test** (not a
+threshold) to tell a genuine bad node from noise:
+
+```
+● DISTRIBUTED — 4 ranks, 60 aligned steps
+  wall lost to imbalance 18.6% · median sync skew 4.7 ms/step
+    rank 0:  10.0 ms ·   0% (z=-4.5)
+    rank 2:  12.0 ms ·  99% (z=+13.4)  <- straggler
+
+● FINDINGS (1)
+  [HIGH] Rank 2 is a persistent straggler  (DIST.STRAGGLER)
+        Rank 2 is the slowest (critical-path) rank in 99% of steps across 4
+        ranks (expected 25% by chance; z=13.4) and runs 20% slower than the
+        median rank. Synchronous all-reduce makes every other rank wait for it
+        — 18.6% of wall time is lost to this imbalance.
+        -> Investigate rank 2's device/host: thermal throttling, a slower GPU,
+           NUMA placement, or an unbalanced data shard.
+```
+
+This is a **real** distributed system — reproduce it on your laptop (CPU, no GPU)
+with genuine multi-process gloo all-reduce:
+
+```bash
+pip install -e ".[torch]"
+python examples/ddp_gloo.py --ranks 4 --straggler-rank 2
+pytscope analyze runs/ddp_gloo
+```
+
+For **pipeline parallelism**, pytscope measures the achieved bubble and compares
+it to the inherent GPipe minimum `(p-1)/(m+p-1)`, so it flags only the *excess*
+bubble you can actually fix — not the bubble that's just the cost of your `p`
+and `m`.
+
+### Exposed communication: the metric that decides large-scale efficiency
+
+Gradient all-reduce *can* run concurrently with backward compute — the part that
+overlaps is free, the part that doesn't is **exposed** and sits on the critical
+path. pytscope ingests a `torch.profiler`/Kineto trace and computes the split
+exactly (interval arithmetic over the kernel timeline):
+
+```
+● COMMUNICATION OVERLAP — from kernel trace
+  comm 36.0 ms · overlapped 67% · exposed 12.0 ms (20% of wall)
+
+  [HIGH] Communication is not overlapped with compute  (DIST.EXPOSED_COMM)
+        20% of wall time is exposed communication. Only 67% of the 36.0 ms of
+        communication is hidden behind compute.
+        -> DDP gradient bucketing (bucket_cap_mb), overlap optimizer/all-reduce,
+           or increase per-GPU compute so backward hides the all-reduce.
+```
+
+```bash
+pytscope analyze runs/job --trace trace.json   # from torch.profiler
+python examples/exposed_comm.py && pytscope analyze runs/trace_demo  # no GPU
+```
+
+---
+
+## Overhead
+
+Measured on `tests/test_overhead.py` (run `pytest -s`):
+
+| Path | Cost |
+|------|------|
+| Pure instrumentation (begin/mark×3/end) | **~0.7 µs/step** |
+| End-to-end incl. JSONL disk write | **~3 µs/step** |
+| Disabled DDP rank (no-op) | **~0.06 µs/step** |
+
+On a 50 ms training step that's **~0.006% overhead** — versus trace-dumping
+profilers (Kineto/HTA) that add real overhead and emit multi-MB artifacts.
+Memory bounded (live writer retains nothing); batched flushes; DDP-safe.
+
+## Install
+
+```bash
+pip install -e ".[dev]"          # core + tests
+pip install -e ".[torch,lightning,huggingface]"   # framework integrations
+```
+
+## Quickstart
+
+**Automatic — zero changes to your loop (recommended):**
+
+```python
+from pytscope.auto import AutoProfiler
+
+prof = AutoProfiler("runs/exp1", model, optimizer, warmup=10)
+prof.start()
+for x, y in loader:                           # <- your loop, untouched
+    loss = loss_fn(model(x), y)
+    loss.backward()
+    optimizer.step(); optimizer.zero_grad()
+prof.finish()
+```
+
+`AutoProfiler` registers PyTorch hooks (forward, `optimizer.step`, and
+synchronous collectives) to attribute **data / forward / backward / optimizer /
+comm** automatically — no `mark()` calls anywhere in your training code. All
+hooks/patches are removed on `finish()`.
+
+**Manual loop** (full control, or gradient accumulation):
+
+```python
+from pytscope import Profiler
+
+prof = Profiler("runs/exp1", warmup=10)
+prof.start()
+for batch in prof.iter_data(loader):          # times data fetch
+    with prof.step():
+        loss = loss_fn(model(batch))
+        prof.mark("forward")
+        loss.backward();   prof.mark("backward")
+        opt.step(); opt.zero_grad(); prof.mark("optimizer")
+prof.finish()
+```
+
+**Lightning (one line):**
+
+```python
+from pytscope.integrations.lightning import PytscopeCallback
+trainer = pl.Trainer(callbacks=[PytscopeCallback("runs/exp1")])
+```
+
+**Hugging Face (one line):**
+
+```python
+from pytscope.integrations.huggingface import PytscopeCallback
+trainer = Trainer(..., callbacks=[PytscopeCallback("runs/exp1")])
+```
+
+**Then analyze, or compare two runs:**
+
+```bash
+pytscope analyze runs/exp1
+pytscope diff runs/exp1 runs/exp2   # reproducibility / drift: why do they differ?
+```
+
+Reports lean into a compact, amber-LED hardware-panel aesthetic — every
+section is a "lit panel" (a colored ● indicator that reads red/amber/green by
+severity), with gradient meter bars, severity-coded findings, and
+loss/step-time sparklines. They auto-colorize in a real terminal and degrade
+to byte-identical plain text when piped, in CI, or under
+`NO_COLOR`/`--color=never` — never garbled, either way, and nothing written
+to disk besides the run itself.
+
+## Try the demos
+
+No ML deps:
+
+```bash
+python examples/manual_loop.py     && pytscope analyze runs/demo    # timing
+python examples/cross_signal.py    && pytscope analyze runs/cross   # cross-signal
+```
+
+Real PyTorch (CUDA / Apple MPS / CPU, auto-detected), with real device timing
+and memory:
+
+```bash
+pip install -e ".[torch]"
+python examples/pytorch_real.py            && pytscope analyze runs/pytorch  # healthy
+python examples/pytorch_real.py --leak     && pytscope analyze runs/pytorch  # catches the leak
+```
+
+The `--leak` run reports `MEMORY.GROWTH [HIGH]` from genuinely captured device
+memory. (Memory attribution is most accurate on CUDA, which exposes true in-step
+peaks; on MPS we sample resident memory at the step boundary.)
+
+---
+
+## Architecture
+
+```
+training loop → collectors → RunStore (aligned timeline)
+                                  ↓
+              analyzers (timing | memory | convergence | repro)
+                                  ↓
+              diagnosis engine (ranked, cross-signal findings)
+                                  ↓
+                  reporters (CLI — amber-LED hardware-panel terminal report)
+```
+
+Adding a heuristic is one decorated function (`@rule`); adding a vertical is one
+analyzer over the existing timeline.
+
+## Status & validation
+
+**v0.1, validated on real multi-GPU NCCL hardware** — straggler attribution and
+exposed-comm now have a clean run on 2× T4 (Kaggle, free tier, no paid rental):
+an exact pass on straggler detection (`z=14.1`, named the injected rank
+correctly) and a directionally-correct exposed-comm read that also surfaced a
+genuine finding about PCIe-only interconnects. MFU-on-GPU is the last gap —
+unblocked (a demo bug found and fixed) with a rerun pending.
+[Full report →](docs/validation-runs/2026-06-08-kaggle-2xT4/RESULTS.md) ·
+[Validation matrix & protocol →](docs/VALIDATION.md)
+
+DDP is first-class; FSDP/tensor/pipeline parallelism are not yet.
+
+## Documentation
+
+- [Usage guide](docs/usage.md) — install, instrument, and the CLI.
+- [Architecture](docs/architecture.md) — the one-timeline design.
+- [Diagnostics reference](docs/diagnostics.md) — every finding and its fix.
+- [Validation](docs/VALIDATION.md) — what's proven, and the multi-GPU protocol.
+
+## Contributing
+
+Contributions are welcome — adding a diagnosis rule is the most approachable
+first PR. See [CONTRIBUTING.md](CONTRIBUTING.md) and the
+[Code of Conduct](CODE_OF_CONDUCT.md). Releases follow [RELEASING.md](RELEASING.md).
+
+## License
+
+[MIT](LICENSE) © 2026 Sumukh Chaluvaraju