pulseml 0.2.2__tar.gz → 0.2.3__tar.gz

pulseml-0.2.3/PKG-INFO

@@ -0,0 +1,648 @@
+Metadata-Version: 2.4
+Name: pulseml
+Version: 0.2.3
+Summary: A live ML training debugger for tracking, diagnosing, verifying, and patching training runs.
+Author-email: Yash Patel <codeyash09@gmail.com>
+License: Proprietary
+Project-URL: Homepage, https://pulsedb.netlify.app/
+Project-URL: Repository, https://github.com/codeyash09/PulseML
+Project-URL: Issues, https://github.com/codeyash09/PulseML/issues
+Project-URL: Documentation, https://pulsedb.netlify.app/
+Keywords: machine-learning,deep-learning,debugging,ml-debugging,training,pytorch,tensorflow,numpy,cupy,jax,gradients,training-debugger
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: License :: Other/Proprietary License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.9
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Software Development :: Debuggers
+Requires-Python: >=3.9
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: numpy
+Requires-Dist: matplotlib
+Requires-Dist: pillow
+Requires-Dist: litellm
+Requires-Dist: fpdf2
+Provides-Extra: torch
+Requires-Dist: torch; extra == "torch"
+Provides-Extra: tensorflow
+Requires-Dist: tensorflow; extra == "tensorflow"
+Provides-Extra: cupy
+Requires-Dist: cupy-cuda12x; extra == "cupy"
+Provides-Extra: jax
+Requires-Dist: jax; extra == "jax"
+Dynamic: license-file
+
+# PulseML
+
+**Pulse** is a live ML training debugger for CLI and headless environments. It combines runtime observability, deterministic numerical checks, ML-specific linting, and an AI debugging agent to investigate failures while training is running.
+
+**Track → Diagnose → Verify → Patch**
+
+[Dashboard](https://pulsedashb.netlify.app/) · [GitHub](https://github.com/codeyash09/PulseML) · [PyPI](https://pypi.org/project/pulseml/)
+
+---
+
+## Installation
+
+Pulse is available on PyPI.
+
+### Requirements
+
+- Python 3.9+
+- A supported backend: NumPy, PyTorch, TensorFlow, CuPy, or JAX
+- An API key for an AI provider if you want to use the AI debugging agent
+
+### Install
+
+```bash
+pip install pulseml
+```
+
+### Verify
+
+```bash
+python -c "import pulse; print('Pulse installed successfully')"
+```
+
+### Start Pulse
+
+Import `auto_track` and call it immediately before your training loop:
+
+```python
+from pulse import auto_track
+
+if __name__ == "__main__":
+    auto_track()
+
+    # Your training loop
+    for epoch in range(num_epochs):
+        # Training logic here
+        pass
+```
+
+The `__main__` guard is particularly important in multiprocessing or process-spawning environments.
+
+Once the process is running, Pulse discovers numeric variables available to the training process and exposes them through the CLI.
+
+### AI provider configuration
+
+Pulse supports cloud and local AI providers.
+
+Common provider environment variables include:
+
+```text
+ANTHROPIC_API_KEY
+OPENAI_API_KEY
+GEMINI_API_KEY
+DEEPSEEK_API_KEY
+MISTRAL_API_KEY
+OPENROUTER_API_KEY
+```
+
+For example:
+
+```bash
+export GEMINI_API_KEY="your-api-key"
+```
+
+On Windows PowerShell:
+
+```powershell
+$env:GEMINI_API_KEY="your-api-key"
+```
+
+Do not place API keys directly in source code or commit them to a repository.
+
+---
+
+## Why Pulse
+
+Machine-learning failures are often diagnosed from the wrong end of the problem:
+
+```text
+Train
+  ↓
+Wait
+  ↓
+Training fails
+  ↓
+Read logs
+  ↓
+Guess
+  ↓
+Change code
+  ↓
+Train again
+```
+
+Pulse is designed to move the debugging process into the training run itself:
+
+```text
+Train
+  ↓
+Track
+  ↓
+Detect
+  ↓
+Measure
+  ↓
+Verify
+  ↓
+Diagnose
+  ↓
+Patch
+```
+
+The useful evidence behind an ML failure may be a gradient change, activation distribution, tensor shape, normalization relationship, parameter update, or the exact point at which a numerical value began to diverge.
+
+Pulse gives the debugging agent access to that evidence instead of requiring it to reconstruct the run from logs alone.
+
+---
+
+## Benchmark
+
+Pulse was evaluated against Claude Code on a **28-case ML debugging benchmark** designed to test whether an AI coding agent could identify and repair injected ML faults.
+
+### Results
+
+| Agent | Model | Score |
+|---|---|---:|
+| **Pulse** | Gemini 3.5 Flash Lite | **15 / 28** |
+| **Claude Code** | Opus 5 | **18 / 28** |
+
+Pulse achieved **15/28**, compared with **18/28** for Claude Code.
+
+The benchmark was particularly informative because the task was not simply to generate code that ran. The agent needed to identify the injected mutation, determine whether it was actually responsible for the observed behavior, and apply the necessary correction.
+
+### What the benchmark exposed
+
+Pulse's largest weakness was **diagnostic precision**.
+
+In difficult cases, Pulse could recognize that something was wrong but fail to consistently:
+
+1. identify the exact mutation responsible for the failure;
+2. isolate the smallest relevant code region;
+3. distinguish the root cause from surrounding symptoms; or
+4. apply only the necessary fix instead of proposing a broader change.
+
+This matters because ML debugging is different from generic code generation.
+
+A debugger should not rewrite a training system simply because it can. It should be able to identify the specific value, argument, operation, or line responsible for the failure and make the smallest correct change.
+
+### How the benchmark changed Pulse
+
+The benchmark directly motivated upgrades to the debugging pipeline:
+
+- **Expanded `mllint`** with additional deterministic ML-specific checks.
+- **Minimal-fix-first behavior**, prioritizing single-line or small adjacent-line fixes before refactors.
+- **Whitespace-tolerant patch matching**, preventing valid fixes from being rejected because of formatting differences.
+- **Retry-on-mismatch patching**, allowing the agent to re-quote the exact current code when a proposed patch does not match.
+- **Stronger verification**, checking both logical correctness and whether the scope of the patch is unnecessarily broad.
+- **More agentic debugging workflows**, reducing the amount of manual investigation the underlying model has to perform.
+
+The benchmark is not presented as proof that Pulse is already better than general-purpose coding agents. It provides a measurable baseline, exposes a concrete failure mode, and gives Pulse a data-driven direction for improvement.
+
+---
+
+## Key Features
+
+### Live ML Training Monitoring
+
+Track losses, metrics, tensors, gradients, activations, weights, and other numerical values while training is running.
+
+Inspect:
+
+- shapes
+- dtypes
+- devices
+- norms
+- statistics
+- NaN / Inf counts
+- scalar histories
+- gradient information
+- activation information
+
+### CLI / Headless First
+
+Pulse is built for:
+
+- terminals
+- SSH sessions
+- Google Colab
+- containers
+- remote servers
+- long-running training jobs
+- cloud GPU machines
+
+No graphical interface is required.
+
+### CPU-First Tracking
+
+Pulse is intentionally conservative around GPU access.
+
+By default:
+
+```text
+TRACKING_MODE = CPU_DEFAULT
+GPU_TRACKING = OPT-IN
+DEFAULT_GPU_OVERHEAD = 0
+```
+
+If a variable already exists on a GPU, Pulse does not automatically copy it back to the CPU on every iteration.
+
+GPU tracking is explicitly requested:
+
+```text
+/gputrack <variable>
+```
+
+This can introduce device-to-host transfer overhead. That is expected when inspecting GPU-resident data.
+
+The design principle is:
+
+> **If nobody asks Pulse to touch the GPU, Pulse doesn't touch the GPU.**
+
+### Dynamic Variable Tracking
+
+Variables can be added, removed, promoted, or demoted while training is running.
+
+```text
+/vars
+/tracked
+/add <variable>
+/track <variable>
+/lotrack <variable>
+/gputrack <variable>
+/gpuuntrack <variable>
+/delete <variable>
+```
+
+### Deterministic Numerical Verification
+
+Pulse separates AI reasoning from exact numerical calculation.
+
+Instead of asking an AI model to estimate whether a gradient or update is unusually large, Pulse can calculate the relevant quantity directly.
+
+```text
+AI hypothesis
+      ↓
+Numerical calculation
+      ↓
+Exact result
+      ↓
+Evidence-backed diagnosis
+```
+
+This can be used for:
+
+- gradient/update ratios
+- scaling factors
+- normalization calculations
+- parameter changes
+- numerical thresholds
+- restricted mathematical expressions
+
+The AI reasons about what the numbers mean. Pulse provides deterministic measurements for the arithmetic.
+
+### ML Linting
+
+`mllint` provides deterministic, code-level checks for common ML failure patterns before an agent has to reason about them from scratch.
+
+Checks include patterns involving:
+
+- incompatible loss/metric combinations
+- redundant activation/loss combinations
+- `backward()` without the expected optimizer update
+- training/evaluation mode issues
+- suspicious learning-rate configurations
+- other ML-specific static patterns
+
+The goal is not to replace the debugging agent. It is to provide high-confidence evidence and eliminate classes of mistakes that do not require probabilistic reasoning.
+
+### Agentic Debugging
+
+Pulse can combine:
+
+- live training state
+- tensor statistics
+- scalar histories
+- gradients
+- activations
+- tracebacks
+- source code
+- deterministic numerical checks
+- static lint findings
+
+The goal is to move from:
+
+```text
+"What might be wrong?"
+```
+
+toward:
+
+```text
+"Here is the evidence.
+Here is the mutation.
+Here is why it caused the failure.
+Here is the smallest necessary fix."
+```
+
+### Minimal Fixes by Default
+
+Pulse's debugging pipeline is designed to prefer the **smallest correct change**.
+
+The agent should first consider:
+
+1. a single-line change;
+2. a small adjacent-line change;
+3. a larger edit only when the root cause genuinely requires it.
+
+Unrelated cleanup and refactoring should not be bundled into a debugging patch.
+
+This is especially important for ML debugging because broad changes can hide whether the actual fault was correctly identified.
+
+### Automatic Intervention
+
+With:
+
+```text
+/autofix on
+```
+
+Pulse can pause training when configured detection logic identifies serious numerical or training problems, allowing the debugging agent to investigate before additional compute is wasted.
+
+---
+
+## CLI
+
+Useful commands include:
+
+```text
+/help
+
+/vars
+/tracked
+
+/add <variable>
+/track <variable>
+/lotrack <variable>
+
+/gputrack <variable>
+/gpuuntrack <variable>
+
+/delete <variable>
+
+/autofix on|off
+
+/code
+/cloud
+```
+
+Pulse can pause training for investigation, allowing you to inspect the current state, add variables, ask the AI agent questions, and investigate a failure before continuing.
+
+---
+
+## Quickstart
+
+After installation, import `auto_track` immediately before your training loop:
+
+```python
+from pulse import auto_track
+
+if __name__ == "__main__":
+    auto_track()
+
+    # Your training loop
+    for epoch in range(num_epochs):
+        # Training logic here
+        pass
+```
+
+Pulse discovers numeric variables available to the training process and provides them through the CLI.
+
+You can then control tracking while the run is active:
+
+```text
+/vars
+/tracked
+/add <variable>
+/track <variable>
+/lotrack <variable>
+/gputrack <variable>
+/gpuuntrack <variable>
+/delete <variable>
+```
+
+---
+
+## Supported Backends
+
+| Backend | Support |
+|---|---|
+| NumPy | Yes |
+| PyTorch | Yes |
+| TensorFlow | Yes |
+| CuPy | Yes |
+| JAX | Yes |
+
+Pulse uses a shared backend abstraction so the debugging workflow can remain consistent across frameworks.
+
+---
+
+## Past Debugs
+
+Pulse has been used to investigate numerical failures in custom ML systems.
+
+### Residual normalization failure
+
+A custom LLM became unstable after a 2.5× vocabulary increase. Pulse helped identify a normalization issue in which residual growth was divided by `math.sqrt(num_layers)` rather than `num_layers`.
+
+The important part was not simply observing that training became unstable. The debugging process connected the observed activation behavior to the mathematical relationship causing the instability.
+
+### Attention numerical failure
+
+A custom attention implementation produced NaN loss. Pulse traced the failure to a missing infinity check before a division operation.
+
+These cases represent the intended workflow:
+
+**observe the behavior → inspect the evidence → verify the math → identify the actual failure**
+
+---
+
+## Performance
+
+A debugger should not become the training bottleneck.
+
+Pulse is designed around:
+
+- CPU-first inspection
+- opt-in GPU probing
+- selective variable tracking
+- lightweight tracking modes
+- separate probe cadences
+- cached statistics
+- host-side numerical processing where possible
+- minimal intervention in the training loop
+
+The objective is:
+
+```text
+MORE VISIBILITY
+      +
+LESS OVERHEAD
+```
+
+Rather than collecting everything continuously, Pulse lets you decide what information is worth monitoring.
+
+---
+
+## Cloud Workspaces
+
+Pulse can optionally synchronize debugging information through shared workspaces.
+
+Depending on configuration, a workspace can provide shared access to:
+
+- debugging sessions
+- training incidents
+- tracebacks
+- agent conversations
+- repository metadata
+- team membership
+- telemetry
+
+Workspaces can be given custom names and managed from the CLI. Members can leave a workspace, while workspace administrators can delete a workspace.
+
+Cloud synchronization is best-effort and is not intended to block the training loop.
+
+For sensitive projects, review your cloud and telemetry configuration carefully.
+
+Telemetry can be disabled with:
+
+```text
+PULSE_TELEMETRY=off
+```
+
+Using a local AI model keeps model requests local, but does not automatically disable separately enabled Pulse workspace synchronization.
+
+---
+
+## AI Providers
+
+Pulse can use supported cloud or local AI providers for its debugging agent.
+
+Common provider environment variables include:
+
+```text
+ANTHROPIC_API_KEY
+OPENAI_API_KEY
+GEMINI_API_KEY
+DEEPSEEK_API_KEY
+MISTRAL_API_KEY
+OPENROUTER_API_KEY
+```
+
+Local and self-hosted models can also be used where supported.
+
+Do not place API keys directly in source code or commit them to a repository.
+
+---
+
+## Why This Matters
+
+The central problem Pulse is trying to solve is not code generation.
+
+Modern coding agents are increasingly capable of writing large amounts of code. ML debugging has a different requirement: **causal precision**.
+
+A useful ML debugger needs to connect three things:
+
+```text
+CODE
+  ↕
+RUNTIME BEHAVIOR
+  ↕
+MATHEMATICS
+```
+
+If an agent only sees code, it can miss what actually happened during training.
+
+If it only sees runtime values, it may not know which source-level mutation produced them.
+
+If it only reasons probabilistically, it can make confident but numerically unsupported claims.
+
+Pulse is designed to put those pieces together.
+
+The benchmark results reinforce why that distinction matters. At the initial 28-case benchmark, Pulse scored 15/28 versus 18/28 for Claude Code with Opus 5. Rather than treating that gap as a dead end, the failures identified a concrete engineering problem: **the system needed to make the agent better at isolating the exact mutation and applying the smallest necessary fix.**
+
+That is the direction of Pulse's development.
+
+---
+
+## Project Direction
+
+Pulse is being built around a debugging stack in which runtime observation, deterministic computation, static ML analysis, and AI reasoning reinforce each other:
+
+```text
+TRAINING
+   ↓
+OBSERVATION
+   ↓
+STATIC ANALYSIS
+   ↓
+NUMERICAL EVIDENCE
+   ↓
+AI REASONING
+   ↓
+VERIFICATION
+   ↓
+MINIMAL PATCH
+```
+
+The goal is not simply to report:
+
+```text
+"Your training is broken."
+```
+
+It is to answer:
+
+```text
+What broke?
+
+Why did it break?
+
+When did it start?
+
+Can the numbers prove it?
+
+What exactly changed?
+
+What is the smallest correct fix?
+
+Can that fix be verified?
+```
+
+---
+
+## Links
+
+- [Dashboard](https://pulsedashb.netlify.app/)
+- [GitHub](https://github.com/codeyash09/PulseML)
+- [PyPI](https://pypi.org/project/pulseml/)
+
+---
+
+## License
+
+Proprietary. See `LICENSE`.
+
+Use of this software is governed by the terms in that file. Copying, redistribution, and reverse engineering are not permitted.