modal-progress 0.1.0__tar.gz

modal_progress-0.1.0/.gitignore

@@ -0,0 +1,8 @@
+.venv/
+__pycache__/
+*.pyc
+*.egg-info/
+build/
+dist/
+.pytest_cache/
+.ruff_cache/

modal_progress-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Enes Yilmaz
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

modal_progress-0.1.0/PKG-INFO

@@ -0,0 +1,128 @@
+Metadata-Version: 2.5
+Name: modal-progress
+Version: 0.1.0
+Summary: Live progress from inside Modal containers, for people at a terminal and agents reading a log
+Project-URL: Homepage, https://github.com/EnesYilmazcode/modal-progress
+Author: Enes Yilmaz
+License-Expression: MIT
+License-File: LICENSE
+Requires-Python: >=3.10
+Requires-Dist: modal>=1.0
+Requires-Dist: rich>=13
+Provides-Extra: dev
+Requires-Dist: pytest>=8; extra == 'dev'
+Description-Content-Type: text/markdown
+
+<h1 align="center">modal-progress</h1>
+
+<p align="center"><b>Live progress from inside Modal containers.</b><br>
+Progress bars per container in a terminal, plain progress lines in a log file.</p>
+
+<p align="center">
+  <img src="https://raw.githubusercontent.com/EnesYilmazcode/modal-progress/main/assets/media/with-and-without.gif" width="820" alt="The same 60-input Modal job logged to a file: without modal-progress the log is silent until the end, with it there is a progress line every 5 seconds">
+</p>
+
+`modal run` draws `.map` progress as a live terminal bar. When the output goes to a file or a pipe, which is how coding agents run commands, nothing is printed until the job ends, so there is no way to tell a slow job from a stuck one. modal-progress prints a plain line every few seconds instead, and lets each container report its own progress.
+
+## Install
+
+```bash
+pip install git+https://github.com/EnesYilmazcode/modal-progress
+```
+
+Requires Python 3.10+ and modal 1.x.
+
+## Usage
+
+### No code changes
+
+Use `modal-progress run` in place of `modal run`. Arguments are passed through unchanged.
+
+```bash
+modal-progress run app.py > log.txt
+```
+
+```
+[progress] work 11/60 done (18%), 0:13 elapsed, eta 1:00
+[progress] work 19/60 done (32%), 0:20 elapsed, eta 0:44
+[progress] work 29/60 done (48%), 0:25 elapsed, eta 0:27
+[progress] work 60/60 done, 0:47
+```
+
+In a terminal it behaves exactly like `modal run`.
+
+### Progress from inside each container
+
+Wrap a loop in `bar()` inside the function, and wrap the map in `watch()` on the caller's side.
+
+```python
+import modal
+from modal_progress import bar, watch
+
+app = modal.App("example")
+image = modal.Image.debian_slim().add_local_python_source("modal_progress")
+
+@app.function(image=image)
+def work(i: int) -> int:
+    for step in bar(range(40), desc=f"input {i}"):
+        ...  # one step of work
+    return i
+
+@app.local_entrypoint()
+def main():
+    xs = range(60)
+    with watch(app, name="work") as w:
+        results = list(w.track(work.map(xs, return_exceptions=True), total=len(xs)))
+```
+
+In a terminal, this shows an overall bar plus one bar per container. In a log file, it prints:
+
+```
+[progress] work 20/60 done (33%), 0 failed, 6 running, 0:25 elapsed, eta 0:31
+[progress] work: an input failed: ValueError: bad input 37
+[progress] work 40/60 done (67%), 1 failed, 6 running, 0:40 elapsed, eta 0:14
+[progress] work finished: 60/60 done, 1 failed, 0:57
+```
+
+Failures are reported the moment they happen. Every line starts with `[progress]`, so it can be filtered with `grep`. A full example is in [`examples/fanout.py`](https://github.com/EnesYilmazcode/modal-progress/blob/main/examples/fanout.py).
+
+## API
+
+| Call | Where | What it does |
+|---|---|---|
+| `bar(iterable=None, total=None, desc="", every=0.5)` | container | tqdm-style wrapper. Also supports `update(n)`, `set_description(s)`, `close()`, and `with`. Sends at most one update every `every` seconds. |
+| `report(n, total=None, desc="")` | container | Sets progress for the current input directly, for loops `bar()` can't wrap. |
+| `watch(app, name="map", mode=None, interval=None, record=None)` | caller | Context manager that collects and draws progress. `record` writes every event to a JSON lines file. |
+| `w.track(results, total=None)` | caller | Passes map results through and counts them. Exceptions count as failures. |
+
+## Configuration
+
+| Variable | Values | Default |
+|---|---|---|
+| `MODAL_PROGRESS` | `rich`, `plain`, `off` | `rich` in a terminal, `plain` otherwise. `off` also stops containers from sending. |
+| `MODAL_PROGRESS_INTERVAL` | seconds between plain lines | `5` |
+
+## How it works
+
+Each container sends its latest progress to a [Modal Queue](https://modal.com/docs/reference/modal.Queue) named `modal-progress`, in a partition keyed by the app ID, batched every 0.25 seconds from a background thread. `watch()` reads that partition and combines it with the map's results as they come back. A `bar()` update costs about 0.2 µs, so it can wrap tight loops.
+
+`modal-progress run` needs no queue. It hooks Modal's own count of finished inputs and prints it as plain lines when stdout isn't a terminal.
+
+## Limitations
+
+- Uses three private Modal internals: `App._get_container_app()` to find the app ID in a container, Modal's live display to add the bars to it, and its map progress callback for `modal-progress run`. All are guarded, so if one changes you lose progress output, not the job. Tested on modal 1.6.1.
+- Works with `modal run` and `app.run()`. A deployed function called from a different app reports to its own app's partition, so the caller won't see it.
+- Progress shows up within about a second of happening.
+
+## Development
+
+```bash
+pip install -e ".[dev]"
+python -m pytest -q
+python -m modal_progress demo        # simulated map, no Modal account needed
+modal run examples/fanout.py         # real map on Modal
+```
+
+## License
+
+MIT

modal_progress-0.1.0/README.md

@@ -0,0 +1,113 @@
+<h1 align="center">modal-progress</h1>
+
+<p align="center"><b>Live progress from inside Modal containers.</b><br>
+Progress bars per container in a terminal, plain progress lines in a log file.</p>
+
+<p align="center">
+  <img src="https://raw.githubusercontent.com/EnesYilmazcode/modal-progress/main/assets/media/with-and-without.gif" width="820" alt="The same 60-input Modal job logged to a file: without modal-progress the log is silent until the end, with it there is a progress line every 5 seconds">
+</p>
+
+`modal run` draws `.map` progress as a live terminal bar. When the output goes to a file or a pipe, which is how coding agents run commands, nothing is printed until the job ends, so there is no way to tell a slow job from a stuck one. modal-progress prints a plain line every few seconds instead, and lets each container report its own progress.
+
+## Install
+
+```bash
+pip install git+https://github.com/EnesYilmazcode/modal-progress
+```
+
+Requires Python 3.10+ and modal 1.x.
+
+## Usage
+
+### No code changes
+
+Use `modal-progress run` in place of `modal run`. Arguments are passed through unchanged.
+
+```bash
+modal-progress run app.py > log.txt
+```
+
+```
+[progress] work 11/60 done (18%), 0:13 elapsed, eta 1:00
+[progress] work 19/60 done (32%), 0:20 elapsed, eta 0:44
+[progress] work 29/60 done (48%), 0:25 elapsed, eta 0:27
+[progress] work 60/60 done, 0:47
+```
+
+In a terminal it behaves exactly like `modal run`.
+
+### Progress from inside each container
+
+Wrap a loop in `bar()` inside the function, and wrap the map in `watch()` on the caller's side.
+
+```python
+import modal
+from modal_progress import bar, watch
+
+app = modal.App("example")
+image = modal.Image.debian_slim().add_local_python_source("modal_progress")
+
+@app.function(image=image)
+def work(i: int) -> int:
+    for step in bar(range(40), desc=f"input {i}"):
+        ...  # one step of work
+    return i
+
+@app.local_entrypoint()
+def main():
+    xs = range(60)
+    with watch(app, name="work") as w:
+        results = list(w.track(work.map(xs, return_exceptions=True), total=len(xs)))
+```
+
+In a terminal, this shows an overall bar plus one bar per container. In a log file, it prints:
+
+```
+[progress] work 20/60 done (33%), 0 failed, 6 running, 0:25 elapsed, eta 0:31
+[progress] work: an input failed: ValueError: bad input 37
+[progress] work 40/60 done (67%), 1 failed, 6 running, 0:40 elapsed, eta 0:14
+[progress] work finished: 60/60 done, 1 failed, 0:57
+```
+
+Failures are reported the moment they happen. Every line starts with `[progress]`, so it can be filtered with `grep`. A full example is in [`examples/fanout.py`](https://github.com/EnesYilmazcode/modal-progress/blob/main/examples/fanout.py).
+
+## API
+
+| Call | Where | What it does |
+|---|---|---|
+| `bar(iterable=None, total=None, desc="", every=0.5)` | container | tqdm-style wrapper. Also supports `update(n)`, `set_description(s)`, `close()`, and `with`. Sends at most one update every `every` seconds. |
+| `report(n, total=None, desc="")` | container | Sets progress for the current input directly, for loops `bar()` can't wrap. |
+| `watch(app, name="map", mode=None, interval=None, record=None)` | caller | Context manager that collects and draws progress. `record` writes every event to a JSON lines file. |
+| `w.track(results, total=None)` | caller | Passes map results through and counts them. Exceptions count as failures. |
+
+## Configuration
+
+| Variable | Values | Default |
+|---|---|---|
+| `MODAL_PROGRESS` | `rich`, `plain`, `off` | `rich` in a terminal, `plain` otherwise. `off` also stops containers from sending. |
+| `MODAL_PROGRESS_INTERVAL` | seconds between plain lines | `5` |
+
+## How it works
+
+Each container sends its latest progress to a [Modal Queue](https://modal.com/docs/reference/modal.Queue) named `modal-progress`, in a partition keyed by the app ID, batched every 0.25 seconds from a background thread. `watch()` reads that partition and combines it with the map's results as they come back. A `bar()` update costs about 0.2 µs, so it can wrap tight loops.
+
+`modal-progress run` needs no queue. It hooks Modal's own count of finished inputs and prints it as plain lines when stdout isn't a terminal.
+
+## Limitations
+
+- Uses three private Modal internals: `App._get_container_app()` to find the app ID in a container, Modal's live display to add the bars to it, and its map progress callback for `modal-progress run`. All are guarded, so if one changes you lose progress output, not the job. Tested on modal 1.6.1.
+- Works with `modal run` and `app.run()`. A deployed function called from a different app reports to its own app's partition, so the caller won't see it.
+- Progress shows up within about a second of happening.
+
+## Development
+
+```bash
+pip install -e ".[dev]"
+python -m pytest -q
+python -m modal_progress demo        # simulated map, no Modal account needed
+modal run examples/fanout.py         # real map on Modal
+```
+
+## License
+
+MIT

modal_progress-0.1.0/examples/fanout.py

@@ -0,0 +1,34 @@
+# 60 inputs over 6 containers. Each input walks 40 steps, input 37 fails
+# two thirds of the way in. Run from the repo root:
+#
+#   modal run examples/fanout.py                    rich bars in a terminal
+#   modal run examples/fanout.py > log.txt          plain lines, like an agent sees
+#   modal run examples/fanout.py --record run.jsonl  also save every event
+import random
+import time
+
+import modal
+
+from modal_progress import bar, watch
+
+app = modal.App("modal-progress-example")
+image = modal.Image.debian_slim().add_local_python_source("modal_progress")
+
+
+@app.function(image=image, max_containers=6)
+def work(i: int) -> int:
+    speed = random.Random(i).uniform(0.6, 1.6)
+    for step in bar(range(40), desc=f"input {i}"):
+        time.sleep(0.1 * speed)
+        if i == 37 and step == 26:
+            raise ValueError("bad input 37")
+    return i * i
+
+
+@app.local_entrypoint()
+def main(record: str = ""):
+    xs = range(60)
+    with watch(app, name="work", record=record or None) as w:
+        results = list(w.track(work.map(xs, return_exceptions=True, order_outputs=False), total=len(xs)))
+    failed = sum(isinstance(r, Exception) for r in results)
+    print(f"{len(results)} results, {failed} failed")

modal_progress-0.1.0/modal_progress/__init__.py

@@ -0,0 +1,7 @@
+"""Live progress from inside Modal containers."""
+
+from .container import bar, report
+from .watch import Watcher, watch
+
+__all__ = ["bar", "report", "watch", "Watcher"]
+__version__ = "0.1.0"

modal_progress-0.1.0/modal_progress/__main__.py

@@ -0,0 +1,63 @@
+"""python -m modal_progress run app.py   modal run, plus plain progress lines in logs
+python -m modal_progress demo         a simulated fan-out, no Modal account needed
+"""
+
+from __future__ import annotations
+
+import argparse
+import sys
+
+
+def demo(args: argparse.Namespace) -> None:
+    from .sim import simulate
+    from .watch import watch
+
+    fail = tuple(int(x) for x in args.fail.split(",") if x)
+    with watch(name="work", mode=args.mode, interval=args.interval) as w:
+        for _ in w.track(
+            simulate(inputs=args.inputs, containers=args.containers, steps=args.steps, step_time=args.step_time, fail=fail),
+            total=args.inputs,
+        ):
+            pass
+
+
+def run(rest: list[str], interval: float) -> None:
+    import modal.__main__
+
+    from .agent import install
+
+    install(interval=interval)
+    sys.argv = ["modal", "run", *rest]
+    modal.__main__.main()
+
+
+def main(argv: list[str] | None = None) -> None:
+    argv = sys.argv[1:] if argv is None else argv
+    # Everything after `run` goes to modal untouched, flags included.
+    if argv[:1] == ["run"]:
+        rest = argv[1:]
+        interval = 5.0
+        if rest[:1] == ["--interval"] and len(rest) > 1:
+            interval, rest = float(rest[1]), rest[2:]
+        return run(rest, interval)
+
+    parser = argparse.ArgumentParser(prog="modal-progress")
+    sub = parser.add_subparsers(dest="cmd", required=True)
+    sub.add_parser("run", help="modal run with plain progress lines (args go to modal run)")
+
+    d = sub.add_parser("demo", help="simulate a fan-out locally")
+    d.add_argument("--inputs", type=int, default=60)
+    d.add_argument("--containers", type=int, default=6)
+    d.add_argument("--steps", type=int, default=40)
+    d.add_argument("--step-time", type=float, default=0.1)
+    d.add_argument("--fail", default="37", help="comma-separated inputs that raise")
+    d.add_argument("--mode", choices=["rich", "plain"], default=None)
+    d.add_argument("--interval", type=float, default=None, help="seconds between plain lines")
+
+    args = parser.parse_args(argv)
+    if args.cmd == "demo":
+        demo(args)
+
+
+if __name__ == "__main__":
+    sys.exit(main())

modal_progress-0.1.0/modal_progress/agent.py

@@ -0,0 +1,80 @@
+"""Plain map progress for `modal run` when nobody is at a terminal.
+
+Modal tracks map progress with its own counter and draws it as a rich live
+bar. When stdout is a file or a pipe, which is how a background agent runs
+it, rich draws nothing until the end, so the log has no progress at all.
+install() wraps that counter so it also prints a throttled plain line.
+No change to the user's code is needed.
+"""
+
+from __future__ import annotations
+
+import sys
+import time
+from typing import Callable, TextIO
+
+from .render import PREFIX
+from .state import clock
+
+
+class LineThrottle:
+    """Turns (completed, total) updates into at most one line per interval."""
+
+    def __init__(self, tag: str, interval: float = 5.0, out: TextIO | None = None, now=time.monotonic):
+        self.tag = tag
+        self.interval = interval
+        self.out = out
+        self.now = now
+        self.started = now()
+        self.last = self.started
+        self.last_line: tuple[int, int] | None = None
+
+    def __call__(self, completed: int, total: int) -> None:
+        t = self.now()
+        finished = total > 0 and completed >= total
+        if (completed, total) == self.last_line:
+            return
+        if not finished and t - self.last < self.interval:
+            return
+        self.last = t
+        self.last_line = (completed, total)
+        elapsed = t - self.started
+        if finished:
+            line = f"{PREFIX} {self.tag} {completed}/{total} done, {clock(elapsed)}"
+        else:
+            pct = f" ({100 * completed / total:.0f}%)" if total else ""
+            rate = completed / elapsed if elapsed > 0 else 0
+            # One finished input says little about the rest, so wait for 5%.
+            trusted = total and completed >= max(2, total * 0.05)
+            eta = f", eta {clock((total - completed) / rate)}" if rate > 0 and trusted else ""
+            line = f"{PREFIX} {self.tag} {completed}/{total} done{pct}, {clock(elapsed)} elapsed{eta}"
+        out = self.out or sys.stdout
+        out.write(line + "\n")
+        out.flush()
+
+
+def install(interval: float = 5.0, manager_cls=None) -> Callable[[], None]:
+    """Patches Modal's rich output manager. Returns a function that undoes it."""
+    if manager_cls is None:
+        from modal._output.rich import RichOutputManager as manager_cls
+
+    original = manager_cls.function_progress_callback
+
+    def function_progress_callback(self, tag: str, total: int | None):
+        update = original(self, tag, total)
+        if self.is_terminal:
+            return update
+        line = LineThrottle(tag, interval)
+
+        def both(completed: int, total: int) -> None:
+            update(completed, total)
+            line(completed, total)
+
+        return both
+
+    manager_cls.function_progress_callback = function_progress_callback
+
+    def undo() -> None:
+        manager_cls.function_progress_callback = original
+
+    return undo