expr-tracker 0.2.1__tar.gz → 0.2.3__tar.gz

{expr_tracker-0.2.1 → expr_tracker-0.2.3}/.gitignore

@@ -14,3 +14,8 @@ test.ipynb
 .coverage
 htmlcov/
 site/
+
+# Tracker run output: the default dir, and whatever people point --dir at
+tracker/
+outputs/
+runs/

expr_tracker-0.2.3/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Hangxing Wei
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

expr_tracker-0.2.1/README.md → expr_tracker-0.2.3/PKG-INFO

@@ -1,5 +1,57 @@
+Metadata-Version: 2.4
+Name: expr_tracker
+Version: 0.2.3
+Summary: Local-first experiment tracking with queryable history and expression-based alerts on your training metrics
+Project-URL: Homepage, https://hspk.github.io/expr_tracker/
+Project-URL: Documentation, https://hspk.github.io/expr_tracker/
+Project-URL: Repository, https://github.com/HSPK/expr_tracker
+Project-URL: Issues, https://github.com/HSPK/expr_tracker/issues
+Project-URL: Changelog, https://github.com/HSPK/expr_tracker/releases
+Author-email: HSPK <whxway@whu.edu.cn>
+License-Expression: MIT
+License-File: LICENSE
+Keywords: alerting,deep-learning,experiment-tracking,machine-learning,mlops,monitoring,training,wandb
+Classifier: Development Status :: 4 - Beta
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: System :: Monitoring
+Classifier: Typing :: Typed
+Requires-Python: >=3.10
+Requires-Dist: click>=8.1.0
+Requires-Dist: loguru>=0.7.3
+Requires-Dist: pydantic>=2.0
+Provides-Extra: all
+Requires-Dist: pandas>=1.5; extra == 'all'
+Requires-Dist: polars>=0.20; extra == 'all'
+Requires-Dist: slark>=0.1.28; extra == 'all'
+Requires-Dist: trackio>=0.4.0; extra == 'all'
+Requires-Dist: wandb>=0.21.0; extra == 'all'
+Provides-Extra: lark
+Requires-Dist: slark>=0.1.28; extra == 'lark'
+Provides-Extra: pandas
+Requires-Dist: pandas>=1.5; extra == 'pandas'
+Provides-Extra: polars
+Requires-Dist: polars>=0.20; extra == 'polars'
+Provides-Extra: trackio
+Requires-Dist: trackio>=0.4.0; extra == 'trackio'
+Provides-Extra: wandb
+Requires-Dist: wandb>=0.21.0; extra == 'wandb'
+Description-Content-Type: text/markdown
+
 # Experiment Tracker
 
+[![PyPI](https://img.shields.io/pypi/v/expr-tracker)](https://pypi.org/project/expr-tracker/)
+[![Python](https://img.shields.io/pypi/pyversions/expr-tracker)](https://pypi.org/project/expr-tracker/)
+[![License](https://img.shields.io/pypi/l/expr-tracker)](LICENSE)
+[![Docs](https://img.shields.io/badge/docs-github.io-blue)](https://hspk.github.io/expr_tracker/)
+
 A local-first experiment tracker. Metrics land in a JSONL file you own, stay
 queryable while the run is live, and can trigger alerts from an expression language.
 `wandb` and `trackio` are optional mirrors, not requirements.
@@ -54,6 +106,8 @@ install command; it never crashes a run.
 | [Alerts](https://hspk.github.io/expr_tracker/guide/alerts/) | An expression DSL with rolling windows, three-valued logic (no false alarms during warm-up), a rule state machine, and a watchdog that catches a hung run. |
 | [Channels](https://hspk.github.io/expr_tracker/guide/alerts/#channels) | Lark, Slack, DingTalk, WeCom, generic webhook, email — with rate limiting, dedup, retries and per-channel routing. |
 | [Artifacts](https://hspk.github.io/expr_tracker/guide/artifacts/) | Versioned file sets, deduplicated by content, shared across a project's runs, with lineage. |
+| [Spans](https://hspk.github.io/expr_tracker/guide/spans/) | Time the parts of a step, and their parts. Each duration becomes a metric, so alerts and queries work on it unchanged; `et trace` exports the timeline for Perfetto. |
+| [Streams](https://hspk.github.io/expr_tracker/guide/streams/) | Independent producers — a data worker and a training loop — each with their own step cursor and file inside one run. |
 | [Distributed](https://hspk.github.io/expr_tracker/guide/distributed/) | Per-rank shards so concurrent appends cannot corrupt step order; only rank 0 alerts by default. |
 | [CLI](https://hspk.github.io/expr_tracker/guide/cli/) | `et history`, `et rules explain`, `et rules test`, `et alert`. |
 
@@ -96,6 +150,9 @@ uv run ruff format src tests
 | `test_hot_paths.py` | contracts and defaults of `et.log` / `et.history` / summary / alerts |
 | `test_value_encoding.py` | numpy, pydantic, datetime, Path, Enum round trips; output types; query bounds |
 | `test_expr_properties.py` | DSL properties: render round-trip stability, precedence, the whole `M` builder |
+| `test_trace.py` | Chrome Trace export: lane layout, stream and step selection, the CLI |
+| `test_spans.py` | nesting, aggregation, decorator and async forms, errors, thread and task isolation |
+| `test_streams.py` | stream naming and validation, isolation, resolution order, backend grouping, two-process runs |
 | `test_distributed.py` | rank shards, `alert_on_rank`, real multi-process runs |
 | `test_wandb.py` | real wandb in offline mode: parameter mapping, step alignment, artifacts |
 | `test_trackio.py` | trackio contract, resume mapping, real end-to-end |
@@ -114,3 +171,7 @@ Published to GitHub Pages by `.github/workflows/docs.yaml` on every push to `mai
 Internals: [`docs/design.md`](docs/design.md) (data model and key invariants) and
 [`docs/architecture.md`](docs/architecture.md) (module map, read/write paths,
 concurrency model).
+
+## License
+
+[MIT](LICENSE)

expr_tracker-0.2.1/PKG-INFO → expr_tracker-0.2.3/README.md

@@ -1,32 +1,10 @@
-Metadata-Version: 2.4
-Name: expr_tracker
-Version: 0.2.1
-Summary: Add your description here
-Author-email: HSPK <whxway@whu.edu.cn>
-Requires-Python: >=3.10
-Requires-Dist: click>=8.1.0
-Requires-Dist: loguru>=0.7.3
-Requires-Dist: pydantic>=2.0
-Provides-Extra: all
-Requires-Dist: pandas>=1.5; extra == 'all'
-Requires-Dist: polars>=0.20; extra == 'all'
-Requires-Dist: slark>=0.1.28; extra == 'all'
-Requires-Dist: trackio>=0.4.0; extra == 'all'
-Requires-Dist: wandb>=0.21.0; extra == 'all'
-Provides-Extra: lark
-Requires-Dist: slark>=0.1.28; extra == 'lark'
-Provides-Extra: pandas
-Requires-Dist: pandas>=1.5; extra == 'pandas'
-Provides-Extra: polars
-Requires-Dist: polars>=0.20; extra == 'polars'
-Provides-Extra: trackio
-Requires-Dist: trackio>=0.4.0; extra == 'trackio'
-Provides-Extra: wandb
-Requires-Dist: wandb>=0.21.0; extra == 'wandb'
-Description-Content-Type: text/markdown
-
 # Experiment Tracker
 
+[![PyPI](https://img.shields.io/pypi/v/expr-tracker)](https://pypi.org/project/expr-tracker/)
+[![Python](https://img.shields.io/pypi/pyversions/expr-tracker)](https://pypi.org/project/expr-tracker/)
+[![License](https://img.shields.io/pypi/l/expr-tracker)](LICENSE)
+[![Docs](https://img.shields.io/badge/docs-github.io-blue)](https://hspk.github.io/expr_tracker/)
+
 A local-first experiment tracker. Metrics land in a JSONL file you own, stay
 queryable while the run is live, and can trigger alerts from an expression language.
 `wandb` and `trackio` are optional mirrors, not requirements.
@@ -81,6 +59,8 @@ install command; it never crashes a run.
 | [Alerts](https://hspk.github.io/expr_tracker/guide/alerts/) | An expression DSL with rolling windows, three-valued logic (no false alarms during warm-up), a rule state machine, and a watchdog that catches a hung run. |
 | [Channels](https://hspk.github.io/expr_tracker/guide/alerts/#channels) | Lark, Slack, DingTalk, WeCom, generic webhook, email — with rate limiting, dedup, retries and per-channel routing. |
 | [Artifacts](https://hspk.github.io/expr_tracker/guide/artifacts/) | Versioned file sets, deduplicated by content, shared across a project's runs, with lineage. |
+| [Spans](https://hspk.github.io/expr_tracker/guide/spans/) | Time the parts of a step, and their parts. Each duration becomes a metric, so alerts and queries work on it unchanged; `et trace` exports the timeline for Perfetto. |
+| [Streams](https://hspk.github.io/expr_tracker/guide/streams/) | Independent producers — a data worker and a training loop — each with their own step cursor and file inside one run. |
 | [Distributed](https://hspk.github.io/expr_tracker/guide/distributed/) | Per-rank shards so concurrent appends cannot corrupt step order; only rank 0 alerts by default. |
 | [CLI](https://hspk.github.io/expr_tracker/guide/cli/) | `et history`, `et rules explain`, `et rules test`, `et alert`. |
 
@@ -123,6 +103,9 @@ uv run ruff format src tests
 | `test_hot_paths.py` | contracts and defaults of `et.log` / `et.history` / summary / alerts |
 | `test_value_encoding.py` | numpy, pydantic, datetime, Path, Enum round trips; output types; query bounds |
 | `test_expr_properties.py` | DSL properties: render round-trip stability, precedence, the whole `M` builder |
+| `test_trace.py` | Chrome Trace export: lane layout, stream and step selection, the CLI |
+| `test_spans.py` | nesting, aggregation, decorator and async forms, errors, thread and task isolation |
+| `test_streams.py` | stream naming and validation, isolation, resolution order, backend grouping, two-process runs |
 | `test_distributed.py` | rank shards, `alert_on_rank`, real multi-process runs |
 | `test_wandb.py` | real wandb in offline mode: parameter mapping, step alignment, artifacts |
 | `test_trackio.py` | trackio contract, resume mapping, real end-to-end |
@@ -141,3 +124,7 @@ Published to GitHub Pages by `.github/workflows/docs.yaml` on every push to `mai
 Internals: [`docs/design.md`](docs/design.md) (data model and key invariants) and
 [`docs/architecture.md`](docs/architecture.md) (module map, read/write paths,
 concurrency model).
+
+## License
+
+[MIT](LICENSE)

{expr_tracker-0.2.1 → expr_tracker-0.2.3}/docs/design.md

@@ -75,7 +75,48 @@ last line. Patch lines and `step_policy="allow"` both let a file end on a lower
 and continuing from that would reuse an existing one. `max_step` is maintained
 incrementally and persisted in the sidecar.
 
-### A.4 Multiple processes
+### A.4 Streams
+
+Independent producers — a training loop and a data worker — have unrelated step
+semantics, so each gets its own file, cursor and resume state under one run
+directory: `metrics[.stream][.rankN].jsonl`, plus per-stream `summary` and
+`config` sidecars so concurrent processes cannot clobber each other.
+
+The store is instantiated once per stream rather than made multi-tenant, so every
+invariant below holds unchanged within a stream; only the filename differs.
+
+`resolve_run_path()` matches names exactly rather than taking the first glob hit:
+`metrics.data.jsonl` sorts before `metrics.jsonl`, so sort order would silently
+return a stream in place of the default producer.
+
+A stream is forwarded to a backend as its own run, grouped under the run name.
+Neither wandb nor trackio can merge two step axes into one run; wandb's shared
+mode can, but requires a live server and has no trackio equivalent.
+
+### A.5 Spans
+
+`et.span` times a region and its children. A closed span takes the same ingress as
+`log()` — `HistoryStore.ingest()` — so the step policy, the open-row merge and the
+commit rules exist in one place regardless of which API produced the metrics.
+
+Two differences from a log call:
+
+- the merge **accumulates** rather than replaces, because the same span can run
+  many times in one step;
+- the metrics are built here from floats and ints, so they skip `RecordCodec`.
+  That encoder costs 5 µs, which a per-sub-step call cannot afford.
+
+A span never commits a step: durations ride along with whatever `log()` commits,
+so timing a region adds no row. The tree, with timestamps and attributes, goes to
+`spans[.stream][.rankN].jsonl` through a second `JsonlWriter`, enqueued rather
+than appended so a span does not pay for a flush decision of its own.
+
+The nesting stack is a `ContextVar`, not a `threading.local`: a new thread starts
+from the default and each asyncio task gets its own copy, which is exactly the
+scoping spans need. (The run singleton went the other way, from `ContextVar` to a
+locked global, because there the goal was for worker threads to *share* it.)
+
+### A.6 Multiple processes
 
 The supported model is **rank 0 tracks**; there is no cross-rank merging.
 

{expr_tracker-0.2.1 → expr_tracker-0.2.3}/docs/guide/alerts.md

@@ -93,6 +93,69 @@ Built-in types: `lark`, `slack`, `dingtalk`, `wecom`, `webhook` (a generic JSON
 template), `email`, `callable`. All but `lark` use only the standard library. Add
 your own with `register_backend()`.
 
+### Email
+
+Email needs an SMTP server to send *from*, even when the recipient is Gmail.
+
+```python
+et.init(
+    project="demo",
+    alert={
+        "channels": [
+            {
+                "type": "email",
+                "name": "inbox",
+                "options": {
+                    "host": "smtp.gmail.com",
+                    "port": 587,
+                    "tls": True,
+                    "user": "you@gmail.com",
+                    "password": os.environ["SMTP_PASSWORD"],
+                    "sender": "you@gmail.com",
+                    "to": ["you@gmail.com", "teammate@example.com"],
+                },
+                "min_level": "error",
+            }
+        ]
+    },
+    alert_rules=["isnan(loss) => critical: loss diverged"],
+)
+```
+
+| Option | Default | Meaning |
+| --- | --- | --- |
+| `host` | — | SMTP server, **required** |
+| `to` | — | one address or a list, **required** |
+| `port` | 465 with `ssl`, else 25 | |
+| `tls` | `false` | STARTTLS on a plain connection (port 587) |
+| `ssl` | `false` | implicit TLS from the start (port 465) |
+| `user` / `password` | — | omit both for an unauthenticated relay |
+| `sender` | `user`, else `expr-tracker` | the `From` address |
+| `html` | `true` | send the HTML part as well as the text |
+
+Use `tls` **or** `ssl`, not both: `tls` upgrades a plain connection, `ssl` starts
+encrypted.
+
+!!! warning "Keep the password out of your code"
+    Read it from the environment, as above. Gmail additionally rejects account
+    passwords for SMTP — turn on 2-step verification and create an
+    [app password](https://myaccount.google.com/apppasswords), which you can
+    revoke independently of your account.
+
+Mail is sent as `multipart/alternative`: a severity-coloured HTML card with the
+fields as a table, plus the plain text as a fallback for clients that will not
+render HTML. Set `html: false` for text only.
+
+Common servers:
+
+| Provider | host | port | setting |
+| --- | --- | --- | --- |
+| Gmail | `smtp.gmail.com` | 587 | `tls: true` (app password required) |
+| Outlook / Office 365 | `smtp.office365.com` | 587 | `tls: true` |
+| QQ / 163 | `smtp.qq.com`, `smtp.163.com` | 465 | `ssl: true` (authorisation code) |
+| SendGrid | `smtp.sendgrid.net` | 587 | `tls: true`, user `apikey` |
+| Internal relay | your host | 25 | often no `user`/`password` |
+
 ### Routing
 
 Each channel filters independently:

{expr_tracker-0.2.1 → expr_tracker-0.2.3}/docs/guide/cli.md

@@ -22,6 +22,25 @@ et history tracker/jsonl/demo/run-1 -n -1 --format csv > run.csv
 
 The argument is a run directory or a `metrics.jsonl` file.
 
+## `et trace`
+
+Export recorded [spans](spans.md) as a Chrome Trace, for
+[Perfetto](https://ui.perfetto.dev) or `chrome://tracing`.
+
+```bash
+et trace runs/llm/sft-1                              # every stream -> trace.json
+et trace runs/llm/sft-1 -o pipeline.json
+et trace runs/llm/sft-1 --stream data                # one stream
+et trace runs/llm/sft-1 --stream default,data        # several
+et trace runs/llm/sft-1 --step-range 100:200         # a slice of the run
+```
+
+| Option | Default | Meaning |
+| --- | --- | --- |
+| `-o`, `--output` | `trace.json` | where to write |
+| `--stream` | every stream | comma separated; `default` is the unnamed one |
+| `--step-range` | all | `start:end`, end exclusive |
+
 ## `et rules explain`
 
 Show how an expression parses, and what it references. Useful when precedence is in

expr_tracker-0.2.3/docs/guide/spans.md

@@ -0,0 +1,169 @@
+# Spans
+
+A step is rarely one thing. `et.span` times the parts, and their parts, and turns
+each duration into an ordinary metric — so `history()`, alert rules and plots all
+work on it with no extra machinery.
+
+```python
+for batch in loader:
+    with et.span("forward"):
+        with et.span("attention"):
+            attn = attention(batch)
+        with et.span("mlp"):
+            out = mlp(attn)
+    with et.span("backward"):
+        loss.backward()
+    et.log({"loss": loss.item()})
+```
+
+The step's row then carries:
+
+```jsonc
+{"_step": 42, "_time": ...,
+ "forward/duration_ms": 31.2,  "forward/count": 1,
+ "forward/attention/duration_ms": 18.4, "forward/attention/count": 1,
+ "forward/mlp/duration_ms": 9.1, "forward/mlp/count": 1,
+ "backward/duration_ms": 44.7, "backward/count": 1,
+ "loss": 0.31}
+```
+
+Nested names join with `/`, so `forward/norm` and `backward/norm` stay distinct,
+and the alert language reads them directly.
+
+## Forms
+
+```python
+with et.span("forward"): ...                # context manager
+
+async with et.span("fetch"): ...            # async
+
+@et.span("preprocess")                      # decorator, sync or async
+def preprocess(batch): ...
+
+span = et.start_span("epoch")               # manual, for crossing scopes
+...
+span.end()
+```
+
+## Repeated spans
+
+The same span several times in one step sums, and counts:
+
+```python
+for layer in layers:            # 32 layers
+    with et.span("layer"):
+        x = layer(x)
+```
+
+```jsonc
+{"layer/duration_ms": 412.8, "layer/count": 32}
+```
+
+The total is usually what you want; divide by the count for the mean.
+
+## A span never commits a step
+
+Durations join whatever `log()` commits, so a span costs no row of its own and
+you can time things before you know what to log:
+
+```python
+with et.span("forward"):
+    ...
+# nothing written yet
+et.log({"loss": loss})     # one row, with the metrics and the durations
+```
+
+## Attributes
+
+Attributes describe an individual span. They reach `spans.jsonl`, not the metrics,
+because they are usually not numbers:
+
+```python
+with et.span("load", batch_size=32) as span:
+    rows = read()
+    span.set(rows=len(rows))
+```
+
+## Alerting on durations
+
+Because a duration is a metric, the [expression language](../reference/expressions.md)
+already handles it:
+
+```python
+et.init(..., alert_rules=[
+    "mean(data/load_ms[50]) > 200 => warning: data loading is slowing down",
+    "forward/duration_ms > 3 * mean(forward/duration_ms[100]) => error: slow step",
+])
+```
+
+## The span file
+
+The full tree is written to `spans.jsonl` beside the metrics, one line per span:
+
+```jsonc
+{"_step": 42, "name": "forward/attention", "depth": 1,
+ "start": 1754323200.123456, "dur_ms": 18.4, "args": {"batch_size": 32}}
+```
+
+Children appear before their parents, because a parent closes last. It follows
+the [stream](streams.md): a data worker writes `spans.data.jsonl`.
+
+Turn it off if you only want the metrics:
+
+```python
+et.init(..., spans=False)
+```
+
+## Viewing the timeline
+
+`et trace` turns the span file into a Chrome Trace, which
+[Perfetto](https://ui.perfetto.dev) and `chrome://tracing` open directly:
+
+```bash
+et trace runs/llm/sft-1 -o trace.json
+et trace runs/llm/sft-1 --stream data --step-range 100:200
+```
+
+Each stream becomes a process and each thread a track, so a data worker and a
+training loop sit on one timeline and the gap where one waited for the other is
+visible. Spans keep their nesting, their step and their attributes.
+
+Exporting a standard format rather than drawing our own view means the result
+can be loaded beside a `torch.profiler` trace, which is usually where the real
+question is: what were the GPUs doing while the loader stalled.
+
+## Errors
+
+An exception is recorded and re-raised — the span never swallows it:
+
+```jsonc
+{"_step": 42, "name": "risky", "dur_ms": 3.1, "error": "ValueError"}
+```
+
+A failure inside the recording itself is logged and ignored: measuring something
+must not break it.
+
+## Threads and tasks
+
+The nesting stack is per thread and per asyncio task, so concurrent work does not
+nest inside unrelated spans:
+
+```python
+async def work(name):
+    async with et.span(name):        # "a" and "b", never "a/b"
+        await asyncio.sleep(1)
+
+await asyncio.gather(work("a"), work("b"))
+```
+
+## Cost
+
+| | per span |
+| --- | --- |
+| default | ~13 µs |
+| `spans=False` | ~6 µs |
+| no active run | ~3 µs |
+
+For comparison, `et.log()` is ~24 µs. Twenty spans on a 100 ms step is 0.26% of
+the step. If your step is closer to a millisecond, set `spans=False` and keep the
+metrics, or time fewer regions.

expr_tracker-0.2.3/docs/guide/streams.md

@@ -0,0 +1,113 @@
+# Streams
+
+A training loop and a data worker measure different things on different clocks:
+training step 100 and data batch 100 are unrelated. A **stream** gives each
+producer its own step cursor and its own file inside one run directory.
+
+```python
+# the training process
+et.init(project="llm", name="sft-1")
+et.log({"train/loss": loss})                    # steps 0, 1, 2, ...
+
+# the data worker, a separate process
+et.init(project="llm", name="sft-1", stream="data")
+et.log({"data/produce_ms": 12.4})               # its own steps 0, 1, 2, ...
+```
+
+Without this, both producers share one cursor: their steps interleave, the merge
+puts unrelated metrics on one row, and `step_policy="monotonic"` drops whichever
+producer happens to fall behind.
+
+## Layout
+
+```
+runs/llm/sft-1/
+├── metrics.jsonl            # the default producer
+├── metrics.data.jsonl       # stream "data"
+├── metrics.meta.json
+├── metrics.data.meta.json
+├── summary.json             # per stream, so processes cannot clobber each other
+├── summary.data.json
+├── config.json
+└── config.data.json
+```
+
+Streams compose with [rank shards](distributed.md): rank 1 of the data worker
+writes `metrics.data.rank1.jsonl`.
+
+Stream names become part of a filename, so they must be letters, digits, `_` or
+`-`, starting with a letter or digit. `rank1` and friends are rejected because
+they already mean a rank shard.
+
+## Reading
+
+```python
+et.history(50)                       # whichever stream this process writes
+et.history(50, stream=None)          # the default producer
+et.history(50, stream="data")        # another stream
+et.history(50, run=path, stream="data")   # offline
+```
+
+Omitting `stream` reads the running stream; passing `stream=None` explicitly means
+the default, unnamed one. Another process's stream is read from its file, so you
+see what it has flushed rather than what it has buffered.
+
+```python
+from expr_tracker.history import list_streams
+list_streams("runs/llm/sft-1")       # [None, "data"]
+```
+
+## Alerts
+
+Each process alerts on what it can see, which is its own stream. That is usually
+what you want — the data worker is the thing that knows its pipeline stalled:
+
+```python
+et.init(
+    project="llm", name="sft-1", stream="data",
+    alert_rules=[
+        "produce_ms > 1000        => warning: data pipeline slow",
+        "no_data(5m)              => error: data worker stopped producing",
+    ],
+)
+```
+
+!!! note
+    A rule cannot span streams that live in different processes, because neither
+    process holds the other's metrics. If you need that, log both from one
+    process, or evaluate the rule downstream against the files.
+
+## Backends
+
+A stream is forwarded as its own backend run, grouped under the run name:
+
+```python
+et.init(project="llm", name="sft-1", stream="data", backends=["wandb"])
+# -> wandb.init(name="sft-1-data", group="sft-1", job_type="data")
+```
+
+Both wandb and trackio understand `group`, and neither can merge two step axes
+into a single run. wandb's shared mode can, but it needs a live server and has no
+trackio equivalent, so grouping is the default.
+
+Override it per backend if you want something else:
+
+```python
+et.init(..., stream="data", backend_kwargs={"wandb": {"group": "my-group"}})
+```
+
+## When you do not need a stream
+
+If the producers are in **one process at different cadences** — an eval loop
+every 100 steps, say — you do not need a stream. Log with the training step and
+the sparse metric simply appears on the steps where you logged it:
+
+```python
+et.log({"train/loss": loss})
+if step % 100 == 0:
+    et.log({"eval/acc": acc}, step=step)
+```
+
+Window functions already work on that, because `eval.acc[20]` counts points of
+that metric, not rows. Reach for a stream when producers are genuinely concurrent
+and their step numbers mean different things.

{expr_tracker-0.2.1 → expr_tracker-0.2.3}/docs/reference/api.md

@@ -23,6 +23,7 @@ et.init(
     backends: Sequence[str] = ("wandb", "jsonl"),
     backend_kwargs: dict[str, dict] | None = None,
     print_to_screen: bool = False,
+    stream: str | None = None,
     alert=None,
     alert_rules: Sequence = (),
     **history_options,
@@ -77,9 +78,13 @@ et.history(
     fill_missing: bool = False,
     dropna: bool = False,
     run: str | Path | None = None,
+    stream: str | None = ...,
 )
 ```
 
+Omit `stream` to read the running one; pass `None` for the default producer. See
+[Streams](../guide/streams.md).
+
 `n=-1` or `None` returns everything. With `run=`, reads that run offline and no
 `init()` is needed. See [Querying history](../guide/history.md).
 
@@ -148,6 +153,27 @@ et.register_backend(kind: str, cls: type[AlertBackend])
 
 Registers a custom channel type.
 
+## Spans
+
+```python
+with et.span(name, **attributes) as span: ...   # also async, also a decorator
+span = et.start_span(name, **attributes)        # ends with span.end()
+span.set(**attributes)
+span.duration_ms
+```
+
+A closed span adds `<path>/duration_ms` and `<path>/count` to the open row and
+appends the full record to `spans.jsonl`. See [Spans](../guide/spans.md).
+
+### Trace export
+
+```python
+from expr_tracker.trace import build_trace, write_trace
+
+write_trace(run, "trace.json", stream="*", step_range=None)   # returns the span count
+build_trace(run, stream="*")                                  # the dict, unwritten
+```
+
 ## Artifacts
 
 ```python