bmsdna-devtools 0.42.0__tar.gz → 0.43.0__tar.gz

bmsdna_devtools-0.42.0/README.md → bmsdna_devtools-0.43.0/PKG-INFO

@@ -1,3 +1,20 @@
+Metadata-Version: 2.5
+Name: bmsdna-devtools
+Version: 0.43.0
+Summary: Shared Azure DevOps / GitHub / git / Azure Monitor developer tooling for BMS projects
+Requires-Python: >=3.14
+Requires-Dist: pgdevkit[db]>=0.7.1
+Requires-Dist: requests>=2.32.0
+Requires-Dist: rich>=13.0.0
+Requires-Dist: tomli-w>=1.0.0
+Requires-Dist: typer>=0.26.7
+Provides-Extra: lint
+Requires-Dist: sqlglot>=27.0.0; extra == 'lint'
+Provides-Extra: usage
+Requires-Dist: duckdb>=1.3.0; extra == 'usage'
+Requires-Dist: plotly>=5.24.0; extra == 'usage'
+Description-Content-Type: text/markdown
+
 # bmsdna-devtools
 
 Shared developer tooling for BMS projects: PR build/check status and a
@@ -29,6 +46,8 @@ uv tool install bmsdna-devtools
 
 Or as a project dependency: `uv add bmsdna-devtools`.
 
+Optional extras: `bmsdna-devtools[lint]` (`bdt lint` SQL rules), `bmsdna-devtools[usage]` (`bdt usage`, below).
+
 ## `bdt pr status`
 
 Find the PR opened from the current branch and report build/check status
@@ -1024,6 +1043,39 @@ such as `'assets/**/*.js'`) or `[tool.bdt.lint] exclude_dirs = [...]` in `pyproj
 Identical findings on the same line are reported once. Silence a confirmed-safe finding with
 `# bdt-lint: ignore <rule>` (Python) or `// bdt-lint: ignore <rule>` (TS/JS) on, or directly above, the line.
 
+## `bdt usage analyze` / `bdt usage sql`
+
+Where do the tokens of your Claude Code and codex sessions go, per step of the `dev-workflow`? Needs the extra:
+`uv tool install 'bmsdna-devtools[usage]'` (DuckDB reads the logs, plotly draws the report; neither is imported by other commands).
+
+```bash
+bdt usage analyze [--scope workflow|all] [--since 2026-10-01] [--price gpt-6-sol=1.25,10,0.125] [--no-codex] [--rebuild]
+bdt usage sql "select label, round(pct_cost, 1) from v_steps"    # ad hoc SQL on the last analysis
+```
+
+It reads `~/.claude/projects` (`--logs`) and `~/.codex/sessions` (`--codex-logs`, skipped if missing), prints a summary and
+writes `report.html` (charts plus a table view, light/dark) with CSV / Parquet exports next to the database
+(`~/.cache/bdt/usage/`, or `--db` / `--out`). The report is self-contained and contains your session titles and agent
+descriptions, so look at it before sharing it.
+
+- **Steps:** setup, plan, plan review, implement, test, screenshots, design review, quick review, full review, publish / CI.
+  Every model turn is assigned to one by markers in its tool calls: skill calls (`bms-code-review --quick|--full`,
+  `design-review`, `bms-plan-review`; for codex, reading the skill's `SKILL.md`), `bdt` commands, edits (`apply_patch` for
+  codex), pytest / lint, screenshot tooling, image reads. A step stays until the next marker; test runs only count for
+  the turn that runs them. Subagents inherit the step of the turn that spawned them. Only threads that follow the workflow
+  (and what they spawn) are counted, `--scope all` takes every session. It is a heuristic: read differences of a few percent
+  as noise. The rules are SQL in `bmsdna/devtools/usage/sql/03_markers.sql`.
+- **Numbers:** cost is the API list price (cache writes 1.25x / 2x, cache reads, output, per model), not your actual bill;
+  "% tokens" counts cache reads too, "% fresh tokens" does not. The report also estimates what screenshots cost by sitting in
+  the context of every later turn.
+- **codex:** a codex run that Claude started with `codex exec` / `codex review` (same directory, within 15 minutes) is counted
+  in that Claude session and step, so reviews run by codex show up under "Review" even though Claude's own log doesn't contain
+  them; codex subagents follow their parent thread. codex models have no built-in price: give one with
+  `--price MODEL=INPUT,OUTPUT,CACHED` (USD per million tokens), repeatable. Turns without a price are counted in tokens
+  but missing from every cost number, and the summary says how many.
+- **Cache:** parsed records are kept in the database, so the next run only parses new or changed log files (a few seconds).
+  `--rebuild` re-reads everything. Message text is not stored.
+
 ## Releasing
 
 Bump `version` in `pyproject.toml` as part of your PR, same as any other

bmsdna_devtools-0.42.0/PKG-INFO → bmsdna_devtools-0.43.0/README.md

@@ -1,17 +1,3 @@
-Metadata-Version: 2.5
-Name: bmsdna-devtools
-Version: 0.42.0
-Summary: Shared Azure DevOps / GitHub / git / Azure Monitor developer tooling for BMS projects
-Requires-Python: >=3.14
-Requires-Dist: pgdevkit[db]>=0.7.1
-Requires-Dist: requests>=2.32.0
-Requires-Dist: rich>=13.0.0
-Requires-Dist: tomli-w>=1.0.0
-Requires-Dist: typer>=0.26.7
-Provides-Extra: lint
-Requires-Dist: sqlglot>=27.0.0; extra == 'lint'
-Description-Content-Type: text/markdown
-
 # bmsdna-devtools
 
 Shared developer tooling for BMS projects: PR build/check status and a
@@ -43,6 +29,8 @@ uv tool install bmsdna-devtools
 
 Or as a project dependency: `uv add bmsdna-devtools`.
 
+Optional extras: `bmsdna-devtools[lint]` (`bdt lint` SQL rules), `bmsdna-devtools[usage]` (`bdt usage`, below).
+
 ## `bdt pr status`
 
 Find the PR opened from the current branch and report build/check status
@@ -1038,6 +1026,39 @@ such as `'assets/**/*.js'`) or `[tool.bdt.lint] exclude_dirs = [...]` in `pyproj
 Identical findings on the same line are reported once. Silence a confirmed-safe finding with
 `# bdt-lint: ignore <rule>` (Python) or `// bdt-lint: ignore <rule>` (TS/JS) on, or directly above, the line.
 
+## `bdt usage analyze` / `bdt usage sql`
+
+Where do the tokens of your Claude Code and codex sessions go, per step of the `dev-workflow`? Needs the extra:
+`uv tool install 'bmsdna-devtools[usage]'` (DuckDB reads the logs, plotly draws the report; neither is imported by other commands).
+
+```bash
+bdt usage analyze [--scope workflow|all] [--since 2026-10-01] [--price gpt-6-sol=1.25,10,0.125] [--no-codex] [--rebuild]
+bdt usage sql "select label, round(pct_cost, 1) from v_steps"    # ad hoc SQL on the last analysis
+```
+
+It reads `~/.claude/projects` (`--logs`) and `~/.codex/sessions` (`--codex-logs`, skipped if missing), prints a summary and
+writes `report.html` (charts plus a table view, light/dark) with CSV / Parquet exports next to the database
+(`~/.cache/bdt/usage/`, or `--db` / `--out`). The report is self-contained and contains your session titles and agent
+descriptions, so look at it before sharing it.
+
+- **Steps:** setup, plan, plan review, implement, test, screenshots, design review, quick review, full review, publish / CI.
+  Every model turn is assigned to one by markers in its tool calls: skill calls (`bms-code-review --quick|--full`,
+  `design-review`, `bms-plan-review`; for codex, reading the skill's `SKILL.md`), `bdt` commands, edits (`apply_patch` for
+  codex), pytest / lint, screenshot tooling, image reads. A step stays until the next marker; test runs only count for
+  the turn that runs them. Subagents inherit the step of the turn that spawned them. Only threads that follow the workflow
+  (and what they spawn) are counted, `--scope all` takes every session. It is a heuristic: read differences of a few percent
+  as noise. The rules are SQL in `bmsdna/devtools/usage/sql/03_markers.sql`.
+- **Numbers:** cost is the API list price (cache writes 1.25x / 2x, cache reads, output, per model), not your actual bill;
+  "% tokens" counts cache reads too, "% fresh tokens" does not. The report also estimates what screenshots cost by sitting in
+  the context of every later turn.
+- **codex:** a codex run that Claude started with `codex exec` / `codex review` (same directory, within 15 minutes) is counted
+  in that Claude session and step, so reviews run by codex show up under "Review" even though Claude's own log doesn't contain
+  them; codex subagents follow their parent thread. codex models have no built-in price: give one with
+  `--price MODEL=INPUT,OUTPUT,CACHED` (USD per million tokens), repeatable. Turns without a price are counted in tokens
+  but missing from every cost number, and the summary says how many.
+- **Cache:** parsed records are kept in the database, so the next run only parses new or changed log files (a few seconds).
+  `--rebuild` re-reads everything. Message text is not stored.
+
 ## Releasing
 
 Bump `version` in `pyproject.toml` as part of your PR, same as any other

{bmsdna_devtools-0.42.0 → bmsdna_devtools-0.43.0}/bmsdna/devtools/cli.py

@@ -35,6 +35,7 @@ from . import pr_build, pr_info as pr_info_mod, pr_issue_link, pr_labels, pull a
 from .ado_auth import auth_header
 from .bdt_config import find_pyproject, load_bdt_table
 from .cli_tools import CLI_TIMEOUT_SECS, EXIT_WAIT_TIMEOUT, detect_agent_session, require_az, require_gh
+from .usage.cli import usage_app
 from .gitrepo import (
     AdoRemote,
     GitHubRemote,
@@ -103,6 +104,8 @@ issue_app.add_typer(issue_comment_app, name="comment")
 logs_app = typer.Typer(name="logs", help="Application Insights / Log Analytics queries")
 app.add_typer(logs_app, name="logs")
 
+app.add_typer(usage_app, name="usage")
+
 cleanup_app = typer.Typer(
     name="cleanup",
     help="Prune merged git worktrees and their orphaned pgdevkit Postgres test databases",

bmsdna_devtools-0.43.0/bmsdna/devtools/usage/__init__.py

@@ -0,0 +1,5 @@
+"""`bdt usage`: where do the tokens of Claude Code / codex sessions go, per step of the dev-workflow?
+
+Reads the local session logs with DuckDB, attributes every model turn to a workflow step and prices it.
+The heavy dependencies (duckdb, plotly) are the optional `usage` extra and only imported when a command runs.
+"""

bmsdna_devtools-0.43.0/bmsdna/devtools/usage/cli.py

@@ -0,0 +1,122 @@
+"""`bdt usage`: profile Claude Code / codex session logs per step of the dev-workflow."""
+
+from __future__ import annotations
+
+import os
+import sys
+from datetime import date
+from pathlib import Path
+
+import typer
+
+from . import pricing
+
+usage_app = typer.Typer(
+    name="usage",
+    help="Where do the tokens of Claude Code / codex sessions go, per step of the dev-workflow? "
+    "(needs the `usage` extra: `uv tool install 'bmsdna-devtools\\[usage]'`)",
+)
+
+INSTALL_HINT = "bdt usage needs duckdb and plotly: install the extra, e.g. `uv tool install --reinstall 'bmsdna-devtools[usage]'`"
+DEFAULT_CLAUDE_LOGS = "~/.claude/projects"
+DEFAULT_CODEX_LOGS = "~/.codex/sessions"
+
+
+def default_data_dir() -> Path:
+    cache = Path(os.environ.get("XDG_CACHE_HOME") or "~/.cache").expanduser()
+    return cache / "bdt" / "usage"
+
+
+def require_duckdb():
+    try:
+        import duckdb
+    except ImportError:
+        print(INSTALL_HINT, file=sys.stderr)
+        raise typer.Exit(1) from None
+    return duckdb
+
+
+@usage_app.command("analyze")
+def analyze(
+    logs: str = typer.Option(DEFAULT_CLAUDE_LOGS, "--logs", help="Claude Code projects directory"),
+    codex_logs: str = typer.Option(DEFAULT_CODEX_LOGS, "--codex-logs", help="codex sessions directory (skipped if it does not exist)"),
+    no_codex: bool = typer.Option(False, "--no-codex", help="Ignore codex logs"),
+    db: str | None = typer.Option(None, "--db", help="DuckDB file (default: <cache dir>/bdt/usage/profile.duckdb). Doubles as the parse cache"),
+    out: str | None = typer.Option(None, "--out", help="Directory for report.html and the CSV / Parquet exports (default: next to the database)"),
+    scope: str = typer.Option("workflow", "--scope", help="workflow: only threads that follow dev-workflow (and what they spawn); all: every session"),
+    since: str = typer.Option("1970-01-01", "--since", help="Only messages from this date on (YYYY-MM-DD)"),
+    rebuild: bool = typer.Option(False, "--rebuild", help="Ignore the parse cache in the database and re-read every log file"),
+    no_report: bool = typer.Option(False, "--no-report", help="Only print the summary"),
+    price: list[str] = typer.Option(
+        [], "--price", metavar="MODEL=IN,OUT,CACHED",
+        help="USD per million tokens for a model without a built-in price (codex / OpenAI models), repeatable. "
+        "Example: --price gpt-6-sol=1.25,10,0.125",
+    ),
+) -> None:
+    """Load the session logs, attribute every model turn to a workflow step, price it, and print / render the result.
+
+    The first run parses every log file; later runs only parse new or changed ones. Costs are API list prices
+    (not your actual bill); turns of a model without a price are counted in tokens but not in cost.
+    """
+    if scope not in ("workflow", "all"):
+        raise typer.BadParameter("Must be 'workflow' or 'all'", param_hint="--scope")
+    try:
+        date.fromisoformat(since)
+    except ValueError:
+        raise typer.BadParameter("Must be a date like 2026-10-01", param_hint="--since") from None
+    try:
+        prices = dict(pricing.parse_price(p) for p in price)
+    except ValueError as e:
+        raise typer.BadParameter(str(e), param_hint="--price") from None
+    duckdb = require_duckdb()
+    from . import pipeline
+
+    data_dir = default_data_dir()
+    db_path = Path(db).expanduser() if db else data_dir / "profile.duckdb"
+    db_path.parent.mkdir(parents=True, exist_ok=True)
+    con = duckdb.connect(str(db_path))
+    print(f"Reading {logs}" + ("" if no_codex else f" and {codex_logs}") + " ...", file=sys.stderr)
+    stats = pipeline.build(
+        con, Path(logs), None if no_codex else Path(codex_logs), scope=scope, since=since, rebuild=rebuild, prices=prices
+    )
+    print(f"{stats.files} log files, {stats.ingested} parsed ({stats.files - stats.ingested} from cache), {stats.removed} removed", file=sys.stderr)
+
+    summary = con.execute("SELECT * FROM v_summary").fetchone()
+    if not summary or not summary[2]:
+        print("No sessions in scope. Nothing used dev-workflow in these logs (try --scope all)?", file=sys.stderr)
+        raise typer.Exit(1)
+    con.sql("SELECT * EXCLUDE (unpriced_msgs, unpriced_models) FROM v_summary").show(max_width=200)
+    con.sql("""
+        SELECT label AS step, msgs AS turns, sessions, round(cost, 2) AS usd,
+               round(pct_cost, 1) AS "% cost", round(pct_tokens, 1) AS "% tokens",
+               round(pct_fresh_tokens, 1) AS "% fresh tokens",
+               round(pct_of_step_via_subagents) AS "% via subagents"
+        FROM v_steps""").show(max_width=200)
+    con.sql("SELECT provider, msgs AS turns, tokens, round(cost, 2) AS usd, round(pct_tokens, 1) AS \"% tokens\" FROM v_providers").show(max_width=200)
+    unpriced = con.execute("SELECT unpriced_msgs, unpriced_models FROM v_summary").fetchone()
+    if unpriced and unpriced[0]:
+        print(
+            f"Note: {unpriced[0]} turns of {unpriced[1]} have no price, so they are missing from the cost columns. "
+            "Add one with --price MODEL=IN,OUT,CACHED (USD per million tokens).",
+            file=sys.stderr,
+        )
+    if not no_report:
+        from . import report
+
+        out_dir = Path(out).expanduser() if out else db_path.parent
+        print(f"Report: {report.write_report(con, out_dir)}", file=sys.stderr)
+
+
+@usage_app.command("sql")
+def sql(
+    query: str = typer.Argument(..., help="SQL to run against a database built by `bdt usage analyze` (tables: facts, msgs, threads, v_*)"),
+    db: str | None = typer.Option(None, "--db", help="DuckDB file (default: the one `analyze` writes)"),
+) -> None:
+    """Run ad-hoc SQL on the last analysis, e.g. `bdt usage sql "select label, round(pct_cost, 1) from v_steps"`."""
+    duckdb = require_duckdb()
+    db_path = Path(db).expanduser() if db else default_data_dir() / "profile.duckdb"
+    if not db_path.exists():
+        print(f"{db_path} does not exist yet: run `bdt usage analyze` first.", file=sys.stderr)
+        raise typer.Exit(1)
+    con = duckdb.connect(str(db_path), read_only=True)
+    con.sql(query).show(max_width=200, max_rows=100)

bmsdna_devtools-0.43.0/bmsdna/devtools/usage/pipeline.py

@@ -0,0 +1,255 @@
+"""Build the usage database: load the session logs with DuckDB, assign workflow steps, derive views."""
+
+from __future__ import annotations
+
+import os
+from collections import defaultdict
+from dataclasses import dataclass
+from datetime import datetime
+from importlib import resources
+from pathlib import Path
+from typing import TYPE_CHECKING
+
+from . import pricing
+
+if TYPE_CHECKING:
+    import duckdb
+
+REVIEW_PHASES = {"review_quick", "review_full"}
+KIND_RANK = {"start": 0, "skill": 1, "review_agent": 2, "cmd": 3, "edit": 4, "soft": 5, "once": 6}
+
+# Bump when the rec_* tables or their parsing change: caches with another version are rebuilt.
+CACHE_VERSION = "2"
+REC_TABLES = ("rec_assistant", "rec_images", "rec_sessions", "rec_cmds", "rec_threads")
+
+# provider -> (ingest script, glob below the log directory)
+PROVIDERS = {
+    "claude": ("01_ingest_claude.sql", "*.jsonl"),
+    "codex": ("01_ingest_codex.sql", "*.jsonl"),
+}
+
+
+@dataclass
+class BuildStats:
+    files: int
+    ingested: int
+    removed: int
+
+
+def _sql(name: str, **subs: str) -> str:
+    text = resources.files("bmsdna.devtools.usage").joinpath("sql", name).read_text(encoding="utf-8")
+    for key, value in subs.items():
+        text = text.replace(f"__{key}__", value)
+    return text
+
+
+def _quote(value: str) -> str:
+    return value.replace("'", "''")
+
+
+def _sql_list(values: list[str]) -> str:
+    return "[" + ", ".join(f"'{_quote(v)}'" for v in values) + "]"
+
+
+def assign_phases(con: duckdb.DuckDBPyConnection, scope: str) -> None:
+    """Walk each thread in time order and give every message a workflow phase.
+
+    A thread that follows the workflow (invoked dev-workflow, or scope='all') starts in 'setup' and
+    switches phase on the next marker; the phase stays "sticky" until the next one (test runs are the
+    exception: they only count for the turn that runs them). Other spawned threads inherit the phase of
+    the message that spawned them and only react to strong markers (review/design/plan skills).
+    """
+    msgs_by_thread: dict[str, list[tuple[str, datetime]]] = defaultdict(list)
+    thread_of_mid: dict[str, str] = {}
+    for tid, mid, ts in con.execute("SELECT thread_id, mid, ts FROM msgs ORDER BY thread_id, ts, mid").fetchall():
+        msgs_by_thread[tid].append((mid, ts))
+        thread_of_mid[mid] = tid
+
+    markers: dict[str, list[tuple[str, str]]] = defaultdict(list)
+    for mid, phase, kind in con.execute("SELECT mid, phase, kind FROM tool_markers ORDER BY tool_use_id").fetchall():
+        markers[mid].append((phase, kind))
+
+    meta = {r[0]: (r[1], r[2] or "") for r in con.execute("SELECT thread_id, parent_mid, agent_type FROM threads").fetchall()}
+
+    start_ts: dict[str, datetime] = {}
+    for tid, ts in con.execute(
+        "SELECT m.thread_id, min(m.ts) FROM msgs m JOIN tool_markers k USING (mid) WHERE k.kind = 'start' GROUP BY 1"
+    ).fetchall():
+        start_ts[tid] = ts
+    for tid, ts in con.execute("SELECT thread_id, ts FROM workflow_commands").fetchall():
+        if tid in msgs_by_thread:
+            start_ts[tid] = min(start_ts.get(tid, ts), ts)
+    # Spawned threads that run the whole workflow themselves: told so in the spawn prompt (Claude), or seen doing it.
+    for tid, ts in con.execute(
+        """
+        SELECT th.thread_id, min(m.ts)
+        FROM threads th
+        JOIN msgs m ON m.thread_id = th.thread_id
+        WHERE th.thread_id IN (SELECT m2.thread_id FROM msgs m2 JOIN tool_markers k USING (mid) WHERE k.wf_signal)
+           OR th.thread_id IN (
+                SELECT am.thread_id
+                FROM agent_meta am
+                JOIN tool_uses tu ON tu.tool_use_id = am.parent_tool_use_id
+                WHERE tu.name IN ('Agent', 'Task')
+                  AND regexp_matches(lower(json_extract_string(tu.input, '$.prompt')),
+                                     'follow (the )?[`~/.\\w-]{0,40}dev-workflow|follow the workflow')
+                  AND coalesce(json_extract_string(tu.input, '$.subagent_type'), '') NOT LIKE '%bms-review-%')
+        GROUP BY 1
+        """
+    ).fetchall():
+        start_ts.setdefault(tid, ts)
+    if scope == "all":
+        for tid, items in msgs_by_thread.items():
+            if tid not in meta:
+                start_ts[tid] = items[0][1]
+
+    phase_of: dict[str, str] = {}
+    sticky_of: dict[str, str] = {}  # the phase a spawned thread inherits
+    pending = set(msgs_by_thread)
+
+    def run_thread(tid: str, inherited: str) -> None:
+        agent_type = meta.get(tid, (None, ""))[1]
+        if "bms-review-" in agent_type and inherited not in REVIEW_PHASES and inherited != "outside":
+            inherited = "review_full"
+        phase, started, begin = inherited, False, start_ts.get(tid)
+        for mid, ts in msgs_by_thread[tid]:
+            if begin is not None and not started and ts >= begin:
+                phase, started = "setup", True
+            once = None
+            for ph, kind in sorted(markers.get(mid, []), key=lambda m: KIND_RANK[m[1]]):
+                if kind == "skill" and phase != "outside":
+                    phase = ph
+                elif kind == "review_agent" and phase != "outside" and phase not in REVIEW_PHASES:
+                    phase = ph
+                elif kind in ("cmd", "edit") and started:
+                    phase = ph
+                elif kind == "soft" and started and phase != "design_review":
+                    phase = ph
+                elif kind == "once" and started:
+                    once = ph
+            phase_of[mid] = once or phase
+            sticky_of[mid] = phase
+
+    while pending:
+        progressed = False
+        for tid in sorted(pending):
+            parent_mid = meta.get(tid, (None, None))[0]
+            parent_thread = thread_of_mid.get(parent_mid) if parent_mid else None
+            if parent_thread in pending and parent_thread != tid:
+                continue
+            inherited = sticky_of.get(parent_mid, "outside") if parent_mid else "outside"
+            run_thread(tid, inherited)
+            pending.discard(tid)
+            progressed = True
+        if not progressed:
+            for tid in sorted(pending):
+                run_thread(tid, "outside")
+            break
+
+    con.execute("CREATE OR REPLACE TABLE msg_phase (mid VARCHAR, phase VARCHAR)")
+    con.execute("INSERT INTO msg_phase SELECT unnest(?), unnest(?)", [list(phase_of), list(phase_of.values())])
+
+
+def ingest(
+    con: duckdb.DuckDBPyConnection, sources: dict[str, Path], rebuild: bool = False
+) -> BuildStats:
+    """Bring the rec_* cache tables up to date, parsing only new or changed log files.
+
+    `sources` maps a provider ("claude", "codex") to the directory holding its logs; a provider that is not
+    listed is left alone (its cached rows stay, they are just not analysed, see `build`).
+    """
+    con.execute(_sql("00_schema.sql"))
+    row = con.execute("SELECT value FROM cache_meta WHERE key = 'version'").fetchone()
+    if rebuild or (row and row[0] != CACHE_VERSION):
+        for table in (*REC_TABLES, "src_files"):
+            con.execute(f"DELETE FROM {table}")
+    con.execute("DELETE FROM cache_meta WHERE key = 'version'")
+    con.execute("INSERT INTO cache_meta VALUES ('version', ?)", [CACHE_VERSION])
+
+    files = ingested = removed = 0
+    for provider, root in sources.items():
+        script, pattern = PROVIDERS[provider]
+        on_disk: dict[str, tuple[int, int]] = {}
+        for path in root.rglob(pattern):
+            st = path.stat()
+            on_disk[str(path)] = (st.st_size, st.st_mtime_ns)
+        prefix = str(root) + "/"
+        known = {
+            f: (size, mtime)
+            for f, size, mtime in con.execute(
+                "SELECT filename, size, mtime_ns FROM src_files WHERE starts_with(filename, ?)", [prefix]
+            ).fetchall()
+        }
+        changed = sorted(f for f, stat in on_disk.items() if known.get(f) != stat)
+        gone = sorted(set(known) - set(on_disk))
+
+        stale = changed + gone
+        if stale:
+            con.execute("CREATE OR REPLACE TEMP TABLE stale AS SELECT unnest(?) AS filename", [stale])
+            for table in (*REC_TABLES, "src_files"):
+                con.execute(f"DELETE FROM {table} WHERE filename IN (SELECT filename FROM stale)")
+        if changed:
+            con.execute(_sql(script, FILES=_sql_list(changed)))
+            con.execute(
+                "INSERT INTO src_files SELECT unnest(?), unnest(?), unnest(?)",
+                [changed, [on_disk[f][0] for f in changed], [on_disk[f][1] for f in changed]],
+            )
+        files += len(on_disk)
+        ingested += len(changed)
+        removed += len(gone)
+    return BuildStats(files=files, ingested=ingested, removed=removed)
+
+
+_META_COLUMNS = {"agentType": "VARCHAR", "description": "VARCHAR", "toolUseId": "VARCHAR",
+                 "spawnDepth": "INTEGER", "model": "VARCHAR"}
+
+
+def _meta_source(claude_dir: Path | None) -> str:
+    """FROM-clause for the Claude subagent meta files (an empty table if there are none: read_json errors on no match)."""
+    cols = ", ".join(f"{k}: '{v}'" for k, v in _META_COLUMNS.items())
+    files = sorted(str(p) for p in claude_dir.rglob("subagents/*.meta.json")) if claude_dir else []
+    if files:
+        return (f"read_json({_sql_list(files)}, union_by_name = true, ignore_errors = true, filename = true, "
+                f"columns = {{{cols}}})")
+    empty = ", ".join(f"NULL::{v} AS {k}" for k, v in _META_COLUMNS.items())
+    return f"(SELECT NULL::VARCHAR AS filename, {empty} WHERE false)"
+
+
+def build(
+    con: duckdb.DuckDBPyConnection,
+    claude_dir: Path | None,
+    codex_dir: Path | None = None,
+    scope: str = "workflow",
+    since: str = "1970-01-01",
+    rebuild: bool = False,
+    prices: dict[str, pricing.Price] | None = None,
+) -> BuildStats:
+    """Ingest the given log directories (None = skip that provider) and derive everything from the cache."""
+    sources = {
+        provider: directory.expanduser()
+        for provider, directory in (("claude", claude_dir), ("codex", codex_dir))
+        if directory is not None and directory.expanduser().is_dir()
+    }
+    # The JSON reader keeps a buffer of `maximum_object_size` (16 MB in the ingest scripts) per thread: on a busy,
+    # many-core machine that is gigabytes for no gain, since the log files are read in a few seconds anyway.
+    con.execute(f"SET threads = {min(os.cpu_count() or 4, 8)}")
+    stats = ingest(con, sources, rebuild=rebuild)
+    con.execute(_sql(
+        "02_messages.sql",
+        META_SOURCE=_meta_source(sources.get("claude")),
+        SINCE=_quote(since),
+        PROVIDERS=", ".join(f"'{p}'" for p in sources) or "''",
+    ))
+    con.execute(_sql("03_markers.sql"))
+    assign_phases(con, scope)
+    rows = ", ".join(f"('{_quote(m)}', {i}, {o}, {c})" for m, i, o, c in pricing.price_rows(prices))
+    con.execute(_sql(
+        "04_facts.sql",
+        SINCE=_quote(since),
+        PRICE_ROWS=rows,
+        FALLBACK_MODEL=pricing.FALLBACK_MODEL,
+        CW5=str(pricing.CACHE_WRITE_5M_FACTOR),
+        CW1H=str(pricing.CACHE_WRITE_1H_FACTOR),
+    ))
+    con.execute(_sql("05_views.sql"))
+    return stats

bmsdna_devtools-0.43.0/bmsdna/devtools/usage/pricing.py

@@ -0,0 +1,59 @@
+"""USD per million tokens. Edit freely, or pass `--price`. An unknown Claude model falls back to FALLBACK_MODEL;
+any other model (codex / OpenAI) has no built-in price and stays unpriced until you give it one.
+
+Cache writes are priced relative to input: 1.25x for the 5 minute TTL, 2x for the 1 hour TTL.
+Prices are API list prices, so the result is an "API-equivalent cost" (useful for relative
+shares even if you are on a flat subscription).
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+
+CACHE_WRITE_5M_FACTOR = 1.25
+CACHE_WRITE_1H_FACTOR = 2.0
+FALLBACK_MODEL = "claude-sonnet-5-5"
+
+
+@dataclass(frozen=True)
+class Price:
+    input: float
+    output: float
+    cache_read: float
+
+
+PRICES: dict[str, Price] = {
+    "claude-fable-5-1": Price(10.0, 50.0, 0.25),
+    "claude-fable-5": Price(10.0, 50.0, 1.0),
+    "claude-opus-5-5": Price(4.0, 20.0, 0.20),
+    "claude-opus-5": Price(5.0, 25.0, 0.50),
+    "claude-opus-4-8": Price(5.0, 25.0, 0.50),
+    "claude-opus-4-7": Price(5.0, 25.0, 0.50),
+    "claude-opus-4-6": Price(5.0, 25.0, 0.50),
+    "claude-sonnet-5-5": Price(2.0, 10.0, 0.20),
+    "claude-sonnet-5": Price(2.0, 10.0, 0.20),
+    "claude-sonnet-4-6": Price(3.0, 15.0, 0.30),
+    "claude-haiku-4-5": Price(1.0, 5.0, 0.10),
+    "claude-haiku-4-5-20251001": Price(1.0, 5.0, 0.10),
+    "<synthetic>": Price(0.0, 0.0, 0.0),
+}
+
+
+def parse_price(option: str) -> tuple[str, Price]:
+    """`--price MODEL=INPUT,OUTPUT,CACHE_READ` (USD per million tokens), e.g. `gpt-6-sol=1.25,10,0.125`."""
+    model, sep, rest = option.partition("=")
+    parts = rest.split(",")
+    try:
+        if not (sep and model.strip() and len(parts) == 3):
+            raise ValueError
+        inp, out, cached = (float(x) for x in parts)
+    except ValueError:
+        raise ValueError(f"Expected MODEL=INPUT,OUTPUT,CACHE_READ (USD per million tokens), got {option!r}") from None
+    if min(inp, out, cached) < 0:
+        raise ValueError(f"Prices must not be negative: {option!r}")
+    return model.strip(), Price(inp, out, cached)
+
+
+def price_rows(extra: dict[str, Price] | None = None) -> list[tuple[str, float, float, float]]:
+    """Rows for the `pricing` table: the built-in prices, overridden / extended by `extra`."""
+    return [(m, p.input, p.output, p.cache_read) for m, p in {**PRICES, **(extra or {})}.items()]