witan-code 0.13.5__tar.gz → 0.14.0__tar.gz

{witan_code-0.13.5 → witan_code-0.14.0}/CHANGELOG.md

@@ -6,6 +6,45 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/) (pre-1.0:
 a MINOR bump may include breaking changes).
 
+## [0.14.0] - 2026-08-21
+
+### Fixed
+
+- **A failed index now reports at least as much as a successful one.** On
+  success `witan code index` prints `scanned=… indexed=… skipped=… symbols=…`;
+  on failure the exception replaced that line entirely, so the run you most
+  need numbers from was the only one that produced none. Attributing the CI
+  indexer's `ol-infrastructure` timeout (0.13.3) therefore took five days and
+  needed a traceback line number, a source read to find the batch constant, row
+  counts from three graphs, an ECR check, and a live timing experiment —
+  nearly all of it answerable from numbers the process already had.
+
+  `index_path` now raises `IndexFailed`, carrying the partial `IndexStats`, the
+  write phase that died (the stale-row delete, or either of the two loads), how
+  long it ran, and the sizes it was working with — statements composed, chunk
+  size in use, records loaded. The CLI prints those stats under `partial …` and
+  re-raises, so the exit code and traceback are unchanged. The session hooks
+  and the store-missing fallbacks already catch broadly and are unaffected;
+  `code_reindex` still propagates, and its caller now gets a message carrying
+  the phase and the sizes rather than a bare transport error.
+
+  Requires `witan-core>=0.31` for the transport and batching halves of the same
+  change: a timeout that names its budget, and a chunk failure that names its
+  position. (0.31, not 0.30 — agent-kit#280 took 0.30.0 for unrelated work while
+  this was open, so the release carrying those halves is 0.31.0.)
+
+### Changed
+
+- **The CI sweep's last line names the failed repos, not just the count.**
+  `indexed 13, failed 1` is the first thing an operator reads and did not say
+  which of fourteen repos was a run staler than it should be, so answering that
+  meant scrolling back through fourteen repos' output.
+
+  The exit status stays non-zero for a partial sweep, and the reasoning is now
+  recorded next to it: a Job that reports Complete with one repo stale is a Job
+  nothing can alert on, and a stale index has no other symptom — which is why
+  three consecutive failures from 2026-08-07 went unnoticed for two days.
+
 ## [0.13.5] - 2026-08-21
 
 ### Changed

{witan_code-0.13.5 → witan_code-0.14.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: witan-code
-Version: 0.13.5
+Version: 0.14.0
 Summary: witan-code — tree-sitter code graph + cross-repo bridge (mounts under `witan code`)
 Project-URL: Homepage, https://github.com/mitodl/agent-kit/tree/main/mcp/servers/witan-code
 Project-URL: Repository, https://github.com/mitodl/agent-kit
@@ -26,7 +26,7 @@ Requires-Dist: tree-sitter-sql<0.4,>=0.3
 Requires-Dist: tree-sitter-typescript<0.24,>=0.23
 Requires-Dist: tree-sitter-yaml<0.8,>=0.7
 Requires-Dist: tree-sitter<0.26,>=0.25
-Requires-Dist: witan-core[cli,observability,remote,sentry]<1,>=0.28
+Requires-Dist: witan-core[cli,observability,remote,sentry]<1,>=0.31
 Provides-Extra: tasks
 Requires-Dist: fastmcp-tasks<5,>=4.0.0b1; extra == 'tasks'
 Description-Content-Type: text/markdown

{witan_code-0.13.5 → witan_code-0.14.0}/docs/USER_GUIDE.md

@@ -110,6 +110,13 @@ copy the matching snippet from `config/` into your agent's config — see
    errors=…`). Re-running `index` is incremental — unchanged files are
    skipped by content hash; use `reindex` to force a full rebuild.
 
+   A run that fails while writing prints the same counts under `partial …`,
+   followed by the phase that failed, how long it ran, and the batch sizes it
+   was working with. Those numbers are the diagnosis: they say whether the
+   write was large, how far it got, and which of the deletes or the two loads
+   died. The exit status is unchanged, so a sweep still counts the repo as
+   failed.
+
 3. **Ask "where is this defined"** — via the CLI or, more commonly, through
    an agent calling the MCP tool directly:
 

{witan_code-0.13.5 → witan_code-0.14.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "witan-code"
-version = "0.13.5"
+version = "0.14.0"
 description = "witan-code — tree-sitter code graph + cross-repo bridge (mounts under `witan code`)"
 readme = "README.md"
 license = "BSD-3-Clause"
@@ -18,10 +18,18 @@ dependencies = [
     # >=0.12: remote/store.py imports RemotePayloadTooLarge and
     # payload_too_large from witan_core.remote.proxy at module scope, and 0.11
     # exports neither. A looser floor lets an external install resolve a
-    # witan-core the server cannot even import; that is what this pin is for,
-    # and it has now been tripped THREE times. If you add a witan_core symbol in
-    # the same change as its caller, bump this line in that change: the
-    # workspace resolves witan-core by path, so nothing here can catch it.
+    # witan-core the server cannot even import; that is what this pin is for.
+    # If you add a witan_core symbol in the same change as its caller, bump this
+    # line in that change: the workspace resolves witan-core by path, so no test
+    # run in it can catch you.
+    #
+    # ★ `just check-core-floor` NOW CATCHES THAT, on ordinary PR CI
+    # (.github/workflows/core-floor.yml). It installs this package's wheel into
+    # a clean venv with witan-core pinned to exactly the floor below and imports
+    # every module — the failure mode itself, not a model of it. It escaped
+    # three times before that check existed (#198, #200, #201), each time with a
+    # green CI; the tally used to live in this comment, which was a comment
+    # doing a test's job.
     # (NOT raised for witan-core 0.19/0.20's trace-context work, deliberately.
     # Nothing here imports `trace_context_middleware`, and standalone
     # `witan-code serve` is stdio-only — the ASGI middleware is installed by
@@ -87,7 +95,28 @@ dependencies = [
     # Expect this floor to keep moving while omnigraph 0.10.0 stays on the
     # moving `edge` tag; see
     # packages/witan-core/witan_core/omnigraph_install.py.
-    "witan-core[cli,remote,observability,sentry]>=0.28,<1",
+    #
+    # ★ >=0.31 IS THE THIRD NON-IMPORT FLOOR, and the narrowest: it buys
+    # message content, not correctness. Nothing new is imported and an older
+    # witan-core runs fine. But witan-code 0.14.0 is ENTIRELY a diagnostics
+    # release, and two of the four questions a failed index has to answer are
+    # answered on the witan-core side — which chunk of how many failed
+    # (`change_many`), and which timeout budget expired (`omnigraph_http`).
+    # Resolved below 0.31 the run still prints its partial summary and names
+    # the phase, and the inner error is still the bare "timed out" this
+    # release exists to replace. A package whose changelog promises a
+    # self-describing failure and delivers half of one is the quiet wrongness
+    # the two floors above were raised to avoid.
+    #
+    # ★ THE NUMBER MOVED 0.30 -> 0.31 ON REBASE, and that is the whole hazard
+    # this note exists to flag. agent-kit#280 took 0.30.0 for unrelated work
+    # while this branch was open, so the release carrying the diagnostics is
+    # 0.31.0 — and a floor left at >=0.30 would resolve #280's release instead
+    # and quietly deliver the half-message described above. `just
+    # check-core-floor` cannot catch it: nothing new is imported, so the
+    # clean-install import check passes at either number. Re-check this line by
+    # hand whenever this branch is rebased.
+    "witan-core[cli,remote,observability,sentry]>=0.31,<1",
     "fastmcp>=3.4.2,<5",
     "cyclopts>=4,<5",
     # [targets.<name>] override models (config.py) — also transitively pulled
@@ -171,7 +200,7 @@ testpaths = ["tests"]
 packages = ["witan_code"]
 
 [tool.bumpversion]
-current_version = "0.13.5"
+current_version = "0.14.0"
 allow_dirty = true
 
 [[tool.bumpversion.files]]

witan_code-0.14.0/tests/test_index_failure_diagnostics.py

@@ -0,0 +1,149 @@
+"""A failed index has to report at least as much as a successful one.
+
+The run you most need numbers from was the only one that produced none: on
+success `witan code index` prints `scanned=… indexed=… skipped=… symbols=…`,
+and on failure the exception replaced that line entirely. Attributing the CI
+indexer's `ol-infrastructure` timeout therefore needed a traceback line number,
+a source read to find the batch constant, row counts from three graphs and a
+live timing experiment — nearly all of which the numbers below answer directly.
+"""
+
+from __future__ import annotations
+
+from pathlib import Path
+
+import pytest
+
+from witan_code import cli as cli_module
+from witan_code import indexer
+
+
+def _stats(**kwargs) -> indexer.IndexStats:
+    return indexer.IndexStats(**kwargs)
+
+
+# ── the exception carries the sizes ────────────────────────────────
+
+
+def test_a_write_failure_keeps_the_partial_stats():
+    stats = _stats(scanned=657, indexed=165, skipped=492, symbols=4102)
+
+    with pytest.raises(indexer.IndexFailed) as raised:
+        with indexer._write_phase("delete of stale rows", stats, statements=330):
+            raise RuntimeError("timed out")
+
+    assert raised.value.stats is stats
+    assert raised.value.stats.scanned == 657
+
+
+def test_the_message_names_the_phase_and_what_it_was_working_with():
+    with pytest.raises(indexer.IndexFailed) as raised:
+        with indexer._write_phase(
+            "delete of stale rows", _stats(), statements=330, chunk_size=128
+        ):
+            raise RuntimeError("mutate failed: timed out")
+
+    message = str(raised.value)
+    assert message.startswith("delete of stale rows failed after ")
+    assert "statements=330" in message
+    assert "chunk_size=128" in message
+    assert "mutate failed: timed out" in message
+
+
+def test_the_real_error_stays_reachable():
+    """The wrapper adds sizes; it must not cost anyone the original."""
+    cause = RuntimeError("timed out")
+
+    with pytest.raises(indexer.IndexFailed) as raised:
+        with indexer._write_phase("load of nodes and edges", _stats(), records=9000):
+            raise cause
+
+    assert raised.value.__cause__ is cause
+
+
+def test_elapsed_is_measured_not_guessed(monkeypatch):
+    """Elapsed is what says whether a client budget was reached or something
+    died early — a server-side cutoff and a client timeout both arrive as a
+    failed write, and only the elapsed time separates them."""
+    clock = {"t": 10.0}
+    monkeypatch.setattr(indexer.time, "monotonic", lambda: clock["t"])
+
+    with pytest.raises(indexer.IndexFailed) as raised:
+        with indexer._write_phase("delete of stale rows", _stats()):
+            clock["t"] += 181.3
+            raise RuntimeError("timed out")
+
+    assert raised.value.elapsed == pytest.approx(181.3)
+    assert "failed after 181.3s" in str(raised.value)
+
+
+def test_a_successful_phase_is_transparent():
+    with indexer._write_phase("load of nodes and edges", _stats(), records=1):
+        pass
+
+
+# ── the CLI prints them ────────────────────────────────────────────
+
+
+def test_the_cli_prints_the_partial_summary_before_re_raising(monkeypatch, capsys):
+    stats = _stats(scanned=657, indexed=165, skipped=492, symbols=4102, purged=3)
+
+    def failing_index(path, *, force=False, **kwargs):
+        raise indexer.IndexFailed(
+            "delete of stale rows",
+            RuntimeError("timed out after 181.0s"),
+            stats=stats,
+            elapsed=181.0,
+            detail={"statements": 330, "chunk_size": 128},
+        )
+
+    monkeypatch.setattr(indexer, "index_path", failing_index)
+
+    with pytest.raises(indexer.IndexFailed):
+        cli_module.index(Path("."))
+
+    captured = capsys.readouterr()
+    # every number the successful path would have printed
+    assert "scanned=657" in captured.out
+    assert "indexed=165" in captured.out
+    assert "symbols=4102" in captured.out
+    assert "purged=3" in captured.out
+    # plus what failed, on stderr where the sweep script already redirects
+    assert "failed in delete of stale rows" in captured.err
+    assert "statements=330" in captured.err
+
+
+def test_the_failure_still_propagates(monkeypatch):
+    """Printing the summary must not turn a failed index into a successful one:
+    the sweep counts a repo as indexed by exit status alone."""
+
+    def failing_index(path, *, force=False, **kwargs):
+        raise indexer.IndexFailed(
+            "load of content hashes",
+            RuntimeError("boom"),
+            stats=_stats(),
+            elapsed=1.0,
+            detail={},
+        )
+
+    monkeypatch.setattr(indexer, "index_path", failing_index)
+
+    with pytest.raises(indexer.IndexFailed):
+        cli_module.reindex(Path("."))
+
+
+def test_a_failure_that_is_not_a_write_phase_is_left_alone(monkeypatch, capsys):
+    """Only a write failure has partial stats worth printing. Anything else —
+    an unreadable config, a refused branch — has none, and inventing a summary
+    of zeros would read as "the repo is empty" rather than "it never started".
+    """
+
+    def failing_index(path, *, force=False, **kwargs):
+        raise RuntimeError("witan code: refusing to write to the shared view")
+
+    monkeypatch.setattr(indexer, "index_path", failing_index)
+
+    with pytest.raises(RuntimeError, match="refusing to write"):
+        cli_module.index(Path("."))
+
+    assert capsys.readouterr().out == ""

{witan_code-0.13.5 → witan_code-0.14.0}/witan_code/cli.py

@@ -15,6 +15,7 @@ unconditionally.
 
 import asyncio
 import inspect
+import sys
 from pathlib import Path
 from typing import Annotated, Literal
 
@@ -138,15 +139,30 @@ def _render_table(
 @app.command
 def index(path: Path = Path(".")) -> None:
     """Incrementally index PATH (file or directory). Unchanged files are skipped."""
-    stats = indexer.index_path(path, force=False)
-    _print_summary("index", path, stats)
+    _print_summary("index", path, _index(path, force=False))
 
 
 @app.command
 def reindex(path: Path = Path(".")) -> None:
     """Force re-index PATH, ignoring content hashes."""
-    stats = indexer.index_path(path, force=True)
-    _print_summary("reindex", path, stats)
+    _print_summary("reindex", path, _index(path, force=True))
+
+
+def _index(path: Path, *, force: bool) -> indexer.IndexStats:
+    """``index_path``, but a failure still prints what the run got through.
+
+    The parse phase's counts are the same numbers a success prints, and they are
+    what says whether a write failure is a big-repo problem or not — so losing
+    them to the traceback is losing the diagnosis. Printed here rather than
+    swallowed: the exception still propagates, so the exit code and the
+    traceback are unchanged.
+    """
+    try:
+        return indexer.index_path(path, force=force)
+    except indexer.IndexFailed as exc:
+        _print_summary("partial", path, exc.stats)
+        print(f"failed in {exc.phase}: {exc}", file=sys.stderr)
+        raise
 
 
 @app.command

{witan_code-0.13.5 → witan_code-0.14.0}/witan_code/indexer.py

@@ -9,11 +9,13 @@ then imported modules, then any repo-wide match. It is intentionally syntactic
 and will miss dynamic dispatch and produce occasional false links.
 """
 
+import contextlib
 import functools
 import hashlib
 import importlib
 import os
-from collections.abc import Callable
+import time
+from collections.abc import Callable, Iterator
 from dataclasses import dataclass, field
 from pathlib import Path
 
@@ -295,6 +297,62 @@ class IndexStats:
     rather than silent."""
 
 
+class IndexFailed(RuntimeError):
+    """A write phase of :func:`index_path` failed, carrying what it knows.
+
+    ★ THE FAILING RUN IS THE ONE THAT REPORTED LEAST. A successful index prints
+    ``scanned=… indexed=… skipped=… symbols=…``; a failing one printed only the
+    exception, because the exception replaced the summary — so the run you most
+    need data from was the only one that gave you none. That is why the CI
+    indexer's ``ol-infrastructure`` timeout sat unexplained for five days
+    (tk-the-ci-indexer-s-failure-says-only-timed-out-no--50f61c): attributing it
+    needed a traceback line number, a source read to find the constant, row
+    counts from three graphs and a live timing experiment, almost all of which
+    the numbers below would have answered directly.
+
+    ``stats`` is the partial :class:`IndexStats` as of the failure, ``phase``
+    names the write that failed, and ``detail`` carries the sizes that phase was
+    working with. ``elapsed`` is the seconds that phase ran before dying, which
+    is what says whether a timeout was hit or something failed early.
+    """
+
+    def __init__(
+        self,
+        phase: str,
+        cause: BaseException,
+        *,
+        stats: IndexStats,
+        elapsed: float,
+        detail: dict[str, object],
+    ) -> None:
+        self.phase = phase
+        self.stats = stats
+        self.elapsed = elapsed
+        self.detail = detail
+        described = " ".join(f"{k}={v}" for k, v in detail.items())
+        super().__init__(
+            f"{phase} failed after {elapsed:.1f}s"
+            + (f" ({described})" if described else "")
+            + f": {cause}"
+        )
+
+
+@contextlib.contextmanager
+def _write_phase(phase: str, stats: IndexStats, **detail: object) -> Iterator[None]:
+    """Re-raise anything from a write as :class:`IndexFailed` with its sizes."""
+    started = time.monotonic()
+    try:
+        yield
+    except Exception as exc:
+        raise IndexFailed(
+            phase,
+            exc,
+            stats=stats,
+            elapsed=time.monotonic() - started,
+            detail=detail,
+        ) from exc
+
+
 # ── Public entry points ───────────────────────────────────────────
 
 
@@ -432,14 +490,24 @@ def index_path(
                 delete_steps += _delete_file_steps(file_id)
                 stats.purged += 1
 
-    client.change_many(delete_steps, chunk_size=_DELETE_BATCH_SIZE)
+    with _write_phase(
+        "delete of stale rows",
+        stats,
+        statements=len(delete_steps),
+        chunk_size=_DELETE_BATCH_SIZE,
+        reindexed_files=len(reindexed_file_ids),
+        purged_files=stats.purged,
+    ):
+        client.change_many(delete_steps, chunk_size=_DELETE_BATCH_SIZE)
     # Two loads, not one: the second commits the content hashes the first held
     # back, so a load that dies part-way is re-done rather than skipped forever.
     # See `_defer_content_hashes`.
     deduped = _dedupe(records)
     real_hash_rows = _defer_content_hashes(deduped)
-    client.load(deduped, mode="merge")
-    client.load(real_hash_rows, mode="merge")
+    with _write_phase("load of nodes and edges", stats, records=len(deduped)):
+        client.load(deduped, mode="merge")
+    with _write_phase("load of content hashes", stats, records=len(real_hash_rows)):
+        client.load(real_hash_rows, mode="merge")
 
     # Cross-repo bridge — a SEPARATE phase after the per-repo store write, so the
     # two stores' write locks never nest. A full-repo index (target is the repo