witan-code 0.22.0__tar.gz → 0.23.0__tar.gz

{witan_code-0.22.0 → witan_code-0.23.0}/CHANGELOG.md

@@ -8,6 +8,57 @@ a MINOR bump may include breaking changes).
 
 <!-- scriv-insert-here -->
 
+## [0.23.0] - 2026-09-28
+
+### Added
+
+- **Python package providers.** A `pyproject.toml` whose directory holds a
+  `mitol/<pkg>/__init__.py` namespace child now provides `mitol.<pkg>`, so
+  `mitol.*` imports join to ol-django. Previously only `@mitodl/*` npm
+  packages had a provider extractor and no Python import could form an edge.
+  An existing graph gains these providers on ol-django's next full-repo
+  `witan-code index` (the provider files are not source files, so the
+  unchanged-file skip does not apply to them); `witan-code reindex` on it
+  works too.
+
+- **Repo-level provider files that stop producing bindings lose their old
+  rows.** A `pyproject.toml`, `package.json`, Pulumi config or OpenAPI spec
+  that emitted nothing on this run was never purged, so on a run that could
+  not purge by the collected file list (e.g. indexing outside a repository
+  root) its previous bindings stayed. Every such file scanned is now purged
+  before its fresh bindings are written.
+
+### Changed
+
+- Allow cyclopts 5 (`cyclopts>=4,<6`). Under cyclopts 5, `--help` shows a
+  value placeholder after each option (e.g. `--target STR`), and a usage error
+  such as an unknown option exits with code 2 instead of 1.
+
+### Fixed
+
+- **`witan code` listings emit `rows: []` on an empty result under
+  `--output-format`.** `symbols`, `stitch`, `stitch --unresolved` and `repos`
+  each returned early with a Rich sentence on stdout before the format was
+  consulted, so an empty result was not JSON/TOML/YAML. `_render_table` now
+  takes the empty message and prints it in `txt` mode only. `branches` never
+  consulted the format at all and now dumps `{repo, views, error}` rows, with
+  `--prune`'s progress lines on stderr under a structured format; the `repos`
+  unreadable-store warning and `doctor`'s summary and error lines move to
+  stderr there too.
+- **Table titles and cells are escaped for Rich.** A symbol, path or title
+  containing `[...]` was read as markup and lost from the rendered table.
+
+- **Endpoint bindings form cross-repo edges again.** Every endpoint consumer
+  scored 0.0 because the extractor only matches relative paths and the
+  `relative_url` heuristic subtracted 0.5 from the 0.5 baseline, so
+  `code_repo_dependencies` and the other heuristic-tier views filtered every
+  one out at their 0.5 cutoff. The `relative_url` penalty, its unreachable
+  `explicit_hostname` boost, and the `generated_file` penalty are removed. A
+  generated client for a repo's own API is still suppressed by
+  `self_provided_key`. Stored confidences are computed at index time, so
+  existing graphs need `witan-code reindex` (not `index`, which skips
+  unchanged files) on the consumer repos before these edges appear.
+
 ## [0.22.0] - 2026-09-25
 
 ### Added

{witan_code-0.22.0 → witan_code-0.23.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: witan-code
-Version: 0.22.0
+Version: 0.23.0
 Summary: witan-code — tree-sitter code graph + cross-repo bridge (mounts under `witan code`)
 Project-URL: Homepage, https://github.com/mitodl/agent-kit/tree/main/mcp/servers/witan-code
 Project-URL: Repository, https://github.com/mitodl/agent-kit
@@ -11,7 +11,7 @@ Keywords: agent,code-graph,coding-agent,mcp,static-analysis,tree-sitter
 Classifier: Programming Language :: Python :: 3
 Requires-Python: >=3.11
 Requires-Dist: agent-config-kit<1,>=0.10
-Requires-Dist: cyclopts<5,>=4
+Requires-Dist: cyclopts<6,>=4
 Requires-Dist: fastmcp<5,>=4
 Requires-Dist: joserfc<2,>=1.0
 Requires-Dist: pydantic<3,>=2
@@ -136,7 +136,7 @@ edges):
 |---|---|---|---|
 | `env_var` | Pulumi `<app>:env_vars` keys in `Pulumi.*.yaml` | `get_string("NAME",…)` / `os.environ` / `process.env.X` / `env("NAME")` | the `UPPER_SNAKE` name |
 | `endpoint` | drf-spectacular OpenAPI spec (`paths.<path>.<method>`) | `/api/…` path literals in TS (generated client / fetch) | normalized path (params → `{}`) |
-| `package` | a repo whose `package.json` name is `@mitodl/*` | `from mitol.*` / `include("mitol.*")` / `@mitodl/*` imports | the package name |
+| `package` | a repo whose `package.json` name is `@mitodl/*`, or a `pyproject.toml` next to a `mitol/<pkg>/__init__.py` namespace child (provides `mitol.<pkg>`) | `from mitol.*` / `include("mitol.*")` / `@mitodl/*` imports | the package name |
 | `service` | `applications/<svc>/__main__.py` (repo / image / service name) | the deployed repo itself (key_norm = its canonical URI) | `repo:<uri>` / `image:…` / `name:…` |
 
 Generic env names (`DEBUG`, `PORT`, `SECRET_KEY`, …) are flagged `generic` and
@@ -146,7 +146,7 @@ every repo.
 The bridge is written in a **separate phase after the per-repo store write**, so
 the two stores' advisory write locks never nest (no deadlock) and a bridge
 failure never corrupts a per-repo store. A full-repo index runs the repo-level
-provider extractors (OpenAPI / Pulumi / service / `package.json`) and clears
+provider extractors (OpenAPI / Pulumi / service / `package.json` / `pyproject.toml`) and clears
 bindings for files deleted from disk; all purging is per-file, so unchanged
 (skipped) files — and, for a narrow target like the reindex hook, sibling
 files — keep their bindings.

{witan_code-0.22.0 → witan_code-0.23.0}/README.md

@@ -103,7 +103,7 @@ edges):
 |---|---|---|---|
 | `env_var` | Pulumi `<app>:env_vars` keys in `Pulumi.*.yaml` | `get_string("NAME",…)` / `os.environ` / `process.env.X` / `env("NAME")` | the `UPPER_SNAKE` name |
 | `endpoint` | drf-spectacular OpenAPI spec (`paths.<path>.<method>`) | `/api/…` path literals in TS (generated client / fetch) | normalized path (params → `{}`) |
-| `package` | a repo whose `package.json` name is `@mitodl/*` | `from mitol.*` / `include("mitol.*")` / `@mitodl/*` imports | the package name |
+| `package` | a repo whose `package.json` name is `@mitodl/*`, or a `pyproject.toml` next to a `mitol/<pkg>/__init__.py` namespace child (provides `mitol.<pkg>`) | `from mitol.*` / `include("mitol.*")` / `@mitodl/*` imports | the package name |
 | `service` | `applications/<svc>/__main__.py` (repo / image / service name) | the deployed repo itself (key_norm = its canonical URI) | `repo:<uri>` / `image:…` / `name:…` |
 
 Generic env names (`DEBUG`, `PORT`, `SECRET_KEY`, …) are flagged `generic` and
@@ -113,7 +113,7 @@ every repo.
 The bridge is written in a **separate phase after the per-repo store write**, so
 the two stores' advisory write locks never nest (no deadlock) and a bridge
 failure never corrupts a per-repo store. A full-repo index runs the repo-level
-provider extractors (OpenAPI / Pulumi / service / `package.json`) and clears
+provider extractors (OpenAPI / Pulumi / service / `package.json` / `pyproject.toml`) and clears
 bindings for files deleted from disk; all purging is per-file, so unchanged
 (skipped) files — and, for a narrow target like the reindex hook, sibling
 files — keep their bindings.

{witan_code-0.22.0 → witan_code-0.23.0}/conftest.py

@@ -23,4 +23,4 @@ sys.path.insert(0, str(Path(__file__).resolve().parents[3]))
 
 # The import is the point: `testsupport.hermetic` redirects the environment at
 # module scope. The hook re-export is what lets this conftest report a leak.
-from testsupport.hermetic import pytest_sessionfinish  # noqa: E402,F401
+from testsupport.hermetic import pytest_sessionfinish  # noqa: F401

{witan_code-0.22.0 → witan_code-0.23.0}/docs/EDGE_PRECISION_TIERS.md

@@ -35,12 +35,14 @@ up twice at two different trust levels.
 
 ```python
 TypedEdge(
-    precision,           # "precise" | "heuristic" | "fuzzy"
-    consumer_repo, provider_repo,
-    kind, key_norm,
-    canonical_symbol,     # consumer's symbol string; precise tier only, else None
-    confidence,           # 1.0 for precise; the heuristic tier's score otherwise
-    evidence,             # tuple[dict] of {repo, file, line} — see below
+    precision,  # "precise" | "heuristic" | "fuzzy"
+    consumer_repo,
+    provider_repo,
+    kind,
+    key_norm,
+    canonical_symbol,  # consumer's symbol string; precise tier only, else None
+    confidence,  # 1.0 for precise; the heuristic tier's score otherwise
+    evidence,  # tuple[dict] of {repo, file, line} — see below
 )
 ```
 

{witan_code-0.22.0 → witan_code-0.23.0}/docs/STAGE2_STITCHING.md

@@ -45,10 +45,15 @@ way a written edge could.
 
 ```python
 PreciseEdge(
-    consumer_repo, consumer_symbol,
-    provider_repo, provider_symbol,
-    kind, scheme,
-    match_count, preferred, ambiguous_version,
+    consumer_repo,
+    consumer_symbol,
+    provider_repo,
+    provider_symbol,
+    kind,
+    scheme,
+    match_count,
+    preferred,
+    ambiguous_version,
 )
 ```
 

{witan_code-0.22.0 → witan_code-0.23.0}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "witan-code"
-version = "0.22.0"
+version = "0.23.0"
 description = "witan-code — tree-sitter code graph + cross-repo bridge (mounts under `witan code`)"
 readme = "README.md"
 license = "BSD-3-Clause"
@@ -163,7 +163,7 @@ dependencies = [
     # and silently drops the warning.
     "witan-core[cli,remote,observability,sentry]>=0.40,<1",
     "fastmcp>=4,<5",
-    "cyclopts>=4,<5",
+    "cyclopts>=4,<6",
     # [targets.<name>] override models (config.py) — also transitively pulled
     # in by fastmcp, declared explicitly since config.py imports it directly.
     "pydantic>=2,<3",
@@ -244,7 +244,7 @@ testpaths = ["tests"]
 packages = ["witan_code"]
 
 [tool.bumpversion]
-current_version = "0.22.0"
+current_version = "0.23.0"
 allow_dirty = true
 
 [[tool.bumpversion.files]]

{witan_code-0.22.0 → witan_code-0.23.0}/tests/test_branches.py

@@ -6,9 +6,8 @@ import subprocess
 
 import pytest
 
-from witan_core.omnigraph import OmnigraphConflict
-
 from witan_code import repo as repo_module
+from witan_core.omnigraph import OmnigraphConflict
 
 from .conftest import SAMPLE, requires_stack
 

{witan_code-0.22.0 → witan_code-0.23.0}/tests/test_bridge.py

@@ -14,128 +14,104 @@ from witan_code.bridge_extractors import (
     _compute_file_confidence,
     adjust_confidence,
     extract_file_bindings,
+    extract_repo_bindings,
 )
 
 from .conftest import requires_stack
 
-
 # ── Confidence scoring unit tests ─────────────────────────────────
 # All tests operate on _compute_file_confidence directly; no omnigraph needed.
 
 
 def test_load_test_path_penalizes_score():
     """load_test_path penalty: file under a load_testing/ directory."""
-    score = _compute_file_confidence(
-        "load_testing/backend/client/v0/api.ts", "", "/api/v0/courses/"
-    )
+    score = _compute_file_confidence("load_testing/backend/client/v0/api.ts", "")
     assert score < 0.5, f"load_test_path should penalize; got {score}"
 
 
 def test_k6_path_segment_penalizes_score():
     """load_test_path penalty applies to /k6/ path segments."""
-    score = _compute_file_confidence("perf/k6/scenarios.ts", "", "/api/v0/courses/")
+    score = _compute_file_confidence("perf/k6/scenarios.ts", "")
     assert score < 0.5, f"k6 path segment should penalize; got {score}"
 
 
 def test_load_test_framework_import_penalizes():
     """load_test_framework penalty: file imports k6."""
     content = "import http from 'k6/http';\nimport { sleep } from 'k6';\n"
-    score = _compute_file_confidence("client.ts", content, "/api/v0/courses/")
+    score = _compute_file_confidence("client.ts", content)
     assert score < 0.5, f"k6 framework import should penalize; got {score}"
 
 
 def test_load_test_framework_locust_penalizes():
     """load_test_framework penalty applies to locust imports."""
     content = "from 'locust' import HttpUser\n"
-    score = _compute_file_confidence("test_load.ts", content, "/api/v0/things/")
+    score = _compute_file_confidence("test_load.ts", content)
     assert score < 0.5, f"locust import should penalize; got {score}"
 
 
 def test_load_test_framework_artillery_penalizes():
     """load_test_framework penalty applies to artillery imports."""
     content = "import stuff from 'artillery';\n"
-    score = _compute_file_confidence("perf.ts", content, "/api/v0/things/")
+    score = _compute_file_confidence("perf.ts", content)
     assert score < 0.5, f"artillery import should penalize; got {score}"
 
 
-def test_relative_url_penalizes_score():
-    """relative_url penalty: URL has no scheme."""
-    score = _compute_file_confidence("src/client.ts", "", "/api/v0/profiles/")
-    assert score < 0.5, f"relative URL should penalize; got {score}"
+def test_plain_client_clears_the_edge_cutoff():
+    """A consumer with no signal either way sits at the cutoff, so it forms an edge.
 
-
-def test_generated_file_header_heavily_penalizes():
-    """generated_file penalty: file with DO NOT EDIT header."""
-    content = "// DO NOT EDIT — generated by openapi-generator\nexport const x = 1;"
-    score = _compute_file_confidence("src/generated.ts", content, "/api/v0/courses/")
-    assert score < 0.1, f"generated file header should heavily penalize; got {score}"
+    Every path `_TS_ENDPOINT` extracts is relative; penalizing that put every
+    endpoint consumer below `build_graph`'s 0.5 cutoff.
+    """
+    assert _compute_file_confidence("src/client.ts", "") == 0.5
 
 
-def test_generated_file_at_generated_penalizes():
-    """generated_file penalty: file with @generated annotation."""
-    content = "/* @generated */\nexport const x = 1;"
-    score = _compute_file_confidence("src/gen.ts", content, "/api/v0/courses/")
-    assert score < 0.1, f"@generated file should heavily penalize; got {score}"
+def test_generated_client_is_not_penalized():
+    """A generated OpenAPI client is exact consumer evidence; self_provided_key
+    handles one generated for the repo's own API."""
+    content = "// DO NOT EDIT — generated by openapi-generator\nexport const x = 1;"
+    assert _compute_file_confidence("src/generated.ts", content) == 0.5
 
 
 def test_same_origin_helper_penalizes():
     """same_origin_helper penalty: relative import to a csrf/fetch wrapper."""
     content = "import { fetchJSONWithCSRF } from '../utils/csrf';\n"
-    score = _compute_file_confidence("src/actions.ts", content, "/api/v0/profiles/")
+    score = _compute_file_confidence("src/actions.ts", content)
     assert score < 0.5, f"same-origin helper import should penalize; got {score}"
 
 
 def test_same_origin_relative_api_client_penalizes():
     """same_origin_helper penalty: relative import to an api-client file."""
     content = "import { apiFetch } from './api-client';\n"
-    score = _compute_file_confidence("src/actions.ts", content, "/api/v0/courses/")
+    score = _compute_file_confidence("src/actions.ts", content)
     assert score < 0.5, f"relative api-client import should penalize; got {score}"
 
 
-def test_explicit_hostname_boosts_score():
-    """explicit_hostname boost: URL has scheme + host (absolute URL)."""
-    score = _compute_file_confidence(
-        "src/client.ts", "", "https://api.example.com/api/v0/courses/"
-    )
-    assert score > 0.5, f"explicit hostname should boost; got {score}"
-
-
 def test_client_directory_boosts_score():
     """client_directory boost: file path contains /clients/."""
-    score = _compute_file_confidence("src/clients/courses.ts", "", "/api/v0/courses/")
-    assert score > 0.0, "client_directory should boost"
-    # relative URL still penalizes but client_directory partially offsets
-    baseline = _compute_file_confidence("src/other/courses.ts", "", "/api/v0/courses/")
+    score = _compute_file_confidence("src/clients/courses.ts", "")
+    baseline = _compute_file_confidence("src/other/courses.ts", "")
     assert score > baseline, (
         f"client dir path ({score}) should score higher than non-client ({baseline})"
     )
 
 
-def test_client_and_hostname_both_boost():
-    """client_directory + explicit_hostname compound boost > 0.7."""
-    score = _compute_file_confidence(
-        "src/clients/courses.ts",
-        "",
-        "https://api.example.com/api/v0/courses/",
-    )
-    assert score > 0.7, f"client dir + hostname should push score > 0.7; got {score}"
-
-
 def test_score_clamped_to_zero():
     """Multiple penalties cannot push score below 0.0."""
-    content = "// DO NOT EDIT — generated\nimport http from 'k6/http';\n"
-    score = _compute_file_confidence("load_testing/k6/api.ts", content, "/api/v0/x/")
+    content = "import { f } from '../csrf';\nimport http from 'k6/http';\n"
+    score = _compute_file_confidence("load_testing/k6/api.ts", content)
     assert score == 0.0, f"overlapping penalties should clamp at 0.0; got {score}"
 
 
 def test_score_clamped_to_one():
     """Score cannot exceed 1.0 even with all boosts stacking."""
-    score = _compute_file_confidence(
-        "src/clients/integrations/service.ts",
-        "",
-        "https://api.example.com/api/v0/courses/",
+    b = _ep_consumer(confidence=0.9)
+    adjust_confidence(
+        b,
+        consumer_repo="https://github.com/test/repo-a",
+        provider_keys=frozenset(),
+        has_known_provider_package=True,
     )
-    assert score <= 1.0, f"score must not exceed 1.0; got {score}"
+    assert b.confidence == 1.0, f"score must clamp at 1.0; got {b.confidence}"
 
 
 # ── Store-level adjust_confidence tests ───────────────────────────
@@ -312,6 +288,89 @@ def test_csrf_client_still_extracts_package_and_env():
         assert b.confidence == 1.0
 
 
+OPENAPI_SPEC = """\
+openapi: 3.0.3
+paths:
+  /api/v1/courses/{id}/:
+    get: {}
+"""
+
+
+def _rows(bindings, repo):
+    return [
+        {
+            "kind": b.kind,
+            "key_norm": b.key_norm,
+            "role": b.role,
+            "repo": repo,
+            "confidence": b.confidence,
+        }
+        for b in bindings
+    ]
+
+
+def test_extracted_endpoint_and_package_bindings_form_edges(tmp_path):
+    """What the extractors actually emit must survive build_graph's cutoff.
+
+    Regression: every endpoint consumer scored 0.0 (relative_url penalty on
+    paths that are always relative), and no Python package provider was ever
+    emitted, so neither kind produced a single edge on the deployed graph.
+    """
+    from witan_code import visualize
+
+    consumer = "https://github.com/test/frontend"
+    provider = "https://github.com/test/backend"
+    lib = "https://github.com/test/ol-django"
+
+    backend = tmp_path / "backend"
+    (backend / "openapi" / "specs").mkdir(parents=True)
+    (backend / "openapi" / "specs" / "v1.yaml").write_text(OPENAPI_SPEC)
+    ol_django = tmp_path / "ol-django"
+    pkg = ol_django / "src" / "common" / "mitol" / "common"
+    pkg.mkdir(parents=True)
+    (pkg / "__init__.py").write_text("")
+    (ol_django / "src" / "common" / "pyproject.toml").write_text(
+        '[project]\nname = "mitol-django-common"\n'
+    )
+
+    rows = (
+        _rows(
+            extract_file_bindings(NORMAL_TS_CLIENT, "typescript", "src/client.ts"),
+            consumer,
+        )
+        + _rows(
+            extract_file_bindings(
+                "from mitol.common.utils import now_in_utc\n", "python", "app.py"
+            ),
+            consumer,
+        )
+        + _rows(extract_repo_bindings(backend, provider)[0], provider)
+        + _rows(extract_repo_bindings(ol_django, lib)[0], lib)
+    )
+    graph = visualize.build_graph(rows)
+
+    assert dict(graph.edges[(consumer, provider)].kinds) == {"endpoint": 1}
+    assert dict(graph.edges[(consumer, lib)].kinds) == {"package": 1}
+
+
+def test_python_package_provider_only_for_namespace_children(tmp_path):
+    """An app repo's own top-level packages are not published providers."""
+    app = tmp_path / "app"
+    (app / "courses").mkdir(parents=True)
+    (app / "courses" / "__init__.py").write_text("")
+    (app / "pyproject.toml").write_text('[project]\nname = "app"\n')
+    assert extract_repo_bindings(app, "https://github.com/test/app") == (
+        [],
+        {"pyproject.toml"},
+    )
+
+    # A regular (non-namespace) `mitol` package is the app's own, not shared.
+    (app / "mitol" / "thing").mkdir(parents=True)
+    (app / "mitol" / "__init__.py").write_text("")
+    (app / "mitol" / "thing" / "__init__.py").write_text("")
+    assert extract_repo_bindings(app, "https://github.com/test/app")[0] == []
+
+
 # Repo A (mit-learn-like): a Django settings consumer + a NextJS endpoint/env
 # consumer.
 A_SETTINGS = """\
@@ -472,10 +531,9 @@ def test_known_provider_package_boost_sourced_from_stage1_symbol_table(
     re-derivation from raw InterfaceBinding rows — verify the confidence
     boost actually lands end-to-end through a real index + bridge write.
 
-    D_CLIENT_TS's endpoint consumer has only the relative_url penalty
-    (-0.5, no other signal fires): base 0.5 - 0.5 = 0.0. The co-located
-    package import of the repo-c-provided package should then boost it by
-    +0.3 to 0.3.
+    No file-level signal fires on D_CLIENT_TS's endpoint consumer, so it
+    starts at the 0.5 baseline. The co-located package import of the
+    repo-c-provided package should then boost it by +0.3 to 0.8.
     """
     from witan_code import config as cfg_mod
     from witan_code import indexer
@@ -513,7 +571,53 @@ def test_known_provider_package_boost_sourced_from_stage1_symbol_table(
         if r["repo"] == "https://github.com/test/repo-d" and r["kind"] == "endpoint"
     ]
     assert len(rows) == 1
-    assert rows[0]["confidence"] == pytest.approx(0.3)
+    assert rows[0]["confidence"] == pytest.approx(0.8)
+
+
+@requires_stack
+def test_tier_b_source_that_stops_emitting_loses_its_rows(tmp_path, monkeypatch):
+    """A pyproject.toml whose mitol/<pkg>/ was deleted must not keep providing it.
+
+    The toml is unchanged and still on disk. On a run that cannot purge by the
+    collected file list (here: no repository root), nothing put it in the
+    purge set until Tier B started reporting the sources it scanned.
+    """
+    import shutil
+
+    from witan_code import config as cfg_mod
+    from witan_code import indexer
+    from witan_code import server as srv
+    from witan_code.graph import OmnigraphClient
+
+    monkeypatch.setenv("WITAN_CODE_DIR", str(tmp_path / "code"))
+    monkeypatch.setenv("WITAN_REPO", "https://github.com/test/ol-django")
+    cfg = cfg_mod.load()
+    monkeypatch.setattr(srv, "cfg", cfg)
+    srv._clients.clear()
+
+    repo = tmp_path / "ol-django"
+    pkg = repo / "mitol" / "common"
+    pkg.mkdir(parents=True)
+    (pkg / "__init__.py").write_text("")
+    (repo / "pyproject.toml").write_text('[project]\nname = "mitol-django-common"\n')
+
+    client = OmnigraphClient(
+        str(cfg_mod.bridge_store_path(cfg.code_dir)), cfg.queries_dir
+    )
+
+    def providers():
+        return [
+            r["key_norm"]
+            for r in client.read("bridge.gq", "all_bindings", {})
+            if r["kind"] == "package" and r["role"] == "provider"
+        ]
+
+    _index(srv, indexer, repo)
+    assert providers() == ["mitol.common"]
+
+    shutil.rmtree(pkg)
+    _index(srv, indexer, repo)
+    assert providers() == []
 
 
 @requires_stack