codegraph-brain 0.28.0__tar.gz → 0.30.0__tar.gz

codegraph_brain-0.30.0/.github/test-count-baseline

@@ -0,0 +1 @@
+3252

{codegraph_brain-0.28.0 → codegraph_brain-0.30.0}/.github/workflows/ci.yml

@@ -162,6 +162,20 @@ jobs:
           # this exemption hides it — that is the cost, and it is the reason the
           # rule stays on for src/.
           #
+          # pythonsecurity:S8706 (connection-string injection) is the same
+          # finding at sqlite3.connect: graph_identity.py opens a database named
+          # on its own command line. `dump()` requires an existing .db/.sqlite/
+          # .sqlite3 regular file before connecting, and passes no `uri=True`, so
+          # the argument is a file name and never a connection string. Exempted
+          # for scripts/ on the S2083 reasoning above — the engine does not model
+          # the check — and kept on for src/.
+          #
+          # tests/fixtures/ is excluded from test analysis
+          # (`sonar.test.exclusions`; it sits under `sonar.tests`, so
+          # `sonar.exclusions` would not reach it): it holds input corpora, not
+          # code. Its package.json files exist to be parsed by cgis, and a lock
+          # file next to them (text:S8564) would be a fixture nothing reads.
+          #
           # `sonar.newCode.referenceBranch` overrides the server-side new-code
           # definition for this analysis. The instance default is
           # `previous_version` (confirmed: sonar.leak.period=previous_version,
@@ -218,13 +232,16 @@ jobs:
             -Dsonar.newCode.referenceBranch=main
             -Dsonar.sources=src,scripts,.github
             -Dsonar.tests=tests
+            -Dsonar.test.exclusions=tests/fixtures/**
             -Dsonar.python.version=3.12
             -Dsonar.python.coverage.reportPaths=coverage.xml
-            -Dsonar.issue.ignore.multicriteria=e1,e2,e3,e4
+            -Dsonar.issue.ignore.multicriteria=e1,e2,e3,e4,e5
             -Dsonar.issue.ignore.multicriteria.e1.ruleKey=pythonsecurity:S8707
             -Dsonar.issue.ignore.multicriteria.e1.resourceKey=scripts/**/*.py
             -Dsonar.issue.ignore.multicriteria.e4.ruleKey=pythonsecurity:S2083
             -Dsonar.issue.ignore.multicriteria.e4.resourceKey=scripts/**/*.py
+            -Dsonar.issue.ignore.multicriteria.e5.ruleKey=pythonsecurity:S8706
+            -Dsonar.issue.ignore.multicriteria.e5.resourceKey=scripts/**/*.py
             -Dsonar.issue.ignore.multicriteria.e2.ruleKey=githubactions:S8541
             -Dsonar.issue.ignore.multicriteria.e2.resourceKey=.github/workflows/*.yml
             -Dsonar.issue.ignore.multicriteria.e3.ruleKey=githubactions:S8544

{codegraph_brain-0.28.0 → codegraph_brain-0.30.0}/.gitignore

@@ -15,6 +15,8 @@ downloads/
 eggs/
 .eggs/
 lib/
+# The graph-identity corpus has lib/ directories that are inputs, not build output.
+!tests/fixtures/graph_identity/corpus/**/lib/
 lib64/
 parts/
 sdist/

{codegraph_brain-0.28.0 → codegraph_brain-0.30.0}/.pre-commit-config.yaml

@@ -40,6 +40,7 @@ repos:
           - tree-sitter>=0.25.2
           - tree-sitter-python>=0.25.0
           - tree-sitter-typescript>=0.23.2
+          - tree-sitter-go>=0.25.0
           - pytest>=9.0.3
           - "mcp[cli]>=2"  # must match the pyproject bound
           - duckdb>=1.1.0

codegraph_brain-0.30.0/.release-please-manifest.json

@@ -0,0 +1,3 @@
+{
+  ".": "0.30.0"
+}

{codegraph_brain-0.28.0 → codegraph_brain-0.30.0}/CHANGELOG.md

@@ -1,5 +1,63 @@
 # Changelog
 
+## [0.30.0](https://github.com/zaebee/codegraph-brain/compare/codegraph-brain-v0.29.0...codegraph-brain-v0.30.0) (2026-10-15)
+
+
+### Features
+
+* **languages:** Go field and call-result types resolve receiver calls ([#608](https://github.com/zaebee/codegraph-brain/issues/608)) ([#614](https://github.com/zaebee/codegraph-brain/issues/614)) ([96ec5f3](https://github.com/zaebee/codegraph-brain/commit/96ec5f380d52ba1560c63b8b84af673627bc88d0))
+* **languages:** step 7b — the Go pilot ([#607](https://github.com/zaebee/codegraph-brain/issues/607)) ([4a62703](https://github.com/zaebee/codegraph-brain/commit/4a62703f41a4dfb74d1f96bee132b3f78f84c895))
+* **pipeline:** a graph built by other extraction code is rebuilt in full ([#617](https://github.com/zaebee/codegraph-brain/issues/617)) ([dc33153](https://github.com/zaebee/codegraph-brain/commit/dc331531556cc2dc49c8adfacbec079584953eb8))
+* **python:** FastAPI handlers play the ROUTE_HANDLER role ([#566](https://github.com/zaebee/codegraph-brain/issues/566)) ([#616](https://github.com/zaebee/codegraph-brain/issues/616)) ([9098ccd](https://github.com/zaebee/codegraph-brain/commit/9098ccd87ef08fbb3139243fda426b62e8d1b32b))
+
+
+### Bug Fixes
+
+* **docs:** coverage claims name Go — the MCP instructions are built from the registry ([#615](https://github.com/zaebee/codegraph-brain/issues/615)) ([8ea6362](https://github.com/zaebee/codegraph-brain/commit/8ea636239122b24c4a42b2a65517ebb12dad1b03))
+
+
+### Documentation
+
+* auto-sync architecture graph and MCP reference ([#598](https://github.com/zaebee/codegraph-brain/issues/598)) ([3feb1ac](https://github.com/zaebee/codegraph-brain/commit/3feb1ac40b1896acb0e57109c9bd1f2bf941d7bc))
+* auto-sync architecture graph and MCP reference ([#620](https://github.com/zaebee/codegraph-brain/issues/620)) ([3f36e15](https://github.com/zaebee/codegraph-brain/commit/3f36e15c30dca7280b57bebd70a8793764cd872e))
+* **spec:** language packs and a two-plane topology ([#593](https://github.com/zaebee/codegraph-brain/issues/593)) ([d9e7d85](https://github.com/zaebee/codegraph-brain/commit/d9e7d852b17d4cab4a2a4a0f2519f11f8de9236e))
+
+## [0.29.0](https://github.com/zaebee/codegraph-brain/compare/codegraph-brain-v0.28.0...codegraph-brain-v0.29.0) (2026-10-14)
+
+
+### Features
+
+* **bench:** cgis-narrow arm — the cgis arm with three tools listed ([#586](https://github.com/zaebee/codegraph-brain/issues/586)) ([5a5c836](https://github.com/zaebee/codegraph-brain/commit/5a5c836e2fda1f41e6420e0d1160a56a0859ef94))
+* **mcp:** agent-sized cgis_analyze_impact — no test callers, files as dead ends, node budget ([#563](https://github.com/zaebee/codegraph-brain/issues/563)) ([d173d57](https://github.com/zaebee/codegraph-brain/commit/d173d5791e30983b58e7cbe7fb91cac8c7059154))
+* **mcp:** output_format="text" for traversals; trace_flow gets impact's node budget and test screen ([#581](https://github.com/zaebee/codegraph-brain/issues/581)) ([a1dd2b8](https://github.com/zaebee/codegraph-brain/commit/a1dd2b871d34c22e29f25103b9ee349a004ea28a))
+* **mcp:** server instructions in initialize, CGIS_MCP_TOOLS allowlist ([#565](https://github.com/zaebee/codegraph-brain/issues/565)) ([3c37874](https://github.com/zaebee/codegraph-brain/commit/3c37874f78e4a3d439be153641afe1fc19a32f03))
+* **query:** resolve an FQN given with the directories above the graph ([#571](https://github.com/zaebee/codegraph-brain/issues/571)) ([ea43ba9](https://github.com/zaebee/codegraph-brain/commit/ea43ba9f5451e0ee55575570eb8a5341eb906115))
+* **resolver:** DISPATCHES edges from a base-class method to each override ([#575](https://github.com/zaebee/codegraph-brain/issues/575)) ([827143e](https://github.com/zaebee/codegraph-brain/commit/827143ee2a13866602daaea84f1142106c69d49d))
+* **resolver:** type a parameter annotated with an Annotated[T, Depends(...)] alias as T ([#572](https://github.com/zaebee/codegraph-brain/issues/572)) ([fc33b20](https://github.com/zaebee/codegraph-brain/commit/fc33b202b607aecda9d3b317edc1d16f2ba350c6))
+* **workspaces:** read pnpm-workspace.yaml above the ingest root ([#591](https://github.com/zaebee/codegraph-brain/issues/591)) ([b0d67f8](https://github.com/zaebee/codegraph-brain/commit/b0d67f8124da3c414b852f045f6127ab2eec5c3e))
+
+
+### Bug Fixes
+
+* **bench:** agent A/B runner survives a failed session and continues run numbers ([#578](https://github.com/zaebee/codegraph-brain/issues/578)) ([bc303eb](https://github.com/zaebee/codegraph-brain/commit/bc303eb82e3ef25c01093452c2d68e8c3a53d1d9))
+* **bench:** guard — no false refusal on a grep for an import, no bypass through env/newline/prefix ([#590](https://github.com/zaebee/codegraph-brain/issues/590)) ([73ee4e2](https://github.com/zaebee/codegraph-brain/commit/73ee4e260cf3a65a31630f4cc6a46b618872ce7f))
+* **ingest:** read the tsconfig.json and package.json above a sub-directory ingest root ([#560](https://github.com/zaebee/codegraph-brain/issues/560)) ([9722f8c](https://github.com/zaebee/codegraph-brain/commit/9722f8c18da97cba20400cbd109fd8fea684a5e3))
+* **query:** one effective_prefix rule for every FQN-prefix entry point ([#585](https://github.com/zaebee/codegraph-brain/issues/585)) ([b5f10b3](https://github.com/zaebee/codegraph-brain/commit/b5f10b360ab021ba2dcd8dd9a211b222c989e183))
+* **resolver:** TypeScript imports of declared npm packages are EXTERNAL ([#557](https://github.com/zaebee/codegraph-brain/issues/557)) ([73d380a](https://github.com/zaebee/codegraph-brain/commit/73d380acb915671e9597fc01eaf509ba29e78a8b))
+
+
+### Documentation
+
+* auto-sync architecture graph and MCP reference ([#558](https://github.com/zaebee/codegraph-brain/issues/558)) ([ee275a4](https://github.com/zaebee/codegraph-brain/commit/ee275a412b9af8b2d900e16b84335a343f68a7fa))
+* auto-sync architecture graph and MCP reference ([#564](https://github.com/zaebee/codegraph-brain/issues/564)) ([50637b3](https://github.com/zaebee/codegraph-brain/commit/50637b3b8d01cb4be01cbae3710f21a1cb5f1b02))
+* auto-sync architecture graph and MCP reference ([#573](https://github.com/zaebee/codegraph-brain/issues/573)) ([225499d](https://github.com/zaebee/codegraph-brain/commit/225499dd1a17163160e270b9dbeb73a7ad376b7d))
+* **bench:** agent A/B after [#568](https://github.com/zaebee/codegraph-brain/issues/568) and [#550](https://github.com/zaebee/codegraph-brain/issues/550) — cgis-forced at $0.184 against control $0.298 ([#574](https://github.com/zaebee/codegraph-brain/issues/574)) ([8e0ef37](https://github.com/zaebee/codegraph-brain/commit/8e0ef37b76fffc01e32338c67325535f8cb501f4))
+* **bench:** agent A/B on a task no fix was tuned on (occupied_window) ([#576](https://github.com/zaebee/codegraph-brain/issues/576)) ([a6eb679](https://github.com/zaebee/codegraph-brain/commit/a6eb679b51ae7fd3f52c73eb2da005857f2e6126))
+* **bench:** agent A/B on three deep-chain cgis tasks — impact pays, trace_flow's 32k answer does not ([#579](https://github.com/zaebee/codegraph-brain/issues/579)) ([eb7e951](https://github.com/zaebee/codegraph-brain/commit/eb7e9516a8e952fdbd2266c43accfee1a16109e7))
+* **bench:** agent A/B results for owner-impact-credit-wallet after [#563](https://github.com/zaebee/codegraph-brain/issues/563) and [#565](https://github.com/zaebee/codegraph-brain/issues/565) ([#570](https://github.com/zaebee/codegraph-brain/issues/570)) ([ae84feb](https://github.com/zaebee/codegraph-brain/commit/ae84febcc2621a92d00233e81e994171b6b14cd8))
+* **bench:** flow A/B after [#581](https://github.com/zaebee/codegraph-brain/issues/581) — text cut the answer 4x and the cost by a third; precision unchanged ([#583](https://github.com/zaebee/codegraph-brain/issues/583)) ([e36c850](https://github.com/zaebee/codegraph-brain/commit/e36c85099a85300f46a9b6fa28d375fa68f67227))
+* **bench:** unprompted agents do not call cgis — a short menu changes nothing, one instruction line 2 of 6 ([#587](https://github.com/zaebee/codegraph-brain/issues/587)) ([6534f74](https://github.com/zaebee/codegraph-brain/commit/6534f74f23fcd3d1193bda2201bbf598959cac8e))
+
 ## [0.28.0](https://github.com/zaebee/codegraph-brain/compare/codegraph-brain-v0.27.0...codegraph-brain-v0.28.0) (2026-10-02)
 
 

{codegraph_brain-0.28.0 → codegraph_brain-0.30.0}/CLAUDE.md

@@ -45,21 +45,24 @@ make coverage-html   # same, writes htmlcov/
 The pipeline runs in three phases: **Extract → Resolve → Store**.
 
 ```
-IngestionPipeline (pipeline.py)
-  ├── BaseExtractor subclasses (extractors/)   ← language-specific AST → Nodes+Edges
-  │     └── PythonExtractor                    ← only language implemented so far
+IngestionPipeline (pipeline/__init__.py)
+  ├── cgis.languages (registry)                ← lazy: a pack is imported when its first file is met
+  │     ├── languages/python/                  ← PythonExtractor + PACK
+  │     ├── languages/typescript/              ← TypeScriptExtractor (.ts/.tsx) + PACK
+  │     └── languages/go/                      ← GoExtractor (.go) + go.mod reader + PACK (pilot)
   ├── ResolverEngine (resolver/engine.py)      ← raw_call: targets → resolved FQN edges
-  └── SQLiteStore (storage/sqlite_store.py)    ← persists graph; WAL mode
+  ├── SQLiteStore (storage/sqlite_store.py)    ← persists graph; WAL mode
+  └── post-write passes (pipeline/)            ← refresh_dispatch, SemanticUpliftEngine (pipeline/uplift.py)
         └── QueryEngine (query/engine.py)      ← BFS traversals (impact / flow)
 ```
 
-**Node FQN format**: `module.ClassName.method_name` — fully dot-separated, derived from the file path (e.g., `src/cgis/pipeline.py` → `src.cgis.pipeline.IngestionPipeline.run`). Use `file_path_to_module_fqn(path)` from `extractors/python_extractor.py` to convert paths. `__init__.py` strips the `/__init__` suffix.
+**Node FQN format**: `module.ClassName.method_name` — fully dot-separated, derived from the file path (e.g., `src/cgis/pipeline.py` → `src.cgis.pipeline.IngestionPipeline.run`). Use `cgis.languages.scope_of(path, source_root)` to convert paths (it picks the file's own pack; the Python rule lives in `languages/python/extractor.py`). `__init__.py` strips the `/__init__` suffix.
 
 **Raw call convention**: Extractors emit edges with `target = "raw_call:<name>"`. The `ResolverEngine` then resolves these to actual FQNs or leaves them unresolved (keeps the `raw_call:` prefix in output).
 
 **`ResolverEngine` resolution order** for a call to `name`:
 1. `self.method` → look up `method` on the enclosing class via `_class_methods` index
-2. Unqualified call → `_global_symbols` by name; if ambiguous, prefer same-file candidate via `_file_global_symbols`
+2. Unqualified call → `_global_symbols` by name; if ambiguous, prefer the same-scope candidate via `file_global_symbols` (a file's scope is its path, or the `metadata["scope"]` its FILE node declares — a Go package)
 
 ## Core Models (`src/cgis/core/models.py`)
 
@@ -68,12 +71,30 @@ IngestionPipeline (pipeline.py)
 - `NodeType` — structural (`FILE`, `MODULE`, `CLASS`, `FUNCTION`, `METHOD`…), runtime (`API_ENDPOINT`, `DB_TABLE`…), semantic (`DOMAIN_CONCEPT`).
 - `EdgeType` — structural (`CONTAINS`, `DECLARES`, `IMPORTS`), behavioral (`CALLS`, `REFERENCES`), semantic (`HANDLES`, `PERSISTS`, `AUTHORIZES`).
 
-## Adding a New Language Extractor
-
-1. Subclass `BaseExtractor` in `src/cgis/extractors/`.
-2. Implement `parse(code: str, file_path: str) -> tuple[list[Node], list[Edge]]`.
-3. Emit `CALLS` edges with `target="raw_call:<name>"` — the resolver handles the rest.
-4. Register the extractor in `cli.py`'s `extractors` dict with the file extension key.
+## Adding a Language Pack
+
+1. Create `src/cgis/languages/<name>/` with an extractor subclassing `cgis.core.extractor.BaseExtractor`
+   (`parse(code, file_path) -> tuple[list[Node], list[Edge]]`; emit `CALLS` edges with
+   `target="raw_call:<name>"`, set `Node.language = "<name>"`).
+2. In its `__init__.py`, define `PACK`, subclassing `cgis.core.extractor.LanguagePack` (mark methods `@override`)
+   (`name`, `extensions`, `code_fence`, `is_test`, `scope_of`, `extractor`, `project_reader`,
+   `index_facts`, `classify`, `resolution`). `resolution()` returns `ResolutionRules`
+   (`cgis.core.resolution`): type normalizers, steps at a position (`after:import_map`), a
+   fallback with its confidence, `reconcile_import` — all reading the index as `ResolutionView`.
+   Metadata other files resolve against goes in the manifest's `cross_file_metadata`. `classify` answers for references from the pack's own files and
+   runs the whole chain; a source with no pack gets the generic tail in `SymbolIndex.classify_fqn`.
+3. Add one `Manifest` to `MANIFESTS` in `src/cgis/languages/__init__.py`, with a `load` function that
+   imports the pack statically inside it (lazy, yet visible to the review-fingerprint walk). Nothing
+   else names the pack: CLI, MCP server, guardian and scripts all go through the registry, and the
+   `surface-packs` forbidden-edge gate fails if a surface imports a pack directly.
+4. A pack never imports another pack or `cgis.storage` (`cross-pack`, `pack-isolation` gates).
+5. A pack's `__init__` imports no grammar: import the extractor inside `extractor()`. Config reading
+   goes in a `ProjectReader` (`cgis.core.project`) whose facts are data keyed by `ingest_state`
+   key — the collector (`cgis.import_names`), not the pack, writes them. A key other than the four
+   built-ins (`_FACT_SHAPES`) travels as `ImportNames.extra`: recorded, compared, and merged into the
+   `state` the pack's `index_facts` reads. Its value must read back equal from JSON (no tuples).
+6. A FILE node's id comes from `file_node_id`, its definitions' namespace from `scope_of`; they differ
+   only where several files share a scope (Go), which the FILE node declares as `metadata["scope"]`.
 
 ## Type Discipline
 

{codegraph_brain-0.28.0 → codegraph_brain-0.30.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: codegraph-brain
-Version: 0.28.0
+Version: 0.30.0
 Summary: Semantic code graph for AI agents — deterministic FQN resolution, impact analysis and architectural drift gates, exposed over MCP.
 Project-URL: Homepage, https://github.com/zaebee/codegraph-brain
 Project-URL: Repository, https://github.com/zaebee/codegraph-brain
@@ -29,6 +29,7 @@ Requires-Dist: pydantic>=2.13.4
 Requires-Dist: pyyaml>=6.0.2
 Requires-Dist: rich>=15.0.0
 Requires-Dist: structlog>=25.5.0
+Requires-Dist: tree-sitter-go>=0.25.0
 Requires-Dist: tree-sitter-python>=0.25.0
 Requires-Dist: tree-sitter-typescript>=0.23.2
 Requires-Dist: tree-sitter>=0.25.2
@@ -60,7 +61,7 @@ Description-Content-Type: text/markdown
 
 CGIS parses a repository with tree-sitter into a graph of fully qualified symbols and the calls, imports and containment between them, stores it in SQLite, and serves it to AI agents over MCP. An agent that would otherwise grep and read whole files asks the graph instead.
 
-- **Languages:** Python · TypeScript / TSX
+- **Languages:** Python · TypeScript / TSX · Go (pilot)
 - **Runs:** locally — no account, no telemetry; the graph never leaves your disk ([the one opt-in exception](https://github.com/zaebee/codegraph-brain/blob/main/PRIVACY.md))
 
 ```console

{codegraph_brain-0.28.0 → codegraph_brain-0.30.0}/README.md

@@ -10,7 +10,7 @@
 
 CGIS parses a repository with tree-sitter into a graph of fully qualified symbols and the calls, imports and containment between them, stores it in SQLite, and serves it to AI agents over MCP. An agent that would otherwise grep and read whole files asks the graph instead.
 
-- **Languages:** Python · TypeScript / TSX
+- **Languages:** Python · TypeScript / TSX · Go (pilot)
 - **Runs:** locally — no account, no telemetry; the graph never leaves your disk ([the one opt-in exception](https://github.com/zaebee/codegraph-brain/blob/main/PRIVACY.md))
 
 ```console

{codegraph_brain-0.28.0 → codegraph_brain-0.30.0}/benchmarks/agent_ab/README.md

@@ -40,9 +40,17 @@ resolver gets wrong. Each task's `notes` say how it was checked.
 |---|---|---|---|---|
 | `control` | none (`--strict-mcp-config`, empty config) | no | all deleted | no |
 | `cgis` | this checkout's `cgis-mcp` | yes (the plugin minus its `.mcp.json`) | `graph.db` built before the clock starts | no |
+| `cgis-narrow` | as `cgis`, listing only `analyze_impact`, `find_symbol`, `trace_flow` (`CGIS_MCP_TOOLS`) | as `cgis` | as `cgis` | no |
 | `cgis-instructed` | as `cgis` | as `cgis` | as `cgis` | one line: query cgis first |
 | `cgis-forced` | as `cgis` | as `cgis` | as `cgis` | as `cgis-instructed`, and the guard refuses Read/Grep/Glob/Bash until one cgis call |
 
+`cgis-narrow` asks whether a short tool menu gets an unprompted agent to call the
+graph (#542). The server instructions and the plugin skill still name all fifteen
+tools, so it measures a short menu with a longer guide beside it. Each results line
+records `cgis_tools`, the number of `mcp__cgis__*` tools the session was offered,
+so an `env` the client ignored would show as 15. With no `--arm` a run now covers
+five arms.
+
 `cgis-instructed` exists because the first pilot's `cgis` arm made no cgis call in
 12 of 12 sessions: with the server connected and the skill loaded, Sonnet still
 went straight to Grep. It stands in for #542's MCP server instructions. It barely moved the agent (2 cgis
@@ -75,8 +83,19 @@ uv run python scripts/agent_ab.py report
 
 `--model` defaults to `claude-sonnet-5-5`; `--max-budget-usd` (default 2.00)
 caps each session. Every non-dry run is a paid session, so this never runs in
-CI. Transcripts are written to `transcripts/` (git-ignored); `results.jsonl`
-gets one line per (task, arm, run) and is meant to be committed with the write-up.
+CI. Transcripts are written to `transcripts/<task>/<arm>-<sha7>-<run>.jsonl`, with
+the first 7 characters of the cgis commit (git-ignored); `results.jsonl` gets one
+line per (task, arm, cgis commit, run) and is meant to be committed with the
+write-up.
+
+Run numbers continue from the highest one already in `--results` for the same
+task, arm and cgis commit, so a second invocation that finishes an interrupted
+batch adds runs instead of reusing them. A session that fails is written as a
+line with `failed` set to the reason, and the batch goes on: an ingest or
+worktree that raises gives a line with no metrics, and a `claude -p` that exits
+non-zero keeps its scored metrics and cost but is marked failed too. `run` exits
+1 at the end if any session failed; `report` counts failed sessions in their own
+column and leaves them out of the medians. A dry run writes nothing.
 
 ## Tasks
 

codegraph_brain-0.30.0/benchmarks/agent_ab/RESULTS.md

@@ -0,0 +1,177 @@
+# Agent A/B results
+
+One section per run, newest first. The raw lines are in `results.jsonl`; the
+transcripts they point at are git-ignored and stay on the machine that ran them.
+
+## 2026-10-14: does one instruction line get the agent to call cgis?
+
+cgis at 5a5c836, `cgis-instructed` — the `cgis` arm plus one appended system-prompt
+line, "query cgis first, then read source only to confirm", the nearest stand-in for
+a line in a project's CLAUDE.md. Same two impact tasks, 3 runs each, about $0.8.
+
+| task | cgis calls per run | cost per run |
+|---|---|---|
+| `cgis-impact-drift-same-name` | 0, 0, **1** | 0.184, 0.148, **0.075** |
+| `cgis-impact-transitive-tsconfig` | **1**, 0, 0 | **0.099**, 0.154, 0.153 |
+
+- **The line moves the agent a little: 2 of 6 sessions called the graph**, against 0
+  of 18 without it. Both made one call, and both were the cheapest runs of the
+  six, at the level of `cgis-forced` (0.085 / 0.091). The other four ignored the
+  line and cost more than plain `cgis` (median 0.150 against 0.126–0.138).
+- So an instruction is a coin toss weighted against the graph; the hook that refuses
+  Read/Grep/Bash until one cgis call (`cgis-forced`) is what delivers the saving
+  reliably. A project wanting the gain needs that hook, not a sentence.
+- One run is marked `contaminated` with no cgis CLI call in it: on a task in this
+  repository every path contains `cgis/`, and the guard's predicate took the agent's
+  own `grep`/`sed` over `src/cgis/...` for one. A harness false positive, kept out of
+  the medians by the report as usual.
+
+## 2026-10-14: does a short tool menu get an unprompted agent to call cgis? (#542 item 3)
+
+cgis at dbf3de9 (the `cgis-narrow` arm of #586), Sonnet 5.5, 3 runs per arm, 0 failed,
+0 contaminated, about $1.6. One probe session confirmed the narrowing reached the
+client: the `init` event listed exactly `mcp__cgis__cgis_analyze_impact`,
+`mcp__cgis__cgis_find_symbol`, `mcp__cgis__cgis_trace_flow`.
+
+| task | arm | cgis calls | recall | precision | median $ |
+|---|---|---|---|---|---|
+| `cgis-impact-drift-same-name` | cgis (15 tools) | 0, 0, 0 | 1.00 | 1.00 | 0.126 |
+| | cgis-narrow (3 tools) | 0, 0, 0 | 1.00 | 1.00 | 0.125 |
+| `cgis-impact-transitive-tsconfig` | cgis | 0, 0, 0 | 1.00 | 1.00 | 0.138 |
+| | cgis-narrow | 0, 0, 0 | 1.00 | 1.00 | 0.154 |
+
+- **Refuted.** Three tools instead of fifteen changed nothing: no unprompted
+  session called the graph, here or in any of the six `cgis` sessions before it —
+  0 of 18 today, across four tasks. The server instructions (#542 item 1) did not
+  move it either. Item 3's smaller default set has no evidence behind it.
+- These are the two tasks where `cgis-forced` was about 30% cheaper than control
+  (#579), so the graph would have paid; the agent never looked.
+- A likely reason sits in the client: Claude Code defers MCP tools, listing names
+  only, so a first call needs a ToolSearch to load the schema — every forced
+  session made one. An agent holding Grep and Bash, with a task grep finishes in
+  7–13 calls, does not take that step unasked.
+- Confound, as the README notes: the instructions and the plugin skill still name
+  all fifteen tools beside the three offered.
+
+## 2026-10-14: `cgis-flow-ingest-to-sqlite` after #581 (text format, trace budget)
+
+cgis at a1dd2b8, Sonnet 5.5, 3 runs per arm, 0 failed, 0 contaminated, about $0.6.
+
+| arm | recall | precision | median $ | trace answer |
+|---|---|---|---|---|
+| control | 1.00 | 0.94 | **0.101** | — |
+| cgis-forced, before (#579) | 1.00 | 0.52 | 0.167 | 32,637 chars of json |
+| cgis-forced, now | 1.00 | 0.58 | 0.112 | 6.3–8.9k chars of text |
+
+- **Size: fixed.** All three sessions asked for `output_format="text"` on their
+  own (the server instructions now say to), and the answer shrank about 4×.
+  Cost fell by a third, to 11% above control.
+- **Precision: not fixed.** Given every node the root reaches, the agent still
+  lists most of them (17–21 outside the key: `_compute_hash`,
+  `_is_noop_incremental`, `ResolverEngine` helpers). Control reads the code and
+  names the spine. The format no longer costs the agent; deciding which
+  branches are the main path is the part the graph does not do.
+- Two sessions first called `cgis_find_symbol(query="ingest", fqn_prefix="src.cgis")`
+  and got nothing: #568 taught `resolve_fqn` to drop a directory above the
+  graph, but `find_symbol`'s `fqn_prefix` filter was not given the same rule.
+
+## 2026-10-14: three cgis tasks with a deep chain
+
+cgis at bc303eb, Sonnet 5.5, 3 runs per arm, 0 failed, 0 contaminated, about $2.3.
+None of these tasks was used to find the recent fixes.
+
+| task | arm | recall | precision | median $ | median tool calls | cgis calls |
+|---|---|---|---|---|---|---|
+| `cgis-impact-drift-same-name` | control | 1.00 | 1.00 | 0.124 | 9 | — |
+| | cgis-forced | 1.00 | 1.00 | **0.085** | 3 | 1, 1, 1 |
+| `cgis-impact-transitive-tsconfig` | control | 1.00 | 1.00 | 0.126 | 7 | — |
+| | cgis-forced | 1.00 | 1.00 | **0.091** | 4 | 1, 2, 1 |
+| `cgis-flow-ingest-to-sqlite` | control | 1.00 | 0.86 | **0.112** | 5 | — |
+| | cgis-forced | 1.00 | 0.52 | 0.167 | 4 | 2, 2, 2 |
+
+- **Impact across several hops pays:** about 30% cheaper on both impact tasks,
+  with one cgis call doing what took control 7–9 greps and reads. Together with
+  `credit_wallet`, that makes three impact tasks out of three where the graph
+  wins once the question needs more than one hop.
+- **Flow does not, and the cause is the answer's size.** Every forced session
+  called `cgis_trace_flow(cgis.cli.ingest, depth=3, json)` and got **32,637
+  characters**, the bloat #551 fixed for impact only: trace has no node budget
+  and no test filter. Given everything downstream, the agent cannot tell the
+  main path from side branches and names them all (precision 0.52, 20 extra
+  symbols such as every `ResolverEngine._resolved_*_edge`).
+- `occupied_window` (above) and these together say where cgis earns its place:
+  multi-hop impact. A one-hop caller list and a "main path" flow question are
+  not where it helps today.
+
+## 2026-10-14: `owner-impact-occupied-window-same-name`, a task no fix was tuned on
+
+cgis at 827143e (after #551, #568, #550, #545), Sonnet 5.5, 0 contaminated. The
+three fixes before this were all found in `credit_wallet` transcripts and measured
+there; this task checks whether the gain carries to a question they never saw.
+
+| arm | runs | recall | precision | median $ | median tool calls | cgis calls per run |
+|---|---|---|---|---|---|---|
+| control | 3 | 1.00 | 1.00 | **0.130** | 2 | — |
+| cgis | 3 | 1.00 | 1.00 | 0.145 | 4 | 0, 0, 0 |
+| cgis-forced | 2 | 1.00 | 1.00 | 0.135 | 3.5 | 2, 1 |
+
+- **It does not carry.** On "list the direct callers", with a same-named
+  function in another module as the trap, control answered with one grep and
+  one look at the imports, and was the cheapest arm. The graph names the four
+  callers exactly (`impact depth=1`, 2.5k characters), but there is nothing
+  for it to save on a one-hop question that grep plus the imports already
+  answer.
+- The `credit_wallet` gain came from a deep transitive chain through DI. That
+  is where the graph pays, so the question for #542's default tool set is
+  which kinds of question agents ask, not whether cgis is cheaper in general.
+- `cgis` made no graph call again: 0 in 6 sessions across the two tasks.
+- `cgis-forced` has 2 runs, not 3. The host ran out of memory, and earlyoom
+  sent SIGTERM to the third session's `cgis ingest`, which aborted the runner
+  twice. Not a cgis failure. The runner stops at the first failed ingest
+  instead of recording it and going on.
+
+## 2026-10-14: `owner-impact-credit-wallet`, after #568 and #550
+
+cgis at fc33b20, Sonnet 5.5, 3 runs per arm, 0 contaminated, about $1.5 in total.
+`cgis` was not re-run: it made no cgis call on this task in the run below.
+
+| arm | recall | precision | median $ | median tool calls | cgis calls per run |
+|---|---|---|---|---|---|
+| control | 1.00 | 1.00 | 0.298 | 7 | — |
+| cgis-forced | 1.00 | 1.00 | **0.184** | 5 | 2, 5, 1 |
+
+- `cgis-forced` went from $0.235 to $0.184 median and is now about 38% cheaper
+  than control. Control was unchanged ($0.30 to $0.298).
+- **#568.** Two of the three sessions opened with
+  `app.domains.finance.service.credit_wallet` again, and this time it resolved:
+  the first answer was the full chain (6,746 characters), with no not-found
+  first. The third tried the bare `credit_wallet`, which is still ambiguous.
+- **#550.** That first answer now includes both routes, `get_top_up` and
+  `stripe_webhook`. One session finished on a single cgis call. The greps that
+  remain are the agent confirming that `FinanceService.top_up` and the worker
+  tasks have no further callers, not looking for missing ones.
+
+## 2026-10-13: `owner-impact-credit-wallet`, after #563 and #565
+
+cgis at 3c37874, Sonnet 5.5, 3 runs per arm, 0 contaminated, about $2.5 in total.
+
+| arm | recall | precision | median $ | median tool calls | cgis calls per run |
+|---|---|---|---|---|---|
+| control | 1.00 | 1.00 | 0.30 | 7 | — |
+| cgis | 1.00 | 1.00 | 0.30 | 6 | 0, 0, 0 |
+| cgis-forced | 1.00 | 1.00 | 0.235 | 8 | 3, 11, 3 |
+
+- **#551.** The call that returned 51,624 characters in the first pilot,
+  `cgis_analyze_impact(domains.finance.service.credit_wallet, depth=8, json)`,
+  returned 6,136 here. `cgis-forced` went from more expensive than control
+  ($0.207 vs $0.195) to cheaper. Control ranged from $0.198 to $0.327 across
+  three runs, so this shows the direction, not the size.
+- **#542.** The `cgis` arm now gets server instructions in `initialize`, and its
+  server reported `connected` every time. It still made no cgis call: the task
+  is solvable by grep in about seven calls, and the agent took that route.
+- **#568.** Every forced session's first call was
+  `app.domains.finance.service.credit_wallet`, built from the file path, which
+  is not found because the graph is ingested from `ownima-backend/app`. That
+  cost one or two calls per session before the agent found the real FQN.
+- Every arm recovered the two routes the graph misses at this sha
+  (`get_top_up`, `stripe_webhook`) by grepping for them.