@pmaddire/gcie 0.1.4 → 0.1.5

package/AGENT.md

@@ -1,10 +1,14 @@
 # AGENT.md
 
-Agent Operating Instructions for GraphCode Intelligence Engine (GCIE)
+Agent Operating Instructions for GraphCode Intelligence Engine (GCIE)
 
 This file provides persistent architectural context for coding agents working on this repository.
 
-Agents must read this file before performing any development tasks.
+Agents must read this file before performing any development tasks.
+
+Agent trigger:
+Use GCIE for context lookup before reading files or making edits.
+Follow `GCIE_USAGE.md` for the adaptive retrieval workflow and mode-routing rules.
 
 ---
 

package/GCIE_USAGE.md

@@ -1,231 +1,290 @@
-# GCIE Agent Usage (Portable Default)
+# GCIE Agent Usage (Portable, Accuracy-First)
 
-This file is designed to be dropped into any repository and used immediately.
+Trigger line for agent instructions:
+`Use GCIE for context lookup before reading files or making edits. Follow GCIE_USAGE.md.`
 
 ## Goal
 
-Retrieve the smallest useful context while preserving edit safety.
+Retrieve the smallest useful context without sacrificing edit safety.
 
 Priority order:
 1. accuracy (must-have coverage)
 2. full-hit reliability
 3. token efficiency
 
-## Quick Start (Any Repo)
+## Core Rules
 
-1. Identify must-have context categories for the task:
-- implementation file(s)
-- wiring/orchestration file(s)
-- validation surface when risk is non-trivial
-- this may be a test, spec, schema, contract, migration, config, or CLI surface depending on the repo
+1. Do not trade recall for token savings.
+2. Stop retrieval as soon as must-have categories are covered.
+3. Adapt per task family, not per one-off query.
+4. Keep defaults portable; keep repo-specific learning in `.gcie` state.
+
+## Commands (Tool-Synced)
+
+Primary retrieval:
+```powershell
+gcie.cmd context <path> "<query>" --intent <edit|debug|refactor|explore> --budget <auto|int> --mode <basic|adaptive>
+```
+
+Sliced retrieval:
+```powershell
+gcie.cmd context-slices <path> "<query>" --intent <edit|debug|refactor|explore> --profile <low|recall|adaptive>
+```
+
+Adaptive profile state:
+```powershell
+gcie.cmd adaptive-profile .
+gcie.cmd adaptive-profile . --clear
+```
+
+Post-init adaptation pipeline:
+```powershell
+gcie.cmd adapt . --benchmark-size 10 --efficiency-iterations 5 --clear-profile
+```
 
-2. Run one primary retrieval with a file-first, symbol-heavy query:
+One-shot setup + adaptation:
 ```powershell
-gcie.cmd context <path> "<file-first symbol-heavy query>" --intent <edit|debug|refactor|explore> --budget <shape budget>
+gcie.cmd setup . --adapt --adapt-benchmark-size 10 --adapt-efficiency-iterations 5
 ```
 
-3. Check must-have coverage.
+Setup and index:
+```powershell
+gcie.cmd setup .
+gcie.cmd index .
+```
+
+Direct verification:
+```powershell
+rg -n "symbol1|symbol2|symbol3" <likely files or subtree>
+```
+
+## Must-Have Coverage Gate (Required)
+
+Context is sufficient only when all needed categories are present:
+- implementation file(s)
+- wiring/orchestration/caller file(s)
+- validation surface when risk is non-trivial (test/spec/schema/config/contract/CLI surface)
+
+If any must-have file is missing, retrieval is incomplete.
+
+If a must-have file appears only as compact/skeleton context, re-query that file explicitly (pin or targeted query) before editing.
+
+Note: tests/spec files are often excluded by default. Add `--include-tests` only when test context is required.
+
+## Query Construction (Portable, High-Signal)
+
+Preferred pattern:
+`<file-a> <file-b> <function/component> <route/flag> <state/config-key>`
 
-4. If one must-have file is missing, run targeted gap-fill for only that file.
+Rules:
+1. Use file-first, symbol-heavy phrasing.
+2. Include explicit file paths when known.
+3. Include 2-6 distinctive symbols.
+4. Add caller/entry anchor when target is indirect.
+5. Avoid natural-language question phrasing.
 
-5. Stop immediately when must-have coverage is complete.
+Example:
+- Bad: `How does architecture routing decide when to fall back?`
+- Good: `context/context_router.py context/fallback_evaluator.py architecture routing fallback confidence`
 
 ## Retrieval Modes (Adaptive Router)
 
-Use three modes and choose by task family:
+Use three modes and route by observed outcomes:
 
-1. `plain-context-first` (default for most tasks)
-2. `slicer-first` (for hard routed architecture or multi-hop families)
-3. `direct-file-check` (verification and fast gap closure)
+1. `slicer-first`
+2. `plain-context-first`
+3. `direct-file-check`
 
-Plain-context command:
+Slicer-first:
 ```powershell
-gcie.cmd context <path> "<query>" --intent <edit|debug|refactor|explore> --budget <shape budget>
+gcie.cmd context-slices <path> "<query>" --profile low --intent <intent>
 ```
 
-Slicer-first command:
+Plain-context-first:
 ```powershell
-gcie.cmd context-slices <path> "<query>" --intent <edit|debug|refactor|explore>
+gcie.cmd context <path> "<query>" --mode basic --intent <intent> --budget auto
 ```
 
-Direct-file-check command:
+Direct-file-check:
 ```powershell
-rg -n "<symbol1|symbol2|symbol3>" <likely files or subtree>
+rg -n "<symbols>" <files-or-subtree>
 ```
 
-Mode-switch rule:
-- start with `plain-context-first` unless setup calibration proved another mode is better for that family
-- use `slicer-first` only for families where routing/architecture slices repeatedly outperform plain context
-- use `direct-file-check` whenever must-have coverage is uncertain or one file remains missing
-- do not keep retrying the same mode indefinitely; switch after one weak result
+Routing policy:
+1. Start new repos in `slicer-first` bootstrap mode.
+2. If must-have coverage is incomplete after one slicer pass, switch that task to `plain-context-first`.
+3. If a task family misses with slicer 2+ times in calibration, set that family default to `plain-context-first`.
+4. Keep slicer for families where it is both accurate and cheaper.
+5. If two GCIE attempts still miss required files, use `direct-file-check` and mark family `manual-verify-required` until recalibrated.
 
-Portable starter policy:
-- default all families to `plain-context-first`
-- after first 10-20 tasks, promote individual families to `slicer-first` only if benchmarked better
-- keep a family on plain-context if slicer is more expensive with no accuracy gain
+## Scope and Budget Baseline (Portable)
 
-## Architecture Tracking (Portable, In-Repo)
+Scope rule:
+1. Use the smallest path scope that still contains expected files.
+2. Use repo root `.` only for true cross-layer recovery.
+3. If explicit targets cluster in one subtree, subtree scope is usually better than root.
 
-To make slicer mode adapt as the repo changes, keep architecture tracking inside the repo where GCIE runs.
+Profile ladder (concrete, portable):
+1. `context-slices --profile low`
+2. if miss: `context-slices --profile recall`
+3. if miss: `context-slices --profile recall --pin <missing-file>`
+4. if miss: `rg` direct check and targeted file retrieval
 
-Track these files under `.gcie/`:
-- `.gcie/architecture.md`
-- `.gcie/architecture_index.json`
-- `.gcie/context_config.json`
+Plain-context budget baseline:
+- `auto`: simple same-layer or strong single-file lookup
+- `900`: same-family two-file lookup
+- `1100`: backend/config pair or same-layer backend pair
+- `1150`: cross-layer UI/API flow
+- `1300-1400`: explicit multi-hop chain
 
-How to keep it adaptive:
-1. Bootstrap from user docs once (read-only):
-- `ARCHITECTURE.md`, `README.md`, `PROJECT.md`, `docs/architecture.md`, `docs/system_design.md`
-2. Use `.gcie/architecture.md` as GCIE-owned working architecture map.
-3. Refresh `.gcie/architecture.md` and `.gcie/architecture_index.json` when structural changes happen:
-- new subsystem
-- major module split/merge
-- interface/boundary change
-- dependency-direction change
-- active work-area shift
-4. Do not overwrite user-owned docs unless explicitly asked.
+Gap-fill baseline:
+- general implementation/wiring file: `900`
+- small entry/orchestrator file: `500`
 
-Architecture confidence rule:
-- if architecture slice confidence is low or required mappings are stale/missing, fallback to plain `context` automatically
-- record fallback reason in `.gcie/context_config.json` when bypassing slicer mode
+## Adaptive Recovery Order (One Change At A Time)
 
-## Portable Defaults (Task-Shape Based)
+When retrieval is weak, apply in this exact order:
 
-Use these as a starting point in new repos.
+1. Query upgrade: add explicit files, symbols, caller/entry anchor
+2. Scope correction: subtree vs root
+3. One profile/budget escalation
+4. Targeted gap-fill for only missing must-have file(s)
+5. Multi-hop decomposition only if still incomplete
 
-Primary pass budgets:
-- `auto`: simple same-layer or strong single-file lookup
-- `900`: same-family two-file lookup, frontend-local component lookup
-- `1100`: backend/config pair, same-layer backend pair
-- `1150`: cross-layer UI/API flow
-- `1300-1400`: explicit multi-hop chain (3+ linked files)
+Stop condition:
+- If a required file is still missing after two GCIE attempts (with query+scope corrected), stop GCIE retries and use `rg`.
 
-Gap-fill budgets:
-- missing general implementation/wiring file: `900`
-- missing small orchestration or entry file: `500`
+## Architecture Tracking (Portable, In-Repo)
 
-Scope rule:
-- use the smallest path scope that still contains the expected files
-- use repo root (`.`) only for true cross-layer or backend orchestration recovery
-- if explicit targets cluster in one subtree, broad repo-root retrieval is often worse than subtree retrieval
+Track these under `.gcie/`:
+- `.gcie/architecture.md`
+- `.gcie/architecture_index.json`
+- `.gcie/context_config.json`
 
-## Query Construction (Portable)
+Keep adaptive:
+1. Bootstrap from user docs once (`ARCHITECTURE.md`, `README.md`, `PROJECT.md`, `docs/*architecture*`).
+2. Treat `.gcie/architecture.md` as GCIE-owned working map.
+3. Refresh architecture files when boundaries/subsystems/interfaces change.
+4. Do not overwrite user-owned docs unless explicitly asked.
 
-Use this pattern:
+Fallback confidence rule:
+- If architecture confidence is low or mappings are stale/missing, fallback to plain context and record reason in `.gcie/context_config.json`.
 
-`<file-a> <file-b> <function/component> <state-or-arg> <route/flag> <config-key>`
+## Pre-Calibration Readiness Gate (Required)
 
-Guidelines:
-- include explicit file paths when known
-- include 2 to 6 distinctive symbols
-- include a caller or entry anchor when the target is indirect
-- avoid vague summaries and long laundry-list queries
+Run before full adaptation:
 
-## Adaptive Loop (When Retrieval Is Weak)
+1. Index + architecture refresh:
+```powershell
+gcie.cmd index .
+```
 
-Treat retrieval as weak if any are true:
-- missing implementation or wiring category
-- generic entry/support files dominate
-- only tiny snippets from the target file appear, with no useful implementation body
-- expected cross-layer endpoint is missing
+2. Readiness probe:
+- Run 10-20 stratified queries across major families using `context-slices` and `context`.
 
-Adapt in this order, one change at a time:
+3. Readiness adaptation loop:
+- Apply recovery order (query -> scope -> escalation -> targeted gap-fill -> decomposition).
 
-1. Query upgrade:
-- add explicit file paths
-- add missing symbols such as functions, props, routes, flags, or keys
-- add caller or entry anchor
+4. Gate decision:
+- Proceed to calibration only after coverage is reachable with stable behavior.
+- If not reachable, keep safer fallback mode for affected families and continue tracking.
 
-2. Scope correction:
-- noisy root results: move to subtree scope
-- missing cross-layer or backend anchor: use a targeted root query for that file
+## Automatic Post-Trigger Adaptation (Required)
 
-3. Budget bump:
-- raise one rung only, roughly `+100` to `+250`
+After trigger detection in a repo session:
 
-4. Targeted gap-fill:
-- fetch only the missing must-have file(s)
+1. `checkpoint: trigger_detected`
+2. Run `gcie.cmd index .` -> `checkpoint: index_complete`
+3. Run readiness probe -> `checkpoint: readiness_probe_complete`
+4. Run accuracy calibration to 100% must-have hit -> `checkpoint: accuracy_lock_complete`
+5. Run efficiency iterations only under hard accuracy gate -> `checkpoint: efficiency_complete`
+6. Run final stress validation (recommended 50-query) -> `checkpoint: stress_validation_complete`
+7. Write back results (`.planning`, `.gcie/context_config.json`, learned overrides section) -> `checkpoint: write_back_complete`
 
-5. Decompose chain, only if needed:
-- for 4+ hops, split into adjacent 2-3 file hops
+If any checkpoint fails, mark run `incomplete`, record failure artifact in `.planning/`, and continue recovery/fallback flow.
 
-## Safe Efficiency Mode
+## Mandatory Bootstrap Calibration Sequence
 
-Use only after stable coverage is achieved.
+1. Recall calibration stage (required):
+- Tune mode/scope/query/profile until overall and per-family hit rates are 100%.
 
-Rules:
-- do not lower primary budgets for known hard shapes
-- for a single missing file, try `800` before `900` only if the first pass already found same-family context
-- if `800` misses, immediately retry the stable default
-- if any miss persists, revert that task family to stable settings
+2. Recall lock verification (required):
+- Require 2 consecutive 100% lock runs.
 
-Note:
-- `800` is an experimental efficiency step-down, not a portable default truth
-- keep it only if it preserves full must-have coverage in the current repo
+3. Efficiency stage (optional, only after lock):
+- Test controlled reductions one change at a time.
+- Immediately rollback any hit-rate regression.
 
-## Verification Rule
+4. Activation rule (required):
+- Activate only if lock/stress pass.
+- If stress fails, rollback to last known 100%-hit config.
 
-Always verify with a quick local symbol check before editing:
+## Metrics and Decision Rules
 
-```powershell
-rg -n "symbol1|symbol2|symbol3" <likely files>
-```
+Per query, record:
+- must-have hit (true/false)
+- tokens used
+- retrieved files
+- escalations performed
 
-GCIE is a context compressor, not the final truth gate.
+Track overall and by family:
+- hit rate
+- average and median tokens
+- tokens-per-hit (`total_tokens / hit_count`)
 
-If one required file is still missing after retrieval, do direct-file-check first, then run one targeted GCIE call only for that file.
+Selection rule per family:
+1. highest hit rate
+2. if tie: lowest tokens-per-hit
+3. if tie: lowest median tokens
 
-## Portable Stop Rule
+Demotion rules:
+- If slicer miss-rate > 0% during recall calibration, do not keep slicer as default for that family.
+- If both slicer and plain fail, route family to manual-verify until recalibration.
 
-Stop retrieval when all must-have categories are covered:
-- implementation
-- wiring/orchestration
-- validation surface, when risk justifies it
+Promotion rules:
+- Promote only configurations that preserve 100% hit.
+- Efficiency changes must improve tokens without reducing hit rate.
 
-Do not continue increasing budgets after sufficiency is reached.
+## Continuous Adaptation Over Time
 
-## First 5 Tasks Calibration (Minimal)
+Trigger recalibration when any are true:
+1. major repo-change signal (large refactor/churn)
+2. savings decay (rolling savings drops materially vs active baseline)
+3. repeated family misses (2+ in recent window)
 
-For a new repo, track these fields for the first 5 tasks:
-- task shape
-- primary budget
-- gap-fill used (Y/N)
-- must-have full-hit (Y/N)
-- total tokens
+Guardrails:
+1. Use a minimum evidence window (recommended: 20 retrieval events).
+2. Run in quiet/background mode when possible.
+3. Cap adaptation budget per cycle.
+4. Early-stop efficiency loop after 2 non-improving iterations.
+5. Prefer family-scoped recalibration before full recalibration.
 
-If a miss pattern repeats 2+ times in one task family:
-- add one local override for that family only
-- keep all other families on portable defaults
+## Persistence
 
-Update necessity rule:
-- explicit workflow updates are optional, not required for baseline operation
-- if results are stable, keep using portable defaults without changes
-- add or update a local override only when the same miss pattern repeats 2-3 times
+Persist learned defaults in `.gcie/context_config.json` and `.gcie/retrieval_profile.json` with:
+- family
+- default mode/profile
+- last benchmark date
+- hit/token metrics
 
-## Optional Appendix: Repo-Specific Overrides (Example)
+Write repo-local learned routing here:
 
-These are examples from one mixed-layer repo and are not universal defaults.
+## Learned Routing Overrides (Repo-Local, Mutable)
 
-1. `cross_layer_ui_api` override:
-```powershell
-gcie.cmd context frontend "src/App.jsx src/main.jsx <symbols>" --intent edit --budget 900
-gcie.cmd context . "app.py start_convert selected_theme selectedTheme no_ai" --intent edit --budget 900
-```
+No active learned overrides yet.
+Populate after first full adaptation cycle.
 
-2. Stage 3/4 planner-builder pair override (`Plan_slides.py` + `Build_pptx.py`):
-```powershell
-gcie.cmd context . "Plan_slides.py content_slides section_divider figure_slides table_slide" --intent <intent> --budget 900
-gcie.cmd context . "Build_pptx.py build_pptx render_eq_png apply_theme THEME_CHOICES" --intent <intent> --budget 900
-```
+## Agent Instructions Snippet (Copy/Paste)
 
-3. Stage 1/2 with `main.py` override:
-```powershell
-gcie.cmd context . "Analyze_pdf_structure.py Extract_pdf_content.py extract_pages split_into_sections extract_images enrich_with_ai" --intent explore --budget 1100
-gcie.cmd context . "main.py Stage 1 Stage 2 extract_pages enrich_with_ai" --intent explore --budget 500
+```text
+Use GCIE for context lookup before reading files or making edits. Follow GCIE_USAGE.md.
+Prioritize must-have coverage over token savings.
+Start with context-slices --profile low, then adapt using recovery order:
+query -> scope -> profile/budget escalation -> targeted gap-fill -> rg fallback.
 ```
 
-4. Guardrail example:
-- keep the stable workflow for families that regress under split retrieval
-- example: `llm_client.py + Analyze_pdf_structure.py + Extract_pdf_content.py` in one benchmarked repo
+## Notes
 
-If this appendix does not match your repo, ignore it and use only the portable sections above.
+1. This file is intentionally generalized and adaptive for any repo.
+2. Keep repo-specific tuning in learned overrides and `.gcie` state, not in global defaults.
+3. If in doubt, choose the higher-accuracy path first, then optimize tokens after lock.

package/README.md

@@ -58,7 +58,7 @@ Use this when you want a fast drop-in setup for coding agents.
 No heavy upfront tuning is required. The workflow starts portable-first and only adds local overrides after repeated miss patterns.
 
 One-command repo bootstrap:
-- `gcie.cmd setup .`
+- `npx -y @pmaddire/gcie@latest setup .`
 
 This creates `.gcie` architecture tracking files, copies portable agent workflow docs, and runs an initial index pass.
 
@@ -101,7 +101,7 @@ Net: adapted protocol preserved full accuracy while reducing average tokens by ~
 After publishing to npm, users can set up any repo with one command:
 
 ```powershell
-npx gcie@latest
+npx -y @pmaddire/gcie@latest setup .
 ```
 
 This runs `gcie setup .` in the current repo by default.
@@ -110,10 +110,32 @@ If Python deps are missing, GCIE now bootstraps a local package venv and install
 Optional setup flags are passed through:
 
 ```powershell
-npx gcie@latest --no-index
-npx gcie@latest --force
+npx -y @pmaddire/gcie@latest setup . --no-index
+npx -y @pmaddire/gcie@latest setup . --force
 ```
 
+For command-only usage without setup:
+
+```powershell
+npx -y @pmaddire/gcie@latest --help
+```
+
+## Agent Integration
+
+To make your coding agent use GCIE automatically, add this trigger line to your
+agent instructions (system prompt / repo instruction file):
+
+`Use GCIE for context lookup before reading files or making edits. Follow GCIE_USAGE.md.`
+
+Required file:
+- keep `GCIE_USAGE.md` in the target repo root
+
+Recommended setup:
+1. Run one-command setup:
+   - `npx -y @pmaddire/gcie@latest setup .`
+2. Add the trigger line above to your agent instruction file.
+3. Start normal coding tasks; the agent should use GCIE-first retrieval workflow.
+
 ## One-Command GitHub Bootstrap
 
 Run this from the target repo to download GCIE from GitHub and set it up automatically:
@@ -146,7 +168,7 @@ What it does:
 1. In the GCIE repo:
    - `npm link`
 2. In your target repo:
-   - `npm link gcie`
+   - `npm link @pmaddire/gcie`
 3. Verify:
    - `gcie --help`
 
@@ -162,7 +184,7 @@ This repo includes a lightweight npm wrapper so you can run `gcie` like other np
 2. In target repo: `gcie --help`
 
 Local option:
-- `npm install` then `npx gcie --help`
+- `npm install` then `npx @pmaddire/gcie@latest --help`
 
 The wrapper prefers `.venv` in the GCIE repo and falls back to system Python.
 
@@ -216,7 +238,7 @@ Important note:
 - `gcie index <path>`
 - `gcie query <file.py> "<question>"`
 - `gcie debug <file.py> "<question>"`
-- `gcie context <repo|file> "<task>" --budget auto --intent <edit|debug|refactor|explore>`
+- `gcie context <repo|file> "<task>" --budget auto --intent <edit|debug|refactor|explore> --mode basic`
 - `gcie context-slices <repo> "<task>" --intent <edit|debug|refactor|explore> [--profile recall|low] [--stage-a 400] [--stage-b 800] [--max-total 1200] [--pin frontend/src/App.jsx] [--pin-budget 300] [--include-tests]`
 
 ## How To Use It
@@ -367,6 +389,6 @@ npm publish --access public
 Then users can run:
 
 ```powershell
-npx gcie@latest
+npx -y @pmaddire/gcie@latest setup .
 ```
 

package/bench_questions.py

@@ -0,0 +1,69 @@
+import os
+import pathlib
+from parser.ast_parser import parse_python_file
+from graphs.call_graph import build_call_graph
+from graphs.variable_graph import build_variable_graph
+from retrieval.hybrid_retriever import hybrid_retrieve
+from llm_context.snippet_selector import RankedSnippet, estimate_tokens
+from llm_context.context_builder import build_context
+
+ROOT=pathlib.Path('.')
+EXCLUDE={'__pycache__','.venv','venv'}
+py_files=[]
+for path in ROOT.rglob('*.py'):
+    if any(part in EXCLUDE for part in path.parts):
+        continue
+    py_files.append(path)
+
+snippets_by_node={}
+modules=[]
+for path in py_files:
+    try:
+        module=parse_python_file(path)
+    except Exception:
+        continue
+    modules.append(module)
+    text=path.read_text()
+    lines=text.splitlines()
+    for fn in module.functions:
+        start=max(0,fn.start_line-1)
+        end=min(len(lines),fn.end_line)
+        snippet='\n'.join(lines[start:end])
+        node=f"function:{path.as_posix()}::{fn.name}"
+        snippets_by_node[node]=snippet
+
+call_graph=build_call_graph(modules)
+var_graph=build_variable_graph(modules)
+graph=call_graph
+for node,attrs in var_graph.nodes(data=True):
+    if not graph.has_node(node):
+        graph.add_node(node,**attrs)
+for u,v,data in var_graph.edges(data=True):
+    graph.add_edge(u,v,**data)
+
+prompts=[
+    "Why is variable diff exploding?",
+    "How does git history mining handle empty repositories?",
+    "How do CLI index/query/debug commands work?",
+]
+
+def naive_tokens():
+    total=0
+    for path in py_files:
+        total+=estimate_tokens(path.read_text())
+    return total
+
+naive=naive_tokens()
+print('Prompt|GCIE tokens|Naive tokens|Reduction%|Selected snippets|Notes')
+for prompt in prompts:
+    hybrid=hybrid_retrieve(graph,prompt,top_k=10,git_recency_by_node={},coverage_risk_by_node={},max_hops=2)
+    ranked=[]
+    for cand in hybrid:
+        text=snippets_by_node.get(cand.node_id)
+        if not text:
+            continue
+        ranked.append(RankedSnippet(cand.node_id,text,cand.score))
+    context=build_context(prompt,ranked,token_budget=300)
+    reduction=(1-context.total_tokens_estimate/naive)*100 if naive else 0
+    note='good' if ranked else 'empty'
+    print(f"{prompt}|{context.total_tokens_estimate:.1f}|{naive:.1f}|{reduction:.1f}%|{len(context.snippets)}|{note}")