@muthuishere/ctx-optimize 0.2.0 → 0.2.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +285 -0
- package/package.json +6 -6
package/README.md
ADDED
|
@@ -0,0 +1,285 @@
|
|
|
1
|
+
# ctx-optimize
|
|
2
|
+
|
|
3
|
+
[](https://github.com/muthuishere/ctx-optimize/actions/workflows/ci.yml)
|
|
4
|
+
[](https://www.npmjs.com/package/@muthuishere/ctx-optimize)
|
|
5
|
+
[](https://muthuishere.github.io/ctx-optimize-site/proof/agent/)
|
|
6
|
+
[](https://www.npmjs.com/package/@muthuishere/ctx-optimize)
|
|
7
|
+
[](LICENSE)
|
|
8
|
+
|
|
9
|
+
**Gather a codebase — and its world — into one local knowledge store an AI agent answers from. Deterministic. No LLM API. No DB. Gather once, refresh cheaply, never go everywhere every time.**
|
|
10
|
+
|
|
11
|
+
Your coding agent burns its context window on grep-and-read: to answer one
|
|
12
|
+
question it greps, opens files, chases callers, re-reads. ctx-optimize turns a
|
|
13
|
+
repo — plus, via adapters, database schemas, messaging topics, log shapes,
|
|
14
|
+
documents — into a queryable graph stored as plain files in a central
|
|
15
|
+
per-module store, and your agent (Claude Code, Codex, Devin — any skill-capable
|
|
16
|
+
harness) answers *from the store* in a single call. The binary never touches a
|
|
17
|
+
model, a database, or the network: it's deterministic, and the only
|
|
18
|
+
intelligence in the system is the agent you already run.
|
|
19
|
+
|
|
20
|
+
> **Status: v0.1.x — published.** On npm (`@muthuishere/ctx-optimize`) with
|
|
21
|
+
> prebuilt binaries for macOS / Linux / Windows; CI green; benchmarks
|
|
22
|
+
> reproducible (see [Proof](#proof--reproducible-not-our-word)). Working
|
|
23
|
+
> today: code extraction for **12 embedded languages** (Go, Python, JS,
|
|
24
|
+
> TS/TSX, Java, C, C++, C#, Rust, Zig, SQL — tree-sitter compiled to WASM,
|
|
25
|
+
> zero setup) plus **drop-in grammar packs** for any other language
|
|
26
|
+
> (kotlin/swift/dart ship in `grammars/`), markdown docs, the universal
|
|
27
|
+
> adapter door, `query`/`path`/`explain`/`affected`/`hubs`, **symbol cards**
|
|
28
|
+
> (`card X`: signature + doc + callers/callees, no file read), the
|
|
29
|
+
> **deterministic wiki** (regenerated on every add), the save-result/reflect
|
|
30
|
+
> learning loop, merge/export (json/dot/graphml/csv/obsidian), the live
|
|
31
|
+
> dashboard, remote init/push/pull, and **multi-module monorepo support**
|
|
32
|
+
> (`scan` / `init --scan` / parallel fan-out `add` / navigator + federated
|
|
33
|
+
> queries — see below). Exact call edges (x/tools + LSP) are
|
|
34
|
+
> next — see `openspec/` for the plan.
|
|
35
|
+
|
|
36
|
+
**Site, demos, benchmarks:** https://muthuishere.github.io/ctx-optimize-site/
|
|
37
|
+
— landing page, unedited demos, and the full proof write-up. Everything below
|
|
38
|
+
is reproducible; see [Proof](#proof--reproducible-not-our-word).
|
|
39
|
+
|
|
40
|
+
## Install
|
|
41
|
+
|
|
42
|
+
npm (recommended — thin JS launcher resolves a prebuilt platform binary via
|
|
43
|
+
`optionalDependencies`; no postinstall script, no download):
|
|
44
|
+
|
|
45
|
+
```sh
|
|
46
|
+
npm install -g @muthuishere/ctx-optimize
|
|
47
|
+
```
|
|
48
|
+
|
|
49
|
+
Go:
|
|
50
|
+
|
|
51
|
+
```sh
|
|
52
|
+
go install github.com/muthuishere/ctx-optimize/cmd/ctx-optimize@latest
|
|
53
|
+
```
|
|
54
|
+
|
|
55
|
+
Then install the agent skill (writes to `~/.claude/skills`, and
|
|
56
|
+
`~/.agents/skills` when codex is present):
|
|
57
|
+
|
|
58
|
+
```sh
|
|
59
|
+
ctx-optimize install --skills
|
|
60
|
+
```
|
|
61
|
+
|
|
62
|
+
## Usage
|
|
63
|
+
|
|
64
|
+
```sh
|
|
65
|
+
# first time in a repo: scaffold .ctxoptimize/ (config + adapters dir) + the store
|
|
66
|
+
ctx-optimize init
|
|
67
|
+
|
|
68
|
+
# gather a repo into the central store (~/ctxoptimize/<repo-name>/)
|
|
69
|
+
ctx-optimize add .
|
|
70
|
+
|
|
71
|
+
# ask the store — complete, citable hits under a token budget
|
|
72
|
+
ctx-optimize query "where is the refund flow" --json
|
|
73
|
+
|
|
74
|
+
# feed ANY system through the universal adapter door (strictly validated)
|
|
75
|
+
./my-postgres-adapter | ctx-optimize add --json -
|
|
76
|
+
|
|
77
|
+
# combine module stores into one view; dump for other tools
|
|
78
|
+
ctx-optimize merge api worker billing --into everything
|
|
79
|
+
ctx-optimize export --format dot --out graph.dot
|
|
80
|
+
|
|
81
|
+
# see it: local dashboard (embedded single file, zero external requests)
|
|
82
|
+
ctx-optimize serve # → http://127.0.0.1:4747 — graph, search, details
|
|
83
|
+
|
|
84
|
+
# share the store: sync-only remotes (S3-compatible or any folder)
|
|
85
|
+
ctx-optimize remote init s3://team-bucket/ctx/myrepo # writes .ctxoptimize/config.json — commit it
|
|
86
|
+
ctx-optimize remote push # incremental — only changed artifacts move
|
|
87
|
+
ctx-optimize remote pull # a teammate who cloned the repo: this is ALL they run
|
|
88
|
+
|
|
89
|
+
ctx-optimize status --json
|
|
90
|
+
```
|
|
91
|
+
|
|
92
|
+
- The store is **plain files** (ndjson/json/md) — diffable, portable, at
|
|
93
|
+
`~/ctxoptimize/<repo-name>/`. The only thing in your repo is the
|
|
94
|
+
committable `.ctxoptimize/` directory.
|
|
95
|
+
- **Remotes are for sync only.** Queries always run on the local folder.
|
|
96
|
+
`push`/`pull` take no URL — the remote is whatever the config says.
|
|
97
|
+
|
|
98
|
+
## Multi-module — monorepos get one graph per module, plus a navigator
|
|
99
|
+
|
|
100
|
+
One giant graph for a 300-module monorepo helps nobody: people work in one
|
|
101
|
+
module at a time, and an agent that loads the whole repo's graph pays for
|
|
102
|
+
299 modules it isn't asking about. ctx-optimize builds **one store per
|
|
103
|
+
module** and a small **navigator** that routes questions instead:
|
|
104
|
+
|
|
105
|
+
```sh
|
|
106
|
+
# find every project in the tree — read-only, prints the exact config it would write
|
|
107
|
+
ctx-optimize scan # markers: go.mod/go.work, package.json, gradle,
|
|
108
|
+
# maven, Cargo.toml, pyproject… (--depth N, default 5)
|
|
109
|
+
|
|
110
|
+
# write ALL found modules into the committed config — generated once, then the
|
|
111
|
+
# list is yours: edit, add, prune (.ctxoptimize/config.json modules[])
|
|
112
|
+
ctx-optimize init --scan --yes
|
|
113
|
+
|
|
114
|
+
# gather: one worker per module, in parallel; stores mirror the repo tree
|
|
115
|
+
ctx-optimize add . # → ~/ctxoptimize/<repo>/<module-path>/, each with
|
|
116
|
+
# its own graph + wiki [--jobs N]
|
|
117
|
+
```
|
|
118
|
+
|
|
119
|
+
Measured on `apache/beam`: **310 modules discovered at depth 8, all gathered
|
|
120
|
+
in 14.5s at ~9× CPU, zero failures** — including maven modules nested inside
|
|
121
|
+
other modules' resource trees.
|
|
122
|
+
|
|
123
|
+
The root store holds a **navigator**, not a merged giant graph:
|
|
124
|
+
`modules.json` + `navigator.md` — every module's path, node/edge counts, top
|
|
125
|
+
hub symbols, and README one-liner — plus a unified wiki front page linking
|
|
126
|
+
into each module's own wiki. Query scope then follows your cwd:
|
|
127
|
+
|
|
128
|
+
```sh
|
|
129
|
+
cd sdks/java/transform-service
|
|
130
|
+
ctx-optimize query "expansion service" # answers from THIS module's graph, labeled;
|
|
131
|
+
# zero hits auto-escalate repo-wide (--root forces)
|
|
132
|
+
cd - # back at the repo root:
|
|
133
|
+
ctx-optimize query "kafka read" # navigator ranks modules, federates across the
|
|
134
|
+
# best matches [--modules all|a,b]
|
|
135
|
+
ctx-optimize card SomeSymbol # not in your module? answered from the owning
|
|
136
|
+
# module, labeled "[not in X — found in Y]"
|
|
137
|
+
```
|
|
138
|
+
|
|
139
|
+
`merge <mod>... --into <name>` stays opt-in for when you actually want one
|
|
140
|
+
combined graph. (graphify's monorepo story is manual per-directory builds —
|
|
141
|
+
no discovery, no parallel gather, no navigator.)
|
|
142
|
+
|
|
143
|
+
## Proof — reproducible, not our word
|
|
144
|
+
|
|
145
|
+
Two kinds of evidence, both runnable.
|
|
146
|
+
|
|
147
|
+
**Speed vs graphify** (raw data in [`benchmarks/`](benchmarks/)): a 12k-file
|
|
148
|
+
corpus gathered in **0.67s vs 8.88s**, queries **~4× faster**, a smaller
|
|
149
|
+
store. Methodology on the site.
|
|
150
|
+
|
|
151
|
+
**What an agent actually saves.** A headless harness lets the *same* model
|
|
152
|
+
answer a set of questions **three ways** over OpenRouter — plain shell,
|
|
153
|
+
ctx-optimize, and graphify — and reports the provider's own token/cost
|
|
154
|
+
accounting (`usage.include=true`), not our estimate. Last public CI run on
|
|
155
|
+
`gorilla/mux` (a small, well-named repo — plain grep's *best* case, i.e. the
|
|
156
|
+
hardest terrain for a graph to win on):
|
|
157
|
+
|
|
158
|
+
| comparison | result |
|
|
159
|
+
|---|---|
|
|
160
|
+
| ctx-optimize **vs plain shell** | **−31% cost · −64% tool calls · −36% tokens** |
|
|
161
|
+
| ctx-optimize **vs graphify** | **~half the tokens & tool calls** |
|
|
162
|
+
| graphify **vs plain shell** | **+22% tokens** — its `query` returns a raw node dump that costs *more* than grep |
|
|
163
|
+
|
|
164
|
+
ctx-optimize answers most questions in a single `query`/`card` call; both arms
|
|
165
|
+
answered correctly with `file:line` citations (a cheaper wrong answer is a
|
|
166
|
+
loss, not a saving).
|
|
167
|
+
|
|
168
|
+
**Run it yourself — no source needed**, it uses the published CLI:
|
|
169
|
+
|
|
170
|
+
```sh
|
|
171
|
+
npm i -g @muthuishere/ctx-optimize # the store CLI
|
|
172
|
+
pipx install graphifyy # the competitor (arm c; optional)
|
|
173
|
+
export OPENROUTER_API_KEY=sk-or-... # read from env only, never logged
|
|
174
|
+
bash proof/agent/run-bench.sh # defaults: gorilla/mux, openai/gpt-4o-mini
|
|
175
|
+
```
|
|
176
|
+
|
|
177
|
+
Or fork and click **Run workflow** — [`.github/workflows/benchmark.yml`](.github/workflows/benchmark.yml)
|
|
178
|
+
runs it headless on a clean runner and publishes the table to the job summary.
|
|
179
|
+
Harness + full write-up: https://muthuishere.github.io/ctx-optimize-site/proof/agent/
|
|
180
|
+
|
|
181
|
+
## .ctxoptimize/ — config that travels with the repo
|
|
182
|
+
|
|
183
|
+
```
|
|
184
|
+
.ctxoptimize/
|
|
185
|
+
config.json name + remote
|
|
186
|
+
adapters/ drop scripts here — every .js/.py/.sh runs on `add`
|
|
187
|
+
```
|
|
188
|
+
|
|
189
|
+
`config.json`:
|
|
190
|
+
|
|
191
|
+
```json
|
|
192
|
+
{
|
|
193
|
+
"name": "my-module",
|
|
194
|
+
"remote": {
|
|
195
|
+
"type": "s3",
|
|
196
|
+
"url": "s3://team-bucket/ctx/my-module",
|
|
197
|
+
"credentials": {
|
|
198
|
+
"access_key_id": "${TEAM_R2_KEY_ID}",
|
|
199
|
+
"secret_access_key": "${TEAM_R2_SECRET}",
|
|
200
|
+
"region": "auto",
|
|
201
|
+
"endpoint": "${R2_ENDPOINT}"
|
|
202
|
+
}
|
|
203
|
+
}
|
|
204
|
+
}
|
|
205
|
+
```
|
|
206
|
+
|
|
207
|
+
Commit the directory — it is safe by construction:
|
|
208
|
+
|
|
209
|
+
- `name` picks the store folder under `~/ctxoptimize/` (default: repo basename).
|
|
210
|
+
- `remote` is a plain string URL or the full object above. `${VAR}` anywhere
|
|
211
|
+
in the url/credentials resolves from the environment **at sync time** — the
|
|
212
|
+
file holds variable names, never secret values; resolved values are never
|
|
213
|
+
written or printed. Omitted credentials fall back to the standard `AWS_*`
|
|
214
|
+
env vars (endpoint override covers R2/Hetzner/MinIO).
|
|
215
|
+
- **Adapters are files**: dropping `kafka.js` into `.ctxoptimize/adapters/`
|
|
216
|
+
is the whole registration (`.js`/`.mjs` → node, `.py` → python3, `.sh` →
|
|
217
|
+
sh; other extensions inert — `init` seeds an `example.js.sample` template).
|
|
218
|
+
Each script prints batch JSON to stdout; `ctx-optimize add` runs the
|
|
219
|
+
built-in extractors **and** every adapter through the fail-closed door. One
|
|
220
|
+
command refreshes the whole world; a fresh clone needs zero setup to `pull`.
|
|
221
|
+
|
|
222
|
+
## Grammar packs — add any language without recompiling
|
|
223
|
+
|
|
224
|
+
A language is just a grammar + a node-type mapping. The 12 embedded ones
|
|
225
|
+
cover the mainstream; anything else is a **pack**: `<name>.wasm` +
|
|
226
|
+
`<name>.json` dropped into `~/ctxoptimize/grammars/` (machine-wide) or
|
|
227
|
+
`.ctxoptimize/grammars/` (travels with the repo). Next `add` picks it up;
|
|
228
|
+
pack extensions override embedded ones. kotlin, swift and dart ship as packs
|
|
229
|
+
in `grammars/` — copy the pair in to enable.
|
|
230
|
+
|
|
231
|
+
Build your own from ANY tree-sitter grammar with one command — no toolchain
|
|
232
|
+
to install (zig auto-downloads once, sha256-verified; grammar fetched as a
|
|
233
|
+
tarball, no git):
|
|
234
|
+
|
|
235
|
+
```sh
|
|
236
|
+
ctx-optimize languages add kotlin # known names resolve to the right repo/branch/exts
|
|
237
|
+
ctx-optimize languages add https://github.com/tree-sitter-grammars/tree-sitter-lua
|
|
238
|
+
# → ~/ctxoptimize/grammars/<name>.wasm + <name>.json (mapping auto-suggested
|
|
239
|
+
# from the grammar's node-types.json — review it, then `add` just works)
|
|
240
|
+
ctx-optimize languages list # embedded + packs + addable names
|
|
241
|
+
ctx-optimize languages remove <name>
|
|
242
|
+
```
|
|
243
|
+
|
|
244
|
+
## Adapters — the open door
|
|
245
|
+
|
|
246
|
+
Everything external is an adapter emitting one JSON schema into
|
|
247
|
+
`ctx-optimize add --json -`: nodes (`id`, `label`, `kind`, `file_type`,
|
|
248
|
+
`source`, `location`) and edges (`source`, `target`, `relation`,
|
|
249
|
+
`confidence` ∈ `EXTRACTED|INFERRED|AMBIGUOUS`). The door validates strictly and
|
|
250
|
+
tags provenance per producer. Your agent can write a new adapter on demand —
|
|
251
|
+
point it at any system with the schema and it gathers it. Make it permanent by
|
|
252
|
+
dropping the script into `.ctxoptimize/adapters/` — every future `add` runs it.
|
|
253
|
+
|
|
254
|
+
## Design
|
|
255
|
+
|
|
256
|
+
Evidence-first: every product decision traces to a measured spike
|
|
257
|
+
(`openspec/changes/2026-07-11-graphify-gaps/spikes.md`) — including honest
|
|
258
|
+
benchmarks against a real agent baseline (not corpus-stuffing strawmen), the
|
|
259
|
+
terrain law (graph value is inverse to a codebase's greppability), and the
|
|
260
|
+
symbol-card finding (agents' reads are pointer-chases a complete answer
|
|
261
|
+
eliminates). Extensibility is a verified differentiator, not a slogan: a
|
|
262
|
+
source audit of graphify (2026-07-11) found its languages, data-source lanes
|
|
263
|
+
and exporters are all fork-required static registries (only its remote hooks
|
|
264
|
+
are user-pluggable); here languages are drop-in packs, adapters are dropped
|
|
265
|
+
scripts, and the batch door takes any producer. Vision: `docs/VISION.md`.
|
|
266
|
+
Standing critique: `docs/CRITIQUE.md`.
|
|
267
|
+
|
|
268
|
+
## Lineage
|
|
269
|
+
|
|
270
|
+
With all due respect to graphify — a project we learned a great deal from —
|
|
271
|
+
there is a direct line between it and this tool: graphify's central graph
|
|
272
|
+
store and its pluggable remote push/pull hooks (the one part of graphify an
|
|
273
|
+
end user can extend without forking) were contributed upstream by this
|
|
274
|
+
project's author (graphify #1751 / #1752; git-verifiable). ctx-optimize is
|
|
275
|
+
that same idea carried through the whole product: the store, the languages,
|
|
276
|
+
the adapters, and the sync are all open seams by design — nothing here
|
|
277
|
+
requires a fork to extend.
|
|
278
|
+
|
|
279
|
+
## License
|
|
280
|
+
|
|
281
|
+
MIT © 2026 Muthukumaran Navaneethakrishnan
|
|
282
|
+
|
|
283
|
+
---
|
|
284
|
+
|
|
285
|
+
Made by [muthuishere](https://github.com/muthuishere).
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "@muthuishere/ctx-optimize",
|
|
3
|
-
"version": "0.2.
|
|
3
|
+
"version": "0.2.2",
|
|
4
4
|
"description": "Gather a codebase (and its world) into one local knowledge store an AI agent answers from — deterministic, no LLM API, no DB.",
|
|
5
5
|
"bin": {
|
|
6
6
|
"ctx-optimize": "bin/ctx-optimize.js"
|
|
@@ -15,10 +15,10 @@
|
|
|
15
15
|
"url": "git+https://github.com/muthuishere/ctx-optimize.git"
|
|
16
16
|
},
|
|
17
17
|
"optionalDependencies": {
|
|
18
|
-
"@muthuishere/ctx-optimize-darwin-arm64": "0.2.
|
|
19
|
-
"@muthuishere/ctx-optimize-darwin-x64": "0.2.
|
|
20
|
-
"@muthuishere/ctx-optimize-linux-arm64": "0.2.
|
|
21
|
-
"@muthuishere/ctx-optimize-linux-x64": "0.2.
|
|
22
|
-
"@muthuishere/ctx-optimize-windows-x64": "0.2.
|
|
18
|
+
"@muthuishere/ctx-optimize-darwin-arm64": "0.2.2",
|
|
19
|
+
"@muthuishere/ctx-optimize-darwin-x64": "0.2.2",
|
|
20
|
+
"@muthuishere/ctx-optimize-linux-arm64": "0.2.2",
|
|
21
|
+
"@muthuishere/ctx-optimize-linux-x64": "0.2.2",
|
|
22
|
+
"@muthuishere/ctx-optimize-windows-x64": "0.2.2"
|
|
23
23
|
}
|
|
24
24
|
}
|