@muthuishere/ctx-optimize 0.4.2 → 0.5.1

package/README.md

@@ -3,7 +3,7 @@
 [![CI](https://github.com/muthuishere/ctx-optimize/actions/workflows/ci.yml/badge.svg)](https://github.com/muthuishere/ctx-optimize/actions/workflows/ci.yml)
 [![Go Reference](https://pkg.go.dev/badge/github.com/muthuishere/ctx-optimize.svg)](https://pkg.go.dev/github.com/muthuishere/ctx-optimize)
 [![npm](https://img.shields.io/npm/v/@muthuishere/ctx-optimize?logo=npm)](https://www.npmjs.com/package/@muthuishere/ctx-optimize)
-[![benchmark](https://img.shields.io/badge/benchmark-run%20it%20yourself-4ade80)](https://muthuishere.github.io/ctx-optimize-site/proof/agent/)
+[![benchmark](https://img.shields.io/badge/benchmark-run%20it%20yourself-4ade80)](proof/agent/)
 [![platforms](https://img.shields.io/badge/platforms-macOS%20%7C%20Linux%20%7C%20Windows-blue)](https://www.npmjs.com/package/@muthuishere/ctx-optimize)
 [![license](https://img.shields.io/badge/license-MIT-green)](LICENSE)
 
@@ -36,6 +36,10 @@ already run.
 > learning loop, merge/export (json/dot/graphml/csv/obsidian), **scripted
 > remote push/pull**, and **multi-module monorepo support** (`scan` /
 > `init --scan` / parallel fan-out `add` / navigator + federated queries).
+> New on main (unreleased): **native sources** — databases, buckets, queues,
+> and external APIs enter the store by env-var name (`ctx-optimize add
+> BILLING_DB_URL`); 9 wire-protocol connectors in a companion binary — see
+> [Databases, buckets, queues, APIs](#databases-buckets-queues-apis--native-sources).
 > New in v0.4 (**breaking**): **the remote is your script** — the binary
 > ships no transport of its own; `remote push`/`pull` run the commands you
 > declare in the committed config (`remote init` and the built-in
@@ -56,7 +60,7 @@ already run.
 > packs (`routes add` / `manifests add`, name or GitHub URL) exactly like
 > grammar packs. Exact call edges (x/tools + LSP) are next — see `openspec/`.
 
-**Site, demos, benchmarks:** https://muthuishere.github.io/ctx-optimize-site/
+**Demos, benchmarks, proof:** [`benchmarks/`](benchmarks/) · [`proof/`](proof/) — everything reproducible from this repo.
 — landing page, unedited demos, and the full proof write-up. Everything below
 is reproducible; see [Proof](#proof--reproducible-not-our-word).
 
@@ -134,8 +138,12 @@ ctx-optimize change-plan "RefundService"
 ctx-optimize sync
 ctx-optimize adapters run
 
-# feed ANY system through the universal adapter door (strictly validated)
-./my-postgres-adapter | ctx-optimize add --json -
+# native sources: a database/bucket/queue/API by env-var name — the value
+# is a URL, its scheme picks the connector; recorded, refreshed on every up
+ctx-optimize add BILLING_DB_URL
+
+# feed ANY other system through the universal adapter door (strictly validated)
+./my-exotic-adapter | ctx-optimize add --json -
 
 # combine module stores into one view; dump for other tools
 ctx-optimize merge api worker billing --into everything
@@ -198,6 +206,41 @@ Upgrading from v0.3: `remote init` and the built-in `file://`/`s3://`
 transports are gone. A legacy URL-shaped config still loads but is inert —
 `push`/`pull` print the migration pointer.
 
+## Databases, buckets, queues, APIs — native sources
+
+**A source is an environment variable name. Its value is a URL. The URL
+scheme picks the connector.** One command from zero to "refreshed on every
+`up`":
+
+```sh
+ctx-optimize adapters help postgres    # setup card: value format, credential params, paste-ready commands
+export BILLING_DB_URL='postgres://reader:$PG_PASS@db.internal:5432/billing'   # or .ctxoptimize/.env (gitignored by construction)
+ctx-optimize add BILLING_DB_URL        # resolve → dial → capture → merge → recorded in config sources
+```
+
+Nine wire-protocol-native connectors — **postgres, mysql, mongodb, redis,
+kafka, nats, s3** (MinIO/R2 via endpoint hosts, bare AWS via the credential
+chain), **mssql**, and **openapi** (http(s) URL or a spec file path) — no
+pg_dump/atlas/tbls needed on any machine. Captures are the **logical shape**
+a developer reasons about: system schemas skipped, a partitioned table is one
+node with `partitions: N`, bounded samples with every cap reported. Measured:
+a 100-table / 3-schema postgres captures in **31 ms** including connect
+(pg_dump 101 ms, atlas 248 ms, tbls 1356 ms) — and where a 100-partition
+table plus 500 Timescale chunks bloat other tools to 600–716 raw tables, it
+emits **101 logical tables**.
+
+Secret hygiene is structural: argv and committed config carry env-var
+**names** only (a literal password in an entry is a hard error), values
+resolve process env → `.ctxoptimize/.env` → root `.env` in memory at dial
+time, stored ids are sanitized, and every output is scrubbed. A teammate
+without the credentials still runs `up` cleanly — that source is a one-line
+skip and the nodes arrive via `remote pull`; `--strict` turns skips into CI
+failures. Recorded sources refresh on `up` under a 24h TTL
+(`--sources=always|never`). The drivers live in a **companion binary**,
+`ctx-optimize-adapters`, shipped beside the main one in every archive and
+npm package — the main binary stays driver-free and exactly as fast, and
+execs the sibling only when a source dials.
+
 ## Multi-module — monorepos get one graph per module, plus a navigator
 
 One giant graph for a 300-module monorepo helps nobody: people work in one
@@ -249,7 +292,7 @@ Two kinds of evidence, both runnable.
 
 **Speed vs graphify** (raw data in [`benchmarks/`](benchmarks/)): a 12k-file
 corpus gathered in **0.67s vs 8.88s**, queries **~4× faster**, a smaller
-store. Methodology on the site.
+store. Methodology beside the raw data in [`benchmarks/`](benchmarks/).
 
 **What an agent actually saves.** A headless harness lets the *same* model
 answer a set of questions **three ways** over OpenRouter — plain shell,
@@ -279,16 +322,42 @@ bash proof/agent/run-bench.sh           # defaults: gorilla/mux, openai/gpt-4o-m
 
 Or fork and click **Run workflow** — [`.github/workflows/benchmark.yml`](.github/workflows/benchmark.yml)
 runs it headless on a clean runner and publishes the table to the job summary.
-Harness + full write-up: https://muthuishere.github.io/ctx-optimize-site/proof/agent/
+Harness + full write-up: [`proof/agent/`](proof/agent/)
+
+**The model ladder** ([`benchmarks/agent-model-bench/`](benchmarks/agent-model-bench/)):
+same prebuilt linux-kernel store (~274k nodes), same 8 block-layer questions,
+one fresh agent session per model, answers judged blind against withheld
+golden keys:
+
+| model · harness | score /80 | avg s/question | tool calls (8 q) |
+|---|---|---|---|
+| Fable 5 · Claude Code | **80** | 24.6 | 23 |
+| Sonnet 5 · Claude Code | **80** | 17.5 | 25 |
+| Opus 4.8 · Claude Code | **79** | 19.0 | 22 |
+| Haiku 4.5 · Claude Code | **72** | 13.6 | 18 |
+| gpt-4o-mini · toolnexus, one-shot | **54** | 9.4 | 24 |
+
+The store is model-portable: the cheapest Claude tier lands 90% of frontier
+quality at half the wall time, and a $0.15/M-token model reaches ~70% — for
+$0.015 total — when the mandatory protocol from the committed
+`.ctxoptimize/instructions.md` card ("Small models & custom runtimes") is
+pinned in its system prompt. Without that protocol the same small model
+scored 23/80. One-shot per question beats a continuous loop: same score,
+7× cheaper, no cross-question bleed.
 
 ## .ctxoptimize/ — config that travels with the repo
 
 ```
 .ctxoptimize/
-  config.json          name + remote commands (+ modules[] in a monorepo)
+  config.json          name + remote commands + sources[] (+ modules[] in a monorepo)
+  instructions.md      the committed usage card agents read — managed block,
+                       version-stamped, refreshed by `up` (upgrade-only; your
+                       edits outside the markers are never touched)
   adapters/            drop scripts here — every .js/.py/.sh runs on `add`
   push.js / pull.js    your transport scripts (init writes an inert *.sample pair)
   remote.example.md    transport recipes: git lane, s3 lane, custom
+  .gitignore           covers .env* by construction — put source URLs with
+                       secrets in .ctxoptimize/.env, never in config
 ```
 
 `config.json`:

package/bin/ctx-optimize-adapters.js

@@ -0,0 +1,46 @@
+#!/usr/bin/env node
+// Thin launcher for the capture companion: resolves the platform sub-package
+// installed via optionalDependencies and execs its ctx-optimize-adapters
+// binary (shipped beside ctx-optimize in the same bin/). No postinstall.
+"use strict";
+const { spawn } = require("child_process");
+const path = require("path");
+
+const platformMap = {
+  "darwin-arm64": "@muthuishere/ctx-optimize-darwin-arm64",
+  "darwin-x64": "@muthuishere/ctx-optimize-darwin-x64",
+  "linux-arm64": "@muthuishere/ctx-optimize-linux-arm64",
+  "linux-x64": "@muthuishere/ctx-optimize-linux-x64",
+  "win32-x64": "@muthuishere/ctx-optimize-windows-x64",
+};
+
+const key = `${process.platform}-${process.arch}`;
+const pkg = platformMap[key];
+if (!pkg) {
+  console.error(`ctx-optimize-adapters: unsupported platform ${key}`);
+  process.exit(1);
+}
+
+const exe =
+  process.platform === "win32" ? "ctx-optimize-adapters.exe" : "ctx-optimize-adapters";
+let binPath;
+try {
+  binPath = path.join(path.dirname(require.resolve(`${pkg}/package.json`)), "bin", exe);
+} catch {
+  console.error(
+    `ctx-optimize-adapters: platform package ${pkg} is not installed.\n` +
+      `Your package manager skipped an optional dependency — reinstall with:\n` +
+      `  npm install -g @muthuishere/ctx-optimize`
+  );
+  process.exit(1);
+}
+
+const child = spawn(binPath, process.argv.slice(2), { stdio: "inherit" });
+child.on("exit", (code, signal) => {
+  if (signal) process.kill(process.pid, signal);
+  process.exit(code === null ? 1 : code);
+});
+child.on("error", (err) => {
+  console.error(`ctx-optimize-adapters: failed to start binary: ${err.message}`);
+  process.exit(1);
+});

package/package.json

@@ -1,9 +1,10 @@
 {
   "name": "@muthuishere/ctx-optimize",
-  "version": "0.4.2",
+  "version": "0.5.1",
   "description": "Gather a codebase (and its world) into one local knowledge store an AI agent answers from — deterministic, no LLM API, no DB.",
   "bin": {
-    "ctx-optimize": "bin/ctx-optimize.js"
+    "ctx-optimize": "bin/ctx-optimize.js",
+    "ctx-optimize-adapters": "bin/ctx-optimize-adapters.js"
   },
   "files": [
     "bin/",
@@ -15,10 +16,10 @@
     "url": "git+https://github.com/muthuishere/ctx-optimize.git"
   },
   "optionalDependencies": {
-    "@muthuishere/ctx-optimize-darwin-arm64": "0.4.2",
-    "@muthuishere/ctx-optimize-darwin-x64": "0.4.2",
-    "@muthuishere/ctx-optimize-linux-arm64": "0.4.2",
-    "@muthuishere/ctx-optimize-linux-x64": "0.4.2",
-    "@muthuishere/ctx-optimize-windows-x64": "0.4.2"
+    "@muthuishere/ctx-optimize-darwin-arm64": "0.5.1",
+    "@muthuishere/ctx-optimize-darwin-x64": "0.5.1",
+    "@muthuishere/ctx-optimize-linux-arm64": "0.5.1",
+    "@muthuishere/ctx-optimize-linux-x64": "0.5.1",
+    "@muthuishere/ctx-optimize-windows-x64": "0.5.1"
   }
 }