@chroxify/eunoe 1.2.2 → 1.3.1

package/README.md

@@ -243,7 +243,10 @@ a small and very fast model, reorders the top 30 in about 300 ms, which
 lifted the right excerpt from 88% to 95% of questions and beat every other
 method and model that was tested. Without a key the BM25 order is used and
 nothing else changes. The recall is computed once per message and pinned to
-it, so it never disturbs the cache.
+it, so it never disturbs the cache. Once that message is folded its recall is
+dropped with it, and the recall on the messages still in context stays under
+30K characters in total and under a quarter of the context (with a 6K floor),
+so it never crowds out the conversation itself.
 
 ## How it plugs in
 
@@ -430,6 +433,143 @@ reads, and every token a fold rewrote is charged back at the cache-write price.
 deletes sessions that have been idle for 30 days unless `cleanupPeriodDays` in
 its `settings.json` says otherwise.
 
+## Embedding
+
+The proxy is one host of a small library. Anything that sends Anthropic
+Messages requests can use the same folding, recall and estimates: a
+forwarding server, an agent loop of your own, a Cloudflare Durable Object,
+another proxy. The core reads no files, no environment and no global
+config, and it runs in Node, Bun, Deno and Workers.
+
+```sh
+npm install @chroxify/eunoe
+```
+
+```ts
+import { CORE_DEFAULTS, memoryStore, rewrite } from "@chroxify/eunoe"
+
+const store = memoryStore()
+const { body, note } = rewrite({
+  body: request,             // the Messages request you were about to send
+  threadId: conversationId,  // anything stable per conversation
+  config: { ...CORE_DEFAULTS, mode: "compact" },
+  store,
+  window: 200_000,           // the model's context window
+  mayCompact: true,          // false for count_tokens and other side calls
+})
+// send `body` instead of `request`; `note` says what happened
+```
+
+`rewrite` is a pure function of the request and the thread's saved state,
+so the same history always produces the same bytes and the prompt cache
+holds. Everything it needs from outside comes in through three small
+pieces:
+
+| | |
+|---|---|
+| `ThreadStore` | `read(key)` and `save(key, state)`, synchronous. `memoryStore()` keeps state in memory; give it your own to persist across restarts. |
+| `ThreadSources` | optional: `readHistory()` for the whole conversation when you send a shortened one, `findHandoff` and `readHandoff` for `{ee:continue:…}`, `guide` and `transcriptDir` when the agent can search a transcript. |
+| `Reranker` | optional, for `rewriteAsync`: reorders recall candidates. `jevReranker(model, key)` is the one the proxy uses. |
+
+In a Durable Object, SQLite-backed storage is synchronous, so the store is
+two lines:
+
+```ts
+const store = {
+  read: (key) => this.ctx.storage.kv.get(key) ?? {},
+  save: (key, state) => this.ctx.storage.kv.put(key, state),
+}
+```
+
+### Folding at a boundary you know
+
+A host often knows better than any token count when a step is over. After a
+planning step, fold every earlier turn and keep only the plan whole:
+
+```ts
+import { foldThread, pinTurns } from "@chroxify/eunoe"
+
+messages.push({ role: "user", content: "Build step one." })
+foldThread({ store, threadId, messages }, { at: "latest", pin: [planTurn] })
+```
+
+Turns are numbered as `note.cutTurn` numbers them, from 0 for the first
+prompt, and the boundary must be a turn that already exists, so fold once
+the next step's prompt is in the list. Every turn before the boundary is
+folded to the user's message, the evidence and the reply. Pinned turns are
+sent whole, with their tool calls, and stay whole even when a later
+automatic cut drops the turns around them. `pinTurns(target, [n], false)`
+releases one. A host fold applies in every mode, `off` included, and its
+output stays byte-stable from one request to the next.
+
+### The note
+
+Every call returns a note: `action` (`pass`, `full` or `compact`),
+`cutTurn`, `folded`, `omitted`, `pinned`, `estimate` in tokens, `fit` when
+the request had to be trimmed to fit (`recut`, `bare` or `over`), and
+`stuck` when even the cut leaves it over the limit. `stuck` gives the
+tokens in each part (`fixed`, `head`, `folded`, `live`) so the host can
+act: start a fresh conversation with a handoff, or fold at its own
+boundary. It clears as soon as a request fits. While a thread is stuck,
+eunoe holds what it cleared still instead of clearing a little more on
+every request, and clears again only when that makes the request fit or
+brings it back under the window, so the cache keeps hitting.
+
+### The whole proxy as one call
+
+`createForwarder` is the proxy's request loop as a fetch handler: it
+rewrites each `/v1/messages` request, forwards everything to the API,
+streams the answer back, learns the thread's real token ratio from the
+answer's `usage`, retries once after a "prompt is too long" against the
+limit the error names (and keeps that limit for the thread and model), retries a
+stalled `count_tokens`, and looks the context window up from the Models API. `eunoe serve` is this call with a file-backed
+store, Claude Code's session files as sources and a request log.
+
+```ts
+import { CORE_DEFAULTS, createForwarder, memoryStore } from "@chroxify/eunoe"
+
+const forward = createForwarder({ store: memoryStore(), config: CORE_DEFAULTS })
+Bun.serve({ port: 8787, fetch: forward })          // or Deno.serve, a Worker's fetch, …
+```
+
+Threads are told apart by an `x-thread-id` header, or Claude Code's own
+session id; pass `threadIdOf`, `sources`, `reranker`, `window` or `record`
+to change any part, or `prepare` to take the rewrite over entirely. A host
+that calls the API itself gets the same calibration with
+`recordUsage(store, threadKey, sentBody, usage)` and
+`recordOverflow(store, threadKey, sentBody, errorText)`.
+
+| export | |
+|---|---|
+| `rewrite`, `rewriteAsync` | rewrite one request for one thread; the async one takes a reranker |
+| `foldThread`, `pinTurns` | fold at your own boundary, keep chosen turns whole |
+| `createForwarder` | the full proxy loop as a `(Request) => Promise<Response>` |
+| `recordUsage`, `recordOverflow` | feed what the API said back into the thread's estimates |
+| `createWindowLookup` | a model's context window from the Models API, cached, with a per-family fallback |
+| `memoryStore`, `threadKeyOf` | in-memory state, and the key a thread is saved under |
+| `listSubagentRecall` | recall over what your own subagents printed |
+| `renderTranscriptFiles`, `transcriptGuide` | the searchable per-turn transcript from your own messages, and the text that points the agent at it |
+| `jevReranker`, `continueToken`, `watchUsage`, `claudeCodeSessionId` | the pieces the proxy uses, for hosts that want them |
+
+What stays in the proxy is what only makes sense next to Claude Code on
+your machine: reading its session files and subagent runs, finding earlier
+sessions for `{ee:continue:…}`, rendering the markdown transcript, the
+config file and the CLI.
+
+Every export, with its signature, what it does and an example, is in
+[`docs/api.md`](docs/api.md), which starts with where each kind of state and
+data lives. [`examples/`](examples) runs as it is. [`tour.ts`](examples/tour.ts) walks
+through every feature on real inputs and prints what each did: folding,
+the full cut, evidence, recall, recall from host history, reranking,
+handoffs, host folds and pins, `stuck`, calibration, custom storage, the
+forwarder and transcripts. The others are hosts: a
+[Durable Object](examples/durable-object.ts) per thread on its own storage,
+tested in real `workerd`, a
+[forwarding server](examples/forwarding-server.ts) on `node:http` with a
+`/fold` endpoint, a [custom agent loop](examples/agent-loop.ts) that folds
+at each step and keeps its plan pinned, and a
+[Claude Code proxy](examples/proxy.ts) with a file-backed store.
+
 ## What it doesn't do
 
 - **It is a proxy, which makes it a single point of failure.** If it isn't
@@ -442,17 +582,21 @@ its `settings.json` says otherwise.
   502 that Claude Code retries, never a hang. Sessions that were already
   running keep the old address until restarted, so they wait for the proxy
   rather than falling back.
-- **It only understands Claude Code's shape of conversation.** Other
-  Anthropic API clients pass through unchanged and get nothing out of it.
+- **The proxy only understands Claude Code's shape of conversation.** Other
+  Anthropic API clients pass through it unchanged and get nothing out of
+  it; they can [embed the library](#embedding) instead.
 - **It can't recover what Claude Code already compacted,** so turn it on
   before a long session rather than after.
 - **Recall is keyword search.** Ask about "the deploy that failed" and it
   finds the deploy output, but ask about "that thing from before" and it
   finds nothing, leaving the agent to search the transcript itself. Jev
   narrows this gap without closing it.
-- **It never cuts against a limit it can't verify.** The context window size
-  comes from the Models API for each model, and if that lookup fails the
-  budget folds still happen but the full cut does not.
+- **The window can be a guess.** The context window size comes from the
+  Models API for each model. When that lookup fails or takes over 1.5 s,
+  eunoe assumes the model family's window (1M for Claude 5 models, Opus
+  4.8 and `[1m]` ids, 200K for other Claude models) and asks again ten
+  minutes later. A guess that is too big is caught by the "prompt is too
+  long" retry, which then keeps the real limit for that thread and model.
 
 ## Credits