@tokz/ai-sdk 0.1.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 tokz
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,120 @@
+# `@tokz/ai-sdk`
+
+Drop-in wrapper for the [Vercel AI SDK](https://ai-sdk.dev). It intercepts the
+tool-to-model boundary: your tools keep returning what they always returned, and the
+model sees a compressed view of it.
+
+Compression is structural — every byte the model reads is a byte-exact substring of
+the tool's own output, chosen deterministically on the CPU. Nothing is rewritten, and
+nothing leaves the process.
+
+## Install
+
+```bash
+pnpm add @tokz/ai-sdk ai
+```
+
+`ai` is a peer dependency (`>=5`). This package builds and typechecks without it; you
+only need it at runtime, and even then only if you do not pass your own `delegate`.
+
+## Usage
+
+```ts
+import { generateText } from "@tokz/ai-sdk";
+import { openai } from "@ai-sdk/openai";
+import { tool } from "ai";
+import { z } from "zod";
+
+const listPods = tool({
+  description: "List pods in a namespace",
+  inputSchema: z.object({ namespace: z.string() }),
+  execute: async ({ namespace }) => {
+    const res = await fetch(`https://k8s.internal/api/v1/namespaces/${namespace}/pods`);
+    return res.json(); // 40 KB of near-identical rows
+  },
+});
+
+const result = await generateText({
+  model: openai("gpt-5.4"),
+  messages: [{ role: "user", content: "Which pods in prod are unhealthy?" }],
+  tools: { listPods },
+  tokz: {
+    targetRatio: 0.45,
+    autoExpand: "on-reference",
+    conversationCompaction: true,
+  },
+});
+
+console.log(result.text);
+console.log(result.tokz.originalBytes, "->", result.tokz.renderedBytes);
+```
+
+Everything except `tokz` is forwarded to `ai.generateText` untouched, and the result
+is the SDK's result with one extra field, `result.tokz`.
+
+## The `tokz` option block
+
+| Option | Default | Effect |
+| --- | --- | --- |
+| `targetRatio` | `0.45` | Share of tool-output bytes to keep. The structural skeleton is never dropped, so the achieved ratio can be higher. |
+| `autoExpand` | `"never"` | `"on-reference"` recovers elided content the answer refers to and re-prompts **once**. |
+| `conversationCompaction` | `false` | Re-compresses retained tool results in `messages` before delegating. |
+| `compactionRatio` | `0.6 * targetRatio` | Budget used by that compaction. |
+| `store` | shared store | Where originals and span maps are retained. Pass one per conversation if a process serves many. |
+| `delegate` | `ai.generateText` | The underlying call. Injectable for tests. |
+
+## Recovering the original
+
+The original text never reaches the model, but it never leaves you either:
+
+```ts
+import { expand } from "@tokz/ai-sdk";
+
+const record = result.tokz.store.get(toolCallId)!;
+record.originalText;                                  // exactly what the tool returned
+expand(record.originalText, record.result.spanMap!);  // verified round-trip
+expand(record.originalText, record.result.spanMap!, { elision: 0 }); // just the dropped run
+```
+
+## Compacting history
+
+```ts
+import { compactHistory } from "@tokz/ai-sdk";
+
+const tighter = compactHistory(messages, 0.2);
+```
+
+This is a pure local function. The original text and its span map are already in the
+store, so a smaller view of an old tool result is the same deterministic CPU pass that
+produced the first view — **zero API calls, zero inference**. A wrapper that discarded
+the original would have to ask a model to summarise history, which costs more than the
+tokens it saves.
+
+Messages are not mutated; changed ones come back as new objects.
+
+## Auto-expand is a heuristic
+
+`detectReferencedElisions` decides that the model "referred to" elided content by
+vocabulary overlap: it recovers each elided run locally, takes the identifiers in it,
+and flags the run when the response mentions one at a word boundary. It over-fires on
+generic words and under-fires when the model paraphrases without naming anything.
+
+Treat a hit as a cheap reason to re-supply context, not as proof. The *recovery* is
+exact — `expand` verifies `srcSha256` and returns the dropped bytes verbatim — only
+the *trigger* is fuzzy.
+
+```ts
+import { detectReferencedElisions, expansionMessage } from "@tokz/ai-sdk";
+
+const hits = detectReferencedElisions(result.text!, result.tokz.store);
+if (hits.length) messages.push(expansionMessage(hits));
+```
+
+## What the model actually sees
+
+Wrapped tools set `toModelOutput` — the AI SDK's documented tool-to-model hook — *and*
+return the compressed render from `execute`. Both, deliberately: the hook is the
+correct mechanism, and returning the render as well means the original cannot reach a
+message even on an SDK version that skips the hook. Tools with no server-side
+`execute` (client-side or provider-executed) are left alone, as are tools that stream
+their output as an `AsyncIterable`.

package/dist/index.d.ts

@@ -0,0 +1,342 @@
+/**
+ * The wire contract. Every SDK, API response, and customer integration binds to
+ * these shapes. Versioned; changes must be additive.
+ *
+ * All offsets are UTF-8 **byte** offsets, half-open [s, e) — not JS string
+ * indices — so "byte-exact substring" survives non-ASCII payloads and any future
+ * non-JS SDK.
+ */
+type Span = {
+    s: number;
+    e: number;
+};
+/** An elided run. `after` indexes into `spans[]`: the elision sits immediately
+ *  after that span (`-1` means it precedes the first). `bytes` counts dropped
+ *  source bytes. */
+type Elision = {
+    after: number;
+    bytes: number;
+    note?: string;
+};
+/** Content types the structural path can parse. `unknown` is an honest
+ *  passthrough, not an error. */
+type StructuralContentType = "json" | "jsonl" | "markdown-table" | "csv" | "diff" | "unknown";
+/** Content types the semantic path handles. */
+type SemanticContentType = "prose" | "rag" | "chat";
+type ContentType = StructuralContentType | "prose";
+type SpanMap = {
+    version: 1;
+    srcBytes: number;
+    /** SHA-256 hex of the source bytes. Verified before expanding, so a map can
+     *  never be applied to text it was not built from. */
+    srcSha256: string;
+    contentType: ContentType;
+    /** Sorted ascending, non-overlapping, adjacency-merged. */
+    spans: Span[];
+    elisions: Elision[];
+    stats: {
+        keptBytes: number;
+        ratio: number;
+    };
+};
+/**
+ * One run of rewritten output traced back to the input ranges that produced it.
+ * This is what a black-box rewriter cannot offer: even when the text is not
+ * byte-identical, every part of it is attributable.
+ */
+type ProvenanceChunk = {
+    /** Byte offsets into the compressed output. */
+    outS: number;
+    outE: number;
+    /** Input ranges that contributed. */
+    inSpans: Span[];
+    /** Model confidence in [0, 1]. */
+    confidence: number;
+};
+type ProvenanceMap = {
+    version: 1;
+    srcBytes: number;
+    srcSha256: string;
+    contentType: SemanticContentType;
+    chunks: ProvenanceChunk[];
+    stats: {
+        keptBytes: number;
+        ratio: number;
+    };
+};
+/**
+ * Discriminated on `method` so a consumer cannot read `compressedText` off a
+ * structural result: the structural path never produces text server-side, and
+ * the type system should enforce that rather than a convention.
+ */
+type CompressionResult = {
+    method: "structural";
+    spanMap: SpanMap;
+    compressedText?: never;
+    provenanceMap?: never;
+} | {
+    method: "semantic";
+    compressedText: string;
+    provenanceMap: ProvenanceMap;
+    spanMap?: never;
+} | {
+    method: "passthrough";
+    spanMap: SpanMap;
+    compressedText?: never;
+    provenanceMap?: never;
+};
+type CompressOpts = {
+    /** Fraction of source bytes to aim to keep. Structural scaffolding is never
+     *  dropped, so the achieved ratio may exceed this — read `stats.ratio`. */
+    targetRatio?: number;
+    /** Optional query. Boosts salience of nodes matching its terms, and is passed
+     *  to the semantic classifier when the semantic path runs. */
+    query?: string;
+    /** Skip detection and force a handler. */
+    contentType?: ContentType | "auto";
+};
+type ExpandOpts = {
+    /** Recover only the run dropped by this elision, rather than the whole source. */
+    elision?: number;
+};
+
+/**
+ * Local, network-free reconstruction from the caller's own source plus a span
+ * map. This is what lets a stateless API offer recovery at all: we hold nothing,
+ * because the caller already has everything.
+ *
+ * - no opts       -> the full original text, after verifying `srcSha256`.
+ * - {elision: i}  -> just the bytes elision `i` dropped, recovered verbatim.
+ *
+ * Throws if the map was not built from this source, rather than returning
+ * plausible-looking misaligned output.
+ */
+declare function expand(source: string | Uint8Array, spanMap: SpanMap, opts?: ExpandOpts): string;
+
+/**
+ * The Vercel AI SDK surface we bind to, restated structurally.
+ *
+ * Verified 2026-07-22 against ai@7.0.34 (the copy in this workspace's pnpm store,
+ * `dist/index.js` — `createToolModelOutput` awaits `tool.toModelOutput({ toolCallId,
+ * input, output })` when building tool-result message parts) and the published docs:
+ * - `generateText(options)` takes `{ model, messages | prompt, tools, stopWhen, ... }`
+ *   and resolves to `{ text, toolCalls, toolResults, steps, response, ... }`.
+ *   Ref: https://ai-sdk.dev/docs/reference/ai-sdk-core/generate-text
+ * - a tool is `{ description, inputSchema, execute?, toModelOutput? }`. `inputSchema`
+ *   replaced v4's `parameters`; the `tool()` helper is type-inference only and has
+ *   no runtime behaviour (packages/provider-utils/src/types/tool.ts), so wrapping a
+ *   plain object is equivalent to wrapping a `tool()` result.
+ * - `ToolExecuteFunction` is `(input, { toolCallId, messages, abortSignal, ... }) =>
+ *   OUTPUT | PromiseLike<OUTPUT> | AsyncIterable<OUTPUT>`; the resolved value becomes
+ *   the tool output.
+ * - `toModelOutput({ toolCallId, input, output }) => ToolResultOutput` maps the tool
+ *   output to what the *model* sees, independently of what the app sees. That is the
+ *   tool-to-model boundary this package intercepts.
+ *
+ * Restated rather than imported because `ai` is an optional peer: the package must
+ * typecheck and build with no `ai` installed, and the shapes above have been renamed
+ * across majors (parameters -> inputSchema, LanguageModelV2 -> V4) while these three
+ * fields have not.
+ */
+type ToolResultOutput = {
+    type: "text";
+    value: string;
+} | {
+    type: "json";
+    value: unknown;
+} | {
+    type: "error-text";
+    value: string;
+} | {
+    type: "error-json";
+    value: unknown;
+} | {
+    type: "content";
+    value: unknown[];
+};
+type ToolExecuteOptions = {
+    toolCallId?: string;
+    messages?: unknown;
+    abortSignal?: AbortSignal;
+};
+type ToolLike = {
+    description?: string;
+    inputSchema?: unknown;
+    execute?: (input: never, options: ToolExecuteOptions) => unknown;
+    toModelOutput?: (options: {
+        toolCallId: string;
+        input: unknown;
+        output: unknown;
+    }) => ToolResultOutput | PromiseLike<ToolResultOutput>;
+};
+type ToolSetLike = Record<string, ToolLike>;
+type ToolResultPart = {
+    type: "tool-result";
+    toolCallId: string;
+    toolName?: string;
+    /** v5+ shape. v4 put the value on `result` instead; both are handled. */
+    output?: ToolResultOutput;
+    result?: unknown;
+};
+type ModelMessageLike = {
+    role: string;
+    content: unknown;
+};
+type GenerateTextFn = (options: Record<string, unknown>) => Promise<GenerateTextResultLike>;
+type GenerateTextResultLike = {
+    text?: string;
+    toolCalls?: unknown;
+    toolResults?: unknown;
+    steps?: unknown;
+    response?: {
+        messages?: ModelMessageLike[];
+    };
+};
+type AutoExpandMode = "on-reference" | "never";
+type TokzOptions = {
+    /** Share of source bytes to keep. Defaults to `DEFAULT_TARGET_RATIO` (0.45). */
+    targetRatio?: number;
+    /** Expand elisions the model's answer refers to, and re-prompt once. Default `never`. */
+    autoExpand?: AutoExpandMode;
+    /** Re-compress retained tool results at a tighter budget before delegating. */
+    conversationCompaction?: boolean;
+    /** Budget used when `conversationCompaction` is on. Default 0.6 * targetRatio. */
+    compactionRatio?: number;
+    /** Where originals and span maps are retained. Defaults to the shared store, so
+     *  `compactHistory` and auto-expand work across turns without threading state. */
+    store?: RetentionStore;
+    /** Underlying `ai.generateText`. Injected in tests; otherwise imported from `ai`. */
+    delegate?: GenerateTextFn;
+};
+/**
+ * The original text plus its map, held next to the compressed render. This pair is
+ * what makes re-compaction and expansion local: nothing has to be fetched or
+ * re-inferred to produce a different view of the same tool result.
+ */
+type RetentionRecord = {
+    toolName: string;
+    toolCallId: string;
+    originalText: string;
+    result: CompressionResult;
+    rendered: string;
+    targetRatio: number;
+};
+interface RetentionStore {
+    get(toolCallId: string): RetentionRecord | undefined;
+    set(record: RetentionRecord): void;
+    records(): RetentionRecord[];
+    clear(): void;
+}
+
+type ReferencedElision = {
+    toolCallId: string;
+    toolName: string;
+    elisionIndex: number;
+    /** Terms from the elided bytes that appear in the model's response. */
+    matchedTerms: string[];
+    /** The elided bytes, recovered verbatim from the caller's own text. */
+    text: string;
+    note?: string;
+};
+type DetectOptions = {
+    /** Shortest token considered a reference. Below 4 characters the match rate is
+     *  dominated by English words that happen to appear in JSON. */
+    minTermLength?: number;
+    /** Cap on terms sampled per elision, to bound work on multi-megabyte drops. */
+    maxTermsPerElision?: number;
+};
+/**
+ * HEURISTIC. There is no reliable signal that a model "meant" an elided field, so
+ * this matches vocabulary: recover each elided run locally, take the identifiers it
+ * contains, and flag the elision when the response mentions one of them. It over-
+ * fires on generic words (mitigated by a stopword list and a length floor) and
+ * under-fires when the model paraphrases without naming anything. Treat a hit as a
+ * cheap prompt to re-supply context, never as proof of a reference.
+ *
+ * Recovery itself is exact: `expand` verifies `srcSha256` and returns the dropped
+ * bytes verbatim, with no network call.
+ */
+declare function detectReferencedElisions(responseText: string, store?: RetentionStore, opts?: DetectOptions): ReferencedElision[];
+/** A user-role message re-supplying the recovered bytes, ready to append before a
+ *  follow-up turn. Separate from detection so callers can inspect first. */
+declare function expansionMessage(expansions: readonly ReferencedElision[]): ModelMessageLike;
+
+type TokzGenerateTextOptions = Record<string, unknown> & {
+    tokz?: TokzOptions;
+};
+type TokzReport = {
+    store: RetentionStore;
+    targetRatio: number;
+    /** Bytes of tool output before and after compression, this call only. */
+    originalBytes: number;
+    renderedBytes: number;
+    expansions: ReferencedElision[];
+    /** Whether a second delegate call was made to re-supply expanded content. */
+    reprompted: boolean;
+};
+type TokzGenerateTextResult = GenerateTextResultLike & {
+    tokz: TokzReport;
+};
+/**
+ * Drop-in replacement for `ai.generateText`. Every option is forwarded untouched
+ * except `tools`, which is wrapped so results reach the model compressed, and
+ * `messages`, which is optionally compacted first.
+ */
+declare function generateText(options: TokzGenerateTextOptions): Promise<TokzGenerateTextResult>;
+
+/**
+ * Re-compress retained tool results at a tighter budget.
+ *
+ * This is a pure local function: the original text and its span map are already in
+ * the store, so producing a smaller view of an old tool result costs zero API calls
+ * and zero inference — it is the same deterministic CPU pass that produced the first
+ * view. That property is the whole point of retaining the pair; a wrapper that threw
+ * the original away would have to re-ask a model to shrink history, which costs more
+ * than the tokens it saves.
+ *
+ * Input messages are not mutated; changed messages are returned as new objects.
+ */
+declare function compactHistory(messages: readonly ModelMessageLike[], targetRatio: number, store?: RetentionStore): ModelMessageLike[];
+
+type WrapContext = {
+    store: RetentionStore;
+    targetRatio: number;
+};
+/** Tool output as text the compressor can work on. Pretty-printed because the
+ *  structural detector reads JSON shape, and key order follows the object, so the
+ *  same result always serialises to the same bytes. */
+declare function stringifyToolResult(value: unknown): string;
+/**
+ * Wrap one tool so the model receives the compressed render while the original text
+ * and its span map stay in the store.
+ *
+ * `execute` returns the render rather than the raw value, and `toModelOutput` maps
+ * to the same string. Doing both is deliberate: `toModelOutput` is the documented
+ * tool-to-model hook, but returning the render from `execute` means the original
+ * cannot reach a message even under an SDK version that ignores the hook. The
+ * original is one `store.get(toolCallId)` away for the caller.
+ */
+declare function wrapTool(toolName: string, tool: ToolLike, ctx: WrapContext): ToolLike;
+declare function wrapTools(tools: ToolSetLike, ctx: WrapContext): ToolSetLike;
+
+/** Insertion-ordered so `records()` reads back in call order — history compaction
+ *  and any future oldest-first budget both depend on that order being stable. */
+declare class MapRetentionStore implements RetentionStore {
+    #private;
+    get(toolCallId: string): RetentionRecord | undefined;
+    set(record: RetentionRecord): void;
+    records(): RetentionRecord[];
+    clear(): void;
+}
+/**
+ * Process-wide default so `compactHistory(messages, ratio)` matches the spec'd
+ * two-argument signature and still finds the originals a previous `generateText`
+ * retained. Pass an explicit store per conversation when one process serves many.
+ */
+declare const sharedStore: RetentionStore;
+
+declare function compressText(text: string, opts: CompressOpts): CompressionResult;
+/** What the model should see for a result, whichever path produced it. */
+declare function renderResult(result: CompressionResult, source: string): string;
+
+export { type AutoExpandMode, type DetectOptions, type GenerateTextFn, type GenerateTextResultLike, MapRetentionStore, type ModelMessageLike, type ReferencedElision, type RetentionRecord, type RetentionStore, type TokzGenerateTextOptions, type TokzGenerateTextResult, type TokzOptions, type TokzReport, type ToolLike, type ToolResultOutput, type ToolResultPart, type ToolSetLike, type WrapContext, compactHistory, compressText, detectReferencedElisions, expand, expansionMessage, generateText, renderResult, sharedStore, stringifyToolResult, wrapTool, wrapTools };

package/dist/index.js

@@ -0,0 +1,367 @@
+// ../core/dist/index.js
+import { createHash } from "crypto";
+var enc = new TextEncoder();
+var dec = new TextDecoder("utf-8");
+function sha256hex(bytes) {
+  return createHash("sha256").update(bytes).digest("hex");
+}
+function toBytes(text) {
+  return enc.encode(text);
+}
+function fromBytes(bytes) {
+  return dec.decode(bytes);
+}
+function byteLength(text) {
+  return enc.encode(text).length;
+}
+function assemble(src, spans) {
+  let len = 0;
+  for (const sp of spans) len += sp.e - sp.s;
+  const out = new Uint8Array(len);
+  let o = 0;
+  for (const sp of spans) {
+    out.set(src.subarray(sp.s, sp.e), o);
+    o += sp.e - sp.s;
+  }
+  return dec.decode(out);
+}
+function elisionRange(spanMap, elision) {
+  const start = elision.after < 0 ? 0 : spanMap.spans[elision.after].e;
+  return { start, end: start + elision.bytes };
+}
+function expand(source, spanMap, opts = {}) {
+  const src = typeof source === "string" ? toBytes(source) : source;
+  if (sha256hex(src) !== spanMap.srcSha256) {
+    throw new Error(
+      "tokz: srcSha256 mismatch \u2014 this span map was not built from the given source (was the text edited?)"
+    );
+  }
+  if (opts.elision !== void 0) {
+    const el = spanMap.elisions[opts.elision];
+    if (!el) throw new Error(`tokz: no elision at index ${opts.elision}`);
+    const { start, end } = elisionRange(spanMap, el);
+    return fromBytes(src.subarray(start, end));
+  }
+  return fromBytes(src);
+}
+var DEFAULT_TARGET_RATIO = 0.45;
+var MAX_PAYLOAD_BYTES = 1024 * 1024;
+var LATENCY_BUDGET_PAYLOAD_BYTES = 64 * 1024;
+
+// src/engine.ts
+import * as engineModule from "@tokz/engine";
+var engine = engineModule;
+function compressText(text, opts) {
+  return normalize(engine.compress(text, opts));
+}
+function renderResult(result, source) {
+  if (typeof engine.render === "function") return engine.render(result, source);
+  if (result.method === "semantic") return result.compressedText;
+  return assemble(toBytes(source), result.spanMap.spans);
+}
+function normalize(raw) {
+  if ("method" in raw) return raw;
+  const { spanMap } = raw;
+  const untouched = spanMap.elisions.length === 0 && spanMap.stats.keptBytes === spanMap.srcBytes;
+  return untouched ? { method: "passthrough", spanMap } : { method: "structural", spanMap };
+}
+
+// src/store.ts
+var MapRetentionStore = class {
+  #byId = /* @__PURE__ */ new Map();
+  get(toolCallId) {
+    return this.#byId.get(toolCallId);
+  }
+  set(record) {
+    this.#byId.set(record.toolCallId, record);
+  }
+  records() {
+    return [...this.#byId.values()];
+  }
+  clear() {
+    this.#byId.clear();
+  }
+};
+var sharedStore = new MapRetentionStore();
+
+// src/wrap.ts
+function stringifyToolResult(value) {
+  if (typeof value === "string") return value;
+  return JSON.stringify(value ?? null, null, 2);
+}
+function wrapTool(toolName, tool, ctx) {
+  const inner = tool.execute;
+  if (typeof inner !== "function") return tool;
+  const wrapped = {
+    ...tool,
+    execute: async (input, options) => {
+      const raw = await inner.call(tool, input, options);
+      if (isAsyncIterable(raw)) return raw;
+      const originalText = stringifyToolResult(raw);
+      const result = compressText(originalText, { targetRatio: ctx.targetRatio });
+      const rendered = renderResult(result, originalText);
+      const toolCallId = options.toolCallId ?? contentAddressedId(toolName, originalText);
+      const record = {
+        toolName,
+        toolCallId,
+        originalText,
+        result,
+        rendered,
+        targetRatio: ctx.targetRatio
+      };
+      ctx.store.set(record);
+      return rendered;
+    },
+    toModelOutput: ({ toolCallId, output }) => {
+      const record = ctx.store.get(toolCallId);
+      return { type: "text", value: record ? record.rendered : stringifyToolResult(output) };
+    }
+  };
+  return wrapped;
+}
+function wrapTools(tools, ctx) {
+  const out = {};
+  for (const name of Object.keys(tools)) {
+    const tool = tools[name];
+    if (tool) out[name] = wrapTool(name, tool, ctx);
+  }
+  return out;
+}
+function resolveTargetRatio(ratio) {
+  return ratio ?? DEFAULT_TARGET_RATIO;
+}
+function isAsyncIterable(value) {
+  return typeof value === "object" && value !== null && typeof value[Symbol.asyncIterator] === "function";
+}
+function contentAddressedId(toolName, originalText) {
+  return `tokz-${sha256hex(toBytes(`${toolName}\0${originalText}`)).slice(0, 16)}`;
+}
+
+// src/compact.ts
+function compactHistory(messages, targetRatio, store = sharedStore) {
+  return messages.map((message) => {
+    if (!Array.isArray(message.content)) return message;
+    let changed = false;
+    const content = message.content.map((part) => {
+      if (!isToolResultPart(part)) return part;
+      const record = store.get(part.toolCallId);
+      if (!record) return part;
+      const tighter = compressText(record.originalText, { targetRatio });
+      const rendered = renderResult(tighter, record.originalText);
+      if (rendered === currentValue(part)) return part;
+      store.set({ ...record, result: tighter, rendered, targetRatio });
+      changed = true;
+      return part.output !== void 0 ? { ...part, output: { type: "text", value: rendered } } : { ...part, result: rendered };
+    });
+    return changed ? { ...message, content } : message;
+  });
+}
+function currentValue(part) {
+  if (part.output && "value" in part.output && typeof part.output.value === "string") {
+    return part.output.value;
+  }
+  if (part.result !== void 0) return stringifyToolResult(part.result);
+  return void 0;
+}
+function isToolResultPart(part) {
+  if (typeof part !== "object" || part === null) return false;
+  const p = part;
+  return p.type === "tool-result" && typeof p.toolCallId === "string";
+}
+
+// src/autoexpand.ts
+function detectReferencedElisions(responseText, store = sharedStore, opts = {}) {
+  const minTermLength = opts.minTermLength ?? 4;
+  const maxTerms = opts.maxTermsPerElision ?? 256;
+  const haystack = responseText.toLowerCase();
+  const hits = [];
+  for (const record of store.records()) {
+    if (record.result.method === "semantic") continue;
+    const spanMap = record.result.spanMap;
+    for (let i = 0; i < spanMap.elisions.length; i++) {
+      const elision = spanMap.elisions[i];
+      if (!elision) continue;
+      const text = expand(record.originalText, spanMap, { elision: i });
+      const matchedTerms = candidateTerms(text, minTermLength, maxTerms).filter(
+        (term) => containsWord(haystack, term)
+      );
+      if (matchedTerms.length === 0) continue;
+      hits.push({
+        toolCallId: record.toolCallId,
+        toolName: record.toolName,
+        elisionIndex: i,
+        matchedTerms,
+        text,
+        ...elision.note === void 0 ? {} : { note: elision.note }
+      });
+    }
+  }
+  return hits;
+}
+function expansionMessage(expansions) {
+  const blocks = expansions.map(
+    (e) => `Recovered from the elided portion of tool result "${e.toolName}"${e.note ? ` (${e.note})` : ""}:
+${e.text}`
+  );
+  return { role: "user", content: blocks.join("\n\n") };
+}
+var STOPWORDS = /* @__PURE__ */ new Set([
+  "true",
+  "false",
+  "null",
+  "http",
+  "https",
+  "this",
+  "that",
+  "with",
+  "from",
+  "have",
+  "been",
+  "were",
+  "they",
+  "there",
+  "then",
+  "than",
+  "your",
+  "when",
+  "what",
+  "which",
+  "would",
+  "could",
+  "should",
+  "about",
+  "into",
+  "more",
+  "some",
+  "only",
+  "also",
+  "each",
+  "over",
+  "just",
+  "like",
+  "them",
+  "these",
+  "those"
+]);
+var JSON_KEY = /"([^"\\]+)"\s*:/g;
+var WORD = /[A-Za-z][A-Za-z0-9_.-]*[A-Za-z0-9]/g;
+function candidateTerms(text, minLength, cap) {
+  const terms = [];
+  const seen = /* @__PURE__ */ new Set();
+  const add = (term) => {
+    if (term.length < minLength || STOPWORDS.has(term) || seen.has(term)) return;
+    seen.add(term);
+    terms.push(term);
+  };
+  const push = (raw) => {
+    add(raw.toLowerCase());
+    const spaced = raw.replace(/([a-z0-9])([A-Z])/g, "$1 $2").replace(/[_-]+/g, " ").toLowerCase();
+    if (spaced !== raw.toLowerCase()) add(spaced);
+  };
+  for (const m of text.matchAll(JSON_KEY)) {
+    if (terms.length >= cap) return terms;
+    if (m[1]) push(m[1]);
+  }
+  for (const m of text.matchAll(WORD)) {
+    if (terms.length >= cap) return terms;
+    push(m[0]);
+  }
+  return terms;
+}
+function containsWord(haystack, term) {
+  let from = 0;
+  for (; ; ) {
+    const at = haystack.indexOf(term, from);
+    if (at === -1) return false;
+    const before = at === 0 ? "" : haystack[at - 1];
+    const after = haystack[at + term.length] ?? "";
+    if (!isWordChar(before) && !isWordChar(after)) return true;
+    from = at + 1;
+  }
+}
+function isWordChar(ch) {
+  return ch !== "" && /[A-Za-z0-9_]/.test(ch);
+}
+
+// src/generate.ts
+async function generateText(options) {
+  const { tokz = {}, ...rest } = options;
+  const store = tokz.store ?? sharedStore;
+  const targetRatio = resolveTargetRatio(tokz.targetRatio);
+  const delegate = tokz.delegate ?? await loadDelegate();
+  const forwarded = { ...rest };
+  if (isToolSet(rest["tools"])) {
+    forwarded["tools"] = wrapTools(rest["tools"], { store, targetRatio });
+  }
+  if (tokz.conversationCompaction && Array.isArray(rest["messages"])) {
+    const ratio = tokz.compactionRatio ?? targetRatio * 0.6;
+    forwarded["messages"] = compactHistory(rest["messages"], ratio, store);
+  }
+  const before = snapshotBytes(store);
+  let result = await delegate(forwarded);
+  const usage = deltaBytes(store, before);
+  let expansions = [];
+  let reprompted = false;
+  if (tokz.autoExpand === "on-reference" && typeof result.text === "string") {
+    expansions = detectReferencedElisions(result.text, store);
+    if (expansions.length > 0 && Array.isArray(forwarded["messages"])) {
+      const messages = [
+        ...forwarded["messages"],
+        { role: "assistant", content: result.text },
+        expansionMessage(expansions)
+      ];
+      result = await delegate({ ...forwarded, messages });
+      reprompted = true;
+    }
+  }
+  return {
+    ...result,
+    tokz: {
+      store,
+      targetRatio,
+      originalBytes: usage.original,
+      renderedBytes: usage.rendered,
+      expansions,
+      reprompted
+    }
+  };
+}
+async function loadDelegate() {
+  const specifier = "ai";
+  const mod = await import(specifier);
+  if (typeof mod.generateText !== "function") {
+    throw new Error("tokz: the `ai` package is required unless `tokz.delegate` is supplied");
+  }
+  return mod.generateText;
+}
+function isToolSet(value) {
+  return typeof value === "object" && value !== null && !Array.isArray(value);
+}
+function snapshotBytes(store) {
+  return new Set(store.records().map((r) => r.toolCallId));
+}
+function deltaBytes(store, before) {
+  let original = 0;
+  let rendered = 0;
+  for (const record of store.records()) {
+    if (before.has(record.toolCallId)) continue;
+    original += byteLength(record.originalText);
+    rendered += byteLength(record.rendered);
+  }
+  return { original, rendered };
+}
+export {
+  MapRetentionStore,
+  compactHistory,
+  compressText,
+  detectReferencedElisions,
+  expand,
+  expansionMessage,
+  generateText,
+  renderResult,
+  sharedStore,
+  stringifyToolResult,
+  wrapTool,
+  wrapTools
+};
+//# sourceMappingURL=index.js.map

package/dist/index.js.map

@@ -0,0 +1 @@
+{"version":3,"sources":["../../core/src/bytes.ts","../../core/src/expand.ts","../../core/src/constants.ts","../src/engine.ts","../src/store.ts","../src/wrap.ts","../src/compact.ts","../src/autoexpand.ts","../src/generate.ts"],"sourcesContent":["import { createHash } from \"node:crypto\";\nimport type { Span, SpanMap, Elision } from \"./types.js\";\n\nconst enc = new TextEncoder();\nconst dec = new TextDecoder(\"utf-8\");\n\nexport function sha256hex(bytes: Uint8Array): string {\n  return createHash(\"sha256\").update(bytes).digest(\"hex\");\n}\n\nexport function toBytes(text: string): Uint8Array {\n  return enc.encode(text);\n}\n\nexport function fromBytes(bytes: Uint8Array): string {\n  return dec.decode(bytes);\n}\n\nexport function byteLength(text: string): number {\n  return enc.encode(text).length;\n}\n\nexport function isAscii(text: string): boolean {\n  for (let i = 0; i < text.length; i++) {\n    if (text.charCodeAt(i) > 127) return false;\n  }\n  return true;\n}\n\n/**\n * Concatenate the source bytes covered by `spans` and decode as UTF-8. This is\n * exactly the compressed text: only surviving bytes, in source order, verbatim.\n */\nexport function assemble(src: Uint8Array, spans: Span[]): string {\n  let len = 0;\n  for (const sp of spans) len += sp.e - sp.s;\n  const out = new Uint8Array(len);\n  let o = 0;\n  for (const sp of spans) {\n    out.set(src.subarray(sp.s, sp.e), o);\n    o += sp.e - sp.s;\n  }\n  return dec.decode(out);\n}\n\n/** Byte range in the source that an elision dropped. */\nexport function elisionRange(spanMap: SpanMap, elision: Elision): { start: number; end: number } {\n  const start = elision.after < 0 ? 0 : spanMap.spans[elision.after]!.e;\n  return { start, end: start + elision.bytes };\n}\n\n/**\n * jsonc-parser and most tokenizers report offsets in UTF-16 code units; the span\n * map speaks UTF-8 bytes. Build a converter from string index to byte offset.\n *\n * For ASCII — which most tool output is — the two are the same number, so the\n * prefix table is skipped entirely rather than allocating 4 bytes per character\n * to answer `i => i`.\n */\nexport function byteMapper(text: string, ascii?: boolean): (strIdx: number) => number {\n  if (ascii ?? isAscii(text)) {\n    const len = text.length;\n    return (strIdx: number) => Math.max(0, Math.min(strIdx, len));\n  }\n\n  const prefix = new Int32Array(text.length + 1);\n  let bytes = 0;\n  let i = 0;\n  const buf = new Uint8Array(4);\n  for (const cp of text) {\n    const units = cp.length; // 1 or 2 UTF-16 units\n    const n = enc.encodeInto(cp, buf).written ?? 0;\n    // Every UTF-16 index inside a code point maps to the byte offset at its start.\n    for (let u = 0; u < units; u++) {\n      prefix[i] = bytes;\n      i++;\n    }\n    bytes += n;\n  }\n  prefix[i] = bytes;\n  return (strIdx: number) => prefix[Math.max(0, Math.min(strIdx, text.length))]!;\n}\n","import type { SpanMap, ExpandOpts } from \"./types.js\";\nimport { sha256hex, toBytes, fromBytes, elisionRange } from \"./bytes.js\";\n\n/**\n * Local, network-free reconstruction from the caller's own source plus a span\n * map. This is what lets a stateless API offer recovery at all: we hold nothing,\n * because the caller already has everything.\n *\n * - no opts       -> the full original text, after verifying `srcSha256`.\n * - {elision: i}  -> just the bytes elision `i` dropped, recovered verbatim.\n *\n * Throws if the map was not built from this source, rather than returning\n * plausible-looking misaligned output.\n */\nexport function expand(source: string | Uint8Array, spanMap: SpanMap, opts: ExpandOpts = {}): string {\n  const src = typeof source === \"string\" ? toBytes(source) : source;\n  if (sha256hex(src) !== spanMap.srcSha256) {\n    throw new Error(\n      \"tokz: srcSha256 mismatch — this span map was not built from the given source (was the text edited?)\",\n    );\n  }\n  if (opts.elision !== undefined) {\n    const el = spanMap.elisions[opts.elision];\n    if (!el) throw new Error(`tokz: no elision at index ${opts.elision}`);\n    const { start, end } = elisionRange(spanMap, el);\n    return fromBytes(src.subarray(start, end));\n  }\n  return fromBytes(src);\n}\n","/** Default share of source bytes to keep when the caller does not say. */\nexport const DEFAULT_TARGET_RATIO = 0.45;\n\n/** Max request body the hosted API accepts. Compression cost is linear in input\n *  size, so this is where predictable latency stops, not an arbitrary number. */\nexport const MAX_PAYLOAD_BYTES = 1024 * 1024;\n\n/**\n * Nesting past this is declined rather than compressed. Parsers and the\n * selection walk both recurse per level, so a deeply nested document overflows\n * the stack — and `\"[\".repeat(6000)` reaches that in ~12 KB, well inside any\n * request body limit. Declining costs the caller nothing; throwing would hand a\n * remote caller a way to kill a request slot.\n */\nexport const MAX_NESTING_DEPTH = 200;\n\n/** Latency budgets from the design doc, used by the eval kill gates. */\nexport const STRUCTURAL_BUDGET_MS = 10;\nexport const SEMANTIC_BUDGET_MS = 30;\n\n/** Payload size the structural latency budget is quoted against. Cost is linear\n *  above this, so a latency number without a size attached means little. */\nexport const LATENCY_BUDGET_PAYLOAD_BYTES = 64 * 1024;\n\n/** API key format: `tokz_live_<id8>_<secret>`. The id is public and indexed; only\n *  a SHA-256 of the secret is stored. */\nexport const KEY_PREFIX = \"tokz_live_\";\n","import * as engineModule from \"@tokz/engine\";\nimport { assemble, toBytes, type CompressionResult, type CompressOpts, type SpanMap } from \"@tokz/core\";\n\n/**\n * `@tokz/engine` is mid-migration from `compress() -> { text, spanMap }` to the\n * unified `CompressionResult` union, and `render` lands with it. Binding through a\n * declared surface with one cast keeps this package compiling and running against\n * either shape instead of pinning it to whichever version happens to be built in\n * `dist/` at the moment.\n */\ntype EngineSurface = {\n  compress: (text: string, opts?: CompressOpts) => CompressionResult | LegacyCompressResult;\n  render?: (result: CompressionResult, source: string) => string;\n};\n\ntype LegacyCompressResult = { text: string; spanMap: SpanMap };\n\nconst engine = engineModule as unknown as EngineSurface;\n\nexport function compressText(text: string, opts: CompressOpts): CompressionResult {\n  return normalize(engine.compress(text, opts));\n}\n\n/** What the model should see for a result, whichever path produced it. */\nexport function renderResult(result: CompressionResult, source: string): string {\n  if (typeof engine.render === \"function\") return engine.render(result, source);\n  if (result.method === \"semantic\") return result.compressedText;\n  return assemble(toBytes(source), result.spanMap.spans);\n}\n\nfunction normalize(raw: CompressionResult | LegacyCompressResult): CompressionResult {\n  if (\"method\" in raw) return raw;\n  const { spanMap } = raw;\n  // A map that kept everything and dropped nothing is the honest passthrough case,\n  // which the union spells out and the legacy shape did not.\n  const untouched = spanMap.elisions.length === 0 && spanMap.stats.keptBytes === spanMap.srcBytes;\n  return untouched ? { method: \"passthrough\", spanMap } : { method: \"structural\", spanMap };\n}\n","import type { RetentionRecord, RetentionStore } from \"./types.js\";\n\n/** Insertion-ordered so `records()` reads back in call order — history compaction\n *  and any future oldest-first budget both depend on that order being stable. */\nexport class MapRetentionStore implements RetentionStore {\n  readonly #byId = new Map<string, RetentionRecord>();\n\n  get(toolCallId: string): RetentionRecord | undefined {\n    return this.#byId.get(toolCallId);\n  }\n\n  set(record: RetentionRecord): void {\n    this.#byId.set(record.toolCallId, record);\n  }\n\n  records(): RetentionRecord[] {\n    return [...this.#byId.values()];\n  }\n\n  clear(): void {\n    this.#byId.clear();\n  }\n}\n\n/**\n * Process-wide default so `compactHistory(messages, ratio)` matches the spec'd\n * two-argument signature and still finds the originals a previous `generateText`\n * retained. Pass an explicit store per conversation when one process serves many.\n */\nexport const sharedStore: RetentionStore = new MapRetentionStore();\n","import { DEFAULT_TARGET_RATIO, sha256hex, toBytes } from \"@tokz/core\";\nimport { compressText, renderResult } from \"./engine.js\";\nimport type {\n  RetentionRecord,\n  RetentionStore,\n  ToolExecuteOptions,\n  ToolLike,\n  ToolSetLike,\n} from \"./types.js\";\n\nexport type WrapContext = {\n  store: RetentionStore;\n  targetRatio: number;\n};\n\n/** Tool output as text the compressor can work on. Pretty-printed because the\n *  structural detector reads JSON shape, and key order follows the object, so the\n *  same result always serialises to the same bytes. */\nexport function stringifyToolResult(value: unknown): string {\n  if (typeof value === \"string\") return value;\n  return JSON.stringify(value ?? null, null, 2);\n}\n\n/**\n * Wrap one tool so the model receives the compressed render while the original text\n * and its span map stay in the store.\n *\n * `execute` returns the render rather than the raw value, and `toModelOutput` maps\n * to the same string. Doing both is deliberate: `toModelOutput` is the documented\n * tool-to-model hook, but returning the render from `execute` means the original\n * cannot reach a message even under an SDK version that ignores the hook. The\n * original is one `store.get(toolCallId)` away for the caller.\n */\nexport function wrapTool(toolName: string, tool: ToolLike, ctx: WrapContext): ToolLike {\n  const inner = tool.execute;\n  // Client-side and provider-executed tools have no server-side execute to intercept.\n  if (typeof inner !== \"function\") return tool;\n\n  const wrapped: ToolLike = {\n    ...tool,\n    execute: async (input: never, options: ToolExecuteOptions): Promise<unknown> => {\n      const raw = await inner.call(tool, input, options);\n      // A tool may return an AsyncIterable to stream its output. Compressing that\n      // would mean buffering it, which changes the tool's semantics, so it passes\n      // through uncompressed rather than silently becoming \"{}\".\n      if (isAsyncIterable(raw)) return raw;\n      const originalText = stringifyToolResult(raw);\n      const result = compressText(originalText, { targetRatio: ctx.targetRatio });\n      const rendered = renderResult(result, originalText);\n      const toolCallId = options.toolCallId ?? contentAddressedId(toolName, originalText);\n      const record: RetentionRecord = {\n        toolName,\n        toolCallId,\n        originalText,\n        result,\n        rendered,\n        targetRatio: ctx.targetRatio,\n      };\n      ctx.store.set(record);\n      return rendered;\n    },\n    toModelOutput: ({ toolCallId, output }) => {\n      const record = ctx.store.get(toolCallId);\n      return { type: \"text\", value: record ? record.rendered : stringifyToolResult(output) };\n    },\n  };\n  return wrapped;\n}\n\nexport function wrapTools(tools: ToolSetLike, ctx: WrapContext): ToolSetLike {\n  const out: ToolSetLike = {};\n  for (const name of Object.keys(tools)) {\n    const tool = tools[name];\n    if (tool) out[name] = wrapTool(name, tool, ctx);\n  }\n  return out;\n}\n\nexport function resolveTargetRatio(ratio: number | undefined): number {\n  return ratio ?? DEFAULT_TARGET_RATIO;\n}\n\nfunction isAsyncIterable(value: unknown): value is AsyncIterable<unknown> {\n  return (\n    typeof value === \"object\" &&\n    value !== null &&\n    typeof (value as AsyncIterable<unknown>)[Symbol.asyncIterator] === \"function\"\n  );\n}\n\n/** Content-addressed, not counter- or clock-based: two runs of the same tool call\n *  must key the same record, or a customer's prompt cache stops hitting. */\nfunction contentAddressedId(toolName: string, originalText: string): string {\n  return `tokz-${sha256hex(toBytes(`${toolName}\u0000${originalText}`)).slice(0, 16)}`;\n}\n","import { compressText, renderResult } from \"./engine.js\";\nimport { sharedStore } from \"./store.js\";\nimport { stringifyToolResult } from \"./wrap.js\";\nimport type { ModelMessageLike, RetentionStore, ToolResultPart } from \"./types.js\";\n\n/**\n * Re-compress retained tool results at a tighter budget.\n *\n * This is a pure local function: the original text and its span map are already in\n * the store, so producing a smaller view of an old tool result costs zero API calls\n * and zero inference — it is the same deterministic CPU pass that produced the first\n * view. That property is the whole point of retaining the pair; a wrapper that threw\n * the original away would have to re-ask a model to shrink history, which costs more\n * than the tokens it saves.\n *\n * Input messages are not mutated; changed messages are returned as new objects.\n */\nexport function compactHistory(\n  messages: readonly ModelMessageLike[],\n  targetRatio: number,\n  store: RetentionStore = sharedStore,\n): ModelMessageLike[] {\n  return messages.map((message) => {\n    if (!Array.isArray(message.content)) return message;\n\n    let changed = false;\n    const content = (message.content as unknown[]).map((part) => {\n      if (!isToolResultPart(part)) return part;\n      const record = store.get(part.toolCallId);\n      if (!record) return part;\n\n      const tighter = compressText(record.originalText, { targetRatio });\n      const rendered = renderResult(tighter, record.originalText);\n      if (rendered === currentValue(part)) return part;\n\n      store.set({ ...record, result: tighter, rendered, targetRatio });\n      changed = true;\n      return part.output !== undefined\n        ? { ...part, output: { type: \"text\" as const, value: rendered } }\n        : { ...part, result: rendered };\n    });\n\n    return changed ? { ...message, content } : message;\n  });\n}\n\nfunction currentValue(part: ToolResultPart): string | undefined {\n  if (part.output && \"value\" in part.output && typeof part.output.value === \"string\") {\n    return part.output.value;\n  }\n  if (part.result !== undefined) return stringifyToolResult(part.result);\n  return undefined;\n}\n\nfunction isToolResultPart(part: unknown): part is ToolResultPart {\n  if (typeof part !== \"object\" || part === null) return false;\n  const p = part as { type?: unknown; toolCallId?: unknown };\n  return p.type === \"tool-result\" && typeof p.toolCallId === \"string\";\n}\n","import { expand } from \"@tokz/core\";\nimport { sharedStore } from \"./store.js\";\nimport type { ModelMessageLike, RetentionStore } from \"./types.js\";\n\nexport type ReferencedElision = {\n  toolCallId: string;\n  toolName: string;\n  elisionIndex: number;\n  /** Terms from the elided bytes that appear in the model's response. */\n  matchedTerms: string[];\n  /** The elided bytes, recovered verbatim from the caller's own text. */\n  text: string;\n  note?: string;\n};\n\nexport type DetectOptions = {\n  /** Shortest token considered a reference. Below 4 characters the match rate is\n   *  dominated by English words that happen to appear in JSON. */\n  minTermLength?: number;\n  /** Cap on terms sampled per elision, to bound work on multi-megabyte drops. */\n  maxTermsPerElision?: number;\n};\n\n/**\n * HEURISTIC. There is no reliable signal that a model \"meant\" an elided field, so\n * this matches vocabulary: recover each elided run locally, take the identifiers it\n * contains, and flag the elision when the response mentions one of them. It over-\n * fires on generic words (mitigated by a stopword list and a length floor) and\n * under-fires when the model paraphrases without naming anything. Treat a hit as a\n * cheap prompt to re-supply context, never as proof of a reference.\n *\n * Recovery itself is exact: `expand` verifies `srcSha256` and returns the dropped\n * bytes verbatim, with no network call.\n */\nexport function detectReferencedElisions(\n  responseText: string,\n  store: RetentionStore = sharedStore,\n  opts: DetectOptions = {},\n): ReferencedElision[] {\n  const minTermLength = opts.minTermLength ?? 4;\n  const maxTerms = opts.maxTermsPerElision ?? 256;\n  const haystack = responseText.toLowerCase();\n  const hits: ReferencedElision[] = [];\n\n  for (const record of store.records()) {\n    // Semantic results carry a provenance map, not recoverable byte ranges.\n    if (record.result.method === \"semantic\") continue;\n    const spanMap = record.result.spanMap;\n\n    for (let i = 0; i < spanMap.elisions.length; i++) {\n      const elision = spanMap.elisions[i];\n      if (!elision) continue;\n      const text = expand(record.originalText, spanMap, { elision: i });\n      const matchedTerms = candidateTerms(text, minTermLength, maxTerms).filter((term) =>\n        containsWord(haystack, term),\n      );\n      if (matchedTerms.length === 0) continue;\n      hits.push({\n        toolCallId: record.toolCallId,\n        toolName: record.toolName,\n        elisionIndex: i,\n        matchedTerms,\n        text,\n        ...(elision.note === undefined ? {} : { note: elision.note }),\n      });\n    }\n  }\n  return hits;\n}\n\n/** A user-role message re-supplying the recovered bytes, ready to append before a\n *  follow-up turn. Separate from detection so callers can inspect first. */\nexport function expansionMessage(expansions: readonly ReferencedElision[]): ModelMessageLike {\n  const blocks = expansions.map(\n    (e) =>\n      `Recovered from the elided portion of tool result \"${e.toolName}\"` +\n      `${e.note ? ` (${e.note})` : \"\"}:\\n${e.text}`,\n  );\n  return { role: \"user\", content: blocks.join(\"\\n\\n\") };\n}\n\nconst STOPWORDS = new Set([\n  \"true\",\n  \"false\",\n  \"null\",\n  \"http\",\n  \"https\",\n  \"this\",\n  \"that\",\n  \"with\",\n  \"from\",\n  \"have\",\n  \"been\",\n  \"were\",\n  \"they\",\n  \"there\",\n  \"then\",\n  \"than\",\n  \"your\",\n  \"when\",\n  \"what\",\n  \"which\",\n  \"would\",\n  \"could\",\n  \"should\",\n  \"about\",\n  \"into\",\n  \"more\",\n  \"some\",\n  \"only\",\n  \"also\",\n  \"each\",\n  \"over\",\n  \"just\",\n  \"like\",\n  \"them\",\n  \"these\",\n  \"those\",\n]);\n\nconst JSON_KEY = /\"([^\"\\\\]+)\"\\s*:/g;\n/** Hyphens and dots are inside the token, not between tokens: the identifiers worth\n *  matching are `api-5`, `node-2`, `registry.example.com`, not their fragments. */\nconst WORD = /[A-Za-z][A-Za-z0-9_.-]*[A-Za-z0-9]/g;\n\nfunction candidateTerms(text: string, minLength: number, cap: number): string[] {\n  const terms: string[] = [];\n  const seen = new Set<string>();\n  const add = (term: string): void => {\n    if (term.length < minLength || STOPWORDS.has(term) || seen.has(term)) return;\n    seen.add(term);\n    terms.push(term);\n  };\n  const push = (raw: string): void => {\n    add(raw.toLowerCase());\n    // A model asked about `restartCount` usually writes \"restart count\", so the\n    // separated spelling is a term in its own right.\n    const spaced = raw.replace(/([a-z0-9])([A-Z])/g, \"$1 $2\").replace(/[_-]+/g, \" \").toLowerCase();\n    if (spaced !== raw.toLowerCase()) add(spaced);\n  };\n\n  // Keys first: a model asking about elided data almost always names the field.\n  for (const m of text.matchAll(JSON_KEY)) {\n    if (terms.length >= cap) return terms;\n    if (m[1]) push(m[1]);\n  }\n  for (const m of text.matchAll(WORD)) {\n    if (terms.length >= cap) return terms;\n    push(m[0]);\n  }\n  return terms;\n}\n\nfunction containsWord(haystack: string, term: string): boolean {\n  let from = 0;\n  for (;;) {\n    const at = haystack.indexOf(term, from);\n    if (at === -1) return false;\n    const before = at === 0 ? \"\" : haystack[at - 1]!;\n    const after = haystack[at + term.length] ?? \"\";\n    if (!isWordChar(before) && !isWordChar(after)) return true;\n    from = at + 1;\n  }\n}\n\nfunction isWordChar(ch: string): boolean {\n  return ch !== \"\" && /[A-Za-z0-9_]/.test(ch);\n}\n","import { byteLength } from \"@tokz/core\";\nimport { compactHistory } from \"./compact.js\";\nimport { detectReferencedElisions, expansionMessage, type ReferencedElision } from \"./autoexpand.js\";\nimport { sharedStore } from \"./store.js\";\nimport { resolveTargetRatio, wrapTools } from \"./wrap.js\";\nimport type {\n  GenerateTextFn,\n  GenerateTextResultLike,\n  ModelMessageLike,\n  RetentionStore,\n  TokzOptions,\n  ToolSetLike,\n} from \"./types.js\";\n\nexport type TokzGenerateTextOptions = Record<string, unknown> & { tokz?: TokzOptions };\n\nexport type TokzReport = {\n  store: RetentionStore;\n  targetRatio: number;\n  /** Bytes of tool output before and after compression, this call only. */\n  originalBytes: number;\n  renderedBytes: number;\n  expansions: ReferencedElision[];\n  /** Whether a second delegate call was made to re-supply expanded content. */\n  reprompted: boolean;\n};\n\nexport type TokzGenerateTextResult = GenerateTextResultLike & { tokz: TokzReport };\n\n/**\n * Drop-in replacement for `ai.generateText`. Every option is forwarded untouched\n * except `tools`, which is wrapped so results reach the model compressed, and\n * `messages`, which is optionally compacted first.\n */\nexport async function generateText(\n  options: TokzGenerateTextOptions,\n): Promise<TokzGenerateTextResult> {\n  const { tokz = {}, ...rest } = options;\n  const store = tokz.store ?? sharedStore;\n  const targetRatio = resolveTargetRatio(tokz.targetRatio);\n  const delegate = tokz.delegate ?? (await loadDelegate());\n\n  const forwarded: Record<string, unknown> = { ...rest };\n\n  if (isToolSet(rest[\"tools\"])) {\n    forwarded[\"tools\"] = wrapTools(rest[\"tools\"], { store, targetRatio });\n  }\n\n  if (tokz.conversationCompaction && Array.isArray(rest[\"messages\"])) {\n    const ratio = tokz.compactionRatio ?? targetRatio * 0.6;\n    forwarded[\"messages\"] = compactHistory(rest[\"messages\"] as ModelMessageLike[], ratio, store);\n  }\n\n  const before = snapshotBytes(store);\n  let result = await delegate(forwarded);\n  const usage = deltaBytes(store, before);\n\n  let expansions: ReferencedElision[] = [];\n  let reprompted = false;\n  if (tokz.autoExpand === \"on-reference\" && typeof result.text === \"string\") {\n    expansions = detectReferencedElisions(result.text, store);\n    if (expansions.length > 0 && Array.isArray(forwarded[\"messages\"])) {\n      // One re-prompt only. The expansion is local and free, but the extra turn is\n      // not, and a loop that re-expands on its own answer never terminates.\n      const messages = [\n        ...(forwarded[\"messages\"] as ModelMessageLike[]),\n        { role: \"assistant\", content: result.text },\n        expansionMessage(expansions),\n      ];\n      result = await delegate({ ...forwarded, messages });\n      reprompted = true;\n    }\n  }\n\n  return {\n    ...result,\n    tokz: {\n      store,\n      targetRatio,\n      originalBytes: usage.original,\n      renderedBytes: usage.rendered,\n      expansions,\n      reprompted,\n    },\n  };\n}\n\n/**\n * `ai` is an optional peer, so the specifier is held in a variable: a literal would\n * make TypeScript resolve the module at build time and fail in a workspace that has\n * not installed it, which is exactly the install this package must still compile in.\n */\nasync function loadDelegate(): Promise<GenerateTextFn> {\n  const specifier = \"ai\";\n  const mod = (await import(specifier)) as { generateText?: GenerateTextFn };\n  if (typeof mod.generateText !== \"function\") {\n    throw new Error(\"tokz: the `ai` package is required unless `tokz.delegate` is supplied\");\n  }\n  return mod.generateText;\n}\n\nfunction isToolSet(value: unknown): value is ToolSetLike {\n  return typeof value === \"object\" && value !== null && !Array.isArray(value);\n}\n\nfunction snapshotBytes(store: RetentionStore): Set<string> {\n  return new Set(store.records().map((r) => r.toolCallId));\n}\n\nfunction deltaBytes(\n  store: RetentionStore,\n  before: Set<string>,\n): { original: number; rendered: number } {\n  let original = 0;\n  let rendered = 0;\n  for (const record of store.records()) {\n    if (before.has(record.toolCallId)) continue;\n    original += byteLength(record.originalText);\n    rendered += byteLength(record.rendered);\n  }\n  return { original, rendered };\n}\n"],"mappings":";AAAA,SAAS,kBAAkB;AAG3B,IAAM,MAAM,IAAI,YAAY;AAC5B,IAAM,MAAM,IAAI,YAAY,OAAO;AAE5B,SAAS,UAAU,OAA2B;AACnD,SAAO,WAAW,QAAQ,EAAE,OAAO,KAAK,EAAE,OAAO,KAAK;AACxD;AAEO,SAAS,QAAQ,MAA0B;AAChD,SAAO,IAAI,OAAO,IAAI;AACxB;AAEO,SAAS,UAAU,OAA2B;AACnD,SAAO,IAAI,OAAO,KAAK;AACzB;AAEO,SAAS,WAAW,MAAsB;AAC/C,SAAO,IAAI,OAAO,IAAI,EAAE;AAC1B;AAaO,SAAS,SAAS,KAAiB,OAAuB;AAC/D,MAAI,MAAM;AACV,aAAW,MAAM,MAAO,QAAO,GAAG,IAAI,GAAG;AACzC,QAAM,MAAM,IAAI,WAAW,GAAG;AAC9B,MAAI,IAAI;AACR,aAAW,MAAM,OAAO;AACtB,QAAI,IAAI,IAAI,SAAS,GAAG,GAAG,GAAG,CAAC,GAAG,CAAC;AACnC,SAAK,GAAG,IAAI,GAAG;EACjB;AACA,SAAO,IAAI,OAAO,GAAG;AACvB;AAGO,SAAS,aAAa,SAAkB,SAAkD;AAC/F,QAAM,QAAQ,QAAQ,QAAQ,IAAI,IAAI,QAAQ,MAAM,QAAQ,KAAK,EAAG;AACpE,SAAO,EAAE,OAAO,KAAK,QAAQ,QAAQ,MAAM;AAC7C;ACnCO,SAAS,OAAO,QAA6B,SAAkB,OAAmB,CAAC,GAAW;AACnG,QAAM,MAAM,OAAO,WAAW,WAAW,QAAQ,MAAM,IAAI;AAC3D,MAAI,UAAU,GAAG,MAAM,QAAQ,WAAW;AACxC,UAAM,IAAI;MACR;IACF;EACF;AACA,MAAI,KAAK,YAAY,QAAW;AAC9B,UAAM,KAAK,QAAQ,SAAS,KAAK,OAAO;AACxC,QAAI,CAAC,GAAI,OAAM,IAAI,MAAM,6BAA6B,KAAK,OAAO,EAAE;AACpE,UAAM,EAAE,OAAO,IAAI,IAAI,aAAa,SAAS,EAAE;AAC/C,WAAO,UAAU,IAAI,SAAS,OAAO,GAAG,CAAC;EAC3C;AACA,SAAO,UAAU,GAAG;AACtB;AC3BO,IAAM,uBAAuB;AAI7B,IAAM,oBAAoB,OAAO;AAiBjC,IAAM,+BAA+B,KAAK;;;ACtBjD,YAAY,kBAAkB;AAiB9B,IAAM,SAAS;AAER,SAAS,aAAa,MAAc,MAAuC;AAChF,SAAO,UAAU,OAAO,SAAS,MAAM,IAAI,CAAC;AAC9C;AAGO,SAAS,aAAa,QAA2B,QAAwB;AAC9E,MAAI,OAAO,OAAO,WAAW,WAAY,QAAO,OAAO,OAAO,QAAQ,MAAM;AAC5E,MAAI,OAAO,WAAW,WAAY,QAAO,OAAO;AAChD,SAAO,SAAS,QAAQ,MAAM,GAAG,OAAO,QAAQ,KAAK;AACvD;AAEA,SAAS,UAAU,KAAkE;AACnF,MAAI,YAAY,IAAK,QAAO;AAC5B,QAAM,EAAE,QAAQ,IAAI;AAGpB,QAAM,YAAY,QAAQ,SAAS,WAAW,KAAK,QAAQ,MAAM,cAAc,QAAQ;AACvF,SAAO,YAAY,EAAE,QAAQ,eAAe,QAAQ,IAAI,EAAE,QAAQ,cAAc,QAAQ;AAC1F;;;ACjCO,IAAM,oBAAN,MAAkD;AAAA,EAC9C,QAAQ,oBAAI,IAA6B;AAAA,EAElD,IAAI,YAAiD;AACnD,WAAO,KAAK,MAAM,IAAI,UAAU;AAAA,EAClC;AAAA,EAEA,IAAI,QAA+B;AACjC,SAAK,MAAM,IAAI,OAAO,YAAY,MAAM;AAAA,EAC1C;AAAA,EAEA,UAA6B;AAC3B,WAAO,CAAC,GAAG,KAAK,MAAM,OAAO,CAAC;AAAA,EAChC;AAAA,EAEA,QAAc;AACZ,SAAK,MAAM,MAAM;AAAA,EACnB;AACF;AAOO,IAAM,cAA8B,IAAI,kBAAkB;;;ACX1D,SAAS,oBAAoB,OAAwB;AAC1D,MAAI,OAAO,UAAU,SAAU,QAAO;AACtC,SAAO,KAAK,UAAU,SAAS,MAAM,MAAM,CAAC;AAC9C;AAYO,SAAS,SAAS,UAAkB,MAAgB,KAA4B;AACrF,QAAM,QAAQ,KAAK;AAEnB,MAAI,OAAO,UAAU,WAAY,QAAO;AAExC,QAAM,UAAoB;AAAA,IACxB,GAAG;AAAA,IACH,SAAS,OAAO,OAAc,YAAkD;AAC9E,YAAM,MAAM,MAAM,MAAM,KAAK,MAAM,OAAO,OAAO;AAIjD,UAAI,gBAAgB,GAAG,EAAG,QAAO;AACjC,YAAM,eAAe,oBAAoB,GAAG;AAC5C,YAAM,SAAS,aAAa,cAAc,EAAE,aAAa,IAAI,YAAY,CAAC;AAC1E,YAAM,WAAW,aAAa,QAAQ,YAAY;AAClD,YAAM,aAAa,QAAQ,cAAc,mBAAmB,UAAU,YAAY;AAClF,YAAM,SAA0B;AAAA,QAC9B;AAAA,QACA;AAAA,QACA;AAAA,QACA;AAAA,QACA;AAAA,QACA,aAAa,IAAI;AAAA,MACnB;AACA,UAAI,MAAM,IAAI,MAAM;AACpB,aAAO;AAAA,IACT;AAAA,IACA,eAAe,CAAC,EAAE,YAAY,OAAO,MAAM;AACzC,YAAM,SAAS,IAAI,MAAM,IAAI,UAAU;AACvC,aAAO,EAAE,MAAM,QAAQ,OAAO,SAAS,OAAO,WAAW,oBAAoB,MAAM,EAAE;AAAA,IACvF;AAAA,EACF;AACA,SAAO;AACT;AAEO,SAAS,UAAU,OAAoB,KAA+B;AAC3E,QAAM,MAAmB,CAAC;AAC1B,aAAW,QAAQ,OAAO,KAAK,KAAK,GAAG;AACrC,UAAM,OAAO,MAAM,IAAI;AACvB,QAAI,KAAM,KAAI,IAAI,IAAI,SAAS,MAAM,MAAM,GAAG;AAAA,EAChD;AACA,SAAO;AACT;AAEO,SAAS,mBAAmB,OAAmC;AACpE,SAAO,SAAS;AAClB;AAEA,SAAS,gBAAgB,OAAiD;AACxE,SACE,OAAO,UAAU,YACjB,UAAU,QACV,OAAQ,MAAiC,OAAO,aAAa,MAAM;AAEvE;AAIA,SAAS,mBAAmB,UAAkB,cAA8B;AAC1E,SAAO,QAAQ,UAAU,QAAQ,GAAG,QAAQ,KAAI,YAAY,EAAE,CAAC,EAAE,MAAM,GAAG,EAAE,CAAC;AAC/E;;;AC7EO,SAAS,eACd,UACA,aACA,QAAwB,aACJ;AACpB,SAAO,SAAS,IAAI,CAAC,YAAY;AAC/B,QAAI,CAAC,MAAM,QAAQ,QAAQ,OAAO,EAAG,QAAO;AAE5C,QAAI,UAAU;AACd,UAAM,UAAW,QAAQ,QAAsB,IAAI,CAAC,SAAS;AAC3D,UAAI,CAAC,iBAAiB,IAAI,EAAG,QAAO;AACpC,YAAM,SAAS,MAAM,IAAI,KAAK,UAAU;AACxC,UAAI,CAAC,OAAQ,QAAO;AAEpB,YAAM,UAAU,aAAa,OAAO,cAAc,EAAE,YAAY,CAAC;AACjE,YAAM,WAAW,aAAa,SAAS,OAAO,YAAY;AAC1D,UAAI,aAAa,aAAa,IAAI,EAAG,QAAO;AAE5C,YAAM,IAAI,EAAE,GAAG,QAAQ,QAAQ,SAAS,UAAU,YAAY,CAAC;AAC/D,gBAAU;AACV,aAAO,KAAK,WAAW,SACnB,EAAE,GAAG,MAAM,QAAQ,EAAE,MAAM,QAAiB,OAAO,SAAS,EAAE,IAC9D,EAAE,GAAG,MAAM,QAAQ,SAAS;AAAA,IAClC,CAAC;AAED,WAAO,UAAU,EAAE,GAAG,SAAS,QAAQ,IAAI;AAAA,EAC7C,CAAC;AACH;AAEA,SAAS,aAAa,MAA0C;AAC9D,MAAI,KAAK,UAAU,WAAW,KAAK,UAAU,OAAO,KAAK,OAAO,UAAU,UAAU;AAClF,WAAO,KAAK,OAAO;AAAA,EACrB;AACA,MAAI,KAAK,WAAW,OAAW,QAAO,oBAAoB,KAAK,MAAM;AACrE,SAAO;AACT;AAEA,SAAS,iBAAiB,MAAuC;AAC/D,MAAI,OAAO,SAAS,YAAY,SAAS,KAAM,QAAO;AACtD,QAAM,IAAI;AACV,SAAO,EAAE,SAAS,iBAAiB,OAAO,EAAE,eAAe;AAC7D;;;ACxBO,SAAS,yBACd,cACA,QAAwB,aACxB,OAAsB,CAAC,GACF;AACrB,QAAM,gBAAgB,KAAK,iBAAiB;AAC5C,QAAM,WAAW,KAAK,sBAAsB;AAC5C,QAAM,WAAW,aAAa,YAAY;AAC1C,QAAM,OAA4B,CAAC;AAEnC,aAAW,UAAU,MAAM,QAAQ,GAAG;AAEpC,QAAI,OAAO,OAAO,WAAW,WAAY;AACzC,UAAM,UAAU,OAAO,OAAO;AAE9B,aAAS,IAAI,GAAG,IAAI,QAAQ,SAAS,QAAQ,KAAK;AAChD,YAAM,UAAU,QAAQ,SAAS,CAAC;AAClC,UAAI,CAAC,QAAS;AACd,YAAM,OAAO,OAAO,OAAO,cAAc,SAAS,EAAE,SAAS,EAAE,CAAC;AAChE,YAAM,eAAe,eAAe,MAAM,eAAe,QAAQ,EAAE;AAAA,QAAO,CAAC,SACzE,aAAa,UAAU,IAAI;AAAA,MAC7B;AACA,UAAI,aAAa,WAAW,EAAG;AAC/B,WAAK,KAAK;AAAA,QACR,YAAY,OAAO;AAAA,QACnB,UAAU,OAAO;AAAA,QACjB,cAAc;AAAA,QACd;AAAA,QACA;AAAA,QACA,GAAI,QAAQ,SAAS,SAAY,CAAC,IAAI,EAAE,MAAM,QAAQ,KAAK;AAAA,MAC7D,CAAC;AAAA,IACH;AAAA,EACF;AACA,SAAO;AACT;AAIO,SAAS,iBAAiB,YAA4D;AAC3F,QAAM,SAAS,WAAW;AAAA,IACxB,CAAC,MACC,qDAAqD,EAAE,QAAQ,IAC5D,EAAE,OAAO,KAAK,EAAE,IAAI,MAAM,EAAE;AAAA,EAAM,EAAE,IAAI;AAAA,EAC/C;AACA,SAAO,EAAE,MAAM,QAAQ,SAAS,OAAO,KAAK,MAAM,EAAE;AACtD;AAEA,IAAM,YAAY,oBAAI,IAAI;AAAA,EACxB;AAAA,EACA;AAAA,EACA;AAAA,EACA;AAAA,EACA;AAAA,EACA;AAAA,EACA;AAAA,EACA;AAAA,EACA;AAAA,EACA;AAAA,EACA;AAAA,EACA;AAAA,EACA;AAAA,EACA;AAAA,EACA;AAAA,EACA;AAAA,EACA;AAAA,EACA;AAAA,EACA;AAAA,EACA;AAAA,EACA;AAAA,EACA;AAAA,EACA;AAAA,EACA;AAAA,EACA;AAAA,EACA;AAAA,EACA;AAAA,EACA;AAAA,EACA;AAAA,EACA;AAAA,EACA;AAAA,EACA;AAAA,EACA;AAAA,EACA;AAAA,EACA;AAAA,EACA;AACF,CAAC;AAED,IAAM,WAAW;AAGjB,IAAM,OAAO;AAEb,SAAS,eAAe,MAAc,WAAmB,KAAuB;AAC9E,QAAM,QAAkB,CAAC;AACzB,QAAM,OAAO,oBAAI,IAAY;AAC7B,QAAM,MAAM,CAAC,SAAuB;AAClC,QAAI,KAAK,SAAS,aAAa,UAAU,IAAI,IAAI,KAAK,KAAK,IAAI,IAAI,EAAG;AACtE,SAAK,IAAI,IAAI;AACb,UAAM,KAAK,IAAI;AAAA,EACjB;AACA,QAAM,OAAO,CAAC,QAAsB;AAClC,QAAI,IAAI,YAAY,CAAC;AAGrB,UAAM,SAAS,IAAI,QAAQ,sBAAsB,OAAO,EAAE,QAAQ,UAAU,GAAG,EAAE,YAAY;AAC7F,QAAI,WAAW,IAAI,YAAY,EAAG,KAAI,MAAM;AAAA,EAC9C;AAGA,aAAW,KAAK,KAAK,SAAS,QAAQ,GAAG;AACvC,QAAI,MAAM,UAAU,IAAK,QAAO;AAChC,QAAI,EAAE,CAAC,EAAG,MAAK,EAAE,CAAC,CAAC;AAAA,EACrB;AACA,aAAW,KAAK,KAAK,SAAS,IAAI,GAAG;AACnC,QAAI,MAAM,UAAU,IAAK,QAAO;AAChC,SAAK,EAAE,CAAC,CAAC;AAAA,EACX;AACA,SAAO;AACT;AAEA,SAAS,aAAa,UAAkB,MAAuB;AAC7D,MAAI,OAAO;AACX,aAAS;AACP,UAAM,KAAK,SAAS,QAAQ,MAAM,IAAI;AACtC,QAAI,OAAO,GAAI,QAAO;AACtB,UAAM,SAAS,OAAO,IAAI,KAAK,SAAS,KAAK,CAAC;AAC9C,UAAM,QAAQ,SAAS,KAAK,KAAK,MAAM,KAAK;AAC5C,QAAI,CAAC,WAAW,MAAM,KAAK,CAAC,WAAW,KAAK,EAAG,QAAO;AACtD,WAAO,KAAK;AAAA,EACd;AACF;AAEA,SAAS,WAAW,IAAqB;AACvC,SAAO,OAAO,MAAM,eAAe,KAAK,EAAE;AAC5C;;;ACrIA,eAAsB,aACpB,SACiC;AACjC,QAAM,EAAE,OAAO,CAAC,GAAG,GAAG,KAAK,IAAI;AAC/B,QAAM,QAAQ,KAAK,SAAS;AAC5B,QAAM,cAAc,mBAAmB,KAAK,WAAW;AACvD,QAAM,WAAW,KAAK,YAAa,MAAM,aAAa;AAEtD,QAAM,YAAqC,EAAE,GAAG,KAAK;AAErD,MAAI,UAAU,KAAK,OAAO,CAAC,GAAG;AAC5B,cAAU,OAAO,IAAI,UAAU,KAAK,OAAO,GAAG,EAAE,OAAO,YAAY,CAAC;AAAA,EACtE;AAEA,MAAI,KAAK,0BAA0B,MAAM,QAAQ,KAAK,UAAU,CAAC,GAAG;AAClE,UAAM,QAAQ,KAAK,mBAAmB,cAAc;AACpD,cAAU,UAAU,IAAI,eAAe,KAAK,UAAU,GAAyB,OAAO,KAAK;AAAA,EAC7F;AAEA,QAAM,SAAS,cAAc,KAAK;AAClC,MAAI,SAAS,MAAM,SAAS,SAAS;AACrC,QAAM,QAAQ,WAAW,OAAO,MAAM;AAEtC,MAAI,aAAkC,CAAC;AACvC,MAAI,aAAa;AACjB,MAAI,KAAK,eAAe,kBAAkB,OAAO,OAAO,SAAS,UAAU;AACzE,iBAAa,yBAAyB,OAAO,MAAM,KAAK;AACxD,QAAI,WAAW,SAAS,KAAK,MAAM,QAAQ,UAAU,UAAU,CAAC,GAAG;AAGjE,YAAM,WAAW;AAAA,QACf,GAAI,UAAU,UAAU;AAAA,QACxB,EAAE,MAAM,aAAa,SAAS,OAAO,KAAK;AAAA,QAC1C,iBAAiB,UAAU;AAAA,MAC7B;AACA,eAAS,MAAM,SAAS,EAAE,GAAG,WAAW,SAAS,CAAC;AAClD,mBAAa;AAAA,IACf;AAAA,EACF;AAEA,SAAO;AAAA,IACL,GAAG;AAAA,IACH,MAAM;AAAA,MACJ;AAAA,MACA;AAAA,MACA,eAAe,MAAM;AAAA,MACrB,eAAe,MAAM;AAAA,MACrB;AAAA,MACA;AAAA,IACF;AAAA,EACF;AACF;AAOA,eAAe,eAAwC;AACrD,QAAM,YAAY;AAClB,QAAM,MAAO,MAAM,OAAO;AAC1B,MAAI,OAAO,IAAI,iBAAiB,YAAY;AAC1C,UAAM,IAAI,MAAM,uEAAuE;AAAA,EACzF;AACA,SAAO,IAAI;AACb;AAEA,SAAS,UAAU,OAAsC;AACvD,SAAO,OAAO,UAAU,YAAY,UAAU,QAAQ,CAAC,MAAM,QAAQ,KAAK;AAC5E;AAEA,SAAS,cAAc,OAAoC;AACzD,SAAO,IAAI,IAAI,MAAM,QAAQ,EAAE,IAAI,CAAC,MAAM,EAAE,UAAU,CAAC;AACzD;AAEA,SAAS,WACP,OACA,QACwC;AACxC,MAAI,WAAW;AACf,MAAI,WAAW;AACf,aAAW,UAAU,MAAM,QAAQ,GAAG;AACpC,QAAI,OAAO,IAAI,OAAO,UAAU,EAAG;AACnC,gBAAY,WAAW,OAAO,YAAY;AAC1C,gBAAY,WAAW,OAAO,QAAQ;AAAA,EACxC;AACA,SAAO,EAAE,UAAU,SAAS;AAC9B;","names":[]}

package/package.json

@@ -0,0 +1,52 @@
+{
+  "name": "@tokz/ai-sdk",
+  "version": "0.1.0",
+  "description": "Vercel AI SDK wrapper for tokz: compresses tool results in place, compacts history, and expands on reference.",
+  "type": "module",
+  "license": "MIT",
+  "homepage": "https://tokz.dev",
+  "keywords": [
+    "tokz",
+    "ai-sdk",
+    "vercel-ai",
+    "prompt-compression",
+    "context-window",
+    "tool-results",
+    "agents"
+  ],
+  "main": "./dist/index.js",
+  "types": "./dist/index.d.ts",
+  "exports": {
+    ".": { "types": "./dist/index.d.ts", "default": "./dist/index.js" }
+  },
+  "files": [
+    "dist"
+  ],
+  "sideEffects": false,
+  "engines": {
+    "node": ">=20"
+  },
+  "publishConfig": {
+    "access": "public"
+  },
+  "scripts": {
+    "build": "tsup",
+    "test": "vitest run",
+    "prepublishOnly": "pnpm build"
+  },
+  "dependencies": {
+    "@tokz/engine": "workspace:*"
+  },
+  "peerDependencies": {
+    "ai": ">=5"
+  },
+  "peerDependenciesMeta": {
+    "ai": { "optional": true }
+  },
+  "devDependencies": {
+    "@tokz/core": "workspace:*",
+    "tsup": "^8.3.5",
+    "typescript": "^5.7.2",
+    "vitest": "^2.1.8"
+  }
+}