@blackwell-systems/gcf 0.3.0 → 0.4.0

package/README.md

@@ -8,7 +8,7 @@
 
 TypeScript implementation of [GCF (Graph Compact Format)](https://gcformat.com/) — the most token-efficient wire format for LLMs. A drop-in alternative to JSON and TOON for any structured data.
 
-**79% fewer input tokens than JSON. 75% fewer output tokens. 52% smaller than TOON. 100% LLM comprehension at 500 symbols, where JSON fails at 66.7%.**
+**79% fewer input tokens than JSON. 75% fewer output tokens. 52% smaller than TOON. 100% LLM comprehension at 500 symbols, where JSON scores 76.9% and TOON scores 92.3%.**
 
 Docs: [gcformat.com](https://gcformat.com/) · [Playground](https://gcformat.com/playground.html) · [GCF vs TOON](https://gcformat.com/guide/vs-toon.html)
 
@@ -97,6 +97,40 @@ const out2 = encodeWithSession(payload2, sess); // reused symbols as "@N  # prev
 
 By the 5th call in a session: 92.7% token savings vs JSON.
 
+## Streaming Encode
+
+Write GCF output incrementally as symbols and edges arrive. Zero buffering, O(1) memory per row. Ideal for MCP servers that walk large graphs or paginate results:
+
+```typescript
+import { StreamEncoder } from '@blackwell-systems/gcf';
+
+const enc = new StreamEncoder(writer, 'context_for_task', { tokenBudget: 5000 });
+
+// Symbols emit immediately as they're discovered.
+enc.writeSymbol({ qualifiedName: 'pkg.Auth', kind: 'function', score: 0.95, provenance: 'lsp', distance: 0 });
+enc.writeSymbol({ qualifiedName: 'pkg.Server', kind: 'function', score: 0.60, provenance: 'lsp', distance: 1 });
+
+// Edges emit immediately too.
+enc.writeEdge({ source: 'pkg.Server', target: 'pkg.Auth', edgeType: 'calls' });
+
+// Close emits the ## _summary trailer with final counts.
+enc.close();
+```
+
+Output:
+```
+GCF tool=context_for_task budget=5000
+## targets
+@0 fn pkg.Auth 0.95 lsp
+## related
+@1 fn pkg.Server 0.60 lsp
+## edges [?]
+@0<@1 calls
+## _summary symbols=2 edges=1 sections=targets:1,related:1,edges:1
+```
+
+The `writer` is any object with a `write(s: string)` method (Node.js streams, web WritableStreams, or a simple callback). Standard `decode()` handles streaming output with no changes.
+
 ## Delta Encoding
 
 When the consumer already has a prior context pack, send only what changed:
@@ -153,6 +187,7 @@ Works on objects, arrays, and primitives. Arrays of uniform objects get tabular
 | `encodeGeneric(data: unknown): string` | Encode any value to GCF tabular format |
 | `decode(input: string): Payload` | Parse GCF text back to a Payload |
 | `encodeWithSession(p: Payload, s: Session): string` | Encode with session deduplication |
+| `new StreamEncoder(w, tool, opts)` | Create a streaming encoder (zero-buffering) |
 | `encodeDelta(d: DeltaPayload): string` | Encode a delta (added/removed only) |
 | `new Session()` | Create a new session tracker |
 
@@ -169,15 +204,17 @@ Works on objects, arrays, and primitives. Arrays of uniform objects get tabular
 
 ## Comprehension Eval
 
-A rigorous 3-way benchmark (GCF vs TOON vs JSON) at 500 symbols, 200 edges. Six structured extraction questions sent to an LLM:
+Rigorous 3-way benchmark (GCF vs TOON vs JSON) at 500 symbols, 200 edges. 13 structured extraction questions sent to an LLM with zero format instructions:
 
 | Format | Accuracy | Tokens | vs JSON |
 |--------|----------|--------|---------|
-| **GCF** | **100%** (6/6) | **11,090** | **79% fewer** |
-| TOON | 100% (6/6) | 16,378 | 69% fewer |
-| JSON | 66.7% (4/6) | 53,341 | baseline |
+| **GCF** | **100%** (13/13) | **11,090** | **79% fewer** |
+| TOON | 92.3% (12/13) | 16,378 | 69% fewer |
+| JSON | 76.9% (10/13) | 53,341 | baseline |
+
+GCF is the only format with perfect accuracy at scale, at 32% fewer tokens than TOON.
 
-JSON failed on counting tasks. GCF and TOON both achieved perfect accuracy. GCF does it in 32% fewer tokens.
+Reproduce: `git clone https://github.com/blackwell-systems/gcf-go && cd gcf-go/eval && GOWORK=off go test -run TestComprehension -v -timeout 0`
 
 ## Token Efficiency (TOON's Own Benchmark)
 
@@ -185,13 +222,13 @@ Running [TOON's benchmark harness](https://github.com/blackwell-systems/toon/tre
 
 | Track | GCF | TOON | Result |
 |-------|-----|------|--------|
-| Mixed-structure (nested, semi-uniform) | 169,554 | 227,896 | **GCF 34% smaller** |
-| Flat-only (tabular) | 66,026 | 67,837 | **GCF 3% smaller** |
-| Semi-uniform event logs | 107,269 | 154,032 | **GCF 44% smaller** |
+| Mixed-structure (nested, semi-uniform) | 170,367 | 227,896 | **GCF 34% smaller** |
+| Flat-only (tabular) | 66,029 | 67,837 | **GCF 3% smaller** |
+| Semi-uniform event logs | 108,158 | 154,032 | **GCF 42% smaller** |
 
-GCF wins on every dataset except deeply nested config (75 tokens on a 618-token payload). On semi-uniform data, GCF uses 44% fewer tokens than TOON.
+GCF wins all 6 datasets. On semi-uniform data (the most common real-world pattern), GCF uses 42% fewer tokens than TOON.
 
-Reproducible: [blackwell-systems/toon@gcf-comparison](https://github.com/blackwell-systems/toon/tree/gcf-comparison)
+Reproduce: `git clone https://github.com/blackwell-systems/toon && cd toon && git checkout gcf-comparison && cd benchmarks && pnpm install && pnpm benchmark:tokens`
 
 ## Links
 

package/dist/index.d.ts

@@ -5,4 +5,5 @@ export { decode } from './decode.js';
 export { Session, encodeWithSession } from './session.js';
 export { encodeDelta } from './delta.js';
 export { encodeGeneric } from './generic.js';
+export { StreamEncoder, type StreamWriter, type StreamOptions } from './stream.js';
 //# sourceMappingURL=index.d.ts.map

package/dist/index.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA,YAAY,EAAE,MAAM,EAAE,IAAI,EAAE,OAAO,EAAE,YAAY,EAAE,UAAU,EAAE,MAAM,YAAY,CAAC;AAClF,OAAO,EAAE,WAAW,EAAE,WAAW,EAAE,MAAM,gBAAgB,CAAC;AAC1D,OAAO,EAAE,MAAM,EAAE,MAAM,aAAa,CAAC;AACrC,OAAO,EAAE,MAAM,EAAE,MAAM,aAAa,CAAC;AACrC,OAAO,EAAE,OAAO,EAAE,iBAAiB,EAAE,MAAM,cAAc,CAAC;AAC1D,OAAO,EAAE,WAAW,EAAE,MAAM,YAAY,CAAC;AACzC,OAAO,EAAE,aAAa,EAAE,MAAM,cAAc,CAAC"}
+{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA,YAAY,EAAE,MAAM,EAAE,IAAI,EAAE,OAAO,EAAE,YAAY,EAAE,UAAU,EAAE,MAAM,YAAY,CAAC;AAClF,OAAO,EAAE,WAAW,EAAE,WAAW,EAAE,MAAM,gBAAgB,CAAC;AAC1D,OAAO,EAAE,MAAM,EAAE,MAAM,aAAa,CAAC;AACrC,OAAO,EAAE,MAAM,EAAE,MAAM,aAAa,CAAC;AACrC,OAAO,EAAE,OAAO,EAAE,iBAAiB,EAAE,MAAM,cAAc,CAAC;AAC1D,OAAO,EAAE,WAAW,EAAE,MAAM,YAAY,CAAC;AACzC,OAAO,EAAE,aAAa,EAAE,MAAM,cAAc,CAAC;AAC7C,OAAO,EAAE,aAAa,EAAE,KAAK,YAAY,EAAE,KAAK,aAAa,EAAE,MAAM,aAAa,CAAC"}

package/dist/index.js

@@ -4,4 +4,5 @@ export { decode } from './decode.js';
 export { Session, encodeWithSession } from './session.js';
 export { encodeDelta } from './delta.js';
 export { encodeGeneric } from './generic.js';
+export { StreamEncoder } from './stream.js';
 //# sourceMappingURL=index.js.map

package/dist/index.js.map

@@ -1 +1 @@
-{"version":3,"file":"index.js","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AACA,OAAO,EAAE,WAAW,EAAE,WAAW,EAAE,MAAM,gBAAgB,CAAC;AAC1D,OAAO,EAAE,MAAM,EAAE,MAAM,aAAa,CAAC;AACrC,OAAO,EAAE,MAAM,EAAE,MAAM,aAAa,CAAC;AACrC,OAAO,EAAE,OAAO,EAAE,iBAAiB,EAAE,MAAM,cAAc,CAAC;AAC1D,OAAO,EAAE,WAAW,EAAE,MAAM,YAAY,CAAC;AACzC,OAAO,EAAE,aAAa,EAAE,MAAM,cAAc,CAAC"}
+{"version":3,"file":"index.js","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AACA,OAAO,EAAE,WAAW,EAAE,WAAW,EAAE,MAAM,gBAAgB,CAAC;AAC1D,OAAO,EAAE,MAAM,EAAE,MAAM,aAAa,CAAC;AACrC,OAAO,EAAE,MAAM,EAAE,MAAM,aAAa,CAAC;AACrC,OAAO,EAAE,OAAO,EAAE,iBAAiB,EAAE,MAAM,cAAc,CAAC;AAC1D,OAAO,EAAE,WAAW,EAAE,MAAM,YAAY,CAAC;AACzC,OAAO,EAAE,aAAa,EAAE,MAAM,cAAc,CAAC;AAC7C,OAAO,EAAE,aAAa,EAAyC,MAAM,aAAa,CAAC"}

package/dist/stream.d.ts

@@ -0,0 +1,67 @@
+import type { Symbol, Edge } from './types.js';
+/**
+ * Options for the streaming encoder.
+ */
+export interface StreamOptions {
+    tokenBudget?: number;
+    tokensUsed?: number;
+    packRoot?: string;
+    session?: boolean;
+}
+/**
+ * A writable sink for streaming output. Accepts string chunks.
+ * Compatible with Node.js streams, web WritableStreams, or simple callbacks.
+ */
+export interface StreamWriter {
+    write(chunk: string): void;
+}
+/**
+ * StreamEncoder writes GCF output incrementally as symbols and edges arrive.
+ * Zero buffering: each symbol/edge is written immediately. A trailer summary
+ * is emitted on close() with the final counts.
+ *
+ * @example
+ * ```ts
+ * const chunks: string[] = [];
+ * const enc = new StreamEncoder({ write: (s) => chunks.push(s) }, 'context_for_task', { tokenBudget: 5000 });
+ * enc.writeSymbol({ qualifiedName: 'pkg.Auth', kind: 'function', score: 0.95, provenance: 'lsp', distance: 0 });
+ * enc.writeEdge({ source: 'pkg.Server', target: 'pkg.Auth', edgeType: 'calls' });
+ * enc.close();
+ * ```
+ */
+export declare class StreamEncoder {
+    private w;
+    private symIndex;
+    private nextID;
+    private currentGroup;
+    private groupCounts;
+    private edgeCount;
+    private edgesStarted;
+    constructor(w: StreamWriter, tool: string, opts?: StreamOptions);
+    private writeHeader;
+    /**
+     * Emit a symbol line immediately. Group headers are emitted automatically
+     * when the distance changes.
+     */
+    writeSymbol(s: Symbol): void;
+    /**
+     * Emit an edge line immediately. The edges section header is emitted
+     * automatically on the first edge (with [?] deferred count).
+     * Source and target must reference previously-written symbols.
+     */
+    writeEdge(e: Edge): void;
+    /**
+     * Emit a bare reference for a previously-transmitted symbol (session mode).
+     */
+    writeBareRef(qname: string, distance: number): void;
+    /**
+     * Emit the ## _summary trailer with final counts. Must be called after all
+     * symbols and edges have been written.
+     */
+    close(): void;
+    /** Number of symbols written so far. */
+    get symbolCount(): number;
+    /** Number of edges written so far. */
+    get edgeCount_(): number;
+}
+//# sourceMappingURL=stream.d.ts.map

package/dist/stream.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"stream.d.ts","sourceRoot":"","sources":["../src/stream.ts"],"names":[],"mappings":"AACA,OAAO,KAAK,EAAE,MAAM,EAAE,IAAI,EAAE,MAAM,YAAY,CAAC;AAE/C;;GAEG;AACH,MAAM,WAAW,aAAa;IAC5B,WAAW,CAAC,EAAE,MAAM,CAAC;IACrB,UAAU,CAAC,EAAE,MAAM,CAAC;IACpB,QAAQ,CAAC,EAAE,MAAM,CAAC;IAClB,OAAO,CAAC,EAAE,OAAO,CAAC;CACnB;AAED;;;GAGG;AACH,MAAM,WAAW,YAAY;IAC3B,KAAK,CAAC,KAAK,EAAE,MAAM,GAAG,IAAI,CAAC;CAC5B;AAED;;;;;;;;;;;;;GAaG;AACH,qBAAa,aAAa;IACxB,OAAO,CAAC,CAAC,CAAe;IACxB,OAAO,CAAC,QAAQ,CAAkC;IAClD,OAAO,CAAC,MAAM,CAAK;IACnB,OAAO,CAAC,YAAY,CAAM;IAC1B,OAAO,CAAC,WAAW,CAAkC;IACrD,OAAO,CAAC,SAAS,CAAK;IACtB,OAAO,CAAC,YAAY,CAAS;gBAEjB,CAAC,EAAE,YAAY,EAAE,IAAI,EAAE,MAAM,EAAE,IAAI,GAAE,aAAkB;IAKnE,OAAO,CAAC,WAAW;IASnB;;;OAGG;IACH,WAAW,CAAC,CAAC,EAAE,MAAM,GAAG,IAAI;IAoB5B;;;;OAIG;IACH,SAAS,CAAC,CAAC,EAAE,IAAI,GAAG,IAAI;IAkBxB;;OAEG;IACH,YAAY,CAAC,KAAK,EAAE,MAAM,EAAE,QAAQ,EAAE,MAAM,GAAG,IAAI;IAkBnD;;;OAGG;IACH,KAAK,IAAI,IAAI;IAkBb,wCAAwC;IACxC,IAAI,WAAW,IAAI,MAAM,CAAwB;IAEjD,sCAAsC;IACtC,IAAI,UAAU,IAAI,MAAM,CAA2B;CACpD"}

package/dist/stream.js

@@ -0,0 +1,123 @@
+import { KIND_ABBREV } from './constants.js';
+/**
+ * StreamEncoder writes GCF output incrementally as symbols and edges arrive.
+ * Zero buffering: each symbol/edge is written immediately. A trailer summary
+ * is emitted on close() with the final counts.
+ *
+ * @example
+ * ```ts
+ * const chunks: string[] = [];
+ * const enc = new StreamEncoder({ write: (s) => chunks.push(s) }, 'context_for_task', { tokenBudget: 5000 });
+ * enc.writeSymbol({ qualifiedName: 'pkg.Auth', kind: 'function', score: 0.95, provenance: 'lsp', distance: 0 });
+ * enc.writeEdge({ source: 'pkg.Server', target: 'pkg.Auth', edgeType: 'calls' });
+ * enc.close();
+ * ```
+ */
+export class StreamEncoder {
+    w;
+    symIndex = new Map();
+    nextID = 0;
+    currentGroup = '';
+    groupCounts = new Map();
+    edgeCount = 0;
+    edgesStarted = false;
+    constructor(w, tool, opts = {}) {
+        this.w = w;
+        this.writeHeader(tool, opts);
+    }
+    writeHeader(tool, opts) {
+        const parts = [`GCF tool=${tool}`];
+        if (opts.tokenBudget)
+            parts.push(`budget=${opts.tokenBudget}`);
+        if (opts.tokensUsed)
+            parts.push(`tokens=${opts.tokensUsed}`);
+        if (opts.packRoot)
+            parts.push(`pack_root=${opts.packRoot}`);
+        if (opts.session)
+            parts.push('session=true');
+        this.w.write(parts.join(' ') + '\n');
+    }
+    /**
+     * Emit a symbol line immediately. Group headers are emitted automatically
+     * when the distance changes.
+     */
+    writeSymbol(s) {
+        const groupNames = ['targets', 'related', 'extended'];
+        const groupName = s.distance < groupNames.length
+            ? groupNames[s.distance]
+            : `distance_${s.distance}`;
+        if (groupName !== this.currentGroup) {
+            this.w.write(`## ${groupName}\n`);
+            this.currentGroup = groupName;
+        }
+        const id = this.nextID++;
+        this.symIndex.set(s.qualifiedName, id);
+        const kind = KIND_ABBREV[s.kind] || s.kind;
+        this.w.write(`@${id} ${kind} ${s.qualifiedName} ${s.score.toFixed(2)} ${s.provenance}\n`);
+        this.groupCounts.set(groupName, (this.groupCounts.get(groupName) || 0) + 1);
+    }
+    /**
+     * Emit an edge line immediately. The edges section header is emitted
+     * automatically on the first edge (with [?] deferred count).
+     * Source and target must reference previously-written symbols.
+     */
+    writeEdge(e) {
+        const srcIdx = this.symIndex.get(e.source);
+        const tgtIdx = this.symIndex.get(e.target);
+        if (srcIdx === undefined || tgtIdx === undefined)
+            return;
+        if (!this.edgesStarted) {
+            this.w.write('## edges [?]\n');
+            this.edgesStarted = true;
+        }
+        let line = `@${tgtIdx}<@${srcIdx} ${e.edgeType}`;
+        if (e.status && e.status !== 'unchanged') {
+            line += ` ${e.status}`;
+        }
+        this.w.write(line + '\n');
+        this.edgeCount++;
+    }
+    /**
+     * Emit a bare reference for a previously-transmitted symbol (session mode).
+     */
+    writeBareRef(qname, distance) {
+        const groupNames = ['targets', 'related', 'extended'];
+        const groupName = distance < groupNames.length
+            ? groupNames[distance]
+            : `distance_${distance}`;
+        if (groupName !== this.currentGroup) {
+            this.w.write(`## ${groupName}\n`);
+            this.currentGroup = groupName;
+        }
+        const id = this.nextID++;
+        this.symIndex.set(qname, id);
+        this.w.write(`@${id}  # previously transmitted\n`);
+        this.groupCounts.set(groupName, (this.groupCounts.get(groupName) || 0) + 1);
+    }
+    /**
+     * Emit the ## _summary trailer with final counts. Must be called after all
+     * symbols and edges have been written.
+     */
+    close() {
+        const sections = [];
+        const groupOrder = ['targets', 'related', 'extended'];
+        for (const g of groupOrder) {
+            const c = this.groupCounts.get(g);
+            if (c && c > 0)
+                sections.push(`${g}:${c}`);
+        }
+        for (const [g, c] of this.groupCounts) {
+            if (!groupOrder.includes(g) && c > 0)
+                sections.push(`${g}:${c}`);
+        }
+        if (this.edgeCount > 0) {
+            sections.push(`edges:${this.edgeCount}`);
+        }
+        this.w.write(`## _summary symbols=${this.nextID} edges=${this.edgeCount} sections=${sections.join(',')}\n`);
+    }
+    /** Number of symbols written so far. */
+    get symbolCount() { return this.nextID; }
+    /** Number of edges written so far. */
+    get edgeCount_() { return this.edgeCount; }
+}
+//# sourceMappingURL=stream.js.map

package/dist/stream.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"stream.js","sourceRoot":"","sources":["../src/stream.ts"],"names":[],"mappings":"AAAA,OAAO,EAAE,WAAW,EAAE,MAAM,gBAAgB,CAAC;AAqB7C;;;;;;;;;;;;;GAaG;AACH,MAAM,OAAO,aAAa;IAChB,CAAC,CAAe;IAChB,QAAQ,GAAwB,IAAI,GAAG,EAAE,CAAC;IAC1C,MAAM,GAAG,CAAC,CAAC;IACX,YAAY,GAAG,EAAE,CAAC;IAClB,WAAW,GAAwB,IAAI,GAAG,EAAE,CAAC;IAC7C,SAAS,GAAG,CAAC,CAAC;IACd,YAAY,GAAG,KAAK,CAAC;IAE7B,YAAY,CAAe,EAAE,IAAY,EAAE,OAAsB,EAAE;QACjE,IAAI,CAAC,CAAC,GAAG,CAAC,CAAC;QACX,IAAI,CAAC,WAAW,CAAC,IAAI,EAAE,IAAI,CAAC,CAAC;IAC/B,CAAC;IAEO,WAAW,CAAC,IAAY,EAAE,IAAmB;QACnD,MAAM,KAAK,GAAG,CAAC,YAAY,IAAI,EAAE,CAAC,CAAC;QACnC,IAAI,IAAI,CAAC,WAAW;YAAE,KAAK,CAAC,IAAI,CAAC,UAAU,IAAI,CAAC,WAAW,EAAE,CAAC,CAAC;QAC/D,IAAI,IAAI,CAAC,UAAU;YAAE,KAAK,CAAC,IAAI,CAAC,UAAU,IAAI,CAAC,UAAU,EAAE,CAAC,CAAC;QAC7D,IAAI,IAAI,CAAC,QAAQ;YAAE,KAAK,CAAC,IAAI,CAAC,aAAa,IAAI,CAAC,QAAQ,EAAE,CAAC,CAAC;QAC5D,IAAI,IAAI,CAAC,OAAO;YAAE,KAAK,CAAC,IAAI,CAAC,cAAc,CAAC,CAAC;QAC7C,IAAI,CAAC,CAAC,CAAC,KAAK,CAAC,KAAK,CAAC,IAAI,CAAC,GAAG,CAAC,GAAG,IAAI,CAAC,CAAC;IACvC,CAAC;IAED;;;OAGG;IACH,WAAW,CAAC,CAAS;QACnB,MAAM,UAAU,GAAG,CAAC,SAAS,EAAE,SAAS,EAAE,UAAU,CAAC,CAAC;QACtD,MAAM,SAAS,GAAG,CAAC,CAAC,QAAQ,GAAG,UAAU,CAAC,MAAM;YAC9C,CAAC,CAAC,UAAU,CAAC,CAAC,CAAC,QAAQ,CAAC;YACxB,CAAC,CAAC,YAAY,CAAC,CAAC,QAAQ,EAAE,CAAC;QAE7B,IAAI,SAAS,KAAK,IAAI,CAAC,YAAY,EAAE,CAAC;YACpC,IAAI,CAAC,CAAC,CAAC,KAAK,CAAC,MAAM,SAAS,IAAI,CAAC,CAAC;YAClC,IAAI,CAAC,YAAY,GAAG,SAAS,CAAC;QAChC,CAAC;QAED,MAAM,EAAE,GAAG,IAAI,CAAC,MAAM,EAAE,CAAC;QACzB,IAAI,CAAC,QAAQ,CAAC,GAAG,CAAC,CAAC,CAAC,aAAa,EAAE,EAAE,CAAC,CAAC;QAEvC,MAAM,IAAI,GAAG,WAAW,CAAC,CAAC,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC,IAAI,CAAC;QAC3C,IAAI,CAAC,CAAC,CAAC,KAAK,CAAC,IAAI,EAAE,IAAI,IAAI,IAAI,CAAC,CAAC,aAAa,IAAI,CAAC,CAAC,KAAK,CAAC,OAAO,CAAC,CAAC,CAAC,IAAI,CAAC,CAAC,UAAU,IAAI,CAAC,CAAC;QAE1F,IAAI,CAAC,WAAW,CAAC,GAAG,CAAC,SAAS,EAAE,CAAC,IAAI,CAAC,WAAW,CAAC,GAAG,CAAC,SAAS,CAAC,IAAI,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC;IAC9E,CAAC;IAED;;;;OAIG;IACH,SAAS,CAAC,CAAO;QACf,MAAM,MAAM,GAAG,IAAI,CAAC,QAAQ,CAAC,GAAG,CAAC,CAAC,CAAC,MAAM,CAAC,CAAC;QAC3C,MAAM,MAAM,GAAG,IAAI,CAAC,QAAQ,CAAC,GAAG,CAAC,CAAC,CAAC,MAAM,CAAC,CAAC;QAC3C,IAAI,MAAM,KAAK,SAAS,IAAI,MAAM,KAAK,SAAS;YAAE,OAAO;QAEzD,IAAI,CAAC,IAAI,CAAC,YAAY,EAAE,CAAC;YACvB,IAAI,CAAC,CAAC,CAAC,KAAK,CAAC,gBAAgB,CAAC,CAAC;YAC/B,IAAI,CAAC,YAAY,GAAG,IAAI,CAAC;QAC3B,CAAC;QAED,IAAI,IAAI,GAAG,IAAI,MAAM,KAAK,MAAM,IAAI,CAAC,CAAC,QAAQ,EAAE,CAAC;QACjD,IAAI,CAAC,CAAC,MAAM,IAAI,CAAC,CAAC,MAAM,KAAK,WAAW,EAAE,CAAC;YACzC,IAAI,IAAI,IAAI,CAAC,CAAC,MAAM,EAAE,CAAC;QACzB,CAAC;QACD,IAAI,CAAC,CAAC,CAAC,KAAK,CAAC,IAAI,GAAG,IAAI,CAAC,CAAC;QAC1B,IAAI,CAAC,SAAS,EAAE,CAAC;IACnB,CAAC;IAED;;OAEG;IACH,YAAY,CAAC,KAAa,EAAE,QAAgB;QAC1C,MAAM,UAAU,GAAG,CAAC,SAAS,EAAE,SAAS,EAAE,UAAU,CAAC,CAAC;QACtD,MAAM,SAAS,GAAG,QAAQ,GAAG,UAAU,CAAC,MAAM;YAC5C,CAAC,CAAC,UAAU,CAAC,QAAQ,CAAC;YACtB,CAAC,CAAC,YAAY,QAAQ,EAAE,CAAC;QAE3B,IAAI,SAAS,KAAK,IAAI,CAAC,YAAY,EAAE,CAAC;YACpC,IAAI,CAAC,CAAC,CAAC,KAAK,CAAC,MAAM,SAAS,IAAI,CAAC,CAAC;YAClC,IAAI,CAAC,YAAY,GAAG,SAAS,CAAC;QAChC,CAAC;QAED,MAAM,EAAE,GAAG,IAAI,CAAC,MAAM,EAAE,CAAC;QACzB,IAAI,CAAC,QAAQ,CAAC,GAAG,CAAC,KAAK,EAAE,EAAE,CAAC,CAAC;QAC7B,IAAI,CAAC,CAAC,CAAC,KAAK,CAAC,IAAI,EAAE,8BAA8B,CAAC,CAAC;QAEnD,IAAI,CAAC,WAAW,CAAC,GAAG,CAAC,SAAS,EAAE,CAAC,IAAI,CAAC,WAAW,CAAC,GAAG,CAAC,SAAS,CAAC,IAAI,CAAC,CAAC,GAAG,CAAC,CAAC,CAAC;IAC9E,CAAC;IAED;;;OAGG;IACH,KAAK;QACH,MAAM,QAAQ,GAAa,EAAE,CAAC;QAC9B,MAAM,UAAU,GAAG,CAAC,SAAS,EAAE,SAAS,EAAE,UAAU,CAAC,CAAC;QAEtD,KAAK,MAAM,CAAC,IAAI,UAAU,EAAE,CAAC;YAC3B,MAAM,CAAC,GAAG,IAAI,CAAC,WAAW,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC;YAClC,IAAI,CAAC,IAAI,CAAC,GAAG,CAAC;gBAAE,QAAQ,CAAC,IAAI,CAAC,GAAG,CAAC,IAAI,CAAC,EAAE,CAAC,CAAC;QAC7C,CAAC;QACD,KAAK,MAAM,CAAC,CAAC,EAAE,CAAC,CAAC,IAAI,IAAI,CAAC,WAAW,EAAE,CAAC;YACtC,IAAI,CAAC,UAAU,CAAC,QAAQ,CAAC,CAAC,CAAC,IAAI,CAAC,GAAG,CAAC;gBAAE,QAAQ,CAAC,IAAI,CAAC,GAAG,CAAC,IAAI,CAAC,EAAE,CAAC,CAAC;QACnE,CAAC;QACD,IAAI,IAAI,CAAC,SAAS,GAAG,CAAC,EAAE,CAAC;YACvB,QAAQ,CAAC,IAAI,CAAC,SAAS,IAAI,CAAC,SAAS,EAAE,CAAC,CAAC;QAC3C,CAAC;QAED,IAAI,CAAC,CAAC,CAAC,KAAK,CAAC,uBAAuB,IAAI,CAAC,MAAM,UAAU,IAAI,CAAC,SAAS,aAAa,QAAQ,CAAC,IAAI,CAAC,GAAG,CAAC,IAAI,CAAC,CAAC;IAC9G,CAAC;IAED,wCAAwC;IACxC,IAAI,WAAW,KAAa,OAAO,IAAI,CAAC,MAAM,CAAC,CAAC,CAAC;IAEjD,sCAAsC;IACtC,IAAI,UAAU,KAAa,OAAO,IAAI,CAAC,SAAS,CAAC,CAAC,CAAC;CACpD"}

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@blackwell-systems/gcf",
-  "version": "0.3.0",
+  "version": "0.4.0",
   "description": "TypeScript implementation of GCF (Graph Compact Format) - token-optimized wire format for LLM tool responses",
   "type": "module",
   "main": "./dist/index.js",

package/src/index.ts

@@ -5,3 +5,4 @@ export { decode } from './decode.js';
 export { Session, encodeWithSession } from './session.js';
 export { encodeDelta } from './delta.js';
 export { encodeGeneric } from './generic.js';
+export { StreamEncoder, type StreamWriter, type StreamOptions } from './stream.js';

package/src/stream.ts

@@ -0,0 +1,154 @@
+import { KIND_ABBREV } from './constants.js';
+import type { Symbol, Edge } from './types.js';
+
+/**
+ * Options for the streaming encoder.
+ */
+export interface StreamOptions {
+  tokenBudget?: number;
+  tokensUsed?: number;
+  packRoot?: string;
+  session?: boolean;
+}
+
+/**
+ * A writable sink for streaming output. Accepts string chunks.
+ * Compatible with Node.js streams, web WritableStreams, or simple callbacks.
+ */
+export interface StreamWriter {
+  write(chunk: string): void;
+}
+
+/**
+ * StreamEncoder writes GCF output incrementally as symbols and edges arrive.
+ * Zero buffering: each symbol/edge is written immediately. A trailer summary
+ * is emitted on close() with the final counts.
+ *
+ * @example
+ * ```ts
+ * const chunks: string[] = [];
+ * const enc = new StreamEncoder({ write: (s) => chunks.push(s) }, 'context_for_task', { tokenBudget: 5000 });
+ * enc.writeSymbol({ qualifiedName: 'pkg.Auth', kind: 'function', score: 0.95, provenance: 'lsp', distance: 0 });
+ * enc.writeEdge({ source: 'pkg.Server', target: 'pkg.Auth', edgeType: 'calls' });
+ * enc.close();
+ * ```
+ */
+export class StreamEncoder {
+  private w: StreamWriter;
+  private symIndex: Map<string, number> = new Map();
+  private nextID = 0;
+  private currentGroup = '';
+  private groupCounts: Map<string, number> = new Map();
+  private edgeCount = 0;
+  private edgesStarted = false;
+
+  constructor(w: StreamWriter, tool: string, opts: StreamOptions = {}) {
+    this.w = w;
+    this.writeHeader(tool, opts);
+  }
+
+  private writeHeader(tool: string, opts: StreamOptions): void {
+    const parts = [`GCF tool=${tool}`];
+    if (opts.tokenBudget) parts.push(`budget=${opts.tokenBudget}`);
+    if (opts.tokensUsed) parts.push(`tokens=${opts.tokensUsed}`);
+    if (opts.packRoot) parts.push(`pack_root=${opts.packRoot}`);
+    if (opts.session) parts.push('session=true');
+    this.w.write(parts.join(' ') + '\n');
+  }
+
+  /**
+   * Emit a symbol line immediately. Group headers are emitted automatically
+   * when the distance changes.
+   */
+  writeSymbol(s: Symbol): void {
+    const groupNames = ['targets', 'related', 'extended'];
+    const groupName = s.distance < groupNames.length
+      ? groupNames[s.distance]
+      : `distance_${s.distance}`;
+
+    if (groupName !== this.currentGroup) {
+      this.w.write(`## ${groupName}\n`);
+      this.currentGroup = groupName;
+    }
+
+    const id = this.nextID++;
+    this.symIndex.set(s.qualifiedName, id);
+
+    const kind = KIND_ABBREV[s.kind] || s.kind;
+    this.w.write(`@${id} ${kind} ${s.qualifiedName} ${s.score.toFixed(2)} ${s.provenance}\n`);
+
+    this.groupCounts.set(groupName, (this.groupCounts.get(groupName) || 0) + 1);
+  }
+
+  /**
+   * Emit an edge line immediately. The edges section header is emitted
+   * automatically on the first edge (with [?] deferred count).
+   * Source and target must reference previously-written symbols.
+   */
+  writeEdge(e: Edge): void {
+    const srcIdx = this.symIndex.get(e.source);
+    const tgtIdx = this.symIndex.get(e.target);
+    if (srcIdx === undefined || tgtIdx === undefined) return;
+
+    if (!this.edgesStarted) {
+      this.w.write('## edges [?]\n');
+      this.edgesStarted = true;
+    }
+
+    let line = `@${tgtIdx}<@${srcIdx} ${e.edgeType}`;
+    if (e.status && e.status !== 'unchanged') {
+      line += ` ${e.status}`;
+    }
+    this.w.write(line + '\n');
+    this.edgeCount++;
+  }
+
+  /**
+   * Emit a bare reference for a previously-transmitted symbol (session mode).
+   */
+  writeBareRef(qname: string, distance: number): void {
+    const groupNames = ['targets', 'related', 'extended'];
+    const groupName = distance < groupNames.length
+      ? groupNames[distance]
+      : `distance_${distance}`;
+
+    if (groupName !== this.currentGroup) {
+      this.w.write(`## ${groupName}\n`);
+      this.currentGroup = groupName;
+    }
+
+    const id = this.nextID++;
+    this.symIndex.set(qname, id);
+    this.w.write(`@${id}  # previously transmitted\n`);
+
+    this.groupCounts.set(groupName, (this.groupCounts.get(groupName) || 0) + 1);
+  }
+
+  /**
+   * Emit the ## _summary trailer with final counts. Must be called after all
+   * symbols and edges have been written.
+   */
+  close(): void {
+    const sections: string[] = [];
+    const groupOrder = ['targets', 'related', 'extended'];
+
+    for (const g of groupOrder) {
+      const c = this.groupCounts.get(g);
+      if (c && c > 0) sections.push(`${g}:${c}`);
+    }
+    for (const [g, c] of this.groupCounts) {
+      if (!groupOrder.includes(g) && c > 0) sections.push(`${g}:${c}`);
+    }
+    if (this.edgeCount > 0) {
+      sections.push(`edges:${this.edgeCount}`);
+    }
+
+    this.w.write(`## _summary symbols=${this.nextID} edges=${this.edgeCount} sections=${sections.join(',')}\n`);
+  }
+
+  /** Number of symbols written so far. */
+  get symbolCount(): number { return this.nextID; }
+
+  /** Number of edges written so far. */
+  get edgeCount_(): number { return this.edgeCount; }
+}