cut-at-k 0.1.0

package/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Redouane (ROSH Company Labs)
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

package/README.md

@@ -0,0 +1,127 @@
+# cut-at-k
+
+Cut a stream at every point and check what the consumer is left with.
+
+A stream that stops early is not a rare case. A connection drops, a proxy times out,
+a producer dies, someone presses stop. So take the same stream, run it through your
+client once whole and once for every prefix of it, and ask two questions of each cut:
+
+- **Did the head change?** Truncation may lose the tail. It must never alter what came
+  before the cut.
+- **Does the consumer say so?** A run that was cut short must not report what the same
+  run reports when it finishes.
+
+The second one is where things are usually wrong, and it is quiet when it is wrong,
+which is why a loop is better at finding it than reading the code.
+
+```bash
+git clone https://github.com/roshcompanylabs/cut-at-k
+```
+
+Not on npm yet. Zero dependencies, so a clone is the whole install.
+
+```js
+import { severAtEveryPoint, summarise, format } from './src/index.js';
+
+const report = await severAtEveryPoint({
+  events,                                  // the whole stream, in order
+  label: 'my-stream',
+  replay: async (prefix) => {              // run this prefix through your client
+    const seen = [];
+    const settled = await drive(prefix, seen);
+    return { delivered: seen, settled, text: render(seen) };
+  },
+});
+
+console.log(format(summarise([report], {
+  lostContent: (cut, whole) => cut.text !== whole.text,
+})));
+```
+
+Nothing here knows about any protocol. You supply the events and a function that
+replays them; this supplies the loop and the comparison.
+
+## It will not call something a defect on your behalf
+
+Three things look identical in the raw numbers and only one of them is a finding:
+
+- A cut that **lost nothing**. Cutting after the last content event and before some
+  trailing bookkeeping removes no information, so of course the consumer reports the
+  same thing. Correct behaviour.
+- A stream that is **invalid on purpose**. Cutting it before its deliberate error
+  removes the error, so the whole stream delivers *less* than the prefix and every
+  comparison against it is noise. Conformance corpora are full of these.
+- A cut that **lost content and was reported as though it had not**. The finding.
+
+So `summarise` reports nothing as a defect unless you pass `lostContent`, and it
+reports excluded streams as a count rather than dropping them silently. The first
+version of `summarise` did not make these distinctions and called 154 cuts a
+problem on a corpus where 54 were. A tool that cries wolf is worse than no tool.
+
+## Worked example: AG-UI's own conformance corpus
+
+[AG-UI](https://github.com/ag-ui-protocol/ag-ui) ships 68 conformance fixtures — real
+event streams, written by the protocol's authors, with the outcome each one requires.
+Two are a single event long and have no cut point, so 66 were replayed: each through a
+real `HttpAgent` over real HTTP/SSE, whole and then cut at every event boundary. Both
+steps are in `results/` and the recording is committed, so the block below is one
+command away:
+
+```
+node results/run-ag-ui.mjs results/sever-results.json
+```
+
+
+```
+48 streams, 227 cuts, 18 streams excluded as invalid on purpose
+4 cuts threw during replay
+
+reported the same as the whole run    : 154
+  of which something was actually lost : 54
+  of those, no terminal event either   : 52
+spread across                          : 34 streams
+```
+
+A cut here is the specification's own definition of a truncated run: the stream ends
+after k events with no further frames and no terminal event. From
+[`/spec/1.0/basic/transports#truncation`](https://github.com/ag-ui-protocol/ag-ui):
+
+> A consumer whose stream ends without a terminal event has a truncated run … A
+> consumer MUST NOT synthesize a `RUN_FINISHED` for it and MUST NOT report it as
+> having succeeded.
+
+The smallest case, from their `conformant-run-is-quiet` fixture cut after its fourth
+event:
+
+```
+truncated   settled {"result":null,"newMessages":1}   text "Hello, "
+            events  RUN_STARTED STEP_STARTED TEXT_MESSAGE_START TEXT_MESSAGE_CONTENT
+completed   settled {"result":null,"newMessages":1}   text "Hello, world."
+```
+
+The awaited call reports the same value either way. A subscriber sees no terminal
+event at all. Neither channel says the run was cut.
+
+`results/run-ag-ui.mjs` reproduces the table from the recorded observations, through
+this library's own summary rather than separate arithmetic, so the numbers above and
+the numbers the tool prints cannot drift apart.
+
+### What this is not
+
+It is one protocol, one client, at one commit, cut at event boundaries. Cutting inside
+a frame — mid-UTF-8 sequence, mid-SSE frame, mid tool-call JSON — is not covered yet
+and is where the more interesting failures probably live. Other frameworks have no
+validator of their own to use as an oracle, so for those the harness would have to
+assert its own expectation, which is weaker evidence.
+
+Findings belong in the tracker of the project that carries them, not in a scoreboard
+kept outside it.
+
+## Status
+
+Early. The core loop, the byte-level severing and the summary are tested
+(`npm test`, 21 tests, no dependencies). The API will change.
+
+## Licence
+
+MIT © Redouane (ROSH Company Labs)

package/package.json

@@ -0,0 +1,37 @@
+{
+  "name": "cut-at-k",
+  "version": "0.1.0",
+  "description": "Cut a stream at every point and check what the consumer is left with",
+  "type": "module",
+  "exports": {
+    ".": "./src/index.js"
+  },
+  "files": [
+    "src",
+    "results",
+    "README.md",
+    "LICENSE"
+  ],
+  "scripts": {
+    "test": "node --test"
+  },
+  "keywords": [
+    "streaming",
+    "truncation",
+    "conformance",
+    "sse",
+    "testing",
+    "agent"
+  ],
+  "license": "MIT",
+  "author": {
+    "name": "Redouane - ROSH Company Labs"
+  },
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/roshcompanylabs/cut-at-k.git"
+  },
+  "engines": {
+    "node": ">=18"
+  }
+}

package/results/record.mjs

@@ -0,0 +1,144 @@
+/**
+ * The recording step for the AG-UI worked example.
+ *
+ * Replays every AG-UI conformance fixture through a real HttpAgent over real
+ * HTTP/SSE — once whole, then once for every prefix — and writes the raw
+ * observations as the JSON that results/run-ag-ui.mjs consumes.
+ *
+ *   node results/record.mjs <ag-ui>/spec/1.0/conformance/streams results/sever-results.json
+ *   node results/run-ag-ui.mjs results/sever-results.json
+ *
+ * Only this file needs @ag-ui/client. Its output is committed beside it, so the
+ * numbers in the README can be re-derived without installing their client at all.
+ */
+import { createServer } from 'node:http';
+import { readFileSync, readdirSync, writeFileSync } from 'node:fs';
+import { join, basename } from 'node:path';
+import { HttpAgent } from '@ag-ui/client';
+
+const CORPUS = process.argv[2];
+const OUT = process.argv[3];
+if (!CORPUS || !OUT) {
+  console.error('usage: node record.mjs <corpus-dir> <out.json>');
+  process.exit(1);
+}
+
+const fixtures = readdirSync(CORPUS)
+  .filter((f) => f.endsWith('.json'))
+  .sort()
+  .map((f) => {
+    const j = JSON.parse(readFileSync(join(CORPUS, f), 'utf8'));
+    return { name: j.name ?? basename(f, '.json'), file: f, stream: j.stream ?? [] };
+  });
+
+const byName = new Map(fixtures.map((f) => [f.name, f]));
+
+// A real HTTP server speaking real SSE. Truncation = the response simply ends
+// after k events: no further frames, no terminal event, clean close.
+const server = createServer((req, res) => {
+  const u = new URL(req.url, 'http://127.0.0.1');
+  const fx = byName.get(u.searchParams.get('fixture'));
+  const k = Number(u.searchParams.get('k'));
+  if (!fx) {
+    res.writeHead(404).end();
+    return;
+  }
+  // Drain the request body before answering.
+  req.on('data', () => {});
+  req.on('end', () => {
+    res.writeHead(200, {
+      'content-type': 'text/event-stream',
+      'cache-control': 'no-cache',
+      connection: 'close',
+    });
+    const prefix = fx.stream.slice(0, k);
+    for (const ev of prefix) res.write(`data: ${JSON.stringify(ev)}\n\n`);
+    res.end();
+  });
+});
+
+await new Promise((r) => server.listen(0, '127.0.0.1', r));
+const port = server.address().port;
+
+function textOf(messages) {
+  return messages
+    .map((m) => {
+      const c = typeof m.content === 'string' ? m.content : JSON.stringify(m.content ?? null);
+      const tc = (m.toolCalls ?? []).map((t) => `${t.function?.name}(${t.function?.arguments})`).join(',');
+      return `${m.role}:${c}${tc ? `|${tc}` : ''}`;
+    })
+    .join('\u0001');
+}
+
+function contentOnly(messages) {
+  return messages
+    .map((m) => (typeof m.content === 'string' ? m.content : JSON.stringify(m.content ?? null)))
+    .join('\u0001');
+}
+
+async function replay(fixture, k) {
+  const delivered = [];
+  const agent = new HttpAgent({
+    url: `http://127.0.0.1:${port}/?fixture=${encodeURIComponent(fixture)}&k=${k}`,
+  });
+  try {
+    const settled = await agent.runAgent({}, { onEvent: ({ event }) => delivered.push(event.type) });
+    const msgs = settled.newMessages ?? [];
+    return {
+      delivered: delivered.join(' '),
+      settled: JSON.stringify({ result: settled.result ?? null, newMessages: msgs.length }),
+      messagesCount: msgs.length,
+      messagesText: textOf(msgs),
+      messagesContentOnly: contentOnly(msgs),
+      messagesJson: JSON.stringify(msgs),
+    };
+  } catch (err) {
+    const m = err instanceof Error ? err.message : String(err);
+    return {
+      threw: m,
+      delivered: delivered.join(' '),
+      // Alternative recording discipline: an errored run recorded as a settled
+      // value rather than a throw, which is what their table's zero throws implies.
+      errSettled: JSON.stringify({ error: m }),
+    };
+  }
+}
+
+const rows = [];
+for (const fx of fixtures) {
+  const of = fx.stream.length;
+  if (of < 2) continue; // no cut exists: k runs 1..of-1
+  const whole = await replay(fx.name, of);
+  for (let k = 1; k < of; k++) {
+    const cut = await replay(fx.name, k);
+    rows.push({
+      fixture: fx.name,
+      k,
+      of,
+      threw: cut.threw,
+      cutDelivered: cut.delivered,
+      cutSettled: cut.settled,
+      cutMessages: cut.messagesText,
+      cutMessagesCount: cut.messagesCount,
+      cutMessagesContentOnly: cut.messagesContentOnly,
+      cutMessagesJson: cut.messagesJson,
+      cutErrSettled: cut.errSettled,
+      wholeDelivered: whole.delivered,
+      wholeSettled: whole.settled,
+      wholeMessages: whole.messagesText,
+      wholeMessagesCount: whole.messagesCount,
+      wholeMessagesContentOnly: whole.messagesContentOnly,
+      wholeMessagesJson: whole.messagesJson,
+      wholeErrSettled: whole.errSettled,
+      wholeThrew: whole.threw,
+    });
+  }
+  process.stderr.write('.');
+}
+process.stderr.write('\n');
+
+writeFileSync(OUT, JSON.stringify({ rows }, null, 1));
+console.log(`fixtures seen: ${fixtures.length}`);
+console.log(`fixtures replayed (>=2 events): ${new Set(rows.map((r) => r.fixture)).size}`);
+console.log(`rows (cuts): ${rows.length}`);
+await new Promise((r) => server.close(r));

package/results/run-ag-ui.mjs

@@ -0,0 +1,84 @@
+/**
+ * The worked example: AG-UI's own conformance corpus, cut at every event.
+ *
+ * This reads the raw observations recorded by replaying each fixture through a
+ * real HttpAgent over real HTTP/SSE, and runs them through this library's own
+ * summary so the numbers in the README come from the tool rather than from a
+ * separate piece of arithmetic.
+ *
+ *   node results/run-ag-ui.mjs <path-to-sever-results.json>
+ *
+ * The recording step lives in the AG-UI checkout because it needs their client;
+ * the file it writes is the input here.
+ */
+import { readFileSync } from 'node:fs';
+import { severAtEveryPoint } from '../src/sever.js';
+import { summarise, format } from '../src/report.js';
+
+const input = process.argv[2];
+if (!input) {
+  console.error('usage: node results/run-ag-ui.mjs <sever-results.json>');
+  process.exit(1);
+}
+
+const raw = JSON.parse(readFileSync(input, 'utf8'));
+
+// Rebuild one report per fixture from the recorded observations. Replaying is
+// already done; this replays from the recording so the analysis is the
+// library's, not a bespoke script's.
+const byFixture = new Map();
+for (const row of raw.rows) {
+  if (row.k < 0) continue;
+  if (!byFixture.has(row.fixture)) byFixture.set(row.fixture, []);
+  byFixture.get(row.fixture).push(row);
+}
+
+const reports = [];
+for (const [fixture, rows] of byFixture) {
+  rows.sort((a, b) => a.k - b.k);
+  const of = rows[0].of;
+  const whole = {
+    delivered: String(rows[0].wholeDelivered).split(' ').filter(Boolean),
+    settled: rows[0].wholeSettled,
+    messages: rows[0].wholeMessages,
+  };
+  const recorded = new Map(rows.map((r) => [r.k, r]));
+
+  reports.push(
+    await severAtEveryPoint({
+      events: Array.from({ length: of }, (_, i) => i),
+      label: fixture,
+      replay: async (prefix) => {
+        if (prefix.length === of) return whole;
+        const r = recorded.get(prefix.length);
+        if (!r) throw new Error(`no recording for k=${prefix.length}`);
+        if (r.threw) throw new Error(r.threw);
+        return {
+          delivered: String(r.cutDelivered).split(' ').filter(Boolean),
+          settled: r.cutSettled,
+          messages: r.cutMessages,
+        };
+      },
+    }),
+  );
+}
+
+// Fixtures whose whole stream is invalid on purpose. Cutting one before its
+// deliberate error removes the error, so every comparison against it is noise.
+const summary = summarise(reports, {
+  ignore: (label) => label.includes('fatal'),
+  // Did the cut actually remove something the application would have seen?
+  // Cutting after the last content event and before some trailing bookkeeping
+  // removes nothing, and a consumer reporting the same for those is right.
+  lostContent: (cut, whole) => cut.messages !== whole.messages,
+});
+
+console.log(format(summary));
+console.log('\nstreams with at least one silent cut:');
+for (const [label, n] of Object.entries(summary.byStream).sort((a, b) => b[1] - a[1]).slice(0, 10)) {
+  console.log(`  ${String(n).padStart(3)}  ${label}`);
+}
+console.log('\nlast event delivered on a silent cut:');
+for (const [ev, n] of Object.entries(summary.byTerminal).sort((a, b) => b[1] - a[1])) {
+  console.log(`  ${String(n).padStart(3)}  ${ev}`);
+}