cut-at-k 0.1.0 → 0.1.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +41 -12
- package/package.json +2 -1
- package/results/record.mjs +11 -1
- package/results/sever-results.json +20 -16
package/README.md
CHANGED
|
@@ -15,13 +15,11 @@ The second one is where things are usually wrong, and it is quiet when it is wro
|
|
|
15
15
|
which is why a loop is better at finding it than reading the code.
|
|
16
16
|
|
|
17
17
|
```bash
|
|
18
|
-
|
|
18
|
+
npm install cut-at-k
|
|
19
19
|
```
|
|
20
20
|
|
|
21
|
-
Not on npm yet. Zero dependencies, so a clone is the whole install.
|
|
22
|
-
|
|
23
21
|
```js
|
|
24
|
-
import { severAtEveryPoint, summarise, format } from '
|
|
22
|
+
import { severAtEveryPoint, summarise, format } from 'cut-at-k';
|
|
25
23
|
|
|
26
24
|
const report = await severAtEveryPoint({
|
|
27
25
|
events, // the whole stream, in order
|
|
@@ -63,7 +61,8 @@ problem on a corpus where 54 were. A tool that cries wolf is worse than no tool.
|
|
|
63
61
|
[AG-UI](https://github.com/ag-ui-protocol/ag-ui) ships 68 conformance fixtures — real
|
|
64
62
|
event streams, written by the protocol's authors, with the outcome each one requires.
|
|
65
63
|
Two are a single event long and have no cut point, so 66 were replayed: each through a
|
|
66
|
-
real `HttpAgent` over real HTTP/SSE
|
|
64
|
+
real `HttpAgent` over real HTTP/SSE — `@ag-ui/client@1.0.0` — whole and then cut at every
|
|
65
|
+
event boundary. Both
|
|
67
66
|
steps are in `results/` and the recording is committed, so the block below is one
|
|
68
67
|
command away:
|
|
69
68
|
|
|
@@ -91,16 +90,43 @@ after k events with no further frames and no terminal event. From
|
|
|
91
90
|
> having succeeded.
|
|
92
91
|
|
|
93
92
|
The smallest case, from their `conformant-run-is-quiet` fixture cut after its fourth
|
|
94
|
-
event:
|
|
93
|
+
event. What `runAgent()` actually resolves to, verbatim:
|
|
95
94
|
|
|
96
95
|
```
|
|
97
|
-
truncated
|
|
98
|
-
|
|
99
|
-
completed
|
|
96
|
+
truncated events RUN_STARTED STEP_STARTED TEXT_MESSAGE_START TEXT_MESSAGE_CONTENT
|
|
97
|
+
resolved {"newMessages":[{"id":"m-1","role":"assistant","content":"Hello, "}]}
|
|
98
|
+
completed resolved {"newMessages":[{"id":"m-1","role":"assistant","content":"Hello, world."}]}
|
|
100
99
|
```
|
|
101
100
|
|
|
102
|
-
|
|
103
|
-
|
|
101
|
+
Both calls resolve. Neither rejects, and `RunAgentResult` has no field that says which
|
|
102
|
+
one was cut short — its `result` member is `undefined` in both, so `JSON.stringify`
|
|
103
|
+
omits it entirely. The two values are not identical: the truncated one carries less
|
|
104
|
+
text. But the caller has nothing to compare it against, and the message count is 1
|
|
105
|
+
either way, so a caller that checks whether the call succeeded, or how many messages
|
|
106
|
+
came back, is told the same thing by a run that finished and a run that stopped
|
|
107
|
+
halfway through a sentence.
|
|
108
|
+
|
|
109
|
+
Be precise about the limit of that, because it matters: a subscriber **can** detect
|
|
110
|
+
this today. `onRunFinishedEvent` fires on the completed run and not on the truncated
|
|
111
|
+
one, while `onRunFinalized` fires on both. So the claim is not that the client hides
|
|
112
|
+
the truncation everywhere — it is that the awaited call does not surface it, which is
|
|
113
|
+
the channel most callers use.
|
|
114
|
+
|
|
115
|
+
### This is already known, and someone has written the fix
|
|
116
|
+
|
|
117
|
+
None of the above is a discovery. It was reported on 2026-08-03 as
|
|
118
|
+
[ag-ui-protocol/ag-ui#2300](https://github.com/ag-ui-protocol/ag-ui/issues/2300) —
|
|
119
|
+
"a stream that ends without RUN_FINISHED/RUN_ERROR resolves as success" — and
|
|
120
|
+
[#2354](https://github.com/ag-ui-protocol/ag-ui/pull/2354) has been open since
|
|
121
|
+
2026-08-07 with the fix: `verifyEvents` asserting that a terminal event arrived, so a
|
|
122
|
+
stream that just stops becomes an error instead of a silent success. Both were open at
|
|
123
|
+
the time of writing.
|
|
124
|
+
|
|
125
|
+
That is the argument for this tool rather than against it. The bug was found once, by
|
|
126
|
+
hand, on one stream. Running the same corpus through this loop surfaces **54** cuts of
|
|
127
|
+
that shape across **34** streams without anyone guessing where to look — and when
|
|
128
|
+
#2354 merges, re-running the recording is how you find out which of the 54 it actually
|
|
129
|
+
fixed.
|
|
104
130
|
|
|
105
131
|
`results/run-ag-ui.mjs` reproduces the table from the recorded observations, through
|
|
106
132
|
this library's own summary rather than separate arithmetic, so the numbers above and
|
|
@@ -108,7 +134,10 @@ the numbers the tool prints cannot drift apart.
|
|
|
108
134
|
|
|
109
135
|
### What this is not
|
|
110
136
|
|
|
111
|
-
It is one protocol,
|
|
137
|
+
It is one protocol, cut at event boundaries, and one client at one version:
|
|
138
|
+
`@ag-ui/client@1.0.0`. The figures move with the client, so the version is part of the
|
|
139
|
+
result rather than a footnote — replaying the same corpus through `0.0.58` gives
|
|
140
|
+
141/49/47/31, because that older client rejects four fixtures this one parses. Cutting inside
|
|
112
141
|
a frame — mid-UTF-8 sequence, mid-SSE frame, mid tool-call JSON — is not covered yet
|
|
113
142
|
and is where the more interesting failures probably live. Other frameworks have no
|
|
114
143
|
validator of their own to use as an oracle, so for those the harness would have to
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "cut-at-k",
|
|
3
|
-
"version": "0.1.
|
|
3
|
+
"version": "0.1.2",
|
|
4
4
|
"description": "Cut a stream at every point and check what the consumer is left with",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"exports": {
|
|
@@ -31,6 +31,7 @@
|
|
|
31
31
|
"type": "git",
|
|
32
32
|
"url": "git+https://github.com/roshcompanylabs/cut-at-k.git"
|
|
33
33
|
},
|
|
34
|
+
"funding": "https://github.com/sponsors/roshcompanylabs",
|
|
34
35
|
"engines": {
|
|
35
36
|
"node": ">=18"
|
|
36
37
|
}
|
package/results/record.mjs
CHANGED
|
@@ -14,6 +14,7 @@
|
|
|
14
14
|
import { createServer } from 'node:http';
|
|
15
15
|
import { readFileSync, readdirSync, writeFileSync } from 'node:fs';
|
|
16
16
|
import { join, basename } from 'node:path';
|
|
17
|
+
import { createRequire } from 'node:module';
|
|
17
18
|
import { HttpAgent } from '@ag-ui/client';
|
|
18
19
|
|
|
19
20
|
const CORPUS = process.argv[2];
|
|
@@ -137,7 +138,16 @@ for (const fx of fixtures) {
|
|
|
137
138
|
}
|
|
138
139
|
process.stderr.write('\n');
|
|
139
140
|
|
|
140
|
-
|
|
141
|
+
// Which client produced this. Without it the figures cannot be re-recorded by anyone,
|
|
142
|
+
// which is the whole point of committing the observations rather than the numbers.
|
|
143
|
+
const clientVersion = JSON.parse(
|
|
144
|
+
readFileSync(createRequire(import.meta.url).resolve('@ag-ui/client/package.json'), 'utf8'),
|
|
145
|
+
).version;
|
|
146
|
+
|
|
147
|
+
writeFileSync(
|
|
148
|
+
OUT,
|
|
149
|
+
JSON.stringify({ client: { name: '@ag-ui/client', version: clientVersion }, rows }, null, 1),
|
|
150
|
+
);
|
|
141
151
|
console.log(`fixtures seen: ${fixtures.length}`);
|
|
142
152
|
console.log(`fixtures replayed (>=2 events): ${new Set(rows.map((r) => r.fixture)).size}`);
|
|
143
153
|
console.log(`rows (cuts): ${rows.length}`);
|
|
@@ -1,4 +1,8 @@
|
|
|
1
1
|
{
|
|
2
|
+
"client": {
|
|
3
|
+
"name": "@ag-ui/client",
|
|
4
|
+
"version": "1.0.0"
|
|
5
|
+
},
|
|
2
6
|
"rows": [
|
|
3
7
|
{
|
|
4
8
|
"fixture": "activity-delta-missing-target-tolerated",
|
|
@@ -817,7 +821,7 @@
|
|
|
817
821
|
"wholeMessages": "reasoning:Let me check.\u0001assistant:Checked.",
|
|
818
822
|
"wholeMessagesCount": 2,
|
|
819
823
|
"wholeMessagesContentOnly": "Let me check.\u0001Checked.",
|
|
820
|
-
"wholeMessagesJson": "[{\"id\":\"
|
|
824
|
+
"wholeMessagesJson": "[{\"id\":\"91d1f950-44ea-4c5b-af92-a5639d912c4b\",\"role\":\"reasoning\",\"content\":\"Let me check.\"},{\"id\":\"era45-m1\",\"role\":\"assistant\",\"content\":\"Checked.\"}]"
|
|
821
825
|
},
|
|
822
826
|
{
|
|
823
827
|
"fixture": "era-0-0-45-thinking-translated",
|
|
@@ -834,7 +838,7 @@
|
|
|
834
838
|
"wholeMessages": "reasoning:Let me check.\u0001assistant:Checked.",
|
|
835
839
|
"wholeMessagesCount": 2,
|
|
836
840
|
"wholeMessagesContentOnly": "Let me check.\u0001Checked.",
|
|
837
|
-
"wholeMessagesJson": "[{\"id\":\"
|
|
841
|
+
"wholeMessagesJson": "[{\"id\":\"91d1f950-44ea-4c5b-af92-a5639d912c4b\",\"role\":\"reasoning\",\"content\":\"Let me check.\"},{\"id\":\"era45-m1\",\"role\":\"assistant\",\"content\":\"Checked.\"}]"
|
|
838
842
|
},
|
|
839
843
|
{
|
|
840
844
|
"fixture": "era-0-0-45-thinking-translated",
|
|
@@ -845,13 +849,13 @@
|
|
|
845
849
|
"cutMessages": "reasoning:",
|
|
846
850
|
"cutMessagesCount": 1,
|
|
847
851
|
"cutMessagesContentOnly": "",
|
|
848
|
-
"cutMessagesJson": "[{\"id\":\"
|
|
852
|
+
"cutMessagesJson": "[{\"id\":\"982cc4a1-d6bb-4e41-aee4-142c44a626dc\",\"role\":\"reasoning\",\"content\":\"\"}]",
|
|
849
853
|
"wholeDelivered": "RUN_STARTED REASONING_START REASONING_MESSAGE_START REASONING_MESSAGE_CONTENT REASONING_MESSAGE_END REASONING_END TEXT_MESSAGE_START TEXT_MESSAGE_CONTENT TEXT_MESSAGE_END RUN_FINISHED",
|
|
850
854
|
"wholeSettled": "{\"result\":null,\"newMessages\":2}",
|
|
851
855
|
"wholeMessages": "reasoning:Let me check.\u0001assistant:Checked.",
|
|
852
856
|
"wholeMessagesCount": 2,
|
|
853
857
|
"wholeMessagesContentOnly": "Let me check.\u0001Checked.",
|
|
854
|
-
"wholeMessagesJson": "[{\"id\":\"
|
|
858
|
+
"wholeMessagesJson": "[{\"id\":\"91d1f950-44ea-4c5b-af92-a5639d912c4b\",\"role\":\"reasoning\",\"content\":\"Let me check.\"},{\"id\":\"era45-m1\",\"role\":\"assistant\",\"content\":\"Checked.\"}]"
|
|
855
859
|
},
|
|
856
860
|
{
|
|
857
861
|
"fixture": "era-0-0-45-thinking-translated",
|
|
@@ -862,13 +866,13 @@
|
|
|
862
866
|
"cutMessages": "reasoning:Let me check.",
|
|
863
867
|
"cutMessagesCount": 1,
|
|
864
868
|
"cutMessagesContentOnly": "Let me check.",
|
|
865
|
-
"cutMessagesJson": "[{\"id\":\"
|
|
869
|
+
"cutMessagesJson": "[{\"id\":\"bbf406ed-71b2-4d36-8598-63068d7c4d61\",\"role\":\"reasoning\",\"content\":\"Let me check.\"}]",
|
|
866
870
|
"wholeDelivered": "RUN_STARTED REASONING_START REASONING_MESSAGE_START REASONING_MESSAGE_CONTENT REASONING_MESSAGE_END REASONING_END TEXT_MESSAGE_START TEXT_MESSAGE_CONTENT TEXT_MESSAGE_END RUN_FINISHED",
|
|
867
871
|
"wholeSettled": "{\"result\":null,\"newMessages\":2}",
|
|
868
872
|
"wholeMessages": "reasoning:Let me check.\u0001assistant:Checked.",
|
|
869
873
|
"wholeMessagesCount": 2,
|
|
870
874
|
"wholeMessagesContentOnly": "Let me check.\u0001Checked.",
|
|
871
|
-
"wholeMessagesJson": "[{\"id\":\"
|
|
875
|
+
"wholeMessagesJson": "[{\"id\":\"91d1f950-44ea-4c5b-af92-a5639d912c4b\",\"role\":\"reasoning\",\"content\":\"Let me check.\"},{\"id\":\"era45-m1\",\"role\":\"assistant\",\"content\":\"Checked.\"}]"
|
|
872
876
|
},
|
|
873
877
|
{
|
|
874
878
|
"fixture": "era-0-0-45-thinking-translated",
|
|
@@ -879,13 +883,13 @@
|
|
|
879
883
|
"cutMessages": "reasoning:Let me check.",
|
|
880
884
|
"cutMessagesCount": 1,
|
|
881
885
|
"cutMessagesContentOnly": "Let me check.",
|
|
882
|
-
"cutMessagesJson": "[{\"id\":\"
|
|
886
|
+
"cutMessagesJson": "[{\"id\":\"71dbfb22-7fef-42a3-b4f1-27b895858747\",\"role\":\"reasoning\",\"content\":\"Let me check.\"}]",
|
|
883
887
|
"wholeDelivered": "RUN_STARTED REASONING_START REASONING_MESSAGE_START REASONING_MESSAGE_CONTENT REASONING_MESSAGE_END REASONING_END TEXT_MESSAGE_START TEXT_MESSAGE_CONTENT TEXT_MESSAGE_END RUN_FINISHED",
|
|
884
888
|
"wholeSettled": "{\"result\":null,\"newMessages\":2}",
|
|
885
889
|
"wholeMessages": "reasoning:Let me check.\u0001assistant:Checked.",
|
|
886
890
|
"wholeMessagesCount": 2,
|
|
887
891
|
"wholeMessagesContentOnly": "Let me check.\u0001Checked.",
|
|
888
|
-
"wholeMessagesJson": "[{\"id\":\"
|
|
892
|
+
"wholeMessagesJson": "[{\"id\":\"91d1f950-44ea-4c5b-af92-a5639d912c4b\",\"role\":\"reasoning\",\"content\":\"Let me check.\"},{\"id\":\"era45-m1\",\"role\":\"assistant\",\"content\":\"Checked.\"}]"
|
|
889
893
|
},
|
|
890
894
|
{
|
|
891
895
|
"fixture": "era-0-0-45-thinking-translated",
|
|
@@ -896,13 +900,13 @@
|
|
|
896
900
|
"cutMessages": "reasoning:Let me check.",
|
|
897
901
|
"cutMessagesCount": 1,
|
|
898
902
|
"cutMessagesContentOnly": "Let me check.",
|
|
899
|
-
"cutMessagesJson": "[{\"id\":\"
|
|
903
|
+
"cutMessagesJson": "[{\"id\":\"32b14f8b-b9a9-479d-87ff-eefb7794ed58\",\"role\":\"reasoning\",\"content\":\"Let me check.\"}]",
|
|
900
904
|
"wholeDelivered": "RUN_STARTED REASONING_START REASONING_MESSAGE_START REASONING_MESSAGE_CONTENT REASONING_MESSAGE_END REASONING_END TEXT_MESSAGE_START TEXT_MESSAGE_CONTENT TEXT_MESSAGE_END RUN_FINISHED",
|
|
901
905
|
"wholeSettled": "{\"result\":null,\"newMessages\":2}",
|
|
902
906
|
"wholeMessages": "reasoning:Let me check.\u0001assistant:Checked.",
|
|
903
907
|
"wholeMessagesCount": 2,
|
|
904
908
|
"wholeMessagesContentOnly": "Let me check.\u0001Checked.",
|
|
905
|
-
"wholeMessagesJson": "[{\"id\":\"
|
|
909
|
+
"wholeMessagesJson": "[{\"id\":\"91d1f950-44ea-4c5b-af92-a5639d912c4b\",\"role\":\"reasoning\",\"content\":\"Let me check.\"},{\"id\":\"era45-m1\",\"role\":\"assistant\",\"content\":\"Checked.\"}]"
|
|
906
910
|
},
|
|
907
911
|
{
|
|
908
912
|
"fixture": "era-0-0-45-thinking-translated",
|
|
@@ -913,13 +917,13 @@
|
|
|
913
917
|
"cutMessages": "reasoning:Let me check.\u0001assistant:",
|
|
914
918
|
"cutMessagesCount": 2,
|
|
915
919
|
"cutMessagesContentOnly": "Let me check.\u0001",
|
|
916
|
-
"cutMessagesJson": "[{\"id\":\"
|
|
920
|
+
"cutMessagesJson": "[{\"id\":\"517d746e-90a2-4f2b-a9ab-b67f417f1fa1\",\"role\":\"reasoning\",\"content\":\"Let me check.\"},{\"id\":\"era45-m1\",\"role\":\"assistant\",\"content\":\"\"}]",
|
|
917
921
|
"wholeDelivered": "RUN_STARTED REASONING_START REASONING_MESSAGE_START REASONING_MESSAGE_CONTENT REASONING_MESSAGE_END REASONING_END TEXT_MESSAGE_START TEXT_MESSAGE_CONTENT TEXT_MESSAGE_END RUN_FINISHED",
|
|
918
922
|
"wholeSettled": "{\"result\":null,\"newMessages\":2}",
|
|
919
923
|
"wholeMessages": "reasoning:Let me check.\u0001assistant:Checked.",
|
|
920
924
|
"wholeMessagesCount": 2,
|
|
921
925
|
"wholeMessagesContentOnly": "Let me check.\u0001Checked.",
|
|
922
|
-
"wholeMessagesJson": "[{\"id\":\"
|
|
926
|
+
"wholeMessagesJson": "[{\"id\":\"91d1f950-44ea-4c5b-af92-a5639d912c4b\",\"role\":\"reasoning\",\"content\":\"Let me check.\"},{\"id\":\"era45-m1\",\"role\":\"assistant\",\"content\":\"Checked.\"}]"
|
|
923
927
|
},
|
|
924
928
|
{
|
|
925
929
|
"fixture": "era-0-0-45-thinking-translated",
|
|
@@ -930,13 +934,13 @@
|
|
|
930
934
|
"cutMessages": "reasoning:Let me check.\u0001assistant:Checked.",
|
|
931
935
|
"cutMessagesCount": 2,
|
|
932
936
|
"cutMessagesContentOnly": "Let me check.\u0001Checked.",
|
|
933
|
-
"cutMessagesJson": "[{\"id\":\"
|
|
937
|
+
"cutMessagesJson": "[{\"id\":\"2c1bea1e-769d-4568-8925-a231053d19a5\",\"role\":\"reasoning\",\"content\":\"Let me check.\"},{\"id\":\"era45-m1\",\"role\":\"assistant\",\"content\":\"Checked.\"}]",
|
|
934
938
|
"wholeDelivered": "RUN_STARTED REASONING_START REASONING_MESSAGE_START REASONING_MESSAGE_CONTENT REASONING_MESSAGE_END REASONING_END TEXT_MESSAGE_START TEXT_MESSAGE_CONTENT TEXT_MESSAGE_END RUN_FINISHED",
|
|
935
939
|
"wholeSettled": "{\"result\":null,\"newMessages\":2}",
|
|
936
940
|
"wholeMessages": "reasoning:Let me check.\u0001assistant:Checked.",
|
|
937
941
|
"wholeMessagesCount": 2,
|
|
938
942
|
"wholeMessagesContentOnly": "Let me check.\u0001Checked.",
|
|
939
|
-
"wholeMessagesJson": "[{\"id\":\"
|
|
943
|
+
"wholeMessagesJson": "[{\"id\":\"91d1f950-44ea-4c5b-af92-a5639d912c4b\",\"role\":\"reasoning\",\"content\":\"Let me check.\"},{\"id\":\"era45-m1\",\"role\":\"assistant\",\"content\":\"Checked.\"}]"
|
|
940
944
|
},
|
|
941
945
|
{
|
|
942
946
|
"fixture": "era-0-0-45-thinking-translated",
|
|
@@ -947,13 +951,13 @@
|
|
|
947
951
|
"cutMessages": "reasoning:Let me check.\u0001assistant:Checked.",
|
|
948
952
|
"cutMessagesCount": 2,
|
|
949
953
|
"cutMessagesContentOnly": "Let me check.\u0001Checked.",
|
|
950
|
-
"cutMessagesJson": "[{\"id\":\"
|
|
954
|
+
"cutMessagesJson": "[{\"id\":\"47127635-ebf9-4ed2-8912-fa06aac11e12\",\"role\":\"reasoning\",\"content\":\"Let me check.\"},{\"id\":\"era45-m1\",\"role\":\"assistant\",\"content\":\"Checked.\"}]",
|
|
951
955
|
"wholeDelivered": "RUN_STARTED REASONING_START REASONING_MESSAGE_START REASONING_MESSAGE_CONTENT REASONING_MESSAGE_END REASONING_END TEXT_MESSAGE_START TEXT_MESSAGE_CONTENT TEXT_MESSAGE_END RUN_FINISHED",
|
|
952
956
|
"wholeSettled": "{\"result\":null,\"newMessages\":2}",
|
|
953
957
|
"wholeMessages": "reasoning:Let me check.\u0001assistant:Checked.",
|
|
954
958
|
"wholeMessagesCount": 2,
|
|
955
959
|
"wholeMessagesContentOnly": "Let me check.\u0001Checked.",
|
|
956
|
-
"wholeMessagesJson": "[{\"id\":\"
|
|
960
|
+
"wholeMessagesJson": "[{\"id\":\"91d1f950-44ea-4c5b-af92-a5639d912c4b\",\"role\":\"reasoning\",\"content\":\"Let me check.\"},{\"id\":\"era45-m1\",\"role\":\"assistant\",\"content\":\"Checked.\"}]"
|
|
957
961
|
},
|
|
958
962
|
{
|
|
959
963
|
"fixture": "era-0-0-47-upgrades-binary-content",
|