@sema-agent/core 7.38.2 → 7.38.3

package/CHANGELOG.md

@@ -1,3 +1,19 @@
+## 7.38.3 — 2026-10-02
+
+**Hard-breaking changes:** none.
+
+**Closed sets:** none. **Package exports:** +0 / −0.
+
+### The JSONL tail repair reads the file's last segment, not the whole file (#1286; behavior (widen: a store file of 2 GiB or more opens and takes appends; an unterminated last segment past the engine's string bound no longer throws at open); +0 exports) (@server @test)
+
+- **Before (v7.38.2).** `AppendLog` repaired a JSONL file's tail at three stations (open, the lazy reopen after a compaction swap, the append door when the file had moved off a record boundary) by reading the whole file to look at its last line. A file of 2 GiB or more could not be opened at all (`ERR_FS_FILE_TOO_LARGE`), so an append-only dataset that grows without bound (an outcomes log) refused every write once it crossed that size; an unterminated last segment whose text was longer than V8's maximum string (`buffer.constants.MAX_STRING_LENGTH`, 536,870,888 characters; for ASCII that is 512 MiB) threw `ERR_STRING_TOO_LONG` at open.
+- **Now.** One function, `readTailSegment(fd, size)` (`src/stores/file/fs-atomic.ts`), walks back from the end in 64 KiB blocks to the last newline and returns the segment after it; the three stations read their tail through it, so a repair costs the length of the last line, not of the file. What the segment IS stays `jsonlTextFormOf`'s call, unchanged: a whole record is ended with its newline, a half record or a non-object is cut back to the last newline, a file that ends on a newline is not touched. A segment longer than that bound, MEASURED IN BYTES (536,870,888 bytes), is not judged: it is ended as a line (one `\n` appended, its bytes untouched, never cut) and one `console.warn` line names the file; the next append stands on its own line behind it, where leaving it open would have glued the next acknowledged record onto it.
+- **Who is hurt.** Nobody who worked before: every file that opened still gets the same repair, byte for byte, with one exception: an unterminated last segment of more than 536,870,888 BYTES of multi-byte text (up to about three times that) could still be decoded before, and a torn one was cut back; it is now ended as a line unjudged like an ASCII one, so a torn one stays as an unparseable line (a reader skips it) instead of being cut. A file that used to refuse to open (2 GiB or more, or an ASCII last segment above the string bound) now opens and appends. The repair is only half of what such a file needs: a store that replays its whole file when it boots (`readJsonlRecords`, still one whole-file read) still cannot read a file of 2 GiB or more, nor a line above the string bound; only a consumer that appends without replaying (a sink) benefits end to end. That read is tracked as its own backlog row (1287), not changed here.
+- **Compensation.** None needed; there is no switch for the old whole read.
+- **Known limit (KL-1286-a).** A last segment longer than the string bound is ended as a line and not judged: if it was a real record, it stays on disk and is not read by the repair (a reader meets it as an ordinary line; what the whole-file read does with it is backlog row 1287's). The bound is the runtime's `buffer.constants.MAX_STRING_LENGTH` applied to the segment's length in BYTES: a segment within it always decodes (a byte is at most one UTF-16 unit), one past it might still decode when it is multi-byte text, and is left unjudged either way (judging by decoded length would mean reading and decoding up to three times the bound to find out). A record that long is half a gigabyte of text: this store's writers do not produce one in practice, so the shape is a foreign writer's or damage.
+- **Known limit (KL-1286-b).** The repair is not serialized with an appender that is writing the same file: it samples the last segment, judges it, then cuts by offset. A peer that completes the half record after the sample and before the cut has that record cut with the remnant. The window is the same read-to-cut one v7.38.2 had (it read the whole file first); it is now the length of the last line, not of the file. The directory fence (one writer per ledger directory) is what keeps a peer from being there; closing the window itself needs the repair to run under that fence.
+- Pins: `test/backlog1286-jsonl-tail-large-file.test.ts` — a 2.2 GB sparse file (nothing is written; `truncateSync` extends it) with ① a whole unterminated last line (ended, one byte added, the next append is its own line), ② a half last line / a non-object tail (cut back), ③ a clean file (not one byte moved), and the other two stations on it (lazy reopen after a swap; append door met by a peer's whole / half tail); ④ tails around read-block edges (100 KiB; every length one byte either side of one and two blocks), ⑤ a line longer than three blocks, ⑥ a file with no newline at all (small, and larger than a block), whole and torn; ⑦ a segment one byte past the string bound (ended, never cut, one warning, the next append not glued); ⑨ the repair reads at most a couple of blocks and never goes through `readFileSync` on the log; ⑪ the present form of KL-1286-b (a peer completing the half record between the read and the cut loses it); `readTailSegment` itself (segment starts, newline at a block's first / last byte, the bound judged at its exact edge). `test/backlog1278-jsonl-tail-newline.test.ts` PIN 14 (a multi-byte character straddling a read-block edge does not tear a whole record) and PINs 1–13 unchanged and green.
+
 ## 7.38.2 — 2026-10-02
 
 **Hard-breaking changes:** none.

package/dist/stores/file/fs-atomic.d.ts

@@ -133,10 +133,11 @@ export declare class AppendLog {
      * onto the remnant, so replay dropped BOTH — after that second append had returned and fsync'd. Measured:
      * records [1,2,3] + a torn remnant + appends 5,6 replayed as [1,2,3,6].
      *
-     * Cost on the healthy path is ONE `fstat` per append (the sizes match and we return); the whole-file read
-     * only happens when the file actually moved off a boundary. A legitimate concurrent appender (another
-     * process under the boot-lock handoff) leaves the file newline-terminated, so it is adopted, not truncated; a
-     * whole record it left without its newline is kept and ended, and only a partial one is cut.
+     * Cost on the healthy path is ONE `fstat` per append (the sizes match and we return); the tail read only
+     * happens when the file actually moved off a boundary, and it reads the last segment, not the file. A
+     * legitimate concurrent appender (another process under the boot-lock handoff) leaves the file
+     * newline-terminated, so it is adopted, not truncated; a whole record it left without its newline is kept
+     * and ended, and only a partial one is cut.
      */
     private repairTail;
     /**
@@ -171,6 +172,30 @@ export declare class AppendLog {
     closeForSwap(): void;
     private releaseFd;
 }
+/** Bytes read per positioned read while a file's tail is walked back to its last newline. */
+export declare const JSONL_TAIL_READ_BLOCK: number;
+/**
+ * The longest last segment (in BYTES) a repair station will judge: V8's maximum string length in UTF-16 code units
+ * (Node's `buffer.constants.MAX_STRING_LENGTH`). A segment of N bytes decodes to at most N code units, so one within
+ * the bound can never make `toString` throw. One past it MAY still decode (multi-byte text has fewer units than
+ * bytes) but is left unjudged either way: telling would mean reading and decoding up to three times the bound.
+ */
+export declare const JSONL_TAIL_SEGMENT_LIMIT: number;
+/** A file's last segment (the bytes after its last `\n`, or the whole file when it holds none) — or the fact that it is
+ *  too long to judge. `start` is the segment's offset; a file that ends on a newline has an empty segment at `size`. */
+export type JsonlTailSegment = {
+    unjudgeable: false;
+    start: number;
+    bytes: Buffer;
+} | {
+    unjudgeable: true;
+};
+/**
+ * The last segment of an open file of `size` bytes, found by walking back from the end one block at a time to the
+ * last newline: a read is bounded by the segment's length, never by the file's. The one reader of "what is the tail"
+ * for every repair station; whether the segment is a record is {@link jsonlTextFormOf}'s to say.
+ */
+export declare function readTailSegment(fd: number, size: number, limit?: number): JsonlTailSegment;
 /** Why a {@link BootLock}/{@link acquireStoreDirLock} acquisition was refused, as a branchable code. */
 export type FileStoreLockErrorCode = 
 /** A LIVE process (this one or another) already holds the directory. Steady state: it ends when that

package/dist/stores/file/fs-atomic.js

@@ -1,4 +1,5 @@
 import { appendFileSync, closeSync, constants as FS, existsSync, fstatSync, ftruncateSync, linkSync, mkdirSync, openSync, readFileSync, readSync, renameSync, truncateSync, unlinkSync, writeFileSync, writeSync, fsyncSync, } from "node:fs";
+import { constants as bufferConstants } from "node:buffer";
 import { homedir, hostname, uptime } from "node:os";
 import { hrtime } from "node:process";
 import { basename, dirname, join, resolve as resolvePath } from "node:path";
@@ -285,23 +286,71 @@ export class AppendLog {
         }
     }
 }
+export const JSONL_TAIL_READ_BLOCK = 64 * 1024;
+export const JSONL_TAIL_SEGMENT_LIMIT = bufferConstants.MAX_STRING_LENGTH;
+function readExactly(fd, into, length, position) {
+    let got = 0;
+    while (got < length) {
+        const n = readSync(fd, into, got, length - got, position + got);
+        if (n <= 0)
+            throw new Error(`jsonl tail: the file ended ${length - got} bytes early while its last line was read — it shrank under the reader`);
+        got += n;
+    }
+}
+export function readTailSegment(fd, size, limit = JSONL_TAIL_SEGMENT_LIMIT) {
+    const block = Buffer.allocUnsafe(Math.min(JSONL_TAIL_READ_BLOCK, size));
+    let start = 0;
+    for (let end = size; end > 0;) {
+        const from = Math.max(0, end - JSONL_TAIL_READ_BLOCK);
+        readExactly(fd, block, end - from, from);
+        const newline = block.subarray(0, end - from).lastIndexOf(0x0a);
+        if (newline >= 0) {
+            start = from + newline + 1;
+            break;
+        }
+        end = from;
+        if (size - end > limit)
+            return { unjudgeable: true };
+    }
+    if (size - start > limit)
+        return { unjudgeable: true };
+    const bytes = Buffer.allocUnsafe(size - start);
+    readExactly(fd, bytes, bytes.length, start);
+    return { unjudgeable: false, start, bytes };
+}
 function repairJsonlTail(path) {
-    let bytes;
+    let fd;
     try {
-        bytes = readFileSync(path);
+        fd = openSync(path, "r");
     }
     catch (err) {
         if (err.code === "ENOENT")
             return;
         throw err;
     }
-    if (bytes.length === 0 || bytes[bytes.length - 1] === 0x0a)
+    let segment;
+    try {
+        segment = readTailSegment(fd, fstatSync(fd).size);
+    }
+    finally {
+        closeSync(fd);
+    }
+    if (segment.unjudgeable) {
+        appendFileSync(path, "\n");
+        try {
+            console.warn(`[sema] file store: a tail segment larger than the judgeable bound (${JSONL_TAIL_SEGMENT_LIMIT} bytes) was ended as a line, not judged — ` +
+                `${path}. Its bytes are untouched; a reader meets it as an ordinary line.`);
+        }
+        catch {
+        }
+        return;
+    }
+    if (segment.bytes.length === 0)
         return;
-    const segmentStart = bytes.lastIndexOf(0x0a) + 1;
-    if (jsonlTextFormOf(bytes.toString("utf8", segmentStart)).form === "record")
+    if (jsonlTextFormOf(segment.bytes.toString("utf8")).form === "record")
         appendFileSync(path, "\n");
     else
-        truncateSync(path, segmentStart);
+        truncateSync(path, segment.start);
 }
 export class FileStoreLockError extends Error {
     code;

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@sema-agent/core",
-  "version": "7.38.2",
+  "version": "7.38.3",
   "description": "Stateless, task-oriented AI agent core",
   "type": "module",
   "license": "BUSL-1.1",

package/wire-changes/core@7.38.2.json

@@ -2,7 +2,7 @@
   "repo": "core",
   "version": "7.38.2",
   "from": "a8493dd6305fe6f8b3fea1174b56c30667f3ff1b",
-  "to": "201b3b096bc8086ff09caca61683c69acdc8bcbe",
+  "to": "25b09b0232b290da289a6d9697f0d3b72826713c",
   "added": [],
   "removed": [],
   "changed": [],
@@ -14,7 +14,25 @@
   "typeKeysFlattenedCount": 220,
   "declarationMoves": [],
   "mintMoves": [],
-  "criteriaObserves": [],
+  "criteriaObserves": [
+    {
+      "kind": "typeMember",
+      "name": "AppendLog.append",
+      "assertions": [
+        "G1.b",
+        "G1.c",
+        "G1.d"
+      ]
+    },
+    {
+      "kind": "typeMember",
+      "name": "CheckpointStore.get",
+      "assertions": [
+        "G1.a",
+        "G1.c"
+      ]
+    }
+  ],
   "consumersHint": {
     "cli": [],
     "server": [],

package/wire-changes/core@7.38.3.json

@@ -0,0 +1,36 @@
+{
+  "repo": "core",
+  "version": "7.38.3",
+  "from": "25b09b0232b290da289a6d9697f0d3b72826713c",
+  "to": "e0ab70decfe37f3dcdcb4aff4f8e694df14a7efe",
+  "added": [],
+  "removed": [],
+  "changed": [],
+  "narrowed": [],
+  "closedSetMembers": [],
+  "closedSetsUnreadable": [],
+  "typeKeys": [],
+  "typeKeysUnreadableArms": [],
+  "typeKeysFlattenedCount": 220,
+  "declarationMoves": [],
+  "mintMoves": [],
+  "criteriaObserves": [
+    {
+      "kind": "typeMember",
+      "name": "AppendLog.append",
+      "assertions": [
+        "G1.a",
+        "G1.b",
+        "G1.c"
+      ]
+    }
+  ],
+  "consumersHint": {
+    "cli": [],
+    "server": [],
+    "sdk": [],
+    "note": "scanned cli@64a76326 server@f4ce4085 sdk@2756723 — a name listed is MENTIONED there today, which is evidence of consumption, not proof of it"
+  },
+  "breakingCount": 0,
+  "breakingItems": []
+}