agent-sanitizer 2.29.0 → 2.30.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- package/README.md +20 -20
- package/THREAT-MODEL.md +19 -0
- package/claude-hooks/sanitize-output.mjs +17 -1
- package/package.json +1 -1
- package/src/html.mjs +6 -2
- package/src/output.mjs +11 -2
- package/src/rehydrate.mjs +333 -101
- package/src/view-map.mjs +70 -0
- package/src/warnings.mjs +14 -0
- package/types/claude-hooks/sanitize-output.d.mts +1 -0
- package/types/html.d.mts +6 -2
- package/types/rehydrate.d.mts +16 -0
- package/types/view-map.d.mts +31 -0
- package/types/warnings.d.mts +12 -0
package/README.md
CHANGED
|
@@ -53,17 +53,17 @@ Split into subpaths so the heavy HTML dependency stays opt-in. **Seam** names
|
|
|
53
53
|
the callback you inject for the agent-specific concern; `—` is a pure transform,
|
|
54
54
|
`fs (direct)` does its own file I/O instead of taking one.
|
|
55
55
|
|
|
56
|
-
| # | Import | Purpose
|
|
57
|
-
| --- | --------------- |
|
|
58
|
-
| 1 | `/invisible` | Strip zero-width, bidi, variation-selector and tag chars + ANSI/SGR escapes. Preserves ZWNJ/ZWJ for Arabic/Indic/emoji. Zero deps.
|
|
59
|
-
| 2 | `/html` | Splice out instructions hidden in comments, `display:none`, off-screen, white-on-white, `hidden`. Leaves a placeholder.
|
|
60
|
-
| 3 | `/html` | Detect exfil-shaped URLs (payloads in query/path, embedded creds, `data:`/`javascript:`, off-origin redirects). Reports only.
|
|
61
|
-
| 4 | `/confusables` | Fold look-alike glyphs in tool-call input (paths, commands) to ASCII, closing a cross-script deny-rule bypass. Gated per token, so non-Latin prose passes through unfolded.
|
|
62
|
-
| 5 | `/instructions` | Scan/auto-clean `CLAUDE.md`, `AGENTS.md`, `SKILL.md`, etc., decoding Unicode-tag + zero-width-binary payloads.
|
|
63
|
-
| 6 | `/prompt` | Classify a prompt pass / note / block on payload-capable invisible/ANSI content (inert escapes get the note).
|
|
64
|
-
| 7 | `/output` | Run Layers 1–4 over structured tool output, preserving shape. The Layer-5 slot takes a delete-only filter.
|
|
65
|
-
| 8 | `/rehydrate` | Re-anchor a model Edit
|
|
66
|
-
| — | `/view-map` | Pure offset/text machinery mapping a file's on-disk bytes ↔ the sanitized view (Layer-1 deletions, Layer-4 redactions). No I/O — consumed by `/rehydrate`.
|
|
56
|
+
| # | Import | Purpose | Seam |
|
|
57
|
+
| --- | --------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | --------------------------- |
|
|
58
|
+
| 1 | `/invisible` | Strip zero-width, bidi, variation-selector and tag chars + ANSI/SGR escapes. Preserves ZWNJ/ZWJ for Arabic/Indic/emoji. Zero deps. | — |
|
|
59
|
+
| 2 | `/html` | Splice out instructions hidden in comments, `display:none`, off-screen, white-on-white, `hidden`. Leaves a placeholder. | — |
|
|
60
|
+
| 3 | `/html` | Detect exfil-shaped URLs (payloads in query/path, embedded creds, `data:`/`javascript:`, off-origin redirects). Reports only. | — |
|
|
61
|
+
| 4 | `/confusables` | Fold look-alike glyphs in tool-call input (paths, commands) to ASCII, closing a cross-script deny-rule bypass. Gated per token, so non-Latin prose passes through unfolded. | `scan` |
|
|
62
|
+
| 5 | `/instructions` | Scan/auto-clean `CLAUDE.md`, `AGENTS.md`, `SKILL.md`, etc., decoding Unicode-tag + zero-width-binary payloads. | `fs` (direct) |
|
|
63
|
+
| 6 | `/prompt` | Classify a prompt pass / note / block on payload-capable invisible/ANSI content (inert escapes get the note). | — |
|
|
64
|
+
| 7 | `/output` | Run Layers 1–4 over structured tool output, preserving shape. The Layer-5 slot takes a delete-only filter. | `redact`, `filterInjection` |
|
|
65
|
+
| 8 | `/rehydrate` | Re-anchor a model Edit or whole-file Write composed from the _sanitized_ view back onto real bytes; gate MultiEdit on a verified view==disk; deny anything ambiguous or secret-exposing. | `io` |
|
|
66
|
+
| — | `/view-map` | Pure offset/text machinery mapping a file's on-disk bytes ↔ the sanitized view (Layer-1 deletions, Layer-4 redactions). No I/O — consumed by `/rehydrate`. | — |
|
|
67
67
|
|
|
68
68
|
See [`THREAT-MODEL.md`](./THREAT-MODEL.md) for per-vector detail.
|
|
69
69
|
|
|
@@ -377,15 +377,15 @@ Guard, Meta's Prompt Guard, Rebuff, NeMo Guardrails) and PII redactors
|
|
|
377
377
|
classifier never "sees" as suspicious because it renders as blank space or
|
|
378
378
|
doesn't render at all.
|
|
379
379
|
|
|
380
|
-
| | `agent-sanitizer`
|
|
381
|
-
| ----------------------------- |
|
|
382
|
-
| **What it catches** | Payload-capable invisible chars, ANSI/SGR, hidden HTML, confusable glyphs, exfil-shaped URLs
|
|
383
|
-
| **How it decides** | Deterministic parsing/regex over real tokenizer output—no model call
|
|
384
|
-
| **Failure mode** | Fails open on ambiguous input (see [`THREAT-MODEL.md`](./THREAT-MODEL.md)); false negative over false positive by design
|
|
385
|
-
| **Latency / infra** | Pure JS, mostly zero-dep (`/html` lazy-loads ~200 ms once)
|
|
386
|
-
| **Determinism / testability** | Exact-equality unit tests, no flakiness across runs
|
|
387
|
-
| **Reversibility** | `/rehydrate` re-anchors a model's
|
|
388
|
-
| **Non-JS support** | Same verdicts via a bundled CLI/worker—Python client included, no reimplementation
|
|
380
|
+
| | `agent-sanitizer` | Semantic guard/classifier (Lakera, Prompt Guard, Rebuff, NeMo rails) | PII redactor (Presidio) |
|
|
381
|
+
| ----------------------------- | ----------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------ |
|
|
382
|
+
| **What it catches** | Payload-capable invisible chars, ANSI/SGR, hidden HTML, confusable glyphs, exfil-shaped URLs | Malicious _intent_—jailbreaks, injected instructions, off-topic asks | Names, emails, SSNs, and other PII spans |
|
|
383
|
+
| **How it decides** | Deterministic parsing/regex over real tokenizer output—no model call | ML/LLM classification—probabilistic, needs a threshold and retuning as attacks shift | NER + pattern matching |
|
|
384
|
+
| **Failure mode** | Fails open on ambiguous input (see [`THREAT-MODEL.md`](./THREAT-MODEL.md)); false negative over false positive by design | False positives silently mangle or block legitimate prompts; false negatives are invisible until exploited | Under/over-redaction depending on locale and entity coverage |
|
|
385
|
+
| **Latency / infra** | Pure JS, mostly zero-dep (`/html` lazy-loads ~200 ms once) | Network round-trip to a hosted model, or a local model to host yourself | Local, but heavier NLP pipeline |
|
|
386
|
+
| **Determinism / testability** | Exact-equality unit tests, no flakiness across runs | Same input can classify differently across model versions | Deterministic per rule, but rule coverage varies |
|
|
387
|
+
| **Reversibility** | `/rehydrate` re-anchors a model's Edit or whole-file Write from the sanitized view back onto real bytes, denying anything ambiguous | N/A—classifiers only pass/block, they don't rewrite-and-reverse | N/A |
|
|
388
|
+
| **Non-JS support** | Same verdicts via a bundled CLI/worker—Python client included, no reimplementation | Usually a hosted API (language-agnostic) or Python-only SDK | Python-first (spaCy-based) |
|
|
389
389
|
|
|
390
390
|
These are complementary: a semantic guard for intent, Presidio for PII, and this
|
|
391
391
|
for the hidden channel both are blind to.
|
package/THREAT-MODEL.md
CHANGED
|
@@ -367,6 +367,25 @@ character-extraction oracle.
|
|
|
367
367
|
File access and the redactor are injected via `io`; the package performs no I/O
|
|
368
368
|
of its own and bundles no secret engine.
|
|
369
369
|
|
|
370
|
+
**Whole-file Writes are re-anchored too.** A model that reads a file whose
|
|
371
|
+
legitimate content includes stripped characters (ANSI-colored logs, zero-width
|
|
372
|
+
runs, a lone surrogate) and writes it back would otherwise silently persist the
|
|
373
|
+
stripped version. Every well-formed `Write` to an existing file is diffed
|
|
374
|
+
against the sanitized view by position (longest common prefix/suffix, snapped
|
|
375
|
+
off placeholder and surrogate boundaries): the unchanged regions are restored
|
|
376
|
+
to their exact on-disk bytes — stripped runs and redacted secrets included —
|
|
377
|
+
while the genuinely-changed middle keeps the model's bytes (Layer-1 strips of
|
|
378
|
+
_new_ text stay stripped; that is the sanitizer working). Each restored region
|
|
379
|
+
passes the same re-clean soundness gate as an Edit span. On gate failure the
|
|
380
|
+
outcome follows the precision doctrine: a placeholder-free region falls back to
|
|
381
|
+
the model's bytes (**fail open** — the write merely loses stripped characters,
|
|
382
|
+
exactly the pre-restoration behavior), while a placeholder-bearing region is
|
|
383
|
+
**denied** (restoring at a misattributed anchor could graft secret bytes
|
|
384
|
+
wrongly; not restoring persists placeholder text over the secret — neither open
|
|
385
|
+
option is safe). An empty view (an all-invisible file, the archetypal
|
|
386
|
+
hidden-payload artifact) is never restored: a Write there is the model
|
|
387
|
+
replacing content it was told is suspicious, not echoing it back.
|
|
388
|
+
|
|
370
389
|
`MultiEdit` is a rehydration candidate but never re-anchored: its edits apply
|
|
371
390
|
sequentially, each against the result of the previous, which the span machinery
|
|
372
391
|
(one `old_string` against one static view) cannot model. A MultiEdit against a
|
|
@@ -82,6 +82,19 @@ export const { describeRemoved, describeWarned, suppressToolOutput } = _output;
|
|
|
82
82
|
|
|
83
83
|
const HOOK_NAME = "sanitize-output";
|
|
84
84
|
|
|
85
|
+
// Model-facing warning for a reveal the persistence loop had to drop: the
|
|
86
|
+
// pre-splice text could not be re-vetted (redactor unreachable mid-run), so no
|
|
87
|
+
// sidecar was written and the "preserved for later inspection" promise the
|
|
88
|
+
// splice/withhold warnings make is NOT kept for this output. Fixed prose, no
|
|
89
|
+
// error text — the redactor runs on attacker-influenced content and this line
|
|
90
|
+
// reaches the model-facing context. Exported so tests assert it by reference.
|
|
91
|
+
// Deliberately a LOCAL constant rather than a shared engine builder alongside
|
|
92
|
+
// output.mjs's "Withheld the ${label}" template: the plugin bundle resolves
|
|
93
|
+
// the engine to the pinned registry release, so hook code cannot use a new
|
|
94
|
+
// engine export until the pin advances past it.
|
|
95
|
+
export const REVEAL_WITHHELD_WARNING =
|
|
96
|
+
"Withheld the reveal sidecar: it could not be vetted for secrets";
|
|
97
|
+
|
|
85
98
|
// Total wall-clock budget for one hook invocation's blocking daemon calls — the
|
|
86
99
|
// Layer-4 redactor — SHARED across every string leaf of the tool output. Each
|
|
87
100
|
// call is handed the budget remaining at that moment; once it is spent, a further
|
|
@@ -888,7 +901,10 @@ export async function evaluateToolOutput(input, ext = {}) {
|
|
|
888
901
|
// hidden only inside a comment reaches the redactor here for the first
|
|
889
902
|
// time (the post-splice scan never saw it). If the daemon is unreachable
|
|
890
903
|
// we must neither write that unvetted text nor suppress the already-safe
|
|
891
|
-
// primary output — drop this one convenience reveal
|
|
904
|
+
// primary output — drop this one convenience reveal, but SAY so: the
|
|
905
|
+
// splice warning has just promised the model a reveal it can Read back,
|
|
906
|
+
// and a silent drop leaves that promise dangling.
|
|
907
|
+
warnings.push(REVEAL_WITHHELD_WARNING);
|
|
892
908
|
continue;
|
|
893
909
|
}
|
|
894
910
|
const hint = persistReveal(stored);
|
package/package.json
CHANGED
|
@@ -1,6 +1,6 @@
|
|
|
1
1
|
{
|
|
2
2
|
"name": "agent-sanitizer",
|
|
3
|
-
"version": "2.
|
|
3
|
+
"version": "2.30.0",
|
|
4
4
|
"description": "Defend an agent against hidden-content injection: strip payload-capable invisible Unicode and ANSI, splice out human-invisible HTML, and flag data-exfil URLs in untrusted text before any model sees it.",
|
|
5
5
|
"type": "module",
|
|
6
6
|
"repository": {
|
package/src/html.mjs
CHANGED
|
@@ -1814,9 +1814,12 @@ export function looksLikeHtmlSource(text) {
|
|
|
1814
1814
|
* Layer 2 over web-ingress text: splice out HTML comments and hidden elements
|
|
1815
1815
|
* (placeholders mark the cuts; all other bytes are preserved verbatim) and
|
|
1816
1816
|
* count preserved scripting/resource tags for the caller's warning. Returns
|
|
1817
|
-
* null when there is nothing to strip and nothing to report.
|
|
1817
|
+
* null when there is nothing to strip and nothing to report. `unparseable` is
|
|
1818
|
+
* set (true) only on the fail-closed path below, where the whole input was
|
|
1819
|
+
* withheld behind {@link UNPARSEABLE_PLACEHOLDER} rather than spliced — the
|
|
1820
|
+
* caller's warning must describe a whole-output withhold, not a splice.
|
|
1818
1821
|
* @param {string} text
|
|
1819
|
-
* @returns {{ text: string, removed: { comments: number, hidden: number }, warned: { tags: Record<string, number>, dataSrc: number } } | null}
|
|
1822
|
+
* @returns {{ text: string, removed: { comments: number, hidden: number }, warned: { tags: Record<string, number>, dataSrc: number }, unparseable?: true } | null}
|
|
1820
1823
|
*/
|
|
1821
1824
|
export function sanitizeHtml(text) {
|
|
1822
1825
|
if (!HTML_TAG_PRESENT.test(text)) return null;
|
|
@@ -1836,6 +1839,7 @@ export function sanitizeHtml(text) {
|
|
|
1836
1839
|
text: UNPARSEABLE_PLACEHOLDER,
|
|
1837
1840
|
removed: { comments: 0, hidden: 1 },
|
|
1838
1841
|
warned: newWarned(),
|
|
1842
|
+
unparseable: true,
|
|
1839
1843
|
};
|
|
1840
1844
|
}
|
|
1841
1845
|
const { ranges, warned } = scan;
|
package/src/output.mjs
CHANGED
|
@@ -39,6 +39,7 @@ import {
|
|
|
39
39
|
import {
|
|
40
40
|
describeExfil,
|
|
41
41
|
describeHtmlSanitized,
|
|
42
|
+
HTML_UNPARSEABLE_WARNING,
|
|
42
43
|
describeWarned,
|
|
43
44
|
LONE_SURROGATE_WARNING,
|
|
44
45
|
} from "./warnings.mjs";
|
|
@@ -429,8 +430,16 @@ async function applyMarkdownPipeline(state, { html, exfilScan }) {
|
|
|
429
430
|
// A WARNING: these bytes were invisible to a human reading the rendered
|
|
430
431
|
// page and are now gone from the model's view too — the exact shape of
|
|
431
432
|
// a hidden-instruction payload, and the model cannot check what it was
|
|
432
|
-
// without the reveal sidecar.
|
|
433
|
-
|
|
433
|
+
// without the reveal sidecar. The unparseable fail-closed path withheld
|
|
434
|
+
// the WHOLE output, not a spliced span, so it gets its own sentence
|
|
435
|
+
// rather than a misleading "1 hidden element(s) replaced".
|
|
436
|
+
state.findings.push(
|
|
437
|
+
warning(
|
|
438
|
+
layer2.unparseable
|
|
439
|
+
? HTML_UNPARSEABLE_WARNING
|
|
440
|
+
: describeHtmlSanitized(layer2.removed),
|
|
441
|
+
),
|
|
442
|
+
);
|
|
434
443
|
}
|
|
435
444
|
// A NOTE: nothing was removed and nothing was hidden. This line says "the
|
|
436
445
|
// page had scripts, treat their contents as data", which is true of nearly
|
package/src/rehydrate.mjs
CHANGED
|
@@ -7,8 +7,9 @@
|
|
|
7
7
|
* characters, and secret redaction replaces secrets with [REDACTED…]
|
|
8
8
|
* placeholders. An Edit whose old_string was copied from that view then fails
|
|
9
9
|
* exact-match against the real file, and a whole-file Write would persist
|
|
10
|
-
* placeholder text over the real secret
|
|
11
|
-
*
|
|
10
|
+
* placeholder text over the real secret — or silently drop the stripped
|
|
11
|
+
* characters from every region it faithfully echoed back. This module closes
|
|
12
|
+
* the loop without ever showing the model a secret: it re-derives the sanitized view of the
|
|
12
13
|
* target file (the shared {@link applyLayer1}, then the injected redactor's
|
|
13
14
|
* map mode), locates the model's old_string in that view, and maps it
|
|
14
15
|
* span-exact back to the on-disk bytes — across both placeholder expansion and
|
|
@@ -65,6 +66,7 @@ import {
|
|
|
65
66
|
makeFileView,
|
|
66
67
|
toUtf16View,
|
|
67
68
|
pairDiskSpans,
|
|
69
|
+
anchorSpans,
|
|
68
70
|
viewMapDefect,
|
|
69
71
|
} from "./view-map.mjs";
|
|
70
72
|
|
|
@@ -307,12 +309,11 @@ async function rehydrateEdit(
|
|
|
307
309
|
layer1View(span.diskText).cleaned !== span.cleanedText ||
|
|
308
310
|
(span.diskText !== oldS && content.includes(oldS));
|
|
309
311
|
if (anchorAmbiguous)
|
|
310
|
-
return
|
|
311
|
-
|
|
312
|
-
|
|
313
|
-
|
|
314
|
-
|
|
315
|
-
};
|
|
312
|
+
return anchorAmbiguityDeny(
|
|
313
|
+
"the matched region",
|
|
314
|
+
ti.file_path,
|
|
315
|
+
"edit a smaller region away from them",
|
|
316
|
+
);
|
|
316
317
|
const newRes = rehydrateNewString(
|
|
317
318
|
oldS,
|
|
318
319
|
ti.new_string,
|
|
@@ -362,6 +363,25 @@ async function rehydrateEdit(
|
|
|
362
363
|
};
|
|
363
364
|
}
|
|
364
365
|
|
|
366
|
+
/**
|
|
367
|
+
* The shared anchor-ambiguity refusal: a stripped run abuts kept text it
|
|
368
|
+
* resembles, so greedy deletion alignment cannot prove which bytes the region
|
|
369
|
+
* owns. Edit's searched spans and Write's position-anchored regions hit the
|
|
370
|
+
* same soundness gate and must speak the same language — one builder so the
|
|
371
|
+
* two deny sentences cannot drift apart.
|
|
372
|
+
* @param {string} lead what could not be anchored ("the matched region", …)
|
|
373
|
+
* @param {string} filePath
|
|
374
|
+
* @param {string} guidance the caller-specific way out, without trailing punctuation
|
|
375
|
+
*/
|
|
376
|
+
function anchorAmbiguityDeny(lead, filePath, guidance) {
|
|
377
|
+
return {
|
|
378
|
+
deny:
|
|
379
|
+
`${lead} sits next to stripped control sequences that cannot be ` +
|
|
380
|
+
`re-anchored unambiguously in ${filePath}; ${guidance}, or ask the user ` +
|
|
381
|
+
`to make this change`,
|
|
382
|
+
};
|
|
383
|
+
}
|
|
384
|
+
|
|
365
385
|
/**
|
|
366
386
|
* Foreign redaction placeholders surviving in post-substitution content `out`:
|
|
367
387
|
* hint-prefixed, placeholder-shaped tokens that are neither introduced by a
|
|
@@ -399,39 +419,171 @@ function foreignPlaceholders(out, hint, viewText, secretSpans) {
|
|
|
399
419
|
}
|
|
400
420
|
|
|
401
421
|
/**
|
|
422
|
+
* Restore one position-anchored view region of a Write — the common prefix or
|
|
423
|
+
* suffix `anchorSpans` proved unchanged — to its on-disk bytes. Unlike Edit's
|
|
424
|
+
* searched spans, the region is position-fixed, so the only residual hazard is
|
|
425
|
+
* greedy-alignment run misattribution, caught by the same re-clean soundness
|
|
426
|
+
* gate Edit uses (Edit's clause (b), the verbatim-collision check, cannot
|
|
427
|
+
* apply to a span that was never searched for). On gate failure the outcome
|
|
428
|
+
* depends on what the region holds: a placeholder-free region falls back to
|
|
429
|
+
* the model's own bytes (fail open — the write merely loses stripped
|
|
430
|
+
* characters, exactly today's behavior), while a placeholder-bearing region is
|
|
431
|
+
* denied (restoring at a misattributed anchor could graft secret bytes
|
|
432
|
+
* wrongly; not restoring persists placeholder text over the secret — neither
|
|
433
|
+
* open option is safe).
|
|
434
|
+
*
|
|
435
|
+
* `resolveSpan`'s boundary semantics deliberately drop a stripped run sitting
|
|
436
|
+
* exactly at the region's interior edge (it may belong to the changed middle),
|
|
437
|
+
* but a run at the very start or end OF THE FILE is unambiguous when the
|
|
438
|
+
* region reaches that edge — re-attach those explicitly, since `diskOffset`
|
|
439
|
+
* keeps boundary runs outside the span in both directions.
|
|
440
|
+
*
|
|
441
|
+
* `restoredChars` counts UTF-16 code units, matching the "character(s)" prose
|
|
442
|
+
* in {@link writeContext}.
|
|
443
|
+
* @param {WriteRestoreContext} ctx per-Write invariants shared by both regions
|
|
444
|
+
* @param {number} viewStart
|
|
445
|
+
* @param {number} viewEnd
|
|
446
|
+
* @param {string} fallback the model's own bytes for this region
|
|
447
|
+
* @returns {{text: string, pairs: readonly {placeholder: string, original: string, start: number}[], restoredChars: number} | {deny: string}}
|
|
448
|
+
*/
|
|
449
|
+
function restoreWriteRegion(ctx, viewStart, viewEnd, fallback) {
|
|
450
|
+
const { content, cleaned, view, deletions } = ctx;
|
|
451
|
+
if (viewStart >= viewEnd) return { text: "", pairs: [], restoredChars: 0 };
|
|
452
|
+
const span = resolveSpan(
|
|
453
|
+
content,
|
|
454
|
+
cleaned,
|
|
455
|
+
view,
|
|
456
|
+
deletions,
|
|
457
|
+
viewStart,
|
|
458
|
+
viewEnd,
|
|
459
|
+
);
|
|
460
|
+
/* c8 ignore start -- anchorSpans snapped both boundaries out of placeholder
|
|
461
|
+
interiors, the only way resolveSpan returns null; kept as a fail-loud
|
|
462
|
+
guard against a future regression in that snapping. */
|
|
463
|
+
if (span === null)
|
|
464
|
+
throw new Error("write anchor cut a placeholder despite boundary snapping");
|
|
465
|
+
/* c8 ignore stop */
|
|
466
|
+
const first = deletions[0];
|
|
467
|
+
const atStart = viewStart === 0 && first?.start === 0 ? first.deleted : "";
|
|
468
|
+
const last = deletions[deletions.length - 1];
|
|
469
|
+
const atEnd =
|
|
470
|
+
viewEnd === view.text.length && last?.start === cleaned.length
|
|
471
|
+
? last.deleted
|
|
472
|
+
: "";
|
|
473
|
+
const diskText = atStart + span.diskText + atEnd;
|
|
474
|
+
if (layer1View(diskText).cleaned !== span.cleanedText) {
|
|
475
|
+
if (span.pairs.length > 0)
|
|
476
|
+
return anchorAmbiguityDeny(
|
|
477
|
+
`the unchanged region around a ${ctx.hint}…] placeholder`,
|
|
478
|
+
ctx.filePath,
|
|
479
|
+
"use Edit for the changed region",
|
|
480
|
+
);
|
|
481
|
+
return { text: fallback, pairs: [], restoredChars: 0 };
|
|
482
|
+
}
|
|
483
|
+
return {
|
|
484
|
+
text: diskText,
|
|
485
|
+
pairs: span.pairs,
|
|
486
|
+
restoredChars: diskText.length - span.cleanedText.length,
|
|
487
|
+
};
|
|
488
|
+
}
|
|
489
|
+
|
|
490
|
+
/**
|
|
491
|
+
* The per-Write invariants `restoreWriteRegion` needs for both regions,
|
|
492
|
+
* bundled once in {@link rehydrateWrite} instead of threaded as positional
|
|
493
|
+
* parameters.
|
|
494
|
+
* @typedef {{
|
|
495
|
+
* filePath: string,
|
|
496
|
+
* content: string,
|
|
497
|
+
* cleaned: string,
|
|
498
|
+
* view: import("./view-map.mjs").FileView<"utf16">,
|
|
499
|
+
* deletions: {start: number, deleted: string}[],
|
|
500
|
+
* hint: string,
|
|
501
|
+
* }} WriteRestoreContext
|
|
502
|
+
*/
|
|
503
|
+
|
|
504
|
+
/**
|
|
505
|
+
* Re-anchor a whole-file Write against the target's sanitized view. The
|
|
506
|
+
* regions `anchorSpans` proves unchanged (common prefix/suffix, in view space)
|
|
507
|
+
* are restored to their on-disk bytes — redacted secrets AND Layer-1-stripped
|
|
508
|
+
* runs come back position-exact — while the genuinely-changed middle keeps the
|
|
509
|
+
* model's bytes, with this file's placeholders substituted for their secrets
|
|
510
|
+
* (Layer-1 strips of NEW text stay stripped: that is the sanitizer working).
|
|
402
511
|
* @param {{file_path: string, content: string}} ti
|
|
512
|
+
* @param {string} content disk bytes
|
|
513
|
+
* @param {string} cleaned Layer-1 view of `content`
|
|
403
514
|
* @param {import("./view-map.mjs").FileView<"utf16">} view
|
|
515
|
+
* @param {{start: number, deleted: string}[]} deletions
|
|
404
516
|
* @param {RehydrateIo} io
|
|
405
517
|
* @param {string} hint placeholder prefix
|
|
406
518
|
*/
|
|
407
|
-
async function rehydrateWrite(ti, view, io, hint) {
|
|
408
|
-
const
|
|
409
|
-
|
|
410
|
-
|
|
411
|
-
//
|
|
412
|
-
//
|
|
413
|
-
//
|
|
414
|
-
|
|
415
|
-
|
|
416
|
-
|
|
417
|
-
|
|
418
|
-
|
|
519
|
+
async function rehydrateWrite(ti, content, cleaned, view, deletions, io, hint) {
|
|
520
|
+
const viewText = view.text;
|
|
521
|
+
// An EMPTY view means the model can see nothing of the file — an
|
|
522
|
+
// all-invisible file, the archetypal hidden-payload artifact. A Write there
|
|
523
|
+
// (of "" or of anything else) is the model replacing content it was told is
|
|
524
|
+
// suspicious, not echoing content back; restoring the hidden bytes would
|
|
525
|
+
// actively defeat that cleanup. Keep the model's bytes (fail open).
|
|
526
|
+
if (ti.content === viewText && viewText !== "") {
|
|
527
|
+
// A hinted Write of a PRISTINE file's view (the hint is literal prose the
|
|
528
|
+
// file already had) — nothing diverges, nothing to do.
|
|
529
|
+
if (content === ti.content) return null;
|
|
530
|
+
// Faithful whole-file round-trip: the incoming content IS the sanitized
|
|
531
|
+
// view, so the write becomes the disk bytes themselves — placeholders
|
|
532
|
+
// resolve to their secrets and every stripped run comes back. Provably
|
|
533
|
+
// sound with no gate: the view was derived from exactly these bytes, so
|
|
534
|
+
// re-sanitizing reproduces it, nothing new can be exposed, and no foreign
|
|
535
|
+
// placeholder can appear.
|
|
419
536
|
return {
|
|
420
|
-
|
|
421
|
-
|
|
422
|
-
|
|
423
|
-
|
|
424
|
-
|
|
537
|
+
updatedInput: { ...ti, content },
|
|
538
|
+
context: writeContext(
|
|
539
|
+
ti.file_path,
|
|
540
|
+
content.length - cleaned.length,
|
|
541
|
+
view.pairs.length,
|
|
542
|
+
hint,
|
|
543
|
+
),
|
|
425
544
|
};
|
|
545
|
+
}
|
|
546
|
+
const { prefixEnd, suffixStart } = anchorSpans(ti.content, view);
|
|
547
|
+
const suffixLen = viewText.length - suffixStart;
|
|
548
|
+
const ctx = {
|
|
549
|
+
filePath: ti.file_path,
|
|
550
|
+
content,
|
|
551
|
+
cleaned,
|
|
552
|
+
view,
|
|
553
|
+
deletions,
|
|
554
|
+
hint,
|
|
555
|
+
};
|
|
556
|
+
const prefix = restoreWriteRegion(
|
|
557
|
+
ctx,
|
|
558
|
+
0,
|
|
559
|
+
prefixEnd,
|
|
560
|
+
ti.content.slice(0, prefixEnd),
|
|
561
|
+
);
|
|
562
|
+
if ("deny" in prefix) return prefix;
|
|
563
|
+
const suffix = restoreWriteRegion(
|
|
564
|
+
ctx,
|
|
565
|
+
suffixStart,
|
|
566
|
+
viewText.length,
|
|
567
|
+
suffixLen === 0 ? "" : ti.content.slice(-suffixLen),
|
|
568
|
+
);
|
|
569
|
+
if ("deny" in suffix) return suffix;
|
|
426
570
|
|
|
427
|
-
//
|
|
428
|
-
//
|
|
429
|
-
//
|
|
430
|
-
//
|
|
571
|
+
// The changed middle, in content space (prefix/suffix are common substrings,
|
|
572
|
+
// so their view-space lengths index ti.content directly). Only this file's
|
|
573
|
+
// OWN placeholder texts occurring here are substituted; placeholders wholly
|
|
574
|
+
// inside the restored prefix/suffix need no substitution — resolveSpan
|
|
575
|
+
// already brought back the real disk secrets byte-exact.
|
|
576
|
+
const middle = ti.content.slice(prefixEnd, ti.content.length - suffixLen);
|
|
577
|
+
const texts = [...new Set(view.pairs.map((pair) => pair.placeholder))].filter(
|
|
578
|
+
(phText) => middle.includes(phText),
|
|
579
|
+
);
|
|
580
|
+
// Resolve each placeholder text to its single secret first, then splice in
|
|
581
|
+
// ONE ordered pass (R6) via the shared `spliceOrdered` — see its doc for why
|
|
582
|
+
// a chained `out.split(ph).join(secret)` per placeholder is unsound.
|
|
431
583
|
const valueByPh = new Map();
|
|
432
584
|
for (const phText of texts) {
|
|
433
585
|
const produced = view.pairs.filter((pair) => pair.placeholder === phText);
|
|
434
|
-
if (occurrences(
|
|
586
|
+
if (occurrences(viewText, phText).length > produced.length)
|
|
435
587
|
return {
|
|
436
588
|
deny:
|
|
437
589
|
`${ti.file_path} mixes literal "${phText}" text with a redacted secret sharing ` +
|
|
@@ -448,46 +600,112 @@ async function rehydrateWrite(ti, view, io, hint) {
|
|
|
448
600
|
};
|
|
449
601
|
valueByPh.set(phText, values[0]);
|
|
450
602
|
}
|
|
451
|
-
|
|
452
|
-
|
|
453
|
-
|
|
454
|
-
// excluded from the foreign-placeholder scan below.
|
|
455
|
-
const { text: out, spans: secretSpans } = spliceOrdered(
|
|
456
|
-
ti.content,
|
|
457
|
-
orderedMatches(ti.content, texts),
|
|
603
|
+
const { text: middleOut, spans: middleSpans } = spliceOrdered(
|
|
604
|
+
middle,
|
|
605
|
+
orderedMatches(middle, texts),
|
|
458
606
|
(match) => valueByPh.get(match.text),
|
|
459
607
|
);
|
|
460
|
-
const
|
|
608
|
+
const out = prefix.text + middleOut + suffix.text;
|
|
461
609
|
|
|
462
|
-
// R3: the
|
|
463
|
-
//
|
|
464
|
-
//
|
|
465
|
-
//
|
|
466
|
-
//
|
|
467
|
-
//
|
|
468
|
-
//
|
|
469
|
-
//
|
|
470
|
-
|
|
471
|
-
|
|
472
|
-
|
|
473
|
-
|
|
474
|
-
|
|
475
|
-
|
|
476
|
-
|
|
477
|
-
|
|
610
|
+
// R3: the content may still carry a FOREIGN placeholder — one pasted from
|
|
611
|
+
// another file/context that shares the hint prefix but is not one of this
|
|
612
|
+
// file's own. It would be persisted verbatim over a real secret. Compare the
|
|
613
|
+
// ACTUAL placeholder STRINGS, not scalar hint counts: a scalar comparison is
|
|
614
|
+
// defeated by an edit that drops one literal hint and adds one foreign
|
|
615
|
+
// placeholder (the counts net to zero). `secretSpans` are the byte ranges in
|
|
616
|
+
// `out` occupied by secret VALUES (substituted in the middle, or restored
|
|
617
|
+
// with the prefix/suffix) — a hint occurrence inside one is a pathological
|
|
618
|
+
// secret whose bytes contain the hint prefix, NOT a pasted placeholder, so
|
|
619
|
+
// it is excluded from the scan.
|
|
620
|
+
if (out.includes(hint)) {
|
|
621
|
+
const diskSpans = pairDiskSpans(view, deletions);
|
|
622
|
+
const secretSpans = [];
|
|
623
|
+
// A restored prefix starts at disk offset 0, so disk offsets ARE out
|
|
624
|
+
// offsets there; a restored suffix is the file's tail, shifted by however
|
|
625
|
+
// much the content ahead of it grew or shrank.
|
|
626
|
+
for (const pair of prefix.pairs)
|
|
627
|
+
secretSpans.push(diskSpans[view.pairs.indexOf(pair)]);
|
|
628
|
+
const suffixShift = out.length - content.length;
|
|
629
|
+
for (const pair of suffix.pairs) {
|
|
630
|
+
const diskSpan = diskSpans[view.pairs.indexOf(pair)];
|
|
631
|
+
secretSpans.push({
|
|
632
|
+
start: diskSpan.start + suffixShift,
|
|
633
|
+
end: diskSpan.end + suffixShift,
|
|
634
|
+
});
|
|
635
|
+
}
|
|
636
|
+
for (const span of middleSpans)
|
|
637
|
+
secretSpans.push({
|
|
638
|
+
start: span.start + prefix.text.length,
|
|
639
|
+
end: span.end + prefix.text.length,
|
|
640
|
+
});
|
|
641
|
+
if (foreignPlaceholders(out, hint, viewText, secretSpans).length > 0)
|
|
642
|
+
return {
|
|
643
|
+
deny:
|
|
644
|
+
`the new content still carries a ${hint}…] placeholder that does not match any ` +
|
|
645
|
+
`secret in ${ti.file_path}, so a whole-file Write cannot copy a placeholder from ` +
|
|
646
|
+
`another file or context; request the source file's content and rehydrate a ` +
|
|
647
|
+
`same-file Edit instead, or write the secret's real value directly`,
|
|
648
|
+
};
|
|
649
|
+
}
|
|
478
650
|
|
|
479
|
-
|
|
480
|
-
|
|
651
|
+
// Nothing restored, nothing substituted: the write proceeds with the
|
|
652
|
+
// model's own bytes exactly as it would have without this layer.
|
|
653
|
+
if (out === ti.content) return null;
|
|
654
|
+
|
|
655
|
+
const secrets = [
|
|
656
|
+
...valueByPh.values(),
|
|
657
|
+
...prefix.pairs.map((pair) => pair.original),
|
|
658
|
+
...suffix.pairs.map((pair) => pair.original),
|
|
659
|
+
];
|
|
660
|
+
// Writing back the file's exact disk bytes cannot expose anything: the next
|
|
661
|
+
// sanitized view is byte-identical to the prior one. Skip the redactor
|
|
662
|
+
// round-trip on that (common) faithful-round-trip case.
|
|
663
|
+
if (out !== content) {
|
|
664
|
+
const exposed = await exposedSecrets(secrets, viewText, out, io);
|
|
665
|
+
if (exposed > 0) return { deny: exposureDeny(exposed) };
|
|
666
|
+
}
|
|
481
667
|
|
|
482
668
|
return {
|
|
483
669
|
updatedInput: { ...ti, content: out },
|
|
484
|
-
context:
|
|
485
|
-
|
|
486
|
-
|
|
487
|
-
|
|
670
|
+
context: writeContext(
|
|
671
|
+
ti.file_path,
|
|
672
|
+
prefix.restoredChars + suffix.restoredChars,
|
|
673
|
+
middleSpans.length + prefix.pairs.length + suffix.pairs.length,
|
|
674
|
+
hint,
|
|
675
|
+
),
|
|
488
676
|
};
|
|
489
677
|
}
|
|
490
678
|
|
|
679
|
+
/**
|
|
680
|
+
* Model-facing context line for a rewritten Write input.
|
|
681
|
+
* @param {string} filePath
|
|
682
|
+
* @param {number} restoredChars UTF-16 code units of stripped characters restored with the unchanged regions
|
|
683
|
+
* @param {number} secretCount placeholders resolved (spliced or restored)
|
|
684
|
+
* @param {string} hint placeholder prefix
|
|
685
|
+
*/
|
|
686
|
+
function writeContext(filePath, restoredChars, secretCount, hint) {
|
|
687
|
+
const parts = [];
|
|
688
|
+
if (restoredChars > 0)
|
|
689
|
+
parts.push(
|
|
690
|
+
`${restoredChars} invisible/control character(s) stripped from your view of ` +
|
|
691
|
+
`${filePath} were restored from disk in the regions your Write left unchanged.`,
|
|
692
|
+
);
|
|
693
|
+
if (secretCount > 0)
|
|
694
|
+
parts.push(
|
|
695
|
+
`Write content contained ${hint}…] placeholders; they were resolved to the ` +
|
|
696
|
+
`file's real secret values on disk (still hidden from you), so the secrets ` +
|
|
697
|
+
`are preserved in the written file.`,
|
|
698
|
+
);
|
|
699
|
+
// Reachable with both counts zero: a lone surrogate the view normalized to
|
|
700
|
+
// U+FFFD restores same-length, changing bytes but neither counter.
|
|
701
|
+
if (parts.length === 0)
|
|
702
|
+
parts.push(
|
|
703
|
+
`unchanged regions of your Write were restored to the exact on-disk bytes of ` +
|
|
704
|
+
`${filePath} (differing only by characters hidden from your view).`,
|
|
705
|
+
);
|
|
706
|
+
return parts.join(" ");
|
|
707
|
+
}
|
|
708
|
+
|
|
491
709
|
/**
|
|
492
710
|
* The single MultiEdit refusal: covers a sanitized view that diverges
|
|
493
711
|
* from disk (redacted secrets, stripped invisible characters, a lone
|
|
@@ -511,20 +729,21 @@ function multiEditDeny(filePath) {
|
|
|
511
729
|
|
|
512
730
|
/**
|
|
513
731
|
* True when this tool call could need re-anchoring against the target file's
|
|
514
|
-
* sanitized view: any well-formed Edit (the view may differ from
|
|
515
|
-
* without placeholders, via stripped invisible characters
|
|
516
|
-
*
|
|
517
|
-
*
|
|
732
|
+
* sanitized view: any well-formed Edit or Write (the view may differ from
|
|
733
|
+
* disk even without placeholders, via stripped invisible characters — a
|
|
734
|
+
* hint-free whole-file Write of such a file would silently persist the
|
|
735
|
+
* stripped bytes), and any well-formed MultiEdit (gated on the same
|
|
736
|
+
* grounds).
|
|
518
737
|
* @param {string} tool
|
|
519
738
|
* @param {any} ti
|
|
520
|
-
* @param {string} hint
|
|
521
739
|
*/
|
|
522
|
-
function isCandidate(tool, ti
|
|
740
|
+
function isCandidate(tool, ti) {
|
|
523
741
|
if (typeof ti?.file_path !== "string") return false;
|
|
524
742
|
if (tool === "Edit")
|
|
525
743
|
return (
|
|
526
744
|
typeof ti.old_string === "string" && typeof ti.new_string === "string"
|
|
527
745
|
);
|
|
746
|
+
if (tool === "Write") return typeof ti.content === "string";
|
|
528
747
|
// MultiEdit applies its edits SEQUENTIALLY, each against the result of the
|
|
529
748
|
// previous, so the span machinery below (which maps one old_string against
|
|
530
749
|
// one static view) cannot re-anchor it. It is still a candidate: on a
|
|
@@ -543,8 +762,6 @@ function isCandidate(tool, ti, hint) {
|
|
|
543
762
|
typeof edit?.new_string === "string",
|
|
544
763
|
)
|
|
545
764
|
);
|
|
546
|
-
if (tool === "Write")
|
|
547
|
-
return typeof ti.content === "string" && ti.content.includes(hint);
|
|
548
765
|
return false;
|
|
549
766
|
}
|
|
550
767
|
|
|
@@ -586,16 +803,17 @@ export async function rehydrateRedacted(
|
|
|
586
803
|
`hidden from your view; rehydration is not supported for notebooks. Keep ` +
|
|
587
804
|
`the secret-bearing cell unchanged, or ask the user to edit it.`,
|
|
588
805
|
};
|
|
589
|
-
if (!isCandidate(tool, toolInput
|
|
806
|
+
if (!isCandidate(tool, toolInput)) return null;
|
|
590
807
|
const hinted =
|
|
591
|
-
tool === "Write"
|
|
592
|
-
|
|
593
|
-
|
|
594
|
-
|
|
595
|
-
|
|
596
|
-
|
|
597
|
-
|
|
598
|
-
toolInput.
|
|
808
|
+
tool === "Write"
|
|
809
|
+
? toolInput.content.includes(hint)
|
|
810
|
+
: tool === "MultiEdit"
|
|
811
|
+
? toolInput.edits.some(
|
|
812
|
+
(/** @type {{old_string: string, new_string: string}} */ edit) =>
|
|
813
|
+
edit.old_string.includes(hint) || edit.new_string.includes(hint),
|
|
814
|
+
)
|
|
815
|
+
: toolInput.old_string.includes(hint) ||
|
|
816
|
+
toolInput.new_string.includes(hint);
|
|
599
817
|
|
|
600
818
|
let content;
|
|
601
819
|
try {
|
|
@@ -609,14 +827,15 @@ export async function rehydrateRedacted(
|
|
|
609
827
|
// its own (nothing to re-anchor), so pass through — a hint-free call, an
|
|
610
828
|
// Edit whose old_string is non-empty, or a MultiEdit whose FIRST edit's
|
|
611
829
|
// old_string is non-empty (only an empty first old_string is the create
|
|
612
|
-
// form; anything else errors not-found in the real tool).
|
|
613
|
-
//
|
|
614
|
-
//
|
|
615
|
-
//
|
|
616
|
-
//
|
|
617
|
-
//
|
|
618
|
-
//
|
|
619
|
-
//
|
|
830
|
+
// form; anything else errors not-found in the real tool). A hint-free
|
|
831
|
+
// Write is file CREATION too — there is no prior view and nothing to
|
|
832
|
+
// restore, so it passes through on the `!hinted` arm. But any call that
|
|
833
|
+
// WOULD create the file with hinted content — a hinted Write, a hinted
|
|
834
|
+
// Edit-create, or a hinted MultiEdit-create — persists its placeholder
|
|
835
|
+
// verbatim, standing for a secret that does NOT exist on this new path.
|
|
836
|
+
// R4: that is the same cross-file/stale-placeholder mistake a same-file
|
|
837
|
+
// Write is denied for; refuse with the same guidance rather than write
|
|
838
|
+
// the placeholder text as a real value.
|
|
620
839
|
if (nodeErr?.code === "ENOENT") {
|
|
621
840
|
const creates =
|
|
622
841
|
tool === "Write" ||
|
|
@@ -637,11 +856,19 @@ export async function rehydrateRedacted(
|
|
|
637
856
|
// failed, the file didn't vanish. A hinted call's content may carry
|
|
638
857
|
// placeholder text that must never be persisted literally over whatever
|
|
639
858
|
// secret is actually there, so fail closed with a deny instead of the
|
|
640
|
-
// silent pass-through above. A non-hinted
|
|
641
|
-
// secret-shaped placeholder
|
|
642
|
-
// hit this exact same
|
|
643
|
-
// swallow an unexpected failure.
|
|
644
|
-
|
|
859
|
+
// silent pass-through above. A non-hinted Edit/MultiEdit was never going
|
|
860
|
+
// to write a secret-shaped placeholder, and the underlying tool call
|
|
861
|
+
// reads the file itself, so it will hit this exact same error — let it
|
|
862
|
+
// propagate rather than swallow an unexpected failure. A non-hinted
|
|
863
|
+
// WRITE never reads its target: pre-restoration it would simply proceed,
|
|
864
|
+
// so blocking it on a read error this layer alone performed would fail
|
|
865
|
+
// closed on a placeholder-free ambiguity. Pass it through (fail open —
|
|
866
|
+
// restoration is best-effort; the write merely loses stripped
|
|
867
|
+
// characters, exactly the pre-restoration behavior).
|
|
868
|
+
if (!hinted) {
|
|
869
|
+
if (tool !== "Write") throw err;
|
|
870
|
+
return null;
|
|
871
|
+
}
|
|
645
872
|
return {
|
|
646
873
|
deny:
|
|
647
874
|
`could not read ${toolInput.file_path} to rehydrate its secrets ` +
|
|
@@ -655,9 +882,11 @@ export async function rehydrateRedacted(
|
|
|
655
882
|
// R1: if nothing is redacted, a hint-free old_string cannot touch a hidden
|
|
656
883
|
// span, so keep the fast pass-through (a verbatim match needs no translation;
|
|
657
884
|
// a mismatch is an ordinary stale old_string Edit reports itself) and never
|
|
658
|
-
// invoke the redactor's map mode.
|
|
659
|
-
//
|
|
660
|
-
//
|
|
885
|
+
// invoke the redactor's map mode. The same holds for a hint-free Write:
|
|
886
|
+
// with no stripped run and no secret, the view IS the disk bytes and there
|
|
887
|
+
// is nothing to restore. But if the file DOES hold secrets, a hint-free
|
|
888
|
+
// old_string can still match disk bytes INSIDE a redacted span the model
|
|
889
|
+
// never saw — the char-by-char extraction oracle. Fall through to the
|
|
661
890
|
// resolver so its overlap/exposure guards run before any such byte is spliced
|
|
662
891
|
// raw. `io.redact` (plain mode) is the cheap secrets-present probe; it returns
|
|
663
892
|
// null exactly when the file has no secrets.
|
|
@@ -733,16 +962,19 @@ export async function rehydrateRedacted(
|
|
|
733
962
|
};
|
|
734
963
|
}
|
|
735
964
|
// View identical to disk: any placeholders in an Edit's old_string are
|
|
736
|
-
// literal text, so there is nothing to re-anchor
|
|
737
|
-
//
|
|
738
|
-
//
|
|
965
|
+
// literal text, so there is nothing to re-anchor — and a hint-free Write of
|
|
966
|
+
// a pristine file has nothing to restore. `cleaned === content` also rules
|
|
967
|
+
// out a lone-surrogate-only divergence (view.pairs/deletions alone would
|
|
968
|
+
// miss that, since the normalization is neither a redaction pair nor a
|
|
739
969
|
// Layer-1 deletion). HINTED Write and MultiEdit are the exceptions: their
|
|
740
970
|
// content still carries the hint prefix, and with no own placeholder to
|
|
741
|
-
// resolve that hint
|
|
971
|
+
// resolve that hint may be a FOREIGN [REDACTED…] placeholder that would be
|
|
742
972
|
// persisted verbatim over pristine bytes. A Write falls through to
|
|
743
|
-
// rehydrateWrite's
|
|
744
|
-
//
|
|
745
|
-
// the
|
|
973
|
+
// rehydrateWrite's foreign-placeholder scan (which denies a genuinely
|
|
974
|
+
// foreign token and passes literal prose the file already had); a MultiEdit
|
|
975
|
+
// to the MultiEdit deny below — without this a hinted MultiEdit on a
|
|
976
|
+
// pristine file silently persists the foreign placeholder a byte-identical
|
|
977
|
+
// Write is denied for.
|
|
746
978
|
const viewEqualsDisk =
|
|
747
979
|
view.pairs.length === 0 && deletions.length === 0 && cleaned === content;
|
|
748
980
|
// Precision refinement for a hinted MultiEdit on that PRISTINE file: a
|
|
@@ -782,5 +1014,5 @@ export async function rehydrateRedacted(
|
|
|
782
1014
|
hinted,
|
|
783
1015
|
hint,
|
|
784
1016
|
)
|
|
785
|
-
: rehydrateWrite(toolInput, view, io, hint);
|
|
1017
|
+
: rehydrateWrite(toolInput, content, cleaned, view, deletions, io, hint);
|
|
786
1018
|
}
|
package/src/view-map.mjs
CHANGED
|
@@ -451,6 +451,76 @@ export function spliceOrdered(text, matches, replacementFor) {
|
|
|
451
451
|
return { text: out + text.slice(last), spans };
|
|
452
452
|
}
|
|
453
453
|
|
|
454
|
+
/**
|
|
455
|
+
* Anchor a whole-file Write's content against the sanitized view it was
|
|
456
|
+
* composed from: the longest common prefix and suffix are the regions the
|
|
457
|
+
* model left unchanged, so their on-disk bytes (stripped runs, redacted
|
|
458
|
+
* secrets, lone surrogates included) can be restored position-exact — no
|
|
459
|
+
* search, no anchor ambiguity. Returns view-space `{prefixEnd, suffixStart}`;
|
|
460
|
+
* because the prefix and suffix are common substrings, the same lengths index
|
|
461
|
+
* `content` (prefix `[0, prefixEnd)`, suffix `[content.length - (view.text.length
|
|
462
|
+
* - suffixStart))`).
|
|
463
|
+
*
|
|
464
|
+
* Both boundaries are snapped OUT of hazards, always shrinking the restored
|
|
465
|
+
* region (the fail-open direction — a smaller restore only loses stripped
|
|
466
|
+
* characters, never corrupts):
|
|
467
|
+
* - a boundary strictly inside a placeholder moves to the placeholder's
|
|
468
|
+
* edge, so `resolveSpan` (which returns null on a placeholder-cutting
|
|
469
|
+
* boundary) always resolves;
|
|
470
|
+
* - a boundary splitting a surrogate pair moves off it. The view never
|
|
471
|
+
* carries lone surrogates (they were normalized to U+FFFD), so a high
|
|
472
|
+
* surrogate at `prefixEnd - 1` is always a genuine pair's first half; and
|
|
473
|
+
* since the prefix/suffix are common substrings, checking the view covers
|
|
474
|
+
* the content side too.
|
|
475
|
+
* The prefix is computed first and the suffix capped so they never overlap
|
|
476
|
+
* (prefix wins — deterministic).
|
|
477
|
+
* @param {string} content incoming Write content (view space)
|
|
478
|
+
* @param {FileView<"utf16">} view sanitized view of the target file
|
|
479
|
+
* @returns {{prefixEnd: number, suffixStart: number}}
|
|
480
|
+
*/
|
|
481
|
+
export function anchorSpans(content, view) {
|
|
482
|
+
assertFileView(view, "utf16", "anchorSpans");
|
|
483
|
+
const viewText = view.text;
|
|
484
|
+
const maxPrefix = Math.min(content.length, viewText.length);
|
|
485
|
+
let p = 0;
|
|
486
|
+
while (p < maxPrefix && content[p] === viewText[p]) p++;
|
|
487
|
+
const interior = (/** @type {number} */ offset) =>
|
|
488
|
+
view.pairs.find(
|
|
489
|
+
(pair) =>
|
|
490
|
+
pair.start < offset && offset < pair.start + pair.placeholder.length,
|
|
491
|
+
);
|
|
492
|
+
const cut = interior(p);
|
|
493
|
+
if (cut) p = cut.start;
|
|
494
|
+
if (p > 0 && isHighSurrogate(viewText.charCodeAt(p - 1))) p--;
|
|
495
|
+
|
|
496
|
+
const maxSuffix = maxPrefix - p;
|
|
497
|
+
let s = 0;
|
|
498
|
+
while (
|
|
499
|
+
s < maxSuffix &&
|
|
500
|
+
content[content.length - 1 - s] === viewText[viewText.length - 1 - s]
|
|
501
|
+
)
|
|
502
|
+
s++;
|
|
503
|
+
let suffixStart = viewText.length - s;
|
|
504
|
+
const cutEnd = interior(suffixStart);
|
|
505
|
+
if (cutEnd) suffixStart = cutEnd.start + cutEnd.placeholder.length;
|
|
506
|
+
if (
|
|
507
|
+
suffixStart < viewText.length &&
|
|
508
|
+
isLowSurrogate(viewText.charCodeAt(suffixStart))
|
|
509
|
+
)
|
|
510
|
+
suffixStart++;
|
|
511
|
+
return { prefixEnd: p, suffixStart };
|
|
512
|
+
}
|
|
513
|
+
|
|
514
|
+
/** @param {number} code */
|
|
515
|
+
function isHighSurrogate(code) {
|
|
516
|
+
return code >= 0xd800 && code <= 0xdbff;
|
|
517
|
+
}
|
|
518
|
+
|
|
519
|
+
/** @param {number} code */
|
|
520
|
+
function isLowSurrogate(code) {
|
|
521
|
+
return code >= 0xdc00 && code <= 0xdfff;
|
|
522
|
+
}
|
|
523
|
+
|
|
454
524
|
/**
|
|
455
525
|
* On-disk [start, end) span of every redaction pair, mapped from its view
|
|
456
526
|
* offset through placeholder expansion (view → cleaned) and stripped invisible
|
package/src/warnings.mjs
CHANGED
|
@@ -51,6 +51,20 @@ export function describeHtmlSanitized(removed) {
|
|
|
51
51
|
return `HTML sanitized: ${describeRemoved(removed)} replaced with placeholders`;
|
|
52
52
|
}
|
|
53
53
|
|
|
54
|
+
/**
|
|
55
|
+
* The Layer-2 warning for the fail-closed unparseable path: the parse itself
|
|
56
|
+
* blew up, so nothing was spliced — the ENTIRE output was withheld behind one
|
|
57
|
+
* placeholder. {@link describeHtmlSanitized}'s "N hidden element(s) replaced"
|
|
58
|
+
* would misstate that as a routine splice, so this path gets its own sentence.
|
|
59
|
+
* A bare constant like {@link LONE_SURROGATE_WARNING} (no counts to
|
|
60
|
+
* interpolate). The sentence deliberately promises nothing about a reveal
|
|
61
|
+
* sidecar: the library facade drops `reveal`, and the hook can withhold the
|
|
62
|
+
* sidecar when it fails vetting — either way a persistence promise here would
|
|
63
|
+
* be false.
|
|
64
|
+
*/
|
|
65
|
+
export const HTML_UNPARSEABLE_WARNING =
|
|
66
|
+
"HTML unparseable — the entire output was withheld behind a placeholder; nothing was spliced";
|
|
67
|
+
|
|
54
68
|
/**
|
|
55
69
|
* Full warning for Layer 2's preserved-but-reported content (scripting and
|
|
56
70
|
* resource tags, data: URIs), or "" when there is nothing to report. Callers
|
|
@@ -260,6 +260,7 @@ export const SECRET_HINT_EXT: RegExp;
|
|
|
260
260
|
export const describeRemoved: typeof import("agent-sanitizer/output").describeRemoved;
|
|
261
261
|
export const describeWarned: typeof import("agent-sanitizer/output").describeWarned;
|
|
262
262
|
export const suppressToolOutput: typeof import("agent-sanitizer/output").suppressToolOutput;
|
|
263
|
+
export const REVEAL_WITHHELD_WARNING: "Withheld the reveal sidecar: it could not be vetted for secrets";
|
|
263
264
|
export const COLLISION_WITHHELD_MESSAGE: "[WITHHELD \u2014 this field's name collided with another after sanitization]";
|
|
264
265
|
export const ON_DISK_PLACEHOLDER_WARNING: string;
|
|
265
266
|
/**
|
package/types/html.d.mts
CHANGED
|
@@ -61,9 +61,12 @@ export function looksLikeHtmlSource(text: string): boolean;
|
|
|
61
61
|
* Layer 2 over web-ingress text: splice out HTML comments and hidden elements
|
|
62
62
|
* (placeholders mark the cuts; all other bytes are preserved verbatim) and
|
|
63
63
|
* count preserved scripting/resource tags for the caller's warning. Returns
|
|
64
|
-
* null when there is nothing to strip and nothing to report.
|
|
64
|
+
* null when there is nothing to strip and nothing to report. `unparseable` is
|
|
65
|
+
* set (true) only on the fail-closed path below, where the whole input was
|
|
66
|
+
* withheld behind {@link UNPARSEABLE_PLACEHOLDER} rather than spliced — the
|
|
67
|
+
* caller's warning must describe a whole-output withhold, not a splice.
|
|
65
68
|
* @param {string} text
|
|
66
|
-
* @returns {{ text: string, removed: { comments: number, hidden: number }, warned: { tags: Record<string, number>, dataSrc: number } } | null}
|
|
69
|
+
* @returns {{ text: string, removed: { comments: number, hidden: number }, warned: { tags: Record<string, number>, dataSrc: number }, unparseable?: true } | null}
|
|
67
70
|
*/
|
|
68
71
|
export function sanitizeHtml(text: string): {
|
|
69
72
|
text: string;
|
|
@@ -75,6 +78,7 @@ export function sanitizeHtml(text: string): {
|
|
|
75
78
|
tags: Record<string, number>;
|
|
76
79
|
dataSrc: number;
|
|
77
80
|
};
|
|
81
|
+
unparseable?: true;
|
|
78
82
|
} | null;
|
|
79
83
|
/**
|
|
80
84
|
* @param {string} url
|
package/types/rehydrate.d.mts
CHANGED
|
@@ -54,3 +54,19 @@ export type RehydrateIo = {
|
|
|
54
54
|
redactMap: (text: string) => Promise<RedactMapView> | RedactMapView;
|
|
55
55
|
redact: (text: string) => Promise<string | null> | (string | null);
|
|
56
56
|
};
|
|
57
|
+
/**
|
|
58
|
+
* The per-Write invariants `restoreWriteRegion` needs for both regions,
|
|
59
|
+
* bundled once in {@link rehydrateWrite} instead of threaded as positional
|
|
60
|
+
* parameters.
|
|
61
|
+
*/
|
|
62
|
+
export type WriteRestoreContext = {
|
|
63
|
+
filePath: string;
|
|
64
|
+
content: string;
|
|
65
|
+
cleaned: string;
|
|
66
|
+
view: import("./view-map.mjs").FileView<"utf16">;
|
|
67
|
+
deletions: {
|
|
68
|
+
start: number;
|
|
69
|
+
deleted: string;
|
|
70
|
+
}[];
|
|
71
|
+
hint: string;
|
|
72
|
+
};
|
package/types/view-map.d.mts
CHANGED
|
@@ -198,6 +198,37 @@ export function spliceOrdered(text: string, matches: {
|
|
|
198
198
|
end: number;
|
|
199
199
|
}[];
|
|
200
200
|
};
|
|
201
|
+
/**
|
|
202
|
+
* Anchor a whole-file Write's content against the sanitized view it was
|
|
203
|
+
* composed from: the longest common prefix and suffix are the regions the
|
|
204
|
+
* model left unchanged, so their on-disk bytes (stripped runs, redacted
|
|
205
|
+
* secrets, lone surrogates included) can be restored position-exact — no
|
|
206
|
+
* search, no anchor ambiguity. Returns view-space `{prefixEnd, suffixStart}`;
|
|
207
|
+
* because the prefix and suffix are common substrings, the same lengths index
|
|
208
|
+
* `content` (prefix `[0, prefixEnd)`, suffix `[content.length - (view.text.length
|
|
209
|
+
* - suffixStart))`).
|
|
210
|
+
*
|
|
211
|
+
* Both boundaries are snapped OUT of hazards, always shrinking the restored
|
|
212
|
+
* region (the fail-open direction — a smaller restore only loses stripped
|
|
213
|
+
* characters, never corrupts):
|
|
214
|
+
* - a boundary strictly inside a placeholder moves to the placeholder's
|
|
215
|
+
* edge, so `resolveSpan` (which returns null on a placeholder-cutting
|
|
216
|
+
* boundary) always resolves;
|
|
217
|
+
* - a boundary splitting a surrogate pair moves off it. The view never
|
|
218
|
+
* carries lone surrogates (they were normalized to U+FFFD), so a high
|
|
219
|
+
* surrogate at `prefixEnd - 1` is always a genuine pair's first half; and
|
|
220
|
+
* since the prefix/suffix are common substrings, checking the view covers
|
|
221
|
+
* the content side too.
|
|
222
|
+
* The prefix is computed first and the suffix capped so they never overlap
|
|
223
|
+
* (prefix wins — deterministic).
|
|
224
|
+
* @param {string} content incoming Write content (view space)
|
|
225
|
+
* @param {FileView<"utf16">} view sanitized view of the target file
|
|
226
|
+
* @returns {{prefixEnd: number, suffixStart: number}}
|
|
227
|
+
*/
|
|
228
|
+
export function anchorSpans(content: string, view: FileView<"utf16">): {
|
|
229
|
+
prefixEnd: number;
|
|
230
|
+
suffixStart: number;
|
|
231
|
+
};
|
|
201
232
|
/**
|
|
202
233
|
* On-disk [start, end) span of every redaction pair, mapped from its view
|
|
203
234
|
* offset through placeholder expansion (view → cleaned) and stripped invisible
|
package/types/warnings.d.mts
CHANGED
|
@@ -69,3 +69,15 @@ export function describeExfil(threats: {
|
|
|
69
69
|
* entry points, and two typed copies is one typo away from two warnings.
|
|
70
70
|
*/
|
|
71
71
|
export const LONE_SURROGATE_WARNING: "Normalized lone UTF-16 surrogates";
|
|
72
|
+
/**
|
|
73
|
+
* The Layer-2 warning for the fail-closed unparseable path: the parse itself
|
|
74
|
+
* blew up, so nothing was spliced — the ENTIRE output was withheld behind one
|
|
75
|
+
* placeholder. {@link describeHtmlSanitized}'s "N hidden element(s) replaced"
|
|
76
|
+
* would misstate that as a routine splice, so this path gets its own sentence.
|
|
77
|
+
* A bare constant like {@link LONE_SURROGATE_WARNING} (no counts to
|
|
78
|
+
* interpolate). The sentence deliberately promises nothing about a reveal
|
|
79
|
+
* sidecar: the library facade drops `reveal`, and the hook can withhold the
|
|
80
|
+
* sidecar when it fails vetting — either way a persistence promise here would
|
|
81
|
+
* be false.
|
|
82
|
+
*/
|
|
83
|
+
export const HTML_UNPARSEABLE_WARNING: "HTML unparseable \u2014 the entire output was withheld behind a placeholder; nothing was spliced";
|