@framers/agentos-ext-gmail-source 0.5.1 → 0.5.2

package/README.md

@@ -54,7 +54,7 @@ const filename = sanitizeMailMetadata(decodedFilename, { ...policy, maxOutputCha
 
 `sanitizeMailText(input, policy)` removes address tokens and credential-shaped content without a network call. `sanitizeMailMetadata` applies the same rules and folds whitespace for filenames, label names and display fields. `sanitizeMailHeaders` retains only Subject, Date, From, To and Cc; its Subject output is capped at 200 characters. MIME header decoding and structural validation must happen before these functions receive strings. Bcc is never returned.
 
-`policy` supplies `maxInputCharacters`, `maxLineCharacters` and `maxOutputCharacters` as positive UTF-16 code-unit counts. The input ceiling is 2,097,152 and the line ceiling is 65,536. The output cap cannot exceed the input cap. Oversized input throws `GmailTextFault` with a fixed reason; an overlong line is suppressed. Processing never scans only a truncated prefix for credentials. Repeated percent and common entity decoding is bounded, and unresolved encoded-byte lines are removed. Output truncation avoids splitting a surrogate pair and is marked in the result.
+`policy` supplies `maxInputCharacters`, `maxLineCharacters` and `maxOutputCharacters` as positive UTF-16 code-unit counts. The input ceiling is 2,097,152 and the line ceiling is 65,536. The output cap cannot exceed the input cap. Oversized input throws `GmailTextFault` with a fixed reason; an overlong line is suppressed. Processing never scans only a truncated prefix for credentials. Percent and character-reference decoding repeats at most four times, every named reference of the HTML standard and numeric references among them; a line that still holds an encoded byte or a known reference after those passes is removed. Padding, a run of spaces and invisible characters that holds anything other than ASCII spaces and tabs, such as the no-break spaces, zero-width non-joiners, combining grapheme joiners and braille blanks mail templates repeat after a preheader, becomes one space, or nothing when it holds no space at all, so zero-width characters between letters join them. Padding that holds an invisible character and ends a line leaves no space there; a run of no-break spaces alone keeps one, so a signature separator written with one still reads `-- `. Runs of ASCII spaces and tabs stay as written. Output truncation avoids splitting a surrogate pair and is marked in the result.
 
 `messageText(input, policy)` sanitizes first, trims recognized reply history and short signatures, then applies the output cap. Explicit forwarded content and ambiguous prose are retained. Its `trimmedQuote`, `trimmedSignature` and `truncated` flags describe omissions. Quote spans must refer to its returned string, without inserted summaries or an ellipsis presented as source text. A host should disclose that attachments are listed and their contents are not read when its mapper implements that boundary.
 
@@ -87,12 +87,14 @@ Construct `GmailThreadMapper(bounds)` with explicit response, decoded-byte, MIME
 Supply a fixed selection cutoff, message window, original read start and completion times, actual response cache headers, independently established cache permission and a host-generated relative detail path. The path must resolve authentication and current source generation in the host. It is never a Gmail deep link or a request-derived external URL. A successful OAuth grant does not establish cache or model-processing permission.
 Pass `labelCache` as the separate labels response’s cache decision. Missing, refused or expired label permission keeps ids without names. Permitted names shorten the entire document’s expiry when their allowed lifetime ends first.
 
-Each message is filtered by its internal timestamp and excluded labels. Retained messages are selected newest first with stable id ties, within the explicit cap, and rendered chronologically. Plain text wins within a MIME alternative. Other multipart containers retain their non-attachment text children in MIME order. HTML uses the public AgentOS `HtmlLoader` with an in-memory Buffer. No browser or remote resource is loaded. The resulting text, structured headers, filenames, media types and label names all pass the text sanitizer.
+Each message is filtered by its internal timestamp and excluded labels. Retained messages are selected newest first with stable id ties, within the explicit cap, and rendered chronologically. Plain text wins within a MIME alternative. Other multipart containers retain their non-attachment text children in MIME order. HTML uses the public AgentOS `HtmlLoader` with an in-memory Buffer; every character reference the loader leaves is then decoded, padding collapses and whitespace collapses again, with at most one blank line between paragraphs. No browser or remote resource is loaded. The resulting text, structured headers, filenames, media types and label names all pass the text sanitizer.
 
 Attachments are listed; their contents are not read. An external body or attachment contributes only sanitized filename, media type and size. Inline attachment bytes and attachment ids are discarded. Bcc, raw payloads, snippets and unknown fields are not returned. The mapper never calls any Gmail endpoint. Unsupported charset, invalid bytes and omitted content have fixed counted reasons. Invalid protocol shape is a failure rather than an empty successful source.
 
 A returned document has matching item/ref versions, a normalized-content hash, message ids and body spans, earliest allowed expiry and explicit omission/truncation metadata. Spans count UTF-16 code units in the returned text and exclude the header prefix. Quote trimming preserves normalized source text; no generated summary or ellipsis is inserted into a quote. Header-only messages can have an empty body span. Render structured metadata as escaped text and treat all mail as untrusted data.
 
+The version is the thread's history id and `contentHash`, the SHA-256 of the normalized text, metadata and message spans, so the hash is the field that carries a change to how text is normalized. A release that normalizes text differently changes the hash of exactly the threads whose normalized text, metadata or spans it changes. At the next read of such a thread, when history names it or a reconcile scan lists it, the account-source store finds a new version and writes the copy again. A thread whose history id, normalized text, metadata and message spans (each message's window end among them, so a changed `windowMs` changes the version too) are unchanged keeps its version, and the store, given a host's `refresh` binding, keeps that copy in place without embedding it again. No separate normalizer version is kept.
+
 A history-only change does not extend cache life or change the normalized hash. Storage still requires the account-source transactional fence, fresh expiry checks and host allowances. Derived results retain the whole source's personal-use, expiry and deletion restrictions. Never persist the transient MIME types or bypass the account-source store with a mapper result.
 
 ## Read-only API

package/dist/characters.d.ts

@@ -0,0 +1,39 @@
+/**
+ * @fileoverview One pass of character-reference decoding and the collapse of padding runs, shared by the HTML reader
+ * and the text sanitizer. A named reference is read from the HTML standard's table, and a numeric one when it names a
+ * Unicode scalar value. Padding is what mail templates repeat after a preheader so that a client's preview shows
+ * nothing more of the body: no-break spaces, zero-width non-joiners, combining grapheme joiners and braille blanks.
+ * Nothing here logs or fetches.
+ * @module agentos-ext-gmail-source/characters
+ */
+/**
+ * Decodes each character reference in `text` once, in one left-to-right pass: a named reference the HTML standard's
+ * table holds, and a decimal or hexadecimal one that names a Unicode scalar value. Any other reference stays as
+ * written, `&constructor;` and a number past U+10FFFF or naming a surrogate among them. Decoding never lengthens the
+ * text.
+ *
+ * @param text Any text: an HTML body after its tags are removed, a plain body or a header value.
+ * @returns The text with its known references replaced by the characters they stand for.
+ */
+export declare function decodeCharacterReferences(text: string): string;
+/**
+ * Whether `text` still holds a named reference the table decodes, as text nested deeper than the decoding passes
+ * does. One scan, each name looked up once.
+ *
+ * @param text One line of decoded text.
+ */
+export declare function holdsNamedReference(text: string): boolean;
+/**
+ * Collapses each run of spaces and invisible characters that holds anything other than ASCII spaces and tabs: to one
+ * space when the run holds a character with the width of a space (a no-break or other Unicode space, a tab or a
+ * braille blank), and to nothing when it holds only invisible characters, so zero-width characters between letters
+ * join them. A run that holds an invisible character and ends a line or the text also becomes nothing, so a
+ * preheader's padding leaves no space at the end of its line; a run of no-break spaces alone keeps its one space
+ * there, so a signature separator written with one still reads `-- `. A run of ASCII spaces and tabs alone stays as
+ * written, so a plain body keeps its indentation and alignment.
+ *
+ * @param text Text whose character references are already decoded.
+ * @returns The text with each padding run collapsed.
+ */
+export declare function collapsePadding(text: string): string;
+//# sourceMappingURL=characters.d.ts.map

package/dist/characters.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"characters.d.ts","sourceRoot":"","sources":["../src/characters.ts"],"names":[],"mappings":"AAAA;;;;;;;GAOG;AAwBH;;;;;;;;GAQG;AACH,wBAAgB,yBAAyB,CAAC,IAAI,EAAE,MAAM,GAAG,MAAM,CAQ9D;AAED;;;;;GAKG;AACH,wBAAgB,mBAAmB,CAAC,IAAI,EAAE,MAAM,GAAG,OAAO,CAKzD;AAED;;;;;;;;;;;GAWG;AACH,wBAAgB,eAAe,CAAC,IAAI,EAAE,MAAM,GAAG,MAAM,CAOpD"}

package/dist/characters.js

@@ -0,0 +1,83 @@
+/**
+ * @fileoverview One pass of character-reference decoding and the collapse of padding runs, shared by the HTML reader
+ * and the text sanitizer. A named reference is read from the HTML standard's table, and a numeric one when it names a
+ * Unicode scalar value. Padding is what mail templates repeat after a preheader so that a client's preview shows
+ * nothing more of the body: no-break spaces, zero-width non-joiners, combining grapheme joiners and braille blanks.
+ * Nothing here logs or fetches.
+ * @module agentos-ext-gmail-source/characters
+ */
+import { HTML_NAMED_REFERENCES } from './htmlNamedReferences.js';
+// The standard's names, with `‐` read as the ASCII hyphen-minus, as this sanitizer has read it since its first
+// release, so a credential prefix spelled with it is still recognized. A Map, so no name reads an object's prototype.
+const names = new Map([...Object.entries(HTML_NAMED_REFERENCES), ['hyphen', '-']]);
+// A hexadecimal, decimal or named reference that ends with a semicolon.
+const reference = /&(#x[0-9a-f]+|#[0-9]+|[a-z][a-z0-9]+);/gi;
+// A named reference that ends with a semicolon.
+const namedReference = /&([a-z][a-z0-9]+);/gi;
+// A run of horizontal spaces and invisible characters: tabs, every space separator (the no-break spaces among them),
+// the braille blank, every format character (the zero-width space, the joiners, the word joiner, the byte order mark,
+// the soft hyphen and the direction marks among them) and the combining grapheme joiner.
+const paddingRun = /[\t\p{Zs}\u2800\p{Cf}\u034f]+/gu;
+// A character of such a run that takes up the width of a space.
+const spacing = /[\t\p{Zs}\u2800]/u;
+// A character of such a run with no width of its own.
+const invisible = /[\p{Cf}\u034f]/u;
+// A line break, which ends a line.
+const lineBreak = /[\n\r\u2028\u2029]/;
+// A run of ASCII spaces and tabs alone.
+const asciiSpacing = /^[ \t]+$/;
+/**
+ * Decodes each character reference in `text` once, in one left-to-right pass: a named reference the HTML standard's
+ * table holds, and a decimal or hexadecimal one that names a Unicode scalar value. Any other reference stays as
+ * written, `&constructor;` and a number past U+10FFFF or naming a surrogate among them. Decoding never lengthens the
+ * text.
+ *
+ * @param text Any text: an HTML body after its tags are removed, a plain body or a header value.
+ * @returns The text with its known references replaced by the characters they stand for.
+ */
+export function decodeCharacterReferences(text) {
+    return text.replace(reference, (whole, entity) => {
+        if (!entity.startsWith('#'))
+            return names.get(entity) ?? whole;
+        const hex = entity[1]?.toLowerCase() === 'x';
+        const point = Number.parseInt(entity.slice(hex ? 2 : 1), hex ? 16 : 10);
+        return Number.isSafeInteger(point) && point >= 0 && point <= 0x10ffff && !(point >= 0xd800 && point <= 0xdfff)
+            ? String.fromCodePoint(point) : whole;
+    });
+}
+/**
+ * Whether `text` still holds a named reference the table decodes, as text nested deeper than the decoding passes
+ * does. One scan, each name looked up once.
+ *
+ * @param text One line of decoded text.
+ */
+export function holdsNamedReference(text) {
+    for (const match of text.matchAll(namedReference)) {
+        if (names.has(match[1] ?? ''))
+            return true;
+    }
+    return false;
+}
+/**
+ * Collapses each run of spaces and invisible characters that holds anything other than ASCII spaces and tabs: to one
+ * space when the run holds a character with the width of a space (a no-break or other Unicode space, a tab or a
+ * braille blank), and to nothing when it holds only invisible characters, so zero-width characters between letters
+ * join them. A run that holds an invisible character and ends a line or the text also becomes nothing, so a
+ * preheader's padding leaves no space at the end of its line; a run of no-break spaces alone keeps its one space
+ * there, so a signature separator written with one still reads `-- `. A run of ASCII spaces and tabs alone stays as
+ * written, so a plain body keeps its indentation and alignment.
+ *
+ * @param text Text whose character references are already decoded.
+ * @returns The text with each padding run collapsed.
+ */
+export function collapsePadding(text) {
+    return text.replace(paddingRun, (run, offset) => {
+        if (asciiSpacing.test(run))
+            return run;
+        if (!spacing.test(run))
+            return '';
+        const end = offset + run.length;
+        return invisible.test(run) && (end === text.length || lineBreak.test(text.charAt(end))) ? '' : ' ';
+    });
+}
+//# sourceMappingURL=characters.js.map

package/dist/characters.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"characters.js","sourceRoot":"","sources":["../src/characters.ts"],"names":[],"mappings":"AAAA;;;;;;;GAOG;AAEH,OAAO,EAAE,qBAAqB,EAAE,MAAM,0BAA0B,CAAC;AAEjE,sHAAsH;AACtH,sHAAsH;AACtH,MAAM,KAAK,GAAG,IAAI,GAAG,CAAiB,CAAC,GAAG,MAAM,CAAC,OAAO,CAAC,qBAAqB,CAAC,EAAE,CAAC,QAAQ,EAAE,GAAG,CAAC,CAAC,CAAC,CAAC;AACnG,wEAAwE;AACxE,MAAM,SAAS,GAAG,0CAA0C,CAAC;AAC7D,gDAAgD;AAChD,MAAM,cAAc,GAAG,sBAAsB,CAAC;AAC9C,qHAAqH;AACrH,sHAAsH;AACtH,yFAAyF;AACzF,MAAM,UAAU,GAAG,iCAAiC,CAAC;AACrD,gEAAgE;AAChE,MAAM,OAAO,GAAG,mBAAmB,CAAC;AACpC,sDAAsD;AACtD,MAAM,SAAS,GAAG,iBAAiB,CAAC;AACpC,mCAAmC;AACnC,MAAM,SAAS,GAAG,oBAAoB,CAAC;AACvC,wCAAwC;AACxC,MAAM,YAAY,GAAG,UAAU,CAAC;AAEhC;;;;;;;;GAQG;AACH,MAAM,UAAU,yBAAyB,CAAC,IAAY;IACpD,OAAO,IAAI,CAAC,OAAO,CAAC,SAAS,EAAE,CAAC,KAAK,EAAE,MAAc,EAAE,EAAE;QACvD,IAAI,CAAC,MAAM,CAAC,UAAU,CAAC,GAAG,CAAC;YAAE,OAAO,KAAK,CAAC,GAAG,CAAC,MAAM,CAAC,IAAI,KAAK,CAAC;QAC/D,MAAM,GAAG,GAAG,MAAM,CAAC,CAAC,CAAC,EAAE,WAAW,EAAE,KAAK,GAAG,CAAC;QAC7C,MAAM,KAAK,GAAG,MAAM,CAAC,QAAQ,CAAC,MAAM,CAAC,KAAK,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAE,GAAG,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC;QACxE,OAAO,MAAM,CAAC,aAAa,CAAC,KAAK,CAAC,IAAI,KAAK,IAAI,CAAC,IAAI,KAAK,IAAI,QAAQ,IAAI,CAAC,CAAC,KAAK,IAAI,MAAM,IAAI,KAAK,IAAI,MAAM,CAAC;YAC5G,CAAC,CAAC,MAAM,CAAC,aAAa,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,KAAK,CAAC;IAC1C,CAAC,CAAC,CAAC;AACL,CAAC;AAED;;;;;GAKG;AACH,MAAM,UAAU,mBAAmB,CAAC,IAAY;IAC9C,KAAK,MAAM,KAAK,IAAI,IAAI,CAAC,QAAQ,CAAC,cAAc,CAAC,EAAE,CAAC;QAClD,IAAI,KAAK,CAAC,GAAG,CAAC,KAAK,CAAC,CAAC,CAAC,IAAI,EAAE,CAAC;YAAE,OAAO,IAAI,CAAC;IAC7C,CAAC;IACD,OAAO,KAAK,CAAC;AACf,CAAC;AAED;;;;;;;;;;;GAWG;AACH,MAAM,UAAU,eAAe,CAAC,IAAY;IAC1C,OAAO,IAAI,CAAC,OAAO,CAAC,UAAU,EAAE,CAAC,GAAW,EAAE,MAAc,EAAE,EAAE;QAC9D,IAAI,YAAY,CAAC,IAAI,CAAC,GAAG,CAAC;YAAE,OAAO,GAAG,CAAC;QACvC,IAAI,CAAC,OAAO,CAAC,IAAI,CAAC,GAAG,CAAC;YAAE,OAAO,EAAE,CAAC;QAClC,MAAM,GAAG,GAAG,MAAM,GAAG,GAAG,CAAC,MAAM,CAAC;QAChC,OAAO,SAAS,CAAC,IAAI,CAAC,GAAG,CAAC,IAAI,CAAC,GAAG,KAAK,IAAI,CAAC,MAAM,IAAI,SAAS,CAAC,IAAI,CAAC,IAAI,CAAC,MAAM,CAAC,GAAG,CAAC,CAAC,CAAC,CAAC,CAAC,CAAC,EAAE,CAAC,CAAC,CAAC,GAAG,CAAC;IACrG,CAAC,CAAC,CAAC;AACL,CAAC"}

package/dist/htmlNamedReferences.d.ts

@@ -0,0 +1,13 @@
+/**
+ * @fileoverview Every named character reference the HTML standard decodes when it ends with a semicolon, 2,125
+ * names, each with the characters it stands for. Generated from https://html.spec.whatwg.org/entities.json, read 18
+ * October 2026 (SHA-256 d741d877ac77c4194c4ad526b5b4a19aef8dfe411ab840a466891cdbb9f362e6), and identical to the map
+ * of the `entities` npm package 2.2.0. The standard says the list "is static and will not be expanded or changed in the
+ * future" (https://html.spec.whatwg.org/multipage/named-characters.html). The names a browser also reads without the
+ * semicolon are read here only with it. Characters outside printable ASCII are written as escapes, so no invisible
+ * character sits in the source.
+ * @module agentos-ext-gmail-source/htmlNamedReferences
+ */
+/** The characters each named reference stands for, by its name without the `&` and the `;`. */
+export declare const HTML_NAMED_REFERENCES: Readonly<Record<string, string>>;
+//# sourceMappingURL=htmlNamedReferences.d.ts.map

package/dist/htmlNamedReferences.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"htmlNamedReferences.d.ts","sourceRoot":"","sources":["../src/htmlNamedReferences.ts"],"names":[],"mappings":"AAAA;;;;;;;;;GASG;AAEH,+FAA+F;AAC/F,eAAO,MAAM,qBAAqB,EAAE,QAAQ,CAAC,MAAM,CAAC,MAAM,EAAE,MAAM,CAAC,CAwXjE,CAAC"}