@substrat-run/attachment-extractors 0.1.0 → 0.1.2

package/README.md

@@ -30,13 +30,26 @@ extractors" a valid configuration, not a broken one.
 | Extractor | Reads |
 |---|---|
 | `textExtractor` | `text/*` other than HTML, and `.txt` `.md` `.csv` `.tsv` when the type says nothing |
-| `htmlExtractor` | `text/html`, `application/xhtml+xml`: the visible text, with script, style and comments dropped |
+| `htmlExtractor` | `text/html`, `application/xhtml+xml`: the visible text, each construct ended where a browser's tokenizer ends it; script, style, template content and comments dropped |
 | `docxExtractor` | Word documents: the body, then footnotes, endnotes, headers and footers |
 | `xlsxExtractor` | Spreadsheets: shared and inline strings, never a cell's number |
 | `pptxExtractor` | Presentations: slides in order, then speaker notes |
 
 There is no PDF extractor yet, and nothing is OCR'd.
 
+The HTML extractor **never indexes text the HTML parser keeps out of the rendered document**:
+comments, `script`, `style`, the other elements whose content a browser does not render, and
+`template` content. Where it does not model how a browser parses, it **under-indexes** rather
+than guess. It indexes nothing inside a `select` or an inline `svg` or `math`, and nothing after
+a `frameset` unless it can prove a browser ignores it (and an honoured frameset discards the
+body, so what was read in it is dropped too). After such a context it resumes only where it is certain a browser ends
+it, and otherwise indexes nothing more of the file. Its tests hold this against parse5, a
+browser-grade HTML parser.
+
+It follows the parser, not the renderer. It does not evaluate the `hidden` attribute, CSS or
+interactive state (a closed `<details>`), so text those hide **is** indexed. That is never more
+than a searcher can read: search is gated by the same read permission as opening the file.
+
 ## Bounds
 
 The bounds that protect the parsing process live here. `maxInputBytes` (32 MiB) is declared to
@@ -47,8 +60,9 @@ extractor returns.
 
 Every parser is a single forward scan over a capped input, so its CPU is bounded by the code
 rather than the file. The kernel's time budget is cooperative. The parsers check the `signal`
-they are handed and yield between zip entries and every 256 KiB of progress, so an aborted
-extraction stops within one such step.
+they are handed and yield between zip entries and at least every `EXTRACTION_STRIDE` (the
+kernel's, 256 K) units of work — every search included, so a single long comment or tag is
+no exception — and an aborted extraction stops within one stride.
 
 ## License
 

package/dist/index.d.ts

@@ -0,0 +1,98 @@
+/**
+ * `@substrat-run/attachment-extractors` — the parsers behind the kernel's attachment
+ * extractor seam (#1575, K-43).
+ *
+ * **The kernel indexes attachment text; it does not parse file formats** (K-43). It owns the
+ * text table, its index, the job, the outcome states, the output cap and the gated search,
+ * and defines one seam, `AttachmentExtractor`. Everything that reads a file FORMAT lives
+ * here, and the kernel and the adapters never import this package (`lint:deps` refuses it):
+ * whoever constructs the host passes `defaultAttachmentExtractors()` in, or a list of its
+ * own. Untrusted-input parsing belongs at the edge, not beside the permission checker.
+ *
+ * Best-effort by content type, and explicit about what it does not do:
+ *
+ * - **text** (`text/*` other than HTML): decoded as the declared charset, UTF-8 otherwise.
+ * - **html** (`text/html`, `application/xhtml+xml`): tags stripped, each construct ended where
+ *   the HTML tokenizer ends it; comments, `script`, `style`, the other raw-text elements a
+ *   browser does not render, and `template` content, dropped — an unclosed one through to the
+ *   end; entities decoded. **Its contract: it never indexes text the HTML parser keeps out of
+ *   the rendered document**, and it may under-index what it does not model — inline SVG and
+ *   MathML, `select`, a honoured frameset — indexing nothing inside them, and nothing after one
+ *   whose end it cannot be sure of (`htmlText` lists the rules). It does not evaluate the
+ *   `hidden` attribute, CSS or interactive state, so text those hide is indexed.
+ * - **docx / xlsx / pptx**: an OOXML file is a zip of XML parts, so this reads the zip's
+ *   central directory, inflates only the parts that carry text, and keeps the text of
+ *   their `t` elements. The inflate is the web-standard `DecompressionStream('deflate-raw')`,
+ *   present in Node and workerd alike, so the whole format costs no dependency.
+ * - **No PDF yet, and no OCR.** A host given no extractor for a type records `unsupported`
+ *   with that reason — a legible outcome, never "indexed, no match".
+ *
+ * ## The bounds this package owns, and why
+ *
+ * The bounds that protect the PROCESS doing the parsing live here (K-43); the one that
+ * protects the scope — the output cap — is the kernel's, enforced on whatever comes back.
+ *
+ * - `maxInputBytes` is declared on each extractor, so the kernel can refuse an oversized
+ *   file on its RECORDED size before fetching a byte; each extractor checks it again.
+ * - `maxInflatedBytes` is the zip-bomb cap: a counter on the bytes the inflater actually
+ *   produces, across every part of one file. An entry's declared size is checked first as
+ *   a cheap refusal, and never trusted, because a bomb is exactly a file that lies about it.
+ *
+ * A file that is not what it says, or breaks a bound, is answered `{ failed }` with a
+ * content-free reason — never a throw, and never a quote of the file: the reason lands in
+ * the scope, and the content it would quote is what the read gate protects.
+ *
+ * ## CPU, bounded by construction — and stoppable
+ *
+ * Every parser here is a LINEAR scan (no regular expression that can rescan from each
+ * opener), over an input with a fixed cap, so the worst case is a property of the code rather
+ * than of the file. Per format, at the default bounds:
+ *
+ * - **text**: a decode of at most 4 × the kernel's output cap (2 MiB).
+ * - **html**: a decode of at most 16 × the output cap (8 MiB), one forward scan of it, one
+ *   entity pass over what survives.
+ * - **docx / xlsx / pptx**: the central directory (at most 20 000 entries), inflate and decode
+ *   of at most `maxInflatedBytes` (16 MiB) in total, and one forward scan of each inflated
+ *   part; parts stop being read once the text collected is twice the output cap.
+ *
+ * Each extractor honours the kernel's time budget cooperatively (`Pace`): it checks the
+ * `signal`, and yields a turn of the event loop so the kernel's timer can fire, between zip
+ * entries and at least every `EXTRACTION_STRIDE` (the kernel's, 256 Ki) units of work. That
+ * holds for every pass over the content — decoding, inflating, the zip directory, the entity
+ * pass, and each search for a construct's end, a native `indexOf` included, which is cut to
+ * the window left before the next check. An aborted extraction answers
+ * `{ failed: 'the extraction was aborted' }` within one stride. The steps between two checks
+ * that are NOT cut to a stride are copies (joining the text collected) and the name tests
+ * over the zip directory, at most 20 000 names of at most 255 bytes each.
+ */
+import { type AttachmentExtractor } from '@substrat-run/kernel';
+/** The bounds that protect the parsing process (K-43). */
+export interface ExtractorBounds {
+    /** Largest file an extractor will read, declared to the kernel and checked again here. */
+    readonly maxInputBytes: number;
+    /** Bytes the inflater may produce across every part of one file (the zip-bomb cap). */
+    readonly maxInflatedBytes: number;
+}
+/**
+ * The defaults. 16 MiB of inflated XML is far more than the text of any ordinary office
+ * document, and small enough that decoding it fits a Worker's memory.
+ */
+export declare const DEFAULT_EXTRACTOR_BOUNDS: ExtractorBounds;
+/** `text/*` other than HTML — and `.txt`, `.md`, `.csv`, `.tsv` when the type says nothing. */
+export declare function textExtractor(bounds?: ExtractorBounds): AttachmentExtractor;
+/** HTML and XHTML — the text a reader of the page would see. */
+export declare function htmlExtractor(bounds?: ExtractorBounds): AttachmentExtractor;
+/** Word documents: the body, then footnotes, endnotes, headers and footers. */
+export declare const docxExtractor: (bounds?: ExtractorBounds) => AttachmentExtractor;
+/** Spreadsheets: shared strings and inline strings — never a cell's number. */
+export declare const xlsxExtractor: (bounds?: ExtractorBounds) => AttachmentExtractor;
+/** Presentations: slides in order, then their speaker notes. */
+export declare const pptxExtractor: (bounds?: ExtractorBounds) => AttachmentExtractor;
+/**
+ * Every extractor this package has, in the order a host should try them — what a host is
+ * constructed with when it has no list of its own:
+ *
+ *     new SqliteScopeHost({ dir, attachmentExtractors: defaultAttachmentExtractors() })
+ */
+export declare function defaultAttachmentExtractors(bounds?: ExtractorBounds): AttachmentExtractor[];
+//# sourceMappingURL=index.d.ts.map

package/dist/index.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;;GAkEG;AACH,OAAO,EAEL,KAAK,mBAAmB,EAIzB,MAAM,sBAAsB,CAAC;AAsB9B,0DAA0D;AAC1D,MAAM,WAAW,eAAe;IAC9B,0FAA0F;IAC1F,QAAQ,CAAC,aAAa,EAAE,MAAM,CAAC;IAC/B,uFAAuF;IACvF,QAAQ,CAAC,gBAAgB,EAAE,MAAM,CAAC;CACnC;AAED;;;GAGG;AACH,eAAO,MAAM,wBAAwB,EAAE,eAGtC,CAAC;AAmrCF,+FAA+F;AAC/F,wBAAgB,aAAa,CAAC,MAAM,GAAE,eAA0C,GAAG,mBAAmB,CAUrG;AAED,gEAAgE;AAChE,wBAAgB,aAAa,CAAC,MAAM,GAAE,eAA0C,GAAG,mBAAmB,CAOrG;AAWD,+EAA+E;AAC/E,eAAO,MAAM,aAAa,YAAY,eAAe,wBAAoE,CAAC;AAC1H,+EAA+E;AAC/E,eAAO,MAAM,aAAa,YAAY,eAAe,wBAAoE,CAAC;AAC1H,gEAAgE;AAChE,eAAO,MAAM,aAAa,YAAY,eAAe,wBAAoE,CAAC;AAE1H;;;;;GAKG;AACH,wBAAgB,2BAA2B,CAAC,MAAM,GAAE,eAA0C,GAAG,mBAAmB,EAAE,CAErH"}