@cyanheads/secedgar-mcp-server 0.15.12 → 0.15.13

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (74) hide show
  1. package/AGENTS.md +7 -6
  2. package/CLAUDE.md +7 -6
  3. package/README.md +36 -36
  4. package/changelog/0.15.x/0.15.13.md +33 -0
  5. package/dist/config/server-config.d.ts.map +1 -1
  6. package/dist/config/server-config.js +47 -0
  7. package/dist/config/server-config.js.map +1 -1
  8. package/dist/mcp-server/tools/definitions/company-search.tool.d.ts +7 -0
  9. package/dist/mcp-server/tools/definitions/company-search.tool.d.ts.map +1 -1
  10. package/dist/mcp-server/tools/definitions/company-search.tool.js +8 -0
  11. package/dist/mcp-server/tools/definitions/company-search.tool.js.map +1 -1
  12. package/dist/mcp-server/tools/definitions/compare-companies.tool.d.ts +7 -0
  13. package/dist/mcp-server/tools/definitions/compare-companies.tool.d.ts.map +1 -1
  14. package/dist/mcp-server/tools/definitions/compare-companies.tool.js +8 -0
  15. package/dist/mcp-server/tools/definitions/compare-companies.tool.js.map +1 -1
  16. package/dist/mcp-server/tools/definitions/fetch-frames.tool.d.ts +7 -0
  17. package/dist/mcp-server/tools/definitions/fetch-frames.tool.d.ts.map +1 -1
  18. package/dist/mcp-server/tools/definitions/fetch-frames.tool.js +8 -0
  19. package/dist/mcp-server/tools/definitions/fetch-frames.tool.js.map +1 -1
  20. package/dist/mcp-server/tools/definitions/find-holders.tool.d.ts +7 -0
  21. package/dist/mcp-server/tools/definitions/find-holders.tool.d.ts.map +1 -1
  22. package/dist/mcp-server/tools/definitions/find-holders.tool.js +8 -0
  23. package/dist/mcp-server/tools/definitions/find-holders.tool.js.map +1 -1
  24. package/dist/mcp-server/tools/definitions/get-beneficial-owners.tool.d.ts +7 -0
  25. package/dist/mcp-server/tools/definitions/get-beneficial-owners.tool.d.ts.map +1 -1
  26. package/dist/mcp-server/tools/definitions/get-beneficial-owners.tool.js +8 -0
  27. package/dist/mcp-server/tools/definitions/get-beneficial-owners.tool.js.map +1 -1
  28. package/dist/mcp-server/tools/definitions/get-filing.tool.d.ts +7 -0
  29. package/dist/mcp-server/tools/definitions/get-filing.tool.d.ts.map +1 -1
  30. package/dist/mcp-server/tools/definitions/get-filing.tool.js +177 -45
  31. package/dist/mcp-server/tools/definitions/get-filing.tool.js.map +1 -1
  32. package/dist/mcp-server/tools/definitions/get-financials.tool.d.ts +7 -0
  33. package/dist/mcp-server/tools/definitions/get-financials.tool.d.ts.map +1 -1
  34. package/dist/mcp-server/tools/definitions/get-financials.tool.js +8 -0
  35. package/dist/mcp-server/tools/definitions/get-financials.tool.js.map +1 -1
  36. package/dist/mcp-server/tools/definitions/get-fund-holdings.tool.d.ts +7 -0
  37. package/dist/mcp-server/tools/definitions/get-fund-holdings.tool.d.ts.map +1 -1
  38. package/dist/mcp-server/tools/definitions/get-fund-holdings.tool.js +8 -0
  39. package/dist/mcp-server/tools/definitions/get-fund-holdings.tool.js.map +1 -1
  40. package/dist/mcp-server/tools/definitions/get-insider-transactions.tool.d.ts +7 -0
  41. package/dist/mcp-server/tools/definitions/get-insider-transactions.tool.d.ts.map +1 -1
  42. package/dist/mcp-server/tools/definitions/get-insider-transactions.tool.js +8 -0
  43. package/dist/mcp-server/tools/definitions/get-insider-transactions.tool.js.map +1 -1
  44. package/dist/mcp-server/tools/definitions/get-institutional-holdings.tool.d.ts +7 -0
  45. package/dist/mcp-server/tools/definitions/get-institutional-holdings.tool.d.ts.map +1 -1
  46. package/dist/mcp-server/tools/definitions/get-institutional-holdings.tool.js +8 -0
  47. package/dist/mcp-server/tools/definitions/get-institutional-holdings.tool.js.map +1 -1
  48. package/dist/mcp-server/tools/definitions/get-material-events.tool.d.ts +7 -0
  49. package/dist/mcp-server/tools/definitions/get-material-events.tool.d.ts.map +1 -1
  50. package/dist/mcp-server/tools/definitions/get-material-events.tool.js +8 -0
  51. package/dist/mcp-server/tools/definitions/get-material-events.tool.js.map +1 -1
  52. package/dist/mcp-server/tools/definitions/get-snapshot.tool.d.ts +7 -0
  53. package/dist/mcp-server/tools/definitions/get-snapshot.tool.d.ts.map +1 -1
  54. package/dist/mcp-server/tools/definitions/get-snapshot.tool.js +8 -0
  55. package/dist/mcp-server/tools/definitions/get-snapshot.tool.js.map +1 -1
  56. package/dist/mcp-server/tools/definitions/index.d.ts +91 -0
  57. package/dist/mcp-server/tools/definitions/index.d.ts.map +1 -1
  58. package/dist/mcp-server/tools/definitions/search-filings.tool.d.ts +7 -0
  59. package/dist/mcp-server/tools/definitions/search-filings.tool.d.ts.map +1 -1
  60. package/dist/mcp-server/tools/definitions/search-filings.tool.js +8 -0
  61. package/dist/mcp-server/tools/definitions/search-filings.tool.js.map +1 -1
  62. package/dist/services/edgar/concept-series.d.ts.map +1 -1
  63. package/dist/services/edgar/concept-series.js +5 -1
  64. package/dist/services/edgar/concept-series.js.map +1 -1
  65. package/dist/services/edgar/edgar-api-service.d.ts +21 -11
  66. package/dist/services/edgar/edgar-api-service.d.ts.map +1 -1
  67. package/dist/services/edgar/edgar-api-service.js +132 -33
  68. package/dist/services/edgar/edgar-api-service.js.map +1 -1
  69. package/dist/services/edgar/filing-to-text.d.ts +73 -8
  70. package/dist/services/edgar/filing-to-text.d.ts.map +1 -1
  71. package/dist/services/edgar/filing-to-text.js +179 -31
  72. package/dist/services/edgar/filing-to-text.js.map +1 -1
  73. package/package.json +3 -3
  74. package/server.json +3 -3
package/AGENTS.md CHANGED
@@ -1,7 +1,7 @@
1
1
  # Agent Protocol
2
2
 
3
3
  **Server:** secedgar-mcp-server
4
- **Version:** 0.15.12
4
+ **Version:** 0.15.13
5
5
  **Framework:** [@cyanheads/mcp-ts-core](https://www.npmjs.com/package/@cyanheads/mcp-ts-core) `^0.13.15`
6
6
  **Engines:** Bun ≥1.4.0, Node ≥24.0.0
7
7
  **MCP SDK:** `@modelcontextprotocol/server` ^2.3.1
@@ -82,7 +82,7 @@ Tailor suggestions to what's actually missing or stale — don't recite the full
82
82
 
83
83
  ## Domain Notes
84
84
 
85
- - **Rate limit:** 10 req/s per IP. Every SEC request goes through one process-wide `createPacer` queue in `EdgarApiService` that spaces request *starts* `1000 / EDGAR_RATE_LIMIT_RPS` ms apart, with no concurrency cap and no wait budget — an ordinary burst drains at the configured rate and is never shed, and this server's own traffic can never exceed it, so `rateLimitRps` stays at 10 and headroom for a co-tenant on the outbound IP is the operator's call via `EDGAR_RATE_LIMIT_RPS`. A 429 is **not** retried: SEC holds the block until the request rate has stayed under the threshold for ten minutes and every request sent meanwhile restarts that clock, so the ~3s retry budget (`MAX_RETRIES` 3 × `BASE_BACKOFF_MS` 1000, exponential) cannot outlast it (#112). Instead a 429 closes the service's **block gate** for `EDGAR_RATE_LIMIT_COOLDOWN_SECONDS` (default 600). While it is closed nothing is sent: a call fails locally as `RateLimited` with `retryAfter` set to the whole seconds remaining (never below 1), and the gate is re-read at dispatch, so a call already queued in the pacer when the 429 landed — or a 5xx retry sleeping through its backoff — is refused too. The first call after the cool-down is sent alone as a probe while every other call waits on it: any HTTP answer but a 429 reopens normal pacing, a probe 429 closes the gate for the full cool-down again, and a probe that gets no HTTP answer (a network error) proves nothing, so the next call probes in its place. **The gate lives in the service, not in the pacer's `cooldown` option** — that gate holds queued calls until it reopens and then releases them all at the pacing rate, so it can neither refuse locally nor keep the reopening to one request. The upstream 429 and the local refusal carry one `data.reason: 'rate_limited'`, declared (`thrownBy: 'service'`, `retryable: true`) on every tool that reaches SEC, plus `data.retryable: true` — `ctx.fail` copies a contract entry's `retryable` onto `data`, a service throw does not, so both throw sites set it themselves (#122) — and one recovery hint naming SEC's ten-minute window and the seconds to wait; SEC's HTML block page stays out of the payload (`captureBody: false`). Mirror-served reads (the ticker base behind `resolveCik`, and `tryGetCompanyFacts` / `tryGetCompanyConcept` / `tryGetFrames` through `mirrorOrLive`) never reach `rawFetch`, so they keep answering during a block. 500/502/503/504 still retry with the backoff. `createApp({ teardown })` disposes the pacer through `disposeEdgarApiService()` (#116)
85
+ - **Rate limit:** 10 req/s per IP. Every SEC request goes through one process-wide `createPacer` queue in `EdgarApiService` that spaces request *starts* `1000 / EDGAR_RATE_LIMIT_RPS` ms apart, with no concurrency cap and no wait budget — an ordinary burst drains at the configured rate and is never shed, and this server's own traffic can never exceed it, so `rateLimitRps` stays at 10 and headroom for a co-tenant on the outbound IP is the operator's call via `EDGAR_RATE_LIMIT_RPS`. A 429 is **not** retried: SEC holds the block until the request rate has stayed under the threshold for ten minutes and every request sent meanwhile restarts that clock, so the ~3s retry budget (`MAX_RETRIES` 3 × `BASE_BACKOFF_MS` 1000, exponential) cannot outlast it (#112). Instead a 429 closes the service's **block gate** for `EDGAR_RATE_LIMIT_COOLDOWN_SECONDS` (default 600). While it is closed nothing is sent: a call fails locally as `RateLimited` with `retryAfter` set to the whole seconds remaining (never below 1), and the gate is re-read at dispatch, so a call already queued in the pacer when the 429 landed — or a 5xx or no-response retry sleeping through its backoff — is refused too. The first call after the cool-down is sent alone as a probe while every other call waits on it: any HTTP answer but a 429 reopens normal pacing, a probe 429 closes the gate for the full cool-down again, and a probe that gets no HTTP answer (a network error, or no headers within the per-attempt cap) proves nothing, so the next call probes in its place — and every call that waited on it counts that probe as one of its own no-response attempts, so calls queued behind a silent SEC each settle within their own three-attempt ladder instead of probing in turn, 15 s apiece (#173). **The gate lives in the service, not in the pacer's `cooldown` option** — that gate holds queued calls until it reopens and then releases them all at the pacing rate, so it can neither refuse locally nor keep the reopening to one request. The upstream 429 and the local refusal carry one `data.reason: 'rate_limited'`, declared (`thrownBy: 'service'`, `retryable: true`) on every tool that reaches SEC, plus `data.retryable: true` — `ctx.fail` copies a contract entry's `retryable` onto `data`, a service throw does not, so both throw sites set it themselves (#122) — and one recovery hint naming SEC's ten-minute window and the seconds to wait; SEC's HTML block page stays out of the payload (`captureBody: false`). Mirror-served reads (the ticker base behind `resolveCik`, and `tryGetCompanyFacts` / `tryGetCompanyConcept` / `tryGetFrames` through `mirrorOrLive`) never reach `rawFetch`, so they keep answering during a block. 500/502/503/504 still retry with the backoff, and so does an attempt that gets no HTTP response at all — a `fetch` rejection (Bun sets the transport code on the error itself, Node's undici on its `cause`) or no response headers within `ATTEMPT_HEADERS_TIMEOUT_MS` (15 s from dispatch; pacer queue time and the body read are outside it), which holds one request to 48 s against the MCP client's 60 s default. When the last attempt gets no HTTP response — whatever the earlier ones got — the call fails as `ServiceUnavailable` with `data.reason: 'sec_unreachable'`, `retryable: true`, `url`, and a hint, declared beside `rate_limited` on the same tools; the message names host, attempts, and the transport code or the timeout, and the runtime's own text rides `cause` to the log only (#173). `createApp({ teardown })` disposes the pacer through `disposeEdgarApiService()` (#116)
86
86
  - **Parameter names:** one name per concept across tools — `company` (the entity a tool reads; on `get_institutional_holdings` that is the 13F filer), `filed_after` / `filed_before` (inclusive filing-date bounds), `forms`. Every tool carrying one of those keys declares `inputAliases` for the other spellings in use — `ticker`, `cik`, `ticker_or_cik` → `company`; `start_date`, `date_from` → `filed_after`; `end_date`, `date_to` → `filed_before`; `form_types` → `forms` — so a caller carrying a name from one tool to the next is not rejected (#115). An alias is declared only where it maps to exactly one key of that tool and its value shape fits: a singular `form_type` cannot fill the `forms` array, so it is left out rather than widening the schema. A tool whose only company input carries another name takes the same company-identifier spellings onto it — `company`, `ticker`, `cik`, `ticker_or_cik` → `issuer` on `find_holders` and `get_beneficial_owners`, and those plus `name` and `search` → `query` on `company_search` — since the key it moves into has no other reading there; the canonical names stay, because renaming them would break existing callers (#152). `fund` (a fund or series) and `companies` (an array) are not aliased. `search_filings` still filters by date only with both bounds; `company_search`, `get_material_events`, and `get_insider_transactions` accept either alone
87
87
  - **User-Agent required:** `"AppName contact@email.com"` on every SEC request or IP gets blocked
88
88
  - **CIK zero-padding:** the `data.sec.gov` APIs (submissions, companyfacts, companyconcept) require the 10-digit zero-padded CIK (`String(cik).padStart(10, '0')`). Archive paths under `www.sec.gov/Archives/edgar/data/` take it unpadded — SEC answers the padded path with a 301, and `fetch` follows it inside the same pacer slot, so every archive read would cost two requests. `filingArchiveUrl` (exported from `edgar-api-service.ts`) strips the zeros for all six archive reads and for `get_filing`'s `filing_url` (#156)
@@ -102,9 +102,9 @@ Tailor suggestions to what's actually missing or stale — don't recite the full
102
102
  - **Submissions archive walk:** the submissions feed's `filings.recent` holds the last year or 1,000 filings, whichever is more — so 26,000+ rows for JPMorgan, and a filing day is never split (Microsoft's window runs to 1,002 rows to finish its oldest day); everything older lives on `filings.files[]` archive pages of 200–360 KB each. Every tool that reads past `recent` goes through `SubmissionsArchiveWalk` (`submissions-archive.ts`): one `ARCHIVE_PAGE_SCAN_CAP` (10) for all, pages read one at a time through the service's pacer and page cache, and no page selected when the lower bound falls on or after `recent`'s oldest date — so a call answered by `recent` sends no archive request. The caller picks the order — `newest-first` (`company_search`, `search_filings`' pre-2001 arm, `get_material_events`, `get_insider_transactions`' window), or `oldest-first` forward from a date (`get_institutional_holdings`' quarter lookup, since newest-first cannot reach JPMorgan's 2020-Q4 13F-HR on page 41 of 70 under the cap) — and can stop after any page; `scannedThrough` (the oldest page start read) and `truncated` (selected pages left unread, whether the cap, an early stop, or a caller that never started the walk left them) are reported the same way everywhere. Worst case per call is the submissions document plus 10 pages plus the tool's own document reads (`get_insider_transactions`: 111). `get_material_events` without a date window walks only to fill `limit`: it enters the walk when fewer than `limit` `recent` rows pass the `items` filter — the same count as the stop test, so unrelated recent 8-Ks never stand in for matches (#139) — and stops on the page that fills it with rows passing that filter, and its `dataset.truncated` is true whenever selected pages went unread — including when `recent` alone fills `limit` and no page is read, since the dataframe then holds only the recent window (#134); a result short of `limit` with selected pages unread (only the cap ends a walk that short) gets a notice naming the pages read, the oldest date reached, and the lever — `filed_before=<that date>` alone or in a window when undated, narrowing the window with `filed_before=<that date>` when dated, since a newest-first walk reaches older filings only through an earlier upper bound — appended to the zero-match or empty-window notice when one fires, since `ctx.enrich.notice` is last-wins; an empty-window notice the cap cut short claims no 8-K only "as far back as the scan reached" and drops its widen-the-range advice (#139); `get_institutional_holdings` stops on the first page holding a 13F-HR or 13F-NT for the quarter, and runs the #86 operating-company routing only when neither `recent` nor the pages read hold any 13F-HR or 13F-NT — a notice-only manager asked for a quarter it filed nothing for gets the plain quarter miss (#117, #133); without a `quarter`, a manager whose `recent` holds 13F-NT notices and no 13F-HR gets the notice message for its newest notice rather than the #86 classification (#133). **Page selection widens both date bounds by a day:** a manifest page's `filingFrom` / `filingTo` can sit a day off its own rows in either direction (JPMorgan's page 041 is listed to 2021-02-17 but holds 42 filings dated 2021-02-18; Apple's page 001 is listed a day past its newest row), so an exact overlap test skipped the page holding a window's edge day. The slack costs at most one extra page per bound, and every caller filters rows by their own filing dates (#137)
103
103
  - **Filing metadata outside `recent`:** `get_filing` takes `form`, `filing_date`, and `period_ending` from `recent` when the accession is there, else from the filing's own SEC header — the raw `<TYPE>` / `<FILING-DATE>` / `<PERIOD>` in `index-headers.html`, which the tool already fetches to type the documents, and only when that page 404s (common before 2014) the sub-1 KB `<accession>.hdr.sgml`. `FILING-DATE` is the filing date, not the acceptance timestamp, and matches the submissions feed; an S-8 or Form 4 header has no `PERIOD`. `company_name` stays the current registrant name (#126)
104
104
  - **Primary document (#161):** `get_filing`'s `primary_document`, the document it reads when `document` is omitted, and the primary its error catalogs name is the name SEC records (`recordedPrimaryNames`): the submissions feed's `primaryDocument` when the accession sits in `recent` (last path segment — an XML form is recorded through its stylesheet path, `xslF345X06/form4.xml`), else the document the header page types with the submission's form. Only with neither does the largest readable document stand in, and the viewer-page guard there matches `R\d+.html?`, not every name starting with `R` (`Report10K.htm`). Size alone picked an 8-K's press-release exhibit over the form and a 13F-HR's information table over its cover page. The header page and the feed are read together right after an index is found and before the body, since every outcome — the error catalogs included — names the primary from them
105
- - **Filing content:** HTML → text via `html-to-text` library; pre-2005 filings produce noisier output. The library walks the DOM recursively, so `CONVERT_OPTIONS` caps it at `limits.maxDepth: 512` with a `[…]` ellipsis where the cut falls — without it deep markup overflowed the stack and one such candidate failed `search_filings`' whole pre-2001 text scan (#118). The bound must hold on both runtimes the server ships under: a fresh Node process overflows near 1,360 levels of nested lists (its most stack-hungry shape), Bun near 13,000; real HTML filings measure under 25 levels. Legacy SGML `<PAGE>` markers are replaced with a line break before parsing (`neutralizePageMarkers`) — the parser never closes them, so a text document otherwise nests one level per page and a long one would be cut at the cap; with them gone the deepest measured `.txt` sits under 100 (an EX-27 schedule's unclosed tags). Both changes leave every sampled filing's text byte-identical. `limits` deep-merges with the library defaults, so its 16,777,216-character `maxInputLength` (silent truncation, no ellipsis) still applies. **Plain-text SGML bodies keep their lines (#136):** the parser collapses every newline run to a space, so a plain `<TEXT>` body — a pre-2001 full submission, a 2000s ASCII primary document — lost the line structure `detectHeadings` anchors on (Apple's 1996 10-K came back as 104 lines with an empty outline). `preservePlainTextBodies` runs inside the one `convert()` call: a body is HTML when `<html`, `<body`, `<p`, `<div`, `<br`, `<font`, `<tr`, `<td`, or `<xml` opens anywhere in it (`\b`-bounded, so `<PAGE>`, `<PDF>`, and EX-27's `<PP&E>` don't trip `<p`; no `<table`, since EDGAR ASCII tables are `<TABLE>`); any other body has every tag dropped, `&`/`<`/`>` escaped, and goes in a `<pre>`. The list is broad because misreading HTML as plain wrecks it, while the reverse is only the old output. Tags are dropped rather than parsed: the parser formats `<TABLE>`/`<CAPTION>` as blocks inside the `<pre>`, and a stray `<word` in prose would swallow text up to the next `>`. The `<TEXT>` wrapper matches case-sensitively — EDGAR writes it uppercase, and an inline SVG `<text>` is not a body — and the scan is one forward `indexOf` pass, since a lazy `<TEXT>([\s\S]*?)</TEXT>` regex goes quadratic on openers with no closer. Input with no plain body reaches the parser unchanged, so current HTML and 2000–2004 SGML-wrapped HTML convert byte-identically; their outlines change only where an unpunctuated Item cross-reference opens a line and drops out as wrapped prose (see Heading detection). Kept indentation grows plain extractions about 19%. **Every document of a submission holding an HTML document (#159):** the parser converts only `<body>` elements when the input has one, so an HTML-era submission (HTML primary, plain EX-10/EX-27 exhibits) lost every plain exhibit, and the pre-2001 scan could not match exhibit text. When the input holds `<body`, each plain body's `<pre>` goes inside a `<body>` of its own, so every document converts in document order, separated by the block break the parser already puts between two HTML documents — no per-document label, as multi-HTML submissions never had one. Uuencoded payloads (GRAPHIC, a PDF behind `<PDF>`, whose encoded bytes can pass the HTML test) and machine-readable files by `<FILENAME>` extension (`.xml`/`.xsd`/`.css`/`.js`/`.json` — a modern full submission's XBRL and renderer files read as plain) stay outside, dropped as before, and so does the SEC header, as it always was for these submissions. Input with no `<body>` — a plain-only submission — converts whole, header included. Over 93 corpus files only the 10 submissions mixing HTML and plain documents change: each old text is a prefix of the new one and all 186 old outline entries keep their offsets. The primary is document 1, so its headings come first and exhibit headings only fill what the 50-entry cap leaves; cross-document keep-later dedup moved none. **The XBRL renderer's files are cut (#160):** in the same `<body>` case, a document whose `<TYPE>` is `XML` is removed from the input. That is the type EDGAR gives every file its renderer generates (`R1.htm`…, `Show.js`, `report.css`, `FilingSummary.xml`, described `IDEA: XBRL DOCUMENT`). The viewer pages carry `<body>` elements, and so does 2011's `Show.js`, in a string literal, so the R-file tables followed the filed documents. The rule keys on the type, not the `R\d+.htm` name: across 79 sampled 2014–2026 indexes every one of 1,081 R files is typed `XML`, no filed document is, and inline-XBRL primaries carry their form type. Only the 4 corpus submissions holding renderer files change, each new text a prefix of the old. Indented headings stay undetected (centered S-4 headings; Items indented in 2 of 29 sampled 10-Ks): admitting indentation fills the outline cap with centered cover legends and promotes indented TOC rows. **A primary the archive does not serve (#158):** many 2000–2001 indexes list sequence-numbered documents (`0001.htm`, `0001.txt`, `0002.txt`) that answer 404 while `<accession>.txt` is served. When no candidate CIK serves the index's primary, `resolveFilingArchive` reads the full submission instead: `primary_document` and `filing_url` name the `.txt`, and the notice (composed into the truncation guidance, since `notice` is last-wins) names the unserved file. The fallback runs after the CIK loop, so a CIK that serves the primary still wins, and a served primary costs no extra request. A caller-named `document` is never swapped — a listed-but-unserved one fails `document_not_found` with a hint naming `<accession>.txt`. The extract cache stores the document and CIK each text came from, so a cache hit reports the same primary, notice, `cik`, and `filing_url` as the miss that filled it, with no candidate-CIK search
106
- - **Heading detection:** `detectHeadings` runs three line-anchored patterns — the all-caps forms, the mixed-case same-line `Item N. Title` / `Part II` forms (#71), and a bare `Item N` marker whose title lands on a later line. Every whitespace match is horizontal-only: `\s` matches `\n`, which fused runs of consecutive all-caps lines into composite entries (`"TABLE OF CONTENTS\n\nPART I"`) (#105). Horizontal-only is not space-or-tab — filings put NBSP inside all-caps headings too (`MICROSOFT<NBSP>CORPORATION`), so the all-caps arm nests the class-set difference `[\s--\n]` in its class (the `v` flag). It is one class, not the class alternated with `[^\S\n]`: the quantified alternation slows sharply on very long lines, while a single class scans in linear time. The all-caps line admits `-`, `–`, `—`, digits, and apostrophes, and `isAllCapsHeadingText` bounds it and narrows the last two. Its heading text runs at most `MAX_TITLE_CHARS` (140): past it the line is a paragraph set in capitals, which the widened class otherwise admitted (Alternative Ballistics' 308-character risk-factor coda took `section: "business"` by substring from `OUR BUSINESS`; EuroSPAC's 860-character tax legend; GT Biopharma's 288- and 326-character voting notices). A digit is admitted only inside a whitespace-delimited `\d{1,2}[A-Z]?\.?` token judged on the whole line, a TOC page cell included (`NOTE 10`, `SCHEDULE 14A`; `USD 2,929`, `FMR LLC(4)`, `WASHINGTON, D.C. 20549`, and `…ACT OF 1934` stay out), and an apostrophe only inside a word or after a word-final `S` (`MANAGEMENT’S`, `STOCKHOLDERS’ EQUITY`), never opening or closing the heading. The all-caps `ITEM …` arm takes the rest of its line whatever it holds, and neither Item arm admits a line that runs a table of contents together: a second Item marker, or a page cell mid-line followed by two or more spaces and the next cell (`isRunTogetherToc`). Without it a 238-character 8-K table of contents (`ITEM 8.01. OTHER EVENTS. 3 ITEM 9.01. …`, Amazon's 0001104659-26-107526) was one heading and took `item 8`, `item 9`, and `exhibits`. The test is on shape, not length: a 140-character bound on that arm dropped a proxy's one proposal heading, Modine's 185-character `ITEM 1 – TO APPROVE THE PROPOSAL TO AMEND THE Amended and Restated Articles…` (DEF 14A 0001104659-26-115309), and turned `item 1` and `proposal` into misses. Across 104 cached filings (244 all-caps and 568 mixed-case Item lines) the shape test flags only the Amazon line, and no all-caps Item line runs past 240 characters, so the arm carries no outer length bound. The mixed-case same-line arm keeps its title bound as its prose guard. A colon stays out: every colon line it admitted across 26 sampled documents was a statement caption (`CURRENT ASSETS:`). A Part line may carry a title — whitespace, an optional `-`/`–`/`—`/`:`/`.`, then an uppercase letter, at most 140 characters — and is a cross-reference rather than a heading (`isPartCrossReference`), on the mixed-case and the all-caps arm alike, when it sits under a line ending on a comma or semicolon like an unpunctuated Item, or when its title opens on another marker (`Part II — Item 5. Market for Registrant’s Common Equity`, a bank 10-K's cross-reference index row), which would otherwise also displace the bare body Part it names. After dedup, a bare Part entry gives way to a later titled heading for the same numeral (`dropBarePartsTitledLater`, which applies the same marker-title test): a 10-Q's TOC row `Part II` and its body `PART II — OTHER INFORMATION` have different keys, so without the merge `section: "part ii"` landed on the TOC row 50,045 characters early (Apple Q3 FY2026). Over the 26-document corpus these rules add 72 headings and remove 4 TOC Part rows: 66 section headings (all 18 of Microsoft FY2026's `NOTE N — TITLE` lines, statement titles, titled Parts) and 6 furniture (statement captions, `SCHEDULE 14A`, a `PAGE` stamp, a wrapped legend line), no numeric table row (#170). The Item code spans `16A`–`16K` and the current 8-K's dotted `N.NN` (`ITEM_CODE`, shared by all four Item patterns), and the marker's period is optional — 20-F filers write the lettered markers unpunctuated. A dotted code may also abut its title or carry a paragraph suffix (`Item 9.01Financial Statements…`, `Item 9.01(d). Financial…`, the shapes html-to-text leaves when the marker and title are separate spans), and then the title must start with a letter, so a bare `Item 5.02.` line never matches on its own. A code in `CURRENT_EIGHT_K_ITEMS` takes a 160-character title bound (`EIGHT_K_MAX_TITLE_CHARS`) on the same-line arm and in `usableTitle` — Item 5.02's 146-character title is the longest across the 53 sampled 8-Ks, and the 14-character margin admits a short trailing note while a paragraph opening on the marker line (`Item 5.02(b) On June 2, 2026, the Board…`) runs to hundreds of characters and stays out — and every other marker keeps the 140-character prose guard, a dotted code outside the table (`Item 7.09`) included. Across 53 current 8-Ks the outline holds all 112 item codes EDGAR lists for them (24 before); without the bound exemption it holds 89, without the abutting/suffix form 103 (#165). When an all-caps marker line (`ITEM 2.02`) both matches the all-caps arm alone and joins the title below it, and both entries survive dedup at one offset, the bare entry is dropped — one heading, listed once. Because the period is optional, an unpunctuated same-line Item is dropped as wrapped prose (`isWrappedProse`) when its next word starts lowercase or the line above ends on a comma or semicolon. A line ending on a lowercase word is no evidence: a running page header (`Table of Contents`) sits directly above an unpunctuated body heading (`Item 5 — Other information`) at the top of a page. A plain-text 10-K's cross-reference that wraps to start a line with `Item 8 on this Form 10-K…` otherwise takes the `item 8` landing ahead of the real heading (#136). **A bare marker is admitted on the shape of what follows, not just on a title being there:** a blank line between marker and title admits the pair; without one, the pair is admitted only when no bare page-number line sits under the title, since marker/title/page-number on three tight lines is the TOC-cell artifact #71's guard exists to reject. That test is one-way — it rejects a TOC row, it does not certify a heading — so a marker line anywhere else (a page footer, an unwrapped paragraph) is admitted too, taking the next line as its title; the entry carries the marker's own offset, so `section` still lands in the right place. A heading text repeated `RUNNING_HEADER_MIN_OCCURRENCES` (5) times at distinct offsets is page furniture — C3is' FY2025 20-F repeats `TABLE OF CONTENTS` 129 times and a 10-Q stamps `PART I` on all 47 of its pages, while a heading printed once in the TOC and once in the body occurs twice. **Furniture is then handled by shape, two ways:** a bare marker (`Item 1`, `PART I`) carries no navigable text, so no occurrence of it earns an entry and every one is dropped; anything else keeps its FIRST occurrence and drops the rest, inverting the keep-later rule — a filing that runs a section's own heading across every page of that section (`NOTES TO THE CONSOLIDATED FINANCIAL STATEMENTS` through an S-1's financial statements) prints it first at the section start, and keep-later would land it on the section's last page instead. So the outline carries no fused composite and at most one entry per running header, at its first occurrence. **A bare marker is counted on its own line, never on the joined heading:** a page footer's marker takes its title from whatever prose the next page opens with, so every composite it produces is a different string and only the marker line repeats — count composites instead and a 10-Q's per-page `Item 1` footer yields one bogus entry per page, crowding the real Items past `maxEntries`. That count is per rendered form (`Item 1` and `Item 1.` are separate), since a filing stamping the bare form on every page still writes the punctuated one for the real heading. Dedup (keep the later, i.e. body, occurrence) keys on `foldForHeadingMatch` plus a terminal-punctuation strip, never raw lowercase: the same filing prints one marker with different space characters and a trailing period in its TOC and its body (`Item 3.` with U+0020 vs U+2009; `…Policies` vs `…Policies.`), and a raw key leaves both in the outline. A TOC row's trailing page cell — a page number or a lettered page (`F-3`), optionally after a `Page` word (`Page 1`), set off by a tab or two or more spaces (`TOC_PAGE_TAIL_RE`) — is stripped from every heading for the same reason. Without the `Page` form, First Bancorp's 8-K TOC row `Item 8.01 Other Events Page 1` (0000765207-26-000110) kept its cell, never deduped against the body `Item 8.01 Other Events.`, and took `item 8` at 1998 instead of 2227. A number after a single space is the heading's own (`PROPOSAL NO. 1`, `ARTICLE 5`, `…Securities Exchange Act of 1934`): the single-space strip cut it off, and dedup then merged GT Biopharma's three `PROPOSAL NO. N` headings into one `PROPOSAL NO.` at the last (#172). Across 81 cached filings the change moved nothing else in any outline. Lettered sub-item markers (a bare `A.`/`D.` inside an Item) are deliberately not detected — single letters recur throughout filing prose.
107
- - **Section matching:** `get_filing`'s `section` compares through `foldForHeadingMatch` (`filing-to-text.ts`) on **both** operands — Unicode whitespace runs collapse to one space, U+2018/U+2019 → `'`, U+201C/U+201D → `"`, then lowercase. Comparison-time only: `FilingHeading.heading`, the `outline` output, `content[]`, and every character offset keep the document's own bytes. EDGAR headings carry NBSP runs and typographic quotes, so a caller re-sending an outline heading through a UI that normalizes either one never byte-matched the heading it came from — which broke the `section_not_found` recovery loop the error's own hint points at (#106). No fuzzy matching — `"item 8"` resolves only to Item 8 — but every tier runs on folded operands, so a single-spaced needle also reaches a heading whose marker carries an NBSP run or a double space (`Item 9.` in a 2004 10-K). Folding only one operand does not fix the round trip. One boundary (`sectionMatches`): a needle ending in a number never continues into a longer one. A trailing digit never runs on into another digit, so `item 1` matches `Item 1.` and `Item 1A` but never `Item 12`, and `item 5` reaches an 8-K's `Item 5.02` while `item 5.0` does not. A trailing roman numeral (a final word of `i`, `v`, `x`) must end at a space, punctuation, or the heading's end, so `part i` never matches `PART II`, `PART III`, `PART IV`, or `PART II, III`: Sono-Tek's 10-Q (0001171520-26-000305), which prints Part I only inside a one-line TOC, landed `part i` on `PART II - OTHER INFORMATION`, and Microsoft's FY2026 10-K on `PART II, III` (348204). `part i` now falls through to the line tier (Microsoft: the body `PART I` at 6900) or misses with the outline (Sono-Tek; an S-1, whose Part I is the unlabeled prospectus). Plain-text outlines keep TOC rows worded differently from their body heading, and a TOC row for Item 10–16 precedes the body Item 1 — a bare substring test landed `item 1` on another Item in 6 of 37 plain 10-K/10-Q outlines and 1 of 270 HTML landings (#136). **Resolution order (#169, #166)**, first hit wins, over every detected heading (`detectHeadings(text)`, which takes no cap, run once per call — the 50-entry cap belongs to the rendered outline alone; with the first 50 only, a heavy 10-K's financial-statement captions put Items 9B–16 out of reach): (1) **exact** (`resolveSectionHeading`) — a heading whose `headingKey` (the fold plus a terminal-punctuation strip, the dedup key) equals the needle's, or whose title after its Item/Part marker does; a bare marker (`ITEM 1.`, `Part II`) has no title and is never a candidate, or GSBD's `item 1` jumps to a bare TOC `ITEM 1.`. The tier exists for #170's widened all-caps class: CCC's `SPECIAL NOTE REGARDING FORWARD-LOOKING STATEMENTS AND RISK FACTORS` (7562) otherwise takes `risk factors` from `Item 1A. Risk Factors.` (67351). (2) **substring** — the first heading by offset that `sectionMatches`. (3) **a document line** (`findSectionLine`, `filing-to-text.ts`), for a mixed-case subheading the outline never lists (`Liquidity and Capital Resources`): a line whose `headingKey` equals the needle's, else — only when no line is equal — one that starts with it at a word boundary within `MAX_TITLE_CHARS` (a date-suffixed title: `Consolidated Schedule of Investments as of June 30, 2026`), unless a one-word needle runs on into a longer Title Case title (`continuesIntoLongerTitle`: the next word opens on a capital in a line that is not all caps). One word heads many unrelated titles — `business` landed on `Business Combinations.` (C3is), `Business Seasonality and Product Introductions` (Apple 10-Q), and a cross-reference line `Business Overview—Governmental regulations…` (DRDGOLD); a needle of two or more words keeps its continuation (`Human Capital` → `Human Capital Management`), and a suffix opening lowercase or on punctuation (`(continued)`, `—Parent Company Only`) always does. The cost: `forward-looking` no longer reaches `Forward-Looking Statements`, nor `revenue` `Revenue Recognition`; both miss with the outline. Equality goes first because a prefix match alone lands `Competition` on an earlier line that merely begins with the word. In both line tiers a line is skipped when it opens lowercase (a wrapped sentence or bullet); when it sits under a line ending on a comma or semicolon, or opens on an unpunctuated Item marker followed by a lowercase word (a wrapped cross-reference, the `followsMidSentence` / `isWrappedProse` guards heading detection applies: `…in Part II,\nItem 8 on this Form 10-K…`); or when it is a TOC row (`isTocRow`). A TOC row ends in a page cell set off by **two or more** spaces or a tab (`TOC_PAGE_TAIL_RE`; one space would read the year in `as of June 30, 2026` as a page number), or its next line with content is a page cell (`PAGE_CELL_LINE_RE`: a number, `F-3`, or a lowercase roman `ii`, which counts only on its own line — `Part iv` is not a page cell); either form may put a `Page` word before the number (`Page 3`), spacer lines passed over (an HTML TOC table puts NBSP rows between its cells). A line that is not a marker, followed directly by an Item/Part marker, is a TOC title cell with no page column. A marker line — a Part over its first Item — is a TOC row when that Item row is: the Item line ends in a page cell, the line below it holds or ends in one (a title cell, or Aware's wrapped `of Equity Securities....... 12`), or, for a bare marker, the line below its title holds one. Microsoft's FY2026 10-K prints body `PART IV` only as page furniture, so `part iv` reaches the line tier, where `PART IV / Item 15. / Exhibit and Financial Statement Schedules / 93` was taken; it now lands on the body's first `PART IV` (352884). A body Part sits over its Item too, but prose follows, not a page cell. The first remaining line wins, at its first non-blank character; one linear pass, run only after both heading tiers miss. A line landing says so in the notice, composed into the truncation guidance when the page is cut (`notice` is last-wins): `No detected heading matches section '<value>', so this page starts at the first document line that does.` — no new output field. Every #166 acceptance offset holds. A heading hit always beats a line hit (Reddit's `use of proceeds` → the `USE OF PROCEEDS` heading). (4) `section_not_found`. **The outline cut (#169):** past 50 detected headings the outline is every Item/Part heading (`STRUCTURAL_HEADING_RE`, bare markers included) plus the earliest other headings up to 50, in offset order (`selectOutline`) — shared by the first-page `outline` and the `section_not_found` message and data. When the Item/Part headings alone pass 50, each bare marker that a later titled heading for the same Item or Part repeats is left out first (`outlineStructuralHeadings`): DRDGOLD's FY2026 20-F renders all 31 TOC Items as bare markers (`ITEM 3.`), which filled the cut and pushed body Items 16A–19 out; its outline now runs `ITEM 1.` through `ITEM 19. EXHIBITS` from the body. A bare marker no titled heading repeats stays, and `section` still matches every heading. The miss message heads a cut outline `Outline (50 of N detected headings):` (else `Outline:`), and a cut first page appends `The outline lists 50 of N detected headings, every Item and Part heading among them; section matches all of them.` to the truncation guidance — `…among them, bare table-of-contents copies of titled ones aside; …` when bare markers were left out (DRDGOLD: `The outline lists 50 of 142 detected headings, every Item and Part heading among them, bare table-of-contents copies of titled ones aside; section matches all of them.`), and no Item/Part clause when the kept Item/Part headings alone pass 50, which reaches `structuredContent.notice` and `content[]` — no new output field, since the outline is emitted only on a truncated first page. A document of 50 or fewer headings renders as before.
105
+ - **Filing content:** HTML → text via `html-to-text` library; pre-2005 filings produce noisier output. The library walks the DOM recursively, so `CONVERT_OPTIONS` caps it at `limits.maxDepth: 512` with a `[…]` ellipsis where the cut falls — without it deep markup overflowed the stack and one such candidate failed `search_filings`' whole pre-2001 text scan (#118). The bound must hold on both runtimes the server ships under: a fresh Node process overflows near 1,360 levels of nested lists (its most stack-hungry shape), Bun near 13,000; real HTML filings measure under 25 levels. Legacy SGML `<PAGE>` markers are replaced with a line break before parsing (`neutralizePageMarkers`) — the parser never closes them, so a text document otherwise nests one level per page and a long one would be cut at the cap; with them gone the deepest measured `.txt` sits under 100 (an EX-27 schedule's unclosed tags). Both changes leave every sampled filing's text byte-identical. `limits` deep-merges with the library defaults, so its 16,777,216-character `maxInputLength` (silent truncation, no ellipsis) still applies. **Plain-text SGML bodies keep their lines (#136):** the parser collapses every newline run to a space, so a plain `<TEXT>` body — a pre-2001 full submission, a 2000s ASCII primary document — lost the line structure `detectHeadings` anchors on (Apple's 1996 10-K came back as 104 lines with an empty outline). `preservePlainTextBodies` runs inside the one `convert()` call: a body is HTML when `<html`, `<body`, `<p`, `<div`, `<br`, `<font`, `<tr`, `<td`, or `<xml` opens anywhere in it (`\b`-bounded, so `<PAGE>`, `<PDF>`, and EX-27's `<PP&E>` don't trip `<p`; no `<table`, since EDGAR ASCII tables are `<TABLE>`); any other body has every tag dropped, `&`/`<`/`>` escaped, and goes in a `<pre>`. The list is broad because misreading HTML as plain wrecks it, while the reverse is only the old output. Tags are dropped rather than parsed: the parser formats `<TABLE>`/`<CAPTION>` as blocks inside the `<pre>`, and a stray `<word` in prose would swallow text up to the next `>`. The `<TEXT>` wrapper matches case-sensitively — EDGAR writes it uppercase, and an inline SVG `<text>` is not a body — and the scan is one forward `indexOf` pass, since a lazy `<TEXT>([\s\S]*?)</TEXT>` regex goes quadratic on openers with no closer. Input with no plain body reaches the parser unchanged, so current HTML and 2000–2004 SGML-wrapped HTML convert byte-identically; their outlines change only where an unpunctuated Item cross-reference opens a line and drops out as wrapped prose (see Heading detection). Kept indentation grows plain extractions about 19%. **Every document of a submission holding an HTML document (#159):** the parser converts only `<body>` elements when the input has one, so an HTML-era submission (HTML primary, plain EX-10/EX-27 exhibits) lost every plain exhibit, and the pre-2001 scan could not match exhibit text. When the input holds `<body`, each plain body's `<pre>` goes inside a `<body>` of its own, so every document converts in document order, separated by the block break the parser already puts between two HTML documents — no per-document label, as multi-HTML submissions never had one. Uuencoded payloads (GRAPHIC, a PDF behind `<PDF>`, whose encoded bytes can pass the HTML test) and machine-readable files by `<FILENAME>` extension (`.xml`/`.xsd`/`.css`/`.js`/`.json` — a modern full submission's XBRL and renderer files read as plain) stay outside, dropped as before, and so does the SEC header, as it always was for these submissions. Input with no `<body>` — a plain-only submission — converts whole, header included. Over 93 corpus files only the 10 submissions mixing HTML and plain documents change: each old text is a prefix of the new one and all 186 old outline entries keep their offsets. The primary is document 1, so its headings come first and exhibit headings only fill what the 50-entry cap leaves; cross-document keep-later dedup moved none. **The XBRL renderer's files are cut (#160):** in the same `<body>` case, a document whose `<TYPE>` is `XML` is removed from the input. That is the type EDGAR gives every file its renderer generates (`R1.htm`…, `Show.js`, `report.css`, `FilingSummary.xml`, described `IDEA: XBRL DOCUMENT`). The viewer pages carry `<body>` elements, and so does 2011's `Show.js`, in a string literal, so the R-file tables followed the filed documents. The rule keys on the type, not the `R\d+.htm` name: across 79 sampled 2014–2026 indexes every one of 1,081 R files is typed `XML`, no filed document is, and inline-XBRL primaries carry their form type. Only the 4 corpus submissions holding renderer files change, each new text a prefix of the old. Indented headings stay undetected (centered S-4 headings; Items indented in 2 of 29 sampled 10-Ks): admitting indentation fills the outline cap with centered cover legends and promotes indented TOC rows. The one exception is a same-line titled Item opening on a single NBSP (#176, see Heading detection). **A primary the archive does not serve (#158):** many 2000–2001 indexes list sequence-numbered documents (`0001.htm`, `0001.txt`, `0002.txt`) that answer 404 while `<accession>.txt` is served. When no candidate CIK serves the index's primary, `resolveFilingArchive` reads the full submission instead: `primary_document` and `filing_url` name the `.txt`, and the notice (composed into the truncation guidance, since `notice` is last-wins) names the unserved file. The fallback runs after the CIK loop, so a CIK that serves the primary still wins, and a served primary costs no extra request. A caller-named `document` is never swapped — a listed-but-unserved one fails `document_not_found` with a hint naming `<accession>.txt`. The extract cache stores the document and CIK each text came from, so a cache hit reports the same primary, notice, `cik`, and `filing_url` as the miss that filled it, with no candidate-CIK search
106
+ - **Heading detection:** `detectHeadings` runs three line-anchored patterns — the all-caps forms, the mixed-case same-line `Item N. Title` / `Part II` forms (#71), and a bare `Item N` marker whose title lands on a later line. **One NBSP may open a same-line Item line (#176):** both Item arms admit a single U+00A0 ahead of the marker, since an `&nbsp;` before a styled span left Horizon Space's 8-K line `<NBSP>Item 1.01. Entry into a Material Definitive Agreement.` (0001929980-26-000542) out of an outline that listed Items 7.01 and 9.01. A lookahead holds the NBSP to an Item marker; the entry's offset is the marker's (1960, not the NBSP's 1959), `isWrappedProse` and `followsMidSentence` still read the line from its real start, and a line that is a bare marker once its page cell is stripped (`<NBSP>ITEM 2.02`, `<NBSP>Item 1. 3`) yields nothing (`itemEntryOffset`). Nothing wider: ASCII-space or tab indentation, two NBSPs, and NBSP-led Part lines stay out — Microsoft's 2004 10-K (0001193125-04-150689) stamps an NBSP-led `Part I` on every page, and admitting those turns its body `PART I` (3832), `PART II` (64076), and `PART IV` (308565) into running-header furniture. Across the 67-filing corpus the change adds that one entry and removes none. Every whitespace match is horizontal-only: `\s` matches `\n`, which fused runs of consecutive all-caps lines into composite entries (`"TABLE OF CONTENTS\n\nPART I"`) (#105). Horizontal-only is not space-or-tab — filings put NBSP inside all-caps headings too (`MICROSOFT<NBSP>CORPORATION`), so the all-caps arm nests the class-set difference `[\s--\n]` in its class (the `v` flag). It is one class, not the class alternated with `[^\S\n]`: the quantified alternation slows sharply on very long lines, while a single class scans in linear time. The all-caps line admits `-`, `–`, `—`, digits, and apostrophes, and `isAllCapsHeadingText` bounds it and narrows the last two. Its heading text runs at most `MAX_TITLE_CHARS` (140): past it the line is a paragraph set in capitals, which the widened class otherwise admitted (Alternative Ballistics' 308-character risk-factor coda took `section: "business"` by substring from `OUR BUSINESS`; EuroSPAC's 860-character tax legend; GT Biopharma's 288- and 326-character voting notices). A digit is admitted only inside a whitespace-delimited `\d{1,2}[A-Z]?\.?` token judged on the whole line, a TOC page cell included (`NOTE 10`, `SCHEDULE 14A`; `USD 2,929`, `FMR LLC(4)`, `WASHINGTON, D.C. 20549`, and `…ACT OF 1934` stay out), and an apostrophe only inside a word or after a word-final `S` (`MANAGEMENT’S`, `STOCKHOLDERS’ EQUITY`), never opening or closing the heading. The all-caps `ITEM …` arm takes the rest of its line whatever it holds, and neither Item arm admits a line that runs a table of contents together: a second Item marker, or a page cell mid-line followed by two or more spaces and the next cell (`isRunTogetherToc`). Without it a 238-character 8-K table of contents (`ITEM 8.01. OTHER EVENTS. 3 ITEM 9.01. …`, Amazon's 0001104659-26-107526) was one heading and took `item 8`, `item 9`, and `exhibits`. The test is on shape, not length: a 140-character bound on that arm dropped a proxy's one proposal heading, Modine's 185-character `ITEM 1 – TO APPROVE THE PROPOSAL TO AMEND THE Amended and Restated Articles…` (DEF 14A 0001104659-26-115309), and turned `item 1` and `proposal` into misses. Across 104 cached filings (244 all-caps and 568 mixed-case Item lines) the shape test flags only the Amazon line, and no all-caps Item line runs past 240 characters, so the arm carries no outer length bound. The mixed-case same-line arm keeps its title bound as its prose guard. A colon stays out: every colon line it admitted across 26 sampled documents was a statement caption (`CURRENT ASSETS:`). A Part line may carry a title — whitespace, an optional `-`/`–`/`—`/`:`/`.`, then an uppercase letter, at most 140 characters — and is a cross-reference rather than a heading (`isPartCrossReference`), on the mixed-case and the all-caps arm alike, when it sits under a line ending on a comma or semicolon like an unpunctuated Item, or when its title opens on another marker (`Part II — Item 5. Market for Registrant’s Common Equity`, a bank 10-K's cross-reference index row), which would otherwise also displace the bare body Part it names. After dedup, a bare Part entry gives way to a later titled heading for the same numeral (`dropBarePartsTitledLater`, which applies the same marker-title test): a 10-Q's TOC row `Part II` and its body `PART II — OTHER INFORMATION` have different keys, so without the merge `section: "part ii"` landed on the TOC row 50,045 characters early (Apple Q3 FY2026). Over the 26-document corpus these rules add 72 headings and remove 4 TOC Part rows: 66 section headings (all 18 of Microsoft FY2026's `NOTE N — TITLE` lines, statement titles, titled Parts) and 6 furniture (statement captions, `SCHEDULE 14A`, a `PAGE` stamp, a wrapped legend line), no numeric table row (#170). The Item code spans `16A`–`16K` and the current 8-K's dotted `N.NN` (`ITEM_CODE`, shared by all four Item patterns), and the marker's period is optional — 20-F filers write the lettered markers unpunctuated. A dotted code may also abut its title or carry a paragraph suffix (`Item 9.01Financial Statements…`, `Item 9.01(d). Financial…`, the shapes html-to-text leaves when the marker and title are separate spans), and then the title must start with a letter, so a bare `Item 5.02.` line never matches on its own. A code in `CURRENT_EIGHT_K_ITEMS` takes a 160-character title bound (`EIGHT_K_MAX_TITLE_CHARS`) on the same-line arm and in `usableTitle` — Item 5.02's 146-character title is the longest across the 53 sampled 8-Ks, and the 14-character margin admits a short trailing note while a paragraph opening on the marker line (`Item 5.02(b) On June 2, 2026, the Board…`) runs to hundreds of characters and stays out — and every other marker keeps the 140-character prose guard, a dotted code outside the table (`Item 7.09`) included. Across 53 current 8-Ks the outline holds all 112 item codes EDGAR lists for them (24 before); without the bound exemption it holds 89, without the abutting/suffix form 103 (#165). When an all-caps marker line (`ITEM 2.02`) both matches the all-caps arm alone and joins the title below it, and both entries survive dedup at one offset, the bare entry is dropped — one heading, listed once. Because the period is optional, an unpunctuated same-line Item is dropped as wrapped prose (`isWrappedProse`) when its next word starts lowercase or the line above ends on a comma or semicolon. A line ending on a lowercase word is no evidence: a running page header (`Table of Contents`) sits directly above an unpunctuated body heading (`Item 5 — Other information`) at the top of a page. A plain-text 10-K's cross-reference that wraps to start a line with `Item 8 on this Form 10-K…` otherwise takes the `item 8` landing ahead of the real heading (#136). **A bare marker is admitted on the shape of what follows, not just on a title being there:** a blank line between marker and title admits the pair; without one, the pair is admitted only when no bare page-number line sits under the title, since marker/title/page-number on three tight lines is the TOC-cell artifact #71's guard exists to reject. That test is one-way — it rejects a TOC row, it does not certify a heading — so a marker line anywhere else (a page footer, an unwrapped paragraph) is admitted too, taking the next line as its title; the entry carries the marker's own offset, so `section` still lands in the right place. A heading text repeated `RUNNING_HEADER_MIN_OCCURRENCES` (5) times at distinct offsets is page furniture — C3is' FY2025 20-F repeats `TABLE OF CONTENTS` 129 times and a 10-Q stamps `PART I` on all 47 of its pages, while a heading printed once in the TOC and once in the body occurs twice. **Furniture is then handled by shape, two ways:** a bare marker (`Item 1`, `PART I`) carries no navigable text, so no occurrence of it earns an entry and every one is dropped; anything else keeps its FIRST occurrence and drops the rest, inverting the keep-later rule — a filing that runs a section's own heading across every page of that section (`NOTES TO THE CONSOLIDATED FINANCIAL STATEMENTS` through an S-1's financial statements) prints it first at the section start, and keep-later would land it on the section's last page instead. So the outline carries no fused composite and at most one entry per running header, at its first occurrence. **A bare marker is counted on its own line, never on the joined heading:** a page footer's marker takes its title from whatever prose the next page opens with, so every composite it produces is a different string and only the marker line repeats — count composites instead and a 10-Q's per-page `Item 1` footer yields one bogus entry per page, crowding the real Items past `maxEntries`. That count is per rendered form (`Item 1` and `Item 1.` are separate), since a filing stamping the bare form on every page still writes the punctuated one for the real heading. Dedup (keep the later, i.e. body, occurrence) keys on `headingKey` — `foldForHeadingMatch`, a terminal-punctuation strip, and the separator between an Item/Part marker and its title folded to one space — never raw lowercase: the same filing prints one marker with different space characters and a trailing period in its TOC and its body (`Item 3.` with U+0020 vs U+2009; `…Policies` vs `…Policies.`), and a raw key leaves both in the outline. The separator is the third difference: Worthington Steel's 10-Q (0001968487-26-000036) prints `Item 1A. Risk Factors` in its TOC and `Item 1A. – Risk Factors` in its body, so both kept an entry and `risk factors` landed on the TOC row at 4292 instead of 250397. A marker key is rebuilt as the marker, one space, and the title (`item 1a risk factors`), whatever period, dash, or colon stood between them (`MARKER_TITLE_RE`, exported for the section matcher); across the 67-filing corpus that drops the 10-Q's 11 TOC copies (34 → 23 entries) and moves no other outline entry (#174). The furniture count, and the copy furniture keeps, key on the printed form instead — `headingKey` short of the separator fold (`printedKey`) — so a TOC row and a body heading never pool with a running header printed with another separator and lose the body heading to the TOC row. A TOC row's trailing page cell — a page number or a lettered page (`F-3`), optionally after a `Page` word (`Page 1`), set off by a tab or two or more spaces (`TOC_PAGE_TAIL_RE`) — is stripped from every heading for the same reason. Without the `Page` form, First Bancorp's 8-K TOC row `Item 8.01 Other Events Page 1` (0000765207-26-000110) kept its cell, never deduped against the body `Item 8.01 Other Events.`, and took `item 8` at 1998 instead of 2227. A number after a single space is the heading's own (`PROPOSAL NO. 1`, `ARTICLE 5`, `…Securities Exchange Act of 1934`): the single-space strip cut it off, and dedup then merged GT Biopharma's three `PROPOSAL NO. N` headings into one `PROPOSAL NO.` at the last (#172). Across 81 cached filings the change moved nothing else in any outline. Lettered sub-item markers (a bare `A.`/`D.` inside an Item) are deliberately not detected — single letters recur throughout filing prose — and neither is a digit-led sub-item line (`3D. RISK FACTORS`; the all-caps arm needs a leading letter): `section` reaches a labelled line by stripping the label at comparison time instead (#175, see Section matching), so no detected heading is added.
107
+ - **Section matching:** `get_filing`'s `section` compares through `foldForHeadingMatch` (`filing-to-text.ts`) on **both** operands — Unicode whitespace runs collapse to one space, U+2018/U+2019 → `'`, U+201C/U+201D → `"`, then lowercase. Comparison-time only: `FilingHeading.heading`, the `outline` output, `content[]`, and every character offset keep the document's own bytes. EDGAR headings carry NBSP runs and typographic quotes, so a caller re-sending an outline heading through a UI that normalizes either one never byte-matched the heading it came from — which broke the `section_not_found` recovery loop the error's own hint points at (#106). No fuzzy matching — `"item 8"` resolves only to Item 8 — but every tier runs on folded operands, so a single-spaced needle also reaches a heading whose marker carries an NBSP run or a double space (`Item 9.` in a 2004 10-K). Folding only one operand does not fix the round trip. One boundary (`sectionMatches`): a needle ending in a number never continues into a longer one. A trailing digit never runs on into another digit, so `item 1` matches `Item 1.` and `Item 1A` but never `Item 12`, and `item 5` reaches an 8-K's `Item 5.02` while `item 5.0` does not. A trailing roman numeral (a final word of `i`, `v`, `x`) must end at a space, punctuation, or the heading's end, so `part i` never matches `PART II`, `PART III`, `PART IV`, or `PART II, III`: Sono-Tek's 10-Q (0001171520-26-000305), which prints Part I only inside a one-line TOC, landed `part i` on `PART II - OTHER INFORMATION`, and Microsoft's FY2026 10-K on `PART II, III` (348204). `part i` now falls through to the line tier (Microsoft: the body `PART I` at 6900) or misses with the outline (Sono-Tek; an S-1, whose Part I is the unlabeled prospectus). Plain-text outlines keep TOC rows worded differently from their body heading, and a TOC row for Item 10–16 precedes the body Item 1 — a bare substring test landed `item 1` on another Item in 6 of 37 plain 10-K/10-Q outlines and 1 of 270 HTML landings (#136). **Resolution order (#169, #166, #177)**, first hit wins, over every detected heading (`detectHeadings(text)`, which takes no cap, run once per call — the 50-entry cap belongs to the rendered outline alone; with the first 50 only, a heavy 10-K's financial-statement captions put Items 9B–16 out of reach) and the document's own lines: (1) **exact** (`resolveSection`) — a heading whose `headingKey` (the dedup key: the fold, a terminal-punctuation strip, and the marker separator folded to one space) equals the needle's, or whose title after its Item/Part marker does — failing that, the same with a leading sub-item label removed (below); a bare marker (`ITEM 1.`, `Part II`) has no title and is never a candidate, or GSBD's `item 1` jumps to a bare TOC `ITEM 1.`. The tier exists for #170's widened all-caps class: CCC's `SPECIAL NOTE REGARDING FORWARD-LOOKING STATEMENTS AND RISK FACTORS` (7562) otherwise takes `risk factors` from `Item 1A. Risk Factors.` (67351). (2) **Item/Part substring** — the first Item or Part heading (`STRUCTURAL_HEADING_RE`, bare markers included) that `sectionMatches`: a value inside an Item title names that Item, while a line equal to it is often a table-row label (CCC's `compensation` → `Item 11. Executive Compensation` at 493087, not the `Compensation` cell at 418658). (3) **an equal line** and (5) **a prefix line** — a document line (`matchSectionLine`, `filing-to-text.ts`, which reports which of the two it found; `findSectionLine` is its offset alone), for a mixed-case subheading the outline never lists (`Liquidity and Capital Resources`): a line whose `headingKey` equals the needle's (3), else — only when no line is equal — one that starts with it (5) at a word boundary within `MAX_TITLE_CHARS` (a date-suffixed title: `Consolidated Schedule of Investments as of June 30, 2026`), unless a one-word needle runs on into a longer Title Case title (`continuesIntoLongerTitle`: the next word opens on a capital in a line that is not all caps). One word heads many unrelated titles — `business` landed on `Business Combinations.` (C3is), `Business Seasonality and Product Introductions` (Apple 10-Q), and a cross-reference line `Business Overview—Governmental regulations…` (DRDGOLD); a needle of two or more words keeps its continuation (`Human Capital` → `Human Capital Management`), and a suffix opening lowercase or on punctuation (`(continued)`, `—Parent Company Only`) always does. The cost: `forward-looking` no longer reaches `Forward-Looking Statements`, nor `revenue` `Revenue Recognition`; both miss with the outline. Equality goes first because a prefix match alone lands `Competition` on an earlier line that merely begins with the word. In both line tiers a line is skipped when it opens lowercase (a wrapped sentence or bullet); when it sits under a line ending on a comma or semicolon, or opens on an unpunctuated Item marker followed by a lowercase word (a wrapped cross-reference, the `followsMidSentence` / `isWrappedProse` guards heading detection applies: `…in Part II,\nItem 8 on this Form 10-K…`); or when it is a TOC row (`isTocRow`). A TOC row ends in a page cell set off by **two or more** spaces or a tab (`TOC_PAGE_TAIL_RE`; one space would read the year in `as of June 30, 2026` as a page number), or its next line with content is a page cell (`PAGE_CELL_LINE_RE`: a number, `F-3`, or a lowercase roman `ii`, which counts only on its own line — `Part iv` is not a page cell); either form may put a `Page` word before the number (`Page 3`), spacer lines passed over (an HTML TOC table puts NBSP rows between its cells). A line that is not a marker, followed directly by an Item/Part marker, is a TOC title cell with no page column. A marker line — a Part over its first Item — is a TOC row when that Item row is: the Item line ends in a page cell, the line below it holds or ends in one (a title cell, or Aware's wrapped `of Equity Securities....... 12`), or, for a bare marker, the line below its title holds one. Microsoft's FY2026 10-K prints body `PART IV` only as page furniture, so `part iv` reaches the line tier, where `PART IV / Item 15. / Exhibit and Financial Statement Schedules / 93` was taken; it now lands on the body's first `PART IV` (352884). A body Part sits over its Item too, but prose follows, not a page cell. The first remaining line wins, at its first non-blank character; one linear pass serves tiers 3 and 5, run only after tiers 1 and 2 miss, so a value landing on a tier-4 heading now pays that pass (7–11 ms on a 1 MB S-1). A cut page's guidance after a `section` landing says to pass `next_offset` as `offset` without `section`, which outranks `offset` — kept, it returns the same page. A line landing says so in the notice, composed into the truncation guidance when the page is cut (`notice` is last-wins): `No detected heading matches section '<value>', so this page starts at the first document line that does.` — no new output field. An equal line that outranked a heading merely containing the value says `No detected heading equals section '<value>', so this page starts at the first document line that does, ahead of headings that only contain it.` instead, since that heading does match by substring. Every #166 acceptance offset holds. (4) **other substring** — the first other heading that `sectionMatches`, after the equal line: a heading that merely holds the word took Arxis' S-1 `business` to `NOTE 3. BUSINESS COMBINATIONS` (544705) past its `Business` heading line (323578), and CCC's `revenue` to `COST OF REVENUES` (332106) past the MD&A `Revenue` subheading (256790). An exact or Item/Part heading still beats an equal line (Reddit's `use of proceeds` → the `USE OF PROCEEDS` heading); every other heading ranks after one. Over 67 corpus filings × 103 section values the order moved 24 landings, each one the validation predicted: 17 onto the section named (Microsoft FY2026 `general` → `GENERAL` at 8720, not `GENERAL RISKS`; DRDGOLD's `risk`, `directors`, `management`, and `employees` → body Items instead of TOC rows), and 7 onto a different place that is defensible but that a reader may like less, since an Item heading or a line equal to the value now outranks a heading that only holds it — JPM's 10-Q `risk` lands on Item 3, a one-sentence cross-reference, not `FIRMWIDE RISK MANAGEMENT`; `revenue` in Microsoft's 2004 and FY2026 10-Ks and Aware's on a selected-data, MD&A-table, or income-statement row label rather than a heading containing the word; `summary` on Item 16 `Form 10-K Summary`; Intel's 2000 `consolidated balance sheets` on an Item 14 index row. (6) `section_not_found`. **A proxy statement keeps its headings ahead of the lines:** for a proxy or information statement form (`PROXY_FORM_RE`: DEF 14A, DEFA14A, DEFM14A, PRE 14A, DEF 14C, …), tier 1 is followed by the first heading of any kind that `sectionMatches`, then the line tiers — its ITEM headings are ballot proposals, not the Regulation S-K item taxonomy, and its lines equal to a value are mostly table cells. Under tiers 2–4 Goldman Sachs' DEF 14A (0001193125-26-117433) took `compensation` from the CD&A (123680) to `ITEM 2. … (SAY ON PAY)` (274744) and `summary` from `EXECUTIVE SUMMARY` (16953) to a Pay versus Performance column header (282286). The form is read before `section` resolves, so the `.hdr.sgml` fallback request (#126) now precedes a `section_not_found`. **Inside each heading tier a body heading ranks ahead of a table-of-contents row (#174):** a heading is a TOC row when its title line — its own line, or the next line with content after a bare or joined marker — is one by `isTocRow` (the line tiers' test), or the next line with content below it ends in a page cell (a TOC title wrapped onto a second line, Aware's `of Equity Securities....... 12`); `isTocHeading` in `filing-to-text.ts` owns the test, finding the line start from the newline before the offset since an NBSP-led Item's offset sits one past it. A TOC row still lands when it is its tier's only match. Three TOC shapes survive dedup: a row worded apart from its body heading (First Bancorp's 8-K `Item 9.01 Financial Statements and Exhibits. Page 1` took `item 9` at 2041 from the body Item at 2402), a bare all-caps marker over its title and page cells (DRDGOLD's FY2026 20-F `ITEM 3.` / `KEY INFORMATION` / `6` took `item 3` at 6174 from `ITEM 3. KEY INFORMATION` at 34346, and nearly every other `item N` the same way), and the title cell itself (`key information`). TOC rows are ranked, never dropped or pushed behind the line tiers: the test also reads a body heading over a page number (CCC's `Item 6. Reserved.` at 234199, Reddit's S-1 `PART II` at 1021631), and behind the line tiers CCC's and Microsoft's `item 6` left the body heading for a bare TOC marker line. With the dedup-key fold, the ranking moved 83 landings over the corpus, each from a TOC row onto a body heading (DRDGOLD 35, Worthington Steel 20, GSBD 9, Amazon's 2003 10-K 8, Aware 6, First Bancorp 4, CCC 1), and no other: 78 onto the body heading the row copies, 5 onto another the value names (Amazon 2003's `part ii`–`part iv` onto its body Parts, Aware's `management` from its TOC Item 12 to body Item 7, Worthington Steel's `other information` from its TOC Item 5 to body `PART II. OTHER INFORMATION`). **A leading 20-F sub-item label is ignored (#175):** 20-F filers label sub-items (`3D. RISK FACTORS`, `B. Business Overview`), so a plain-phrase value never equalled the line. Tier 1 and both line tiers also compare the heading or line key with a leading label removed (`withoutSubItemLabel`: `/^(?:\d{1,2}[a-z]|[a-z])\. /` on the folded key, anchored, so constant-time), and the one-word longer-title rule judges the line after its label — Regencell's `B. Business Overview` keeps `business` a miss while `business overview` lands (151573). The value itself is never stripped, so `3D. RISK FACTORS` still matches its own line; a line opening on an initial (`L. John Doerr`) strips the same way and so matches only a value equal to the rest of the line. The landing is the line's first character, label included (DRDGOLD `risk factors` → 34503). Tier 1 strips only once no heading equals the value as printed, so an earlier `A. RISK FACTORS` never takes `risk factors` from a later `Item 1A. Risk Factors`; a labelled body heading still ranks ahead of a TOC row. The line tiers keep the first line equal either way: across the corpus the labelled line they land on is the 20-F sub-item itself and a later unlabelled one is narrower (DRDGOLD `business overview` → `4B. BUSINESS OVERVIEW` at 176975, not `Business overview` at 269017). Over the corpus the strip moved 23 landings, all in the three 20-Fs: 19 misses now land on the labelled sub-item, and 4 earlier landings move onto one (DRDGOLD `compensation` from the `Compensation of key management` subheading inside Item 6B, 328266, to `6B. COMPENSATION` at 324690); no filing without labelled sub-item lines changed. **The outline cut (#169):** past 50 detected headings the outline is every Item/Part heading (`STRUCTURAL_HEADING_RE`, bare markers included) plus the earliest other headings up to 50, in offset order (`selectOutline`) — shared by the first-page `outline` and the `section_not_found` message and data. When the Item/Part headings alone pass 50, each bare marker that a later titled heading for the same Item or Part repeats is left out first (`outlineStructuralHeadings`): DRDGOLD's FY2026 20-F renders all 31 TOC Items as bare markers (`ITEM 3.`), which filled the cut and pushed body Items 16A–19 out; its outline now runs `ITEM 1.` through `ITEM 19. EXHIBITS` from the body. A bare marker no titled heading repeats stays, and `section` still matches every heading. The miss message heads a cut outline `Outline (50 of N detected headings):` (else `Outline:`), and a cut first page appends `The outline lists 50 of N detected headings, every Item and Part heading among them; section matches all of them.` to the truncation guidance — `…among them, bare table-of-contents copies of titled ones aside; …` when bare markers were left out (DRDGOLD: `The outline lists 50 of 142 detected headings, every Item and Part heading among them, bare table-of-contents copies of titled ones aside; section matches all of them.`), and no Item/Part clause when the kept Item/Part headings alone pass 50, which reaches `structuredContent.notice` and `content[]` — no new output field, since the outline is emitted only on a truncated first page. A document of 50 or fewer headings renders as before.
108
108
  - **Binary filing documents:** a filing index routinely carries scanned pages, PDF exhibits, `Financial_Report.xlsx`, and `-xbrl.zip` — `filingToExtract` runs `html-to-text` over whatever bytes arrive and never inspects a content type, so requesting one used to return decoded binary as `content` with no error (#96). `get-filing.tool.ts` detects them with one rule, `binaryType`: the filename extension (`binaryTypeFromName`), the one signal every filing carries, plus a header-page `GRAPHIC` type for a scan named oddly — the header page is read before the body (#161). Two halves, both needed — `documents[]` entries carry `binary: true`, and `resolveFilingArchive` throws `binary_document` before `tryGetFilingDocument` runs, since the `document` input is an unconstrained string and a catalog marker alone cannot stop a caller. Extension-driven and bucket-independent on purpose: a PDF exhibit is typed `EX-99.*` and files under `exhibits`, not among the graphics.
109
109
  - **Dataframes:** `secedgar_search_filings`, `secedgar_get_financials`, `secedgar_fetch_frames`, `secedgar_compare_companies`, `secedgar_get_insider_transactions`, `secedgar_get_institutional_holdings`, `secedgar_get_material_events`, `secedgar_find_holders`, `secedgar_get_beneficial_owners`, and `secedgar_get_fund_holdings` materialize their full result set as `df_<id>` on a shared DuckDB-backed canvas (one per tenant). The two ownership tools register the full parsed set (inline response is a preview capped at `limit`); `get_insider_transactions` scans at least `INSIDER_CANVAS_FILING_SCAN` recent filings when canvas is available (so small inline limits don't under-scan the window), capped at 100, and flags `dataset.truncated` when more Form 4 filings exist beyond the scanned window (#63); with a `filed_after` / `filed_before` window it reads the in-window Form 4 / 4-A rows with an XML primary document — `recent`, then the archive pages overlapping the window, newest first, stopping once the scan cap plus a sentinel row are in hand — and with a canvas parses every one up to 100 instead of the 40-filing floor, since the window already bounds the set; `dataset.truncated` is then the sentinel or unread overlapping pages, and `history_scanned_through` the oldest filing parsed (#127); a company's submissions feed also lists the Form 4s it filed as a reporting owner of another issuer (JPMorgan's 10% stake in a BlackRock fund), so a parsed document whose `issuerCik` differs from the resolved CIK, compared with zero-padding stripped, adds no row inline or to the dataframe and is counted in `filings_other_issuer` — it still counts toward `filings_scanned`, the scan floor, and `history_scanned_through`, and the no-canvas stop rule is unchanged — though a skipped filing adds no transactions, so a no-canvas call reads on through such filings until `limit` transactions are in hand, within the same scan cap (#157); its `shares_traded` is an unsigned magnitude paired with a `direction` (acquire/dispose) column, so net activity is `SUM(CASE WHEN direction='dispose' THEN -shares_traded ELSE shares_traded END)` (#46); `get_financials`' dataframe materializes the source-filing fiscal keys as `source_filing_fy`/`source_filing_fp` (renamed from bare `fiscal_year`/`fiscal_period` — every comparative period restated in one filing carries that filing's fy/fp, so `period_end` is the time key for ORDER BY/GROUP BY/windowing) (#72); `compare_companies`' dataframe holds the FULL aligned series across every period while the inline matrix is capped at `periods`, and each row carries both the aligned `period` key and the underlying XBRL `frame` (they differ for point-in-time concepts) (#85). In an annual matrix an instant enters only on one of the filer's fiscal-year ends and takes the key of the annual duration ending that day. The ends come from the companyfacts payload already in hand (`filerFiscalYearEndsWithComparatives` in `concept-series.ts`, shared with `get_financials`' annual view of an instant): each annual report's own year (the #148 rule) plus the comparative years it restates — its year-long facts ending a whole number of years earlier, within a week — so the first XBRL 10-K's prior-year balance keys too, while a year-long figure ending off the report's calendar (a December filer's trailing year to 09-30) stays out. The #148 test of a 10-Q-held annual frame keeps `filerFiscalYearEnds` (each report's own year only): it matches day of year, where a comparative year adds nothing but a wider tolerance. The key is the first `CY####` frame on any concept in the payload ending then, so a balance lands beside its fiscal year's income lines even when none is requested; with no framed duration ending on it, the calendar year in which that fiscal year's second half starts (SEC's own framing: a year ending 06-30 is that year, one ending 06-27 or 05-31 the year before — Seagate's and Western Digital's years ending 2025-06-27 are `CY2024`; SEC departs from it for the year both ended 2012-06-29, framed `CY2012`, which is why a framed key wins). Interim and cover-date instants stay out of the annual matrix, and a company-concept left with none gets a gap plus a caveat pointing at quarterly; when that empties the whole matrix, `no_comparable_data` says the values were all off fiscal-year ends and its hint names `period_type: "quarterly"` (omitting `period_type` keeps annual). Quarterly keys are unchanged. The year ends are read lazily, only when an annual matrix meets an instant (#171). The period-ends caveat checks every inline column — the newest can hold one company alone — spelling out the newest column whose ends differ and naming the others; the inline window is the newest periods of the union across companies, so a company-concept pair whose values all predate it is named in `caveats` — one line per concept, each company with its newest period and the dataframe pointer — and never in `gaps`, which keeps meaning no value in any period (#144). Each row set carries an all-nullable schema (framework's `inferSchemaFromRows` always emits `nullable: true` since mcp-ts-core 0.10.4). Per-table TTL is delegated to the framework via `RegisterTableOptions.ttlMs` (resolved in mcp-ts-core#140); bridge still tracks metadata expiry in `ctx.state` for provenance. Every producer emits the describe-then-query pointer on the branch that registers the dataframe, composed once by `dataframeGuidance(dataset)` in the bridge so the wording cannot drift; where a truncation call already fires there it rides as that call's `guidance`, since `ctx.enrich.notice` is last-wins across `notice`/`truncated` and a second call would clobber the first (#104). `secedgar_dataframe_query` detects a `row_limit`-bound result from `QueryResult.truncated` rather than `rowCount > rows.length` — `row_limit` is pushed into the query as the provider's cap, so a capped result comes back with `rowCount === rows.length` and the arithmetic cannot see it; `rowCount` is then the cap and not a total, so that branch names the cap instead of printing "of N rows", `row_count_capped` carries the same disclosure into `format()`, and the fallback comparison stays for the `registerAs` path, where `truncated` is never set. A SQL `LIMIT` exactly equal to the cap is indistinguishable from a result that holds that many rows and is reported as exact (#109). `secedgar_dataframe_query` runs framework's SQL gate with `denySystemCatalogs: true` to block `information_schema`, `pg_catalog`, `sqlite_master`, and `duckdb_*` catalog access. Raw DuckDB execution errors (missing table, syntax) are caught in `bridge.query` and re-thrown as structured `missing_table`/`invalid_sql` reasons, with DuckDB's "Did you mean …?" hint stripped so internal catalog names don't leak (#47). The framework's own classifications pass through untouched and are declared on the tool's contract (`thrownBy: 'service'`, since the handler never names them): a SELECT that prepares and then fails on the data — a `Conversion` / `Invalid Input` / `Out of Range` error, e.g. `CAST('abc' AS INTEGER)` — is `ValidationError` with `sql_execution_error` (mcp-ts-core 0.13.4; `DatabaseError` with no reason before), and the gate's `multi_statement`, `denied_function`, `sql_read_only`, and `plan_operator_not_allowed` rejections arrive with their own recovery hints — since mcp-ts-core 0.13.15 the gate also denies engine-setting and SQL-from-string functions (`current_setting`, `query_table`) as `denied_function` and a file named as a table (`FROM 'data.csv'`) as `sql_read_only`. `get_beneficial_owners` registers one row per reporting person rather than one per filing, since powers and percent of class are per person; `get_fund_holdings` registers every position in the report while the inline list is one `limit`-sized page from `offset`, and its rows carry the fund keys (`series_id`, `registrant_cik`, `report_period_date`, `accession_number`) so a portfolio joins the 13F and insider rows on `cusip`.
110
110
  - **Beneficial ownership (13D/13G):** SEC replaced the legacy `SC 13D` / `SC 13G` text filings with structured XML under the `SCHEDULE 13D` / `SCHEDULE 13G` names on **2024-12-18**; the changeover is total, so a legacy-name filter matches nothing recent and a current-name filter matches nothing older. The two schedules are **separate schemas, not one shape with optional fields** — 13D repeats `reportingPersons/reportingPersonInfo` with the power fields flat on each person and the percentage in `percentOfClass`; 13G repeats `coverPageHeaderReportingPersonDetails` with the powers nested inside `reportingPersonBeneficiallyOwnedNumberOfShares`, the percentage in `classPercent`, no reporting-person CIK anywhere, and no purpose-of-transaction field at all (its item 4 is structured ownership data, which is what makes it the passive form). `beneficial-ownership-parser.ts` therefore branches per schema and dispatches on the submission type, falling back to whichever repeated reporting-person element the document carries. **Ownership powers and percent of class are per reporting person, including on joint filings** — a fund, its adviser, and its controlling principal each report the same underlying shares, so the parsed owner list is never collapsed to a group total and the percentages must not be summed. Discovery runs off the issuer's own `getSubmissions()`: SEC cross-lists a schedule under every CIK party to it, so no entity-scoped EFTS step is needed.
@@ -117,7 +117,7 @@ Tailor suggestions to what's actually missing or stale — don't recite the full
117
117
 
118
118
  | Env Var | Required | Default | Description |
119
119
  |:--------|:---------|:--------|:------------|
120
- | `EDGAR_USER_AGENT` | **Yes** | — | User-Agent for SEC compliance. Format: `"AppName contact@email.com"` |
120
+ | `EDGAR_USER_AGENT` | **Yes** | — | User-Agent for SEC compliance. Format: `"AppName contact@email.com"`. A value this runtime's `fetch` cannot send as a header (a character past Latin-1 such as an em dash, or an inner line break) fails startup as a `ConfigurationError` naming the variable, rather than every request failing later (#173). |
121
121
  | `EDGAR_RATE_LIMIT_RPS` | No | `10` | Max requests/second to SEC APIs. Do not exceed 10. |
122
122
  | `EDGAR_RATE_LIMIT_COOLDOWN_SECONDS` | No | `600` | Seconds to stop sending to SEC after a 429. Calls are refused locally for this long, then the first one goes out alone as a probe. SEC lifts its block only after ten quiet minutes, so a shorter value just probes into it. |
123
123
  | `EDGAR_TICKER_CACHE_TTL` | No | `3600` | Seconds to cache the ticker index (company_tickers.json + company_tickers_mf.json). A failed fund-file load is retried after 60 s (or the rate-limit cool-down) instead of standing for the TTL. |
@@ -398,6 +398,7 @@ After a version bump and final commit, publish to both npm and GHCR:
398
398
  bun publish --access public
399
399
 
400
400
  docker buildx build --platform linux/amd64,linux/arm64 \
401
+ --build-arg APP_VERSION=<version> \
401
402
  -t ghcr.io/cyanheads/secedgar-mcp-server:<version> \
402
403
  -t ghcr.io/cyanheads/secedgar-mcp-server:latest \
403
404
  --push .
package/CLAUDE.md CHANGED
@@ -1,7 +1,7 @@
1
1
  # Agent Protocol
2
2
 
3
3
  **Server:** secedgar-mcp-server
4
- **Version:** 0.15.12
4
+ **Version:** 0.15.13
5
5
  **Framework:** [@cyanheads/mcp-ts-core](https://www.npmjs.com/package/@cyanheads/mcp-ts-core) `^0.13.15`
6
6
  **Engines:** Bun ≥1.4.0, Node ≥24.0.0
7
7
  **MCP SDK:** `@modelcontextprotocol/server` ^2.3.1
@@ -82,7 +82,7 @@ Tailor suggestions to what's actually missing or stale — don't recite the full
82
82
 
83
83
  ## Domain Notes
84
84
 
85
- - **Rate limit:** 10 req/s per IP. Every SEC request goes through one process-wide `createPacer` queue in `EdgarApiService` that spaces request *starts* `1000 / EDGAR_RATE_LIMIT_RPS` ms apart, with no concurrency cap and no wait budget — an ordinary burst drains at the configured rate and is never shed, and this server's own traffic can never exceed it, so `rateLimitRps` stays at 10 and headroom for a co-tenant on the outbound IP is the operator's call via `EDGAR_RATE_LIMIT_RPS`. A 429 is **not** retried: SEC holds the block until the request rate has stayed under the threshold for ten minutes and every request sent meanwhile restarts that clock, so the ~3s retry budget (`MAX_RETRIES` 3 × `BASE_BACKOFF_MS` 1000, exponential) cannot outlast it (#112). Instead a 429 closes the service's **block gate** for `EDGAR_RATE_LIMIT_COOLDOWN_SECONDS` (default 600). While it is closed nothing is sent: a call fails locally as `RateLimited` with `retryAfter` set to the whole seconds remaining (never below 1), and the gate is re-read at dispatch, so a call already queued in the pacer when the 429 landed — or a 5xx retry sleeping through its backoff — is refused too. The first call after the cool-down is sent alone as a probe while every other call waits on it: any HTTP answer but a 429 reopens normal pacing, a probe 429 closes the gate for the full cool-down again, and a probe that gets no HTTP answer (a network error) proves nothing, so the next call probes in its place. **The gate lives in the service, not in the pacer's `cooldown` option** — that gate holds queued calls until it reopens and then releases them all at the pacing rate, so it can neither refuse locally nor keep the reopening to one request. The upstream 429 and the local refusal carry one `data.reason: 'rate_limited'`, declared (`thrownBy: 'service'`, `retryable: true`) on every tool that reaches SEC, plus `data.retryable: true` — `ctx.fail` copies a contract entry's `retryable` onto `data`, a service throw does not, so both throw sites set it themselves (#122) — and one recovery hint naming SEC's ten-minute window and the seconds to wait; SEC's HTML block page stays out of the payload (`captureBody: false`). Mirror-served reads (the ticker base behind `resolveCik`, and `tryGetCompanyFacts` / `tryGetCompanyConcept` / `tryGetFrames` through `mirrorOrLive`) never reach `rawFetch`, so they keep answering during a block. 500/502/503/504 still retry with the backoff. `createApp({ teardown })` disposes the pacer through `disposeEdgarApiService()` (#116)
85
+ - **Rate limit:** 10 req/s per IP. Every SEC request goes through one process-wide `createPacer` queue in `EdgarApiService` that spaces request *starts* `1000 / EDGAR_RATE_LIMIT_RPS` ms apart, with no concurrency cap and no wait budget — an ordinary burst drains at the configured rate and is never shed, and this server's own traffic can never exceed it, so `rateLimitRps` stays at 10 and headroom for a co-tenant on the outbound IP is the operator's call via `EDGAR_RATE_LIMIT_RPS`. A 429 is **not** retried: SEC holds the block until the request rate has stayed under the threshold for ten minutes and every request sent meanwhile restarts that clock, so the ~3s retry budget (`MAX_RETRIES` 3 × `BASE_BACKOFF_MS` 1000, exponential) cannot outlast it (#112). Instead a 429 closes the service's **block gate** for `EDGAR_RATE_LIMIT_COOLDOWN_SECONDS` (default 600). While it is closed nothing is sent: a call fails locally as `RateLimited` with `retryAfter` set to the whole seconds remaining (never below 1), and the gate is re-read at dispatch, so a call already queued in the pacer when the 429 landed — or a 5xx or no-response retry sleeping through its backoff — is refused too. The first call after the cool-down is sent alone as a probe while every other call waits on it: any HTTP answer but a 429 reopens normal pacing, a probe 429 closes the gate for the full cool-down again, and a probe that gets no HTTP answer (a network error, or no headers within the per-attempt cap) proves nothing, so the next call probes in its place — and every call that waited on it counts that probe as one of its own no-response attempts, so calls queued behind a silent SEC each settle within their own three-attempt ladder instead of probing in turn, 15 s apiece (#173). **The gate lives in the service, not in the pacer's `cooldown` option** — that gate holds queued calls until it reopens and then releases them all at the pacing rate, so it can neither refuse locally nor keep the reopening to one request. The upstream 429 and the local refusal carry one `data.reason: 'rate_limited'`, declared (`thrownBy: 'service'`, `retryable: true`) on every tool that reaches SEC, plus `data.retryable: true` — `ctx.fail` copies a contract entry's `retryable` onto `data`, a service throw does not, so both throw sites set it themselves (#122) — and one recovery hint naming SEC's ten-minute window and the seconds to wait; SEC's HTML block page stays out of the payload (`captureBody: false`). Mirror-served reads (the ticker base behind `resolveCik`, and `tryGetCompanyFacts` / `tryGetCompanyConcept` / `tryGetFrames` through `mirrorOrLive`) never reach `rawFetch`, so they keep answering during a block. 500/502/503/504 still retry with the backoff, and so does an attempt that gets no HTTP response at all — a `fetch` rejection (Bun sets the transport code on the error itself, Node's undici on its `cause`) or no response headers within `ATTEMPT_HEADERS_TIMEOUT_MS` (15 s from dispatch; pacer queue time and the body read are outside it), which holds one request to 48 s against the MCP client's 60 s default. When the last attempt gets no HTTP response — whatever the earlier ones got — the call fails as `ServiceUnavailable` with `data.reason: 'sec_unreachable'`, `retryable: true`, `url`, and a hint, declared beside `rate_limited` on the same tools; the message names host, attempts, and the transport code or the timeout, and the runtime's own text rides `cause` to the log only (#173). `createApp({ teardown })` disposes the pacer through `disposeEdgarApiService()` (#116)
86
86
  - **Parameter names:** one name per concept across tools — `company` (the entity a tool reads; on `get_institutional_holdings` that is the 13F filer), `filed_after` / `filed_before` (inclusive filing-date bounds), `forms`. Every tool carrying one of those keys declares `inputAliases` for the other spellings in use — `ticker`, `cik`, `ticker_or_cik` → `company`; `start_date`, `date_from` → `filed_after`; `end_date`, `date_to` → `filed_before`; `form_types` → `forms` — so a caller carrying a name from one tool to the next is not rejected (#115). An alias is declared only where it maps to exactly one key of that tool and its value shape fits: a singular `form_type` cannot fill the `forms` array, so it is left out rather than widening the schema. A tool whose only company input carries another name takes the same company-identifier spellings onto it — `company`, `ticker`, `cik`, `ticker_or_cik` → `issuer` on `find_holders` and `get_beneficial_owners`, and those plus `name` and `search` → `query` on `company_search` — since the key it moves into has no other reading there; the canonical names stay, because renaming them would break existing callers (#152). `fund` (a fund or series) and `companies` (an array) are not aliased. `search_filings` still filters by date only with both bounds; `company_search`, `get_material_events`, and `get_insider_transactions` accept either alone
87
87
  - **User-Agent required:** `"AppName contact@email.com"` on every SEC request or IP gets blocked
88
88
  - **CIK zero-padding:** the `data.sec.gov` APIs (submissions, companyfacts, companyconcept) require the 10-digit zero-padded CIK (`String(cik).padStart(10, '0')`). Archive paths under `www.sec.gov/Archives/edgar/data/` take it unpadded — SEC answers the padded path with a 301, and `fetch` follows it inside the same pacer slot, so every archive read would cost two requests. `filingArchiveUrl` (exported from `edgar-api-service.ts`) strips the zeros for all six archive reads and for `get_filing`'s `filing_url` (#156)
@@ -102,9 +102,9 @@ Tailor suggestions to what's actually missing or stale — don't recite the full
102
102
  - **Submissions archive walk:** the submissions feed's `filings.recent` holds the last year or 1,000 filings, whichever is more — so 26,000+ rows for JPMorgan, and a filing day is never split (Microsoft's window runs to 1,002 rows to finish its oldest day); everything older lives on `filings.files[]` archive pages of 200–360 KB each. Every tool that reads past `recent` goes through `SubmissionsArchiveWalk` (`submissions-archive.ts`): one `ARCHIVE_PAGE_SCAN_CAP` (10) for all, pages read one at a time through the service's pacer and page cache, and no page selected when the lower bound falls on or after `recent`'s oldest date — so a call answered by `recent` sends no archive request. The caller picks the order — `newest-first` (`company_search`, `search_filings`' pre-2001 arm, `get_material_events`, `get_insider_transactions`' window), or `oldest-first` forward from a date (`get_institutional_holdings`' quarter lookup, since newest-first cannot reach JPMorgan's 2020-Q4 13F-HR on page 41 of 70 under the cap) — and can stop after any page; `scannedThrough` (the oldest page start read) and `truncated` (selected pages left unread, whether the cap, an early stop, or a caller that never started the walk left them) are reported the same way everywhere. Worst case per call is the submissions document plus 10 pages plus the tool's own document reads (`get_insider_transactions`: 111). `get_material_events` without a date window walks only to fill `limit`: it enters the walk when fewer than `limit` `recent` rows pass the `items` filter — the same count as the stop test, so unrelated recent 8-Ks never stand in for matches (#139) — and stops on the page that fills it with rows passing that filter, and its `dataset.truncated` is true whenever selected pages went unread — including when `recent` alone fills `limit` and no page is read, since the dataframe then holds only the recent window (#134); a result short of `limit` with selected pages unread (only the cap ends a walk that short) gets a notice naming the pages read, the oldest date reached, and the lever — `filed_before=<that date>` alone or in a window when undated, narrowing the window with `filed_before=<that date>` when dated, since a newest-first walk reaches older filings only through an earlier upper bound — appended to the zero-match or empty-window notice when one fires, since `ctx.enrich.notice` is last-wins; an empty-window notice the cap cut short claims no 8-K only "as far back as the scan reached" and drops its widen-the-range advice (#139); `get_institutional_holdings` stops on the first page holding a 13F-HR or 13F-NT for the quarter, and runs the #86 operating-company routing only when neither `recent` nor the pages read hold any 13F-HR or 13F-NT — a notice-only manager asked for a quarter it filed nothing for gets the plain quarter miss (#117, #133); without a `quarter`, a manager whose `recent` holds 13F-NT notices and no 13F-HR gets the notice message for its newest notice rather than the #86 classification (#133). **Page selection widens both date bounds by a day:** a manifest page's `filingFrom` / `filingTo` can sit a day off its own rows in either direction (JPMorgan's page 041 is listed to 2021-02-17 but holds 42 filings dated 2021-02-18; Apple's page 001 is listed a day past its newest row), so an exact overlap test skipped the page holding a window's edge day. The slack costs at most one extra page per bound, and every caller filters rows by their own filing dates (#137)
103
103
  - **Filing metadata outside `recent`:** `get_filing` takes `form`, `filing_date`, and `period_ending` from `recent` when the accession is there, else from the filing's own SEC header — the raw `<TYPE>` / `<FILING-DATE>` / `<PERIOD>` in `index-headers.html`, which the tool already fetches to type the documents, and only when that page 404s (common before 2014) the sub-1 KB `<accession>.hdr.sgml`. `FILING-DATE` is the filing date, not the acceptance timestamp, and matches the submissions feed; an S-8 or Form 4 header has no `PERIOD`. `company_name` stays the current registrant name (#126)
104
104
  - **Primary document (#161):** `get_filing`'s `primary_document`, the document it reads when `document` is omitted, and the primary its error catalogs name is the name SEC records (`recordedPrimaryNames`): the submissions feed's `primaryDocument` when the accession sits in `recent` (last path segment — an XML form is recorded through its stylesheet path, `xslF345X06/form4.xml`), else the document the header page types with the submission's form. Only with neither does the largest readable document stand in, and the viewer-page guard there matches `R\d+.html?`, not every name starting with `R` (`Report10K.htm`). Size alone picked an 8-K's press-release exhibit over the form and a 13F-HR's information table over its cover page. The header page and the feed are read together right after an index is found and before the body, since every outcome — the error catalogs included — names the primary from them
105
- - **Filing content:** HTML → text via `html-to-text` library; pre-2005 filings produce noisier output. The library walks the DOM recursively, so `CONVERT_OPTIONS` caps it at `limits.maxDepth: 512` with a `[…]` ellipsis where the cut falls — without it deep markup overflowed the stack and one such candidate failed `search_filings`' whole pre-2001 text scan (#118). The bound must hold on both runtimes the server ships under: a fresh Node process overflows near 1,360 levels of nested lists (its most stack-hungry shape), Bun near 13,000; real HTML filings measure under 25 levels. Legacy SGML `<PAGE>` markers are replaced with a line break before parsing (`neutralizePageMarkers`) — the parser never closes them, so a text document otherwise nests one level per page and a long one would be cut at the cap; with them gone the deepest measured `.txt` sits under 100 (an EX-27 schedule's unclosed tags). Both changes leave every sampled filing's text byte-identical. `limits` deep-merges with the library defaults, so its 16,777,216-character `maxInputLength` (silent truncation, no ellipsis) still applies. **Plain-text SGML bodies keep their lines (#136):** the parser collapses every newline run to a space, so a plain `<TEXT>` body — a pre-2001 full submission, a 2000s ASCII primary document — lost the line structure `detectHeadings` anchors on (Apple's 1996 10-K came back as 104 lines with an empty outline). `preservePlainTextBodies` runs inside the one `convert()` call: a body is HTML when `<html`, `<body`, `<p`, `<div`, `<br`, `<font`, `<tr`, `<td`, or `<xml` opens anywhere in it (`\b`-bounded, so `<PAGE>`, `<PDF>`, and EX-27's `<PP&E>` don't trip `<p`; no `<table`, since EDGAR ASCII tables are `<TABLE>`); any other body has every tag dropped, `&`/`<`/`>` escaped, and goes in a `<pre>`. The list is broad because misreading HTML as plain wrecks it, while the reverse is only the old output. Tags are dropped rather than parsed: the parser formats `<TABLE>`/`<CAPTION>` as blocks inside the `<pre>`, and a stray `<word` in prose would swallow text up to the next `>`. The `<TEXT>` wrapper matches case-sensitively — EDGAR writes it uppercase, and an inline SVG `<text>` is not a body — and the scan is one forward `indexOf` pass, since a lazy `<TEXT>([\s\S]*?)</TEXT>` regex goes quadratic on openers with no closer. Input with no plain body reaches the parser unchanged, so current HTML and 2000–2004 SGML-wrapped HTML convert byte-identically; their outlines change only where an unpunctuated Item cross-reference opens a line and drops out as wrapped prose (see Heading detection). Kept indentation grows plain extractions about 19%. **Every document of a submission holding an HTML document (#159):** the parser converts only `<body>` elements when the input has one, so an HTML-era submission (HTML primary, plain EX-10/EX-27 exhibits) lost every plain exhibit, and the pre-2001 scan could not match exhibit text. When the input holds `<body`, each plain body's `<pre>` goes inside a `<body>` of its own, so every document converts in document order, separated by the block break the parser already puts between two HTML documents — no per-document label, as multi-HTML submissions never had one. Uuencoded payloads (GRAPHIC, a PDF behind `<PDF>`, whose encoded bytes can pass the HTML test) and machine-readable files by `<FILENAME>` extension (`.xml`/`.xsd`/`.css`/`.js`/`.json` — a modern full submission's XBRL and renderer files read as plain) stay outside, dropped as before, and so does the SEC header, as it always was for these submissions. Input with no `<body>` — a plain-only submission — converts whole, header included. Over 93 corpus files only the 10 submissions mixing HTML and plain documents change: each old text is a prefix of the new one and all 186 old outline entries keep their offsets. The primary is document 1, so its headings come first and exhibit headings only fill what the 50-entry cap leaves; cross-document keep-later dedup moved none. **The XBRL renderer's files are cut (#160):** in the same `<body>` case, a document whose `<TYPE>` is `XML` is removed from the input. That is the type EDGAR gives every file its renderer generates (`R1.htm`…, `Show.js`, `report.css`, `FilingSummary.xml`, described `IDEA: XBRL DOCUMENT`). The viewer pages carry `<body>` elements, and so does 2011's `Show.js`, in a string literal, so the R-file tables followed the filed documents. The rule keys on the type, not the `R\d+.htm` name: across 79 sampled 2014–2026 indexes every one of 1,081 R files is typed `XML`, no filed document is, and inline-XBRL primaries carry their form type. Only the 4 corpus submissions holding renderer files change, each new text a prefix of the old. Indented headings stay undetected (centered S-4 headings; Items indented in 2 of 29 sampled 10-Ks): admitting indentation fills the outline cap with centered cover legends and promotes indented TOC rows. **A primary the archive does not serve (#158):** many 2000–2001 indexes list sequence-numbered documents (`0001.htm`, `0001.txt`, `0002.txt`) that answer 404 while `<accession>.txt` is served. When no candidate CIK serves the index's primary, `resolveFilingArchive` reads the full submission instead: `primary_document` and `filing_url` name the `.txt`, and the notice (composed into the truncation guidance, since `notice` is last-wins) names the unserved file. The fallback runs after the CIK loop, so a CIK that serves the primary still wins, and a served primary costs no extra request. A caller-named `document` is never swapped — a listed-but-unserved one fails `document_not_found` with a hint naming `<accession>.txt`. The extract cache stores the document and CIK each text came from, so a cache hit reports the same primary, notice, `cik`, and `filing_url` as the miss that filled it, with no candidate-CIK search
106
- - **Heading detection:** `detectHeadings` runs three line-anchored patterns — the all-caps forms, the mixed-case same-line `Item N. Title` / `Part II` forms (#71), and a bare `Item N` marker whose title lands on a later line. Every whitespace match is horizontal-only: `\s` matches `\n`, which fused runs of consecutive all-caps lines into composite entries (`"TABLE OF CONTENTS\n\nPART I"`) (#105). Horizontal-only is not space-or-tab — filings put NBSP inside all-caps headings too (`MICROSOFT<NBSP>CORPORATION`), so the all-caps arm nests the class-set difference `[\s--\n]` in its class (the `v` flag). It is one class, not the class alternated with `[^\S\n]`: the quantified alternation slows sharply on very long lines, while a single class scans in linear time. The all-caps line admits `-`, `–`, `—`, digits, and apostrophes, and `isAllCapsHeadingText` bounds it and narrows the last two. Its heading text runs at most `MAX_TITLE_CHARS` (140): past it the line is a paragraph set in capitals, which the widened class otherwise admitted (Alternative Ballistics' 308-character risk-factor coda took `section: "business"` by substring from `OUR BUSINESS`; EuroSPAC's 860-character tax legend; GT Biopharma's 288- and 326-character voting notices). A digit is admitted only inside a whitespace-delimited `\d{1,2}[A-Z]?\.?` token judged on the whole line, a TOC page cell included (`NOTE 10`, `SCHEDULE 14A`; `USD 2,929`, `FMR LLC(4)`, `WASHINGTON, D.C. 20549`, and `…ACT OF 1934` stay out), and an apostrophe only inside a word or after a word-final `S` (`MANAGEMENT’S`, `STOCKHOLDERS’ EQUITY`), never opening or closing the heading. The all-caps `ITEM …` arm takes the rest of its line whatever it holds, and neither Item arm admits a line that runs a table of contents together: a second Item marker, or a page cell mid-line followed by two or more spaces and the next cell (`isRunTogetherToc`). Without it a 238-character 8-K table of contents (`ITEM 8.01. OTHER EVENTS. 3 ITEM 9.01. …`, Amazon's 0001104659-26-107526) was one heading and took `item 8`, `item 9`, and `exhibits`. The test is on shape, not length: a 140-character bound on that arm dropped a proxy's one proposal heading, Modine's 185-character `ITEM 1 – TO APPROVE THE PROPOSAL TO AMEND THE Amended and Restated Articles…` (DEF 14A 0001104659-26-115309), and turned `item 1` and `proposal` into misses. Across 104 cached filings (244 all-caps and 568 mixed-case Item lines) the shape test flags only the Amazon line, and no all-caps Item line runs past 240 characters, so the arm carries no outer length bound. The mixed-case same-line arm keeps its title bound as its prose guard. A colon stays out: every colon line it admitted across 26 sampled documents was a statement caption (`CURRENT ASSETS:`). A Part line may carry a title — whitespace, an optional `-`/`–`/`—`/`:`/`.`, then an uppercase letter, at most 140 characters — and is a cross-reference rather than a heading (`isPartCrossReference`), on the mixed-case and the all-caps arm alike, when it sits under a line ending on a comma or semicolon like an unpunctuated Item, or when its title opens on another marker (`Part II — Item 5. Market for Registrant’s Common Equity`, a bank 10-K's cross-reference index row), which would otherwise also displace the bare body Part it names. After dedup, a bare Part entry gives way to a later titled heading for the same numeral (`dropBarePartsTitledLater`, which applies the same marker-title test): a 10-Q's TOC row `Part II` and its body `PART II — OTHER INFORMATION` have different keys, so without the merge `section: "part ii"` landed on the TOC row 50,045 characters early (Apple Q3 FY2026). Over the 26-document corpus these rules add 72 headings and remove 4 TOC Part rows: 66 section headings (all 18 of Microsoft FY2026's `NOTE N — TITLE` lines, statement titles, titled Parts) and 6 furniture (statement captions, `SCHEDULE 14A`, a `PAGE` stamp, a wrapped legend line), no numeric table row (#170). The Item code spans `16A`–`16K` and the current 8-K's dotted `N.NN` (`ITEM_CODE`, shared by all four Item patterns), and the marker's period is optional — 20-F filers write the lettered markers unpunctuated. A dotted code may also abut its title or carry a paragraph suffix (`Item 9.01Financial Statements…`, `Item 9.01(d). Financial…`, the shapes html-to-text leaves when the marker and title are separate spans), and then the title must start with a letter, so a bare `Item 5.02.` line never matches on its own. A code in `CURRENT_EIGHT_K_ITEMS` takes a 160-character title bound (`EIGHT_K_MAX_TITLE_CHARS`) on the same-line arm and in `usableTitle` — Item 5.02's 146-character title is the longest across the 53 sampled 8-Ks, and the 14-character margin admits a short trailing note while a paragraph opening on the marker line (`Item 5.02(b) On June 2, 2026, the Board…`) runs to hundreds of characters and stays out — and every other marker keeps the 140-character prose guard, a dotted code outside the table (`Item 7.09`) included. Across 53 current 8-Ks the outline holds all 112 item codes EDGAR lists for them (24 before); without the bound exemption it holds 89, without the abutting/suffix form 103 (#165). When an all-caps marker line (`ITEM 2.02`) both matches the all-caps arm alone and joins the title below it, and both entries survive dedup at one offset, the bare entry is dropped — one heading, listed once. Because the period is optional, an unpunctuated same-line Item is dropped as wrapped prose (`isWrappedProse`) when its next word starts lowercase or the line above ends on a comma or semicolon. A line ending on a lowercase word is no evidence: a running page header (`Table of Contents`) sits directly above an unpunctuated body heading (`Item 5 — Other information`) at the top of a page. A plain-text 10-K's cross-reference that wraps to start a line with `Item 8 on this Form 10-K…` otherwise takes the `item 8` landing ahead of the real heading (#136). **A bare marker is admitted on the shape of what follows, not just on a title being there:** a blank line between marker and title admits the pair; without one, the pair is admitted only when no bare page-number line sits under the title, since marker/title/page-number on three tight lines is the TOC-cell artifact #71's guard exists to reject. That test is one-way — it rejects a TOC row, it does not certify a heading — so a marker line anywhere else (a page footer, an unwrapped paragraph) is admitted too, taking the next line as its title; the entry carries the marker's own offset, so `section` still lands in the right place. A heading text repeated `RUNNING_HEADER_MIN_OCCURRENCES` (5) times at distinct offsets is page furniture — C3is' FY2025 20-F repeats `TABLE OF CONTENTS` 129 times and a 10-Q stamps `PART I` on all 47 of its pages, while a heading printed once in the TOC and once in the body occurs twice. **Furniture is then handled by shape, two ways:** a bare marker (`Item 1`, `PART I`) carries no navigable text, so no occurrence of it earns an entry and every one is dropped; anything else keeps its FIRST occurrence and drops the rest, inverting the keep-later rule — a filing that runs a section's own heading across every page of that section (`NOTES TO THE CONSOLIDATED FINANCIAL STATEMENTS` through an S-1's financial statements) prints it first at the section start, and keep-later would land it on the section's last page instead. So the outline carries no fused composite and at most one entry per running header, at its first occurrence. **A bare marker is counted on its own line, never on the joined heading:** a page footer's marker takes its title from whatever prose the next page opens with, so every composite it produces is a different string and only the marker line repeats — count composites instead and a 10-Q's per-page `Item 1` footer yields one bogus entry per page, crowding the real Items past `maxEntries`. That count is per rendered form (`Item 1` and `Item 1.` are separate), since a filing stamping the bare form on every page still writes the punctuated one for the real heading. Dedup (keep the later, i.e. body, occurrence) keys on `foldForHeadingMatch` plus a terminal-punctuation strip, never raw lowercase: the same filing prints one marker with different space characters and a trailing period in its TOC and its body (`Item 3.` with U+0020 vs U+2009; `…Policies` vs `…Policies.`), and a raw key leaves both in the outline. A TOC row's trailing page cell — a page number or a lettered page (`F-3`), optionally after a `Page` word (`Page 1`), set off by a tab or two or more spaces (`TOC_PAGE_TAIL_RE`) — is stripped from every heading for the same reason. Without the `Page` form, First Bancorp's 8-K TOC row `Item 8.01 Other Events Page 1` (0000765207-26-000110) kept its cell, never deduped against the body `Item 8.01 Other Events.`, and took `item 8` at 1998 instead of 2227. A number after a single space is the heading's own (`PROPOSAL NO. 1`, `ARTICLE 5`, `…Securities Exchange Act of 1934`): the single-space strip cut it off, and dedup then merged GT Biopharma's three `PROPOSAL NO. N` headings into one `PROPOSAL NO.` at the last (#172). Across 81 cached filings the change moved nothing else in any outline. Lettered sub-item markers (a bare `A.`/`D.` inside an Item) are deliberately not detected — single letters recur throughout filing prose.
107
- - **Section matching:** `get_filing`'s `section` compares through `foldForHeadingMatch` (`filing-to-text.ts`) on **both** operands — Unicode whitespace runs collapse to one space, U+2018/U+2019 → `'`, U+201C/U+201D → `"`, then lowercase. Comparison-time only: `FilingHeading.heading`, the `outline` output, `content[]`, and every character offset keep the document's own bytes. EDGAR headings carry NBSP runs and typographic quotes, so a caller re-sending an outline heading through a UI that normalizes either one never byte-matched the heading it came from — which broke the `section_not_found` recovery loop the error's own hint points at (#106). No fuzzy matching — `"item 8"` resolves only to Item 8 — but every tier runs on folded operands, so a single-spaced needle also reaches a heading whose marker carries an NBSP run or a double space (`Item 9.` in a 2004 10-K). Folding only one operand does not fix the round trip. One boundary (`sectionMatches`): a needle ending in a number never continues into a longer one. A trailing digit never runs on into another digit, so `item 1` matches `Item 1.` and `Item 1A` but never `Item 12`, and `item 5` reaches an 8-K's `Item 5.02` while `item 5.0` does not. A trailing roman numeral (a final word of `i`, `v`, `x`) must end at a space, punctuation, or the heading's end, so `part i` never matches `PART II`, `PART III`, `PART IV`, or `PART II, III`: Sono-Tek's 10-Q (0001171520-26-000305), which prints Part I only inside a one-line TOC, landed `part i` on `PART II - OTHER INFORMATION`, and Microsoft's FY2026 10-K on `PART II, III` (348204). `part i` now falls through to the line tier (Microsoft: the body `PART I` at 6900) or misses with the outline (Sono-Tek; an S-1, whose Part I is the unlabeled prospectus). Plain-text outlines keep TOC rows worded differently from their body heading, and a TOC row for Item 10–16 precedes the body Item 1 — a bare substring test landed `item 1` on another Item in 6 of 37 plain 10-K/10-Q outlines and 1 of 270 HTML landings (#136). **Resolution order (#169, #166)**, first hit wins, over every detected heading (`detectHeadings(text)`, which takes no cap, run once per call — the 50-entry cap belongs to the rendered outline alone; with the first 50 only, a heavy 10-K's financial-statement captions put Items 9B–16 out of reach): (1) **exact** (`resolveSectionHeading`) — a heading whose `headingKey` (the fold plus a terminal-punctuation strip, the dedup key) equals the needle's, or whose title after its Item/Part marker does; a bare marker (`ITEM 1.`, `Part II`) has no title and is never a candidate, or GSBD's `item 1` jumps to a bare TOC `ITEM 1.`. The tier exists for #170's widened all-caps class: CCC's `SPECIAL NOTE REGARDING FORWARD-LOOKING STATEMENTS AND RISK FACTORS` (7562) otherwise takes `risk factors` from `Item 1A. Risk Factors.` (67351). (2) **substring** — the first heading by offset that `sectionMatches`. (3) **a document line** (`findSectionLine`, `filing-to-text.ts`), for a mixed-case subheading the outline never lists (`Liquidity and Capital Resources`): a line whose `headingKey` equals the needle's, else — only when no line is equal — one that starts with it at a word boundary within `MAX_TITLE_CHARS` (a date-suffixed title: `Consolidated Schedule of Investments as of June 30, 2026`), unless a one-word needle runs on into a longer Title Case title (`continuesIntoLongerTitle`: the next word opens on a capital in a line that is not all caps). One word heads many unrelated titles — `business` landed on `Business Combinations.` (C3is), `Business Seasonality and Product Introductions` (Apple 10-Q), and a cross-reference line `Business Overview—Governmental regulations…` (DRDGOLD); a needle of two or more words keeps its continuation (`Human Capital` → `Human Capital Management`), and a suffix opening lowercase or on punctuation (`(continued)`, `—Parent Company Only`) always does. The cost: `forward-looking` no longer reaches `Forward-Looking Statements`, nor `revenue` `Revenue Recognition`; both miss with the outline. Equality goes first because a prefix match alone lands `Competition` on an earlier line that merely begins with the word. In both line tiers a line is skipped when it opens lowercase (a wrapped sentence or bullet); when it sits under a line ending on a comma or semicolon, or opens on an unpunctuated Item marker followed by a lowercase word (a wrapped cross-reference, the `followsMidSentence` / `isWrappedProse` guards heading detection applies: `…in Part II,\nItem 8 on this Form 10-K…`); or when it is a TOC row (`isTocRow`). A TOC row ends in a page cell set off by **two or more** spaces or a tab (`TOC_PAGE_TAIL_RE`; one space would read the year in `as of June 30, 2026` as a page number), or its next line with content is a page cell (`PAGE_CELL_LINE_RE`: a number, `F-3`, or a lowercase roman `ii`, which counts only on its own line — `Part iv` is not a page cell); either form may put a `Page` word before the number (`Page 3`), spacer lines passed over (an HTML TOC table puts NBSP rows between its cells). A line that is not a marker, followed directly by an Item/Part marker, is a TOC title cell with no page column. A marker line — a Part over its first Item — is a TOC row when that Item row is: the Item line ends in a page cell, the line below it holds or ends in one (a title cell, or Aware's wrapped `of Equity Securities....... 12`), or, for a bare marker, the line below its title holds one. Microsoft's FY2026 10-K prints body `PART IV` only as page furniture, so `part iv` reaches the line tier, where `PART IV / Item 15. / Exhibit and Financial Statement Schedules / 93` was taken; it now lands on the body's first `PART IV` (352884). A body Part sits over its Item too, but prose follows, not a page cell. The first remaining line wins, at its first non-blank character; one linear pass, run only after both heading tiers miss. A line landing says so in the notice, composed into the truncation guidance when the page is cut (`notice` is last-wins): `No detected heading matches section '<value>', so this page starts at the first document line that does.` — no new output field. Every #166 acceptance offset holds. A heading hit always beats a line hit (Reddit's `use of proceeds` → the `USE OF PROCEEDS` heading). (4) `section_not_found`. **The outline cut (#169):** past 50 detected headings the outline is every Item/Part heading (`STRUCTURAL_HEADING_RE`, bare markers included) plus the earliest other headings up to 50, in offset order (`selectOutline`) — shared by the first-page `outline` and the `section_not_found` message and data. When the Item/Part headings alone pass 50, each bare marker that a later titled heading for the same Item or Part repeats is left out first (`outlineStructuralHeadings`): DRDGOLD's FY2026 20-F renders all 31 TOC Items as bare markers (`ITEM 3.`), which filled the cut and pushed body Items 16A–19 out; its outline now runs `ITEM 1.` through `ITEM 19. EXHIBITS` from the body. A bare marker no titled heading repeats stays, and `section` still matches every heading. The miss message heads a cut outline `Outline (50 of N detected headings):` (else `Outline:`), and a cut first page appends `The outline lists 50 of N detected headings, every Item and Part heading among them; section matches all of them.` to the truncation guidance — `…among them, bare table-of-contents copies of titled ones aside; …` when bare markers were left out (DRDGOLD: `The outline lists 50 of 142 detected headings, every Item and Part heading among them, bare table-of-contents copies of titled ones aside; section matches all of them.`), and no Item/Part clause when the kept Item/Part headings alone pass 50, which reaches `structuredContent.notice` and `content[]` — no new output field, since the outline is emitted only on a truncated first page. A document of 50 or fewer headings renders as before.
105
+ - **Filing content:** HTML → text via `html-to-text` library; pre-2005 filings produce noisier output. The library walks the DOM recursively, so `CONVERT_OPTIONS` caps it at `limits.maxDepth: 512` with a `[…]` ellipsis where the cut falls — without it deep markup overflowed the stack and one such candidate failed `search_filings`' whole pre-2001 text scan (#118). The bound must hold on both runtimes the server ships under: a fresh Node process overflows near 1,360 levels of nested lists (its most stack-hungry shape), Bun near 13,000; real HTML filings measure under 25 levels. Legacy SGML `<PAGE>` markers are replaced with a line break before parsing (`neutralizePageMarkers`) — the parser never closes them, so a text document otherwise nests one level per page and a long one would be cut at the cap; with them gone the deepest measured `.txt` sits under 100 (an EX-27 schedule's unclosed tags). Both changes leave every sampled filing's text byte-identical. `limits` deep-merges with the library defaults, so its 16,777,216-character `maxInputLength` (silent truncation, no ellipsis) still applies. **Plain-text SGML bodies keep their lines (#136):** the parser collapses every newline run to a space, so a plain `<TEXT>` body — a pre-2001 full submission, a 2000s ASCII primary document — lost the line structure `detectHeadings` anchors on (Apple's 1996 10-K came back as 104 lines with an empty outline). `preservePlainTextBodies` runs inside the one `convert()` call: a body is HTML when `<html`, `<body`, `<p`, `<div`, `<br`, `<font`, `<tr`, `<td`, or `<xml` opens anywhere in it (`\b`-bounded, so `<PAGE>`, `<PDF>`, and EX-27's `<PP&E>` don't trip `<p`; no `<table`, since EDGAR ASCII tables are `<TABLE>`); any other body has every tag dropped, `&`/`<`/`>` escaped, and goes in a `<pre>`. The list is broad because misreading HTML as plain wrecks it, while the reverse is only the old output. Tags are dropped rather than parsed: the parser formats `<TABLE>`/`<CAPTION>` as blocks inside the `<pre>`, and a stray `<word` in prose would swallow text up to the next `>`. The `<TEXT>` wrapper matches case-sensitively — EDGAR writes it uppercase, and an inline SVG `<text>` is not a body — and the scan is one forward `indexOf` pass, since a lazy `<TEXT>([\s\S]*?)</TEXT>` regex goes quadratic on openers with no closer. Input with no plain body reaches the parser unchanged, so current HTML and 2000–2004 SGML-wrapped HTML convert byte-identically; their outlines change only where an unpunctuated Item cross-reference opens a line and drops out as wrapped prose (see Heading detection). Kept indentation grows plain extractions about 19%. **Every document of a submission holding an HTML document (#159):** the parser converts only `<body>` elements when the input has one, so an HTML-era submission (HTML primary, plain EX-10/EX-27 exhibits) lost every plain exhibit, and the pre-2001 scan could not match exhibit text. When the input holds `<body`, each plain body's `<pre>` goes inside a `<body>` of its own, so every document converts in document order, separated by the block break the parser already puts between two HTML documents — no per-document label, as multi-HTML submissions never had one. Uuencoded payloads (GRAPHIC, a PDF behind `<PDF>`, whose encoded bytes can pass the HTML test) and machine-readable files by `<FILENAME>` extension (`.xml`/`.xsd`/`.css`/`.js`/`.json` — a modern full submission's XBRL and renderer files read as plain) stay outside, dropped as before, and so does the SEC header, as it always was for these submissions. Input with no `<body>` — a plain-only submission — converts whole, header included. Over 93 corpus files only the 10 submissions mixing HTML and plain documents change: each old text is a prefix of the new one and all 186 old outline entries keep their offsets. The primary is document 1, so its headings come first and exhibit headings only fill what the 50-entry cap leaves; cross-document keep-later dedup moved none. **The XBRL renderer's files are cut (#160):** in the same `<body>` case, a document whose `<TYPE>` is `XML` is removed from the input. That is the type EDGAR gives every file its renderer generates (`R1.htm`…, `Show.js`, `report.css`, `FilingSummary.xml`, described `IDEA: XBRL DOCUMENT`). The viewer pages carry `<body>` elements, and so does 2011's `Show.js`, in a string literal, so the R-file tables followed the filed documents. The rule keys on the type, not the `R\d+.htm` name: across 79 sampled 2014–2026 indexes every one of 1,081 R files is typed `XML`, no filed document is, and inline-XBRL primaries carry their form type. Only the 4 corpus submissions holding renderer files change, each new text a prefix of the old. Indented headings stay undetected (centered S-4 headings; Items indented in 2 of 29 sampled 10-Ks): admitting indentation fills the outline cap with centered cover legends and promotes indented TOC rows. The one exception is a same-line titled Item opening on a single NBSP (#176, see Heading detection). **A primary the archive does not serve (#158):** many 2000–2001 indexes list sequence-numbered documents (`0001.htm`, `0001.txt`, `0002.txt`) that answer 404 while `<accession>.txt` is served. When no candidate CIK serves the index's primary, `resolveFilingArchive` reads the full submission instead: `primary_document` and `filing_url` name the `.txt`, and the notice (composed into the truncation guidance, since `notice` is last-wins) names the unserved file. The fallback runs after the CIK loop, so a CIK that serves the primary still wins, and a served primary costs no extra request. A caller-named `document` is never swapped — a listed-but-unserved one fails `document_not_found` with a hint naming `<accession>.txt`. The extract cache stores the document and CIK each text came from, so a cache hit reports the same primary, notice, `cik`, and `filing_url` as the miss that filled it, with no candidate-CIK search
106
+ - **Heading detection:** `detectHeadings` runs three line-anchored patterns — the all-caps forms, the mixed-case same-line `Item N. Title` / `Part II` forms (#71), and a bare `Item N` marker whose title lands on a later line. **One NBSP may open a same-line Item line (#176):** both Item arms admit a single U+00A0 ahead of the marker, since an `&nbsp;` before a styled span left Horizon Space's 8-K line `<NBSP>Item 1.01. Entry into a Material Definitive Agreement.` (0001929980-26-000542) out of an outline that listed Items 7.01 and 9.01. A lookahead holds the NBSP to an Item marker; the entry's offset is the marker's (1960, not the NBSP's 1959), `isWrappedProse` and `followsMidSentence` still read the line from its real start, and a line that is a bare marker once its page cell is stripped (`<NBSP>ITEM 2.02`, `<NBSP>Item 1. 3`) yields nothing (`itemEntryOffset`). Nothing wider: ASCII-space or tab indentation, two NBSPs, and NBSP-led Part lines stay out — Microsoft's 2004 10-K (0001193125-04-150689) stamps an NBSP-led `Part I` on every page, and admitting those turns its body `PART I` (3832), `PART II` (64076), and `PART IV` (308565) into running-header furniture. Across the 67-filing corpus the change adds that one entry and removes none. Every whitespace match is horizontal-only: `\s` matches `\n`, which fused runs of consecutive all-caps lines into composite entries (`"TABLE OF CONTENTS\n\nPART I"`) (#105). Horizontal-only is not space-or-tab — filings put NBSP inside all-caps headings too (`MICROSOFT<NBSP>CORPORATION`), so the all-caps arm nests the class-set difference `[\s--\n]` in its class (the `v` flag). It is one class, not the class alternated with `[^\S\n]`: the quantified alternation slows sharply on very long lines, while a single class scans in linear time. The all-caps line admits `-`, `–`, `—`, digits, and apostrophes, and `isAllCapsHeadingText` bounds it and narrows the last two. Its heading text runs at most `MAX_TITLE_CHARS` (140): past it the line is a paragraph set in capitals, which the widened class otherwise admitted (Alternative Ballistics' 308-character risk-factor coda took `section: "business"` by substring from `OUR BUSINESS`; EuroSPAC's 860-character tax legend; GT Biopharma's 288- and 326-character voting notices). A digit is admitted only inside a whitespace-delimited `\d{1,2}[A-Z]?\.?` token judged on the whole line, a TOC page cell included (`NOTE 10`, `SCHEDULE 14A`; `USD 2,929`, `FMR LLC(4)`, `WASHINGTON, D.C. 20549`, and `…ACT OF 1934` stay out), and an apostrophe only inside a word or after a word-final `S` (`MANAGEMENT’S`, `STOCKHOLDERS’ EQUITY`), never opening or closing the heading. The all-caps `ITEM …` arm takes the rest of its line whatever it holds, and neither Item arm admits a line that runs a table of contents together: a second Item marker, or a page cell mid-line followed by two or more spaces and the next cell (`isRunTogetherToc`). Without it a 238-character 8-K table of contents (`ITEM 8.01. OTHER EVENTS. 3 ITEM 9.01. …`, Amazon's 0001104659-26-107526) was one heading and took `item 8`, `item 9`, and `exhibits`. The test is on shape, not length: a 140-character bound on that arm dropped a proxy's one proposal heading, Modine's 185-character `ITEM 1 – TO APPROVE THE PROPOSAL TO AMEND THE Amended and Restated Articles…` (DEF 14A 0001104659-26-115309), and turned `item 1` and `proposal` into misses. Across 104 cached filings (244 all-caps and 568 mixed-case Item lines) the shape test flags only the Amazon line, and no all-caps Item line runs past 240 characters, so the arm carries no outer length bound. The mixed-case same-line arm keeps its title bound as its prose guard. A colon stays out: every colon line it admitted across 26 sampled documents was a statement caption (`CURRENT ASSETS:`). A Part line may carry a title — whitespace, an optional `-`/`–`/`—`/`:`/`.`, then an uppercase letter, at most 140 characters — and is a cross-reference rather than a heading (`isPartCrossReference`), on the mixed-case and the all-caps arm alike, when it sits under a line ending on a comma or semicolon like an unpunctuated Item, or when its title opens on another marker (`Part II — Item 5. Market for Registrant’s Common Equity`, a bank 10-K's cross-reference index row), which would otherwise also displace the bare body Part it names. After dedup, a bare Part entry gives way to a later titled heading for the same numeral (`dropBarePartsTitledLater`, which applies the same marker-title test): a 10-Q's TOC row `Part II` and its body `PART II — OTHER INFORMATION` have different keys, so without the merge `section: "part ii"` landed on the TOC row 50,045 characters early (Apple Q3 FY2026). Over the 26-document corpus these rules add 72 headings and remove 4 TOC Part rows: 66 section headings (all 18 of Microsoft FY2026's `NOTE N — TITLE` lines, statement titles, titled Parts) and 6 furniture (statement captions, `SCHEDULE 14A`, a `PAGE` stamp, a wrapped legend line), no numeric table row (#170). The Item code spans `16A`–`16K` and the current 8-K's dotted `N.NN` (`ITEM_CODE`, shared by all four Item patterns), and the marker's period is optional — 20-F filers write the lettered markers unpunctuated. A dotted code may also abut its title or carry a paragraph suffix (`Item 9.01Financial Statements…`, `Item 9.01(d). Financial…`, the shapes html-to-text leaves when the marker and title are separate spans), and then the title must start with a letter, so a bare `Item 5.02.` line never matches on its own. A code in `CURRENT_EIGHT_K_ITEMS` takes a 160-character title bound (`EIGHT_K_MAX_TITLE_CHARS`) on the same-line arm and in `usableTitle` — Item 5.02's 146-character title is the longest across the 53 sampled 8-Ks, and the 14-character margin admits a short trailing note while a paragraph opening on the marker line (`Item 5.02(b) On June 2, 2026, the Board…`) runs to hundreds of characters and stays out — and every other marker keeps the 140-character prose guard, a dotted code outside the table (`Item 7.09`) included. Across 53 current 8-Ks the outline holds all 112 item codes EDGAR lists for them (24 before); without the bound exemption it holds 89, without the abutting/suffix form 103 (#165). When an all-caps marker line (`ITEM 2.02`) both matches the all-caps arm alone and joins the title below it, and both entries survive dedup at one offset, the bare entry is dropped — one heading, listed once. Because the period is optional, an unpunctuated same-line Item is dropped as wrapped prose (`isWrappedProse`) when its next word starts lowercase or the line above ends on a comma or semicolon. A line ending on a lowercase word is no evidence: a running page header (`Table of Contents`) sits directly above an unpunctuated body heading (`Item 5 — Other information`) at the top of a page. A plain-text 10-K's cross-reference that wraps to start a line with `Item 8 on this Form 10-K…` otherwise takes the `item 8` landing ahead of the real heading (#136). **A bare marker is admitted on the shape of what follows, not just on a title being there:** a blank line between marker and title admits the pair; without one, the pair is admitted only when no bare page-number line sits under the title, since marker/title/page-number on three tight lines is the TOC-cell artifact #71's guard exists to reject. That test is one-way — it rejects a TOC row, it does not certify a heading — so a marker line anywhere else (a page footer, an unwrapped paragraph) is admitted too, taking the next line as its title; the entry carries the marker's own offset, so `section` still lands in the right place. A heading text repeated `RUNNING_HEADER_MIN_OCCURRENCES` (5) times at distinct offsets is page furniture — C3is' FY2025 20-F repeats `TABLE OF CONTENTS` 129 times and a 10-Q stamps `PART I` on all 47 of its pages, while a heading printed once in the TOC and once in the body occurs twice. **Furniture is then handled by shape, two ways:** a bare marker (`Item 1`, `PART I`) carries no navigable text, so no occurrence of it earns an entry and every one is dropped; anything else keeps its FIRST occurrence and drops the rest, inverting the keep-later rule — a filing that runs a section's own heading across every page of that section (`NOTES TO THE CONSOLIDATED FINANCIAL STATEMENTS` through an S-1's financial statements) prints it first at the section start, and keep-later would land it on the section's last page instead. So the outline carries no fused composite and at most one entry per running header, at its first occurrence. **A bare marker is counted on its own line, never on the joined heading:** a page footer's marker takes its title from whatever prose the next page opens with, so every composite it produces is a different string and only the marker line repeats — count composites instead and a 10-Q's per-page `Item 1` footer yields one bogus entry per page, crowding the real Items past `maxEntries`. That count is per rendered form (`Item 1` and `Item 1.` are separate), since a filing stamping the bare form on every page still writes the punctuated one for the real heading. Dedup (keep the later, i.e. body, occurrence) keys on `headingKey` — `foldForHeadingMatch`, a terminal-punctuation strip, and the separator between an Item/Part marker and its title folded to one space — never raw lowercase: the same filing prints one marker with different space characters and a trailing period in its TOC and its body (`Item 3.` with U+0020 vs U+2009; `…Policies` vs `…Policies.`), and a raw key leaves both in the outline. The separator is the third difference: Worthington Steel's 10-Q (0001968487-26-000036) prints `Item 1A. Risk Factors` in its TOC and `Item 1A. – Risk Factors` in its body, so both kept an entry and `risk factors` landed on the TOC row at 4292 instead of 250397. A marker key is rebuilt as the marker, one space, and the title (`item 1a risk factors`), whatever period, dash, or colon stood between them (`MARKER_TITLE_RE`, exported for the section matcher); across the 67-filing corpus that drops the 10-Q's 11 TOC copies (34 → 23 entries) and moves no other outline entry (#174). The furniture count, and the copy furniture keeps, key on the printed form instead — `headingKey` short of the separator fold (`printedKey`) — so a TOC row and a body heading never pool with a running header printed with another separator and lose the body heading to the TOC row. A TOC row's trailing page cell — a page number or a lettered page (`F-3`), optionally after a `Page` word (`Page 1`), set off by a tab or two or more spaces (`TOC_PAGE_TAIL_RE`) — is stripped from every heading for the same reason. Without the `Page` form, First Bancorp's 8-K TOC row `Item 8.01 Other Events Page 1` (0000765207-26-000110) kept its cell, never deduped against the body `Item 8.01 Other Events.`, and took `item 8` at 1998 instead of 2227. A number after a single space is the heading's own (`PROPOSAL NO. 1`, `ARTICLE 5`, `…Securities Exchange Act of 1934`): the single-space strip cut it off, and dedup then merged GT Biopharma's three `PROPOSAL NO. N` headings into one `PROPOSAL NO.` at the last (#172). Across 81 cached filings the change moved nothing else in any outline. Lettered sub-item markers (a bare `A.`/`D.` inside an Item) are deliberately not detected — single letters recur throughout filing prose — and neither is a digit-led sub-item line (`3D. RISK FACTORS`; the all-caps arm needs a leading letter): `section` reaches a labelled line by stripping the label at comparison time instead (#175, see Section matching), so no detected heading is added.
107
+ - **Section matching:** `get_filing`'s `section` compares through `foldForHeadingMatch` (`filing-to-text.ts`) on **both** operands — Unicode whitespace runs collapse to one space, U+2018/U+2019 → `'`, U+201C/U+201D → `"`, then lowercase. Comparison-time only: `FilingHeading.heading`, the `outline` output, `content[]`, and every character offset keep the document's own bytes. EDGAR headings carry NBSP runs and typographic quotes, so a caller re-sending an outline heading through a UI that normalizes either one never byte-matched the heading it came from — which broke the `section_not_found` recovery loop the error's own hint points at (#106). No fuzzy matching — `"item 8"` resolves only to Item 8 — but every tier runs on folded operands, so a single-spaced needle also reaches a heading whose marker carries an NBSP run or a double space (`Item 9.` in a 2004 10-K). Folding only one operand does not fix the round trip. One boundary (`sectionMatches`): a needle ending in a number never continues into a longer one. A trailing digit never runs on into another digit, so `item 1` matches `Item 1.` and `Item 1A` but never `Item 12`, and `item 5` reaches an 8-K's `Item 5.02` while `item 5.0` does not. A trailing roman numeral (a final word of `i`, `v`, `x`) must end at a space, punctuation, or the heading's end, so `part i` never matches `PART II`, `PART III`, `PART IV`, or `PART II, III`: Sono-Tek's 10-Q (0001171520-26-000305), which prints Part I only inside a one-line TOC, landed `part i` on `PART II - OTHER INFORMATION`, and Microsoft's FY2026 10-K on `PART II, III` (348204). `part i` now falls through to the line tier (Microsoft: the body `PART I` at 6900) or misses with the outline (Sono-Tek; an S-1, whose Part I is the unlabeled prospectus). Plain-text outlines keep TOC rows worded differently from their body heading, and a TOC row for Item 10–16 precedes the body Item 1 — a bare substring test landed `item 1` on another Item in 6 of 37 plain 10-K/10-Q outlines and 1 of 270 HTML landings (#136). **Resolution order (#169, #166, #177)**, first hit wins, over every detected heading (`detectHeadings(text)`, which takes no cap, run once per call — the 50-entry cap belongs to the rendered outline alone; with the first 50 only, a heavy 10-K's financial-statement captions put Items 9B–16 out of reach) and the document's own lines: (1) **exact** (`resolveSection`) — a heading whose `headingKey` (the dedup key: the fold, a terminal-punctuation strip, and the marker separator folded to one space) equals the needle's, or whose title after its Item/Part marker does — failing that, the same with a leading sub-item label removed (below); a bare marker (`ITEM 1.`, `Part II`) has no title and is never a candidate, or GSBD's `item 1` jumps to a bare TOC `ITEM 1.`. The tier exists for #170's widened all-caps class: CCC's `SPECIAL NOTE REGARDING FORWARD-LOOKING STATEMENTS AND RISK FACTORS` (7562) otherwise takes `risk factors` from `Item 1A. Risk Factors.` (67351). (2) **Item/Part substring** — the first Item or Part heading (`STRUCTURAL_HEADING_RE`, bare markers included) that `sectionMatches`: a value inside an Item title names that Item, while a line equal to it is often a table-row label (CCC's `compensation` → `Item 11. Executive Compensation` at 493087, not the `Compensation` cell at 418658). (3) **an equal line** and (5) **a prefix line** — a document line (`matchSectionLine`, `filing-to-text.ts`, which reports which of the two it found; `findSectionLine` is its offset alone), for a mixed-case subheading the outline never lists (`Liquidity and Capital Resources`): a line whose `headingKey` equals the needle's (3), else — only when no line is equal — one that starts with it (5) at a word boundary within `MAX_TITLE_CHARS` (a date-suffixed title: `Consolidated Schedule of Investments as of June 30, 2026`), unless a one-word needle runs on into a longer Title Case title (`continuesIntoLongerTitle`: the next word opens on a capital in a line that is not all caps). One word heads many unrelated titles — `business` landed on `Business Combinations.` (C3is), `Business Seasonality and Product Introductions` (Apple 10-Q), and a cross-reference line `Business Overview—Governmental regulations…` (DRDGOLD); a needle of two or more words keeps its continuation (`Human Capital` → `Human Capital Management`), and a suffix opening lowercase or on punctuation (`(continued)`, `—Parent Company Only`) always does. The cost: `forward-looking` no longer reaches `Forward-Looking Statements`, nor `revenue` `Revenue Recognition`; both miss with the outline. Equality goes first because a prefix match alone lands `Competition` on an earlier line that merely begins with the word. In both line tiers a line is skipped when it opens lowercase (a wrapped sentence or bullet); when it sits under a line ending on a comma or semicolon, or opens on an unpunctuated Item marker followed by a lowercase word (a wrapped cross-reference, the `followsMidSentence` / `isWrappedProse` guards heading detection applies: `…in Part II,\nItem 8 on this Form 10-K…`); or when it is a TOC row (`isTocRow`). A TOC row ends in a page cell set off by **two or more** spaces or a tab (`TOC_PAGE_TAIL_RE`; one space would read the year in `as of June 30, 2026` as a page number), or its next line with content is a page cell (`PAGE_CELL_LINE_RE`: a number, `F-3`, or a lowercase roman `ii`, which counts only on its own line — `Part iv` is not a page cell); either form may put a `Page` word before the number (`Page 3`), spacer lines passed over (an HTML TOC table puts NBSP rows between its cells). A line that is not a marker, followed directly by an Item/Part marker, is a TOC title cell with no page column. A marker line — a Part over its first Item — is a TOC row when that Item row is: the Item line ends in a page cell, the line below it holds or ends in one (a title cell, or Aware's wrapped `of Equity Securities....... 12`), or, for a bare marker, the line below its title holds one. Microsoft's FY2026 10-K prints body `PART IV` only as page furniture, so `part iv` reaches the line tier, where `PART IV / Item 15. / Exhibit and Financial Statement Schedules / 93` was taken; it now lands on the body's first `PART IV` (352884). A body Part sits over its Item too, but prose follows, not a page cell. The first remaining line wins, at its first non-blank character; one linear pass serves tiers 3 and 5, run only after tiers 1 and 2 miss, so a value landing on a tier-4 heading now pays that pass (7–11 ms on a 1 MB S-1). A cut page's guidance after a `section` landing says to pass `next_offset` as `offset` without `section`, which outranks `offset` — kept, it returns the same page. A line landing says so in the notice, composed into the truncation guidance when the page is cut (`notice` is last-wins): `No detected heading matches section '<value>', so this page starts at the first document line that does.` — no new output field. An equal line that outranked a heading merely containing the value says `No detected heading equals section '<value>', so this page starts at the first document line that does, ahead of headings that only contain it.` instead, since that heading does match by substring. Every #166 acceptance offset holds. (4) **other substring** — the first other heading that `sectionMatches`, after the equal line: a heading that merely holds the word took Arxis' S-1 `business` to `NOTE 3. BUSINESS COMBINATIONS` (544705) past its `Business` heading line (323578), and CCC's `revenue` to `COST OF REVENUES` (332106) past the MD&A `Revenue` subheading (256790). An exact or Item/Part heading still beats an equal line (Reddit's `use of proceeds` → the `USE OF PROCEEDS` heading); every other heading ranks after one. Over 67 corpus filings × 103 section values the order moved 24 landings, each one the validation predicted: 17 onto the section named (Microsoft FY2026 `general` → `GENERAL` at 8720, not `GENERAL RISKS`; DRDGOLD's `risk`, `directors`, `management`, and `employees` → body Items instead of TOC rows), and 7 onto a different place that is defensible but that a reader may like less, since an Item heading or a line equal to the value now outranks a heading that only holds it — JPM's 10-Q `risk` lands on Item 3, a one-sentence cross-reference, not `FIRMWIDE RISK MANAGEMENT`; `revenue` in Microsoft's 2004 and FY2026 10-Ks and Aware's on a selected-data, MD&A-table, or income-statement row label rather than a heading containing the word; `summary` on Item 16 `Form 10-K Summary`; Intel's 2000 `consolidated balance sheets` on an Item 14 index row. (6) `section_not_found`. **A proxy statement keeps its headings ahead of the lines:** for a proxy or information statement form (`PROXY_FORM_RE`: DEF 14A, DEFA14A, DEFM14A, PRE 14A, DEF 14C, …), tier 1 is followed by the first heading of any kind that `sectionMatches`, then the line tiers — its ITEM headings are ballot proposals, not the Regulation S-K item taxonomy, and its lines equal to a value are mostly table cells. Under tiers 2–4 Goldman Sachs' DEF 14A (0001193125-26-117433) took `compensation` from the CD&A (123680) to `ITEM 2. … (SAY ON PAY)` (274744) and `summary` from `EXECUTIVE SUMMARY` (16953) to a Pay versus Performance column header (282286). The form is read before `section` resolves, so the `.hdr.sgml` fallback request (#126) now precedes a `section_not_found`. **Inside each heading tier a body heading ranks ahead of a table-of-contents row (#174):** a heading is a TOC row when its title line — its own line, or the next line with content after a bare or joined marker — is one by `isTocRow` (the line tiers' test), or the next line with content below it ends in a page cell (a TOC title wrapped onto a second line, Aware's `of Equity Securities....... 12`); `isTocHeading` in `filing-to-text.ts` owns the test, finding the line start from the newline before the offset since an NBSP-led Item's offset sits one past it. A TOC row still lands when it is its tier's only match. Three TOC shapes survive dedup: a row worded apart from its body heading (First Bancorp's 8-K `Item 9.01 Financial Statements and Exhibits. Page 1` took `item 9` at 2041 from the body Item at 2402), a bare all-caps marker over its title and page cells (DRDGOLD's FY2026 20-F `ITEM 3.` / `KEY INFORMATION` / `6` took `item 3` at 6174 from `ITEM 3. KEY INFORMATION` at 34346, and nearly every other `item N` the same way), and the title cell itself (`key information`). TOC rows are ranked, never dropped or pushed behind the line tiers: the test also reads a body heading over a page number (CCC's `Item 6. Reserved.` at 234199, Reddit's S-1 `PART II` at 1021631), and behind the line tiers CCC's and Microsoft's `item 6` left the body heading for a bare TOC marker line. With the dedup-key fold, the ranking moved 83 landings over the corpus, each from a TOC row onto a body heading (DRDGOLD 35, Worthington Steel 20, GSBD 9, Amazon's 2003 10-K 8, Aware 6, First Bancorp 4, CCC 1), and no other: 78 onto the body heading the row copies, 5 onto another the value names (Amazon 2003's `part ii`–`part iv` onto its body Parts, Aware's `management` from its TOC Item 12 to body Item 7, Worthington Steel's `other information` from its TOC Item 5 to body `PART II. OTHER INFORMATION`). **A leading 20-F sub-item label is ignored (#175):** 20-F filers label sub-items (`3D. RISK FACTORS`, `B. Business Overview`), so a plain-phrase value never equalled the line. Tier 1 and both line tiers also compare the heading or line key with a leading label removed (`withoutSubItemLabel`: `/^(?:\d{1,2}[a-z]|[a-z])\. /` on the folded key, anchored, so constant-time), and the one-word longer-title rule judges the line after its label — Regencell's `B. Business Overview` keeps `business` a miss while `business overview` lands (151573). The value itself is never stripped, so `3D. RISK FACTORS` still matches its own line; a line opening on an initial (`L. John Doerr`) strips the same way and so matches only a value equal to the rest of the line. The landing is the line's first character, label included (DRDGOLD `risk factors` → 34503). Tier 1 strips only once no heading equals the value as printed, so an earlier `A. RISK FACTORS` never takes `risk factors` from a later `Item 1A. Risk Factors`; a labelled body heading still ranks ahead of a TOC row. The line tiers keep the first line equal either way: across the corpus the labelled line they land on is the 20-F sub-item itself and a later unlabelled one is narrower (DRDGOLD `business overview` → `4B. BUSINESS OVERVIEW` at 176975, not `Business overview` at 269017). Over the corpus the strip moved 23 landings, all in the three 20-Fs: 19 misses now land on the labelled sub-item, and 4 earlier landings move onto one (DRDGOLD `compensation` from the `Compensation of key management` subheading inside Item 6B, 328266, to `6B. COMPENSATION` at 324690); no filing without labelled sub-item lines changed. **The outline cut (#169):** past 50 detected headings the outline is every Item/Part heading (`STRUCTURAL_HEADING_RE`, bare markers included) plus the earliest other headings up to 50, in offset order (`selectOutline`) — shared by the first-page `outline` and the `section_not_found` message and data. When the Item/Part headings alone pass 50, each bare marker that a later titled heading for the same Item or Part repeats is left out first (`outlineStructuralHeadings`): DRDGOLD's FY2026 20-F renders all 31 TOC Items as bare markers (`ITEM 3.`), which filled the cut and pushed body Items 16A–19 out; its outline now runs `ITEM 1.` through `ITEM 19. EXHIBITS` from the body. A bare marker no titled heading repeats stays, and `section` still matches every heading. The miss message heads a cut outline `Outline (50 of N detected headings):` (else `Outline:`), and a cut first page appends `The outline lists 50 of N detected headings, every Item and Part heading among them; section matches all of them.` to the truncation guidance — `…among them, bare table-of-contents copies of titled ones aside; …` when bare markers were left out (DRDGOLD: `The outline lists 50 of 142 detected headings, every Item and Part heading among them, bare table-of-contents copies of titled ones aside; section matches all of them.`), and no Item/Part clause when the kept Item/Part headings alone pass 50, which reaches `structuredContent.notice` and `content[]` — no new output field, since the outline is emitted only on a truncated first page. A document of 50 or fewer headings renders as before.
108
108
  - **Binary filing documents:** a filing index routinely carries scanned pages, PDF exhibits, `Financial_Report.xlsx`, and `-xbrl.zip` — `filingToExtract` runs `html-to-text` over whatever bytes arrive and never inspects a content type, so requesting one used to return decoded binary as `content` with no error (#96). `get-filing.tool.ts` detects them with one rule, `binaryType`: the filename extension (`binaryTypeFromName`), the one signal every filing carries, plus a header-page `GRAPHIC` type for a scan named oddly — the header page is read before the body (#161). Two halves, both needed — `documents[]` entries carry `binary: true`, and `resolveFilingArchive` throws `binary_document` before `tryGetFilingDocument` runs, since the `document` input is an unconstrained string and a catalog marker alone cannot stop a caller. Extension-driven and bucket-independent on purpose: a PDF exhibit is typed `EX-99.*` and files under `exhibits`, not among the graphics.
109
109
  - **Dataframes:** `secedgar_search_filings`, `secedgar_get_financials`, `secedgar_fetch_frames`, `secedgar_compare_companies`, `secedgar_get_insider_transactions`, `secedgar_get_institutional_holdings`, `secedgar_get_material_events`, `secedgar_find_holders`, `secedgar_get_beneficial_owners`, and `secedgar_get_fund_holdings` materialize their full result set as `df_<id>` on a shared DuckDB-backed canvas (one per tenant). The two ownership tools register the full parsed set (inline response is a preview capped at `limit`); `get_insider_transactions` scans at least `INSIDER_CANVAS_FILING_SCAN` recent filings when canvas is available (so small inline limits don't under-scan the window), capped at 100, and flags `dataset.truncated` when more Form 4 filings exist beyond the scanned window (#63); with a `filed_after` / `filed_before` window it reads the in-window Form 4 / 4-A rows with an XML primary document — `recent`, then the archive pages overlapping the window, newest first, stopping once the scan cap plus a sentinel row are in hand — and with a canvas parses every one up to 100 instead of the 40-filing floor, since the window already bounds the set; `dataset.truncated` is then the sentinel or unread overlapping pages, and `history_scanned_through` the oldest filing parsed (#127); a company's submissions feed also lists the Form 4s it filed as a reporting owner of another issuer (JPMorgan's 10% stake in a BlackRock fund), so a parsed document whose `issuerCik` differs from the resolved CIK, compared with zero-padding stripped, adds no row inline or to the dataframe and is counted in `filings_other_issuer` — it still counts toward `filings_scanned`, the scan floor, and `history_scanned_through`, and the no-canvas stop rule is unchanged — though a skipped filing adds no transactions, so a no-canvas call reads on through such filings until `limit` transactions are in hand, within the same scan cap (#157); its `shares_traded` is an unsigned magnitude paired with a `direction` (acquire/dispose) column, so net activity is `SUM(CASE WHEN direction='dispose' THEN -shares_traded ELSE shares_traded END)` (#46); `get_financials`' dataframe materializes the source-filing fiscal keys as `source_filing_fy`/`source_filing_fp` (renamed from bare `fiscal_year`/`fiscal_period` — every comparative period restated in one filing carries that filing's fy/fp, so `period_end` is the time key for ORDER BY/GROUP BY/windowing) (#72); `compare_companies`' dataframe holds the FULL aligned series across every period while the inline matrix is capped at `periods`, and each row carries both the aligned `period` key and the underlying XBRL `frame` (they differ for point-in-time concepts) (#85). In an annual matrix an instant enters only on one of the filer's fiscal-year ends and takes the key of the annual duration ending that day. The ends come from the companyfacts payload already in hand (`filerFiscalYearEndsWithComparatives` in `concept-series.ts`, shared with `get_financials`' annual view of an instant): each annual report's own year (the #148 rule) plus the comparative years it restates — its year-long facts ending a whole number of years earlier, within a week — so the first XBRL 10-K's prior-year balance keys too, while a year-long figure ending off the report's calendar (a December filer's trailing year to 09-30) stays out. The #148 test of a 10-Q-held annual frame keeps `filerFiscalYearEnds` (each report's own year only): it matches day of year, where a comparative year adds nothing but a wider tolerance. The key is the first `CY####` frame on any concept in the payload ending then, so a balance lands beside its fiscal year's income lines even when none is requested; with no framed duration ending on it, the calendar year in which that fiscal year's second half starts (SEC's own framing: a year ending 06-30 is that year, one ending 06-27 or 05-31 the year before — Seagate's and Western Digital's years ending 2025-06-27 are `CY2024`; SEC departs from it for the year both ended 2012-06-29, framed `CY2012`, which is why a framed key wins). Interim and cover-date instants stay out of the annual matrix, and a company-concept left with none gets a gap plus a caveat pointing at quarterly; when that empties the whole matrix, `no_comparable_data` says the values were all off fiscal-year ends and its hint names `period_type: "quarterly"` (omitting `period_type` keeps annual). Quarterly keys are unchanged. The year ends are read lazily, only when an annual matrix meets an instant (#171). The period-ends caveat checks every inline column — the newest can hold one company alone — spelling out the newest column whose ends differ and naming the others; the inline window is the newest periods of the union across companies, so a company-concept pair whose values all predate it is named in `caveats` — one line per concept, each company with its newest period and the dataframe pointer — and never in `gaps`, which keeps meaning no value in any period (#144). Each row set carries an all-nullable schema (framework's `inferSchemaFromRows` always emits `nullable: true` since mcp-ts-core 0.10.4). Per-table TTL is delegated to the framework via `RegisterTableOptions.ttlMs` (resolved in mcp-ts-core#140); bridge still tracks metadata expiry in `ctx.state` for provenance. Every producer emits the describe-then-query pointer on the branch that registers the dataframe, composed once by `dataframeGuidance(dataset)` in the bridge so the wording cannot drift; where a truncation call already fires there it rides as that call's `guidance`, since `ctx.enrich.notice` is last-wins across `notice`/`truncated` and a second call would clobber the first (#104). `secedgar_dataframe_query` detects a `row_limit`-bound result from `QueryResult.truncated` rather than `rowCount > rows.length` — `row_limit` is pushed into the query as the provider's cap, so a capped result comes back with `rowCount === rows.length` and the arithmetic cannot see it; `rowCount` is then the cap and not a total, so that branch names the cap instead of printing "of N rows", `row_count_capped` carries the same disclosure into `format()`, and the fallback comparison stays for the `registerAs` path, where `truncated` is never set. A SQL `LIMIT` exactly equal to the cap is indistinguishable from a result that holds that many rows and is reported as exact (#109). `secedgar_dataframe_query` runs framework's SQL gate with `denySystemCatalogs: true` to block `information_schema`, `pg_catalog`, `sqlite_master`, and `duckdb_*` catalog access. Raw DuckDB execution errors (missing table, syntax) are caught in `bridge.query` and re-thrown as structured `missing_table`/`invalid_sql` reasons, with DuckDB's "Did you mean …?" hint stripped so internal catalog names don't leak (#47). The framework's own classifications pass through untouched and are declared on the tool's contract (`thrownBy: 'service'`, since the handler never names them): a SELECT that prepares and then fails on the data — a `Conversion` / `Invalid Input` / `Out of Range` error, e.g. `CAST('abc' AS INTEGER)` — is `ValidationError` with `sql_execution_error` (mcp-ts-core 0.13.4; `DatabaseError` with no reason before), and the gate's `multi_statement`, `denied_function`, `sql_read_only`, and `plan_operator_not_allowed` rejections arrive with their own recovery hints — since mcp-ts-core 0.13.15 the gate also denies engine-setting and SQL-from-string functions (`current_setting`, `query_table`) as `denied_function` and a file named as a table (`FROM 'data.csv'`) as `sql_read_only`. `get_beneficial_owners` registers one row per reporting person rather than one per filing, since powers and percent of class are per person; `get_fund_holdings` registers every position in the report while the inline list is one `limit`-sized page from `offset`, and its rows carry the fund keys (`series_id`, `registrant_cik`, `report_period_date`, `accession_number`) so a portfolio joins the 13F and insider rows on `cusip`.
110
110
  - **Beneficial ownership (13D/13G):** SEC replaced the legacy `SC 13D` / `SC 13G` text filings with structured XML under the `SCHEDULE 13D` / `SCHEDULE 13G` names on **2024-12-18**; the changeover is total, so a legacy-name filter matches nothing recent and a current-name filter matches nothing older. The two schedules are **separate schemas, not one shape with optional fields** — 13D repeats `reportingPersons/reportingPersonInfo` with the power fields flat on each person and the percentage in `percentOfClass`; 13G repeats `coverPageHeaderReportingPersonDetails` with the powers nested inside `reportingPersonBeneficiallyOwnedNumberOfShares`, the percentage in `classPercent`, no reporting-person CIK anywhere, and no purpose-of-transaction field at all (its item 4 is structured ownership data, which is what makes it the passive form). `beneficial-ownership-parser.ts` therefore branches per schema and dispatches on the submission type, falling back to whichever repeated reporting-person element the document carries. **Ownership powers and percent of class are per reporting person, including on joint filings** — a fund, its adviser, and its controlling principal each report the same underlying shares, so the parsed owner list is never collapsed to a group total and the percentages must not be summed. Discovery runs off the issuer's own `getSubmissions()`: SEC cross-lists a schedule under every CIK party to it, so no entity-scoped EFTS step is needed.
@@ -117,7 +117,7 @@ Tailor suggestions to what's actually missing or stale — don't recite the full
117
117
 
118
118
  | Env Var | Required | Default | Description |
119
119
  |:--------|:---------|:--------|:------------|
120
- | `EDGAR_USER_AGENT` | **Yes** | — | User-Agent for SEC compliance. Format: `"AppName contact@email.com"` |
120
+ | `EDGAR_USER_AGENT` | **Yes** | — | User-Agent for SEC compliance. Format: `"AppName contact@email.com"`. A value this runtime's `fetch` cannot send as a header (a character past Latin-1 such as an em dash, or an inner line break) fails startup as a `ConfigurationError` naming the variable, rather than every request failing later (#173). |
121
121
  | `EDGAR_RATE_LIMIT_RPS` | No | `10` | Max requests/second to SEC APIs. Do not exceed 10. |
122
122
  | `EDGAR_RATE_LIMIT_COOLDOWN_SECONDS` | No | `600` | Seconds to stop sending to SEC after a 429. Calls are refused locally for this long, then the first one goes out alone as a probe. SEC lifts its block only after ten quiet minutes, so a shorter value just probes into it. |
123
123
  | `EDGAR_TICKER_CACHE_TTL` | No | `3600` | Seconds to cache the ticker index (company_tickers.json + company_tickers_mf.json). A failed fund-file load is retried after 60 s (or the rate-limit cool-down) instead of standing for the TTL. |
@@ -398,6 +398,7 @@ After a version bump and final commit, publish to both npm and GHCR:
398
398
  bun publish --access public
399
399
 
400
400
  docker buildx build --platform linux/amd64,linux/arm64 \
401
+ --build-arg APP_VERSION=<version> \
401
402
  -t ghcr.io/cyanheads/secedgar-mcp-server:<version> \
402
403
  -t ghcr.io/cyanheads/secedgar-mcp-server:latest \
403
404
  --push .