settfex 0.22.0__tar.gz → 0.22.2__tar.gz

{settfex-0.22.0 → settfex-0.22.2}/AGENTS.md

@@ -126,11 +126,20 @@ window. `docs.accounting` is what the *parser* did with the rows it got: `accoun
 is the one flag to branch on, `accounting.no_link` counts rows whose download link was missing, and
 `accounting.unmapped_headers` names columns the site serves that this library does not model. A
 listing where **every** usable row was lost raises `IncompleteListingError` rather than returning
-`[]`, so an empty list you actually receive means an empty result.
+`[]`, so an empty list you actually receive means an empty result. The same now holds for the
+transport: a failed request raises `FetchError`, and a body that is not a listing page raises
+`ParseError`, instead of parsing to zero rows and looking like "this issuer filed nothing".
+
+One thing deliberately does **not** raise: a broken "display all results" page. That section keeps
+its truncated inline rows and the shortfall shows in `completeness()` — so check that before
+treating a section as complete. As of 2026-09-20 SEC's Thai `fs-kf` page answers HTTP 500, which is
+an upstream bug, not a settfex one.
 
 `download_sec_documents(...)` returns a `DownloadResult` — a list of the files that downloaded,
 which also carries `.failed` (each with its target and reason) and `.is_complete`. Check it before
 reporting a batch as done; dead links on the SEC host arrive as HTML under HTTP 200, not as errors.
+⚠️ Read `.failed` off **that** object: it is a `list` subclass, so slicing, `sorted()`, `list()` and
+comprehensions all return a plain `list` and drop the failure report without warning.
 
 ### ThaiBMA bonds — `from settfex.services.thaibma import ...`
 

{settfex-0.22.0 → settfex-0.22.2}/CHANGELOG.md

@@ -7,6 +7,109 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.22.2] - 2026-09-20
+
+### Fixed
+
+- **56-1 and 56-2 sections truncated even with `follow_view_more=True`, because their
+  "display all results" slugs were unmapped.** `_CATEGORY_FOR_VIEWMORE_SLUG` covered `fs-norm`,
+  `fs-kf` and `fs-mda` — the three sections a single `FS` search returns — and nothing else. A
+  56-1 or 56-2 section over the site's ~10-row inline cap therefore linked to its complete list
+  through a slug the map did not recognise, and the link was skipped.
+
+  0.22.0's own notes predicted this and said the shortfall WARNING would be what surfaced it. It
+  was, from the other side of the API: a consumer read `completeness()` and reported 10 of 12 on a
+  Thai 56-1. Both slugs were then **live-probed rather than guessed** (CPALL / PTT / SCB × th / en,
+  2026-09-20), which is how the second one was found at all.
+
+  The gap was larger than first reported, and not Thai-only. Over a 2000–2026 window:
+
+  | Listing | Before | After |
+  |---|---|---|
+  | CPALL 56-1 `th` | 11 of 23 | **23 of 23** |
+  | PTT 56-1 `th` | 11 of 25 | **25 of 25** |
+  | PTT 56-2 `th` | 10 of 15 | **15 of 15** |
+  | PTT 56-2 `en` | 10 of 15 | **15 of 15** |
+
+  Note the `fs-` prefix is the site's own and is **not** a category hint — `fs-r561` is served by
+  the 56-1 search, not the FS one — so a mapping keyed on the prefix would be wrong.
+
+  No API change: the mapping is module-private, and the public `SEC_VIEWMORE_SLUGS` constant (which
+  has no caller, and so would not notice itself going stale) is updated to match, with a test
+  pinning the two equal. The generated API surface is byte-identical to 0.22.1.
+
+
+## [0.22.1] - 2026-09-20
+
+### Fixed
+
+- **The SEC listing path never checked the HTTP status, so an error page came back as an empty
+  list** (issue #131). `_run_search` returned `post_resp.text` without ever reading
+  `status_code`, so a failed request parsed to zero rows and was returned as `[]` with
+  `has_losses == False` — indistinguishable from an issuer that genuinely filed nothing. Worse
+  than a parse bug, because it is *intermittent*: the same query succeeds an hour later, so a
+  backfill records the window as covered and nothing ever marks the gap. Evidenced by a real
+  **HTTP 505** captured from the listing host on 2026-09-20 (committed as
+  `idisc_505_error_page.html`).
+
+  settfex already had the answer everywhere else — `raise_for_status` at ~25 call sites across SET
+  and ThaiBMA, and an explicit check in `download.py`. `financial_report.py` was the one module
+  that did neither. Two rules now, because a status code cannot see everything:
+
+  - **non-2xx → `FetchError`**, naming the status, the report code and the URL;
+  - **a 2xx body that is no listing page at all → `ParseError`**. The rule is the weakest one that
+    separates the populations (no result table *and* no section heading), because the case that
+    must never trip it is a **genuinely empty listing**. Verified against every page this repo
+    holds, including the one whose section is genuinely empty.
+
+  **Behavior change:** these entry points now raise where they previously returned `[]`. Both are
+  `FetchError` subclasses and both were already documented outcomes of the listing call, so an
+  existing `except FetchError` already covers them.
+
+- **A broken "display all results" page silently deleted its whole section** — found by the fix
+  above, on the live site, the same day. A ViewMore page *replaces* its section's inline rows, and
+  0.22.0 applied that replacement unconditionally: a failing page parsed to zero rows, superseded
+  the tally, and the section vanished with the inline rows included. The live Thai `fs-kf` page
+  answered **HTTP 500 on every attempt** on 2026-09-20 while the other five slug/language pairs
+  answered 200, so a Thai listing was losing its entire Key Financial Ratio section in silence.
+
+  An unusable ViewMore page is now **degraded, not raised**: the truncated inline rows are kept —
+  they are the real answer `follow_view_more=False` would have given — a WARNING names the page
+  and its status, and the shortfall shows through `completeness()`. Raising would cost the caller
+  the whole listing over one section. A CPALL Thai listing that returned 167 documents with a
+  silent hole now returns 177 with `key_financial_ratio: (10, 15)` stated in the result.
+
+- **A fifth download-link shape was unrecognised, so the oldest filings were unreachable**
+  (issue #133). `capital.sec.or.th/…/get_zip_all_public_page.php` carries pre-2014 `.DOC`/`.XLS`
+  filings — the history that cannot be re-derived from anywhere else. 0.22.0 counted those rows as
+  `no_link` and warned (that is #127's mechanism working, and it is how the shape was reported),
+  but could not reach the data.
+
+  It is an **indirection**, so recognising the URL is necessary and not sufficient: it answers a
+  2 KB HTML page whose entire body is a JavaScript redirect to a server-minted zip.
+  `classify_download_href` now returns it as `capfin:<comp_id>-<year>-<period>-<lang>`, and
+  `download()` resolves the redirect (one extra request) and returns the real archive — 334 KB
+  across four members, verified live. Three properties the code respects: the target is **minted
+  per request** (so `file_url` stays the stable indirection URL and the `/tmp/…` one is never
+  stored), it is **per-language** (the Thai request mints a different archive), and the pattern is
+  strict with a live off-host check rather than a decorative one.
+
+### Documentation
+
+- **`DownloadResult` loses `.failed` when you derive a new list** (issue #134). Slicing,
+  `sorted()`, `list()`, concatenation and comprehensions all return a plain `list` without
+  `.failed`, `.requested` or `.is_complete`, because that is what the `list` methods construct;
+  `copy.copy()` is the exception. So `for f in sorted(result)` quietly gives up the failure report
+  that #128 added. Documented on the class and in the service docs for now — removing the edge is
+  a breaking change and is tracked separately.
+
+- **The old-format archives are MEMBER-STABLE, not byte-stable.** Re-fetching the same filing
+  legitimately yields different container bytes — 12 bytes of 334,498, the Info-ZIP pack timestamp
+  — while every member payload is byte-identical. **Judge integrity per member; the container
+  sha256 is not an identity for this shape.** An archive keyed on the container hash will see a
+  re-fetch as a new or corrupted object.
+
+
 ## [0.22.0] - 2026-09-20
 
 ### Fixed

{settfex-0.22.0 → settfex-0.22.2}/CLAUDE.md

@@ -412,12 +412,18 @@ See [`CHANGELOG.md`](CHANGELOG.md) for the full, versioned release history — t
 - **A stock-info 404 means "unknown symbol" — unlike the DR-profile 404:** `/api/set/stock/{sym}/info` answers an unknown ticker with HTTP 404 `{"message": "Invalid Stock Name"}` and resolves every real one, so `SymbolNotFoundError` is raised **with** the "did you mean?" suggester enabled (the opposite of `/api/set/dr/{sym}/profile`, which 404s for valid non-DRs and must pass `suggest=False`).
 - **The session cache dir is forced to `0700`, and a caller-supplied one is deliberately NOT — do not "make it consistent":** `diskcache` reads cached values back with **pickle**, so write access to `~/.settfex/cache` is code execution in the caller's process (CVE-2025-69872 / PYSEC-2026-2447 — **no fixed release**, upstream's last release was 2023; it is the only advisory `pip-audit` still reports and the only one on a *runtime* dependency). `_secure_cache_dir()` therefore creates the directory `0700` and **chmods an existing default one in place** (a real side effect on the user's filesystem, logged at INFO). A directory the caller passed is created `0700` when new but, if it already exists group/other-writable, is **reported and left alone** — a shared cache can be deliberate, and silently breaking a multi-account layout is worse than the warning. Windows is skipped outright (POSIX bits do not express this), and every permission step is inside `try/except OSError` because a cache that cannot be chmod'd is still a usable cache. Tests pin all four behaviours in `tests/utils/test_session_cache.py`.
 - **SEC serves FOUR download-URL shapes; two of them were silently dropping live filings:** `classify_download_href` knew `Download?FILEID=<path>` and `ipos/Common/IPOSGetFile.aspx?id=`. **Key Financial Ratio** rows use two more, and both fell through to "not a download link" and made the row vanish — live-probed 2026-09-20 on CPALL, both real downloads: (1) recent rows (2026 Q1/Q2) link through `/public/idisc/Views/FinancialStatementDownload?query=<opaque base64>`, answering `application/zip` — a **134 KB zip of the real PDFs**; (2) older rows (2018, 2019) link through `/public/idisc/views/viewdoc?…&TransId=<id>&FileSeq=<n>`, answering `application/.tif` — a **36 KB scan of the original**. Mapped now as `file_id="fsdl:<blob>"` / `"viewdoc:<id>-<n>"`, both `file_kind=None` (neither URL states a type; `Content-Disposition` names the file at download time). After the fix, CPALL 2015-2026 returns **177 documents with `completeness()` exact in all five categories** — `key_financial_ratio` was `(13, 15)`. ⚠️ **Three things this should teach the next reader.** (a) **It falsifies the reporting issue's own finding:** its corpus scan of 39 captured pages found *zero* natural instances of a dropped row and stated P1 "has not been observed in the wild"; the first two live runs of the new accounting found four. A captured corpus can only show what was captured. (b) **The window is load-bearing:** a narrow recent window passed clean while the 2018/2019 rows were being dropped — **old filings use old URL shapes**, so a probe that only checks recent data cannot see them. (c) **Do NOT generalise the allowlist** into "any link under /public/idisc/ with a query is a download": the "display all results" ViewMore link has exactly that shape, and would become a phantom filing in every truncated section. Expect the list to grow; `ListingAccounting.no_link` is the mechanism that makes the next one visible, and a test pins the ViewMore link as not-a-download. Note `_fallback_filename` keys on the **colon**, not on an `ipos:` prefix: a real FILEID path never contains one, so every synthetic id is safely excluded from being sliced like a path.
+- **The SEC LISTING path checks its HTTP status now — and a ViewMore failure DEGRADES rather than raising (issue #131):** `_run_search` used to return `post_resp.text` without ever reading `status_code`, so an error page parsed to zero rows and came back as `[]` with `has_losses == False` — indistinguishable from an issuer that filed nothing, and intermittent, so a backfill recorded the window as covered. Evidenced by a real **HTTP 505** from the listing host (2026-09-20). Two rules: non-2xx → `FetchError`; a 2xx body with **no result table and no section heading** → `ParseError`. The content rule is deliberately the weakest one that separates the populations, because the case it must never trip is a **genuinely empty listing** — a test pins it against every page this repo holds. It does **not** key on `ctl00_CPH_pnlControl`: the real ViewMore capture has that id and the trimmed `FS_VIEWMORE_HTML` constant does not, so a panel-id rule would call a legitimate page an error. ⚠️ **The severity ladder differs by leg, on purpose.** The POST leg raises (a total loss — nothing to fall back on). A **ViewMore** page is different: it *replaces* its section's inline rows, so 0.22.0's unconditional replacement meant a failing page **deleted the whole section, inline rows included, in silence**. A broken ViewMore is a *partial* loss, so it now keeps the inline rows, warns, and lets `completeness()` show the shortfall. `_map_rows` additionally refuses a **zero-row** input: within this package that can only be a non-listing body, since every captured page carries at least the placeholder row an empty section serves.
+- **The live Thai `fs-kf` ViewMore page has been answering HTTP 500 since at least 2026-09-20 — an UPSTREAM bug, and it is why the degrade path exists:** probed 3× each, `th/fs-kf` returned 500 every time while `th/fs-norm`, `th/fs-mda` and all three `en/*` returned 200. On 0.22.0 this silently removed CPALL's entire Thai Key Financial Ratio section; now the listing returns 177 documents with `key_financial_ratio: (10, 15)` stated in `completeness()` plus two WARNINGs. **If you see a Thai KFR shortfall, check whether SEC has fixed their page before looking in settfex.**
+- **~~`fs-r561` is an UNMAPPED ViewMore slug~~ — CLOSED in 0.22.2, and `fs-r562` with it:** `_CATEGORY_FOR_VIEWMORE_SLUG` used to map only `fs-norm`/`fs-kf`/`fs-mda`, the three sections one `FS` search returns, so a **56-1 or 56-2** section over the site's ~10-row inline cap truncated **even with `follow_view_more=True`**. 0.22.0's notes predicted it and said the shortfall WARNING would surface it — it did, reported by a downstream consumer reading `completeness()`. Both slugs were **live-probed, never guessed** (2026-09-20, CPALL/PTT/SCB × th/en), and the scale was larger than first reported: over a 2000-2026 window CPALL th 56-1 served **11 of 23**, PTT th 56-1 **11 of 25**, PTT 56-2 **10 of 15 in BOTH languages** — so it was never a Thai-only gap. Now 23/23, 25/25, 15/15, 15/15. ⚠️ **The `fs-` prefix is the site's own and is NOT a category hint** — `fs-r561` is served by the *56-1* search, not the FS one; a mapping keyed on the prefix would be wrong. Two surfaces state this fact: `_CATEGORY_FOR_VIEWMORE_SLUG` (used) and the public `SEC_VIEWMORE_SLUGS` in `constants.py` (**used nowhere**, so nothing would notice it going stale) — a test pins them equal.
+- **SEC's old-format filings are on a FIFTH host and behind a JavaScript indirection (issue #133):** `capital.sec.or.th/…/get_zip_all_public_page.php` carries pre-2014 `.DOC`/`.XLS` filings. The URL answers **HTTP 200 with a 2 KB HTML page whose entire body is `document.location = '/tmp/<id>.zip'`** — so recognising the link is necessary but not sufficient, and `download()` resolves the redirect at download time (one extra request). Three properties the code depends on: the target is **minted per request** (`XP`/`sD`/`V1` for the same filing across three captures), so `file_url` stays the stable indirection URL and the `/tmp/…` one is never stored; it is **per-language** (the Thai request mints a different archive, so `lang` is part of the `capfin:<comp_id>-<year>-<period>-<lang>` id); and `comp_id` is a **different id space from `set_id`** in the same query — they are not interchangeable. The redirect pattern accepts an absolute URL on purpose, so the same-host check is live rather than dead code.
+- **These old-format zips are MEMBER-STABLE, not byte-stable — do not key an archive on the container hash:** re-minting the same filing produced a different container sha256 while every member payload stayed byte-identical. Exactly **12 bytes of 334,498** differ, in four 3-byte runs, one per member: the Info-ZIP extended-timestamp (`UT`) extra field, i.e. the pack time. **Judge integrity per member.** A consumer that hashes the container will see a legitimate re-fetch as a new — or corrupted — object. (Also observed: one sample's embedded pack time *predated* the request that minted it, suggesting the server sometimes hands back a previously generated archive. INFERRED from two samples.)
+- **`DownloadResult` loses `.failed` the moment you derive a new list (issue #134, documented not fixed):** `.failed`, `.requested` and `.is_complete` are instance attributes on a `list` subclass, so slicing, `sorted()`, `list()`, concatenation and comprehensions all return a plain `list` without them — 6 of 7 ordinary operations, `copy.copy()` being the exception. `for f in sorted(result)` therefore quietly gives up the failure report #128 added. **Read `.failed` from the object `download_all` returned, before deriving anything from it.** `SecDocumentList` has the identical hazard for `.accounting`/`.reported_counts`. Removing the edge is a breaking change and is tracked for a later release.
 - **`no_href` was three different things, and only one is a defect:** a dropped SEC listing row is a `ไม่พบข้อมูล`/`Data not found` **placeholder** (the site saying the section is empty), a "display all results" **ViewMore navigation row** (how a long section truncates — 24% of live sections), or a **real data row whose download link is gone** (`no_link`). They shared one counter, which is why P1 could not be escalated: any rule strict enough to catch a genuine loss also fired on every truncated section. Split (`ListingAccounting.placeholders`/`navigation`/`no_link`), `no_link` is **0 on every real page captured** and non-zero only on a page derived to exhibit the bug — so it can be escalated without a record-count marker at all. `_MapResult.no_href` is kept as the sum of the three; **branch on `accounting.no_link`, never on `no_href`** (it is non-zero on perfectly healthy pages).
 - **Two cross-checks on a SEC listing, and they answer different questions:** `completeness()`/`reported_counts` compares what you hold against **the number the site printed** — a shortfall there is usually legitimate "view more" truncation, so it is reported and never raised. `docs.accounting` is **ours**: what the parser did with each row it received. A shortfall is a maybe; `accounting.no_link > 0` is always a loss. Do not fold one into the other.
 - **`IncompleteListingError` fires only on a TOTAL loss:** rows classified fine and *every* usable one was dropped for want of a download link. A **partial** loss warns and lands on `accounting.no_link` instead — raising there would turn one bad row in a 200-row listing into no listing at all, and unlike a total loss a partial one is visible in the return value. Same severity ladder as the 0.21.0 `ParseError` for unrecognised headings, and it is a `ParseError` subclass so `except ParseError`/`except FetchError` keep working. The escalation is scoped to the **requested** categories (a single `FS` search returns three sections) — the same principle PR #125 applied to `completeness()`.
 - **An unmapped COLUMN is reported, never raised — and the ignore-list is 5 entries, not 13:** the issue that reported this measured "100% of sections carry an unmapped header", but that count includes the revision-tracking sections, **whose rows are skipped before a single cell is read**. Restricted to sections actually read, the whole corpus serves exactly six unmapped headers, five of which have no model field (`Details`, `รายละเอียด`, `Link`, `Time`, `เวลา` → `_IGNORED_HEADERS`) and the sixth was the real loss (`วันที่ได้รับข้อมูล`, now mapped). So the corpus is 100% clean and the first header this check ever names will be a genuine site change. It **warns**, because an unmodelled column is "the site has more than we model", not lost data — raising would turn a cosmetic change into an outage. Do **not** add the revision-section headers (`Order Date`, `Company Name`, `ชื่อบริษัท`, …) to the ignore-list: they never reach the check, and listing them would make it look broader than it is.
 - **`download_all` returns a `DownloadResult`, not a bare list — a partial batch used to be shaped like a complete one:** with `continue_on_error=True` (still the default) a failure became `None` and was filtered out, so the difference between "I downloaded the filings" and "I downloaded some of the filings" existed only in a log line. `DownloadResult` **is** a `list[DownloadedFile]` (same trick as `SecDocumentList`), so `len()`/iteration/indexing/pass-through are unchanged; it additionally carries `.failed` (target, source document, error, error type), `.requested` and `.is_complete`. `continue_on_error=False` still propagates the first failure.
-- **The SEC listing summary log is derived from the accounting, and its LEVEL is load-bearing:** `Listed 0 SEC document(s)` used to be the same sentence at the same level as a complete success, with the contradicting number already in scope three lines above. Now: a lossy parse → WARNING; a shortfall **with** `follow_view_more=True` → WARNING (the ViewMore page should have returned the section in full, so nothing is left to explain the gap); a shortfall with `follow_view_more=False` → INFO (that truncation is what the caller asked for — warning about it trains people to ignore warnings). Note `_CATEGORY_FOR_VIEWMORE_SLUG` maps only `fs-norm`/`fs-kf`/`fs-mda`, so a 56-1/56-2 section with more than ~10 records would truncate even with `follow_view_more=True`; no capture shows one, and this WARNING is what would surface it.
+- **The SEC listing summary log is derived from the accounting, and its LEVEL is load-bearing:** `Listed 0 SEC document(s)` used to be the same sentence at the same level as a complete success, with the contradicting number already in scope three lines above. Now: a lossy parse → WARNING; a shortfall **with** `follow_view_more=True` → WARNING (the ViewMore page should have returned the section in full, so nothing is left to explain the gap); a shortfall with `follow_view_more=False` → INFO (that truncation is what the caller asked for — warning about it trains people to ignore warnings). This WARNING is not theoretical: it is what surfaced the unmapped `fs-r561`/`fs-r562` slugs (closed in 0.22.2, see above) — a downstream consumer read the shortfall out of `completeness()` and reported it.
 - **`Stock.get_latest_price()` is DR-aware by default:** for DRs it returns a `DrIndicativeQuotation` (TradingView indicative price; `volume`/`change` are `None` — it's a fair value, not a SET trade) and falls back to SET chart data on ANY TradingView failure. The switch only applies when `as_of is None` (TV can't answer historical instants); opt out per-call with `prefer_dr_indicative=False`. DR-ness is detected by one cached DR-profile probe per `Stock` instance (a 404 marks non-DR permanently for that instance; transient errors are never cached). Indicative vs SET-close divergence is EXPECTED (probe: 5.94 indicative vs 5.75 SET close after a US-session move) — not a bug.
 
 ---

{settfex-0.22.0 → settfex-0.22.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: settfex
-Version: 0.22.0
+Version: 0.22.2
 Summary: Async Python library for fetching real-time and historical data from the Stock Exchange of Thailand (SET) and Thailand Futures Exchange (TFEX)
 Project-URL: Homepage, https://github.com/lumduan/settfex
 Project-URL: Documentation, https://github.com/lumduan/settfex#readme

{settfex-0.22.0 → settfex-0.22.2}/docs/settfex/services/sec/financial_report.md

@@ -229,6 +229,27 @@ files.requested                            # unique files attempted (duplicates
 Before that, a failed item was logged and dropped from the result, so a partial batch was shaped
 exactly like a complete one.
 
+> ⚠️ **Read `.failed` from the object `download_all` returned.** `DownloadResult` is a `list`
+> subclass, so the failure report lives on the instance — and **every operation that derives a new
+> list drops it**: slicing, `sorted()`, `list()`, concatenation and comprehensions all return a
+> plain `list` with no `.failed`, `.requested` or `.is_complete`, and nothing warns.
+> `copy.copy()` is the one exception. So `for f in sorted(files)` silently gives up the report:
+> check `files.is_complete` first, then sort.
+
+### Old-format filings live on another host, behind an indirection
+
+Some pre-2014 financial statements link to `capital.sec.or.th` instead of IDISC. That URL answers a
+2 KB HTML page whose entire body is a JavaScript redirect to a **server-minted** zip, so
+`download()` resolves it (one extra request) and returns the real archive. Two consequences:
+
+* **The minted target is single-use** — the same filing mints `/tmp/0653XP.zip`, `/tmp/0653sD.zip`,
+  `/tmp/0653V1.zip` on successive requests. `file_url` therefore stays the stable indirection URL;
+  never store the `/tmp/…` one.
+* **The container sha256 is not an identity for these.** Re-fetching the same filing legitimately
+  yields different container bytes — 12 bytes of 334,498, the Info-ZIP pack timestamp — while every
+  member payload is byte-identical. **Judge integrity per member, not by hashing the zip.** An
+  archive that keys on the container hash will see a re-fetch as a new or corrupted object.
+
 ### Tuning downloads (timeout & concurrency)
 
 Form 56‑1/56‑2 "One Reports" are large (**15–25 MB**). Downloads default to a **180 s** per-file
@@ -351,7 +372,9 @@ file = await download_sec_document(doc)  # dead link -> FetchError ("soft 404")
 POST /public/idisc/api/company/valuebyuniqueId      {"lang","content"} -> [{"Text","Value","Flag"}]
 GET  /public/idisc/{lang}/FinancialReport/{TYPE}     -> page + __VIEWSTATE tokens
 POST /public/idisc/{lang}/FinancialReport/{TYPE}     form postback -> result HTML
-GET  /public/idisc/{lang}/ViewMore/{slug}?...        -> complete section (fs-norm/fs-kf/fs-mda)
+GET  /public/idisc/{lang}/ViewMore/{slug}?...        -> complete section
+     slug in fs-norm | fs-kf | fs-mda | fs-r561 | fs-r562   ('fs-' is the site's prefix,
+                                                             not a category hint)
 GET  /public/idisc/Download?FILEID=<path>            -> zip/pdf bytes
 GET  /ipos/Common/IPOSGetFile.aspx?id=<id>&sq=0&v=10 -> zip bytes
 GET  /public/idisc/Views/FinancialStatementDownload?query=<blob> -> zip bytes (recent KFR rows)

{settfex-0.22.0 → settfex-0.22.2}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "settfex"
-version = "0.22.0"
+version = "0.22.2"
 description = "Async Python library for fetching real-time and historical data from the Stock Exchange of Thailand (SET) and Thailand Futures Exchange (TFEX)"
 readme = "README.md"
 requires-python = ">=3.11"

{settfex-0.22.0 → settfex-0.22.2}/settfex/__init__.py

@@ -35,7 +35,7 @@ Usage:
     >>> asyncio.run(main())
 """
 
-__version__ = "0.22.0"
+__version__ = "0.22.2"
 __author__ = "batt"
 __license__ = "MIT"
 

{settfex-0.22.0 → settfex-0.22.2}/settfex/services/sec/constants.py

@@ -44,11 +44,19 @@ SEC_REPORT_TYPE_FORM_56_1 = "R561"
 SEC_REPORT_TYPE_FORM_56_2 = "R562"
 SEC_REPORT_TYPE_KEY_FINANCIAL_RATIO = "KFR"
 
-# ViewMore slugs (seen live) for the sections a FS search can truncate.
+# ViewMore slugs (seen live) for every section that can truncate behind "display all results".
+# Note the `fs-` prefix is the site's own and does not mean "financial statement": `fs-r561` and
+# `fs-r562` are served by the 56-1 / 56-2 searches. Live-probed 2026-09-20.
+#
+# This is a reference constant; the mapping the listing service actually consults is
+# `_CATEGORY_FOR_VIEWMORE_SLUG` in `financial_report.py`. Keep the two in step -- they state the
+# same fact, and this one has no caller to notice when it goes stale.
 SEC_VIEWMORE_SLUGS: dict[str, str] = {
     "financial_statement": "fs-norm",
     "key_financial_ratio": "fs-kf",
     "mda": "fs-mda",
+    "form_56_1": "fs-r561",
+    "form_56_2": "fs-r562",
 }
 
 # The SEC search form's field names (ASP.NET control ids).

{settfex-0.22.0 → settfex-0.22.2}/settfex/services/sec/download.py

@@ -13,12 +13,12 @@ import re
 from collections.abc import Iterable, Sequence
 from pathlib import Path
 from typing import Any
-from urllib.parse import unquote, urlparse
+from urllib.parse import unquote, urljoin, urlparse
 
 from loguru import logger
 from pydantic import BaseModel, ConfigDict, Field
 
-from settfex.exceptions import FetchError
+from settfex.exceptions import FetchError, ParseError
 from settfex.services.sec.constants import (
     SEC_BASE_URL,
     SEC_DOWNLOAD_ENDPOINT,
@@ -26,7 +26,7 @@ from settfex.services.sec.constants import (
     SEC_REFERER,
 )
 from settfex.services.sec.financial_report import SecDocument
-from settfex.services.sec.utils import build_sec_headers
+from settfex.services.sec.utils import _CAPITAL_HOST, build_sec_headers
 from settfex.utils.data_fetcher import AsyncDataFetcher, FetcherConfig
 
 
@@ -95,6 +95,21 @@ class DownloadResult(list[DownloadedFile]):
 
     Same shape as :class:`~settfex.services.sec.financial_report.SecDocumentList`, for the same
     reason: the extra information rides along instead of breaking every existing caller.
+
+    .. warning::
+       **Deriving a new list drops the failure report** (issue #134). ``.failed``, ``.requested``
+       and ``.is_complete`` live on the instance, so every operation that *builds a new list* —
+       slicing, ``sorted()``, ``list()``, concatenation, a comprehension — returns a plain
+       ``list`` without them, and nothing warns::
+
+           for f in sorted(result, key=lambda f: f.size):   # <- .failed is gone from here
+               ...
+           if not result.is_complete:                       # <- read it from the ORIGINAL
+               print([f.target for f in result.failed])
+
+       ``copy.copy(result)`` is the exception and does preserve them. Read the failure report from
+       the object ``download_all`` returned, before deriving anything from it. A redesign that
+       removes this edge is a breaking change and is tracked for a later release.
     """
 
     def __init__(
@@ -165,6 +180,50 @@ def _effective_download_config(config: FetcherConfig | None, timeout: int | None
     return base.model_copy(update={"use_session": False})
 
 
+# The capital.sec.or.th shape answers HTML whose entire body is a JavaScript redirect:
+#
+#     <script language = 'javascript' >
+#             document.location = '/tmp/0653XP.zip';
+#     </script>
+#
+# Strict on purpose, but in two parts rather than one. The pattern pins the assignment and the
+# `.zip` suffix; the HOST and the path are then checked against the page's own origin after the
+# join. Doing it that way keeps the host check LIVE: a pattern that also required the `/tmp/`
+# prefix could never match an absolute URL, so the off-host branch below would have been dead
+# code that looked like a safeguard. Every capture so far is a bare `/tmp/<id>.zip` path.
+_CAPITAL_REDIRECT = re.compile(r"""document\.location\s*=\s*['"]([^'"]+\.zip)['"]""", re.IGNORECASE)
+
+
+def _capital_redirect_target(body: bytes, url: str) -> str:
+    """Resolve the JavaScript indirection to its absolute target, or raise.
+
+    The target is **minted per request** — the same filing yielded ``/tmp/0653XP.zip``,
+    ``/tmp/0653sD.zip`` and ``/tmp/0653V1.zip`` across three captures — so it is resolved at
+    download time and never stored on the listing.
+
+    The page is TIS-620, not UTF-8, so the body is decoded permissively; the line this reads is
+    pure ASCII either way.
+    """
+    text = body.decode("utf-8", "replace")
+    match = _CAPITAL_REDIRECT.search(text)
+    if match is None:
+        raise ParseError(
+            f"The old-format filing page at {url} carried no JavaScript redirect to a zip. Its "
+            f"shape may have changed; raised rather than returning the 2 KB HTML page as if it "
+            f"were the document.",
+            url=url,
+        )
+    target: str = urljoin(url, match.group(1))
+    parsed = urlparse(target)
+    if parsed.hostname != urlparse(url).hostname:
+        raise FetchError(
+            f"The redirect on {url} points off-host, to {target} — refusing to follow it."
+        )
+    if not parsed.path.lower().endswith(".zip"):
+        raise ParseError(f"The redirect on {url} points at {target}, which is not a zip.", url=url)
+    return target
+
+
 class DocumentDownloadService:
     """Download SEC documents to bytes (and optionally disk). Stateless host — no SessionManager."""
 
@@ -212,6 +271,13 @@ class DocumentDownloadService:
         fetcher = fetcher or AsyncDataFetcher(config=self.config)
         try:
             resp = await fetcher.fetch(url, headers=headers, decode_text=False)
+            if resp.status_code == 200 and _CAPITAL_HOST in (urlparse(url).hostname or ""):
+                # The old-format shape is an indirection, so this costs one extra request. It is
+                # done HERE, inside the fetcher's lifetime, rather than on the listing: the target
+                # is minted per request and would be stale by the time anyone used it.
+                url = _capital_redirect_target(resp.content, url)
+                logger.debug(f"Resolved the old-format indirection to {url}")
+                resp = await fetcher.fetch(url, headers=headers, decode_text=False)
         finally:
             if owns_fetcher:
                 await fetcher.__aexit__(None, None, None)

{settfex-0.22.0 → settfex-0.22.2}/settfex/services/sec/financial_report.py

@@ -20,7 +20,12 @@ from urllib.parse import urljoin
 from loguru import logger
 from pydantic import BaseModel, ConfigDict, Field, computed_field
 
-from settfex.exceptions import IncompleteListingError, InvalidDateError, ParseError
+from settfex.exceptions import (
+    FetchError,
+    IncompleteListingError,
+    InvalidDateError,
+    ParseError,
+)
 from settfex.services.sec.company import resolve_company
 from settfex.services.sec.constants import (
     SEC_BASE_URL,
@@ -581,6 +586,88 @@ def _drop_reason(row: ReportRow, *, reported: int | None) -> _DropReason:
     return "no_link"
 
 
+# A listing response is an ASP.NET result panel: a sequence of `card-heading` divs, each with a
+# `<table>`. An error page has neither. Measured over every page this repo holds -- 8 committed
+# fixtures, 3 listing test constants and a real 104 KB ViewMore capture -- every genuine listing
+# body carries at least one of each, INCLUDING the one whose section is genuinely empty (MOTHER's
+# Key Financial Ratio, which reports 0 records). The two error-shaped bodies in the evidence, a
+# real HTTP 505 page and a capital.sec.or.th indirection page, carry neither.
+#
+# NOTE it deliberately does NOT key on `ctl00_CPH_pnlControl` alone: the real ViewMore capture has
+# that id and the trimmed `FS_VIEWMORE_HTML` test constant does not, so a panel-id rule would call
+# a legitimate page an error.
+_LISTING_MARKERS = ("card-heading", "<table")
+
+
+def _require_ok(status_code: int, url: str, code: str) -> None:
+    """Raise ``FetchError`` unless the listing response is 2xx (issue #131).
+
+    The listing path used to read ``response.text`` without ever looking at the status, so an
+    error page parsed to zero rows and was returned as an empty list -- indistinguishable from an
+    issuer with no filings, and worse than a parse bug because it is intermittent.
+
+    Raised as a plain :class:`FetchError`, **not** through ``raise_for_status``: that helper maps
+    404 to ``SymbolNotFoundError`` and would consult the symbol suggester, which is wrong here.
+    The unique_id was already resolved, so a 404 on this endpoint means the route or the host is
+    wrong, not that an issuer does not exist -- the same reasoning that gives the DR-profile and
+    analyst-consensus endpoints their own handling (see CLAUDE.md's Known Gotchas).
+    """
+    if 200 <= status_code < 300:
+        return
+    message = (
+        f"SEC listing request failed: HTTP {status_code} for report code {code!r} "
+        f"at {url}. The response was not parsed — an error page yields zero rows, which is "
+        f"indistinguishable from an issuer that filed nothing."
+    )
+    logger.error(message)
+    raise FetchError(message, status_code=status_code)
+
+
+def _require_listing_page(html: str, url: str, code: str) -> None:
+    """Raise ``ParseError`` when a 2xx body is not a listing page at all (issue #131).
+
+    The status check above catches the evidenced case (a real HTTP 505). This is the companion for
+    an error or interstitial page served with HTTP 200, which no status check can see. It is
+    deliberately the weakest rule that separates the two populations, so that a genuinely empty
+    listing -- the thing that must never raise -- cannot trip it. See :data:`_LISTING_MARKERS`.
+    """
+    lowered = html.lower()
+    if any(marker in lowered for marker in _LISTING_MARKERS):
+        return
+    message = (
+        f"SEC listing response for report code {code!r} at {url} carries no result table and no "
+        f"section heading, so it is not a listing page (HTTP 200 error or interstitial page?). "
+        f"Raised rather than returned as an empty list, which would be indistinguishable from an "
+        f"issuer that filed nothing."
+    )
+    logger.error(message)
+    raise ParseError(message, url=url, rows_parsed=0)
+
+
+def _view_more_is_usable(status_code: int, html: str, url: str, code: str) -> bool:
+    """Is this "display all results" page fit to replace its section's inline rows?
+
+    Returns False (with a WARNING) instead of raising, so a broken ViewMore page costs the caller
+    the *completeness* of one section rather than the whole listing. See the call site for why.
+    """
+    if not 200 <= status_code < 300:
+        logger.warning(
+            f"The 'display all results' page for report code {code!r} at {url} answered HTTP "
+            f"{status_code}. Keeping the truncated inline rows for that section instead of "
+            f"replacing them; `completeness()` will show the shortfall."
+        )
+        return False
+    lowered = html.lower()
+    if not any(marker in lowered for marker in _LISTING_MARKERS):
+        logger.warning(
+            f"The 'display all results' page for report code {code!r} at {url} is not a listing "
+            f"page (no result table, no section heading). Keeping the truncated inline rows for "
+            f"that section; `completeness()` will show the shortfall."
+        )
+        return False
+    return True
+
+
 # ViewMore slug -> the category whose complete list that page holds.
 @dataclass
 class _MapResult:
@@ -640,6 +727,19 @@ def _map_rows(
     single "FS" search returns three sections, so without it a loss in a section that was filtered
     out on request would raise for a caller who never wanted it.
     """
+    if not rows:
+        # Reachable only for a body that looked like a listing and yielded no data row at all.
+        # Within this package that cannot be a real listing: every captured page — including the
+        # ones whose sections are empty — carries at least the revision-tracking placeholder row.
+        # Returning [] here is the exact silent loss of issue #131, one layer below the transport.
+        message = (
+            f"No result rows were parsed from {source}. A SEC listing always carries at least one "
+            f"row, including the placeholder an empty section serves, so this body is not a "
+            f"listing — raised rather than returned as an empty list."
+        )
+        logger.error(message)
+        raise ParseError(message, url=source, rows_parsed=0)
+
     result = _MapResult()
     accounting = result.accounting
     accounting.rows = len(rows)
@@ -780,10 +880,21 @@ def _escalate_losses(
         )
 
 
+# Every slug the site serves a "display all results" page under. The `fs-` prefix is the site's,
+# not a category hint: `fs-r561` and `fs-r562` belong to the 56-1 / 56-2 searches, not to the FS
+# one. Live-probed 2026-09-20 across CPALL/PTT/SCB in both languages.
+#
+# The last two were MISSING until 0.22.2, so a 56-1 or 56-2 section over the ~10-row inline cap
+# was truncated even with `follow_view_more=True`. 0.22.0's notes predicted exactly this and said
+# the shortfall WARNING would surface it; it did, from a downstream consumer reading
+# `completeness()`. Scale, with a 2000-2026 window: CPALL Thai 56-1 served 11 of 23, PTT Thai 56-1
+# 11 of 25, PTT 56-2 10 of 15 in BOTH languages -- so this was never a Thai-only gap.
 _CATEGORY_FOR_VIEWMORE_SLUG: dict[str, DocumentCategory] = {
     "fs-norm": DocumentCategory.FINANCIAL_STATEMENT,
     "fs-kf": DocumentCategory.KEY_FINANCIAL_RATIO,
     "fs-mda": DocumentCategory.MDA,
+    "fs-r561": DocumentCategory.FORM_56_1,
+    "fs-r562": DocumentCategory.FORM_56_2,
 }
 
 _VIEWMORE_HREF = re.compile(r'href="([^"]*?/ViewMore/([a-z0-9-]+)[^"]*)"', re.IGNORECASE)
@@ -987,10 +1098,12 @@ class FinancialReportService:
             f"{SEC_BASE_URL}{SEC_FINANCIAL_REPORT_ENDPOINT.format(lang=lang, report_type=code)}"
         )
         get_resp = await fetcher.fetch(report_url, headers=build_sec_headers(referer=SEC_REFERER))
+        # This leg was already guarded by accident: an error page carries no __VIEWSTATE, so the
+        # check below fired. It reported the wrong cause, though — "the page structure may have
+        # changed" for what was really an HTTP 500 — so the status is now read first.
+        _require_ok(get_resp.status_code, report_url, code)
         tokens = extract_aspnet_tokens(get_resp.text)
         if not tokens.get("__VIEWSTATE"):
-            from settfex.exceptions import FetchError
-
             raise FetchError(
                 "SEC search page returned no __VIEWSTATE token — the page structure may have "
                 "changed, or the request was blocked."
@@ -1011,6 +1124,7 @@ class FinancialReportService:
             method="POST",
             data=form,
         )
+        _require_ok(post_resp.status_code, report_url, code)
         return post_resp.text, report_url
 
     async def _search_code(
@@ -1030,6 +1144,7 @@ class FinancialReportService:
             fetcher, code, unique_id, company_name, date_from, date_to, lang
         )
         wanted_values = {c.value for c in wanted}
+        _require_listing_page(html, url, code)
         mapped = _map_rows(
             parse_report_tables(html),
             unique_id,
@@ -1068,6 +1183,22 @@ class FinancialReportService:
                 )
             )
             for (cat, url), page in zip(vm_targets, pages, strict=True):
+                # A ViewMore page holds the COMPLETE list for its section and REPLACES the inline
+                # rows, so an unusable one here does not merely add nothing -- on 0.22.0 it
+                # silently deleted the whole section, the inline rows included.
+                #
+                # It is DEGRADED rather than raised, which is the same severity ladder the rest of
+                # this module uses: a failed ViewMore is a PARTIAL loss, because the truncated
+                # inline rows are still a real answer -- exactly the one `follow_view_more=False`
+                # would have given. Skipping the replacement keeps them, and the shortfall then
+                # shows up on its own through `completeness()` and the WARNING in
+                # `_log_listing_summary`, which already treats "short even after following the
+                # view-more pages" as the case worth warning about.
+                #
+                # This is not hypothetical: on 2026-09-20 the live Thai `fs-kf` page answered
+                # HTTP 500 on every attempt while the other five slug/language pairs answered 200.
+                if not _view_more_is_usable(page.status_code, page.text, url, code):
+                    continue
                 vm = _map_rows(
                     parse_report_tables(page.text),
                     unique_id,

{settfex-0.22.0 → settfex-0.22.2}/settfex/services/sec/utils.py

@@ -273,6 +273,14 @@ def parse_int(value: str | None) -> int | None:
     return int(match.group(0)) if match else None
 
 
+# The fifth download shape lives on a different host and is an indirection, not a file. Kept
+# module-private on purpose: the other four shapes have public `SEC_*_ENDPOINT` constants in
+# `constants.py`, but adding a fifth public symbol would put this fix outside a patch release.
+# Promoting it is a recorded 0.23.0 consistency item.
+_CAPITAL_HOST = "capital.sec.or.th"
+_CAPITAL_ZIP_PATH = "get_zip_all_public_page.php"
+
+
 def classify_download_href(href: str | None) -> tuple[str, str | None, str | None]:
     """
     Resolve a row href to (absolute_url, file_id, file_kind).
@@ -283,6 +291,13 @@ def classify_download_href(href: str | None) -> tuple[str, str | None, str | Non
       file_id="fsdl:<blob>", kind=None. The blob is opaque, so nothing can be derived from it.
     - Scanned originals: ``/public/idisc/views/viewdoc?…&TransId=<id>&FileSeq=<n>`` →
       file_id="viewdoc:<id>-<n>", kind=None (it answers a TIFF, but the URL does not say so).
+    - Old-format filings on a DIFFERENT host: ``capital.sec.or.th/…/get_zip_all_public_page.php``
+      → file_id="capfin:<comp_id>-<year>-<period>-<lang>", kind=None. This one is an
+      **indirection**: the URL answers HTML whose whole body is a JavaScript redirect to a
+      server-minted zip, so the download path resolves it (see
+      :meth:`~settfex.services.sec.download.DocumentDownloadService.download`). The returned
+      ``file_url`` is the STABLE indirection URL, never the minted target — that target is
+      per-request and must not be cached.
     Relative hrefs are resolved against the SEC base URL.
 
     Anything else returns ``("", None, None)`` and the row produces no document — which is the
@@ -310,5 +325,16 @@ def classify_download_href(href: str | None) -> tuple[str, str | None, str | Non
         return absolute, f"fsdl:{query['query'][0]}", None
     if parsed.path.lower().endswith("viewdoc") and "TransId" in query:
         return absolute, f"viewdoc:{query['TransId'][0]}-{query.get('FileSeq', ['1'])[0]}", None
+    if parsed.path.lower().endswith(_CAPITAL_ZIP_PATH) and "comp_id" in query:
+        # `comp_id` is a DIFFERENT id space from the IDISC `set_id` in the same query — they are
+        # not interchangeable, so the file_id is built from comp_id and never from set_id. `lang`
+        # is load-bearing: the Thai request mints a different archive for the same filing.
+        parts = (
+            query["comp_id"][0],
+            query.get("year", ["?"])[0],
+            query.get("period", ["?"])[0],
+            query.get("lang", ["?"])[0],
+        )
+        return absolute, "capfin:" + "-".join(parts), None
     # Not a recognized download link (e.g. a "display all results" ViewMore link) — not a doc.
     return "", None, None

{settfex-0.22.0 → settfex-0.22.2}/tests/golden/api_surface.json

@@ -1063,6 +1063,7 @@
         "FailedDownload",
         "FetchError",
         "FetcherConfig",
+        "ParseError",
         "SEC_BASE_URL",
         "SEC_DOWNLOAD_ENDPOINT",
         "SEC_FILE_NOT_FOUND_MARKER",
@@ -1711,6 +1712,7 @@
         "AsyncDataFetcher",
         "CATEGORY_TO_REPORT_TYPE",
         "DocumentCategory",
+        "FetchError",
         "FetcherConfig",
         "FinancialReportService",
         "IncompleteListingError",