settfex 0.22.1__tar.gz → 0.22.2__tar.gz

{settfex-0.22.1 → settfex-0.22.2}/CHANGELOG.md

@@ -7,6 +7,38 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+## [0.22.2] - 2026-09-20
+
+### Fixed
+
+- **56-1 and 56-2 sections truncated even with `follow_view_more=True`, because their
+  "display all results" slugs were unmapped.** `_CATEGORY_FOR_VIEWMORE_SLUG` covered `fs-norm`,
+  `fs-kf` and `fs-mda` — the three sections a single `FS` search returns — and nothing else. A
+  56-1 or 56-2 section over the site's ~10-row inline cap therefore linked to its complete list
+  through a slug the map did not recognise, and the link was skipped.
+
+  0.22.0's own notes predicted this and said the shortfall WARNING would be what surfaced it. It
+  was, from the other side of the API: a consumer read `completeness()` and reported 10 of 12 on a
+  Thai 56-1. Both slugs were then **live-probed rather than guessed** (CPALL / PTT / SCB × th / en,
+  2026-09-20), which is how the second one was found at all.
+
+  The gap was larger than first reported, and not Thai-only. Over a 2000–2026 window:
+
+  | Listing | Before | After |
+  |---|---|---|
+  | CPALL 56-1 `th` | 11 of 23 | **23 of 23** |
+  | PTT 56-1 `th` | 11 of 25 | **25 of 25** |
+  | PTT 56-2 `th` | 10 of 15 | **15 of 15** |
+  | PTT 56-2 `en` | 10 of 15 | **15 of 15** |
+
+  Note the `fs-` prefix is the site's own and is **not** a category hint — `fs-r561` is served by
+  the 56-1 search, not the FS one — so a mapping keyed on the prefix would be wrong.
+
+  No API change: the mapping is module-private, and the public `SEC_VIEWMORE_SLUGS` constant (which
+  has no caller, and so would not notice itself going stale) is updated to match, with a test
+  pinning the two equal. The generated API surface is byte-identical to 0.22.1.
+
+
 ## [0.22.1] - 2026-09-20
 
 ### Fixed

{settfex-0.22.1 → settfex-0.22.2}/CLAUDE.md

@@ -414,7 +414,7 @@ See [`CHANGELOG.md`](CHANGELOG.md) for the full, versioned release history — t
 - **SEC serves FOUR download-URL shapes; two of them were silently dropping live filings:** `classify_download_href` knew `Download?FILEID=<path>` and `ipos/Common/IPOSGetFile.aspx?id=`. **Key Financial Ratio** rows use two more, and both fell through to "not a download link" and made the row vanish — live-probed 2026-09-20 on CPALL, both real downloads: (1) recent rows (2026 Q1/Q2) link through `/public/idisc/Views/FinancialStatementDownload?query=<opaque base64>`, answering `application/zip` — a **134 KB zip of the real PDFs**; (2) older rows (2018, 2019) link through `/public/idisc/views/viewdoc?…&TransId=<id>&FileSeq=<n>`, answering `application/.tif` — a **36 KB scan of the original**. Mapped now as `file_id="fsdl:<blob>"` / `"viewdoc:<id>-<n>"`, both `file_kind=None` (neither URL states a type; `Content-Disposition` names the file at download time). After the fix, CPALL 2015-2026 returns **177 documents with `completeness()` exact in all five categories** — `key_financial_ratio` was `(13, 15)`. ⚠️ **Three things this should teach the next reader.** (a) **It falsifies the reporting issue's own finding:** its corpus scan of 39 captured pages found *zero* natural instances of a dropped row and stated P1 "has not been observed in the wild"; the first two live runs of the new accounting found four. A captured corpus can only show what was captured. (b) **The window is load-bearing:** a narrow recent window passed clean while the 2018/2019 rows were being dropped — **old filings use old URL shapes**, so a probe that only checks recent data cannot see them. (c) **Do NOT generalise the allowlist** into "any link under /public/idisc/ with a query is a download": the "display all results" ViewMore link has exactly that shape, and would become a phantom filing in every truncated section. Expect the list to grow; `ListingAccounting.no_link` is the mechanism that makes the next one visible, and a test pins the ViewMore link as not-a-download. Note `_fallback_filename` keys on the **colon**, not on an `ipos:` prefix: a real FILEID path never contains one, so every synthetic id is safely excluded from being sliced like a path.
 - **The SEC LISTING path checks its HTTP status now — and a ViewMore failure DEGRADES rather than raising (issue #131):** `_run_search` used to return `post_resp.text` without ever reading `status_code`, so an error page parsed to zero rows and came back as `[]` with `has_losses == False` — indistinguishable from an issuer that filed nothing, and intermittent, so a backfill recorded the window as covered. Evidenced by a real **HTTP 505** from the listing host (2026-09-20). Two rules: non-2xx → `FetchError`; a 2xx body with **no result table and no section heading** → `ParseError`. The content rule is deliberately the weakest one that separates the populations, because the case it must never trip is a **genuinely empty listing** — a test pins it against every page this repo holds. It does **not** key on `ctl00_CPH_pnlControl`: the real ViewMore capture has that id and the trimmed `FS_VIEWMORE_HTML` constant does not, so a panel-id rule would call a legitimate page an error. ⚠️ **The severity ladder differs by leg, on purpose.** The POST leg raises (a total loss — nothing to fall back on). A **ViewMore** page is different: it *replaces* its section's inline rows, so 0.22.0's unconditional replacement meant a failing page **deleted the whole section, inline rows included, in silence**. A broken ViewMore is a *partial* loss, so it now keeps the inline rows, warns, and lets `completeness()` show the shortfall. `_map_rows` additionally refuses a **zero-row** input: within this package that can only be a non-listing body, since every captured page carries at least the placeholder row an empty section serves.
 - **The live Thai `fs-kf` ViewMore page has been answering HTTP 500 since at least 2026-09-20 — an UPSTREAM bug, and it is why the degrade path exists:** probed 3× each, `th/fs-kf` returned 500 every time while `th/fs-norm`, `th/fs-mda` and all three `en/*` returned 200. On 0.22.0 this silently removed CPALL's entire Thai Key Financial Ratio section; now the listing returns 177 documents with `key_financial_ratio: (10, 15)` stated in `completeness()` plus two WARNINGs. **If you see a Thai KFR shortfall, check whether SEC has fixed their page before looking in settfex.**
-- **`fs-r561` is an UNMAPPED ViewMore slug — a known, visible 56-1 shortfall (not yet fixed):** `_CATEGORY_FOR_VIEWMORE_SLUG` maps only `fs-norm`/`fs-kf`/`fs-mda`, so a 56-1/56-2 section with more than ~10 records truncates even with `follow_view_more=True`. 0.22.0's CLAUDE.md predicted this and said the shortfall WARNING is what would surface it — **it did**: CPALL's Thai 56-1 reports 12 records and yields 10, and the page carries `ViewMore/fs-r561`, which serves exactly those 12 rows (probed 2026-09-20; the English page has only 6 records so it carries no ViewMore link at all). This is **reported, not silent** — `completeness()` shows `(10, 12)` — so it is a data gap, not a silent-loss bug. Closing it is one entry in the slug map plus whatever `fs-r562` turns out to be.
+- **~~`fs-r561` is an UNMAPPED ViewMore slug~~ — CLOSED in 0.22.2, and `fs-r562` with it:** `_CATEGORY_FOR_VIEWMORE_SLUG` used to map only `fs-norm`/`fs-kf`/`fs-mda`, the three sections one `FS` search returns, so a **56-1 or 56-2** section over the site's ~10-row inline cap truncated **even with `follow_view_more=True`**. 0.22.0's notes predicted it and said the shortfall WARNING would surface it — it did, reported by a downstream consumer reading `completeness()`. Both slugs were **live-probed, never guessed** (2026-09-20, CPALL/PTT/SCB × th/en), and the scale was larger than first reported: over a 2000-2026 window CPALL th 56-1 served **11 of 23**, PTT th 56-1 **11 of 25**, PTT 56-2 **10 of 15 in BOTH languages** — so it was never a Thai-only gap. Now 23/23, 25/25, 15/15, 15/15. ⚠️ **The `fs-` prefix is the site's own and is NOT a category hint** — `fs-r561` is served by the *56-1* search, not the FS one; a mapping keyed on the prefix would be wrong. Two surfaces state this fact: `_CATEGORY_FOR_VIEWMORE_SLUG` (used) and the public `SEC_VIEWMORE_SLUGS` in `constants.py` (**used nowhere**, so nothing would notice it going stale) — a test pins them equal.
 - **SEC's old-format filings are on a FIFTH host and behind a JavaScript indirection (issue #133):** `capital.sec.or.th/…/get_zip_all_public_page.php` carries pre-2014 `.DOC`/`.XLS` filings. The URL answers **HTTP 200 with a 2 KB HTML page whose entire body is `document.location = '/tmp/<id>.zip'`** — so recognising the link is necessary but not sufficient, and `download()` resolves the redirect at download time (one extra request). Three properties the code depends on: the target is **minted per request** (`XP`/`sD`/`V1` for the same filing across three captures), so `file_url` stays the stable indirection URL and the `/tmp/…` one is never stored; it is **per-language** (the Thai request mints a different archive, so `lang` is part of the `capfin:<comp_id>-<year>-<period>-<lang>` id); and `comp_id` is a **different id space from `set_id`** in the same query — they are not interchangeable. The redirect pattern accepts an absolute URL on purpose, so the same-host check is live rather than dead code.
 - **These old-format zips are MEMBER-STABLE, not byte-stable — do not key an archive on the container hash:** re-minting the same filing produced a different container sha256 while every member payload stayed byte-identical. Exactly **12 bytes of 334,498** differ, in four 3-byte runs, one per member: the Info-ZIP extended-timestamp (`UT`) extra field, i.e. the pack time. **Judge integrity per member.** A consumer that hashes the container will see a legitimate re-fetch as a new — or corrupted — object. (Also observed: one sample's embedded pack time *predated* the request that minted it, suggesting the server sometimes hands back a previously generated archive. INFERRED from two samples.)
 - **`DownloadResult` loses `.failed` the moment you derive a new list (issue #134, documented not fixed):** `.failed`, `.requested` and `.is_complete` are instance attributes on a `list` subclass, so slicing, `sorted()`, `list()`, concatenation and comprehensions all return a plain `list` without them — 6 of 7 ordinary operations, `copy.copy()` being the exception. `for f in sorted(result)` therefore quietly gives up the failure report #128 added. **Read `.failed` from the object `download_all` returned, before deriving anything from it.** `SecDocumentList` has the identical hazard for `.accounting`/`.reported_counts`. Removing the edge is a breaking change and is tracked for a later release.
@@ -423,7 +423,7 @@ See [`CHANGELOG.md`](CHANGELOG.md) for the full, versioned release history — t
 - **`IncompleteListingError` fires only on a TOTAL loss:** rows classified fine and *every* usable one was dropped for want of a download link. A **partial** loss warns and lands on `accounting.no_link` instead — raising there would turn one bad row in a 200-row listing into no listing at all, and unlike a total loss a partial one is visible in the return value. Same severity ladder as the 0.21.0 `ParseError` for unrecognised headings, and it is a `ParseError` subclass so `except ParseError`/`except FetchError` keep working. The escalation is scoped to the **requested** categories (a single `FS` search returns three sections) — the same principle PR #125 applied to `completeness()`.
 - **An unmapped COLUMN is reported, never raised — and the ignore-list is 5 entries, not 13:** the issue that reported this measured "100% of sections carry an unmapped header", but that count includes the revision-tracking sections, **whose rows are skipped before a single cell is read**. Restricted to sections actually read, the whole corpus serves exactly six unmapped headers, five of which have no model field (`Details`, `รายละเอียด`, `Link`, `Time`, `เวลา` → `_IGNORED_HEADERS`) and the sixth was the real loss (`วันที่ได้รับข้อมูล`, now mapped). So the corpus is 100% clean and the first header this check ever names will be a genuine site change. It **warns**, because an unmodelled column is "the site has more than we model", not lost data — raising would turn a cosmetic change into an outage. Do **not** add the revision-section headers (`Order Date`, `Company Name`, `ชื่อบริษัท`, …) to the ignore-list: they never reach the check, and listing them would make it look broader than it is.
 - **`download_all` returns a `DownloadResult`, not a bare list — a partial batch used to be shaped like a complete one:** with `continue_on_error=True` (still the default) a failure became `None` and was filtered out, so the difference between "I downloaded the filings" and "I downloaded some of the filings" existed only in a log line. `DownloadResult` **is** a `list[DownloadedFile]` (same trick as `SecDocumentList`), so `len()`/iteration/indexing/pass-through are unchanged; it additionally carries `.failed` (target, source document, error, error type), `.requested` and `.is_complete`. `continue_on_error=False` still propagates the first failure.
-- **The SEC listing summary log is derived from the accounting, and its LEVEL is load-bearing:** `Listed 0 SEC document(s)` used to be the same sentence at the same level as a complete success, with the contradicting number already in scope three lines above. Now: a lossy parse → WARNING; a shortfall **with** `follow_view_more=True` → WARNING (the ViewMore page should have returned the section in full, so nothing is left to explain the gap); a shortfall with `follow_view_more=False` → INFO (that truncation is what the caller asked for — warning about it trains people to ignore warnings). Note `_CATEGORY_FOR_VIEWMORE_SLUG` maps only `fs-norm`/`fs-kf`/`fs-mda`, so a 56-1/56-2 section with more than ~10 records would truncate even with `follow_view_more=True`; no capture shows one, and this WARNING is what would surface it.
+- **The SEC listing summary log is derived from the accounting, and its LEVEL is load-bearing:** `Listed 0 SEC document(s)` used to be the same sentence at the same level as a complete success, with the contradicting number already in scope three lines above. Now: a lossy parse → WARNING; a shortfall **with** `follow_view_more=True` → WARNING (the ViewMore page should have returned the section in full, so nothing is left to explain the gap); a shortfall with `follow_view_more=False` → INFO (that truncation is what the caller asked for — warning about it trains people to ignore warnings). This WARNING is not theoretical: it is what surfaced the unmapped `fs-r561`/`fs-r562` slugs (closed in 0.22.2, see above) — a downstream consumer read the shortfall out of `completeness()` and reported it.
 - **`Stock.get_latest_price()` is DR-aware by default:** for DRs it returns a `DrIndicativeQuotation` (TradingView indicative price; `volume`/`change` are `None` — it's a fair value, not a SET trade) and falls back to SET chart data on ANY TradingView failure. The switch only applies when `as_of is None` (TV can't answer historical instants); opt out per-call with `prefer_dr_indicative=False`. DR-ness is detected by one cached DR-profile probe per `Stock` instance (a 404 marks non-DR permanently for that instance; transient errors are never cached). Indicative vs SET-close divergence is EXPECTED (probe: 5.94 indicative vs 5.75 SET close after a US-session move) — not a bug.
 
 ---

{settfex-0.22.1 → settfex-0.22.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: settfex
-Version: 0.22.1
+Version: 0.22.2
 Summary: Async Python library for fetching real-time and historical data from the Stock Exchange of Thailand (SET) and Thailand Futures Exchange (TFEX)
 Project-URL: Homepage, https://github.com/lumduan/settfex
 Project-URL: Documentation, https://github.com/lumduan/settfex#readme

{settfex-0.22.1 → settfex-0.22.2}/docs/settfex/services/sec/financial_report.md

@@ -372,7 +372,9 @@ file = await download_sec_document(doc)  # dead link -> FetchError ("soft 404")
 POST /public/idisc/api/company/valuebyuniqueId      {"lang","content"} -> [{"Text","Value","Flag"}]
 GET  /public/idisc/{lang}/FinancialReport/{TYPE}     -> page + __VIEWSTATE tokens
 POST /public/idisc/{lang}/FinancialReport/{TYPE}     form postback -> result HTML
-GET  /public/idisc/{lang}/ViewMore/{slug}?...        -> complete section (fs-norm/fs-kf/fs-mda)
+GET  /public/idisc/{lang}/ViewMore/{slug}?...        -> complete section
+     slug in fs-norm | fs-kf | fs-mda | fs-r561 | fs-r562   ('fs-' is the site's prefix,
+                                                             not a category hint)
 GET  /public/idisc/Download?FILEID=<path>            -> zip/pdf bytes
 GET  /ipos/Common/IPOSGetFile.aspx?id=<id>&sq=0&v=10 -> zip bytes
 GET  /public/idisc/Views/FinancialStatementDownload?query=<blob> -> zip bytes (recent KFR rows)

{settfex-0.22.1 → settfex-0.22.2}/pyproject.toml

@@ -1,6 +1,6 @@
 [project]
 name = "settfex"
-version = "0.22.1"
+version = "0.22.2"
 description = "Async Python library for fetching real-time and historical data from the Stock Exchange of Thailand (SET) and Thailand Futures Exchange (TFEX)"
 readme = "README.md"
 requires-python = ">=3.11"

{settfex-0.22.1 → settfex-0.22.2}/settfex/__init__.py

@@ -35,7 +35,7 @@ Usage:
     >>> asyncio.run(main())
 """
 
-__version__ = "0.22.1"
+__version__ = "0.22.2"
 __author__ = "batt"
 __license__ = "MIT"
 

{settfex-0.22.1 → settfex-0.22.2}/settfex/services/sec/constants.py

@@ -44,11 +44,19 @@ SEC_REPORT_TYPE_FORM_56_1 = "R561"
 SEC_REPORT_TYPE_FORM_56_2 = "R562"
 SEC_REPORT_TYPE_KEY_FINANCIAL_RATIO = "KFR"
 
-# ViewMore slugs (seen live) for the sections a FS search can truncate.
+# ViewMore slugs (seen live) for every section that can truncate behind "display all results".
+# Note the `fs-` prefix is the site's own and does not mean "financial statement": `fs-r561` and
+# `fs-r562` are served by the 56-1 / 56-2 searches. Live-probed 2026-09-20.
+#
+# This is a reference constant; the mapping the listing service actually consults is
+# `_CATEGORY_FOR_VIEWMORE_SLUG` in `financial_report.py`. Keep the two in step -- they state the
+# same fact, and this one has no caller to notice when it goes stale.
 SEC_VIEWMORE_SLUGS: dict[str, str] = {
     "financial_statement": "fs-norm",
     "key_financial_ratio": "fs-kf",
     "mda": "fs-mda",
+    "form_56_1": "fs-r561",
+    "form_56_2": "fs-r562",
 }
 
 # The SEC search form's field names (ASP.NET control ids).

{settfex-0.22.1 → settfex-0.22.2}/settfex/services/sec/financial_report.py

@@ -880,10 +880,21 @@ def _escalate_losses(
         )
 
 
+# Every slug the site serves a "display all results" page under. The `fs-` prefix is the site's,
+# not a category hint: `fs-r561` and `fs-r562` belong to the 56-1 / 56-2 searches, not to the FS
+# one. Live-probed 2026-09-20 across CPALL/PTT/SCB in both languages.
+#
+# The last two were MISSING until 0.22.2, so a 56-1 or 56-2 section over the ~10-row inline cap
+# was truncated even with `follow_view_more=True`. 0.22.0's notes predicted exactly this and said
+# the shortfall WARNING would surface it; it did, from a downstream consumer reading
+# `completeness()`. Scale, with a 2000-2026 window: CPALL Thai 56-1 served 11 of 23, PTT Thai 56-1
+# 11 of 25, PTT 56-2 10 of 15 in BOTH languages -- so this was never a Thai-only gap.
 _CATEGORY_FOR_VIEWMORE_SLUG: dict[str, DocumentCategory] = {
     "fs-norm": DocumentCategory.FINANCIAL_STATEMENT,
     "fs-kf": DocumentCategory.KEY_FINANCIAL_RATIO,
     "fs-mda": DocumentCategory.MDA,
+    "fs-r561": DocumentCategory.FORM_56_1,
+    "fs-r562": DocumentCategory.FORM_56_2,
 }
 
 _VIEWMORE_HREF = re.compile(r'href="([^"]*?/ViewMore/([a-z0-9-]+)[^"]*)"', re.IGNORECASE)

{settfex-0.22.1 → settfex-0.22.2}/tests/services/sec/derive_th_fixtures.py

@@ -1,11 +1,12 @@
 #!/usr/bin/env python3
-"""Derive the committed SEC listing fixtures from the captured response bodies of #123 / #127 / #131.
+"""Derive the committed SEC listing fixtures from captured bodies (#123 / #127 / #131 / #133 / 0.22.2).
 
 Run from the repo root, with the evidence bundles extracted somewhere gitignored. Any number of
 bundle roots may be given; each source is resolved against whichever one carries it::
 
     uv run python tests/services/sec/derive_th_fixtures.py \\
-        tmp/issue-123-bundle tmp/issue-127-bundle tmp/issue-batch-r4/extracted/0004-evidence
+        tmp/issue-123-bundle tmp/issue-127-bundle \\
+        tmp/issue-batch-r4/extracted/0004-evidence tmp/issue-fs-r561
 
 What this does, and what it deliberately does not do:
 
@@ -57,6 +58,11 @@ SELECTED = [
     ("F3_en_PTT_56-2", "en_ptt_56_2.html", "#127", "panel"),
     ("G5_idisc_505_error_page", "idisc_505_error_page.html", "#131", "verbatim"),
     ("G2_capital_indirection_en_2013_p12", "capital_indirection_en.html", "#133", "verbatim"),
+    # The "display all results" pages for the two slugs that were unmapped until 0.22.2. H1 is the
+    # page whose section reports 12 and serves 12. H2 is ENGLISH on purpose: it is the evidence
+    # that the 56-2 half of this gap was never Thai-only.
+    ("H1_th_CPALL_viewmore_fs-r561", "th_cpall_viewmore_56_1.html", "0.22.2", "panel"),
+    ("H2_en_PTT_viewmore_fs-r562", "en_ptt_viewmore_56_2.html", "0.22.2", "panel"),
 ]
 
 PANEL_START = '<div id="ctl00_CPH_pnlControl"'
@@ -139,7 +145,8 @@ def main(argv: list[str]) -> int:
 
     readme = [
         "# SEC listing fixtures — provenance\n",
-        "Derived from the response bodies captured for issues #123, #127, #131 and #133 by",
+        "Derived from the response bodies captured for issues #123, #127, #131, #133 and the",
+        "0.22.2 slug fix by",
         "`tests/services/sec/derive_th_fixtures.py`, which verifies every source sha256 against the",
         "bundle's `MANIFEST.md` and then either slices the result panel out of the original bytes",
         "or copies the whole body, verbatim in both cases.",

{settfex-0.22.1 → settfex-0.22.2}/tests/services/sec/fixtures_sec/README.md

@@ -1,6 +1,7 @@
 # SEC listing fixtures — provenance
 
-Derived from the response bodies captured for issues #123, #127, #131 and #133 by
+Derived from the response bodies captured for issues #123, #127, #131, #133 and the
+0.22.2 slug fix by
 `tests/services/sec/derive_th_fixtures.py`, which verifies every source sha256 against the
 bundle's `MANIFEST.md` and then either slices the result panel out of the original bytes
 or copies the whole body, verbatim in both cases.
@@ -31,3 +32,5 @@ uv run python tests/services/sec/derive_th_fixtures.py \
 | #127 | `F3_en_PTT_56-2.html` | `0be33ee80ad2ec303d699d74d1b9d6ba8fbd5cff33fd0b557e825827deb1548a` | `en_ptt_56_2.html` | `84cec174807540f4e03cdc7a2853bcb96a65a58d2ae5f6293c64e12f6bf64ae1` | 3,056 |
 | #131 | `G5_idisc_505_error_page.html` | `3c6c080037bc084f14cb1cd2a89c09698dd00ad831e2efb0d553618b051d2794` | `idisc_505_error_page.html` | `3c6c080037bc084f14cb1cd2a89c09698dd00ad831e2efb0d553618b051d2794` | 369 |
 | #133 | `G2_capital_indirection_en_2013_p12.html` | `e0e114533be26b36e796880130ce4b3650f20122d0f4164a838faf93498ff35f` | `capital_indirection_en.html` | `e0e114533be26b36e796880130ce4b3650f20122d0f4164a838faf93498ff35f` | 2,263 |
+| 0.22.2 | `H1_th_CPALL_viewmore_fs-r561.html` | `c413c1ce84e12315a01f80fdd6db8d478102d206412916eff7d48e958f36c622` | `th_cpall_viewmore_56_1.html` | `2756b5585f08b761bf3c93a2dc36b2d525f43457223d3004b438695056ba7f9c` | 5,640 |
+| 0.22.2 | `H2_en_PTT_viewmore_fs-r562.html` | `c54c4b883c7e48193d254f3cea4735b5b23f1cd2c3bac7ae7d7b80814eac880e` | `en_ptt_viewmore_56_2.html` | `4880199be20a556cb3bb69840a694beeb6e9d42b7b1ae4c157e2c2e341a03c10` | 5,857 |

settfex-0.22.2/tests/services/sec/fixtures_sec/en_ptt_viewmore_56_2.html

@@ -0,0 +1 @@
+<div id="ctl00_CPH_pnlControl"><div class="card card-table"><div class="card-heading">Form 56-2 : Annual Reports  ( 15 record(s) found)</div><div class="table-responsive"><div><table class="table table-striped table-hover" cellspacing="0" rules="all" border="1" id="gPP06T06" style="border-style:None;border-collapse:collapse;"><tr><th class="RgCol_Left" scope="col">Name</th><th class="RgCol_Center" scope="col">Year</th><th class="RgCol_Center" scope="col">Receive Date</th><th class="RgCol_Center" scope="col">Details</th></tr><tr><td class="RgCol_Left">PTT PUBLIC COMPANY LIMITED</td><td class="RgCol_Center">2025</td><td class="RgCol_Center">12/03/2026</td><td class="icon30 text-center"><a href="https://market.sec.or.th/public/idisc/Download?FILEID=dat/annual/0646ONE120320262215100764E.zip" target="_blank"><img src="https://market.sec.or.th/Documents/images/icon_view.jpg" /></a></td></tr><tr><td class="RgCol_Left">PTT PUBLIC COMPANY LIMITED</td><td class="RgCol_Center">2024</td><td class="RgCol_Center">13/03/2025</td><td class="icon30 text-center"><a href="https://market.sec.or.th/public/idisc/Download?FILEID=dat/annual/0646ONE130320251942270461E.zip" target="_blank"><img src="https://market.sec.or.th/Documents/images/icon_view.jpg" /></a></td></tr><tr><td class="RgCol_Left">PTT PUBLIC COMPANY LIMITED</td><td class="RgCol_Center">2023</td><td class="RgCol_Center">14/03/2024</td><td class="icon30 text-center"><a href="https://market.sec.or.th/public/idisc/Download?FILEID=dat/annual/0646ONE140320241008470575E.zip" target="_blank"><img src="https://market.sec.or.th/Documents/images/icon_view.jpg" /></a></td></tr><tr><td class="RgCol_Left">PTT PUBLIC COMPANY LIMITED</td><td class="RgCol_Center">2022</td><td class="RgCol_Center">15/03/2023</td><td class="icon30 text-center"><a href="https://market.sec.or.th/public/idisc/Download?FILEID=dat/f56/F0646E22.zip" target="_blank"><img src="https://market.sec.or.th/Documents/images/icon_view.jpg" /></a></td></tr><tr><td class="RgCol_Left">PTT PUBLIC COMPANY LIMITED</td><td class="RgCol_Center">2021</td><td class="RgCol_Center">31/05/2022</td><td class="icon30 text-center"><a href="https://market.sec.or.th/public/idisc/Download?FILEID=dat/f56/F0646E21.zip" target="_blank"><img src="https://market.sec.or.th/Documents/images/icon_view.jpg" /></a></td></tr><tr><td class="RgCol_Left">PTT PUBLIC COMPANY LIMITED</td><td class="RgCol_Center">2020</td><td class="RgCol_Center">15/03/2021</td><td class="icon30 text-center"><a href="https://market.sec.or.th/public/idisc/Download?FILEID=dat/f56/F0646E20.zip" target="_blank"><img src="https://market.sec.or.th/Documents/images/icon_view.jpg" /></a></td></tr><tr><td class="RgCol_Left">PTT PUBLIC COMPANY LIMITED</td><td class="RgCol_Center">2019</td><td class="RgCol_Center">12/03/2020</td><td class="icon30 text-center"><a href="https://market.sec.or.th/public/idisc/Download?FILEID=dat/annual/A0646E19.zip" target="_blank"><img src="https://market.sec.or.th/Documents/images/icon_view.jpg" /></a></td></tr><tr><td class="RgCol_Left">PTT PUBLIC COMPANY LIMITED</td><td class="RgCol_Center">2018</td><td class="RgCol_Center">15/03/2019</td><td class="icon30 text-center"><a href="https://market.sec.or.th/public/idisc/Download?FILEID=dat/annual/A0646E18.zip" target="_blank"><img src="https://market.sec.or.th/Documents/images/icon_view.jpg" /></a></td></tr><tr><td class="RgCol_Left">PTT PUBLIC COMPANY LIMITED</td><td class="RgCol_Center">2017</td><td class="RgCol_Center">21/03/2018</td><td class="icon30 text-center"><a href="https://market.sec.or.th/public/idisc/Download?FILEID=dat/annual/A0646E17.zip" target="_blank"><img src="https://market.sec.or.th/Documents/images/icon_view.jpg" /></a></td></tr><tr><td class="RgCol_Left">PTT PUBLIC COMPANY LIMITED</td><td class="RgCol_Center">2016</td><td class="RgCol_Center">03/04/2017</td><td class="icon30 text-center"><a href="https://market.sec.or.th/public/idisc/Download?FILEID=dat/annual/A0646E16.zip" target="_blank"><img src="https://market.sec.or.th/Documents/images/icon_view.jpg" /></a></td></tr><tr><td class="RgCol_Left">PTT PUBLIC COMPANY LIMITED</td><td class="RgCol_Center">2015</td><td class="RgCol_Center">22/03/2016</td><td class="icon30 text-center"><a href="https://market.sec.or.th/public/idisc/Download?FILEID=dat/annual/A0646E15.zip" target="_blank"><img src="https://market.sec.or.th/Documents/images/icon_view.jpg" /></a></td></tr><tr><td class="RgCol_Left">PTT PUBLIC COMPANY LIMITED</td><td class="RgCol_Center">2014</td><td class="RgCol_Center">17/03/2015</td><td class="icon30 text-center"><a href="https://market.sec.or.th/public/idisc/Download?FILEID=dat/annual/A0646E14.zip" target="_blank"><img src="https://market.sec.or.th/Documents/images/icon_view.jpg" /></a></td></tr><tr><td class="RgCol_Left">PTT PUBLIC COMPANY LIMITED</td><td class="RgCol_Center">2013</td><td class="RgCol_Center">04/04/2014</td><td class="icon30 text-center"><a href="https://market.sec.or.th/public/idisc/Download?FILEID=dat/annual/A0646E13.zip" target="_blank"><img src="https://market.sec.or.th/Documents/images/icon_view.jpg" /></a></td></tr><tr><td class="RgCol_Left">PTT PUBLIC COMPANY LIMITED</td><td class="RgCol_Center">2012</td><td class="RgCol_Center">29/03/2013</td><td class="icon30 text-center"><a href="https://market.sec.or.th/public/idisc/Download?FILEID=dat/annual/APTTE12.zip " target="_blank"><img src="https://market.sec.or.th/Documents/images/icon_view.jpg" /></a></td></tr><tr><td class="RgCol_Left">PTT PUBLIC COMPANY LIMITED</td><td class="RgCol_Center">2011</td><td class="RgCol_Center">11/04/2012</td><td class="icon30 text-center"><a href="https://market.sec.or.th/public/idisc/Download?FILEID=dat/annual/A0646E11.zip" target="_blank"><img src="https://market.sec.or.th/Documents/images/icon_view.jpg" /></a></td></tr></table></div></div></div></div>

settfex-0.22.2/tests/services/sec/fixtures_sec/th_cpall_viewmore_56_1.html

@@ -0,0 +1 @@
+<div id="ctl00_CPH_pnlControl"><div class="card card-table"><div class="card-heading">56-1 : แบบแสดงรายการข้อมูลประจำปี  (จำนวนรายการที่พบ 12 รายการ)</div><div class="table-responsive"><div><table class="table table-striped table-hover" cellspacing="0" rules="all" border="1" id="gPP06T05" style="border-style:None;border-collapse:collapse;"><tr><th class="RgCol_Left" scope="col">ชื่อ</th><th class="RgCol_Center" scope="col">ประจำปี</th><th class="RgCol_Center" scope="col">วันที่ได้รับข้อมูล</th><th class="RgCol_Center" scope="col">รายละเอียด</th></tr><tr><td class="RgCol_Left">บริษัท ซีพี ออลล์ จำกัด (มหาชน)</td><td class="RgCol_Center">2568</td><td class="RgCol_Center">26/03/2569</td><td class="icon30 text-center"><a href="https://market.sec.or.th/public/idisc/Download?FILEID=dat/f56/0737E1N260320261522390725T.zip" target="_blank"><img src="https://market.sec.or.th/Documents/images/icon_view.jpg" /></a></td></tr><tr><td class="RgCol_Left">บริษัท ซีพี ออลล์ จำกัด (มหาชน)</td><td class="RgCol_Center">2567</td><td class="RgCol_Center">31/03/2568</td><td class="icon30 text-center"><a href="https://market.sec.or.th/public/idisc/Download?FILEID=dat/f56/0737ONE310320252137250194T.zip" target="_blank"><img src="https://market.sec.or.th/Documents/images/icon_view.jpg" /></a></td></tr><tr><td class="RgCol_Left">บริษัท ซีพี ออลล์ จำกัด (มหาชน)</td><td class="RgCol_Center">2566</td><td class="RgCol_Center">03/04/2567</td><td class="icon30 text-center"><a href="https://market.sec.or.th/public/idisc/Download?FILEID=dat/f56/0737ONE030420241559440549T.zip" target="_blank"><img src="https://market.sec.or.th/Documents/images/icon_view.jpg" /></a></td></tr><tr><td class="RgCol_Left">บริษัท ซีพี ออลล์ จำกัด (มหาชน)</td><td class="RgCol_Center">2565</td><td class="RgCol_Center">27/03/2566</td><td class="icon30 text-center"><a href="https://market.sec.or.th/public/idisc/Download?FILEID=dat/f56/F0737T22.zip" target="_blank"><img src="https://market.sec.or.th/Documents/images/icon_view.jpg" /></a></td></tr><tr><td class="RgCol_Left">บริษัท ซีพี ออลล์ จำกัด (มหาชน)</td><td class="RgCol_Center">2564</td><td class="RgCol_Center">30/03/2565</td><td class="icon30 text-center"><a href="https://market.sec.or.th/public/idisc/Download?FILEID=dat/f56/F0737T21.zip" target="_blank"><img src="https://market.sec.or.th/Documents/images/icon_view.jpg" /></a></td></tr><tr><td class="RgCol_Left">บริษัท ซีพี ออลล์ จำกัด (มหาชน)</td><td class="RgCol_Center">2563</td><td class="RgCol_Center">22/03/2564</td><td class="icon30 text-center"><a href="https://market.sec.or.th/public/idisc/Download?FILEID=dat/f56/F0737T20.zip" target="_blank"><img src="https://market.sec.or.th/Documents/images/icon_view.jpg" /></a></td></tr><tr><td class="RgCol_Left">บริษัท ซีพี ออลล์ จำกัด (มหาชน)</td><td class="RgCol_Center">2562</td><td class="RgCol_Center">25/03/2563</td><td class="icon30 text-center"><a href="https://market.sec.or.th/public/idisc/Download?FILEID=dat/f56/F0737T19.zip" target="_blank"><img src="https://market.sec.or.th/Documents/images/icon_view.jpg" /></a></td></tr><tr><td class="RgCol_Left">บริษัท ซีพี ออลล์ จำกัด (มหาชน)</td><td class="RgCol_Center">2561</td><td class="RgCol_Center">28/05/2562</td><td class="icon30 text-center"><a href="https://market.sec.or.th/public/idisc/Download?FILEID=dat/f56/F0737T18.zip" target="_blank"><img src="https://market.sec.or.th/Documents/images/icon_view.jpg" /></a></td></tr><tr><td class="RgCol_Left">บริษัท ซีพี ออลล์ จำกัด (มหาชน)</td><td class="RgCol_Center">2560</td><td class="RgCol_Center">02/04/2561</td><td class="icon30 text-center"><a href="https://market.sec.or.th/public/idisc/Download?FILEID=dat/f56/F0737T17.zip" target="_blank"><img src="https://market.sec.or.th/Documents/images/icon_view.jpg" /></a></td></tr><tr><td class="RgCol_Left">บริษัท ซีพี ออลล์ จำกัด (มหาชน)</td><td class="RgCol_Center">2559</td><td class="RgCol_Center">15/03/2560</td><td class="icon30 text-center"><a href="https://market.sec.or.th/public/idisc/Download?FILEID=dat/f56/F0737T16.zip" target="_blank"><img src="https://market.sec.or.th/Documents/images/icon_view.jpg" /></a></td></tr><tr><td class="RgCol_Left">บริษัท ซีพี ออลล์ จำกัด (มหาชน)</td><td class="RgCol_Center">2558</td><td class="RgCol_Center">24/08/2559</td><td class="icon30 text-center"><a href="https://market.sec.or.th/public/idisc/Download?FILEID=dat/f56/F0737T15.zip" target="_blank"><img src="https://market.sec.or.th/Documents/images/icon_view.jpg" /></a></td></tr><tr><td class="RgCol_Left">บริษัท ซีพี ออลล์ จำกัด (มหาชน)</td><td class="RgCol_Center">2557</td><td class="RgCol_Center">27/03/2558</td><td class="icon30 text-center"><a href="https://market.sec.or.th/public/idisc/Download?FILEID=dat/f56/F0737T14.zip" target="_blank"><img src="https://market.sec.or.th/Documents/images/icon_view.jpg" /></a></td></tr></table></div></div></div></div>

settfex-0.22.2/tests/services/sec/test_view_more_slugs.py

@@ -0,0 +1,166 @@
+"""The 56-1 / 56-2 "display all results" slugs, unmapped until 0.22.2.
+
+`_CATEGORY_FOR_VIEWMORE_SLUG` covered `fs-norm` / `fs-kf` / `fs-mda` — the three sections a single
+`FS` search returns — and nothing else. A 56-1 or 56-2 section over the site's ~10-row inline cap
+therefore truncated **even with `follow_view_more=True`**, because the link to its complete list
+carried a slug the map did not recognise.
+
+0.22.0's own notes predicted this and said the shortfall WARNING would be what surfaced it. It was,
+from the other side of the API: a consumer reading `completeness()` saw 10 of 12 on a Thai 56-1.
+
+The scale, live-probed 2026-09-20 over a 2000–2026 window:
+
+    CPALL  56-1  th   served 11 of 23
+    PTT    56-1  th   served 11 of 25
+    PTT    56-2  th   served 10 of 15
+    PTT    56-2  en   served 10 of 15     <- not a Thai-only gap
+
+Both fixtures here are real captured pages (see ``fixtures_sec/README.md``), and the 56-2 one is
+**English on purpose**: it is the evidence for that last line.
+
+Note the `fs-` prefix is the site's own and is not a category hint — `fs-r561` is served by the
+56-1 search, not by the FS one. A mapping keyed on the prefix would be wrong.
+"""
+
+from __future__ import annotations
+
+from typing import Any
+from unittest.mock import patch
+
+import pytest
+
+from settfex.services.sec.constants import SEC_VIEWMORE_SLUGS
+from settfex.services.sec.financial_report import (
+    _CATEGORY_FOR_VIEWMORE_SLUG,
+    DocumentCategory,
+    FinancialReportService,
+)
+from settfex.services.sec.utils import parse_report_tables, split_section_count
+
+from .fixtures import REPORT_PAGE_HTML, load_fixture
+from .test_financial_report import _make_fetcher, _resp
+
+# A 56-1 search result truncated to one row, with the link to its complete list. Same shape as the
+# existing FS_TRUNCATED_HTML constant, which models the equivalent `fs-norm` case.
+FORM_56_1_TRUNCATED_HTML = """
+<div id="ctl00_CPH_pnlControl">
+  <div class="card card-table"><div class="card-heading">56-1 : Annual Registration Statements ( 12 record(s) found)</div>
+    <table id="g561"><tbody>
+      <tr><th>Name</th><th>Year</th><th>Receive Date</th><th>Details</th></tr>
+      <tr><td>CP ALL PUBLIC COMPANY LIMITED</td><td>2025</td><td>26/03/2026</td>
+          <td><a href="https://market.sec.or.th/public/idisc/Download?FILEID=dat/f56/inline.zip"><img></a></td></tr>
+      <tr><td colspan="4"><a href="/public/idisc/th/ViewMore/fs-r561?UniqueIdReference=0000003875&amp;DateFrom=20150101&amp;DateTo=20261231">Click here to display all results</a></td></tr>
+    </tbody></table>
+  </div>
+</div>
+"""
+
+
+class TestTheSlugMap:
+    """Both surfaces state the same fact, and only one of them has a caller."""
+
+    @pytest.mark.parametrize(
+        ("slug", "category"),
+        [
+            ("fs-norm", DocumentCategory.FINANCIAL_STATEMENT),
+            ("fs-kf", DocumentCategory.KEY_FINANCIAL_RATIO),
+            ("fs-mda", DocumentCategory.MDA),
+            ("fs-r561", DocumentCategory.FORM_56_1),
+            ("fs-r562", DocumentCategory.FORM_56_2),
+        ],
+    )
+    def test_every_live_slug_is_mapped(self, slug: str, category: DocumentCategory) -> None:
+        assert _CATEGORY_FOR_VIEWMORE_SLUG[slug] is category
+
+    def test_every_category_can_be_completed(self) -> None:
+        """A category with no slug is one that silently truncates — there must be none left."""
+        assert set(_CATEGORY_FOR_VIEWMORE_SLUG.values()) == set(DocumentCategory)
+
+    def test_the_public_constant_agrees(self) -> None:
+        """`SEC_VIEWMORE_SLUGS` has no caller, so nothing else would notice it going stale."""
+        assert {c.value: s for s, c in _CATEGORY_FOR_VIEWMORE_SLUG.items()} == SEC_VIEWMORE_SLUGS
+
+
+class TestTheCapturedPages:
+    """What the site actually serves behind those two links."""
+
+    @pytest.mark.parametrize(
+        ("name", "expected"),
+        [("th_cpall_viewmore_56_1.html", 12), ("en_ptt_viewmore_56_2.html", 15)],
+    )
+    def test_the_page_serves_exactly_what_its_section_reports(
+        self, name: str, expected: int
+    ) -> None:
+        """The premise of following the link at all: it holds the COMPLETE list."""
+        rows = parse_report_tables(load_fixture(name))
+        counts = {split_section_count(str(r["section"]))[1] for r in rows}
+        assert counts == {expected}
+        assert len(rows) == expected
+
+
+class TestItIsFollowedEndToEnd:
+    @staticmethod
+    async def _list(post_body: str, view_more_body: str, **kwargs: Any) -> Any:
+        async def router(url, headers=None, *, method="GET", json_body=None, data=None, **kw):
+            if "ViewMore" in url:
+                return _resp(view_more_body)
+            return _resp(post_body if method == "POST" else REPORT_PAGE_HTML)
+
+        with patch("settfex.services.sec.financial_report.AsyncDataFetcher") as cls:
+            _make_fetcher(cls, router)
+            return await FinancialReportService().fetch_documents("uid", **kwargs)
+
+    @pytest.mark.asyncio
+    async def test_a_truncated_56_1_is_completed(self) -> None:
+        """The whole point: 1 inline row becomes the 12 the section reports."""
+        docs = await self._list(
+            FORM_56_1_TRUNCATED_HTML,
+            load_fixture("th_cpall_viewmore_56_1.html"),
+            types="form_56_1",
+            lang="th",
+            follow_view_more=True,
+        )
+        assert len(docs) == 12
+        assert docs.completeness() == {"form_56_1": (12, 12)}
+        assert not docs.accounting.has_losses
+        assert docs.accounting.is_balanced
+
+    @pytest.mark.asyncio
+    async def test_without_following_it_still_truncates(self) -> None:
+        """`follow_view_more=False` is unchanged — the shortfall is expected and reported."""
+        docs = await self._list(
+            FORM_56_1_TRUNCATED_HTML,
+            load_fixture("th_cpall_viewmore_56_1.html"),
+            types="form_56_1",
+            lang="th",
+            follow_view_more=False,
+        )
+        assert len(docs) == 1
+        assert docs.completeness() == {"form_56_1": (1, 12)}
+
+    @pytest.mark.asyncio
+    async def test_the_accounting_replaces_rather_than_doubles(self) -> None:
+        """The ViewMore rows REPLACE the inline one; counting both would break the identity."""
+        docs = await self._list(
+            FORM_56_1_TRUNCATED_HTML,
+            load_fixture("th_cpall_viewmore_56_1.html"),
+            types="form_56_1",
+            lang="th",
+            follow_view_more=True,
+        )
+        tally = docs.accounting.by_category["form_56_1"]
+        assert tally.rows == 12
+        assert tally.documents == 12
+        assert tally.navigation == 0, "the inline page's nav row is superseded with its rows"
+
+    @pytest.mark.asyncio
+    async def test_a_category_nobody_asked_for_is_not_followed(self) -> None:
+        """Unchanged behaviour, re-pinned now that two more slugs can match."""
+        docs = await self._list(
+            FORM_56_1_TRUNCATED_HTML,
+            load_fixture("th_cpall_viewmore_56_1.html"),
+            types="form_56_2",
+            lang="th",
+            follow_view_more=True,
+        )
+        assert list(docs) == []

{settfex-0.22.1 → settfex-0.22.2}/uv.lock

@@ -3175,7 +3175,7 @@ wheels = [
 
 [[package]]
 name = "settfex"
-version = "0.22.1"
+version = "0.22.2"
 source = { editable = "." }
 dependencies = [
     { name = "curl-cffi" },