substack-saved-mcp 0.3.0__tar.gz → 0.3.2__tar.gz

{substack_saved_mcp-0.3.0 → substack_saved_mcp-0.3.2}/CLAUDE.md

@@ -29,12 +29,13 @@ After any change to dependencies or tracked files, run `uv lock` to keep `uv.loc
 - `mcp_server.py` exposes the same application operations as FastMCP stdio tools and resources. Read tools query the local cache; sync and save/unsave operations use the authenticated browser session.
 - `substack_client.py` owns Playwright authentication and remote Substack interaction. `login` is the only intended headful workflow; normal sync and write paths use `storage_state.json` headlessly. Synchronous Playwright API calls are routed through `_run_playwright_sync()` to safely execute in a worker thread if an `asyncio` event loop is active (e.g. under FastMCP). Saved-post fetching prefers the reader inbox API (`GET /api/v1/reader/posts?inboxType=saved`), which exposes the real bookmark timestamp (`saved_at`) and an ISO publication date (`post_date`) per post. `_fetch_all_saved_via_reader_api()` cursor-paginates that endpoint (each page's oldest `saved_at` becomes the next `after=` cursor, until `more` is false), dedupes by canonical URL, and enriches each flat post with its `publication` object (from the response's `publications` array, matched by `publication_id`) and `author_name` (from `publishedBylines`). The full result is cached in `_api_cache` and sliced by offset; `_fetch_saved_posts_page_impl()` accepts an optional `playwright_instance` (same pattern as the notes fetcher and `_fetch_via_dom`) so this caching/slicing/fallback branch is directly testable with a Playwright double instead of only reachable through a real browser. Each page request goes through `_reader_api_get()`, which retries transient failures (HTTP 429 and 5xx) up to `max_retries` (default 3) with `Retry-After`-aware backoff: `_retry_after_seconds()` honors an integer `Retry-After` header (clamped to a 30s cap so a hostile value can't hang the sync) and otherwise uses capped exponential backoff (0.5s, 1s, 2s, ...); 401/403 and other 4xx are returned unretried so the existing auth/fallback handling applies. A 429 that survives all retries is deliberately treated as "unavailable" (partial list, or `None` → DOM fallback) rather than as a silent empty-success, so rate-limiting never masquerades as "you have no saved posts." Both `_reader_api_get()` and `_fetch_all_saved_via_reader_api()` accept an injectable `sleep_func` (default `time.sleep`) so tests exercise the backoff without real delays. If that endpoint is unavailable, it falls back to headless DOM extraction on `https://substack.com/saved`, which scrapes `div.reader2-post-container` cards, caches scrolling results in `_dom_cache`, and marks each dict with `_dom: True`; DOM cards only expose a localized relative publish string and no bookmark time, so `saved_at` stays unknown on that path. Both `save_post()` and `unsave_post()` were originally guesswork (an unverified DOM button selector plus an unverified `POST /api/v1/bookmark` call) and were fixed once real endpoints were captured via `inspect-network`. Every post page server-renders a `window._preloads` blob containing that post's own numeric ID and rich metadata (`preloads.post.id`, `.title`, `.audience`, `.description`/`.subtitle`, `.post_date`, plus `preloads.pub.name`); `_save_post_impl` reads this via `page.evaluate("() => window._preloads")` right after page load and, when an ID is found, calls the real `POST https://substack.com/api/v1/posts/saved` endpoint (body `{"post_id": ...}`) directly via the Playwright API request context — this also lets it populate the returned `SavedPost` with accurate title/publication/audience/excerpt instead of parsing `page.title()`. `_unsave_post_impl` does the mirror image: when the post's `substack_post_id` is already known (true for any post that has been through a normal `sync`, since the reader API's `id` field populates it), it calls `DELETE https://substack.com/api/v1/posts/saved` with the same body shape, no DOM interaction at all. Both treat an `ok` response as `"confirmed"` and only fall back to the old best-effort DOM click (`_click_bookmark_toggle()`, used when the numeric ID can't be obtained or the direct call doesn't confirm) — note its selector is English-only (`aria-label*='save'/'bookmark'`) and can silently fail on non-English Substack UIs, which is part of why the direct API path is preferred whenever possible. `_click_bookmark_toggle()` fingerprints the button's `aria-label`/`aria-pressed`/`class` before and after the click and returns `"confirmed"` only if that fingerprint changed, else `"unconfirmed"`, `"not_found"`, or `"click_failed"` — `save_post()` returns `(SavedPost, confirmation)` and `unsave_post()` returns just the confirmation string; both accept an optional `playwright_instance` for test injection (same pattern as `_fetch_via_dom`). `fetch_post_content()` reuses the same `window._preloads` mechanism to retrieve a saved post's full content: `_fetch_post_content_impl` navigates to the post's page and reads `preloads.post.body_html` (Substack's field name for full content — `parse_remote_post()` already expects this key from the reader API, though the saved-list payload never actually populates it, only individual post pages do), returning `None` for `body_html` if the page's embed format doesn't expose it (frontend change) or the account lacks paywall access; this is the case where the caller should be told to run `inspect-network` against an open post page to re-discover the real content source. `cli.py`'s `inspect-network` command (which logs any `api/v1`/`bookmark`/`saved`/`notes?`/`comment`/`reader`/`feed`/`restack` request's method, URL, status, and JSON response body) is the tool for discovering/re-verifying these endpoints when Substack's frontend changes. It supports `--authenticated/--anonymous` (reuses `storage_state.json` via `get_storage_state_path()` so `/saved` renders the real logged-in page instead of the marketing page), `--url` to target any page, `--filter` to override the response-matching regex, `--max-body` to cap logged response bytes, and `--out PATH` to append each exchange as JSON Lines for later grepping. Response bodies are captured via `context.route("**/*", handle_route)` + `route.fetch()`/`route.fulfill()`, not the more obvious `page.on("response", ...)` + `response.text()`: the latter can deadlock calling `response.text()` inside a sync-API response event handler on the same driver thread, and in practice silently produced zero captured bodies across an entire 108-request session before this fix (every `response_body` came back `null` with no error surfaced) — `route.fetch()` reads the body outside that handler and `route.fulfill(response=...)` re-serves the exact response so page behavior is unaffected; a body that still can't be read now logs a visible yellow warning instead of writing `null` silently.
 
-Saved-posts fetching has a three-source chain, controlled by `get_saved_posts_source()`/`SUBSTACK_SAVED_POSTS_SOURCE` (`auto` default, or a forced `legacy`/`unified`/`dom`): the unified `GET /api/v1/reader/saved?filter=posts` endpoint (the same one notes uses with `filter=notes`) is now tried first, then the legacy reader-posts API above as an automatic fallback, then the DOM fallback. This was a migration decision, not a default guess — see `PLAN-newapi.md` for the full evidence-gathering and decision-gate process. Two tools were built to make that decision on evidence rather than guesswork: `probe-api URL [--out PATH]` headlessly GETs any known API URL with the saved session (no browser page) and dumps the raw JSON — a lighter complement to `inspect-network`, which is for *discovering* unknown endpoints by watching a live browsing session; and `compare-saved-apis [--out PATH]` fetches the full saved list from both sources in one session and reports counts, URL-set differences, per-field presence on the unified payload, and whether `post.saved_at` is present and ordered. A live run against a real ~1000-post account (after a first attempt was invalidated by rate limiting immediately following a large legacy fetch — `compare-saved-apis` now pauses and retries harder around the second fetch for exactly this reason) found the unified endpoint a **strict superset**: 1079 posts vs. legacy's 985, with only 1 post legacy had that unified lacked, `saved_at`/`id`/`canonical_url`/`title`/`post_date`/`audience`/`wordcount` all on 100% of items, correctly ordered newest-first. A follow-up shadow sync (via the real `sync --force` code path, into throwaway databases, same live session) showed *why*: legacy's own count is unstable run-to-run (985, then 1080 across two consecutive syncs), consistent with a real latent bug — `_fetch_all_saved_via_reader_api()` returns `all_posts if all_posts else None` when a mid-pagination 429 survives every retry, silently returning a partial list with no "this is incomplete" signal, rather than surfacing the truncation. The unified endpoint held steady at 1079 across both the comparison and the shadow sync. `_fetch_all_saved_posts_via_unified_api()` is a structural sibling of `_fetch_all_saved_notes_via_api()` (opaque `nextCursor` pagination, not the legacy `after=<ISO saved_at>` scheme), cached separately in `_unified_api_cache`/`_unified_api_failed` so one source's failure can't poison another's mid-run cache, same as the posts/notes cache split. `parse_remote_post()`'s `saved_at` lookup checks both the item level (legacy) and `post_obj` (unified: `post.saved_at`), since the two APIs nest it differently. The mid-pagination silent-truncation bug is now fixed, and turned out to be shared by all three cursor-paginated fetchers (legacy posts, unified posts, and — pre-existing, since before this session — notes): each still returns `all_posts`/`all_notes` `if ... else None` when a page request fails after every retry (so a persistent 429 doesn't throw away progress already made), but now also sets a per-source instance flag (`_api_truncated` / `_unified_api_truncated` / `_notes_api_truncated`, reset at the top of each fetch and in `reset_cache()`) whenever that partial-but-nonempty return fires. `is_posts_fetch_truncated()`/`is_notes_fetch_truncated()` read whichever cache is currently populated and report its flag (using `getattr()` defaults so a test double that skips `__init__`, the established pattern in this repo's sync tests, safely reports "not truncated" instead of raising). `sync_saved_posts()`/`sync_saved_notes()` check this after the fetch loop: on a `--force` sync, a truncated fetch **skips reconciliation entirely** rather than soft-deleting posts/notes that are merely absent from an incomplete list (they may still be saved remotely — the fetch just couldn't reach them this run), and the returned `SyncRun` gets the previously-unused `status = "partial"` (already documented in the model's comment, never actually emitted before this fix) with an `error_message` explaining what happened. The CLI's `sync` command treats `"partial"` as a qualified success (yellow, not red) — `posts_ok`/`notes_ok` still count it as having run — and prints the warning; `status` needs no change since it already displays whatever `last_sync_status` string is stored, and `get_status()`'s `status = 'success'` filter for "last successful sync" already correctly excludes a `partial` run.
+Saved-posts fetching has a three-source chain, controlled by `get_saved_posts_source()`/`SUBSTACK_SAVED_POSTS_SOURCE` (`auto` default, or a forced `legacy`/`unified`/`dom`): the unified `GET /api/v1/reader/saved?filter=posts` endpoint (the same one notes uses with `filter=notes`) is now tried first, then the legacy reader-posts API above as an automatic fallback, then the DOM fallback. This was a migration decision, not a default guess — see `PLAN.md`'s Decisions Ledger and Evidence sections for the full evidence-gathering and decision-gate process. Two tools were built to make that decision on evidence rather than guesswork: `probe-api URL [--out PATH]` headlessly GETs any known API URL with the saved session (no browser page) and dumps the raw JSON — a lighter complement to `inspect-network`, which is for *discovering* unknown endpoints by watching a live browsing session; and `compare-saved-apis [--out PATH]` fetches the full saved list from both sources in one session and reports counts, URL-set differences, per-field presence on the unified payload, and whether `post.saved_at` is present and ordered. A live run against a real ~1000-post account (after a first attempt was invalidated by rate limiting immediately following a large legacy fetch — `compare-saved-apis` now pauses and retries harder around the second fetch for exactly this reason) found the unified endpoint a **strict superset**: 1079 posts vs. legacy's 985, with only 1 post legacy had that unified lacked, `saved_at`/`id`/`canonical_url`/`title`/`post_date`/`audience`/`wordcount` all on 100% of items, correctly ordered newest-first. A follow-up shadow sync (via the real `sync --force` code path, into throwaway databases, same live session) showed *why*: legacy's own count is unstable run-to-run (985, then 1080 across two consecutive syncs), consistent with a real latent bug — `_fetch_all_saved_via_reader_api()` returns `all_posts if all_posts else None` when a mid-pagination 429 survives every retry, silently returning a partial list with no "this is incomplete" signal, rather than surfacing the truncation. The unified endpoint held steady at 1079 across both the comparison and the shadow sync. `_fetch_all_saved_posts_via_unified_api()` is a structural sibling of `_fetch_all_saved_notes_via_api()` (opaque `nextCursor` pagination, not the legacy `after=<ISO saved_at>` scheme), cached separately in `_unified_api_cache`/`_unified_api_failed` so one source's failure can't poison another's mid-run cache, same as the posts/notes cache split. `parse_remote_post()`'s `saved_at` lookup checks both the item level (legacy) and `post_obj` (unified: `post.saved_at`), since the two APIs nest it differently. The mid-pagination silent-truncation bug is now fixed, and turned out to be shared by all three cursor-paginated fetchers (legacy posts, unified posts, and — pre-existing, since before this session — notes): each still returns `all_posts`/`all_notes` `if ... else None` when a page request fails after every retry (so a persistent 429 doesn't throw away progress already made), but now also sets a per-source instance flag (`_api_truncated` / `_unified_api_truncated` / `_notes_api_truncated`, reset at the top of each fetch and in `reset_cache()`) whenever that partial-but-nonempty return fires. `is_posts_fetch_truncated()`/`is_notes_fetch_truncated()` read whichever cache is currently populated and report its flag (using `getattr()` defaults so a test double that skips `__init__`, the established pattern in this repo's sync tests, safely reports "not truncated" instead of raising). `sync_saved_posts()`/`sync_saved_notes()` check this after the fetch loop: on a `--force` sync, a truncated fetch **skips reconciliation entirely** rather than soft-deleting posts/notes that are merely absent from an incomplete list (they may still be saved remotely — the fetch just couldn't reach them this run), and the returned `SyncRun` gets the previously-unused `status = "partial"` (already documented in the model's comment, never actually emitted before this fix) with an `error_message` explaining what happened. The CLI's `sync` command treats `"partial"` as a qualified success (yellow, not red) — `posts_ok`/`notes_ok` still count it as having run — and prints the warning; `status` needs no change since it already displays whatever `last_sync_status` string is stored, and `get_status()`'s `status = 'success'` filter for "last successful sync" already correctly excludes a `partial` run.
 - **Saved notes** (Substack's short-form posts, distinct from long-form posts) are modeled and synced separately from posts, in their own `notes` table/`SavedNote`/`NoteSummary` models, discovered and confirmed live against a real account rather than guessed. The saved-notes list is `GET https://substack.com/api/v1/reader/saved?filter=notes` (also `filter=all`/`filter=posts` — a newer, unified endpoint Substack has added alongside the legacy posts-only `/api/v1/reader/posts?inboxType=saved` that `_fetch_all_saved_via_reader_api()` still uses; posts intentionally were not migrated to it). Its response is `{"items": [...], "nextCursor": <opaque base64 JSON token> | null}`; pagination resubmits the server's own `nextCursor` value as a `cursor=` query param rather than the `after=<ISO timestamp>` scheme posts uses, and the `limit=` query param has no effect (the server returns a fixed page size), so `_fetch_all_saved_notes_via_api()` is a standalone paginator, not a generalization of `_fetch_all_saved_via_reader_api()`. Each item is `{"entity_key": "c-<id>", "publication": null|{...}, "post": null|{...}, "comment": {...}}`; a restack is represented by the top-level `publication`/`post` fields being populated instead of null (unconfirmed by a live example in the account used to discover this, but structurally solid since that item shape is shared with `filter=posts`). Critically, **the endpoint never exposes a bookmark timestamp** — only `comment.is_saved: bool` — so `SavedNote.saved_at` is always `None`, and `sync_saved_notes()`'s incremental early-stop check compares "already saved locally" (`existing.is_saved == 1`) rather than a `saved_at` match the way `sync_saved_posts()` does. A note's identity is its numeric `comment.id`, which is also embedded in its permalink (`https://substack.com/@<handle>/note/c-<id>`, confirmed live via click-tracking) — `_extract_note_id()` parses it back out of a URL via regex, and `upsert_note()`/`reconcile_unsaved_notes()`/`get_note_by_substack_id()` key on that id (not URL, unlike posts) since a note may lack a stable permalink. `comment.body` is already Substack-flattened plain text; `comment.body_json` is a ProseMirror doc (`{type:"doc", content:[{type:"paragraph"|"orderedList"|"listItem",...}]}`, text nodes carrying `marks:[{type:"bold"|"italic"|"link"}]`) that `content_utils.note_body_to_text()`/`prosemirror_to_llm_text()` convert to the same `**bold**`/`*italic*`/`text (url)` conventions `html_to_llm_text()` uses for posts; `body_raw` stores the source verbatim as a hedge against the body-format guess needing correction later. The individual-note-fetch endpoint `GET /api/v1/reader/comment/{id}` → `{"item": {"comment": {...}}}` and the bookmark endpoints `POST`/`DELETE https://substack.com/api/v1/note/c-{id}/save` (id in the URL path, no request body) are all plain authenticated calls through Playwright's `p.request` API context — **no browser page or `window._preloads` lookup is needed for notes at all**, unlike posts, so `save_note()`/`unsave_note()`/`fetch_note_content()` in `substack_client.py` never launch `p.chromium`. Notes have no DOM fallback and no `_click_bookmark_toggle`-style guesswork: the notes card markup is uncaptured, and since the note id is the identity, a DOM-scraped row without one could never reconcile, so an unavailable notes API raises `SubstackClientError` naming `inspect-network` rather than degrading silently. `sync_runs.entity` (`'post'` | `'note'`, default `'post'` for pre-existing rows) scopes `get_status()`'s "last sync" queries per entity so a notes sync can't be reported as the posts' last sync result.
 - `content_utils.py` converts a post's raw `body_html` into clean text for LLM consumption. `html_to_llm_text()` is a small `html.parser.HTMLParser` subclass (no external HTML library dependency) that renders headings as markdown `#` prefixes, list items as `- ` bullets, links as `text (url)`, keeps minimal `**bold**`/`*italic*` markers, drops `script`/`style`/`iframe` content entirely, and collapses excess blank lines. `format_post_for_llm()` prepends a plain metadata header (title, publication, author, published date, URL) to the cleaned body text — this combined string is what both the CLI's `get-content` command and the MCP `get_post_content` tool return. `note_body_to_text()` handles the same problem for notes' `body_json`/`body`, dispatching a dict (or JSON string) to `prosemirror_to_llm_text()` (a small recursive walker over Substack's ProseMirror doc shape — `paragraph`/`heading`/`bulletList`/`orderedList`/`listItem`/`hard_break` nodes, `bold`/`italic`/`link` marks — producing the same `**bold**`/`*italic*`/`text (url)` conventions as `html_to_llm_text()`) and a plain string to either the HTML converter (if it looks like HTML) or straight through; `format_note_for_llm()` builds the note-specific header (author/handle/posted date/URL, plus a `Restacked: <title> (<url>)` line when applicable) rather than reusing `format_post_for_llm()`, since `publication_name` is required there but meaningless for most notes.
 - `sync.py` converts API/DOM payloads to `SavedPost` models and coordinates paginated incremental or full syncs. `parse_remote_post()` routes DOM dicts (identified by the `_dom` marker) through a minimal mapping and everything else through the full reader-API/legacy mapping; it never fabricates a `saved_at` from the sync moment (unknown save times stay `None`). `word_count` is mapped from `wordcount`/`word_count`/`words` via `_first_positive_int()` (which ignores non-positive/uncoercible values); the `compare-saved-apis` parity run confirmed the real field is `wordcount` (100% populated on the unified endpoint's payload, 0% on `word_count`/`words`) — the defensive multi-key mapping is kept since `_first_positive_int()` already checks `wordcount` first and the other two cost nothing to keep as a hedge. `reading_time_minutes` is intentionally *derived* (ceil of `word_count / WORDS_PER_MINUTE`, ~200 wpm) rather than mapped from a field, because a wrong guess about that field's unit (seconds vs minutes) would persist a badly wrong value; deriving is unit-unambiguous. `image_url` is mapped from `cover_image`/`image_url` on the reader-API payload and is populated for the large majority of synced posts (confirmed live: ~98%); it was previously stored but invisible through every read path except `get_saved_post`/the full `SavedPost` — `PostSummary` and the `list_posts()`/`search_posts()` SELECTs now include it too; the CLI `search` command prints it when present, but `list` deliberately omits it since it's a long, uninformative CDN URL in that terser view. `content_text` remains unpopulated by sync (only `get-content`/`get_post_content` populate it) because the saved-list payload doesn't carry post body HTML, only individual post pages do. There is no `metadata_json` column: it was a never-populated "raw source JSON" placeholder from the original PLAN and has been removed from the model and schema. Existing databases created before its removal may still have an inert `metadata_json` column; reads tolerate it because `SavedPost` (Pydantic) ignores unknown columns from `SELECT *`, and `upsert_post` no longer references it. A force/full sync (`--force`) collects every fetched post's URL and, after the fetch loop completes, calls `reconcile_unsaved_posts()` to soft-delete any locally `is_saved = 1` post absent from that complete remote set — this is how posts unsaved directly on Substack (outside this tool) get reflected locally. An incremental sync never reconciles, since its early-stop-on-matches optimization means it only sees a partial remote list. It records each run in SQLite (including `reconciled_count`) and returns a `SyncRun` with `success`, `auth_required`, or `failed` status instead of propagating expected authentication failures. `sync_saved_notes()` is a sibling of `sync_saved_posts()`, not a generalization of it (notes reconcile by Substack note id via `reconcile_unsaved_notes()`, not URL, and the incremental early-stop check compares `existing.is_saved == 1` rather than a `saved_at` match, since notes carry none); both write `sync_runs.entity` (`'post'`/`'note'`) and share a `_build_sync_run()` helper instead of each hand-constructing `SyncRun` three times; `_build_sync_run()` takes the real `started_at` (captured once at the top of each sync function, right before `start_sync_run()`) rather than stamping it at finish time as all three constructions originally did, which had made every returned `SyncRun.started_at` equal to `completed_at` regardless of how long the sync actually took. `parse_remote_note()` unwraps a saved-notes list item (or a single-note-fetch `{"item": {...}}` wrapper) into a `SavedNote`, returning `None` (so the sync loop skips it) rather than inventing an id when `comment.id` is missing; it delegates body conversion to `content_utils.note_body_to_text()` and derives `word_count` from the resulting text rather than any source field.
 - `database.py` is the SQLite repository and schema owner. The `posts` table is the cache; `sync_runs` records sync history; `posts_fts` is an external-content FTS5 index maintained by database triggers. `reconcile_unsaved_posts()` bulk soft-deletes posts missing from a given complete remote URL set and is a no-op on an empty list (an empty list is more likely a fetch problem than genuine mass-unsaving). `init_db()` runs additive column migrations (e.g. `posts.audience`, `sync_runs.reconciled_count`) via `ALTER TABLE` guarded by an existence check or `try`/`except sqlite3.OperationalError`, since `CREATE TABLE IF NOT EXISTS` never adds columns to an already-existing table; the `posts.audience` check runs before the `executescript` block because the `idx_posts_audience` index creation inside it would otherwise fail on a pre-migration table. `list_posts()`/`search_posts()` accept an `audience` filter (exact match, case-insensitive), and `list_audiences()` returns the distinct audience values actually present in the cache with post counts, rather than hardcoding Substack's (undocumented, possibly-growing) audience enum. `PostSummary` (returned by `list_posts()`/`search_posts()` and the MCP list/search tools) carries `reading_time_minutes` and `word_count`, and the CLI `list`/`search` commands display them — without this the values were populated in the `posts` table but invisible through every read path except `get_saved_post`/the full `SavedPost`. A parallel `notes` table (with its own `notes_fts` external-content index and `notes_ai`/`notes_ad`/`notes_au` triggers) holds saved notes; unlike `posts`, its unique identity column is `substack_note_id`, not `url` (a note may have no stable permalink), so `upsert_note()` does an explicit lookup-then-branch UPDATE/INSERT on `substack_note_id` rather than `INSERT ... ON CONFLICT(url)`. `upsert_post()` was originally the opposite (looked up by `substack_post_id` *or* `url` but written via `INSERT ... ON CONFLICT(url)`), which meant a post found by id whose canonical URL had since changed (slug rename, custom-domain migration) couldn't conflict on the new URL and tripped the `substack_post_id UNIQUE` constraint on the fresh INSERT instead; it now uses the same explicit lookup-then-branch UPDATE/INSERT pattern as `upsert_note()`, closing that crash path. `search_posts()`'s LIKE fallback (used when an FTS5 query string is malformed) shares its WHERE-clause construction with the main FTS branch via `_post_search_filters()` — mirroring `_note_search_filters()` — so a malformed query never silently drops the `publication`/`audience`/date-range filters or an `is_saved_only=False` caller the way the fallback originally did. `get_note_by_substack_id()` exists because `get_note()`'s polymorphic `url_or_id` dispatch treats any all-digit string as a local row id — and `substack_note_id` values are always digits — so callers that only have the Substack note id (the sync loop's incremental check) must use it instead of risking a collision with an unrelated local row. `sync_runs.entity` (`'post'`/`'note'`, additive column defaulting existing rows to `'post'`) scopes `get_status()`'s "last successful sync"/"last sync status" queries per entity, which is the one existing posts read path notes support required changing (it previously read the last `sync_runs` row unscoped, which was harmless with one entity and would otherwise let a notes sync be reported as the posts' last sync).
-- `config.py` centralizes application paths. By default, data lives under `~/.local/share/substack-saved-mcp`; `SUBSTACK_SAVED_DB_PATH`, `SUBSTACK_SAVED_DATA_DIR`, and `SUBSTACK_SAVED_BROWSER_DIR` override those paths. `get_saved_posts_source()`/`SUBSTACK_SAVED_POSTS_SOURCE` (`auto`/`legacy`/`unified`/`dom`, falling back to `auto` on an unrecognized value) selects which of the three saved-posts fetch sources `_fetch_saved_posts_page_impl()` tries, so a source-specific regression is one env var away from a revert rather than a code rollback.
+- `config.py` centralizes application paths. By default, data lives under `~/.local/share/substack-saved-mcp`; `SUBSTACK_SAVED_DB_PATH`, `SUBSTACK_SAVED_DATA_DIR`, and `SUBSTACK_SAVED_BROWSER_DIR` override those paths. `get_saved_posts_source()`/`SUBSTACK_SAVED_POSTS_SOURCE` (`auto`/`legacy`/`unified`/`dom`, falling back to `auto` on an unrecognized value) selects which of the three saved-posts fetch sources `_fetch_saved_posts_page_impl()` tries, so a source-specific regression is one env var away from a revert rather than a code rollback. `get_fully_read_threshold()`/`SUBSTACK_SAVED_FULLY_READ_THRESHOLD` (default `0.95`, falling back to the default on an unparseable or out-of-`(0, 1]` value) sets the `max_read_progress` fraction at or above which a post counts as fully read.
+- **Reading progress.** Substack exposes per-post reading progress (visible in its mobile apps, never the web UI) through the same saved-posts payloads this project already fetches: `post.read_progress` (current scroll position, 0.0–1.0), `post.max_read_progress` (high-water mark, 0.0–1.0), and `post.is_viewed` (bool) — confirmed live on both the unified and legacy reader APIs (identically mirrored under `post.inboxItem`, so that nested object is never read), absent from the DOM fallback. `parse_remote_post()` maps all three off `post_obj` via a new `_float_or_none()` helper (clamped to `[0.0, 1.0]`, tolerant of a real `0.0` unlike `_first_positive_int()`). The three raw values are stored as columns (`posts.read_progress`/`max_read_progress REAL`, `posts.is_viewed INTEGER NOT NULL DEFAULT 0`); `is_fully_read` and `minutes_remaining` are deliberately **not** columns — they're Pydantic `@computed_field` properties on `SavedPost`/`PostSummary`, derived from `max_read_progress` (the high-water mark, not the current position, since that's what "have I read this" means) and `word_count` at read time, same rationale as `reading_time_minutes`: real observed values like `0.9999`/`0.9867`/`0.9822` rule out an exact `== 1.0` test, and a stored derivative would go stale the moment the threshold changed. `database.py`'s `_READ_STATE_PREDICATES`/`_read_state_clause()` classify a post into `unread`/`in_progress`/`finished`/`started` via SQL predicates wrapping every comparison in `COALESCE(..., 0)` — load-bearing, not defensive styling: after the additive migration every pre-existing row is `NULL`, and `max_read_progress = 0` would otherwise evaluate to `NULL` (not true) and silently exclude the entire legacy cache from every read_state filter. `list_posts()`/`search_posts()`/`_post_search_filters()` accept `read_state`, threaded through both the FTS and LIKE-fallback branches of `search_posts()` the same way `audience` already was; `list_posts()`'s sort-key whitelist also gained `read_progress` and a `minutes_remaining` SQL expression (`COALESCE(word_count,0) * (1 - COALESCE(max_read_progress,0))`) so a caller can find something short to finish. `upsert_post()` coalesces these three fields with explicit `is not None` (`is_viewed`: truthy-sticky) checks rather than the pre-existing `or`-based idiom used for other optional fields, since a real `0.0` is falsy and the `or` pattern would have silently discarded it in favor of a stale stored value — a DOM-sourced or otherwise progress-free payload (e.g. `_save_post_impl`'s construction site) preserves whatever progress is already stored rather than clobbering it. `get_status()`/`SavedPostsStatus` gained `posts_unread`/`posts_in_progress`/`posts_fully_read`/`minutes_remaining_total` aggregates built from the same predicates and remaining-words expression. Progress refreshes whenever a post is re-fetched during sync; since incremental sync only touches recently-saved posts (its early-stop fires after `MAX_CONSECUTIVE_MATCHES` matches), older posts' progress goes stale between full syncs — `sync --force` is the documented way to refresh it for the whole backlog, and no dedicated refresh mode was added.
 
 ## Repository Conventions
 

{substack_saved_mcp-0.3.0 → substack_saved_mcp-0.3.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: substack-saved-mcp
-Version: 0.3.0
+Version: 0.3.2
 Summary: Local stdio-based MCP server and sync engine for Substack saved posts
 Author-email: Toni Hermoso Pulido <toniher@cau.cat>
 License-Expression: MIT
@@ -11,7 +11,6 @@ Requires-Dist: fastmcp>=0.1.0
 Requires-Dist: playwright>=1.40.0
 Requires-Dist: pydantic>=2.0.0
 Provides-Extra: dev
-Requires-Dist: pytest-asyncio>=0.23.0; extra == 'dev'
 Requires-Dist: pytest>=8.0.0; extra == 'dev'
 Requires-Dist: ruff==0.16.1; extra == 'dev'
 Description-Content-Type: text/markdown
@@ -24,7 +23,8 @@ A local, stdio-based Model Context Protocol (MCP) server and sync engine for you
 
 ## Features
 
-- **Read & Search**: Full-text search (SQLite FTS5) across saved post titles, excerpts, authors, and publications. Filter by publication, audience tier (e.g. `everyone`, `only_paid`), and date ranges (`published_at` vs `saved_at`). Search also covers a post's **full body text**, but only for posts whose content has already been fetched once via `get-content` / the `get_post_content` tool — a normal `sync` stores metadata and excerpts, not full bodies, so posts you haven't opened yet are matched on their title/excerpt/metadata only, not their full text.
+- **Read & Search**: Full-text search (SQLite FTS5) across saved post titles, excerpts, authors, and publications. Filter by publication, audience tier (e.g. `everyone`, `only_paid`), reading progress (`--read-state unread`/`in_progress`/`finished`/`started`), and date ranges (`published_at` vs `saved_at`). Search also covers a post's **full body text**, but only for posts whose content has already been fetched once via `get-content` / the `get_post_content` tool — a normal `sync` stores metadata and excerpts, not full bodies, so posts you haven't opened yet are matched on their title/excerpt/metadata only, not their full text.
+- **Reading Progress**: Substack tracks how far you've read each saved post (visible in its mobile apps, not the web UI) — this tool surfaces it. Each post reports `is_fully_read` and `minutes_remaining`, derived from the stored `max_read_progress` high-water mark at a configurable threshold (default 0.95, `SUBSTACK_SAVED_FULLY_READ_THRESHOLD`). Filter with `--read-state`, or sort a list by `read_progress`/`minutes_remaining` to find something short to finish. Progress refreshes on every sync a post is touched by; run `sync --force` to refresh it for your whole backlog.
 - **Saved Notes, too**: Substack's short-form notes are synced, searched, and cached separately from posts (they carry an author and body rather than a title or publication tier). Full-text search covers note bodies, authors, and restacked-post titles. Notes never require a browser at all — every notes operation (sync, save, unsave, full-content fetch) is a plain authenticated API call.
 - **Full Content for LLMs**: Fetch a saved post's or note's full content and get it back cleaned and formatted (headings, lists, links) for feeding directly to an LLM, with the result cached locally for next time.
 - **Save & Unsave**: Bookmark new Substack posts and notes, or unbookmark existing ones. Posts go through an authenticated browser session; notes are API-only.
@@ -122,6 +122,11 @@ substack-saved-mcp audiences
 substack-saved-mcp list --audience only_paid
 substack-saved-mcp search "artificial intelligence" --audience everyone
 
+# 4c. Filter or sort by reading progress
+substack-saved-mcp list --read-state finished
+substack-saved-mcp list --read-state in_progress --sort-by minutes_remaining
+substack-saved-mcp search "artificial intelligence" --read-state unread
+
 # 5. Save or unsave a post
 substack-saved-mcp save "https://example.substack.com/p/post-slug"
 substack-saved-mcp unsave "https://example.substack.com/p/post-slug"
@@ -225,6 +230,19 @@ or note that merely couldn't be fetched is never mistaken for one you actually
 unsaved on Substack. Just run `sync` again later — a subsequent successful run
 picks up anything that was missed.
 
+### How is "fully read" determined, and how fresh is it?
+
+Substack reports a `max_read_progress` high-water mark (0.0–1.0) per saved post; a
+post counts as fully read once that crosses a threshold (default `0.95` — real
+posts top out around `0.98`–`0.9999` rather than an exact `1.0`). Override it with:
+```bash
+export SUBSTACK_SAVED_FULLY_READ_THRESHOLD="0.90"
+```
+Progress is refreshed whenever a post is re-fetched during sync. An incremental
+sync only touches recently-saved posts, so progress on older posts in your
+backlog can go stale between reads; run `substack-saved-mcp sync --force` to
+refresh it for everything. Notes have no reading-progress concept.
+
 ### Will a browser window pop up when running as an MCP server?
 
 **No, a visible browser window will not open during normal MCP operations.**