substack-saved-mcp 0.2.0__tar.gz → 0.3.0__tar.gz

{substack_saved_mcp-0.2.0 → substack_saved_mcp-0.3.0}/CLAUDE.md

@@ -21,16 +21,20 @@ uv sync --extra dev
 
 `ruff` is configured in `pyproject.toml` (rules: `E4`, `E7`, `E9`, `F`, `I`, `UP`, `B`, `RUF`, line length 88, target py311). Lint with `uvx ruff check .` or `uv run ruff check .`.
 
+After any change to dependencies or tracked files, run `uv lock` to keep `uv.lock` current, then `prek run --all-files` (using the repo's `.pre-commit-config.yaml`) before considering the work done.
+
 ## Architecture
 
 - `cli.py` is the Click entry point (`substack-saved-mcp`). Its commands initialize the database and then delegate to the repository, sync engine, Playwright client, or MCP server.
 - `mcp_server.py` exposes the same application operations as FastMCP stdio tools and resources. Read tools query the local cache; sync and save/unsave operations use the authenticated browser session.
-- `substack_client.py` owns Playwright authentication and remote Substack interaction. `login` is the only intended headful workflow; normal sync and write paths use `storage_state.json` headlessly. Synchronous Playwright API calls are routed through `_run_playwright_sync()` to safely execute in a worker thread if an `asyncio` event loop is active (e.g. under FastMCP). Saved-post fetching prefers the reader inbox API (`GET /api/v1/reader/posts?inboxType=saved`), which exposes the real bookmark timestamp (`saved_at`) and an ISO publication date (`post_date`) per post. `_fetch_all_saved_via_reader_api()` cursor-paginates that endpoint (each page's oldest `saved_at` becomes the next `after=` cursor, until `more` is false), dedupes by canonical URL, and enriches each flat post with its `publication` object (from the response's `publications` array, matched by `publication_id`) and `author_name` (from `publishedBylines`). The full result is cached in `_api_cache` and sliced by offset; `_fetch_saved_posts_page_impl()` accepts an optional `playwright_instance` (same pattern as the notes fetcher and `_fetch_via_dom`) so this caching/slicing/fallback branch is directly testable with a Playwright double instead of only reachable through a real browser. Each page request goes through `_reader_api_get()`, which retries transient failures (HTTP 429 and 5xx) up to `max_retries` (default 3) with `Retry-After`-aware backoff: `_retry_after_seconds()` honors an integer `Retry-After` header (clamped to a 30s cap so a hostile value can't hang the sync) and otherwise uses capped exponential backoff (0.5s, 1s, 2s, ...); 401/403 and other 4xx are returned unretried so the existing auth/fallback handling applies. A 429 that survives all retries is deliberately treated as "unavailable" (partial list, or `None` → DOM fallback) rather than as a silent empty-success, so rate-limiting never masquerades as "you have no saved posts." Both `_reader_api_get()` and `_fetch_all_saved_via_reader_api()` accept an injectable `sleep_func` (default `time.sleep`) so tests exercise the backoff without real delays. If that endpoint is unavailable, it falls back to headless DOM extraction on `https://substack.com/saved`, which scrapes `div.reader2-post-container` cards, caches scrolling results in `_dom_cache`, and marks each dict with `_dom: True`; DOM cards only expose a localized relative publish string and no bookmark time, so `saved_at` stays unknown on that path. Both `save_post()` and `unsave_post()` were originally guesswork (an unverified DOM button selector plus an unverified `POST /api/v1/bookmark` call) and were fixed once real endpoints were captured via `inspect-network`. Every post page server-renders a `window._preloads` blob containing that post's own numeric ID and rich metadata (`preloads.post.id`, `.title`, `.audience`, `.description`/`.subtitle`, `.post_date`, plus `preloads.pub.name`); `_save_post_impl` reads this via `page.evaluate("() => window._preloads")` right after page load and, when an ID is found, calls the real `POST https://substack.com/api/v1/posts/saved` endpoint (body `{"post_id": ...}`) directly via the Playwright API request context — this also lets it populate the returned `SavedPost` with accurate title/publication/audience/excerpt instead of parsing `page.title()`. `_unsave_post_impl` does the mirror image: when the post's `substack_post_id` is already known (true for any post that has been through a normal `sync`, since the reader API's `id` field populates it), it calls `DELETE https://substack.com/api/v1/posts/saved` with the same body shape, no DOM interaction at all. Both treat an `ok` response as `"confirmed"` and only fall back to the old best-effort DOM click (`_click_bookmark_toggle()`, used when the numeric ID can't be obtained or the direct call doesn't confirm) — note its selector is English-only (`aria-label*='save'/'bookmark'`) and can silently fail on non-English Substack UIs, which is part of why the direct API path is preferred whenever possible. `_click_bookmark_toggle()` fingerprints the button's `aria-label`/`aria-pressed`/`class` before and after the click and returns `"confirmed"` only if that fingerprint changed, else `"unconfirmed"`, `"not_found"`, or `"click_failed"` — `save_post()` returns `(SavedPost, confirmation)` and `unsave_post()` returns just the confirmation string; both accept an optional `playwright_instance` for test injection (same pattern as `_fetch_via_dom`). `fetch_post_content()` reuses the same `window._preloads` mechanism to retrieve a saved post's full content: `_fetch_post_content_impl` navigates to the post's page and reads `preloads.post.body_html` (Substack's field name for full content — `parse_remote_post()` already expects this key from the reader API, though the saved-list payload never actually populates it, only individual post pages do), returning `None` for `body_html` if the page's embed format doesn't expose it (frontend change) or the account lacks paywall access; this is the case where the caller should be told to run `inspect-network` against an open post page to re-discover the real content source. `cli.py`'s `inspect-network` command (which logs any `api/v1`/`bookmark`/`saved`/`notes?`/`comment`/`reader`/`feed`/`restack` request's method, URL, status, and — since a Playwright sync-API response event handler can't safely call `response.text()` inside the handler itself (it can deadlock on the same driver thread) — a JSON response body captured via a guarded `try`/`except`) is the tool for discovering/re-verifying these endpoints when Substack's frontend changes. It supports `--authenticated/--anonymous` (reuses `storage_state.json` via `get_storage_state_path()` so `/saved` renders the real logged-in page instead of the marketing page), `--url` to target any page, `--filter` to override the response-matching regex, `--max-body` to cap logged response bytes, and `--out PATH` to append each exchange as JSON Lines for later grepping.
+- `substack_client.py` owns Playwright authentication and remote Substack interaction. `login` is the only intended headful workflow; normal sync and write paths use `storage_state.json` headlessly. Synchronous Playwright API calls are routed through `_run_playwright_sync()` to safely execute in a worker thread if an `asyncio` event loop is active (e.g. under FastMCP). Saved-post fetching prefers the reader inbox API (`GET /api/v1/reader/posts?inboxType=saved`), which exposes the real bookmark timestamp (`saved_at`) and an ISO publication date (`post_date`) per post. `_fetch_all_saved_via_reader_api()` cursor-paginates that endpoint (each page's oldest `saved_at` becomes the next `after=` cursor, until `more` is false), dedupes by canonical URL, and enriches each flat post with its `publication` object (from the response's `publications` array, matched by `publication_id`) and `author_name` (from `publishedBylines`). The full result is cached in `_api_cache` and sliced by offset; `_fetch_saved_posts_page_impl()` accepts an optional `playwright_instance` (same pattern as the notes fetcher and `_fetch_via_dom`) so this caching/slicing/fallback branch is directly testable with a Playwright double instead of only reachable through a real browser. Each page request goes through `_reader_api_get()`, which retries transient failures (HTTP 429 and 5xx) up to `max_retries` (default 3) with `Retry-After`-aware backoff: `_retry_after_seconds()` honors an integer `Retry-After` header (clamped to a 30s cap so a hostile value can't hang the sync) and otherwise uses capped exponential backoff (0.5s, 1s, 2s, ...); 401/403 and other 4xx are returned unretried so the existing auth/fallback handling applies. A 429 that survives all retries is deliberately treated as "unavailable" (partial list, or `None` → DOM fallback) rather than as a silent empty-success, so rate-limiting never masquerades as "you have no saved posts." Both `_reader_api_get()` and `_fetch_all_saved_via_reader_api()` accept an injectable `sleep_func` (default `time.sleep`) so tests exercise the backoff without real delays. If that endpoint is unavailable, it falls back to headless DOM extraction on `https://substack.com/saved`, which scrapes `div.reader2-post-container` cards, caches scrolling results in `_dom_cache`, and marks each dict with `_dom: True`; DOM cards only expose a localized relative publish string and no bookmark time, so `saved_at` stays unknown on that path. Both `save_post()` and `unsave_post()` were originally guesswork (an unverified DOM button selector plus an unverified `POST /api/v1/bookmark` call) and were fixed once real endpoints were captured via `inspect-network`. Every post page server-renders a `window._preloads` blob containing that post's own numeric ID and rich metadata (`preloads.post.id`, `.title`, `.audience`, `.description`/`.subtitle`, `.post_date`, plus `preloads.pub.name`); `_save_post_impl` reads this via `page.evaluate("() => window._preloads")` right after page load and, when an ID is found, calls the real `POST https://substack.com/api/v1/posts/saved` endpoint (body `{"post_id": ...}`) directly via the Playwright API request context — this also lets it populate the returned `SavedPost` with accurate title/publication/audience/excerpt instead of parsing `page.title()`. `_unsave_post_impl` does the mirror image: when the post's `substack_post_id` is already known (true for any post that has been through a normal `sync`, since the reader API's `id` field populates it), it calls `DELETE https://substack.com/api/v1/posts/saved` with the same body shape, no DOM interaction at all. Both treat an `ok` response as `"confirmed"` and only fall back to the old best-effort DOM click (`_click_bookmark_toggle()`, used when the numeric ID can't be obtained or the direct call doesn't confirm) — note its selector is English-only (`aria-label*='save'/'bookmark'`) and can silently fail on non-English Substack UIs, which is part of why the direct API path is preferred whenever possible. `_click_bookmark_toggle()` fingerprints the button's `aria-label`/`aria-pressed`/`class` before and after the click and returns `"confirmed"` only if that fingerprint changed, else `"unconfirmed"`, `"not_found"`, or `"click_failed"` — `save_post()` returns `(SavedPost, confirmation)` and `unsave_post()` returns just the confirmation string; both accept an optional `playwright_instance` for test injection (same pattern as `_fetch_via_dom`). `fetch_post_content()` reuses the same `window._preloads` mechanism to retrieve a saved post's full content: `_fetch_post_content_impl` navigates to the post's page and reads `preloads.post.body_html` (Substack's field name for full content — `parse_remote_post()` already expects this key from the reader API, though the saved-list payload never actually populates it, only individual post pages do), returning `None` for `body_html` if the page's embed format doesn't expose it (frontend change) or the account lacks paywall access; this is the case where the caller should be told to run `inspect-network` against an open post page to re-discover the real content source. `cli.py`'s `inspect-network` command (which logs any `api/v1`/`bookmark`/`saved`/`notes?`/`comment`/`reader`/`feed`/`restack` request's method, URL, status, and JSON response body) is the tool for discovering/re-verifying these endpoints when Substack's frontend changes. It supports `--authenticated/--anonymous` (reuses `storage_state.json` via `get_storage_state_path()` so `/saved` renders the real logged-in page instead of the marketing page), `--url` to target any page, `--filter` to override the response-matching regex, `--max-body` to cap logged response bytes, and `--out PATH` to append each exchange as JSON Lines for later grepping. Response bodies are captured via `context.route("**/*", handle_route)` + `route.fetch()`/`route.fulfill()`, not the more obvious `page.on("response", ...)` + `response.text()`: the latter can deadlock calling `response.text()` inside a sync-API response event handler on the same driver thread, and in practice silently produced zero captured bodies across an entire 108-request session before this fix (every `response_body` came back `null` with no error surfaced) — `route.fetch()` reads the body outside that handler and `route.fulfill(response=...)` re-serves the exact response so page behavior is unaffected; a body that still can't be read now logs a visible yellow warning instead of writing `null` silently.
+
+Saved-posts fetching has a three-source chain, controlled by `get_saved_posts_source()`/`SUBSTACK_SAVED_POSTS_SOURCE` (`auto` default, or a forced `legacy`/`unified`/`dom`): the unified `GET /api/v1/reader/saved?filter=posts` endpoint (the same one notes uses with `filter=notes`) is now tried first, then the legacy reader-posts API above as an automatic fallback, then the DOM fallback. This was a migration decision, not a default guess — see `PLAN-newapi.md` for the full evidence-gathering and decision-gate process. Two tools were built to make that decision on evidence rather than guesswork: `probe-api URL [--out PATH]` headlessly GETs any known API URL with the saved session (no browser page) and dumps the raw JSON — a lighter complement to `inspect-network`, which is for *discovering* unknown endpoints by watching a live browsing session; and `compare-saved-apis [--out PATH]` fetches the full saved list from both sources in one session and reports counts, URL-set differences, per-field presence on the unified payload, and whether `post.saved_at` is present and ordered. A live run against a real ~1000-post account (after a first attempt was invalidated by rate limiting immediately following a large legacy fetch — `compare-saved-apis` now pauses and retries harder around the second fetch for exactly this reason) found the unified endpoint a **strict superset**: 1079 posts vs. legacy's 985, with only 1 post legacy had that unified lacked, `saved_at`/`id`/`canonical_url`/`title`/`post_date`/`audience`/`wordcount` all on 100% of items, correctly ordered newest-first. A follow-up shadow sync (via the real `sync --force` code path, into throwaway databases, same live session) showed *why*: legacy's own count is unstable run-to-run (985, then 1080 across two consecutive syncs), consistent with a real latent bug — `_fetch_all_saved_via_reader_api()` returns `all_posts if all_posts else None` when a mid-pagination 429 survives every retry, silently returning a partial list with no "this is incomplete" signal, rather than surfacing the truncation. The unified endpoint held steady at 1079 across both the comparison and the shadow sync. `_fetch_all_saved_posts_via_unified_api()` is a structural sibling of `_fetch_all_saved_notes_via_api()` (opaque `nextCursor` pagination, not the legacy `after=<ISO saved_at>` scheme), cached separately in `_unified_api_cache`/`_unified_api_failed` so one source's failure can't poison another's mid-run cache, same as the posts/notes cache split. `parse_remote_post()`'s `saved_at` lookup checks both the item level (legacy) and `post_obj` (unified: `post.saved_at`), since the two APIs nest it differently. The mid-pagination silent-truncation bug is now fixed, and turned out to be shared by all three cursor-paginated fetchers (legacy posts, unified posts, and — pre-existing, since before this session — notes): each still returns `all_posts`/`all_notes` `if ... else None` when a page request fails after every retry (so a persistent 429 doesn't throw away progress already made), but now also sets a per-source instance flag (`_api_truncated` / `_unified_api_truncated` / `_notes_api_truncated`, reset at the top of each fetch and in `reset_cache()`) whenever that partial-but-nonempty return fires. `is_posts_fetch_truncated()`/`is_notes_fetch_truncated()` read whichever cache is currently populated and report its flag (using `getattr()` defaults so a test double that skips `__init__`, the established pattern in this repo's sync tests, safely reports "not truncated" instead of raising). `sync_saved_posts()`/`sync_saved_notes()` check this after the fetch loop: on a `--force` sync, a truncated fetch **skips reconciliation entirely** rather than soft-deleting posts/notes that are merely absent from an incomplete list (they may still be saved remotely — the fetch just couldn't reach them this run), and the returned `SyncRun` gets the previously-unused `status = "partial"` (already documented in the model's comment, never actually emitted before this fix) with an `error_message` explaining what happened. The CLI's `sync` command treats `"partial"` as a qualified success (yellow, not red) — `posts_ok`/`notes_ok` still count it as having run — and prints the warning; `status` needs no change since it already displays whatever `last_sync_status` string is stored, and `get_status()`'s `status = 'success'` filter for "last successful sync" already correctly excludes a `partial` run.
 - **Saved notes** (Substack's short-form posts, distinct from long-form posts) are modeled and synced separately from posts, in their own `notes` table/`SavedNote`/`NoteSummary` models, discovered and confirmed live against a real account rather than guessed. The saved-notes list is `GET https://substack.com/api/v1/reader/saved?filter=notes` (also `filter=all`/`filter=posts` — a newer, unified endpoint Substack has added alongside the legacy posts-only `/api/v1/reader/posts?inboxType=saved` that `_fetch_all_saved_via_reader_api()` still uses; posts intentionally were not migrated to it). Its response is `{"items": [...], "nextCursor": <opaque base64 JSON token> | null}`; pagination resubmits the server's own `nextCursor` value as a `cursor=` query param rather than the `after=<ISO timestamp>` scheme posts uses, and the `limit=` query param has no effect (the server returns a fixed page size), so `_fetch_all_saved_notes_via_api()` is a standalone paginator, not a generalization of `_fetch_all_saved_via_reader_api()`. Each item is `{"entity_key": "c-<id>", "publication": null|{...}, "post": null|{...}, "comment": {...}}`; a restack is represented by the top-level `publication`/`post` fields being populated instead of null (unconfirmed by a live example in the account used to discover this, but structurally solid since that item shape is shared with `filter=posts`). Critically, **the endpoint never exposes a bookmark timestamp** — only `comment.is_saved: bool` — so `SavedNote.saved_at` is always `None`, and `sync_saved_notes()`'s incremental early-stop check compares "already saved locally" (`existing.is_saved == 1`) rather than a `saved_at` match the way `sync_saved_posts()` does. A note's identity is its numeric `comment.id`, which is also embedded in its permalink (`https://substack.com/@<handle>/note/c-<id>`, confirmed live via click-tracking) — `_extract_note_id()` parses it back out of a URL via regex, and `upsert_note()`/`reconcile_unsaved_notes()`/`get_note_by_substack_id()` key on that id (not URL, unlike posts) since a note may lack a stable permalink. `comment.body` is already Substack-flattened plain text; `comment.body_json` is a ProseMirror doc (`{type:"doc", content:[{type:"paragraph"|"orderedList"|"listItem",...}]}`, text nodes carrying `marks:[{type:"bold"|"italic"|"link"}]`) that `content_utils.note_body_to_text()`/`prosemirror_to_llm_text()` convert to the same `**bold**`/`*italic*`/`text (url)` conventions `html_to_llm_text()` uses for posts; `body_raw` stores the source verbatim as a hedge against the body-format guess needing correction later. The individual-note-fetch endpoint `GET /api/v1/reader/comment/{id}` → `{"item": {"comment": {...}}}` and the bookmark endpoints `POST`/`DELETE https://substack.com/api/v1/note/c-{id}/save` (id in the URL path, no request body) are all plain authenticated calls through Playwright's `p.request` API context — **no browser page or `window._preloads` lookup is needed for notes at all**, unlike posts, so `save_note()`/`unsave_note()`/`fetch_note_content()` in `substack_client.py` never launch `p.chromium`. Notes have no DOM fallback and no `_click_bookmark_toggle`-style guesswork: the notes card markup is uncaptured, and since the note id is the identity, a DOM-scraped row without one could never reconcile, so an unavailable notes API raises `SubstackClientError` naming `inspect-network` rather than degrading silently. `sync_runs.entity` (`'post'` | `'note'`, default `'post'` for pre-existing rows) scopes `get_status()`'s "last sync" queries per entity so a notes sync can't be reported as the posts' last sync result.
 - `content_utils.py` converts a post's raw `body_html` into clean text for LLM consumption. `html_to_llm_text()` is a small `html.parser.HTMLParser` subclass (no external HTML library dependency) that renders headings as markdown `#` prefixes, list items as `- ` bullets, links as `text (url)`, keeps minimal `**bold**`/`*italic*` markers, drops `script`/`style`/`iframe` content entirely, and collapses excess blank lines. `format_post_for_llm()` prepends a plain metadata header (title, publication, author, published date, URL) to the cleaned body text — this combined string is what both the CLI's `get-content` command and the MCP `get_post_content` tool return. `note_body_to_text()` handles the same problem for notes' `body_json`/`body`, dispatching a dict (or JSON string) to `prosemirror_to_llm_text()` (a small recursive walker over Substack's ProseMirror doc shape — `paragraph`/`heading`/`bulletList`/`orderedList`/`listItem`/`hard_break` nodes, `bold`/`italic`/`link` marks — producing the same `**bold**`/`*italic*`/`text (url)` conventions as `html_to_llm_text()`) and a plain string to either the HTML converter (if it looks like HTML) or straight through; `format_note_for_llm()` builds the note-specific header (author/handle/posted date/URL, plus a `Restacked: <title> (<url>)` line when applicable) rather than reusing `format_post_for_llm()`, since `publication_name` is required there but meaningless for most notes.
-- `sync.py` converts API/DOM payloads to `SavedPost` models and coordinates paginated incremental or full syncs. `parse_remote_post()` routes DOM dicts (identified by the `_dom` marker) through a minimal mapping and everything else through the full reader-API/legacy mapping; it never fabricates a `saved_at` from the sync moment (unknown save times stay `None`). `word_count` is mapped defensively from several unconfirmed candidate keys (`wordcount`/`word_count`/`words`, via `_first_positive_int()` which ignores non-positive/uncoercible values) — the exact reader-API field name hasn't been captured, so if these stay empty an `inspect-network` capture is needed to find the real one. `reading_time_minutes` is intentionally *derived* (ceil of `word_count / WORDS_PER_MINUTE`, ~200 wpm) rather than mapped from a field, because a wrong guess about that field's unit (seconds vs minutes) would persist a badly wrong value; deriving is unit-unambiguous. `image_url` is mapped from `cover_image`/`image_url` on the reader-API payload and is populated for the large majority of synced posts (confirmed live: ~98%); it was previously stored but invisible through every read path except `get_saved_post`/the full `SavedPost` — `PostSummary` and the `list_posts()`/`search_posts()` SELECTs now include it too; the CLI `search` command prints it when present, but `list` deliberately omits it since it's a long, uninformative CDN URL in that terser view. `content_text` remains unpopulated by sync (only `get-content`/`get_post_content` populate it) because the saved-list payload doesn't carry post body HTML, only individual post pages do. There is no `metadata_json` column: it was a never-populated "raw source JSON" placeholder from the original PLAN and has been removed from the model and schema. Existing databases created before its removal may still have an inert `metadata_json` column; reads tolerate it because `SavedPost` (Pydantic) ignores unknown columns from `SELECT *`, and `upsert_post` no longer references it. A force/full sync (`--force`) collects every fetched post's URL and, after the fetch loop completes, calls `reconcile_unsaved_posts()` to soft-delete any locally `is_saved = 1` post absent from that complete remote set — this is how posts unsaved directly on Substack (outside this tool) get reflected locally. An incremental sync never reconciles, since its early-stop-on-matches optimization means it only sees a partial remote list. It records each run in SQLite (including `reconciled_count`) and returns a `SyncRun` with `success`, `auth_required`, or `failed` status instead of propagating expected authentication failures. `sync_saved_notes()` is a sibling of `sync_saved_posts()`, not a generalization of it (notes reconcile by Substack note id via `reconcile_unsaved_notes()`, not URL, and the incremental early-stop check compares `existing.is_saved == 1` rather than a `saved_at` match, since notes carry none); both write `sync_runs.entity` (`'post'`/`'note'`) and share a `_build_sync_run()` helper instead of each hand-constructing `SyncRun` three times; `_build_sync_run()` takes the real `started_at` (captured once at the top of each sync function, right before `start_sync_run()`) rather than stamping it at finish time as all three constructions originally did, which had made every returned `SyncRun.started_at` equal to `completed_at` regardless of how long the sync actually took. `parse_remote_note()` unwraps a saved-notes list item (or a single-note-fetch `{"item": {...}}` wrapper) into a `SavedNote`, returning `None` (so the sync loop skips it) rather than inventing an id when `comment.id` is missing; it delegates body conversion to `content_utils.note_body_to_text()` and derives `word_count` from the resulting text rather than any source field.
+- `sync.py` converts API/DOM payloads to `SavedPost` models and coordinates paginated incremental or full syncs. `parse_remote_post()` routes DOM dicts (identified by the `_dom` marker) through a minimal mapping and everything else through the full reader-API/legacy mapping; it never fabricates a `saved_at` from the sync moment (unknown save times stay `None`). `word_count` is mapped from `wordcount`/`word_count`/`words` via `_first_positive_int()` (which ignores non-positive/uncoercible values); the `compare-saved-apis` parity run confirmed the real field is `wordcount` (100% populated on the unified endpoint's payload, 0% on `word_count`/`words`) — the defensive multi-key mapping is kept since `_first_positive_int()` already checks `wordcount` first and the other two cost nothing to keep as a hedge. `reading_time_minutes` is intentionally *derived* (ceil of `word_count / WORDS_PER_MINUTE`, ~200 wpm) rather than mapped from a field, because a wrong guess about that field's unit (seconds vs minutes) would persist a badly wrong value; deriving is unit-unambiguous. `image_url` is mapped from `cover_image`/`image_url` on the reader-API payload and is populated for the large majority of synced posts (confirmed live: ~98%); it was previously stored but invisible through every read path except `get_saved_post`/the full `SavedPost` — `PostSummary` and the `list_posts()`/`search_posts()` SELECTs now include it too; the CLI `search` command prints it when present, but `list` deliberately omits it since it's a long, uninformative CDN URL in that terser view. `content_text` remains unpopulated by sync (only `get-content`/`get_post_content` populate it) because the saved-list payload doesn't carry post body HTML, only individual post pages do. There is no `metadata_json` column: it was a never-populated "raw source JSON" placeholder from the original PLAN and has been removed from the model and schema. Existing databases created before its removal may still have an inert `metadata_json` column; reads tolerate it because `SavedPost` (Pydantic) ignores unknown columns from `SELECT *`, and `upsert_post` no longer references it. A force/full sync (`--force`) collects every fetched post's URL and, after the fetch loop completes, calls `reconcile_unsaved_posts()` to soft-delete any locally `is_saved = 1` post absent from that complete remote set — this is how posts unsaved directly on Substack (outside this tool) get reflected locally. An incremental sync never reconciles, since its early-stop-on-matches optimization means it only sees a partial remote list. It records each run in SQLite (including `reconciled_count`) and returns a `SyncRun` with `success`, `auth_required`, or `failed` status instead of propagating expected authentication failures. `sync_saved_notes()` is a sibling of `sync_saved_posts()`, not a generalization of it (notes reconcile by Substack note id via `reconcile_unsaved_notes()`, not URL, and the incremental early-stop check compares `existing.is_saved == 1` rather than a `saved_at` match, since notes carry none); both write `sync_runs.entity` (`'post'`/`'note'`) and share a `_build_sync_run()` helper instead of each hand-constructing `SyncRun` three times; `_build_sync_run()` takes the real `started_at` (captured once at the top of each sync function, right before `start_sync_run()`) rather than stamping it at finish time as all three constructions originally did, which had made every returned `SyncRun.started_at` equal to `completed_at` regardless of how long the sync actually took. `parse_remote_note()` unwraps a saved-notes list item (or a single-note-fetch `{"item": {...}}` wrapper) into a `SavedNote`, returning `None` (so the sync loop skips it) rather than inventing an id when `comment.id` is missing; it delegates body conversion to `content_utils.note_body_to_text()` and derives `word_count` from the resulting text rather than any source field.
 - `database.py` is the SQLite repository and schema owner. The `posts` table is the cache; `sync_runs` records sync history; `posts_fts` is an external-content FTS5 index maintained by database triggers. `reconcile_unsaved_posts()` bulk soft-deletes posts missing from a given complete remote URL set and is a no-op on an empty list (an empty list is more likely a fetch problem than genuine mass-unsaving). `init_db()` runs additive column migrations (e.g. `posts.audience`, `sync_runs.reconciled_count`) via `ALTER TABLE` guarded by an existence check or `try`/`except sqlite3.OperationalError`, since `CREATE TABLE IF NOT EXISTS` never adds columns to an already-existing table; the `posts.audience` check runs before the `executescript` block because the `idx_posts_audience` index creation inside it would otherwise fail on a pre-migration table. `list_posts()`/`search_posts()` accept an `audience` filter (exact match, case-insensitive), and `list_audiences()` returns the distinct audience values actually present in the cache with post counts, rather than hardcoding Substack's (undocumented, possibly-growing) audience enum. `PostSummary` (returned by `list_posts()`/`search_posts()` and the MCP list/search tools) carries `reading_time_minutes` and `word_count`, and the CLI `list`/`search` commands display them — without this the values were populated in the `posts` table but invisible through every read path except `get_saved_post`/the full `SavedPost`. A parallel `notes` table (with its own `notes_fts` external-content index and `notes_ai`/`notes_ad`/`notes_au` triggers) holds saved notes; unlike `posts`, its unique identity column is `substack_note_id`, not `url` (a note may have no stable permalink), so `upsert_note()` does an explicit lookup-then-branch UPDATE/INSERT on `substack_note_id` rather than `INSERT ... ON CONFLICT(url)`. `upsert_post()` was originally the opposite (looked up by `substack_post_id` *or* `url` but written via `INSERT ... ON CONFLICT(url)`), which meant a post found by id whose canonical URL had since changed (slug rename, custom-domain migration) couldn't conflict on the new URL and tripped the `substack_post_id UNIQUE` constraint on the fresh INSERT instead; it now uses the same explicit lookup-then-branch UPDATE/INSERT pattern as `upsert_note()`, closing that crash path. `search_posts()`'s LIKE fallback (used when an FTS5 query string is malformed) shares its WHERE-clause construction with the main FTS branch via `_post_search_filters()` — mirroring `_note_search_filters()` — so a malformed query never silently drops the `publication`/`audience`/date-range filters or an `is_saved_only=False` caller the way the fallback originally did. `get_note_by_substack_id()` exists because `get_note()`'s polymorphic `url_or_id` dispatch treats any all-digit string as a local row id — and `substack_note_id` values are always digits — so callers that only have the Substack note id (the sync loop's incremental check) must use it instead of risking a collision with an unrelated local row. `sync_runs.entity` (`'post'`/`'note'`, additive column defaulting existing rows to `'post'`) scopes `get_status()`'s "last successful sync"/"last sync status" queries per entity, which is the one existing posts read path notes support required changing (it previously read the last `sync_runs` row unscoped, which was harmless with one entity and would otherwise let a notes sync be reported as the posts' last sync).
-- `config.py` centralizes application paths. By default, data lives under `~/.local/share/substack-saved-mcp`; `SUBSTACK_SAVED_DB_PATH`, `SUBSTACK_SAVED_DATA_DIR`, and `SUBSTACK_SAVED_BROWSER_DIR` override those paths.
+- `config.py` centralizes application paths. By default, data lives under `~/.local/share/substack-saved-mcp`; `SUBSTACK_SAVED_DB_PATH`, `SUBSTACK_SAVED_DATA_DIR`, and `SUBSTACK_SAVED_BROWSER_DIR` override those paths. `get_saved_posts_source()`/`SUBSTACK_SAVED_POSTS_SOURCE` (`auto`/`legacy`/`unified`/`dom`, falling back to `auto` on an unrecognized value) selects which of the three saved-posts fetch sources `_fetch_saved_posts_page_impl()` tries, so a source-specific regression is one env var away from a revert rather than a code rollback.
 
 ## Repository Conventions
 

{substack_saved_mcp-0.2.0 → substack_saved_mcp-0.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: substack-saved-mcp
-Version: 0.2.0
+Version: 0.3.0
 Summary: Local stdio-based MCP server and sync engine for Substack saved posts
 Author-email: Toni Hermoso Pulido <toniher@cau.cat>
 License-Expression: MIT
@@ -202,6 +202,29 @@ export SUBSTACK_SAVED_DB_PATH="/path/to/my/custom_database.sqlite"
 export SUBSTACK_SAVED_DATA_DIR="/path/to/my/data_dir"
 ```
 
+### Which Substack API does syncing saved posts use?
+
+Saved posts are fetched from Substack's newer unified reader API by default, with
+automatic fallback to an older, posts-only API and finally to headless browser
+scraping if needed — you shouldn't normally need to think about this. If you ever
+want to force a specific source (e.g. while troubleshooting), set:
+```bash
+export SUBSTACK_SAVED_POSTS_SOURCE="unified"  # or "legacy" or "dom"
+```
+Leave it unset (or `"auto"`) for the default, self-healing behavior.
+
+### What does a "partial" sync status mean?
+
+If Substack rate-limits (HTTP 429) a sync so heavily that a page of results
+can't be fetched even after retrying, the sync keeps whatever it already
+fetched rather than failing outright, and reports `status: partial` (instead
+of `success`) in `substack-saved-mcp status` or the sync tool's response.
+When this happens on a `sync --force`, reconciliation (soft-deleting posts/notes
+no longer in the remote list) is automatically skipped for that run, so a post
+or note that merely couldn't be fetched is never mistaken for one you actually
+unsaved on Substack. Just run `sync` again later — a subsequent successful run
+picks up anything that was missed.
+
 ### Will a browser window pop up when running as an MCP server?
 
 **No, a visible browser window will not open during normal MCP operations.**

{substack_saved_mcp-0.2.0 → substack_saved_mcp-0.3.0}/README.md

@@ -184,6 +184,29 @@ export SUBSTACK_SAVED_DB_PATH="/path/to/my/custom_database.sqlite"
 export SUBSTACK_SAVED_DATA_DIR="/path/to/my/data_dir"
 ```
 
+### Which Substack API does syncing saved posts use?
+
+Saved posts are fetched from Substack's newer unified reader API by default, with
+automatic fallback to an older, posts-only API and finally to headless browser
+scraping if needed — you shouldn't normally need to think about this. If you ever
+want to force a specific source (e.g. while troubleshooting), set:
+```bash
+export SUBSTACK_SAVED_POSTS_SOURCE="unified"  # or "legacy" or "dom"
+```
+Leave it unset (or `"auto"`) for the default, self-healing behavior.
+
+### What does a "partial" sync status mean?
+
+If Substack rate-limits (HTTP 429) a sync so heavily that a page of results
+can't be fetched even after retrying, the sync keeps whatever it already
+fetched rather than failing outright, and reports `status: partial` (instead
+of `success`) in `substack-saved-mcp status` or the sync tool's response.
+When this happens on a `sync --force`, reconciliation (soft-deleting posts/notes
+no longer in the remote list) is automatically skipped for that run, so a post
+or note that merely couldn't be fetched is never mistaken for one you actually
+unsaved on Substack. Just run `sync` again later — a subsequent successful run
+picks up anything that was missed.
+
 ### Will a browser window pop up when running as an MCP server?
 
 **No, a visible browser window will not open during normal MCP operations.**

{substack_saved_mcp-0.2.0 → substack_saved_mcp-0.3.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "substack-saved-mcp"
-version = "0.2.0"
+version = "0.3.0"
 description = "Local stdio-based MCP server and sync engine for Substack saved posts"
 readme = "README.md"
 requires-python = ">=3.11"

{substack_saved_mcp-0.2.0 → substack_saved_mcp-0.3.0}/src/substack_saved_mcp/__init__.py

@@ -1,3 +1,3 @@
 """Substack Saved Posts MCP & Sync Application."""
 
-__version__ = "0.1.0"
+__version__ = "0.3.0"

{substack_saved_mcp-0.2.0 → substack_saved_mcp-0.3.0}/src/substack_saved_mcp/cli.py

@@ -1,12 +1,16 @@
 """Command Line Interface (CLI) for Substack Saved Posts MCP & Sync tool."""
 
+import itertools
 import json
+import logging
 import re
 import sys
+import time
 from datetime import UTC, datetime
 
 import click
 
+from substack_saved_mcp import __version__
 from substack_saved_mcp.config import get_storage_state_path
 from substack_saved_mcp.content_utils import (
     format_note_for_llm,
@@ -52,10 +56,13 @@ from substack_saved_mcp.substack_client import (
 )
 from substack_saved_mcp.sync import sync_saved_notes as run_sync_notes
 from substack_saved_mcp.sync import sync_saved_posts as run_sync
+from substack_saved_mcp.url_utils import canonicalize_url
+
+logger = logging.getLogger(__name__)
 
 
 @click.group()
-@click.version_option(version="0.1.0")
+@click.version_option(version=__version__)
 def cli() -> None:
     """Substack Saved Posts MCP & Sync Engine."""
     pass
@@ -98,12 +105,14 @@ def sync(force: bool, only: str | None) -> None:
     if only in (None, "posts"):
         click.echo("Starting Substack saved posts sync...")
         result = run_sync(force=force)
-        if result.status == "success":
+        if result.status in ("success", "partial"):
             posts_ok = True
             msg = f"Sync complete! Fetched {result.fetched_count} posts, upserted {result.upserted_count} posts."
             if result.reconciled_count:
                 msg += f" Unsaved {result.reconciled_count} post(s) no longer on Substack's saved list."
-            click.secho(msg, fg="green")
+            click.secho(msg, fg="green" if result.status == "success" else "yellow")
+            if result.status == "partial":
+                click.secho(f"Warning: {result.error_message}", fg="yellow")
         elif result.status == "auth_required":
             click.secho(f"Authentication required: {result.error_message}", fg="yellow")
         else:
@@ -112,12 +121,16 @@ def sync(force: bool, only: str | None) -> None:
     if only in (None, "notes"):
         click.echo("Starting Substack saved notes sync...")
         note_result = run_sync_notes(force=force)
-        if note_result.status == "success":
+        if note_result.status in ("success", "partial"):
             notes_ok = True
             msg = f"Sync complete! Fetched {note_result.fetched_count} notes, upserted {note_result.upserted_count} notes."
             if note_result.reconciled_count:
                 msg += f" Unsaved {note_result.reconciled_count} note(s) no longer on Substack's saved list."
-            click.secho(msg, fg="green")
+            click.secho(
+                msg, fg="green" if note_result.status == "success" else "yellow"
+            )
+            if note_result.status == "partial":
+                click.secho(f"Warning: {note_result.error_message}", fg="yellow")
         elif note_result.status == "auth_required":
             click.secho(
                 f"Authentication required: {note_result.error_message}", fg="yellow"
@@ -748,38 +761,56 @@ def inspect_network(
         context = browser.new_context(storage_state=storage_state)
         page = context.new_page()
 
-        def handle_response(response):
-            if response.url.lower().split("?")[0].endswith(_INSPECT_ASSET_SUFFIXES):
+        def handle_route(route):
+            request = route.request
+            req_url = request.url
+            if req_url.lower().split("?")[0].endswith(
+                _INSPECT_ASSET_SUFFIXES
+            ) or not pattern.search(req_url):
+                route.continue_()
                 return
-            if not pattern.search(response.url):
+
+            # A sync-API page.on("response") handler can deadlock calling
+            # response.text() on the same driver thread that produced it. Routing
+            # the request through route.fetch() reads the body safely outside that
+            # handler, and route.fulfill() re-serves the exact response to the page
+            # so navigation/rendering behaves identically to an unrouted request.
+            try:
+                response = route.fetch()
+            except Exception as e:
+                logger.warning(f"Could not fetch intercepted request {req_url}: {e}")
+                route.continue_()
                 return
 
-            request_body = response.request.post_data
+            request_body = request.post_data
+            content_type = response.headers.get("content-type") or ""
             response_body = None
-            if max_body:
-                content_type = response.headers.get("content-type") or ""
-                if "json" in content_type:
-                    try:
-                        response_body = response.text()[:max_body]
-                    except Exception:
-                        response_body = None
+            body_read_failed = False
+            if max_body and "json" in content_type:
+                try:
+                    response_body = response.text()[:max_body]
+                except Exception as e:
+                    body_read_failed = True
+                    logger.warning(f"Could not read response body for {req_url}: {e}")
 
             click.echo(
-                f"[Network Intercept] {response.request.method} {response.url} "
+                f"[Network Intercept] {request.method} {req_url} "
                 f"(Status: {response.status})"
             )
             if request_body:
                 click.echo(f"    Request Body : {request_body}")
             if response_body:
                 click.echo(f"    Response Body: {response_body}")
+            elif body_read_failed:
+                click.secho("    Response Body: <could not be read>", fg="yellow")
 
             if out_file:
                 out_file.write(
                     json.dumps(
                         {
                             "ts": datetime.now(UTC).isoformat(),
-                            "method": response.request.method,
-                            "url": response.url,
+                            "method": request.method,
+                            "url": req_url,
                             "status": response.status,
                             "request_body": request_body,
                             "response_body": response_body,
@@ -789,7 +820,9 @@ def inspect_network(
                 )
                 out_file.flush()
 
-        page.on("response", handle_response)
+            route.fulfill(response=response)
+
+        context.route("**/*", handle_route)
         page.goto(url)
         click.echo(
             "Navigate around the page (try the Notes toggle, save/unsave, open an "
@@ -803,5 +836,192 @@ def inspect_network(
         click.echo(f"Capture written to {out_path}")
 
 
+@cli.command(name="probe-api")
+@click.argument("url")
+@click.option(
+    "--out",
+    "out_path",
+    type=click.Path(dir_okay=False),
+    help="Write the raw JSON response to this file instead of stdout.",
+)
+def probe_api(url: str, out_path: str | None) -> None:
+    """Headlessly GET a Substack API URL with the saved session (no browser page).
+
+    Complements 'inspect-network' (which discovers unknown endpoints by watching
+    a live browsing session): use this to re-verify a URL already known, or to
+    inspect a specific reader-API response while assessing a migration.
+    """
+    client = SubstackSavedPostsClient()
+    try:
+        data = client.probe_api_get(url)
+    except AuthRequiredError as e:
+        click.secho(f"Authentication required: {e}", fg="yellow")
+        return
+    except Exception as e:
+        click.secho(f"Error probing {url}: {e}", fg="red")
+        return
+
+    text = json.dumps(data, indent=2)
+    if out_path:
+        with open(out_path, "w") as f:
+            f.write(text)
+        click.echo(f"Response written to {out_path}")
+    else:
+        click.echo(text)
+
+
+# Fields parse_remote_post() reads off a unified-endpoint post object, used by
+# compare-saved-apis to report per-field presence without printing post content.
+_UNIFIED_POST_FIELDS = (
+    "id",
+    "canonical_url",
+    "title",
+    "post_date",
+    "saved_at",
+    "audience",
+    "description",
+    "subtitle",
+    "cover_image",
+    "wordcount",
+    "word_count",
+    "words",
+)
+
+
+@cli.command(name="compare-saved-apis")
+@click.option(
+    "--out",
+    "out_path",
+    type=click.Path(dir_okay=False),
+    help=(
+        "Write the full report (including mismatched canonical URLs) as JSON to "
+        "this file. Without it, only aggregate counts/percentages are printed — "
+        "no post URLs or content."
+    ),
+)
+def compare_saved_apis(out_path: str | None) -> None:
+    """Fetch the full saved-posts list from both the legacy and unified reader
+    APIs and report parity: counts, set differences, per-field presence on the
+    unified payload, and bookmark-timestamp/ordering — the evidence needed to
+    decide whether posts can migrate off the legacy reader-posts API.
+    """
+    try:
+        from playwright.sync_api import sync_playwright
+    except ImportError:
+        click.secho("Playwright not installed.", fg="red")
+        return
+
+    client = SubstackSavedPostsClient()
+    try:
+        client._ensure_authenticated()
+    except AuthRequiredError as e:
+        click.secho(f"Authentication required: {e}", fg="yellow")
+        return
+
+    with sync_playwright() as p:
+        api_context = p.request.new_context(storage_state=str(client.state_path))
+        try:
+            click.echo("Fetching via legacy reader-posts API...")
+            legacy_posts = client._fetch_all_saved_via_reader_api(api_context) or []
+
+            # This tool runs once, interactively, rather than in a background
+            # sync loop, so it can afford to be far more patient than the
+            # default sync retry policy: pause briefly before the second full
+            # paginated fetch (fetching legacy's full list can itself trigger
+            # rate limiting that would otherwise make the very next request
+            # 429 through all retries and be misread as "unified has no
+            # posts"), and retry the unified fetch harder (max_retries=6,
+            # exponential backoff up to ~30s per attempt) before giving up.
+            click.echo("Pausing briefly before the unified reader/saved API...")
+            time.sleep(5)
+            click.echo("Fetching via unified reader/saved API...")
+            unified_items = (
+                client._fetch_all_saved_posts_via_unified_api(
+                    api_context, max_retries=6
+                )
+                or []
+            )
+        except AuthRequiredError as e:
+            click.secho(f"Authentication required: {e}", fg="yellow")
+            return
+
+    legacy_urls = {
+        canonicalize_url(item.get("canonical_url") or "") for item in legacy_posts
+    }
+    legacy_urls.discard("")
+    unified_posts = [item.get("post") or {} for item in unified_items]
+    unified_urls = {
+        canonicalize_url(post.get("canonical_url") or post.get("url") or "")
+        for post in unified_posts
+    }
+    unified_urls.discard("")
+
+    missing_from_unified = legacy_urls - unified_urls
+    missing_from_legacy = unified_urls - legacy_urls
+
+    n_unified = len(unified_posts) or 1
+    field_presence = {
+        field: sum(1 for post in unified_posts if post.get(field))
+        for field in _UNIFIED_POST_FIELDS
+    }
+
+    saved_ats = [post.get("saved_at") for post in unified_posts if post.get("saved_at")]
+    ordered = all(a >= b for a, b in itertools.pairwise(saved_ats))
+
+    if legacy_posts and not unified_items:
+        click.echo("")
+        click.secho(
+            "Unified API returned 0 posts while legacy returned "
+            f"{len(legacy_posts)} — before reading this as 'the endpoint lacks "
+            "posts data', re-run compare-saved-apis on its own (not right "
+            "after another full sync/fetch): a 429 that survives every retry "
+            "reads identically to a genuinely empty/unavailable endpoint, and "
+            "the large legacy fetch just before it can trigger exactly that.",
+            fg="yellow",
+        )
+
+    click.echo("")
+    click.echo(f"Legacy reader-posts API : {len(legacy_posts)} posts")
+    click.echo(f"Unified reader/saved API: {len(unified_items)} posts")
+    click.echo(f"In both                : {len(legacy_urls & unified_urls)}")
+    click.echo(f"Only in legacy          : {len(missing_from_unified)}")
+    click.echo(f"Only in unified         : {len(missing_from_legacy)}")
+    click.echo("")
+    click.echo("Unified payload field presence:")
+    for field in _UNIFIED_POST_FIELDS:
+        pct = 100 * field_presence[field] / n_unified
+        click.echo(
+            f"  {field:<16} {field_presence[field]:>5}/{len(unified_posts):<5} ({pct:5.1f}%)"
+        )
+    click.echo("")
+    if saved_ats:
+        click.echo(
+            f"post.saved_at present on {len(saved_ats)}/{len(unified_posts)} items; "
+            f"{'monotonically newest-first' if ordered else 'NOT monotonically ordered'}."
+        )
+    else:
+        click.secho(
+            "No post.saved_at found anywhere in the unified payload — a real "
+            "bookmark timestamp for posts may be absent from this endpoint.",
+            fg="yellow",
+        )
+
+    if out_path:
+        report = {
+            "legacy_count": len(legacy_posts),
+            "unified_count": len(unified_items),
+            "overlap_count": len(legacy_urls & unified_urls),
+            "only_in_legacy": sorted(missing_from_unified),
+            "only_in_unified": sorted(missing_from_legacy),
+            "unified_field_presence": field_presence,
+            "unified_field_presence_denominator": len(unified_posts),
+            "saved_at_present_count": len(saved_ats),
+            "saved_at_ordered_newest_first": ordered if saved_ats else None,
+        }
+        with open(out_path, "w") as f:
+            json.dump(report, f, indent=2)
+        click.echo(f"\nFull report (including mismatched URLs) written to {out_path}")
+
+
 if __name__ == "__main__":
     cli()

{substack_saved_mcp-0.2.0 → substack_saved_mcp-0.3.0}/src/substack_saved_mcp/config.py

@@ -5,6 +5,17 @@ from pathlib import Path
 
 APP_NAME = "substack-saved-mcp"
 
+_SAVED_POSTS_SOURCES = frozenset({"auto", "legacy", "unified", "dom"})
+
+
+def get_saved_posts_source() -> str:
+    """Return which saved-posts fetch source to use: 'auto' (default, tries legacy
+    then the newer unified reader API then DOM), or a forced 'legacy'/'unified'/'dom'
+    for testing and rollback. Falls back to 'auto' on an unrecognized value rather
+    than raising, since a typo here shouldn't break every sync."""
+    value = (os.getenv("SUBSTACK_SAVED_POSTS_SOURCE") or "auto").strip().lower()
+    return value if value in _SAVED_POSTS_SOURCES else "auto"
+
 
 def get_default_data_dir() -> Path:
     """Return OS-appropriate application data directory."""