substack-saved-mcp 0.3.3__tar.gz → 0.4.0__tar.gz

{substack_saved_mcp-0.3.3 → substack_saved_mcp-0.4.0}/.gitignore

@@ -17,3 +17,9 @@ storage_state.json
 *.sqlite-wal
 *.sqlite-shm
 .env
+
+# tokenade — per-project cache / scaffold flag / dedup refs
+.tokenade/
+
+# tokenade — legacy per-project embedding model cache (now global)
+.fastembed_cache/

{substack_saved_mcp-0.3.3 → substack_saved_mcp-0.4.0}/CLAUDE.md

@@ -27,15 +27,16 @@ After any change to dependencies or tracked files, run `uv lock` to keep `uv.loc
 
 - `cli.py` is the Click entry point (`substack-saved-mcp`). Its commands initialize the database and then delegate to the repository, sync engine, Playwright client, or MCP server.
 - `mcp_server.py` exposes the same application operations as FastMCP stdio tools and resources. Read tools query the local cache; sync and save/unsave operations use the authenticated browser session.
-- `substack_client.py` owns Playwright authentication and remote Substack interaction. `login` is the only intended headful workflow; normal sync and write paths use `storage_state.json` headlessly. Synchronous Playwright API calls are routed through `_run_playwright_sync()` to safely execute in a worker thread if an `asyncio` event loop is active (e.g. under FastMCP). The legacy reader inbox API (`GET /api/v1/reader/posts?inboxType=saved`, second in the source chain described below) exposes the real bookmark timestamp (`saved_at`) and an ISO publication date (`post_date`) per post. `_fetch_all_saved_via_reader_api()` cursor-paginates that endpoint (each page's oldest `saved_at` becomes the next `after=` cursor, until `more` is false), dedupes by canonical URL, and enriches each flat post with its `publication` object (from the response's `publications` array, matched by `publication_id`) and `author_name` (from `publishedBylines`). The full result is cached in `_api_cache` and sliced by offset; `_fetch_saved_posts_page_impl()` accepts an optional `playwright_instance` (same pattern as the notes fetcher and `_fetch_via_dom`) so this caching/slicing/fallback branch is directly testable with a Playwright double instead of only reachable through a real browser. Each page request goes through `_reader_api_get()`, which retries transient failures (HTTP 429 and 5xx) up to `max_retries` (default 3) with `Retry-After`-aware backoff: `_retry_after_seconds()` honors an integer `Retry-After` header (clamped to a 30s cap so a hostile value can't hang the sync) and otherwise uses capped exponential backoff (0.5s, 1s, 2s, ...); 401/403 and other 4xx are returned unretried so the existing auth/fallback handling applies. A 429 that survives all retries is deliberately treated as "unavailable" (partial list, or `None` → DOM fallback) rather than as a silent empty-success, so rate-limiting never masquerades as "you have no saved posts." Both `_reader_api_get()` and `_fetch_all_saved_via_reader_api()` accept an injectable `sleep_func` (default `time.sleep`) so tests exercise the backoff without real delays. If that endpoint is unavailable, it falls back to headless DOM extraction on `https://substack.com/saved`, which scrapes `div.reader2-post-container` cards, caches scrolling results in `_dom_cache`, and marks each dict with `_dom: True`; DOM cards only expose a localized relative publish string and no bookmark time, so `saved_at` stays unknown on that path. Every post page server-renders a `window._preloads` blob containing that post's own numeric ID and rich metadata (`preloads.post.id`, `.title`, `.audience`, `.description`/`.subtitle`, `.post_date`, plus `preloads.pub.name`); `_save_post_impl` reads this via `page.evaluate("() => window._preloads")` right after page load and, when an ID is found, calls the real `POST https://substack.com/api/v1/posts/saved` endpoint (body `{"post_id": ...}`) directly via the Playwright API request context — this also lets it populate the returned `SavedPost` with accurate title/publication/audience/excerpt instead of parsing `page.title()`. `_unsave_post_impl` does the mirror image: when the post's `substack_post_id` is already known (true for any post that has been through a normal `sync`, since the reader API's `id` field populates it), it calls `DELETE https://substack.com/api/v1/posts/saved` with the same body shape, no DOM interaction at all. Both treat an `ok` response as `"confirmed"` and only fall back to the best-effort DOM click (`_click_bookmark_toggle()`) when the numeric ID can't be obtained or the direct call doesn't confirm — never click before the API call, since the button toggles and would unsave an already-saved post. Note its selector is English-only (`aria-label*='save'/'bookmark'`) and can silently fail on non-English Substack UIs, which is part of why the direct API path is preferred whenever possible. `_click_bookmark_toggle()` fingerprints the button's `aria-label`/`aria-pressed`/`class` before and after the click and returns `"confirmed"` only if that fingerprint changed, else `"unconfirmed"`, `"not_found"`, or `"click_failed"` — `save_post()` returns `(SavedPost, confirmation)` and `unsave_post()` returns just the confirmation string; both accept an optional `playwright_instance` for test injection (same pattern as `_fetch_via_dom`). `fetch_post_content()` reuses the same `window._preloads` mechanism to retrieve a saved post's full content: `_fetch_post_content_impl` navigates to the post's page and reads `preloads.post.body_html` (Substack's field name for full content — `parse_remote_post()` already expects this key from the reader API, though the saved-list payload never actually populates it, only individual post pages do), returning `None` for `body_html` if the page's embed format doesn't expose it (frontend change) or the account lacks paywall access; this is the case where the caller should be told to run `inspect-network` against an open post page to re-discover the real content source. `cli.py`'s `inspect-network` command (which logs any `api/v1`/`bookmark`/`saved`/`notes?`/`comment`/`reader`/`feed`/`restack` request's method, URL, status, and JSON response body) is the tool for discovering/re-verifying these endpoints when Substack's frontend changes. It supports `--authenticated/--anonymous` (reuses `storage_state.json` via `get_storage_state_path()` so `/saved` renders the real logged-in page instead of the marketing page), `--url` to target any page, `--filter` to override the response-matching regex, `--max-body` to cap logged response bytes, and `--out PATH` to append each exchange as JSON Lines for later grepping. Response bodies are captured via `context.route("**/*", handle_route)` + `route.fetch()`/`route.fulfill()`, not the more obvious `page.on("response", ...)` + `response.text()`: the latter can deadlock calling `response.text()` inside a sync-API response event handler on the same driver thread, silently yielding `null` bodies — `route.fetch()` reads the body outside that handler and `route.fulfill(response=...)` re-serves the exact response so page behavior is unaffected; a body that still can't be read logs a visible yellow warning.
+- `substack_client.py` owns Playwright authentication and remote Substack interaction. `login` is the only intended headful workflow; normal sync and write paths use `storage_state.json` headlessly. Synchronous Playwright API calls are routed through `_run_playwright_sync()` to safely execute in a worker thread if an `asyncio` event loop is active (e.g. under FastMCP). The legacy reader inbox API (`GET /api/v1/reader/posts?inboxType=saved`, second in the source chain described below) exposes the real bookmark timestamp (`saved_at`) and an ISO publication date (`post_date`) per post. `_fetch_all_saved_via_reader_api()` cursor-paginates that endpoint (each page's oldest `saved_at` becomes the next `after=` cursor, until `more` is false), dedupes by canonical URL, and enriches each flat post with its `publication` object (from the response's `publications` array, matched by `publication_id`) and `author_name` (from `publishedBylines`). The full result is cached in `_api_cache` and sliced by offset; `_fetch_saved_posts_page_impl()` accepts an optional `playwright_instance` (same pattern as the notes fetcher and `_fetch_via_dom`) so this caching/slicing/fallback branch is directly testable with a Playwright double instead of only reachable through a real browser. Each page request goes through `_reader_api_get()`, which retries transient failures (HTTP 429 and 5xx) up to `max_retries` (default 5) with `Retry-After`-aware backoff: `_retry_after_seconds()` honors an integer `Retry-After` header (clamped to a 30s cap so a hostile value can't hang the sync) and otherwise uses capped exponential backoff (0.5s, 1s, 2s, ...); 401/403 and other 4xx are returned unretried so the existing auth/fallback handling applies. A 429 that survives all retries is deliberately treated as "unavailable" (partial list, or `None` → DOM fallback) rather than as a silent empty-success, so rate-limiting never masquerades as "you have no saved posts." Both `_reader_api_get()` and `_fetch_all_saved_via_reader_api()` accept an injectable `sleep_func` (default `time.sleep`) so tests exercise the backoff without real delays. If that endpoint is unavailable, it falls back to headless DOM extraction on `https://substack.com/saved`, which scrapes `div.reader2-post-container` cards, caches scrolling results in `_dom_cache`, and marks each dict with `_dom: True`; DOM cards only expose a localized relative publish string and no bookmark time, so `saved_at` stays unknown on that path. Every post page server-renders a `window._preloads` blob containing that post's own numeric ID and rich metadata (`preloads.post.id`, `.title`, `.audience`, `.description`/`.subtitle`, `.post_date`, plus `preloads.pub.name`); `_save_post_impl` reads this via `page.evaluate("() => window._preloads")` right after page load and, when an ID is found, calls the real `POST https://substack.com/api/v1/posts/saved` endpoint (body `{"post_id": ...}`) directly via the Playwright API request context — this also lets it populate the returned `SavedPost` with accurate title/publication/audience/excerpt instead of parsing `page.title()`. `_unsave_post_impl` does the mirror image: when the post's `substack_post_id` is already known (true for any post that has been through a normal `sync`, since the reader API's `id` field populates it), it calls `DELETE https://substack.com/api/v1/posts/saved` with the same body shape, no DOM interaction at all. Both treat an `ok` response as `"confirmed"` and only fall back to the best-effort DOM click (`_click_bookmark_toggle()`) when the numeric ID can't be obtained or the direct call doesn't confirm — never click before the API call, since the button toggles and would unsave an already-saved post. Note its selector is English-only (`aria-label*='save'/'bookmark'`) and can silently fail on non-English Substack UIs, which is part of why the direct API path is preferred whenever possible. `_click_bookmark_toggle()` fingerprints the button's `aria-label`/`aria-pressed`/`class` before and after the click and returns `"confirmed"` only if that fingerprint changed, else `"unconfirmed"`, `"not_found"`, or `"click_failed"` — `save_post()` returns `(SavedPost, confirmation)` and `unsave_post()` returns just the confirmation string; their `_save_post_impl()`/`_unsave_post_impl()` bodies accept an optional `playwright_instance` for test injection (same pattern as `_fetch_via_dom`). `fetch_post_content()` reuses the same `window._preloads` mechanism to retrieve a saved post's full content: `_fetch_post_content_impl` navigates to the post's page and reads `preloads.post.body_html` (Substack's field name for full content — `parse_remote_post()` already expects this key from the reader API, though the saved-list payload never actually populates it, only individual post pages do), returning `None` for `body_html` if the page's embed format doesn't expose it (frontend change) or the account lacks paywall access; this is the case where the caller should be told to run `inspect-network` against an open post page to re-discover the real content source. `cli.py`'s `inspect-network` command (which logs any `api/v1`/`bookmark`/`saved`/`notes?`/`comment`/`reader`/`feed`/`restack` request's method, URL, status, and JSON response body) is the tool for discovering/re-verifying these endpoints when Substack's frontend changes. It supports `--authenticated/--anonymous` (reuses `storage_state.json` via `get_storage_state_path()` so `/saved` renders the real logged-in page instead of the marketing page), `--url` to target any page, `--filter` to override the response-matching regex, `--max-body` to cap logged response bytes, and `--out PATH` to append each exchange as JSON Lines for later grepping. Response bodies are captured via `context.route("**/*", handle_route)` + `route.fetch()`/`route.fulfill()`, not the more obvious `page.on("response", ...)` + `response.text()`: the latter can deadlock calling `response.text()` inside a sync-API response event handler on the same driver thread, silently yielding `null` bodies — `route.fetch()` reads the body outside that handler and `route.fulfill(response=...)` re-serves the exact response so page behavior is unaffected; a body that still can't be read logs a visible yellow warning.
 
-Saved-posts fetching has a three-source chain, controlled by `get_saved_posts_source()`/`SUBSTACK_SAVED_POSTS_SOURCE` (`auto` default, or a forced `legacy`/`unified`/`dom`): the unified `GET /api/v1/reader/saved?filter=posts` endpoint (the same one notes uses with `filter=notes`) is now tried first, then the legacy reader-posts API above as an automatic fallback, then the DOM fallback. This was a migration decision, not a default guess — see `PLAN.md`'s Decisions Ledger and Evidence sections for the full evidence-gathering and decision-gate process. `probe-api URL [--out PATH]` headlessly GETs any known API URL with the saved session (no browser page) and dumps the raw JSON, a lighter complement to `inspect-network`, which is for *discovering* unknown endpoints by watching a live browsing session. `PLAN.md` § Evidence records the live comparison: the unified endpoint was a stable superset of legacy, and the legacy count varied from run to run because of silent mid-pagination truncation. `_fetch_all_saved_posts_via_unified_api()` is a structural sibling of `_fetch_all_saved_notes_via_api()` (opaque `nextCursor` pagination, not the legacy `after=<ISO saved_at>` scheme), cached separately in `_unified_api_cache`/`_unified_api_failed` so one source's failure can't poison another's mid-run cache. `parse_remote_post()` reads `saved_at` from both the item level (legacy) and `post_obj` (unified: `post.saved_at`), since the two APIs nest it differently. All three cursor-paginated fetchers (legacy posts, unified posts, notes) return a partial list rather than `None` when a page fails after every retry, so a persistent 429 doesn't throw away progress, and set a per-source flag (`_api_truncated` / `_unified_api_truncated` / `_notes_api_truncated`, reset at the top of each fetch and in `reset_cache()`). `is_posts_fetch_truncated()`/`is_notes_fetch_truncated()` report the populated cache's flag, using `getattr()` defaults so test doubles that skip `__init__` report "not truncated". On a truncated fetch, `sync_saved_posts()`/`sync_saved_notes()` **skip reconciliation** (absent items may still be saved remotely) and return `status = "partial"` with an `error_message`. The CLI's `sync` shows `partial` in yellow as a qualified success (`posts_ok`/`notes_ok` still count it as having run); `get_status()`'s "last successful sync" (`status = 'success'`) excludes it.
+Saved-posts fetching has a three-source chain, controlled by `get_saved_posts_source()`/`SUBSTACK_SAVED_POSTS_SOURCE` (`auto` default, or a forced `legacy`/`unified`/`dom`): the unified `GET /api/v1/reader/saved?filter=posts` endpoint (the same one notes uses with `filter=notes`) is now tried first, then the legacy reader-posts API above as an automatic fallback, then the DOM fallback. `PLAN.md`'s Decisions Ledger and Evidence sections record why this order was chosen. `probe-api URL [--out PATH]` headlessly GETs any known API URL with the saved session (no browser page) and dumps the raw JSON, a lighter complement to `inspect-network`, which is for *discovering* unknown endpoints by watching a live browsing session. `PLAN.md` § Evidence records the live comparison: the unified endpoint was a stable superset of legacy, and the legacy count varied from run to run because of silent mid-pagination truncation. `_fetch_all_saved_posts_via_unified_api()` is a structural sibling of `_fetch_all_saved_notes_via_api()` (opaque `nextCursor` pagination, not the legacy `after=<ISO saved_at>` scheme), cached separately in `_unified_api_cache`/`_unified_api_failed` so one source's failure can't poison another's mid-run cache. `parse_remote_post()` reads `saved_at` from both the item level (legacy) and `post_obj` (unified: `post.saved_at`), since the two APIs nest it differently. All three cursor-paginated fetchers (legacy posts, unified posts, notes) return a partial list rather than `None` when a page fails after every retry, so a persistent 429 doesn't throw away progress, and set a per-source flag (`_api_truncated` / `_unified_api_truncated` / `_notes_api_truncated`, reset at the top of each fetch and in `reset_cache()`). `is_posts_fetch_truncated()`/`is_notes_fetch_truncated()` report the populated cache's flag, using `getattr()` defaults so test doubles that skip `__init__` report "not truncated". On a truncated fetch, `sync_saved_posts()`/`sync_saved_notes()` **skip reconciliation** (absent items may still be saved remotely) and return `status = "partial"` with an `error_message`. The CLI's `sync` shows `partial` in yellow as a qualified success (`posts_ok`/`notes_ok` still count it as having run); `get_status()`'s "last successful sync" (`status = 'success'`) excludes it.
 - **Saved notes** (Substack's short-form posts, distinct from long-form posts) are modeled and synced separately from posts, in their own `notes` table/`SavedNote`/`NoteSummary` models, discovered and confirmed live against a real account rather than guessed. The saved-notes list is `GET https://substack.com/api/v1/reader/saved?filter=notes` (also `filter=all`/`filter=posts` — a newer, unified endpoint Substack has added alongside the legacy posts-only `/api/v1/reader/posts?inboxType=saved` that `_fetch_all_saved_via_reader_api()` uses; posts now use `filter=posts` first, see the source chain above). Its response is `{"items": [...], "nextCursor": <opaque base64 JSON token> | null}`; pagination resubmits the server's own `nextCursor` value as a `cursor=` query param rather than the `after=<ISO timestamp>` scheme posts uses, and the `limit=` query param has no effect (the server returns a fixed page size), so `_fetch_all_saved_notes_via_api()` is a standalone paginator, not a generalization of `_fetch_all_saved_via_reader_api()`. Each item is `{"entity_key": "c-<id>", "publication": null|{...}, "post": null|{...}, "comment": {...}}`; a restack is represented by the top-level `publication`/`post` fields being populated instead of null (unconfirmed by a live example in the account used to discover this, but structurally solid since that item shape is shared with `filter=posts`). Critically, **the endpoint never exposes a bookmark timestamp** — only `comment.is_saved: bool` — so `SavedNote.saved_at` is always `None`, and `sync_saved_notes()`'s incremental early-stop check compares "already saved locally" (`existing.is_saved == 1`) rather than a `saved_at` match the way `sync_saved_posts()` does. A note's identity is its numeric `comment.id`, which is also embedded in its permalink (`https://substack.com/@<handle>/note/c-<id>`, confirmed live via click-tracking) — `_extract_note_id()` parses it back out of a URL via regex, and `upsert_note()`/`reconcile_unsaved_notes()`/`get_note_by_substack_id()` key on that id (not URL, unlike posts) since a note may lack a stable permalink. `comment.body` is already Substack-flattened plain text; `comment.body_json` is a ProseMirror doc (`{type:"doc", content:[{type:"paragraph"|"orderedList"|"listItem",...}]}`, text nodes carrying `marks:[{type:"bold"|"italic"|"link"}]`) that `content_utils.note_body_to_text()`/`prosemirror_to_llm_text()` convert to the same `**bold**`/`*italic*`/`text (url)` conventions `html_to_llm_text()` uses for posts; `body_raw` stores the source verbatim as a hedge against the body-format guess needing correction later. The individual-note-fetch endpoint `GET /api/v1/reader/comment/{id}` → `{"item": {"comment": {...}}}` and the bookmark endpoints `POST`/`DELETE https://substack.com/api/v1/note/c-{id}/save` (id in the URL path, no request body) are all plain authenticated calls through Playwright's `p.request` API context — **no browser page or `window._preloads` lookup is needed for notes at all**, unlike posts, so `save_note()`/`unsave_note()`/`fetch_note_content()` in `substack_client.py` never launch `p.chromium`. Notes have no DOM fallback and no `_click_bookmark_toggle`-style guesswork: the notes card markup is uncaptured, and since the note id is the identity, a DOM-scraped row without one could never reconcile, so an unavailable notes API raises `SubstackClientError` naming `inspect-network` rather than degrading silently. `sync_runs.entity` (`'post'` | `'note'`, default `'post'` for pre-existing rows) scopes `get_status()`'s "last sync" queries per entity so a notes sync can't be reported as the posts' last sync result.
 - `content_utils.py` converts a post's raw `body_html` into clean text for LLM consumption. `html_to_llm_text()` is a small `html.parser.HTMLParser` subclass (no external HTML library dependency) that renders headings as markdown `#` prefixes, list items as `- ` bullets, links as `text (url)`, keeps minimal `**bold**`/`*italic*` markers, drops `script`/`style`/`iframe` content entirely, and collapses excess blank lines. `format_post_for_llm()` prepends a plain metadata header (title, publication, author, published date, URL) to the cleaned body text — this combined string is what both the CLI's `get-content` command and the MCP `get_post_content` tool return, through the shared `sync.load_post_content()`/`load_note_content()` (cache check, fetch, convert, store; they raise `ContentUnavailableError` when Substack returns no body). `note_body_to_text()` handles the same problem for notes' `body_json`/`body`, dispatching a dict (or JSON string) to `prosemirror_to_llm_text()` (a small recursive walker over Substack's ProseMirror doc shape — `paragraph`/`heading`/`bulletList`/`orderedList`/`listItem`/`hard_break` nodes, `bold`/`italic`/`link` marks — producing the same `**bold**`/`*italic*`/`text (url)` conventions as `html_to_llm_text()`) and a plain string to either the HTML converter (if it looks like HTML) or straight through; `format_note_for_llm()` builds the note-specific header (author/handle/posted date/URL, plus a `Restacked: <title> (<url>)` line when applicable) rather than reusing `format_post_for_llm()`, since `publication_name` is required there but meaningless for most notes.
 - `sync.py` converts API/DOM payloads to `SavedPost` models and coordinates paginated incremental or full syncs. `parse_remote_post()` routes DOM dicts (identified by the `_dom` marker) through a minimal mapping and everything else through the full reader-API/legacy mapping; it never fabricates a `saved_at` from the sync moment (unknown save times stay `None`). `word_count` is mapped from `wordcount`/`word_count`/`words` via `_first_positive_int()` (which ignores non-positive/uncoercible values); a live parity run (see `PLAN.md` § Evidence) confirmed the real field is `wordcount` (100% populated on the unified endpoint's payload, 0% on `word_count`/`words`) — the defensive multi-key mapping is kept since `_first_positive_int()` already checks `wordcount` first and the other two cost nothing to keep as a hedge. `reading_time_minutes` is intentionally *derived* (ceil of `word_count / WORDS_PER_MINUTE`, ~200 wpm) rather than mapped from a field, because a wrong guess about that field's unit (seconds vs minutes) would persist a badly wrong value; deriving is unit-unambiguous. `image_url` is mapped from `cover_image`/`image_url` on the reader-API payload and is populated for the large majority of synced posts (confirmed live: ~98%); `PostSummary` and the `list_posts()`/`search_posts()` SELECTs include it; the CLI `search` command prints it when present, but `list` deliberately omits it since it's a long, uninformative CDN URL in that terser view. `content_text` remains unpopulated by sync (only `get-content`/`get_post_content` populate it) because the saved-list payload doesn't carry post body HTML, only individual post pages do. There is no `metadata_json` column. Older databases may still have an inert `metadata_json` column; reads tolerate it because `SavedPost` (Pydantic) ignores unknown columns from `SELECT *`, and `upsert_post` no longer references it. A force/full sync (`--force`) collects every fetched post's URL and, after the fetch loop completes, calls `reconcile_unsaved_posts()` to soft-delete any locally `is_saved = 1` post absent from that complete remote set — this is how posts unsaved directly on Substack (outside this tool) get reflected locally. An incremental sync never reconciles, since its early-stop-on-matches optimization means it only sees a partial remote list. It records each run in SQLite (including `reconciled_count`) and returns a `SyncRun` with `success`, `auth_required`, or `failed` status instead of propagating expected authentication failures. `sync_saved_notes()` is a sibling of `sync_saved_posts()`, not a generalization of it (notes reconcile by Substack note id via `reconcile_unsaved_notes()`, not URL, and the incremental early-stop check compares `existing.is_saved == 1` rather than a `saved_at` match, since notes carry none); both write `sync_runs.entity` (`'post'`/`'note'`) and share a `_build_sync_run()` helper instead of each hand-constructing `SyncRun` three times; `_build_sync_run()` takes the real `started_at`, captured once at the top of each sync function right before `start_sync_run()`; don't stamp it at finish time. `parse_remote_note()` unwraps a saved-notes list item (or a single-note-fetch `{"item": {...}}` wrapper) into a `SavedNote`, returning `None` (so the sync loop skips it) rather than inventing an id when `comment.id` is missing; it delegates body conversion to `content_utils.note_body_to_text()` and derives `word_count` from the resulting text rather than any source field.
-- `database.py` is the SQLite repository and schema owner. The `posts` table is the cache; `sync_runs` records sync history; `posts_fts` is an external-content FTS5 index maintained by database triggers. `reconcile_unsaved_posts()` bulk soft-deletes posts missing from a given complete remote URL set and is a no-op on an empty list (an empty list is more likely a fetch problem than genuine mass-unsaving). `init_db()` runs additive column migrations (e.g. `posts.audience`, `sync_runs.reconciled_count`) via `ALTER TABLE` guarded by an existence check or `try`/`except sqlite3.OperationalError`, since `CREATE TABLE IF NOT EXISTS` never adds columns to an already-existing table; the `posts.audience` check runs before the `executescript` block because the `idx_posts_audience` index creation inside it would otherwise fail on a pre-migration table. `list_posts()`/`search_posts()` accept an `audience` filter (exact match, case-insensitive), and `list_audiences()` returns the distinct audience values actually present in the cache with post counts, rather than hardcoding Substack's (undocumented, possibly-growing) audience enum. `PostSummary` (returned by `list_posts()`/`search_posts()` and the MCP list/search tools) carries `reading_time_minutes` and `word_count`, and the CLI `list`/`search` commands display them — without this the values were populated in the `posts` table but invisible through every read path except `get_saved_post`/the full `SavedPost`. A parallel `notes` table (with its own `notes_fts` external-content index and `notes_ai`/`notes_ad`/`notes_au` triggers) holds saved notes; unlike `posts`, its unique identity column is `substack_note_id`, not `url` (a note may have no stable permalink), so `upsert_note()` does an explicit lookup-then-branch UPDATE/INSERT on `substack_note_id` rather than `INSERT ... ON CONFLICT(url)`. `upsert_post()` uses the same lookup-then-branch UPDATE/INSERT (by `substack_post_id` or `url`); don't switch it to `INSERT ... ON CONFLICT(url)`, which crashes on the `substack_post_id UNIQUE` constraint when a post's canonical URL changes (slug rename, custom-domain migration). `search_posts()`'s LIKE fallback (used when an FTS5 query string is malformed) shares its WHERE-clause construction with the main FTS branch via `_post_search_filters()` — mirroring `_note_search_filters()` — so a malformed query never silently drops the `publication`/`audience`/date-range filters or an `is_saved_only=False` caller. `get_note_by_substack_id()` exists because `get_note()`'s polymorphic `url_or_id` dispatch treats any all-digit string as a local row id — and `substack_note_id` values are always digits — so callers that only have the Substack note id (the sync loop's incremental check) must use it instead of risking a collision with an unrelated local row. `sync_runs.entity` (`'post'`/`'note'`, additive column defaulting existing rows to `'post'`) scopes `get_status()`'s "last successful sync"/"last sync status" queries per entity, which is the one existing posts read path notes support required changing (it previously read the last `sync_runs` row unscoped, which was harmless with one entity and would otherwise let a notes sync be reported as the posts' last sync).
+- `database.py` is the SQLite repository and schema owner. The `posts` table is the cache; `sync_runs` records sync history; `posts_fts` is an external-content FTS5 index maintained by database triggers. `reconcile_unsaved_posts()` bulk soft-deletes posts missing from a given complete remote URL set and is a no-op on an empty list (an empty list is more likely a fetch problem than genuine mass-unsaving). `init_db()` runs additive column migrations (e.g. `posts.audience`, `sync_runs.reconciled_count`) via `ALTER TABLE` guarded by an existence check or `try`/`except sqlite3.OperationalError`, since `CREATE TABLE IF NOT EXISTS` never adds columns to an already-existing table; the `posts.audience` check runs before the `executescript` block because the `idx_posts_audience` index creation inside it would otherwise fail on a pre-migration table. `list_posts()`/`search_posts()` accept an `audience` filter (exact match, case-insensitive), and `list_audiences()` returns the distinct audience values actually present in the cache with post counts, rather than hardcoding Substack's (undocumented, possibly-growing) audience enum. `PostSummary` (returned by `list_posts()`/`search_posts()` and the MCP list/search tools) carries `reading_time_minutes` and `word_count`, and the CLI `list`/`search` commands display them. A parallel `notes` table (with its own `notes_fts` external-content index and `notes_ai`/`notes_ad`/`notes_au` triggers) holds saved notes; unlike `posts`, its unique identity column is `substack_note_id`, not `url` (a note may have no stable permalink), so `upsert_note()` does an explicit lookup-then-branch UPDATE/INSERT on `substack_note_id` rather than `INSERT ... ON CONFLICT(url)`. `upsert_post()` uses the same lookup-then-branch UPDATE/INSERT (by `substack_post_id` or `url`); don't switch it to `INSERT ... ON CONFLICT(url)`, which crashes on the `substack_post_id UNIQUE` constraint when a post's canonical URL changes (slug rename, custom-domain migration). `search_posts()`'s LIKE fallback (used when an FTS5 query string is malformed) shares its WHERE-clause construction with the main FTS branch via `_post_search_filters()` — mirroring `_note_search_filters()` — so a malformed query never silently drops the `publication`/`audience`/date-range filters or an `is_saved_only=False` caller. `get_note_by_substack_id()` exists because `get_note()`'s polymorphic `url_or_id` dispatch treats any all-digit string as a local row id — and `substack_note_id` values are always digits — so callers that only have the Substack note id (the sync loop's incremental check) must use it instead of risking a collision with an unrelated local row. `sync_runs.entity` (`'post'`/`'note'`, additive column defaulting existing rows to `'post'`) scopes `get_status()`'s "last successful sync"/"last sync status" queries per entity, so a notes sync is never reported as the posts' last sync.
 - `config.py` centralizes application paths. By default, data lives under `~/.local/share/substack-saved-mcp`; `SUBSTACK_SAVED_DB_PATH`, `SUBSTACK_SAVED_DATA_DIR`, and `SUBSTACK_SAVED_BROWSER_DIR` override those paths. `get_saved_posts_source()`/`SUBSTACK_SAVED_POSTS_SOURCE` (`auto`/`legacy`/`unified`/`dom`, falling back to `auto` on an unrecognized value) selects which of the three saved-posts fetch sources `_fetch_saved_posts_page_impl()` tries, so a source-specific regression is one env var away from a revert rather than a code rollback. `get_fully_read_threshold()`/`SUBSTACK_SAVED_FULLY_READ_THRESHOLD` (default `0.95`, falling back to the default on an unparseable or out-of-`(0, 1]` value) sets the `max_read_progress` fraction at or above which a post counts as fully read.
 - **Reading progress.** Substack exposes per-post reading progress (visible in its mobile apps, never the web UI) through the same saved-posts payloads this project already fetches: `post.read_progress` (current scroll position, 0.0–1.0), `post.max_read_progress` (high-water mark, 0.0–1.0), and `post.is_viewed` (bool) — confirmed live on both the unified and legacy reader APIs (identically mirrored under `post.inboxItem`, so that nested object is never read), absent from the DOM fallback. `parse_remote_post()` maps all three off `post_obj` via a new `_float_or_none()` helper (clamped to `[0.0, 1.0]`, tolerant of a real `0.0` unlike `_first_positive_int()`). The three raw values are stored as columns (`posts.read_progress`/`max_read_progress REAL`, `posts.is_viewed INTEGER NOT NULL DEFAULT 0`); `is_fully_read` and `minutes_remaining` are deliberately **not** columns — they're Pydantic `@computed_field` properties on `SavedPost`/`PostSummary`, derived from `max_read_progress` (the high-water mark, not the current position, since that's what "have I read this" means) and `word_count` at read time, same rationale as `reading_time_minutes`: real observed values like `0.9999`/`0.9867`/`0.9822` rule out an exact `== 1.0` test, and a stored derivative would go stale the moment the threshold changed. `database.py`'s `_READ_STATE_PREDICATES`/`_read_state_clause()` classify a post into `unread`/`in_progress`/`finished`/`unfinished` (below the threshold, i.e. unread + in progress)/`started` via SQL predicates wrapping every comparison in `COALESCE(..., 0)` — load-bearing, not defensive styling: after the additive migration every pre-existing row is `NULL`, and `max_read_progress = 0` would otherwise evaluate to `NULL` (not true) and silently exclude the entire legacy cache from every read_state filter. `list_posts()`/`search_posts()`/`_post_search_filters()` accept `read_state`, threaded through both the FTS and LIKE-fallback branches of `search_posts()` the same way `audience` already was; `list_posts()`'s sort-key whitelist also gained `read_progress` and a `minutes_remaining` SQL expression (`COALESCE(word_count,0) * (1 - COALESCE(max_read_progress,0))`) so a caller can find something short to finish. `upsert_post()` coalesces these three fields with explicit `is not None` (`is_viewed`: truthy-sticky) checks rather than the pre-existing `or`-based idiom used for other optional fields, since a real `0.0` is falsy and the `or` pattern would have silently discarded it in favor of a stale stored value — a DOM-sourced or otherwise progress-free payload (e.g. `_save_post_impl`'s construction site) preserves whatever progress is already stored rather than clobbering it. `get_status()`/`SavedPostsStatus` gained `posts_unread`/`posts_in_progress`/`posts_fully_read`/`minutes_remaining_total` aggregates built from the same predicates and remaining-words expression. Progress refreshes whenever a post is re-fetched during sync; since incremental sync only touches recently-saved posts (its early-stop fires after `MAX_CONSECUTIVE_MATCHES` matches), older posts' progress goes stale between full syncs — `sync --force` is the documented way to refresh it for the whole backlog, and no dedicated refresh mode was added.
+- **Reading history.** Substack's inbox History tab (every post the user opened, saved or not) is `GET https://substack.com/api/v1/reader/posts?inboxType=seen&limit=20[&cursor=...]`, confirmed live: the legacy reader-posts payload shape (`posts`/`publications`/`more`), but paginated by resubmitting the server's opaque `cursor` (base64 `{"inbox_type":"seen","before":<ISO>}`) rather than `after=`. Each post carries `is_saved`/`saved_at`/reading progress plus its view time at `post.inboxItem.seen_at`, which `parse_remote_post()` maps to `SavedPost.seen_at` (`posts.seen_at` column, additive migration, coalesced by `upsert_post()` like `saved_at`). Items are **not** strictly ordered by `seen_at`. `_fetch_all_history_via_reader_api()` shares publication/byline enrichment with the legacy saved fetcher via `_enrich_reader_posts()`, has its own `_history_api_cache`/`_failed`/`_truncated` state and `is_history_fetch_truncated()`, and like notes has **no DOM fallback** (raises `SubstackClientError` naming `inspect-network`). Stopping at its `max_posts=2000` cap while Substack still reports `more` (a `while`/`else` in the paginator) sets both `_history_api_truncated` and `_history_api_capped`; `is_history_fetch_capped()` lets `sync_reading_history()` report `partial` with a cap-specific message (older history not synced) instead of the mid-pagination-failure one. A truncated or capped fetch is ignored when an incremental run stopped early (it never needed the unfetched remainder), so routine syncs don't warn every time. `sync_reading_history()` (`sync_runs.entity = 'history'`) stores browsed posts in the same `posts` table: an existing row keeps its `is_saved`/`unsaved_at` (history never saves, unsaves, or resurrects — the saved-posts sync and its reconciliation own that), a new row takes `is_saved` from the payload; it **never reconciles**, and its incremental early stop (`MAX_CONSECUTIVE_MATCHES` unchanged `seen_at`s) is only a heuristic given the loose ordering, so `--force` walks the whole history. The CLI `sync` runs posts → notes → history (cooldown between each, `--only history` for one, `--no-history` to skip it); `list`/`search --history` and the MCP `history=True` parameter map to `is_saved_only=False, seen_only=True` (`seen_at IS NOT NULL`), with `seen_at` added to the sort whitelist and default sort under `--history`. Default saved-only queries, `list_publications()`, `list_audiences()`, and `get_status()` remain `is_saved = 1`.
 
 ## Repository Conventions
 
@@ -45,6 +46,6 @@ Saved-posts fetching has a three-source chain, controlled by `get_saved_posts_so
 - Treat unsaving as a soft delete: use `soft_delete_post()` (single post, e.g. the `unsave` command/tool) or `reconcile_unsaved_posts()` (bulk, called from a force sync) to set `is_saved = 0` and `unsaved_at`, retaining the record for history. Default list/search/publication queries intentionally include only `is_saved = 1`.
 - The local cache always reflects the user's intent for `save`/`unsave`, regardless of whether the remote toggle on Substack could be confirmed — never block the local write on remote confirmation. Instead, surface the confirmation status (`"confirmed"` vs. `"unconfirmed"/"not_found"/"click_failed"`) as a clearly-labeled warning to the caller (CLI: yellow `click.secho`; MCP: a `warning` key and `remote_confirmed: bool` in the tool's dict response) so an unconfirmed remote action isn't silently reported as a full success. A subsequent `sync --force` will self-correct the local cache via `reconcile_unsaved_posts()` if a "confirmed" save/unsave didn't actually happen remotely.
 - Do not write directly to `posts_fts` or `notes_fts`; use the repository operations so the SQLite insert/update/delete triggers maintain the FTS5 indexes.
-- Notes are identified by `substack_note_id`, not URL — a note may lack a stable permalink, unlike a post. Never give notes a DOM-scrape fallback: the notes card markup is unverified, and since a note's id *is* its identity, a DOM-scraped row without one could never be reconciled; an unavailable notes API should raise a clear error naming `inspect-network` instead of degrading to guesswork the way the old `save_post`/`unsave_post` once did before their real endpoints were captured.
+- Notes are identified by `substack_note_id`, not URL — a note may lack a stable permalink, unlike a post. Never give notes a DOM-scrape fallback: the notes card markup is unverified, and since a note's id *is* its identity, a DOM-scraped row without one could never be reconciled; an unavailable notes API should raise a clear error naming `inspect-network` instead of degrading to guesswork.
 - Database-facing tests use `tmp_path` databases. CLI and MCP tests set `SUBSTACK_SAVED_DB_PATH` with `monkeypatch` so they never access a developer's real cache. Sync tests inject a `SubstackSavedPostsClient` substitute rather than opening Playwright or contacting Substack.
 - Browser state and SQLite data are credentials or user data. Keep `storage_state.json`, browser-state directories, `.env` files, and database files untracked; configuration creates data directories with user-only permissions on POSIX systems.

{substack_saved_mcp-0.3.3 → substack_saved_mcp-0.4.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.5
 Name: substack-saved-mcp
-Version: 0.3.3
+Version: 0.4.0
 Summary: Local stdio-based MCP server and sync engine for Substack saved posts
 Author-email: Toni Hermoso Pulido <toniher@cau.cat>
 License-Expression: MIT
@@ -21,10 +21,13 @@ Description-Content-Type: text/markdown
 
 A local stdio Model Context Protocol (MCP) server and sync engine for the Substack **posts and notes** you have saved.
 
+> **Behavior change in 0.4.0:** a plain `substack-saved-mcp sync` now syncs your **reading history** too: every post you opened on Substack, saved or not. Run `sync --no-history` to sync only saved posts and notes as before. Unsaved posts from your history stay out of default `list`/`search` results.
+
 ## Features
 
 - **Read & Search**: Full-text search (SQLite FTS5) over saved post titles, excerpts, authors, and publications. Filter by publication, audience tier (`everyone`, `only_paid`), reading progress, and date ranges (`published_at` or `saved_at`). Search covers a post's full body text only after you fetch it once with `get-content` or the `get_post_content` tool. A normal `sync` stores metadata and excerpts, so search matches posts you haven't opened on their title, excerpt, and metadata.
 - **Reading Progress**: Substack records how far you've read each saved post. Its mobile apps show this; the web UI doesn't. This tool stores it, so you can list the posts you haven't read or haven't finished, and sort by the minutes left. See [Which saved posts haven't I read?](#which-saved-posts-havent-i-read).
+- **Reading History**: `sync` also pulls Substack's inbox History tab: every post you opened, saved or not, with when you last viewed it (`seen_at`) and your reading progress. Browsed-but-unsaved posts are cached with `is_saved = 0`, so default `list`/`search` results stay saved-only; add `--history` (MCP: `history=True`) to browse or search what you've read.
 - **Saved Notes**: The tool syncs, searches, and caches Substack's short-form notes apart from posts. A note has an author and a body instead of a title or audience tier. Search covers note bodies, authors, and the titles of restacked posts. Notes skip the browser: sync, save, unsave, and content fetch are all plain authenticated API calls.
 - **Full Content for LLMs**: Fetch a saved post or note and get its content back as clean text, with headings, lists, and links kept readable. The tool caches the result for next time.
 - **Save & Unsave**: Bookmark posts and notes, or remove bookmarks. Posts use a headless browser session; notes use the API.
@@ -107,12 +110,15 @@ substack-saved-mcp init
 # 2. Authenticate with Substack (opens interactive browser window once)
 substack-saved-mcp login
 
-# 3. Sync saved posts AND notes into local cache (both entities by default)
+# 3. Sync saved posts, saved notes, AND reading history (all three by default;
+#    add --no-history to skip history)
 substack-saved-mcp sync
 
 # 3b. Or sync just one entity
 substack-saved-mcp sync --only posts
 substack-saved-mcp sync --only notes
+substack-saved-mcp sync --only history
+substack-saved-mcp sync --no-history   # saved posts and notes, skip history
 
 # 4. Search saved posts via CLI
 substack-saved-mcp search "artificial intelligence"
@@ -125,6 +131,10 @@ substack-saved-mcp search "artificial intelligence" --audience everyone
 # 4c. Find posts you haven't read or haven't finished
 substack-saved-mcp list --read-state unread
 substack-saved-mcp list --read-state unfinished --sort-by minutes_remaining
+
+# 4d. Browse or search your reading history (viewed posts, saved or not)
+substack-saved-mcp list --history
+substack-saved-mcp search "democracy" --history
 substack-saved-mcp search "artificial intelligence" --read-state unread
 
 # 5. Save or unsave a post
@@ -253,7 +263,7 @@ No. During normal MCP use you won't see a browser window.
 
 - **Read & Search Tools** (`search_saved_posts`, `list_saved_posts`, `get_saved_post`, `search_saved_notes`, `list_saved_notes`, `get_saved_note`, `list_publications`, `list_audiences`, `saved_posts_status`):  
   Query the local SQLite database offline. No browser.
-- **Post Sync & Write Tools** (`sync_saved_posts`, `save_post`, `unsave_post`, `get_post_content`):  
+- **Post Sync & Write Tools** (`sync_saved_posts`, `sync_reading_history`, `save_post`, `unsave_post`, `get_post_content`):  
   Run headless with the session stored in `storage_state.json`.
 - **Note Sync & Write Tools** (`sync_saved_notes`, `save_note`, `unsave_note`, `get_note_content`):  
   Make direct API requests with `storage_state.json`, with no browser page, headless or visible.

{substack_saved_mcp-0.3.3 → substack_saved_mcp-0.4.0}/README.md

@@ -4,10 +4,13 @@
 
 A local stdio Model Context Protocol (MCP) server and sync engine for the Substack **posts and notes** you have saved.
 
+> **Behavior change in 0.4.0:** a plain `substack-saved-mcp sync` now syncs your **reading history** too: every post you opened on Substack, saved or not. Run `sync --no-history` to sync only saved posts and notes as before. Unsaved posts from your history stay out of default `list`/`search` results.
+
 ## Features
 
 - **Read & Search**: Full-text search (SQLite FTS5) over saved post titles, excerpts, authors, and publications. Filter by publication, audience tier (`everyone`, `only_paid`), reading progress, and date ranges (`published_at` or `saved_at`). Search covers a post's full body text only after you fetch it once with `get-content` or the `get_post_content` tool. A normal `sync` stores metadata and excerpts, so search matches posts you haven't opened on their title, excerpt, and metadata.
 - **Reading Progress**: Substack records how far you've read each saved post. Its mobile apps show this; the web UI doesn't. This tool stores it, so you can list the posts you haven't read or haven't finished, and sort by the minutes left. See [Which saved posts haven't I read?](#which-saved-posts-havent-i-read).
+- **Reading History**: `sync` also pulls Substack's inbox History tab: every post you opened, saved or not, with when you last viewed it (`seen_at`) and your reading progress. Browsed-but-unsaved posts are cached with `is_saved = 0`, so default `list`/`search` results stay saved-only; add `--history` (MCP: `history=True`) to browse or search what you've read.
 - **Saved Notes**: The tool syncs, searches, and caches Substack's short-form notes apart from posts. A note has an author and a body instead of a title or audience tier. Search covers note bodies, authors, and the titles of restacked posts. Notes skip the browser: sync, save, unsave, and content fetch are all plain authenticated API calls.
 - **Full Content for LLMs**: Fetch a saved post or note and get its content back as clean text, with headings, lists, and links kept readable. The tool caches the result for next time.
 - **Save & Unsave**: Bookmark posts and notes, or remove bookmarks. Posts use a headless browser session; notes use the API.
@@ -90,12 +93,15 @@ substack-saved-mcp init
 # 2. Authenticate with Substack (opens interactive browser window once)
 substack-saved-mcp login
 
-# 3. Sync saved posts AND notes into local cache (both entities by default)
+# 3. Sync saved posts, saved notes, AND reading history (all three by default;
+#    add --no-history to skip history)
 substack-saved-mcp sync
 
 # 3b. Or sync just one entity
 substack-saved-mcp sync --only posts
 substack-saved-mcp sync --only notes
+substack-saved-mcp sync --only history
+substack-saved-mcp sync --no-history   # saved posts and notes, skip history
 
 # 4. Search saved posts via CLI
 substack-saved-mcp search "artificial intelligence"
@@ -108,6 +114,10 @@ substack-saved-mcp search "artificial intelligence" --audience everyone
 # 4c. Find posts you haven't read or haven't finished
 substack-saved-mcp list --read-state unread
 substack-saved-mcp list --read-state unfinished --sort-by minutes_remaining
+
+# 4d. Browse or search your reading history (viewed posts, saved or not)
+substack-saved-mcp list --history
+substack-saved-mcp search "democracy" --history
 substack-saved-mcp search "artificial intelligence" --read-state unread
 
 # 5. Save or unsave a post
@@ -236,7 +246,7 @@ No. During normal MCP use you won't see a browser window.
 
 - **Read & Search Tools** (`search_saved_posts`, `list_saved_posts`, `get_saved_post`, `search_saved_notes`, `list_saved_notes`, `get_saved_note`, `list_publications`, `list_audiences`, `saved_posts_status`):  
   Query the local SQLite database offline. No browser.
-- **Post Sync & Write Tools** (`sync_saved_posts`, `save_post`, `unsave_post`, `get_post_content`):  
+- **Post Sync & Write Tools** (`sync_saved_posts`, `sync_reading_history`, `save_post`, `unsave_post`, `get_post_content`):  
   Run headless with the session stored in `storage_state.json`.
 - **Note Sync & Write Tools** (`sync_saved_notes`, `save_note`, `unsave_note`, `get_note_content`):  
   Make direct API requests with `storage_state.json`, with no browser page, headless or visible.

{substack_saved_mcp-0.3.3 → substack_saved_mcp-0.4.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "substack-saved-mcp"
-version = "0.3.3"
+version = "0.4.0"
 description = "Local stdio-based MCP server and sync engine for Substack saved posts"
 readme = "README.md"
 requires-python = ">=3.11"

{substack_saved_mcp-0.3.3 → substack_saved_mcp-0.4.0}/src/substack_saved_mcp/__init__.py

@@ -1,3 +1,3 @@
 """Substack Saved Posts MCP & Sync Application."""
 
-__version__ = "0.3.3"
+__version__ = "0.4.0"

{substack_saved_mcp-0.3.3 → substack_saved_mcp-0.4.0}/src/substack_saved_mcp/cli.py

@@ -55,6 +55,7 @@ from substack_saved_mcp.sync import (
     load_note_content,
     load_post_content,
 )
+from substack_saved_mcp.sync import sync_reading_history as run_sync_history
 from substack_saved_mcp.sync import sync_saved_notes as run_sync_notes
 from substack_saved_mcp.sync import sync_saved_posts as run_sync
 
@@ -88,66 +89,74 @@ def login() -> None:
         sys.exit(1)
 
 
+def _report_sync(label: str, result) -> bool:
+    """Print a sync result; return True if it ran (success or partial)."""
+    if result.status in ("success", "partial"):
+        msg = f"Sync complete! Fetched {result.fetched_count} {label}, upserted {result.upserted_count} {label}."
+        if result.reconciled_count:
+            msg += f" Unsaved {result.reconciled_count} {label} no longer on Substack's saved list."
+        click.secho(msg, fg="green" if result.status == "success" else "yellow")
+        if result.status == "partial":
+            click.secho(f"Warning: {result.error_message}", fg="yellow")
+        return True
+    if result.status == "auth_required":
+        click.secho(f"Authentication required: {result.error_message}", fg="yellow")
+    else:
+        click.secho(f"Sync failed: {result.error_message}", fg="red")
+    return False
+
+
 @cli.command()
 @click.option(
     "--force", is_flag=True, help="Force full resync instead of incremental stop."
 )
 @click.option(
     "--only",
-    type=click.Choice(["posts", "notes"]),
-    help="Sync only one entity instead of both.",
-)
-def sync(force: bool, only: str | None) -> None:
-    """Sync saved posts and notes from Substack account into local SQLite cache."""
-    posts_ok = only == "notes"
-    notes_ok = only == "posts"
-
-    if only in (None, "posts"):
-        click.echo("Starting Substack saved posts sync...")
-        result = run_sync(force=force)
-        if result.status in ("success", "partial"):
-            posts_ok = True
-            msg = f"Sync complete! Fetched {result.fetched_count} posts, upserted {result.upserted_count} posts."
-            if result.reconciled_count:
-                msg += f" Unsaved {result.reconciled_count} post(s) no longer on Substack's saved list."
-            click.secho(msg, fg="green" if result.status == "success" else "yellow")
-            if result.status == "partial":
-                click.secho(f"Warning: {result.error_message}", fg="yellow")
-        elif result.status == "auth_required":
-            click.secho(f"Authentication required: {result.error_message}", fg="yellow")
-        else:
-            click.secho(f"Sync failed: {result.error_message}", fg="red")
-
-    if only is None:
-        # Fetching the full saved-posts list can itself trigger rate limiting
-        # that would make the very next request (the notes fetch) 429 through
-        # all its retries too. Pause to let that limit cool down.
-        cooldown = (
-            15 if (posts_ok and getattr(result, "status", None) == "partial") else 5
+    type=click.Choice(["posts", "notes", "history"]),
+    help="Sync only one entity instead of all.",
+)
+@click.option(
+    "--no-history",
+    is_flag=True,
+    help="Skip the reading-history sync (saved posts and notes only).",
+)
+def sync(force: bool, only: str | None, no_history: bool) -> None:
+    """Sync saved posts, saved notes, and reading history into the local cache.
+
+    By default this includes the reading history: every post opened on
+    Substack, saved or not. Use --no-history to sync only saved posts and notes.
+    """
+    if only == "history" and no_history:
+        raise click.UsageError("--only history and --no-history are contradictory.")
+    steps = [
+        ("posts", "saved posts", "posts", run_sync),
+        ("notes", "saved notes", "notes", run_sync_notes),
+        ("history", "reading history", "viewed posts", run_sync_history),
+    ]
+    if only:
+        steps = [s for s in steps if s[0] == only]
+    if no_history:
+        steps = [s for s in steps if s[0] != "history"]
+
+    if not only and not no_history:
+        click.secho(
+            "Note: sync now includes your reading history (all posts opened on "
+            "Substack, saved or not). Use --no-history to skip it.",
+            fg="blue",
         )
-        time.sleep(cooldown)
-
-    if only in (None, "notes"):
-        click.echo("Starting Substack saved notes sync...")
-        note_result = run_sync_notes(force=force)
-        if note_result.status in ("success", "partial"):
-            notes_ok = True
-            msg = f"Sync complete! Fetched {note_result.fetched_count} notes, upserted {note_result.upserted_count} notes."
-            if note_result.reconciled_count:
-                msg += f" Unsaved {note_result.reconciled_count} note(s) no longer on Substack's saved list."
-            click.secho(
-                msg, fg="green" if note_result.status == "success" else "yellow"
-            )
-            if note_result.status == "partial":
-                click.secho(f"Warning: {note_result.error_message}", fg="yellow")
-        elif note_result.status == "auth_required":
-            click.secho(
-                f"Authentication required: {note_result.error_message}", fg="yellow"
-            )
-        else:
-            click.secho(f"Sync failed: {note_result.error_message}", fg="red")
 
-    if not (posts_ok or notes_ok):
+    any_ok = False
+    previous = None
+    for _name, title, noun, run in steps:
+        if previous is not None:
+            # A full list fetch can itself trigger rate limiting that would make
+            # the next request 429 through all its retries too. Let it cool down.
+            time.sleep(15 if previous.status == "partial" else 5)
+        click.echo(f"Starting Substack {title} sync...")
+        previous = run(force=force)
+        any_ok = _report_sync(noun, previous) or any_ok
+
+    if not any_ok:
         sys.exit(1)
 
 
@@ -301,6 +310,11 @@ def unsave_note(url_or_id: str) -> None:
     type=click.Choice(["unread", "in_progress", "finished", "unfinished", "started"]),
     help="Filter by reading progress.",
 )
+@click.option(
+    "--history",
+    is_flag=True,
+    help="Search the reading history (viewed posts, saved or not) instead of saved posts.",
+)
 @click.option("--limit", default=10, help="Maximum search results.")
 def search(
     query: str,
@@ -311,9 +325,10 @@ def search(
     saved_after: str | None,
     saved_before: str | None,
     read_state: str | None,
+    history: bool,
     limit: int,
 ) -> None:
-    """Perform full-text search across cached saved posts."""
+    """Perform full-text search across cached saved posts (or reading history)."""
     init_db()
     results = db_search_posts(
         query=query,
@@ -325,9 +340,13 @@ def search(
         saved_before=saved_before,
         read_state=read_state,
         limit=limit,
+        is_saved_only=not history,
+        seen_only=history,
     )
     if not results:
-        click.echo(f"No saved posts matched query '{query}'.")
+        click.echo(
+            f"No {'viewed' if history else 'saved'} posts matched query '{query}'."
+        )
         return
 
     click.echo(f"Found {len(results)} matching post(s):\n")
@@ -337,6 +356,8 @@ def search(
         click.echo(
             f"   Published   : {p.published_at or 'N/A'} | Saved: {p.saved_at or 'N/A'}"
         )
+        if p.seen_at:
+            click.echo(f"   Seen        : {p.seen_at}")
         click.echo(f"   Audience    : {p.audience or 'N/A'}")
         if p.reading_time_minutes or p.word_count:
             click.echo(
@@ -375,9 +396,15 @@ def search(
 @click.option(
     "--sort-by",
     type=click.Choice(
-        ["saved_at", "published_at", "read_progress", "minutes_remaining"]
+        ["saved_at", "published_at", "read_progress", "minutes_remaining", "seen_at"]
     ),
-    default="saved_at",
+    default=None,
+    help="Sort key (default: saved_at, or seen_at with --history).",
+)
+@click.option(
+    "--history",
+    is_flag=True,
+    help="List the reading history (viewed posts, saved or not) instead of saved posts.",
 )
 def list_cmd(
     limit: int,
@@ -385,9 +412,10 @@ def list_cmd(
     publication: str | None,
     audience: str | None,
     read_state: str | None,
-    sort_by: str,
+    sort_by: str | None,
+    history: bool,
 ) -> None:
-    """List saved posts ordered by saved date or publication date."""
+    """List saved posts (or reading history with --history)."""
     init_db()
     posts = db_list_posts(
         limit=limit,
@@ -395,13 +423,16 @@ def list_cmd(
         publication=publication,
         audience=audience,
         read_state=read_state,
-        sort_by=sort_by,
+        sort_by=sort_by or ("seen_at" if history else "saved_at"),
+        is_saved_only=not history,
+        seen_only=history,
     )
     if not posts:
-        click.echo("No saved posts found.")
+        click.echo(f"No {'viewed' if history else 'saved'} posts found.")
         return
 
-    click.echo(f"Saved Posts ({len(posts)} displayed):\n")
+    heading = "Reading History" if history else "Saved Posts"
+    click.echo(f"{heading} ({len(posts)} displayed):\n")
     for idx, p in enumerate(posts, offset + 1):
         reading = (
             f" | {p.reading_time_minutes} min ({p.word_count} words)"
@@ -412,6 +443,8 @@ def list_cmd(
             progress_pct = round((p.max_read_progress or 0.0) * 100)
             status_word = "finished" if p.is_fully_read else "in progress"
             reading += f" | {progress_pct}% ({status_word})"
+        if history:
+            reading += f" | Seen: {p.seen_at}" + (" [saved]" if p.is_saved else "")
         click.secho(f"{idx}. {p.title}", fg="cyan")
         click.echo(
             f"   Pub: {p.publication_name} | Saved: {p.saved_at or 'N/A'} | Published: {p.published_at or 'N/A'} | Audience: {p.audience or 'N/A'}{reading}"