substack-saved-mcp 0.1.0__tar.gz → 0.3.0__tar.gz

substack_saved_mcp-0.3.0/CLAUDE.md

@@ -0,0 +1,49 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code when working with this repository.
+
+## Commands
+
+Use `uv` for all development commands. Set up the environment with:
+
+```bash
+uv sync --extra dev
+```
+
+| Task | Command |
+| --- | --- |
+| Run all tests | `uv run python -m pytest` |
+| Run one test file | `uv run python -m pytest tests/test_database.py` |
+| Run one test | `uv run python -m pytest tests/test_database.py::test_fts5_search` |
+| Build the package | `uv build` |
+| Run the CLI | `uv run substack-saved-mcp --help` |
+| Reinstall tool globally | `uv tool install . --no-cache --force` |
+
+`ruff` is configured in `pyproject.toml` (rules: `E4`, `E7`, `E9`, `F`, `I`, `UP`, `B`, `RUF`, line length 88, target py311). Lint with `uvx ruff check .` or `uv run ruff check .`.
+
+After any change to dependencies or tracked files, run `uv lock` to keep `uv.lock` current, then `prek run --all-files` (using the repo's `.pre-commit-config.yaml`) before considering the work done.
+
+## Architecture
+
+- `cli.py` is the Click entry point (`substack-saved-mcp`). Its commands initialize the database and then delegate to the repository, sync engine, Playwright client, or MCP server.
+- `mcp_server.py` exposes the same application operations as FastMCP stdio tools and resources. Read tools query the local cache; sync and save/unsave operations use the authenticated browser session.
+- `substack_client.py` owns Playwright authentication and remote Substack interaction. `login` is the only intended headful workflow; normal sync and write paths use `storage_state.json` headlessly. Synchronous Playwright API calls are routed through `_run_playwright_sync()` to safely execute in a worker thread if an `asyncio` event loop is active (e.g. under FastMCP). Saved-post fetching prefers the reader inbox API (`GET /api/v1/reader/posts?inboxType=saved`), which exposes the real bookmark timestamp (`saved_at`) and an ISO publication date (`post_date`) per post. `_fetch_all_saved_via_reader_api()` cursor-paginates that endpoint (each page's oldest `saved_at` becomes the next `after=` cursor, until `more` is false), dedupes by canonical URL, and enriches each flat post with its `publication` object (from the response's `publications` array, matched by `publication_id`) and `author_name` (from `publishedBylines`). The full result is cached in `_api_cache` and sliced by offset; `_fetch_saved_posts_page_impl()` accepts an optional `playwright_instance` (same pattern as the notes fetcher and `_fetch_via_dom`) so this caching/slicing/fallback branch is directly testable with a Playwright double instead of only reachable through a real browser. Each page request goes through `_reader_api_get()`, which retries transient failures (HTTP 429 and 5xx) up to `max_retries` (default 3) with `Retry-After`-aware backoff: `_retry_after_seconds()` honors an integer `Retry-After` header (clamped to a 30s cap so a hostile value can't hang the sync) and otherwise uses capped exponential backoff (0.5s, 1s, 2s, ...); 401/403 and other 4xx are returned unretried so the existing auth/fallback handling applies. A 429 that survives all retries is deliberately treated as "unavailable" (partial list, or `None` → DOM fallback) rather than as a silent empty-success, so rate-limiting never masquerades as "you have no saved posts." Both `_reader_api_get()` and `_fetch_all_saved_via_reader_api()` accept an injectable `sleep_func` (default `time.sleep`) so tests exercise the backoff without real delays. If that endpoint is unavailable, it falls back to headless DOM extraction on `https://substack.com/saved`, which scrapes `div.reader2-post-container` cards, caches scrolling results in `_dom_cache`, and marks each dict with `_dom: True`; DOM cards only expose a localized relative publish string and no bookmark time, so `saved_at` stays unknown on that path. Both `save_post()` and `unsave_post()` were originally guesswork (an unverified DOM button selector plus an unverified `POST /api/v1/bookmark` call) and were fixed once real endpoints were captured via `inspect-network`. Every post page server-renders a `window._preloads` blob containing that post's own numeric ID and rich metadata (`preloads.post.id`, `.title`, `.audience`, `.description`/`.subtitle`, `.post_date`, plus `preloads.pub.name`); `_save_post_impl` reads this via `page.evaluate("() => window._preloads")` right after page load and, when an ID is found, calls the real `POST https://substack.com/api/v1/posts/saved` endpoint (body `{"post_id": ...}`) directly via the Playwright API request context — this also lets it populate the returned `SavedPost` with accurate title/publication/audience/excerpt instead of parsing `page.title()`. `_unsave_post_impl` does the mirror image: when the post's `substack_post_id` is already known (true for any post that has been through a normal `sync`, since the reader API's `id` field populates it), it calls `DELETE https://substack.com/api/v1/posts/saved` with the same body shape, no DOM interaction at all. Both treat an `ok` response as `"confirmed"` and only fall back to the old best-effort DOM click (`_click_bookmark_toggle()`, used when the numeric ID can't be obtained or the direct call doesn't confirm) — note its selector is English-only (`aria-label*='save'/'bookmark'`) and can silently fail on non-English Substack UIs, which is part of why the direct API path is preferred whenever possible. `_click_bookmark_toggle()` fingerprints the button's `aria-label`/`aria-pressed`/`class` before and after the click and returns `"confirmed"` only if that fingerprint changed, else `"unconfirmed"`, `"not_found"`, or `"click_failed"` — `save_post()` returns `(SavedPost, confirmation)` and `unsave_post()` returns just the confirmation string; both accept an optional `playwright_instance` for test injection (same pattern as `_fetch_via_dom`). `fetch_post_content()` reuses the same `window._preloads` mechanism to retrieve a saved post's full content: `_fetch_post_content_impl` navigates to the post's page and reads `preloads.post.body_html` (Substack's field name for full content — `parse_remote_post()` already expects this key from the reader API, though the saved-list payload never actually populates it, only individual post pages do), returning `None` for `body_html` if the page's embed format doesn't expose it (frontend change) or the account lacks paywall access; this is the case where the caller should be told to run `inspect-network` against an open post page to re-discover the real content source. `cli.py`'s `inspect-network` command (which logs any `api/v1`/`bookmark`/`saved`/`notes?`/`comment`/`reader`/`feed`/`restack` request's method, URL, status, and JSON response body) is the tool for discovering/re-verifying these endpoints when Substack's frontend changes. It supports `--authenticated/--anonymous` (reuses `storage_state.json` via `get_storage_state_path()` so `/saved` renders the real logged-in page instead of the marketing page), `--url` to target any page, `--filter` to override the response-matching regex, `--max-body` to cap logged response bytes, and `--out PATH` to append each exchange as JSON Lines for later grepping. Response bodies are captured via `context.route("**/*", handle_route)` + `route.fetch()`/`route.fulfill()`, not the more obvious `page.on("response", ...)` + `response.text()`: the latter can deadlock calling `response.text()` inside a sync-API response event handler on the same driver thread, and in practice silently produced zero captured bodies across an entire 108-request session before this fix (every `response_body` came back `null` with no error surfaced) — `route.fetch()` reads the body outside that handler and `route.fulfill(response=...)` re-serves the exact response so page behavior is unaffected; a body that still can't be read now logs a visible yellow warning instead of writing `null` silently.
+
+Saved-posts fetching has a three-source chain, controlled by `get_saved_posts_source()`/`SUBSTACK_SAVED_POSTS_SOURCE` (`auto` default, or a forced `legacy`/`unified`/`dom`): the unified `GET /api/v1/reader/saved?filter=posts` endpoint (the same one notes uses with `filter=notes`) is now tried first, then the legacy reader-posts API above as an automatic fallback, then the DOM fallback. This was a migration decision, not a default guess — see `PLAN-newapi.md` for the full evidence-gathering and decision-gate process. Two tools were built to make that decision on evidence rather than guesswork: `probe-api URL [--out PATH]` headlessly GETs any known API URL with the saved session (no browser page) and dumps the raw JSON — a lighter complement to `inspect-network`, which is for *discovering* unknown endpoints by watching a live browsing session; and `compare-saved-apis [--out PATH]` fetches the full saved list from both sources in one session and reports counts, URL-set differences, per-field presence on the unified payload, and whether `post.saved_at` is present and ordered. A live run against a real ~1000-post account (after a first attempt was invalidated by rate limiting immediately following a large legacy fetch — `compare-saved-apis` now pauses and retries harder around the second fetch for exactly this reason) found the unified endpoint a **strict superset**: 1079 posts vs. legacy's 985, with only 1 post legacy had that unified lacked, `saved_at`/`id`/`canonical_url`/`title`/`post_date`/`audience`/`wordcount` all on 100% of items, correctly ordered newest-first. A follow-up shadow sync (via the real `sync --force` code path, into throwaway databases, same live session) showed *why*: legacy's own count is unstable run-to-run (985, then 1080 across two consecutive syncs), consistent with a real latent bug — `_fetch_all_saved_via_reader_api()` returns `all_posts if all_posts else None` when a mid-pagination 429 survives every retry, silently returning a partial list with no "this is incomplete" signal, rather than surfacing the truncation. The unified endpoint held steady at 1079 across both the comparison and the shadow sync. `_fetch_all_saved_posts_via_unified_api()` is a structural sibling of `_fetch_all_saved_notes_via_api()` (opaque `nextCursor` pagination, not the legacy `after=<ISO saved_at>` scheme), cached separately in `_unified_api_cache`/`_unified_api_failed` so one source's failure can't poison another's mid-run cache, same as the posts/notes cache split. `parse_remote_post()`'s `saved_at` lookup checks both the item level (legacy) and `post_obj` (unified: `post.saved_at`), since the two APIs nest it differently. The mid-pagination silent-truncation bug is now fixed, and turned out to be shared by all three cursor-paginated fetchers (legacy posts, unified posts, and — pre-existing, since before this session — notes): each still returns `all_posts`/`all_notes` `if ... else None` when a page request fails after every retry (so a persistent 429 doesn't throw away progress already made), but now also sets a per-source instance flag (`_api_truncated` / `_unified_api_truncated` / `_notes_api_truncated`, reset at the top of each fetch and in `reset_cache()`) whenever that partial-but-nonempty return fires. `is_posts_fetch_truncated()`/`is_notes_fetch_truncated()` read whichever cache is currently populated and report its flag (using `getattr()` defaults so a test double that skips `__init__`, the established pattern in this repo's sync tests, safely reports "not truncated" instead of raising). `sync_saved_posts()`/`sync_saved_notes()` check this after the fetch loop: on a `--force` sync, a truncated fetch **skips reconciliation entirely** rather than soft-deleting posts/notes that are merely absent from an incomplete list (they may still be saved remotely — the fetch just couldn't reach them this run), and the returned `SyncRun` gets the previously-unused `status = "partial"` (already documented in the model's comment, never actually emitted before this fix) with an `error_message` explaining what happened. The CLI's `sync` command treats `"partial"` as a qualified success (yellow, not red) — `posts_ok`/`notes_ok` still count it as having run — and prints the warning; `status` needs no change since it already displays whatever `last_sync_status` string is stored, and `get_status()`'s `status = 'success'` filter for "last successful sync" already correctly excludes a `partial` run.
+- **Saved notes** (Substack's short-form posts, distinct from long-form posts) are modeled and synced separately from posts, in their own `notes` table/`SavedNote`/`NoteSummary` models, discovered and confirmed live against a real account rather than guessed. The saved-notes list is `GET https://substack.com/api/v1/reader/saved?filter=notes` (also `filter=all`/`filter=posts` — a newer, unified endpoint Substack has added alongside the legacy posts-only `/api/v1/reader/posts?inboxType=saved` that `_fetch_all_saved_via_reader_api()` still uses; posts intentionally were not migrated to it). Its response is `{"items": [...], "nextCursor": <opaque base64 JSON token> | null}`; pagination resubmits the server's own `nextCursor` value as a `cursor=` query param rather than the `after=<ISO timestamp>` scheme posts uses, and the `limit=` query param has no effect (the server returns a fixed page size), so `_fetch_all_saved_notes_via_api()` is a standalone paginator, not a generalization of `_fetch_all_saved_via_reader_api()`. Each item is `{"entity_key": "c-<id>", "publication": null|{...}, "post": null|{...}, "comment": {...}}`; a restack is represented by the top-level `publication`/`post` fields being populated instead of null (unconfirmed by a live example in the account used to discover this, but structurally solid since that item shape is shared with `filter=posts`). Critically, **the endpoint never exposes a bookmark timestamp** — only `comment.is_saved: bool` — so `SavedNote.saved_at` is always `None`, and `sync_saved_notes()`'s incremental early-stop check compares "already saved locally" (`existing.is_saved == 1`) rather than a `saved_at` match the way `sync_saved_posts()` does. A note's identity is its numeric `comment.id`, which is also embedded in its permalink (`https://substack.com/@<handle>/note/c-<id>`, confirmed live via click-tracking) — `_extract_note_id()` parses it back out of a URL via regex, and `upsert_note()`/`reconcile_unsaved_notes()`/`get_note_by_substack_id()` key on that id (not URL, unlike posts) since a note may lack a stable permalink. `comment.body` is already Substack-flattened plain text; `comment.body_json` is a ProseMirror doc (`{type:"doc", content:[{type:"paragraph"|"orderedList"|"listItem",...}]}`, text nodes carrying `marks:[{type:"bold"|"italic"|"link"}]`) that `content_utils.note_body_to_text()`/`prosemirror_to_llm_text()` convert to the same `**bold**`/`*italic*`/`text (url)` conventions `html_to_llm_text()` uses for posts; `body_raw` stores the source verbatim as a hedge against the body-format guess needing correction later. The individual-note-fetch endpoint `GET /api/v1/reader/comment/{id}` → `{"item": {"comment": {...}}}` and the bookmark endpoints `POST`/`DELETE https://substack.com/api/v1/note/c-{id}/save` (id in the URL path, no request body) are all plain authenticated calls through Playwright's `p.request` API context — **no browser page or `window._preloads` lookup is needed for notes at all**, unlike posts, so `save_note()`/`unsave_note()`/`fetch_note_content()` in `substack_client.py` never launch `p.chromium`. Notes have no DOM fallback and no `_click_bookmark_toggle`-style guesswork: the notes card markup is uncaptured, and since the note id is the identity, a DOM-scraped row without one could never reconcile, so an unavailable notes API raises `SubstackClientError` naming `inspect-network` rather than degrading silently. `sync_runs.entity` (`'post'` | `'note'`, default `'post'` for pre-existing rows) scopes `get_status()`'s "last sync" queries per entity so a notes sync can't be reported as the posts' last sync result.
+- `content_utils.py` converts a post's raw `body_html` into clean text for LLM consumption. `html_to_llm_text()` is a small `html.parser.HTMLParser` subclass (no external HTML library dependency) that renders headings as markdown `#` prefixes, list items as `- ` bullets, links as `text (url)`, keeps minimal `**bold**`/`*italic*` markers, drops `script`/`style`/`iframe` content entirely, and collapses excess blank lines. `format_post_for_llm()` prepends a plain metadata header (title, publication, author, published date, URL) to the cleaned body text — this combined string is what both the CLI's `get-content` command and the MCP `get_post_content` tool return. `note_body_to_text()` handles the same problem for notes' `body_json`/`body`, dispatching a dict (or JSON string) to `prosemirror_to_llm_text()` (a small recursive walker over Substack's ProseMirror doc shape — `paragraph`/`heading`/`bulletList`/`orderedList`/`listItem`/`hard_break` nodes, `bold`/`italic`/`link` marks — producing the same `**bold**`/`*italic*`/`text (url)` conventions as `html_to_llm_text()`) and a plain string to either the HTML converter (if it looks like HTML) or straight through; `format_note_for_llm()` builds the note-specific header (author/handle/posted date/URL, plus a `Restacked: <title> (<url>)` line when applicable) rather than reusing `format_post_for_llm()`, since `publication_name` is required there but meaningless for most notes.
+- `sync.py` converts API/DOM payloads to `SavedPost` models and coordinates paginated incremental or full syncs. `parse_remote_post()` routes DOM dicts (identified by the `_dom` marker) through a minimal mapping and everything else through the full reader-API/legacy mapping; it never fabricates a `saved_at` from the sync moment (unknown save times stay `None`). `word_count` is mapped from `wordcount`/`word_count`/`words` via `_first_positive_int()` (which ignores non-positive/uncoercible values); the `compare-saved-apis` parity run confirmed the real field is `wordcount` (100% populated on the unified endpoint's payload, 0% on `word_count`/`words`) — the defensive multi-key mapping is kept since `_first_positive_int()` already checks `wordcount` first and the other two cost nothing to keep as a hedge. `reading_time_minutes` is intentionally *derived* (ceil of `word_count / WORDS_PER_MINUTE`, ~200 wpm) rather than mapped from a field, because a wrong guess about that field's unit (seconds vs minutes) would persist a badly wrong value; deriving is unit-unambiguous. `image_url` is mapped from `cover_image`/`image_url` on the reader-API payload and is populated for the large majority of synced posts (confirmed live: ~98%); it was previously stored but invisible through every read path except `get_saved_post`/the full `SavedPost` — `PostSummary` and the `list_posts()`/`search_posts()` SELECTs now include it too; the CLI `search` command prints it when present, but `list` deliberately omits it since it's a long, uninformative CDN URL in that terser view. `content_text` remains unpopulated by sync (only `get-content`/`get_post_content` populate it) because the saved-list payload doesn't carry post body HTML, only individual post pages do. There is no `metadata_json` column: it was a never-populated "raw source JSON" placeholder from the original PLAN and has been removed from the model and schema. Existing databases created before its removal may still have an inert `metadata_json` column; reads tolerate it because `SavedPost` (Pydantic) ignores unknown columns from `SELECT *`, and `upsert_post` no longer references it. A force/full sync (`--force`) collects every fetched post's URL and, after the fetch loop completes, calls `reconcile_unsaved_posts()` to soft-delete any locally `is_saved = 1` post absent from that complete remote set — this is how posts unsaved directly on Substack (outside this tool) get reflected locally. An incremental sync never reconciles, since its early-stop-on-matches optimization means it only sees a partial remote list. It records each run in SQLite (including `reconciled_count`) and returns a `SyncRun` with `success`, `auth_required`, or `failed` status instead of propagating expected authentication failures. `sync_saved_notes()` is a sibling of `sync_saved_posts()`, not a generalization of it (notes reconcile by Substack note id via `reconcile_unsaved_notes()`, not URL, and the incremental early-stop check compares `existing.is_saved == 1` rather than a `saved_at` match, since notes carry none); both write `sync_runs.entity` (`'post'`/`'note'`) and share a `_build_sync_run()` helper instead of each hand-constructing `SyncRun` three times; `_build_sync_run()` takes the real `started_at` (captured once at the top of each sync function, right before `start_sync_run()`) rather than stamping it at finish time as all three constructions originally did, which had made every returned `SyncRun.started_at` equal to `completed_at` regardless of how long the sync actually took. `parse_remote_note()` unwraps a saved-notes list item (or a single-note-fetch `{"item": {...}}` wrapper) into a `SavedNote`, returning `None` (so the sync loop skips it) rather than inventing an id when `comment.id` is missing; it delegates body conversion to `content_utils.note_body_to_text()` and derives `word_count` from the resulting text rather than any source field.
+- `database.py` is the SQLite repository and schema owner. The `posts` table is the cache; `sync_runs` records sync history; `posts_fts` is an external-content FTS5 index maintained by database triggers. `reconcile_unsaved_posts()` bulk soft-deletes posts missing from a given complete remote URL set and is a no-op on an empty list (an empty list is more likely a fetch problem than genuine mass-unsaving). `init_db()` runs additive column migrations (e.g. `posts.audience`, `sync_runs.reconciled_count`) via `ALTER TABLE` guarded by an existence check or `try`/`except sqlite3.OperationalError`, since `CREATE TABLE IF NOT EXISTS` never adds columns to an already-existing table; the `posts.audience` check runs before the `executescript` block because the `idx_posts_audience` index creation inside it would otherwise fail on a pre-migration table. `list_posts()`/`search_posts()` accept an `audience` filter (exact match, case-insensitive), and `list_audiences()` returns the distinct audience values actually present in the cache with post counts, rather than hardcoding Substack's (undocumented, possibly-growing) audience enum. `PostSummary` (returned by `list_posts()`/`search_posts()` and the MCP list/search tools) carries `reading_time_minutes` and `word_count`, and the CLI `list`/`search` commands display them — without this the values were populated in the `posts` table but invisible through every read path except `get_saved_post`/the full `SavedPost`. A parallel `notes` table (with its own `notes_fts` external-content index and `notes_ai`/`notes_ad`/`notes_au` triggers) holds saved notes; unlike `posts`, its unique identity column is `substack_note_id`, not `url` (a note may have no stable permalink), so `upsert_note()` does an explicit lookup-then-branch UPDATE/INSERT on `substack_note_id` rather than `INSERT ... ON CONFLICT(url)`. `upsert_post()` was originally the opposite (looked up by `substack_post_id` *or* `url` but written via `INSERT ... ON CONFLICT(url)`), which meant a post found by id whose canonical URL had since changed (slug rename, custom-domain migration) couldn't conflict on the new URL and tripped the `substack_post_id UNIQUE` constraint on the fresh INSERT instead; it now uses the same explicit lookup-then-branch UPDATE/INSERT pattern as `upsert_note()`, closing that crash path. `search_posts()`'s LIKE fallback (used when an FTS5 query string is malformed) shares its WHERE-clause construction with the main FTS branch via `_post_search_filters()` — mirroring `_note_search_filters()` — so a malformed query never silently drops the `publication`/`audience`/date-range filters or an `is_saved_only=False` caller the way the fallback originally did. `get_note_by_substack_id()` exists because `get_note()`'s polymorphic `url_or_id` dispatch treats any all-digit string as a local row id — and `substack_note_id` values are always digits — so callers that only have the Substack note id (the sync loop's incremental check) must use it instead of risking a collision with an unrelated local row. `sync_runs.entity` (`'post'`/`'note'`, additive column defaulting existing rows to `'post'`) scopes `get_status()`'s "last successful sync"/"last sync status" queries per entity, which is the one existing posts read path notes support required changing (it previously read the last `sync_runs` row unscoped, which was harmless with one entity and would otherwise let a notes sync be reported as the posts' last sync).
+- `config.py` centralizes application paths. By default, data lives under `~/.local/share/substack-saved-mcp`; `SUBSTACK_SAVED_DB_PATH`, `SUBSTACK_SAVED_DATA_DIR`, and `SUBSTACK_SAVED_BROWSER_DIR` override those paths. `get_saved_posts_source()`/`SUBSTACK_SAVED_POSTS_SOURCE` (`auto`/`legacy`/`unified`/`dom`, falling back to `auto` on an unrecognized value) selects which of the three saved-posts fetch sources `_fetch_saved_posts_page_impl()` tries, so a source-specific regression is one env var away from a revert rather than a code rollback.
+
+## Repository Conventions
+
+- Model persisted posts with the Pydantic `SavedPost` schema. Keep `published_at` (post publication time) distinct from `saved_at` (bookmark time). Both come from the source payload; never fabricate `saved_at` from the sync/DB-insert moment — leave it `None` when the source does not expose the original bookmark time (`upsert_post` preserves a known value and stores `NULL` otherwise).
+- Store the raw Substack `audience` string (e.g. `everyone`, `only_paid`) verbatim on `SavedPost.audience` rather than only collapsing it into the `is_paywalled` boolean; `parse_remote_post()` still derives `is_paywalled` from it (`audience == "only_paid"`) for backward-compatible boolean filtering, but the original value is preserved for exact-match filtering and discovery via `list_audiences()`.
+- Canonicalize every post URL through `canonicalize_url()` before storage or lookup. It strips known tracking parameters, fragments, and non-root trailing slashes; SQLite uniqueness and upsert behavior depend on that canonical form.
+- Treat unsaving as a soft delete: use `soft_delete_post()` (single post, e.g. the `unsave` command/tool) or `reconcile_unsaved_posts()` (bulk, called from a force sync) to set `is_saved = 0` and `unsaved_at`, retaining the record for history. Default list/search/publication queries intentionally include only `is_saved = 1`.
+- The local cache always reflects the user's intent for `save`/`unsave`, regardless of whether the remote toggle on Substack could be confirmed — never block the local write on remote confirmation. Instead, surface the confirmation status (`"confirmed"` vs. `"unconfirmed"/"not_found"/"click_failed"`) as a clearly-labeled warning to the caller (CLI: yellow `click.secho`; MCP: a `warning` key and `remote_confirmed: bool` in the tool's dict response) so an unconfirmed remote action isn't silently reported as a full success. A subsequent `sync --force` will self-correct the local cache via `reconcile_unsaved_posts()` if a "confirmed" save/unsave didn't actually happen remotely.
+- Do not write directly to `posts_fts` or `notes_fts`; use the repository operations so the SQLite insert/update/delete triggers maintain the FTS5 indexes.
+- Notes are identified by `substack_note_id`, not URL — a note may lack a stable permalink, unlike a post. Never give notes a DOM-scrape fallback: the notes card markup is unverified, and since a note's id *is* its identity, a DOM-scraped row without one could never be reconciled; an unavailable notes API should raise a clear error naming `inspect-network` instead of degrading to guesswork the way the old `save_post`/`unsave_post` once did before their real endpoints were captured.
+- Database-facing tests use `tmp_path` databases. CLI and MCP tests set `SUBSTACK_SAVED_DB_PATH` with `monkeypatch` so they never access a developer's real cache. Sync tests inject a `SubstackSavedPostsClient` substitute rather than opening Playwright or contacting Substack.
+- Browser state and SQLite data are credentials or user data. Keep `storage_state.json`, browser-state directories, `.env` files, and database files untracked; configuration creates data directories with user-only permissions on POSIX systems.

{substack_saved_mcp-0.1.0 → substack_saved_mcp-0.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
-Metadata-Version: 2.4
+Metadata-Version: 2.5
 Name: substack-saved-mcp
-Version: 0.1.0
+Version: 0.3.0
 Summary: Local stdio-based MCP server and sync engine for Substack saved posts
 Author-email: Toni Hermoso Pulido <toniher@cau.cat>
 License-Expression: MIT
@@ -16,18 +16,21 @@ Requires-Dist: pytest>=8.0.0; extra == 'dev'
 Requires-Dist: ruff==0.16.1; extra == 'dev'
 Description-Content-Type: text/markdown
 
-# Substack Saved Posts MCP & CLI
+# Substack Saved Posts & Notes MCP & CLI
 
-A local, stdio-based Model Context Protocol (MCP) server and sync engine for your saved/bookmarked Substack posts.
+[![PyPI version](https://img.shields.io/pypi/v/substack-saved-mcp.svg)](https://pypi.org/project/substack-saved-mcp/)
+
+A local, stdio-based Model Context Protocol (MCP) server and sync engine for your saved/bookmarked Substack **posts and notes**.
 
 ## Features
 
 - **Read & Search**: Full-text search (SQLite FTS5) across saved post titles, excerpts, authors, and publications. Filter by publication, audience tier (e.g. `everyone`, `only_paid`), and date ranges (`published_at` vs `saved_at`). Search also covers a post's **full body text**, but only for posts whose content has already been fetched once via `get-content` / the `get_post_content` tool — a normal `sync` stores metadata and excerpts, not full bodies, so posts you haven't opened yet are matched on their title/excerpt/metadata only, not their full text.
-- **Full Content for LLMs**: Fetch a saved post's full content and get it back cleaned and formatted (headings, lists, links) for feeding directly to an LLM, with the result cached locally for next time.
-- **Save & Unsave**: Bookmark new Substack posts or unbookmark existing ones via authenticated browser sessions.
+- **Saved Notes, too**: Substack's short-form notes are synced, searched, and cached separately from posts (they carry an author and body rather than a title or publication tier). Full-text search covers note bodies, authors, and restacked-post titles. Notes never require a browser at all — every notes operation (sync, save, unsave, full-content fetch) is a plain authenticated API call.
+- **Full Content for LLMs**: Fetch a saved post's or note's full content and get it back cleaned and formatted (headings, lists, links) for feeding directly to an LLM, with the result cached locally for next time.
+- **Save & Unsave**: Bookmark new Substack posts and notes, or unbookmark existing ones. Posts go through an authenticated browser session; notes are API-only.
 - **Offline First**: Fast, offline queries directly from local SQLite cache.
 - **Privacy & Security**: Keeps session credentials local, redacting tokens from logs.
-- **FastMCP Protocol**: Stdio MCP interface with rich tool suite and resources.
+- **FastMCP Protocol**: Stdio MCP interface with rich tool suite and resources for both posts and notes.
 
 ---
 
@@ -35,9 +38,16 @@ A local, stdio-based Model Context Protocol (MCP) server and sync engine for you
 
 [`uv`](https://github.com/astral-sh/uv) is the recommended fast Python package manager for installing and running `substack-saved-mcp`.
 
-### Option A: Install System-wide as a Tool (`uv tool install`)
+### Option A: Install from PyPI as a System-wide Tool (`uv tool install`)
+
+Install the published package from [PyPI](https://pypi.org/project/substack-saved-mcp/):
+
+```bash
+# Install system-wide into an isolated uv environment
+uv tool install substack-saved-mcp
+```
 
-Install directly from your local repository folder:
+To install directly from a local repository folder instead:
 
 ```bash
 # Navigate to the repository
@@ -97,9 +107,13 @@ substack-saved-mcp init
 # 2. Authenticate with Substack (opens interactive browser window once)
 substack-saved-mcp login
 
-# 3. Sync saved posts into local cache
+# 3. Sync saved posts AND notes into local cache (both entities by default)
 substack-saved-mcp sync
 
+# 3b. Or sync just one entity
+substack-saved-mcp sync --only posts
+substack-saved-mcp sync --only notes
+
 # 4. Search saved posts via CLI
 substack-saved-mcp search "artificial intelligence"
 
@@ -115,7 +129,18 @@ substack-saved-mcp unsave "https://example.substack.com/p/post-slug"
 # 6. Get a saved post's full content, cleaned up and ready for an LLM
 substack-saved-mcp get-content "https://example.substack.com/p/post-slug"
 
-# 7. Launch stdio MCP server
+# 7. Work with saved notes the same way
+substack-saved-mcp list-notes --limit 10
+substack-saved-mcp search-notes "kubernetes" --author alice
+substack-saved-mcp note-authors
+substack-saved-mcp save-note "https://substack.com/@handle/note/c-123456"
+substack-saved-mcp unsave-note "https://substack.com/@handle/note/c-123456"
+substack-saved-mcp get-note "https://substack.com/@handle/note/c-123456"
+
+# 8. Check combined status (posts and notes counts, last sync per entity)
+substack-saved-mcp status
+
+# 9. Launch stdio MCP server
 substack-saved-mcp serve
 ```
 
@@ -177,14 +202,39 @@ export SUBSTACK_SAVED_DB_PATH="/path/to/my/custom_database.sqlite"
 export SUBSTACK_SAVED_DATA_DIR="/path/to/my/data_dir"
 ```
 
+### Which Substack API does syncing saved posts use?
+
+Saved posts are fetched from Substack's newer unified reader API by default, with
+automatic fallback to an older, posts-only API and finally to headless browser
+scraping if needed — you shouldn't normally need to think about this. If you ever
+want to force a specific source (e.g. while troubleshooting), set:
+```bash
+export SUBSTACK_SAVED_POSTS_SOURCE="unified"  # or "legacy" or "dom"
+```
+Leave it unset (or `"auto"`) for the default, self-healing behavior.
+
+### What does a "partial" sync status mean?
+
+If Substack rate-limits (HTTP 429) a sync so heavily that a page of results
+can't be fetched even after retrying, the sync keeps whatever it already
+fetched rather than failing outright, and reports `status: partial` (instead
+of `success`) in `substack-saved-mcp status` or the sync tool's response.
+When this happens on a `sync --force`, reconciliation (soft-deleting posts/notes
+no longer in the remote list) is automatically skipped for that run, so a post
+or note that merely couldn't be fetched is never mistaken for one you actually
+unsaved on Substack. Just run `sync` again later — a subsequent successful run
+picks up anything that was missed.
+
 ### Will a browser window pop up when running as an MCP server?
 
 **No, a visible browser window will not open during normal MCP operations.**
 
-- **Read & Search Tools** (`search_saved_posts`, `list_saved_posts`, `get_saved_post`, `list_publications`, `saved_posts_status`):  
+- **Read & Search Tools** (`search_saved_posts`, `list_saved_posts`, `get_saved_post`, `search_saved_notes`, `list_saved_notes`, `get_saved_note`, `list_publications`, `list_audiences`, `saved_posts_status`):  
   Operate 100% offline using the local SQLite database. Zero browser activity.
-- **Sync & Write Tools** (`sync_saved_posts`, `save_post`, `unsave_post`, `get_post_content`):  
+- **Post Sync & Write Tools** (`sync_saved_posts`, `save_post`, `unsave_post`, `get_post_content`):  
   Run in **headless background mode** using the pre-authenticated session stored in `storage_state.json`.
+- **Note Sync & Write Tools** (`sync_saved_notes`, `save_note`, `unsave_note`, `get_note_content`):  
+  Never open a browser page at all, headless or otherwise — Substack's notes endpoints are plain authenticated HTTP calls, so these tools only ever make direct API requests using `storage_state.json`.
 - **Interactive Login**:  
   A visible browser window opens **only** when you manually run `substack-saved-mcp login` from your terminal. If your session expires while using an MCP client, the tool will return a clear error message instructing you to re-authenticate via `substack-saved-mcp login` instead of popping open a browser window unexpectedly.
 

{substack_saved_mcp-0.1.0 → substack_saved_mcp-0.3.0}/README.md

@@ -1,15 +1,18 @@
-# Substack Saved Posts MCP & CLI
+# Substack Saved Posts & Notes MCP & CLI
 
-A local, stdio-based Model Context Protocol (MCP) server and sync engine for your saved/bookmarked Substack posts.
+[![PyPI version](https://img.shields.io/pypi/v/substack-saved-mcp.svg)](https://pypi.org/project/substack-saved-mcp/)
+
+A local, stdio-based Model Context Protocol (MCP) server and sync engine for your saved/bookmarked Substack **posts and notes**.
 
 ## Features
 
 - **Read & Search**: Full-text search (SQLite FTS5) across saved post titles, excerpts, authors, and publications. Filter by publication, audience tier (e.g. `everyone`, `only_paid`), and date ranges (`published_at` vs `saved_at`). Search also covers a post's **full body text**, but only for posts whose content has already been fetched once via `get-content` / the `get_post_content` tool — a normal `sync` stores metadata and excerpts, not full bodies, so posts you haven't opened yet are matched on their title/excerpt/metadata only, not their full text.
-- **Full Content for LLMs**: Fetch a saved post's full content and get it back cleaned and formatted (headings, lists, links) for feeding directly to an LLM, with the result cached locally for next time.
-- **Save & Unsave**: Bookmark new Substack posts or unbookmark existing ones via authenticated browser sessions.
+- **Saved Notes, too**: Substack's short-form notes are synced, searched, and cached separately from posts (they carry an author and body rather than a title or publication tier). Full-text search covers note bodies, authors, and restacked-post titles. Notes never require a browser at all — every notes operation (sync, save, unsave, full-content fetch) is a plain authenticated API call.
+- **Full Content for LLMs**: Fetch a saved post's or note's full content and get it back cleaned and formatted (headings, lists, links) for feeding directly to an LLM, with the result cached locally for next time.
+- **Save & Unsave**: Bookmark new Substack posts and notes, or unbookmark existing ones. Posts go through an authenticated browser session; notes are API-only.
 - **Offline First**: Fast, offline queries directly from local SQLite cache.
 - **Privacy & Security**: Keeps session credentials local, redacting tokens from logs.
-- **FastMCP Protocol**: Stdio MCP interface with rich tool suite and resources.
+- **FastMCP Protocol**: Stdio MCP interface with rich tool suite and resources for both posts and notes.
 
 ---
 
@@ -17,9 +20,16 @@ A local, stdio-based Model Context Protocol (MCP) server and sync engine for you
 
 [`uv`](https://github.com/astral-sh/uv) is the recommended fast Python package manager for installing and running `substack-saved-mcp`.
 
-### Option A: Install System-wide as a Tool (`uv tool install`)
+### Option A: Install from PyPI as a System-wide Tool (`uv tool install`)
+
+Install the published package from [PyPI](https://pypi.org/project/substack-saved-mcp/):
+
+```bash
+# Install system-wide into an isolated uv environment
+uv tool install substack-saved-mcp
+```
 
-Install directly from your local repository folder:
+To install directly from a local repository folder instead:
 
 ```bash
 # Navigate to the repository
@@ -79,9 +89,13 @@ substack-saved-mcp init
 # 2. Authenticate with Substack (opens interactive browser window once)
 substack-saved-mcp login
 
-# 3. Sync saved posts into local cache
+# 3. Sync saved posts AND notes into local cache (both entities by default)
 substack-saved-mcp sync
 
+# 3b. Or sync just one entity
+substack-saved-mcp sync --only posts
+substack-saved-mcp sync --only notes
+
 # 4. Search saved posts via CLI
 substack-saved-mcp search "artificial intelligence"
 
@@ -97,7 +111,18 @@ substack-saved-mcp unsave "https://example.substack.com/p/post-slug"
 # 6. Get a saved post's full content, cleaned up and ready for an LLM
 substack-saved-mcp get-content "https://example.substack.com/p/post-slug"
 
-# 7. Launch stdio MCP server
+# 7. Work with saved notes the same way
+substack-saved-mcp list-notes --limit 10
+substack-saved-mcp search-notes "kubernetes" --author alice
+substack-saved-mcp note-authors
+substack-saved-mcp save-note "https://substack.com/@handle/note/c-123456"
+substack-saved-mcp unsave-note "https://substack.com/@handle/note/c-123456"
+substack-saved-mcp get-note "https://substack.com/@handle/note/c-123456"
+
+# 8. Check combined status (posts and notes counts, last sync per entity)
+substack-saved-mcp status
+
+# 9. Launch stdio MCP server
 substack-saved-mcp serve
 ```
 
@@ -159,14 +184,39 @@ export SUBSTACK_SAVED_DB_PATH="/path/to/my/custom_database.sqlite"
 export SUBSTACK_SAVED_DATA_DIR="/path/to/my/data_dir"
 ```
 
+### Which Substack API does syncing saved posts use?
+
+Saved posts are fetched from Substack's newer unified reader API by default, with
+automatic fallback to an older, posts-only API and finally to headless browser
+scraping if needed — you shouldn't normally need to think about this. If you ever
+want to force a specific source (e.g. while troubleshooting), set:
+```bash
+export SUBSTACK_SAVED_POSTS_SOURCE="unified"  # or "legacy" or "dom"
+```
+Leave it unset (or `"auto"`) for the default, self-healing behavior.
+
+### What does a "partial" sync status mean?
+
+If Substack rate-limits (HTTP 429) a sync so heavily that a page of results
+can't be fetched even after retrying, the sync keeps whatever it already
+fetched rather than failing outright, and reports `status: partial` (instead
+of `success`) in `substack-saved-mcp status` or the sync tool's response.
+When this happens on a `sync --force`, reconciliation (soft-deleting posts/notes
+no longer in the remote list) is automatically skipped for that run, so a post
+or note that merely couldn't be fetched is never mistaken for one you actually
+unsaved on Substack. Just run `sync` again later — a subsequent successful run
+picks up anything that was missed.
+
 ### Will a browser window pop up when running as an MCP server?
 
 **No, a visible browser window will not open during normal MCP operations.**
 
-- **Read & Search Tools** (`search_saved_posts`, `list_saved_posts`, `get_saved_post`, `list_publications`, `saved_posts_status`):  
+- **Read & Search Tools** (`search_saved_posts`, `list_saved_posts`, `get_saved_post`, `search_saved_notes`, `list_saved_notes`, `get_saved_note`, `list_publications`, `list_audiences`, `saved_posts_status`):  
   Operate 100% offline using the local SQLite database. Zero browser activity.
-- **Sync & Write Tools** (`sync_saved_posts`, `save_post`, `unsave_post`, `get_post_content`):  
+- **Post Sync & Write Tools** (`sync_saved_posts`, `save_post`, `unsave_post`, `get_post_content`):  
   Run in **headless background mode** using the pre-authenticated session stored in `storage_state.json`.
+- **Note Sync & Write Tools** (`sync_saved_notes`, `save_note`, `unsave_note`, `get_note_content`):  
+  Never open a browser page at all, headless or otherwise — Substack's notes endpoints are plain authenticated HTTP calls, so these tools only ever make direct API requests using `storage_state.json`.
 - **Interactive Login**:  
   A visible browser window opens **only** when you manually run `substack-saved-mcp login` from your terminal. If your session expires while using an MCP client, the tool will return a clear error message instructing you to re-authenticate via `substack-saved-mcp login` instead of popping open a browser window unexpectedly.
 

{substack_saved_mcp-0.1.0 → substack_saved_mcp-0.3.0}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "substack-saved-mcp"
-version = "0.1.0"
+version = "0.3.0"
 description = "Local stdio-based MCP server and sync engine for Substack saved posts"
 readme = "README.md"
 requires-python = ">=3.11"

{substack_saved_mcp-0.1.0 → substack_saved_mcp-0.3.0}/src/substack_saved_mcp/__init__.py

@@ -1,3 +1,3 @@
 """Substack Saved Posts MCP & Sync Application."""
 
-__version__ = "0.1.0"
+__version__ = "0.3.0"