substack-saved-mcp 0.3.2__tar.gz → 0.3.3__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- substack_saved_mcp-0.3.3/CLAUDE.md +50 -0
- substack_saved_mcp-0.3.3/PKG-INFO +286 -0
- substack_saved_mcp-0.3.3/README.md +269 -0
- {substack_saved_mcp-0.3.2 → substack_saved_mcp-0.3.3}/pyproject.toml +1 -1
- {substack_saved_mcp-0.3.2 → substack_saved_mcp-0.3.3}/src/substack_saved_mcp/__init__.py +1 -1
- {substack_saved_mcp-0.3.2 → substack_saved_mcp-0.3.3}/src/substack_saved_mcp/cli.py +25 -271
- {substack_saved_mcp-0.3.2 → substack_saved_mcp-0.3.3}/src/substack_saved_mcp/config.py +2 -2
- {substack_saved_mcp-0.3.2 → substack_saved_mcp-0.3.3}/src/substack_saved_mcp/database.py +2 -2
- {substack_saved_mcp-0.3.2 → substack_saved_mcp-0.3.3}/src/substack_saved_mcp/mcp_server.py +54 -150
- {substack_saved_mcp-0.3.2 → substack_saved_mcp-0.3.3}/src/substack_saved_mcp/substack_client.py +28 -27
- {substack_saved_mcp-0.3.2 → substack_saved_mcp-0.3.3}/src/substack_saved_mcp/sync.py +96 -1
- {substack_saved_mcp-0.3.2 → substack_saved_mcp-0.3.3}/tests/test_database.py +2 -0
- {substack_saved_mcp-0.3.2 → substack_saved_mcp-0.3.3}/tests/test_substack_client.py +50 -0
- {substack_saved_mcp-0.3.2 → substack_saved_mcp-0.3.3}/uv.lock +1 -1
- substack_saved_mcp-0.3.2/CLAUDE.md +0 -50
- substack_saved_mcp-0.3.2/PKG-INFO +0 -286
- substack_saved_mcp-0.3.2/README.md +0 -269
- {substack_saved_mcp-0.3.2 → substack_saved_mcp-0.3.3}/.github/workflows/ci.yml +0 -0
- {substack_saved_mcp-0.3.2 → substack_saved_mcp-0.3.3}/.github/workflows/pypi.yml +0 -0
- {substack_saved_mcp-0.3.2 → substack_saved_mcp-0.3.3}/.gitignore +0 -0
- {substack_saved_mcp-0.3.2 → substack_saved_mcp-0.3.3}/.pre-commit-config.yaml +0 -0
- {substack_saved_mcp-0.3.2 → substack_saved_mcp-0.3.3}/LICENSE +0 -0
- {substack_saved_mcp-0.3.2 → substack_saved_mcp-0.3.3}/PLAN.md +0 -0
- {substack_saved_mcp-0.3.2 → substack_saved_mcp-0.3.3}/src/substack_saved_mcp/content_utils.py +0 -0
- {substack_saved_mcp-0.3.2 → substack_saved_mcp-0.3.3}/src/substack_saved_mcp/models.py +0 -0
- {substack_saved_mcp-0.3.2 → substack_saved_mcp-0.3.3}/src/substack_saved_mcp/url_utils.py +0 -0
- {substack_saved_mcp-0.3.2 → substack_saved_mcp-0.3.3}/tests/test_cli.py +0 -0
- {substack_saved_mcp-0.3.2 → substack_saved_mcp-0.3.3}/tests/test_content_utils.py +0 -0
- {substack_saved_mcp-0.3.2 → substack_saved_mcp-0.3.3}/tests/test_mcp_server.py +0 -0
- {substack_saved_mcp-0.3.2 → substack_saved_mcp-0.3.3}/tests/test_normalization.py +0 -0
- {substack_saved_mcp-0.3.2 → substack_saved_mcp-0.3.3}/tests/test_sync.py +0 -0
- {substack_saved_mcp-0.3.2 → substack_saved_mcp-0.3.3}/tests/test_url_utils.py +0 -0
|
@@ -0,0 +1,50 @@
|
|
|
1
|
+
# CLAUDE.md
|
|
2
|
+
|
|
3
|
+
This file provides guidance to Claude Code when working with this repository.
|
|
4
|
+
|
|
5
|
+
## Commands
|
|
6
|
+
|
|
7
|
+
Use `uv` for all development commands. Set up the environment with:
|
|
8
|
+
|
|
9
|
+
```bash
|
|
10
|
+
uv sync --extra dev
|
|
11
|
+
```
|
|
12
|
+
|
|
13
|
+
| Task | Command |
|
|
14
|
+
| --- | --- |
|
|
15
|
+
| Run all tests | `uv run python -m pytest` |
|
|
16
|
+
| Run one test file | `uv run python -m pytest tests/test_database.py` |
|
|
17
|
+
| Run one test | `uv run python -m pytest tests/test_database.py::test_fts5_search` |
|
|
18
|
+
| Build the package | `uv build` |
|
|
19
|
+
| Run the CLI | `uv run substack-saved-mcp --help` |
|
|
20
|
+
| Reinstall tool globally | `uv tool install . --no-cache --force` |
|
|
21
|
+
|
|
22
|
+
`ruff` is configured in `pyproject.toml` (rules: `E4`, `E7`, `E9`, `F`, `I`, `UP`, `B`, `RUF`, line length 88, target py311). Lint with `uvx ruff check .` or `uv run ruff check .`.
|
|
23
|
+
|
|
24
|
+
After any change to dependencies or tracked files, run `uv lock` to keep `uv.lock` current, then `prek run --all-files` (using the repo's `.pre-commit-config.yaml`) before considering the work done.
|
|
25
|
+
|
|
26
|
+
## Architecture
|
|
27
|
+
|
|
28
|
+
- `cli.py` is the Click entry point (`substack-saved-mcp`). Its commands initialize the database and then delegate to the repository, sync engine, Playwright client, or MCP server.
|
|
29
|
+
- `mcp_server.py` exposes the same application operations as FastMCP stdio tools and resources. Read tools query the local cache; sync and save/unsave operations use the authenticated browser session.
|
|
30
|
+
- `substack_client.py` owns Playwright authentication and remote Substack interaction. `login` is the only intended headful workflow; normal sync and write paths use `storage_state.json` headlessly. Synchronous Playwright API calls are routed through `_run_playwright_sync()` to safely execute in a worker thread if an `asyncio` event loop is active (e.g. under FastMCP). The legacy reader inbox API (`GET /api/v1/reader/posts?inboxType=saved`, second in the source chain described below) exposes the real bookmark timestamp (`saved_at`) and an ISO publication date (`post_date`) per post. `_fetch_all_saved_via_reader_api()` cursor-paginates that endpoint (each page's oldest `saved_at` becomes the next `after=` cursor, until `more` is false), dedupes by canonical URL, and enriches each flat post with its `publication` object (from the response's `publications` array, matched by `publication_id`) and `author_name` (from `publishedBylines`). The full result is cached in `_api_cache` and sliced by offset; `_fetch_saved_posts_page_impl()` accepts an optional `playwright_instance` (same pattern as the notes fetcher and `_fetch_via_dom`) so this caching/slicing/fallback branch is directly testable with a Playwright double instead of only reachable through a real browser. Each page request goes through `_reader_api_get()`, which retries transient failures (HTTP 429 and 5xx) up to `max_retries` (default 3) with `Retry-After`-aware backoff: `_retry_after_seconds()` honors an integer `Retry-After` header (clamped to a 30s cap so a hostile value can't hang the sync) and otherwise uses capped exponential backoff (0.5s, 1s, 2s, ...); 401/403 and other 4xx are returned unretried so the existing auth/fallback handling applies. A 429 that survives all retries is deliberately treated as "unavailable" (partial list, or `None` → DOM fallback) rather than as a silent empty-success, so rate-limiting never masquerades as "you have no saved posts." Both `_reader_api_get()` and `_fetch_all_saved_via_reader_api()` accept an injectable `sleep_func` (default `time.sleep`) so tests exercise the backoff without real delays. If that endpoint is unavailable, it falls back to headless DOM extraction on `https://substack.com/saved`, which scrapes `div.reader2-post-container` cards, caches scrolling results in `_dom_cache`, and marks each dict with `_dom: True`; DOM cards only expose a localized relative publish string and no bookmark time, so `saved_at` stays unknown on that path. Every post page server-renders a `window._preloads` blob containing that post's own numeric ID and rich metadata (`preloads.post.id`, `.title`, `.audience`, `.description`/`.subtitle`, `.post_date`, plus `preloads.pub.name`); `_save_post_impl` reads this via `page.evaluate("() => window._preloads")` right after page load and, when an ID is found, calls the real `POST https://substack.com/api/v1/posts/saved` endpoint (body `{"post_id": ...}`) directly via the Playwright API request context — this also lets it populate the returned `SavedPost` with accurate title/publication/audience/excerpt instead of parsing `page.title()`. `_unsave_post_impl` does the mirror image: when the post's `substack_post_id` is already known (true for any post that has been through a normal `sync`, since the reader API's `id` field populates it), it calls `DELETE https://substack.com/api/v1/posts/saved` with the same body shape, no DOM interaction at all. Both treat an `ok` response as `"confirmed"` and only fall back to the best-effort DOM click (`_click_bookmark_toggle()`) when the numeric ID can't be obtained or the direct call doesn't confirm — never click before the API call, since the button toggles and would unsave an already-saved post. Note its selector is English-only (`aria-label*='save'/'bookmark'`) and can silently fail on non-English Substack UIs, which is part of why the direct API path is preferred whenever possible. `_click_bookmark_toggle()` fingerprints the button's `aria-label`/`aria-pressed`/`class` before and after the click and returns `"confirmed"` only if that fingerprint changed, else `"unconfirmed"`, `"not_found"`, or `"click_failed"` — `save_post()` returns `(SavedPost, confirmation)` and `unsave_post()` returns just the confirmation string; both accept an optional `playwright_instance` for test injection (same pattern as `_fetch_via_dom`). `fetch_post_content()` reuses the same `window._preloads` mechanism to retrieve a saved post's full content: `_fetch_post_content_impl` navigates to the post's page and reads `preloads.post.body_html` (Substack's field name for full content — `parse_remote_post()` already expects this key from the reader API, though the saved-list payload never actually populates it, only individual post pages do), returning `None` for `body_html` if the page's embed format doesn't expose it (frontend change) or the account lacks paywall access; this is the case where the caller should be told to run `inspect-network` against an open post page to re-discover the real content source. `cli.py`'s `inspect-network` command (which logs any `api/v1`/`bookmark`/`saved`/`notes?`/`comment`/`reader`/`feed`/`restack` request's method, URL, status, and JSON response body) is the tool for discovering/re-verifying these endpoints when Substack's frontend changes. It supports `--authenticated/--anonymous` (reuses `storage_state.json` via `get_storage_state_path()` so `/saved` renders the real logged-in page instead of the marketing page), `--url` to target any page, `--filter` to override the response-matching regex, `--max-body` to cap logged response bytes, and `--out PATH` to append each exchange as JSON Lines for later grepping. Response bodies are captured via `context.route("**/*", handle_route)` + `route.fetch()`/`route.fulfill()`, not the more obvious `page.on("response", ...)` + `response.text()`: the latter can deadlock calling `response.text()` inside a sync-API response event handler on the same driver thread, silently yielding `null` bodies — `route.fetch()` reads the body outside that handler and `route.fulfill(response=...)` re-serves the exact response so page behavior is unaffected; a body that still can't be read logs a visible yellow warning.
|
|
31
|
+
|
|
32
|
+
Saved-posts fetching has a three-source chain, controlled by `get_saved_posts_source()`/`SUBSTACK_SAVED_POSTS_SOURCE` (`auto` default, or a forced `legacy`/`unified`/`dom`): the unified `GET /api/v1/reader/saved?filter=posts` endpoint (the same one notes uses with `filter=notes`) is now tried first, then the legacy reader-posts API above as an automatic fallback, then the DOM fallback. This was a migration decision, not a default guess — see `PLAN.md`'s Decisions Ledger and Evidence sections for the full evidence-gathering and decision-gate process. `probe-api URL [--out PATH]` headlessly GETs any known API URL with the saved session (no browser page) and dumps the raw JSON, a lighter complement to `inspect-network`, which is for *discovering* unknown endpoints by watching a live browsing session. `PLAN.md` § Evidence records the live comparison: the unified endpoint was a stable superset of legacy, and the legacy count varied from run to run because of silent mid-pagination truncation. `_fetch_all_saved_posts_via_unified_api()` is a structural sibling of `_fetch_all_saved_notes_via_api()` (opaque `nextCursor` pagination, not the legacy `after=<ISO saved_at>` scheme), cached separately in `_unified_api_cache`/`_unified_api_failed` so one source's failure can't poison another's mid-run cache. `parse_remote_post()` reads `saved_at` from both the item level (legacy) and `post_obj` (unified: `post.saved_at`), since the two APIs nest it differently. All three cursor-paginated fetchers (legacy posts, unified posts, notes) return a partial list rather than `None` when a page fails after every retry, so a persistent 429 doesn't throw away progress, and set a per-source flag (`_api_truncated` / `_unified_api_truncated` / `_notes_api_truncated`, reset at the top of each fetch and in `reset_cache()`). `is_posts_fetch_truncated()`/`is_notes_fetch_truncated()` report the populated cache's flag, using `getattr()` defaults so test doubles that skip `__init__` report "not truncated". On a truncated fetch, `sync_saved_posts()`/`sync_saved_notes()` **skip reconciliation** (absent items may still be saved remotely) and return `status = "partial"` with an `error_message`. The CLI's `sync` shows `partial` in yellow as a qualified success (`posts_ok`/`notes_ok` still count it as having run); `get_status()`'s "last successful sync" (`status = 'success'`) excludes it.
|
|
33
|
+
- **Saved notes** (Substack's short-form posts, distinct from long-form posts) are modeled and synced separately from posts, in their own `notes` table/`SavedNote`/`NoteSummary` models, discovered and confirmed live against a real account rather than guessed. The saved-notes list is `GET https://substack.com/api/v1/reader/saved?filter=notes` (also `filter=all`/`filter=posts` — a newer, unified endpoint Substack has added alongside the legacy posts-only `/api/v1/reader/posts?inboxType=saved` that `_fetch_all_saved_via_reader_api()` uses; posts now use `filter=posts` first, see the source chain above). Its response is `{"items": [...], "nextCursor": <opaque base64 JSON token> | null}`; pagination resubmits the server's own `nextCursor` value as a `cursor=` query param rather than the `after=<ISO timestamp>` scheme posts uses, and the `limit=` query param has no effect (the server returns a fixed page size), so `_fetch_all_saved_notes_via_api()` is a standalone paginator, not a generalization of `_fetch_all_saved_via_reader_api()`. Each item is `{"entity_key": "c-<id>", "publication": null|{...}, "post": null|{...}, "comment": {...}}`; a restack is represented by the top-level `publication`/`post` fields being populated instead of null (unconfirmed by a live example in the account used to discover this, but structurally solid since that item shape is shared with `filter=posts`). Critically, **the endpoint never exposes a bookmark timestamp** — only `comment.is_saved: bool` — so `SavedNote.saved_at` is always `None`, and `sync_saved_notes()`'s incremental early-stop check compares "already saved locally" (`existing.is_saved == 1`) rather than a `saved_at` match the way `sync_saved_posts()` does. A note's identity is its numeric `comment.id`, which is also embedded in its permalink (`https://substack.com/@<handle>/note/c-<id>`, confirmed live via click-tracking) — `_extract_note_id()` parses it back out of a URL via regex, and `upsert_note()`/`reconcile_unsaved_notes()`/`get_note_by_substack_id()` key on that id (not URL, unlike posts) since a note may lack a stable permalink. `comment.body` is already Substack-flattened plain text; `comment.body_json` is a ProseMirror doc (`{type:"doc", content:[{type:"paragraph"|"orderedList"|"listItem",...}]}`, text nodes carrying `marks:[{type:"bold"|"italic"|"link"}]`) that `content_utils.note_body_to_text()`/`prosemirror_to_llm_text()` convert to the same `**bold**`/`*italic*`/`text (url)` conventions `html_to_llm_text()` uses for posts; `body_raw` stores the source verbatim as a hedge against the body-format guess needing correction later. The individual-note-fetch endpoint `GET /api/v1/reader/comment/{id}` → `{"item": {"comment": {...}}}` and the bookmark endpoints `POST`/`DELETE https://substack.com/api/v1/note/c-{id}/save` (id in the URL path, no request body) are all plain authenticated calls through Playwright's `p.request` API context — **no browser page or `window._preloads` lookup is needed for notes at all**, unlike posts, so `save_note()`/`unsave_note()`/`fetch_note_content()` in `substack_client.py` never launch `p.chromium`. Notes have no DOM fallback and no `_click_bookmark_toggle`-style guesswork: the notes card markup is uncaptured, and since the note id is the identity, a DOM-scraped row without one could never reconcile, so an unavailable notes API raises `SubstackClientError` naming `inspect-network` rather than degrading silently. `sync_runs.entity` (`'post'` | `'note'`, default `'post'` for pre-existing rows) scopes `get_status()`'s "last sync" queries per entity so a notes sync can't be reported as the posts' last sync result.
|
|
34
|
+
- `content_utils.py` converts a post's raw `body_html` into clean text for LLM consumption. `html_to_llm_text()` is a small `html.parser.HTMLParser` subclass (no external HTML library dependency) that renders headings as markdown `#` prefixes, list items as `- ` bullets, links as `text (url)`, keeps minimal `**bold**`/`*italic*` markers, drops `script`/`style`/`iframe` content entirely, and collapses excess blank lines. `format_post_for_llm()` prepends a plain metadata header (title, publication, author, published date, URL) to the cleaned body text — this combined string is what both the CLI's `get-content` command and the MCP `get_post_content` tool return, through the shared `sync.load_post_content()`/`load_note_content()` (cache check, fetch, convert, store; they raise `ContentUnavailableError` when Substack returns no body). `note_body_to_text()` handles the same problem for notes' `body_json`/`body`, dispatching a dict (or JSON string) to `prosemirror_to_llm_text()` (a small recursive walker over Substack's ProseMirror doc shape — `paragraph`/`heading`/`bulletList`/`orderedList`/`listItem`/`hard_break` nodes, `bold`/`italic`/`link` marks — producing the same `**bold**`/`*italic*`/`text (url)` conventions as `html_to_llm_text()`) and a plain string to either the HTML converter (if it looks like HTML) or straight through; `format_note_for_llm()` builds the note-specific header (author/handle/posted date/URL, plus a `Restacked: <title> (<url>)` line when applicable) rather than reusing `format_post_for_llm()`, since `publication_name` is required there but meaningless for most notes.
|
|
35
|
+
- `sync.py` converts API/DOM payloads to `SavedPost` models and coordinates paginated incremental or full syncs. `parse_remote_post()` routes DOM dicts (identified by the `_dom` marker) through a minimal mapping and everything else through the full reader-API/legacy mapping; it never fabricates a `saved_at` from the sync moment (unknown save times stay `None`). `word_count` is mapped from `wordcount`/`word_count`/`words` via `_first_positive_int()` (which ignores non-positive/uncoercible values); a live parity run (see `PLAN.md` § Evidence) confirmed the real field is `wordcount` (100% populated on the unified endpoint's payload, 0% on `word_count`/`words`) — the defensive multi-key mapping is kept since `_first_positive_int()` already checks `wordcount` first and the other two cost nothing to keep as a hedge. `reading_time_minutes` is intentionally *derived* (ceil of `word_count / WORDS_PER_MINUTE`, ~200 wpm) rather than mapped from a field, because a wrong guess about that field's unit (seconds vs minutes) would persist a badly wrong value; deriving is unit-unambiguous. `image_url` is mapped from `cover_image`/`image_url` on the reader-API payload and is populated for the large majority of synced posts (confirmed live: ~98%); `PostSummary` and the `list_posts()`/`search_posts()` SELECTs include it; the CLI `search` command prints it when present, but `list` deliberately omits it since it's a long, uninformative CDN URL in that terser view. `content_text` remains unpopulated by sync (only `get-content`/`get_post_content` populate it) because the saved-list payload doesn't carry post body HTML, only individual post pages do. There is no `metadata_json` column. Older databases may still have an inert `metadata_json` column; reads tolerate it because `SavedPost` (Pydantic) ignores unknown columns from `SELECT *`, and `upsert_post` no longer references it. A force/full sync (`--force`) collects every fetched post's URL and, after the fetch loop completes, calls `reconcile_unsaved_posts()` to soft-delete any locally `is_saved = 1` post absent from that complete remote set — this is how posts unsaved directly on Substack (outside this tool) get reflected locally. An incremental sync never reconciles, since its early-stop-on-matches optimization means it only sees a partial remote list. It records each run in SQLite (including `reconciled_count`) and returns a `SyncRun` with `success`, `auth_required`, or `failed` status instead of propagating expected authentication failures. `sync_saved_notes()` is a sibling of `sync_saved_posts()`, not a generalization of it (notes reconcile by Substack note id via `reconcile_unsaved_notes()`, not URL, and the incremental early-stop check compares `existing.is_saved == 1` rather than a `saved_at` match, since notes carry none); both write `sync_runs.entity` (`'post'`/`'note'`) and share a `_build_sync_run()` helper instead of each hand-constructing `SyncRun` three times; `_build_sync_run()` takes the real `started_at`, captured once at the top of each sync function right before `start_sync_run()`; don't stamp it at finish time. `parse_remote_note()` unwraps a saved-notes list item (or a single-note-fetch `{"item": {...}}` wrapper) into a `SavedNote`, returning `None` (so the sync loop skips it) rather than inventing an id when `comment.id` is missing; it delegates body conversion to `content_utils.note_body_to_text()` and derives `word_count` from the resulting text rather than any source field.
|
|
36
|
+
- `database.py` is the SQLite repository and schema owner. The `posts` table is the cache; `sync_runs` records sync history; `posts_fts` is an external-content FTS5 index maintained by database triggers. `reconcile_unsaved_posts()` bulk soft-deletes posts missing from a given complete remote URL set and is a no-op on an empty list (an empty list is more likely a fetch problem than genuine mass-unsaving). `init_db()` runs additive column migrations (e.g. `posts.audience`, `sync_runs.reconciled_count`) via `ALTER TABLE` guarded by an existence check or `try`/`except sqlite3.OperationalError`, since `CREATE TABLE IF NOT EXISTS` never adds columns to an already-existing table; the `posts.audience` check runs before the `executescript` block because the `idx_posts_audience` index creation inside it would otherwise fail on a pre-migration table. `list_posts()`/`search_posts()` accept an `audience` filter (exact match, case-insensitive), and `list_audiences()` returns the distinct audience values actually present in the cache with post counts, rather than hardcoding Substack's (undocumented, possibly-growing) audience enum. `PostSummary` (returned by `list_posts()`/`search_posts()` and the MCP list/search tools) carries `reading_time_minutes` and `word_count`, and the CLI `list`/`search` commands display them — without this the values were populated in the `posts` table but invisible through every read path except `get_saved_post`/the full `SavedPost`. A parallel `notes` table (with its own `notes_fts` external-content index and `notes_ai`/`notes_ad`/`notes_au` triggers) holds saved notes; unlike `posts`, its unique identity column is `substack_note_id`, not `url` (a note may have no stable permalink), so `upsert_note()` does an explicit lookup-then-branch UPDATE/INSERT on `substack_note_id` rather than `INSERT ... ON CONFLICT(url)`. `upsert_post()` uses the same lookup-then-branch UPDATE/INSERT (by `substack_post_id` or `url`); don't switch it to `INSERT ... ON CONFLICT(url)`, which crashes on the `substack_post_id UNIQUE` constraint when a post's canonical URL changes (slug rename, custom-domain migration). `search_posts()`'s LIKE fallback (used when an FTS5 query string is malformed) shares its WHERE-clause construction with the main FTS branch via `_post_search_filters()` — mirroring `_note_search_filters()` — so a malformed query never silently drops the `publication`/`audience`/date-range filters or an `is_saved_only=False` caller. `get_note_by_substack_id()` exists because `get_note()`'s polymorphic `url_or_id` dispatch treats any all-digit string as a local row id — and `substack_note_id` values are always digits — so callers that only have the Substack note id (the sync loop's incremental check) must use it instead of risking a collision with an unrelated local row. `sync_runs.entity` (`'post'`/`'note'`, additive column defaulting existing rows to `'post'`) scopes `get_status()`'s "last successful sync"/"last sync status" queries per entity, which is the one existing posts read path notes support required changing (it previously read the last `sync_runs` row unscoped, which was harmless with one entity and would otherwise let a notes sync be reported as the posts' last sync).
|
|
37
|
+
- `config.py` centralizes application paths. By default, data lives under `~/.local/share/substack-saved-mcp`; `SUBSTACK_SAVED_DB_PATH`, `SUBSTACK_SAVED_DATA_DIR`, and `SUBSTACK_SAVED_BROWSER_DIR` override those paths. `get_saved_posts_source()`/`SUBSTACK_SAVED_POSTS_SOURCE` (`auto`/`legacy`/`unified`/`dom`, falling back to `auto` on an unrecognized value) selects which of the three saved-posts fetch sources `_fetch_saved_posts_page_impl()` tries, so a source-specific regression is one env var away from a revert rather than a code rollback. `get_fully_read_threshold()`/`SUBSTACK_SAVED_FULLY_READ_THRESHOLD` (default `0.95`, falling back to the default on an unparseable or out-of-`(0, 1]` value) sets the `max_read_progress` fraction at or above which a post counts as fully read.
|
|
38
|
+
- **Reading progress.** Substack exposes per-post reading progress (visible in its mobile apps, never the web UI) through the same saved-posts payloads this project already fetches: `post.read_progress` (current scroll position, 0.0–1.0), `post.max_read_progress` (high-water mark, 0.0–1.0), and `post.is_viewed` (bool) — confirmed live on both the unified and legacy reader APIs (identically mirrored under `post.inboxItem`, so that nested object is never read), absent from the DOM fallback. `parse_remote_post()` maps all three off `post_obj` via a new `_float_or_none()` helper (clamped to `[0.0, 1.0]`, tolerant of a real `0.0` unlike `_first_positive_int()`). The three raw values are stored as columns (`posts.read_progress`/`max_read_progress REAL`, `posts.is_viewed INTEGER NOT NULL DEFAULT 0`); `is_fully_read` and `minutes_remaining` are deliberately **not** columns — they're Pydantic `@computed_field` properties on `SavedPost`/`PostSummary`, derived from `max_read_progress` (the high-water mark, not the current position, since that's what "have I read this" means) and `word_count` at read time, same rationale as `reading_time_minutes`: real observed values like `0.9999`/`0.9867`/`0.9822` rule out an exact `== 1.0` test, and a stored derivative would go stale the moment the threshold changed. `database.py`'s `_READ_STATE_PREDICATES`/`_read_state_clause()` classify a post into `unread`/`in_progress`/`finished`/`unfinished` (below the threshold, i.e. unread + in progress)/`started` via SQL predicates wrapping every comparison in `COALESCE(..., 0)` — load-bearing, not defensive styling: after the additive migration every pre-existing row is `NULL`, and `max_read_progress = 0` would otherwise evaluate to `NULL` (not true) and silently exclude the entire legacy cache from every read_state filter. `list_posts()`/`search_posts()`/`_post_search_filters()` accept `read_state`, threaded through both the FTS and LIKE-fallback branches of `search_posts()` the same way `audience` already was; `list_posts()`'s sort-key whitelist also gained `read_progress` and a `minutes_remaining` SQL expression (`COALESCE(word_count,0) * (1 - COALESCE(max_read_progress,0))`) so a caller can find something short to finish. `upsert_post()` coalesces these three fields with explicit `is not None` (`is_viewed`: truthy-sticky) checks rather than the pre-existing `or`-based idiom used for other optional fields, since a real `0.0` is falsy and the `or` pattern would have silently discarded it in favor of a stale stored value — a DOM-sourced or otherwise progress-free payload (e.g. `_save_post_impl`'s construction site) preserves whatever progress is already stored rather than clobbering it. `get_status()`/`SavedPostsStatus` gained `posts_unread`/`posts_in_progress`/`posts_fully_read`/`minutes_remaining_total` aggregates built from the same predicates and remaining-words expression. Progress refreshes whenever a post is re-fetched during sync; since incremental sync only touches recently-saved posts (its early-stop fires after `MAX_CONSECUTIVE_MATCHES` matches), older posts' progress goes stale between full syncs — `sync --force` is the documented way to refresh it for the whole backlog, and no dedicated refresh mode was added.
|
|
39
|
+
|
|
40
|
+
## Repository Conventions
|
|
41
|
+
|
|
42
|
+
- Model persisted posts with the Pydantic `SavedPost` schema. Keep `published_at` (post publication time) distinct from `saved_at` (bookmark time). Both come from the source payload; never fabricate `saved_at` from the sync/DB-insert moment — leave it `None` when the source does not expose the original bookmark time (`upsert_post` preserves a known value and stores `NULL` otherwise).
|
|
43
|
+
- Store the raw Substack `audience` string (e.g. `everyone`, `only_paid`) verbatim on `SavedPost.audience` rather than only collapsing it into the `is_paywalled` boolean; `parse_remote_post()` still derives `is_paywalled` from it (`audience == "only_paid"`) for backward-compatible boolean filtering, but the original value is preserved for exact-match filtering and discovery via `list_audiences()`.
|
|
44
|
+
- Canonicalize every post URL through `canonicalize_url()` before storage or lookup. It strips known tracking parameters, fragments, and non-root trailing slashes; SQLite uniqueness and upsert behavior depend on that canonical form.
|
|
45
|
+
- Treat unsaving as a soft delete: use `soft_delete_post()` (single post, e.g. the `unsave` command/tool) or `reconcile_unsaved_posts()` (bulk, called from a force sync) to set `is_saved = 0` and `unsaved_at`, retaining the record for history. Default list/search/publication queries intentionally include only `is_saved = 1`.
|
|
46
|
+
- The local cache always reflects the user's intent for `save`/`unsave`, regardless of whether the remote toggle on Substack could be confirmed — never block the local write on remote confirmation. Instead, surface the confirmation status (`"confirmed"` vs. `"unconfirmed"/"not_found"/"click_failed"`) as a clearly-labeled warning to the caller (CLI: yellow `click.secho`; MCP: a `warning` key and `remote_confirmed: bool` in the tool's dict response) so an unconfirmed remote action isn't silently reported as a full success. A subsequent `sync --force` will self-correct the local cache via `reconcile_unsaved_posts()` if a "confirmed" save/unsave didn't actually happen remotely.
|
|
47
|
+
- Do not write directly to `posts_fts` or `notes_fts`; use the repository operations so the SQLite insert/update/delete triggers maintain the FTS5 indexes.
|
|
48
|
+
- Notes are identified by `substack_note_id`, not URL — a note may lack a stable permalink, unlike a post. Never give notes a DOM-scrape fallback: the notes card markup is unverified, and since a note's id *is* its identity, a DOM-scraped row without one could never be reconciled; an unavailable notes API should raise a clear error naming `inspect-network` instead of degrading to guesswork the way the old `save_post`/`unsave_post` once did before their real endpoints were captured.
|
|
49
|
+
- Database-facing tests use `tmp_path` databases. CLI and MCP tests set `SUBSTACK_SAVED_DB_PATH` with `monkeypatch` so they never access a developer's real cache. Sync tests inject a `SubstackSavedPostsClient` substitute rather than opening Playwright or contacting Substack.
|
|
50
|
+
- Browser state and SQLite data are credentials or user data. Keep `storage_state.json`, browser-state directories, `.env` files, and database files untracked; configuration creates data directories with user-only permissions on POSIX systems.
|
|
@@ -0,0 +1,286 @@
|
|
|
1
|
+
Metadata-Version: 2.5
|
|
2
|
+
Name: substack-saved-mcp
|
|
3
|
+
Version: 0.3.3
|
|
4
|
+
Summary: Local stdio-based MCP server and sync engine for Substack saved posts
|
|
5
|
+
Author-email: Toni Hermoso Pulido <toniher@cau.cat>
|
|
6
|
+
License-Expression: MIT
|
|
7
|
+
License-File: LICENSE
|
|
8
|
+
Requires-Python: >=3.11
|
|
9
|
+
Requires-Dist: click>=8.1.0
|
|
10
|
+
Requires-Dist: fastmcp>=0.1.0
|
|
11
|
+
Requires-Dist: playwright>=1.40.0
|
|
12
|
+
Requires-Dist: pydantic>=2.0.0
|
|
13
|
+
Provides-Extra: dev
|
|
14
|
+
Requires-Dist: pytest>=8.0.0; extra == 'dev'
|
|
15
|
+
Requires-Dist: ruff==0.16.1; extra == 'dev'
|
|
16
|
+
Description-Content-Type: text/markdown
|
|
17
|
+
|
|
18
|
+
# Substack Saved Posts & Notes MCP & CLI
|
|
19
|
+
|
|
20
|
+
[](https://pypi.org/project/substack-saved-mcp/)
|
|
21
|
+
|
|
22
|
+
A local stdio Model Context Protocol (MCP) server and sync engine for the Substack **posts and notes** you have saved.
|
|
23
|
+
|
|
24
|
+
## Features
|
|
25
|
+
|
|
26
|
+
- **Read & Search**: Full-text search (SQLite FTS5) over saved post titles, excerpts, authors, and publications. Filter by publication, audience tier (`everyone`, `only_paid`), reading progress, and date ranges (`published_at` or `saved_at`). Search covers a post's full body text only after you fetch it once with `get-content` or the `get_post_content` tool. A normal `sync` stores metadata and excerpts, so search matches posts you haven't opened on their title, excerpt, and metadata.
|
|
27
|
+
- **Reading Progress**: Substack records how far you've read each saved post. Its mobile apps show this; the web UI doesn't. This tool stores it, so you can list the posts you haven't read or haven't finished, and sort by the minutes left. See [Which saved posts haven't I read?](#which-saved-posts-havent-i-read).
|
|
28
|
+
- **Saved Notes**: The tool syncs, searches, and caches Substack's short-form notes apart from posts. A note has an author and a body instead of a title or audience tier. Search covers note bodies, authors, and the titles of restacked posts. Notes skip the browser: sync, save, unsave, and content fetch are all plain authenticated API calls.
|
|
29
|
+
- **Full Content for LLMs**: Fetch a saved post or note and get its content back as clean text, with headings, lists, and links kept readable. The tool caches the result for next time.
|
|
30
|
+
- **Save & Unsave**: Bookmark posts and notes, or remove bookmarks. Posts use a headless browser session; notes use the API.
|
|
31
|
+
- **Offline Queries**: Read and search tools query the local SQLite cache, with no network calls.
|
|
32
|
+
- **Local Credentials**: Your Substack session stays on your machine, in a data directory only your user can read.
|
|
33
|
+
- **MCP Tools and Resources** for both posts and notes over stdio.
|
|
34
|
+
|
|
35
|
+
---
|
|
36
|
+
|
|
37
|
+
## Installation with `uv`
|
|
38
|
+
|
|
39
|
+
Install and run `substack-saved-mcp` with [`uv`](https://github.com/astral-sh/uv).
|
|
40
|
+
|
|
41
|
+
### Option A: Install from PyPI as a system-wide tool (`uv tool install`)
|
|
42
|
+
|
|
43
|
+
Install the published package from [PyPI](https://pypi.org/project/substack-saved-mcp/):
|
|
44
|
+
|
|
45
|
+
```bash
|
|
46
|
+
# Install system-wide into an isolated uv environment
|
|
47
|
+
uv tool install substack-saved-mcp
|
|
48
|
+
```
|
|
49
|
+
|
|
50
|
+
To install from a local checkout instead:
|
|
51
|
+
|
|
52
|
+
```bash
|
|
53
|
+
# Navigate to the repository
|
|
54
|
+
cd /path/to/substack-saved-mcp
|
|
55
|
+
|
|
56
|
+
# Install system-wide into an isolated uv environment
|
|
57
|
+
uv tool install .
|
|
58
|
+
|
|
59
|
+
# Or install from the Git repository:
|
|
60
|
+
# uv tool install git+https://github.com/toniher/substack-saved-mcp.git
|
|
61
|
+
```
|
|
62
|
+
|
|
63
|
+
The install puts `substack-saved-mcp` on your PATH:
|
|
64
|
+
|
|
65
|
+
```bash
|
|
66
|
+
# Verify installation
|
|
67
|
+
substack-saved-mcp --help
|
|
68
|
+
```
|
|
69
|
+
|
|
70
|
+
To update or uninstall:
|
|
71
|
+
```bash
|
|
72
|
+
# Upgrade installed tool
|
|
73
|
+
uv tool upgrade substack-saved-mcp
|
|
74
|
+
|
|
75
|
+
# Uninstall tool
|
|
76
|
+
uv tool uninstall substack-saved-mcp
|
|
77
|
+
```
|
|
78
|
+
|
|
79
|
+
---
|
|
80
|
+
|
|
81
|
+
### Option B: Development environment (`uv sync`)
|
|
82
|
+
|
|
83
|
+
To develop or modify the code:
|
|
84
|
+
|
|
85
|
+
```bash
|
|
86
|
+
# Clone and enter directory
|
|
87
|
+
cd substack-saved-mcp
|
|
88
|
+
|
|
89
|
+
# Install dependencies and dev tools (pytest)
|
|
90
|
+
uv sync --extra dev
|
|
91
|
+
|
|
92
|
+
# Run CLI commands using uv run
|
|
93
|
+
uv run substack-saved-mcp --help
|
|
94
|
+
|
|
95
|
+
# Run tests
|
|
96
|
+
uv run pytest
|
|
97
|
+
```
|
|
98
|
+
|
|
99
|
+
---
|
|
100
|
+
|
|
101
|
+
## Quick Start
|
|
102
|
+
|
|
103
|
+
```bash
|
|
104
|
+
# 1. Initialize local database
|
|
105
|
+
substack-saved-mcp init
|
|
106
|
+
|
|
107
|
+
# 2. Authenticate with Substack (opens interactive browser window once)
|
|
108
|
+
substack-saved-mcp login
|
|
109
|
+
|
|
110
|
+
# 3. Sync saved posts AND notes into local cache (both entities by default)
|
|
111
|
+
substack-saved-mcp sync
|
|
112
|
+
|
|
113
|
+
# 3b. Or sync just one entity
|
|
114
|
+
substack-saved-mcp sync --only posts
|
|
115
|
+
substack-saved-mcp sync --only notes
|
|
116
|
+
|
|
117
|
+
# 4. Search saved posts via CLI
|
|
118
|
+
substack-saved-mcp search "artificial intelligence"
|
|
119
|
+
|
|
120
|
+
# 4b. Filter by publication or audience tier (see which tiers are cached with `audiences`)
|
|
121
|
+
substack-saved-mcp audiences
|
|
122
|
+
substack-saved-mcp list --audience only_paid
|
|
123
|
+
substack-saved-mcp search "artificial intelligence" --audience everyone
|
|
124
|
+
|
|
125
|
+
# 4c. Find posts you haven't read or haven't finished
|
|
126
|
+
substack-saved-mcp list --read-state unread
|
|
127
|
+
substack-saved-mcp list --read-state unfinished --sort-by minutes_remaining
|
|
128
|
+
substack-saved-mcp search "artificial intelligence" --read-state unread
|
|
129
|
+
|
|
130
|
+
# 5. Save or unsave a post
|
|
131
|
+
substack-saved-mcp save "https://example.substack.com/p/post-slug"
|
|
132
|
+
substack-saved-mcp unsave "https://example.substack.com/p/post-slug"
|
|
133
|
+
|
|
134
|
+
# 6. Get a saved post's full content, cleaned up and ready for an LLM
|
|
135
|
+
substack-saved-mcp get-content "https://example.substack.com/p/post-slug"
|
|
136
|
+
|
|
137
|
+
# 7. Work with saved notes the same way
|
|
138
|
+
substack-saved-mcp list-notes --limit 10
|
|
139
|
+
substack-saved-mcp search-notes "kubernetes" --author alice
|
|
140
|
+
substack-saved-mcp note-authors
|
|
141
|
+
substack-saved-mcp save-note "https://substack.com/@handle/note/c-123456"
|
|
142
|
+
substack-saved-mcp unsave-note "https://substack.com/@handle/note/c-123456"
|
|
143
|
+
substack-saved-mcp get-note "https://substack.com/@handle/note/c-123456"
|
|
144
|
+
|
|
145
|
+
# 8. Check combined status (counts, unread totals, last sync per entity)
|
|
146
|
+
substack-saved-mcp status
|
|
147
|
+
|
|
148
|
+
# 9. Launch stdio MCP server
|
|
149
|
+
substack-saved-mcp serve
|
|
150
|
+
```
|
|
151
|
+
|
|
152
|
+
---
|
|
153
|
+
|
|
154
|
+
## Configuring MCP Clients (Claude Desktop, Goose, Cursor, etc.)
|
|
155
|
+
|
|
156
|
+
Add `substack-saved-mcp` to your MCP client's configuration file (e.g. `claude_desktop_config.json`).
|
|
157
|
+
|
|
158
|
+
### Using the installed tool (`uv tool` or global binary)
|
|
159
|
+
|
|
160
|
+
```json
|
|
161
|
+
{
|
|
162
|
+
"mcpServers": {
|
|
163
|
+
"substack-saved": {
|
|
164
|
+
"command": "substack-saved-mcp",
|
|
165
|
+
"args": ["serve"]
|
|
166
|
+
}
|
|
167
|
+
}
|
|
168
|
+
}
|
|
169
|
+
```
|
|
170
|
+
|
|
171
|
+
### Using `uv` from the repository path
|
|
172
|
+
|
|
173
|
+
To run from your checkout without installing:
|
|
174
|
+
|
|
175
|
+
```json
|
|
176
|
+
{
|
|
177
|
+
"mcpServers": {
|
|
178
|
+
"substack-saved": {
|
|
179
|
+
"command": "uv",
|
|
180
|
+
"args": [
|
|
181
|
+
"--directory",
|
|
182
|
+
"/path/to/substack-saved-mcp",
|
|
183
|
+
"run",
|
|
184
|
+
"substack-saved-mcp",
|
|
185
|
+
"serve"
|
|
186
|
+
]
|
|
187
|
+
}
|
|
188
|
+
}
|
|
189
|
+
}
|
|
190
|
+
```
|
|
191
|
+
|
|
192
|
+
---
|
|
193
|
+
|
|
194
|
+
## Frequently Asked Questions (FAQ)
|
|
195
|
+
|
|
196
|
+
### Where is the database saved?
|
|
197
|
+
|
|
198
|
+
The SQLite database lives in your OS application data directory:
|
|
199
|
+
|
|
200
|
+
- **Linux / macOS**: `~/.local/share/substack-saved-mcp/saved_posts.sqlite`
|
|
201
|
+
*(or `$XDG_DATA_HOME/substack-saved-mcp/saved_posts.sqlite` if you set `XDG_DATA_HOME`)*
|
|
202
|
+
|
|
203
|
+
Set a custom database path or directory with environment variables:
|
|
204
|
+
```bash
|
|
205
|
+
export SUBSTACK_SAVED_DB_PATH="/path/to/my/custom_database.sqlite"
|
|
206
|
+
# or
|
|
207
|
+
export SUBSTACK_SAVED_DATA_DIR="/path/to/my/data_dir"
|
|
208
|
+
```
|
|
209
|
+
|
|
210
|
+
### Which saved posts haven't I read?
|
|
211
|
+
|
|
212
|
+
Filter by reading state on the CLI with `--read-state`, or pass `read_state` to the `list_saved_posts` and `search_saved_posts` MCP tools. In a chat, ask your assistant to "list my unread saved posts".
|
|
213
|
+
|
|
214
|
+
| State | Posts it matches |
|
|
215
|
+
|-------|------------------|
|
|
216
|
+
| `unread` | You never opened them, and Substack recorded no progress. |
|
|
217
|
+
| `in_progress` | You read part, below the finished threshold. |
|
|
218
|
+
| `finished` | You read past the threshold (default 95%). |
|
|
219
|
+
| `unfinished` | Everything below the threshold: `unread` plus `in_progress`. |
|
|
220
|
+
| `started` | You opened them or read any part. |
|
|
221
|
+
|
|
222
|
+
```bash
|
|
223
|
+
substack-saved-mcp sync --force # refresh progress for every post first
|
|
224
|
+
substack-saved-mcp list --read-state unfinished --sort-by minutes_remaining --limit 20
|
|
225
|
+
substack-saved-mcp status # unread count and total minutes remaining
|
|
226
|
+
```
|
|
227
|
+
|
|
228
|
+
Sorting `unfinished` posts by `minutes_remaining` puts the quickest ones to finish at the top. Each post also reports `is_fully_read` and `minutes_remaining`. Notes have no reading progress.
|
|
229
|
+
|
|
230
|
+
### How does the tool decide a post is fully read, and how fresh is that?
|
|
231
|
+
|
|
232
|
+
Substack reports a `max_read_progress` high-water mark (0.0 to 1.0) for each saved post. The tool counts a post as fully read once that value reaches a threshold, `0.95` by default, because finished posts on Substack tend to stop around `0.98` to `0.9999` rather than at `1.0`. Change it with:
|
|
233
|
+
```bash
|
|
234
|
+
export SUBSTACK_SAVED_FULLY_READ_THRESHOLD="0.90"
|
|
235
|
+
```
|
|
236
|
+
A sync refreshes progress for each post it fetches. An incremental sync stops at the first posts it already has, so progress on older posts goes stale until you run `substack-saved-mcp sync --force`. A post you finished last week can show as `unread` until then.
|
|
237
|
+
|
|
238
|
+
### Which Substack API does syncing saved posts use?
|
|
239
|
+
|
|
240
|
+
Sync tries Substack's unified reader API first. If that fails, it falls back to an older posts-only API, then to headless browser scraping. To force one source while troubleshooting, set:
|
|
241
|
+
```bash
|
|
242
|
+
export SUBSTACK_SAVED_POSTS_SOURCE="unified" # or "legacy" or "dom"
|
|
243
|
+
```
|
|
244
|
+
Leave it unset, or set `"auto"`, for the default fallback chain.
|
|
245
|
+
|
|
246
|
+
### What does a "partial" sync status mean?
|
|
247
|
+
|
|
248
|
+
Substack sometimes rate-limits a sync (HTTP 429) so hard that a page of results fails even after retries. The sync then keeps what it fetched and reports `status: partial` instead of `success`, in `substack-saved-mcp status` and in the sync tool's response. On a `sync --force`, a partial run skips reconciliation, so the tool won't mark a post or note as unsaved because it couldn't fetch it. Run `sync` again later to pick up what it missed.
|
|
249
|
+
|
|
250
|
+
### Will a browser window pop up when running as an MCP server?
|
|
251
|
+
|
|
252
|
+
No. During normal MCP use you won't see a browser window.
|
|
253
|
+
|
|
254
|
+
- **Read & Search Tools** (`search_saved_posts`, `list_saved_posts`, `get_saved_post`, `search_saved_notes`, `list_saved_notes`, `get_saved_note`, `list_publications`, `list_audiences`, `saved_posts_status`):
|
|
255
|
+
Query the local SQLite database offline. No browser.
|
|
256
|
+
- **Post Sync & Write Tools** (`sync_saved_posts`, `save_post`, `unsave_post`, `get_post_content`):
|
|
257
|
+
Run headless with the session stored in `storage_state.json`.
|
|
258
|
+
- **Note Sync & Write Tools** (`sync_saved_notes`, `save_note`, `unsave_note`, `get_note_content`):
|
|
259
|
+
Make direct API requests with `storage_state.json`, with no browser page, headless or visible.
|
|
260
|
+
- **Interactive Login**:
|
|
261
|
+
A browser window opens only when you run `substack-saved-mcp login` in your terminal. If your session expires while you use an MCP client, the tool returns an error telling you to run `substack-saved-mcp login`.
|
|
262
|
+
|
|
263
|
+
### What if I get a Playwright "Executable doesn't exist" error?
|
|
264
|
+
|
|
265
|
+
An error like `BrowserType.launch: Executable doesn't exist` (often during `login`) means Playwright hasn't installed its browsers in the tool's isolated environment. Run `playwright install` inside that environment.
|
|
266
|
+
|
|
267
|
+
For a system-wide install (`uv tool install`):
|
|
268
|
+
```bash
|
|
269
|
+
~/.local/share/uv/tools/substack-saved-mcp/bin/playwright install
|
|
270
|
+
```
|
|
271
|
+
|
|
272
|
+
For a development environment (`uv sync`):
|
|
273
|
+
```bash
|
|
274
|
+
uv run playwright install
|
|
275
|
+
```
|
|
276
|
+
|
|
277
|
+
### I edited the source code, but the installed `substack-saved-mcp` command still runs the old version. Why?
|
|
278
|
+
|
|
279
|
+
`uv tool install` copies the package into its own environment at install time and doesn't track your working tree. After you edit files under `src/` or pull new commits, the installed copy keeps running the old code. `uv run substack-saved-mcp ...` from the repo uses your current source.
|
|
280
|
+
|
|
281
|
+
Reinstall from your working tree to pick up the changes:
|
|
282
|
+
```bash
|
|
283
|
+
uv tool install . --no-cache --force
|
|
284
|
+
```
|
|
285
|
+
- `--force` replaces the installed version instead of skipping because one is present.
|
|
286
|
+
- `--no-cache` builds fresh instead of reusing a cached wheel from before your edits.
|
|
@@ -0,0 +1,269 @@
|
|
|
1
|
+
# Substack Saved Posts & Notes MCP & CLI
|
|
2
|
+
|
|
3
|
+
[](https://pypi.org/project/substack-saved-mcp/)
|
|
4
|
+
|
|
5
|
+
A local stdio Model Context Protocol (MCP) server and sync engine for the Substack **posts and notes** you have saved.
|
|
6
|
+
|
|
7
|
+
## Features
|
|
8
|
+
|
|
9
|
+
- **Read & Search**: Full-text search (SQLite FTS5) over saved post titles, excerpts, authors, and publications. Filter by publication, audience tier (`everyone`, `only_paid`), reading progress, and date ranges (`published_at` or `saved_at`). Search covers a post's full body text only after you fetch it once with `get-content` or the `get_post_content` tool. A normal `sync` stores metadata and excerpts, so search matches posts you haven't opened on their title, excerpt, and metadata.
|
|
10
|
+
- **Reading Progress**: Substack records how far you've read each saved post. Its mobile apps show this; the web UI doesn't. This tool stores it, so you can list the posts you haven't read or haven't finished, and sort by the minutes left. See [Which saved posts haven't I read?](#which-saved-posts-havent-i-read).
|
|
11
|
+
- **Saved Notes**: The tool syncs, searches, and caches Substack's short-form notes apart from posts. A note has an author and a body instead of a title or audience tier. Search covers note bodies, authors, and the titles of restacked posts. Notes skip the browser: sync, save, unsave, and content fetch are all plain authenticated API calls.
|
|
12
|
+
- **Full Content for LLMs**: Fetch a saved post or note and get its content back as clean text, with headings, lists, and links kept readable. The tool caches the result for next time.
|
|
13
|
+
- **Save & Unsave**: Bookmark posts and notes, or remove bookmarks. Posts use a headless browser session; notes use the API.
|
|
14
|
+
- **Offline Queries**: Read and search tools query the local SQLite cache, with no network calls.
|
|
15
|
+
- **Local Credentials**: Your Substack session stays on your machine, in a data directory only your user can read.
|
|
16
|
+
- **MCP Tools and Resources** for both posts and notes over stdio.
|
|
17
|
+
|
|
18
|
+
---
|
|
19
|
+
|
|
20
|
+
## Installation with `uv`
|
|
21
|
+
|
|
22
|
+
Install and run `substack-saved-mcp` with [`uv`](https://github.com/astral-sh/uv).
|
|
23
|
+
|
|
24
|
+
### Option A: Install from PyPI as a system-wide tool (`uv tool install`)
|
|
25
|
+
|
|
26
|
+
Install the published package from [PyPI](https://pypi.org/project/substack-saved-mcp/):
|
|
27
|
+
|
|
28
|
+
```bash
|
|
29
|
+
# Install system-wide into an isolated uv environment
|
|
30
|
+
uv tool install substack-saved-mcp
|
|
31
|
+
```
|
|
32
|
+
|
|
33
|
+
To install from a local checkout instead:
|
|
34
|
+
|
|
35
|
+
```bash
|
|
36
|
+
# Navigate to the repository
|
|
37
|
+
cd /path/to/substack-saved-mcp
|
|
38
|
+
|
|
39
|
+
# Install system-wide into an isolated uv environment
|
|
40
|
+
uv tool install .
|
|
41
|
+
|
|
42
|
+
# Or install from the Git repository:
|
|
43
|
+
# uv tool install git+https://github.com/toniher/substack-saved-mcp.git
|
|
44
|
+
```
|
|
45
|
+
|
|
46
|
+
The install puts `substack-saved-mcp` on your PATH:
|
|
47
|
+
|
|
48
|
+
```bash
|
|
49
|
+
# Verify installation
|
|
50
|
+
substack-saved-mcp --help
|
|
51
|
+
```
|
|
52
|
+
|
|
53
|
+
To update or uninstall:
|
|
54
|
+
```bash
|
|
55
|
+
# Upgrade installed tool
|
|
56
|
+
uv tool upgrade substack-saved-mcp
|
|
57
|
+
|
|
58
|
+
# Uninstall tool
|
|
59
|
+
uv tool uninstall substack-saved-mcp
|
|
60
|
+
```
|
|
61
|
+
|
|
62
|
+
---
|
|
63
|
+
|
|
64
|
+
### Option B: Development environment (`uv sync`)
|
|
65
|
+
|
|
66
|
+
To develop or modify the code:
|
|
67
|
+
|
|
68
|
+
```bash
|
|
69
|
+
# Clone and enter directory
|
|
70
|
+
cd substack-saved-mcp
|
|
71
|
+
|
|
72
|
+
# Install dependencies and dev tools (pytest)
|
|
73
|
+
uv sync --extra dev
|
|
74
|
+
|
|
75
|
+
# Run CLI commands using uv run
|
|
76
|
+
uv run substack-saved-mcp --help
|
|
77
|
+
|
|
78
|
+
# Run tests
|
|
79
|
+
uv run pytest
|
|
80
|
+
```
|
|
81
|
+
|
|
82
|
+
---
|
|
83
|
+
|
|
84
|
+
## Quick Start
|
|
85
|
+
|
|
86
|
+
```bash
|
|
87
|
+
# 1. Initialize local database
|
|
88
|
+
substack-saved-mcp init
|
|
89
|
+
|
|
90
|
+
# 2. Authenticate with Substack (opens interactive browser window once)
|
|
91
|
+
substack-saved-mcp login
|
|
92
|
+
|
|
93
|
+
# 3. Sync saved posts AND notes into local cache (both entities by default)
|
|
94
|
+
substack-saved-mcp sync
|
|
95
|
+
|
|
96
|
+
# 3b. Or sync just one entity
|
|
97
|
+
substack-saved-mcp sync --only posts
|
|
98
|
+
substack-saved-mcp sync --only notes
|
|
99
|
+
|
|
100
|
+
# 4. Search saved posts via CLI
|
|
101
|
+
substack-saved-mcp search "artificial intelligence"
|
|
102
|
+
|
|
103
|
+
# 4b. Filter by publication or audience tier (see which tiers are cached with `audiences`)
|
|
104
|
+
substack-saved-mcp audiences
|
|
105
|
+
substack-saved-mcp list --audience only_paid
|
|
106
|
+
substack-saved-mcp search "artificial intelligence" --audience everyone
|
|
107
|
+
|
|
108
|
+
# 4c. Find posts you haven't read or haven't finished
|
|
109
|
+
substack-saved-mcp list --read-state unread
|
|
110
|
+
substack-saved-mcp list --read-state unfinished --sort-by minutes_remaining
|
|
111
|
+
substack-saved-mcp search "artificial intelligence" --read-state unread
|
|
112
|
+
|
|
113
|
+
# 5. Save or unsave a post
|
|
114
|
+
substack-saved-mcp save "https://example.substack.com/p/post-slug"
|
|
115
|
+
substack-saved-mcp unsave "https://example.substack.com/p/post-slug"
|
|
116
|
+
|
|
117
|
+
# 6. Get a saved post's full content, cleaned up and ready for an LLM
|
|
118
|
+
substack-saved-mcp get-content "https://example.substack.com/p/post-slug"
|
|
119
|
+
|
|
120
|
+
# 7. Work with saved notes the same way
|
|
121
|
+
substack-saved-mcp list-notes --limit 10
|
|
122
|
+
substack-saved-mcp search-notes "kubernetes" --author alice
|
|
123
|
+
substack-saved-mcp note-authors
|
|
124
|
+
substack-saved-mcp save-note "https://substack.com/@handle/note/c-123456"
|
|
125
|
+
substack-saved-mcp unsave-note "https://substack.com/@handle/note/c-123456"
|
|
126
|
+
substack-saved-mcp get-note "https://substack.com/@handle/note/c-123456"
|
|
127
|
+
|
|
128
|
+
# 8. Check combined status (counts, unread totals, last sync per entity)
|
|
129
|
+
substack-saved-mcp status
|
|
130
|
+
|
|
131
|
+
# 9. Launch stdio MCP server
|
|
132
|
+
substack-saved-mcp serve
|
|
133
|
+
```
|
|
134
|
+
|
|
135
|
+
---
|
|
136
|
+
|
|
137
|
+
## Configuring MCP Clients (Claude Desktop, Goose, Cursor, etc.)
|
|
138
|
+
|
|
139
|
+
Add `substack-saved-mcp` to your MCP client's configuration file (e.g. `claude_desktop_config.json`).
|
|
140
|
+
|
|
141
|
+
### Using the installed tool (`uv tool` or global binary)
|
|
142
|
+
|
|
143
|
+
```json
|
|
144
|
+
{
|
|
145
|
+
"mcpServers": {
|
|
146
|
+
"substack-saved": {
|
|
147
|
+
"command": "substack-saved-mcp",
|
|
148
|
+
"args": ["serve"]
|
|
149
|
+
}
|
|
150
|
+
}
|
|
151
|
+
}
|
|
152
|
+
```
|
|
153
|
+
|
|
154
|
+
### Using `uv` from the repository path
|
|
155
|
+
|
|
156
|
+
To run from your checkout without installing:
|
|
157
|
+
|
|
158
|
+
```json
|
|
159
|
+
{
|
|
160
|
+
"mcpServers": {
|
|
161
|
+
"substack-saved": {
|
|
162
|
+
"command": "uv",
|
|
163
|
+
"args": [
|
|
164
|
+
"--directory",
|
|
165
|
+
"/path/to/substack-saved-mcp",
|
|
166
|
+
"run",
|
|
167
|
+
"substack-saved-mcp",
|
|
168
|
+
"serve"
|
|
169
|
+
]
|
|
170
|
+
}
|
|
171
|
+
}
|
|
172
|
+
}
|
|
173
|
+
```
|
|
174
|
+
|
|
175
|
+
---
|
|
176
|
+
|
|
177
|
+
## Frequently Asked Questions (FAQ)
|
|
178
|
+
|
|
179
|
+
### Where is the database saved?
|
|
180
|
+
|
|
181
|
+
The SQLite database lives in your OS application data directory:
|
|
182
|
+
|
|
183
|
+
- **Linux / macOS**: `~/.local/share/substack-saved-mcp/saved_posts.sqlite`
|
|
184
|
+
*(or `$XDG_DATA_HOME/substack-saved-mcp/saved_posts.sqlite` if you set `XDG_DATA_HOME`)*
|
|
185
|
+
|
|
186
|
+
Set a custom database path or directory with environment variables:
|
|
187
|
+
```bash
|
|
188
|
+
export SUBSTACK_SAVED_DB_PATH="/path/to/my/custom_database.sqlite"
|
|
189
|
+
# or
|
|
190
|
+
export SUBSTACK_SAVED_DATA_DIR="/path/to/my/data_dir"
|
|
191
|
+
```
|
|
192
|
+
|
|
193
|
+
### Which saved posts haven't I read?
|
|
194
|
+
|
|
195
|
+
Filter by reading state on the CLI with `--read-state`, or pass `read_state` to the `list_saved_posts` and `search_saved_posts` MCP tools. In a chat, ask your assistant to "list my unread saved posts".
|
|
196
|
+
|
|
197
|
+
| State | Posts it matches |
|
|
198
|
+
|-------|------------------|
|
|
199
|
+
| `unread` | You never opened them, and Substack recorded no progress. |
|
|
200
|
+
| `in_progress` | You read part, below the finished threshold. |
|
|
201
|
+
| `finished` | You read past the threshold (default 95%). |
|
|
202
|
+
| `unfinished` | Everything below the threshold: `unread` plus `in_progress`. |
|
|
203
|
+
| `started` | You opened them or read any part. |
|
|
204
|
+
|
|
205
|
+
```bash
|
|
206
|
+
substack-saved-mcp sync --force # refresh progress for every post first
|
|
207
|
+
substack-saved-mcp list --read-state unfinished --sort-by minutes_remaining --limit 20
|
|
208
|
+
substack-saved-mcp status # unread count and total minutes remaining
|
|
209
|
+
```
|
|
210
|
+
|
|
211
|
+
Sorting `unfinished` posts by `minutes_remaining` puts the quickest ones to finish at the top. Each post also reports `is_fully_read` and `minutes_remaining`. Notes have no reading progress.
|
|
212
|
+
|
|
213
|
+
### How does the tool decide a post is fully read, and how fresh is that?
|
|
214
|
+
|
|
215
|
+
Substack reports a `max_read_progress` high-water mark (0.0 to 1.0) for each saved post. The tool counts a post as fully read once that value reaches a threshold, `0.95` by default, because finished posts on Substack tend to stop around `0.98` to `0.9999` rather than at `1.0`. Change it with:
|
|
216
|
+
```bash
|
|
217
|
+
export SUBSTACK_SAVED_FULLY_READ_THRESHOLD="0.90"
|
|
218
|
+
```
|
|
219
|
+
A sync refreshes progress for each post it fetches. An incremental sync stops at the first posts it already has, so progress on older posts goes stale until you run `substack-saved-mcp sync --force`. A post you finished last week can show as `unread` until then.
|
|
220
|
+
|
|
221
|
+
### Which Substack API does syncing saved posts use?
|
|
222
|
+
|
|
223
|
+
Sync tries Substack's unified reader API first. If that fails, it falls back to an older posts-only API, then to headless browser scraping. To force one source while troubleshooting, set:
|
|
224
|
+
```bash
|
|
225
|
+
export SUBSTACK_SAVED_POSTS_SOURCE="unified" # or "legacy" or "dom"
|
|
226
|
+
```
|
|
227
|
+
Leave it unset, or set `"auto"`, for the default fallback chain.
|
|
228
|
+
|
|
229
|
+
### What does a "partial" sync status mean?
|
|
230
|
+
|
|
231
|
+
Substack sometimes rate-limits a sync (HTTP 429) so hard that a page of results fails even after retries. The sync then keeps what it fetched and reports `status: partial` instead of `success`, in `substack-saved-mcp status` and in the sync tool's response. On a `sync --force`, a partial run skips reconciliation, so the tool won't mark a post or note as unsaved because it couldn't fetch it. Run `sync` again later to pick up what it missed.
|
|
232
|
+
|
|
233
|
+
### Will a browser window pop up when running as an MCP server?
|
|
234
|
+
|
|
235
|
+
No. During normal MCP use you won't see a browser window.
|
|
236
|
+
|
|
237
|
+
- **Read & Search Tools** (`search_saved_posts`, `list_saved_posts`, `get_saved_post`, `search_saved_notes`, `list_saved_notes`, `get_saved_note`, `list_publications`, `list_audiences`, `saved_posts_status`):
|
|
238
|
+
Query the local SQLite database offline. No browser.
|
|
239
|
+
- **Post Sync & Write Tools** (`sync_saved_posts`, `save_post`, `unsave_post`, `get_post_content`):
|
|
240
|
+
Run headless with the session stored in `storage_state.json`.
|
|
241
|
+
- **Note Sync & Write Tools** (`sync_saved_notes`, `save_note`, `unsave_note`, `get_note_content`):
|
|
242
|
+
Make direct API requests with `storage_state.json`, with no browser page, headless or visible.
|
|
243
|
+
- **Interactive Login**:
|
|
244
|
+
A browser window opens only when you run `substack-saved-mcp login` in your terminal. If your session expires while you use an MCP client, the tool returns an error telling you to run `substack-saved-mcp login`.
|
|
245
|
+
|
|
246
|
+
### What if I get a Playwright "Executable doesn't exist" error?
|
|
247
|
+
|
|
248
|
+
An error like `BrowserType.launch: Executable doesn't exist` (often during `login`) means Playwright hasn't installed its browsers in the tool's isolated environment. Run `playwright install` inside that environment.
|
|
249
|
+
|
|
250
|
+
For a system-wide install (`uv tool install`):
|
|
251
|
+
```bash
|
|
252
|
+
~/.local/share/uv/tools/substack-saved-mcp/bin/playwright install
|
|
253
|
+
```
|
|
254
|
+
|
|
255
|
+
For a development environment (`uv sync`):
|
|
256
|
+
```bash
|
|
257
|
+
uv run playwright install
|
|
258
|
+
```
|
|
259
|
+
|
|
260
|
+
### I edited the source code, but the installed `substack-saved-mcp` command still runs the old version. Why?
|
|
261
|
+
|
|
262
|
+
`uv tool install` copies the package into its own environment at install time and doesn't track your working tree. After you edit files under `src/` or pull new commits, the installed copy keeps running the old code. `uv run substack-saved-mcp ...` from the repo uses your current source.
|
|
263
|
+
|
|
264
|
+
Reinstall from your working tree to pick up the changes:
|
|
265
|
+
```bash
|
|
266
|
+
uv tool install . --no-cache --force
|
|
267
|
+
```
|
|
268
|
+
- `--force` replaces the installed version instead of skipping because one is present.
|
|
269
|
+
- `--no-cache` builds fresh instead of reusing a cached wheel from before your edits.
|