substack-saved-mcp 0.1.0__tar.gz

substack_saved_mcp-0.1.0/.github/workflows/ci.yml

@@ -0,0 +1,32 @@
+name: CI
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+permissions:
+  contents: read
+
+jobs:
+  quality:
+    runs-on: ubuntu-latest
+    strategy:
+      matrix:
+        python-version: ["3.11", "3.12"]
+
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: astral-sh/setup-uv@v6
+        with:
+          enable-cache: true
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: ${{ matrix.python-version }}
+
+      - run: uv sync --extra dev --locked
+      - run: uvx prek@0.4.11 run --all-files
+      - run: uv run pytest
+      - run: uv build

substack_saved_mcp-0.1.0/.github/workflows/pypi.yml

@@ -0,0 +1,61 @@
+# Upload a Python Package using Twine when a release is created
+# Adapted from mne-bids-pipeline
+# From https://github.com/sphinx-gallery/sphinx-gallery/blob/master/.github/workflows/release.yml
+
+name: Build package
+on:  # yamllint disable-line rule:truthy
+  release:
+    types: [published]
+  push:
+    branches:
+      - main
+  pull_request:
+    branches:
+      - main
+
+permissions:
+  contents: read
+  id-token: write
+
+jobs:
+  package:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v5
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: '3.12'
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install build twine
+      - name: Build package
+        run: python -m build --sdist --wheel
+      - name: Check package
+        run: twine check --strict dist/*
+      - name: Check env vars
+        run: |
+          echo "Triggered by: ${{ github.event_name }}"
+      - uses: actions/upload-artifact@v4
+        with:
+          name: dist
+          path: dist
+
+  # PyPI on release
+  pypi:
+    needs: package
+    runs-on: ubuntu-latest
+    if: github.event_name == 'release'
+    environment:
+      name: pypi
+    steps:
+      - uses: actions/download-artifact@v5
+        with:
+          name: dist
+          path: dist
+      - name: Publish to PyPI
+        uses: pypa/gh-action-pypi-publish@release/v1
+        with:
+          user: __token__
+          password: ${{ secrets.PYPI_API_TOKEN }}

substack_saved_mcp-0.1.0/.gitignore

@@ -0,0 +1,19 @@
+# Python & environments
+.venv/
+__pycache__/
+*.py[cod]
+*$py.class
+*.egg-info/
+dist/
+build/
+.pytest_cache/
+
+# Application Data & Credentials (DO NOT COMMIT)
+data/
+browser_state/
+browser-state/
+storage_state.json
+*.sqlite
+*.sqlite-wal
+*.sqlite-shm
+.env

substack_saved_mcp-0.1.0/.pre-commit-config.yaml

@@ -0,0 +1,6 @@
+repos:
+  - repo: https://github.com/astral-sh/ruff-pre-commit
+    rev: v0.16.1
+    hooks:
+      - id: ruff-check
+      - id: ruff-format

substack_saved_mcp-0.1.0/CLAUDE.md

@@ -0,0 +1,43 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code when working with this repository.
+
+## Commands
+
+Use `uv` for all development commands. Set up the environment with:
+
+```bash
+uv sync --extra dev
+```
+
+| Task | Command |
+| --- | --- |
+| Run all tests | `uv run python -m pytest` |
+| Run one test file | `uv run python -m pytest tests/test_database.py` |
+| Run one test | `uv run python -m pytest tests/test_database.py::test_fts5_search` |
+| Build the package | `uv build` |
+| Run the CLI | `uv run substack-saved-mcp --help` |
+| Reinstall tool globally | `uv tool install . --no-cache --force` |
+
+`ruff` is configured in `pyproject.toml` (rules: `E4`, `E7`, `E9`, `F`, `I`, `UP`, `B`, `RUF`, line length 88, target py311). Lint with `uvx ruff check .` or `uv run ruff check .`.
+
+## Architecture
+
+- `cli.py` is the Click entry point (`substack-saved-mcp`). Its commands initialize the database and then delegate to the repository, sync engine, Playwright client, or MCP server.
+- `mcp_server.py` exposes the same application operations as FastMCP stdio tools and resources. Read tools query the local cache; sync and save/unsave operations use the authenticated browser session.
+- `substack_client.py` owns Playwright authentication and remote Substack interaction. `login` is the only intended headful workflow; normal sync and write paths use `storage_state.json` headlessly. Synchronous Playwright API calls are routed through `_run_playwright_sync()` to safely execute in a worker thread if an `asyncio` event loop is active (e.g. under FastMCP). Saved-post fetching prefers the reader inbox API (`GET /api/v1/reader/posts?inboxType=saved`), which exposes the real bookmark timestamp (`saved_at`) and an ISO publication date (`post_date`) per post. `_fetch_all_saved_via_reader_api()` cursor-paginates that endpoint (each page's oldest `saved_at` becomes the next `after=` cursor, until `more` is false), dedupes by canonical URL, and enriches each flat post with its `publication` object (from the response's `publications` array, matched by `publication_id`) and `author_name` (from `publishedBylines`). The full result is cached in `_api_cache` and sliced by offset. Each page request goes through `_reader_api_get()`, which retries transient failures (HTTP 429 and 5xx) up to `max_retries` (default 3) with `Retry-After`-aware backoff: `_retry_after_seconds()` honors an integer `Retry-After` header (clamped to a 30s cap so a hostile value can't hang the sync) and otherwise uses capped exponential backoff (0.5s, 1s, 2s, ...); 401/403 and other 4xx are returned unretried so the existing auth/fallback handling applies. A 429 that survives all retries is deliberately treated as "unavailable" (partial list, or `None` → DOM fallback) rather than as a silent empty-success, so rate-limiting never masquerades as "you have no saved posts." Both `_reader_api_get()` and `_fetch_all_saved_via_reader_api()` accept an injectable `sleep_func` (default `time.sleep`) so tests exercise the backoff without real delays. If that endpoint is unavailable, it falls back to headless DOM extraction on `https://substack.com/saved`, which scrapes `div.reader2-post-container` cards, caches scrolling results in `_dom_cache`, and marks each dict with `_dom: True`; DOM cards only expose a localized relative publish string and no bookmark time, so `saved_at` stays unknown on that path. Both `save_post()` and `unsave_post()` were originally guesswork (an unverified DOM button selector plus an unverified `POST /api/v1/bookmark` call) and were fixed once real endpoints were captured via `inspect-network`. Every post page server-renders a `window._preloads` blob containing that post's own numeric ID and rich metadata (`preloads.post.id`, `.title`, `.audience`, `.description`/`.subtitle`, `.post_date`, plus `preloads.pub.name`); `_save_post_impl` reads this via `page.evaluate("() => window._preloads")` right after page load and, when an ID is found, calls the real `POST https://substack.com/api/v1/posts/saved` endpoint (body `{"post_id": ...}`) directly via the Playwright API request context — this also lets it populate the returned `SavedPost` with accurate title/publication/audience/excerpt instead of parsing `page.title()`. `_unsave_post_impl` does the mirror image: when the post's `substack_post_id` is already known (true for any post that has been through a normal `sync`, since the reader API's `id` field populates it), it calls `DELETE https://substack.com/api/v1/posts/saved` with the same body shape, no DOM interaction at all. Both treat an `ok` response as `"confirmed"` and only fall back to the old best-effort DOM click (`_click_bookmark_toggle()`, used when the numeric ID can't be obtained or the direct call doesn't confirm) — note its selector is English-only (`aria-label*='save'/'bookmark'`) and can silently fail on non-English Substack UIs, which is part of why the direct API path is preferred whenever possible. `_click_bookmark_toggle()` fingerprints the button's `aria-label`/`aria-pressed`/`class` before and after the click and returns `"confirmed"` only if that fingerprint changed, else `"unconfirmed"`, `"not_found"`, or `"click_failed"` — `save_post()` returns `(SavedPost, confirmation)` and `unsave_post()` returns just the confirmation string; both accept an optional `playwright_instance` for test injection (same pattern as `_fetch_via_dom`). `fetch_post_content()` reuses the same `window._preloads` mechanism to retrieve a saved post's full content: `_fetch_post_content_impl` navigates to the post's page and reads `preloads.post.body_html` (Substack's field name for full content — `parse_remote_post()` already expects this key from the reader API, though the saved-list payload never actually populates it, only individual post pages do), returning `None` for `body_html` if the page's embed format doesn't expose it (frontend change) or the account lacks paywall access; this is the case where the caller should be told to run `inspect-network` against an open post page to re-discover the real content source. `cli.py`'s `inspect-network` command (which logs any `api/v1`/`bookmark`/`saved` request's method, URL, status, and JSON body while the user manually clicks around a headful browser) is the tool for discovering/re-verifying these endpoints when Substack's frontend changes.
+- `content_utils.py` converts a post's raw `body_html` into clean text for LLM consumption. `html_to_llm_text()` is a small `html.parser.HTMLParser` subclass (no external HTML library dependency) that renders headings as markdown `#` prefixes, list items as `- ` bullets, links as `text (url)`, keeps minimal `**bold**`/`*italic*` markers, drops `script`/`style`/`iframe` content entirely, and collapses excess blank lines. `format_post_for_llm()` prepends a plain metadata header (title, publication, author, published date, URL) to the cleaned body text — this combined string is what both the CLI's `get-content` command and the MCP `get_post_content` tool return.
+- `sync.py` converts API/DOM payloads to `SavedPost` models and coordinates paginated incremental or full syncs. `parse_remote_post()` routes DOM dicts (identified by the `_dom` marker) through a minimal mapping and everything else through the full reader-API/legacy mapping; it never fabricates a `saved_at` from the sync moment (unknown save times stay `None`). `word_count` is mapped defensively from several unconfirmed candidate keys (`wordcount`/`word_count`/`words`, via `_first_positive_int()` which ignores non-positive/uncoercible values) — the exact reader-API field name hasn't been captured, so if these stay empty an `inspect-network` capture is needed to find the real one. `reading_time_minutes` is intentionally *derived* (ceil of `word_count / WORDS_PER_MINUTE`, ~200 wpm) rather than mapped from a field, because a wrong guess about that field's unit (seconds vs minutes) would persist a badly wrong value; deriving is unit-unambiguous. `image_url` is mapped from `cover_image`/`image_url` on the reader-API payload and is populated for the large majority of synced posts (confirmed live: ~98%); it was previously stored but invisible through every read path except `get_saved_post`/the full `SavedPost` — `PostSummary` and the `list_posts()`/`search_posts()` SELECTs now include it too; the CLI `search` command prints it when present, but `list` deliberately omits it since it's a long, uninformative CDN URL in that terser view. `content_text` remains unpopulated by sync (only `get-content`/`get_post_content` populate it) because the saved-list payload doesn't carry post body HTML, only individual post pages do. There is no `metadata_json` column: it was a never-populated "raw source JSON" placeholder from the original PLAN and has been removed from the model and schema. Existing databases created before its removal may still have an inert `metadata_json` column; reads tolerate it because `SavedPost` (Pydantic) ignores unknown columns from `SELECT *`, and `upsert_post` no longer references it. A force/full sync (`--force`) collects every fetched post's URL and, after the fetch loop completes, calls `reconcile_unsaved_posts()` to soft-delete any locally `is_saved = 1` post absent from that complete remote set — this is how posts unsaved directly on Substack (outside this tool) get reflected locally. An incremental sync never reconciles, since its early-stop-on-matches optimization means it only sees a partial remote list. It records each run in SQLite (including `reconciled_count`) and returns a `SyncRun` with `success`, `auth_required`, or `failed` status instead of propagating expected authentication failures.
+- `database.py` is the SQLite repository and schema owner. The `posts` table is the cache; `sync_runs` records sync history; `posts_fts` is an external-content FTS5 index maintained by database triggers. `reconcile_unsaved_posts()` bulk soft-deletes posts missing from a given complete remote URL set and is a no-op on an empty list (an empty list is more likely a fetch problem than genuine mass-unsaving). `init_db()` runs additive column migrations (e.g. `posts.audience`, `sync_runs.reconciled_count`) via `ALTER TABLE` guarded by an existence check or `try`/`except sqlite3.OperationalError`, since `CREATE TABLE IF NOT EXISTS` never adds columns to an already-existing table; the `posts.audience` check runs before the `executescript` block because the `idx_posts_audience` index creation inside it would otherwise fail on a pre-migration table. `list_posts()`/`search_posts()` accept an `audience` filter (exact match, case-insensitive), and `list_audiences()` returns the distinct audience values actually present in the cache with post counts, rather than hardcoding Substack's (undocumented, possibly-growing) audience enum. `PostSummary` (returned by `list_posts()`/`search_posts()` and the MCP list/search tools) carries `reading_time_minutes` and `word_count`, and the CLI `list`/`search` commands display them — without this the values were populated in the `posts` table but invisible through every read path except `get_saved_post`/the full `SavedPost`.
+- `config.py` centralizes application paths. By default, data lives under `~/.local/share/substack-saved-mcp`; `SUBSTACK_SAVED_DB_PATH`, `SUBSTACK_SAVED_DATA_DIR`, and `SUBSTACK_SAVED_BROWSER_DIR` override those paths.
+
+## Repository Conventions
+
+- Model persisted posts with the Pydantic `SavedPost` schema. Keep `published_at` (post publication time) distinct from `saved_at` (bookmark time). Both come from the source payload; never fabricate `saved_at` from the sync/DB-insert moment — leave it `None` when the source does not expose the original bookmark time (`upsert_post` preserves a known value and stores `NULL` otherwise).
+- Store the raw Substack `audience` string (e.g. `everyone`, `only_paid`) verbatim on `SavedPost.audience` rather than only collapsing it into the `is_paywalled` boolean; `parse_remote_post()` still derives `is_paywalled` from it (`audience == "only_paid"`) for backward-compatible boolean filtering, but the original value is preserved for exact-match filtering and discovery via `list_audiences()`.
+- Canonicalize every post URL through `canonicalize_url()` before storage or lookup. It strips known tracking parameters, fragments, and non-root trailing slashes; SQLite uniqueness and upsert behavior depend on that canonical form.
+- Treat unsaving as a soft delete: use `soft_delete_post()` (single post, e.g. the `unsave` command/tool) or `reconcile_unsaved_posts()` (bulk, called from a force sync) to set `is_saved = 0` and `unsaved_at`, retaining the record for history. Default list/search/publication queries intentionally include only `is_saved = 1`.
+- The local cache always reflects the user's intent for `save`/`unsave`, regardless of whether the remote toggle on Substack could be confirmed — never block the local write on remote confirmation. Instead, surface the confirmation status (`"confirmed"` vs. `"unconfirmed"/"not_found"/"click_failed"`) as a clearly-labeled warning to the caller (CLI: yellow `click.secho`; MCP: a `warning` key and `remote_confirmed: bool` in the tool's dict response) so an unconfirmed remote action isn't silently reported as a full success. A subsequent `sync --force` will self-correct the local cache via `reconcile_unsaved_posts()` if a "confirmed" save/unsave didn't actually happen remotely.
+- Do not write directly to `posts_fts`; use the `posts` repository operations so the SQLite insert/update/delete triggers maintain the FTS5 index.
+- Database-facing tests use `tmp_path` databases. CLI and MCP tests set `SUBSTACK_SAVED_DB_PATH` with `monkeypatch` so they never access a developer's real cache. Sync tests inject a `SubstackSavedPostsClient` substitute rather than opening Playwright or contacting Substack.
+- Browser state and SQLite data are credentials or user data. Keep `storage_state.json`, browser-state directories, `.env` files, and database files untracked; configuration creates data directories with user-only permissions on POSIX systems.

substack_saved_mcp-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Toni Hermoso Pulido
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

substack_saved_mcp-0.1.0/PKG-INFO

@@ -0,0 +1,218 @@
+Metadata-Version: 2.4
+Name: substack-saved-mcp
+Version: 0.1.0
+Summary: Local stdio-based MCP server and sync engine for Substack saved posts
+Author-email: Toni Hermoso Pulido <toniher@cau.cat>
+License-Expression: MIT
+License-File: LICENSE
+Requires-Python: >=3.11
+Requires-Dist: click>=8.1.0
+Requires-Dist: fastmcp>=0.1.0
+Requires-Dist: playwright>=1.40.0
+Requires-Dist: pydantic>=2.0.0
+Provides-Extra: dev
+Requires-Dist: pytest-asyncio>=0.23.0; extra == 'dev'
+Requires-Dist: pytest>=8.0.0; extra == 'dev'
+Requires-Dist: ruff==0.16.1; extra == 'dev'
+Description-Content-Type: text/markdown
+
+# Substack Saved Posts MCP & CLI
+
+A local, stdio-based Model Context Protocol (MCP) server and sync engine for your saved/bookmarked Substack posts.
+
+## Features
+
+- **Read & Search**: Full-text search (SQLite FTS5) across saved post titles, excerpts, authors, and publications. Filter by publication, audience tier (e.g. `everyone`, `only_paid`), and date ranges (`published_at` vs `saved_at`). Search also covers a post's **full body text**, but only for posts whose content has already been fetched once via `get-content` / the `get_post_content` tool — a normal `sync` stores metadata and excerpts, not full bodies, so posts you haven't opened yet are matched on their title/excerpt/metadata only, not their full text.
+- **Full Content for LLMs**: Fetch a saved post's full content and get it back cleaned and formatted (headings, lists, links) for feeding directly to an LLM, with the result cached locally for next time.
+- **Save & Unsave**: Bookmark new Substack posts or unbookmark existing ones via authenticated browser sessions.
+- **Offline First**: Fast, offline queries directly from local SQLite cache.
+- **Privacy & Security**: Keeps session credentials local, redacting tokens from logs.
+- **FastMCP Protocol**: Stdio MCP interface with rich tool suite and resources.
+
+---
+
+## Installation with `uv`
+
+[`uv`](https://github.com/astral-sh/uv) is the recommended fast Python package manager for installing and running `substack-saved-mcp`.
+
+### Option A: Install System-wide as a Tool (`uv tool install`)
+
+Install directly from your local repository folder:
+
+```bash
+# Navigate to the repository
+cd /path/to/substack-saved-mcp
+
+# Install system-wide into an isolated uv environment
+uv tool install .
+
+# Or install directly from a remote Git repository:
+# uv tool install git+https://github.com/your-username/substack-saved-mcp.git
+```
+
+After installation, `substack-saved-mcp` is immediately available in your PATH:
+
+```bash
+# Verify installation
+substack-saved-mcp --help
+```
+
+To update or uninstall:
+```bash
+# Upgrade installed tool
+uv tool upgrade substack-saved-mcp
+
+# Uninstall tool
+uv tool uninstall substack-saved-mcp
+```
+
+---
+
+### Option B: Local Development / Development Environment (`uv sync`)
+
+If you are developing or modifying the codebase:
+
+```bash
+# Clone and enter directory
+cd substack-saved-mcp
+
+# Install dependencies and dev tools (pytest)
+uv sync --extra dev
+
+# Run CLI commands using uv run
+uv run substack-saved-mcp --help
+
+# Run tests
+uv run pytest
+```
+
+---
+
+## Quick Start
+
+```bash
+# 1. Initialize local database
+substack-saved-mcp init
+
+# 2. Authenticate with Substack (opens interactive browser window once)
+substack-saved-mcp login
+
+# 3. Sync saved posts into local cache
+substack-saved-mcp sync
+
+# 4. Search saved posts via CLI
+substack-saved-mcp search "artificial intelligence"
+
+# 4b. Filter by publication or audience tier (see which tiers are cached with `audiences`)
+substack-saved-mcp audiences
+substack-saved-mcp list --audience only_paid
+substack-saved-mcp search "artificial intelligence" --audience everyone
+
+# 5. Save or unsave a post
+substack-saved-mcp save "https://example.substack.com/p/post-slug"
+substack-saved-mcp unsave "https://example.substack.com/p/post-slug"
+
+# 6. Get a saved post's full content, cleaned up and ready for an LLM
+substack-saved-mcp get-content "https://example.substack.com/p/post-slug"
+
+# 7. Launch stdio MCP server
+substack-saved-mcp serve
+```
+
+---
+
+## Configuring MCP Clients (Claude Desktop, Goose, Cursor, etc.)
+
+Add `substack-saved-mcp` to your MCP client's configuration file (e.g. `claude_desktop_config.json`).
+
+### Using System-Wide Installed Tool (`uv tool` or global binary)
+
+```json
+{
+  "mcpServers": {
+    "substack-saved": {
+      "command": "substack-saved-mcp",
+      "args": ["serve"]
+    }
+  }
+}
+```
+
+### Using `uv` directly from the Repository Path
+
+If you prefer running directly from your repository path without installing system-wide:
+
+```json
+{
+  "mcpServers": {
+    "substack-saved": {
+      "command": "uv",
+      "args": [
+        "--directory",
+        "/path/to/substack-saved-mcp",
+        "run",
+        "substack-saved-mcp",
+        "serve"
+      ]
+    }
+  }
+}
+```
+
+---
+
+## Frequently Asked Questions (FAQ)
+
+### Where is the database saved?
+
+By default, the SQLite database is saved in your OS application data directory:
+
+- **Linux / macOS**: `~/.local/share/substack-saved-mcp/saved_posts.sqlite`  
+  *(or `$XDG_DATA_HOME/substack-saved-mcp/saved_posts.sqlite` if `XDG_DATA_HOME` is set)*
+
+You can specify a custom database path or directory using environment variables:
+```bash
+export SUBSTACK_SAVED_DB_PATH="/path/to/my/custom_database.sqlite"
+# or
+export SUBSTACK_SAVED_DATA_DIR="/path/to/my/data_dir"
+```
+
+### Will a browser window pop up when running as an MCP server?
+
+**No, a visible browser window will not open during normal MCP operations.**
+
+- **Read & Search Tools** (`search_saved_posts`, `list_saved_posts`, `get_saved_post`, `list_publications`, `saved_posts_status`):  
+  Operate 100% offline using the local SQLite database. Zero browser activity.
+- **Sync & Write Tools** (`sync_saved_posts`, `save_post`, `unsave_post`, `get_post_content`):  
+  Run in **headless background mode** using the pre-authenticated session stored in `storage_state.json`.
+- **Interactive Login**:  
+  A visible browser window opens **only** when you manually run `substack-saved-mcp login` from your terminal. If your session expires while using an MCP client, the tool will return a clear error message instructing you to re-authenticate via `substack-saved-mcp login` instead of popping open a browser window unexpectedly.
+
+### What if I get a Playwright "Executable doesn't exist" error?
+
+If you encounter an error like `BrowserType.launch: Executable doesn't exist` when running commands (especially `login`), it means Playwright hasn't installed its required browsers in the isolated environment.
+
+To fix this, you need to run the `playwright install` command *inside* the environment where the tool is installed. 
+
+For a system-wide tool installation (via `uv tool install`), run:
+```bash
+~/.local/share/uv/tools/substack-saved-mcp/bin/playwright install
+```
+
+If you are using a local development environment (via `uv sync`), run:
+```bash
+uv run playwright install
+```
+
+### I edited the source code, but the installed `substack-saved-mcp` command still behaves like the old version. Why?
+
+`uv tool install` copies the package into its own isolated environment at install time — it does **not** track your working tree. If you edited files under `src/` (or pulled new commits) after installing the tool system-wide, the globally installed copy is stale and keeps running the old code, even though `uv run substack-saved-mcp ...` from the repo would use the latest source.
+
+Reinstall from your current working tree to pick up the changes:
+```bash
+uv tool install . --no-cache --force
+```
+- `--force` replaces the existing installed version instead of skipping the install because a version is already present.
+- `--no-cache` ensures a fresh build rather than reusing a cached wheel/build artifact from before your edits.
+
+Do this any time after modifying the codebase and before relying on the globally installed `substack-saved-mcp` binary (as opposed to `uv run substack-saved-mcp`, which always reflects the working tree).

substack_saved_mcp-0.1.0/PLAN.md

@@ -0,0 +1,232 @@
+# Saved Substack Posts MCP Plan
+
+## Goal
+
+Create a local, stdio-based MCP server that lets an MCP client (such as Claude Desktop or Goose) search, retrieve, save, and unsave a user's Substack posts. A background/CLI sync process retrieves saved posts through the user's authenticated Substack browser session and maintains a local SQLite cache.
+
+The server must work offline against cached data for search and read queries, while write operations (`save_post` and `unsave_post`) use the authenticated Playwright browser session to sync actions directly with Substack and update the local database.
+
+The server tracks both **when the post was saved** (`saved_at`) and **the original publication date of the post** (`published_at`).
+
+---
+
+## Assumptions and Boundaries
+
+- "Favourite" / "Saved" means a post saved or bookmarked in the user's Substack account (`https://substack.com/saved`).
+- Substack does not provide a documented, stable public API for bookmarking. Sync and write actions use Playwright authenticated contexts or reverse-engineered private API endpoints.
+- Read operations (list, search, get) work 100% offline using the local SQLite cache.
+- Write operations (`save_post`, `unsave_post`) require a valid authenticated session to update Substack remotely, followed immediately by updating the local SQLite cache.
+- The project will use Python 3.11+, Playwright, SQLite (with FTS5), and FastMCP over stdio transport.
+
+---
+
+## Key Architecture & Data Flow
+
+```text
+Substack Authenticated Session (Playwright persistent profile / storage_state.json)
+                                |
+                   +------------+------------+
+                   |                         |
+          (Read / Sync Engine)      (Write Operations)
+                   |                         |
+                   v                         v
+           sync_saved_posts.py       save_post() / unsave_post()
+           - Paginated sync          - Calls Substack bookmark API / DOM
+           - Auth expiration check   - Updates remote bookmark state
+           - Normalizes posts        - Updates local DB immediately
+                   |                         |
+                   +------------+------------+
+                                |
+                                v
+                saved_posts.sqlite (SQLite + FTS5)
+                - Stores published_at & saved_at
+                - Tracks is_saved (1 = active, 0 = unsaved)
+                                |
+                                v
+                       FastMCP stdio Server
+           - Tools: list, search, get_post, save_post, unsave_post, list_publications, status, sync
+           - Resources: substack://posts/{id}, substack://publications
+                                |
+                                v
+                            MCP Client
+```
+
+---
+
+## Repository Layout
+
+```text
+substack-saved-mcp/
+  pyproject.toml
+  README.md
+  PLAN.md
+  src/substack_saved_mcp/
+    __init__.py
+    config.py                  # Environment & default OS app-data paths
+    models.py                  # Dataclasses/Pydantic schemas for posts & sync runs
+    database.py                # SQLite schema, FTS5 virtual table, migrations, queries
+    url_utils.py               # Canonicalization & tracking query param stripping
+    substack_client.py         # Playwright reader & writer for Substack posts
+    sync.py                    # Incremental & full sync engine with rate limiting
+    mcp_server.py              # FastMCP server, read/write tools, and resources
+    cli.py                     # Click/Typer CLI (login, sync, serve, save, unsave, status, inspect)
+  tests/
+    test_database.py
+    test_url_utils.py
+    test_normalization.py
+    test_search.py
+    test_write_actions.py      # Tests for save and unsave database & API handlers
+    test_mcp_server.py
+    fixtures/                  # Sanitized sample API response JSON payloads
+  data/                        # Gitignored default SQLite location
+  browser-state/               # Gitignored Playwright storage_state.json & user profile
+```
+
+---
+
+## Data Model & Schema Design
+
+Use SQLite with FTS5 enabled from initial setup to provide fast full-text search across titles, publication names, authors, and excerpts/content.
+
+### Table: `posts`
+
+| Column | Type | Constraints / Description |
+|---|---|---|
+| `id` | INTEGER | Primary Key AUTOINCREMENT |
+| `substack_post_id` | TEXT | Remote identifier when available; `UNIQUE` when populated |
+| `url` | TEXT | Canonical post URL (tracking params stripped); `UNIQUE` |
+| `title` | TEXT | Post title |
+| `publication_name` | TEXT | Publication display name |
+| `publication_url` | TEXT | Publication URL |
+| `author_name` | TEXT | Author display name |
+| `published_at` | TEXT | **ISO-8601 UTC timestamp of original post publication** |
+| `saved_at` | TEXT | **ISO-8601 UTC timestamp when post was saved/bookmarked** |
+| `unsaved_at` | TEXT | Optional ISO-8601 UTC timestamp when unsaved |
+| `is_saved` | INTEGER | **1 = active saved post, 0 = unsaved post** (default `1`) |
+| `excerpt` | TEXT | Plain-text summary or lead snippet |
+| `content_text` | TEXT | Optional full body text (plain text or Markdown) |
+| `image_url` | TEXT | Optional lead thumbnail/cover image URL |
+| `is_paywalled` | INTEGER | Boolean flag (0 = free, 1 = paywalled) |
+| `reading_time_minutes` | INTEGER | Optional estimated reading time |
+| `word_count` | INTEGER | Optional word count |
+| `metadata_json` | TEXT | ~~Raw sanitized source JSON retained for future migrations~~ **Removed during implementation** — was never populated; the model and schema no longer include it (pre-existing DBs tolerate the leftover inert column). |
+| `created_at` | TEXT | Local record insertion timestamp (ISO-8601) |
+| `updated_at` | TEXT | Local record update timestamp (ISO-8601) |
+
+### Indexes
+- `idx_posts_url` ON `posts(url)`
+- `idx_posts_published_at` ON `posts(published_at DESC)`
+- `idx_posts_saved_at` ON `posts(saved_at DESC)`
+- `idx_posts_is_saved` ON `posts(is_saved)`
+
+### Virtual Table: `posts_fts` (FTS5)
+
+Created over `title`, `publication_name`, `author_name`, `excerpt`, and `content_text` with triggers to keep `posts_fts` synchronized on `INSERT`, `UPDATE`, and `DELETE`. Search queries filter by default on `is_saved = 1`.
+
+---
+
+## MCP Tool Suite Specification
+
+| Tool Name | Action | Parameters | Description & Output |
+|---|---|---|---|
+| `search_saved_posts` | **Read** | `query` (str, req), `publication` (opt), `published_after` (opt), `published_before` (opt), `saved_after` (opt), `saved_before` (opt), `limit` (int, default 20) | Full-text FTS5 search across saved posts. Returns list of concise posts with `title`, `url`, `publication_name`, `published_at`, `saved_at`, `excerpt`. |
+| `list_saved_posts` | **Read** | `limit` (int, default 20), `offset` (int, default 0), `publication` (opt), `sort_by` (`saved_at` \| `published_at`, default `saved_at`) | Paginated list of active saved posts ordered by `saved_at` or `published_at`. |
+| `get_saved_post` | **Read** | `url_or_id` (str, req) | Detailed view of a single post, including full metadata, timestamps (`published_at`, `saved_at`), and `content_text` if cached. |
+| `save_post` | **Write** | `url` (str, req) | Saves/bookmarks a Substack post remotely on Substack and updates local DB. Returns updated post object with `saved_at` set to current UTC time. |
+| `unsave_post` | **Write** | `url_or_id` (str, req) | Unsaves/unbookmarks a Substack post remotely on Substack and updates local DB (`is_saved = 0`, `unsaved_at = current UTC time`). |
+| `list_publications` | **Read** | None | Returns list of all publications present in cache with count of saved posts for each. |
+| `saved_posts_status` | **Read** | None | Cache statistics: active saved posts count, total unsaved count, last sync run details. |
+| `sync_saved_posts` | **Sync** | `force` (bool, default False) | Triggers background/incremental sync from Substack account into local DB. |
+
+---
+
+## Detailed Write Action & Date Tracking Design
+
+### 1. Date Tracking Strategy (`published_at` vs `saved_at`)
+- **`published_at`**: Extracted from Substack post payload (`post.post_date` or HTML `<time datetime="...">` metadata). Preserves when the article was original authored/published.
+- **`saved_at`**:
+  - When synced from Substack: Extracted from Substack's bookmark object (e.g. `bookmark.created_at`).
+  - When saved via `save_post` tool: Recorded immediately as local ISO-8601 UTC timestamp (`datetime.now(timezone.utc)`).
+
+### 2. Implementation of `save_post(url)`
+1. **Canonicalization**: Strip tracking parameters (`utm_*`, `r`, `s`, etc.) from `url`.
+2. **Remote Execution**:
+   - Using `SubstackSavedPostsClient` with `storage_state.json`:
+   - Issue authenticated POST request to Substack bookmark endpoint (e.g. `/api/v1/bookmark` or post page action).
+   - If session is expired (401/403), fail early with `auth_required` error message.
+3. **Database Update**:
+   - Parse returned post metadata (title, publication, `published_at`).
+   - Set `saved_at = datetime.now(timezone.utc).isoformat()`, `is_saved = 1`, `unsaved_at = NULL`.
+   - Upsert record into `posts` table and update `posts_fts`.
+4. **Return**: Structured output with confirmation and full post summary.
+
+### 3. Implementation of `unsave_post(url_or_id)`
+1. **Resolution**: Lookup canonical post in SQLite by ID or canonical URL.
+2. **Remote Execution**:
+   - Issue authenticated DELETE request to Substack bookmark endpoint for `substack_post_id`.
+   - If session is expired (401/403), fail early with `auth_required` error message.
+3. **Database Update**:
+   - Set `is_saved = 0` and `unsaved_at = datetime.now(timezone.utc).isoformat()`.
+   - Keep post record in database (soft delete) so history is preserved and sync does not re-add it unexpectedly.
+4. **Return**: Structured confirmation containing unsaved post title and canonical URL.
+
+---
+
+## Execution Phases
+
+### Phase 1: Package Setup, Schema & Date/URL Utilities
+1. Initialize Python package (`pyproject.toml` with `fastmcp`, `playwright`, `pytest`).
+2. Implement `url_utils.py` for query param stripping and canonical URL normalization.
+3. Implement `database.py` with SQLite schema including `published_at`, `saved_at`, `unsaved_at`, `is_saved`, FTS5 triggers, and CRUD queries.
+4. Build `cli.py` skeleton with subcommands: `init`, `login`, `sync`, `serve`, `save`, `unsave`, `status`, `inspect-network`.
+
+### Phase 2: Session Capture & Network Inspection (Reads & Writes)
+1. Implement `substack-saved-mcp login` using Playwright in headful mode (`headless=False`).
+2. Store authenticated browser storage state in `browser-state/storage_state.json`.
+3. Implement `inspect-network` command to record network payloads for:
+   - Reading saved posts page (`GET /api/v1/saved_posts`)
+   - Bookmarking a post (`POST /api/v1/bookmark`)
+   - Unbookmarking a post (`DELETE /api/v1/bookmark`)
+4. Implement `SubstackSavedPostsClient` with `fetch_saved_posts()`, `save_post()`, and `unsave_post()` methods.
+
+### Phase 3: Incremental Sync Engine & Error Guardrails
+1. Implement paginated sync with rate-limit backoff (500ms delay between pages).
+2. Store `published_at` (original post date) and `saved_at` (bookmark date) for each normalized post.
+3. Support incremental sync cutoff when encountering existing saved posts.
+4. Handle session expiration cleanly with actionable error messages.
+
+### Phase 4: Local Query Engine with FTS5 & Date Filtering
+1. Implement repository methods in `database.py`:
+   - `search_posts(query, publication, published_after, published_before, saved_after, saved_before, limit)`
+   - `list_posts(limit, offset, publication, sort_by)`
+   - `get_post(url_or_id)`
+   - `list_publications()`
+   - `save_post_db(post_data)`
+   - `unsave_post_db(url_or_id)`
+2. Ensure search and list queries filter by default on `is_saved = 1`.
+
+### Phase 5: FastMCP Tool & Resource Suite
+1. Implement `mcp_server.py` with all 8 tools (`search_saved_posts`, `list_saved_posts`, `get_saved_post`, `save_post`, `unsave_post`, `list_publications`, `saved_posts_status`, `sync_saved_posts`).
+2. Expose resources `substack://posts/{id}` and `substack://publications`.
+3. Bind `substack-saved-mcp serve` CLI command.
+
+### Phase 6: Privacy, Hardening & Security Audit
+1. Verify storage paths use standard OS app-data directories.
+2. Audit log sanitization to ensure tokens/cookies are never logged.
+3. Validate write actions prevent unexpected side-effects.
+
+### Phase 7: Comprehensive Testing Suite
+1. **Unit Tests**: Test URL canonicalization, date parsing, FTS queries, and soft-deletion logic.
+2. **Write Action Tests**: Test `save_post` and `unsave_post` database state transitions and mock API client calls.
+3. **MCP Integration Tests**: Verify all 8 tools over stdio with fixture database states.
+
+---
+
+## Definition of Done (v1.0)
+
+- User can authenticate via `login`.
+- MCP client can search saved posts with full-text FTS5 query matching and date range filters (`published_at`, `saved_at`).
+- MCP client can issue `save_post` to bookmark a new Substack URL remotely and cache it locally.
+- MCP client can issue `unsave_post` to unbookmark a Substack post remotely and soft-delete it locally.
+- `published_at` (original post date) and `saved_at` (bookmark date) are tracked and returned for every post.
+- Fixture tests pass for all read, search, write, and sync tool workflows.