mdfetch 0.2.2__tar.gz → 0.3.0__tar.gz

mdfetch-0.3.0/.specify/feature.json

@@ -0,0 +1,3 @@
+{
+  "feature_directory": "specs/005-substack-provider"
+}

{mdfetch-0.2.2 → mdfetch-0.3.0}/.specify/memory/changelog.md

@@ -2,6 +2,32 @@
 
 ---
 
+### mdfetch — Substack Provider — 2026-05-15
+
+**Branch**: `005-substack-provider`
+**Spec**: specs/005-substack-provider
+
+**What was added**:
+- `SubstackExtractor` provider for `substack.com` and all `*.substack.com` subdomain articles, auto-discovered via `@register` decorator
+- Article body isolation from `div.body.markup`; title prepended from `h1.post-title` in `div.post-header` (unconditional — structurally outside body); optional subtitle from `h3.subtitle` prepended after title
+- `div.subscription-widget-wrap` stripped (inline subscribe CTAs and paywall terminal widget — achieves silent free-preview truncation for paywalled posts without error or marker)
+- `<iframe>` and unknown `div[data-component-name]` embed containers converted to plain anchor links (consistent with dev.to embed handling)
+- HTTP 429 retried via base-class fixed-delay retry (no `_no_retry_status_codes` override — no Freedium-style mirror for Substack)
+- `UnsupportedContentTypeError` raised when `div.body.markup` is absent; `EmptyContentError` raised when body yields no extractable text
+- 18 unit tests in `tests/unit/test_substack_extractor.py`; 3 integration tests in `tests/integration/test_substack_integration.py` with 2 snapshot golden files
+- `test_router.py` unsupported-domain fixture updated from `substack.com` to `wordpress.com`
+
+**New Components**:
+- `src/mdfetch/providers/substack.py` — SubstackExtractor
+- `tests/unit/test_substack_extractor.py` — 18 unit tests
+- `tests/integration/test_substack_integration.py` — 3 integration tests
+- `tests/integration/snapshots/substack-kafka-topic-types.md`
+- `tests/integration/snapshots/substack-api-trends-2025.md`
+
+**Tasks Completed**: 21/21
+
+---
+
 ### mdfetch — Remove Exponential Backoff — 2026-05-15
 
 **Branch**: `004-remove-backoff`

{mdfetch-0.2.2 → mdfetch-0.3.0}/.specify/memory/plan.md

@@ -1,7 +1,7 @@
 # mdfetch — Main Implementation Plan
 
 **Last Updated**: 2026-05-15
-**Sources**: [specs/001-mdfetch-medium-extractor/plan.md], [specs/002-devto-provider/plan.md], [specs/003-medium-freedium-fallback/plan.md], [specs/004-remove-backoff/plan.md]
+**Sources**: [specs/001-mdfetch-medium-extractor/plan.md], [specs/002-devto-provider/plan.md], [specs/003-medium-freedium-fallback/plan.md], [specs/004-remove-backoff/plan.md], [specs/005-substack-provider/plan.md]
 
 ---
 
@@ -63,6 +63,17 @@ DevToExtractor(BaseExtractor)     — src/mdfetch/providers/devto.py
 │                  replaces iframes and ltag embeds with anchor links; strips empty anchor-name
 │                  elements; prepends h1 + cover image from crayons-article__header
 └── convert_to_markdown() → markdownify with ATX headings; collapses 3+ newlines to 2
+
+SubstackExtractor(BaseExtractor)  — src/mdfetch/providers/substack.py  [005-substack-provider]
+├── DOMAINS = frozenset({"substack.com"})  — also matches *.substack.com via suffix routing
+├── _no_retry_status_codes = frozenset()  — HTTP 429 retried (no Freedium-style fallback)
+├── clean_html() → locates div.body.markup; raises UnsupportedContentTypeError if absent;
+│                  strips div.subscription-widget-wrap (inline CTAs + paywall terminal);
+│                  replaces iframes with anchor links using src/data-src;
+│                  replaces div[data-component-name] (except SubscribeWidget, Image2ToDOM) with anchor links;
+│                  prepends h3.subtitle from div.post-header (if present);
+│                  prepends h1.post-title from div.post-header (unconditional — structurally outside body)
+└── convert_to_markdown() → markdownify with ATX headings; collapses 3+ newlines to 2; raises EmptyContentError if empty
 ```
 
 ### Router / Auto-Discovery
@@ -99,7 +110,8 @@ src/
     └── providers/
         ├── __init__.py      # Empty — auto-discovery handles registration
         ├── medium.py        # MediumExtractor
-        └── devto.py         # DevToExtractor [002-devto-provider]
+        ├── devto.py         # DevToExtractor [002-devto-provider]
+        └── substack.py      # SubstackExtractor [005-substack-provider]
 
 tests/
 ├── unit/
@@ -107,17 +119,21 @@ tests/
 │   ├── test_medium_extractor.py
 │   ├── test_fetch_errors.py
 │   ├── test_silent.py
-│   └── test_devto_extractor.py   # [002-devto-provider]
+│   ├── test_devto_extractor.py      # [002-devto-provider]
+│   └── test_substack_extractor.py   # [005-substack-provider]
 └── integration/
     ├── snapshots/           # Golden Markdown files (article body snapshots)
     │   ├── from-drift-to-parity.md
     │   ├── architecting-the-asynchronous-agent.md
     │   ├── integration-digest-december-2025.md
-    │   ├── devto-integration-digest-december-2025.md   # [002-devto-provider]
-    │   ├── devto-integration-digest-july-2025.md       # [002-devto-provider]
-    │   └── devto-integration-digest-march-2026.md      # [002-devto-provider]
+    │   ├── devto-integration-digest-december-2025.md      # [002-devto-provider]
+    │   ├── devto-integration-digest-july-2025.md          # [002-devto-provider]
+    │   ├── devto-integration-digest-march-2026.md         # [002-devto-provider]
+    │   ├── substack-kafka-topic-types.md                  # [005-substack-provider]
+    │   └── substack-api-trends-2025.md                    # [005-substack-provider]
     ├── test_medium_integration.py
-    └── test_devto_integration.py    # [002-devto-provider]
+    ├── test_devto_integration.py        # [002-devto-provider]
+    └── test_substack_integration.py     # [005-substack-provider]
 
 specs/                       # Speckit feature specifications
 pyproject.toml               # hatchling build backend, uv package manager
@@ -140,17 +156,19 @@ Makefile                     # setup / test / integration / lint / typecheck / f
 
 ## Testing Strategy
 
-**Unit tests** (63 tests, offline):
+**Unit tests** (84 tests, offline):
 - Router: domain routing, subdomain suffix matching, duplicate registration, invalid URLs, unsupported platforms
 - MediumExtractor: clean_html, convert_to_markdown, empty content, non-article pages, _parse_freedium (heading remap, missing main-content), fallback on 403/429 (URL construction, exc.url contract, no-sleep on 429), no-fallback on 200, UnsupportedContentTypeError.url on Freedium path [003-medium-freedium-fallback]
 - DevToExtractor: clean_html (title/cover/heading/image preservation, iframe/ltag embed→link, anchor stripping, non-article error), convert_to_markdown (headings/code/lists/images, no raw HTML, empty content error) [002-devto-provider]
+- SubstackExtractor: routing (subdomain + root domain + _no_retry_status_codes assertion), clean_html (body.markup tag return, subscription-widget strip, title prepend, subtitle prepend, prose preservation, iframe→anchor), convert_to_markdown (title heading, no triple blank lines, image syntax, link preservation), paywalled post (non-empty, Subscribe text absent, free preview present), error cases (UnsupportedContentTypeError on no body, EmptyContentError on whitespace body) [005-substack-provider]
 - Fetch errors: HTTP 404, 503, timeout, connection error, size limit exceeded; `_no_retry_status_codes` immediate-raise + `_no_retry_codes` override [003-medium-freedium-fallback]
 - Silent: no stdout/stderr output, no logging during extraction
 
-**Integration tests** (6 tests, network required):
+**Integration tests** (9 tests, network required):
 - Parametrized over 3 real stn1slv.medium.com articles (including a known paywalled URL that exercises the Freedium fallback when medium.com returns 403) [003-medium-freedium-fallback]
 - Parametrized over 3 real dev.to/stn1slv articles [002-devto-provider]
-- Snapshot-based containment check: `expected_body in extracted_result` — tests pass regardless of whether medium.com or Freedium served the content (heading normalisation ensures identical output)
+- Parametrized over 2 real Substack articles + 1 homepage error test (`UnsupportedContentTypeError`) [005-substack-provider]
+- Snapshot-based containment check: `expected_body in extracted_result` — tests pass regardless of which source served the content
 - 3 retries with 2-second **fixed** delay on `FetchError` (hardcoded; not env-var configurable) [004-remove-backoff]
 - Run with: `make integration` or `uv run pytest tests/integration/ --override-ini=addopts=`
 - Excluded from default `pytest` run via `addopts = "-m 'not integration'"` in pyproject.toml
@@ -187,6 +205,11 @@ Makefile                     # setup / test / integration / lint / typecheck / f
 | Routing | `pkgutil.iter_modules` auto-discovery + `@register` | SC-006: one new file = one new platform |
 | Integration tests | Snapshot containment + retry | Durable against minor HTML changes; resilient to transient 403s |
 | test_router.py domain example | Changed from `dev.to` to `substack.com` for "unsupported domain" test | Once DevToExtractor registers `dev.to`, those tests would no longer raise UnsupportedPlatformError | [002-devto-provider]
+| test_router.py domain example | Changed from `substack.com` to `wordpress.com` for "unsupported domain" test | Once SubstackExtractor registers `substack.com`, those tests would no longer raise UnsupportedPlatformError | [005-substack-provider]
+| Substack article targeting | `div.body.markup` as extraction root | Contains article prose only; `div.available-content` is a transparent wrapper; `div.post-footer` and `div.visibility-check` are sibling elements never encountered when using body as root | [005-substack-provider]
+| Substack title prepend | Unconditional prepend of `h1.post-title` from `div.post-header` | Structurally guaranteed outside `div.body.markup`; section headings use distinct class `header-anchor-post` — no deduplication needed | [005-substack-provider]
+| Substack subtitle | Prepend `h3.subtitle` after title (inserted at index 0 first, then title at index 0 displaces it to index 1) | Author intent preserved; subtitle rendered as `###` heading | [005-substack-provider]
+| Substack HTTP 429 | No `_no_retry_status_codes` override — base class `frozenset()` applies | Unlike Medium, Substack has no Freedium-style mirror; retry is the correct fallback | [005-substack-provider]
 | Medium 403/429 fallback | Override `extract()` in `MediumExtractor`; `_no_retry_status_codes=frozenset({403,429})` on class | Immediate fallback with no medium.com retries; `BaseExtractor` extended with `_no_retry_codes` param for thread safety | [003-medium-freedium-fallback]
 | Freedium HTML parsing | Dedicated `_parse_freedium()` method; `div.main-content`; h4→h3 remap | Freedium HTML is structurally incompatible with `clean_html()` (no `<article>`); heading remap ensures snapshot tests pass for both paths | [003-medium-freedium-fallback]
 | Freedium exc.url contract | `inner_exc.url = url` unconditionally; error message is source-agnostic ("Fallback page…") | Preserves transparent-fallback contract (FR-028); `exc.url` is the authoritative field; message content is internal | [003-medium-freedium-fallback]
@@ -199,9 +222,9 @@ Makefile                     # setup / test / integration / lint / typecheck / f
 - [x] Provider Pattern Architecture — `BaseExtractor` ABC with concrete `fetch_html` and abstract `clean_html`, `convert_to_markdown`; `MediumExtractor` inherits
 - [x] Technology Stack — `httpx`, `beautifulsoup4`/`lxml`, `markdownify`, `pytest`, `ruff`; `uv` for all dev workflows
 - [x] Coding Standards — PEP 8, strict type hints, `mypy --strict` passes
-- [x] Integration Testing — real Medium URLs, snapshot-based containment assertions
+- [x] Integration Testing — real Medium, dev.to, and Substack URLs, snapshot-based containment assertions
 - [x] Packaging and Distribution — `pyproject.toml` + `src/` layout + `hatchling`; all Makefile targets use `uv run`
 
 ---
 
-*Last Updated: 2026-05-15 | Sources appended: [specs/004-remove-backoff/plan.md]*
+*Last Updated: 2026-05-15 | Sources appended: [specs/004-remove-backoff/plan.md], [specs/005-substack-provider/plan.md]*

{mdfetch-0.2.2 → mdfetch-0.3.0}/.specify/memory/spec.md

@@ -1,13 +1,13 @@
 # mdfetch — Main Specification
 
 **Last Updated**: 2026-05-15
-**Sources**: [specs/001-mdfetch-medium-extractor/spec.md], [specs/002-devto-provider/spec.md], [specs/003-medium-freedium-fallback/spec.md], [specs/004-remove-backoff/spec.md]
+**Sources**: [specs/001-mdfetch-medium-extractor/spec.md], [specs/002-devto-provider/spec.md], [specs/003-medium-freedium-fallback/spec.md], [specs/004-remove-backoff/spec.md], [specs/005-substack-provider/spec.md]
 
 ---
 
 ## Overview
 
-`mdfetch` is a Python library that extracts article content from web platforms and returns it as clean, well-structured Markdown. The library enforces a provider pattern — an abstract base defines the extraction contract, and each supported platform is implemented as a separate, independent provider. Supported platforms: `medium.com` (and subdomains), `dev.to`.
+`mdfetch` is a Python library that extracts article content from web platforms and returns it as clean, well-structured Markdown. The library enforces a provider pattern — an abstract base defines the extraction contract, and each supported platform is implemented as a separate, independent provider. Supported platforms: `medium.com` (and subdomains), `dev.to`, `substack.com` (and `*.substack.com` subdomains).
 
 ---
 
@@ -117,6 +117,40 @@ A developer calling `extract()` encounters a transient network error. The librar
 
 ---
 
+### US-011 — Extract a Free Substack Article to Markdown (P1)
+[Source: specs/005-substack-provider]
+
+A developer calls `extract()` with a free public `*.substack.com/p/...` URL. The function fetches the article and returns its content as clean Markdown — with no subscription banners, navigation menus, author bios, share buttons, or page chrome.
+
+**Acceptance Scenarios**:
+1. Given a valid URL pointing to a free public Substack post, when `extract()` is called, then it returns a non-empty Markdown string containing the article title as a top-level heading followed by the body content.
+2. Given a Substack post with multiple headings, paragraphs, lists, and inline links, when `extract()` is called, then the returned Markdown preserves all headings, paragraphs, lists, and hyperlinks while stripping subscription CTAs and navigation elements.
+3. Given a Substack post containing images, when `extract()` is called, then images appear in the output as Markdown image syntax (`![alt](url)`).
+
+---
+
+### US-012 — Handle a Paywalled Substack Post Gracefully (P2)
+[Source: specs/005-substack-provider]
+
+A developer calls `extract()` with a URL for a subscriber-only Substack post. The function returns the visible free-preview content as Markdown without raising an error.
+
+**Acceptance Scenarios**:
+1. Given a Substack post that is subscriber-only, when `extract()` is called, then it returns the freely available preview section as Markdown without raising an exception.
+2. Given a paywalled post whose free preview contains at least one paragraph, when `extract()` is called, then the output does not contain the paywall call-to-action text (e.g., "Subscribe to read the full post").
+
+---
+
+### US-013 — Reject Non-Article Substack Pages (P3)
+[Source: specs/005-substack-provider]
+
+A developer accidentally passes a Substack URL that does not point to an article (e.g., a publication homepage). The function raises a typed exception rather than returning empty or garbage Markdown.
+
+**Acceptance Scenarios**:
+1. Given a Substack publication homepage URL, when `extract()` is called, then `UnsupportedContentTypeError` is raised.
+2. Given a Substack post whose extractable text is empty after stripping all chrome, when `extract()` is called, then `EmptyContentError` is raised.
+
+---
+
 ### US-006 — Integration Tests Pass Against Real dev.to Article URLs (P3)
 [Source: specs/002-devto-provider]
 
@@ -164,6 +198,19 @@ A developer runs the integration test suite and all dev.to integration tests pas
 - **FR-027**: The Freedium fallback MUST be unconditionally active for all Medium URL extractions — no caller configuration, opt-in flag, or extractor parameter is required or supported. [Source: specs/003-medium-freedium-fallback]
 - **FR-028**: The Freedium fallback MUST be fully transparent to the caller — no warning, signal, metadata, or result field shall indicate which source (medium.com or Freedium) provided the content. [Source: specs/003-medium-freedium-fallback]
 
+### Substack Platform
+- **FR-030**: The library MUST route all `substack.com` and `*.substack.com` URLs to the Substack provider using the existing domain-registration mechanism. [Source: specs/005-substack-provider]
+- **FR-031**: The library MUST extract the main article body from a Substack post page (`div.body.markup`) and return it as clean Markdown. [Source: specs/005-substack-provider]
+- **FR-032**: The library MUST strip all non-content elements from a Substack article page before conversion, including: navigation headers, subscription call-to-action blocks, paywall nag prompts, social share buttons, author bio sections, comment sections, and page footers. [Source: specs/005-substack-provider]
+- **FR-033**: The library MUST prepend the article title as a top-level Markdown heading (`# Title`) from `h1.post-title` in `div.post-header`. Because Substack's HTML structure always places the post title outside `div.body.markup`, unconditional prepend achieves exactly-once inclusion. [Source: specs/005-substack-provider]
+- **FR-034**: The library MUST preserve the article's structural content: headings (all levels), paragraphs, ordered and unordered lists, inline code, fenced code blocks, blockquotes, hyperlinks, images, and article subtitle (when present as `h3.subtitle` in the post header). [Source: specs/005-substack-provider]
+- **FR-035**: The library MUST raise `UnsupportedContentTypeError` when the fetched Substack page does not contain a recognisable article body element (`div.body.markup`). [Source: specs/005-substack-provider]
+- **FR-036**: The library MUST raise `EmptyContentError` when the Substack article body is present but yields no extractable text after stripping. [Source: specs/005-substack-provider]
+- **FR-037**: For paywalled Substack posts, the library MUST silently extract only the publicly visible free-preview section without raising an exception and without appending any truncation marker, provided the preview contains at least some text. [Source: specs/005-substack-provider]
+- **FR-038**: The library MUST collapse runs of three or more consecutive blank lines to a single blank line in the Substack output Markdown. [Source: specs/005-substack-provider]
+- **FR-039**: The library MUST NOT treat HTTP 429 responses from Substack as a non-retryable condition; 429 MUST be retried up to the configured retry count with the standard fixed delay. [Source: specs/005-substack-provider]
+- **FR-040**: The library MUST convert embedded third-party content in Substack posts (e.g., tweet embeds, YouTube video iframes, and similar rich-media widgets) to plain anchor links using the embed's source URL, matching the pattern used by the dev.to provider. [Source: specs/005-substack-provider]
+
 ### dev.to Platform
 - **FR-015**: The library MUST add `dev.to` to the provider router so that any URL with the `dev.to` domain is dispatched to the dev.to provider without any change to the caller's code. [Source: specs/002-devto-provider]
 - **FR-016**: The library MUST include a dev.to provider that fetches the article page, isolates the main article body from `<div id="article-body">`, removes all non-content elements (navigation, social reaction widgets, comments, author sidebar, tag links), and returns the body as Markdown. [Source: specs/002-devto-provider]
@@ -190,7 +237,26 @@ A developer runs the integration test suite and all dev.to integration tests pas
 |-----------|------|-------------|
 | `DOMAINS` | `frozenset[str]` | Domain suffixes this provider handles (e.g., `{"medium.com"}`, `{"dev.to"}`) |
 
-**Invariants**: Each domain suffix registered to exactly one provider. Stateless — every call is independent. Registered providers: `MediumExtractor` (medium.com), `DevToExtractor` (dev.to).
+**Invariants**: Each domain suffix registered to exactly one provider. Stateless — every call is independent. Registered providers: `MediumExtractor` (medium.com), `DevToExtractor` (dev.to), `SubstackExtractor` (substack.com and all `*.substack.com` subdomains).
+
+### Substack Post
+[Source: specs/005-substack-provider]
+| Attribute | Type | Description |
+|-----------|------|-------------|
+| `url` | `str` | A `*.substack.com/p/<slug>` URL (or equivalent custom-domain path) |
+| `title` | `str` | Article title from `h1.post-title` in `div.post-header` |
+| `subtitle` | `str \| None` | Optional subtitle/deck from `h3.subtitle` in `div.post-header` |
+| `body` | `Tag` | Prose content inside `div.body.markup` |
+
+**Validation**: Page must contain `div.body.markup`; absent → `UnsupportedContentTypeError`. Body must yield non-empty text after stripping → else `EmptyContentError`.
+
+### Free Preview
+[Source: specs/005-substack-provider]
+| Attribute | Type | Description |
+|-----------|------|-------------|
+| `content` | `str` | Portion of a paywalled post publicly readable without a subscription |
+
+**Boundary**: In the DOM, bounded by the last `div.subscription-widget-wrap` at the truncation point. Stripping that element silently achieves truncation.
 
 ### ExtractionResult (Output)
 | Attribute | Type | Description |
@@ -257,6 +323,11 @@ caller provides URL string
 - **dev.to liquid-tag embeds**: Embedded third-party widgets (GitHub Gists, CodePen, YouTube) serialised as `<div class="ltag__*" data-url="...">` are replaced with plain Markdown links; they are never silently dropped.
 - **dev.to cover image**: The cover image lives in `<header class="crayons-article__header">`, not in `div#article-body` — the provider explicitly extracts and prepends it.
 - **dev.to HTML structure changes**: If `div#article-body` is absent, `UnsupportedContentTypeError` is raised.
+- **Substack paywalled posts**: `div.body.markup` already contains only the free-preview content; stripping `div.subscription-widget-wrap` achieves silent truncation (no error, no marker). [Source: specs/005-substack-provider]
+- **Substack homepage URLs**: `div.body.markup` is absent → `UnsupportedContentTypeError` is raised immediately. [Source: specs/005-substack-provider]
+- **Substack rich embeds**: `<iframe>` elements and `div[data-component-name]` containers (excluding `SubscribeWidget` and `Image2ToDOM`) are converted to plain anchor links using the embed's source URL. [Source: specs/005-substack-provider]
+- **Substack HTTP 429**: Treated as a retryable transient error (no `_no_retry_status_codes` override) — contrasts with `MediumExtractor` which uses `frozenset({403, 429})` to trigger Freedium fallback. [Source: specs/005-substack-provider]
+- **Substack HTML structure changes**: If Substack redesigns and removes `div.body.markup`, the extractor will require an update.
 
 ---
 
@@ -275,6 +346,12 @@ caller provides URL string
 - **SC-011**: 100% of integration tests pass against the three provided reference dev.to article URLs at the time of release. [Source: specs/002-devto-provider]
 - **SC-012**: The dev.to provider is delivered as exactly one new file; no existing source files are modified (except `test_router.py` for expected domain-example maintenance when the provider registers `dev.to`). [Source: specs/002-devto-provider]
 
+- **SC-021**: A free public Substack article returns Markdown that contains the full article title and body text with zero subscription prompt phrases (e.g., "Subscribe", "This post is for paid subscribers"). [Source: specs/005-substack-provider]
+- **SC-022**: A paywalled Substack post returns a non-empty Markdown string (the free preview) without raising an exception, provided the free preview contains at least one paragraph. [Source: specs/005-substack-provider]
+- **SC-023**: A Substack homepage URL raises `UnsupportedContentTypeError` within the normal fetch timeout. [Source: specs/005-substack-provider]
+- **SC-024**: The extracted Markdown for any Substack article contains no consecutive blank-line runs of three or more lines. [Source: specs/005-substack-provider]
+- **SC-025**: The Substack provider is exercised by at least one integration test using a real network request, matching the pattern established by existing providers. [Source: specs/005-substack-provider]
+
 ---
 
 ## Assumptions
@@ -297,4 +374,4 @@ caller provides URL string
 
 ---
 
-*Last Updated: 2026-05-15 | Sources appended: [specs/004-remove-backoff/spec.md]*
+*Last Updated: 2026-05-15 | Sources appended: [specs/004-remove-backoff/spec.md], [specs/005-substack-provider/spec.md]*

{mdfetch-0.2.2 → mdfetch-0.3.0}/CLAUDE.md

@@ -13,11 +13,12 @@ src/mdfetch/
 └── providers/
     ├── __init__.py
     ├── medium.py     # MediumExtractor (medium.com + *.medium.com)
-    └── devto.py      # DevToExtractor (dev.to)
+    ├── devto.py      # DevToExtractor (dev.to)
+    └── substack.py   # SubstackExtractor (substack.com + *.substack.com)
 
 tests/
 ├── unit/             # pytest unit tests (no network)
-└── integration/      # real network tests (Medium + dev.to URLs + snapshots)
+└── integration/      # real network tests (Medium + dev.to + Substack URLs + snapshots)
 
 .github/workflows/
 ├── ci.yml            # lint + unit tests on push/PR (Python 3.12–3.14)
@@ -51,7 +52,10 @@ uv run mypy src/  # type check
 ```
 
 <!-- SPECKIT START -->
-## Recent Changes
+## Known Issues & Gotchas
 
-- **004-remove-backoff**: Removed exponential backoff from `fetch_html`; restored fixed-delay retries (`retry_delay` seconds per attempt, constant between attempts). Deleted `MDFETCH_RETRIES`/`MDFETCH_RETRY_DELAY` env-var support from CI and integration fixtures; hardcoded `retries=3, retry_delay=2.0` in all integration tests. Deleted `tests/integration/conftest.py`.
+### ⚠️ test_router.py "unsupported domain" fixture must be updated per new provider
+**Issue:** When a new provider registers a domain (e.g., `dev.to`, then `substack.com`), the existing `test_raises_for_unsupported_domain` and `test_unsupported_error_includes_domain` tests in `tests/unit/test_router.py` use that domain as their "unsupported" example and start routing successfully instead of raising `UnsupportedPlatformError`.
+**Root Cause:** The domain used in the router tests was `substack.com` after the dev.to feature; adding SubstackExtractor made it valid too.
+**Prevention Rule:** When adding a new provider, update the unsupported-domain fixture in `test_router.py` to use a domain not registered by any provider (currently `wordpress.com`).
 <!-- SPECKIT END -->

{mdfetch-0.2.2 → mdfetch-0.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: mdfetch
-Version: 0.2.2
+Version: 0.3.0
 Summary: Extract article content from web platforms and return it as clean Markdown.
 Project-URL: Homepage, https://github.com/stn1slv/md-fetch
 Project-URL: Source, https://github.com/stn1slv/md-fetch
@@ -8,7 +8,7 @@ Project-URL: Issues, https://github.com/stn1slv/md-fetch/issues
 Author-email: Stanislav Deviatov <devyatov@gmail.com>
 License: MIT
 License-File: LICENSE
-Keywords: article,dev.to,extraction,markdown,medium,scraping
+Keywords: article,dev.to,extraction,markdown,medium,scraping,substack
 Classifier: Development Status :: 3 - Alpha
 Classifier: Intended Audience :: Developers
 Classifier: License :: OSI Approved :: MIT License
@@ -47,6 +47,7 @@ from mdfetch import extract
 # Works with any supported platform — just pass the URL
 markdown = extract("https://medium.com/some-publication/article-slug-abc123")
 markdown = extract("https://dev.to/username/article-slug")
+markdown = extract("https://example.substack.com/p/article-slug")
 print(markdown)
 ```
 
@@ -87,6 +88,7 @@ except EmptyContentError as e:
 |----------|---------|
 | Medium   | `medium.com`, `*.medium.com` |
 | dev.to   | `dev.to` |
+| Substack | `substack.com`, `*.substack.com` |
 
 ## Development
 

{mdfetch-0.2.2 → mdfetch-0.3.0}/README.md

@@ -16,6 +16,7 @@ from mdfetch import extract
 # Works with any supported platform — just pass the URL
 markdown = extract("https://medium.com/some-publication/article-slug-abc123")
 markdown = extract("https://dev.to/username/article-slug")
+markdown = extract("https://example.substack.com/p/article-slug")
 print(markdown)
 ```
 
@@ -56,6 +57,7 @@ except EmptyContentError as e:
 |----------|---------|
 | Medium   | `medium.com`, `*.medium.com` |
 | dev.to   | `dev.to` |
+| Substack | `substack.com`, `*.substack.com` |
 
 ## Development
 

{mdfetch-0.2.2 → mdfetch-0.3.0}/pyproject.toml

@@ -4,12 +4,12 @@ build-backend = "hatchling.build"
 
 [project]
 name = "mdfetch"
-version = "0.2.2"
+version = "0.3.0"
 description = "Extract article content from web platforms and return it as clean Markdown."
 readme = "README.md"
 license = { text = "MIT" }
 authors = [{ name = "Stanislav Deviatov", email = "devyatov@gmail.com" }]
-keywords = ["markdown", "scraping", "medium", "dev.to", "article", "extraction"]
+keywords = ["markdown", "scraping", "medium", "dev.to", "substack", "article", "extraction"]
 classifiers = [
     "Development Status :: 3 - Alpha",
     "Intended Audience :: Developers",

mdfetch-0.3.0/specs/005-substack-provider/checklists/requirements.md

@@ -0,0 +1,36 @@
+# Specification Quality Checklist: Substack Platform Provider
+
+**Purpose**: Validate specification completeness and quality before proceeding to planning
+**Created**: 2026-05-15
+**Feature**: [spec.md](../spec.md)
+
+## Content Quality
+
+- [x] No implementation details (languages, frameworks, APIs)
+- [x] Focused on user value and business needs
+- [x] Written for non-technical stakeholders
+- [x] All mandatory sections completed
+
+## Requirement Completeness
+
+- [x] No [NEEDS CLARIFICATION] markers remain
+- [x] Requirements are testable and unambiguous
+- [x] Success criteria are measurable
+- [x] Success criteria are technology-agnostic (no implementation details)
+- [x] All acceptance scenarios are defined
+- [x] Edge cases are identified
+- [x] Scope is clearly bounded
+- [x] Dependencies and assumptions identified
+
+## Feature Readiness
+
+- [x] All functional requirements have clear acceptance criteria
+- [x] User scenarios cover primary flows
+- [x] Feature meets measurable outcomes defined in Success Criteria
+- [x] No implementation details leak into specification
+
+## Notes
+
+- All items pass. Spec is ready for `/speckit-clarify` or `/speckit-plan`.
+- Assumption documented: `netapinotes.com` (Ghost) is out of scope; only `substack.com` / `*.substack.com` are targeted.
+- Custom Substack domains (non-`*.substack.com`) deferred to v2 per Assumptions section.

mdfetch-0.3.0/specs/005-substack-provider/contracts/extractor-api.md

@@ -0,0 +1,59 @@
+# Contract: SubstackExtractor Public API
+
+This document defines the public contract for `SubstackExtractor`, which follows the same interface as all mdfetch providers.
+
+## Registration
+
+```python
+from mdfetch.providers.substack import SubstackExtractor
+
+SubstackExtractor.DOMAINS  # frozenset({"substack.com"})
+```
+
+The `@register` decorator auto-registers the extractor for `substack.com` and all `*.substack.com` subdomains when the module is imported. The router's autodiscovery imports it automatically.
+
+## Method: `extract`
+
+```python
+def extract(self, url: str, *, retries: int = 3, retry_delay: float = 2.0) -> str
+```
+
+**Parameters**
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `url` | `str` | — | A fully qualified `http(s)://` URL on `substack.com` or `*.substack.com` |
+| `retries` | `int` | `3` | Total fetch attempts (including first) on transient errors |
+| `retry_delay` | `float` | `2.0` | Fixed delay in seconds between retries |
+
+**Returns**: `str` — the article content as ATX-style Markdown. Begins with `# Article Title`. Subtitle (if present) follows as `### Subtitle`. Body prose, headings, lists, code blocks, images, and hyperlinks follow.
+
+**Raises**
+
+| Exception | Condition |
+|-----------|-----------|
+| `UnsupportedContentTypeError` | Fetched page lacks `div.body.markup` (not an article) |
+| `EmptyContentError` | Article body is present but yields no extractable text |
+| `HTTPStatusError` | Non-2xx HTTP response after all retries exhausted |
+| `FetchError` | Network error or timeout after all retries exhausted |
+| `InvalidURLError` | URL scheme is not `http`/`https` or hostname is empty (raised by router before extractor is invoked) |
+
+**HTTP 429 behaviour**: treated as a transient error and retried up to `retries` times with `retry_delay` seconds between attempts (no immediate-raise override).
+
+## Top-level `extract()` function
+
+The public entry point in `mdfetch.__init__` routes Substack URLs to this extractor automatically:
+
+```python
+from mdfetch import extract
+
+markdown = extract("https://getkafkanated.substack.com/p/kafka-deserves-topic-types")
+```
+
+## Output Guarantees
+
+- No consecutive blank-line runs of three or more lines.
+- No subscription CTAs, navigation, author bio, or engagement buttons.
+- For paywalled posts: returns only the freely visible preview; no exception; no truncation marker.
+- Images rendered as `![alt text](https://…)`.
+- Embeds (iframes, Substack embed containers) rendered as plain hyperlinks `[url](url)`.

mdfetch-0.3.0/specs/005-substack-provider/data-model.md

@@ -0,0 +1,34 @@
+# Data Model: Substack Platform Provider
+
+This feature introduces no persistent data. The entities below describe in-memory objects used during extraction.
+
+## SubstackPost (in-memory, read-only)
+
+Represents the parsed structure of a single Substack article page as observed in the DOM.
+
+| Field | Source element | Required | Notes |
+|-------|---------------|----------|-------|
+| `title` | `h1.post-title` in `div.post-header` | Yes | Prepended as `# Title` in output |
+| `subtitle` | `h3.subtitle` in `div.post-header` | No | Prepended as `### Subtitle` if present |
+| `body` | `div.body.markup` | Yes | Prose content after stripping chrome |
+
+## DOM Element Taxonomy
+
+| Element | CSS selector | Action |
+|---------|-------------|--------|
+| Article body | `div.body.markup` | **Keep** — extraction root |
+| Post title | `div.post-header h1.post-title` | **Prepend** to body before conversion |
+| Subtitle/deck | `div.post-header h3.subtitle` | **Prepend** (after title) if present |
+| Section headings | `h1.header-anchor-post` inside body | **Keep** at original `h1` level |
+| Captioned images | `div.captioned-image-container` | **Keep** — markdownify handles `img` tags |
+| Inline CTA / paywall boundary | `div.subscription-widget-wrap` | **Strip** |
+| Subscribe widget | `div[data-component-name=SubscribeWidget]` | **Strip** (child of above; covered by parent strip) |
+| Image link wrapper | `a[data-component-name=Image2ToDOM]` | **Keep** — transparent anchor around `img` |
+| iframe embeds | `iframe` | **Replace** with `<a href=src>src</a>` |
+| Other embed containers | `div[data-component-name]` (excluding known safe values) | **Replace** with anchor link |
+| Post footer | `div.post-footer` | Outside `div.body.markup`; not encountered |
+| Byline / author bio | `div.post-header div` (byline wrapper) | Outside extraction root; not encountered |
+
+## State Transitions
+
+None — the extractor is stateless. Each `extract()` call is independent.