mdfetch 0.2.1__tar.gz → 0.3.0__tar.gz

{mdfetch-0.2.1 → mdfetch-0.3.0}/.github/workflows/integration.yml

@@ -30,9 +30,6 @@ jobs:
 
       - name: Run integration tests
         id: integration
-        env:
-          MDFETCH_RETRIES: "6"
-          MDFETCH_RETRY_DELAY: "2.0"
         run: make integration
 
       - name: Create issue on failure

mdfetch-0.3.0/.specify/feature.json

@@ -0,0 +1,3 @@
+{
+  "feature_directory": "specs/005-substack-provider"
+}

{mdfetch-0.2.1 → mdfetch-0.3.0}/.specify/memory/changelog.md

@@ -2,6 +2,49 @@
 
 ---
 
+### mdfetch — Substack Provider — 2026-05-15
+
+**Branch**: `005-substack-provider`
+**Spec**: specs/005-substack-provider
+
+**What was added**:
+- `SubstackExtractor` provider for `substack.com` and all `*.substack.com` subdomain articles, auto-discovered via `@register` decorator
+- Article body isolation from `div.body.markup`; title prepended from `h1.post-title` in `div.post-header` (unconditional — structurally outside body); optional subtitle from `h3.subtitle` prepended after title
+- `div.subscription-widget-wrap` stripped (inline subscribe CTAs and paywall terminal widget — achieves silent free-preview truncation for paywalled posts without error or marker)
+- `<iframe>` and unknown `div[data-component-name]` embed containers converted to plain anchor links (consistent with dev.to embed handling)
+- HTTP 429 retried via base-class fixed-delay retry (no `_no_retry_status_codes` override — no Freedium-style mirror for Substack)
+- `UnsupportedContentTypeError` raised when `div.body.markup` is absent; `EmptyContentError` raised when body yields no extractable text
+- 18 unit tests in `tests/unit/test_substack_extractor.py`; 3 integration tests in `tests/integration/test_substack_integration.py` with 2 snapshot golden files
+- `test_router.py` unsupported-domain fixture updated from `substack.com` to `wordpress.com`
+
+**New Components**:
+- `src/mdfetch/providers/substack.py` — SubstackExtractor
+- `tests/unit/test_substack_extractor.py` — 18 unit tests
+- `tests/integration/test_substack_integration.py` — 3 integration tests
+- `tests/integration/snapshots/substack-kafka-topic-types.md`
+- `tests/integration/snapshots/substack-api-trends-2025.md`
+
+**Tasks Completed**: 21/21
+
+---
+
+### mdfetch — Remove Exponential Backoff — 2026-05-15
+
+**Branch**: `004-remove-backoff`
+**Spec**: specs/004-remove-backoff
+
+**What was added**:
+- Fixed-delay retry behaviour: `fetch_html` now sleeps exactly `retry_delay` seconds between attempts (previously exponential `retry_delay × 2ⁿ`, capped at 60 s). The change makes retry timing predictable for callers and aligns with the Freedium fallback that already absorbs 403/429 errors.
+- Removed `MDFETCH_RETRIES` and `MDFETCH_RETRY_DELAY` env-var support from integration test fixtures (`conftest.py` deleted) and CI workflow (`integration.yml`). Integration tests now hardcode `retries=3, retry_delay=2.0`.
+- Deleted two unit tests (`test_exponential_backoff_sleep_sequence`, `test_exponential_backoff_capped_at_max_delay`) that verified the removed exponential schedule. Added sleep-value assertion to `test_status_code_not_in_no_retry_set_still_retries` to close the FR-029 gap.
+
+**New Components**:
+- No new files. Pure removal: `tests/integration/conftest.py` deleted; `_MAX_RETRY_DELAY` constant removed from `src/mdfetch/base.py`.
+
+**Tasks Completed**: 16/16
+
+---
+
 ### mdfetch — Medium Freedium Fallback — 2026-05-15
 
 **Branch**: `003-medium-freedium-fallback`

{mdfetch-0.2.1 → mdfetch-0.3.0}/.specify/memory/plan.md

@@ -1,7 +1,7 @@
 # mdfetch — Main Implementation Plan
 
-**Last Updated**: 2026-05-14
-**Sources**: [specs/001-mdfetch-medium-extractor/plan.md], [specs/002-devto-provider/plan.md]
+**Last Updated**: 2026-05-15
+**Sources**: [specs/001-mdfetch-medium-extractor/plan.md], [specs/002-devto-provider/plan.md], [specs/003-medium-freedium-fallback/plan.md], [specs/004-remove-backoff/plan.md], [specs/005-substack-provider/plan.md]
 
 ---
 
@@ -41,6 +41,7 @@ BaseExtractor (ABC)               — src/mdfetch/base.py
 ├── _no_retry_status_codes: frozenset[int] = frozenset()  — codes that skip retry; overridden by providers
 ├── fetch_html(url, *, retries, retry_delay, _no_retry_codes=None) → str
 │                                 — streaming HTTP, 30s timeout, 10 MB cap;
+│                                   fixed delay of retry_delay seconds between attempts (not exponential);
 │                                   codes in _no_retry_codes (or class attribute) raise immediately
 ├── clean_html(soup) → Tag        — abstract: platform-specific HTML isolation
 ├── convert_to_markdown(tag) → str— abstract: platform-specific Markdown conversion
@@ -62,6 +63,17 @@ DevToExtractor(BaseExtractor)     — src/mdfetch/providers/devto.py
 │                  replaces iframes and ltag embeds with anchor links; strips empty anchor-name
 │                  elements; prepends h1 + cover image from crayons-article__header
 └── convert_to_markdown() → markdownify with ATX headings; collapses 3+ newlines to 2
+
+SubstackExtractor(BaseExtractor)  — src/mdfetch/providers/substack.py  [005-substack-provider]
+├── DOMAINS = frozenset({"substack.com"})  — also matches *.substack.com via suffix routing
+├── _no_retry_status_codes = frozenset()  — HTTP 429 retried (no Freedium-style fallback)
+├── clean_html() → locates div.body.markup; raises UnsupportedContentTypeError if absent;
+│                  strips div.subscription-widget-wrap (inline CTAs + paywall terminal);
+│                  replaces iframes with anchor links using src/data-src;
+│                  replaces div[data-component-name] (except SubscribeWidget, Image2ToDOM) with anchor links;
+│                  prepends h3.subtitle from div.post-header (if present);
+│                  prepends h1.post-title from div.post-header (unconditional — structurally outside body)
+└── convert_to_markdown() → markdownify with ATX headings; collapses 3+ newlines to 2; raises EmptyContentError if empty
 ```
 
 ### Router / Auto-Discovery
@@ -98,7 +110,8 @@ src/
     └── providers/
         ├── __init__.py      # Empty — auto-discovery handles registration
         ├── medium.py        # MediumExtractor
-        └── devto.py         # DevToExtractor [002-devto-provider]
+        ├── devto.py         # DevToExtractor [002-devto-provider]
+        └── substack.py      # SubstackExtractor [005-substack-provider]
 
 tests/
 ├── unit/
@@ -106,17 +119,21 @@ tests/
 │   ├── test_medium_extractor.py
 │   ├── test_fetch_errors.py
 │   ├── test_silent.py
-│   └── test_devto_extractor.py   # [002-devto-provider]
+│   ├── test_devto_extractor.py      # [002-devto-provider]
+│   └── test_substack_extractor.py   # [005-substack-provider]
 └── integration/
     ├── snapshots/           # Golden Markdown files (article body snapshots)
     │   ├── from-drift-to-parity.md
     │   ├── architecting-the-asynchronous-agent.md
     │   ├── integration-digest-december-2025.md
-    │   ├── devto-integration-digest-december-2025.md   # [002-devto-provider]
-    │   ├── devto-integration-digest-july-2025.md       # [002-devto-provider]
-    │   └── devto-integration-digest-march-2026.md      # [002-devto-provider]
+    │   ├── devto-integration-digest-december-2025.md      # [002-devto-provider]
+    │   ├── devto-integration-digest-july-2025.md          # [002-devto-provider]
+    │   ├── devto-integration-digest-march-2026.md         # [002-devto-provider]
+    │   ├── substack-kafka-topic-types.md                  # [005-substack-provider]
+    │   └── substack-api-trends-2025.md                    # [005-substack-provider]
     ├── test_medium_integration.py
-    └── test_devto_integration.py    # [002-devto-provider]
+    ├── test_devto_integration.py        # [002-devto-provider]
+    └── test_substack_integration.py     # [005-substack-provider]
 
 specs/                       # Speckit feature specifications
 pyproject.toml               # hatchling build backend, uv package manager
@@ -139,18 +156,20 @@ Makefile                     # setup / test / integration / lint / typecheck / f
 
 ## Testing Strategy
 
-**Unit tests** (65 tests, offline):
+**Unit tests** (84 tests, offline):
 - Router: domain routing, subdomain suffix matching, duplicate registration, invalid URLs, unsupported platforms
 - MediumExtractor: clean_html, convert_to_markdown, empty content, non-article pages, _parse_freedium (heading remap, missing main-content), fallback on 403/429 (URL construction, exc.url contract, no-sleep on 429), no-fallback on 200, UnsupportedContentTypeError.url on Freedium path [003-medium-freedium-fallback]
 - DevToExtractor: clean_html (title/cover/heading/image preservation, iframe/ltag embed→link, anchor stripping, non-article error), convert_to_markdown (headings/code/lists/images, no raw HTML, empty content error) [002-devto-provider]
+- SubstackExtractor: routing (subdomain + root domain + _no_retry_status_codes assertion), clean_html (body.markup tag return, subscription-widget strip, title prepend, subtitle prepend, prose preservation, iframe→anchor), convert_to_markdown (title heading, no triple blank lines, image syntax, link preservation), paywalled post (non-empty, Subscribe text absent, free preview present), error cases (UnsupportedContentTypeError on no body, EmptyContentError on whitespace body) [005-substack-provider]
 - Fetch errors: HTTP 404, 503, timeout, connection error, size limit exceeded; `_no_retry_status_codes` immediate-raise + `_no_retry_codes` override [003-medium-freedium-fallback]
 - Silent: no stdout/stderr output, no logging during extraction
 
-**Integration tests** (6 tests, network required):
+**Integration tests** (9 tests, network required):
 - Parametrized over 3 real stn1slv.medium.com articles (including a known paywalled URL that exercises the Freedium fallback when medium.com returns 403) [003-medium-freedium-fallback]
 - Parametrized over 3 real dev.to/stn1slv articles [002-devto-provider]
-- Snapshot-based containment check: `expected_body in extracted_result` — tests pass regardless of whether medium.com or Freedium served the content (heading normalisation ensures identical output)
-- 3 retries with 2-second delay on `FetchError` (covers transient 403s/timeouts)
+- Parametrized over 2 real Substack articles + 1 homepage error test (`UnsupportedContentTypeError`) [005-substack-provider]
+- Snapshot-based containment check: `expected_body in extracted_result` — tests pass regardless of which source served the content
+- 3 retries with 2-second **fixed** delay on `FetchError` (hardcoded; not env-var configurable) [004-remove-backoff]
 - Run with: `make integration` or `uv run pytest tests/integration/ --override-ini=addopts=`
 - Excluded from default `pytest` run via `addopts = "-m 'not integration'"` in pyproject.toml
 
@@ -186,9 +205,15 @@ Makefile                     # setup / test / integration / lint / typecheck / f
 | Routing | `pkgutil.iter_modules` auto-discovery + `@register` | SC-006: one new file = one new platform |
 | Integration tests | Snapshot containment + retry | Durable against minor HTML changes; resilient to transient 403s |
 | test_router.py domain example | Changed from `dev.to` to `substack.com` for "unsupported domain" test | Once DevToExtractor registers `dev.to`, those tests would no longer raise UnsupportedPlatformError | [002-devto-provider]
+| test_router.py domain example | Changed from `substack.com` to `wordpress.com` for "unsupported domain" test | Once SubstackExtractor registers `substack.com`, those tests would no longer raise UnsupportedPlatformError | [005-substack-provider]
+| Substack article targeting | `div.body.markup` as extraction root | Contains article prose only; `div.available-content` is a transparent wrapper; `div.post-footer` and `div.visibility-check` are sibling elements never encountered when using body as root | [005-substack-provider]
+| Substack title prepend | Unconditional prepend of `h1.post-title` from `div.post-header` | Structurally guaranteed outside `div.body.markup`; section headings use distinct class `header-anchor-post` — no deduplication needed | [005-substack-provider]
+| Substack subtitle | Prepend `h3.subtitle` after title (inserted at index 0 first, then title at index 0 displaces it to index 1) | Author intent preserved; subtitle rendered as `###` heading | [005-substack-provider]
+| Substack HTTP 429 | No `_no_retry_status_codes` override — base class `frozenset()` applies | Unlike Medium, Substack has no Freedium-style mirror; retry is the correct fallback | [005-substack-provider]
 | Medium 403/429 fallback | Override `extract()` in `MediumExtractor`; `_no_retry_status_codes=frozenset({403,429})` on class | Immediate fallback with no medium.com retries; `BaseExtractor` extended with `_no_retry_codes` param for thread safety | [003-medium-freedium-fallback]
 | Freedium HTML parsing | Dedicated `_parse_freedium()` method; `div.main-content`; h4→h3 remap | Freedium HTML is structurally incompatible with `clean_html()` (no `<article>`); heading remap ensures snapshot tests pass for both paths | [003-medium-freedium-fallback]
 | Freedium exc.url contract | `inner_exc.url = url` unconditionally; error message is source-agnostic ("Fallback page…") | Preserves transparent-fallback contract (FR-028); `exc.url` is the authoritative field; message content is internal | [003-medium-freedium-fallback]
+| Retry strategy | Fixed delay (`retry_delay` seconds per attempt, unchanged between attempts) | Exponential backoff removed (PR #8); Freedium fallback absorbs 403/429 at a higher level making exponential growth unnecessary; integration tests use hardcoded defaults (`retries=3, retry_delay=2.0`) with no env-var override | [004-remove-backoff]
 
 ---
 
@@ -197,9 +222,9 @@ Makefile                     # setup / test / integration / lint / typecheck / f
 - [x] Provider Pattern Architecture — `BaseExtractor` ABC with concrete `fetch_html` and abstract `clean_html`, `convert_to_markdown`; `MediumExtractor` inherits
 - [x] Technology Stack — `httpx`, `beautifulsoup4`/`lxml`, `markdownify`, `pytest`, `ruff`; `uv` for all dev workflows
 - [x] Coding Standards — PEP 8, strict type hints, `mypy --strict` passes
-- [x] Integration Testing — real Medium URLs, snapshot-based containment assertions
+- [x] Integration Testing — real Medium, dev.to, and Substack URLs, snapshot-based containment assertions
 - [x] Packaging and Distribution — `pyproject.toml` + `src/` layout + `hatchling`; all Makefile targets use `uv run`
 
 ---
 
-*Last Updated: 2026-05-15 | Sources appended: [specs/003-medium-freedium-fallback/plan.md]*
+*Last Updated: 2026-05-15 | Sources appended: [specs/004-remove-backoff/plan.md], [specs/005-substack-provider/plan.md]*

{mdfetch-0.2.1 → mdfetch-0.3.0}/.specify/memory/spec.md

@@ -1,13 +1,13 @@
 # mdfetch — Main Specification
 
-**Last Updated**: 2026-05-14
-**Sources**: [specs/001-mdfetch-medium-extractor/spec.md], [specs/002-devto-provider/spec.md]
+**Last Updated**: 2026-05-15
+**Sources**: [specs/001-mdfetch-medium-extractor/spec.md], [specs/002-devto-provider/spec.md], [specs/003-medium-freedium-fallback/spec.md], [specs/004-remove-backoff/spec.md], [specs/005-substack-provider/spec.md]
 
 ---
 
 ## Overview
 
-`mdfetch` is a Python library that extracts article content from web platforms and returns it as clean, well-structured Markdown. The library enforces a provider pattern — an abstract base defines the extraction contract, and each supported platform is implemented as a separate, independent provider. Supported platforms: `medium.com` (and subdomains), `dev.to`.
+`mdfetch` is a Python library that extracts article content from web platforms and returns it as clean, well-structured Markdown. The library enforces a provider pattern — an abstract base defines the extraction contract, and each supported platform is implemented as a separate, independent provider. Supported platforms: `medium.com` (and subdomains), `dev.to`, `substack.com` (and `*.substack.com` subdomains).
 
 ---
 
@@ -105,6 +105,52 @@ A developer calls the library's extract function for a publicly accessible Mediu
 
 ---
 
+### US-010 — Predictable Fixed-Delay Retry Behaviour (P1)
+[Source: specs/004-remove-backoff]
+
+A developer calling `extract()` encounters a transient network error. The library retries using a simple, predictable fixed delay — exactly `retry_delay` seconds between every attempt — rather than an exponentially growing delay. The retry timing is constant and easy to reason about regardless of which attempt number is being made.
+
+**Acceptance Scenarios**:
+1. Given `fetch_html` is called with `retries=3` and `retry_delay=2.0`, when the first two attempts raise a transient error, then the library sleeps exactly `2.0` seconds before each retry (not `2.0` then `4.0`).
+2. Given `fetch_html` is called with `retries=1`, when the attempt fails, then no sleep occurs and the exception is raised immediately.
+3. Given a status code in `_no_retry_status_codes`, when that error is raised, then no sleep or retry occurs.
+
+---
+
+### US-011 — Extract a Free Substack Article to Markdown (P1)
+[Source: specs/005-substack-provider]
+
+A developer calls `extract()` with a free public `*.substack.com/p/...` URL. The function fetches the article and returns its content as clean Markdown — with no subscription banners, navigation menus, author bios, share buttons, or page chrome.
+
+**Acceptance Scenarios**:
+1. Given a valid URL pointing to a free public Substack post, when `extract()` is called, then it returns a non-empty Markdown string containing the article title as a top-level heading followed by the body content.
+2. Given a Substack post with multiple headings, paragraphs, lists, and inline links, when `extract()` is called, then the returned Markdown preserves all headings, paragraphs, lists, and hyperlinks while stripping subscription CTAs and navigation elements.
+3. Given a Substack post containing images, when `extract()` is called, then images appear in the output as Markdown image syntax (`![alt](url)`).
+
+---
+
+### US-012 — Handle a Paywalled Substack Post Gracefully (P2)
+[Source: specs/005-substack-provider]
+
+A developer calls `extract()` with a URL for a subscriber-only Substack post. The function returns the visible free-preview content as Markdown without raising an error.
+
+**Acceptance Scenarios**:
+1. Given a Substack post that is subscriber-only, when `extract()` is called, then it returns the freely available preview section as Markdown without raising an exception.
+2. Given a paywalled post whose free preview contains at least one paragraph, when `extract()` is called, then the output does not contain the paywall call-to-action text (e.g., "Subscribe to read the full post").
+
+---
+
+### US-013 — Reject Non-Article Substack Pages (P3)
+[Source: specs/005-substack-provider]
+
+A developer accidentally passes a Substack URL that does not point to an article (e.g., a publication homepage). The function raises a typed exception rather than returning empty or garbage Markdown.
+
+**Acceptance Scenarios**:
+1. Given a Substack publication homepage URL, when `extract()` is called, then `UnsupportedContentTypeError` is raised.
+2. Given a Substack post whose extractable text is empty after stripping all chrome, when `extract()` is called, then `EmptyContentError` is raised.
+
+---
+
 ### US-006 — Integration Tests Pass Against Real dev.to Article URLs (P3)
 [Source: specs/002-devto-provider]
 
@@ -135,6 +181,7 @@ A developer runs the integration test suite and all dev.to integration tests pas
 ### Network
 - **FR-006**: The library MUST raise a descriptive error when a network request fails (connection error, timeout, non-2xx HTTP status).
 - **FR-014**: HTTP requests MUST use a standard browser-like User-Agent string so that web servers return readable HTML. The User-Agent MUST NOT identify the library by name or version. The library does not check or respect `robots.txt` in v1.
+- **FR-029**: The `fetch_html` method MUST use a fixed delay of exactly `retry_delay` seconds between retry attempts — no exponential multiplication. The retry delay is constant and does not grow with each attempt number. [Source: specs/004-remove-backoff]
 
 ### Packaging & Testing
 - **FR-009**: The library MUST be packaged and distributed via PyPI using modern Python packaging best practices, enabling installation through the standard package manager without additional steps.
@@ -151,6 +198,19 @@ A developer runs the integration test suite and all dev.to integration tests pas
 - **FR-027**: The Freedium fallback MUST be unconditionally active for all Medium URL extractions — no caller configuration, opt-in flag, or extractor parameter is required or supported. [Source: specs/003-medium-freedium-fallback]
 - **FR-028**: The Freedium fallback MUST be fully transparent to the caller — no warning, signal, metadata, or result field shall indicate which source (medium.com or Freedium) provided the content. [Source: specs/003-medium-freedium-fallback]
 
+### Substack Platform
+- **FR-030**: The library MUST route all `substack.com` and `*.substack.com` URLs to the Substack provider using the existing domain-registration mechanism. [Source: specs/005-substack-provider]
+- **FR-031**: The library MUST extract the main article body from a Substack post page (`div.body.markup`) and return it as clean Markdown. [Source: specs/005-substack-provider]
+- **FR-032**: The library MUST strip all non-content elements from a Substack article page before conversion, including: navigation headers, subscription call-to-action blocks, paywall nag prompts, social share buttons, author bio sections, comment sections, and page footers. [Source: specs/005-substack-provider]
+- **FR-033**: The library MUST prepend the article title as a top-level Markdown heading (`# Title`) from `h1.post-title` in `div.post-header`. Because Substack's HTML structure always places the post title outside `div.body.markup`, unconditional prepend achieves exactly-once inclusion. [Source: specs/005-substack-provider]
+- **FR-034**: The library MUST preserve the article's structural content: headings (all levels), paragraphs, ordered and unordered lists, inline code, fenced code blocks, blockquotes, hyperlinks, images, and article subtitle (when present as `h3.subtitle` in the post header). [Source: specs/005-substack-provider]
+- **FR-035**: The library MUST raise `UnsupportedContentTypeError` when the fetched Substack page does not contain a recognisable article body element (`div.body.markup`). [Source: specs/005-substack-provider]
+- **FR-036**: The library MUST raise `EmptyContentError` when the Substack article body is present but yields no extractable text after stripping. [Source: specs/005-substack-provider]
+- **FR-037**: For paywalled Substack posts, the library MUST silently extract only the publicly visible free-preview section without raising an exception and without appending any truncation marker, provided the preview contains at least some text. [Source: specs/005-substack-provider]
+- **FR-038**: The library MUST collapse runs of three or more consecutive blank lines to a single blank line in the Substack output Markdown. [Source: specs/005-substack-provider]
+- **FR-039**: The library MUST NOT treat HTTP 429 responses from Substack as a non-retryable condition; 429 MUST be retried up to the configured retry count with the standard fixed delay. [Source: specs/005-substack-provider]
+- **FR-040**: The library MUST convert embedded third-party content in Substack posts (e.g., tweet embeds, YouTube video iframes, and similar rich-media widgets) to plain anchor links using the embed's source URL, matching the pattern used by the dev.to provider. [Source: specs/005-substack-provider]
+
 ### dev.to Platform
 - **FR-015**: The library MUST add `dev.to` to the provider router so that any URL with the `dev.to` domain is dispatched to the dev.to provider without any change to the caller's code. [Source: specs/002-devto-provider]
 - **FR-016**: The library MUST include a dev.to provider that fetches the article page, isolates the main article body from `<div id="article-body">`, removes all non-content elements (navigation, social reaction widgets, comments, author sidebar, tag links), and returns the body as Markdown. [Source: specs/002-devto-provider]
@@ -177,7 +237,26 @@ A developer runs the integration test suite and all dev.to integration tests pas
 |-----------|------|-------------|
 | `DOMAINS` | `frozenset[str]` | Domain suffixes this provider handles (e.g., `{"medium.com"}`, `{"dev.to"}`) |
 
-**Invariants**: Each domain suffix registered to exactly one provider. Stateless — every call is independent. Registered providers: `MediumExtractor` (medium.com), `DevToExtractor` (dev.to).
+**Invariants**: Each domain suffix registered to exactly one provider. Stateless — every call is independent. Registered providers: `MediumExtractor` (medium.com), `DevToExtractor` (dev.to), `SubstackExtractor` (substack.com and all `*.substack.com` subdomains).
+
+### Substack Post
+[Source: specs/005-substack-provider]
+| Attribute | Type | Description |
+|-----------|------|-------------|
+| `url` | `str` | A `*.substack.com/p/<slug>` URL (or equivalent custom-domain path) |
+| `title` | `str` | Article title from `h1.post-title` in `div.post-header` |
+| `subtitle` | `str \| None` | Optional subtitle/deck from `h3.subtitle` in `div.post-header` |
+| `body` | `Tag` | Prose content inside `div.body.markup` |
+
+**Validation**: Page must contain `div.body.markup`; absent → `UnsupportedContentTypeError`. Body must yield non-empty text after stripping → else `EmptyContentError`.
+
+### Free Preview
+[Source: specs/005-substack-provider]
+| Attribute | Type | Description |
+|-----------|------|-------------|
+| `content` | `str` | Portion of a paywalled post publicly readable without a subscription |
+
+**Boundary**: In the DOM, bounded by the last `div.subscription-widget-wrap` at the truncation point. Stripping that element silently achieves truncation.
 
 ### ExtractionResult (Output)
 | Attribute | Type | Description |
@@ -238,12 +317,17 @@ caller provides URL string
 - **Network timeouts**: Covered by `FetchError` (30-second fixed timeout).
 - **Oversized responses**: Responses exceeding 10 MB are rejected with `FetchError` to prevent OOM.
 - **Profile/tag pages**: When a `medium.com` URL points to a non-article page, `UnsupportedContentTypeError` is raised (distinct from `UnsupportedPlatformError`).
-- **HTTP 403 / transient failures**: Integration tests use a 3-retry helper with 2-second delay to handle transient rate limits.
+- **HTTP 403 / transient failures**: Integration tests use 3 retries with a 2-second fixed delay (hardcoded; not env-var configurable) to handle transient rate limits. [Source: specs/004-remove-backoff]
 - **dev.to profile pages**: When a `dev.to` URL points to an author profile (no `div#article-body`), `UnsupportedContentTypeError` is raised.
 - **dev.to tag listing pages**: When a `dev.to` URL points to a tag page (e.g., `dev.to/t/kafka`), `UnsupportedContentTypeError` is raised.
 - **dev.to liquid-tag embeds**: Embedded third-party widgets (GitHub Gists, CodePen, YouTube) serialised as `<div class="ltag__*" data-url="...">` are replaced with plain Markdown links; they are never silently dropped.
 - **dev.to cover image**: The cover image lives in `<header class="crayons-article__header">`, not in `div#article-body` — the provider explicitly extracts and prepends it.
 - **dev.to HTML structure changes**: If `div#article-body` is absent, `UnsupportedContentTypeError` is raised.
+- **Substack paywalled posts**: `div.body.markup` already contains only the free-preview content; stripping `div.subscription-widget-wrap` achieves silent truncation (no error, no marker). [Source: specs/005-substack-provider]
+- **Substack homepage URLs**: `div.body.markup` is absent → `UnsupportedContentTypeError` is raised immediately. [Source: specs/005-substack-provider]
+- **Substack rich embeds**: `<iframe>` elements and `div[data-component-name]` containers (excluding `SubscribeWidget` and `Image2ToDOM`) are converted to plain anchor links using the embed's source URL. [Source: specs/005-substack-provider]
+- **Substack HTTP 429**: Treated as a retryable transient error (no `_no_retry_status_codes` override) — contrasts with `MediumExtractor` which uses `frozenset({403, 429})` to trigger Freedium fallback. [Source: specs/005-substack-provider]
+- **Substack HTML structure changes**: If Substack redesigns and removes `div.body.markup`, the extractor will require an update.
 
 ---
 
@@ -262,6 +346,12 @@ caller provides URL string
 - **SC-011**: 100% of integration tests pass against the three provided reference dev.to article URLs at the time of release. [Source: specs/002-devto-provider]
 - **SC-012**: The dev.to provider is delivered as exactly one new file; no existing source files are modified (except `test_router.py` for expected domain-example maintenance when the provider registers `dev.to`). [Source: specs/002-devto-provider]
 
+- **SC-021**: A free public Substack article returns Markdown that contains the full article title and body text with zero subscription prompt phrases (e.g., "Subscribe", "This post is for paid subscribers"). [Source: specs/005-substack-provider]
+- **SC-022**: A paywalled Substack post returns a non-empty Markdown string (the free preview) without raising an exception, provided the free preview contains at least one paragraph. [Source: specs/005-substack-provider]
+- **SC-023**: A Substack homepage URL raises `UnsupportedContentTypeError` within the normal fetch timeout. [Source: specs/005-substack-provider]
+- **SC-024**: The extracted Markdown for any Substack article contains no consecutive blank-line runs of three or more lines. [Source: specs/005-substack-provider]
+- **SC-025**: The Substack provider is exercised by at least one integration test using a real network request, matching the pattern established by existing providers. [Source: specs/005-substack-provider]
+
 ---
 
 ## Assumptions
@@ -273,6 +363,9 @@ caller provides URL string
 - The library supports Python 3.12 and later.
 - The library operates on publicly accessible HTML; it does not execute JavaScript or render dynamic content.
 - Network timeouts use a fixed default of 30 seconds (not user-configurable in v1).
+- **SC-018**: `make test` passes with zero failures (all unit tests green) when run without any `MDFETCH_*` environment variables. [Source: specs/004-remove-backoff]
+- **SC-019**: `make integration` passes with zero failures when run without any `MDFETCH_RETRIES` or `MDFETCH_RETRY_DELAY` environment variables — integration tests use hardcoded defaults (3 retries, 2.0 s delay). [Source: specs/004-remove-backoff]
+- **SC-020**: No reference to `MDFETCH_RETRIES`, `MDFETCH_RETRY_DELAY`, or `_MAX_RETRY_DELAY` appears in `src/`, `tests/`, or `.github/` (specification and documentation files excluded). [Source: specs/004-remove-backoff]
 - **SC-013**: Articles that previously failed with a 403 paywall error are successfully extracted in at least 90% of cases where the Freedium mirror has the content available. [Source: specs/003-medium-freedium-fallback]
 - **SC-014**: Articles that previously failed with a 429 rate-limit error are successfully extracted via fallback without requiring the caller to retry. [Source: specs/003-medium-freedium-fallback]
 - **SC-015**: Zero changes are required in existing caller code to benefit from the Freedium fallback — existing integrations continue to work as-is. [Source: specs/003-medium-freedium-fallback]
@@ -281,4 +374,4 @@ caller provides URL string
 
 ---
 
-*Last Updated: 2026-05-15 | Sources appended: [specs/003-medium-freedium-fallback/spec.md]*
+*Last Updated: 2026-05-15 | Sources appended: [specs/004-remove-backoff/spec.md], [specs/005-substack-provider/spec.md]*

{mdfetch-0.2.1 → mdfetch-0.3.0}/CLAUDE.md

@@ -13,11 +13,12 @@ src/mdfetch/
 └── providers/
     ├── __init__.py
     ├── medium.py     # MediumExtractor (medium.com + *.medium.com)
-    └── devto.py      # DevToExtractor (dev.to)
+    ├── devto.py      # DevToExtractor (dev.to)
+    └── substack.py   # SubstackExtractor (substack.com + *.substack.com)
 
 tests/
 ├── unit/             # pytest unit tests (no network)
-└── integration/      # real network tests (Medium + dev.to URLs + snapshots)
+└── integration/      # real network tests (Medium + dev.to + Substack URLs + snapshots)
 
 .github/workflows/
 ├── ci.yml            # lint + unit tests on push/PR (Python 3.12–3.14)
@@ -51,5 +52,10 @@ uv run mypy src/  # type check
 ```
 
 <!-- SPECKIT START -->
-**Recent changes**: `003-medium-freedium-fallback` — Transparent Freedium mirror fallback for Medium 403/429 responses; `_no_retry_status_codes` hook on `BaseExtractor`; `_parse_freedium()` with h4→h3 heading remap on `MediumExtractor`
+## Known Issues & Gotchas
+
+### ⚠️ test_router.py "unsupported domain" fixture must be updated per new provider
+**Issue:** When a new provider registers a domain (e.g., `dev.to`, then `substack.com`), the existing `test_raises_for_unsupported_domain` and `test_unsupported_error_includes_domain` tests in `tests/unit/test_router.py` use that domain as their "unsupported" example and start routing successfully instead of raising `UnsupportedPlatformError`.
+**Root Cause:** The domain used in the router tests was `substack.com` after the dev.to feature; adding SubstackExtractor made it valid too.
+**Prevention Rule:** When adding a new provider, update the unsupported-domain fixture in `test_router.py` to use a domain not registered by any provider (currently `wordpress.com`).
 <!-- SPECKIT END -->

{mdfetch-0.2.1 → mdfetch-0.3.0}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: mdfetch
-Version: 0.2.1
+Version: 0.3.0
 Summary: Extract article content from web platforms and return it as clean Markdown.
 Project-URL: Homepage, https://github.com/stn1slv/md-fetch
 Project-URL: Source, https://github.com/stn1slv/md-fetch
@@ -8,7 +8,7 @@ Project-URL: Issues, https://github.com/stn1slv/md-fetch/issues
 Author-email: Stanislav Deviatov <devyatov@gmail.com>
 License: MIT
 License-File: LICENSE
-Keywords: article,dev.to,extraction,markdown,medium,scraping
+Keywords: article,dev.to,extraction,markdown,medium,scraping,substack
 Classifier: Development Status :: 3 - Alpha
 Classifier: Intended Audience :: Developers
 Classifier: License :: OSI Approved :: MIT License
@@ -47,6 +47,7 @@ from mdfetch import extract
 # Works with any supported platform — just pass the URL
 markdown = extract("https://medium.com/some-publication/article-slug-abc123")
 markdown = extract("https://dev.to/username/article-slug")
+markdown = extract("https://example.substack.com/p/article-slug")
 print(markdown)
 ```
 
@@ -87,6 +88,7 @@ except EmptyContentError as e:
 |----------|---------|
 | Medium   | `medium.com`, `*.medium.com` |
 | dev.to   | `dev.to` |
+| Substack | `substack.com`, `*.substack.com` |
 
 ## Development
 

{mdfetch-0.2.1 → mdfetch-0.3.0}/README.md

@@ -16,6 +16,7 @@ from mdfetch import extract
 # Works with any supported platform — just pass the URL
 markdown = extract("https://medium.com/some-publication/article-slug-abc123")
 markdown = extract("https://dev.to/username/article-slug")
+markdown = extract("https://example.substack.com/p/article-slug")
 print(markdown)
 ```
 
@@ -56,6 +57,7 @@ except EmptyContentError as e:
 |----------|---------|
 | Medium   | `medium.com`, `*.medium.com` |
 | dev.to   | `dev.to` |
+| Substack | `substack.com`, `*.substack.com` |
 
 ## Development
 

{mdfetch-0.2.1 → mdfetch-0.3.0}/pyproject.toml

@@ -4,12 +4,12 @@ build-backend = "hatchling.build"
 
 [project]
 name = "mdfetch"
-version = "0.2.1"
+version = "0.3.0"
 description = "Extract article content from web platforms and return it as clean Markdown."
 readme = "README.md"
 license = { text = "MIT" }
 authors = [{ name = "Stanislav Deviatov", email = "devyatov@gmail.com" }]
-keywords = ["markdown", "scraping", "medium", "dev.to", "article", "extraction"]
+keywords = ["markdown", "scraping", "medium", "dev.to", "substack", "article", "extraction"]
 classifiers = [
     "Development Status :: 3 - Alpha",
     "Intended Audience :: Developers",

mdfetch-0.3.0/specs/004-remove-backoff/checklists/requirements.md

@@ -0,0 +1,34 @@
+# Specification Quality Checklist: Remove Exponential Backoff and Env-Var Retry Config
+
+**Purpose**: Validate specification completeness and quality before proceeding to planning
+**Created**: 2026-05-15
+**Feature**: [spec.md](../spec.md)
+
+## Content Quality
+
+- [X] No implementation details (languages, frameworks, APIs)
+- [X] Focused on user value and business needs
+- [X] Written for non-technical stakeholders
+- [X] All mandatory sections completed
+
+## Requirement Completeness
+
+- [X] No [NEEDS CLARIFICATION] markers remain
+- [X] Requirements are testable and unambiguous
+- [X] Success criteria are measurable
+- [X] Success criteria are technology-agnostic (no implementation details)
+- [X] All acceptance scenarios are defined
+- [X] Edge cases are identified
+- [X] Scope is clearly bounded
+- [X] Dependencies and assumptions identified
+
+## Feature Readiness
+
+- [X] All functional requirements have clear acceptance criteria
+- [X] User scenarios cover primary flows
+- [X] Feature meets measurable outcomes defined in Success Criteria
+- [X] No implementation details leak into specification
+
+## Notes
+
+- All checklist items pass. Specification is ready for `/speckit-clarify` or `/speckit-plan`.