mdfetch 0.2.0__tar.gz → 0.2.1__tar.gz

mdfetch-0.2.1/.github/workflows/integration.yml

@@ -0,0 +1,86 @@
+name: Integration Tests
+
+on:
+  schedule:
+    - cron: "30 23 * * 5"
+  workflow_dispatch:
+
+permissions:
+  contents: read
+  issues: write
+
+jobs:
+  integration:
+    runs-on: [self-hosted, Linux, ARM64]
+    timeout-minutes: 30
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@v4
+        with:
+          enable-cache: true
+          cache-dependency-glob: "uv.lock"
+
+      - name: Set up Python
+        run: uv python pin 3.12
+
+      - name: Install dependencies
+        run: uv sync --frozen --all-extras
+
+      - name: Run integration tests
+        id: integration
+        env:
+          MDFETCH_RETRIES: "6"
+          MDFETCH_RETRY_DELAY: "2.0"
+        run: make integration
+
+      - name: Create issue on failure
+        if: failure() && steps.integration.outcome == 'failure'
+        uses: actions/github-script@v7
+        with:
+          script: |
+            const runUrl = `${context.serverUrl}/${context.repo.owner}/${context.repo.repo}/actions/runs/${context.runId}`;
+            const date = new Date().toISOString().slice(0, 10);
+
+            const search = await github.rest.search.issuesAndPullRequests({
+              q: `repo:${context.repo.owner}/${context.repo.repo} is:issue is:open label:integration-failure`,
+              per_page: 1,
+            });
+
+            if (search.data.total_count > 0) {
+              await github.rest.issues.createComment({
+                owner: context.repo.owner,
+                repo: context.repo.repo,
+                issue_number: search.data.items[0].number,
+                body: `Integration tests failed again on ${date}.\n\n**Failed run:** ${runUrl}`,
+              });
+              return;
+            }
+
+            const issueParams = {
+              owner: context.repo.owner,
+              repo: context.repo.repo,
+              title: `Integration tests failed on ${date} — possible HTML structure change`,
+              body: [
+                "## Integration test failure",
+                "",
+                "The scheduled integration tests failed. This usually means a provider's",
+                "upstream HTML structure has changed and the extractor needs updating.",
+                "",
+                `**Failed run:** ${runUrl}`,
+                "",
+                "### Suggested steps",
+                "1. Open the failed run above and inspect the test output.",
+                "2. Identify which provider is broken (Medium, dev.to, …).",
+                "3. Update the relevant extractor in `src/mdfetch/providers/`.",
+                "4. Add or update snapshot fixtures in `tests/integration/` if needed.",
+              ].join("\n"),
+            };
+            try {
+              issueParams.labels = ["bug", "integration-failure"];
+              await github.rest.issues.create(issueParams);
+            } catch {
+              delete issueParams.labels;
+              await github.rest.issues.create(issueParams);
+            }

mdfetch-0.2.1/.specify/feature.json

@@ -0,0 +1,3 @@
+{
+  "feature_directory": "specs/003-medium-freedium-fallback"
+}

{mdfetch-0.2.0 → mdfetch-0.2.1}/.specify/memory/changelog.md

@@ -2,6 +2,28 @@
 
 ---
 
+### mdfetch — Medium Freedium Fallback — 2026-05-15
+
+**Branch**: `003-medium-freedium-fallback`
+**Spec**: specs/003-medium-freedium-fallback
+
+**What was added**:
+- Transparent fallback to `https://freedium-mirror.cfd/` when medium.com returns HTTP 403 (paywall) or HTTP 429 (rate limit) — caller sees no difference in the `extract()` interface
+- `_no_retry_status_codes: frozenset[int]` class attribute on `BaseExtractor`; codes in this set skip retry/backoff and raise immediately (defaults to `frozenset()` — safe for all existing providers)
+- `_no_retry_codes: frozenset[int] | None = None` keyword-only parameter on `fetch_html()` for per-call override without instance mutation (thread-safe)
+- `_parse_freedium(soup)` method on `MediumExtractor`: locates `div.main-content`, remaps h4→h3/h5→h4/h6→h5, converts to Markdown; heading remap ensures output is structurally identical to the direct medium.com path
+- `extract()` override on `MediumExtractor`: on 403/429, fetches `freedium_url` with `_no_retry_codes=frozenset()`, routes to `_parse_freedium()`; always sets `exc.url` to the original Medium URL on failure
+- 18 new unit tests across `test_medium_extractor.py` (TestParseFreedium, TestFreediumFallback, TestRateLimitFallback, TestNoFallbackOnSuccess) and `test_fetch_errors.py`
+- Integration test suite now resilient to medium.com 403 responses — paywalled URL included in snapshot tests
+
+**New Components**:
+- Changes to `src/mdfetch/base.py` — `_no_retry_status_codes` attribute + `_no_retry_codes` param on `fetch_html()`
+- Changes to `src/mdfetch/providers/medium.py` — `_parse_freedium()` + `extract()` override + Freedium constants
+
+**Tasks Completed**: 12/12
+
+---
+
 ### mdfetch — dev.to Extractor — 2026-05-14
 
 **Branch**: `002-devto-provider`

{mdfetch-0.2.0 → mdfetch-0.2.1}/.specify/memory/plan.md

@@ -38,15 +38,23 @@
 
 ```
 BaseExtractor (ABC)               — src/mdfetch/base.py
-├── fetch_html(url) → str         — concrete: streaming HTTP with 30s timeout, 10 MB cap
+├── _no_retry_status_codes: frozenset[int] = frozenset()  — codes that skip retry; overridden by providers
+├── fetch_html(url, *, retries, retry_delay, _no_retry_codes=None) → str
+│                                 — streaming HTTP, 30s timeout, 10 MB cap;
+│                                   codes in _no_retry_codes (or class attribute) raise immediately
 ├── clean_html(soup) → Tag        — abstract: platform-specific HTML isolation
 ├── convert_to_markdown(tag) → str— abstract: platform-specific Markdown conversion
 └── extract(url) → str            — concrete template method (orchestrates the above)
 
 MediumExtractor(BaseExtractor)    — src/mdfetch/providers/medium.py
 ├── DOMAINS = frozenset({"medium.com"})
+├── _FREEDIUM_BASE = "https://freedium-mirror.cfd/"
+├── _no_retry_status_codes = frozenset({403, 429})  — immediate fallback, no medium.com retry
 ├── clean_html() → removes nav, clap buttons, sidebars, share elements, post-footer, author bio
-└── convert_to_markdown() → markdownify with ATX headings, fenced code blocks
+├── convert_to_markdown() → markdownify with ATX headings, fenced code blocks
+├── _parse_freedium(soup) → remaps h4→h3/h5→h4/h6→h5; finds div.main-content; convert_to_markdown
+└── extract() → override: on 403/429 calls fetch_html(freedium_url, _no_retry_codes=frozenset());
+                           exc.url always set to original Medium URL on any Freedium failure
 
 DevToExtractor(BaseExtractor)     — src/mdfetch/providers/devto.py
 ├── DOMAINS = frozenset({"dev.to"})
@@ -131,17 +139,17 @@ Makefile                     # setup / test / integration / lint / typecheck / f
 
 ## Testing Strategy
 
-**Unit tests** (47 tests, offline):
+**Unit tests** (65 tests, offline):
 - Router: domain routing, subdomain suffix matching, duplicate registration, invalid URLs, unsupported platforms
-- MediumExtractor: clean_html, convert_to_markdown, empty content, non-article pages
+- MediumExtractor: clean_html, convert_to_markdown, empty content, non-article pages, _parse_freedium (heading remap, missing main-content), fallback on 403/429 (URL construction, exc.url contract, no-sleep on 429), no-fallback on 200, UnsupportedContentTypeError.url on Freedium path [003-medium-freedium-fallback]
 - DevToExtractor: clean_html (title/cover/heading/image preservation, iframe/ltag embed→link, anchor stripping, non-article error), convert_to_markdown (headings/code/lists/images, no raw HTML, empty content error) [002-devto-provider]
-- Fetch errors: HTTP 404, 503, timeout, connection error, size limit exceeded
+- Fetch errors: HTTP 404, 503, timeout, connection error, size limit exceeded; `_no_retry_status_codes` immediate-raise + `_no_retry_codes` override [003-medium-freedium-fallback]
 - Silent: no stdout/stderr output, no logging during extraction
 
 **Integration tests** (6 tests, network required):
-- Parametrized over 3 real stn1slv.medium.com articles
+- Parametrized over 3 real stn1slv.medium.com articles (including a known paywalled URL that exercises the Freedium fallback when medium.com returns 403) [003-medium-freedium-fallback]
 - Parametrized over 3 real dev.to/stn1slv articles [002-devto-provider]
-- Snapshot-based containment check: `expected_body in extracted_result`
+- Snapshot-based containment check: `expected_body in extracted_result` — tests pass regardless of whether medium.com or Freedium served the content (heading normalisation ensures identical output)
 - 3 retries with 2-second delay on `FetchError` (covers transient 403s/timeouts)
 - Run with: `make integration` or `uv run pytest tests/integration/ --override-ini=addopts=`
 - Excluded from default `pytest` run via `addopts = "-m 'not integration'"` in pyproject.toml
@@ -178,6 +186,9 @@ Makefile                     # setup / test / integration / lint / typecheck / f
 | Routing | `pkgutil.iter_modules` auto-discovery + `@register` | SC-006: one new file = one new platform |
 | Integration tests | Snapshot containment + retry | Durable against minor HTML changes; resilient to transient 403s |
 | test_router.py domain example | Changed from `dev.to` to `substack.com` for "unsupported domain" test | Once DevToExtractor registers `dev.to`, those tests would no longer raise UnsupportedPlatformError | [002-devto-provider]
+| Medium 403/429 fallback | Override `extract()` in `MediumExtractor`; `_no_retry_status_codes=frozenset({403,429})` on class | Immediate fallback with no medium.com retries; `BaseExtractor` extended with `_no_retry_codes` param for thread safety | [003-medium-freedium-fallback]
+| Freedium HTML parsing | Dedicated `_parse_freedium()` method; `div.main-content`; h4→h3 remap | Freedium HTML is structurally incompatible with `clean_html()` (no `<article>`); heading remap ensures snapshot tests pass for both paths | [003-medium-freedium-fallback]
+| Freedium exc.url contract | `inner_exc.url = url` unconditionally; error message is source-agnostic ("Fallback page…") | Preserves transparent-fallback contract (FR-028); `exc.url` is the authoritative field; message content is internal | [003-medium-freedium-fallback]
 
 ---
 
@@ -188,3 +199,7 @@ Makefile                     # setup / test / integration / lint / typecheck / f
 - [x] Coding Standards — PEP 8, strict type hints, `mypy --strict` passes
 - [x] Integration Testing — real Medium URLs, snapshot-based containment assertions
 - [x] Packaging and Distribution — `pyproject.toml` + `src/` layout + `hatchling`; all Makefile targets use `uv run`
+
+---
+
+*Last Updated: 2026-05-15 | Sources appended: [specs/003-medium-freedium-fallback/plan.md]*

{mdfetch-0.2.0 → mdfetch-0.2.1}/.specify/memory/spec.md

@@ -72,6 +72,39 @@ A developer passing a dev.to URL that points to a profile page, a tag listing, o
 
 ---
 
+### US-007 — Transparent Fallback on Blocked Medium Article (P1)
+[Source: specs/003-medium-freedium-fallback]
+
+A developer calls the library's extract function with a Medium URL. The article is behind a paywall or the user is geo-blocked, causing Medium to return a 403 error. Without any code changes, the library automatically retrieves the same article via the Freedium mirror and returns clean Markdown content.
+
+**Acceptance Scenarios**:
+1. Given a valid Medium article URL that returns 403 from medium.com, when `extract()` is called, then the library returns clean Markdown content retrieved via the Freedium mirror.
+2. Given a valid Medium article URL that returns 403 from medium.com, when the Freedium mirror also fails, then the library raises an appropriate extraction error with `exc.url` set to the original Medium URL.
+3. Given a valid Medium article URL that returns 403 from medium.com, when the library falls back to Freedium, then the caller receives the result without any knowledge of which source was used.
+
+---
+
+### US-008 — Automatic Fallback on Rate Limiting (P2)
+[Source: specs/003-medium-freedium-fallback]
+
+A developer calls the library's extract function for a Medium URL. Medium responds with 429 Too Many Requests. The library automatically uses the Freedium mirror as a fallback and returns clean Markdown without requiring the caller to retry.
+
+**Acceptance Scenarios**:
+1. Given a valid Medium article URL that returns 429 from medium.com, when `extract()` is called, then the library returns clean Markdown content retrieved via the Freedium mirror.
+2. Given a valid Medium article URL that returns 429 from medium.com, when the Freedium mirror also fails, then the library raises an appropriate extraction error.
+
+---
+
+### US-009 — No Fallback When Primary Succeeds (P3)
+[Source: specs/003-medium-freedium-fallback]
+
+A developer calls the library's extract function for a publicly accessible Medium article. Medium responds successfully. The library returns the content directly without involving the Freedium mirror, preserving the existing happy-path behavior.
+
+**Acceptance Scenarios**:
+1. Given a valid Medium article URL that returns a successful response, when `extract()` is called, then the library returns clean Markdown content without making any request to the Freedium mirror.
+
+---
+
 ### US-006 — Integration Tests Pass Against Real dev.to Article URLs (P3)
 [Source: specs/002-devto-provider]
 
@@ -107,6 +140,17 @@ A developer runs the integration test suite and all dev.to integration tests pas
 - **FR-009**: The library MUST be packaged and distributed via PyPI using modern Python packaging best practices, enabling installation through the standard package manager without additional steps.
 - **FR-010**: The test suite MUST include integration tests that supply real article URLs (Medium and dev.to) to the extraction function and assert that the output matches expected Markdown structure and content.
 
+### Freedium Fallback (Medium)
+- **FR-020**: When a Medium article extraction results in HTTP 403, the system MUST immediately attempt extraction via the Freedium mirror (`https://freedium-mirror.cfd/{url}`) — no retries against medium.com are made first. [Source: specs/003-medium-freedium-fallback]
+- **FR-021**: When a Medium article extraction results in HTTP 429, the system MUST immediately attempt extraction via the Freedium mirror — no retries against medium.com are made first. [Source: specs/003-medium-freedium-fallback]
+- **FR-022**: When the Freedium mirror is used as a fallback, the system MUST return content in the same clean Markdown format as direct extraction; Freedium's `<h4>` headings are remapped to `<h3>` so both paths produce identical heading-level output. [Source: specs/003-medium-freedium-fallback]
+- **FR-023**: When the primary Medium request succeeds (HTTP 200), the system MUST NOT make any request to the Freedium mirror. [Source: specs/003-medium-freedium-fallback]
+- **FR-024**: When both the primary Medium request and the Freedium fallback fail, the system MUST raise an error consistent with the existing exception hierarchy; `exc.url` MUST be set to the original Medium URL (never the Freedium URL). [Source: specs/003-medium-freedium-fallback]
+- **FR-025**: The Freedium fallback mechanism MUST require no changes to the caller's code — the public `extract()` interface remains unchanged. [Source: specs/003-medium-freedium-fallback]
+- **FR-026**: The Freedium fallback MUST only apply to Medium provider URLs; other providers are unaffected. [Source: specs/003-medium-freedium-fallback]
+- **FR-027**: The Freedium fallback MUST be unconditionally active for all Medium URL extractions — no caller configuration, opt-in flag, or extractor parameter is required or supported. [Source: specs/003-medium-freedium-fallback]
+- **FR-028**: The Freedium fallback MUST be fully transparent to the caller — no warning, signal, metadata, or result field shall indicate which source (medium.com or Freedium) provided the content. [Source: specs/003-medium-freedium-fallback]
+
 ### dev.to Platform
 - **FR-015**: The library MUST add `dev.to` to the provider router so that any URL with the `dev.to` domain is dispatched to the dev.to provider without any change to the caller's code. [Source: specs/002-devto-provider]
 - **FR-016**: The library MUST include a dev.to provider that fetches the article page, isolates the main article body from `<div id="article-body">`, removes all non-content elements (navigation, social reaction widgets, comments, author sidebar, tag links), and returns the body as Markdown. [Source: specs/002-devto-provider]
@@ -185,7 +229,10 @@ caller provides URL string
 
 ## Edge Cases and Error Handling
 
-- **Paywalled content**: If Medium gates content behind a paywall and the body is not in the public HTML, the library may return an `EmptyContentError` or a reduced extraction. Full paywall bypass is out of scope for v1.
+- **Paywalled content (HTTP 403)**: When Medium returns HTTP 403 (paywall or geo-block), the library automatically retries via the Freedium mirror (`https://freedium-mirror.cfd/`). If Freedium also fails, the error raised carries `exc.url` set to the original Medium URL. [Source: specs/003-medium-freedium-fallback]
+- **Freedium mirror unreachable**: If the Freedium mirror returns a network error, timeout, or non-2xx status, the fallback fails and the exception is propagated with `exc.url` set to the original Medium URL — Freedium's URL never appears in `exc.url`.
+- **Freedium HTML structure**: Freedium uses `<div class="main-content">` (no `<article>`); if this element is absent, `UnsupportedContentTypeError` is raised with a source-agnostic message ("Fallback page missing main-content element").
+- **Freedium heading levels**: Freedium renders section headings as `<h4>` vs. medium.com's `<h2>`/`<h3>`; `_parse_freedium()` remaps h4→h3, h5→h4, h6→h5 before conversion so snapshot tests pass regardless of which source served the content.
 - **HTML structure changes**: If Medium changes its HTML structure and `<article>` is absent, `UnsupportedContentTypeError` is raised.
 - **Empty article body**: If `<article>` is found but contains no extractable text, `EmptyContentError` is raised.
 - **Network timeouts**: Covered by `FetchError` (30-second fixed timeout).
@@ -226,3 +273,12 @@ caller provides URL string
 - The library supports Python 3.12 and later.
 - The library operates on publicly accessible HTML; it does not execute JavaScript or render dynamic content.
 - Network timeouts use a fixed default of 30 seconds (not user-configurable in v1).
+- **SC-013**: Articles that previously failed with a 403 paywall error are successfully extracted in at least 90% of cases where the Freedium mirror has the content available. [Source: specs/003-medium-freedium-fallback]
+- **SC-014**: Articles that previously failed with a 429 rate-limit error are successfully extracted via fallback without requiring the caller to retry. [Source: specs/003-medium-freedium-fallback]
+- **SC-015**: Zero changes are required in existing caller code to benefit from the Freedium fallback — existing integrations continue to work as-is. [Source: specs/003-medium-freedium-fallback]
+- **SC-016**: When the primary Medium request succeeds, there is no additional latency attributable to the fallback mechanism. [Source: specs/003-medium-freedium-fallback]
+- **SC-017**: All existing unit and integration tests for the Medium extractor continue to pass without modification after the fallback is introduced. [Source: specs/003-medium-freedium-fallback]
+
+---
+
+*Last Updated: 2026-05-15 | Sources appended: [specs/003-medium-freedium-fallback/spec.md]*

{mdfetch-0.2.0 → mdfetch-0.2.1}/CLAUDE.md

@@ -11,11 +11,18 @@ src/mdfetch/
 ├── base.py           # BaseExtractor ABC
 ├── router.py         # Domain-to-provider routing
 └── providers/
-    └── medium.py     # MediumExtractor (medium.com + *.medium.com)
+    ├── __init__.py
+    ├── medium.py     # MediumExtractor (medium.com + *.medium.com)
+    └── devto.py      # DevToExtractor (dev.to)
 
 tests/
 ├── unit/             # pytest unit tests (no network)
-└── integration/      # pytest -m integration (real Medium URLs)
+└── integration/      # real network tests (Medium + dev.to URLs + snapshots)
+
+.github/workflows/
+├── ci.yml            # lint + unit tests on push/PR (Python 3.12–3.14)
+├── integration.yml   # scheduled integration tests every Friday 23:30 UTC
+└── publish.yml       # PyPI publish on release
 
 specs/                # Speckit feature specifications
 pyproject.toml        # hatchling build, uv package manager
@@ -38,16 +45,11 @@ make test         # unit tests only
 make lint         # ruff check
 make format       # ruff format
 make build        # uv build (wheel + sdist)
-make upgrade-deps # uv sync --upgrade
-uv run pytest -m integration   # integration tests (network required)
-uv run mypy src/               # type check
+make upgrade-deps # uv sync --all-extras --upgrade
+make integration  # integration tests (network required)
+uv run mypy src/  # type check
 ```
 
-## Recent Changes
-
-- 002-devto-provider: dev.to article extraction provider — `DevToExtractor`, cover image + embed→link handling, 17 new unit tests, 3 integration tests, version 0.2.0
-- 001-mdfetch-medium-extractor: Initial library release — `extract()` API, Medium provider, typed exceptions, auto-discovery routing, PyPI packaging, snapshot integration tests
-
 <!-- SPECKIT START -->
-**Active feature plan**: none
+**Recent changes**: `003-medium-freedium-fallback` — Transparent Freedium mirror fallback for Medium 403/429 responses; `_no_retry_status_codes` hook on `BaseExtractor`; `_parse_freedium()` with h4→h3 heading remap on `MediumExtractor`
 <!-- SPECKIT END -->

{mdfetch-0.2.0 → mdfetch-0.2.1}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: mdfetch
-Version: 0.2.0
+Version: 0.2.1
 Summary: Extract article content from web platforms and return it as clean Markdown.
 Project-URL: Homepage, https://github.com/stn1slv/md-fetch
 Project-URL: Source, https://github.com/stn1slv/md-fetch

{mdfetch-0.2.0 → mdfetch-0.2.1}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "mdfetch"
-version = "0.2.0"
+version = "0.2.1"
 description = "Extract article content from web platforms and return it as clean Markdown."
 readme = "README.md"
 license = { text = "MIT" }

mdfetch-0.2.1/specs/003-medium-freedium-fallback/checklists/requirements.md

@@ -0,0 +1,34 @@
+# Specification Quality Checklist: Medium Freedium Fallback
+
+**Purpose**: Validate specification completeness and quality before proceeding to planning
+**Created**: 2026-05-14
+**Feature**: [spec.md](../spec.md)
+
+## Content Quality
+
+- [x] No implementation details (languages, frameworks, APIs)
+- [x] Focused on user value and business needs
+- [x] Written for non-technical stakeholders
+- [x] All mandatory sections completed
+
+## Requirement Completeness
+
+- [x] No [NEEDS CLARIFICATION] markers remain
+- [x] Requirements are testable and unambiguous
+- [x] Success criteria are measurable
+- [x] Success criteria are technology-agnostic (no implementation details)
+- [x] All acceptance scenarios are defined
+- [x] Edge cases are identified
+- [x] Scope is clearly bounded
+- [x] Dependencies and assumptions identified
+
+## Feature Readiness
+
+- [x] All functional requirements have clear acceptance criteria
+- [x] User scenarios cover primary flows
+- [x] Feature meets measurable outcomes defined in Success Criteria
+- [x] No implementation details leak into specification
+
+## Notes
+
+- All items pass. Specification is ready for `/speckit-plan`.

mdfetch-0.2.1/specs/003-medium-freedium-fallback/contracts/extract-api.md

@@ -0,0 +1,41 @@
+# Contract: `extract()` Public API
+
+**Feature**: Medium Freedium Fallback | **Status**: Unchanged
+
+## Signature
+
+```python
+def extract(url: str, *, retries: int = 3, retry_delay: float = 2.0) -> str
+```
+
+## Behaviour (unchanged)
+
+| Input | Output |
+|---|---|
+| Valid Medium URL, article accessible | Clean Markdown string |
+| Valid Medium URL, 403 or 429 from medium.com, Freedium succeeds | Clean Markdown string (source is transparent) |
+| Valid Medium URL, 403 or 429 from medium.com, Freedium also fails | Raises `FetchError` or `HTTPStatusError` |
+| Valid Medium URL, other HTTP error (404, 500, …) | Raises `HTTPStatusError` after retries |
+| Valid Medium URL, network timeout/error | Raises `FetchError` after retries |
+| Non-Medium URL | Raises `UnsupportedPlatformError` |
+| Invalid URL | Raises `InvalidURLError` |
+
+## Exception Hierarchy (unchanged)
+
+```
+MdfetchError
+├── InvalidURLError
+├── UnsupportedPlatformError
+├── UnsupportedContentTypeError
+├── EmptyContentError
+└── FetchError
+    └── HTTPStatusError  (status_code: int, url: str | None)
+```
+
+## Guarantees introduced by this feature
+
+- When fallback is invoked, `exc.url` on any raised exception contains the **original Medium URL**, not the Freedium mirror URL.
+- No new exception types are introduced.
+- No new parameters are added to `extract()`.
+- The Freedium mirror URL is never exposed in return values or the `exc.url` field.
+- `exc.url` is the authoritative public URL field and is always set to the original Medium URL when fallback fails. `exc.message` is an internal implementation detail and may contain transport-level information (e.g. the URL that was actually fetched); callers MUST NOT rely on its contents.

mdfetch-0.2.1/specs/003-medium-freedium-fallback/plan.md

@@ -0,0 +1,187 @@
+# Implementation Plan: Medium Freedium Fallback
+
+**Branch**: `003-medium-freedium-fallback` | **Date**: 2026-05-14 | **Spec**: [spec.md](spec.md)
+
+**Input**: Feature specification from `specs/003-medium-freedium-fallback/spec.md`
+
+## Summary
+
+Add transparent fallback to the Freedium mirror (`https://freedium-mirror.cfd/`) within `MediumExtractor` when `medium.com` returns HTTP 403 or 429. On these status codes the fallback is immediate (no retries against `medium.com` first). The public `extract()` interface is unchanged. The implementation requires a minimal extension to `BaseExtractor` (a `_no_retry_status_codes` class attribute) and an override of `extract()` in `MediumExtractor`.
+
+## Technical Context
+
+**Language/Version**: Python 3.12–3.14
+
+**Primary Dependencies**: httpx ≥0.27, beautifulsoup4 ≥4.12, lxml ≥5.0, markdownify ≥0.13 — all existing; no new runtime dependencies
+
+**Storage**: N/A
+
+**Testing**: pytest with `@pytest.mark.integration` marker for network tests
+
+**Target Platform**: Any (Python library distributed via PyPI)
+
+**Project Type**: library
+
+**Performance Goals**: Zero added latency on successful primary fetch; fallback adds exactly one additional HTTP roundtrip
+
+**Constraints**: Public `extract(url, *, retries, retry_delay)` signature unchanged; no new runtime dependencies; all existing tests must pass unmodified
+
+**Scale/Scope**: Per-call behaviour change within `MediumExtractor`; no state, no storage. Thread-safe by design: `_no_retry_codes` override is passed as a parameter to `fetch_html()` rather than mutating instance state.
+
+## Constitution Check
+
+*GATE: Must pass before Phase 0 research. Re-check after Phase 1 design.*
+
+- [x] Validates Provider Pattern Architecture — changes confined to `MediumExtractor` plus one minimal hook in `BaseExtractor`; no code duplication
+- [x] Confirms Technology Stack — httpx (existing), BeautifulSoup (existing); no new deps
+- [x] Adheres to Coding Standards — PEP 8, strict type hints, clear naming required throughout
+- [x] Incorporates Integration Testing — new `@pytest.mark.integration` test for paywalled URL via Freedium
+- [x] Respects Packaging and Distribution standards — `pyproject.toml` + `src/` layout unchanged; all Makefile targets delegate to `uv run`
+
+**Post-design re-check**: All gates remain green. No violations requiring justification.
+
+## Project Structure
+
+### Documentation (this feature)
+
+```text
+specs/003-medium-freedium-fallback/
+├── plan.md              # This file
+├── research.md          # Phase 0 output
+├── contracts/
+│   └── extract-api.md   # Phase 1 output — public extract() contract
+└── tasks.md             # Phase 2 output (/speckit-tasks command)
+```
+
+### Source Code (repository root)
+
+No new files. Changes to existing files only:
+
+```text
+src/mdfetch/
+├── base.py                    # Add _no_retry_status_codes + loop guard
+└── providers/
+    └── medium.py              # Add _parse_freedium(); override _no_retry_status_codes + extract()
+
+tests/
+├── unit/
+│   ├── test_fetch_errors.py   # Add tests: 403/429 not retried for MediumExtractor
+│   └── test_medium_extractor.py  # Add tests: _parse_freedium(), fallback URL + 403/429 paths
+└── integration/
+    └── test_medium_integration.py  # Add test: paywalled URL succeeds via Freedium (keyword assertion)
+```
+
+**Structure Decision**: Single project layout unchanged; all changes are surgical modifications to existing files.
+
+## Phase 0: Research
+
+See [research.md](research.md) for full rationale. Key decisions:
+
+| Topic | Decision |
+|---|---|
+| Preventing retries on 403/429 | Add `_no_retry_status_codes: frozenset[int] = frozenset()` to `BaseExtractor`; override with `frozenset({403, 429})` in `MediumExtractor` |
+| Fallback implementation location | Override `extract()` in `MediumExtractor` |
+| Freedium URL construction | `f"https://freedium-mirror.cfd/{original_url}"` |
+| HTML parsing | Freedium uses `<div class="main-content">` — **incompatible** with `clean_html()` which requires `<article>`; add dedicated `_parse_freedium(soup)` method to `MediumExtractor` |
+| Error when both fail | Raise `HTTPStatusError`/`FetchError` from Freedium attempt; set `exc.url` to original Medium URL |
+| Integration test | Snapshot containment assertion (`expected in result`) — paywalled URL covered by existing snapshot test suite; dedicated keyword-only test removed as redundant |
+
+## Phase 1: Design & Contracts
+
+### BaseExtractor changes (`src/mdfetch/base.py`)
+
+Add class attribute:
+```python
+_no_retry_status_codes: frozenset[int] = frozenset()
+```
+
+Also add a `_no_retry_codes` keyword-only parameter to `fetch_html()` so callers can pass a per-call override without mutating instance state (used for the Freedium fetch):
+
+```python
+def fetch_html(
+    self,
+    url: str,
+    *,
+    retries: int = 3,
+    retry_delay: float = 2.0,
+    _no_retry_codes: frozenset[int] | None = None,
+) -> str: ...
+```
+
+In the retry loop, resolve the effective set before the loop:
+```python
+no_retry = self._no_retry_status_codes if _no_retry_codes is None else _no_retry_codes
+# ...
+except FetchError as exc:
+    if isinstance(exc, HTTPStatusError) and exc.status_code in no_retry:
+        raise
+    last_exc = exc
+    if attempt < retries - 1:
+        time.sleep(...)
+```
+
+These are the only changes to `base.py`. The attribute is empty by default, so all existing providers and tests are unaffected.
+
+### MediumExtractor changes (`src/mdfetch/providers/medium.py`)
+
+**Verified finding**: Freedium HTML uses `<div class="main-content">` as the content root — no `<article>` element exists. The existing `clean_html()` always raises `UnsupportedContentTypeError` on Freedium HTML. A dedicated parser method is required.
+
+Add two class-level constants, a new `_parse_freedium()` method, and override `extract()`:
+
+```python
+_FREEDIUM_BASE = "https://freedium-mirror.cfd/"
+_no_retry_status_codes: frozenset[int] = frozenset({403, 429})
+
+def _parse_freedium(self, soup: BeautifulSoup) -> str:
+    """Parse Freedium mirror HTML, which uses div.main-content instead of <article>."""
+    content = soup.find("div", class_="main-content")
+    if not isinstance(content, Tag):
+        raise UnsupportedContentTypeError(
+            "Fallback page missing main-content element",
+        )
+    # Remap h4→h3, h5→h4, h6→h5 so output matches medium.com's heading levels
+    for level in (4, 5, 6):
+        for tag in list(content.find_all(f"h{level}")):
+            tag.name = f"h{level - 1}"
+    return self.convert_to_markdown(content)
+
+def extract(self, url: str, *, retries: int = 3, retry_delay: float = 2.0) -> str:
+    """Extract article, falling back to Freedium mirror on HTTP 403 or 429."""
+    try:
+        return super().extract(url, retries=retries, retry_delay=retry_delay)
+    except HTTPStatusError as exc:
+        if exc.status_code not in self._no_retry_status_codes:
+            raise
+    freedium_url = f"{self._FREEDIUM_BASE}{url}"
+    # Pass _no_retry_codes=frozenset() so a 429 from Freedium is retried
+    # with backoff rather than raised immediately (thread-safe: no mutation).
+    try:
+        html = self.fetch_html(
+            freedium_url, retries=retries, retry_delay=retry_delay, _no_retry_codes=frozenset()
+        )
+        soup = BeautifulSoup(html, "lxml")
+        return self._parse_freedium(soup)
+    except MdfetchError as inner_exc:
+        inner_exc.url = url
+        raise
+```
+
+Notes:
+- `HTTPStatusError` and `MdfetchError` imported in `medium.py`
+- `_parse_freedium()` skips all Medium-specific stripping — Freedium's `main-content` div contains only article body
+- Error message in `UnsupportedContentTypeError` is source-agnostic ("Fallback page…") to preserve transparent-fallback contract (FR-009)
+- `exc.url` is set unconditionally to the original URL (not guarded by `is None`) — ensures the Freedium URL never escapes to callers regardless of which exception type is raised
+- `_no_retry_codes=frozenset()` on the Freedium fetch prevents class-level 429 no-retry behavior from suppressing retries against the mirror (thread-safe: no instance mutation)
+
+### Data Model
+
+No new entities. This feature is a pure behaviour change with no persistent state.
+
+### Contracts
+
+See [contracts/extract-api.md](contracts/extract-api.md). The public `extract()` signature is unchanged. The Freedium URL is a fully internal implementation detail — never exposed through any public interface or exception message visible to callers.
+
+### Revision: Implementation Sync 2026-05-15
+- Code snippets updated to reflect shipped implementation: `_no_retry_codes` parameter on `fetch_html()`, heading remap in `_parse_freedium()`, unconditional `exc.url` assignment, source-agnostic error message ("Fallback page…"), thread-safe Freedium fetch via parameter override.
+- Integration test strategy updated: keyword-only paywalled test removed; paywalled URL is covered by snapshot containment test.
+- Scale/Scope updated: concurrency safety now explicitly documented.