mdfetch 0.2.0__tar.gz → 0.2.2__tar.gz

mdfetch-0.2.2/.github/workflows/integration.yml

@@ -0,0 +1,83 @@
+name: Integration Tests
+
+on:
+  schedule:
+    - cron: "30 23 * * 5"
+  workflow_dispatch:
+
+permissions:
+  contents: read
+  issues: write
+
+jobs:
+  integration:
+    runs-on: [self-hosted, Linux, ARM64]
+    timeout-minutes: 30
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Install uv
+        uses: astral-sh/setup-uv@v4
+        with:
+          enable-cache: true
+          cache-dependency-glob: "uv.lock"
+
+      - name: Set up Python
+        run: uv python pin 3.12
+
+      - name: Install dependencies
+        run: uv sync --frozen --all-extras
+
+      - name: Run integration tests
+        id: integration
+        run: make integration
+
+      - name: Create issue on failure
+        if: failure() && steps.integration.outcome == 'failure'
+        uses: actions/github-script@v7
+        with:
+          script: |
+            const runUrl = `${context.serverUrl}/${context.repo.owner}/${context.repo.repo}/actions/runs/${context.runId}`;
+            const date = new Date().toISOString().slice(0, 10);
+
+            const search = await github.rest.search.issuesAndPullRequests({
+              q: `repo:${context.repo.owner}/${context.repo.repo} is:issue is:open label:integration-failure`,
+              per_page: 1,
+            });
+
+            if (search.data.total_count > 0) {
+              await github.rest.issues.createComment({
+                owner: context.repo.owner,
+                repo: context.repo.repo,
+                issue_number: search.data.items[0].number,
+                body: `Integration tests failed again on ${date}.\n\n**Failed run:** ${runUrl}`,
+              });
+              return;
+            }
+
+            const issueParams = {
+              owner: context.repo.owner,
+              repo: context.repo.repo,
+              title: `Integration tests failed on ${date} — possible HTML structure change`,
+              body: [
+                "## Integration test failure",
+                "",
+                "The scheduled integration tests failed. This usually means a provider's",
+                "upstream HTML structure has changed and the extractor needs updating.",
+                "",
+                `**Failed run:** ${runUrl}`,
+                "",
+                "### Suggested steps",
+                "1. Open the failed run above and inspect the test output.",
+                "2. Identify which provider is broken (Medium, dev.to, …).",
+                "3. Update the relevant extractor in `src/mdfetch/providers/`.",
+                "4. Add or update snapshot fixtures in `tests/integration/` if needed.",
+              ].join("\n"),
+            };
+            try {
+              issueParams.labels = ["bug", "integration-failure"];
+              await github.rest.issues.create(issueParams);
+            } catch {
+              delete issueParams.labels;
+              await github.rest.issues.create(issueParams);
+            }

mdfetch-0.2.2/.specify/feature.json

@@ -0,0 +1,3 @@
+{
+  "feature_directory": "specs/004-remove-backoff"
+}

mdfetch-0.2.2/.specify/memory/changelog.md

@@ -0,0 +1,96 @@
+# Merged Features Log
+
+---
+
+### mdfetch — Remove Exponential Backoff — 2026-05-15
+
+**Branch**: `004-remove-backoff`
+**Spec**: specs/004-remove-backoff
+
+**What was added**:
+- Fixed-delay retry behaviour: `fetch_html` now sleeps exactly `retry_delay` seconds between attempts (previously exponential `retry_delay × 2ⁿ`, capped at 60 s). The change makes retry timing predictable for callers and aligns with the Freedium fallback that already absorbs 403/429 errors.
+- Removed `MDFETCH_RETRIES` and `MDFETCH_RETRY_DELAY` env-var support from integration test fixtures (`conftest.py` deleted) and CI workflow (`integration.yml`). Integration tests now hardcode `retries=3, retry_delay=2.0`.
+- Deleted two unit tests (`test_exponential_backoff_sleep_sequence`, `test_exponential_backoff_capped_at_max_delay`) that verified the removed exponential schedule. Added sleep-value assertion to `test_status_code_not_in_no_retry_set_still_retries` to close the FR-029 gap.
+
+**New Components**:
+- No new files. Pure removal: `tests/integration/conftest.py` deleted; `_MAX_RETRY_DELAY` constant removed from `src/mdfetch/base.py`.
+
+**Tasks Completed**: 16/16
+
+---
+
+### mdfetch — Medium Freedium Fallback — 2026-05-15
+
+**Branch**: `003-medium-freedium-fallback`
+**Spec**: specs/003-medium-freedium-fallback
+
+**What was added**:
+- Transparent fallback to `https://freedium-mirror.cfd/` when medium.com returns HTTP 403 (paywall) or HTTP 429 (rate limit) — caller sees no difference in the `extract()` interface
+- `_no_retry_status_codes: frozenset[int]` class attribute on `BaseExtractor`; codes in this set skip retry/backoff and raise immediately (defaults to `frozenset()` — safe for all existing providers)
+- `_no_retry_codes: frozenset[int] | None = None` keyword-only parameter on `fetch_html()` for per-call override without instance mutation (thread-safe)
+- `_parse_freedium(soup)` method on `MediumExtractor`: locates `div.main-content`, remaps h4→h3/h5→h4/h6→h5, converts to Markdown; heading remap ensures output is structurally identical to the direct medium.com path
+- `extract()` override on `MediumExtractor`: on 403/429, fetches `freedium_url` with `_no_retry_codes=frozenset()`, routes to `_parse_freedium()`; always sets `exc.url` to the original Medium URL on failure
+- 18 new unit tests across `test_medium_extractor.py` (TestParseFreedium, TestFreediumFallback, TestRateLimitFallback, TestNoFallbackOnSuccess) and `test_fetch_errors.py`
+- Integration test suite now resilient to medium.com 403 responses — paywalled URL included in snapshot tests
+
+**New Components**:
+- Changes to `src/mdfetch/base.py` — `_no_retry_status_codes` attribute + `_no_retry_codes` param on `fetch_html()`
+- Changes to `src/mdfetch/providers/medium.py` — `_parse_freedium()` + `extract()` override + Freedium constants
+
+**Tasks Completed**: 12/12
+
+---
+
+### mdfetch — dev.to Extractor — 2026-05-14
+
+**Branch**: `002-devto-provider`
+**Spec**: specs/002-devto-provider
+
+**What was added**:
+- `DevToExtractor` provider for `dev.to` articles, auto-discovered via `@register` decorator
+- Article body isolation from `<div id="article-body">` with cover image extracted from `<header class="crayons-article__header">` and prepended to output
+- `<iframe>` and liquid-tag embed (`ltag__*`) replacement with plain Markdown links (FR-019)
+- `UnsupportedContentTypeError` raised for non-article dev.to pages (profiles, tag listings)
+- 17 new unit tests in `tests/unit/test_devto_extractor.py`
+- 3 dev.to integration tests in `tests/integration/test_devto_integration.py` with snapshot golden files
+- Library version bumped from `0.1.0` to `0.2.0`
+- `"dev.to"` added to `pyproject.toml` keywords (T013)
+
+**New Components**:
+- `src/mdfetch/providers/devto.py` — DevToExtractor
+- `tests/unit/test_devto_extractor.py` — 17 unit tests
+- `tests/integration/test_devto_integration.py` — 3 integration tests
+- `tests/integration/snapshots/devto-integration-digest-december-2025.md`
+- `tests/integration/snapshots/devto-integration-digest-july-2025.md`
+- `tests/integration/snapshots/devto-integration-digest-march-2026.md`
+
+**Tasks Completed**: 13/13
+
+---
+
+### mdfetch — Medium Extractor (Initial Release) — 2026-05-14
+
+**Branch**: `feature/first-draft`
+**Spec**: specs/001-mdfetch-medium-extractor
+
+**What was added**:
+- `extract(url: str) -> str` public API for converting Medium articles to Markdown
+- Provider pattern with `BaseExtractor` ABC and `MediumExtractor` implementation
+- Auto-discovery routing via `pkgutil.iter_modules` + `@register` decorator (SC-006 compliant)
+- Full typed exception hierarchy: `MdfetchError` → `InvalidURLError`, `UnsupportedPlatformError`, `UnsupportedContentTypeError`, `FetchError` → `HTTPStatusError`, `EmptyContentError`
+- Streaming HTTP fetch with 10 MB response size cap
+- Browser-like User-Agent (FR-014 compliant, no library branding)
+- Snapshot-based integration tests with retry logic (3 retries, 2-second delay)
+- PyPI-ready package: `pyproject.toml` + `src/` layout + `hatchling` build backend
+- `Makefile` with full dev workflow (setup, test, integration, lint, typecheck, format, build, upgrade-deps, clean)
+
+**New Components**:
+- `src/mdfetch/__init__.py` — public surface
+- `src/mdfetch/exceptions.py` — typed exception hierarchy
+- `src/mdfetch/base.py` — BaseExtractor ABC + fetch_html + extract template method
+- `src/mdfetch/router.py` — @register, auto-discovery, route()
+- `src/mdfetch/providers/medium.py` — MediumExtractor
+- `tests/unit/` — 30 unit tests (offline)
+- `tests/integration/` — 3 integration tests with 3 snapshot golden files
+
+**Tasks Completed**: 32/32

{mdfetch-0.2.0 → mdfetch-0.2.2}/.specify/memory/plan.md

@@ -1,7 +1,7 @@
 # mdfetch — Main Implementation Plan
 
-**Last Updated**: 2026-05-14
-**Sources**: [specs/001-mdfetch-medium-extractor/plan.md], [specs/002-devto-provider/plan.md]
+**Last Updated**: 2026-05-15
+**Sources**: [specs/001-mdfetch-medium-extractor/plan.md], [specs/002-devto-provider/plan.md], [specs/003-medium-freedium-fallback/plan.md], [specs/004-remove-backoff/plan.md]
 
 ---
 
@@ -38,15 +38,24 @@
 
 ```
 BaseExtractor (ABC)               — src/mdfetch/base.py
-├── fetch_html(url) → str         — concrete: streaming HTTP with 30s timeout, 10 MB cap
+├── _no_retry_status_codes: frozenset[int] = frozenset()  — codes that skip retry; overridden by providers
+├── fetch_html(url, *, retries, retry_delay, _no_retry_codes=None) → str
+│                                 — streaming HTTP, 30s timeout, 10 MB cap;
+│                                   fixed delay of retry_delay seconds between attempts (not exponential);
+│                                   codes in _no_retry_codes (or class attribute) raise immediately
 ├── clean_html(soup) → Tag        — abstract: platform-specific HTML isolation
 ├── convert_to_markdown(tag) → str— abstract: platform-specific Markdown conversion
 └── extract(url) → str            — concrete template method (orchestrates the above)
 
 MediumExtractor(BaseExtractor)    — src/mdfetch/providers/medium.py
 ├── DOMAINS = frozenset({"medium.com"})
+├── _FREEDIUM_BASE = "https://freedium-mirror.cfd/"
+├── _no_retry_status_codes = frozenset({403, 429})  — immediate fallback, no medium.com retry
 ├── clean_html() → removes nav, clap buttons, sidebars, share elements, post-footer, author bio
-└── convert_to_markdown() → markdownify with ATX headings, fenced code blocks
+├── convert_to_markdown() → markdownify with ATX headings, fenced code blocks
+├── _parse_freedium(soup) → remaps h4→h3/h5→h4/h6→h5; finds div.main-content; convert_to_markdown
+└── extract() → override: on 403/429 calls fetch_html(freedium_url, _no_retry_codes=frozenset());
+                           exc.url always set to original Medium URL on any Freedium failure
 
 DevToExtractor(BaseExtractor)     — src/mdfetch/providers/devto.py
 ├── DOMAINS = frozenset({"dev.to"})
@@ -131,18 +140,18 @@ Makefile                     # setup / test / integration / lint / typecheck / f
 
 ## Testing Strategy
 
-**Unit tests** (47 tests, offline):
+**Unit tests** (63 tests, offline):
 - Router: domain routing, subdomain suffix matching, duplicate registration, invalid URLs, unsupported platforms
-- MediumExtractor: clean_html, convert_to_markdown, empty content, non-article pages
+- MediumExtractor: clean_html, convert_to_markdown, empty content, non-article pages, _parse_freedium (heading remap, missing main-content), fallback on 403/429 (URL construction, exc.url contract, no-sleep on 429), no-fallback on 200, UnsupportedContentTypeError.url on Freedium path [003-medium-freedium-fallback]
 - DevToExtractor: clean_html (title/cover/heading/image preservation, iframe/ltag embed→link, anchor stripping, non-article error), convert_to_markdown (headings/code/lists/images, no raw HTML, empty content error) [002-devto-provider]
-- Fetch errors: HTTP 404, 503, timeout, connection error, size limit exceeded
+- Fetch errors: HTTP 404, 503, timeout, connection error, size limit exceeded; `_no_retry_status_codes` immediate-raise + `_no_retry_codes` override [003-medium-freedium-fallback]
 - Silent: no stdout/stderr output, no logging during extraction
 
 **Integration tests** (6 tests, network required):
-- Parametrized over 3 real stn1slv.medium.com articles
+- Parametrized over 3 real stn1slv.medium.com articles (including a known paywalled URL that exercises the Freedium fallback when medium.com returns 403) [003-medium-freedium-fallback]
 - Parametrized over 3 real dev.to/stn1slv articles [002-devto-provider]
-- Snapshot-based containment check: `expected_body in extracted_result`
-- 3 retries with 2-second delay on `FetchError` (covers transient 403s/timeouts)
+- Snapshot-based containment check: `expected_body in extracted_result` — tests pass regardless of whether medium.com or Freedium served the content (heading normalisation ensures identical output)
+- 3 retries with 2-second **fixed** delay on `FetchError` (hardcoded; not env-var configurable) [004-remove-backoff]
 - Run with: `make integration` or `uv run pytest tests/integration/ --override-ini=addopts=`
 - Excluded from default `pytest` run via `addopts = "-m 'not integration'"` in pyproject.toml
 
@@ -178,6 +187,10 @@ Makefile                     # setup / test / integration / lint / typecheck / f
 | Routing | `pkgutil.iter_modules` auto-discovery + `@register` | SC-006: one new file = one new platform |
 | Integration tests | Snapshot containment + retry | Durable against minor HTML changes; resilient to transient 403s |
 | test_router.py domain example | Changed from `dev.to` to `substack.com` for "unsupported domain" test | Once DevToExtractor registers `dev.to`, those tests would no longer raise UnsupportedPlatformError | [002-devto-provider]
+| Medium 403/429 fallback | Override `extract()` in `MediumExtractor`; `_no_retry_status_codes=frozenset({403,429})` on class | Immediate fallback with no medium.com retries; `BaseExtractor` extended with `_no_retry_codes` param for thread safety | [003-medium-freedium-fallback]
+| Freedium HTML parsing | Dedicated `_parse_freedium()` method; `div.main-content`; h4→h3 remap | Freedium HTML is structurally incompatible with `clean_html()` (no `<article>`); heading remap ensures snapshot tests pass for both paths | [003-medium-freedium-fallback]
+| Freedium exc.url contract | `inner_exc.url = url` unconditionally; error message is source-agnostic ("Fallback page…") | Preserves transparent-fallback contract (FR-028); `exc.url` is the authoritative field; message content is internal | [003-medium-freedium-fallback]
+| Retry strategy | Fixed delay (`retry_delay` seconds per attempt, unchanged between attempts) | Exponential backoff removed (PR #8); Freedium fallback absorbs 403/429 at a higher level making exponential growth unnecessary; integration tests use hardcoded defaults (`retries=3, retry_delay=2.0`) with no env-var override | [004-remove-backoff]
 
 ---
 
@@ -188,3 +201,7 @@ Makefile                     # setup / test / integration / lint / typecheck / f
 - [x] Coding Standards — PEP 8, strict type hints, `mypy --strict` passes
 - [x] Integration Testing — real Medium URLs, snapshot-based containment assertions
 - [x] Packaging and Distribution — `pyproject.toml` + `src/` layout + `hatchling`; all Makefile targets use `uv run`
+
+---
+
+*Last Updated: 2026-05-15 | Sources appended: [specs/004-remove-backoff/plan.md]*

{mdfetch-0.2.0 → mdfetch-0.2.2}/.specify/memory/spec.md

@@ -1,7 +1,7 @@
 # mdfetch — Main Specification
 
-**Last Updated**: 2026-05-14
-**Sources**: [specs/001-mdfetch-medium-extractor/spec.md], [specs/002-devto-provider/spec.md]
+**Last Updated**: 2026-05-15
+**Sources**: [specs/001-mdfetch-medium-extractor/spec.md], [specs/002-devto-provider/spec.md], [specs/003-medium-freedium-fallback/spec.md], [specs/004-remove-backoff/spec.md]
 
 ---
 
@@ -72,6 +72,51 @@ A developer passing a dev.to URL that points to a profile page, a tag listing, o
 
 ---
 
+### US-007 — Transparent Fallback on Blocked Medium Article (P1)
+[Source: specs/003-medium-freedium-fallback]
+
+A developer calls the library's extract function with a Medium URL. The article is behind a paywall or the user is geo-blocked, causing Medium to return a 403 error. Without any code changes, the library automatically retrieves the same article via the Freedium mirror and returns clean Markdown content.
+
+**Acceptance Scenarios**:
+1. Given a valid Medium article URL that returns 403 from medium.com, when `extract()` is called, then the library returns clean Markdown content retrieved via the Freedium mirror.
+2. Given a valid Medium article URL that returns 403 from medium.com, when the Freedium mirror also fails, then the library raises an appropriate extraction error with `exc.url` set to the original Medium URL.
+3. Given a valid Medium article URL that returns 403 from medium.com, when the library falls back to Freedium, then the caller receives the result without any knowledge of which source was used.
+
+---
+
+### US-008 — Automatic Fallback on Rate Limiting (P2)
+[Source: specs/003-medium-freedium-fallback]
+
+A developer calls the library's extract function for a Medium URL. Medium responds with 429 Too Many Requests. The library automatically uses the Freedium mirror as a fallback and returns clean Markdown without requiring the caller to retry.
+
+**Acceptance Scenarios**:
+1. Given a valid Medium article URL that returns 429 from medium.com, when `extract()` is called, then the library returns clean Markdown content retrieved via the Freedium mirror.
+2. Given a valid Medium article URL that returns 429 from medium.com, when the Freedium mirror also fails, then the library raises an appropriate extraction error.
+
+---
+
+### US-009 — No Fallback When Primary Succeeds (P3)
+[Source: specs/003-medium-freedium-fallback]
+
+A developer calls the library's extract function for a publicly accessible Medium article. Medium responds successfully. The library returns the content directly without involving the Freedium mirror, preserving the existing happy-path behavior.
+
+**Acceptance Scenarios**:
+1. Given a valid Medium article URL that returns a successful response, when `extract()` is called, then the library returns clean Markdown content without making any request to the Freedium mirror.
+
+---
+
+### US-010 — Predictable Fixed-Delay Retry Behaviour (P1)
+[Source: specs/004-remove-backoff]
+
+A developer calling `extract()` encounters a transient network error. The library retries using a simple, predictable fixed delay — exactly `retry_delay` seconds between every attempt — rather than an exponentially growing delay. The retry timing is constant and easy to reason about regardless of which attempt number is being made.
+
+**Acceptance Scenarios**:
+1. Given `fetch_html` is called with `retries=3` and `retry_delay=2.0`, when the first two attempts raise a transient error, then the library sleeps exactly `2.0` seconds before each retry (not `2.0` then `4.0`).
+2. Given `fetch_html` is called with `retries=1`, when the attempt fails, then no sleep occurs and the exception is raised immediately.
+3. Given a status code in `_no_retry_status_codes`, when that error is raised, then no sleep or retry occurs.
+
+---
+
 ### US-006 — Integration Tests Pass Against Real dev.to Article URLs (P3)
 [Source: specs/002-devto-provider]
 
@@ -102,11 +147,23 @@ A developer runs the integration test suite and all dev.to integration tests pas
 ### Network
 - **FR-006**: The library MUST raise a descriptive error when a network request fails (connection error, timeout, non-2xx HTTP status).
 - **FR-014**: HTTP requests MUST use a standard browser-like User-Agent string so that web servers return readable HTML. The User-Agent MUST NOT identify the library by name or version. The library does not check or respect `robots.txt` in v1.
+- **FR-029**: The `fetch_html` method MUST use a fixed delay of exactly `retry_delay` seconds between retry attempts — no exponential multiplication. The retry delay is constant and does not grow with each attempt number. [Source: specs/004-remove-backoff]
 
 ### Packaging & Testing
 - **FR-009**: The library MUST be packaged and distributed via PyPI using modern Python packaging best practices, enabling installation through the standard package manager without additional steps.
 - **FR-010**: The test suite MUST include integration tests that supply real article URLs (Medium and dev.to) to the extraction function and assert that the output matches expected Markdown structure and content.
 
+### Freedium Fallback (Medium)
+- **FR-020**: When a Medium article extraction results in HTTP 403, the system MUST immediately attempt extraction via the Freedium mirror (`https://freedium-mirror.cfd/{url}`) — no retries against medium.com are made first. [Source: specs/003-medium-freedium-fallback]
+- **FR-021**: When a Medium article extraction results in HTTP 429, the system MUST immediately attempt extraction via the Freedium mirror — no retries against medium.com are made first. [Source: specs/003-medium-freedium-fallback]
+- **FR-022**: When the Freedium mirror is used as a fallback, the system MUST return content in the same clean Markdown format as direct extraction; Freedium's `<h4>` headings are remapped to `<h3>` so both paths produce identical heading-level output. [Source: specs/003-medium-freedium-fallback]
+- **FR-023**: When the primary Medium request succeeds (HTTP 200), the system MUST NOT make any request to the Freedium mirror. [Source: specs/003-medium-freedium-fallback]
+- **FR-024**: When both the primary Medium request and the Freedium fallback fail, the system MUST raise an error consistent with the existing exception hierarchy; `exc.url` MUST be set to the original Medium URL (never the Freedium URL). [Source: specs/003-medium-freedium-fallback]
+- **FR-025**: The Freedium fallback mechanism MUST require no changes to the caller's code — the public `extract()` interface remains unchanged. [Source: specs/003-medium-freedium-fallback]
+- **FR-026**: The Freedium fallback MUST only apply to Medium provider URLs; other providers are unaffected. [Source: specs/003-medium-freedium-fallback]
+- **FR-027**: The Freedium fallback MUST be unconditionally active for all Medium URL extractions — no caller configuration, opt-in flag, or extractor parameter is required or supported. [Source: specs/003-medium-freedium-fallback]
+- **FR-028**: The Freedium fallback MUST be fully transparent to the caller — no warning, signal, metadata, or result field shall indicate which source (medium.com or Freedium) provided the content. [Source: specs/003-medium-freedium-fallback]
+
 ### dev.to Platform
 - **FR-015**: The library MUST add `dev.to` to the provider router so that any URL with the `dev.to` domain is dispatched to the dev.to provider without any change to the caller's code. [Source: specs/002-devto-provider]
 - **FR-016**: The library MUST include a dev.to provider that fetches the article page, isolates the main article body from `<div id="article-body">`, removes all non-content elements (navigation, social reaction widgets, comments, author sidebar, tag links), and returns the body as Markdown. [Source: specs/002-devto-provider]
@@ -185,13 +242,16 @@ caller provides URL string
 
 ## Edge Cases and Error Handling
 
-- **Paywalled content**: If Medium gates content behind a paywall and the body is not in the public HTML, the library may return an `EmptyContentError` or a reduced extraction. Full paywall bypass is out of scope for v1.
+- **Paywalled content (HTTP 403)**: When Medium returns HTTP 403 (paywall or geo-block), the library automatically retries via the Freedium mirror (`https://freedium-mirror.cfd/`). If Freedium also fails, the error raised carries `exc.url` set to the original Medium URL. [Source: specs/003-medium-freedium-fallback]
+- **Freedium mirror unreachable**: If the Freedium mirror returns a network error, timeout, or non-2xx status, the fallback fails and the exception is propagated with `exc.url` set to the original Medium URL — Freedium's URL never appears in `exc.url`.
+- **Freedium HTML structure**: Freedium uses `<div class="main-content">` (no `<article>`); if this element is absent, `UnsupportedContentTypeError` is raised with a source-agnostic message ("Fallback page missing main-content element").
+- **Freedium heading levels**: Freedium renders section headings as `<h4>` vs. medium.com's `<h2>`/`<h3>`; `_parse_freedium()` remaps h4→h3, h5→h4, h6→h5 before conversion so snapshot tests pass regardless of which source served the content.
 - **HTML structure changes**: If Medium changes its HTML structure and `<article>` is absent, `UnsupportedContentTypeError` is raised.
 - **Empty article body**: If `<article>` is found but contains no extractable text, `EmptyContentError` is raised.
 - **Network timeouts**: Covered by `FetchError` (30-second fixed timeout).
 - **Oversized responses**: Responses exceeding 10 MB are rejected with `FetchError` to prevent OOM.
 - **Profile/tag pages**: When a `medium.com` URL points to a non-article page, `UnsupportedContentTypeError` is raised (distinct from `UnsupportedPlatformError`).
-- **HTTP 403 / transient failures**: Integration tests use a 3-retry helper with 2-second delay to handle transient rate limits.
+- **HTTP 403 / transient failures**: Integration tests use 3 retries with a 2-second fixed delay (hardcoded; not env-var configurable) to handle transient rate limits. [Source: specs/004-remove-backoff]
 - **dev.to profile pages**: When a `dev.to` URL points to an author profile (no `div#article-body`), `UnsupportedContentTypeError` is raised.
 - **dev.to tag listing pages**: When a `dev.to` URL points to a tag page (e.g., `dev.to/t/kafka`), `UnsupportedContentTypeError` is raised.
 - **dev.to liquid-tag embeds**: Embedded third-party widgets (GitHub Gists, CodePen, YouTube) serialised as `<div class="ltag__*" data-url="...">` are replaced with plain Markdown links; they are never silently dropped.
@@ -226,3 +286,15 @@ caller provides URL string
 - The library supports Python 3.12 and later.
 - The library operates on publicly accessible HTML; it does not execute JavaScript or render dynamic content.
 - Network timeouts use a fixed default of 30 seconds (not user-configurable in v1).
+- **SC-018**: `make test` passes with zero failures (all unit tests green) when run without any `MDFETCH_*` environment variables. [Source: specs/004-remove-backoff]
+- **SC-019**: `make integration` passes with zero failures when run without any `MDFETCH_RETRIES` or `MDFETCH_RETRY_DELAY` environment variables — integration tests use hardcoded defaults (3 retries, 2.0 s delay). [Source: specs/004-remove-backoff]
+- **SC-020**: No reference to `MDFETCH_RETRIES`, `MDFETCH_RETRY_DELAY`, or `_MAX_RETRY_DELAY` appears in `src/`, `tests/`, or `.github/` (specification and documentation files excluded). [Source: specs/004-remove-backoff]
+- **SC-013**: Articles that previously failed with a 403 paywall error are successfully extracted in at least 90% of cases where the Freedium mirror has the content available. [Source: specs/003-medium-freedium-fallback]
+- **SC-014**: Articles that previously failed with a 429 rate-limit error are successfully extracted via fallback without requiring the caller to retry. [Source: specs/003-medium-freedium-fallback]
+- **SC-015**: Zero changes are required in existing caller code to benefit from the Freedium fallback — existing integrations continue to work as-is. [Source: specs/003-medium-freedium-fallback]
+- **SC-016**: When the primary Medium request succeeds, there is no additional latency attributable to the fallback mechanism. [Source: specs/003-medium-freedium-fallback]
+- **SC-017**: All existing unit and integration tests for the Medium extractor continue to pass without modification after the fallback is introduced. [Source: specs/003-medium-freedium-fallback]
+
+---
+
+*Last Updated: 2026-05-15 | Sources appended: [specs/004-remove-backoff/spec.md]*

{mdfetch-0.2.0 → mdfetch-0.2.2}/CLAUDE.md

@@ -11,11 +11,18 @@ src/mdfetch/
 ├── base.py           # BaseExtractor ABC
 ├── router.py         # Domain-to-provider routing
 └── providers/
-    └── medium.py     # MediumExtractor (medium.com + *.medium.com)
+    ├── __init__.py
+    ├── medium.py     # MediumExtractor (medium.com + *.medium.com)
+    └── devto.py      # DevToExtractor (dev.to)
 
 tests/
 ├── unit/             # pytest unit tests (no network)
-└── integration/      # pytest -m integration (real Medium URLs)
+└── integration/      # real network tests (Medium + dev.to URLs + snapshots)
+
+.github/workflows/
+├── ci.yml            # lint + unit tests on push/PR (Python 3.12–3.14)
+├── integration.yml   # scheduled integration tests every Friday 23:30 UTC
+└── publish.yml       # PyPI publish on release
 
 specs/                # Speckit feature specifications
 pyproject.toml        # hatchling build, uv package manager
@@ -38,16 +45,13 @@ make test         # unit tests only
 make lint         # ruff check
 make format       # ruff format
 make build        # uv build (wheel + sdist)
-make upgrade-deps # uv sync --upgrade
-uv run pytest -m integration   # integration tests (network required)
-uv run mypy src/               # type check
+make upgrade-deps # uv sync --all-extras --upgrade
+make integration  # integration tests (network required)
+uv run mypy src/  # type check
 ```
 
+<!-- SPECKIT START -->
 ## Recent Changes
 
-- 002-devto-provider: dev.to article extraction provider — `DevToExtractor`, cover image + embed→link handling, 17 new unit tests, 3 integration tests, version 0.2.0
-- 001-mdfetch-medium-extractor: Initial library release — `extract()` API, Medium provider, typed exceptions, auto-discovery routing, PyPI packaging, snapshot integration tests
-
-<!-- SPECKIT START -->
-**Active feature plan**: none
+- **004-remove-backoff**: Removed exponential backoff from `fetch_html`; restored fixed-delay retries (`retry_delay` seconds per attempt, constant between attempts). Deleted `MDFETCH_RETRIES`/`MDFETCH_RETRY_DELAY` env-var support from CI and integration fixtures; hardcoded `retries=3, retry_delay=2.0` in all integration tests. Deleted `tests/integration/conftest.py`.
 <!-- SPECKIT END -->

{mdfetch-0.2.0 → mdfetch-0.2.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: mdfetch
-Version: 0.2.0
+Version: 0.2.2
 Summary: Extract article content from web platforms and return it as clean Markdown.
 Project-URL: Homepage, https://github.com/stn1slv/md-fetch
 Project-URL: Source, https://github.com/stn1slv/md-fetch

{mdfetch-0.2.0 → mdfetch-0.2.2}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "mdfetch"
-version = "0.2.0"
+version = "0.2.2"
 description = "Extract article content from web platforms and return it as clean Markdown."
 readme = "README.md"
 license = { text = "MIT" }

mdfetch-0.2.2/specs/003-medium-freedium-fallback/checklists/requirements.md

@@ -0,0 +1,34 @@
+# Specification Quality Checklist: Medium Freedium Fallback
+
+**Purpose**: Validate specification completeness and quality before proceeding to planning
+**Created**: 2026-05-14
+**Feature**: [spec.md](../spec.md)
+
+## Content Quality
+
+- [x] No implementation details (languages, frameworks, APIs)
+- [x] Focused on user value and business needs
+- [x] Written for non-technical stakeholders
+- [x] All mandatory sections completed
+
+## Requirement Completeness
+
+- [x] No [NEEDS CLARIFICATION] markers remain
+- [x] Requirements are testable and unambiguous
+- [x] Success criteria are measurable
+- [x] Success criteria are technology-agnostic (no implementation details)
+- [x] All acceptance scenarios are defined
+- [x] Edge cases are identified
+- [x] Scope is clearly bounded
+- [x] Dependencies and assumptions identified
+
+## Feature Readiness
+
+- [x] All functional requirements have clear acceptance criteria
+- [x] User scenarios cover primary flows
+- [x] Feature meets measurable outcomes defined in Success Criteria
+- [x] No implementation details leak into specification
+
+## Notes
+
+- All items pass. Specification is ready for `/speckit-plan`.

mdfetch-0.2.2/specs/003-medium-freedium-fallback/contracts/extract-api.md

@@ -0,0 +1,41 @@
+# Contract: `extract()` Public API
+
+**Feature**: Medium Freedium Fallback | **Status**: Unchanged
+
+## Signature
+
+```python
+def extract(url: str, *, retries: int = 3, retry_delay: float = 2.0) -> str
+```
+
+## Behaviour (unchanged)
+
+| Input | Output |
+|---|---|
+| Valid Medium URL, article accessible | Clean Markdown string |
+| Valid Medium URL, 403 or 429 from medium.com, Freedium succeeds | Clean Markdown string (source is transparent) |
+| Valid Medium URL, 403 or 429 from medium.com, Freedium also fails | Raises `FetchError` or `HTTPStatusError` |
+| Valid Medium URL, other HTTP error (404, 500, …) | Raises `HTTPStatusError` after retries |
+| Valid Medium URL, network timeout/error | Raises `FetchError` after retries |
+| Non-Medium URL | Raises `UnsupportedPlatformError` |
+| Invalid URL | Raises `InvalidURLError` |
+
+## Exception Hierarchy (unchanged)
+
+```
+MdfetchError
+├── InvalidURLError
+├── UnsupportedPlatformError
+├── UnsupportedContentTypeError
+├── EmptyContentError
+└── FetchError
+    └── HTTPStatusError  (status_code: int, url: str | None)
+```
+
+## Guarantees introduced by this feature
+
+- When fallback is invoked, `exc.url` on any raised exception contains the **original Medium URL**, not the Freedium mirror URL.
+- No new exception types are introduced.
+- No new parameters are added to `extract()`.
+- The Freedium mirror URL is never exposed in return values or the `exc.url` field.
+- `exc.url` is the authoritative public URL field and is always set to the original Medium URL when fallback fails. `exc.message` is an internal implementation detail and may contain transport-level information (e.g. the URL that was actually fetched); callers MUST NOT rely on its contents.