mdfetch 0.2.1__tar.gz → 0.2.2__tar.gz

{mdfetch-0.2.1 → mdfetch-0.2.2}/.github/workflows/integration.yml

@@ -30,9 +30,6 @@ jobs:
 
       - name: Run integration tests
         id: integration
-        env:
-          MDFETCH_RETRIES: "6"
-          MDFETCH_RETRY_DELAY: "2.0"
         run: make integration
 
       - name: Create issue on failure

mdfetch-0.2.2/.specify/feature.json

@@ -0,0 +1,3 @@
+{
+  "feature_directory": "specs/004-remove-backoff"
+}

{mdfetch-0.2.1 → mdfetch-0.2.2}/.specify/memory/changelog.md

@@ -2,6 +2,23 @@
 
 ---
 
+### mdfetch — Remove Exponential Backoff — 2026-05-15
+
+**Branch**: `004-remove-backoff`
+**Spec**: specs/004-remove-backoff
+
+**What was added**:
+- Fixed-delay retry behaviour: `fetch_html` now sleeps exactly `retry_delay` seconds between attempts (previously exponential `retry_delay × 2ⁿ`, capped at 60 s). The change makes retry timing predictable for callers and aligns with the Freedium fallback that already absorbs 403/429 errors.
+- Removed `MDFETCH_RETRIES` and `MDFETCH_RETRY_DELAY` env-var support from integration test fixtures (`conftest.py` deleted) and CI workflow (`integration.yml`). Integration tests now hardcode `retries=3, retry_delay=2.0`.
+- Deleted two unit tests (`test_exponential_backoff_sleep_sequence`, `test_exponential_backoff_capped_at_max_delay`) that verified the removed exponential schedule. Added sleep-value assertion to `test_status_code_not_in_no_retry_set_still_retries` to close the FR-029 gap.
+
+**New Components**:
+- No new files. Pure removal: `tests/integration/conftest.py` deleted; `_MAX_RETRY_DELAY` constant removed from `src/mdfetch/base.py`.
+
+**Tasks Completed**: 16/16
+
+---
+
 ### mdfetch — Medium Freedium Fallback — 2026-05-15
 
 **Branch**: `003-medium-freedium-fallback`

{mdfetch-0.2.1 → mdfetch-0.2.2}/.specify/memory/plan.md

@@ -1,7 +1,7 @@
 # mdfetch — Main Implementation Plan
 
-**Last Updated**: 2026-05-14
-**Sources**: [specs/001-mdfetch-medium-extractor/plan.md], [specs/002-devto-provider/plan.md]
+**Last Updated**: 2026-05-15
+**Sources**: [specs/001-mdfetch-medium-extractor/plan.md], [specs/002-devto-provider/plan.md], [specs/003-medium-freedium-fallback/plan.md], [specs/004-remove-backoff/plan.md]
 
 ---
 
@@ -41,6 +41,7 @@ BaseExtractor (ABC)               — src/mdfetch/base.py
 ├── _no_retry_status_codes: frozenset[int] = frozenset()  — codes that skip retry; overridden by providers
 ├── fetch_html(url, *, retries, retry_delay, _no_retry_codes=None) → str
 │                                 — streaming HTTP, 30s timeout, 10 MB cap;
+│                                   fixed delay of retry_delay seconds between attempts (not exponential);
 │                                   codes in _no_retry_codes (or class attribute) raise immediately
 ├── clean_html(soup) → Tag        — abstract: platform-specific HTML isolation
 ├── convert_to_markdown(tag) → str— abstract: platform-specific Markdown conversion
@@ -139,7 +140,7 @@ Makefile                     # setup / test / integration / lint / typecheck / f
 
 ## Testing Strategy
 
-**Unit tests** (65 tests, offline):
+**Unit tests** (63 tests, offline):
 - Router: domain routing, subdomain suffix matching, duplicate registration, invalid URLs, unsupported platforms
 - MediumExtractor: clean_html, convert_to_markdown, empty content, non-article pages, _parse_freedium (heading remap, missing main-content), fallback on 403/429 (URL construction, exc.url contract, no-sleep on 429), no-fallback on 200, UnsupportedContentTypeError.url on Freedium path [003-medium-freedium-fallback]
 - DevToExtractor: clean_html (title/cover/heading/image preservation, iframe/ltag embed→link, anchor stripping, non-article error), convert_to_markdown (headings/code/lists/images, no raw HTML, empty content error) [002-devto-provider]
@@ -150,7 +151,7 @@ Makefile                     # setup / test / integration / lint / typecheck / f
 - Parametrized over 3 real stn1slv.medium.com articles (including a known paywalled URL that exercises the Freedium fallback when medium.com returns 403) [003-medium-freedium-fallback]
 - Parametrized over 3 real dev.to/stn1slv articles [002-devto-provider]
 - Snapshot-based containment check: `expected_body in extracted_result` — tests pass regardless of whether medium.com or Freedium served the content (heading normalisation ensures identical output)
-- 3 retries with 2-second delay on `FetchError` (covers transient 403s/timeouts)
+- 3 retries with 2-second **fixed** delay on `FetchError` (hardcoded; not env-var configurable) [004-remove-backoff]
 - Run with: `make integration` or `uv run pytest tests/integration/ --override-ini=addopts=`
 - Excluded from default `pytest` run via `addopts = "-m 'not integration'"` in pyproject.toml
 
@@ -189,6 +190,7 @@ Makefile                     # setup / test / integration / lint / typecheck / f
 | Medium 403/429 fallback | Override `extract()` in `MediumExtractor`; `_no_retry_status_codes=frozenset({403,429})` on class | Immediate fallback with no medium.com retries; `BaseExtractor` extended with `_no_retry_codes` param for thread safety | [003-medium-freedium-fallback]
 | Freedium HTML parsing | Dedicated `_parse_freedium()` method; `div.main-content`; h4→h3 remap | Freedium HTML is structurally incompatible with `clean_html()` (no `<article>`); heading remap ensures snapshot tests pass for both paths | [003-medium-freedium-fallback]
 | Freedium exc.url contract | `inner_exc.url = url` unconditionally; error message is source-agnostic ("Fallback page…") | Preserves transparent-fallback contract (FR-028); `exc.url` is the authoritative field; message content is internal | [003-medium-freedium-fallback]
+| Retry strategy | Fixed delay (`retry_delay` seconds per attempt, unchanged between attempts) | Exponential backoff removed (PR #8); Freedium fallback absorbs 403/429 at a higher level making exponential growth unnecessary; integration tests use hardcoded defaults (`retries=3, retry_delay=2.0`) with no env-var override | [004-remove-backoff]
 
 ---
 
@@ -202,4 +204,4 @@ Makefile                     # setup / test / integration / lint / typecheck / f
 
 ---
 
-*Last Updated: 2026-05-15 | Sources appended: [specs/003-medium-freedium-fallback/plan.md]*
+*Last Updated: 2026-05-15 | Sources appended: [specs/004-remove-backoff/plan.md]*

{mdfetch-0.2.1 → mdfetch-0.2.2}/.specify/memory/spec.md

@@ -1,7 +1,7 @@
 # mdfetch — Main Specification
 
-**Last Updated**: 2026-05-14
-**Sources**: [specs/001-mdfetch-medium-extractor/spec.md], [specs/002-devto-provider/spec.md]
+**Last Updated**: 2026-05-15
+**Sources**: [specs/001-mdfetch-medium-extractor/spec.md], [specs/002-devto-provider/spec.md], [specs/003-medium-freedium-fallback/spec.md], [specs/004-remove-backoff/spec.md]
 
 ---
 
@@ -105,6 +105,18 @@ A developer calls the library's extract function for a publicly accessible Mediu
 
 ---
 
+### US-010 — Predictable Fixed-Delay Retry Behaviour (P1)
+[Source: specs/004-remove-backoff]
+
+A developer calling `extract()` encounters a transient network error. The library retries using a simple, predictable fixed delay — exactly `retry_delay` seconds between every attempt — rather than an exponentially growing delay. The retry timing is constant and easy to reason about regardless of which attempt number is being made.
+
+**Acceptance Scenarios**:
+1. Given `fetch_html` is called with `retries=3` and `retry_delay=2.0`, when the first two attempts raise a transient error, then the library sleeps exactly `2.0` seconds before each retry (not `2.0` then `4.0`).
+2. Given `fetch_html` is called with `retries=1`, when the attempt fails, then no sleep occurs and the exception is raised immediately.
+3. Given a status code in `_no_retry_status_codes`, when that error is raised, then no sleep or retry occurs.
+
+---
+
 ### US-006 — Integration Tests Pass Against Real dev.to Article URLs (P3)
 [Source: specs/002-devto-provider]
 
@@ -135,6 +147,7 @@ A developer runs the integration test suite and all dev.to integration tests pas
 ### Network
 - **FR-006**: The library MUST raise a descriptive error when a network request fails (connection error, timeout, non-2xx HTTP status).
 - **FR-014**: HTTP requests MUST use a standard browser-like User-Agent string so that web servers return readable HTML. The User-Agent MUST NOT identify the library by name or version. The library does not check or respect `robots.txt` in v1.
+- **FR-029**: The `fetch_html` method MUST use a fixed delay of exactly `retry_delay` seconds between retry attempts — no exponential multiplication. The retry delay is constant and does not grow with each attempt number. [Source: specs/004-remove-backoff]
 
 ### Packaging & Testing
 - **FR-009**: The library MUST be packaged and distributed via PyPI using modern Python packaging best practices, enabling installation through the standard package manager without additional steps.
@@ -238,7 +251,7 @@ caller provides URL string
 - **Network timeouts**: Covered by `FetchError` (30-second fixed timeout).
 - **Oversized responses**: Responses exceeding 10 MB are rejected with `FetchError` to prevent OOM.
 - **Profile/tag pages**: When a `medium.com` URL points to a non-article page, `UnsupportedContentTypeError` is raised (distinct from `UnsupportedPlatformError`).
-- **HTTP 403 / transient failures**: Integration tests use a 3-retry helper with 2-second delay to handle transient rate limits.
+- **HTTP 403 / transient failures**: Integration tests use 3 retries with a 2-second fixed delay (hardcoded; not env-var configurable) to handle transient rate limits. [Source: specs/004-remove-backoff]
 - **dev.to profile pages**: When a `dev.to` URL points to an author profile (no `div#article-body`), `UnsupportedContentTypeError` is raised.
 - **dev.to tag listing pages**: When a `dev.to` URL points to a tag page (e.g., `dev.to/t/kafka`), `UnsupportedContentTypeError` is raised.
 - **dev.to liquid-tag embeds**: Embedded third-party widgets (GitHub Gists, CodePen, YouTube) serialised as `<div class="ltag__*" data-url="...">` are replaced with plain Markdown links; they are never silently dropped.
@@ -273,6 +286,9 @@ caller provides URL string
 - The library supports Python 3.12 and later.
 - The library operates on publicly accessible HTML; it does not execute JavaScript or render dynamic content.
 - Network timeouts use a fixed default of 30 seconds (not user-configurable in v1).
+- **SC-018**: `make test` passes with zero failures (all unit tests green) when run without any `MDFETCH_*` environment variables. [Source: specs/004-remove-backoff]
+- **SC-019**: `make integration` passes with zero failures when run without any `MDFETCH_RETRIES` or `MDFETCH_RETRY_DELAY` environment variables — integration tests use hardcoded defaults (3 retries, 2.0 s delay). [Source: specs/004-remove-backoff]
+- **SC-020**: No reference to `MDFETCH_RETRIES`, `MDFETCH_RETRY_DELAY`, or `_MAX_RETRY_DELAY` appears in `src/`, `tests/`, or `.github/` (specification and documentation files excluded). [Source: specs/004-remove-backoff]
 - **SC-013**: Articles that previously failed with a 403 paywall error are successfully extracted in at least 90% of cases where the Freedium mirror has the content available. [Source: specs/003-medium-freedium-fallback]
 - **SC-014**: Articles that previously failed with a 429 rate-limit error are successfully extracted via fallback without requiring the caller to retry. [Source: specs/003-medium-freedium-fallback]
 - **SC-015**: Zero changes are required in existing caller code to benefit from the Freedium fallback — existing integrations continue to work as-is. [Source: specs/003-medium-freedium-fallback]
@@ -281,4 +297,4 @@ caller provides URL string
 
 ---
 
-*Last Updated: 2026-05-15 | Sources appended: [specs/003-medium-freedium-fallback/spec.md]*
+*Last Updated: 2026-05-15 | Sources appended: [specs/004-remove-backoff/spec.md]*

{mdfetch-0.2.1 → mdfetch-0.2.2}/CLAUDE.md

@@ -51,5 +51,7 @@ uv run mypy src/  # type check
 ```
 
 <!-- SPECKIT START -->
-**Recent changes**: `003-medium-freedium-fallback` — Transparent Freedium mirror fallback for Medium 403/429 responses; `_no_retry_status_codes` hook on `BaseExtractor`; `_parse_freedium()` with h4→h3 heading remap on `MediumExtractor`
+## Recent Changes
+
+- **004-remove-backoff**: Removed exponential backoff from `fetch_html`; restored fixed-delay retries (`retry_delay` seconds per attempt, constant between attempts). Deleted `MDFETCH_RETRIES`/`MDFETCH_RETRY_DELAY` env-var support from CI and integration fixtures; hardcoded `retries=3, retry_delay=2.0` in all integration tests. Deleted `tests/integration/conftest.py`.
 <!-- SPECKIT END -->

{mdfetch-0.2.1 → mdfetch-0.2.2}/PKG-INFO

@@ -1,6 +1,6 @@
 Metadata-Version: 2.4
 Name: mdfetch
-Version: 0.2.1
+Version: 0.2.2
 Summary: Extract article content from web platforms and return it as clean Markdown.
 Project-URL: Homepage, https://github.com/stn1slv/md-fetch
 Project-URL: Source, https://github.com/stn1slv/md-fetch

{mdfetch-0.2.1 → mdfetch-0.2.2}/pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "mdfetch"
-version = "0.2.1"
+version = "0.2.2"
 description = "Extract article content from web platforms and return it as clean Markdown."
 readme = "README.md"
 license = { text = "MIT" }

mdfetch-0.2.2/specs/004-remove-backoff/checklists/requirements.md

@@ -0,0 +1,34 @@
+# Specification Quality Checklist: Remove Exponential Backoff and Env-Var Retry Config
+
+**Purpose**: Validate specification completeness and quality before proceeding to planning
+**Created**: 2026-05-15
+**Feature**: [spec.md](../spec.md)
+
+## Content Quality
+
+- [X] No implementation details (languages, frameworks, APIs)
+- [X] Focused on user value and business needs
+- [X] Written for non-technical stakeholders
+- [X] All mandatory sections completed
+
+## Requirement Completeness
+
+- [X] No [NEEDS CLARIFICATION] markers remain
+- [X] Requirements are testable and unambiguous
+- [X] Success criteria are measurable
+- [X] Success criteria are technology-agnostic (no implementation details)
+- [X] All acceptance scenarios are defined
+- [X] Edge cases are identified
+- [X] Scope is clearly bounded
+- [X] Dependencies and assumptions identified
+
+## Feature Readiness
+
+- [X] All functional requirements have clear acceptance criteria
+- [X] User scenarios cover primary flows
+- [X] Feature meets measurable outcomes defined in Success Criteria
+- [X] No implementation details leak into specification
+
+## Notes
+
+- All checklist items pass. Specification is ready for `/speckit-clarify` or `/speckit-plan`.

mdfetch-0.2.2/specs/004-remove-backoff/plan.md

@@ -0,0 +1,146 @@
+# Implementation Plan: Remove Exponential Backoff and Env-Var Retry Config
+
+**Branch**: `004-remove-backoff` | **Date**: 2026-05-15 | **Spec**: [spec.md](spec.md)
+
+**Input**: Feature specification from `specs/004-remove-backoff/spec.md`
+
+## Summary
+
+Replace the exponential backoff formula introduced in PR #6 (`min(60, retry_delay × 2ⁿ)`) with a simple fixed delay (`retry_delay` seconds, unchanged between attempts). Remove the `_MAX_RETRY_DELAY` module constant, remove `MDFETCH_RETRIES`/`MDFETCH_RETRY_DELAY` env-var reads from the integration test fixtures and CI workflow, and delete the two unit tests that verified the exponential schedule. The `extract()` public API signature and exception contract are entirely unchanged.
+
+## Technical Context
+
+**Language/Version**: Python 3.12+
+
+**Primary Dependencies**:
+
+| Package | Role |
+|---------|------|
+| `httpx` | HTTP client |
+| `beautifulsoup4` / `lxml` | HTML parsing |
+| `markdownify` | Markdown conversion |
+| `pytest` | Test runner |
+| `ruff` / `mypy` | Lint, format, type-check |
+
+**Storage**: N/A
+
+**Testing**: pytest; unit tests for retry logic; integration tests against real URLs
+
+**Target Platform**: Cross-platform PyPI library (Linux, macOS, Windows); Python 3.12+
+
+**Project Type**: library
+
+**Performance Goals**: N/A — this change only affects retry sleep duration
+
+**Constraints**: All 65 existing unit tests and 6 integration tests must pass; `mypy --strict` must pass
+
+**Scale/Scope**: Small, targeted removal — 7 files touched, 2 tests deleted, no new files
+
+## Constitution Check
+
+*GATE: Must pass before Phase 0 research. Re-check after Phase 1 design.*
+
+- [X] Validates Provider Pattern Architecture — no changes to `BaseExtractor` ABC or provider pattern
+- [X] Confirms Technology Stack — no dependency additions or removals
+- [X] Adheres to Coding Standards — PEP 8, strict type hints; removing code only improves compliance
+- [X] Incorporates Integration Testing — integration tests remain; fixtures simplified not removed
+- [X] Respects Packaging and Distribution standards — `pyproject.toml`, `src/` layout, `uv` unchanged
+
+**No violations.** Complexity Tracking table not required.
+
+## Project Structure
+
+### Documentation (this feature)
+
+```text
+specs/004-remove-backoff/
+├── plan.md       ← this file
+├── research.md   ← Phase 0 output
+└── tasks.md      ← Phase 2 output (/speckit-tasks)
+```
+
+### Source Code (files changed by this feature)
+
+```text
+src/mdfetch/
+├── base.py                        # remove _MAX_RETRY_DELAY; fix sleep formula; update docstring
+└── __init__.py                    # update docstring (remove exponential reference)
+
+tests/
+├── unit/test_fetch_errors.py      # remove 2 backoff tests; update 2 stale "backoff" docstrings; add sleep-value assertion to test_status_code_not_in_no_retry_set_still_retries
+└── integration/
+    ├── conftest.py                # remove os import + env var reads; simplify/remove fixtures
+    ├── test_medium_integration.py # remove fixture params; inline retries=3, retry_delay=2.0
+    └── test_devto_integration.py  # remove fixture params; inline retries=3, retry_delay=2.0
+
+.github/workflows/integration.yml # remove MDFETCH_RETRIES and MDFETCH_RETRY_DELAY env vars
+```
+
+## Phase 0: Research
+
+No NEEDS CLARIFICATION items and no new external dependencies. All design decisions are determined by the existing codebase.
+
+See [research.md](research.md) for the decision log.
+
+## Phase 1: Design
+
+### Data Model
+
+No entities. This feature removes code; no new data structures are introduced.
+
+### Contracts
+
+The public `extract()` signature and exception contract are **unchanged**:
+
+```python
+def extract(url: str, *, retries: int = 3, retry_delay: float = 2.0) -> str
+```
+
+- Same parameters, same defaults, same return type, same exception hierarchy.
+- The only observable behaviour change: the sleep between retries is now a flat `retry_delay` seconds rather than `retry_delay × 2ⁿ`. This is not part of the public API contract.
+
+No `contracts/` directory needed.
+
+### Integration Test Strategy
+
+**Before** (PR #6 pattern):
+
+```python
+# conftest.py reads env vars
+@pytest.fixture
+def http_retries() -> int:
+    return int(os.environ.get("MDFETCH_RETRIES", "3"))
+
+# test uses fixture params
+def test_extract_contains_snapshot(
+    url: str, snapshot: str, http_retries: int, http_retry_delay: float
+) -> None:
+    result = extract(url, retries=http_retries, retry_delay=http_retry_delay)
+```
+
+**After** (this feature):
+
+```python
+# conftest.py deleted or left empty
+# test uses hardcoded constants
+def test_extract_contains_snapshot(url: str, snapshot: str) -> None:
+    result = extract(url, retries=3, retry_delay=2.0)
+```
+
+Rationale: the fixtures existed solely to expose env-var configurability. Without that purpose they are dead abstraction. Inlining `3` and `2.0` is simpler and leaves no misleading indirection.
+
+### Key Changes by File
+
+| File | Change |
+|------|--------|
+| `src/mdfetch/base.py` | Remove `_MAX_RETRY_DELAY = 60.0`; change `time.sleep(min(_MAX_RETRY_DELAY, retry_delay * (2**attempt)))` → `time.sleep(retry_delay)`; update docstring |
+| `src/mdfetch/__init__.py` | Update docstring: replace "exponential backoff starting at *retry_delay* seconds" |
+| `tests/unit/test_fetch_errors.py` | Delete `test_exponential_backoff_sleep_sequence` and `test_exponential_backoff_capped_at_max_delay`; update 2 stale "backoff" docstrings; add sleep-value assertion to `test_status_code_not_in_no_retry_set_still_retries` |
+| `tests/integration/conftest.py` | Remove `import os`, remove both fixtures (or delete file if empty) |
+| `tests/integration/test_medium_integration.py` | Remove `http_retries`/`http_retry_delay` fixture params; inline `retries=3, retry_delay=2.0` |
+| `tests/integration/test_devto_integration.py` | Same as above |
+| `.github/workflows/integration.yml` | Remove `MDFETCH_RETRIES: "6"` and `MDFETCH_RETRY_DELAY: "2.0"` from `env:` block |
+
+### Revision: Implementation Sync 2026-05-15
+- Two stale "backoff" docstrings in `tests/unit/test_fetch_errors.py` were not captured in the original plan; added to key-changes table.
+- `test_status_code_not_in_no_retry_set_still_retries` lacked a sleep-value assertion; added to key-changes table to close the AC1 gap.

mdfetch-0.2.2/specs/004-remove-backoff/research.md

@@ -0,0 +1,42 @@
+# Research: Remove Exponential Backoff and Env-Var Retry Config
+
+**Feature**: `004-remove-backoff` | **Date**: 2026-05-15
+
+## Summary
+
+No external research required. All decisions are determined by the existing codebase and the motivation documented in the spec.
+
+---
+
+## Decision 1: Fixed vs Exponential Delay
+
+**Decision**: Replace exponential backoff with a fixed `retry_delay` seconds between attempts.
+
+**Rationale**: The exponential schedule was introduced to work around aggressive rate-limiting from medium.com in CI. That problem is now solved at a higher level by the Freedium fallback (PR #7): 403/429 responses trigger an immediate switch to Freedium rather than retrying against medium.com at all. The exponential schedule therefore provides no additional resilience but adds complexity and unpredictability to retry timing.
+
+**Alternatives considered**:
+- Keep exponential, remove only env vars → Rejected. Removes the tooling noise but keeps the unnecessary complexity in the core retry loop.
+- Keep exponential, add deprecation warning → Rejected. A deprecation period is unnecessary for an internal mechanism not part of the public API.
+
+---
+
+## Decision 2: Remove Fixtures vs Hardcode Defaults
+
+**Decision**: Delete the `http_retries` and `http_retry_delay` pytest fixtures from `conftest.py` and inline `retries=3, retry_delay=2.0` directly in integration test calls.
+
+**Rationale**: The fixtures existed as thin wrappers around env-var reads. Once the env vars are removed, the fixtures become single-line constants with no abstraction value. Inlining the defaults removes a layer of indirection that would otherwise require readers to trace through `conftest.py` to understand what values are used.
+
+**Alternatives considered**:
+- Keep fixtures with hardcoded defaults → Rejected. A fixture returning a literal integer is dead abstraction; it signals configurability that doesn't exist.
+- Move constants to a `conftest.py` module-level variable → Rejected. Same problem — adds indirection without benefit.
+
+---
+
+## Decision 3: Handling `conftest.py` After Fixture Removal
+
+**Decision**: Delete `conftest.py` entirely since removing both fixtures leaves the file empty (only `import os` and two functions, all of which are removed).
+
+**Rationale**: An empty `conftest.py` is harmless but misleading — it implies there is shared test configuration when there is none. Deleting it is cleaner.
+
+**Alternatives considered**:
+- Leave empty file → Rejected. No benefit; creates noise in the file tree.

mdfetch-0.2.2/specs/004-remove-backoff/spec.md

@@ -0,0 +1,92 @@
+# Feature Specification: Remove Exponential Backoff and Env-Var Retry Config
+
+**Feature Branch**: `004-remove-backoff`
+
+**Created**: 2026-05-15
+
+**Status**: Completed
+
+**Input**: User description: "let's deprecate and remove the exponential backoff and env.variables which was implemented in https://github.com/stn1slv/md-fetch/pull/6"
+
+## Background
+
+PR #6 introduced two related changes to mdfetch:
+
+1. **Exponential backoff** — the wait before retry attempt *n* was changed from a fixed `retry_delay` seconds to `min(60, retry_delay × 2ⁿ)`. A `_MAX_RETRY_DELAY = 60.0` module-level constant was added to cap the delay.
+2. **Configurable retries via environment variables** — `tests/integration/conftest.py` reads `MDFETCH_RETRIES` and `MDFETCH_RETRY_DELAY` env vars and exposes them as pytest fixtures. CI sets `MDFETCH_RETRIES=6` in `integration.yml`.
+
+Both changes are to be removed, restoring the simpler fixed-delay retry behaviour.
+
+## User Scenarios & Testing *(mandatory)*
+
+### User Story 1 - Fixed-Delay Retry Behaviour Restored (Priority: P1)
+
+A developer calling `extract()` encounters a transient network error. The library retries using a simple, predictable fixed delay — exactly `retry_delay` seconds between every attempt — rather than an exponentially growing delay.
+
+**Why this priority**: This is the core change. Simple, deterministic retry behaviour is easier to reason about and test. The exponential schedule was added to work around aggressive CI rate-limiting; that problem is now solved at the infrastructure level (Freedium fallback for 403/429) rather than in the retry loop.
+
+**Independent Test**: Can be fully tested by calling `fetch_html` with a failing URL and verifying that `time.sleep` is called with the exact `retry_delay` value for every inter-attempt gap (not an exponentially growing value).
+
+**Acceptance Scenarios**:
+
+1. **Given** `fetch_html` is called with `retries=3` and `retry_delay=2.0`, **When** the first two attempts raise a transient error, **Then** the library sleeps exactly `2.0` seconds before each retry (not `2.0` then `4.0`).
+2. **Given** `fetch_html` is called with `retries=1`, **When** the attempt fails, **Then** no sleep occurs and the exception is raised immediately.
+3. **Given** a status code that is in `_no_retry_status_codes`, **When** that error is raised, **Then** no sleep or retry occurs.
+
+---
+
+### User Story 2 - Integration Tests Use Hardcoded Retry Defaults (Priority: P2)
+
+A developer runs `make integration` locally or in CI without setting any `MDFETCH_*` environment variables. The integration tests use fixed defaults (3 retries, 2.0 s delay) and do not consult environment variables.
+
+**Why this priority**: Env vars coupling test behaviour to CI configuration makes tests harder to reason about and reproduce locally. With the Freedium fallback absorbing 403/429 errors, 3 retries at a fixed 2 s delay is sufficient.
+
+**Independent Test**: Can be fully tested by running `make integration` without setting `MDFETCH_RETRIES` or `MDFETCH_RETRY_DELAY` and verifying all 6 tests pass.
+
+**Acceptance Scenarios**:
+
+1. **Given** neither `MDFETCH_RETRIES` nor `MDFETCH_RETRY_DELAY` is set, **When** `make integration` runs, **Then** all integration tests pass using 3 retries and 2.0 s delay.
+2. **Given** the `MDFETCH_RETRIES=6` env var is set in `integration.yml`, **When** this feature is shipped, **Then** that env var is removed from the CI workflow.
+
+---
+
+### Edge Cases
+
+- What happens if tests currently rely on the `http_retries` / `http_retry_delay` fixtures by name? They must be updated to use hardcoded defaults or inline constants.
+- What happens to the two unit tests that assert exponential sleep sequences? They must be removed entirely.
+- What happens to the `_MAX_RETRY_DELAY` module-level constant? It is unused after the change and must be removed to avoid dead code.
+
+## Requirements *(mandatory)*
+
+### Functional Requirements
+
+- **FR-001**: The `fetch_html` method MUST use a fixed delay of exactly `retry_delay` seconds between retry attempts — no exponential multiplication.
+- **FR-002**: The `_MAX_RETRY_DELAY` module-level constant in `base.py` MUST be removed.
+- **FR-003**: The `import time` statement in `base.py` MUST be retained (still needed for `time.sleep`).
+- **FR-004**: The `MDFETCH_RETRIES` and `MDFETCH_RETRY_DELAY` environment variable reads in `tests/integration/conftest.py` MUST be removed.
+- **FR-005**: The `http_retries` and `http_retry_delay` pytest fixtures MUST be simplified to return hardcoded defaults (`3` and `2.0` respectively) or be inlined into the test functions.
+- **FR-006**: The `MDFETCH_RETRIES` and `MDFETCH_RETRY_DELAY` env var entries in `.github/workflows/integration.yml` MUST be removed.
+- **FR-007**: The two unit tests `test_exponential_backoff_sleep_sequence` and `test_exponential_backoff_capped_at_max_delay` in `tests/unit/test_fetch_errors.py` MUST be removed.
+- **FR-008**: The `fetch_html` docstring in `base.py` MUST be updated to describe fixed-delay retry behaviour (remove references to exponential schedule and `min(60, …)` formula).
+- **FR-009**: The `extract()` docstring in `src/mdfetch/__init__.py` MUST be updated if it references exponential backoff.
+- **FR-010**: All remaining unit and integration tests MUST continue to pass after the changes.
+
+## Success Criteria *(mandatory)*
+
+### Measurable Outcomes
+
+- **SC-001**: `make test` passes with zero failures after the changes (all unit tests green).
+- **SC-002**: `make integration` passes with zero failures when run without any `MDFETCH_*` environment variables.
+- **SC-003**: The two exponential-backoff unit tests no longer exist in the test suite.
+- **SC-004**: No reference to `MDFETCH_RETRIES`, `MDFETCH_RETRY_DELAY`, or `_MAX_RETRY_DELAY` remains in `src/`, `tests/`, or `.github/` (spec and documentation files are excluded).
+- **SC-005**: The CI `integration.yml` workflow no longer sets `MDFETCH_RETRIES` or `MDFETCH_RETRY_DELAY`.
+
+## Assumptions
+
+- The Freedium fallback (introduced in PR #7) adequately handles the Medium 403/429 rate-limiting that originally motivated exponential backoff. Fixed delay at 3 retries is therefore sufficient for CI reliability.
+- The `http_retries` and `http_retry_delay` fixtures in `conftest.py` are used only in `test_medium_integration.py` and `test_devto_integration.py`; no other test files depend on them.
+- The `timeout-minutes: 30` setting in `integration.yml` can remain as-is — it provides adequate headroom even with fixed delay retries.
+
+### Revision: Implementation Sync 2026-05-15
+- Stale "backoff" terminology found in two docstrings in `tests/unit/test_fetch_errors.py` (lines 98, 123); these are test comments not covered by FR-004 or FR-007 and require a targeted cleanup task.
+- `test_status_code_not_in_no_retry_set_still_retries` verifies sleep count but not sleep value; US1 AC1 ("sleeps exactly `retry_delay` seconds") implies a value assertion is also needed.

mdfetch-0.2.2/specs/004-remove-backoff/tasks.md

@@ -0,0 +1,147 @@
+# Tasks: Remove Exponential Backoff and Env-Var Retry Config
+
+**Input**: Design documents from `specs/004-remove-backoff/`
+
+**Prerequisites**: plan.md ✅, spec.md ✅, research.md ✅
+
+**Organization**: Tasks are grouped by user story to enable independent implementation and testing.
+
+## Format: `[ID] [P?] [Story] Description`
+
+- **[P]**: Can run in parallel (different files, no dependencies on incomplete tasks)
+- **[Story]**: Which user story this task belongs to (US1, US2)
+
+---
+
+## Phase 1: Setup
+
+**Purpose**: Establish green baseline before making any changes
+
+- [X] T001 Verify baseline by running `make test` — confirm 65 unit tests pass before changes
+
+---
+
+## Phase 2: Foundational (Blocking Prerequisites)
+
+**Purpose**: No shared infrastructure changes needed for this feature — this phase is satisfied by T001.
+
+**⚠️ NOTE**: US1 and US2 are fully independent; either can be implemented first or in parallel.
+
+---
+
+## Phase 3: User Story 1 — Fixed-Delay Retry Behaviour Restored (Priority: P1) 🎯 MVP
+
+**Goal**: Replace the exponential backoff formula with a flat `retry_delay` seconds sleep; remove the `_MAX_RETRY_DELAY` cap constant; update docstrings; delete the two unit tests that verified the exponential schedule.
+
+**Independent Test**: Run `make test` — the remaining retry tests (`TestNoRetryStatusCodes`, `TestFetchErrors`) must pass; the two deleted tests must no longer exist.
+
+### Implementation for User Story 1
+
+- [X] T002 [US1] Remove `_MAX_RETRY_DELAY = 60.0` constant and change `time.sleep(min(_MAX_RETRY_DELAY, retry_delay * (2**attempt)))` to `time.sleep(retry_delay)` in `src/mdfetch/base.py`
+- [X] T003 [US1] Update `fetch_html` docstring in `src/mdfetch/base.py` to say "fixed delay of *retry_delay* seconds" (remove mention of exponential schedule and `min(60, …)` formula)
+- [X] T004 [P] [US1] Update `extract()` docstring in `src/mdfetch/__init__.py` — replace "exponential backoff starting at *retry_delay* seconds" with "fixed delay of *retry_delay* seconds between attempts"
+- [X] T005 [P] [US1] Delete `test_exponential_backoff_sleep_sequence` and `test_exponential_backoff_capped_at_max_delay` test methods from `tests/unit/test_fetch_errors.py`
+
+**Checkpoint**: Run `make test` — all remaining unit tests pass; no reference to exponential backoff remains in `src/`.
+
+---
+
+## Phase 4: User Story 2 — Integration Tests Use Hardcoded Retry Defaults (Priority: P2)
+
+**Goal**: Remove the `MDFETCH_RETRIES` / `MDFETCH_RETRY_DELAY` env-var fixtures from the integration test suite and CI; inline the defaults directly in test call sites; delete `conftest.py`.
+
+**Independent Test**: Run `make integration` without any `MDFETCH_*` env vars set — all 6 integration tests pass.
+
+### Implementation for User Story 2
+
+- [X] T006 [P] [US2] Remove `http_retries` and `http_retry_delay` fixture parameters from the test function signature and inline `retries=3, retry_delay=2.0` in the `extract()` call in `tests/integration/test_medium_integration.py`; remove the env-var docstring line from the test docstring
+- [X] T007 [P] [US2] Same changes as T006 in `tests/integration/test_devto_integration.py`
+- [X] T008 [US2] Delete `tests/integration/conftest.py` (file is empty after T006 + T007 remove all fixture consumers; no other test files reference these fixtures)
+- [X] T009 [US2] Remove the `MDFETCH_RETRIES: "6"` and `MDFETCH_RETRY_DELAY: "2.0"` lines from the `env:` block of the `Run integration tests` step in `.github/workflows/integration.yml`
+
+**Checkpoint**: Run `make integration` — all 6 tests pass; no `MDFETCH_*` variables appear anywhere in the codebase.
+
+---
+
+## Phase 5: Polish & Cross-Cutting Concerns
+
+**Purpose**: Final validation — confirm all quality gates pass together
+
+- [X] T010 Run `make test` to confirm all unit tests pass (expected: 63 tests — 65 minus the 2 deleted backoff tests)
+- [X] T011 Run `uv run mypy src/` to confirm zero type errors after docstring and code changes
+- [X] T012 Run `make lint` to confirm ruff check passes with no violations
+- [X] T013 Verify no remaining references to `_MAX_RETRY_DELAY`, `MDFETCH_RETRIES`, or `MDFETCH_RETRY_DELAY` in source, tests, and CI by running `grep -r "MAX_RETRY_DELAY\|MDFETCH_RETRIES\|MDFETCH_RETRY_DELAY" src/ tests/ .github/` (spec/doc files are intentionally excluded)
+- [X] T014 Run `make integration` without any `MDFETCH_*` env vars set and confirm all 6 integration tests pass (verifies SC-002)
+
+---
+
+## Remediation: Gaps
+
+*Added by Implementation Sync 2026-05-15*
+
+- [X] T015 [P] [US1] Update stale "backoff" docstrings in `tests/unit/test_fetch_errors.py`: line 98 "no backoff sleep" → "no retry sleep"; line 123 "trigger backoff" → "trigger retry with fixed delay" [Sync: Gap Report]
+- [X] T016 [US1] Add sleep-value assertion to `test_status_code_not_in_no_retry_set_still_retries` in `tests/unit/test_fetch_errors.py` — add `assert mock_sleep.call_args_list == [call(1.0), call(1.0)]` after the call_count check (verifies US1 AC1: "sleeps exactly retry_delay seconds") [Sync: Gap Report]
+
+---
+
+## Dependencies & Execution Order
+
+### Phase Dependencies
+
+- **Setup (Phase 1)**: No dependencies — start immediately
+- **US1 (Phase 3)**: Depends on T001 (baseline green) — can start after setup
+- **US2 (Phase 4)**: Depends on T001 (baseline green) — **independent from US1; can run in parallel**
+- **Polish (Phase 5)**: Depends on US1 and US2 both complete
+
+### Within Each User Story
+
+- **US1**: T002 → T003 (same file, sequential); T004 and T005 are [P] (different files, independent of T002/T003)
+- **US2**: T006 and T007 are [P] (different files); T008 depends on T006 + T007 (delete conftest only after test files no longer reference fixtures); T009 is [P] with T006/T007/T008 (different file)
+
+### Parallel Opportunities
+
+```text
+After T001 (baseline green):
+
+Stream A (US1):
+  T002 → T003   # base.py: formula + docstring (same file, sequential)
+  T004          # __init__.py docstring [P with T005]
+  T005          # delete backoff unit tests [P with T004]
+
+Stream B (US2):
+  T006 [P]      # test_medium_integration.py (parallel with T007)
+  T007 [P]      # test_devto_integration.py (parallel with T006)
+  T008          # delete conftest.py (after T006 + T007)
+  T009          # integration.yml env vars [P, independent]
+
+After both streams: T010 → T011 → T012 → T013 → T014
+```
+
+---
+
+## Implementation Strategy
+
+### MVP First (User Story 1 Only)
+
+1. T001 — establish baseline
+2. T002 → T003 — fix retry formula and docstring in `base.py`
+3. T004 + T005 — update `__init__.py` docstring; delete backoff tests
+4. T010 + T011 + T012 — validate
+5. **STOP and VALIDATE**: `make test` passes with 63 tests
+
+### Full Delivery (US1 + US2)
+
+Continue after MVP:
+6. T006 + T007 — update integration test signatures
+7. T008 — delete conftest.py
+8. T009 — remove CI env vars
+9. T013 — confirm no residual references
+
+---
+
+## Notes
+
+- [P] tasks operate on different files with no shared dependencies — safe to run concurrently
+- T008 (delete conftest.py) MUST come after T006 + T007 — deleting conftest.py while test functions still declare `http_retries`/`http_retry_delay` parameters will cause pytest fixture-not-found errors
+- T009 (.github/workflows) is independent of all other tasks; can be done at any point after T001
+- Expected final unit test count: 63 (65 baseline − 2 deleted backoff tests)