harvester-kit 0.1.0__tar.gz

harvester_kit-0.1.0/.github/workflows/ci.yml

@@ -0,0 +1,35 @@
+name: CI
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+
+jobs:
+  lint:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v7
+      - uses: actions/setup-python@v7
+        with:
+          python-version: "3.13"
+      - run: pip install -e ".[dev]"
+      - run: ruff check .
+      - run: ruff format --check .
+      - run: mypy src
+
+  test:
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ubuntu-latest, windows-latest, macos-latest]
+        python: ["3.10", "3.11", "3.12", "3.13", "3.14"]
+    runs-on: ${{ matrix.os }}
+    steps:
+      - uses: actions/checkout@v7
+      - uses: actions/setup-python@v7
+        with:
+          python-version: ${{ matrix.python }}
+          allow-prereleases: true
+      - run: pip install -e ".[dev]"
+      - run: pytest -q

harvester_kit-0.1.0/.github/workflows/release.yml

@@ -0,0 +1,39 @@
+name: Release
+
+# Publishing a GitHub release builds the package and uploads it to PyPI using
+# Trusted Publishing (OIDC): no API token is stored anywhere.
+
+on:
+  release:
+    types: [published]
+
+jobs:
+  build:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v7
+      - uses: actions/setup-python@v7
+        with:
+          python-version: "3.13"
+      - run: pip install build twine
+      - run: python -m build
+      - run: twine check --strict dist/*
+      - uses: actions/upload-artifact@v7
+        with:
+          name: dist
+          path: dist/
+
+  publish:
+    needs: build
+    runs-on: ubuntu-latest
+    environment:
+      name: pypi
+      url: https://pypi.org/project/harvester-kit/
+    permissions:
+      id-token: write
+    steps:
+      - uses: actions/download-artifact@v8
+        with:
+          name: dist
+          path: dist/
+      - uses: pypa/gh-action-pypi-publish@v1.14.2

harvester_kit-0.1.0/.gitignore

@@ -0,0 +1,14 @@
+__pycache__/
+*.py[cod]
+.venv/
+venv/
+dist/
+build/
+*.egg-info/
+.pytest_cache/
+.mypy_cache/
+.ruff_cache/
+.coverage
+htmlcov/
+# harvester's own data directory (stores, queues, runs)
+.harvester/

harvester_kit-0.1.0/CHANGELOG.md

@@ -0,0 +1,14 @@
+# Changelog
+
+## 0.1.0 — 2026-10-03
+
+First release.
+
+- Crawl engine with a persistent SQLite frontier: resumable, de-duplicated, prioritised.
+- Content-addressed raw store with fetch history; offline `reparse`.
+- Per-request cache policies (prefer / revalidate / bypass) with conditional requests.
+- robots.txt per RFC 9309, Crawl-delay, per-host throttling with adaptive back-off and Retry-After.
+- Honest Scrapling-based fetcher (no impersonation, real User-Agent).
+- JSONL output with per-record provenance and data licence; run manifests.
+- Built-in sources: OAI-PMH, DSpace 7+, India Code.
+- CLI: run, reparse, status, changes, show, robots, list, new.

harvester_kit-0.1.0/CONTRIBUTING.md

@@ -0,0 +1,10 @@
+# Contributing
+
+Thanks for helping. A few ground rules keep harvester trustworthy:
+
+1. **Politeness is not optional.** Changes must not add ways to evade blocks:
+   no fingerprint spoofing, CAPTCHA solving or block-dodging proxy rotation.
+2. **Callbacks stay pure.** Anything that would let a parse callback do I/O breaks offline re-parsing.
+3. **Tests come with changes.** `pytest`, `ruff check .`, `ruff format --check .` and `mypy src` must pass.
+
+New sources for public bulk-data protocols (CKAN, Zenodo, sitemaps …) are especially welcome.

harvester_kit-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Sarthak Pahwa
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

harvester_kit-0.1.0/PKG-INFO

@@ -0,0 +1,231 @@
+Metadata-Version: 2.5
+Name: harvester-kit
+Version: 0.1.0
+Summary: Polite, reproducible web harvesting: fetch once, parse forever.
+Project-URL: Homepage, https://github.com/sarthak213/harvester
+Project-URL: Issues, https://github.com/sarthak213/harvester/issues
+Project-URL: Changelog, https://github.com/sarthak213/harvester/blob/main/CHANGELOG.md
+Author: Sarthak Pahwa
+License-Expression: MIT
+License-File: LICENSE
+Keywords: crawling,data-pipeline,dspace,harvesting,oai-pmh,robots.txt,scraping
+Classifier: Development Status :: 3 - Alpha
+Classifier: Environment :: Console
+Classifier: Framework :: AsyncIO
+Classifier: Intended Audience :: Developers
+Classifier: Intended Audience :: Science/Research
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3 :: Only
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Programming Language :: Python :: 3.13
+Classifier: Programming Language :: Python :: 3.14
+Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
+Classifier: Topic :: Scientific/Engineering :: Information Analysis
+Classifier: Typing :: Typed
+Requires-Python: >=3.10
+Requires-Dist: orjson>=3.10
+Requires-Dist: protego>=0.4
+Requires-Dist: pydantic>=2.7
+Requires-Dist: rich>=13.7
+Requires-Dist: scrapling[fetchers]>=0.4.15
+Requires-Dist: typer>=0.12
+Requires-Dist: w3lib>=2.1
+Provides-Extra: dev
+Requires-Dist: mypy>=1.11; extra == 'dev'
+Requires-Dist: pytest-asyncio>=0.24; extra == 'dev'
+Requires-Dist: pytest>=8; extra == 'dev'
+Requires-Dist: ruff>=0.6; extra == 'dev'
+Description-Content-Type: text/markdown
+
+<div align="center">
+
+# harvester
+
+**Polite, reproducible web harvesting. Fetch once, parse forever.**
+
+[![CI](https://github.com/sarthak213/harvester/actions/workflows/ci.yml/badge.svg)](https://github.com/sarthak213/harvester/actions/workflows/ci.yml)
+[![PyPI](https://img.shields.io/pypi/v/harvester-kit)](https://pypi.org/project/harvester-kit/)
+[![Python 3.10+](https://img.shields.io/badge/python-3.10%E2%80%933.14-blue)](https://www.python.org)
+[![License: MIT](https://img.shields.io/badge/license-MIT-green)](LICENSE)
+[![Typed: mypy strict](https://img.shields.io/badge/typed-mypy%20strict-informational)](https://mypy-lang.org)
+
+</div>
+
+Most scrapers are throwaway scripts. They re-download everything whenever a parser changes, start over after a crash, hammer servers until they get blocked, and produce data nobody can trace back to its source.
+
+**harvester** treats harvesting as a data pipeline. Every response goes into a content-addressed store, so parsers can be fixed and re-run **offline**. The crawl queue lives on disk, so interrupted runs **resume** where they stopped. Politeness is built into the engine, not left as an afterthought. Every record carries its **provenance and licence**.
+
+It uses [Scrapling](https://github.com/D4Vinci/Scrapling) for fetching and parsing, and adds the engine around it.
+
+```text
+┌──────────────────────────── quotes | crawl ─────────────────────────────┐
+│      60   pages        150   records      60   fetched   0   from cache │
+│       0   queued         0   retries       0   failed    0   blocked    │
+│ 272.5 KB  downloaded    93   pages/min   39s   elapsed   0   skipped    │
+└─────────────────────────────────────────────────────────────────────────┘
+complete run 20261003T044028Z
+  records: 50 author, 100 quote
+```
+
+## Highlights
+
+| | |
+|---|---|
+| **Fetch once, parse forever** | Raw responses are stored gzip-compressed and keyed by SHA-256. `harvester reparse` replays the whole crawl graph from disk with **zero network requests**, so fixing a parser never costs another crawl. |
+| **Resumable by design** | The frontier is a SQLite queue. Ctrl+C (or a crash, or a reboot) loses nothing; the next `run` continues where it stopped. |
+| **Polite by default** | robots.txt per RFC 9309, including its 4xx/5xx rules and `Crawl-delay`. Per-host rate limits and concurrency caps. `Retry-After` honoured. The delay adapts, backing off on 429/503 and recovering slowly. |
+| **Honest** | Requests carry a real User-Agent with a contact URL. Scrapling's browser impersonation and header forging are **turned off**. |
+| **Smart caching** | Each request picks a cache policy: documents are fetched once, listings are revalidated with `If-None-Match` / `If-Modified-Since`, and a `304` reuses the stored body. |
+| **Provenance and licensing** | Every record states its source URL, fetch time, content hash, parser version and the data's licence. Every run writes a manifest. |
+| **Change tracking** | The store keeps a history of every fetch. `harvester changes` lists pages whose content changed between crawls. |
+| **Bulk-data connectors** | Built-in **OAI-PMH** and **DSpace 7+** sources cover thousands of libraries, archives and government repositories. Downloads are verified against published MD5s. |
+| **Small API** | A source is a class with a start URL and a parse method. Load one from a file, or publish it as a plugin through entry points. |
+
+## Quick start
+
+```bash
+pip install harvester-kit       # installs the `harvester` command and package
+
+harvester run examples/quotes.py          # crawl the scraping sandbox
+harvester reparse examples/quotes.py      # rebuild every record offline
+harvester status examples/quotes.py       # queue, store and run history
+```
+
+Output lands in `.harvester/<source>/runs/<run-id>/records.jsonl`, one record per line:
+
+```json
+{
+  "kind": "author",
+  "id": "Albert-Einstein",
+  "data": {"name": "Albert Einstein", "born": "March 14, 1879", "born_in": "in Ulm, Germany"},
+  "provenance": {
+    "source": "quotes",
+    "url": "https://quotes.toscrape.com/author/Albert-Einstein/",
+    "fetched_at": "2026-10-03T04:41:36Z",
+    "sha256": "9d1c…",
+    "status": 200,
+    "parser_version": "1",
+    "harvester_version": "0.1.0",
+    "license": {"name": "Scraping sandbox by Zyte; sample data"}
+  }
+}
+```
+
+## Writing a source
+
+```python
+from harvester import CachePolicy, Page, Request, Source
+
+
+class Quotes(Source):
+    name = "quotes"
+    start_urls = ("https://quotes.toscrape.com/",)
+    allowed_domains = ("quotes.toscrape.com",)
+
+    def start(self):
+        # Listings change, so revalidate them on each pass.
+        yield Request(url=self.start_urls[0], cache=CachePolicy.REVALIDATE)
+
+    def parse(self, page: Page):
+        for quote in page.css("div.quote"):
+            yield page.record(
+                "quote",
+                id=quote.css("span.text::text").get(),
+                author=quote.css("small.author::text").get(),
+                tags=quote.css("a.tag::text").getall(),
+            )
+        if next_href := page.css("li.next a::attr(href)").get():
+            yield page.follow(next_href, cache=CachePolicy.REVALIDATE)
+```
+
+`harvester new my-site` scaffolds one. Callbacks receive a `Page`, which offers CSS/XPath via Scrapling, `.json()` and `.text`. They yield `Request`s to follow and `Record`s to keep. The one rule: **callbacks must not do their own I/O**. Keeping them pure is what makes offline re-parsing exact.
+
+See [docs/writing-sources.md](docs/writing-sources.md) for options, multiple callbacks, file downloads and packaging a source as a plugin.
+
+## How it works
+
+```mermaid
+flowchart LR
+    S[Source.start] --> F[(Frontier<br/>SQLite queue)]
+    F -->|claim| R[Scope + robots.txt check]
+    R --> C{In raw store?}
+    C -->|PREFER| PG[Page]
+    C -->|REVALIDATE / miss| T[Host throttle] --> H[Scrapling fetch]
+    H -->|200| ST[(Raw store<br/>SHA-256 blobs)] --> PG
+    H -->|304| PG
+    H -->|429 / 503 / 5xx| B[Back off + retry] --> F
+    PG --> CB[Parse callback]
+    CB -->|Request| F
+    CB -->|Record + provenance| O[(records.jsonl<br/>+ manifest)]
+```
+
+```text
+.harvester/<source>/
+├── frontier.sqlite        # queue: pending / done / failed / skipped, retries, priorities
+├── store/
+│   ├── index.sqlite       # latest response per URL + full fetch history
+│   └── blobs/ab/cd/…gz    # bodies, content-addressed and deduplicated
+└── runs/<run-id>/
+    ├── records.jsonl      # output
+    └── manifest.json      # settings, stats, licence, outcome
+```
+
+## Built-in sources
+
+| Source | What it harvests |
+|---|---|
+| `oai-pmh` | Any [OAI-PMH 2.0](https://www.openarchives.org/OAI/openarchivesprotocol.html) repository: `ListRecords` with resumption tokens, deleted records, any metadata format. `-o base_url=… -o set=… -o metadata_prefix=…` |
+| `dspace` | Any DSpace 7+ repository via its REST API: items, then files from chosen bundles, MD5-verified, with text files inlined. `-o base_url=… -o scope=<uuid> -o bundles=ORIGINAL` |
+| `india-code` | India's central Acts from [India Code](https://indiacode.gov.in): clean act records (number, year, ministry, enforcement date, repeal status) plus the official text extraction. `-o in_force_only=true -o pdf=true` |
+
+## CLI
+
+| Command | |
+|---|---|
+| `harvester run SOURCE [-o k=v] [--limit N] [--delay S] [--refresh] [--restart]` | Crawl. Resumes interrupted work; otherwise starts a new pass that reuses the cache. |
+| `harvester reparse SOURCE` | Re-run parsers over the raw store. No network. |
+| `harvester status SOURCE` | Queue counts, store size, failures with reasons, recent runs. |
+| `harvester changes SOURCE` | URLs whose content changed between fetches. |
+| `harvester show URL -s SOURCE [--save FILE]` | Inspect or extract a stored response. |
+| `harvester robots URL` | Explain whether a URL may be fetched, and why. |
+| `harvester list` / `harvester new NAME` | Installed sources / scaffold a new one. |
+
+`SOURCE` is an installed source name, `path/to/file.py`, or `path/to/file.py:ClassName`.
+
+## Politeness, precisely
+
+harvester is meant for collecting data you are entitled to collect, without being a burden on the sites that host it.
+
+- **robots.txt (RFC 9309).**
+  - `2xx`: rules and `Crawl-delay` are obeyed.
+  - `4xx`: no restrictions.
+  - `5xx` or network error: **complete disallow**.
+  - A source may opt out of that last rule only by declaring `robots_unavailable = "allow"` *with a written reason*. The choice is recorded in every run manifest.
+- **Rate limits.** Per-host minimum delay (default 1 s) and concurrency (default 1). On `429`/`503` the delay doubles, honouring `Retry-After`. It recovers by 10% per healthy response.
+- **Identity.** `harvester/<version> (+https://github.com/sarthak213/harvester)`. Change it with `--user-agent`, but keep a contact URL in it.
+- **No evasion.** No browser fingerprint spoofing, no CAPTCHA solving, no proxy rotation to get around blocks. If a site says no, harvester listens.
+
+## Development
+
+```bash
+git clone https://github.com/sarthak213/harvester && cd harvester
+python -m venv .venv && . .venv/bin/activate        # Windows: .venv\Scripts\activate
+pip install -e ".[dev]"
+pytest                   # 41 tests, including a real HTTP server for end-to-end runs
+ruff check . && ruff format --check . && mypy src
+```
+
+## Roadmap
+
+- Browser rendering for JavaScript-only pages (Scrapling `DynamicFetcher`), opt-in per request
+- Sitemap and RSS/Atom discovery sources
+- Parquet / SQLite export with record de-duplication across runs
+- Scheduled incremental runs and change notifications
+- More open-data connectors: CKAN, Zenodo, S3 open-data buckets
+
+## License
+
+MIT. See [LICENSE](LICENSE). The licence covers harvester's code. **The data you harvest has its own terms**; sources declare them, and harvester records them in every output.

harvester_kit-0.1.0/README.md

@@ -0,0 +1,189 @@
+<div align="center">
+
+# harvester
+
+**Polite, reproducible web harvesting. Fetch once, parse forever.**
+
+[![CI](https://github.com/sarthak213/harvester/actions/workflows/ci.yml/badge.svg)](https://github.com/sarthak213/harvester/actions/workflows/ci.yml)
+[![PyPI](https://img.shields.io/pypi/v/harvester-kit)](https://pypi.org/project/harvester-kit/)
+[![Python 3.10+](https://img.shields.io/badge/python-3.10%E2%80%933.14-blue)](https://www.python.org)
+[![License: MIT](https://img.shields.io/badge/license-MIT-green)](LICENSE)
+[![Typed: mypy strict](https://img.shields.io/badge/typed-mypy%20strict-informational)](https://mypy-lang.org)
+
+</div>
+
+Most scrapers are throwaway scripts. They re-download everything whenever a parser changes, start over after a crash, hammer servers until they get blocked, and produce data nobody can trace back to its source.
+
+**harvester** treats harvesting as a data pipeline. Every response goes into a content-addressed store, so parsers can be fixed and re-run **offline**. The crawl queue lives on disk, so interrupted runs **resume** where they stopped. Politeness is built into the engine, not left as an afterthought. Every record carries its **provenance and licence**.
+
+It uses [Scrapling](https://github.com/D4Vinci/Scrapling) for fetching and parsing, and adds the engine around it.
+
+```text
+┌──────────────────────────── quotes | crawl ─────────────────────────────┐
+│      60   pages        150   records      60   fetched   0   from cache │
+│       0   queued         0   retries       0   failed    0   blocked    │
+│ 272.5 KB  downloaded    93   pages/min   39s   elapsed   0   skipped    │
+└─────────────────────────────────────────────────────────────────────────┘
+complete run 20261003T044028Z
+  records: 50 author, 100 quote
+```
+
+## Highlights
+
+| | |
+|---|---|
+| **Fetch once, parse forever** | Raw responses are stored gzip-compressed and keyed by SHA-256. `harvester reparse` replays the whole crawl graph from disk with **zero network requests**, so fixing a parser never costs another crawl. |
+| **Resumable by design** | The frontier is a SQLite queue. Ctrl+C (or a crash, or a reboot) loses nothing; the next `run` continues where it stopped. |
+| **Polite by default** | robots.txt per RFC 9309, including its 4xx/5xx rules and `Crawl-delay`. Per-host rate limits and concurrency caps. `Retry-After` honoured. The delay adapts, backing off on 429/503 and recovering slowly. |
+| **Honest** | Requests carry a real User-Agent with a contact URL. Scrapling's browser impersonation and header forging are **turned off**. |
+| **Smart caching** | Each request picks a cache policy: documents are fetched once, listings are revalidated with `If-None-Match` / `If-Modified-Since`, and a `304` reuses the stored body. |
+| **Provenance and licensing** | Every record states its source URL, fetch time, content hash, parser version and the data's licence. Every run writes a manifest. |
+| **Change tracking** | The store keeps a history of every fetch. `harvester changes` lists pages whose content changed between crawls. |
+| **Bulk-data connectors** | Built-in **OAI-PMH** and **DSpace 7+** sources cover thousands of libraries, archives and government repositories. Downloads are verified against published MD5s. |
+| **Small API** | A source is a class with a start URL and a parse method. Load one from a file, or publish it as a plugin through entry points. |
+
+## Quick start
+
+```bash
+pip install harvester-kit       # installs the `harvester` command and package
+
+harvester run examples/quotes.py          # crawl the scraping sandbox
+harvester reparse examples/quotes.py      # rebuild every record offline
+harvester status examples/quotes.py       # queue, store and run history
+```
+
+Output lands in `.harvester/<source>/runs/<run-id>/records.jsonl`, one record per line:
+
+```json
+{
+  "kind": "author",
+  "id": "Albert-Einstein",
+  "data": {"name": "Albert Einstein", "born": "March 14, 1879", "born_in": "in Ulm, Germany"},
+  "provenance": {
+    "source": "quotes",
+    "url": "https://quotes.toscrape.com/author/Albert-Einstein/",
+    "fetched_at": "2026-10-03T04:41:36Z",
+    "sha256": "9d1c…",
+    "status": 200,
+    "parser_version": "1",
+    "harvester_version": "0.1.0",
+    "license": {"name": "Scraping sandbox by Zyte; sample data"}
+  }
+}
+```
+
+## Writing a source
+
+```python
+from harvester import CachePolicy, Page, Request, Source
+
+
+class Quotes(Source):
+    name = "quotes"
+    start_urls = ("https://quotes.toscrape.com/",)
+    allowed_domains = ("quotes.toscrape.com",)
+
+    def start(self):
+        # Listings change, so revalidate them on each pass.
+        yield Request(url=self.start_urls[0], cache=CachePolicy.REVALIDATE)
+
+    def parse(self, page: Page):
+        for quote in page.css("div.quote"):
+            yield page.record(
+                "quote",
+                id=quote.css("span.text::text").get(),
+                author=quote.css("small.author::text").get(),
+                tags=quote.css("a.tag::text").getall(),
+            )
+        if next_href := page.css("li.next a::attr(href)").get():
+            yield page.follow(next_href, cache=CachePolicy.REVALIDATE)
+```
+
+`harvester new my-site` scaffolds one. Callbacks receive a `Page`, which offers CSS/XPath via Scrapling, `.json()` and `.text`. They yield `Request`s to follow and `Record`s to keep. The one rule: **callbacks must not do their own I/O**. Keeping them pure is what makes offline re-parsing exact.
+
+See [docs/writing-sources.md](docs/writing-sources.md) for options, multiple callbacks, file downloads and packaging a source as a plugin.
+
+## How it works
+
+```mermaid
+flowchart LR
+    S[Source.start] --> F[(Frontier<br/>SQLite queue)]
+    F -->|claim| R[Scope + robots.txt check]
+    R --> C{In raw store?}
+    C -->|PREFER| PG[Page]
+    C -->|REVALIDATE / miss| T[Host throttle] --> H[Scrapling fetch]
+    H -->|200| ST[(Raw store<br/>SHA-256 blobs)] --> PG
+    H -->|304| PG
+    H -->|429 / 503 / 5xx| B[Back off + retry] --> F
+    PG --> CB[Parse callback]
+    CB -->|Request| F
+    CB -->|Record + provenance| O[(records.jsonl<br/>+ manifest)]
+```
+
+```text
+.harvester/<source>/
+├── frontier.sqlite        # queue: pending / done / failed / skipped, retries, priorities
+├── store/
+│   ├── index.sqlite       # latest response per URL + full fetch history
+│   └── blobs/ab/cd/…gz    # bodies, content-addressed and deduplicated
+└── runs/<run-id>/
+    ├── records.jsonl      # output
+    └── manifest.json      # settings, stats, licence, outcome
+```
+
+## Built-in sources
+
+| Source | What it harvests |
+|---|---|
+| `oai-pmh` | Any [OAI-PMH 2.0](https://www.openarchives.org/OAI/openarchivesprotocol.html) repository: `ListRecords` with resumption tokens, deleted records, any metadata format. `-o base_url=… -o set=… -o metadata_prefix=…` |
+| `dspace` | Any DSpace 7+ repository via its REST API: items, then files from chosen bundles, MD5-verified, with text files inlined. `-o base_url=… -o scope=<uuid> -o bundles=ORIGINAL` |
+| `india-code` | India's central Acts from [India Code](https://indiacode.gov.in): clean act records (number, year, ministry, enforcement date, repeal status) plus the official text extraction. `-o in_force_only=true -o pdf=true` |
+
+## CLI
+
+| Command | |
+|---|---|
+| `harvester run SOURCE [-o k=v] [--limit N] [--delay S] [--refresh] [--restart]` | Crawl. Resumes interrupted work; otherwise starts a new pass that reuses the cache. |
+| `harvester reparse SOURCE` | Re-run parsers over the raw store. No network. |
+| `harvester status SOURCE` | Queue counts, store size, failures with reasons, recent runs. |
+| `harvester changes SOURCE` | URLs whose content changed between fetches. |
+| `harvester show URL -s SOURCE [--save FILE]` | Inspect or extract a stored response. |
+| `harvester robots URL` | Explain whether a URL may be fetched, and why. |
+| `harvester list` / `harvester new NAME` | Installed sources / scaffold a new one. |
+
+`SOURCE` is an installed source name, `path/to/file.py`, or `path/to/file.py:ClassName`.
+
+## Politeness, precisely
+
+harvester is meant for collecting data you are entitled to collect, without being a burden on the sites that host it.
+
+- **robots.txt (RFC 9309).**
+  - `2xx`: rules and `Crawl-delay` are obeyed.
+  - `4xx`: no restrictions.
+  - `5xx` or network error: **complete disallow**.
+  - A source may opt out of that last rule only by declaring `robots_unavailable = "allow"` *with a written reason*. The choice is recorded in every run manifest.
+- **Rate limits.** Per-host minimum delay (default 1 s) and concurrency (default 1). On `429`/`503` the delay doubles, honouring `Retry-After`. It recovers by 10% per healthy response.
+- **Identity.** `harvester/<version> (+https://github.com/sarthak213/harvester)`. Change it with `--user-agent`, but keep a contact URL in it.
+- **No evasion.** No browser fingerprint spoofing, no CAPTCHA solving, no proxy rotation to get around blocks. If a site says no, harvester listens.
+
+## Development
+
+```bash
+git clone https://github.com/sarthak213/harvester && cd harvester
+python -m venv .venv && . .venv/bin/activate        # Windows: .venv\Scripts\activate
+pip install -e ".[dev]"
+pytest                   # 41 tests, including a real HTTP server for end-to-end runs
+ruff check . && ruff format --check . && mypy src
+```
+
+## Roadmap
+
+- Browser rendering for JavaScript-only pages (Scrapling `DynamicFetcher`), opt-in per request
+- Sitemap and RSS/Atom discovery sources
+- Parquet / SQLite export with record de-duplication across runs
+- Scheduled incremental runs and change notifications
+- More open-data connectors: CKAN, Zenodo, S3 open-data buckets
+
+## License
+
+MIT. See [LICENSE](LICENSE). The licence covers harvester's code. **The data you harvest has its own terms**; sources declare them, and harvester records them in every output.

harvester_kit-0.1.0/docs/writing-sources.md

@@ -0,0 +1,96 @@
+# Writing sources
+
+A source tells harvester where to start and how to read what comes back. Everything else (queueing, caching, politeness, retries, output) is the engine's job.
+
+## Anatomy
+
+```python
+from harvester import CachePolicy, DataLicense, Page, Request, Source
+
+
+class Gazette(Source):
+    name = "gazette"  # CLI name and data-directory name
+    description = "Notifications from the Example Gazette"
+    allowed_domains = ("gazette.example.org",)  # anything else is skipped
+    license = DataLicense(
+        name="Government open data licence",
+        url="https://gazette.example.org/terms",
+        attribution="Source: Example Gazette",
+        commercial_use=True,
+    )
+    parser_version = "2"  # bump when output changes
+    delay = 2.0  # seconds between requests to this host
+    concurrency = 1  # simultaneous requests to this host
+
+    @classmethod
+    def option_defaults(cls):
+        return {"year": "2026"}  # harvester run gazette -o year=2025
+
+    def start(self):
+        url = f"https://gazette.example.org/{self.options['year']}/"
+        yield Request(url=url, callback="parse_index", cache=CachePolicy.REVALIDATE)
+
+    def parse_index(self, page: Page):
+        for href in page.css("a.notice::attr(href)").getall():
+            yield page.follow(href, callback="parse_notice", meta={"year": self.options["year"]})
+        if nxt := page.css("a[rel=next]::attr(href)").get():
+            yield page.follow(nxt, cache=CachePolicy.REVALIDATE)
+
+    def parse_notice(self, page: Page):
+        yield page.record(
+            "notice",
+            id=page.css("meta[name=notice-id]::attr(content)").get(),
+            title=page.css("h1::text").get(),
+            year=page.meta["year"],
+        )
+        if pdf := page.css("a.pdf::attr(href)").get():
+            yield page.follow(pdf, callback="parse_pdf")
+
+    def parse_pdf(self, page: Page):
+        # The PDF body is already in the raw store; record where to find it.
+        yield page.record("file", id=page.sha256, url=page.url, bytes=len(page.body))
+```
+
+## Callbacks
+
+- A callback is any method named in `Request.callback` (default `parse`).
+- It receives a `Page` and yields `Request` and `Record` objects. Use `page.follow()` and `page.record()` for convenience.
+- **Callbacks must not perform I/O.** Everything they need is in the page: `body`, `headers`, `text`, `json()`, `css()`, `xpath()` and `meta`. Purity is what lets `harvester reparse` reproduce a crawl exactly from the raw store.
+- An exception inside a callback fails only that request. The reason is visible in `harvester status`, and the crawl continues.
+
+## Cache policies
+
+| Policy | Use for | Behaviour on later passes |
+|---|---|---|
+| `PREFER` (default) | documents, files, anything immutable | served from the raw store; no request |
+| `REVALIDATE` | listings, feeds, search results | conditional request; `304` reuses the stored body |
+| `BYPASS` | volatile endpoints | always re-downloaded |
+
+`harvester run --refresh` upgrades every `PREFER` request to `REVALIDATE` for one pass.
+
+## Passing data between callbacks
+
+Put it in `meta`. It is stored with the request in the frontier, so it survives restarts. `page.follow()` merges the current page's meta with any you add.
+
+## robots.txt that cannot be fetched
+
+If a site's robots.txt returns `5xx` or times out, RFC 9309 says crawlers must assume everything is disallowed, and harvester does. If you have good reason to proceed (for example, the operator has confirmed it, or the endpoint is a published bulk API), declare it:
+
+```python
+robots_unavailable = "allow"
+robots_unavailable_reason = "Operator confirmed by email on 2026-10-01; bulk API is public."
+```
+
+The reason is written into every run manifest.
+
+## Packaging a source as a plugin
+
+Any installed package can register sources:
+
+```toml
+# pyproject.toml of your package
+[project.entry-points."harvester.sources"]
+gazette = "my_package.sources:Gazette"
+```
+
+After `pip install`, `harvester list` shows it and `harvester run gazette` works.