scrapeplanner 0.1.0__tar.gz

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (30) hide show
  1. scrapeplanner-0.1.0/.gitignore +14 -0
  2. scrapeplanner-0.1.0/CHANGELOG.md +44 -0
  3. scrapeplanner-0.1.0/LICENSE +21 -0
  4. scrapeplanner-0.1.0/PKG-INFO +452 -0
  5. scrapeplanner-0.1.0/README.md +418 -0
  6. scrapeplanner-0.1.0/pyproject.toml +108 -0
  7. scrapeplanner-0.1.0/src/scrapeplanner/__init__.py +5 -0
  8. scrapeplanner-0.1.0/src/scrapeplanner/__main__.py +5 -0
  9. scrapeplanner-0.1.0/src/scrapeplanner/analyze.py +389 -0
  10. scrapeplanner-0.1.0/src/scrapeplanner/cli.py +108 -0
  11. scrapeplanner-0.1.0/src/scrapeplanner/config.py +25 -0
  12. scrapeplanner-0.1.0/src/scrapeplanner/detectors/__init__.py +0 -0
  13. scrapeplanner-0.1.0/src/scrapeplanner/detectors/antibot.py +259 -0
  14. scrapeplanner-0.1.0/src/scrapeplanner/detectors/embedded.py +142 -0
  15. scrapeplanner-0.1.0/src/scrapeplanner/detectors/feeds.py +259 -0
  16. scrapeplanner-0.1.0/src/scrapeplanner/detectors/platform.py +349 -0
  17. scrapeplanner-0.1.0/src/scrapeplanner/detectors/rendering.py +65 -0
  18. scrapeplanner-0.1.0/src/scrapeplanner/detectors/robots.py +40 -0
  19. scrapeplanner-0.1.0/src/scrapeplanner/detectors/sitemaps.py +119 -0
  20. scrapeplanner-0.1.0/src/scrapeplanner/detectors/structured.py +321 -0
  21. scrapeplanner-0.1.0/src/scrapeplanner/fetch.py +378 -0
  22. scrapeplanner-0.1.0/src/scrapeplanner/html.py +131 -0
  23. scrapeplanner-0.1.0/src/scrapeplanner/models.py +370 -0
  24. scrapeplanner-0.1.0/src/scrapeplanner/output/__init__.py +0 -0
  25. scrapeplanner-0.1.0/src/scrapeplanner/output/json_out.py +9 -0
  26. scrapeplanner-0.1.0/src/scrapeplanner/output/profile.py +80 -0
  27. scrapeplanner-0.1.0/src/scrapeplanner/output/terminal.py +293 -0
  28. scrapeplanner-0.1.0/src/scrapeplanner/planner.py +465 -0
  29. scrapeplanner-0.1.0/src/scrapeplanner/py.typed +0 -0
  30. scrapeplanner-0.1.0/src/scrapeplanner/record.py +135 -0
@@ -0,0 +1,14 @@
1
+ __pycache__/
2
+ *.py[cod]
3
+ .venv/
4
+ dist/
5
+ build/
6
+ *.egg-info/
7
+ .mypy_cache/
8
+ .ruff_cache/
9
+ .pytest_cache/
10
+ .coverage
11
+ htmlcov/
12
+ profile.yaml
13
+ /recordings/
14
+ test-output/
@@ -0,0 +1,44 @@
1
+ # Changelog
2
+
3
+ All notable changes are listed here. The format follows
4
+ [Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and the project uses
5
+ [Semantic Versioning](https://semver.org/). The JSON output has its own
6
+ `schema_version`; a breaking change to it is a major version.
7
+
8
+ ## [Unreleased]
9
+
10
+ ## [0.1.0] - 2026-11-22
11
+
12
+ First release on PyPI. `scrapeplanner URL` reads a site's robots.txt and
13
+ first page, probes the public feeds its platform is known to have, names any
14
+ anti-bot vendor, and prints a collection plan: methods from cheapest to most
15
+ expensive, a polite rate, and the number of requests N records will take. It
16
+ detects and recommends only; it never tries to get past a block.
17
+
18
+ ### Added
19
+
20
+ - `scrapeplanner URL`: terminal report, `--json`, `--profile PATH`.
21
+ - Polite fetcher: robots.txt read first and checked for every URL, 10 requests
22
+ per run by default (every redirect hop counts), at least 1 s between
23
+ requests or the robots.txt crawl-delay, clear User-Agent, stop at the
24
+ first block or challenge, no cookie jar, no proxies.
25
+ - Detectors: robots.txt, sitemaps, platform (Shopify, WooCommerce, Magento,
26
+ BigCommerce, Wix, Squarespace, Next.js, Nuxt), public feeds (Shopify
27
+ products.json and product .js, WooCommerce Store API, WordPress REST,
28
+ RSS/Atom/JSON feeds linked from the page), JSON-LD and microdata,
29
+ embedded state, rendering (raw HTML or browser), anti-bot vendor
30
+ (Cloudflare, Akamai, DataDome, HUMAN, Imperva, Kasada).
31
+ - Planner: methods cheapest first, polite rate, request and time estimate
32
+ for N items, risks and next steps.
33
+ - `--record DIR` saves every exchange for offline replay.
34
+ - Platforms: WordPress, Drupal, Webflow, Salesforce Commerce Cloud, Greenhouse,
35
+ Lever, Ashby, CKAN, Socrata, Hugo, Sphinx, Docusaurus, Gatsby, Remix,
36
+ SvelteKit, Astro, Angular, React, Vue, Lit. "not checked" when no page was
37
+ served.
38
+ - Feeds: Greenhouse, Lever and Ashby job-board APIs; CKAN `package_search`.
39
+ - Anti-bot: AWS WAF, and a site's own CAPTCHA / "are you a robot" page.
40
+ - Stacked report layout for terminals narrower than 80 columns.
41
+ - JSON Schema of the `--json` output in `docs/report.schema.json`.
42
+
43
+ [Unreleased]: https://github.com/siddharthkumarx/scrapeplanner/compare/v0.1.0...HEAD
44
+ [0.1.0]: https://github.com/siddharthkumarx/scrapeplanner/tree/v0.1.0
@@ -0,0 +1,21 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2026 Siddharth Kumar
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
@@ -0,0 +1,452 @@
1
+ Metadata-Version: 2.5
2
+ Name: scrapeplanner
3
+ Version: 0.1.0
4
+ Summary: Give it a URL; it reports the cheapest reliable, polite way to collect that site's data.
5
+ Project-URL: Homepage, https://github.com/siddharthkumarx/scrapeplanner
6
+ Project-URL: Issues, https://github.com/siddharthkumarx/scrapeplanner/issues
7
+ Project-URL: Changelog, https://github.com/siddharthkumarx/scrapeplanner/blob/main/CHANGELOG.md
8
+ Author: Siddharth Kumar
9
+ License-Expression: MIT
10
+ License-File: LICENSE
11
+ Keywords: cli,crawling,json-ld,robots.txt,sitemap,structured data,web scraping
12
+ Classifier: Development Status :: 4 - Beta
13
+ Classifier: Environment :: Console
14
+ Classifier: Intended Audience :: Developers
15
+ Classifier: Operating System :: OS Independent
16
+ Classifier: Programming Language :: Python :: 3
17
+ Classifier: Programming Language :: Python :: 3 :: Only
18
+ Classifier: Programming Language :: Python :: 3.11
19
+ Classifier: Programming Language :: Python :: 3.12
20
+ Classifier: Programming Language :: Python :: 3.13
21
+ Classifier: Programming Language :: Python :: 3.14
22
+ Classifier: Topic :: Internet :: WWW/HTTP
23
+ Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
24
+ Classifier: Typing :: Typed
25
+ Requires-Python: >=3.11
26
+ Requires-Dist: httpx<1,>=0.28
27
+ Requires-Dist: protego>=0.7
28
+ Requires-Dist: pydantic<3,>=2.14
29
+ Requires-Dist: pyyaml>=6.0.3
30
+ Requires-Dist: rich>=15
31
+ Requires-Dist: selectolax<2,>=1.0
32
+ Requires-Dist: typer>=0.27
33
+ Description-Content-Type: text/markdown
34
+
35
+ <p align="center">
36
+ <picture>
37
+ <source media="(prefers-reduced-motion: reduce)" srcset="assets/hero-still.svg">
38
+ <img src="assets/hero.svg" width="720" alt="A terminal runs 'scrapeplanner https://www.deathwishcoffee.com/'. The report says GOOD FIT, best method Public feed / JSON API, platform Shopify, anti-bot Cloudflare (CDN only), polite rate 1 request every 1s, 5 of 10 requests used. It then scrolls to the plan, which recommends Shopify /products.json: 1 request returns all 163 products, against 1,008 requests for HTML parsing. Recorded run 008-sites.">
39
+ </picture>
40
+ </p>
41
+
42
+ <h1 align="center">scrapeplanner</h1>
43
+
44
+ <p align="center"><b>Point it at a URL. It tells you the cheapest polite way to get the data.</b></p>
45
+
46
+ <p align="center">
47
+ <a href="https://pypi.org/project/scrapeplanner/"><img alt="PyPI version" src="https://img.shields.io/pypi/v/scrapeplanner?style=flat&label=PyPI"></a>
48
+ <a href="https://github.com/siddharthkumarx/scrapeplanner/actions/workflows/ci.yml"><img alt="CI" src="https://img.shields.io/github/actions/workflow/status/siddharthkumarx/scrapeplanner/ci.yml?branch=main&style=flat&label=CI"></a>
49
+ <a href="LICENSE"><img alt="License: MIT" src="https://img.shields.io/badge/license-MIT-2FBFAE?style=flat"></a>
50
+ </p>
51
+
52
+ <p align="center">
53
+ <a href="#install">Install</a> ·
54
+ <a href="#quick-start">Quick start</a> ·
55
+ <a href="#example-report">Example report</a> ·
56
+ <a href="#what-it-detects">What it detects</a> ·
57
+ <a href="#how-it-decides">How it decides</a> ·
58
+ <a href="#results-on-30-public-sites">Results</a> ·
59
+ <a href="#faq">FAQ</a>
60
+ </p>
61
+
62
+ ## Why
63
+
64
+ Most of the time lost on a scraping job goes into the wrong method: a
65
+ browser fleet for a store that publishes `products.json`, CSS selectors for a
66
+ job board with a public API, or a week of work on a site that blocks every
67
+ automated request anyway. Those answers are usually on the first page.
68
+
69
+ `scrapeplanner` reads robots.txt and the page, probes the public feeds the
70
+ platform is known to have, names any anti-bot vendor, and prints a plan:
71
+ methods from cheapest to most expensive, a polite rate, and how many requests
72
+ N records will take. On 30 public sites it needed 1 to 8 requests per site
73
+ (122 in total, never more than its cap of 10) and a median of 4.5 seconds
74
+ ([results](docs/results.md)).
75
+
76
+ It is a planning tool. It detects and recommends; it never tries to get past
77
+ anything.
78
+
79
+ ## Install
80
+
81
+ `scrapeplanner` is on [PyPI](https://pypi.org/project/scrapeplanner/) and
82
+ needs Python 3.11 or newer. Install it as a standalone command with pipx or
83
+ uv, or into an environment with pip:
84
+
85
+ ```bash
86
+ pipx install scrapeplanner
87
+ uv tool install scrapeplanner
88
+ pip install scrapeplanner
89
+ ```
90
+
91
+ The latest development version, from source:
92
+
93
+ ```bash
94
+ pipx install git+https://github.com/siddharthkumarx/scrapeplanner
95
+ ```
96
+
97
+ ## Quick start
98
+
99
+ ```bash
100
+ scrapeplanner https://www.deathwishcoffee.com/ # terminal report
101
+ scrapeplanner https://www.deathwishcoffee.com/ --json > report.json # full report as JSON
102
+ scrapeplanner https://www.deathwishcoffee.com/ --profile profile.yaml # starter config
103
+ ```
104
+
105
+ The outputs below come from the recorded run of that site in
106
+ [`checks/results/008-sites/deathwish-coffee/`](checks/results/008-sites/deathwish-coffee/).
107
+
108
+ **Report** (first panel; the full report is in the hero above and in the [example](#example-report)):
109
+
110
+ ```text
111
+ ╭─ scrapeplanner 0.1.0.dev0 · https://www.deathwishcoffee.com/ ──────────────────────────────────╮
112
+ │ Verdict GOOD FIT │
113
+ │ Best method Public feed / JSON API │
114
+ │ Platform Shopify │
115
+ │ Anti-bot Cloudflare (CDN only) │
116
+ │ Rendering data in the raw HTML │
117
+ │ Polite rate 1 request every 1s │
118
+ │ This run 5/10 requests, 5.2s │
119
+ ╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
120
+ ```
121
+
122
+ **`--json`**, a few of the fields (`jq` filter shown):
123
+
124
+ ```bash
125
+ scrapeplanner https://www.deathwishcoffee.com/ --json \
126
+ | jq '{platform: .platform.name, verdict: .plan.verdict, recommended: .plan.recommended,
127
+ requests_used, feed: (.feeds.candidates[0] | {kind, status, items_seen, url})}'
128
+ ```
129
+
130
+ ```json
131
+ {
132
+ "platform": "shopify",
133
+ "verdict": "good",
134
+ "recommended": "feed_api",
135
+ "requests_used": 5,
136
+ "feed": {
137
+ "kind": "shopify_products_json",
138
+ "status": "ok",
139
+ "items_seen": 163,
140
+ "url": "https://www.deathwishcoffee.com/products.json?limit=250"
141
+ }
142
+ }
143
+ ```
144
+
145
+ **`--profile profile.yaml`** (two parts shortened, marked `# ...`):
146
+
147
+ ```yaml
148
+ # Starter profile written by scrapeplanner. Review before use.
149
+ # Every value comes from one analysis run; re-run to refresh it.
150
+ profile_version: 1
151
+ generated_by: scrapeplanner 0.1.0.dev0
152
+ generated_at: '2026-11-04T20:52:19+00:00'
153
+ site: https://www.deathwishcoffee.com
154
+ verdict: good
155
+ method: feed_api
156
+ source:
157
+ urls:
158
+ - https://www.deathwishcoffee.com/products.json?limit=250&page={page}
159
+ pagination:
160
+ param: page
161
+ start: 1
162
+ stop_when: empty page
163
+ fields:
164
+ - id
165
+ - title
166
+ - handle
167
+ # ... 10 more fields ...
168
+ rate:
169
+ delay_seconds: 1.0
170
+ concurrency: 1
171
+ max_requests_per_day: null
172
+ reason: default 1s between requests
173
+ # ... politeness: user agent, robots.txt, stop rules ...
174
+ estimate:
175
+ items: 1000
176
+ requests: 1
177
+ hours: 0.0
178
+ notes:
179
+ - 'the store lists 163 products: one request returns all of them'
180
+ ```
181
+
182
+ Useful options: `--items N` (records to estimate for, default 1000),
183
+ `--delay S` (seconds between requests, minimum 1), `--max-requests N`
184
+ (default 10, hard cap 50), `--timeout S`. Run `scrapeplanner --help` for all.
185
+
186
+ ## Example report
187
+
188
+ GitLab's public job board on Greenhouse, from the recorded run
189
+ [`checks/results/008-sites/gitlab-greenhouse/report.txt`](checks/results/008-sites/gitlab-greenhouse/report.txt)
190
+ (100-column terminal; scroll sideways on a phone):
191
+
192
+ ```text
193
+ ╭─ scrapeplanner 0.1.0.dev0 · https://job-boards.greenhouse.io/gitlab ───────────────────────────╮
194
+ │ Verdict GOOD FIT │
195
+ │ Best method Public feed / JSON API │
196
+ │ Platform Greenhouse (+ Remix) │
197
+ │ Anti-bot none seen │
198
+ │ Rendering data in the raw HTML │
199
+ │ Polite rate 1 request every 1s │
200
+ │ This run 5/10 requests, 4.7s │
201
+ ╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
202
+ ╭─ Plan (cheapest first) ──────────────────────────────────────────────────────────────────────────╮
203
+ │ │
204
+ │ # Method Status Req. for 1,000 Time Details │
205
+ │ ────────────────────────────────────────────────────────────────────────────────────────────── │
206
+ │ 1 Feed / API recommended 1 < 1 min Greenhouse job-board API: │
207
+ │ every open job with its fields │
208
+ │ - one request returns all 244 │
209
+ │ open jobs; the board lists no │
210
+ │ more │
211
+ │ 2 Structured data no - - no schema.org records on this │
212
+ │ page (types seen: none) │
213
+ │ 3 Embedded JSON no - - no large inline state │
214
+ │ 4 HTML usable 1,000 17 min CSS selectors on │
215
+ │ server-rendered HTML (3,931 │
216
+ │ chars of text) │
217
+ │ - one detail page per record; │
218
+ │ URLs from listing pages (count │
219
+ │ not measured) │
220
+ │ - selectors break on │
221
+ │ redesigns; prefer the methods │
222
+ │ above when usable │
223
+ │ 5 Browser no - - not needed: data is available │
224
+ │ without JavaScript │
225
+ │ │
226
+ ╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
227
+ ╭─ Rules ──────────────────────────────────────────────────────────────────────────────────────────╮
228
+ │ robots.txt found (HTTP 200) │
229
+ │ /gitlab allowed │
230
+ │ /sitemap.xml allowed │
231
+ │ Sitemap HTTP 404 │
232
+ │ https://job-boards.greenhouse.io/sitemap.xml │
233
+ ╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
234
+ ╭─ Data found ─────────────────────────────────────────────────────────────────────────────────────╮
235
+ │ Structured none │
236
+ │ greenhouse_api ok, 244 items of 244 │
237
+ │ https://boards-api.greenhouse.io/v1/boards/gitlab/jobs │
238
+ │ Rendering 3,931 characters of text, 52 links │
239
+ ╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
240
+ ╭──────────────────────────────────────────────────────────────────────────────────────────────────╮
241
+ │ Risks │
242
+ │ none │
243
+ │ │
244
+ │ Next steps │
245
+ │ - Check the site's terms of service for data collection and reuse │
246
+ │ - Write a starter profile with --profile profile.yaml and review it │
247
+ ╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
248
+
249
+ # Status Purpose ms URL
250
+ ───────────────────────────────────────────────────────────────────────────────────────────────
251
+ 1 200 robots 92 https://job-boards.greenhouse.io/robots.txt
252
+ 2 200 page 80 https://job-boards.greenhouse.io/gitlab
253
+ 3 404 sitemap 66 https://job-boards.greenhouse.io/sitemap.xml
254
+ 4 200 robots 290 https://boards-api.greenhouse.io/robots.txt
255
+ 5 200 feed:greenhouse_api 63 https://boards-api.greenhouse.io/v1/boards/gitlab/jobs
256
+ ```
257
+
258
+ **How to read it**
259
+
260
+ - **Verdict**: `GOOD FIT` (a usable method and no bot management in the way),
261
+ `POSSIBLE, WITH CARE` (bot management in front, a refused request, or only a
262
+ browser works), `NOT A FIT` (blocked, challenged, or disallowed by robots.txt).
263
+ - **Plan**: every method, cheapest first. *Recommended* is the first one that
264
+ can reach all N records; *usable* ones are fallbacks. "Req. for 1,000" is the
265
+ number of requests for 1,000 records (`--items`); here one API call returns
266
+ all 244 open jobs, against about 1,000 page requests for HTML parsing.
267
+ - **Rules**: what robots.txt allows for the paths the plan would use, plus the
268
+ sitemap it found. A disallowed path is never requested.
269
+ - **Data found**: the evidence: feeds that answered, schema.org records,
270
+ embedded JSON, and how much text the raw HTML holds.
271
+ - **Risks / Next steps**: what to watch and what to do; on a blocked site this
272
+ says to ask the owner for access or an export.
273
+ - **Request log**: every request the tool itself sent. The second robots.txt is
274
+ for the API host: each host's rules are read before its first URL.
275
+
276
+ ## What it detects
277
+
278
+ | Check | What it looks at | Why it matters |
279
+ |---|---|---|
280
+ | robots.txt and sitemaps | Rules for every path the plan would use, crawl-delay, request-rate; one sitemap or sitemap index (URL count, newest `lastmod`) | Decides what may be collected at all, the minimum delay, and how detail URLs can be found without crawling |
281
+ | Platform | Headers, cookies, generator tags, asset paths, the page URL: Shopify, WooCommerce, Magento 2, BigCommerce, Salesforce Commerce Cloud, Greenhouse, Lever, Ashby, CKAN, Socrata, Wix, Squarespace, Webflow, WordPress, Drupal, Hugo, Sphinx, Docusaurus, Next.js, Nuxt, Gatsby, Remix, SvelteKit, Astro, Angular, React, Vue, Lit | The platform says which public feeds exist and which page structure to expect |
282
+ | Public feeds and APIs | Shopify `/products.json` and `/products/<handle>.js`, WooCommerce Store API, WordPress REST, Greenhouse / Lever / Ashby job-board APIs, CKAN `package_search`, RSS / Atom / JSON feeds linked from the page; each is requested once and its shape checked | A working feed is usually the cheapest and most stable source, often all records in one or a few requests |
283
+ | Structured data | JSON-LD and microdata: Product, Offer, JobPosting, Article, Event, Recipe, LocalBusiness, ItemList and more, with which key fields are filled | Clean, typed fields straight from the HTML, no selectors to maintain |
284
+ | Embedded state | `__NEXT_DATA__`, Next.js flight data, Nuxt state, Apollo / Redux state, large `application/json` scripts | The full data model behind a page, often richer than what is shown |
285
+ | Raw HTML or browser | Visible text without JavaScript, empty app shells (`#root`, `#__next`), `<noscript>` warnings | Whether a plain HTTP client is enough or a headless browser is needed |
286
+ | Anti-bot vendor (detection only) | Headers, cookie names and block-page markers of Cloudflare, Akamai, DataDome, HUMAN/PerimeterX, Imperva, Kasada, AWS WAF, and a site's own CAPTCHA page; CDN-only vs bot management vs challenge | Sets a slower rate, caps requests per day, or says "not a fit". It never explains how to get past them |
287
+ | The plan | Methods cheapest first, polite rate, requests and hours for N records, risks, next steps | The decision you need before writing a scraper |
288
+
289
+ ## How it decides
290
+
291
+ <p align="center">
292
+ <picture>
293
+ <source media="(prefers-reduced-motion: reduce)" srcset="assets/ladder-still.svg">
294
+ <img src="assets/ladder.svg" width="640" alt="The method ladder, cheapest first: 1 Feed / API (products.json, job-board APIs, RSS), best on 5 of 30 sites; 2 Structured data (JSON-LD and microdata), best on 3; 3 Embedded JSON (__NEXT_DATA__, Nuxt state), best on 2; 4 HTML (CSS selectors on the raw page), best on 13; 5 Browser (headless, last resort), best on 1. 6 of 30 were not a fit. Counts from run 008-sites.">
295
+ </picture>
296
+ </p>
297
+
298
+ 1. **Method ladder.** Feed / API → structured data → embedded JSON → HTML →
299
+ browser. The first method that works and can reach all N records is
300
+ recommended. A feed that only holds the latest entries (an RSS feed of 24
301
+ posts) is listed but marked partial.
302
+ 2. **Polite rate.** At least 1 second between requests; more if robots.txt asks
303
+ for it (`Crawl-delay`, `Request-rate`). With bot management in front of the
304
+ site, or after a 429 during the run, 5 seconds and at most 5,000 requests a
305
+ day. One request at a time.
306
+ 3. **Request estimate.** Pages needed for N records at the method's page size
307
+ (250 per Shopify `products.json` page, 100 per WooCommerce or WordPress page,
308
+ 1,000 per CKAN page), plus the sitemap files needed to find detail pages.
309
+ Time is requests × delay. When a feed reports its total (163 products, 244
310
+ jobs), the estimate stops there.
311
+
312
+ The rules and thresholds are in [docs/architecture.md](docs/architecture.md)
313
+ and the reasons behind them in [docs/decisions.md](docs/decisions.md).
314
+
315
+ ## Results on 30 public sites
316
+
317
+ One homepage or public listing page per site, run from GitHub Actions on
318
+ 2026-11-04 (run [`008-sites`](checks/results/008-sites/)): independent stores,
319
+ company job boards, open-data portals, news, docs, and scraping practice sites.
320
+
321
+ | Measure | Run `008-sites` |
322
+ |---|---|
323
+ | Verdicts | **21** good fit · **3** with care · **6** not a fit |
324
+ | Recommended method | HTML 13 · Feed / API 5 · Structured data 3 · Embedded JSON 2 · Browser 1 |
325
+ | Platform named | 15 of the 24 sites that served a page |
326
+ | Platform vs labels written before the run | 13 of 14 matched (the miss is an outdated label, see the [FAQ](#faq)) |
327
+ | Anti-bot vendor seen | Cloudflare on 13 sites (challenge page on 4) · Akamai on 1 |
328
+ | Requests per site | 1 to 8, 122 in total; median time 4.5 s |
329
+
330
+ Full table, per-site evidence and notes: **[docs/results.md](docs/results.md)**.
331
+ Every report and every HTTP exchange of that run is committed under
332
+ `checks/results/008-sites/` and replayed offline by the test suite.
333
+
334
+ ## Rules it follows
335
+
336
+ - **robots.txt for every request.** Each host's robots.txt is read before its
337
+ first URL, and every URL is checked for the `scrapeplanner` user agent. If
338
+ robots.txt cannot be read (401, 403, 5xx, timeout), everything is treated as
339
+ disallowed. There is no option to ignore it.
340
+ - **At most 10 requests per site** by default, every redirect hop included
341
+ (`--max-requests` up to 50).
342
+ - **A delay between requests**: 1 second or more, or the site's crawl-delay.
343
+ - **A clear User-Agent**: `scrapeplanner/<version> (+https://github.com/siddharthkumarx/scrapeplanner)`.
344
+ - **Stops at the first block or challenge.** Any 401, 403, 429 or 503, or a
345
+ challenge page, ends the run; nothing more is sent.
346
+ - **Detection only, never evasion.** No proxies, no CAPTCHA solving, no logins,
347
+ no fingerprint tricks, no cookies sent back, no JavaScript executed, no
348
+ personal data collected. Pull requests that add any of these are declined
349
+ ([CONTRIBUTING.md](CONTRIBUTING.md)).
350
+
351
+ ## Output formats
352
+
353
+ - **`--json`**: the full report. Top-level sections: `page`, `robots`,
354
+ `sitemap`, `platform`, `antibot`, `structured`, `embedded`, `rendering`,
355
+ `feeds`, `plan`, plus the request log (`requests`, `skipped`, `stopped`). It
356
+ carries `schema_version`; the JSON Schema is
357
+ [docs/report.schema.json](docs/report.schema.json).
358
+ - **`--profile PATH`**: a starter `profile.yaml` for a collector: `method`,
359
+ `source.urls` (with pagination), `fields`, `rate`, `politeness` (user agent,
360
+ robots, stop rules), `estimate` and `notes`. Review it before use.
361
+
362
+ Field-by-field description: [docs/output-formats.md](docs/output-formats.md).
363
+
364
+ ## FAQ
365
+
366
+ **Is it legal to scrape a site?**
367
+ It depends on the country, the site's terms, and what the data is (personal
368
+ data has its own rules). This tool reads only public pages, follows
369
+ robots.txt and stops at the first refusal, and its report tells you to check
370
+ the terms of service. It is not legal advice.
371
+
372
+ **Does it bypass anti-bot protection?**
373
+ No. It names the vendor from public signals (headers, cookie names, block
374
+ pages) and plans around it: a slower rate, fewer pages, asking the owner, or
375
+ "not a fit". It contains no evasion code and gives no evasion advice.
376
+
377
+ **What does "not a fit" mean?**
378
+ The site blocked or challenged the tool, or robots.txt disallows the page.
379
+ The honest next step is to ask the owner for access, an API key or a data
380
+ export. In the results run, 6 of 30 sites were not a fit: 5 served a block or
381
+ challenge page and 1 disallows the page in robots.txt.
382
+
383
+ **How accurate is platform detection?**
384
+ In run `008-sites` it named a platform on 15 of the 24 sites that served a
385
+ page. 15 sites had an expected platform written down before the run; 14 of
386
+ them served a page and 13 matched. The one miss was a wrong label: Data.gov's
387
+ catalog was expected to be CKAN, but the page served shows no CKAN markers
388
+ (details in [docs/results.md](docs/results.md)). The other 9 served pages
389
+ were custom stacks or static sites with no public marker, and they are
390
+ reported as `unknown` rather than guessed. Every detector is also tested on
391
+ the 30 recorded sites.
392
+
393
+ **Does it run JavaScript or a browser?**
394
+ No. It reads the raw HTML and tells you when a browser would be needed.
395
+
396
+ **Why did a run stop early?**
397
+ It stops at the first 401, 403, 429, 503 or challenge page, even on an
398
+ optional probe. The report says which URL refused and why; the plan keeps what
399
+ was already learned.
400
+
401
+ **Can you collect the data for me?**
402
+ Yes, see below.
403
+
404
+ ### Known limits
405
+
406
+ - It looks at one page per run. Point it at the listing or detail page you
407
+ care about, not only the homepage.
408
+ - Estimates follow stated rules (page sizes, sitemap counts); a site can
409
+ paginate differently. The report shows the inputs.
410
+ - Results depend on where it runs from: these numbers come from GitHub Actions
411
+ runners, which some sites treat differently from a home connection.
412
+ - Platform detection covers the platforms listed above; anything else is
413
+ `unknown`.
414
+
415
+ ## Need the data collected for you?
416
+
417
+ I build and run polite, reliable collectors for e-commerce, job, news and
418
+ open-data sources. You get a free sample of 20-50 real rows before you pay.
419
+ **[hello@siddharthkumar.com](mailto:hello@siddharthkumar.com)** ·
420
+ [siddharthkumar.com](https://siddharthkumar.com)
421
+
422
+ ## Contributing
423
+
424
+ Bug reports, new platform signals and detection fixes are welcome; see
425
+ [CONTRIBUTING.md](CONTRIBUTING.md).
426
+
427
+ - **Architecture**: one polite fetcher, pure detectors (one module each), a
428
+ planner, three outputs. See [docs/architecture.md](docs/architecture.md).
429
+ - **Adding a detector signal**: add a row to the signal table at the top of the
430
+ module in `src/scrapeplanner/detectors/`, add a unit test with a minimal
431
+ input, run `./scripts/check.sh`.
432
+ - **Tests and golden files**: `uv run pytest` runs everything offline.
433
+ `tests/fixtures/` holds the recorded HTTP exchanges of the results run;
434
+ `tests/golden/` replays them through the real analyzer and compares every
435
+ report section. After an intended change, run
436
+ `uv run python scripts/update_golden.py` and explain the diff.
437
+ - **Live checks** run only in GitHub Actions: edit `checks/request.yml`, push,
438
+ and the workflow commits the results.
439
+ - **Releases**: tag `vX.Y.Z`; the release workflow tests, builds and publishes
440
+ to PyPI through Trusted Publishing (no tokens in the repository).
441
+
442
+ [Security policy](SECURITY.md) · [Changelog](CHANGELOG.md) ·
443
+ [Code of conduct](CODE_OF_CONDUCT.md) · [Results](docs/results.md)
444
+
445
+ ## License
446
+
447
+ MIT, see [LICENSE](LICENSE). Fonts and logos in the visuals are listed in
448
+ [assets/LICENSES.md](assets/LICENSES.md).
449
+
450
+ ---
451
+
452
+ <p align="center">Built and maintained by Siddharth Kumar · <a href="https://siddharthkumar.com">siddharthkumar.com</a></p>