scrapeplanner 0.1.0__tar.gz
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- scrapeplanner-0.1.0/.gitignore +14 -0
- scrapeplanner-0.1.0/CHANGELOG.md +44 -0
- scrapeplanner-0.1.0/LICENSE +21 -0
- scrapeplanner-0.1.0/PKG-INFO +452 -0
- scrapeplanner-0.1.0/README.md +418 -0
- scrapeplanner-0.1.0/pyproject.toml +108 -0
- scrapeplanner-0.1.0/src/scrapeplanner/__init__.py +5 -0
- scrapeplanner-0.1.0/src/scrapeplanner/__main__.py +5 -0
- scrapeplanner-0.1.0/src/scrapeplanner/analyze.py +389 -0
- scrapeplanner-0.1.0/src/scrapeplanner/cli.py +108 -0
- scrapeplanner-0.1.0/src/scrapeplanner/config.py +25 -0
- scrapeplanner-0.1.0/src/scrapeplanner/detectors/__init__.py +0 -0
- scrapeplanner-0.1.0/src/scrapeplanner/detectors/antibot.py +259 -0
- scrapeplanner-0.1.0/src/scrapeplanner/detectors/embedded.py +142 -0
- scrapeplanner-0.1.0/src/scrapeplanner/detectors/feeds.py +259 -0
- scrapeplanner-0.1.0/src/scrapeplanner/detectors/platform.py +349 -0
- scrapeplanner-0.1.0/src/scrapeplanner/detectors/rendering.py +65 -0
- scrapeplanner-0.1.0/src/scrapeplanner/detectors/robots.py +40 -0
- scrapeplanner-0.1.0/src/scrapeplanner/detectors/sitemaps.py +119 -0
- scrapeplanner-0.1.0/src/scrapeplanner/detectors/structured.py +321 -0
- scrapeplanner-0.1.0/src/scrapeplanner/fetch.py +378 -0
- scrapeplanner-0.1.0/src/scrapeplanner/html.py +131 -0
- scrapeplanner-0.1.0/src/scrapeplanner/models.py +370 -0
- scrapeplanner-0.1.0/src/scrapeplanner/output/__init__.py +0 -0
- scrapeplanner-0.1.0/src/scrapeplanner/output/json_out.py +9 -0
- scrapeplanner-0.1.0/src/scrapeplanner/output/profile.py +80 -0
- scrapeplanner-0.1.0/src/scrapeplanner/output/terminal.py +293 -0
- scrapeplanner-0.1.0/src/scrapeplanner/planner.py +465 -0
- scrapeplanner-0.1.0/src/scrapeplanner/py.typed +0 -0
- scrapeplanner-0.1.0/src/scrapeplanner/record.py +135 -0
|
@@ -0,0 +1,44 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
All notable changes are listed here. The format follows
|
|
4
|
+
[Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and the project uses
|
|
5
|
+
[Semantic Versioning](https://semver.org/). The JSON output has its own
|
|
6
|
+
`schema_version`; a breaking change to it is a major version.
|
|
7
|
+
|
|
8
|
+
## [Unreleased]
|
|
9
|
+
|
|
10
|
+
## [0.1.0] - 2026-11-22
|
|
11
|
+
|
|
12
|
+
First release on PyPI. `scrapeplanner URL` reads a site's robots.txt and
|
|
13
|
+
first page, probes the public feeds its platform is known to have, names any
|
|
14
|
+
anti-bot vendor, and prints a collection plan: methods from cheapest to most
|
|
15
|
+
expensive, a polite rate, and the number of requests N records will take. It
|
|
16
|
+
detects and recommends only; it never tries to get past a block.
|
|
17
|
+
|
|
18
|
+
### Added
|
|
19
|
+
|
|
20
|
+
- `scrapeplanner URL`: terminal report, `--json`, `--profile PATH`.
|
|
21
|
+
- Polite fetcher: robots.txt read first and checked for every URL, 10 requests
|
|
22
|
+
per run by default (every redirect hop counts), at least 1 s between
|
|
23
|
+
requests or the robots.txt crawl-delay, clear User-Agent, stop at the
|
|
24
|
+
first block or challenge, no cookie jar, no proxies.
|
|
25
|
+
- Detectors: robots.txt, sitemaps, platform (Shopify, WooCommerce, Magento,
|
|
26
|
+
BigCommerce, Wix, Squarespace, Next.js, Nuxt), public feeds (Shopify
|
|
27
|
+
products.json and product .js, WooCommerce Store API, WordPress REST,
|
|
28
|
+
RSS/Atom/JSON feeds linked from the page), JSON-LD and microdata,
|
|
29
|
+
embedded state, rendering (raw HTML or browser), anti-bot vendor
|
|
30
|
+
(Cloudflare, Akamai, DataDome, HUMAN, Imperva, Kasada).
|
|
31
|
+
- Planner: methods cheapest first, polite rate, request and time estimate
|
|
32
|
+
for N items, risks and next steps.
|
|
33
|
+
- `--record DIR` saves every exchange for offline replay.
|
|
34
|
+
- Platforms: WordPress, Drupal, Webflow, Salesforce Commerce Cloud, Greenhouse,
|
|
35
|
+
Lever, Ashby, CKAN, Socrata, Hugo, Sphinx, Docusaurus, Gatsby, Remix,
|
|
36
|
+
SvelteKit, Astro, Angular, React, Vue, Lit. "not checked" when no page was
|
|
37
|
+
served.
|
|
38
|
+
- Feeds: Greenhouse, Lever and Ashby job-board APIs; CKAN `package_search`.
|
|
39
|
+
- Anti-bot: AWS WAF, and a site's own CAPTCHA / "are you a robot" page.
|
|
40
|
+
- Stacked report layout for terminals narrower than 80 columns.
|
|
41
|
+
- JSON Schema of the `--json` output in `docs/report.schema.json`.
|
|
42
|
+
|
|
43
|
+
[Unreleased]: https://github.com/siddharthkumarx/scrapeplanner/compare/v0.1.0...HEAD
|
|
44
|
+
[0.1.0]: https://github.com/siddharthkumarx/scrapeplanner/tree/v0.1.0
|
|
@@ -0,0 +1,21 @@
|
|
|
1
|
+
MIT License
|
|
2
|
+
|
|
3
|
+
Copyright (c) 2026 Siddharth Kumar
|
|
4
|
+
|
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
|
7
|
+
in the Software without restriction, including without limitation the rights
|
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
|
10
|
+
furnished to do so, subject to the following conditions:
|
|
11
|
+
|
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
|
13
|
+
copies or substantial portions of the Software.
|
|
14
|
+
|
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
|
21
|
+
SOFTWARE.
|
|
@@ -0,0 +1,452 @@
|
|
|
1
|
+
Metadata-Version: 2.5
|
|
2
|
+
Name: scrapeplanner
|
|
3
|
+
Version: 0.1.0
|
|
4
|
+
Summary: Give it a URL; it reports the cheapest reliable, polite way to collect that site's data.
|
|
5
|
+
Project-URL: Homepage, https://github.com/siddharthkumarx/scrapeplanner
|
|
6
|
+
Project-URL: Issues, https://github.com/siddharthkumarx/scrapeplanner/issues
|
|
7
|
+
Project-URL: Changelog, https://github.com/siddharthkumarx/scrapeplanner/blob/main/CHANGELOG.md
|
|
8
|
+
Author: Siddharth Kumar
|
|
9
|
+
License-Expression: MIT
|
|
10
|
+
License-File: LICENSE
|
|
11
|
+
Keywords: cli,crawling,json-ld,robots.txt,sitemap,structured data,web scraping
|
|
12
|
+
Classifier: Development Status :: 4 - Beta
|
|
13
|
+
Classifier: Environment :: Console
|
|
14
|
+
Classifier: Intended Audience :: Developers
|
|
15
|
+
Classifier: Operating System :: OS Independent
|
|
16
|
+
Classifier: Programming Language :: Python :: 3
|
|
17
|
+
Classifier: Programming Language :: Python :: 3 :: Only
|
|
18
|
+
Classifier: Programming Language :: Python :: 3.11
|
|
19
|
+
Classifier: Programming Language :: Python :: 3.12
|
|
20
|
+
Classifier: Programming Language :: Python :: 3.13
|
|
21
|
+
Classifier: Programming Language :: Python :: 3.14
|
|
22
|
+
Classifier: Topic :: Internet :: WWW/HTTP
|
|
23
|
+
Classifier: Topic :: Internet :: WWW/HTTP :: Indexing/Search
|
|
24
|
+
Classifier: Typing :: Typed
|
|
25
|
+
Requires-Python: >=3.11
|
|
26
|
+
Requires-Dist: httpx<1,>=0.28
|
|
27
|
+
Requires-Dist: protego>=0.7
|
|
28
|
+
Requires-Dist: pydantic<3,>=2.14
|
|
29
|
+
Requires-Dist: pyyaml>=6.0.3
|
|
30
|
+
Requires-Dist: rich>=15
|
|
31
|
+
Requires-Dist: selectolax<2,>=1.0
|
|
32
|
+
Requires-Dist: typer>=0.27
|
|
33
|
+
Description-Content-Type: text/markdown
|
|
34
|
+
|
|
35
|
+
<p align="center">
|
|
36
|
+
<picture>
|
|
37
|
+
<source media="(prefers-reduced-motion: reduce)" srcset="assets/hero-still.svg">
|
|
38
|
+
<img src="assets/hero.svg" width="720" alt="A terminal runs 'scrapeplanner https://www.deathwishcoffee.com/'. The report says GOOD FIT, best method Public feed / JSON API, platform Shopify, anti-bot Cloudflare (CDN only), polite rate 1 request every 1s, 5 of 10 requests used. It then scrolls to the plan, which recommends Shopify /products.json: 1 request returns all 163 products, against 1,008 requests for HTML parsing. Recorded run 008-sites.">
|
|
39
|
+
</picture>
|
|
40
|
+
</p>
|
|
41
|
+
|
|
42
|
+
<h1 align="center">scrapeplanner</h1>
|
|
43
|
+
|
|
44
|
+
<p align="center"><b>Point it at a URL. It tells you the cheapest polite way to get the data.</b></p>
|
|
45
|
+
|
|
46
|
+
<p align="center">
|
|
47
|
+
<a href="https://pypi.org/project/scrapeplanner/"><img alt="PyPI version" src="https://img.shields.io/pypi/v/scrapeplanner?style=flat&label=PyPI"></a>
|
|
48
|
+
<a href="https://github.com/siddharthkumarx/scrapeplanner/actions/workflows/ci.yml"><img alt="CI" src="https://img.shields.io/github/actions/workflow/status/siddharthkumarx/scrapeplanner/ci.yml?branch=main&style=flat&label=CI"></a>
|
|
49
|
+
<a href="LICENSE"><img alt="License: MIT" src="https://img.shields.io/badge/license-MIT-2FBFAE?style=flat"></a>
|
|
50
|
+
</p>
|
|
51
|
+
|
|
52
|
+
<p align="center">
|
|
53
|
+
<a href="#install">Install</a> ·
|
|
54
|
+
<a href="#quick-start">Quick start</a> ·
|
|
55
|
+
<a href="#example-report">Example report</a> ·
|
|
56
|
+
<a href="#what-it-detects">What it detects</a> ·
|
|
57
|
+
<a href="#how-it-decides">How it decides</a> ·
|
|
58
|
+
<a href="#results-on-30-public-sites">Results</a> ·
|
|
59
|
+
<a href="#faq">FAQ</a>
|
|
60
|
+
</p>
|
|
61
|
+
|
|
62
|
+
## Why
|
|
63
|
+
|
|
64
|
+
Most of the time lost on a scraping job goes into the wrong method: a
|
|
65
|
+
browser fleet for a store that publishes `products.json`, CSS selectors for a
|
|
66
|
+
job board with a public API, or a week of work on a site that blocks every
|
|
67
|
+
automated request anyway. Those answers are usually on the first page.
|
|
68
|
+
|
|
69
|
+
`scrapeplanner` reads robots.txt and the page, probes the public feeds the
|
|
70
|
+
platform is known to have, names any anti-bot vendor, and prints a plan:
|
|
71
|
+
methods from cheapest to most expensive, a polite rate, and how many requests
|
|
72
|
+
N records will take. On 30 public sites it needed 1 to 8 requests per site
|
|
73
|
+
(122 in total, never more than its cap of 10) and a median of 4.5 seconds
|
|
74
|
+
([results](docs/results.md)).
|
|
75
|
+
|
|
76
|
+
It is a planning tool. It detects and recommends; it never tries to get past
|
|
77
|
+
anything.
|
|
78
|
+
|
|
79
|
+
## Install
|
|
80
|
+
|
|
81
|
+
`scrapeplanner` is on [PyPI](https://pypi.org/project/scrapeplanner/) and
|
|
82
|
+
needs Python 3.11 or newer. Install it as a standalone command with pipx or
|
|
83
|
+
uv, or into an environment with pip:
|
|
84
|
+
|
|
85
|
+
```bash
|
|
86
|
+
pipx install scrapeplanner
|
|
87
|
+
uv tool install scrapeplanner
|
|
88
|
+
pip install scrapeplanner
|
|
89
|
+
```
|
|
90
|
+
|
|
91
|
+
The latest development version, from source:
|
|
92
|
+
|
|
93
|
+
```bash
|
|
94
|
+
pipx install git+https://github.com/siddharthkumarx/scrapeplanner
|
|
95
|
+
```
|
|
96
|
+
|
|
97
|
+
## Quick start
|
|
98
|
+
|
|
99
|
+
```bash
|
|
100
|
+
scrapeplanner https://www.deathwishcoffee.com/ # terminal report
|
|
101
|
+
scrapeplanner https://www.deathwishcoffee.com/ --json > report.json # full report as JSON
|
|
102
|
+
scrapeplanner https://www.deathwishcoffee.com/ --profile profile.yaml # starter config
|
|
103
|
+
```
|
|
104
|
+
|
|
105
|
+
The outputs below come from the recorded run of that site in
|
|
106
|
+
[`checks/results/008-sites/deathwish-coffee/`](checks/results/008-sites/deathwish-coffee/).
|
|
107
|
+
|
|
108
|
+
**Report** (first panel; the full report is in the hero above and in the [example](#example-report)):
|
|
109
|
+
|
|
110
|
+
```text
|
|
111
|
+
╭─ scrapeplanner 0.1.0.dev0 · https://www.deathwishcoffee.com/ ──────────────────────────────────╮
|
|
112
|
+
│ Verdict GOOD FIT │
|
|
113
|
+
│ Best method Public feed / JSON API │
|
|
114
|
+
│ Platform Shopify │
|
|
115
|
+
│ Anti-bot Cloudflare (CDN only) │
|
|
116
|
+
│ Rendering data in the raw HTML │
|
|
117
|
+
│ Polite rate 1 request every 1s │
|
|
118
|
+
│ This run 5/10 requests, 5.2s │
|
|
119
|
+
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
|
|
120
|
+
```
|
|
121
|
+
|
|
122
|
+
**`--json`**, a few of the fields (`jq` filter shown):
|
|
123
|
+
|
|
124
|
+
```bash
|
|
125
|
+
scrapeplanner https://www.deathwishcoffee.com/ --json \
|
|
126
|
+
| jq '{platform: .platform.name, verdict: .plan.verdict, recommended: .plan.recommended,
|
|
127
|
+
requests_used, feed: (.feeds.candidates[0] | {kind, status, items_seen, url})}'
|
|
128
|
+
```
|
|
129
|
+
|
|
130
|
+
```json
|
|
131
|
+
{
|
|
132
|
+
"platform": "shopify",
|
|
133
|
+
"verdict": "good",
|
|
134
|
+
"recommended": "feed_api",
|
|
135
|
+
"requests_used": 5,
|
|
136
|
+
"feed": {
|
|
137
|
+
"kind": "shopify_products_json",
|
|
138
|
+
"status": "ok",
|
|
139
|
+
"items_seen": 163,
|
|
140
|
+
"url": "https://www.deathwishcoffee.com/products.json?limit=250"
|
|
141
|
+
}
|
|
142
|
+
}
|
|
143
|
+
```
|
|
144
|
+
|
|
145
|
+
**`--profile profile.yaml`** (two parts shortened, marked `# ...`):
|
|
146
|
+
|
|
147
|
+
```yaml
|
|
148
|
+
# Starter profile written by scrapeplanner. Review before use.
|
|
149
|
+
# Every value comes from one analysis run; re-run to refresh it.
|
|
150
|
+
profile_version: 1
|
|
151
|
+
generated_by: scrapeplanner 0.1.0.dev0
|
|
152
|
+
generated_at: '2026-11-04T20:52:19+00:00'
|
|
153
|
+
site: https://www.deathwishcoffee.com
|
|
154
|
+
verdict: good
|
|
155
|
+
method: feed_api
|
|
156
|
+
source:
|
|
157
|
+
urls:
|
|
158
|
+
- https://www.deathwishcoffee.com/products.json?limit=250&page={page}
|
|
159
|
+
pagination:
|
|
160
|
+
param: page
|
|
161
|
+
start: 1
|
|
162
|
+
stop_when: empty page
|
|
163
|
+
fields:
|
|
164
|
+
- id
|
|
165
|
+
- title
|
|
166
|
+
- handle
|
|
167
|
+
# ... 10 more fields ...
|
|
168
|
+
rate:
|
|
169
|
+
delay_seconds: 1.0
|
|
170
|
+
concurrency: 1
|
|
171
|
+
max_requests_per_day: null
|
|
172
|
+
reason: default 1s between requests
|
|
173
|
+
# ... politeness: user agent, robots.txt, stop rules ...
|
|
174
|
+
estimate:
|
|
175
|
+
items: 1000
|
|
176
|
+
requests: 1
|
|
177
|
+
hours: 0.0
|
|
178
|
+
notes:
|
|
179
|
+
- 'the store lists 163 products: one request returns all of them'
|
|
180
|
+
```
|
|
181
|
+
|
|
182
|
+
Useful options: `--items N` (records to estimate for, default 1000),
|
|
183
|
+
`--delay S` (seconds between requests, minimum 1), `--max-requests N`
|
|
184
|
+
(default 10, hard cap 50), `--timeout S`. Run `scrapeplanner --help` for all.
|
|
185
|
+
|
|
186
|
+
## Example report
|
|
187
|
+
|
|
188
|
+
GitLab's public job board on Greenhouse, from the recorded run
|
|
189
|
+
[`checks/results/008-sites/gitlab-greenhouse/report.txt`](checks/results/008-sites/gitlab-greenhouse/report.txt)
|
|
190
|
+
(100-column terminal; scroll sideways on a phone):
|
|
191
|
+
|
|
192
|
+
```text
|
|
193
|
+
╭─ scrapeplanner 0.1.0.dev0 · https://job-boards.greenhouse.io/gitlab ───────────────────────────╮
|
|
194
|
+
│ Verdict GOOD FIT │
|
|
195
|
+
│ Best method Public feed / JSON API │
|
|
196
|
+
│ Platform Greenhouse (+ Remix) │
|
|
197
|
+
│ Anti-bot none seen │
|
|
198
|
+
│ Rendering data in the raw HTML │
|
|
199
|
+
│ Polite rate 1 request every 1s │
|
|
200
|
+
│ This run 5/10 requests, 4.7s │
|
|
201
|
+
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
|
|
202
|
+
╭─ Plan (cheapest first) ──────────────────────────────────────────────────────────────────────────╮
|
|
203
|
+
│ │
|
|
204
|
+
│ # Method Status Req. for 1,000 Time Details │
|
|
205
|
+
│ ────────────────────────────────────────────────────────────────────────────────────────────── │
|
|
206
|
+
│ 1 Feed / API recommended 1 < 1 min Greenhouse job-board API: │
|
|
207
|
+
│ every open job with its fields │
|
|
208
|
+
│ - one request returns all 244 │
|
|
209
|
+
│ open jobs; the board lists no │
|
|
210
|
+
│ more │
|
|
211
|
+
│ 2 Structured data no - - no schema.org records on this │
|
|
212
|
+
│ page (types seen: none) │
|
|
213
|
+
│ 3 Embedded JSON no - - no large inline state │
|
|
214
|
+
│ 4 HTML usable 1,000 17 min CSS selectors on │
|
|
215
|
+
│ server-rendered HTML (3,931 │
|
|
216
|
+
│ chars of text) │
|
|
217
|
+
│ - one detail page per record; │
|
|
218
|
+
│ URLs from listing pages (count │
|
|
219
|
+
│ not measured) │
|
|
220
|
+
│ - selectors break on │
|
|
221
|
+
│ redesigns; prefer the methods │
|
|
222
|
+
│ above when usable │
|
|
223
|
+
│ 5 Browser no - - not needed: data is available │
|
|
224
|
+
│ without JavaScript │
|
|
225
|
+
│ │
|
|
226
|
+
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
|
|
227
|
+
╭─ Rules ──────────────────────────────────────────────────────────────────────────────────────────╮
|
|
228
|
+
│ robots.txt found (HTTP 200) │
|
|
229
|
+
│ /gitlab allowed │
|
|
230
|
+
│ /sitemap.xml allowed │
|
|
231
|
+
│ Sitemap HTTP 404 │
|
|
232
|
+
│ https://job-boards.greenhouse.io/sitemap.xml │
|
|
233
|
+
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
|
|
234
|
+
╭─ Data found ─────────────────────────────────────────────────────────────────────────────────────╮
|
|
235
|
+
│ Structured none │
|
|
236
|
+
│ greenhouse_api ok, 244 items of 244 │
|
|
237
|
+
│ https://boards-api.greenhouse.io/v1/boards/gitlab/jobs │
|
|
238
|
+
│ Rendering 3,931 characters of text, 52 links │
|
|
239
|
+
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
|
|
240
|
+
╭──────────────────────────────────────────────────────────────────────────────────────────────────╮
|
|
241
|
+
│ Risks │
|
|
242
|
+
│ none │
|
|
243
|
+
│ │
|
|
244
|
+
│ Next steps │
|
|
245
|
+
│ - Check the site's terms of service for data collection and reuse │
|
|
246
|
+
│ - Write a starter profile with --profile profile.yaml and review it │
|
|
247
|
+
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
|
|
248
|
+
|
|
249
|
+
# Status Purpose ms URL
|
|
250
|
+
───────────────────────────────────────────────────────────────────────────────────────────────
|
|
251
|
+
1 200 robots 92 https://job-boards.greenhouse.io/robots.txt
|
|
252
|
+
2 200 page 80 https://job-boards.greenhouse.io/gitlab
|
|
253
|
+
3 404 sitemap 66 https://job-boards.greenhouse.io/sitemap.xml
|
|
254
|
+
4 200 robots 290 https://boards-api.greenhouse.io/robots.txt
|
|
255
|
+
5 200 feed:greenhouse_api 63 https://boards-api.greenhouse.io/v1/boards/gitlab/jobs
|
|
256
|
+
```
|
|
257
|
+
|
|
258
|
+
**How to read it**
|
|
259
|
+
|
|
260
|
+
- **Verdict**: `GOOD FIT` (a usable method and no bot management in the way),
|
|
261
|
+
`POSSIBLE, WITH CARE` (bot management in front, a refused request, or only a
|
|
262
|
+
browser works), `NOT A FIT` (blocked, challenged, or disallowed by robots.txt).
|
|
263
|
+
- **Plan**: every method, cheapest first. *Recommended* is the first one that
|
|
264
|
+
can reach all N records; *usable* ones are fallbacks. "Req. for 1,000" is the
|
|
265
|
+
number of requests for 1,000 records (`--items`); here one API call returns
|
|
266
|
+
all 244 open jobs, against about 1,000 page requests for HTML parsing.
|
|
267
|
+
- **Rules**: what robots.txt allows for the paths the plan would use, plus the
|
|
268
|
+
sitemap it found. A disallowed path is never requested.
|
|
269
|
+
- **Data found**: the evidence: feeds that answered, schema.org records,
|
|
270
|
+
embedded JSON, and how much text the raw HTML holds.
|
|
271
|
+
- **Risks / Next steps**: what to watch and what to do; on a blocked site this
|
|
272
|
+
says to ask the owner for access or an export.
|
|
273
|
+
- **Request log**: every request the tool itself sent. The second robots.txt is
|
|
274
|
+
for the API host: each host's rules are read before its first URL.
|
|
275
|
+
|
|
276
|
+
## What it detects
|
|
277
|
+
|
|
278
|
+
| Check | What it looks at | Why it matters |
|
|
279
|
+
|---|---|---|
|
|
280
|
+
| robots.txt and sitemaps | Rules for every path the plan would use, crawl-delay, request-rate; one sitemap or sitemap index (URL count, newest `lastmod`) | Decides what may be collected at all, the minimum delay, and how detail URLs can be found without crawling |
|
|
281
|
+
| Platform | Headers, cookies, generator tags, asset paths, the page URL: Shopify, WooCommerce, Magento 2, BigCommerce, Salesforce Commerce Cloud, Greenhouse, Lever, Ashby, CKAN, Socrata, Wix, Squarespace, Webflow, WordPress, Drupal, Hugo, Sphinx, Docusaurus, Next.js, Nuxt, Gatsby, Remix, SvelteKit, Astro, Angular, React, Vue, Lit | The platform says which public feeds exist and which page structure to expect |
|
|
282
|
+
| Public feeds and APIs | Shopify `/products.json` and `/products/<handle>.js`, WooCommerce Store API, WordPress REST, Greenhouse / Lever / Ashby job-board APIs, CKAN `package_search`, RSS / Atom / JSON feeds linked from the page; each is requested once and its shape checked | A working feed is usually the cheapest and most stable source, often all records in one or a few requests |
|
|
283
|
+
| Structured data | JSON-LD and microdata: Product, Offer, JobPosting, Article, Event, Recipe, LocalBusiness, ItemList and more, with which key fields are filled | Clean, typed fields straight from the HTML, no selectors to maintain |
|
|
284
|
+
| Embedded state | `__NEXT_DATA__`, Next.js flight data, Nuxt state, Apollo / Redux state, large `application/json` scripts | The full data model behind a page, often richer than what is shown |
|
|
285
|
+
| Raw HTML or browser | Visible text without JavaScript, empty app shells (`#root`, `#__next`), `<noscript>` warnings | Whether a plain HTTP client is enough or a headless browser is needed |
|
|
286
|
+
| Anti-bot vendor (detection only) | Headers, cookie names and block-page markers of Cloudflare, Akamai, DataDome, HUMAN/PerimeterX, Imperva, Kasada, AWS WAF, and a site's own CAPTCHA page; CDN-only vs bot management vs challenge | Sets a slower rate, caps requests per day, or says "not a fit". It never explains how to get past them |
|
|
287
|
+
| The plan | Methods cheapest first, polite rate, requests and hours for N records, risks, next steps | The decision you need before writing a scraper |
|
|
288
|
+
|
|
289
|
+
## How it decides
|
|
290
|
+
|
|
291
|
+
<p align="center">
|
|
292
|
+
<picture>
|
|
293
|
+
<source media="(prefers-reduced-motion: reduce)" srcset="assets/ladder-still.svg">
|
|
294
|
+
<img src="assets/ladder.svg" width="640" alt="The method ladder, cheapest first: 1 Feed / API (products.json, job-board APIs, RSS), best on 5 of 30 sites; 2 Structured data (JSON-LD and microdata), best on 3; 3 Embedded JSON (__NEXT_DATA__, Nuxt state), best on 2; 4 HTML (CSS selectors on the raw page), best on 13; 5 Browser (headless, last resort), best on 1. 6 of 30 were not a fit. Counts from run 008-sites.">
|
|
295
|
+
</picture>
|
|
296
|
+
</p>
|
|
297
|
+
|
|
298
|
+
1. **Method ladder.** Feed / API → structured data → embedded JSON → HTML →
|
|
299
|
+
browser. The first method that works and can reach all N records is
|
|
300
|
+
recommended. A feed that only holds the latest entries (an RSS feed of 24
|
|
301
|
+
posts) is listed but marked partial.
|
|
302
|
+
2. **Polite rate.** At least 1 second between requests; more if robots.txt asks
|
|
303
|
+
for it (`Crawl-delay`, `Request-rate`). With bot management in front of the
|
|
304
|
+
site, or after a 429 during the run, 5 seconds and at most 5,000 requests a
|
|
305
|
+
day. One request at a time.
|
|
306
|
+
3. **Request estimate.** Pages needed for N records at the method's page size
|
|
307
|
+
(250 per Shopify `products.json` page, 100 per WooCommerce or WordPress page,
|
|
308
|
+
1,000 per CKAN page), plus the sitemap files needed to find detail pages.
|
|
309
|
+
Time is requests × delay. When a feed reports its total (163 products, 244
|
|
310
|
+
jobs), the estimate stops there.
|
|
311
|
+
|
|
312
|
+
The rules and thresholds are in [docs/architecture.md](docs/architecture.md)
|
|
313
|
+
and the reasons behind them in [docs/decisions.md](docs/decisions.md).
|
|
314
|
+
|
|
315
|
+
## Results on 30 public sites
|
|
316
|
+
|
|
317
|
+
One homepage or public listing page per site, run from GitHub Actions on
|
|
318
|
+
2026-11-04 (run [`008-sites`](checks/results/008-sites/)): independent stores,
|
|
319
|
+
company job boards, open-data portals, news, docs, and scraping practice sites.
|
|
320
|
+
|
|
321
|
+
| Measure | Run `008-sites` |
|
|
322
|
+
|---|---|
|
|
323
|
+
| Verdicts | **21** good fit · **3** with care · **6** not a fit |
|
|
324
|
+
| Recommended method | HTML 13 · Feed / API 5 · Structured data 3 · Embedded JSON 2 · Browser 1 |
|
|
325
|
+
| Platform named | 15 of the 24 sites that served a page |
|
|
326
|
+
| Platform vs labels written before the run | 13 of 14 matched (the miss is an outdated label, see the [FAQ](#faq)) |
|
|
327
|
+
| Anti-bot vendor seen | Cloudflare on 13 sites (challenge page on 4) · Akamai on 1 |
|
|
328
|
+
| Requests per site | 1 to 8, 122 in total; median time 4.5 s |
|
|
329
|
+
|
|
330
|
+
Full table, per-site evidence and notes: **[docs/results.md](docs/results.md)**.
|
|
331
|
+
Every report and every HTTP exchange of that run is committed under
|
|
332
|
+
`checks/results/008-sites/` and replayed offline by the test suite.
|
|
333
|
+
|
|
334
|
+
## Rules it follows
|
|
335
|
+
|
|
336
|
+
- **robots.txt for every request.** Each host's robots.txt is read before its
|
|
337
|
+
first URL, and every URL is checked for the `scrapeplanner` user agent. If
|
|
338
|
+
robots.txt cannot be read (401, 403, 5xx, timeout), everything is treated as
|
|
339
|
+
disallowed. There is no option to ignore it.
|
|
340
|
+
- **At most 10 requests per site** by default, every redirect hop included
|
|
341
|
+
(`--max-requests` up to 50).
|
|
342
|
+
- **A delay between requests**: 1 second or more, or the site's crawl-delay.
|
|
343
|
+
- **A clear User-Agent**: `scrapeplanner/<version> (+https://github.com/siddharthkumarx/scrapeplanner)`.
|
|
344
|
+
- **Stops at the first block or challenge.** Any 401, 403, 429 or 503, or a
|
|
345
|
+
challenge page, ends the run; nothing more is sent.
|
|
346
|
+
- **Detection only, never evasion.** No proxies, no CAPTCHA solving, no logins,
|
|
347
|
+
no fingerprint tricks, no cookies sent back, no JavaScript executed, no
|
|
348
|
+
personal data collected. Pull requests that add any of these are declined
|
|
349
|
+
([CONTRIBUTING.md](CONTRIBUTING.md)).
|
|
350
|
+
|
|
351
|
+
## Output formats
|
|
352
|
+
|
|
353
|
+
- **`--json`**: the full report. Top-level sections: `page`, `robots`,
|
|
354
|
+
`sitemap`, `platform`, `antibot`, `structured`, `embedded`, `rendering`,
|
|
355
|
+
`feeds`, `plan`, plus the request log (`requests`, `skipped`, `stopped`). It
|
|
356
|
+
carries `schema_version`; the JSON Schema is
|
|
357
|
+
[docs/report.schema.json](docs/report.schema.json).
|
|
358
|
+
- **`--profile PATH`**: a starter `profile.yaml` for a collector: `method`,
|
|
359
|
+
`source.urls` (with pagination), `fields`, `rate`, `politeness` (user agent,
|
|
360
|
+
robots, stop rules), `estimate` and `notes`. Review it before use.
|
|
361
|
+
|
|
362
|
+
Field-by-field description: [docs/output-formats.md](docs/output-formats.md).
|
|
363
|
+
|
|
364
|
+
## FAQ
|
|
365
|
+
|
|
366
|
+
**Is it legal to scrape a site?**
|
|
367
|
+
It depends on the country, the site's terms, and what the data is (personal
|
|
368
|
+
data has its own rules). This tool reads only public pages, follows
|
|
369
|
+
robots.txt and stops at the first refusal, and its report tells you to check
|
|
370
|
+
the terms of service. It is not legal advice.
|
|
371
|
+
|
|
372
|
+
**Does it bypass anti-bot protection?**
|
|
373
|
+
No. It names the vendor from public signals (headers, cookie names, block
|
|
374
|
+
pages) and plans around it: a slower rate, fewer pages, asking the owner, or
|
|
375
|
+
"not a fit". It contains no evasion code and gives no evasion advice.
|
|
376
|
+
|
|
377
|
+
**What does "not a fit" mean?**
|
|
378
|
+
The site blocked or challenged the tool, or robots.txt disallows the page.
|
|
379
|
+
The honest next step is to ask the owner for access, an API key or a data
|
|
380
|
+
export. In the results run, 6 of 30 sites were not a fit: 5 served a block or
|
|
381
|
+
challenge page and 1 disallows the page in robots.txt.
|
|
382
|
+
|
|
383
|
+
**How accurate is platform detection?**
|
|
384
|
+
In run `008-sites` it named a platform on 15 of the 24 sites that served a
|
|
385
|
+
page. 15 sites had an expected platform written down before the run; 14 of
|
|
386
|
+
them served a page and 13 matched. The one miss was a wrong label: Data.gov's
|
|
387
|
+
catalog was expected to be CKAN, but the page served shows no CKAN markers
|
|
388
|
+
(details in [docs/results.md](docs/results.md)). The other 9 served pages
|
|
389
|
+
were custom stacks or static sites with no public marker, and they are
|
|
390
|
+
reported as `unknown` rather than guessed. Every detector is also tested on
|
|
391
|
+
the 30 recorded sites.
|
|
392
|
+
|
|
393
|
+
**Does it run JavaScript or a browser?**
|
|
394
|
+
No. It reads the raw HTML and tells you when a browser would be needed.
|
|
395
|
+
|
|
396
|
+
**Why did a run stop early?**
|
|
397
|
+
It stops at the first 401, 403, 429, 503 or challenge page, even on an
|
|
398
|
+
optional probe. The report says which URL refused and why; the plan keeps what
|
|
399
|
+
was already learned.
|
|
400
|
+
|
|
401
|
+
**Can you collect the data for me?**
|
|
402
|
+
Yes, see below.
|
|
403
|
+
|
|
404
|
+
### Known limits
|
|
405
|
+
|
|
406
|
+
- It looks at one page per run. Point it at the listing or detail page you
|
|
407
|
+
care about, not only the homepage.
|
|
408
|
+
- Estimates follow stated rules (page sizes, sitemap counts); a site can
|
|
409
|
+
paginate differently. The report shows the inputs.
|
|
410
|
+
- Results depend on where it runs from: these numbers come from GitHub Actions
|
|
411
|
+
runners, which some sites treat differently from a home connection.
|
|
412
|
+
- Platform detection covers the platforms listed above; anything else is
|
|
413
|
+
`unknown`.
|
|
414
|
+
|
|
415
|
+
## Need the data collected for you?
|
|
416
|
+
|
|
417
|
+
I build and run polite, reliable collectors for e-commerce, job, news and
|
|
418
|
+
open-data sources. You get a free sample of 20-50 real rows before you pay.
|
|
419
|
+
**[hello@siddharthkumar.com](mailto:hello@siddharthkumar.com)** ·
|
|
420
|
+
[siddharthkumar.com](https://siddharthkumar.com)
|
|
421
|
+
|
|
422
|
+
## Contributing
|
|
423
|
+
|
|
424
|
+
Bug reports, new platform signals and detection fixes are welcome; see
|
|
425
|
+
[CONTRIBUTING.md](CONTRIBUTING.md).
|
|
426
|
+
|
|
427
|
+
- **Architecture**: one polite fetcher, pure detectors (one module each), a
|
|
428
|
+
planner, three outputs. See [docs/architecture.md](docs/architecture.md).
|
|
429
|
+
- **Adding a detector signal**: add a row to the signal table at the top of the
|
|
430
|
+
module in `src/scrapeplanner/detectors/`, add a unit test with a minimal
|
|
431
|
+
input, run `./scripts/check.sh`.
|
|
432
|
+
- **Tests and golden files**: `uv run pytest` runs everything offline.
|
|
433
|
+
`tests/fixtures/` holds the recorded HTTP exchanges of the results run;
|
|
434
|
+
`tests/golden/` replays them through the real analyzer and compares every
|
|
435
|
+
report section. After an intended change, run
|
|
436
|
+
`uv run python scripts/update_golden.py` and explain the diff.
|
|
437
|
+
- **Live checks** run only in GitHub Actions: edit `checks/request.yml`, push,
|
|
438
|
+
and the workflow commits the results.
|
|
439
|
+
- **Releases**: tag `vX.Y.Z`; the release workflow tests, builds and publishes
|
|
440
|
+
to PyPI through Trusted Publishing (no tokens in the repository).
|
|
441
|
+
|
|
442
|
+
[Security policy](SECURITY.md) · [Changelog](CHANGELOG.md) ·
|
|
443
|
+
[Code of conduct](CODE_OF_CONDUCT.md) · [Results](docs/results.md)
|
|
444
|
+
|
|
445
|
+
## License
|
|
446
|
+
|
|
447
|
+
MIT, see [LICENSE](LICENSE). Fonts and logos in the visuals are listed in
|
|
448
|
+
[assets/LICENSES.md](assets/LICENSES.md).
|
|
449
|
+
|
|
450
|
+
---
|
|
451
|
+
|
|
452
|
+
<p align="center">Built and maintained by Siddharth Kumar · <a href="https://siddharthkumar.com">siddharthkumar.com</a></p>
|