crawlutron 0.4.0rc1__py3-none-any.whl

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,601 @@
1
+ Metadata-Version: 2.5
2
+ Name: crawlutron
3
+ Version: 0.4.0rc1
4
+ Summary: Policy-driven, self-hosted web crawler and content-extraction service built around FastAPI
5
+ Project-URL: Homepage, https://github.com/adambarber/crawlutron
6
+ Project-URL: Issues, https://github.com/adambarber/crawlutron/issues
7
+ Project-URL: Changelog, https://github.com/adambarber/crawlutron/blob/main/CHANGELOG.md
8
+ Project-URL: Security, https://github.com/adambarber/crawlutron/blob/main/SECURITY.md
9
+ Author: crawlutron maintainers
10
+ License-Expression: MIT
11
+ License-File: LICENSE
12
+ Keywords: browser,camoufox,crawler,extraction,fastapi,scraper,ssrf-protection
13
+ Classifier: Development Status :: 4 - Beta
14
+ Classifier: Framework :: FastAPI
15
+ Classifier: Intended Audience :: Developers
16
+ Classifier: Operating System :: POSIX :: Linux
17
+ Classifier: Programming Language :: Python :: 3
18
+ Classifier: Programming Language :: Python :: 3.14
19
+ Classifier: Topic :: Internet :: WWW/HTTP
20
+ Classifier: Topic :: Text Processing :: Markup :: HTML
21
+ Requires-Python: >=3.14
22
+ Requires-Dist: crawlutron-core>=0.4.0.dev0
23
+ Requires-Dist: crawlutron-mcp>=0.4.0.dev0
24
+ Requires-Dist: crawlutron-rest>=0.4.0.dev0
25
+ Requires-Dist: crawlutron-safe-url[psl]>=0.4.0.dev0
26
+ Requires-Dist: crawlutron-sdk>=0.4.0.dev0
27
+ Provides-Extra: dev
28
+ Requires-Dist: basedpyright==1.40.2; extra == 'dev'
29
+ Requires-Dist: brotli-stubs==1.2.1; extra == 'dev'
30
+ Requires-Dist: coverage==7.17.0; extra == 'dev'
31
+ Requires-Dist: jscpd==5.3.2; extra == 'dev'
32
+ Requires-Dist: mypy==2.4.1; extra == 'dev'
33
+ Requires-Dist: pre-commit==4.7.0; extra == 'dev'
34
+ Requires-Dist: pytest-asyncio==1.4.1.post1; extra == 'dev'
35
+ Requires-Dist: pytest-cov==7.1.0; extra == 'dev'
36
+ Requires-Dist: pytest-randomly==5.0.0; extra == 'dev'
37
+ Requires-Dist: pytest-timeout==2.4.0; extra == 'dev'
38
+ Requires-Dist: pytest-xdist==3.8.0; extra == 'dev'
39
+ Requires-Dist: pytest==9.2.0; extra == 'dev'
40
+ Requires-Dist: pyyaml==6.0.3; extra == 'dev'
41
+ Requires-Dist: ruff==0.17.3; extra == 'dev'
42
+ Requires-Dist: types-lxml==2026.2.16; extra == 'dev'
43
+ Requires-Dist: types-pyyaml==6.0.12.20261030; extra == 'dev'
44
+ Requires-Dist: types-regex==2026.9.29.20261030; extra == 'dev'
45
+ Requires-Dist: vulture==2.16; extra == 'dev'
46
+ Description-Content-Type: text/markdown
47
+
48
+ # crawlutron
49
+
50
+ A policy-driven, self-hosted web crawler and content-extraction service built around
51
+ FastAPI. It exposes a single, machine-readable crawl contract that supports:
52
+
53
+ **Positioning.** crawlutron is a self-hosted, policy-driven crawler for public pages. Its caller security boundary is built-in per-caller authentication (API keys issued and revoked through `/v1/keys`, operator-token gated; keyed jobs and results are readable only under their key, anonymous crawls share a bounded identity). SSRF is architecture, not an add-on: every egress target is classified, fresh-resolved, IP-pinned, and re-validated per hop and per browser subresource — `when the private-target guard is enabled`, warned once per enabled value at first use (`packages/safe-url/src/safe_url/host_policy.py:88-100`). A transport-auth reverse proxy is recommended when exposing beyond localhost (TLS termination, `Server` header stripping, request-id logging), but built-in per-caller auth is the caller security boundary; the proxy is transport hardening, not a substitute for it. The rule framework (`tests/invariants/`) arms and bites every rule, and the pre-commit hook chain gates every commit. See `docs/architecture-ssrf.md`, `docs/architecture-governance.md`, and `SECURITY.md`.
54
+
55
+ - **HTTP fetch strategies** — transparent declared-crawler HTTP and TLS-fingerprint
56
+ impersonation (curl_cffi).
57
+ - **Browser rendering** — headless/headed Firefox via [Camoufox](https://github.com/daijro/camoufox)
58
+ for JavaScript-dependent pages.
59
+ - **Document extraction** — PDF/DOCX/PPTX/XLSX/TXT via markitdown, plus main-text
60
+ extraction via trafilatura.
61
+ - **Content classification** — 36 crawl classifications (rendered, blocked,
62
+ network_error, robots_disallowed, waf_challenge_detected, etc.) so callers can branch
63
+ on outcome rather than guessing.
64
+ - **Discovery endpoints** — capabilities, agent hints, and a `.well-known` service
65
+ descriptor so agents/bots can discover the contract without prior knowledge.
66
+ - **Web search** — `POST /v1/search` discovers public URLs from a query through a
67
+ keyed provider (Tavily / Brave / Serper) with a keyless DuckDuckGo fallback.
68
+ - **MCP** — `POST /mcp` exposes search/crawl/map/render as tools over the
69
+ Model Context Protocol (2026-07-28, stateless streamable HTTP).
70
+ - **SSRF protection** — target URLs are validated to be public HTTP(S) only.
71
+ - **Webhooks, batching, recursion, rate limiting, robots.txt respect** — best-effort,
72
+ caller-owned frontier semantics.
73
+
74
+ ## Platforms
75
+
76
+ Linux and macOS are supported; Windows is not. The service is built on POSIX
77
+ primitives the Windows runtime does not provide: `fcntl` advisory file locking
78
+ (`packages/core/src/crawlutron/safety/atomic_write.py`, `packages/core/src/crawlutron/orchestration/engine.py`,
79
+ `packages/core/src/crawlutron/transport/content_cache.py`), `resource`/sysconf-based memory
80
+ sizing (`packages/core/src/crawlutron/runtime/pressure_memory.py`), and cgroup filesystem
81
+ readings for the resource gauges (`packages/core/src/crawlutron/runtime/cgroup_sense.py`).
82
+ Deploy it as a Linux container or run it directly on Linux/macOS.
83
+
84
+ ## Security residuals and environment escape hatches
85
+
86
+ Three residual weaknesses are known and accepted, with the assumption that
87
+ guards them stated:
88
+
89
+ - **S1 — HTTPS DNS rebind window.** HTTP targets are IP-pinned end to end.
90
+ HTTPS browser navigation does two fresh DNS resolves but lets the browser
91
+ connect by hostname, so a hostile DNS server can rebind between check and
92
+ connect.
93
+ - **S4 — credential origins ride the egress guard.** Credential origins pass
94
+ the same SSRF egress check as crawl targets: a private/loopback credential
95
+ origin requires `CRAWLUTRON_ALLOW_PRIVATE_TARGETS=1`.
96
+ - **S5 — caller key secrets are verified with salted PBKDF2-SHA256.** Issued
97
+ secrets are `ck_<key_id>.<body>` where the body is `secrets.token_urlsafe(32)`
98
+ (256-bit entropy). At rest the key row stores a salted PBKDF2-SHA256 digest
99
+ (210,000 iterations, per-key 16-byte salt,
100
+ `packages/core/src/crawlutron/storage/caller_key_store.py`); verification is one primary-key
101
+ lookup plus one PBKDF2 run, with a miss burning the same derivation against
102
+ a fixed dummy digest so timing is uniform. Residual: entropy of the secret,
103
+ not the KDF, is the real guard; do not reuse this pattern for human-chosen
104
+ secrets.
105
+
106
+ ### Dangerous environment escape hatches
107
+
108
+ Every `CRAWLUTRON_*` variable that weakens a safety default (derived from the
109
+ live settings/env code — `packages/safe-url/src/safe_url/host_policy.py`,
110
+ `packages/core/src/crawlutron/service/webhooks.py`, `packages/core/src/crawlutron/config/settings.py`,
111
+ `packages/core/src/crawlutron/transport/driver_config.py`):
112
+
113
+ | Variable | Default | Weakened when | Effect |
114
+ | -------- | ------- | ------------- | ------ |
115
+ | `CRAWLUTRON_ALLOW_PRIVATE_TARGETS` | unset | `1` | Disables the SSRF private-target guard: caller URLs may reach private, loopback, and link-local addresses including cloud metadata endpoints. Logs a `SSRF private-target guard DISABLED` WARNING. |
116
+
117
+ ### Ordinary operational knobs
118
+
119
+ Two variables are sometimes mistaken for safety hatches; they are not. They
120
+ change operational behavior only and are listed in the Configuration table
121
+ below:
122
+
123
+ - `CRAWLUTRON_RESPECT_ROBOTS` (default `true`) — turns robots.txt
124
+ enforcement off. Robots compliance is a crawler courtesy/policy choice, not
125
+ a security boundary of this service.
126
+ - `CRAWLUTRON_HEADLESS` (default `true`) — runs the browser headed instead
127
+ of headless. Changes the automation surface, weakens no guard.
128
+
129
+ See `SECURITY.md` for the threat model and reporting process, and
130
+ `docs/reviews/ADVERSARIAL_REVIEW_BROWSER_CRAWLER.md` for the dated adversarial review of
131
+ the crawler.
132
+
133
+ The service is intended to be deployed as a Docker container. A standalone
134
+ `docker-compose.yml` lives at the repository root for local development;
135
+ production deployments typically front it with a transport-auth reverse proxy (TLS termination, `Server` header stripping, request-id logging); built-in per-caller auth (`/v1/keys`, `X-API-Key`/`Authorization: Bearer`) is the caller security boundary, and the proxy is transport hardening, not a substitute for it.
136
+
137
+ ## Security Posture
138
+
139
+ crawlutron is a **localhost-only** service. It binds to `127.0.0.1` by
140
+ default. Callers authenticate with an API key presented as `X-API-Key` or
141
+ `Authorization: Bearer` — jobs and results created under a key are readable
142
+ only under it, and key issuance/revocation is operator-gated
143
+ (`CRAWLUTRON_KEY_ADMIN_TOKEN` via `/v1/keys`). Keyless requests crawl as a
144
+ shared anonymous identity with bounded default quotas. Key verification is
145
+ rate-limited (#653): five consecutive failed verifications from the same
146
+ presented credential trip a 60-second cooldown that refuses further attempts
147
+ with `429 key_verify_throttled` before the PBKDF2 derivation runs again, so
148
+ the expensive pre-auth hash cannot be invoked for free in a loop. Buckets
149
+ key on a hash of the full presented secret (#675), so garbage under another
150
+ caller's key id cannot lock the rightful owner out. Anonymous surface
151
+ metering keys on the REMOTE ADDRESS the ASGI transport reports (`client.host`):
152
+ behind a reverse proxy every request shares the PROXY's address and its one
153
+ bucket — when fronting the service with a proxy, run the metering at the
154
+ proxy or deploy as a single-tenant front. Expose it
155
+ beyond localhost only behind a transport-auth reverse proxy; built-in per-caller
156
+ auth (`/v1/keys`) is the caller security boundary, and the proxy is transport
157
+ hardening, not a substitute for it.
158
+
159
+ What the SSRF defenses cover:
160
+
161
+ - **DNS pinning on every fetch** — every URL is resolved fresh at fetch
162
+ time and the resulting IP is pinned for the connection, so a DNS
163
+ rebind cannot redirect the request mid-flight.
164
+ - **Manual redirect handling** — redirects are followed one hop at a
165
+ time; each hop re-runs the IP-pinning and private-IP check.
166
+ - **Browser-side SSRF** — the Camoufox/Playwright navigation is pinned
167
+ to a freshly-validated IP via `Host:` header rewriting; post-redirect
168
+ URLs are re-validated before being used as document fetch targets.
169
+ - **Body-size guard** — request bodies are capped at 16 MiB before
170
+ pydantic validation runs.
171
+ - **HTML/DOM size guards** — classification and metadata extraction
172
+ truncate inputs at 4 MiB before regex / lxml processing.
173
+ - **Per-tenant cookie namespace** — multi-tenant callers should set
174
+ `request_profile.cookie_namespace` so cookies from different callers
175
+ are isolated in the shared cookie jar.
176
+
177
+
178
+ ## Adversarial Test Coverage
179
+
180
+ The test suite was grown alongside the adversarial reviews; the following
181
+ suites cover the specific attack classes the security posture is designed
182
+ to defeat:
183
+
184
+ - **Truncation and drain** (`tests/test_body_cap_state_machine.py`) — chunked
185
+ handling, `Connection: close` on a truncated request, per-chunk trailers and
186
+ headers, body-size drain after truncation, the rule that no further bytes are
187
+ admitted once a chunk trips the cap, and the 413 the shipped app returns on a
188
+ body over the cap.
189
+ - **File-descriptor ownership** (`tests/invariants/test_ownership.py`) — scratch
190
+ files are owned by the work writing them; the SQLite spill store replaced the
191
+ file spill (`tests/storage/test_crawl_spill_store.py`).
192
+ - **Cache locking** (`tests/transport/test_dns_cache.py`,
193
+ `tests/transport/test_robots_policy_consolidation.py`) — concurrent access to
194
+ the DNS and robots caches, including that robots reads go through the one
195
+ chokepoint.
196
+ - **Cookie namespacing** (`tests/transport/test_cookie_namespace.py`) — per-tenant
197
+ cookies cannot cross a tenant boundary.
198
+ - **Env parsing** (`tests/test_env_parsing.py`) — `parse_bool_env` edge cases,
199
+ including that a typo cannot silently open a policy.
200
+ - **Webhook TOCTOU** (`tests/api/test_webhook_delivery.py`) — one resolved
201
+ target per delivery, consolidated at the chokepoint.
202
+ - **Decompression bomb tests** — a real gzip bomb (`~1 KB` compressed →
203
+ `1 MB` expanded), plus deflate and brotli bomb cases, verify the decompression
204
+ guard rejects over-cap output before OOM.
205
+ - **SSRF tests** (`tests/test_ssrf_route_subresources.py`, `tests/transport/test_dns_cache.py`, `tests/transport/test_redirect_target.py`, `tests/transport/test_browser_target_stability.py`) — DNS
206
+ pinning, route-level interception, WebSocket guard, hostname pinning
207
+ even after a redirect chain, IPv6 private range blocking.
208
+ - **Classification corroboration tests** — soft phrases (e.g. "blocked")
209
+ require ≥2 independent signals; the test asserts a single soft signal
210
+ is NOT enough to trigger a defensive classification.
211
+ - **Resource admission control** — fake cgroup gates verify the
212
+ two-tier weighted admission policy kills stop-the-world pauses under
213
+ saturation.
214
+
215
+ This is unusually thorough for an open-source project. New attack
216
+ classes should land alongside a dedicated adversarial test that fails
217
+ without the fix.
218
+
219
+ **Coverage is measured, not a badge.** No coverage badge or coverage
220
+ ratchet is claimed. The pre-push coverage hook is measured-descriptive
221
+ only — no threshold, no fail-under, no ratchet until the owner rules one
222
+ (`.pre-commit-config.yaml:229-237`). The safety boundary is the invariant
223
+ suite and the fail-closed SSRF/auth guards, not a coverage percentage.
224
+ See `docs/architecture-governance.md`.
225
+
226
+ ---
227
+
228
+ ## Build & Run
229
+
230
+ The service ships with a standalone `docker-compose.yml` and a multi-stage
231
+ `Dockerfile` so it can be built and run from a fresh clone without any parent
232
+ monorepo or shared-helper tree.
233
+
234
+ ```bash
235
+ # Build the image (multi-stage; the Camoufox browser binary is fetched once in an
236
+ # isolated stage so editing requirements.txt does not re-download ~1.3 GB).
237
+ docker compose build crawlutron
238
+
239
+ # Run it (recreates only this service).
240
+ docker compose up -d crawlutron
241
+ ```
242
+
243
+ The container listens on `127.0.0.1:8765`. The served API docs (Swagger UI)
244
+ are available at `/docs` and the machine-readable schema at `/openapi.json`.
245
+ For local-hostname routing (e.g. `http://crawlutron.local`), front the
246
+ container with Caddy or another reverse proxy that owns the `.local` zone.
247
+
248
+ > **Note on the build:** The Camoufox browser binary is downloaded in a dedicated
249
+ > `camoufox-binary` stage that depends *only* on the `camoufox` requirement line,
250
+ > then copied into the runtime image. This isolation means an unrelated change to
251
+ > `requirements.txt` (for example adding `h2`) no longer invalidates the
252
+ > multi-minute binary download.
253
+
254
+ ---
255
+
256
+ ## SQLite schema migrations
257
+
258
+ Migrations live in `packages/core/src/crawlutron/storage/migrations/`. Every store schema — jobs, results, metrics, coordination — lives in the single `cache/crawler.db`; startup applies that database's pending migrations under yoyo's lock, and an exact pre-ledger table is stamped instead of recreated. The CLI names the schema explicitly because they share one file.
259
+
260
+ ```bash
261
+ # Apply pending migrations (normally automatic when the store opens).
262
+ PYTHONPATH=packages/core/src .venv/bin/python -m crawlutron.storage.migrations apply cache/crawler.db crawl_jobs
263
+ PYTHONPATH=packages/core/src .venv/bin/python -m crawlutron.storage.migrations apply cache/crawler.db results
264
+
265
+ # Add a migration: add a versioned schema to manifest.py, point CURRENT_SCHEMAS at it,
266
+ # then add a yoyo step with apply and rollback SQL under the migration directory.
267
+ $EDITOR packages/core/src/crawlutron/storage/migrations/manifest.py
268
+ $EDITOR packages/core/src/crawlutron/storage/migrations/0001_crawl_jobs_initial.py
269
+
270
+ # Stamp an existing database after checking it against the declared schema.
271
+ PYTHONPATH=packages/core/src .venv/bin/python -m crawlutron.storage.migrations stamp cache/crawler.db crawl_jobs
272
+ PYTHONPATH=packages/core/src .venv/bin/python -m crawlutron.storage.migrations stamp cache/crawler.db results
273
+
274
+ # Roll back one schema's applied migrations in reverse order; do not open the
275
+ # store afterward. Rolling one schema back leaves the others applied.
276
+ PYTHONPATH=packages/core/src .venv/bin/python -m crawlutron.storage.migrations rollback cache/crawler.db crawl_jobs
277
+ PYTHONPATH=packages/core/src .venv/bin/python -m crawlutron.storage.migrations rollback cache/crawler.db results
278
+ ```
279
+
280
+ ---
281
+
282
+ ## Endpoints
283
+
284
+ | Method | Path | Purpose |
285
+ | ------ | ---- | ------- |
286
+ | `GET` | `/health` | Liveness probe (always `200` if the process is up). |
287
+ | `GET` | `/ready` | Readiness probe; returns `503` when the self-managed resource guard is saturated (memory or CPU at the gate target). |
288
+ | `POST` | `/v1/crawl` | Submit one URL (or a batch of URLs) for crawling. |
289
+ | `POST` | `/v1/render` | Render a URL with the browser engine and return structured output. |
290
+ | `POST` | `/v1/search` | Discover public URLs from a query via the configured search provider. |
291
+ | `POST` | `/v1/map` | Crawl a site's link graph and return the discovered URL set (no content extraction). |
292
+ | `GET` | `/v1/crawl/{request_id}` | Fetch the result of an async crawl submission. |
293
+ | `GET` | `/v1/crawl/{request_id}/trace` | Per-URL strategy attempt trace (statuses, classifications, timings; no page bodies). |
294
+ | `POST`/`GET`/`DELETE` | `/v1/watches` | Revalidation watches: register, list, get, delete; sweeps recheck due watches and POST webhooks on change. |
295
+ | `GET`/`POST` | `/v1/jobs` | Whole-site resumable jobs: create (with frontier), list, inspect, pause/resume/cancel. |
296
+ | `POST` | `/mcp` | MCP 2026-07-28 endpoint exposing search/crawl/map/render as tools. |
297
+ | `GET` | `/v1/capabilities` | Declares supported strategies, classifications, extraction methods, profiles. |
298
+ | `GET` | `/v1/agent-hints` | Machine-readable guidance for agents/bots (request shapes, caveats). |
299
+ | `GET` | `/.well-known/crawler-service` | Service discovery descriptor (links to OpenAPI, agent hints, capabilities). |
300
+ | `POST`/`GET`/`DELETE` | `/v1/keys` | Caller API keys: issue, list, revoke (operator token required). Callers present the key as `X-API-Key`. |
301
+ | `GET` | `/metrics` | Prometheus-format metrics. Per-caller detail families require the operator token (`X-API-Key: CRAWLUTRON_KEY_ADMIN_TOKEN`); a presented but wrong token records an admin-bucket failure (paced 5/300s), so scrape-guessing griefs the operator doors' cooldown too. |
302
+ | `GET` | `/v1/policy-suggestions` | Candidate host policy rules rolled up from blocked-attempt evidence (proposal only; never applied). |
303
+
304
+ ---
305
+
306
+ ## Web search
307
+
308
+ `POST /v1/search` discovers public URLs from a query and returns SERP-style
309
+ results (`{success, query, provider, limit, data: [{url, title, description}]}`).
310
+ It does **not** crawl result URLs — pipe them into `POST /v1/crawl`.
311
+
312
+ The backend is operator-configured, not caller-selected:
313
+
314
+ - `CRAWLUTRON_SEARCH_PROVIDER` (`auto` | `tavily` | `brave` | `serper` | `duckduckgo`).
315
+ - `auto` (default) activates the first configured keyed provider in priority
316
+ order (Tavily to Brave to Serper) and falls back to the keyless DuckDuckGo
317
+ HTML endpoint when no API key is set. The keyless endpoint may be
318
+ captcha-blocked from datacenter IPs, so configure a key for production use.
319
+ - A selected provider without its key returns `503 search_provider_unavailable`
320
+ naming the missing `CRAWLUTRON_*_API_KEY` variable.
321
+
322
+ Result URLs pass the same public-URL validation as crawl targets (loopback and
323
+ private targets are dropped). `GET /v1/capabilities` reports the active
324
+ provider and the configured provider list.
325
+
326
+ ## MCP
327
+
328
+ `POST /mcp` exposes `search`, `crawl`, `map`, and `render` as tools over the
329
+ **Model Context Protocol (2026-07-28)** — the stateless revision: no
330
+ `initialize` handshake, no sessions. Every request carries its protocol
331
+ version and client capabilities in `_meta`, and the transport validates the
332
+ `MCP-Protocol-Version`, `Mcp-Method`, and `Mcp-Name` headers against the body.
333
+
334
+ - `server/discover` — supported protocol versions, capabilities, server identity.
335
+ - `tools/list` — the fetch, read, and job tools with JSON-Schema inputs.
336
+ - `tools/call` — invokes a tool; returns text `content` plus
337
+ `structuredContent`, with `isError` set for execution failures.
338
+ - The tools mirror the REST endpoints and delegate to the same handlers, so
339
+ **all four strategies and every extraction method are available through MCP**.
340
+ - Webhook/async delivery is not available over MCP.
341
+
342
+ The endpoint is loopback-only and rejects non-loopback `Origin` headers (DNS
343
+ rebinding defense).
344
+
345
+ ## Crawl request shape
346
+
347
+ `POST /v1/crawl` accepts either a single `url` or a list of `urls`, plus optional top-level
348
+ fields: `policy`, `validators` (cached conditional-GET validators), `async_mode`, `webhook_url`,
349
+ and `output_format`. The request/browsing profile knobs — `request_profile`, `browser_profile`,
350
+ `discovery`, `scope_regex`, `css_selector`, `xpath` — live **inside** `policy`.
351
+
352
+ ```json
353
+ {
354
+ "url": "https://example.com/article",
355
+ "policy": {
356
+ "strategies": ["HTTP_DECLARED_CRAWLER", "BROWSER_HEADLESS"],
357
+ "extraction": "full",
358
+ "limits": {
359
+ "timeout_seconds": 30,
360
+ "max_retries": 2,
361
+ "challenge_wait_seconds": 4,
362
+ "max_discovered_links": 10,
363
+ "respect_robots": true
364
+ },
365
+ "request_profile": { "user_agent_family": "generic_crawler" },
366
+ "browser_profile": { "resource_mode": "full" },
367
+ "discovery": { "robots": true, "sitemaps": true, "feeds": true }
368
+ }
369
+ }
370
+ ```
371
+
372
+ ### Strategies (fallback chain)
373
+
374
+ `strategies` is an **ordered fallback chain**. The engine tries each in turn until
375
+ one succeeds, returns a definitive blocked/error classification, or the chain is
376
+ exhausted.
377
+
378
+ | Strategy | Description |
379
+ | -------- | ----------- |
380
+ | `HTTP_DECLARED_CRAWLER` | Transparent direct HTTP fetch with a declared-crawler identity (lightweight, fast). |
381
+ | `HTTP_CFFI_IMPERSONATE` | TLS-fingerprint-impersonated HTTP via `curl_cffi` with browser-shaped headers, for sites that reject non-browser clients. |
382
+ | `BROWSER_HEADLESS` | Firefox rendering in headless mode using Camoufox for JavaScript-dependent pages. |
383
+ | `BROWSER_HEADED_WARMED` | Headed Firefox with configured warmup URLs for challenge-prone sites. |
384
+
385
+ ### Extraction methods (`policy.extraction`)
386
+
387
+ - `full` (default) — best-effort main-text extraction, returns both `markdown` and `text`.
388
+ - `trafilatura` — main-text only via trafilatura.
389
+ - `document` — markitdown-based extraction for PDF/DOCX/PPTX/XLSX/TXT (binary documents).
390
+ - `dom` — raw DOM / selected-element extraction (`css_selector` / `xpath`).
391
+
392
+ ---
393
+
394
+ ## Classifications
395
+
396
+ Every crawl result carries a `classification` from a fixed enum of **36 values**
397
+ (defined in `packages/core/src/crawlutron/vocabulary/taxonomy.py`, returned by `GET /v1/capabilities` under
398
+ `supported_classifications`). Examples: `rendered`, `not_modified`, `captcha_detected`,
399
+ `waf_challenge_detected`, `robots_disallowed`, `navigation_timeout`,
400
+ `private_network_target`, `network_error`, `http_error`, `url_budget_exceeded`,
401
+ `payload_too_large`. Callers should branch on `classification` (whose
402
+ `status: success | blocked | disallowed | retry_exhausted | failed | skipped`
403
+ the taxonomy maps onto) rather than on transport status.
404
+
405
+ ---
406
+
407
+ ## Behavior notes
408
+
409
+ - **SSRF protection** is on by default: only public HTTP(S) targets are allowed.
410
+ Set `CRAWLUTRON_ALLOW_PRIVATE_TARGETS=1` to disable it (local testing only).
411
+ Enabling it logs a WARNING naming the disabled guard, and lets caller-supplied
412
+ URLs reach private, loopback, and link-local addresses including cloud
413
+ metadata endpoints.
414
+ - **robots.txt** is respected by default (`respect_robots`). Disallowed targets
415
+ return `robots_disallowed`.
416
+ - **Recursion / frontier:** `POST /v1/crawl` recursively expands one in-memory
417
+ frontier for the duration of a request and reports discovered links, sitemaps,
418
+ and feeds; cross-run queue ownership remains with the caller. `POST /v1/jobs`
419
+ instead owns a durable whole-site frontier. Each persisted `frontier` and
420
+ `in_flight` entry is `{"url": "...", "depth": N}`; `url` is the canonical page
421
+ identity and depth 0 is the stored canonical root. Legacy string entries load
422
+ at depth 0 when they are the root and at the effective `max_depth` otherwise,
423
+ preventing untracked additional recursion during upgrade.
424
+ - **Whole-site admission:** the engine dispatcher and durable job runner share
425
+ canonical-identity, scope, analytics, document, robots, crawl-depth, and page-cap
426
+ admission. When `discovery.sitemaps` is enabled, root sitemap pages enter the
427
+ durable frontier at depth 0 through that same pipeline. `max_depth=0` is
428
+ single-page. A positive job `max_pages` counts unique canonical page identities
429
+ across the root, pending frontier, in-flight recovery,
430
+ visited pages, and retries. Missing or zero `max_pages` is unbounded for jobs;
431
+ `/v1/crawl` retains its bounded service default. A positive job cap sets
432
+ `cap_truncated=true` in status and `stats.cap_truncated=1` only when it refuses
433
+ or truncates eligible queued or discovered work; normal `BATCH_SIZE` chunking
434
+ does not set it. Already-admitted in-flight work is never dropped: recovery
435
+ preserves at-least-once delivery and records `stats.cap_overcommitted=1`.
436
+ - **Crawl duration & async**: a synchronous `POST /v1/crawl` has **no server-side
437
+ time ceiling** — it returns only when the whole bounded recursion finishes, so
438
+ duration scales with `max_depth`, `max_pages`, per-domain politeness delays, and
439
+ the per-URL budget (`limits.total_url_budget_seconds`, default 180s, which fails a
440
+ single tarpit URL, not the crawl). For large or recursive crawls send
441
+ `async_mode=true` (or `webhook_url`) to get an immediate `202` + `status_url`, then
442
+ poll `GET /v1/crawl/{request_id}`; or use `POST /v1/jobs` for durable, resumable,
443
+ pause/cancel-able whole-site jobs. Read timeouts on the sync path must cover the
444
+ chosen policy and batch size.
445
+ - **Rate limiting** is enforced via `global_concurrency`, `per_domain_concurrency`,
446
+ and `per_domain_delay_ms`, scoped to a single request: each request runs its
447
+ own scheduler, so concurrent requests each enforce the limits independently
448
+ and a service-wide per-domain bound is not provided (callers pacing many
449
+ concurrent requests own the aggregate). Exceeding `max_batch_size` is rejected.
450
+ - **Webhooks / async**: provide `webhook_url` to receive results asynchronously, or
451
+ `async_mode=true` to get a `202 Accepted` with a `status_url` to poll
452
+ `GET /v1/crawl/{request_id}`. Both return immediately instead of blocking on the
453
+ crawl.
454
+ - **Conditional GETs**: `HTTP_DECLARED_CRAWLER` sends `If-None-Match` /
455
+ `If-Modified-Since` when prior ETags are known, returning `not_modified` on a `304`.
456
+ - **Resource self-tuning** (`#self-tuning`): the engine senses its own cgroup
457
+ memory/CPU ceiling and throttles recursive-batch dispatch to ride at the
458
+ targets instead of OOMing or tripping the kernel CPU throttle. No operator
459
+ tuning is required.
460
+ - **HTTP-aware admission (item 7)**: browser crawls are the memory hog (one
461
+ ≈700 MiB (≈600–900 MiB RSS) browser process each), and when browser processes hold memory
462
+ the whole batch used to be throttled — even cheap HTTP work. Today, when the
463
+ upcoming batch is predominantly domains *pinned* to an HTTP strategy (they've
464
+ already proven they fetch without a browser) and CPU has spare headroom, the
465
+ dispatcher uses the **HTTP-aware scale** (`ResourceGate.http_scale_factor`),
466
+ which rides the CPU-implied rate and is *not* dragged down by memory held by
467
+ browser processes of other domains. Browser-bound and cold (unpinned) batches
468
+ keep the conservative combined memory+CPU gate, so the extra concurrency is
469
+ always ramp-bounded (≤ `RECURSIVE_RAMP_MAX × multiplier`) and cannot overshoot
470
+ memory.
471
+
472
+ ---
473
+
474
+ ## Configuration
475
+
476
+ All settings are read from `CRAWLUTRON_*` environment variables (sensible
477
+ defaults shown):
478
+
479
+ | Variable | Default | Meaning |
480
+ | -------- | ------- | ------- |
481
+ | `CRAWLUTRON_GLOBAL_CONCURRENCY` | `4` (env-overridable floor) | Concurrency floor for the recursive dispatcher. Real concurrency is self-managed by the resource guard; `/ready` no longer keys off this value. |
482
+ | `CRAWLUTRON_PER_DOMAIN_CONCURRENCY` | `2` | Max concurrent tasks per host. |
483
+ | `CRAWLUTRON_PER_DOMAIN_DELAY_MS` | `500` | Delay between requests to the same host. |
484
+ | `CRAWLUTRON_HTTP_MAX_CONNECTIONS` | _(RAM-derived; floor 100, cap 512)_ | Max concurrent HTTP connections. The pool default is the floor of 100; the weighted admission derives the real HTTP concurrency from the cgroup limit as `(limit − browser-pool reserve − doc-pool reserve) / 48 MiB per HTTP task` — ~34 at 3 GiB with the 2-instance browser pool (~700 MiB each), ~22 with the default 2 doc workers (300 MiB each). Bound to the weighted admission's HTTP concurrency so the pool can't starve or over-provision. |
485
+ | `CRAWLUTRON_HTTP_MAX_KEEPALIVE` | _(RAM-derived; floor 20)_ | Max idle keep-alive connections (≈ half of the connection ceiling, bounded to [20, 128]). |
486
+ | `CRAWLUTRON_BROWSER_POOL_SIZE` | _(RAM-derived; 2 at 3 GiB)_ | Explicit browser-pool size override. Default is derived from the cgroup limit (reserves ≤50% of RAM for browsers); raising it multiplies RAM use and risks OOM. |
487
+ | `CRAWLUTRON_SEARCH_PROVIDER` | `auto` | Search backend: `auto` (first configured keyed provider) or explicit `tavily` / `brave` / `serper` / `duckduckgo`. |
488
+ | `CRAWLUTRON_TAVILY_API_KEY` | `""` | Tavily Search API key (activates the `tavily` provider). |
489
+ | `CRAWLUTRON_BRAVE_API_KEY` | `""` | Brave Search API key (activates the `brave` provider). |
490
+ | `CRAWLUTRON_SERPER_API_KEY` | `""` | Serper.dev API key (activates the `serper` provider). |
491
+ | `CRAWLUTRON_SEARCH_LIMIT_DEFAULT` | `5` | Default result count per search. |
492
+ | `CRAWLUTRON_SEARCH_LIMIT_MAX` | `20` | Hard cap on results per search. |
493
+ | `CRAWLUTRON_SEARCH_TIMEOUT_SECONDS` | `15` | Provider call timeout. |
494
+ | `CRAWLUTRON_MAX_BATCH_SIZE` | `50` | Max URLs per batch request. |
495
+ | `CRAWLUTRON_WARMUP_URLS` | `""` | Comma-separated URLs fetched once at browser startup to warm engine caches (`browser.py`). Warmup URLs are normalized through the same public-URL policy as crawl targets. Related: `browser_profile.storage_state` paths are honored only under `CRAWLUTRON_CACHE_DIR` (the allowlist root); without a cache dir no storage_state is accepted. |
496
+
497
+ Two DNS posture values are **not** env-tunable — they are module constants in
498
+ `packages/core/src/crawlutron/transport/pinned_dns.py`: `DNS_WORKERS = 32` (the dedicated DNS
499
+ resolver thread pool) and `DNS_WAIT_SECONDS = 5.0` (the ceiling on a resolver
500
+ wait; a request's own timeout tightens it further). They implement the
501
+ pinned-DNS saturation law: when every worker is inside a live resolve, a new
502
+ resolve refuses fast (`ResolverBudgetExceeded`, classifiable as `dns_error`)
503
+ instead of queueing behind an uncancellable `getaddrinfo`.
504
+ | `CRAWLUTRON_DEFAULT_TIMEOUT_SECONDS` | `30` | Default per-request timeout. |
505
+ | `CRAWLUTRON_DEFAULT_MAX_RETRIES` | `5` | Default retry count. |
506
+ | `CRAWLUTRON_RESPECT_ROBOTS` | `true` | Respect robots.txt by default. |
507
+ | `CRAWLUTRON_ALLOWED_HOSTS` | `""` | Optional host allowlist (comma-separated suffix match; case, IDNA, and port normalized). |
508
+ | `CRAWLUTRON_DENIED_HOSTS` | `""` | Optional host denylist (comma-separated suffix match; deny wins over allow). |
509
+ | `CRAWLUTRON_LOG_LEVEL` | `info` | Log verbosity. |
510
+ | `CRAWLUTRON_LOG_FILE` | `./logs/crawlutron.jsonl` (repo-relative; the container image overrides it to `/app/logs/crawlutron.jsonl`) | Structured JSONL log path. |
511
+ | `CRAWLUTRON_LOG_MAX_BYTES` | `10485760` | Log rotation size. |
512
+ | `CRAWLUTRON_LOG_BACKUP_COUNT` | `3` | Number of rotated log files to keep. |
513
+
514
+ Host policy is applied at the shared public-URL chokepoint before the SSRF/private-target check. Entries are normalized as case-insensitive IDNA hostnames (`_policy_host_entry` in `packages/safe-url/src/safe_url/host_policy.py` parses each value via `urlparse(...).hostname`); a port in a config value is stripped during normalization and never matched — an entry is a hostname, not a `host:port` pair. A denied host wins over an allowed host, while an empty allowlist permits every otherwise-valid public host.
515
+
516
+ > The container's production command also reads `CRAWLUTRON_WORKERS` (uvicorn worker count). **Keep this at `1`.** The browser pool is a single global pool sized to the container RAM budget; running more than one uvicorn worker multiplies the browser-pool count across workers (e.g. 4 workers × pool_size 2 = 8 Camoufox processes ≈ 5+ GiB) and blows the 3 GiB memory limit. Do not raise it.
517
+
518
+ > **Secrets & logs:** Logs and prompt JSONL can contain full page content and
519
+ > request metadata. Do not commit generated logs under `logs/`.
520
+
521
+ ---
522
+
523
+ ## API documentation (served)
524
+
525
+ - Swagger UI: `/docs`
526
+ - OpenAPI JSON: `/openapi.json`
527
+ - Agent hints: `/v1/agent-hints`
528
+ - Capabilities: `/v1/capabilities`
529
+ - Service descriptor: `/.well-known/crawler-service`
530
+
531
+ The `/v1/render` request body is a typed `RenderRequest` (`url`/`urls` plus an
532
+ `options` object with `timeout_seconds`, `max_retries`,
533
+ `respect_robots`, `include_html`, `include_text`). Malformed bodies are rejected
534
+ with `422`.
535
+
536
+ ---
537
+
538
+ ## Project layout
539
+
540
+ ```
541
+ . # repository root (clone of crawlutron)
542
+ ├── Dockerfile # Multi-stage build; isolated camoufox-binary stage (context: .)
543
+ ├── docker-compose.yml # Standalone local stack (loopback-only)
544
+ ├── pyproject.toml # Package metadata + deps (single source; uv.lock pins every pin)
545
+ ├── uv.lock # Frozen transitive dependency graph + hashes
546
+ ├── Makefile # Gates + convenience targets mirroring scripts/run-tests.sh
547
+ ├── packages/ # uv workspace member packages, src layout under packages/*/src/
548
+ │ ├── core/
549
+ │ │ └── src/crawlutron/ # core engine: admission kernel, models, transport, storage, service, …
550
+ │ │ ├── api/ # door-plane handler bodies (crawl_endpoint, render_endpoint, map_endpoint)
551
+ │ │ ├── cli/ # crawler CLI entry points
552
+ │ │ ├── config/ # env.py + settings.py (CRAWLUTRON_* environment surface)
553
+ │ │ ├── discovery/ # /v1/capabilities, /v1/agent-hints, /.well-known
554
+ │ │ ├── extraction/ # documents.py (trafilatura + selector extraction)
555
+ │ │ ├── health/ # /health, /ready (READINESS_STATE singleton)
556
+ │ │ ├── mcp/ # MCP protocol support shared with crawlutron_mcp
557
+ │ │ ├── models/ # Pydantic/dataclass wire + domain models
558
+ │ │ ├── orchestration/# engine.py — orchestration + fallback-chain scheduling
559
+ │ │ ├── runtime/ # metrics, memory pressure, worker dispatch, resource lifecycle
560
+ │ │ ├── safety/ # validation, bounded I/O, safe regex, SSRF helpers
561
+ │ │ ├── search/ # /v1/search provider abstraction (Tavily/Brave/Serper/DuckDuckGo)
562
+ │ │ ├── service/ # store_bootstrap, watch_cores, job lifecycle, …
563
+ │ │ ├── storage/ # job_store, result_store, watch_store, sqlite_store, migrations/
564
+ │ │ ├── transport/ # http.py, browser.py, document.py, content_cache.py, scheduler.py, robots.py
565
+ │ │ └── vocabulary/ # taxonomy.py (36-value classification enum), strategy names
566
+ │ ├── rest/
567
+ │ │ └── src/crawlutron_rest/ # REST facade: FastAPI app + routers + CLI
568
+ │ │ ├── main.py # FastAPI factory (lifespan, app.state, router wiring)
569
+ │ │ ├── api/ # one router per concern (crawl, jobs, watches, keys, metrics, policy-suggestions, …)
570
+ │ │ └── cli/ # CLI entry (__main__, output, docker, client)
571
+ │ ├── mcp/
572
+ │ │ └── src/crawlutron_mcp/ # MCP 2026-07-28 endpoint (protocol, tools, server)
573
+ │ └── sdk/
574
+ │ └── src/crawlutron_sdk/ # client SDK mirroring the REST wire surface
575
+ ├── tests/ # Behavior tests (no mocks; real HTTP + real Camoufox)
576
+ ├── docs/ # Reviews, testing guide, agent contract (docs/references/)
577
+ ├── scripts/run-tests.sh # Full battery entrypoint (mirrored by make targets)
578
+ └── .github/workflows/ci.yml # ruff + pytest (fast)
579
+ ```
580
+
581
+ ---
582
+
583
+ ## Testing
584
+
585
+ Behavior tests run against the real stack (a local `http.server`, a real Camoufox
586
+ browser, and real fixtures — no mocks). Use the dedicated venv:
587
+
588
+ ```bash
589
+ pip install -e ".[dev]"
590
+ make test # fast hermetic suite (default)
591
+ # or
592
+ python3 -m pytest -q
593
+ ```
594
+
595
+ The manual test plan lives at `docs/references/crawlutron-manual-test-plan.md`
596
+ and the agent integration contract at `docs/references/crawlutron-agent-contract.md`.
597
+
598
+ ## License
599
+
600
+ MIT — `pyproject.toml` (`license = "MIT"`) and the `LICENSE` file agree; the
601
+ `NOTICE` file carries the third-party attributions.
@@ -0,0 +1,4 @@
1
+ crawlutron-0.4.0rc1.dist-info/METADATA,sha256=Xg8d8wiJXT8A5PI0AVdt4-5jrrSNNuQ60vJhIDBNMZU,37153
2
+ crawlutron-0.4.0rc1.dist-info/WHEEL,sha256=2iZrJkDIkeDh0k9SikKMqFuRP-ua6mb4Ir1AsER4BEw,87
3
+ crawlutron-0.4.0rc1.dist-info/licenses/LICENSE,sha256=ICN6XaVjut4YeGOlhBFaXZcKjUu6SW07ryJsLGzExsI,1322
4
+ crawlutron-0.4.0rc1.dist-info/RECORD,,
@@ -0,0 +1,4 @@
1
+ Wheel-Version: 1.0
2
+ Generator: hatchling 1.33.0
3
+ Root-Is-Purelib: true
4
+ Tag: py3-none-any
@@ -0,0 +1,25 @@
1
+ MIT License
2
+
3
+ Copyright (c) 2024-2026 crawlutron contributors
4
+
5
+ This project contains work adapted from MIT-licensed projects, and depends on
6
+ third-party packages under the MIT, BSD-3-Clause, Apache-2.0, CNRI-Python and
7
+ MPL-2.0 licenses. See NOTICE for the attributions and the full dependency list.
8
+
9
+ Permission is hereby granted, free of charge, to any person obtaining a copy
10
+ of this software and associated documentation files (the "Software"), to deal
11
+ in the Software without restriction, including without limitation the rights
12
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
13
+ copies of the Software, and to permit persons to whom the Software is
14
+ furnished to do so, subject to the following conditions:
15
+
16
+ The above copyright notice and this permission notice shall be included in all
17
+ copies or substantial portions of the Software.
18
+
19
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
20
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
21
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
22
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
23
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
24
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
25
+ SOFTWARE.