dataapp_crawler 0.1.0__py3-none-any.whl

dataapp_crawler-0.1.0.dist-info/METADATA

@@ -0,0 +1,211 @@
+Metadata-Version: 2.5
+Name: dataapp_crawler
+Version: 0.1.0
+Summary: Composable HTTP, browser and proxy acquisition SDK for DataOS DataApps
+Requires-Python: >=3.11
+Requires-Dist: httpx<1,>=0.28
+Provides-Extra: browser
+Requires-Dist: playwright<2,>=1.64; extra == 'browser'
+Provides-Extra: fingerprint
+Requires-Dist: camoufox<0.6,>=0.5.9; extra == 'fingerprint'
+Provides-Extra: parse
+Requires-Dist: parsel<2,>=1.9; extra == 'parse'
+Provides-Extra: resource-proxy
+Requires-Dist: dataapp-resource-proxy<0.2,>=0.1.0; extra == 'resource-proxy'
+Description-Content-Type: text/markdown
+
+# Data App Crawler core
+
+`dataapp-crawler` (imported as `dataapp_crawler`) is the provider-neutral website
+acquisition library for Python 3.11+. It can run independently of the DataApp
+Runtime. Core includes generic CDP/Playwright providers and an optional Camoufox
+adapter, but no Relay- or AWS-specific provider.
+
+> Dependency versions are defined in each package's `pyproject.toml`; an
+> application uses its own `uv.lock`. Local source changes are not delivered to
+> applications until matching packages and browser images are released and the
+> application's lock is updated.
+
+## Basic acquisition
+
+```python
+import asyncio
+
+from dataapp_crawler import Crawler
+
+
+async def main() -> None:
+    async with Crawler() as crawler:
+        capture = await crawler.fetch("https://example.com")
+        print(capture.status)
+
+
+asyncio.run(main())
+```
+
+Use an ordinary synchronous or asynchronous `process(capture, context)` function
+for extraction, OCR/ASR calls, model extraction, or custom logic. The SDK does
+not own a processing engine or Dataset:
+
+```python
+async def process(capture, context):
+    heading = capture.page.locator("h1").first
+    await context.run(heading.wait_for(state="visible"), timeout=10)
+    return {"title": (await context.run(heading.inner_text(), timeout=5)).strip()}
+```
+
+An operation timeout raises `TimeoutError`; exhausting `Limits.max_duration`
+raises `LimitReached`. Diagnostics are bounded and omit raw credentials, headers,
+connection URLs, response bodies, and arbitrary upstream exception text.
+
+## Page-driven flows
+
+`Crawler.run_browser(collect)` hands one native Playwright `Page` and a
+`CrawlContext` to ordinary application code. A single collection function can
+submit a search, traverse list pages, open detail pages, and stop on business
+conditions. It does not require one URL per `Crawler` object.
+
+```python
+from dataapp_crawler import Crawler, Limits
+from dataapp_crawler.browser_tools import open_url, text_of
+
+
+async def collect(page, context):
+    await open_url(page, "https://example.com", context)
+    return {"title": await text_of(page.locator("h1"), context, required=True)}
+
+
+async with Crawler(browser=browser_provider, limits=Limits(max_duration=120)) as crawler:
+    record = await crawler.run_browser(collect)
+```
+
+`browser_tools` includes bounded navigation and extraction helpers such as
+`open_url`, `new_page`, `popup_page`, `same_tab`, `wait_until`, `text_of`,
+`attribute_of`, and `iterate_list`. Page and Locator remain native Playwright
+objects. Raw Playwright actions remain available, but they do not automatically
+gain Crawler accounting, scheduling, diagnostics, or retry semantics.
+
+The application defines selectors, stable business keys, pagination, readiness,
+and side-effect boundaries. Do not automatically retry clicks or submissions
+that can create duplicate actions.
+
+Failure summaries include a fixed `exception_type` without its message. Browser
+flow failures also report `root_host`, the root Page's hostname only. This may be
+the list page rather than a failing detail tab. The original exception still
+propagates, with cancellation and cleanup unchanged. A failure before the root
+Page exists reports `phase=page_creation`.
+
+To inspect a failing detail page, explicitly capture a bounded screenshot inside
+that page's scope before it closes. Keep artifacts private, outside uploaded
+source packages and logs. `diagnostics_dir` applies to request-driven captures;
+it does not automatically snapshot every native Page action in `run_browser`.
+An exception type or screenshot does not by itself establish whether the target
+site blocked access, returned a different layout, or had not finished loading.
+
+## Read-only CAPTCHA observations
+
+`detect_captchas(html, ...)` and `await detect_captchas_on_page(page, ...)` classify
+observable reCAPTCHA v2/Enterprise, Turnstile, Cloudflare full-page Challenge,
+BotDetect, and image-CAPTCHA
+integrations. Results contain fixed evidence codes, component matches, and missing
+inputs. Script-only or unsupported evidence remains `unknown`; a form widget is
+distinct from an access challenge. `CaptchaOverride` accepts explicit site knowledge.
+
+Detection does not navigate, click, capture images, submit forms, or contact paid
+services. A solver match is structural readiness, not authorization or proof of
+website acceptance. `capture_image(locator)` separately captures the currently
+rendered challenge for an explicitly chosen resource call. Keep submission and
+site acceptance checks in ordinary application code.
+
+The candidate `CloudflareChallengeCapture` API observes dynamic full-page
+render parameters when installed before navigation, and applies a managed
+token only to the same active challenge. See the [token workflow](docs/cloudflare-challenge.md).
+The local contract and browser fixture have passed; real-site managed solving
+and deployment acceptance remain pending. Ordinary Turnstile and full-page
+Challenge tasks have different required fields. Detection alone never enables
+a paid call, and the `cf_clearance` cookie mode is not exposed.
+
+See the source template's [API reference](../../examples/crawler/relay_template/help/docs/API.zh-CN.md)
+for parameters and the Runtime/resource boundary.
+
+## Provider extensions
+
+Supplier-specific provider implementations are separate distributions:
+
+| Distribution | Import | Public provider types |
+|---|---|---|
+| `dataapp-crawler-octoparse-browser` | `dataapp_crawler_octoparse` | `BrowserEnvRelay`, `RelayBrowser` |
+| `dataapp-crawler-agentcore` | `dataapp_crawler_agentcore` | `AgentCoreBrowser`, `AWSBrowser` |
+
+The core wheel retains `CDPBrowser`, `PlaywrightBrowser`, and optional
+`CamoufoxBrowser` as generic integrations. It does not contain either supplier
+implementation and does not depend on Relay or boto3. Applications install one
+supplier extension deliberately and import its provider from that extension.
+This makes an accidental supplier-provider import fail at project-install time
+instead of silently exposing an unsupported runtime path.
+
+Source packages may share this repository; that does not merge their wheel
+contents, dependency graphs, project templates, or base images.
+
+## Proxy boundary
+
+Browser and proxy selection are independent. Core provides the common proxy
+contract and supports:
+
+- direct access with `proxy=None`;
+- a developer-provided `StaticProxy`;
+- a managed `DataOSProxy` through the Resource Proxy client;
+- a custom `ProxyProvider.acquire()` returning a `ProxyHandle`.
+
+`DataOSProxy.proxy_group` accepts `auto`, `domestic_basic`,
+`overseas_basic`, or `overseas_enhanced`. `auto` uses the basic managed
+service. The SDK does not select or name proxy suppliers: Resource Proxy owns
+the published supplier order and validates location and duration support.
+
+If managed proxy acquisition fails, `DataOSProxy` raises `ConfigurationError`
+with bounded `code` and `status_code` attributes. These diagnostics contain only
+the documented resource error class and HTTP status; supplier responses, proxy
+URLs, and credentials are never retained in the exception.
+
+A proxy handle is tied to a session lifecycle. It is not a promise to rotate the
+exit IP on every request. Credentials must remain outside inputs, source files,
+logs, and build arguments.
+
+## DataApp integration
+
+Within a Data App, use `DataAppCrawler` from
+`bazhuayu_dataapp.plugins.crawler`:
+
+```python
+from bazhuayu_dataapp import DataApp
+from bazhuayu_dataapp.plugins.crawler import DataAppCrawler
+
+
+async with DataApp:
+    async with DataAppCrawler(browser=browser_provider) as crawler:
+        records = await crawler.run_browser(collect)
+    await DataApp.push_data(records)
+```
+
+Runtime owns Run input, Dataset writes, progress, cancellation, and final state.
+`DataAppCrawler` only joins Crawler cancellation and cleanup to that lifecycle.
+
+## Logging
+
+The standalone Crawler writes INFO stage logs to stderr by default. The DataApp
+adapter uses `DataApp.log`, so the platform receives the same bounded stages.
+Pass a standard logger to change verbosity:
+
+```python
+import logging
+
+logger = logging.getLogger("my_app.crawler")
+logger.setLevel(logging.WARNING)
+
+async with Crawler(logger=logger) as crawler:
+    capture = await crawler.fetch(target_url)
+```
+
+The SDK does not configure the root logger and never logs the value returned by a
+process callback. Acquisition success is not Dataset success; the application
+must push records explicitly and validate the Runtime outcome separately.

dataapp_crawler-0.1.0.dist-info/RECORD

@@ -0,0 +1,36 @@
+dataapp_crawler/__init__.py,sha256=OTt7M2zw3Idh1q5faFWBHoTsth1vexEW0eCC97s4BRs,2204
+dataapp_crawler/__main__.py,sha256=t7XCXtBYjYbAEJ8IPYHn6vksltGKStvU9WqzcAg2Av8,1136
+dataapp_crawler/_logging.py,sha256=opVU_5vMTkpatuNB5wFbNhboy3aR0D-ncIKt5NQgtzM,5522
+dataapp_crawler/_network_diagnostics.py,sha256=zr8JkLY5t7tts1-XpOPwOzk-4fn6oRDCddEz6gCnX0M,6270
+dataapp_crawler/captcha.py,sha256=DBY0gjo6xngyTBFivsC078836Q7UXCv7xeOGpcmLhwM,33763
+dataapp_crawler/checkpoint.py,sha256=1LO3pLF0hdenYas7zwgEhVraydCjrwOGWueK9xmQY-E,18104
+dataapp_crawler/cloudflare.py,sha256=ee7aASD6zSU7pGmle2xBN0J6xnxwyEdrOqKBFIjGMIw,20888
+dataapp_crawler/context.py,sha256=skW6-9TBUCT5EeZP5WsUJXHrsHIdEBDNv_Osa-NcTmA,9370
+dataapp_crawler/cookies.py,sha256=8abIokM5jeIR7QoKxNhsEoKX3dTiXNdwfz6yBHdSeco,6938
+dataapp_crawler/crawler.py,sha256=mkokLOq2mXlJNUhjbfGXMAESn1nYp6wXKAzYwAWmHkA,63818
+dataapp_crawler/errors.py,sha256=y4Jx21weWZonMV62iOwSN__0FSzJxBPA0M8jvAEUPYU,6351
+dataapp_crawler/http.py,sha256=pHmqhAhLbaa9FkEdV7ZIa9ZIPRpVGYolGuhlVE7_9c4,9621
+dataapp_crawler/images.py,sha256=Z5v1pxYHL7nq16DjG8Ih1OUkfjmmBV-sG5qDuvn64XQ,10744
+dataapp_crawler/local_auth.py,sha256=7ZqdCFhSj_P3BqZzHpfiObMcdjxLMfQaObGCmzWbInE,2762
+dataapp_crawler/models.py,sha256=6IGtKimFwJQznj9X1E9Osboz8HvCDGRft09ds63iTdA,11124
+dataapp_crawler/pagination.py,sha256=IqpAr5bK9vqUqdX3we8jOI5RLpkct1YHAQyO_FhJEuk,6675
+dataapp_crawler/parsing.py,sha256=dyTJaPiezNRih8PVF__fY1zjyx0cbcXzWjUgtesmr5s,361
+dataapp_crawler/py.typed,sha256=47DEQpj8HBSa-_TImW-5JCeuQeRkm5NMpJWZG3hSuFU,0
+dataapp_crawler/routing.py,sha256=zfoJo23c5ybPc9OQZlNyA-cKn-8gpF3gvVvdiWU0yrA,2502
+dataapp_crawler/scheduling.py,sha256=o-trbcnwr-am4IkK8fm3P2kchLb4LKoHfiHXJ2suqNU,17103
+dataapp_crawler/sessions.py,sha256=J_WNE800abJYoP7K-GnhbvnpVMixx-3yAZq2fcmngoE,29173
+dataapp_crawler/browser_tools/__init__.py,sha256=K6qSh7wQjsTde0ae31XpdQsQ5zykzVp-LRDiNrfuPTc,645
+dataapp_crawler/browser_tools/_common.py,sha256=py6Z-VMgCc1sb-kayjAMQO8fGUWuKfQgeTU37bcKCjs,1922
+dataapp_crawler/browser_tools/dom.py,sha256=LjACkzOsSjmFnixz2Wa1ZxgoQH9RFKMj5iZBr4Lm2uY,1442
+dataapp_crawler/browser_tools/lists.py,sha256=6o9W9SgBXqDBS2xNt0pkpDzgIQH2g1VHWUQl-GHJGE8,4699
+dataapp_crawler/browser_tools/navigation.py,sha256=-l1PJgx60tmEFs-O8NNQSygYHgXSTomOeHIJu4qW3Gs,8813
+dataapp_crawler/browser_tools/readiness.py,sha256=X-b__NsmNbANT1VDFR-jFbhKAAXldr-FBBdXtyxK8v4,1365
+dataapp_crawler/providers/__init__.py,sha256=4A_xqgasRHuqj3rSmk_y1yrNGbfFuBlw6ancpEe-1zg,537
+dataapp_crawler/providers/browser.py,sha256=RaBhNEtfJ8RqfLHqHAWlM5Qv8FKXJYkti2eL0p5eNg0,10701
+dataapp_crawler/providers/camoufox.py,sha256=kcj84MGSY6m3_Ad5EgsENuSgNpnECKY5ZRw2VCy3gkM,6941
+dataapp_crawler/providers/dataos.py,sha256=xWc_5PIiYmqRuthUcX1bVl5y2rx2X-5InbwhC0Jyu_k,5823
+dataapp_crawler/providers/pool.py,sha256=TBpPHOTBSJfIsG9YuzvGqVrJzauspPItdp7DhqiQqV0,17511
+dataapp_crawler/providers/proxy.py,sha256=_1PPxdAA-Ris9btKYTx3uFBP98N6Zw1ZBRY1GN719mA,7697
+dataapp_crawler-0.1.0.dist-info/METADATA,sha256=dfnqV6FDa1LG2lvfINZaa_DTX9Gm-ebZha7pEBJbfT8,9065
+dataapp_crawler-0.1.0.dist-info/WHEEL,sha256=2iZrJkDIkeDh0k9SikKMqFuRP-ua6mb4Ir1AsER4BEw,87
+dataapp_crawler-0.1.0.dist-info/RECORD,,

dataapp_crawler-0.1.0.dist-info/WHEEL

@@ -0,0 +1,4 @@
+Wheel-Version: 1.0
+Generator: hatchling 1.33.0
+Root-Is-Purelib: true
+Tag: py3-none-any