dataapp-crawler-octoparse-browser 0.1.0__tar.gz

dataapp_crawler_octoparse_browser-0.1.0/PKG-INFO

@@ -0,0 +1,188 @@
+Metadata-Version: 2.5
+Name: dataapp-crawler-octoparse-browser
+Version: 0.1.0
+Summary: Octoparse browser providers for DataApp Crawler
+Requires-Python: >=3.11
+Requires-Dist: dataapp-crawler<0.2,>=0.1.0
+Requires-Dist: octoparse-relay<0.2,>=0.1.10
+Requires-Dist: playwright<2,>=1.64
+Provides-Extra: managed
+Requires-Dist: dataapp-resource-proxy<0.2,>=0.1.0; extra == 'managed'
+Description-Content-Type: text/markdown
+
+# Octoparse Crawler SDK
+
+Octoparse browser providers for DataApp Crawler.
+
+The accompanying browser-image source is maintained in
+[`browserenv/`](browserenv/README.md). Python provider packages and the BrowserEnv
+image share this repository but are built and released separately; BrowserEnv is
+excluded from the Python wheel and sdist.
+
+Install the browser package from the DataOS development index. It installs
+Crawler and the Relay Python transport as dependencies. The Python import path
+is `dataapp_crawler_octoparse`.
+
+## Install
+
+```bash
+pip install --extra-index-url https://nexus.bazhuayu.com/repository/pypi-dataos-dev/simple/ \
+  dataapp-crawler-octoparse-browser==0.1.0.dev4
+```
+
+## Providers
+
+- `BrowserEnvRelay` connects to the Relay browser supplied by the DataApp Relay image.
+- `RelayBrowser` starts and owns a local browser, Relay extension, and Relay server.
+
+```python
+from dataapp_crawler_octoparse import BrowserEnvRelay
+
+browser = BrowserEnvRelay()
+```
+
+The SDK's BrowserEnv image bundles CloakBrowser and runs it in headed mode,
+using Xvfb when needed. Applications connect through `BrowserEnvRelay`.
+
+`RelayBrowser` is an advanced self-managed path, not the Relay Data App template
+default. A fresh post-fix local candidate batch passed full Relay pairing,
+HTTP/HTTPS DOM work, owned-process cleanup, and container cleanup with Chrome
+151 in headless and Xvfb modes, CloakBrowser 146 headless, and Chromium 156
+headless. Earlier Chrome pairing and CloakBrowser headless failures remain useful
+pre-fix history, not the current result. These are versioned local candidate
+checks, not universal browser or target-site guarantees; proxy combinations were
+not repeated across every browser. The published clean `rc.3` template image is
+a separate release artifact from the locally tested
+`dataos-octoparse-integration:20261108-managed` candidate.
+
+BrowserEnv proxy control uses `DATAAPP_BROWSER_PROXY_API_URL`,
+`DATAAPP_BROWSER_PROXY_API_USERNAME`, and `DATAAPP_BROWSER_PROXY_API_PASSWORD`.
+It also accepts the legacy `OCTOPUS_PROXY_SOFTWARE_API_*` settings when their
+canonical equivalents are absent. The control endpoint must be HTTP on
+loopback. An upstream `https://` proxy uses TLS with certificate and hostname
+verification.
+
+In a DataHub Run, choose `DataOSProxy(DataApp.resources())` as the
+`DataAppCrawler` proxy provider to attribute browser traffic to the Run.
+Select a lease duration that covers the browser work. Renewal depends on the
+configured supplier.
+
+## CAPTCHA and images
+
+AWS WAF image selection and controlled extension grants are internal C10/C07
+experiments. The current platform Runtime does not expose their routes.
+Direct local Resource Proxy experiments require matching Relay, extension and
+server candidates. Local protocol checks do not establish real-site submission,
+clearance, data collection or availability through a platform Run.
+
+CAPTCHA solving is opt-in. Install the `managed` extra with the local candidate
+Resource Proxy client. In local development, open `ResourceProxy.from_env()`;
+platform applications use `DataApp.resources()` when the corresponding
+Runtime/DataHub routes are enabled. Model routes still require that platform
+integration; local support does not imply availability on pre.
+
+For local managed-resource testing, save the current environment's DataOS API
+Key in a regular text file containing exactly one non-empty line, then run:
+
+```bash
+python -m dataapp_crawler run --api-key-file /path/to/api-key.txt -- python app.py
+```
+
+The source file cannot be a symlink or directory. The local launcher reads it locally and injects it only into the child process. Resource Proxy
+receives the normal Authorization header, never the source file. Do not paste the
+Key into chat or package the source file with the App. Platform Runs use the
+injected Runtime identity.
+
+The typed API accepts embedded Turnstile, reCAPTCHA v2, Enterprise v2, and
+image-to-text tasks. Widget tasks return `.token`; image tasks return `.text`.
+Module selection chooses a recognizer for a known image technology, rather
+than detecting what a website uses:
+
+```python
+from dataapp_crawler.images import capture_image
+
+image = await context.run(capture_image(page.locator("#captcha-image")))
+task = await context.run(resources.solve_captcha(
+    type="image_to_text",
+    image_body=image.image_body,
+    capmonster_module="botdetect",  # Only for a known BotDetect challenge.
+))
+```
+
+`capture_image()` defaults to a visible rendered-element capture and does not
+request a mutable image URL again. Chromium uses a clipped CDP screenshot when
+available and otherwise falls back to the native Locator screenshot; hidden or
+zero-size elements fail safely. When the developer has established that a
+loaded `IMG` is the current challenge, `source="loaded_image"` explicitly copies
+its pixels through Canvas without another GET and remains subject to cross-origin
+pixel restrictions. The same image's `data_url` can be passed to
+`resources.chat_completion()` as an inline model image. These small helpers do
+not detect, solve, submit, or verify a challenge, and they do not turn `process`
+into a processing framework. The developer still checks the
+challenge identity, applies the candidate, checks the site verdict, and extracts
+business data. Use the matching version's capability acceptance matrix to check
+which task and site flows have been validated.
+
+In the final real Relay check, default capture rejected the hidden zero-size IMG,
+while explicit `loaded_image` returned its current PNG bytes without another
+request. A visible wrapper capture no longer timed out, but that page's wrapper
+pixels were blank; a successful screenshot therefore does not prove it contains
+the challenge image or that a solver or site will accept it.
+
+The managed cloud API through `DataApp.resources()` and Resource Proxy is the
+default public solving path. Application code must explicitly create a bounded
+task after the developer has enabled metered solving; CAPTCHA detection alone
+never creates a paid task. In a local Runtime/DataHub resource-bridge acceptance,
+the managed cloud API
+passed one public reCAPTCHA Enterprise v2 demonstration form through real token
+creation, browser submission, server acceptance, and Dataset output. BotDetect
+image and vision-model candidates were submitted but rejected by the
+demonstration site. Turnstile and regular reCAPTCHA v2 did not receive the same
+acceptance test. Detection only classifies evidence; it does not solve a
+challenge. None of these results establishes success on a business site.
+
+### Native CapMonster extension with a developer-owned key (advanced opt-in)
+
+For trusted local development, a developer may explicitly configure
+BrowserEnv's bundled native CapMonster extension before browser startup:
+
+```bash
+# CAPMONSTER_API_KEY is already loaded from the developer's local secret store.
+export CAPMONSTER_API_KEY
+export CAPMONSTER_CAPTCHA_TYPES='ReCaptcha2,ReCaptchaEnterprise,Turnstile'
+xvfb-run -a octoparse-relay-task -- uv run python app.py
+```
+
+`CAPMONSTER_CAPTCHA_TYPES` is a comma-separated allowlist. Each selected type
+must be supported by the extension bundle in the chosen image. The key belongs
+to the developer and is not a shared platform key. Do not commit it, pass it as
+a build argument, write it to logs or Dataset records, or expose it in errors.
+The launcher prepares a private extension copy before browser startup and removes
+the two variables from the child application environment; the key still enters
+the third-party extension context.
+
+This BYOK path is currently limited to trusted local sessions. The launcher
+rejects raw CapMonster keys when DataHub platform identity is present, so this
+configuration is not a supported production DataApp path. A Resource Proxy
+grant-backed automatic extension integration is under development and is not a
+public product API. Use the explicit managed cloud API where platform credential
+isolation is required. Do not run native automatic solving and explicit API
+solving against the same challenge. Native Enterprise end-to-end site acceptance
+has not yet been established.
+
+## Documentation and examples
+
+The Relay project template ships the following together:
+
+- [Developer guide](../../examples/crawler/relay_template/help/docs/DEVELOPER_GUIDE.zh-CN.md):
+  browser lifecycles, extension input behavior, fingerprints, proxies, and limits.
+- [API reference](../../examples/crawler/relay_template/help/docs/API.zh-CN.md): Provider,
+  Crawler helper, and advanced low-level Relay signatures and return values.
+- [Importable scenarios](../../examples/crawler/relay_template/help/examples/README.md):
+  pointer observation, scoped navigation, readiness, and extraction.
+- [Complete search/list/detail flow](../../examples/crawler/relay_template/site_adapter.py):
+  the site's business flow used by the Runtime `app.py` entry.
+
+`Page` and `Locator` remain native Playwright objects. Pointer trajectory handling
+is implemented by the extension; it is not an additional Python `human_click()`
+method. Browser fingerprint behavior belongs to the selected browser environment.

dataapp_crawler_octoparse_browser-0.1.0/README.md

@@ -0,0 +1,176 @@
+# Octoparse Crawler SDK
+
+Octoparse browser providers for DataApp Crawler.
+
+The accompanying browser-image source is maintained in
+[`browserenv/`](browserenv/README.md). Python provider packages and the BrowserEnv
+image share this repository but are built and released separately; BrowserEnv is
+excluded from the Python wheel and sdist.
+
+Install the browser package from the DataOS development index. It installs
+Crawler and the Relay Python transport as dependencies. The Python import path
+is `dataapp_crawler_octoparse`.
+
+## Install
+
+```bash
+pip install --extra-index-url https://nexus.bazhuayu.com/repository/pypi-dataos-dev/simple/ \
+  dataapp-crawler-octoparse-browser==0.1.0.dev4
+```
+
+## Providers
+
+- `BrowserEnvRelay` connects to the Relay browser supplied by the DataApp Relay image.
+- `RelayBrowser` starts and owns a local browser, Relay extension, and Relay server.
+
+```python
+from dataapp_crawler_octoparse import BrowserEnvRelay
+
+browser = BrowserEnvRelay()
+```
+
+The SDK's BrowserEnv image bundles CloakBrowser and runs it in headed mode,
+using Xvfb when needed. Applications connect through `BrowserEnvRelay`.
+
+`RelayBrowser` is an advanced self-managed path, not the Relay Data App template
+default. A fresh post-fix local candidate batch passed full Relay pairing,
+HTTP/HTTPS DOM work, owned-process cleanup, and container cleanup with Chrome
+151 in headless and Xvfb modes, CloakBrowser 146 headless, and Chromium 156
+headless. Earlier Chrome pairing and CloakBrowser headless failures remain useful
+pre-fix history, not the current result. These are versioned local candidate
+checks, not universal browser or target-site guarantees; proxy combinations were
+not repeated across every browser. The published clean `rc.3` template image is
+a separate release artifact from the locally tested
+`dataos-octoparse-integration:20261108-managed` candidate.
+
+BrowserEnv proxy control uses `DATAAPP_BROWSER_PROXY_API_URL`,
+`DATAAPP_BROWSER_PROXY_API_USERNAME`, and `DATAAPP_BROWSER_PROXY_API_PASSWORD`.
+It also accepts the legacy `OCTOPUS_PROXY_SOFTWARE_API_*` settings when their
+canonical equivalents are absent. The control endpoint must be HTTP on
+loopback. An upstream `https://` proxy uses TLS with certificate and hostname
+verification.
+
+In a DataHub Run, choose `DataOSProxy(DataApp.resources())` as the
+`DataAppCrawler` proxy provider to attribute browser traffic to the Run.
+Select a lease duration that covers the browser work. Renewal depends on the
+configured supplier.
+
+## CAPTCHA and images
+
+AWS WAF image selection and controlled extension grants are internal C10/C07
+experiments. The current platform Runtime does not expose their routes.
+Direct local Resource Proxy experiments require matching Relay, extension and
+server candidates. Local protocol checks do not establish real-site submission,
+clearance, data collection or availability through a platform Run.
+
+CAPTCHA solving is opt-in. Install the `managed` extra with the local candidate
+Resource Proxy client. In local development, open `ResourceProxy.from_env()`;
+platform applications use `DataApp.resources()` when the corresponding
+Runtime/DataHub routes are enabled. Model routes still require that platform
+integration; local support does not imply availability on pre.
+
+For local managed-resource testing, save the current environment's DataOS API
+Key in a regular text file containing exactly one non-empty line, then run:
+
+```bash
+python -m dataapp_crawler run --api-key-file /path/to/api-key.txt -- python app.py
+```
+
+The source file cannot be a symlink or directory. The local launcher reads it locally and injects it only into the child process. Resource Proxy
+receives the normal Authorization header, never the source file. Do not paste the
+Key into chat or package the source file with the App. Platform Runs use the
+injected Runtime identity.
+
+The typed API accepts embedded Turnstile, reCAPTCHA v2, Enterprise v2, and
+image-to-text tasks. Widget tasks return `.token`; image tasks return `.text`.
+Module selection chooses a recognizer for a known image technology, rather
+than detecting what a website uses:
+
+```python
+from dataapp_crawler.images import capture_image
+
+image = await context.run(capture_image(page.locator("#captcha-image")))
+task = await context.run(resources.solve_captcha(
+    type="image_to_text",
+    image_body=image.image_body,
+    capmonster_module="botdetect",  # Only for a known BotDetect challenge.
+))
+```
+
+`capture_image()` defaults to a visible rendered-element capture and does not
+request a mutable image URL again. Chromium uses a clipped CDP screenshot when
+available and otherwise falls back to the native Locator screenshot; hidden or
+zero-size elements fail safely. When the developer has established that a
+loaded `IMG` is the current challenge, `source="loaded_image"` explicitly copies
+its pixels through Canvas without another GET and remains subject to cross-origin
+pixel restrictions. The same image's `data_url` can be passed to
+`resources.chat_completion()` as an inline model image. These small helpers do
+not detect, solve, submit, or verify a challenge, and they do not turn `process`
+into a processing framework. The developer still checks the
+challenge identity, applies the candidate, checks the site verdict, and extracts
+business data. Use the matching version's capability acceptance matrix to check
+which task and site flows have been validated.
+
+In the final real Relay check, default capture rejected the hidden zero-size IMG,
+while explicit `loaded_image` returned its current PNG bytes without another
+request. A visible wrapper capture no longer timed out, but that page's wrapper
+pixels were blank; a successful screenshot therefore does not prove it contains
+the challenge image or that a solver or site will accept it.
+
+The managed cloud API through `DataApp.resources()` and Resource Proxy is the
+default public solving path. Application code must explicitly create a bounded
+task after the developer has enabled metered solving; CAPTCHA detection alone
+never creates a paid task. In a local Runtime/DataHub resource-bridge acceptance,
+the managed cloud API
+passed one public reCAPTCHA Enterprise v2 demonstration form through real token
+creation, browser submission, server acceptance, and Dataset output. BotDetect
+image and vision-model candidates were submitted but rejected by the
+demonstration site. Turnstile and regular reCAPTCHA v2 did not receive the same
+acceptance test. Detection only classifies evidence; it does not solve a
+challenge. None of these results establishes success on a business site.
+
+### Native CapMonster extension with a developer-owned key (advanced opt-in)
+
+For trusted local development, a developer may explicitly configure
+BrowserEnv's bundled native CapMonster extension before browser startup:
+
+```bash
+# CAPMONSTER_API_KEY is already loaded from the developer's local secret store.
+export CAPMONSTER_API_KEY
+export CAPMONSTER_CAPTCHA_TYPES='ReCaptcha2,ReCaptchaEnterprise,Turnstile'
+xvfb-run -a octoparse-relay-task -- uv run python app.py
+```
+
+`CAPMONSTER_CAPTCHA_TYPES` is a comma-separated allowlist. Each selected type
+must be supported by the extension bundle in the chosen image. The key belongs
+to the developer and is not a shared platform key. Do not commit it, pass it as
+a build argument, write it to logs or Dataset records, or expose it in errors.
+The launcher prepares a private extension copy before browser startup and removes
+the two variables from the child application environment; the key still enters
+the third-party extension context.
+
+This BYOK path is currently limited to trusted local sessions. The launcher
+rejects raw CapMonster keys when DataHub platform identity is present, so this
+configuration is not a supported production DataApp path. A Resource Proxy
+grant-backed automatic extension integration is under development and is not a
+public product API. Use the explicit managed cloud API where platform credential
+isolation is required. Do not run native automatic solving and explicit API
+solving against the same challenge. Native Enterprise end-to-end site acceptance
+has not yet been established.
+
+## Documentation and examples
+
+The Relay project template ships the following together:
+
+- [Developer guide](../../examples/crawler/relay_template/help/docs/DEVELOPER_GUIDE.zh-CN.md):
+  browser lifecycles, extension input behavior, fingerprints, proxies, and limits.
+- [API reference](../../examples/crawler/relay_template/help/docs/API.zh-CN.md): Provider,
+  Crawler helper, and advanced low-level Relay signatures and return values.
+- [Importable scenarios](../../examples/crawler/relay_template/help/examples/README.md):
+  pointer observation, scoped navigation, readiness, and extraction.
+- [Complete search/list/detail flow](../../examples/crawler/relay_template/site_adapter.py):
+  the site's business flow used by the Runtime `app.py` entry.
+
+`Page` and `Locator` remain native Playwright objects. Pointer trajectory handling
+is implemented by the extension; it is not an additional Python `human_click()`
+method. Browser fingerprint behavior belongs to the selected browser environment.

dataapp_crawler_octoparse_browser-0.1.0/pyproject.toml

@@ -0,0 +1,35 @@
+[build-system]
+requires = ["hatchling>=1.27"]
+build-backend = "hatchling.build"
+
+[project]
+name = "dataapp-crawler-octoparse-browser"
+version = "0.1.0"
+description = "Octoparse browser providers for DataApp Crawler"
+readme = "README.md"
+requires-python = ">=3.11"
+dependencies = [
+  "dataapp-crawler>=0.1.0,<0.2",
+  "playwright>=1.64,<2",
+  "octoparse-relay>=0.1.10,<0.2",
+]
+
+[project.optional-dependencies]
+managed = ["dataapp-resource-proxy>=0.1.0,<0.2"]
+
+[tool.uv.sources]
+dataapp-crawler = {path = "../crawler", editable = true}
+octoparse-relay = {path = "../../../relay", editable = true}
+
+[tool.hatch.build.targets.wheel]
+packages = ["src/dataapp_crawler_octoparse"]
+
+[tool.hatch.build.targets.sdist]
+only-include = ["src", "README.md", "pyproject.toml"]
+
+[tool.ruff]
+target-version = "py311"
+line-length = 100
+
+[tool.ruff.lint]
+select = ["E", "F", "I", "UP", "B"]

dataapp_crawler_octoparse_browser-0.1.0/src/dataapp_crawler_octoparse/__init__.py

@@ -0,0 +1,13 @@
+"""Octoparse browser provider extension for DataApp Crawler."""
+
+from .browserenv import BrowserEnvRelay
+from .captcha import CaptchaType, ManagedCaptchaPlugin, PreparedCaptchaExtension
+from .relay import RelayBrowser
+
+__all__ = [
+    "BrowserEnvRelay",
+    "CaptchaType",
+    "ManagedCaptchaPlugin",
+    "PreparedCaptchaExtension",
+    "RelayBrowser",
+]