scrapely 1.2.6 → 1.2.8

package/README.md

@@ -1,118 +1,65 @@
-# Scrapely
-
-Scrapely is an Actor-based web scraping platform. Developers push an Actor (a scraping program), the platform builds it into a Docker image, runs it with structured input, and keeps its output in datasets, key-value stores and request queues. The API and SDKs follow Apify's shape, so most Apify Actors run with few or no changes.
-
-This repo holds the platform server, the Actor SDKs, the API clients, the CLI and the MCP servers.
-
-## Repo layout
-
-| Path | What it is |
-|------|------------|
-| `src/` → `dist/` | Node.js Actor SDK, published as `scrapely`. `dist/` is committed because it is the package output. |
-| `scrapely-python/` | Python Actor SDK (`scrapely` on PyPI) |
-| `scrapely-server/` | API server, build/run workers, autoscaler, egress proxy |
-| `scrapely-cli/` | `scrapely` CLI: login, create/init, push, pull, run, builds, runs, storages, secrets |
-| `scrapely-client/` | JavaScript API client (`scrapely-client`, drop-in for `apify-client`) |
-| `scrapely-client-python/` | Python API client (`scrapely-client-python`) |
-| `scrapely-mcp/` | MCP server (TypeScript) |
-| `scrapely-mcp-python/` | MCP server (Python) |
-| `docker/` | Runner and Actor base images, Dragonfly compose file, Kaniko patch |
-| `docs/` | Platform documentation. Start at [docs/README.md](docs/README.md). |
-| `scripts/` | Server setup, deploy, migrations, benchmarks |
-| `tests/` | SDK/runtime tests |
-| `examples/basic-actor/` | Smallest working Actor |
-| `npm-claim-scrapely/` | Old placeholder for the `scrapely` npm name, now superseded by the SDK itself (1.2.4+). |
-
-Each package has its own README with install and usage details.
-
-## How it fits together
-
-- The **API server** (`scrapely-server/`) serves the Apify-style `/v2` API: auth, Actors and versions, builds, runs, tasks, schedules, webhooks, storages, billing and admin routes.
-- **Builds and runs** are queued with BullMQ and picked up by workers. Builds produce Docker images that go to a private registry. Runs execute in Docker on worker servers.
-- **The autoscaler** adds and removes worker servers based on load. See [docs/autoscaling.md](docs/autoscaling.md).
-- **Storage**: run state and storages live in DragonflyDB (Redis-compatible). Accounts, Actors, runs and billing live in SingleStore. Both are backed up to S3.
-- **Browsers**: remote browsers come from Evomi over CDP. See [docs/evomi-integration.md](docs/evomi-integration.md).
-- **Actor code** imports the SDK, which patches Crawlee storage to go through the platform:
+# Scrapely Node.js SDK
 
-```ts
-import { Actor, PlaywrightCrawler, Dataset } from 'scrapely';
-```
-
-```python
-from scrapely import Actor
-```
-
-## Local development
+Build JavaScript and TypeScript Actors for the [Scrapely](https://scrape.ly) platform. The SDK gives you the Actor lifecycle, input and output, datasets, key-value stores and request queues, Crawlee crawlers, remote browsers over CDP, platform events, run control, webhooks and pay-per-event charging.
 
-### Prerequisites
-
-- Node.js 22
-- Docker
-- Doppler CLI with access to the `scrapely` project (all secrets come from Doppler; there are no `.env.example` files)
-- Reachable DragonflyDB and SingleStore instances (Doppler `stg` points at staging)
-
-### Install
+## Install
 
 ```bash
-npm install
-cd scrapely-server && npm install
-cd ../scrapely-cli && npm install
+npm install scrapely
 ```
 
-### Run the API server
-
-```bash
-cd scrapely-server
-doppler run -p scrapely -c stg -- npm run dev
-curl http://localhost:3000/health
-```
+## Quick start
 
-To run a local Dragonfly instead of staging: `docker compose -f docker/docker-compose.dragonfly.yml up -d`.
+```ts
+import { Actor } from 'scrapely';
 
-### Build the runner image
+await Actor.main(async () => {
+    const input = await Actor.getInputOrThrow<{ url: string }>();
+    await Actor.pushData({ url: input.url, status: 'processed' });
+});
+```
 
-Needed before the server can run Actors locally:
+`Actor.main()` initializes the Actor, awaits your function, closes events and storage, and exits the process. Always `await` writes so they reach the platform before the process ends.
 
-```bash
-make build-runner
-```
+## Crawling
 
-### Push and run an Actor
+```ts
+import { Actor, PlaywrightCrawler } from 'scrapely';
 
-```bash
-scrapely login
-cd examples/basic-actor
-scrapely push
-scrapely run --wait
-```
+await Actor.main(async () => {
+    const { startUrls } = await Actor.getInputOrThrow<{ startUrls: string[] }>();
 
-## Tests
+    const crawler = new PlaywrightCrawler({
+        maxConcurrency: 5,
+        async requestHandler({ request, page }) {
+            await Actor.pushData({ url: request.url, title: await page.title() });
+        },
+    });
 
-```bash
-npm test                    # root SDK
-cd scrapely-server && npm test
+    await crawler.run(startUrls);
+});
 ```
 
-Both packages have many more targeted `test:*` scripts in their `package.json`. Most tests are integration-style, and some need a valid `CDP_URL` or live services.
+On the platform, browser crawlers connect to a remote Chromium over CDP instead of launching a local browser. Locally, they launch a normal browser unless you set `CDP_URL` or call `setCdpUrl()`.
 
-## Deploy
+## What's included
 
-| Command | What it does |
-|---------|--------------|
-| `npm run prod` | Full production deploy (`scripts/deploy-prod.sh`) |
-| `npm run prod:update` | Update-only deploy |
-| `npm run setup:server` | Bootstrap a fresh Linux server |
-| `npm run setup:gvisor` | Install gVisor on a Linux host |
-| `npm run migrate` | Run a SingleStore migration |
+- **Actor lifecycle:** `Actor.main()`, `init()`, `exit()`, `fail()`, status messages, reboot, metamorph and abort
+- **Input and output:** `getInput()`, `getInputOrThrow()`, `pushData()`, `getValue()`, `setValue()`
+- **Storage:** `Dataset`, `KeyValueStore` and `RequestQueue`, backed by the Scrapely API on the platform and by local files during development
+- **Crawlers:** `PlaywrightCrawler`, `PuppeteerCrawler`, `CheerioCrawler`, `HttpCrawler`, `BasicCrawler`, `BrowserCrawler`, `JSDOMCrawler` and `LinkeDOMCrawler`
+- **Remote browsers:** automatic CDP wiring through `CDP_URL`, or `setCdpUrl()`
+- **Events:** state persistence, migration and graceful shutdown
+- **Run control:** `Actor.call()`, `Actor.callTask()`, `Actor.start()` and ad-hoc webhooks
+- **Charging:** `Actor.charge()`, `ChargingManager` and `pushDataAndCharge()` for pay-per-event Actors
+- **Logging and utilities:** `log`, `utils` and `enqueueLinks`
 
-The main server uses `scrapely-server/docker-compose.yml` and workers use `docker-compose.worker.yml`. Secrets are injected at container start with `doppler run`.
+## Related packages
 
-`make start` / `stop` / `restart` / `logs` wrap the server compose stack.
+- [`scrapely-client`](https://www.npmjs.com/package/scrapely-client): JavaScript API client
+- [`scrapely-cli`](https://www.npmjs.com/package/scrapely-cli): create, run and push Actors from your terminal
+- [`scrapely-python`](https://pypi.org/project/scrapely-python/): the Python SDK
 
-## Where to look next
+## License
 
-- [docs/README.md](docs/README.md): documentation index (getting started, SDKs, CLI, webhooks, troubleshooting)
-- [scrapely-server/docs/API_ENDPOINTS.md](scrapely-server/docs/API_ENDPOINTS.md): API reference
-- [docs/network-architecture.md](docs/network-architecture.md): fleet topology and data path
-- `scrapely-server/src/index.ts`: server startup and route wiring
-- `src/actor.ts`: Node SDK Actor runtime
+ISC

package/dist/actor.d.ts

@@ -546,8 +546,8 @@ export declare class Actor {
      * });
      * ```
      *
-     * For compatibility with existing Actor Input UI (Input Schema), this function returns
-     * `undefined` when `{ useApifyProxy: false }` is passed.
+     * Pass the proxy input field (`editor: "proxy"`) as is: this function returns
+     * `undefined` when `{ useScrapelyProxy: false }` is passed without custom proxies.
      *
      * @param options - Proxy configuration options
      * @returns Promise resolving to ProxyConfiguration instance, or undefined

package/dist/actor.js

@@ -1015,8 +1015,8 @@ class Actor {
      * });
      * ```
      *
-     * For compatibility with existing Actor Input UI (Input Schema), this function returns
-     * `undefined` when `{ useApifyProxy: false }` is passed.
+     * Pass the proxy input field (`editor: "proxy"`) as is: this function returns
+     * `undefined` when `{ useScrapelyProxy: false }` is passed without custom proxies.
      *
      * @param options - Proxy configuration options
      * @returns Promise resolving to ProxyConfiguration instance, or undefined

package/dist/proxy-configuration.d.ts

@@ -53,14 +53,17 @@ export interface ProxyConfigurationOptions extends CoreProxyConfigurationOptions
     /**
      * Input-schema alias for groups. Normalized to groups.
      */
-    apifyProxyGroups?: string[];
+    scrapelyProxyGroups?: string[];
     /**
      * Input-schema alias for subdivisionCode. Normalized to subdivisionCode.
      */
-    apifyProxySubdivision?: string;
+    scrapelyProxySubdivision?: string;
     /**
      * Input-schema alias for countryCode. Normalized to countryCode.
      */
+    scrapelyProxyCountry?: string;
+    apifyProxyGroups?: string[];
+    apifyProxySubdivision?: string;
     apifyProxyCountry?: string;
     /**
      * Platform proxy password. Defaults to the password injected by the
@@ -73,9 +76,10 @@ export interface ProxyConfigurationOptions extends CoreProxyConfigurationOptions
  */
 export interface CreateProxyConfigurationOptions extends ProxyConfigurationOptions {
     /**
-     * For compatibility with existing Actor Input UI (Input Schema).
-     * When set to `false`, this method returns `undefined`.
+     * The proxy input field's switch (input schema `editor: "proxy"`). When
+     * `false` and no custom proxies are given, this method returns `undefined`.
      */
+    useScrapelyProxy?: boolean;
     useApifyProxy?: boolean;
     /**
      * Whether to verify the proxy configuration by testing a connection.

package/dist/proxy-configuration.js

@@ -39,14 +39,17 @@ function hasCustomProxyOptions(options) {
 }
 function normalizeProxyInput(options) {
     const normalized = { ...options };
-    if (normalized.apifyProxyGroups && !normalized.groups) {
-        normalized.groups = normalized.apifyProxyGroups;
+    const groups = normalized.scrapelyProxyGroups ?? normalized.apifyProxyGroups;
+    if (groups && !normalized.groups) {
+        normalized.groups = groups;
     }
-    if (normalized.apifyProxyCountry && !normalized.countryCode) {
-        normalized.countryCode = normalized.apifyProxyCountry;
+    const countryCode = normalized.scrapelyProxyCountry ?? normalized.apifyProxyCountry;
+    if (countryCode && !normalized.countryCode) {
+        normalized.countryCode = countryCode;
     }
-    if (normalized.apifyProxySubdivision && !normalized.subdivisionCode) {
-        normalized.subdivisionCode = normalized.apifyProxySubdivision;
+    const subdivisionCode = normalized.scrapelyProxySubdivision ?? normalized.apifyProxySubdivision;
+    if (subdivisionCode && !normalized.subdivisionCode) {
+        normalized.subdivisionCode = subdivisionCode;
     }
     return normalized;
 }
@@ -60,17 +63,19 @@ function normalizeProxyInput(options) {
  */
 async function createProxyConfiguration(options = {}) {
     const normalized = normalizeProxyInput(options);
-    const { useApifyProxy, checkAccess = true, ...proxyOptions } = normalized;
-    // Apify parity: `useApifyProxy: false` explicitly disables the PLATFORM proxy
+    const { useScrapelyProxy, useApifyProxy, checkAccess = true, ...proxyOptions } = normalized;
+    // `useScrapelyProxy: false` (or the legacy `useApifyProxy: false`; a boolean
+    // useScrapelyProxy wins) explicitly disables the PLATFORM proxy
     // only — custom proxies from the same input must still be used. The standard
-    // "Custom proxy URLs" input shape is `{ useApifyProxy: false, proxyUrls: [...] }`;
+    // "Custom proxy URLs" input shape is `{ useScrapelyProxy: false, proxyUrls: [...] }`;
     // returning undefined for it would silently drop the user's own proxies and
     // crawl on their real IPs. We only bail when NO custom proxy mode is given —
     // proxyUrls, tieredProxyUrls, tieredProxyConfig and newUrlFunction all count
     // (Python SDK Actor.create_proxy_configuration parity).
-    const dontUseApifyProxy = useApifyProxy === false;
+    const usePlatformProxy = typeof useScrapelyProxy === 'boolean' ? useScrapelyProxy : useApifyProxy;
+    const dontUsePlatformProxy = usePlatformProxy === false;
     const dontUseCustomProxies = !hasCustomProxyOptions(proxyOptions);
-    if (dontUseApifyProxy && dontUseCustomProxies) {
+    if (dontUsePlatformProxy && dontUseCustomProxies) {
         return undefined;
     }
     if (!hasCustomProxyOptions(proxyOptions)) {

package/dist/scrapely-proxy-configuration.d.ts

@@ -89,7 +89,10 @@ export declare class ProxyConfiguration extends CrawleeProxyConfiguration {
     /**
      * Verifies the proxy configuration. For platform proxy mode with an
      * injected status URL, performs the Apify-format access check through the
-     * gateway (`GET {statusUrl}/?format=json`, best-effort, never fatal);
+     * gateway (`GET {statusUrl}/?format=json`). When the gateway answers
+     * `connected: false` (e.g. an unknown proxy group), it throws with the
+     * gateway's `connectionError` on the platform and warns locally, like
+     * Apify's `_checkAccess`; timeouts and HTTP errors only warn.
      * BYOP mode only verifies that a test URL can be generated.
      */
     initialize(options?: {

package/dist/scrapely-proxy-configuration.js

@@ -73,6 +73,12 @@ function validateProxySessionId(sessionId) {
     }
     return sessionId;
 }
+/** `groups`, else the input-schema alias `scrapelyProxyGroups`, else the legacy `apifyProxyGroups`. */
+function platformGroups(options) {
+    if (options.groups?.length)
+        return options.groups;
+    return options.scrapelyProxyGroups ?? options.apifyProxyGroups;
+}
 /**
  * Resolves and validates the country/subdivision options (Apify parity).
  * Country must be a two-letter ISO code; subdivision an ISO 3166-2 code
@@ -80,8 +86,8 @@ function validateProxySessionId(sessionId) {
  * the normalized values used by the username grammar.
  */
 function resolveCountryOptions(options) {
-    const countryCode = options.countryCode ?? options.apifyProxyCountry;
-    const subdivisionCode = options.subdivisionCode ?? options.apifyProxySubdivision;
+    const countryCode = options.countryCode ?? options.scrapelyProxyCountry ?? options.apifyProxyCountry;
+    const subdivisionCode = options.subdivisionCode ?? options.scrapelyProxySubdivision ?? options.apifyProxySubdivision;
     if (countryCode != null && !COUNTRY_CODE_REGEX.test(countryCode)) {
         throw new Error(`ProxyConfiguration: "countryCode" must be a two-letter ISO country code (got "${countryCode}").`);
     }
@@ -225,7 +231,7 @@ async function requestThroughProxy(targetUrl, proxyUrl, timeoutMs) {
  * server.
  */
 function buildPlatformProxyUrl(options, env, sessionId) {
-    const groups = options.groups?.length ? options.groups : options.apifyProxyGroups;
+    const groups = platformGroups(options);
     const { countryCode, subdivisionCode } = resolveCountryOptions(options);
     const password = options.password ?? env.password;
     // Apify username grammar: groups, then session, then country — with the
@@ -334,7 +340,7 @@ function toCrawleeOptions(options, env, placeholderOptions) {
     return crawleeOptions;
 }
 function hasApifyOnlyOptions(options) {
-    const groups = options.groups?.length ? options.groups : options.apifyProxyGroups;
+    const groups = platformGroups(options);
     const { countryCode, subdivisionCode } = resolveCountryOptions(options);
     return !!(groups?.length || countryCode || subdivisionCode || options.password);
 }
@@ -382,10 +388,10 @@ class ProxyConfiguration extends core_1.ProxyConfiguration {
         // values throw at construction, not on the first newUrl() call).
         resolveCountryOptions(options);
         if (hasRealCustomProxyOptions(options) && hasApifyOnlyOptions(options)) {
-            throw new Error('Cannot combine custom proxies with Apify Proxy! ' +
+            throw new Error('Cannot combine custom proxies with Scrapely Proxy! ' +
                 'It is not allowed to set "options.proxyUrls" or "options.newUrlFunction" combined with ' +
-                '"options.groups" or "options.apifyProxyGroups" and "options.countryCode" or ' +
-                '"options.apifyProxyCountry", "options.subdivisionCode" or "options.apifyProxySubdivision".');
+                '"options.groups" or "options.scrapelyProxyGroups" and "options.countryCode" or ' +
+                '"options.scrapelyProxyCountry", "options.subdivisionCode" or "options.scrapelyProxySubdivision".');
         }
         if (needsPlatformProxyUrls(options) && !platformEnv) {
             throw new Error('[Scrapely] The platform proxy was requested (via the "auto" sentinel, tieredProxyConfig tiers without ' +
@@ -445,9 +451,7 @@ class ProxyConfiguration extends core_1.ProxyConfiguration {
         const sid = sessionId != null ? validateProxySessionId(String(sessionId)) : undefined;
         const info = await super.newProxyInfo(sid, options);
         if (info && this.platformEnv && info.hostname === this.platformEnv.hostname) {
-            const groups = this.platformOptions.groups?.length
-                ? this.platformOptions.groups
-                : this.platformOptions.apifyProxyGroups;
+            const groups = platformGroups(this.platformOptions);
             const { countryCode, subdivisionCode } = resolveCountryOptions(this.platformOptions);
             return {
                 ...info,
@@ -483,6 +487,7 @@ class ProxyConfiguration extends core_1.ProxyConfiguration {
                         connectionError: 'Platform proxy authentication failed (HTTP 407). Check the injected ' +
                             'SCRAPELY_PROXY_*/APIFY_PROXY_* credentials.',
                         isManInTheMiddle: false,
+                        fromPayload: false,
                     };
                 }
                 if (statusCode >= 590 && statusCode <= 599) {
@@ -491,6 +496,7 @@ class ProxyConfiguration extends core_1.ProxyConfiguration {
                         connected: false,
                         connectionError: `Platform proxy upstream error (HTTP ${statusCode}: ${description}).`,
                         isManInTheMiddle: false,
+                        fromPayload: false,
                     };
                 }
                 if (statusCode >= 200 && statusCode < 300) {
@@ -500,6 +506,7 @@ class ProxyConfiguration extends core_1.ProxyConfiguration {
                             connected: parsed.connected === true,
                             connectionError: parsed.connectionError ?? '',
                             isManInTheMiddle: parsed.isManInTheMiddle === true,
+                            fromPayload: true,
                         };
                     }
                     catch {
@@ -516,7 +523,10 @@ class ProxyConfiguration extends core_1.ProxyConfiguration {
     /**
      * Verifies the proxy configuration. For platform proxy mode with an
      * injected status URL, performs the Apify-format access check through the
-     * gateway (`GET {statusUrl}/?format=json`, best-effort, never fatal);
+     * gateway (`GET {statusUrl}/?format=json`). When the gateway answers
+     * `connected: false` (e.g. an unknown proxy group), it throws with the
+     * gateway's `connectionError` on the platform and warns locally, like
+     * Apify's `_checkAccess`; timeouts and HTTP errors only warn.
      * BYOP mode only verifies that a test URL can be generated.
      */
     async initialize(options = {}) {
@@ -526,7 +536,7 @@ class ProxyConfiguration extends core_1.ProxyConfiguration {
         }
         // Apify-parity access check: GET {statusUrl}/?format=json *through the
         // platform gateway* so the proxy handshake itself authenticates the
-        // run's credentials. Best-effort — never hard fails the run.
+        // run's credentials.
         if (checkAccess && this.platformEnv?.statusUrl) {
             const status = await this.fetchProxyStatus();
             if (!status) {
@@ -535,6 +545,9 @@ class ProxyConfiguration extends core_1.ProxyConfiguration {
             }
             else {
                 this.isManInTheMiddle = status.isManInTheMiddle;
+                if (!status.connected && status.fromPayload && this.config.get('isAtHome')) {
+                    throw new Error(status.connectionError);
+                }
                 if (!status.connected) {
                     console.warn(`[Scrapely] Platform proxy access check failed: ${status.connectionError}`);
                 }

package/package.json

@@ -1,7 +1,7 @@
 {
     "name": "scrapely",
-    "version": "1.2.6",
-    "description": "Apify-like scraping platform using Evomi browser infrastructure",
+    "version": "1.2.8",
+    "description": "Scrapely SDK for building JavaScript and TypeScript Actors",
     "main": "dist/index.js",
     "types": "dist/index.d.ts",
     "files": [