scrapely 1.2.6 → 1.2.7

package/README.md

@@ -1,118 +1,65 @@
-# Scrapely
-
-Scrapely is an Actor-based web scraping platform. Developers push an Actor (a scraping program), the platform builds it into a Docker image, runs it with structured input, and keeps its output in datasets, key-value stores and request queues. The API and SDKs follow Apify's shape, so most Apify Actors run with few or no changes.
-
-This repo holds the platform server, the Actor SDKs, the API clients, the CLI and the MCP servers.
-
-## Repo layout
-
-| Path | What it is |
-|------|------------|
-| `src/` → `dist/` | Node.js Actor SDK, published as `scrapely`. `dist/` is committed because it is the package output. |
-| `scrapely-python/` | Python Actor SDK (`scrapely` on PyPI) |
-| `scrapely-server/` | API server, build/run workers, autoscaler, egress proxy |
-| `scrapely-cli/` | `scrapely` CLI: login, create/init, push, pull, run, builds, runs, storages, secrets |
-| `scrapely-client/` | JavaScript API client (`scrapely-client`, drop-in for `apify-client`) |
-| `scrapely-client-python/` | Python API client (`scrapely-client-python`) |
-| `scrapely-mcp/` | MCP server (TypeScript) |
-| `scrapely-mcp-python/` | MCP server (Python) |
-| `docker/` | Runner and Actor base images, Dragonfly compose file, Kaniko patch |
-| `docs/` | Platform documentation. Start at [docs/README.md](docs/README.md). |
-| `scripts/` | Server setup, deploy, migrations, benchmarks |
-| `tests/` | SDK/runtime tests |
-| `examples/basic-actor/` | Smallest working Actor |
-| `npm-claim-scrapely/` | Old placeholder for the `scrapely` npm name, now superseded by the SDK itself (1.2.4+). |
-
-Each package has its own README with install and usage details.
-
-## How it fits together
-
-- The **API server** (`scrapely-server/`) serves the Apify-style `/v2` API: auth, Actors and versions, builds, runs, tasks, schedules, webhooks, storages, billing and admin routes.
-- **Builds and runs** are queued with BullMQ and picked up by workers. Builds produce Docker images that go to a private registry. Runs execute in Docker on worker servers.
-- **The autoscaler** adds and removes worker servers based on load. See [docs/autoscaling.md](docs/autoscaling.md).
-- **Storage**: run state and storages live in DragonflyDB (Redis-compatible). Accounts, Actors, runs and billing live in SingleStore. Both are backed up to S3.
-- **Browsers**: remote browsers come from Evomi over CDP. See [docs/evomi-integration.md](docs/evomi-integration.md).
-- **Actor code** imports the SDK, which patches Crawlee storage to go through the platform:
+# Scrapely Node.js SDK
 
-```ts
-import { Actor, PlaywrightCrawler, Dataset } from 'scrapely';
-```
-
-```python
-from scrapely import Actor
-```
-
-## Local development
+Build JavaScript and TypeScript Actors for the [Scrapely](https://scrape.ly) platform. The SDK gives you the Actor lifecycle, input and output, datasets, key-value stores and request queues, Crawlee crawlers, remote browsers over CDP, platform events, run control, webhooks and pay-per-event charging.
 
-### Prerequisites
-
-- Node.js 22
-- Docker
-- Doppler CLI with access to the `scrapely` project (all secrets come from Doppler; there are no `.env.example` files)
-- Reachable DragonflyDB and SingleStore instances (Doppler `stg` points at staging)
-
-### Install
+## Install
 
 ```bash
-npm install
-cd scrapely-server && npm install
-cd ../scrapely-cli && npm install
+npm install scrapely
 ```
 
-### Run the API server
-
-```bash
-cd scrapely-server
-doppler run -p scrapely -c stg -- npm run dev
-curl http://localhost:3000/health
-```
+## Quick start
 
-To run a local Dragonfly instead of staging: `docker compose -f docker/docker-compose.dragonfly.yml up -d`.
+```ts
+import { Actor } from 'scrapely';
 
-### Build the runner image
+await Actor.main(async () => {
+    const input = await Actor.getInputOrThrow<{ url: string }>();
+    await Actor.pushData({ url: input.url, status: 'processed' });
+});
+```
 
-Needed before the server can run Actors locally:
+`Actor.main()` initializes the Actor, awaits your function, closes events and storage, and exits the process. Always `await` writes so they reach the platform before the process ends.
 
-```bash
-make build-runner
-```
+## Crawling
 
-### Push and run an Actor
+```ts
+import { Actor, PlaywrightCrawler } from 'scrapely';
 
-```bash
-scrapely login
-cd examples/basic-actor
-scrapely push
-scrapely run --wait
-```
+await Actor.main(async () => {
+    const { startUrls } = await Actor.getInputOrThrow<{ startUrls: string[] }>();
 
-## Tests
+    const crawler = new PlaywrightCrawler({
+        maxConcurrency: 5,
+        async requestHandler({ request, page }) {
+            await Actor.pushData({ url: request.url, title: await page.title() });
+        },
+    });
 
-```bash
-npm test                    # root SDK
-cd scrapely-server && npm test
+    await crawler.run(startUrls);
+});
 ```
 
-Both packages have many more targeted `test:*` scripts in their `package.json`. Most tests are integration-style, and some need a valid `CDP_URL` or live services.
+On the platform, browser crawlers connect to a remote Chromium over CDP instead of launching a local browser. Locally, they launch a normal browser unless you set `CDP_URL` or call `setCdpUrl()`.
 
-## Deploy
+## What's included
 
-| Command | What it does |
-|---------|--------------|
-| `npm run prod` | Full production deploy (`scripts/deploy-prod.sh`) |
-| `npm run prod:update` | Update-only deploy |
-| `npm run setup:server` | Bootstrap a fresh Linux server |
-| `npm run setup:gvisor` | Install gVisor on a Linux host |
-| `npm run migrate` | Run a SingleStore migration |
+- **Actor lifecycle:** `Actor.main()`, `init()`, `exit()`, `fail()`, status messages, reboot, metamorph and abort
+- **Input and output:** `getInput()`, `getInputOrThrow()`, `pushData()`, `getValue()`, `setValue()`
+- **Storage:** `Dataset`, `KeyValueStore` and `RequestQueue`, backed by the Scrapely API on the platform and by local files during development
+- **Crawlers:** `PlaywrightCrawler`, `PuppeteerCrawler`, `CheerioCrawler`, `HttpCrawler`, `BasicCrawler`, `BrowserCrawler`, `JSDOMCrawler` and `LinkeDOMCrawler`
+- **Remote browsers:** automatic CDP wiring through `CDP_URL`, or `setCdpUrl()`
+- **Events:** state persistence, migration and graceful shutdown
+- **Run control:** `Actor.call()`, `Actor.callTask()`, `Actor.start()` and ad-hoc webhooks
+- **Charging:** `Actor.charge()`, `ChargingManager` and `pushDataAndCharge()` for pay-per-event Actors
+- **Logging and utilities:** `log`, `utils` and `enqueueLinks`
 
-The main server uses `scrapely-server/docker-compose.yml` and workers use `docker-compose.worker.yml`. Secrets are injected at container start with `doppler run`.
+## Related packages
 
-`make start` / `stop` / `restart` / `logs` wrap the server compose stack.
+- [`scrapely-client`](https://www.npmjs.com/package/scrapely-client): JavaScript API client
+- [`scrapely-cli`](https://www.npmjs.com/package/scrapely-cli): create, run and push Actors from your terminal
+- [`scrapely-python`](https://pypi.org/project/scrapely-python/): the Python SDK
 
-## Where to look next
+## License
 
-- [docs/README.md](docs/README.md): documentation index (getting started, SDKs, CLI, webhooks, troubleshooting)
-- [scrapely-server/docs/API_ENDPOINTS.md](scrapely-server/docs/API_ENDPOINTS.md): API reference
-- [docs/network-architecture.md](docs/network-architecture.md): fleet topology and data path
-- `scrapely-server/src/index.ts`: server startup and route wiring
-- `src/actor.ts`: Node SDK Actor runtime
+ISC

package/package.json

@@ -1,7 +1,7 @@
 {
     "name": "scrapely",
-    "version": "1.2.6",
-    "description": "Apify-like scraping platform using Evomi browser infrastructure",
+    "version": "1.2.7",
+    "description": "Scrapely SDK for building JavaScript and TypeScript Actors",
     "main": "dist/index.js",
     "types": "dist/index.d.ts",
     "files": [