scrapely 1.2.3 → 1.2.4

package/README.md

@@ -1 +1,119 @@
-Reserved package name.
+# Scrapely
+
+Scrapely is an Actor-based web scraping platform. Developers push an Actor (a scraping program), the platform builds it into a Docker image, runs it with structured input, and keeps its output in datasets, key-value stores and request queues. The API and SDKs follow Apify's shape, so most Apify Actors run with few or no changes.
+
+This repo holds the platform server, the Actor SDKs, the API clients, the CLI and the MCP servers.
+
+## Repo layout
+
+| Path | What it is |
+|------|------------|
+| `src/` → `dist/` | Node.js Actor SDK, published as `@evomi/scrapely`. `dist/` is committed because it is the package output. |
+| `scrapely-python/` | Python Actor SDK (`scrapely` on PyPI) |
+| `scrapely-server/` | API server, build/run workers, autoscaler, egress proxy |
+| `scrapely-cli/` | `scrapely` CLI: login, create/init, push, pull, run, builds, runs, storages, secrets |
+| `scrapely-client/` | JavaScript API client (`scrapely-client`, drop-in for `apify-client`) |
+| `scrapely-client-python/` | Python API client (`scrapely-client-python`) |
+| `scrapely-mcp/` | MCP server (TypeScript) |
+| `scrapely-mcp-python/` | MCP server (Python) |
+| `docker/` | Runner and Actor base images, Dragonfly compose file, Kaniko patch |
+| `docs/` | Platform documentation. Start at [docs/README.md](docs/README.md). |
+| `scripts/` | Server setup, deploy, migrations, benchmarks |
+| `tests/` | SDK/runtime tests |
+| `examples/basic-actor/` | Smallest working Actor |
+| `Apify_Docs/` | Apify API reference copies, fetched by `getdocs.py` |
+| `npm-claim-scrapely/` | Placeholder that reserves the unscoped `scrapely` npm name. Not the runtime package. |
+
+Each package has its own README with install and usage details.
+
+## How it fits together
+
+- The **API server** (`scrapely-server/`) serves the Apify-style `/v2` API: auth, Actors and versions, builds, runs, tasks, schedules, webhooks, storages, billing and admin routes.
+- **Builds and runs** are queued with BullMQ and picked up by workers. Builds produce Docker images that go to a private registry. Runs execute in Docker on worker servers.
+- **The autoscaler** adds and removes worker servers based on load. See [docs/autoscaling.md](docs/autoscaling.md).
+- **Storage**: run state and storages live in DragonflyDB (Redis-compatible). Accounts, Actors, runs and billing live in SingleStore. Both are backed up to S3.
+- **Browsers**: remote browsers come from Evomi over CDP. See [docs/evomi-integration.md](docs/evomi-integration.md).
+- **Actor code** imports the SDK, which patches Crawlee storage to go through the platform:
+
+```ts
+import { Actor, PlaywrightCrawler, Dataset } from '@evomi/scrapely';
+```
+
+```python
+from scrapely import Actor
+```
+
+## Local development
+
+### Prerequisites
+
+- Node.js 22
+- Docker
+- Doppler CLI with access to the `scrapely` project (all secrets come from Doppler; there are no `.env.example` files)
+- Reachable DragonflyDB and SingleStore instances (Doppler `stg` points at staging)
+
+### Install
+
+```bash
+npm install
+cd scrapely-server && npm install
+cd ../scrapely-cli && npm install
+```
+
+### Run the API server
+
+```bash
+cd scrapely-server
+doppler run -p scrapely -c stg -- npm run dev
+curl http://localhost:3000/health
+```
+
+To run a local Dragonfly instead of staging: `docker compose -f docker/docker-compose.dragonfly.yml up -d`.
+
+### Build the runner image
+
+Needed before the server can run Actors locally:
+
+```bash
+make build-runner
+```
+
+### Push and run an Actor
+
+```bash
+scrapely login
+cd examples/basic-actor
+scrapely push
+scrapely run --wait
+```
+
+## Tests
+
+```bash
+npm test                    # root SDK
+cd scrapely-server && npm test
+```
+
+Both packages have many more targeted `test:*` scripts in their `package.json`. Most tests are integration-style, and some need a valid `CDP_URL` or live services.
+
+## Deploy
+
+| Command | What it does |
+|---------|--------------|
+| `npm run prod` | Full production deploy (`scripts/deploy-prod.sh`) |
+| `npm run prod:update` | Update-only deploy |
+| `npm run setup:server` | Bootstrap a fresh Linux server |
+| `npm run setup:gvisor` | Install gVisor on a Linux host |
+| `npm run migrate` | Run a SingleStore migration |
+
+The main server uses `scrapely-server/docker-compose.yml` and workers use `docker-compose.worker.yml`. Secrets are injected at container start with `doppler run`.
+
+`make start` / `stop` / `restart` / `logs` wrap the server compose stack.
+
+## Where to look next
+
+- [docs/README.md](docs/README.md): documentation index (getting started, SDKs, CLI, webhooks, troubleshooting)
+- [scrapely-server/docs/API_ENDPOINTS.md](scrapely-server/docs/API_ENDPOINTS.md): API reference
+- [docs/network-architecture.md](docs/network-architecture.md): fleet topology and data path
+- `scrapely-server/src/index.ts`: server startup and route wiring
+- `src/actor.ts`: Node SDK Actor runtime