n8n-nodes-scraperapi 0.2.3 → 0.3.0

package/README.md

@@ -1,189 +1,103 @@
-# n8n-nodes-scraperapi
+![Geonode for n8n: Web Scraping & Workflow Automation](assets/banner.svg)
 
-**Get your free API key:** [sign up at geonode.com](https://app.geonode.com/sign-up?utm_source=npm&utm_medium=package&utm_campaign=n8n).
+# Geonode for n8n: Web Scraping & Workflow Automation
 
-Community n8n node package for the **Geonode Scraper API**. Wraps the public REST endpoints (`/v1/extract`, `/v1/map`, `/v1/crawl`, `/v1/webhooks`, `/v1/statistics`) and ships a webhook trigger that registers itself against `/v1/webhooks` automatically.
+Bring web data into self-hosted n8n agents and workflows with Geonode’s community node. Extract pages, crawl sites, discover URLs, search the web and run batches. Use extraction and crawl completion triggers to continue your own AI processing, research or record-update steps.
 
-New credentials default to the production API:
+**[Try Geonode for free](https://app.geonode.com/sign-up)**
 
-```
-https://scraper.geonode.io
-```
-
-You can point it at another deployment by editing the **Base URL** field on the credential. Use a staging or preprod URL only for intentional testing, with an API key for that environment. Existing saved credentials keep their configured Base URL; update it manually if you want to switch environments.
+[Integrations page](https://geonode.com/integrations/integration-n8n) · [Geonode API docs](https://docs.geonode.com/docs/scraper-api) · [n8n setup guide](https://docs.geonode.com/docs/scraper-api/guides/sdks/n8n) · [View package on npm](https://www.npmjs.com/package/n8n-nodes-scraperapi) · [View source on GitHub](https://github.com/geonodecom/integration-n8n) · [Complete guide](docs/guide.md)
 
-See the [official API reference](https://docs.geonode.com/docs/scraper-api/v1/reference) for the production base URL and authentication details.
+**For self-hosted n8n.** This is a community node, not a verified n8n Cloud node. The npm package name remains `n8n-nodes-scraperapi`.
 
----
+<!-- BEGIN README NAV -->
+[Get started](#get-started) · [Build these workflows](#build-these-workflows) · [Example: a URL to a spreadsheet row](#example-a-url-to-a-spreadsheet-row) · [Setup questions](#setup-questions)
+<!-- END README NAV -->
 
-## Package structure
-
-```
-n8n-nodes-scraperapi/
-├─ credentials/
-│  └─ ScraperApiApi.credentials.ts   # X-Api-Key credential, tested against /v1/statistics
-├─ nodes/
-│  ├─ ScraperApi/
-│  │  ├─ ScraperApi.node.ts          # Action node (7 resources, ~20 operations)
-│  │  ├─ GenericFunctions.ts         # Auth + retry/backoff helpers + polling helper
-│  │  ├─ scraperapi.svg              # Node icon
-│  │  └─ descriptions/
-│  │     ├─ ExtractDescription.ts    # Form schema for extract: run / getJob / listJobs
-│  │     ├─ SearchDescription.ts     # Form schema for search: run / getJob / listJobs
-│  │     ├─ BatchDescription.ts      # Form schema for batch: start / getStatus / listJobs / cancel
-│  │     ├─ MapDescription.ts        # Form schema for map: discover URLs
-│  │     ├─ CrawlDescription.ts      # Form schema for crawl: start / getStatus
-│  │     ├─ WebhookDescription.ts    # Form schema for webhook CRUD + deliveries
-│  │     └─ StatisticsDescription.ts # Form schema for statistics: get
-│  └─ ScraperApiTrigger/
-│     ├─ ScraperApiTrigger.node.ts   # Trigger node — auto-registers on activate
-│     └─ scraperapi.svg
-├─ docker-compose.n8n.yml            # Local n8n that mounts this package as a custom extension
-├─ package.json                      # n8n entry points -> dist/**
-├─ tsconfig.json                     # Compiles credentials/** + nodes/** -> dist/
-├─ gulpfile.js                       # Copies SVG icons into dist/
-├─ .eslintrc.js                      # eslint-plugin-n8n-nodes-base rules
-└─ index.js                          # Empty — n8n loads via the "n8n" key in package.json
-```
+## Workflow ideas
 
-### Component descriptions
+- **Feed a knowledge base.** Extract source pages as Markdown, then connect your own chunking, embedding and storage nodes.
+- **Refresh a website dataset.** Start a bounded crawl and process the returned content after the job completes.
+- **React to completed extraction.** Use Geonode Scraper API Trigger to start follow-up processing for extraction or crawl completion.
 
-#### Credential — `ScraperApiApi`
+## Get started
 
-**`X-Api-Key: <key>`** — your personal API key, sent on every request.
+1. In n8n, open **Settings → Community Nodes → Install** and enter `n8n-nodes-scraperapi`.
+2. Create a **Geonode Scraper API** credential. Paste your Geonode API key and keep the base URL `https://scraper.geonode.io`.
+3. Add a **Geonode Scraper API** node. Choose **Extract → Run Extraction**, enter a URL, and choose **Markdown** as the output format.
+4. Execute the step. The result contains the page content in `data.markdown` and its cost in `tokens_charged`.
 
-Credential fields:
+You can obtain a key from **API Keys** in the [Geonode dashboard](https://app.geonode.com/). The credential test checks authenticated usage statistics.
 
-| Field    | Default                                            | When to fill                                          |
-|----------|----------------------------------------------------|-------------------------------------------------------|
-| Base URL | `https://scraper.geonode.io`                         | Root URL, no trailing slash.                          |
-| API Key  | (empty)                                            | Required. Sent as `X-Api-Key`.                        |
+## Build these workflows
 
-Clicking **Test** issues `GET {baseUrl}/v1/statistics` with the credentials applied. This endpoint is auth-gated.
+| Workflow | Node capability |
+| --- | --- |
+| Feed a knowledge base | Extract pages as Markdown with optional JavaScript rendering. |
+| Refresh many sources | Start a batch, then wait for or retrieve its results. |
+| Collect a site | Crawl from a seed URL with depth and page limits. |
+| Discover URLs | Map a site's links before choosing pages to extract. |
+| Search the web | Run search jobs with engine and locale controls. |
+| React to completed jobs | Use **Geonode Scraper API Trigger** for extraction or crawl completion. |
+| Monitor usage | Read statistics and manage webhook subscriptions. |
 
-#### Action node — `ScraperApi`
+For long jobs, use asynchronous extraction or a start/status sequence. The [full guide](docs/guide.md) covers polling, webhook signature checks, errors and retries.
 
-One node with seven resources, each exposing the relevant operations:
+<!-- BEGIN SHARED WORKFLOW GUIDE -->
+## Example: a URL to a spreadsheet row
 
-| Resource     | Operation         | What it does                                                                                       |
-|--------------|-------------------|----------------------------------------------------------------------------------------------------|
-| `extract`    | Run Extraction    | `POST /v1/extract`. Pick formats (html / markdown), JS rendering, sync vs async, proxy, headers.   |
-| `extract`    | Get Job           | `GET /v1/extract/{job_id}` — fetch a single async job result.                                      |
-| `extract`    | List Jobs         | `GET /v1/extract/jobs` with filters (status, URL, date range) and pagination.                      |
-| `search`     | Run Search        | `POST /v1/search` — query plus optional engine, locale, page, safe search and time range.          |
-| `search`     | Get Job           | `GET /v1/search/{job_id}` — fetch a single search job.                                             |
-| `search`     | List Jobs         | `GET /v1/search/jobs` with filters (query, status, date range) and pagination.                     |
-| `batch`      | Start Batch       | `POST /v1/batch` — many URLs in one job; formats, JS rendering, proxy, headers, invalid-URL policy.|
-| `batch`      | Get Status        | `GET /v1/batch/{job_id}` — paginated results; optional client-side polling until terminal status.  |
-| `batch`      | List Jobs         | `GET /v1/batch/jobs` with filters (status, date range) and pagination.                             |
-| `batch`      | Cancel            | `DELETE /v1/batch/{job_id}` — stop a batch that has not finished.                                  |
-| `map`        | Discover URLs     | `POST /v1/map` — sitemap + optional search keyword, returns `links[]`.                             |
-| `crawl`      | Start Crawl       | `POST /v1/crawl` — seed URL, depth, limit, formats, same-domain / subdomain rules, proxy.          |
-| `crawl`      | Get Status        | `GET /v1/crawl/{job_id}` — paginated results; optional client-side polling until terminal status.  |
-| `webhook`    | Create            | `POST /v1/webhooks` — register a target URL for an event.                                          |
-| `webhook`    | Get / List        | `GET /v1/webhooks/{id}` and `GET /v1/webhooks` (paginated).                                        |
-| `webhook`    | Update            | `PATCH /v1/webhooks/{id}` — change URL, description, event type, active flag.                      |
-| `webhook`    | Delete            | `DELETE /v1/webhooks/{id}`.                                                                        |
-| `webhook`    | Rotate Secret     | `POST /v1/webhooks/{id}/rotate-secret` — issues a new HMAC signing secret.                         |
-| `webhook`    | List Deliveries   | `GET /v1/webhooks/{id}/deliveries` with status filter and pagination.                              |
-| `statistics` | Get               | `GET /v1/statistics` for an optional date range.                                                   |
+1. Supply a URL from a trigger or an Edit Fields node, then add Geonode Scraper API → Extract → Run Extraction.
+2. Select your Geonode Scraper API credential, map the incoming URL, choose Markdown and keep synchronous processing for the first test.
+3. Execute the step and inspect data.markdown, metadata.url and tokens_charged in the returned item's JSON.
+4. Connect a separate Google Sheets node and map those fields into Content, Source URL and Usage columns using n8n's expression editor.
+5. Test the destination write before activating the workflow. Add a separate AI node if you want a summary rather than the raw Markdown.
 
-When a sync extract returns inline, the response is the raw `{data, ...}` body. For async (`processing_mode=async`) the API returns `{job_id, status, status_url, estimated_tokens}`; if **Wait for Completion** is on, the node polls `status_url` until the job hits `completed | failed | cancelled` and returns that final body.
+**Expected result:** Geonode retrieves the page content. The Sheets or AI node needs its own connection; this recipe does not install or configure those services.
 
-The shared HTTP helper (`GenericFunctions.scraperApiRequest`) handles:
-- `X-Api-Key` header injection via the credential
-- Up to 5 retries on `429` and `5xx` with exponential backoff
-- `Retry-After` header support (seconds or HTTP-date)
-- Conversion of non-retryable errors into `NodeApiError` so n8n surfaces them in the UI
+## Understand output and completion triggers
 
-#### Trigger node — `ScraperApiTrigger`
+Each successful input item produces an output item whose JSON contains the API response. A queued extraction returns job information, while completed extraction returns content. Do not map a job ID as if it were Markdown.
 
-Use this when you want a workflow to run **whenever the Scraper API emits an event** (`extract_completed`, `crawl_completed`).
+Geonode Scraper API Trigger supports extraction and crawl completion. Activation registers a webhook and deactivation removes it. Keep signature verification enabled and use a callback URL reachable by the API.
 
-On workflow activation it:
-1. Calls `POST /v1/webhooks` with n8n's public webhook URL and the chosen event type.
-2. Stashes the returned `id` and `secret` in workflow static data.
+If Continue On Fail is enabled, a failed item can contain an error string. Route those items separately instead of sending them to your normal content destination.
 
-On webhook delivery it (optionally) verifies `X-Webhook-Signature: sha256=<hex>` using HMAC-SHA256 + the stored secret, then emits `{headers, body}` as workflow input.
+## Setup questions
 
-On deactivation it calls `DELETE /v1/webhooks/{id}` to clean up.
+Can't install on n8n Cloud? This is a self-hosted community node and has not been verified for n8n Cloud. The package name is n8n-nodes-scraperapi.
 
----
+No trigger events? Check workflow activation, the reachable webhook URL and signature verification. A successful credential test alone does not prove that callbacks arrive.
 
-## Running the node
+Need parsed product fields or change alerts? Add your own parsing, storage and comparison nodes after extraction. The content response does not promise those outputs automatically.
 
-A ready-made compose file lives at `docker-compose.n8n.yml` in this directory. It mounts this package into n8n's custom-extensions dir and exposes the editor on `http://localhost:5678`.
-
-From this directory:
-
-```bash
-# 1. Install deps and build the package
-npm install
-npm run build        # produces ./dist/ which docker-compose mounts into n8n
-
-# 2. Start n8n
-docker compose -f docker-compose.n8n.yml up -d
-```
+These recipes describe source-supported workflows. They have not been run as authenticated end-to-end examples in this review.
+<!-- END SHARED WORKFLOW GUIDE -->
 
-> The compose file mounts the package directory read-only into `/home/node/.n8n/custom/n8n-nodes-scraperapi`. n8n reads the `n8n` key from `package.json` and loads the compiled JS from `dist/**`, so **`npm run build` must succeed before starting the container** (or before restarting after code changes).
+## Develop locally
 
-After the container is up:
+Use Node.js 20.15 or newer and npm. Run from this repository root:
 
-1. Open `http://localhost:5678` and finish the n8n owner setup.
-2. **Credentials → New → Scraper API**.
-   - Base URL: `https://scraper.geonode.io` (the default).
-   - API Key: your Geonode Scraper API key for production.
-   - Click **Test** — sends an authenticated `GET /v1/statistics` request.
-3. New workflow → search **Scraper API** → drop it in.
-
-To pick up code changes:
-
-```bash
+```sh
+npm ci
 npm run build
-docker compose -f docker-compose.n8n.yml restart n8n
+npm test
+npm run lint
 ```
 
-To stop / clean up:
+To open a local n8n editor with the built nodes mounted:
 
-```bash
-docker compose -f docker-compose.n8n.yml down            # stop, keep data
-docker compose -f docker-compose.n8n.yml down -v         # also drop the n8n_data volume
+```sh
+docker compose -f docker-compose.n8n.yml up -d
 ```
 
-### Smoke test
-
-In a workflow, add **Scraper API** → resource `extract` → operation `Run Extraction`:
-
-![Creation](./docs/screenshots/node-creating.png)
-![List](./docs/screenshots/nodes-list.png)
+Open `http://localhost:5678`. Rebuild after editing TypeScript, then restart the n8n container. This Compose setup is for local development; webhook triggers require a URL the API can reach.
 
-- URL: `https://example.com`
-- Output Formats: `html`
-- Render JavaScript: off
-- Processing Mode: `Sync (Block Until Done)`
+## Source and releases
 
-![Execution](./docs/screenshots/node-execution.png)
+Package names, node identifiers and credential types are kept stable. Maintainers publish releases to npm with the `Publish` workflow in GitHub Actions.
 
-Click **Execute Node** — you should see a JSON output with `data.html` populated.
-
-Then try the trigger: in a separate workflow add **Scraper API Trigger**, pick `extract_completed`, **save** and **activate**. The node will register a webhook against `/v1/webhooks`. Run an `extract` against the same deployment with `processing_mode=async`; when it finishes, the trigger workflow fires.
-
----
-
-## Scripts reference
-
-```bash
-npm run build      # tsc + copy SVG icons into dist/
-npm run dev        # tsc --watch
-npm run lint       # eslint with eslint-plugin-n8n-nodes-base
-npm run lintfix    # eslint --fix
-npm run format     # prettier on nodes/ and credentials/
-```
+[Contributing](CONTRIBUTING.md) · [Changelog](CHANGELOG.md) · [Contact Geonode](https://geonode.com/contact)
 
-## Notes specific to this API
+MIT licensed. See [LICENSE](LICENSE).
 
-- **Auth** — `X-Api-Key` header (per-user). See `app/api/security/auth_token_handler.py`. Billing user resolution happens in `app/api/security/user_resolver.py`.
-- **Async extract** — `processing_mode=async` returns 202 with `{job_id, status, status_url, estimated_tokens}` (see `app/api/v1/extract/router.py`). The node polls `status_url` when **Wait for Completion** is on.
-- **Throttling (HTTP 429)** — `scraperApiRequest` honors `Retry-After` and retries up to 5 times with exponential backoff (also covers transient 5xx).
-- **Errors** — surfaced as `NodeApiError`. The API returns `{code, message, correlation_id, retryable}` (see `app/api/v1/extract/schemas.py`).
-- **Webhook events** — `extract_completed`, `crawl_completed` (see `app/core/webhooks/event_types.py`).
+**[Try Geonode for free](https://app.geonode.com/sign-up)**

package/dist/credentials/ScraperApiApi.credentials.d.ts

@@ -2,6 +2,7 @@ import { IAuthenticateGeneric, ICredentialTestRequest, ICredentialType, INodePro
 export declare class ScraperApiApi implements ICredentialType {
     name: string;
     displayName: string;
+    icon: "file:scraper-geonode.svg";
     documentationUrl: string;
     properties: INodeProperties[];
     authenticate: IAuthenticateGeneric;

package/dist/credentials/ScraperApiApi.credentials.js

@@ -4,7 +4,8 @@ exports.ScraperApiApi = void 0;
 class ScraperApiApi {
     constructor() {
         this.name = 'scraperApiApi';
-        this.displayName = 'Scraper API';
+        this.displayName = 'Geonode Scraper API';
+        this.icon = 'file:scraper-geonode.svg';
         this.documentationUrl = 'https://docs.geonode.com/docs/scraper-api/v1/reference';
         this.properties = [
             {

package/dist/credentials/ScraperApiApi.credentials.js.map

@@ -1 +1 @@
-{"version":3,"file":"ScraperApiApi.credentials.js","sourceRoot":"","sources":["../../credentials/ScraperApiApi.credentials.ts"],"names":[],"mappings":";;;AAOA,MAAa,aAAa;IAA1B;QACE,SAAI,GAAG,eAAe,CAAC;QACvB,gBAAW,GAAG,aAAa,CAAC;QAC5B,qBAAgB,GAAG,wDAAwD,CAAC;QAE5E,eAAU,GAAsB;YAC9B;gBACE,WAAW,EAAE,UAAU;gBACvB,IAAI,EAAE,SAAS;gBACf,IAAI,EAAE,QAAQ;gBACd,OAAO,EAAE,4BAA4B;gBACrC,WAAW,EAAE,4BAA4B;gBACzC,WAAW,EAAE,4DAA4D;gBACzE,QAAQ,EAAE,IAAI;aACf;YACD;gBACE,WAAW,EAAE,SAAS;gBACtB,IAAI,EAAE,QAAQ;gBACd,IAAI,EAAE,QAAQ;gBACd,WAAW,EAAE,EAAE,QAAQ,EAAE,IAAI,EAAE;gBAC/B,OAAO,EAAE,EAAE;gBACX,QAAQ,EAAE,IAAI;gBACd,WAAW,EACT,+FAA+F;aAClG;SACF,CAAC;QAEF,iBAAY,GAAyB;YACnC,IAAI,EAAE,SAAS;YACf,UAAU,EAAE;gBACV,OAAO,EAAE;oBACP,WAAW,EAAE,0BAA0B;iBACxC;aACF;SACF,CAAC;QAIF,SAAI,GAA2B;YAC7B,OAAO,EAAE;gBACP,OAAO,EAAE,2BAA2B;gBACpC,GAAG,EAAE,gBAAgB;gBACrB,MAAM,EAAE,KAAK;aACd;SACF,CAAC;IACJ,CAAC;CAAA;AA7CD,sCA6CC"}
+{"version":3,"file":"ScraperApiApi.credentials.js","sourceRoot":"","sources":["../../credentials/ScraperApiApi.credentials.ts"],"names":[],"mappings":";;;AAOA,MAAa,aAAa;IAA1B;QACE,SAAI,GAAG,eAAe,CAAC;QACvB,gBAAW,GAAG,qBAAqB,CAAC;QACpC,SAAI,GAAG,0BAAmC,CAAC;QAC3C,qBAAgB,GAAG,wDAAwD,CAAC;QAE5E,eAAU,GAAsB;YAC9B;gBACE,WAAW,EAAE,UAAU;gBACvB,IAAI,EAAE,SAAS;gBACf,IAAI,EAAE,QAAQ;gBACd,OAAO,EAAE,4BAA4B;gBACrC,WAAW,EAAE,4BAA4B;gBACzC,WAAW,EAAE,4DAA4D;gBACzE,QAAQ,EAAE,IAAI;aACf;YACD;gBACE,WAAW,EAAE,SAAS;gBACtB,IAAI,EAAE,QAAQ;gBACd,IAAI,EAAE,QAAQ;gBACd,WAAW,EAAE,EAAE,QAAQ,EAAE,IAAI,EAAE;gBAC/B,OAAO,EAAE,EAAE;gBACX,QAAQ,EAAE,IAAI;gBACd,WAAW,EACT,+FAA+F;aAClG;SACF,CAAC;QAEF,iBAAY,GAAyB;YACnC,IAAI,EAAE,SAAS;YACf,UAAU,EAAE;gBACV,OAAO,EAAE;oBACP,WAAW,EAAE,0BAA0B;iBACxC;aACF;SACF,CAAC;QAIF,SAAI,GAA2B;YAC7B,OAAO,EAAE;gBACP,OAAO,EAAE,2BAA2B;gBACpC,GAAG,EAAE,gBAAgB;gBACrB,MAAM,EAAE,KAAK;aACd;SACF,CAAC;IACJ,CAAC;CAAA;AA9CD,sCA8CC"}