n8n-nodes-scraperapi 0.2.0 → 0.2.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/LICENSE CHANGED
@@ -1,21 +1,21 @@
1
- MIT License
2
-
3
- Copyright (c) 2026 Geonode Team
4
-
5
- Permission is hereby granted, free of charge, to any person obtaining a copy
6
- of this software and associated documentation files (the "Software"), to deal
7
- in the Software without restriction, including without limitation the rights
8
- to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
- copies of the Software, and to permit persons to whom the Software is
10
- furnished to do so, subject to the following conditions:
11
-
12
- The above copyright notice and this permission notice shall be included in all
13
- copies or substantial portions of the Software.
14
-
15
- THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
- IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
- FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
- AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
- LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
- OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
- SOFTWARE.
1
+ MIT License
2
+
3
+ Copyright (c) 2026 Geonode Team
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in all
13
+ copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
21
+ SOFTWARE.
package/README.md CHANGED
@@ -1,185 +1,187 @@
1
- # n8n-nodes-scraperapi
2
-
3
- Community n8n node package for the **Geonode Scraper API**. Wraps the public REST endpoints (`/v1/extract`, `/v1/map`, `/v1/crawl`, `/v1/webhooks`, `/v1/statistics`) and ships a webhook trigger that registers itself against `/v1/webhooks` automatically.
4
-
5
- The default deployment target is the preprod environment:
6
-
7
- ```
8
- https://scraper-preprod-34do9l3uke5.geonode.com
9
- ```
10
-
11
- You can point it at any other deployment by editing the **Base URL** field on the credential.
12
-
13
- ---
14
-
15
- ## Package structure
16
-
17
- ```
18
- n8n-nodes-scraperapi/
19
- ├─ credentials/
20
- │ └─ ScraperApiApi.credentials.ts # X-Api-Key credential, tested against /v1/statistics
21
- ├─ nodes/
22
- │ ├─ ScraperApi/
23
- │ │ ├─ ScraperApi.node.ts # Action node (7 resources, ~20 operations)
24
- │ │ ├─ GenericFunctions.ts # Auth + retry/backoff helpers + polling helper
25
- │ │ ├─ scraperapi.svg # Node icon
26
- │ │ └─ descriptions/
27
- │ │ ├─ ExtractDescription.ts # Form schema for extract: run / getJob / listJobs
28
- │ │ ├─ SearchDescription.ts # Form schema for search: run / getJob / listJobs
29
- │ │ ├─ BatchDescription.ts # Form schema for batch: start / getStatus / listJobs / cancel
30
- │ │ ├─ MapDescription.ts # Form schema for map: discover URLs
31
- │ │ ├─ CrawlDescription.ts # Form schema for crawl: start / getStatus
32
- │ │ ├─ WebhookDescription.ts # Form schema for webhook CRUD + deliveries
33
- │ │ └─ StatisticsDescription.ts # Form schema for statistics: get
34
- │ └─ ScraperApiTrigger/
35
- │ ├─ ScraperApiTrigger.node.ts # Trigger node — auto-registers on activate
36
- │ └─ scraperapi.svg
37
- ├─ docker-compose.n8n.yml # Local n8n that mounts this package as a custom extension
38
- ├─ package.json # n8n entry points -> dist/**
39
- ├─ tsconfig.json # Compiles credentials/** + nodes/** -> dist/
40
- ├─ gulpfile.js # Copies SVG icons into dist/
41
- ├─ .eslintrc.js # eslint-plugin-n8n-nodes-base rules
42
- └─ index.js # Empty — n8n loads via the "n8n" key in package.json
43
- ```
44
-
45
- ### Component descriptions
46
-
47
- #### Credential — `ScraperApiApi`
48
-
49
- **`X-Api-Key: <key>`** — your personal API key, sent on every request.
50
-
51
- Credential fields:
52
-
53
- | Field | Default | When to fill |
54
- |----------|----------------------------------------------------|-------------------------------------------------------|
55
- | Base URL | `https://scraper-preprod-34do9l3uke5.geonode.com` | Root URL, no trailing slash. |
56
- | API Key | (empty) | Required. Sent as `X-Api-Key`. |
57
-
58
- Clicking **Test** issues `GET {baseUrl}/v1/statistics` with the credentials applied. This endpoint is auth-gated.
59
-
60
- #### Action node — `ScraperApi`
61
-
62
- One node with seven resources, each exposing the relevant operations:
63
-
64
- | Resource | Operation | What it does |
65
- |--------------|-------------------|----------------------------------------------------------------------------------------------------|
66
- | `extract` | Run Extraction | `POST /v1/extract`. Pick formats (html / markdown), JS rendering, sync vs async, proxy, headers. |
67
- | `extract` | Get Job | `GET /v1/extract/{job_id}` — fetch a single async job result. |
68
- | `extract` | List Jobs | `GET /v1/extract/jobs` with filters (status, URL, date range) and pagination. |
69
- | `search` | Run Search | `POST /v1/search` — query plus optional engine, locale, page, safe search and time range. |
70
- | `search` | Get Job | `GET /v1/search/{job_id}` — fetch a single search job. |
71
- | `search` | List Jobs | `GET /v1/search/jobs` with filters (query, status, date range) and pagination. |
72
- | `batch` | Start Batch | `POST /v1/batch` — many URLs in one job; formats, JS rendering, proxy, headers, invalid-URL policy.|
73
- | `batch` | Get Status | `GET /v1/batch/{job_id}` — paginated results; optional client-side polling until terminal status. |
74
- | `batch` | List Jobs | `GET /v1/batch/jobs` with filters (status, date range) and pagination. |
75
- | `batch` | Cancel | `DELETE /v1/batch/{job_id}` — stop a batch that has not finished. |
76
- | `map` | Discover URLs | `POST /v1/map` — sitemap + optional search keyword, returns `links[]`. |
77
- | `crawl` | Start Crawl | `POST /v1/crawl` — seed URL, depth, limit, formats, same-domain / subdomain rules, proxy. |
78
- | `crawl` | Get Status | `GET /v1/crawl/{job_id}` — paginated results; optional client-side polling until terminal status. |
79
- | `webhook` | Create | `POST /v1/webhooks` — register a target URL for an event. |
80
- | `webhook` | Get / List | `GET /v1/webhooks/{id}` and `GET /v1/webhooks` (paginated). |
81
- | `webhook` | Update | `PATCH /v1/webhooks/{id}` — change URL, description, event type, active flag. |
82
- | `webhook` | Delete | `DELETE /v1/webhooks/{id}`. |
83
- | `webhook` | Rotate Secret | `POST /v1/webhooks/{id}/rotate-secret` — issues a new HMAC signing secret. |
84
- | `webhook` | List Deliveries | `GET /v1/webhooks/{id}/deliveries` with status filter and pagination. |
85
- | `statistics` | Get | `GET /v1/statistics` for an optional date range. |
86
-
87
- When a sync extract returns inline, the response is the raw `{data, ...}` body. For async (`processing_mode=async`) the API returns `{job_id, status, status_url, estimated_tokens}`; if **Wait for Completion** is on, the node polls `status_url` until the job hits `completed | failed | cancelled` and returns that final body.
88
-
89
- The shared HTTP helper (`GenericFunctions.scraperApiRequest`) handles:
90
- - `X-Api-Key` header injection via the credential
91
- - Up to 5 retries on `429` and `5xx` with exponential backoff
92
- - `Retry-After` header support (seconds or HTTP-date)
93
- - Conversion of non-retryable errors into `NodeApiError` so n8n surfaces them in the UI
94
-
95
- #### Trigger node — `ScraperApiTrigger`
96
-
97
- Use this when you want a workflow to run **whenever the Scraper API emits an event** (`extract_completed`, `crawl_completed`).
98
-
99
- On workflow activation it:
100
- 1. Calls `POST /v1/webhooks` with n8n's public webhook URL and the chosen event type.
101
- 2. Stashes the returned `id` and `secret` in workflow static data.
102
-
103
- On webhook delivery it (optionally) verifies `X-Webhook-Signature: sha256=<hex>` using HMAC-SHA256 + the stored secret, then emits `{headers, body}` as workflow input.
104
-
105
- On deactivation it calls `DELETE /v1/webhooks/{id}` to clean up.
106
-
107
- ---
108
-
109
- ## Running the node
110
-
111
- A ready-made compose file lives at `docker-compose.n8n.yml` in this directory. It mounts this package into n8n's custom-extensions dir and exposes the editor on `http://localhost:5678`.
112
-
113
- From this directory:
114
-
115
- ```bash
116
- # 1. Install deps and build the package
117
- npm install
118
- npm run build # produces ./dist/ which docker-compose mounts into n8n
119
-
120
- # 2. Start n8n
121
- docker compose -f docker-compose.n8n.yml up -d
122
- ```
123
-
124
- > The compose file mounts the package directory read-only into `/home/node/.n8n/custom/n8n-nodes-scraperapi`. n8n reads the `n8n` key from `package.json` and loads the compiled JS from `dist/**`, so **`npm run build` must succeed before starting the container** (or before restarting after code changes).
125
-
126
- After the container is up:
127
-
128
- 1. Open `http://localhost:5678` and finish the n8n owner setup.
129
- 2. **Credentials → New → Scraper API**.
130
- - Base URL: `https://scraper-preprod-34do9l3uke5.geonode.com` (the default).
131
- - API Key: your preprod key.
132
- - Click **Test** — expects 200 from `/v1/health`.
133
- 3. New workflow → search **Scraper API** → drop it in.
134
-
135
- To pick up code changes:
136
-
137
- ```bash
138
- npm run build
139
- docker compose -f docker-compose.n8n.yml restart n8n
140
- ```
141
-
142
- To stop / clean up:
143
-
144
- ```bash
145
- docker compose -f docker-compose.n8n.yml down # stop, keep data
146
- docker compose -f docker-compose.n8n.yml down -v # also drop the n8n_data volume
147
- ```
148
-
149
- ### Smoke test
150
-
151
- In a workflow, add **Scraper API** → resource `extract` → operation `Run Extraction`:
152
-
153
- ![Creation](./docs/screenshots/node-creating.png)
154
- ![List](./docs/screenshots/nodes-list.png)
155
-
156
- - URL: `https://example.com`
157
- - Output Formats: `html`
158
- - Render JavaScript: off
159
- - Processing Mode: `Sync (Block Until Done)`
160
-
161
- ![Execution](./docs/screenshots/node-execution.png)
162
-
163
- Click **Execute Node** — you should see a JSON output with `data.html` populated.
164
-
165
- Then try the trigger: in a separate workflow add **Scraper API Trigger**, pick `extract_completed`, **save** and **activate**. The node will register a webhook against `/v1/webhooks`. Run an `extract` against the same deployment with `processing_mode=async`; when it finishes, the trigger workflow fires.
166
-
167
- ---
168
-
169
- ## Scripts reference
170
-
171
- ```bash
172
- npm run build # tsc + copy SVG icons into dist/
173
- npm run dev # tsc --watch
174
- npm run lint # eslint with eslint-plugin-n8n-nodes-base
175
- npm run lintfix # eslint --fix
176
- npm run format # prettier on nodes/ and credentials/
177
- ```
178
-
179
- ## Notes specific to this API
180
-
181
- - **Auth** — `X-Api-Key` header (per-user). See `app/api/security/auth_token_handler.py`. Billing user resolution happens in `app/api/security/user_resolver.py`.
182
- - **Async extract** — `processing_mode=async` returns 202 with `{job_id, status, status_url, estimated_tokens}` (see `app/api/v1/extract/router.py`). The node polls `status_url` when **Wait for Completion** is on.
183
- - **Throttling (HTTP 429)** — `scraperApiRequest` honors `Retry-After` and retries up to 5 times with exponential backoff (also covers transient 5xx).
184
- - **Errors** — surfaced as `NodeApiError`. The API returns `{code, message, correlation_id, retryable}` (see `app/api/v1/extract/schemas.py`).
185
- - **Webhook events** — `extract_completed`, `crawl_completed` (see `app/core/webhooks/event_types.py`).
1
+ # n8n-nodes-scraperapi
2
+
3
+ Community n8n node package for the **Geonode Scraper API**. Wraps the public REST endpoints (`/v1/extract`, `/v1/map`, `/v1/crawl`, `/v1/webhooks`, `/v1/statistics`) and ships a webhook trigger that registers itself against `/v1/webhooks` automatically.
4
+
5
+ New credentials default to the production API:
6
+
7
+ ```
8
+ https://scraper.geonode.io
9
+ ```
10
+
11
+ You can point it at another deployment by editing the **Base URL** field on the credential. Use a staging or preprod URL only for intentional testing, with an API key for that environment. Existing saved credentials keep their configured Base URL; update it manually if you want to switch environments.
12
+
13
+ See the [official API reference](https://docs.geonode.com/docs/scraper-api/v1/reference) for the production base URL and authentication details.
14
+
15
+ ---
16
+
17
+ ## Package structure
18
+
19
+ ```
20
+ n8n-nodes-scraperapi/
21
+ ├─ credentials/
22
+ │ └─ ScraperApiApi.credentials.ts # X-Api-Key credential, tested against /v1/statistics
23
+ ├─ nodes/
24
+ │ ├─ ScraperApi/
25
+ │ │ ├─ ScraperApi.node.ts # Action node (7 resources, ~20 operations)
26
+ │ │ ├─ GenericFunctions.ts # Auth + retry/backoff helpers + polling helper
27
+ │ │ ├─ scraperapi.svg # Node icon
28
+ │ │ └─ descriptions/
29
+ │ │ ├─ ExtractDescription.ts # Form schema for extract: run / getJob / listJobs
30
+ │ │ ├─ SearchDescription.ts # Form schema for search: run / getJob / listJobs
31
+ │ │ ├─ BatchDescription.ts # Form schema for batch: start / getStatus / listJobs / cancel
32
+ │ │ ├─ MapDescription.ts # Form schema for map: discover URLs
33
+ │ │ ├─ CrawlDescription.ts # Form schema for crawl: start / getStatus
34
+ │ │ ├─ WebhookDescription.ts # Form schema for webhook CRUD + deliveries
35
+ │ │ └─ StatisticsDescription.ts # Form schema for statistics: get
36
+ │ └─ ScraperApiTrigger/
37
+ │ ├─ ScraperApiTrigger.node.ts # Trigger node — auto-registers on activate
38
+ │ └─ scraperapi.svg
39
+ ├─ docker-compose.n8n.yml # Local n8n that mounts this package as a custom extension
40
+ ├─ package.json # n8n entry points -> dist/**
41
+ ├─ tsconfig.json # Compiles credentials/** + nodes/** -> dist/
42
+ ├─ gulpfile.js # Copies SVG icons into dist/
43
+ ├─ .eslintrc.js # eslint-plugin-n8n-nodes-base rules
44
+ └─ index.js # Empty — n8n loads via the "n8n" key in package.json
45
+ ```
46
+
47
+ ### Component descriptions
48
+
49
+ #### Credential — `ScraperApiApi`
50
+
51
+ **`X-Api-Key: <key>`** — your personal API key, sent on every request.
52
+
53
+ Credential fields:
54
+
55
+ | Field | Default | When to fill |
56
+ |----------|----------------------------------------------------|-------------------------------------------------------|
57
+ | Base URL | `https://scraper.geonode.io` | Root URL, no trailing slash. |
58
+ | API Key | (empty) | Required. Sent as `X-Api-Key`. |
59
+
60
+ Clicking **Test** issues `GET {baseUrl}/v1/statistics` with the credentials applied. This endpoint is auth-gated.
61
+
62
+ #### Action node — `ScraperApi`
63
+
64
+ One node with seven resources, each exposing the relevant operations:
65
+
66
+ | Resource | Operation | What it does |
67
+ |--------------|-------------------|----------------------------------------------------------------------------------------------------|
68
+ | `extract` | Run Extraction | `POST /v1/extract`. Pick formats (html / markdown), JS rendering, sync vs async, proxy, headers. |
69
+ | `extract` | Get Job | `GET /v1/extract/{job_id}` — fetch a single async job result. |
70
+ | `extract` | List Jobs | `GET /v1/extract/jobs` with filters (status, URL, date range) and pagination. |
71
+ | `search` | Run Search | `POST /v1/search` — query plus optional engine, locale, page, safe search and time range. |
72
+ | `search` | Get Job | `GET /v1/search/{job_id}` — fetch a single search job. |
73
+ | `search` | List Jobs | `GET /v1/search/jobs` with filters (query, status, date range) and pagination. |
74
+ | `batch` | Start Batch | `POST /v1/batch` — many URLs in one job; formats, JS rendering, proxy, headers, invalid-URL policy.|
75
+ | `batch` | Get Status | `GET /v1/batch/{job_id}` — paginated results; optional client-side polling until terminal status. |
76
+ | `batch` | List Jobs | `GET /v1/batch/jobs` with filters (status, date range) and pagination. |
77
+ | `batch` | Cancel | `DELETE /v1/batch/{job_id}` — stop a batch that has not finished. |
78
+ | `map` | Discover URLs | `POST /v1/map` — sitemap + optional search keyword, returns `links[]`. |
79
+ | `crawl` | Start Crawl | `POST /v1/crawl` — seed URL, depth, limit, formats, same-domain / subdomain rules, proxy. |
80
+ | `crawl` | Get Status | `GET /v1/crawl/{job_id}` — paginated results; optional client-side polling until terminal status. |
81
+ | `webhook` | Create | `POST /v1/webhooks` — register a target URL for an event. |
82
+ | `webhook` | Get / List | `GET /v1/webhooks/{id}` and `GET /v1/webhooks` (paginated). |
83
+ | `webhook` | Update | `PATCH /v1/webhooks/{id}` — change URL, description, event type, active flag. |
84
+ | `webhook` | Delete | `DELETE /v1/webhooks/{id}`. |
85
+ | `webhook` | Rotate Secret | `POST /v1/webhooks/{id}/rotate-secret` — issues a new HMAC signing secret. |
86
+ | `webhook` | List Deliveries | `GET /v1/webhooks/{id}/deliveries` with status filter and pagination. |
87
+ | `statistics` | Get | `GET /v1/statistics` for an optional date range. |
88
+
89
+ When a sync extract returns inline, the response is the raw `{data, ...}` body. For async (`processing_mode=async`) the API returns `{job_id, status, status_url, estimated_tokens}`; if **Wait for Completion** is on, the node polls `status_url` until the job hits `completed | failed | cancelled` and returns that final body.
90
+
91
+ The shared HTTP helper (`GenericFunctions.scraperApiRequest`) handles:
92
+ - `X-Api-Key` header injection via the credential
93
+ - Up to 5 retries on `429` and `5xx` with exponential backoff
94
+ - `Retry-After` header support (seconds or HTTP-date)
95
+ - Conversion of non-retryable errors into `NodeApiError` so n8n surfaces them in the UI
96
+
97
+ #### Trigger node — `ScraperApiTrigger`
98
+
99
+ Use this when you want a workflow to run **whenever the Scraper API emits an event** (`extract_completed`, `crawl_completed`).
100
+
101
+ On workflow activation it:
102
+ 1. Calls `POST /v1/webhooks` with n8n's public webhook URL and the chosen event type.
103
+ 2. Stashes the returned `id` and `secret` in workflow static data.
104
+
105
+ On webhook delivery it (optionally) verifies `X-Webhook-Signature: sha256=<hex>` using HMAC-SHA256 + the stored secret, then emits `{headers, body}` as workflow input.
106
+
107
+ On deactivation it calls `DELETE /v1/webhooks/{id}` to clean up.
108
+
109
+ ---
110
+
111
+ ## Running the node
112
+
113
+ A ready-made compose file lives at `docker-compose.n8n.yml` in this directory. It mounts this package into n8n's custom-extensions dir and exposes the editor on `http://localhost:5678`.
114
+
115
+ From this directory:
116
+
117
+ ```bash
118
+ # 1. Install deps and build the package
119
+ npm install
120
+ npm run build # produces ./dist/ which docker-compose mounts into n8n
121
+
122
+ # 2. Start n8n
123
+ docker compose -f docker-compose.n8n.yml up -d
124
+ ```
125
+
126
+ > The compose file mounts the package directory read-only into `/home/node/.n8n/custom/n8n-nodes-scraperapi`. n8n reads the `n8n` key from `package.json` and loads the compiled JS from `dist/**`, so **`npm run build` must succeed before starting the container** (or before restarting after code changes).
127
+
128
+ After the container is up:
129
+
130
+ 1. Open `http://localhost:5678` and finish the n8n owner setup.
131
+ 2. **Credentials → New → Scraper API**.
132
+ - Base URL: `https://scraper.geonode.io` (the default).
133
+ - API Key: your Geonode Scraper API key for production.
134
+ - Click **Test** — sends an authenticated `GET /v1/statistics` request.
135
+ 3. New workflow → search **Scraper API** → drop it in.
136
+
137
+ To pick up code changes:
138
+
139
+ ```bash
140
+ npm run build
141
+ docker compose -f docker-compose.n8n.yml restart n8n
142
+ ```
143
+
144
+ To stop / clean up:
145
+
146
+ ```bash
147
+ docker compose -f docker-compose.n8n.yml down # stop, keep data
148
+ docker compose -f docker-compose.n8n.yml down -v # also drop the n8n_data volume
149
+ ```
150
+
151
+ ### Smoke test
152
+
153
+ In a workflow, add **Scraper API** → resource `extract` → operation `Run Extraction`:
154
+
155
+ ![Creation](./docs/screenshots/node-creating.png)
156
+ ![List](./docs/screenshots/nodes-list.png)
157
+
158
+ - URL: `https://example.com`
159
+ - Output Formats: `html`
160
+ - Render JavaScript: off
161
+ - Processing Mode: `Sync (Block Until Done)`
162
+
163
+ ![Execution](./docs/screenshots/node-execution.png)
164
+
165
+ Click **Execute Node** — you should see a JSON output with `data.html` populated.
166
+
167
+ Then try the trigger: in a separate workflow add **Scraper API Trigger**, pick `extract_completed`, **save** and **activate**. The node will register a webhook against `/v1/webhooks`. Run an `extract` against the same deployment with `processing_mode=async`; when it finishes, the trigger workflow fires.
168
+
169
+ ---
170
+
171
+ ## Scripts reference
172
+
173
+ ```bash
174
+ npm run build # tsc + copy SVG icons into dist/
175
+ npm run dev # tsc --watch
176
+ npm run lint # eslint with eslint-plugin-n8n-nodes-base
177
+ npm run lintfix # eslint --fix
178
+ npm run format # prettier on nodes/ and credentials/
179
+ ```
180
+
181
+ ## Notes specific to this API
182
+
183
+ - **Auth** — `X-Api-Key` header (per-user). See `app/api/security/auth_token_handler.py`. Billing user resolution happens in `app/api/security/user_resolver.py`.
184
+ - **Async extract** — `processing_mode=async` returns 202 with `{job_id, status, status_url, estimated_tokens}` (see `app/api/v1/extract/router.py`). The node polls `status_url` when **Wait for Completion** is on.
185
+ - **Throttling (HTTP 429)** — `scraperApiRequest` honors `Retry-After` and retries up to 5 times with exponential backoff (also covers transient 5xx).
186
+ - **Errors** — surfaced as `NodeApiError`. The API returns `{code, message, correlation_id, retryable}` (see `app/api/v1/extract/schemas.py`).
187
+ - **Webhook events** — `extract_completed`, `crawl_completed` (see `app/core/webhooks/event_types.py`).
@@ -5,14 +5,14 @@ class ScraperApiApi {
5
5
  constructor() {
6
6
  this.name = 'scraperApiApi';
7
7
  this.displayName = 'Scraper API';
8
- this.documentationUrl = 'https://github.com/geonodecom/scraper-api';
8
+ this.documentationUrl = 'https://docs.geonode.com/docs/scraper-api/v1/reference';
9
9
  this.properties = [
10
10
  {
11
11
  displayName: 'Base URL',
12
12
  name: 'baseUrl',
13
13
  type: 'string',
14
- default: 'https://scraper-preprod-34do9l3uke5.geonode.com',
15
- placeholder: 'https://scraper-preprod-34do9l3uke5.geonode.com',
14
+ default: 'https://scraper.geonode.io',
15
+ placeholder: 'https://scraper.geonode.io',
16
16
  description: 'Root URL of the Scraper API deployment (no trailing slash)',
17
17
  required: true,
18
18
  },
@@ -1 +1 @@
1
- {"version":3,"file":"ScraperApiApi.credentials.js","sourceRoot":"","sources":["../../credentials/ScraperApiApi.credentials.ts"],"names":[],"mappings":";;;AAOA,MAAa,aAAa;IAA1B;QACE,SAAI,GAAG,eAAe,CAAC;QACvB,gBAAW,GAAG,aAAa,CAAC;QAC5B,qBAAgB,GAAG,2CAA2C,CAAC;QAE/D,eAAU,GAAsB;YAC9B;gBACE,WAAW,EAAE,UAAU;gBACvB,IAAI,EAAE,SAAS;gBACf,IAAI,EAAE,QAAQ;gBACd,OAAO,EAAE,iDAAiD;gBAC1D,WAAW,EAAE,iDAAiD;gBAC9D,WAAW,EAAE,4DAA4D;gBACzE,QAAQ,EAAE,IAAI;aACf;YACD;gBACE,WAAW,EAAE,SAAS;gBACtB,IAAI,EAAE,QAAQ;gBACd,IAAI,EAAE,QAAQ;gBACd,WAAW,EAAE,EAAE,QAAQ,EAAE,IAAI,EAAE;gBAC/B,OAAO,EAAE,EAAE;gBACX,QAAQ,EAAE,IAAI;gBACd,WAAW,EACT,+FAA+F;aAClG;SACF,CAAC;QAEF,iBAAY,GAAyB;YACnC,IAAI,EAAE,SAAS;YACf,UAAU,EAAE;gBACV,OAAO,EAAE;oBACP,WAAW,EAAE,0BAA0B;iBACxC;aACF;SACF,CAAC;QAIF,SAAI,GAA2B;YAC7B,OAAO,EAAE;gBACP,OAAO,EAAE,2BAA2B;gBACpC,GAAG,EAAE,gBAAgB;gBACrB,MAAM,EAAE,KAAK;aACd;SACF,CAAC;IACJ,CAAC;CAAA;AA7CD,sCA6CC"}
1
+ {"version":3,"file":"ScraperApiApi.credentials.js","sourceRoot":"","sources":["../../credentials/ScraperApiApi.credentials.ts"],"names":[],"mappings":";;;AAOA,MAAa,aAAa;IAA1B;QACE,SAAI,GAAG,eAAe,CAAC;QACvB,gBAAW,GAAG,aAAa,CAAC;QAC5B,qBAAgB,GAAG,wDAAwD,CAAC;QAE5E,eAAU,GAAsB;YAC9B;gBACE,WAAW,EAAE,UAAU;gBACvB,IAAI,EAAE,SAAS;gBACf,IAAI,EAAE,QAAQ;gBACd,OAAO,EAAE,4BAA4B;gBACrC,WAAW,EAAE,4BAA4B;gBACzC,WAAW,EAAE,4DAA4D;gBACzE,QAAQ,EAAE,IAAI;aACf;YACD;gBACE,WAAW,EAAE,SAAS;gBACtB,IAAI,EAAE,QAAQ;gBACd,IAAI,EAAE,QAAQ;gBACd,WAAW,EAAE,EAAE,QAAQ,EAAE,IAAI,EAAE;gBAC/B,OAAO,EAAE,EAAE;gBACX,QAAQ,EAAE,IAAI;gBACd,WAAW,EACT,+FAA+F;aAClG;SACF,CAAC;QAEF,iBAAY,GAAyB;YACnC,IAAI,EAAE,SAAS;YACf,UAAU,EAAE;gBACV,OAAO,EAAE;oBACP,WAAW,EAAE,0BAA0B;iBACxC;aACF;SACF,CAAC;QAIF,SAAI,GAA2B;YAC7B,OAAO,EAAE;gBACP,OAAO,EAAE,2BAA2B;gBACpC,GAAG,EAAE,gBAAgB;gBACrB,MAAM,EAAE,KAAK;aACd;SACF,CAAC;IACJ,CAAC;CAAA;AA7CD,sCA6CC"}