n8n-nodes-scavio 0.21.0 → 0.22.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
package/README.md CHANGED
@@ -2,7 +2,7 @@
2
2
 
3
3
  This is an n8n community node. It lets you use [Scavio](https://scavio.dev?utm_source=n8n-node&utm_medium=package&utm_campaign=readme) — a real-time structured-data API for search, retail, social, travel, real estate, jobs, app stores, software reviews, ad libraries, public company filings, and any URL — in your n8n workflows.
4
4
 
5
- Scavio returns clean, structured JSON from 48 resources and 370 operations. Use it to power product research, price and review monitoring, AI agent retrieval, lead enrichment, competitive and ad intelligence, and content workflows.
5
+ Scavio returns clean, structured JSON from 48 resources and 377 operations. Use it to power product research, price and review monitoring, AI agent retrieval, lead enrichment, competitive and ad intelligence, and content workflows.
6
6
 
7
7
  [Installation](#installation) - [Operations](#operations) - [What changed in 0.13.0](#what-changed-in-0130) - [Per-resource notes](#per-resource-notes) - [Templates](#templates) - [Credentials](#credentials) - [Compatibility](#compatibility) - [Resources](#resources)
8
8
 
@@ -79,9 +79,22 @@ In n8n: **Settings -> Community Nodes -> Install** -> enter `n8n-nodes-scavio`.
79
79
  | [Meta Ad Library](https://scavio.dev/docs/meta-ads-search?utm_source=n8n-node&utm_medium=package&utm_campaign=readme) | 1 | Search Ad Library, Get Advertiser Ads, Get Ad |
80
80
  | [SEC EDGAR](https://scavio.dev/docs/sec-edgar-lookup?utm_source=n8n-node&utm_medium=package&utm_campaign=readme) | 1 | Look Up Company, Get Company Profile, Get Filings, Get XBRL Concept, Get XBRL Facts Index, Search Filings Full Text |
81
81
  | [Companies House (UK)](https://scavio.dev/docs/companies-house-search?utm_source=n8n-node&utm_medium=package&utm_campaign=readme) | 1 | Search Companies, Get Company, Get Officers, Get Filing History |
82
- | [Extract (Any URL)](https://scavio.dev/docs/extract?utm_source=n8n-node&utm_medium=package&utm_campaign=readme) | 1 normal, 1 advanced, 2 ultra | Extract Page |
82
+ | [Extract (Any URL)](https://scavio.dev/docs/extract?utm_source=n8n-node&utm_medium=package&utm_campaign=readme) | 1 normal, 1 advanced, 2 ultra; crawl and batch bill that per successful page; Map Website 1; status and cancel free | Extract Page, Map Website, Start Crawl, Get Crawl Status, Cancel Crawl, Start Batch, Get Batch Status, Cancel Batch, Screenshot |
83
83
  | Account | Free | Get Usage |
84
84
 
85
+ ## What changed in 0.22.0
86
+
87
+ **Extract gained 7 operations: map a website, crawl it, and extract a list of URLs in one job.** Extract Page behaves as before; only its Mode option wording changed. Nothing else changed.
88
+
89
+ - **Map Website** returns a site's page URLs from its sitemaps and its start page, deduplicated, with optional `Search` text, include / exclude path globs, subdomains and a link cap (`Max Links`, sent as `limit`; API default 1,000; Free 500, PAYG 1,000, paid plans 5,000). Each link says whether it came from the sitemap or the page, with `lastmod` when the sitemap has one; `total_found` and `truncated` say whether there were more. 1 credit.
90
+ - **Start Crawl** extracts same-site pages breadth-first from one URL within `Max Pages` (API default 100), `Max Depth` (0-10, default 3) and path globs, and returns a job ID. URLs under the start path are crawled first. **Start Batch** extracts a list of URLs (one per line or comma-separated) as one job; invalid URLs are listed in the response and never billed.
91
+ - **Billing is per successful page** at the Extract price for the mode: Normal 1 credit, Advanced (renders JavaScript) 1, Ultra (the hardest pages) 2. Failed and skipped pages are free. Credits are taken page by page and a job stops cleanly with `stopped_reason: insufficient_credits` if the balance runs out.
92
+ - **Plan limits** (max pages per crawl / active jobs / URLs per batch): Free 10/1/10, PAYG 100/1/50, Project 500/2/100, Bootstrap 2,000/3/500, Startup 5,000/5/1,000, Growth 10,000/10/1,000. Starting a job past your active-job limit returns 429 with `Retry-After`.
93
+ - **Get Crawl Status / Get Batch Status** return progress (discovered, queued, succeeded, failed, skipped), credits used and the page results paged with `next_cursor` (`Results Per Page` 1-100, API default 25), optionally filtered by status or without content. **Cancel Crawl / Cancel Batch** stop a job; results so far stay available, and a finished job answers 409. Status and cancel are free. Results are kept for 3 days after a job finishes (2 MB per page, 250 MB per job); a job runs for at most 6 hours.
94
+ - **Wait for Completion** (off by default) on Start Crawl and Start Batch polls the job inside the node, every `Poll Interval` seconds (default 5) for up to `Max Wait` seconds (default 300, max 600), then returns the status and first page of results. If the wait runs out, the latest status comes back with `wait.timed_out: true` instead of an error; carry on with Get Status. The start response's fields, including `webhook_secret`, are kept under `created`. Large jobs are better run with the toggle off plus a Wait node, or a webhook.
95
+ - **robots.txt is always obeyed** for crawl and map. Batch reads each URL like Extract Page.
96
+ - **Webhook URL** (optional) receives a signed POST when a job completes, fails or is cancelled, retried after 30 seconds and 5 minutes. Verify the `Scavio-Signature: t=<unix>,v1=<hex>` header: HMAC-SHA256 of `<t>.<raw body>` with the `webhook_secret` returned once by the start operation. `Idempotency Key` makes a repeated start within 24 hours return the original job.
97
+
85
98
  ## What changed in 0.21.0
86
99
 
87
100
  **19 new operations on TikTok, YouTube, Reddit and X, 1 credit each. YouTube Get Transcript now costs 5 credits (was 8).** Nothing else changed.
@@ -1,6 +1,170 @@
1
1
  "use strict";
2
2
  Object.defineProperty(exports, "__esModule", { value: true });
3
3
  exports.extractFields = exports.extractOperations = void 0;
4
+ const ExtractJobWait_1 = require("./ExtractJobWait");
5
+ // Comma-separated text -> string array, blanks dropped.
6
+ const splitList = '={{ String($value).split(",").map(s => s.trim()).filter(Boolean) }}';
7
+ // One URL per line or comma-separated -> string array, blanks dropped.
8
+ const splitUrls = '={{ String($value).split(/[\\n,]/).map(s => s.trim()).filter(Boolean) }}';
9
+ const PRICING = 'Billed per successful page at the Extract price for the mode: Normal 1 credit, Advanced 1, Ultra 2. Failed and skipped pages are free, and the job stops cleanly if your balance runs out.';
10
+ const LIMITS = 'Plan limits (pages per crawl / active jobs / URLs per batch): Free 10/1/10, PAYG 100/1/50, Project 500/2/100, Bootstrap 2,000/3/500, Startup 5,000/5/1,000, Growth 10,000/10/1,000';
11
+ const show = (operation, extra = {}) => ({
12
+ show: { resource: ['extract'], operation, ...extra },
13
+ });
14
+ const JOB_MODES = [
15
+ { name: 'Advanced', value: 'advanced', description: 'Renders JavaScript, 1 credit per page' },
16
+ { name: 'Normal', value: 'normal', description: 'The standard read, 1 credit per page' },
17
+ { name: 'Ultra', value: 'ultra', description: 'For the hardest pages, 2 credits per page' },
18
+ ];
19
+ const JOB_FORMATS = [
20
+ { name: 'HTML', value: 'html', description: 'The raw page exactly as served' },
21
+ { name: 'Markdown', value: 'markdown', description: 'Readability extraction of the main content' },
22
+ { name: 'Plain Text', value: 'text', description: 'The Markdown extraction flattened to plain text' },
23
+ ];
24
+ const SITEMAP_OPTIONS = [
25
+ { name: 'Include', value: 'include', description: 'Use the sitemap and the links on pages' },
26
+ { name: 'Only', value: 'only', description: 'Use the sitemap alone' },
27
+ { name: 'Skip', value: 'skip', description: 'Use the links on pages alone' },
28
+ ];
29
+ const GLOB_HINT = 'Comma-separated path globs, up to 20. * matches within one path segment, ** across segments; a pattern without wildcards matches that path and everything under it. Example: /blog/**';
30
+ function jobFormat() {
31
+ return {
32
+ displayName: 'Format',
33
+ name: 'format',
34
+ type: 'options',
35
+ default: 'markdown',
36
+ options: JOB_FORMATS,
37
+ description: 'Shape each page comes back in, as on Extract Page. Defaults to Markdown.',
38
+ routing: { request: { body: { format: '={{ $value }}' } } },
39
+ };
40
+ }
41
+ function jobMode() {
42
+ return {
43
+ displayName: 'Mode',
44
+ name: 'mode',
45
+ type: 'options',
46
+ default: 'normal',
47
+ options: JOB_MODES,
48
+ description: 'Extraction tier and the per-page price: Normal 1 credit, Advanced 1, Ultra 2. Defaults to Normal.',
49
+ routing: { request: { body: { mode: '={{ $value }}' } } },
50
+ };
51
+ }
52
+ function idempotencyKey() {
53
+ return {
54
+ displayName: 'Idempotency Key',
55
+ name: 'idempotency_key',
56
+ type: 'string',
57
+ default: '',
58
+ description: 'Repeating a start with the same key within 24 hours returns the original job instead of starting a new one. Up to 255 characters.',
59
+ routing: { request: { headers: { 'Idempotency-Key': '={{ $value }}' } } },
60
+ };
61
+ }
62
+ function webhookUrl() {
63
+ return {
64
+ displayName: 'Webhook URL',
65
+ name: 'webhook_url',
66
+ type: 'string',
67
+ default: '',
68
+ placeholder: 'https://example.com/hooks/scavio',
69
+ description: 'HTTPS URL that receives a signed POST when the job completes, fails or is cancelled (retried after 30 seconds and 5 minutes). The signing secret is returned once, as webhook_secret, in this response. Header Scavio-Signature: t=&lt;unix&gt;,v1=&lt;hex HMAC-SHA256 of "&lt;t&gt;.&lt;raw body&gt;"&gt;.',
70
+ routing: { request: { body: { webhook_url: '={{ $value }}' } } },
71
+ };
72
+ }
73
+ function waitFields(operation) {
74
+ return [
75
+ {
76
+ displayName: 'Wait for Completion',
77
+ name: 'waitForCompletion',
78
+ type: 'boolean',
79
+ default: false,
80
+ displayOptions: show([operation]),
81
+ description: 'Whether to poll the job until it finishes and return its status and first page of results. Off returns the job ID at once; use Get Status to collect results. If the wait limit is reached, the latest status is returned with wait.timed_out set to true.',
82
+ },
83
+ {
84
+ displayName: 'Max Wait (Seconds)',
85
+ name: 'maxWaitSeconds',
86
+ type: 'number',
87
+ default: ExtractJobWait_1.DEFAULT_WAIT_SECONDS,
88
+ typeOptions: { minValue: 10, maxValue: ExtractJobWait_1.MAX_WAIT_SECONDS },
89
+ displayOptions: show([operation], { waitForCompletion: [true] }),
90
+ description: `How long to wait before returning the latest status, up to ${ExtractJobWait_1.MAX_WAIT_SECONDS} seconds. Large jobs keep running after the wait ends.`,
91
+ },
92
+ {
93
+ displayName: 'Poll Interval (Seconds)',
94
+ name: 'pollIntervalSeconds',
95
+ type: 'number',
96
+ default: ExtractJobWait_1.DEFAULT_POLL_SECONDS,
97
+ typeOptions: { minValue: 2, maxValue: 60 },
98
+ displayOptions: show([operation], { waitForCompletion: [true] }),
99
+ description: 'Seconds between status checks while waiting. Status checks are free.',
100
+ },
101
+ ];
102
+ }
103
+ function jobIdField(type, operation) {
104
+ return {
105
+ displayName: 'Job ID',
106
+ name: 'jobId',
107
+ type: 'string',
108
+ required: true,
109
+ default: '',
110
+ placeholder: `${type}_01928c1e-...`,
111
+ displayOptions: show([operation]),
112
+ description: `The ID returned by Start ${type === 'crawl' ? 'Crawl' : 'Batch'}`,
113
+ routing: {
114
+ request: { url: `=/api/v1/extract/${type}/{{ String($value).trim() }}` },
115
+ },
116
+ };
117
+ }
118
+ function statusOptions(operation) {
119
+ return {
120
+ displayName: 'Additional Options',
121
+ name: 'additionalOptions',
122
+ type: 'collection',
123
+ placeholder: 'Add Option',
124
+ default: {},
125
+ displayOptions: show([operation]),
126
+ options: [
127
+ {
128
+ displayName: 'Cursor',
129
+ name: 'cursor',
130
+ type: 'string',
131
+ default: '',
132
+ description: 'The next_cursor from the previous response, to fetch the next page of results',
133
+ routing: { request: { qs: { cursor: '={{ $value }}' } } },
134
+ },
135
+ {
136
+ displayName: 'Include Content',
137
+ name: 'include_content',
138
+ type: 'boolean',
139
+ default: true,
140
+ description: 'Whether to return each page\'s content. Turn off to page through results without their content.',
141
+ routing: { request: { qs: { include_content: '={{ $value ? "true" : "false" }}' } } },
142
+ },
143
+ {
144
+ displayName: 'Results Per Page',
145
+ name: 'page_size',
146
+ type: 'number',
147
+ default: 25,
148
+ typeOptions: { minValue: 1, maxValue: 100 },
149
+ description: 'Results per response, 1-100. The API default is 25. Responses are also cut at about 5 MB of content.',
150
+ routing: { request: { qs: { limit: '={{ $value }}' } } },
151
+ },
152
+ {
153
+ displayName: 'Status',
154
+ name: 'status',
155
+ type: 'options',
156
+ default: 'succeeded',
157
+ options: [
158
+ { name: 'Failed', value: 'failed' },
159
+ { name: 'Skipped', value: 'skipped' },
160
+ { name: 'Succeeded', value: 'succeeded' },
161
+ ],
162
+ description: 'Only return results with this status',
163
+ routing: { request: { qs: { status: '={{ $value }}' } } },
164
+ },
165
+ ],
166
+ };
167
+ }
4
168
  exports.extractOperations = [
5
169
  {
6
170
  displayName: 'Operation',
@@ -9,6 +173,20 @@ exports.extractOperations = [
9
173
  noDataExpression: true,
10
174
  displayOptions: { show: { resource: ['extract'] } },
11
175
  options: [
176
+ {
177
+ name: 'Cancel Batch',
178
+ value: 'cancelBatch',
179
+ action: 'Cancel a batch extract job',
180
+ description: 'Stop a queued or running batch. No further pages are fetched or billed; results so far stay available. Returns 409 if the batch already finished. Free.',
181
+ routing: { request: { method: 'DELETE' } },
182
+ },
183
+ {
184
+ name: 'Cancel Crawl',
185
+ value: 'cancelCrawl',
186
+ action: 'Cancel a crawl job',
187
+ description: 'Stop a queued or running crawl. No further pages are fetched or billed; results so far stay available. Returns 409 if the crawl already finished. Free.',
188
+ routing: { request: { method: 'DELETE' } },
189
+ },
12
190
  {
13
191
  name: 'Extract Page',
14
192
  value: 'extract',
@@ -16,6 +194,27 @@ exports.extractOperations = [
16
194
  description: 'Read any URL and get it back as raw HTML, readability Markdown or plain text, alongside the format, mode and content_length it came back with. Mode sets the price: Normal and Advanced cost 1 credit, Ultra costs 2. Nothing is charged unless the extraction succeeds, so a dead link, bot wall or timeout is free.',
17
195
  routing: { request: { method: 'POST', url: '/api/v1/extract' } },
18
196
  },
197
+ {
198
+ name: 'Get Batch Status',
199
+ value: 'batchStatus',
200
+ action: 'Get batch status and results',
201
+ description: 'Progress (page counts, credits used, stopped_reason) and the page results in input order, paged with next_cursor. Results are kept for 3 days after the batch finishes. Free.',
202
+ routing: { request: { method: 'GET' } },
203
+ },
204
+ {
205
+ name: 'Get Crawl Status',
206
+ value: 'crawlStatus',
207
+ action: 'Get crawl status and results',
208
+ description: 'Progress (page counts, credits used, stopped_reason) and the page results, each with URL, depth, status and content, paged with next_cursor. Results are kept for 3 days after the crawl finishes. Free.',
209
+ routing: { request: { method: 'GET' } },
210
+ },
211
+ {
212
+ name: 'Map Website',
213
+ value: 'map',
214
+ action: 'Map a website',
215
+ description: 'Give a site and get back its list of page URLs from its sitemaps and its start page, deduplicated, filtered by path and limited, with total_found and truncated. Pages disallowed by robots.txt are left out. 1 credit.',
216
+ routing: { request: { method: 'POST', url: '/api/v1/extract/map' } },
217
+ },
19
218
  {
20
219
  name: 'Screenshot',
21
220
  value: 'screenshot',
@@ -23,6 +222,26 @@ exports.extractOperations = [
23
222
  description: 'Give any public URL and get back a screenshot of the page as a PNG, returned inline as a base64 data URI',
24
223
  routing: { request: { method: 'POST', url: '/api/v1/extract/screenshot' } },
25
224
  },
225
+ {
226
+ name: 'Start Batch',
227
+ value: 'batch',
228
+ action: 'Start a batch extract job',
229
+ description: `Extract a list of URLs in one job, each returned like Extract Page. Invalid URLs are listed in the response and never billed. Returns a job ID; turn on Wait for Completion or use Get Batch Status for results. ${PRICING}`,
230
+ routing: {
231
+ request: { method: 'POST', url: '/api/v1/extract/batch' },
232
+ output: { postReceive: [(0, ExtractJobWait_1.waitForJob)('batch')] },
233
+ },
234
+ },
235
+ {
236
+ name: 'Start Crawl',
237
+ value: 'crawl',
238
+ action: 'Start a crawl job',
239
+ description: `Crawl a site from one URL: same-site pages are found and extracted breadth-first within your page, depth and path limits, each returned like Extract Page. Pages disallowed by robots.txt are skipped. Returns a job ID; turn on Wait for Completion or use Get Crawl Status for results. ${PRICING}`,
240
+ routing: {
241
+ request: { method: 'POST', url: '/api/v1/extract/crawl' },
242
+ output: { postReceive: [(0, ExtractJobWait_1.waitForJob)('crawl')] },
243
+ },
244
+ },
26
245
  ],
27
246
  default: 'extract',
28
247
  },
@@ -75,21 +294,21 @@ exports.extractFields = [
75
294
  {
76
295
  name: 'Advanced',
77
296
  value: 'advanced',
78
- description: 'Headless browser render for pages built by JavaScript, 1 credit',
297
+ description: 'Renders JavaScript, for pages built in the browser, 1 credit',
79
298
  },
80
299
  {
81
300
  name: 'Normal',
82
301
  value: 'normal',
83
- description: 'Plain datacenter fetch, the cheapest and fastest option, 1 credit',
302
+ description: 'The standard read, the cheapest and fastest option, 1 credit',
84
303
  },
85
304
  {
86
305
  name: 'Ultra',
87
306
  value: 'ultra',
88
- description: 'Premium residential proxy for hard targets, 2 credits',
307
+ description: 'For the hardest pages, 2 credits',
89
308
  },
90
309
  ],
91
310
  routing: { request: { body: { mode: '={{ $value }}' } } },
92
- description: 'Fetch strategy, and the only thing that moves the price. Normal and Advanced cost 1 credit each, Ultra costs 2. Billing happens only on a successful extraction, so a dead link, bot wall or timeout costs nothing.',
311
+ description: 'Extraction tier, and the only thing that moves the price. Normal and Advanced cost 1 credit each, Ultra costs 2. Billing happens only on a successful extraction, so a dead link, bot wall or timeout costs nothing.',
93
312
  },
94
313
  // ── Screenshot fields ──
95
314
  {
@@ -129,5 +348,224 @@ exports.extractFields = [
129
348
  routing: { request: { body: { mode: '={{ $value }}' } } },
130
349
  description: 'Capture tier. Normal works for most sites; advanced waits longer for JS pages; ultra reaches hard anti-bot sites.',
131
350
  },
351
+ // ── Map Website ──
352
+ {
353
+ displayName: 'URL',
354
+ name: 'url',
355
+ type: 'string',
356
+ required: true,
357
+ default: '',
358
+ placeholder: 'https://example.com',
359
+ displayOptions: show(['map']),
360
+ routing: { request: { body: { url: '={{ String($value).trim() }}' } } },
361
+ description: 'The site to map. A bare host is upgraded to HTTPS.',
362
+ },
363
+ {
364
+ displayName: 'Additional Options',
365
+ name: 'additionalOptions',
366
+ type: 'collection',
367
+ placeholder: 'Add Option',
368
+ default: {},
369
+ displayOptions: show(['map']),
370
+ options: [
371
+ {
372
+ displayName: 'Exclude Paths',
373
+ name: 'exclude_paths',
374
+ type: 'string',
375
+ default: '',
376
+ placeholder: '/tag/**, /author/**',
377
+ description: `Paths to drop. ${GLOB_HINT}.`,
378
+ routing: { send: { type: 'body', property: 'exclude_paths', value: splitList } },
379
+ },
380
+ {
381
+ displayName: 'Ignore Query Parameters',
382
+ name: 'ignore_query_parameters',
383
+ type: 'boolean',
384
+ default: true,
385
+ description: 'Whether URLs that differ only by query string count as one URL. The API default is true.',
386
+ routing: { request: { body: { ignore_query_parameters: '={{ $value }}' } } },
387
+ },
388
+ {
389
+ displayName: 'Include Paths',
390
+ name: 'include_paths',
391
+ type: 'string',
392
+ default: '',
393
+ placeholder: '/blog/**',
394
+ description: `Paths to keep. ${GLOB_HINT}.`,
395
+ routing: { send: { type: 'body', property: 'include_paths', value: splitList } },
396
+ },
397
+ {
398
+ displayName: 'Include Subdomains',
399
+ name: 'include_subdomains',
400
+ type: 'boolean',
401
+ default: false,
402
+ description: 'Whether to also return URLs on subdomains of the site. www and the bare domain always count as one site.',
403
+ routing: { request: { body: { include_subdomains: '={{ $value }}' } } },
404
+ },
405
+ {
406
+ displayName: 'Max Links',
407
+ name: 'max_links',
408
+ type: 'number',
409
+ default: 1000,
410
+ typeOptions: { minValue: 1, maxValue: 5000 },
411
+ description: 'Maximum links to return, sent as limit. The API default is 1,000 (or your plan limit if lower). Plan limits: Free 500, PAYG 1,000, paid plans 5,000.',
412
+ routing: { request: { body: { limit: '={{ $value }}' } } },
413
+ },
414
+ {
415
+ displayName: 'Mode',
416
+ name: 'mode',
417
+ type: 'options',
418
+ default: 'normal',
419
+ options: [
420
+ { name: 'Advanced', value: 'advanced', description: 'Renders JavaScript' },
421
+ { name: 'Normal', value: 'normal', description: 'The standard read, 1 credit' },
422
+ { name: 'Ultra', value: 'ultra', description: 'For the hardest pages' },
423
+ ],
424
+ description: 'Extraction tier used to read the start page and sitemaps. Normal costs 1 credit.',
425
+ routing: { request: { body: { mode: '={{ $value }}' } } },
426
+ },
427
+ {
428
+ displayName: 'Search',
429
+ name: 'search',
430
+ type: 'string',
431
+ default: '',
432
+ placeholder: 'pricing',
433
+ description: 'Only return URLs containing every word of this text (case-insensitive), up to 200 characters',
434
+ routing: { request: { body: { search: '={{ $value }}' } } },
435
+ },
436
+ {
437
+ displayName: 'Sitemap',
438
+ name: 'sitemap',
439
+ type: 'options',
440
+ default: 'include',
441
+ options: SITEMAP_OPTIONS,
442
+ description: 'Include (the default) combines the sitemaps with the links on the start page',
443
+ routing: { request: { body: { sitemap: '={{ $value }}' } } },
444
+ },
445
+ ],
446
+ },
447
+ // ── Start Crawl ──
448
+ {
449
+ displayName: 'URL',
450
+ name: 'url',
451
+ type: 'string',
452
+ required: true,
453
+ default: '',
454
+ placeholder: 'https://example.com/docs',
455
+ displayOptions: show(['crawl']),
456
+ routing: { request: { body: { url: '={{ String($value).trim() }}' } } },
457
+ description: 'Where the crawl starts. Only pages on the same site are crawled, and URLs under the start path are crawled first.',
458
+ },
459
+ ...waitFields('crawl'),
460
+ {
461
+ displayName: 'Additional Options',
462
+ name: 'additionalOptions',
463
+ type: 'collection',
464
+ placeholder: 'Add Option',
465
+ default: {},
466
+ displayOptions: show(['crawl']),
467
+ options: [
468
+ {
469
+ displayName: 'Allow Subdomains',
470
+ name: 'allow_subdomains',
471
+ type: 'boolean',
472
+ default: false,
473
+ description: 'Whether to also crawl subdomains of the site',
474
+ routing: { request: { body: { allow_subdomains: '={{ $value }}' } } },
475
+ },
476
+ {
477
+ displayName: 'Exclude Paths',
478
+ name: 'exclude_paths',
479
+ type: 'string',
480
+ default: '',
481
+ placeholder: '/blog/tag/**',
482
+ description: `Paths to skip. Excludes win over includes. ${GLOB_HINT}.`,
483
+ routing: { send: { type: 'body', property: 'exclude_paths', value: splitList } },
484
+ },
485
+ jobFormat(),
486
+ idempotencyKey(),
487
+ {
488
+ displayName: 'Ignore Query Parameters',
489
+ name: 'ignore_query_parameters',
490
+ type: 'boolean',
491
+ default: true,
492
+ description: 'Whether URLs that differ only by query string count as one page. The API default is true.',
493
+ routing: { request: { body: { ignore_query_parameters: '={{ $value }}' } } },
494
+ },
495
+ {
496
+ displayName: 'Include Paths',
497
+ name: 'include_paths',
498
+ type: 'string',
499
+ default: '',
500
+ placeholder: '/docs/**',
501
+ description: `Paths to crawl. ${GLOB_HINT}.`,
502
+ routing: { send: { type: 'body', property: 'include_paths', value: splitList } },
503
+ },
504
+ {
505
+ displayName: 'Max Depth',
506
+ name: 'max_depth',
507
+ type: 'number',
508
+ default: 3,
509
+ typeOptions: { minValue: 0, maxValue: 10 },
510
+ description: 'How many links away from the start page to follow, 0-10. 0 fetches only the start page. The API default is 3.',
511
+ routing: { request: { body: { max_depth: '={{ $value }}' } } },
512
+ },
513
+ {
514
+ displayName: 'Max Pages',
515
+ name: 'max_pages',
516
+ type: 'number',
517
+ default: 100,
518
+ typeOptions: { minValue: 1, maxValue: 10000 },
519
+ description: `Maximum pages to fetch. The API default is 100. ${LIMITS}.`,
520
+ routing: { request: { body: { max_pages: '={{ $value }}' } } },
521
+ },
522
+ jobMode(),
523
+ {
524
+ displayName: 'Sitemap',
525
+ name: 'sitemap',
526
+ type: 'options',
527
+ default: 'include',
528
+ options: [
529
+ { name: 'Include', value: 'include', description: 'Also start from the sitemap' },
530
+ { name: 'Only', value: 'only', description: 'Fetch sitemap pages without following links' },
531
+ { name: 'Skip', value: 'skip', description: 'Follow links only' },
532
+ ],
533
+ description: 'Whether the crawl uses the site\'s sitemap. Defaults to Include.',
534
+ routing: { request: { body: { sitemap: '={{ $value }}' } } },
535
+ },
536
+ webhookUrl(),
537
+ ],
538
+ },
539
+ // ── Get Crawl Status / Cancel Crawl ──
540
+ jobIdField('crawl', 'crawlStatus'),
541
+ statusOptions('crawlStatus'),
542
+ jobIdField('crawl', 'cancelCrawl'),
543
+ // ── Start Batch ──
544
+ {
545
+ displayName: 'URLs',
546
+ name: 'urls',
547
+ type: 'string',
548
+ required: true,
549
+ default: '',
550
+ typeOptions: { rows: 4 },
551
+ placeholder: 'https://example.com/a\nhttps://example.com/b',
552
+ displayOptions: show(['batch']),
553
+ description: `Pages to extract, one per line or comma-separated (duplicates removed). ${LIMITS}.`,
554
+ routing: { send: { type: 'body', property: 'urls', value: splitUrls } },
555
+ },
556
+ ...waitFields('batch'),
557
+ {
558
+ displayName: 'Additional Options',
559
+ name: 'additionalOptions',
560
+ type: 'collection',
561
+ placeholder: 'Add Option',
562
+ default: {},
563
+ displayOptions: show(['batch']),
564
+ options: [jobFormat(), idempotencyKey(), jobMode(), webhookUrl()],
565
+ },
566
+ // ── Get Batch Status / Cancel Batch ──
567
+ jobIdField('batch', 'batchStatus'),
568
+ statusOptions('batchStatus'),
569
+ jobIdField('batch', 'cancelBatch'),
132
570
  ];
133
571
  //# sourceMappingURL=ExtractDescription.js.map
@@ -1 +1 @@
1
- {"version":3,"file":"ExtractDescription.js","sourceRoot":"","sources":["../../../../nodes/Scavio/descriptions/ExtractDescription.ts"],"names":[],"mappings":";;;AAEa,QAAA,iBAAiB,GAAsB;IACnD;QACC,WAAW,EAAE,WAAW;QACxB,IAAI,EAAE,WAAW;QACjB,IAAI,EAAE,SAAS;QACf,gBAAgB,EAAE,IAAI;QACtB,cAAc,EAAE,EAAE,IAAI,EAAE,EAAE,QAAQ,EAAE,CAAC,SAAS,CAAC,EAAE,EAAE;QACnD,OAAO,EAAE;YACR;gBACC,IAAI,EAAE,cAAc;gBACpB,KAAK,EAAE,SAAS;gBAChB,MAAM,EAAE,oBAAoB;gBAC5B,WAAW,EACV,uTAAuT;gBACxT,OAAO,EAAE,EAAE,OAAO,EAAE,EAAE,MAAM,EAAE,MAAM,EAAE,GAAG,EAAE,iBAAiB,EAAE,EAAE;aAChE;YACD;gBACC,IAAI,EAAE,YAAY;gBAClB,KAAK,EAAE,YAAY;gBACnB,MAAM,EAAE,+BAA+B;gBACvC,WAAW,EACV,0GAA0G;gBAC3G,OAAO,EAAE,EAAE,OAAO,EAAE,EAAE,MAAM,EAAE,MAAM,EAAE,GAAG,EAAE,4BAA4B,EAAE,EAAE;aAC3E;SACD;QACD,OAAO,EAAE,SAAS;KAClB;CACD,CAAC;AAEW,QAAA,aAAa,GAAsB;IAC/C;QACC,WAAW,EAAE,KAAK;QAClB,IAAI,EAAE,KAAK;QACX,IAAI,EAAE,QAAQ;QACd,QAAQ,EAAE,IAAI;QACd,OAAO,EAAE,EAAE;QACX,WAAW,EAAE,+BAA+B;QAC5C,cAAc,EAAE,EAAE,IAAI,EAAE,EAAE,QAAQ,EAAE,CAAC,SAAS,CAAC,EAAE,SAAS,EAAE,CAAC,SAAS,CAAC,EAAE,EAAE;QAC3E,OAAO,EAAE,EAAE,OAAO,EAAE,EAAE,IAAI,EAAE,EAAE,GAAG,EAAE,eAAe,EAAE,EAAE,EAAE;QACxD,WAAW,EACV,oJAAoJ;KACrJ;IACD;QACC,WAAW,EAAE,eAAe;QAC5B,IAAI,EAAE,QAAQ;QACd,IAAI,EAAE,SAAS;QACf,OAAO,EAAE,UAAU;QACnB,cAAc,EAAE,EAAE,IAAI,EAAE,EAAE,QAAQ,EAAE,CAAC,SAAS,CAAC,EAAE,SAAS,EAAE,CAAC,SAAS,CAAC,EAAE,EAAE;QAC3E,OAAO,EAAE;YACR;gBACC,IAAI,EAAE,MAAM;gBACZ,KAAK,EAAE,MAAM;gBACb,WAAW,EAAE,4DAA4D;aACzE;YACD;gBACC,IAAI,EAAE,UAAU;gBAChB,KAAK,EAAE,UAAU;gBACjB,WAAW,EAAE,kEAAkE;aAC/E;YACD;gBACC,IAAI,EAAE,YAAY;gBAClB,KAAK,EAAE,MAAM;gBACb,WAAW,EAAE,iDAAiD;aAC9D;SACD;QACD,OAAO,EAAE,EAAE,OAAO,EAAE,EAAE,IAAI,EAAE,EAAE,MAAM,EAAE,eAAe,EAAE,EAAE,EAAE;QAC3D,WAAW,EAAE,yEAAyE;KACtF;IACD;QACC,WAAW,EAAE,MAAM;QACnB,IAAI,EAAE,MAAM;QACZ,IAAI,EAAE,SAAS;QACf,OAAO,EAAE,QAAQ;QACjB,cAAc,EAAE,EAAE,IAAI,EAAE,EAAE,QAAQ,EAAE,CAAC,SAAS,CAAC,EAAE,SAAS,EAAE,CAAC,SAAS,CAAC,EAAE,EAAE;QAC3E,OAAO,EAAE;YACR;gBACC,IAAI,EAAE,UAAU;gBAChB,KAAK,EAAE,UAAU;gBACjB,WAAW,EAAE,iEAAiE;aAC9E;YACD;gBACC,IAAI,EAAE,QAAQ;gBACd,KAAK,EAAE,QAAQ;gBACf,WAAW,EAAE,mEAAmE;aAChF;YACD;gBACC,IAAI,EAAE,OAAO;gBACb,KAAK,EAAE,OAAO;gBACd,WAAW,EAAE,uDAAuD;aACpE;SACD;QACD,OAAO,EAAE,EAAE,OAAO,EAAE,EAAE,IAAI,EAAE,EAAE,IAAI,EAAE,eAAe,EAAE,EAAE,EAAE;QACzD,WAAW,EACV,qNAAqN;KACtN;IAED,0BAA0B;IAC1B;QACC,WAAW,EAAE,KAAK;QAClB,IAAI,EAAE,KAAK;QACX,IAAI,EAAE,QAAQ;QACd,QAAQ,EAAE,IAAI;QACd,OAAO,EAAE,EAAE;QACX,WAAW,EAAE,qBAAqB;QAClC,cAAc,EAAE,EAAE,IAAI,EAAE,EAAE,QAAQ,EAAE,CAAC,SAAS,CAAC,EAAE,SAAS,EAAE,CAAC,YAAY,CAAC,EAAE,EAAE;QAC9E,OAAO,EAAE,EAAE,OAAO,EAAE,EAAE,IAAI,EAAE,EAAE,GAAG,EAAE,eAAe,EAAE,EAAE,EAAE;QACxD,WAAW,EAAE,sEAAsE;KACnF;IACD;QACC,WAAW,EAAE,MAAM;QACnB,IAAI,EAAE,MAAM;QACZ,IAAI,EAAE,SAAS;QACf,OAAO,EAAE,QAAQ;QACjB,cAAc,EAAE,EAAE,IAAI,EAAE,EAAE,QAAQ,EAAE,CAAC,SAAS,CAAC,EAAE,SAAS,EAAE,CAAC,YAAY,CAAC,EAAE,EAAE;QAC9E,OAAO,EAAE;YACR;gBACC,IAAI,EAAE,UAAU;gBAChB,KAAK,EAAE,UAAU;gBACjB,WAAW,EAAE,4DAA4D;aACzE;YACD;gBACC,IAAI,EAAE,QAAQ;gBACd,KAAK,EAAE,QAAQ;gBACf,WAAW,EAAE,sBAAsB;aACnC;YACD;gBACC,IAAI,EAAE,OAAO;gBACb,KAAK,EAAE,OAAO;gBACd,WAAW,EAAE,wDAAwD;aACrE;SACD;QACD,OAAO,EAAE,EAAE,OAAO,EAAE,EAAE,IAAI,EAAE,EAAE,IAAI,EAAE,eAAe,EAAE,EAAE,EAAE;QACzD,WAAW,EAAE,mHAAmH;KAChI;CACD,CAAC"}
1
+ {"version":3,"file":"ExtractDescription.js","sourceRoot":"","sources":["../../../../nodes/Scavio/descriptions/ExtractDescription.ts"],"names":[],"mappings":";;;AAEA,qDAK0B;AAE1B,wDAAwD;AACxD,MAAM,SAAS,GAAG,qEAAqE,CAAC;AACxF,uEAAuE;AACvE,MAAM,SAAS,GAAG,0EAA0E,CAAC;AAE7F,MAAM,OAAO,GACZ,4LAA4L,CAAC;AAC9L,MAAM,MAAM,GACX,oLAAoL,CAAC;AAEtL,MAAM,IAAI,GAAG,CAAC,SAAmB,EAAE,QAAmC,EAAE,EAAE,EAAE,CAAC,CAAC;IAC7E,IAAI,EAAE,EAAE,QAAQ,EAAE,CAAC,SAAS,CAAC,EAAE,SAAS,EAAE,GAAG,KAAK,EAAE;CACpD,CAAC,CAAC;AAEH,MAAM,SAAS,GAA2B;IACzC,EAAE,IAAI,EAAE,UAAU,EAAE,KAAK,EAAE,UAAU,EAAE,WAAW,EAAE,uCAAuC,EAAE;IAC7F,EAAE,IAAI,EAAE,QAAQ,EAAE,KAAK,EAAE,QAAQ,EAAE,WAAW,EAAE,sCAAsC,EAAE;IACxF,EAAE,IAAI,EAAE,OAAO,EAAE,KAAK,EAAE,OAAO,EAAE,WAAW,EAAE,2CAA2C,EAAE;CAC3F,CAAC;AAEF,MAAM,WAAW,GAA2B;IAC3C,EAAE,IAAI,EAAE,MAAM,EAAE,KAAK,EAAE,MAAM,EAAE,WAAW,EAAE,gCAAgC,EAAE;IAC9E,EAAE,IAAI,EAAE,UAAU,EAAE,KAAK,EAAE,UAAU,EAAE,WAAW,EAAE,4CAA4C,EAAE;IAClG,EAAE,IAAI,EAAE,YAAY,EAAE,KAAK,EAAE,MAAM,EAAE,WAAW,EAAE,iDAAiD,EAAE;CACrG,CAAC;AAEF,MAAM,eAAe,GAA2B;IAC/C,EAAE,IAAI,EAAE,SAAS,EAAE,KAAK,EAAE,SAAS,EAAE,WAAW,EAAE,wCAAwC,EAAE;IAC5F,EAAE,IAAI,EAAE,MAAM,EAAE,KAAK,EAAE,MAAM,EAAE,WAAW,EAAE,uBAAuB,EAAE;IACrE,EAAE,IAAI,EAAE,MAAM,EAAE,KAAK,EAAE,MAAM,EAAE,WAAW,EAAE,8BAA8B,EAAE;CAC5E,CAAC;AAEF,MAAM,SAAS,GACd,uLAAuL,CAAC;AAEzL,SAAS,SAAS;IACjB,OAAO;QACN,WAAW,EAAE,QAAQ;QACrB,IAAI,EAAE,QAAQ;QACd,IAAI,EAAE,SAAS;QACf,OAAO,EAAE,UAAU;QACnB,OAAO,EAAE,WAAW;QACpB,WAAW,EAAE,0EAA0E;QACvF,OAAO,EAAE,EAAE,OAAO,EAAE,EAAE,IAAI,EAAE,EAAE,MAAM,EAAE,eAAe,EAAE,EAAE,EAAE;KAC3D,CAAC;AACH,CAAC;AAED,SAAS,OAAO;IACf,OAAO;QACN,WAAW,EAAE,MAAM;QACnB,IAAI,EAAE,MAAM;QACZ,IAAI,EAAE,SAAS;QACf,OAAO,EAAE,QAAQ;QACjB,OAAO,EAAE,SAAS;QAClB,WAAW,EAAE,mGAAmG;QAChH,OAAO,EAAE,EAAE,OAAO,EAAE,EAAE,IAAI,EAAE,EAAE,IAAI,EAAE,eAAe,EAAE,EAAE,EAAE;KACzD,CAAC;AACH,CAAC;AAED,SAAS,cAAc;IACtB,OAAO;QACN,WAAW,EAAE,iBAAiB;QAC9B,IAAI,EAAE,iBAAiB;QACvB,IAAI,EAAE,QAAQ;QACd,OAAO,EAAE,EAAE;QACX,WAAW,EACV,mIAAmI;QACpI,OAAO,EAAE,EAAE,OAAO,EAAE,EAAE,OAAO,EAAE,EAAE,iBAAiB,EAAE,eAAe,EAAE,EAAE,EAAE;KACzE,CAAC;AACH,CAAC;AAED,SAAS,UAAU;IAClB,OAAO;QACN,WAAW,EAAE,aAAa;QAC1B,IAAI,EAAE,aAAa;QACnB,IAAI,EAAE,QAAQ;QACd,OAAO,EAAE,EAAE;QACX,WAAW,EAAE,kCAAkC;QAC/C,WAAW,EACV,6SAA6S;QAC9S,OAAO,EAAE,EAAE,OAAO,EAAE,EAAE,IAAI,EAAE,EAAE,WAAW,EAAE,eAAe,EAAE,EAAE,EAAE;KAChE,CAAC;AACH,CAAC;AAED,SAAS,UAAU,CAAC,SAAiB;IACpC,OAAO;QACN;YACC,WAAW,EAAE,qBAAqB;YAClC,IAAI,EAAE,mBAAmB;YACzB,IAAI,EAAE,SAAS;YACf,OAAO,EAAE,KAAK;YACd,cAAc,EAAE,IAAI,CAAC,CAAC,SAAS,CAAC,CAAC;YACjC,WAAW,EACV,4PAA4P;SAC7P;QACD;YACC,WAAW,EAAE,oBAAoB;YACjC,IAAI,EAAE,gBAAgB;YACtB,IAAI,EAAE,QAAQ;YACd,OAAO,EAAE,qCAAoB;YAC7B,WAAW,EAAE,EAAE,QAAQ,EAAE,EAAE,EAAE,QAAQ,EAAE,iCAAgB,EAAE;YACzD,cAAc,EAAE,IAAI,CAAC,CAAC,SAAS,CAAC,EAAE,EAAE,iBAAiB,EAAE,CAAC,IAAI,CAAC,EAAE,CAAC;YAChE,WAAW,EAAE,8DAA8D,iCAAgB,wDAAwD;SACnJ;QACD;YACC,WAAW,EAAE,yBAAyB;YACtC,IAAI,EAAE,qBAAqB;YAC3B,IAAI,EAAE,QAAQ;YACd,OAAO,EAAE,qCAAoB;YAC7B,WAAW,EAAE,EAAE,QAAQ,EAAE,CAAC,EAAE,QAAQ,EAAE,EAAE,EAAE;YAC1C,cAAc,EAAE,IAAI,CAAC,CAAC,SAAS,CAAC,EAAE,EAAE,iBAAiB,EAAE,CAAC,IAAI,CAAC,EAAE,CAAC;YAChE,WAAW,EAAE,sEAAsE;SACnF;KACD,CAAC;AACH,CAAC;AAED,SAAS,UAAU,CAAC,IAAuB,EAAE,SAAiB;IAC7D,OAAO;QACN,WAAW,EAAE,QAAQ;QACrB,IAAI,EAAE,OAAO;QACb,IAAI,EAAE,QAAQ;QACd,QAAQ,EAAE,IAAI;QACd,OAAO,EAAE,EAAE;QACX,WAAW,EAAE,GAAG,IAAI,eAAe;QACnC,cAAc,EAAE,IAAI,CAAC,CAAC,SAAS,CAAC,CAAC;QACjC,WAAW,EAAE,4BAA4B,IAAI,KAAK,OAAO,CAAC,CAAC,CAAC,OAAO,CAAC,CAAC,CAAC,OAAO,EAAE;QAC/E,OAAO,EAAE;YACR,OAAO,EAAE,EAAE,GAAG,EAAE,oBAAoB,IAAI,8BAA8B,EAAE;SACxE;KACD,CAAC;AACH,CAAC;AAED,SAAS,aAAa,CAAC,SAAiB;IACvC,OAAO;QACN,WAAW,EAAE,oBAAoB;QACjC,IAAI,EAAE,mBAAmB;QACzB,IAAI,EAAE,YAAY;QAClB,WAAW,EAAE,YAAY;QACzB,OAAO,EAAE,EAAE;QACX,cAAc,EAAE,IAAI,CAAC,CAAC,SAAS,CAAC,CAAC;QACjC,OAAO,EAAE;YACR;gBACC,WAAW,EAAE,QAAQ;gBACrB,IAAI,EAAE,QAAQ;gBACd,IAAI,EAAE,QAAQ;gBACd,OAAO,EAAE,EAAE;gBACX,WAAW,EAAE,+EAA+E;gBAC5F,OAAO,EAAE,EAAE,OAAO,EAAE,EAAE,EAAE,EAAE,EAAE,MAAM,EAAE,eAAe,EAAE,EAAE,EAAE;aACzD;YACD;gBACC,WAAW,EAAE,iBAAiB;gBAC9B,IAAI,EAAE,iBAAiB;gBACvB,IAAI,EAAE,SAAS;gBACf,OAAO,EAAE,IAAI;gBACb,WAAW,EAAE,iGAAiG;gBAC9G,OAAO,EAAE,EAAE,OAAO,EAAE,EAAE,EAAE,EAAE,EAAE,eAAe,EAAE,kCAAkC,EAAE,EAAE,EAAE;aACrF;YACD;gBACC,WAAW,EAAE,kBAAkB;gBAC/B,IAAI,EAAE,WAAW;gBACjB,IAAI,EAAE,QAAQ;gBACd,OAAO,EAAE,EAAE;gBACX,WAAW,EAAE,EAAE,QAAQ,EAAE,CAAC,EAAE,QAAQ,EAAE,GAAG,EAAE;gBAC3C,WAAW,EAAE,sGAAsG;gBACnH,OAAO,EAAE,EAAE,OAAO,EAAE,EAAE,EAAE,EAAE,EAAE,KAAK,EAAE,eAAe,EAAE,EAAE,EAAE;aACxD;YACD;gBACC,WAAW,EAAE,QAAQ;gBACrB,IAAI,EAAE,QAAQ;gBACd,IAAI,EAAE,SAAS;gBACf,OAAO,EAAE,WAAW;gBACpB,OAAO,EAAE;oBACR,EAAE,IAAI,EAAE,QAAQ,EAAE,KAAK,EAAE,QAAQ,EAAE;oBACnC,EAAE,IAAI,EAAE,SAAS,EAAE,KAAK,EAAE,SAAS,EAAE;oBACrC,EAAE,IAAI,EAAE,WAAW,EAAE,KAAK,EAAE,WAAW,EAAE;iBACzC;gBACD,WAAW,EAAE,sCAAsC;gBACnD,OAAO,EAAE,EAAE,OAAO,EAAE,EAAE,EAAE,EAAE,EAAE,MAAM,EAAE,eAAe,EAAE,EAAE,EAAE;aACzD;SACD;KACD,CAAC;AACH,CAAC;AAEY,QAAA,iBAAiB,GAAsB;IACnD;QACC,WAAW,EAAE,WAAW;QACxB,IAAI,EAAE,WAAW;QACjB,IAAI,EAAE,SAAS;QACf,gBAAgB,EAAE,IAAI;QACtB,cAAc,EAAE,EAAE,IAAI,EAAE,EAAE,QAAQ,EAAE,CAAC,SAAS,CAAC,EAAE,EAAE;QACnD,OAAO,EAAE;YACR;gBACC,IAAI,EAAE,cAAc;gBACpB,KAAK,EAAE,aAAa;gBACpB,MAAM,EAAE,4BAA4B;gBACpC,WAAW,EACV,yJAAyJ;gBAC1J,OAAO,EAAE,EAAE,OAAO,EAAE,EAAE,MAAM,EAAE,QAAQ,EAAE,EAAE;aAC1C;YACD;gBACC,IAAI,EAAE,cAAc;gBACpB,KAAK,EAAE,aAAa;gBACpB,MAAM,EAAE,oBAAoB;gBAC5B,WAAW,EACV,yJAAyJ;gBAC1J,OAAO,EAAE,EAAE,OAAO,EAAE,EAAE,MAAM,EAAE,QAAQ,EAAE,EAAE;aAC1C;YACD;gBACC,IAAI,EAAE,cAAc;gBACpB,KAAK,EAAE,SAAS;gBAChB,MAAM,EAAE,oBAAoB;gBAC5B,WAAW,EACV,uTAAuT;gBACxT,OAAO,EAAE,EAAE,OAAO,EAAE,EAAE,MAAM,EAAE,MAAM,EAAE,GAAG,EAAE,iBAAiB,EAAE,EAAE;aAChE;YACD;gBACC,IAAI,EAAE,kBAAkB;gBACxB,KAAK,EAAE,aAAa;gBACpB,MAAM,EAAE,8BAA8B;gBACtC,WAAW,EACV,+KAA+K;gBAChL,OAAO,EAAE,EAAE,OAAO,EAAE,EAAE,MAAM,EAAE,KAAK,EAAE,EAAE;aACvC;YACD;gBACC,IAAI,EAAE,kBAAkB;gBACxB,KAAK,EAAE,aAAa;gBACpB,MAAM,EAAE,8BAA8B;gBACtC,WAAW,EACV,0MAA0M;gBAC3M,OAAO,EAAE,EAAE,OAAO,EAAE,EAAE,MAAM,EAAE,KAAK,EAAE,EAAE;aACvC;YACD;gBACC,IAAI,EAAE,aAAa;gBACnB,KAAK,EAAE,KAAK;gBACZ,MAAM,EAAE,eAAe;gBACvB,WAAW,EACV,yNAAyN;gBAC1N,OAAO,EAAE,EAAE,OAAO,EAAE,EAAE,MAAM,EAAE,MAAM,EAAE,GAAG,EAAE,qBAAqB,EAAE,EAAE;aACpE;YACD;gBACC,IAAI,EAAE,YAAY;gBAClB,KAAK,EAAE,YAAY;gBACnB,MAAM,EAAE,+BAA+B;gBACvC,WAAW,EACV,0GAA0G;gBAC3G,OAAO,EAAE,EAAE,OAAO,EAAE,EAAE,MAAM,EAAE,MAAM,EAAE,GAAG,EAAE,4BAA4B,EAAE,EAAE;aAC3E;YACD;gBACC,IAAI,EAAE,aAAa;gBACnB,KAAK,EAAE,OAAO;gBACd,MAAM,EAAE,2BAA2B;gBACnC,WAAW,EAAE,oNAAoN,OAAO,EAAE;gBAC1O,OAAO,EAAE;oBACR,OAAO,EAAE,EAAE,MAAM,EAAE,MAAM,EAAE,GAAG,EAAE,uBAAuB,EAAE;oBACzD,MAAM,EAAE,EAAE,WAAW,EAAE,CAAC,IAAA,2BAAU,EAAC,OAAO,CAAC,CAAC,EAAE;iBAC9C;aACD;YACD;gBACC,IAAI,EAAE,aAAa;gBACnB,KAAK,EAAE,OAAO;gBACd,MAAM,EAAE,mBAAmB;gBAC3B,WAAW,EAAE,6RAA6R,OAAO,EAAE;gBACnT,OAAO,EAAE;oBACR,OAAO,EAAE,EAAE,MAAM,EAAE,MAAM,EAAE,GAAG,EAAE,uBAAuB,EAAE;oBACzD,MAAM,EAAE,EAAE,WAAW,EAAE,CAAC,IAAA,2BAAU,EAAC,OAAO,CAAC,CAAC,EAAE;iBAC9C;aACD;SACD;QACD,OAAO,EAAE,SAAS;KAClB;CACD,CAAC;AAEW,QAAA,aAAa,GAAsB;IAC/C;QACC,WAAW,EAAE,KAAK;QAClB,IAAI,EAAE,KAAK;QACX,IAAI,EAAE,QAAQ;QACd,QAAQ,EAAE,IAAI;QACd,OAAO,EAAE,EAAE;QACX,WAAW,EAAE,+BAA+B;QAC5C,cAAc,EAAE,EAAE,IAAI,EAAE,EAAE,QAAQ,EAAE,CAAC,SAAS,CAAC,EAAE,SAAS,EAAE,CAAC,SAAS,CAAC,EAAE,EAAE;QAC3E,OAAO,EAAE,EAAE,OAAO,EAAE,EAAE,IAAI,EAAE,EAAE,GAAG,EAAE,eAAe,EAAE,EAAE,EAAE;QACxD,WAAW,EACV,oJAAoJ;KACrJ;IACD;QACC,WAAW,EAAE,eAAe;QAC5B,IAAI,EAAE,QAAQ;QACd,IAAI,EAAE,SAAS;QACf,OAAO,EAAE,UAAU;QACnB,cAAc,EAAE,EAAE,IAAI,EAAE,EAAE,QAAQ,EAAE,CAAC,SAAS,CAAC,EAAE,SAAS,EAAE,CAAC,SAAS,CAAC,EAAE,EAAE;QAC3E,OAAO,EAAE;YACR;gBACC,IAAI,EAAE,MAAM;gBACZ,KAAK,EAAE,MAAM;gBACb,WAAW,EAAE,4DAA4D;aACzE;YACD;gBACC,IAAI,EAAE,UAAU;gBAChB,KAAK,EAAE,UAAU;gBACjB,WAAW,EAAE,kEAAkE;aAC/E;YACD;gBACC,IAAI,EAAE,YAAY;gBAClB,KAAK,EAAE,MAAM;gBACb,WAAW,EAAE,iDAAiD;aAC9D;SACD;QACD,OAAO,EAAE,EAAE,OAAO,EAAE,EAAE,IAAI,EAAE,EAAE,MAAM,EAAE,eAAe,EAAE,EAAE,EAAE;QAC3D,WAAW,EAAE,yEAAyE;KACtF;IACD;QACC,WAAW,EAAE,MAAM;QACnB,IAAI,EAAE,MAAM;QACZ,IAAI,EAAE,SAAS;QACf,OAAO,EAAE,QAAQ;QACjB,cAAc,EAAE,EAAE,IAAI,EAAE,EAAE,QAAQ,EAAE,CAAC,SAAS,CAAC,EAAE,SAAS,EAAE,CAAC,SAAS,CAAC,EAAE,EAAE;QAC3E,OAAO,EAAE;YACR;gBACC,IAAI,EAAE,UAAU;gBAChB,KAAK,EAAE,UAAU;gBACjB,WAAW,EAAE,8DAA8D;aAC3E;YACD;gBACC,IAAI,EAAE,QAAQ;gBACd,KAAK,EAAE,QAAQ;gBACf,WAAW,EAAE,8DAA8D;aAC3E;YACD;gBACC,IAAI,EAAE,OAAO;gBACb,KAAK,EAAE,OAAO;gBACd,WAAW,EAAE,kCAAkC;aAC/C;SACD;QACD,OAAO,EAAE,EAAE,OAAO,EAAE,EAAE,IAAI,EAAE,EAAE,IAAI,EAAE,eAAe,EAAE,EAAE,EAAE;QACzD,WAAW,EACV,sNAAsN;KACvN;IAED,0BAA0B;IAC1B;QACC,WAAW,EAAE,KAAK;QAClB,IAAI,EAAE,KAAK;QACX,IAAI,EAAE,QAAQ;QACd,QAAQ,EAAE,IAAI;QACd,OAAO,EAAE,EAAE;QACX,WAAW,EAAE,qBAAqB;QAClC,cAAc,EAAE,EAAE,IAAI,EAAE,EAAE,QAAQ,EAAE,CAAC,SAAS,CAAC,EAAE,SAAS,EAAE,CAAC,YAAY,CAAC,EAAE,EAAE;QAC9E,OAAO,EAAE,EAAE,OAAO,EAAE,EAAE,IAAI,EAAE,EAAE,GAAG,EAAE,eAAe,EAAE,EAAE,EAAE;QACxD,WAAW,EAAE,sEAAsE;KACnF;IACD;QACC,WAAW,EAAE,MAAM;QACnB,IAAI,EAAE,MAAM;QACZ,IAAI,EAAE,SAAS;QACf,OAAO,EAAE,QAAQ;QACjB,cAAc,EAAE,EAAE,IAAI,EAAE,EAAE,QAAQ,EAAE,CAAC,SAAS,CAAC,EAAE,SAAS,EAAE,CAAC,YAAY,CAAC,EAAE,EAAE;QAC9E,OAAO,EAAE;YACR;gBACC,IAAI,EAAE,UAAU;gBAChB,KAAK,EAAE,UAAU;gBACjB,WAAW,EAAE,4DAA4D;aACzE;YACD;gBACC,IAAI,EAAE,QAAQ;gBACd,KAAK,EAAE,QAAQ;gBACf,WAAW,EAAE,sBAAsB;aACnC;YACD;gBACC,IAAI,EAAE,OAAO;gBACb,KAAK,EAAE,OAAO;gBACd,WAAW,EAAE,wDAAwD;aACrE;SACD;QACD,OAAO,EAAE,EAAE,OAAO,EAAE,EAAE,IAAI,EAAE,EAAE,IAAI,EAAE,eAAe,EAAE,EAAE,EAAE;QACzD,WAAW,EAAE,mHAAmH;KAChI;IAED,oBAAoB;IACpB;QACC,WAAW,EAAE,KAAK;QAClB,IAAI,EAAE,KAAK;QACX,IAAI,EAAE,QAAQ;QACd,QAAQ,EAAE,IAAI;QACd,OAAO,EAAE,EAAE;QACX,WAAW,EAAE,qBAAqB;QAClC,cAAc,EAAE,IAAI,CAAC,CAAC,KAAK,CAAC,CAAC;QAC7B,OAAO,EAAE,EAAE,OAAO,EAAE,EAAE,IAAI,EAAE,EAAE,GAAG,EAAE,8BAA8B,EAAE,EAAE,EAAE;QACvE,WAAW,EAAE,oDAAoD;KACjE;IACD;QACC,WAAW,EAAE,oBAAoB;QACjC,IAAI,EAAE,mBAAmB;QACzB,IAAI,EAAE,YAAY;QAClB,WAAW,EAAE,YAAY;QACzB,OAAO,EAAE,EAAE;QACX,cAAc,EAAE,IAAI,CAAC,CAAC,KAAK,CAAC,CAAC;QAC7B,OAAO,EAAE;YACR;gBACC,WAAW,EAAE,eAAe;gBAC5B,IAAI,EAAE,eAAe;gBACrB,IAAI,EAAE,QAAQ;gBACd,OAAO,EAAE,EAAE;gBACX,WAAW,EAAE,qBAAqB;gBAClC,WAAW,EAAE,kBAAkB,SAAS,GAAG;gBAC3C,OAAO,EAAE,EAAE,IAAI,EAAE,EAAE,IAAI,EAAE,MAAM,EAAE,QAAQ,EAAE,eAAe,EAAE,KAAK,EAAE,SAAS,EAAE,EAAE;aAChF;YACD;gBACC,WAAW,EAAE,yBAAyB;gBACtC,IAAI,EAAE,yBAAyB;gBAC/B,IAAI,EAAE,SAAS;gBACf,OAAO,EAAE,IAAI;gBACb,WAAW,EAAE,0FAA0F;gBACvG,OAAO,EAAE,EAAE,OAAO,EAAE,EAAE,IAAI,EAAE,EAAE,uBAAuB,EAAE,eAAe,EAAE,EAAE,EAAE;aAC5E;YACD;gBACC,WAAW,EAAE,eAAe;gBAC5B,IAAI,EAAE,eAAe;gBACrB,IAAI,EAAE,QAAQ;gBACd,OAAO,EAAE,EAAE;gBACX,WAAW,EAAE,UAAU;gBACvB,WAAW,EAAE,kBAAkB,SAAS,GAAG;gBAC3C,OAAO,EAAE,EAAE,IAAI,EAAE,EAAE,IAAI,EAAE,MAAM,EAAE,QAAQ,EAAE,eAAe,EAAE,KAAK,EAAE,SAAS,EAAE,EAAE;aAChF;YACD;gBACC,WAAW,EAAE,oBAAoB;gBACjC,IAAI,EAAE,oBAAoB;gBAC1B,IAAI,EAAE,SAAS;gBACf,OAAO,EAAE,KAAK;gBACd,WAAW,EAAE,0GAA0G;gBACvH,OAAO,EAAE,EAAE,OAAO,EAAE,EAAE,IAAI,EAAE,EAAE,kBAAkB,EAAE,eAAe,EAAE,EAAE,EAAE;aACvE;YACD;gBACC,WAAW,EAAE,WAAW;gBACxB,IAAI,EAAE,WAAW;gBACjB,IAAI,EAAE,QAAQ;gBACd,OAAO,EAAE,IAAI;gBACb,WAAW,EAAE,EAAE,QAAQ,EAAE,CAAC,EAAE,QAAQ,EAAE,IAAI,EAAE;gBAC5C,WAAW,EAAE,sJAAsJ;gBACnK,OAAO,EAAE,EAAE,OAAO,EAAE,EAAE,IAAI,EAAE,EAAE,KAAK,EAAE,eAAe,EAAE,EAAE,EAAE;aAC1D;YACD;gBACC,WAAW,EAAE,MAAM;gBACnB,IAAI,EAAE,MAAM;gBACZ,IAAI,EAAE,SAAS;gBACf,OAAO,EAAE,QAAQ;gBACjB,OAAO,EAAE;oBACR,EAAE,IAAI,EAAE,UAAU,EAAE,KAAK,EAAE,UAAU,EAAE,WAAW,EAAE,oBAAoB,EAAE;oBAC1E,EAAE,IAAI,EAAE,QAAQ,EAAE,KAAK,EAAE,QAAQ,EAAE,WAAW,EAAE,6BAA6B,EAAE;oBAC/E,EAAE,IAAI,EAAE,OAAO,EAAE,KAAK,EAAE,OAAO,EAAE,WAAW,EAAE,uBAAuB,EAAE;iBACvE;gBACD,WAAW,EAAE,kFAAkF;gBAC/F,OAAO,EAAE,EAAE,OAAO,EAAE,EAAE,IAAI,EAAE,EAAE,IAAI,EAAE,eAAe,EAAE,EAAE,EAAE;aACzD;YACD;gBACC,WAAW,EAAE,QAAQ;gBACrB,IAAI,EAAE,QAAQ;gBACd,IAAI,EAAE,QAAQ;gBACd,OAAO,EAAE,EAAE;gBACX,WAAW,EAAE,SAAS;gBACtB,WAAW,EAAE,8FAA8F;gBAC3G,OAAO,EAAE,EAAE,OAAO,EAAE,EAAE,IAAI,EAAE,EAAE,MAAM,EAAE,eAAe,EAAE,EAAE,EAAE;aAC3D;YACD;gBACC,WAAW,EAAE,SAAS;gBACtB,IAAI,EAAE,SAAS;gBACf,IAAI,EAAE,SAAS;gBACf,OAAO,EAAE,SAAS;gBAClB,OAAO,EAAE,eAAe;gBACxB,WAAW,EAAE,8EAA8E;gBAC3F,OAAO,EAAE,EAAE,OAAO,EAAE,EAAE,IAAI,EAAE,EAAE,OAAO,EAAE,eAAe,EAAE,EAAE,EAAE;aAC5D;SACD;KACD;IAED,oBAAoB;IACpB;QACC,WAAW,EAAE,KAAK;QAClB,IAAI,EAAE,KAAK;QACX,IAAI,EAAE,QAAQ;QACd,QAAQ,EAAE,IAAI;QACd,OAAO,EAAE,EAAE;QACX,WAAW,EAAE,0BAA0B;QACvC,cAAc,EAAE,IAAI,CAAC,CAAC,OAAO,CAAC,CAAC;QAC/B,OAAO,EAAE,EAAE,OAAO,EAAE,EAAE,IAAI,EAAE,EAAE,GAAG,EAAE,8BAA8B,EAAE,EAAE,EAAE;QACvE,WAAW,EACV,mHAAmH;KACpH;IACD,GAAG,UAAU,CAAC,OAAO,CAAC;IACtB;QACC,WAAW,EAAE,oBAAoB;QACjC,IAAI,EAAE,mBAAmB;QACzB,IAAI,EAAE,YAAY;QAClB,WAAW,EAAE,YAAY;QACzB,OAAO,EAAE,EAAE;QACX,cAAc,EAAE,IAAI,CAAC,CAAC,OAAO,CAAC,CAAC;QAC/B,OAAO,EAAE;YACR;gBACC,WAAW,EAAE,kBAAkB;gBAC/B,IAAI,EAAE,kBAAkB;gBACxB,IAAI,EAAE,SAAS;gBACf,OAAO,EAAE,KAAK;gBACd,WAAW,EAAE,8CAA8C;gBAC3D,OAAO,EAAE,EAAE,OAAO,EAAE,EAAE,IAAI,EAAE,EAAE,gBAAgB,EAAE,eAAe,EAAE,EAAE,EAAE;aACrE;YACD;gBACC,WAAW,EAAE,eAAe;gBAC5B,IAAI,EAAE,eAAe;gBACrB,IAAI,EAAE,QAAQ;gBACd,OAAO,EAAE,EAAE;gBACX,WAAW,EAAE,cAAc;gBAC3B,WAAW,EAAE,8CAA8C,SAAS,GAAG;gBACvE,OAAO,EAAE,EAAE,IAAI,EAAE,EAAE,IAAI,EAAE,MAAM,EAAE,QAAQ,EAAE,eAAe,EAAE,KAAK,EAAE,SAAS,EAAE,EAAE;aAChF;YACD,SAAS,EAAE;YACX,cAAc,EAAE;YAChB;gBACC,WAAW,EAAE,yBAAyB;gBACtC,IAAI,EAAE,yBAAyB;gBAC/B,IAAI,EAAE,SAAS;gBACf,OAAO,EAAE,IAAI;gBACb,WAAW,EAAE,2FAA2F;gBACxG,OAAO,EAAE,EAAE,OAAO,EAAE,EAAE,IAAI,EAAE,EAAE,uBAAuB,EAAE,eAAe,EAAE,EAAE,EAAE;aAC5E;YACD;gBACC,WAAW,EAAE,eAAe;gBAC5B,IAAI,EAAE,eAAe;gBACrB,IAAI,EAAE,QAAQ;gBACd,OAAO,EAAE,EAAE;gBACX,WAAW,EAAE,UAAU;gBACvB,WAAW,EAAE,mBAAmB,SAAS,GAAG;gBAC5C,OAAO,EAAE,EAAE,IAAI,EAAE,EAAE,IAAI,EAAE,MAAM,EAAE,QAAQ,EAAE,eAAe,EAAE,KAAK,EAAE,SAAS,EAAE,EAAE;aAChF;YACD;gBACC,WAAW,EAAE,WAAW;gBACxB,IAAI,EAAE,WAAW;gBACjB,IAAI,EAAE,QAAQ;gBACd,OAAO,EAAE,CAAC;gBACV,WAAW,EAAE,EAAE,QAAQ,EAAE,CAAC,EAAE,QAAQ,EAAE,EAAE,EAAE;gBAC1C,WAAW,EAAE,+GAA+G;gBAC5H,OAAO,EAAE,EAAE,OAAO,EAAE,EAAE,IAAI,EAAE,EAAE,SAAS,EAAE,eAAe,EAAE,EAAE,EAAE;aAC9D;YACD;gBACC,WAAW,EAAE,WAAW;gBACxB,IAAI,EAAE,WAAW;gBACjB,IAAI,EAAE,QAAQ;gBACd,OAAO,EAAE,GAAG;gBACZ,WAAW,EAAE,EAAE,QAAQ,EAAE,CAAC,EAAE,QAAQ,EAAE,KAAK,EAAE;gBAC7C,WAAW,EAAE,mDAAmD,MAAM,GAAG;gBACzE,OAAO,EAAE,EAAE,OAAO,EAAE,EAAE,IAAI,EAAE,EAAE,SAAS,EAAE,eAAe,EAAE,EAAE,EAAE;aAC9D;YACD,OAAO,EAAE;YACT;gBACC,WAAW,EAAE,SAAS;gBACtB,IAAI,EAAE,SAAS;gBACf,IAAI,EAAE,SAAS;gBACf,OAAO,EAAE,SAAS;gBAClB,OAAO,EAAE;oBACR,EAAE,IAAI,EAAE,SAAS,EAAE,KAAK,EAAE,SAAS,EAAE,WAAW,EAAE,6BAA6B,EAAE;oBACjF,EAAE,IAAI,EAAE,MAAM,EAAE,KAAK,EAAE,MAAM,EAAE,WAAW,EAAE,6CAA6C,EAAE;oBAC3F,EAAE,IAAI,EAAE,MAAM,EAAE,KAAK,EAAE,MAAM,EAAE,WAAW,EAAE,mBAAmB,EAAE;iBACjE;gBACD,WAAW,EAAE,kEAAkE;gBAC/E,OAAO,EAAE,EAAE,OAAO,EAAE,EAAE,IAAI,EAAE,EAAE,OAAO,EAAE,eAAe,EAAE,EAAE,EAAE;aAC5D;YACD,UAAU,EAAE;SACZ;KACD;IAED,wCAAwC;IACxC,UAAU,CAAC,OAAO,EAAE,aAAa,CAAC;IAClC,aAAa,CAAC,aAAa,CAAC;IAC5B,UAAU,CAAC,OAAO,EAAE,aAAa,CAAC;IAElC,oBAAoB;IACpB;QACC,WAAW,EAAE,MAAM;QACnB,IAAI,EAAE,MAAM;QACZ,IAAI,EAAE,QAAQ;QACd,QAAQ,EAAE,IAAI;QACd,OAAO,EAAE,EAAE;QACX,WAAW,EAAE,EAAE,IAAI,EAAE,CAAC,EAAE;QACxB,WAAW,EAAE,8CAA8C;QAC3D,cAAc,EAAE,IAAI,CAAC,CAAC,OAAO,CAAC,CAAC;QAC/B,WAAW,EAAE,2EAA2E,MAAM,GAAG;QACjG,OAAO,EAAE,EAAE,IAAI,EAAE,EAAE,IAAI,EAAE,MAAM,EAAE,QAAQ,EAAE,MAAM,EAAE,KAAK,EAAE,SAAS,EAAE,EAAE;KACvE;IACD,GAAG,UAAU,CAAC,OAAO,CAAC;IACtB;QACC,WAAW,EAAE,oBAAoB;QACjC,IAAI,EAAE,mBAAmB;QACzB,IAAI,EAAE,YAAY;QAClB,WAAW,EAAE,YAAY;QACzB,OAAO,EAAE,EAAE;QACX,cAAc,EAAE,IAAI,CAAC,CAAC,OAAO,CAAC,CAAC;QAC/B,OAAO,EAAE,CAAC,SAAS,EAAE,EAAE,cAAc,EAAE,EAAE,OAAO,EAAE,EAAE,UAAU,EAAE,CAAC;KACjE;IAED,wCAAwC;IACxC,UAAU,CAAC,OAAO,EAAE,aAAa,CAAC;IAClC,aAAa,CAAC,aAAa,CAAC;IAC5B,UAAU,CAAC,OAAO,EAAE,aAAa,CAAC;CAClC,CAAC"}
@@ -0,0 +1,5 @@
1
+ import type { IExecuteSingleFunctions, IN8nHttpFullResponse, INodeExecutionData } from 'n8n-workflow';
2
+ export declare const MAX_WAIT_SECONDS = 600;
3
+ export declare const DEFAULT_WAIT_SECONDS = 300;
4
+ export declare const DEFAULT_POLL_SECONDS = 5;
5
+ export declare function waitForJob(type: 'crawl' | 'batch'): (this: IExecuteSingleFunctions, items: INodeExecutionData[], _response: IN8nHttpFullResponse) => Promise<INodeExecutionData[]>;
@@ -0,0 +1,90 @@
1
+ "use strict";
2
+ Object.defineProperty(exports, "__esModule", { value: true });
3
+ exports.DEFAULT_POLL_SECONDS = exports.DEFAULT_WAIT_SECONDS = exports.MAX_WAIT_SECONDS = void 0;
4
+ exports.waitForJob = waitForJob;
5
+ const n8n_workflow_1 = require("n8n-workflow");
6
+ const version_1 = require("../version");
7
+ // "Wait for Completion" on Start Crawl / Start Batch. The create call returns
8
+ // 202 with a job id; when the toggle is on, this postReceive hook polls the
9
+ // job's status endpoint until it finishes or the wait cap is reached, then
10
+ // returns the status (first page of results plus next_cursor) in place of the
11
+ // 202 body. A timeout is not an error: the latest status comes back with
12
+ // wait.timed_out = true so the workflow can carry on polling with Get Status.
13
+ const BASE_URL = 'https://api.scavio.dev';
14
+ const FINISHED = new Set(['completed', 'failed', 'cancelled']);
15
+ exports.MAX_WAIT_SECONDS = 600;
16
+ exports.DEFAULT_WAIT_SECONDS = 300;
17
+ exports.DEFAULT_POLL_SECONDS = 5;
18
+ const MAX_CONSECUTIVE_POLL_ERRORS = 3;
19
+ function clamp(value, min, max, fallback) {
20
+ const n = Number(value);
21
+ if (!Number.isFinite(n))
22
+ return fallback;
23
+ return Math.min(Math.max(n, min), max);
24
+ }
25
+ function waitForJob(type) {
26
+ return async function (items, _response) {
27
+ if (!this.getNodeParameter('waitForCompletion', false))
28
+ return items;
29
+ const maxWait = clamp(this.getNodeParameter('maxWaitSeconds', exports.DEFAULT_WAIT_SECONDS), 10, exports.MAX_WAIT_SECONDS, exports.DEFAULT_WAIT_SECONDS);
30
+ const interval = clamp(this.getNodeParameter('pollIntervalSeconds', exports.DEFAULT_POLL_SECONDS), 2, 60, exports.DEFAULT_POLL_SECONDS);
31
+ const out = [];
32
+ for (const item of items) {
33
+ const created = (item.json.data ?? {});
34
+ const id = created.id;
35
+ if (typeof id !== 'string' || !id) {
36
+ out.push(item);
37
+ continue;
38
+ }
39
+ const started = Date.now();
40
+ const deadline = started + maxWait * 1000;
41
+ let latest;
42
+ let errors = 0;
43
+ let finished = false;
44
+ for (;;) {
45
+ try {
46
+ latest = (await this.helpers.httpRequestWithAuthentication.call(this, 'scavioApi', {
47
+ method: 'GET',
48
+ url: `${BASE_URL}/api/v1/extract/${type}/${encodeURIComponent(id)}`,
49
+ headers: {
50
+ Accept: 'application/json',
51
+ 'User-Agent': `n8n-nodes-scavio/${version_1.NODE_PACKAGE_VERSION}`,
52
+ 'X-Client-Source': 'n8n-nodes-scavio',
53
+ },
54
+ json: true,
55
+ }));
56
+ errors = 0;
57
+ }
58
+ catch (error) {
59
+ errors++;
60
+ if (errors >= MAX_CONSECUTIVE_POLL_ERRORS) {
61
+ throw new n8n_workflow_1.NodeApiError(this.getNode(), error);
62
+ }
63
+ }
64
+ const status = latest?.data?.status;
65
+ if (typeof status === 'string' && FINISHED.has(status)) {
66
+ finished = true;
67
+ break;
68
+ }
69
+ if (Date.now() + interval * 1000 > deadline)
70
+ break;
71
+ await (0, n8n_workflow_1.sleep)(interval * 1000);
72
+ }
73
+ out.push({
74
+ json: {
75
+ ...(latest ?? item.json),
76
+ // The create response carries webhook_secret exactly once; keep it.
77
+ created,
78
+ wait: {
79
+ completed: finished,
80
+ timed_out: !finished,
81
+ waited_seconds: Math.round((Date.now() - started) / 1000),
82
+ },
83
+ },
84
+ pairedItem: item.pairedItem,
85
+ });
86
+ }
87
+ return out;
88
+ };
89
+ }
90
+ //# sourceMappingURL=ExtractJobWait.js.map
@@ -0,0 +1 @@
1
+ {"version":3,"file":"ExtractJobWait.js","sourceRoot":"","sources":["../../../../nodes/Scavio/descriptions/ExtractJobWait.ts"],"names":[],"mappings":";;;AA+BA,gCAgFC;AA/GD,+CAAmD;AASnD,wCAAkD;AAElD,8EAA8E;AAC9E,4EAA4E;AAC5E,2EAA2E;AAC3E,8EAA8E;AAC9E,yEAAyE;AACzE,8EAA8E;AAE9E,MAAM,QAAQ,GAAG,wBAAwB,CAAC;AAC1C,MAAM,QAAQ,GAAG,IAAI,GAAG,CAAC,CAAC,WAAW,EAAE,QAAQ,EAAE,WAAW,CAAC,CAAC,CAAC;AAClD,QAAA,gBAAgB,GAAG,GAAG,CAAC;AACvB,QAAA,oBAAoB,GAAG,GAAG,CAAC;AAC3B,QAAA,oBAAoB,GAAG,CAAC,CAAC;AACtC,MAAM,2BAA2B,GAAG,CAAC,CAAC;AAEtC,SAAS,KAAK,CAAC,KAAc,EAAE,GAAW,EAAE,GAAW,EAAE,QAAgB;IACxE,MAAM,CAAC,GAAG,MAAM,CAAC,KAAK,CAAC,CAAC;IACxB,IAAI,CAAC,MAAM,CAAC,QAAQ,CAAC,CAAC,CAAC;QAAE,OAAO,QAAQ,CAAC;IACzC,OAAO,IAAI,CAAC,GAAG,CAAC,IAAI,CAAC,GAAG,CAAC,CAAC,EAAE,GAAG,CAAC,EAAE,GAAG,CAAC,CAAC;AACxC,CAAC;AAED,SAAgB,UAAU,CAAC,IAAuB;IACjD,OAAO,KAAK,WAEX,KAA2B,EAC3B,SAA+B;QAE/B,IAAI,CAAE,IAAI,CAAC,gBAAgB,CAAC,mBAAmB,EAAE,KAAK,CAAa;YAAE,OAAO,KAAK,CAAC;QAElF,MAAM,OAAO,GAAG,KAAK,CACpB,IAAI,CAAC,gBAAgB,CAAC,gBAAgB,EAAE,4BAAoB,CAAC,EAC7D,EAAE,EACF,wBAAgB,EAChB,4BAAoB,CACpB,CAAC;QACF,MAAM,QAAQ,GAAG,KAAK,CACrB,IAAI,CAAC,gBAAgB,CAAC,qBAAqB,EAAE,4BAAoB,CAAC,EAClE,CAAC,EACD,EAAE,EACF,4BAAoB,CACpB,CAAC;QAEF,MAAM,GAAG,GAAyB,EAAE,CAAC;QACrC,KAAK,MAAM,IAAI,IAAI,KAAK,EAAE,CAAC;YAC1B,MAAM,OAAO,GAAG,CAAC,IAAI,CAAC,IAAI,CAAC,IAAI,IAAI,EAAE,CAAgB,CAAC;YACtD,MAAM,EAAE,GAAG,OAAO,CAAC,EAAE,CAAC;YACtB,IAAI,OAAO,EAAE,KAAK,QAAQ,IAAI,CAAC,EAAE,EAAE,CAAC;gBACnC,GAAG,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC;gBACf,SAAS;YACV,CAAC;YAED,MAAM,OAAO,GAAG,IAAI,CAAC,GAAG,EAAE,CAAC;YAC3B,MAAM,QAAQ,GAAG,OAAO,GAAG,OAAO,GAAG,IAAI,CAAC;YAC1C,IAAI,MAA+B,CAAC;YACpC,IAAI,MAAM,GAAG,CAAC,CAAC;YACf,IAAI,QAAQ,GAAG,KAAK,CAAC;YAErB,SAAS,CAAC;gBACT,IAAI,CAAC;oBACJ,MAAM,GAAG,CAAC,MAAM,IAAI,CAAC,OAAO,CAAC,6BAA6B,CAAC,IAAI,CAAC,IAAI,EAAE,WAAW,EAAE;wBAClF,MAAM,EAAE,KAAK;wBACb,GAAG,EAAE,GAAG,QAAQ,mBAAmB,IAAI,IAAI,kBAAkB,CAAC,EAAE,CAAC,EAAE;wBACnE,OAAO,EAAE;4BACR,MAAM,EAAE,kBAAkB;4BAC1B,YAAY,EAAE,oBAAoB,8BAAoB,EAAE;4BACxD,iBAAiB,EAAE,kBAAkB;yBACrC;wBACD,IAAI,EAAE,IAAI;qBACV,CAAC,CAAgB,CAAC;oBACnB,MAAM,GAAG,CAAC,CAAC;gBACZ,CAAC;gBAAC,OAAO,KAAK,EAAE,CAAC;oBAChB,MAAM,EAAE,CAAC;oBACT,IAAI,MAAM,IAAI,2BAA2B,EAAE,CAAC;wBAC3C,MAAM,IAAI,2BAAY,CAAC,IAAI,CAAC,OAAO,EAAE,EAAE,KAAmB,CAAC,CAAC;oBAC7D,CAAC;gBACF,CAAC;gBACD,MAAM,MAAM,GAAI,MAAM,EAAE,IAAgC,EAAE,MAAM,CAAC;gBACjE,IAAI,OAAO,MAAM,KAAK,QAAQ,IAAI,QAAQ,CAAC,GAAG,CAAC,MAAM,CAAC,EAAE,CAAC;oBACxD,QAAQ,GAAG,IAAI,CAAC;oBAChB,MAAM;gBACP,CAAC;gBACD,IAAI,IAAI,CAAC,GAAG,EAAE,GAAG,QAAQ,GAAG,IAAI,GAAG,QAAQ;oBAAE,MAAM;gBACnD,MAAM,IAAA,oBAAK,EAAC,QAAQ,GAAG,IAAI,CAAC,CAAC;YAC9B,CAAC;YAED,GAAG,CAAC,IAAI,CAAC;gBACR,IAAI,EAAE;oBACL,GAAG,CAAC,MAAM,IAAI,IAAI,CAAC,IAAI,CAAC;oBACxB,oEAAoE;oBACpE,OAAO;oBACP,IAAI,EAAE;wBACL,SAAS,EAAE,QAAQ;wBACnB,SAAS,EAAE,CAAC,QAAQ;wBACpB,cAAc,EAAE,IAAI,CAAC,KAAK,CAAC,CAAC,IAAI,CAAC,GAAG,EAAE,GAAG,OAAO,CAAC,GAAG,IAAI,CAAC;qBACzD;iBACD;gBACD,UAAU,EAAE,IAAI,CAAC,UAAU;aAC3B,CAAC,CAAC;QACJ,CAAC;QACD,OAAO,GAAG,CAAC;IACZ,CAAC,CAAC;AACH,CAAC"}
@@ -1 +1 @@
1
- export declare const NODE_PACKAGE_VERSION = "0.21.0";
1
+ export declare const NODE_PACKAGE_VERSION = "0.22.0";
@@ -2,5 +2,5 @@
2
2
  Object.defineProperty(exports, "__esModule", { value: true });
3
3
  exports.NODE_PACKAGE_VERSION = void 0;
4
4
  // Generated by scripts/gen-version.js from package.json. Do not edit.
5
- exports.NODE_PACKAGE_VERSION = '0.21.0';
5
+ exports.NODE_PACKAGE_VERSION = '0.22.0';
6
6
  //# sourceMappingURL=version.js.map
package/package.json CHANGED
@@ -1,7 +1,7 @@
1
1
  {
2
2
  "name": "n8n-nodes-scavio",
3
- "version": "0.21.0",
4
- "description": "n8n community node for Scavio — real-time structured JSON from Google, Amazon, Walmart, eBay, Target, Home Depot, Costco, Mercado Libre, YouTube, Reddit, TikTok, Instagram, X, LinkedIn, Threads, Kuaishou, Zillow, Redfin, Realtor.com, Rightmove, Booking.com, Airbnb, Tripadvisor, Yelp, Indeed, Glassdoor, App Store, Google Play, SEC EDGAR, Companies House, G2, Capterra, Trustpilot, ad libraries, and any URL.",
3
+ "version": "0.22.0",
4
+ "description": "n8n community node for Scavio — real-time structured JSON from Google, Amazon, Walmart, eBay, Target, Home Depot, Costco, Mercado Libre, YouTube, Reddit, TikTok, Instagram, X, LinkedIn, Threads, Kuaishou, Zillow, Redfin, Realtor.com, Rightmove, Booking.com, Airbnb, Tripadvisor, Yelp, Indeed, Glassdoor, App Store, Google Play, SEC EDGAR, Companies House, G2, Capterra, Trustpilot, ad libraries, any URL, and whole websites (map, crawl, batch extract).",
5
5
  "keywords": [
6
6
  "n8n-community-node-package",
7
7
  "scavio",
@@ -26,6 +26,9 @@
26
26
  "x",
27
27
  "linkedin",
28
28
  "serp",
29
+ "web-crawler",
30
+ "website-crawl",
31
+ "sitemap",
29
32
  "ai"
30
33
  ],
31
34
  "license": "MIT",