n8n-nodes-crawlbrulee 0.1.0 → 0.3.0

package/README.md

@@ -66,10 +66,35 @@ turn on **Download Screenshot** — it's on **Scrape URL** and on **Get Scrape R
 
 ### what comes back
 
-one item in, one item out: the api's json body becomes the item's `json`, unchanged. scrape, map and finished-job responses carry `response_meta.usage`, so the credits a call cost are in front of you. **Map Website** adds `pagination` and `truncation` there, and its **Limit** starts at 50 in n8n, not the api's 5000 — n8n's rule for a field named Limit sets that, so raising it is usually the first thing you do. the [api docs](https://crawlbrulee.com/docs) have the full response shape.
+one item in, one item out: the api's json body becomes the item's `json`, unchanged. **Map Website** adds `pagination` and `truncation` under `response_meta`, and its **Limit** starts at 50 in n8n, not the api's 5000 — n8n's rule for a field named Limit sets that, so raising it is usually the first thing you do. the [api docs](https://crawlbrulee.com/docs) have the full response shape.
 
 set the node's "On Error" setting to continue and a failed item comes back as `{ "error": "…" }` instead of stopping the run.
 
+### the page's own status
+
+**Scrape URL** and **Get Scrape Result** put the HTTP status the site answered with in `page_status_code`, next to `url`. it is the status of the final page, after redirects.
+
+a page the site really served is a normal item, whatever its status. a 404, 410 or 503 page comes back with its content, like any other page. it is not a node error, so "On Error" never sees it. when the status matters to your workflow, check it before you use the content — for example an **If** node on `{{ $json.page_status_code }}` is less than 400.
+
+### zero data retention
+
+turn on **Zero Data Retention** under **Options** of **Scrape URL**, **Scrape URL (Async)** or **Map Website** to keep the result out of the shared cache. anything stored to deliver it is kept for 24 hours, then deleted. it adds 1 credit and must be enabled for your organization. if it is not, the call fails with a clear message and costs no credits. see [zero data retention](https://crawlbrulee.com/docs/zero-data-retention).
+
+### what a call cost
+
+scrape, map and finished-job responses carry `response_meta.usage`, so the cost of each call is in front of you:
+
+| field | what it is |
+|---|---|
+| `total_credit_cost` | credits this call cost |
+| `engine_credit_cost` | the engine's base price: 1 for `http`, 3 for `browser`, 5 for `screenshot`, 0 for `cache` |
+| `proxy_multiplier` | 1 for the Basic proxy tier, 5 for Advanced |
+| `screenshot_slicing_credit_cost` | 1 when the screenshot was cut into slices on this call, else 0 (reusing slices from the cache costs 0). not on **Map Website** |
+| `zero_data_retention_credit_cost` | 1 when **Zero Data Retention** added its credit to this call, else 0 |
+| `engine`, `proxy` | the engine and the proxy tier that did the work |
+
+`total_credit_cost` is always `engine_credit_cost × proxy_multiplier + screenshot_slicing_credit_cost + zero_data_retention_credit_cost` (map has no slicing part). a page we don't bill, such as one the site answered with a 5xx status, has 0 in every `*_credit_cost` field. the [credits and pricing](https://crawlbrulee.com/docs/credits-and-pricing) page lists what is billed.
+
 ## the crawlbrulee Trigger
 
 use it to react to a background scrape instead of polling for it:
@@ -78,13 +103,13 @@ use it to react to a background scrape instead of polling for it:
 2. paste that url into the **Webhook URL** field of a **Scrape URL (Async)** operation.
 3. when the job finishes, we post one `scrape.complete` event there and the trigger fires.
 
-the item carries the event id, the timestamp and the job data. turn on **Fetch Result** and, for a job that succeeded, the trigger fetches the page content into `result`; if that fetch fails, the reason lands in `result_error` and the item still comes through.
+the item carries the event id, the timestamp and the job data. a job that got a page has status `success` and the page's HTTP status in `page_status_code`, so a 404 page arrives as `success` too. turn on **Fetch Result** and, for a job that succeeded, the trigger fetches the page content into `result`; if that fetch fails, the reason lands in `result_error` and the item still comes through.
 
 with the secret set, the trigger checks the HMAC-SHA256 signature on every delivery. it drops one that fails the check, one that isn't a `scrape.complete` event, and one whose event id is among the last 500 it saw: nothing runs, and we still answer 200 so it is not retried. that id list lives in the node's workflow data, so a fresh copy of the workflow starts empty.
 
 ## errors
 
-the node turns api errors into messages you can act on: a rate limit says how long to wait, out of credits and concurrency limit are each named, an anti-bot block says so with no retry hint, and a scrape error points you at the Advanced proxy tier and at Require JS. the raw response stays on the error for the n8n error view, and every error name is in the [error reference](https://crawlbrulee.com/docs/errors).
+the node turns api errors into messages you can act on: a rate limit says how long to wait, a Zero Data Retention request your organization cannot make yet says to turn the option off or ask us to enable it, out of credits and concurrency limit are each named, an anti-bot block says so with no retry hint, a scrape error points you at the Advanced proxy tier and at Require JS, and a site we could not reach at all (for example it timed out, or its certificate is bad) tells you to check the url and try again later. the raw response stays on the error for the n8n error view, and every error name is in the [error reference](https://crawlbrulee.com/docs/errors).
 
 ## development
 

package/dist/nodes/Crawlbrulee/builders/map.js

@@ -8,6 +8,8 @@ function buildMapBody(_node, _itemIndex, p) {
     const o = (_a = p.options) !== null && _a !== void 0 ? _a : {};
     if (o.sitemapOnly === true)
         body.sitemap_only = true;
+    if (o.zeroDataRetention === true)
+        body.zero_data_retention = true;
     const types = {};
     if (typeof o.internal === 'boolean')
         types.internal = o.internal;

package/dist/nodes/Crawlbrulee/builders/scrape.js

@@ -88,6 +88,8 @@ function buildScrapeBody(node, itemIndex, p) {
         body.proxy = o.proxy;
     if (o.requireJs === true)
         body.require_js = true;
+    if (o.zeroDataRetention === true)
+        body.zero_data_retention = true;
     const cleanup = {};
     if (typeof o.removeAdsAndPopups === 'boolean')
         cleanup.ads_and_popups = o.removeAdsAndPopups;

package/dist/nodes/Crawlbrulee/descriptions/map.js

@@ -92,6 +92,7 @@ exports.mapFields = [
                 default: false,
                 description: 'Whether to read only sitemap.xml and skip homepage link extraction',
             },
+            shared_1.zeroDataRetentionOption,
         ],
     },
 ];

package/dist/nodes/Crawlbrulee/descriptions/scrape.js

@@ -17,7 +17,7 @@ exports.scrapeOperations = {
             name: 'Get Scrape Result',
             value: 'getScrapeResult',
             action: 'Get scrape result',
-            description: 'Fetch the content of a finished async scrape',
+            description: "Fetch the content of a finished async scrape. The site's own HTTP status is in page_status_code.",
         },
         {
             name: 'Get Scrape Status',
@@ -29,7 +29,7 @@ exports.scrapeOperations = {
             name: 'Scrape URL',
             value: 'scrape',
             action: 'Scrape URL',
-            description: 'Fetch one URL and return the requested content',
+            description: "Fetch one URL and return the requested content. The site's own HTTP status is in page_status_code.",
         },
         {
             name: 'Scrape URL (Async)',
@@ -41,6 +41,13 @@ exports.scrapeOperations = {
     default: 'scrape',
 };
 exports.scrapeFields = [
+    {
+        displayName: 'A page the site answers with an error, like a 404, still comes back as a normal item, not as a node error. Its HTTP status is in page_status_code. Check it, for example with an If node, before you use the content.',
+        name: 'pageStatusNotice',
+        type: 'notice',
+        default: '',
+        displayOptions: { show: { resource: ['scrape'], operation: ['scrape', 'getScrapeResult'] } },
+    },
     {
         displayName: 'Job ID',
         name: 'jobId',
@@ -237,6 +244,7 @@ exports.scrapeFields = [
                 default: false,
                 description: 'Whether to render JavaScript in a headless browser first. Adds latency and credits.',
             },
+            shared_1.zeroDataRetentionOption,
         ],
     },
     {

package/dist/nodes/Crawlbrulee/descriptions/shared.js

@@ -1,6 +1,6 @@
 "use strict";
 Object.defineProperty(exports, "__esModule", { value: true });
-exports.localeOption = exports.countryOption = exports.cacheMaxAgeOption = exports.proxyOption = void 0;
+exports.localeOption = exports.countryOption = exports.zeroDataRetentionOption = exports.cacheMaxAgeOption = exports.proxyOption = void 0;
 exports.proxyOption = {
     displayName: 'Proxy',
     name: 'proxy',
@@ -30,6 +30,13 @@ const cacheMaxAgeOption = (defaultText) => ({
     description: `Seconds, or an ISO-8601 cutoff. Cached results older than this are skipped. Default ${defaultText}.`,
 });
 exports.cacheMaxAgeOption = cacheMaxAgeOption;
+exports.zeroDataRetentionOption = {
+    displayName: 'Zero Data Retention',
+    name: 'zeroDataRetention',
+    type: 'boolean',
+    default: false,
+    description: 'Whether to keep the result out of the shared cache. Anything stored to deliver it is kept for 24 hours, then deleted. Adds 1 credit. Must be enabled for your organization.',
+};
 exports.countryOption = {
     displayName: 'Country',
     name: 'country',

package/dist/nodes/Crawlbrulee/transport/errors.js

@@ -4,6 +4,8 @@ exports.describeApiError = describeApiError;
 exports.toNodeApiError = toNodeApiError;
 const n8n_workflow_1 = require("n8n-workflow");
 const KEY_REJECTED = 'crawlbrulee rejected the API key. Check the key on the credential.';
+const UNREACHABLE = 'crawlbrulee could not reach the target site. Check the URL. If it is right, the site may be down or slow, so try again later. This costs no credits.';
+const ZDR_NOT_ENABLED = 'Zero data retention is not enabled for your organization. Turn the Zero Data Retention option off, or contact sales@crawlbrulee.com to have it turned on. This costs no credits.';
 const SCRAPE_HINT = 'The Advanced proxy tier has a higher success rate; enable Require JS for JavaScript-rendered content.';
 function isApiErrorBody(body) {
     return (typeof body === 'object' &&
@@ -39,6 +41,10 @@ function describeApiError(status, body) {
             if (details.reason === 'concurrency_limit')
                 return { message: 'Concurrency limit reached.', description };
             return { message: body.message, description };
+        case 'target_unreachable':
+            return { message: UNREACHABLE, description };
+        case 'zero_data_retention_not_enabled':
+            return { message: ZDR_NOT_ENABLED, description };
         case 'scrape_error':
             return { message: `${body.message.replace(/[.\s]*$/, '')}. ${SCRAPE_HINT}`, description };
         default:

package/dist/nodes/Crawlbrulee/transport/request.js

@@ -3,7 +3,7 @@ Object.defineProperty(exports, "__esModule", { value: true });
 exports.USER_AGENT = void 0;
 exports.crawlbruleeRequest = crawlbruleeRequest;
 const errors_1 = require("./errors");
-exports.USER_AGENT = 'n8n-nodes-crawlbrulee/0.1.0';
+exports.USER_AGENT = 'n8n-nodes-crawlbrulee/0.3.0';
 function stripTrailingSlash(url) {
     return url.replace(/\/+$/, '');
 }

package/dist/nodes/CrawlbruleeTrigger/CrawlbruleeTrigger.node.js

@@ -29,7 +29,7 @@ class CrawlbruleeTrigger {
             ],
             properties: [
                 {
-                    displayName: "Copy this node's webhook URL into the Webhook URL field of a Scrape URL (Async) operation. Each finished job posts one scrape.complete event here. Set the Webhook Signing Secret on the credential to verify deliveries.",
+                    displayName: "Copy this node's webhook URL into the Webhook URL field of a Scrape URL (Async) operation. Each finished job posts one scrape.complete event here. Set the Webhook Signing Secret on the credential to verify deliveries. A page the site answers with an error, like a 404, still arrives with status success; its HTTP status is in page_status_code.",
                     name: 'notice',
                     type: 'notice',
                     default: '',

package/package.json

@@ -1,6 +1,6 @@
 {
 	"name": "n8n-nodes-crawlbrulee",
-	"version": "0.1.0",
+	"version": "0.3.0",
 	"description": "crawlbrulee for n8n — EU-native web scraping for AI agents & developers. scrape any url to markdown, html, links, images, metadata or a screenshot, map a site's urls, and react to finished background scrapes.",
 	"license": "MIT",
 	"homepage": "https://crawlbrulee.com",
@@ -57,7 +57,7 @@
 		]
 	},
 	"devDependencies": {
-		"@crawlbrulee/sdk": "^1.0.1",
+		"@crawlbrulee/sdk": "^1.2.0",
 		"@n8n/node-cli": "0.48.6",
 		"@types/node": "^22.0.0",
 		"eslint": "9.32.0",