@commercelayer/agent-cli 0.7.0 → 0.8.0

package/README.md

@@ -18,7 +18,8 @@ npx -y @commercelayer/agent-cli@latest init --name "The Blue Brand"   # runs c11
 | `init [--name <business name>]` | Create a sandbox store, with no login. The name (the folder name when absent) names the organization; the org slug and store hostname derive from it, numbered when taken. Writes `store.yml` (a commented template) and `credentials`, plus a short `AGENTS.md` and `CLAUDE.md` section (appended to an existing file, never twice) that sends later agents in the repo to [commercelayer.ai/skills.md](https://commercelayer.ai/skills.md). Reuses the store if they exist. |
 | `plan` | Validate `store.yml` and the product files, and list what `apply` would create, update or delete: markets, products, images, and the store's name, policies and theme. `missing_images` lists `img/<file>` references with no file in `.commercelayer/img/` (`apply` refuses them); `unused_images` lists files there that nothing references. |
 | `apply [--allow-delete] [--full]` | Make the store match the files, the website included: the theme settings, and the images in `.commercelayer/img/` that the theme and products reference, uploaded when new or changed (unused ones are dropped from the store host). Refuses deletes without `--allow-delete`. Product changes go through a Commerce Layer import: `apply` waits for it, with progress as JSON lines on stderr, and prints one result. `apply_failed` lists the products Commerce Layer refused. While Commerce Layer is busy or unavailable, `apply` keeps reading (backing off on busy) and writes a `retrying` line. `apply_pending` means the import keeps going after 10 minutes, and `c11r status` follows it. An apply still running is finished before a new one starts. `--full` re-imports every product and price, after edits in the Commerce Layer dashboard. |
-| `catalog pull` | Read the products from the catalog source in `store.yml` (a Supabase table, a Google Sheet or a Google Merchant Center feed) into `.commercelayer/products/catalog.csv`. Then `plan` and `apply` as usual. |
+| `catalog pull` | Read the products from the catalog source in `store.yml` (a Supabase table, a Google Sheet, a Google Merchant Center feed, or a brand's website for a demo store) into `.commercelayer/products/catalog.csv`. Then `plan` and `apply` as usual. `--currency <code>` gives a website's currency when the site does not say it. |
+| `storefront inspect <url>` | Read a brand's website, read-only: title, languages, theme colour, icons, `og:image`, logo candidates, the colours its CSS uses most, the currencies and countries it sells in, the platform (Shopify, WooCommerce or unknown) and whether `robots.txt` lets `catalog pull` read its products. For a demo store that looks like the site: the agent picks `theme.color` and `theme.logo` (a URL, hot-linked) from it. Writes no file. |
 | `storefront init [--force]` | Custom storefronts only. Write a static storefront on [drop-in.js](https://commercelayer.github.io/drop-in.js/), no build step, wired to the sandbox's sales channel. `index.html`: the config block, the products grouped by top-level category with a navigation, a minicart that slides in on add to cart, the hosted cart and checkout, and a product page at `#/p/<sku>` built from `<template id="product-template">` and the `#products` data. `styles.css`: theme tokens first (`--paper`, `--ink`, `--muted`, `--line`, `--accent`, `--on-accent`, `--radius`, `--font`), which also restyle drop-in.js's prices, buttons and links, so a dark theme is a token change; then the drop-in.js overrides, then the page. `app.js`: the market switch, the category links and the product page. Product images are the store's, at `/img/<file>` (from `.commercelayer/img/`, uploaded by `apply`); products with no image show the tile the store host draws for them, at `/tiles/<sku>.svg`. The site's own files go in `storefront/assets/`. Refuses to overwrite without the flag; with it, rewrites the three files. |
 | `storefront publish` | Custom storefronts only. Upload the folder at `storefront.path` (default `storefront`) to the store host, where it replaces the hosted theme, leaving out dotfiles. `/img/` on the store host is always the store's images, so files under the folder's `img/` are not served: publish lists them as `not_served`. |
 | `storefront check` | Custom storefronts only. Read the published storefront at the preview URL and check it: drop-in.js and its config block wired to this store, the drop-in elements, an add to cart for every product and none for unknown skus, every image loading, alt text, stylesheets after drop-in.css, the theme tokens and drop-in overrides (no dead `::part` rules), title, `lang`, viewport, description, and whether the folder changed since publish. Prints `{ status, summary, checks: [{ id, kind, status: pass\|warn\|fail, message, fix, items }], render }` and exits 1 when a check fails; warnings do not fail. `kind` is `static` today; a later rendering phase adds `render` checks and fills `render`. |
@@ -146,14 +147,18 @@ A store has at most 6 languages, the base one included. Every country gets its o
 When the products live in a tool the human edits too, `store.yml` names it and `c11r catalog pull` copies it into `.commercelayer/products/catalog.csv`, sorted by sku: same source, same file. Pull applies the product rules above before it writes, with errors naming the row in the source, and reports the skus added, changed and removed. Removed products leave the store only through `apply --allow-delete`. Every source has the same columns: `sku`, `name`, `description`, `image_url`, `category`, and one price column per market currency, named by its ISO 4217 code in lowercase (`eur`, `usd`).
 ```yaml
 catalog:
-  source: sheets     # sheets | supabase | merchant
+  source: sheets     # sheets | supabase | merchant | website
   url: https://docs.google.com/spreadsheets/d/e/2PACX-.../pub?output=csv
   # table: products  # supabase only
+  # limit: 100       # website only: how many products (at most 500)
 ```
 - `sheets`: a Google Sheet, by its CSV link (File > Share > Publish to web > Comma-separated values) or by its own link when anyone with the link can view it. No key. Translations are `name_<ll>`, `description_<ll>` and `category_<ll>` columns, in sheets and Supabase tables alike.
 - `supabase`: a table (default `products`), read over the REST API with the project's anon key: `C11R_SUPABASE_URL=https://<project>.supabase.co` and `C11R_SUPABASE_KEY=<anon key>` in `.commercelayer/credentials`, or the environment. The key needs a row level security policy that allows select. Never use the `service_role` key. Columns `id`, `created_at` and `updated_at` are skipped.
 - `merchant`: a Google Merchant Center product feed URL, RSS 2.0 (XML, `g:` namespace) or TSV. `id` is the sku, `title` the name, `description`, `image_link` the image, `price` (`14.50 EUR`) the price in its currency, `product_type` the category. An id listed once per currency is one product with several prices. Other attributes, `sale_price` included, are ignored. Other languages: `catalog.translations: { it: <the Italian feed URL> }`, one feed per language, merged by `id` (`title`, `description`, `product_type`).
-Error codes: `no_catalog_source`, `invalid_catalog`, `missing_credentials`, `catalog_unreachable`, `catalog_unauthorized`, `catalog_not_found`, `catalog_not_csv`, `catalog_error`, `empty_catalog`, plus `invalid_product` and `missing_price`.
+- `website`: a brand's own website, for a demo store (the human's site, or a brand they are pitching to). `url` is the site, `limit` how many products (default 100, at most 500). Pull tries, in order, and the first that yields products wins: Shopify's public `products.json` (also `/collections/all/products.json`), WooCommerce's Store API (`/wp-json/wc/store/v1/products`), then the product pages its sitemap lists (`/products/`, `/product/`, `/p/`, `/shop/`, `/item/`) and their schema.org Product JSON-LD. One row per product: the first available variant's sku and price (or the handle in upper snake case), the title, the description as plain text (paragraphs kept, at most 2,000 characters), the first image, the product type or breadcrumb as category. Products without a price are left out, and a sku seen twice keeps the first. Prices are the site's, in the site's currency, never converted: the store's markets must sell in it (`currency_mismatch` otherwise), and when the site does not say it, `currency_unknown` asks for `--currency <code>`. Translations are not pulled yet; `source_site.languages` lists the site's languages. **Images and the logo stay on the brand's servers: `image_url` links them, nothing is copied.** Pull reads politely: `robots.txt` is honoured for every path (`robots_skipped` counts what it disallows), one request at a time and at most 2 a second, 15 s per request and 60 s in all (`budget_exhausted` when it stopped early, with what it read), as `c11r/<version> (+https://commercelayer.ai)`. It prints `strategy`, `source_site: { url, title, platform, product_count_seen, currency, languages }`, `pulled`, `variants_skipped`, `robots_skipped`, `no_price_skipped`, `duplicates_skipped`, `other_currency_skipped`, `tried: [{ strategy, result }]` and `requests`, so the agent can tell the human "I found 340 products, pulled 100".
+Error codes: `no_catalog_source`, `invalid_catalog`, `missing_credentials`, `catalog_unreachable`, `catalog_unauthorized`, `catalog_not_found`, `catalog_not_csv`, `catalog_error`, `empty_catalog`, plus `invalid_product` and `missing_price`; for a website `site_unreachable`, `no_products_found` (ask the human for a feed or a sheet instead), `currency_unknown` and `currency_mismatch`.
+
+`storefront inspect <url>` reads the same way, one page and up to 3 of its stylesheets, and writes nothing: `{ url, title, site_name, description, language, languages, theme_color, icons, og_image, logo_candidates: [{ url, source: json_ld|header_img|img|header_svg, alt, width, height }], colors: [{ hex, source: theme_color|css, count, neutral }], currencies_seen, countries_seen, platform, robots: { products_allowed, paths }, next }`. An inline SVG logo has `url: null`: there is nothing to link. Errors: `missing_argument`, `site_unreachable`, `site_error`, `robots_disallowed`.
 
 ### The website
 The theme renders the home page (logo, hero with name and tagline, products by top-level category), a page per category at `/categories/<slug>/...` and per product at `/products/<sku>`, with the market from `?market=<code>`, cart and checkout. Images written `img/<file>` (`theme.logo`, `theme.hero`, a product's `image_url`) are files in `.commercelayer/img/`, served at `<preview_url>/img/<file>`: paths of letters, digits, `.`, `_`, `-` and `/`, a lowercase `png`, `jpg`, `jpeg`, `webp`, `gif`, `avif` or `svg` extension, at most 5 MB each. `apply` uploads the new and changed ones (by SHA-256) before anything else, and the store host drops the ones nothing references any more.

package/dist/bin.js

@@ -5,7 +5,7 @@
 import * as cmd from './commands.js';
 import { CliError, print } from './output.js';
 import { LATEST_HINT, NPX } from './version.js';
-const VALUE_OPTIONS = new Set(['--name', '--sku', '--quantity', '--country', '--code', '--file', '--store', '--item', '--email', '--first-name', '--last-name', '--phone', '--street', '--city', '--postal-code', '--region', '--shipping', '--category', '--min-price', '--max-price', '--status', '--payment', '--fulfillment', '--limit', '--archived', '--amount', '--carrier', '--tracking', '--tracking-url', '--since', '--until', '--by', '--company', '--line-2', '--query', '--reference', '--notes', '--language', '--source', '--channel', '--label', '--unlabel', '--move']);
+const VALUE_OPTIONS = new Set(['--name', '--sku', '--quantity', '--country', '--code', '--file', '--store', '--item', '--email', '--first-name', '--last-name', '--phone', '--street', '--city', '--postal-code', '--region', '--shipping', '--category', '--min-price', '--max-price', '--status', '--payment', '--fulfillment', '--limit', '--archived', '--amount', '--carrier', '--tracking', '--tracking-url', '--since', '--until', '--by', '--company', '--line-2', '--query', '--reference', '--notes', '--language', '--source', '--channel', '--label', '--unlabel', '--move', '--currency']);
 const argv = process.argv.slice(2);
 const flags = new Set();
 const values = {};
@@ -54,7 +54,7 @@ async function run() {
             return cmd.claim();
         case 'catalog':
             if (sub === 'pull')
-                return cmd.catalogPull();
+                return cmd.catalogPull(values);
             throw new CliError('unknown_command', `There is no command "catalog ${sub ?? ''}".`, 'Use `c11r catalog pull`.');
         case 'storefront':
             if (sub === 'init')
@@ -63,7 +63,9 @@ async function run() {
                 return cmd.storefrontPublish();
             if (sub === 'check')
                 return cmd.storefrontCheckCmd();
-            throw new CliError('unknown_command', `There is no command "storefront ${sub ?? ''}".`, 'Use `c11r storefront init`, `storefront publish` or `storefront check`.');
+            if (sub === 'inspect')
+                return cmd.storefrontInspect(args[2]);
+            throw new CliError('unknown_command', `There is no command "storefront ${sub ?? ''}".`, 'Use `c11r storefront init`, `storefront publish`, `storefront check` or `storefront inspect <url>`.');
         case 'promotions':
             if (sub === 'list')
                 return cmd.promotionsList();

package/dist/catalog.js

@@ -1,6 +1,7 @@
 // c11r catalog pull: the products live in a tool the human edits too (a Supabase
-// table, a Google Sheet, a Google Merchant Center feed). Pull reads them and
-// writes .commercelayer/products/catalog.csv, so plan and apply work as always.
+// table, a Google Sheet, a Google Merchant Center feed), or on a brand's website (a demo
+// store). Pull reads them and writes .commercelayer/products/catalog.csv, so plan and apply
+// work as always.
 // Every source maps onto the same fixed schema: sku, name, description,
 // image_url, category and one price column per currency.
 import fs from 'node:fs';
@@ -11,28 +12,32 @@ import { CliError, fail } from './output.js';
 import { parseCsv, readProductFile, toCsv } from './products.js';
 import { ISO_639_1, setPlain, setTranslated, translatedColumn, TRANSLATED } from './languages.js';
 import { canonical, checkProducts, readStoreFile, storeLanguage } from './store.js';
-export const SOURCES = ['supabase', 'sheets', 'merchant'];
+import { siteOrigin } from './web.js';
+import { fromWebsite, WEBSITE_LIMIT, WEBSITE_MAX } from './website.js';
+export const SOURCES = ['supabase', 'sheets', 'merchant', 'website'];
 export const catalogFile = path.join(paths.products, 'catalog.csv');
 const FIELDS = ['sku', 'name', 'description', 'image_url', 'category'];
 /** Columns tools add on their own (Supabase's defaults): pull skips them. */
 const IGNORED = ['id', 'created_at', 'updated_at'];
 /** Supabase rows per request; pull pages until a page comes back empty. */
 const PAGE = 1000;
-const EXAMPLE = 'For example `catalog: { source: sheets, url: <the sheet\'s CSV link> }`. Sources are supabase, sheets and merchant; see the comments in store.yml.';
+const EXAMPLE = 'For example `catalog: { source: sheets, url: <the sheet\'s CSV link> }`. Sources are supabase, sheets, merchant and website; see the comments in store.yml.';
 function readConfig(doc) {
     const c = doc.catalog;
     if (!c || typeof c !== 'object')
         fail('no_catalog_source', 'store.yml has no catalog source.', `Add a \`catalog:\` entry to ${paths.store}. ${EXAMPLE}`);
     if (!SOURCES.includes(c.source))
-        fail('invalid_catalog', `catalog.source in store.yml is "${c.source ?? ''}".`, `Use supabase, sheets or merchant. ${EXAMPLE}`);
+        fail('invalid_catalog', `catalog.source in store.yml is "${c.source ?? ''}".`, `Use supabase, sheets, merchant or website. ${EXAMPLE}`);
     const table = String(c.table ?? 'products');
     if (!/^[A-Za-z_][A-Za-z0-9_]*$/.test(table))
         fail('invalid_catalog', `catalog.table in store.yml is "${table}".`, 'Use the name of the Supabase table, such as products.');
     const url = String(c.url ?? '');
     if (c.source !== 'supabase' && !/^https?:\/\//.test(url)) {
-        fail('invalid_catalog', url ? `catalog.url in store.yml is "${url}", not a URL.` : 'catalog.url is missing in store.yml.', c.source === 'sheets'
-            ? 'Set url to the Google Sheet\'s CSV link (File > Share > Publish to web > Comma-separated values), or to the sheet\'s own link if it is shared with anyone who has the link.'
-            : 'Set url to the Merchant Center product feed (RSS/XML or TSV), as Merchant Center fetches it.');
+        fail('invalid_catalog', url ? `catalog.url in store.yml is "${url}", not a URL.` : 'catalog.url is missing in store.yml.', c.source === 'website'
+            ? "Set url to the brand's website, such as https://brand.example."
+            : c.source === 'sheets'
+                ? 'Set url to the Google Sheet\'s CSV link (File > Share > Publish to web > Comma-separated values), or to the sheet\'s own link if it is shared with anyone who has the link.'
+                : 'Set url to the Merchant Center product feed (RSS/XML or TSV), as Merchant Center fetches it.');
     }
     const translations = {};
     if (c.translations !== undefined && c.translations !== null) {
@@ -48,7 +53,15 @@ function readConfig(doc) {
             translations[ll] = String(u);
         }
     }
-    return { source: c.source, url, table, translations };
+    let limit = WEBSITE_LIMIT;
+    if (c.limit !== undefined && c.limit !== null) {
+        if (c.source !== 'website')
+            fail('invalid_catalog', `catalog.limit is for a website; this source is ${c.source}.`, 'Pull reads every product of a sheet, a table or a feed: remove limit.');
+        limit = Number(c.limit);
+        if (!Number.isInteger(limit) || limit < 1 || limit > WEBSITE_MAX)
+            fail('invalid_catalog', `catalog.limit in store.yml is "${c.limit}".`, `Use a whole number from 1 to ${WEBSITE_MAX} (default ${WEBSITE_LIMIT}).`);
+    }
+    return { source: c.source, url: c.source === 'website' ? siteOrigin(url, 'catalog.url in store.yml') : url, table, translations, limit };
 }
 async function get(url, what, headers = {}) {
     try {
@@ -271,7 +284,7 @@ async function fromMerchant(c) {
     }
     return out;
 }
-export async function pull() {
+export async function pull(values = {}) {
     if (!fs.existsSync(paths.store))
         fail('no_store', 'There is no store in this repo.', 'Run `c11r init` in the repo root first.');
     let doc;
@@ -285,16 +298,27 @@ export async function pull() {
     const config = readConfig(doc);
     const markets = Array.isArray(doc.markets) ? doc.markets : [];
     const currencies = [...new Set(markets.map((m) => m?.currency).filter(Boolean).map(String))];
-    const sourced = config.source === 'supabase' ? await fromSupabase(config, currencies) : config.source === 'sheets' ? await fromSheet(config, currencies) : await fromMerchant(config);
+    if (values.currency !== undefined && config.source !== 'website')
+        fail('invalid_argument', `--currency is for a website catalog; this source is ${config.source}.`, 'A sheet, a table or a feed says its currencies in its columns: leave out --currency.');
+    let site = null;
+    let sourced;
+    if (config.source === 'website') {
+        const r = await fromWebsite(config, currencies, values.currency);
+        site = r.report;
+        sourced = r.products.map((product) => ({ product, source: `${site.source_site.url}, product ${product.sku}` }));
+    }
+    else
+        sourced = config.source === 'supabase' ? await fromSupabase(config, currencies) : config.source === 'sheets' ? await fromSheet(config, currencies) : await fromMerchant(config);
+    const file = catalogFile;
     if (sourced.length === 0) {
         const rls = config.source === 'supabase' ? ' If the table has rows, row level security hides them from the anon key: ask the human to add a policy that allows select.' : '';
         fail('empty_catalog', `The ${config.source} catalog has no products.`, `Ask the human to add products to it, then pull again.${rls}`);
     }
     const products = checkProducts(sourced, currencies, storeLanguage(doc.language));
     let before = new Map();
-    if (fs.existsSync(catalogFile)) {
+    if (fs.existsSync(file)) {
         try {
-            before = new Map(readProductFile(catalogFile).map((s) => [String(s.product.sku), s.product]));
+            before = new Map(readProductFile(file).map((s) => [String(s.product.sku), s.product]));
         }
         catch (e) {
             if (!(e instanceof CliError))
@@ -303,22 +327,40 @@ export async function pull() {
     }
     const after = new Map(products.map((p) => [p.sku, p]));
     const csv = toCsv(products);
-    const changed = !fs.existsSync(catalogFile) || fs.readFileSync(catalogFile, 'utf8') !== csv;
+    const changed = !fs.existsSync(file) || fs.readFileSync(file, 'utf8') !== csv;
     if (changed) {
         fs.mkdirSync(paths.products, { recursive: true });
-        fs.writeFileSync(catalogFile, csv);
+        fs.writeFileSync(file, csv);
     }
     const removed = [...before.keys()].filter((sku) => !after.has(sku));
     return {
         status: changed ? 'pulled' : 'up_to_date',
         source: config.source,
-        file: catalogFile,
+        ...(site ? { strategy: site.strategy } : {}),
+        file,
         products: products.length,
         added: [...after.keys()].filter((sku) => !before.has(sku)),
         changed: [...after.keys()].filter((sku) => before.has(sku) && canonical(before.get(sku)) !== canonical(after.get(sku))),
         removed,
-        next: changed
-            ? `Run c11r plan to see what apply will change in the store.${removed.length ? ' Removed products are deleted from the store only by c11r apply --allow-delete, once the human agrees.' : ''} ${catalogFile} is rewritten by every pull: change products in the ${config.source} source, not in the file.`
-            : `${catalogFile} already matches the ${config.source} source.`,
+        ...(site ? websiteOutput(site, products.length) : {}),
+        next: site
+            ? websiteNext(site, products.length, changed, removed.length, file)
+            : changed
+                ? `Run c11r plan to see what apply will change in the store.${removed.length ? ' Removed products are deleted from the store only by c11r apply --allow-delete, once the human agrees.' : ''} ${file} is rewritten by every pull: change products in the ${config.source} source, not in the file.`
+                : `${file} already matches the ${config.source} source.`,
     };
 }
+function websiteOutput(r, pulled) {
+    const { strategy: _, ...rest } = r;
+    return { ...rest, pulled };
+}
+function websiteNext(r, pulled, changed, removed, file) {
+    const seen = r.source_site.product_count_seen;
+    const found = seen !== null && seen > pulled ? `I found ${seen} products on the site and pulled ${pulled} (catalog.limit is ${r.limit})` : `I pulled ${pulled} products from the site`;
+    const left = [r.no_price_skipped && `${r.no_price_skipped} without a price`, r.duplicates_skipped && `${r.duplicates_skipped} with a sku already taken`, r.robots_skipped && `${r.robots_skipped} pages robots.txt disallows`, r.other_currency_skipped && `${r.other_currency_skipped} in another currency`].filter(Boolean);
+    return [
+        `Tell the human: "${found}, in ${r.source_site.currency}${r.variants_skipped ? `, one variant each (${r.variants_skipped} other variants left out)` : ''}." ${left.length ? `Left out: ${left.join(', ')}. ` : ''}${r.budget_exhausted ? 'The site was slow: pull stopped at 60 s, so this is what it read by then. ' : ''}`,
+        'Images and the logo stay on the brand\'s servers: image_url links them, nothing is copied. ',
+        changed ? `Run c11r plan to see what apply will change in the store.${removed ? ' Removed products are deleted from the store only by c11r apply --allow-delete, once the human agrees.' : ''} ${file} is rewritten by every pull: never edit it; products to add go in another file in ${paths.products}.` : `${file} already matches the site.`,
+    ].join('');
+}

package/dist/commands.js

@@ -9,6 +9,7 @@ import { diff, hashStore, languageWarnings, loadStore, storefrontMode, storefron
 import { storefrontFiles } from './storefront.js';
 import { imageNotes, planAssets, uploadAssets } from './assets.js';
 import { pull } from './catalog.js';
+import { inspect } from './inspect.js';
 import * as ops from './ops.js';
 import { checkOrderRequest, lineFlag, sourceFlag } from './orderRequest.js';
 import { readInboxCursor, seenKey, writeInboxCursor } from './inbox/cursor.js';
@@ -30,9 +31,10 @@ export const HELP = {
     init: 'Create a sandbox store for this repo, with no login: c11r init --name "<business name>" (the folder name when --name is absent). The name becomes the organization\'s name, and the org slug and store hostname derive from it. Writes .commercelayer/store.yml and .commercelayer/credentials, and a section in AGENTS.md and CLAUDE.md (created, or appended once) that points later agents in this repo to the skills: commit them. Reuses the store if store.yml already exists.',
     plan: 'Show what apply would create, update or delete: markets, products, images, and the store\'s name, language, policies and theme; missing_images lists references with no file in .commercelayer/img/ (apply refuses them), unused_images files nothing references, from .commercelayer/store.yml and the product files under .commercelayer/products/ (*.yml lists, *.csv rows, merged by sku). Products may carry other languages: name, description and category as { <ll>: text } maps (the base language, store.yml language:, default en, among them), or name_<ll>, description_<ll>, category_<ll> CSV columns; at most 6 languages. warnings lists a language no country of the store speaks.',
     apply: 'Make the store match .commercelayer/store.yml and .commercelayer/products/: markets, products, the storefront mode (storefront.mode: hosted, the default, or agent-only: UCP and the pages it needs, no catalog pages), the theme (its settings, badge included, and the images in .commercelayer/img/ that the theme and products reference, uploaded when new or changed), name and policies. The store\'s website is live at preview_url after apply. Refuses deletes unless --allow-delete is given. Product changes go through a Commerce Layer import: apply waits for it (progress as JSON lines on stderr) and prints one result; if it stops with apply_pending the import keeps going, so run `c11r status` to follow it. While Commerce Layer is busy or unavailable it keeps reading, with a retrying line. An apply still running is finished first. --full re-imports every product and price, after edits in the Commerce Layer dashboard.',
-    'catalog pull': 'Read the products from the catalog source in store.yml (catalog: { source: supabase | sheets | merchant, url, table }) and write them to .commercelayer/products/catalog.csv; then plan and apply as usual. Columns: sku, name, description, image_url, category, name_<ll>, description_<ll> and category_<ll> for other languages (name_it), and one price column per market currency, in lowercase (eur, usd). A Merchant feed adds other languages with catalog.translations: { it: <feed url> }, merged by id. Supabase reads C11R_SUPABASE_URL and C11R_SUPABASE_KEY (the anon key) from .commercelayer/credentials or the environment; a Google Sheet needs its CSV link or a link anyone can open; a Merchant Center feed is RSS/XML or TSV.',
+    'catalog pull': 'Read the products from the catalog source in store.yml (catalog: { source: supabase | sheets | merchant | website, url, table, limit }) and write them to .commercelayer/products/catalog.csv; then plan and apply as usual. Columns: sku, name, description, image_url, category, name_<ll>, description_<ll> and category_<ll> for other languages (name_it), and one price column per market currency, in lowercase (eur, usd). A Merchant feed adds other languages with catalog.translations: { it: <feed url> }, merged by id. Supabase reads C11R_SUPABASE_URL and C11R_SUPABASE_KEY (the anon key) from .commercelayer/credentials or the environment; a Google Sheet needs its CSV link or a link anyone can open; a Merchant Center feed is RSS/XML or TSV. source website reads a brand\'s site for a demo store (url: the site, limit: default 100, at most 500): Shopify products.json, then the WooCommerce Store API, then the sitemap\'s product pages and their JSON-LD, first that yields products wins (strategy). One row per product, first available variant (variants_skipped counts the others); prices in the site\'s currency, never converted: currency_mismatch when the markets sell in another, currency_unknown when the site does not say (ask the human, then --currency <code>). It honours robots.txt (robots_skipped), reads one request at a time, at most 2 a second, 60 s in all (budget_exhausted). Images stay on the brand\'s servers: image_url links them, nothing is copied. Prints strategy, source_site { url, title, platform, product_count_seen, currency, languages }, pulled and the skipped counts: tell the human how many products the site has and how many were pulled. no_products_found: ask the human for a feed or a sheet instead.',
     'storefront init': 'Custom storefronts only: the hosted theme needs none of this (set theme: in store.yml and apply). Write a static storefront on drop-in.js, no build step: storefront/index.html (the products by category, a minicart that opens on add to cart, the hosted cart and checkout, and a product page at #/p/<sku>), styles.css (theme tokens first) and app.js. Product images are the store\'s, at /img/ (.commercelayer/img/, uploaded by apply); the site\'s own files go in storefront/assets/. Refuses to overwrite unless --force is given.',
     'storefront check': 'Custom storefronts only. Check the published storefront at the preview URL: drop-in.js and its config block wired to this store, the drop-in elements, an add to cart for every product, every image loading, alt text, stylesheets after drop-in.css, the theme tokens and drop-in overrides, title, lang, viewport, description, and whether the folder changed since publish. Prints { status, summary, checks: [{ id, kind, status: pass|warn|fail, message, fix, items }] }; exits 1 when a check fails.',
+    'storefront inspect': "Read a brand's website for a demo store that looks like it, read-only, nothing written: c11r storefront inspect <url>. Returns { url, title, site_name, description, language, languages, theme_color, icons, og_image, logo_candidates: [{ url, source: json_ld|header_img|img|header_svg, alt, width, height }], colors: [{ hex, source: theme_color|css, count, neutral }], currencies_seen, countries_seen, platform: shopify|woocommerce|unknown, robots: { products_allowed, paths } }. Pick theme.color (the first colour that is not neutral, usually) and theme.logo (a logo candidate's url: hot-linked, never downloaded; an inline SVG has url null) and show the human. robots.products_allowed false: catalog pull cannot read the products, ask for a feed or a sheet. Same polite reading as catalog pull (robots.txt, 2 requests a second). Errors: site_unreachable, site_error, robots_disallowed.",
     'storefront publish': 'Custom storefronts only: upload the storefront folder (store.yml storefront.path, default storefront) to the preview URL, where it replaces the hosted theme.',
     'promotions list': 'The store\'s promotions, live in Commerce Layer (not in store files): id, name, status (active, scheduled, expired, disabled, used_up), rules, actions, uses, starts, ends; managed: false marks one made in the dashboard (its id is Commerce Layer\'s), which can be enabled, disabled or deleted but not updated. warnings flags more than 10 active promotion actions without a coupon (Commerce Layer applies 10).',
     'promotions show': 'One promotion by id, as list shows it: c11r promotions show <id>.',
@@ -242,8 +244,11 @@ export async function apply(flags) {
         apply: job && done.apply ? jobSummary(done.apply) : undefined,
     };
 }
-export async function catalogPull() {
-    return pull();
+export async function catalogPull(values) {
+    return pull(values);
+}
+export async function storefrontInspect(url) {
+    return inspect(url);
 }
 export async function storefrontInit(flags) {
     const api = createApi(requireCredentials());

package/dist/inspect.js

@@ -0,0 +1,230 @@
+// c11r storefront inspect <url>: what a brand's website says about itself, for a demo store that
+// looks like it: title, languages, the logo, the colours, the currencies and countries it sells
+// in, the platform, and whether robots.txt lets catalog pull read its products. Read-only: it
+// writes no file and downloads nothing to keep; the logo is used by its URL (hot-linked).
+import { fail } from './output.js';
+import { absoluteUrl, BudgetExceeded, hreflangs, isType, jsonLd, metas, oneLine, platformOf, robotsAllows, siteOrigin, tags, unreachable, Web } from './web.js';
+const MAX_STYLESHEETS = 3;
+const MAX_LOGOS = 5;
+const MAX_COLORS = 5;
+const hex6 = (h) => {
+    const x = h.replace('#', '').toLowerCase();
+    return `#${x.length === 3 ? [...x].map((c) => c + c).join('') : x}`;
+};
+const rgbHex = (r, g, b) => `#${[r, g, b].map((n) => Math.max(0, Math.min(255, Math.round(n))).toString(16).padStart(2, '0')).join('')}`;
+/** Greys, white and black: a page's text and background, rarely its brand colour. */
+const neutral = (hex) => {
+    const [r, g, b] = [1, 3, 5].map((i) => parseInt(hex.slice(i, i + 2), 16));
+    return Math.max(r, g, b) - Math.min(r, g, b) < 16;
+};
+/** Every colour written in CSS: #rgb, #rrggbb and rgb()/rgba(), counted. */
+export function cssColors(css, into = new Map()) {
+    const add = (h) => into.set(h, (into.get(h) ?? 0) + 1);
+    for (const m of css.matchAll(/#([0-9a-f]{6}|[0-9a-f]{3})(?![0-9a-z_-])/gi))
+        add(hex6(m[1]));
+    for (const m of css.matchAll(/rgba?\(\s*(\d{1,3})[\s,]+(\d{1,3})[\s,]+(\d{1,3})(?:\s*[,/]\s*([\d.]+%?))?\s*\)/gi)) {
+        // A transparent colour is an overlay, not a colour of the brand.
+        if (m[4] !== undefined && parseFloat(m[4]) === 0)
+            continue;
+        add(rgbHex(Number(m[1]), Number(m[2]), Number(m[3])));
+    }
+    return into;
+}
+const num = (v) => {
+    const n = Number(String(v ?? '').replace(/px$/, ''));
+    return Number.isFinite(n) && n > 0 ? n : undefined;
+};
+/** Where the header is: the <header> element, else the top of the body. */
+const headerOf = (html) => {
+    const h = html.match(/<header\b[\s\S]*?<\/header>/i)?.[0];
+    if (h)
+        return h;
+    const body = html.search(/<body\b/i);
+    return html.slice(Math.max(0, body), Math.max(0, body) + 20_000);
+};
+export function logoCandidates(html, base, siteTitle) {
+    const out = [];
+    for (const node of jsonLd(html)) {
+        if (!['Organization', 'Brand', 'OnlineStore', 'Store', 'Corporation', 'LocalBusiness', 'WebSite'].some((t) => isType(node, t)))
+            continue;
+        const l = Array.isArray(node.logo) ? node.logo[0] : node.logo;
+        const url = absoluteUrl(typeof l === 'string' ? l : (l?.url ?? l?.contentUrl), base);
+        if (url)
+            out.push({ url, source: 'json_ld', score: 4, ...(num(l?.width) ? { width: num(l.width) } : {}), ...(num(l?.height) ? { height: num(l.height) } : {}) });
+    }
+    const words = (siteTitle ?? '').toLowerCase();
+    const scan = (region, source, bonus) => {
+        for (const t of tags(region, 'img')) {
+            const a = t.attrs;
+            const src = a.src || a['data-src'] || (a.srcset ?? a['data-srcset'] ?? '').split(',')[0]?.trim().split(/\s+/)[0];
+            // Shopify themes leave {width} in lazy sources.
+            const url = absoluteUrl(src?.replace(/\{width\}/g, '600'), base);
+            if (!url)
+                continue;
+            const own = [a.class, a.id, a.alt, src].join(' ').toLowerCase();
+            // The element around it (a link or a wrapper named logo), not just anything near.
+            const parents = [...region.slice(Math.max(0, t.index - 600), t.index).matchAll(/<(?!img\b|\/)[a-z][^>]*>/gi)].slice(-2).map((m) => m[0]).join(' ').toLowerCase();
+            const alt = (a.alt ?? '').trim().toLowerCase();
+            const score = (own.includes('logo') ? 3 : 0) + (alt && words && (words.includes(alt) || alt.includes(words.split(/\s[|–—-]\s/)[0])) ? 2 : 0) + (parents.includes('logo') ? 1 : 0);
+            if (score === 0)
+                continue;
+            out.push({ url, source, score: score + bonus, ...(a.alt ? { alt: a.alt.trim() } : {}), ...(num(a.width) ? { width: num(a.width) } : {}), ...(num(a.height) ? { height: num(a.height) } : {}) });
+        }
+    };
+    const header = headerOf(html);
+    scan(header, 'header_img', 1);
+    scan(html, 'img', 0);
+    // An inline <svg> logo has no URL to link: say so, so the agent asks for a file or picks another.
+    for (const t of tags(header, 'svg')) {
+        const own = [t.attrs.class, t.attrs.id, t.attrs['aria-label']].join(' ').toLowerCase();
+        const titled = header.slice(t.index, t.index + 400).match(/<title>([^<]*)<\/title>/i)?.[1]?.toLowerCase();
+        if (!own.includes('logo') && !(titled && words && words.includes(titled.trim())))
+            continue;
+        out.push({ url: null, source: 'header_svg', score: 2, note: 'An inline SVG in the header: it has no URL to link. Use another candidate, or ask the human for the logo file (img/logo.svg).' });
+        break;
+    }
+    const seen = new Set();
+    return out
+        .sort((a, b) => b.score - a.score)
+        .filter((l) => {
+        const key = l.url ?? `svg:${l.source}`;
+        if (seen.has(key))
+            return false;
+        seen.add(key);
+        return true;
+    })
+        .slice(0, MAX_LOGOS)
+        .map(({ score: _, ...l }) => l);
+}
+const ISO_CURRENCY = /^[A-Z]{3}$/;
+export function currenciesSeen(html, meta) {
+    const found = new Set();
+    const add = (c) => typeof c === 'string' && ISO_CURRENCY.test(c.trim().toUpperCase()) && found.add(c.trim().toUpperCase());
+    add(meta['og:price:currency']);
+    add(meta['product:price:currency']);
+    for (const m of html.matchAll(/Shopify\.currency\s*=\s*\{[^}]*"active"\s*:\s*"([A-Za-z]{3})"/g))
+        add(m[1]);
+    for (const m of html.matchAll(/"priceCurrency"\s*:\s*"([A-Za-z]{3})"/g))
+        add(m[1]);
+    for (const m of html.matchAll(/"currency_code"\s*:\s*"([A-Za-z]{3})"/g))
+        add(m[1]);
+    for (const m of html.matchAll(/data-currency(?:-code)?\s*=\s*["']([A-Za-z]{3})["']/g))
+        add(m[1]);
+    // A currency selector: <select name="currency"> with ISO codes as values.
+    for (const s of html.matchAll(/<select\b[^>]*name\s*=\s*["']?currency[^>]*>([\s\S]*?)<\/select>/gi))
+        for (const o of tags(s[1], 'option'))
+            add(o.attrs.value);
+    return [...found].sort();
+}
+export function countriesSeen(html, base) {
+    const found = new Set();
+    const add = (c) => typeof c === 'string' && /^[A-Z]{2}$/.test(c.trim()) && found.add(c.trim());
+    for (const h of hreflangs(html, base))
+        if (h.region && /^[A-Z]{2}$/.test(h.region))
+            found.add(h.region);
+    for (const m of html.matchAll(/Shopify\.country\s*=\s*"([A-Z]{2})"/g))
+        add(m[1]);
+    // A country selector: Shopify's localization form, or a <select name="country...">.
+    for (const f of html.matchAll(/<form\b[^>]*localization[^>]*>([\s\S]*?)<\/form>/gi))
+        for (const m of f[1].matchAll(/data-value\s*=\s*["']([A-Z]{2})["']/g))
+            add(m[1]);
+    for (const s of html.matchAll(/<select\b[^>]*name\s*=\s*["']?country[^>]*>([\s\S]*?)<\/select>/gi))
+        for (const o of tags(s[1], 'option'))
+            add(o.attrs.value);
+    return [...found].sort();
+}
+/** The paths catalog pull reads, by strategy: whether robots.txt lets c11r read them. */
+const PRODUCT_PATHS = { shopify: ['/products.json', '/collections/all/products.json'], woocommerce: ['/wp-json/wc/store/v1/products'], structured_data: ['/sitemap.xml'] };
+export async function inspect(raw) {
+    if (!raw)
+        fail('missing_argument', 'storefront inspect needs the website address.', 'Run c11r storefront inspect https://brand.example.');
+    const origin = siteOrigin(raw, 'The address');
+    const withScheme = /^[a-z][a-z0-9+.-]*:\/\//i.test(raw) ? raw : `https://${raw}`;
+    const start = absoluteUrl(withScheme, origin) ?? `${origin}/`;
+    const web = new Web();
+    let page;
+    try {
+        page = await web.get(start);
+    }
+    catch (e) {
+        if (e instanceof BudgetExceeded)
+            fail('site_unreachable', `${start} did not answer in time.`, 'Try again later.');
+        unreachable(start, e);
+    }
+    if (page === null)
+        fail('robots_disallowed', `${start}'s robots.txt does not let c11r read that page.`, 'c11r does not go around robots.txt. Ask the human for the logo, colours and products instead (a feed or a sheet for catalog pull).');
+    if (page.status === 0)
+        fail('site_unreachable', `${start} did not answer within ${Math.round(web.timeout / 1000)} s.`, 'Check the address with the human (open it in a browser), and try again later.');
+    if (!page.ok)
+        fail('site_error', `${start} answered ${page.status}.`, 'Check the address with the human: open it in a browser.');
+    const html = page.text;
+    const base = page.url;
+    const finalOrigin = new URL(base).origin;
+    const meta = metas(html);
+    const title = oneLine(html.match(/<title\b[^>]*>([\s\S]*?)<\/title>/i)?.[1] ?? '') || null;
+    const siteName = meta['og:site_name'] || null;
+    const lang = html.match(/<html\b[^>]*\blang\s*=\s*["']?([A-Za-z]{2,3}(?:[-_][A-Za-z0-9]+)*)/)?.[1] ?? null;
+    const alternates = hreflangs(html, base);
+    const languages = [...new Set([lang?.slice(0, 2).toLowerCase(), ...alternates.map((h) => h.language)].filter((l) => !!l))];
+    const icons = tags(html, 'link')
+        .filter((t) => /\b(icon|apple-touch-icon|mask-icon)\b/i.test(t.attrs.rel ?? ''))
+        .map((t) => ({ url: absoluteUrl(t.attrs.href, base), rel: t.attrs.rel, ...(t.attrs.sizes ? { sizes: t.attrs.sizes } : {}), ...(t.attrs.type ? { type: t.attrs.type } : {}) }))
+        .filter((i) => i.url);
+    // Colours: theme-color first, then the CSS: inline <style>, style="" and the first linked stylesheets.
+    const counts = new Map();
+    for (const m of html.matchAll(/<style\b[^>]*>([\s\S]*?)<\/style>/gi))
+        cssColors(m[1], counts);
+    for (const m of html.matchAll(/\sstyle\s*=\s*"([^"]*)"/gi))
+        cssColors(m[1], counts);
+    const sheets = tags(html, 'link')
+        .filter((t) => /\bstylesheet\b/i.test(t.attrs.rel ?? '') && t.attrs.href)
+        .map((t) => absoluteUrl(t.attrs.href, base))
+        .filter((u) => !!u)
+        .slice(0, MAX_STYLESHEETS);
+    let stylesheetsRead = 0;
+    for (const url of sheets) {
+        try {
+            const css = await web.get(url, 'text/css,*/*;q=0.1');
+            if (css?.ok) {
+                cssColors(css.text, counts);
+                stylesheetsRead++;
+            }
+        }
+        catch (e) {
+            if (e instanceof BudgetExceeded)
+                break;
+        }
+    }
+    const themeColor = /^#([0-9a-f]{3}|[0-9a-f]{6})$/i.test(meta['theme-color'] ?? '') ? hex6(meta['theme-color']) : null;
+    const colors = [
+        ...(themeColor ? [{ hex: themeColor, source: 'theme_color', count: counts.get(themeColor) ?? 0, neutral: neutral(themeColor) }] : []),
+        ...[...counts.entries()]
+            .filter(([h]) => h !== themeColor)
+            .sort((a, b) => b[1] - a[1])
+            .map(([hex, count]) => ({ hex, source: 'css', count, neutral: neutral(hex) })),
+    ].slice(0, MAX_COLORS);
+    const robots = await web.robotsFor(finalOrigin).catch(() => ({ rules: [], sitemaps: [] }));
+    const paths = Object.fromEntries(Object.values(PRODUCT_PATHS).flat().map((p) => [p, robotsAllows(robots, p)]));
+    const platform = platformOf(html, page.headers);
+    const relevant = platform === 'unknown' ? Object.values(PRODUCT_PATHS).flat() : [...PRODUCT_PATHS[platform], ...PRODUCT_PATHS.structured_data];
+    return {
+        url: base,
+        title,
+        site_name: siteName,
+        description: meta.description || meta['og:description'] || null,
+        language: lang,
+        languages,
+        theme_color: themeColor,
+        icons,
+        og_image: absoluteUrl(meta['og:image:secure_url'] || meta['og:image'], base),
+        logo_candidates: logoCandidates(html, base, siteName ?? title),
+        colors,
+        currencies_seen: currenciesSeen(html, meta),
+        countries_seen: countriesSeen(html, base),
+        platform,
+        robots: { products_allowed: relevant.some((p) => paths[p]), paths },
+        stylesheets_read: stylesheetsRead,
+        requests: web.requests,
+        next: 'Pick theme.color from colors (the first one that is not neutral, usually) and theme.logo from logo_candidates (its url, hot-linked: nothing is copied), and show the human what you picked. Then catalog: { source: website, url: <this site> } in store.yml and c11r catalog pull.' + (relevant.some((p) => paths[p]) ? '' : ' robots.txt disallows the paths catalog pull reads: ask the human for a feed or a sheet instead.'),
+    };
+}

package/dist/store.js

@@ -59,6 +59,8 @@ catalog:
   #                    # C11R_SUPABASE_URL and C11R_SUPABASE_KEY in .commercelayer/credentials
   # source: merchant   # a Google Merchant Center feed, RSS (XML) or TSV: url is the feed;
   #                    # translations: { it: <the it feed url> } adds other languages, by id
+  # source: website    # a brand's website, for a demo store: url is the site; limit is how
+  #                    # many products (default 100, at most 500). Images stay on the site.
   # url: https://...
   # table: products
 

package/dist/web.js

@@ -0,0 +1,316 @@
+// Reading a brand's public website, politely: catalog pull (source website) and storefront
+// inspect. robots.txt is honoured for every path, one request at a time and at most 2 a second,
+// 15 s per request and 60 s in all, under a User-Agent that names c11r. Nothing is downloaded to
+// keep: pages are read, and images stay on the brand's servers (their URLs are hot-linked).
+import { fail } from './output.js';
+import { VERSION } from './version.js';
+export const WEB_USER_AGENT = `c11r/${VERSION} (+https://commercelayer.ai)`;
+const ROBOTS_TOKEN = 'c11r';
+/** Pages larger than this are not read: no store page or feed is this big. */
+const MAX_BYTES = 8 * 1024 * 1024;
+const num = (name, fallback) => {
+    const v = Number(process.env[name]);
+    return Number.isFinite(v) && v >= 0 ? v : fallback;
+};
+export function parseRobots(text) {
+    const groups = [];
+    const sitemaps = [];
+    let current = null;
+    let lastWasAgent = false;
+    for (const raw of text.split(/\r?\n/)) {
+        const line = raw.replace(/#.*$/, '').trim();
+        const m = line.match(/^([A-Za-z-]+)\s*:\s*(.*)$/);
+        if (!m)
+            continue;
+        const key = m[1].toLowerCase();
+        const value = m[2].trim();
+        if (key === 'sitemap') {
+            if (value)
+                sitemaps.push(value);
+            continue;
+        }
+        if (key === 'user-agent') {
+            if (!lastWasAgent || !current)
+                groups.push((current = { agents: [], rules: [] }));
+            current.agents.push(value.toLowerCase());
+            lastWasAgent = true;
+            continue;
+        }
+        lastWasAgent = false;
+        if (!current || (key !== 'allow' && key !== 'disallow'))
+            continue;
+        // "Disallow:" with nothing after it allows everything: no rule.
+        if (value)
+            current.rules.push({ allow: key === 'allow', pattern: value });
+    }
+    const mine = groups.filter((g) => g.agents.some((a) => a !== '*' && ROBOTS_TOKEN.startsWith(a)));
+    const chosen = mine.length ? mine : groups.filter((g) => g.agents.includes('*'));
+    return { rules: chosen.flatMap((g) => g.rules), sitemaps };
+}
+const patternRegex = (pattern) => {
+    const anchored = pattern.endsWith('$');
+    const body = (anchored ? pattern.slice(0, -1) : pattern)
+        .split('*')
+        .map((s) => s.replace(/[.+?^${}()|[\]\\]/g, '\\$&'))
+        .join('.*');
+    return new RegExp(`^${body}${anchored ? '$' : ''}`);
+};
+/** Whether robots.txt lets c11r read a path (with its query): the longest matching rule wins, allow on a tie. */
+export function robotsAllows(robots, pathAndQuery) {
+    if (robots.unavailable)
+        return false;
+    let best = null;
+    for (const r of robots.rules) {
+        if (!patternRegex(r.pattern).test(pathAndQuery))
+            continue;
+        const length = r.pattern.length;
+        if (!best || length > best.length || (length === best.length && r.allow))
+            best = { allow: r.allow, length };
+    }
+    return best ? best.allow : true;
+}
+export class BudgetExceeded extends Error {
+}
+/** The one way these commands read the web: robots.txt, pacing, timeouts and the total budget. */
+export class Web {
+    robots = new Map();
+    last = 0;
+    started = Date.now();
+    interval = num('C11R_WEB_INTERVAL_MS', 500);
+    timeout = num('C11R_WEB_TIMEOUT_MS', 15_000);
+    budget = num('C11R_WEB_BUDGET_MS', 60_000);
+    requests = 0;
+    robotsSkipped = 0;
+    budgetExhausted = false;
+    remaining() {
+        return this.budget - (Date.now() - this.started);
+    }
+    async raw(url, accept) {
+        if (this.remaining() <= 0) {
+            this.budgetExhausted = true;
+            throw new BudgetExceeded('budget');
+        }
+        const wait = this.last + this.interval - Date.now();
+        if (wait > 0)
+            await new Promise((r) => setTimeout(r, wait));
+        this.last = Date.now();
+        this.requests++;
+        const ms = Math.max(1, Math.min(this.timeout, this.remaining()));
+        try {
+            const res = await fetch(url, { headers: { 'User-Agent': WEB_USER_AGENT, Accept: accept }, redirect: 'follow', signal: AbortSignal.timeout(ms) });
+            const length = Number(res.headers.get('content-length') ?? 0);
+            const text = length > MAX_BYTES ? '' : await res.text();
+            this.last = Date.now();
+            return { url: res.url || url, status: res.status, ok: res.ok && length <= MAX_BYTES, type: res.headers.get('content-type') ?? '', text: text.length > MAX_BYTES ? '' : text, headers: res.headers };
+        }
+        catch (e) {
+            this.last = Date.now();
+            if (e.name === 'TimeoutError') {
+                if (this.remaining() <= 0) {
+                    this.budgetExhausted = true;
+                    throw new BudgetExceeded('budget');
+                }
+                return { url, status: 0, ok: false, type: '', text: '', headers: new Headers() };
+            }
+            throw e;
+        }
+    }
+    robotsFor(origin) {
+        let r = this.robots.get(origin);
+        if (!r) {
+            r = this.raw(`${origin}/robots.txt`, 'text/plain').then((p) => (p.ok ? parseRobots(p.text) : p.status >= 500 ? { rules: [], sitemaps: [], unavailable: true } : { rules: [], sitemaps: [] }), (e) => {
+                if (e instanceof BudgetExceeded)
+                    throw e;
+                return { rules: [], sitemaps: [] };
+            });
+            this.robots.set(origin, r);
+        }
+        return r;
+    }
+    async allowed(url) {
+        const u = new URL(url);
+        return robotsAllows(await this.robotsFor(u.origin), u.pathname + u.search);
+    }
+    /**
+     * GET a URL robots.txt allows: null when it does not (counted in robotsSkipped), the page
+     * otherwise, whatever its status. A redirect to a path robots.txt disallows is null too.
+     */
+    async get(url, accept = 'text/html,application/xhtml+xml,application/json;q=0.9,*/*;q=0.8') {
+        if (!(await this.allowed(url))) {
+            this.robotsSkipped++;
+            return null;
+        }
+        const page = await this.raw(url, accept);
+        if (page.url !== url && !(await this.allowed(page.url))) {
+            this.robotsSkipped++;
+            return null;
+        }
+        return page;
+    }
+    async json(url) {
+        const page = await this.get(url, 'application/json');
+        if (!page)
+            return null;
+        let body = null;
+        try {
+            body = page.ok ? JSON.parse(page.text) : null;
+        }
+        catch {
+            body = null;
+        }
+        return { page, body };
+    }
+}
+/** The site's origin from what the human gave: a bare host is https. */
+export function siteOrigin(raw, where) {
+    const withScheme = /^[a-z][a-z0-9+.-]*:\/\//i.test(raw) ? raw : `https://${raw}`;
+    let u;
+    try {
+        u = new URL(withScheme);
+    }
+    catch {
+        fail('invalid_argument', `${where} is "${raw}", not a website address.`, 'Use the site\'s address, such as https://brand.example.');
+    }
+    if (u.protocol !== 'https:' && u.protocol !== 'http:')
+        fail('invalid_argument', `${where} is "${raw}", not a website address.`, 'Use an http(s) address, such as https://brand.example.');
+    return u.origin;
+}
+/** The first page could not be read at all. */
+export function unreachable(url, e) {
+    const cause = e.cause;
+    fail('site_unreachable', `Could not reach ${url}: ${cause?.code ?? cause?.message ?? e.message}.`, 'Check the address with the human (open it in a browser), and that this machine is online.');
+}
+const NAMED = { amp: '&', lt: '<', gt: '>', quot: '"', apos: "'", nbsp: ' ', hellip: '…', ndash: '–', mdash: '—', rsquo: '’', lsquo: '‘', rdquo: '”', ldquo: '“', laquo: '«', raquo: '»', copy: '©', reg: '®', trade: '™', eacute: 'é', egrave: 'è', agrave: 'à', ograve: 'ò', ugrave: 'ù', igrave: 'ì', euro: '€', pound: '£', deg: '°', times: '×' };
+export function decodeEntities(s) {
+    return s.replace(/&(#x[0-9a-f]+|#\d+|[a-z]+);/gi, (m, e) => {
+        if (e[0] === '#') {
+            const code = e[1] === 'x' || e[1] === 'X' ? parseInt(e.slice(2), 16) : Number(e.slice(1));
+            return Number.isFinite(code) && code > 0 && code < 0x110000 ? String.fromCodePoint(code) : m;
+        }
+        return NAMED[e.toLowerCase()] ?? m;
+    });
+}
+/** HTML as plain text, paragraphs kept as blank lines. */
+export function htmlToText(html) {
+    return decodeEntities(html
+        .replace(/<(script|style|head|template|noscript)\b[^>]*>[\s\S]*?<\/\1>/gi, '')
+        .replace(/<br\s*\/?>/gi, '\n')
+        .replace(/<li\b[^>]*>/gi, '\n- ')
+        .replace(/<\/li>/gi, '')
+        .replace(/<\/(p|div|tr|ul|ol|h[1-6]|table|section|blockquote)>/gi, '\n\n')
+        .replace(/<[^>]+>/g, ''))
+        .replace(/ /g, ' ')
+        .replace(/[ \t]+/g, ' ')
+        .replace(/ *\n */g, '\n')
+        .replace(/\n{3,}/g, '\n\n')
+        .trim();
+}
+/** Text on one line, trimmed: names. */
+export const oneLine = (s) => decodeEntities(String(s ?? '').replace(/<[^>]+>/g, '')).replace(/\s+/g, ' ').trim();
+/** At most max characters, cut at a word with an ellipsis. */
+export function clip(s, max) {
+    if (s.length <= max)
+        return s;
+    const cut = s.slice(0, max - 1);
+    const at = cut.search(/\s\S*$/);
+    return (at > max * 0.6 ? cut.slice(0, at) : cut).trimEnd() + '…';
+}
+const TRACKING = /^(utm_[a-z_]+|fbclid|gclid|gclsrc|dclid|gbraid|wbraid|msclkid|mc_cid|mc_eid|_ga|_gl|igshid|yclid|_hsenc|_hsmi|mkt_tok|ref|ref_src|srsltid)$/i;
+/** An absolute http(s) URL from a link on a page, tracking parameters removed; null for anything else. */
+export function absoluteUrl(href, base) {
+    if (typeof href !== 'string' || !href.trim() || /^(data|javascript|blob):/i.test(href.trim()))
+        return null;
+    let u;
+    try {
+        u = new URL(decodeEntities(href.trim()), base);
+    }
+    catch {
+        return null;
+    }
+    if (u.protocol !== 'https:' && u.protocol !== 'http:')
+        return null;
+    for (const k of [...u.searchParams.keys()])
+        if (TRACKING.test(k))
+            u.searchParams.delete(k);
+    u.hash = '';
+    return u.toString();
+}
+/** A sku from a handle or slug: upper case, words joined by underscores. */
+export const skuFromHandle = (handle) => handle
+    .normalize('NFKD')
+    .replace(/[̀-ͯ]/g, '')
+    .toUpperCase()
+    .replace(/[^A-Z0-9]+/g, '_')
+    .replace(/^_+|_+$/g, '');
+/** The attributes of one tag, names in lower case. */
+export function attributes(tag) {
+    const out = {};
+    const inner = tag.replace(/^<[\w:-]+/, '').replace(/\/?>$/, '');
+    for (const m of inner.matchAll(/([^\s=/>"']+)(?:\s*=\s*(?:"([^"]*)"|'([^']*)'|([^\s>]+)))?/g)) {
+        const k = m[1].toLowerCase();
+        if (!(k in out))
+            out[k] = decodeEntities(m[2] ?? m[3] ?? m[4] ?? '');
+    }
+    return out;
+}
+/** Every opening tag of a name, with its attributes and where it starts. */
+export function tags(html, name) {
+    return [...html.matchAll(new RegExp(`<${name}\\b[^>]*>`, 'gi'))].map((m) => ({ attrs: attributes(m[0]), index: m.index, raw: m[0] }));
+}
+/** The JSON-LD blocks of a page, flattened: arrays and @graph opened, unparsable blocks left out. */
+export function jsonLd(html) {
+    const out = [];
+    const walk = (v) => {
+        if (Array.isArray(v))
+            v.forEach(walk);
+        else if (v && typeof v === 'object') {
+            const o = v;
+            out.push(o);
+            if (o['@graph'])
+                walk(o['@graph']);
+        }
+    };
+    for (const m of html.matchAll(/<script\b[^>]*type\s*=\s*["']?application\/ld\+json["']?[^>]*>([\s\S]*?)<\/script>/gi)) {
+        try {
+            walk(JSON.parse(m[1].trim().replace(/^<!--|-->$/g, '')));
+        }
+        catch {
+            // A broken block on a page is common: the others still count.
+        }
+    }
+    return out;
+}
+/** Whether a JSON-LD node is of a type: "@type" is a string or a list. */
+export const isType = (node, type) => {
+    const t = node?.['@type'];
+    const has = (x) => typeof x === 'string' && (x === type || x.endsWith(`/${type}`) || x.endsWith(`:${type}`));
+    return Array.isArray(t) ? t.some(has) : has(t);
+};
+/** <meta name|property=...> content, by name in lower case; the first one wins. */
+export function metas(html) {
+    const out = {};
+    for (const { attrs } of tags(html, 'meta')) {
+        const key = (attrs.property ?? attrs.name ?? attrs.itemprop ?? '').toLowerCase();
+        if (key && attrs.content !== undefined && !(key in out))
+            out[key] = attrs.content.trim();
+    }
+    return out;
+}
+/** <link rel=alternate hreflang=...>: language, region, URL. */
+export function hreflangs(html, base) {
+    return tags(html, 'link')
+        .filter((t) => /\balternate\b/i.test(t.attrs.rel ?? '') && t.attrs.hreflang)
+        .map((t) => {
+        const h = t.attrs.hreflang.trim();
+        const m = h.match(/^([a-z]{2,3})(?:[-_]([a-z]{4}))?(?:[-_]([a-z]{2}|\d{3}))?$/i);
+        return { hreflang: h, language: m ? m[1].toLowerCase() : null, region: m?.[3] ? m[3].toUpperCase() : null, url: absoluteUrl(t.attrs.href, base) };
+    });
+}
+/** The platform a page runs on, from what Shopify and WooCommerce always leave in it. */
+export function platformOf(html, headers) {
+    if (headers?.get('x-shopid') || headers?.get('x-shopify-stage') || /cdn\.shopify\.com|Shopify\.shop\s*=|shopify-section|window\.Shopify\b/.test(html))
+        return 'shopify';
+    if (/wp-content\/plugins\/woocommerce|\bwoocommerce\b|wc-block-|wc\/store\/v1/i.test(html))
+        return 'woocommerce';
+    return 'unknown';
+}

package/dist/website.js

@@ -0,0 +1,390 @@
+// catalog pull, source website: the products of a brand's own site, for a demo store. Three ways
+// in, tried in order, and the first that yields products wins: Shopify's public products.json,
+// WooCommerce's Store API, then the sitemap's product pages and their JSON-LD (schema.org
+// Product). Images are hot-linked, never downloaded: they stay on the brand's servers.
+import { fail } from './output.js';
+import { absoluteUrl, BudgetExceeded, clip, htmlToText, hreflangs, isType, jsonLd, oneLine, platformOf, skuFromHandle, unreachable, Web } from './web.js';
+export const WEBSITE_LIMIT = 100;
+export const WEBSITE_MAX = 500;
+const DESCRIPTION_MAX = 2000;
+const SHOPIFY_PAGE = 250;
+const WOO_PAGE = 100;
+/** Child sitemaps read from a sitemap index, product ones first. */
+const MAX_SITEMAPS = 10;
+const PRODUCT_PATH = /\/(products?|p|shop|item)\/[^/?#]+/i;
+/** What a strategy gathered, and what it left out. */
+class Gathered {
+    limit;
+    products = [];
+    skus = new Set();
+    variantsSkipped = 0;
+    noPriceSkipped = 0;
+    duplicatesSkipped = 0;
+    seen = null;
+    currency = null;
+    constructor(limit) {
+        this.limit = limit;
+    }
+    full() {
+        return this.products.length >= this.limit;
+    }
+    add(f) {
+        if (this.full() || !f.name)
+            return;
+        if (f.price === null || !Number.isFinite(f.price) || f.price < 0) {
+            this.noPriceSkipped++;
+            return;
+        }
+        if (!f.sku || this.skus.has(f.sku)) {
+            this.duplicatesSkipped++;
+            return;
+        }
+        this.skus.add(f.sku);
+        this.products.push(f);
+    }
+}
+const text = (html) => {
+    const t = typeof html === 'string' ? clip(htmlToText(html), DESCRIPTION_MAX) : '';
+    return t || undefined;
+};
+/** A category path from site names: "Coffee > Beans"; empty levels and stray > dropped. */
+const category = (...levels) => {
+    const parts = levels
+        .flatMap((l) => String(l ?? '').split('>'))
+        .map((l) => oneLine(l))
+        .filter(Boolean);
+    return parts.length ? parts.join(' > ') : undefined;
+};
+const currencyCode = (v) => (typeof v === 'string' && /^[A-Za-z]{3}$/.test(v.trim()) ? v.trim().toUpperCase() : null);
+async function shopify(web, origin, g) {
+    let base = `${origin}/products.json`;
+    let collection = null;
+    let first = await web.json(`${base}?limit=${SHOPIFY_PAGE}&page=1`);
+    if (!Array.isArray(first?.body?.products)) {
+        const fallback = await web.json(`${origin}/collections/all/products.json?limit=${SHOPIFY_PAGE}&page=1`);
+        if (Array.isArray(fallback?.body?.products)) {
+            base = `${origin}/collections/all/products.json`;
+            collection = 'all';
+            first = fallback;
+        }
+        else
+            return { result: first === null && fallback === null ? 'disallowed' : 'not_found' };
+    }
+    const meta = await web.json(`${origin}/meta.json`).catch((e) => {
+        if (e instanceof BudgetExceeded)
+            throw e;
+        return null;
+    });
+    g.currency = currencyCode(meta?.body?.currency);
+    if (typeof meta?.body?.published_products_count === 'number')
+        g.seen = meta.body.published_products_count;
+    let listed = 0;
+    let page = 1;
+    let body = first.body;
+    for (;;) {
+        const products = body.products;
+        listed += products.length;
+        for (const p of products) {
+            if (g.full())
+                break;
+            const variants = Array.isArray(p.variants) ? p.variants : [];
+            const v = variants.find((x) => x?.available !== false) ?? variants[0];
+            if (variants.length > 1)
+                g.variantsSkipped += variants.length - 1;
+            const tags = Array.isArray(p.tags) ? p.tags : typeof p.tags === 'string' ? p.tags.split(',') : [];
+            const tag = tags.map((t) => t.trim()).find((t) => t && !t.startsWith('_') && !t.includes(':'));
+            g.add({
+                sku: String(v?.sku ?? '').trim() || skuFromHandle(String(p.handle ?? '')),
+                name: oneLine(p.title),
+                description: text(p.body_html),
+                image_url: absoluteUrl(p.images?.[0]?.src, origin) ?? undefined,
+                category: category(p.product_type) ?? category(tag) ?? (collection && collection !== 'all' ? category(collection) : undefined),
+                price: v && v.price !== undefined && v.price !== null && v.price !== '' ? Number(v.price) : null,
+                currency: currencyCode(v?.price_currency ?? v?.price?.currency_code),
+            });
+        }
+        if (g.full() || products.length < SHOPIFY_PAGE)
+            break;
+        page++;
+        const next = await web.json(`${base}?limit=${SHOPIFY_PAGE}&page=${page}`);
+        if (!Array.isArray(next?.body?.products) || next.body.products.length === 0)
+            break;
+        body = next.body;
+    }
+    g.seen ??= listed;
+    g.currency ??= g.products.find((p) => p.currency)?.currency ?? null;
+    return g.products.length ? { result: 'products', gathered: g } : { result: 'no_products' };
+}
+async function woocommerce(web, origin, g) {
+    const url = (page) => `${origin}/wp-json/wc/store/v1/products?per_page=${WOO_PAGE}&page=${page}`;
+    let r = await web.json(url(1));
+    if (r === null)
+        return { result: 'disallowed' };
+    if (!Array.isArray(r.body))
+        return { result: 'not_found' };
+    const total = Number(r.page.headers.get('x-wp-total'));
+    if (Number.isFinite(total) && total > 0)
+        g.seen = total;
+    let listed = 0;
+    for (let page = 1;;) {
+        const products = r.body;
+        listed += products.length;
+        for (const p of products) {
+            if (g.full())
+                break;
+            const prices = p.prices ?? {};
+            const minor = Number(prices.currency_minor_unit ?? 2);
+            const raw = prices.price;
+            const price = raw !== undefined && raw !== null && raw !== '' && Number.isFinite(Number(raw)) ? Number(raw) / 10 ** (Number.isFinite(minor) ? minor : 2) : null;
+            if (Array.isArray(p.variations) && p.variations.length > 1)
+                g.variantsSkipped += p.variations.length - 1;
+            g.add({
+                sku: String(p.sku ?? '').trim() || skuFromHandle(String(p.slug ?? p.id ?? '')),
+                name: oneLine(p.name),
+                description: text(p.description) ?? text(p.short_description),
+                image_url: absoluteUrl(p.images?.[0]?.src, origin) ?? undefined,
+                category: category(p.categories?.[0]?.name),
+                price,
+                currency: currencyCode(prices.currency_code),
+            });
+        }
+        if (g.full() || products.length < WOO_PAGE)
+            break;
+        page++;
+        const next = await web.json(url(page));
+        if (!Array.isArray(next?.body) || next.body.length === 0)
+            break;
+        r = next;
+    }
+    g.seen ??= listed;
+    g.currency = g.products.find((p) => p.currency)?.currency ?? null;
+    return g.products.length ? { result: 'products', gathered: g } : { result: 'no_products' };
+}
+const locs = (xml) => [...xml.matchAll(/<loc>\s*(?:<!\[CDATA\[)?\s*([^<\]]+?)\s*(?:\]\]>)?\s*<\/loc>/gi)].map((m) => m[1].replace(/&amp;/g, '&'));
+/** The product pages a site's sitemaps list: robots.txt's Sitemap lines, else /sitemap.xml; an index is opened. */
+async function productUrls(web, origin) {
+    const robots = await web.robotsFor(origin);
+    const roots = robots.sitemaps.length ? robots.sitemaps.slice(0, 3) : [`${origin}/sitemap.xml`];
+    const urls = new Set();
+    let disallowed = true;
+    const queue = [...roots];
+    let read = 0;
+    while (queue.length && read < MAX_SITEMAPS) {
+        const url = queue.shift();
+        if (/\.gz($|\?)/i.test(url))
+            continue;
+        read++;
+        const page = await web.get(url, 'application/xml,text/xml;q=0.9,*/*;q=0.5');
+        if (page === null)
+            continue;
+        disallowed = false;
+        if (!page.ok)
+            continue;
+        if (/<sitemapindex\b/i.test(page.text)) {
+            // Product sitemaps first (Shopify's sitemap_products_1.xml, WordPress's product-sitemap.xml).
+            const children = locs(page.text).sort((a, b) => Number(/product/i.test(b)) - Number(/product/i.test(a)));
+            queue.push(...children);
+            continue;
+        }
+        for (const loc of locs(page.text)) {
+            const u = absoluteUrl(loc, origin);
+            if (u && new URL(u).origin === origin && PRODUCT_PATH.test(new URL(u).pathname))
+                urls.add(u);
+        }
+    }
+    return { urls: [...urls], disallowed: disallowed && read > 0 };
+}
+const first = (v) => (Array.isArray(v) ? v[0] : v);
+const imageOf = (v, base) => {
+    const i = first(v);
+    const url = typeof i === 'string' ? i : (i?.url ?? i?.contentUrl);
+    return absoluteUrl(url, base) ?? undefined;
+};
+/** "1.234,50", "1,234.50", "14,50", "14.50": the last separator before 1 or 2 digits is the decimal one. */
+export function amount(raw) {
+    const s = raw.replace(/[^\d.,-]/g, '');
+    const m = s.match(/^(-?[\d.,]*?)[.,](\d{1,2})$/);
+    return m ? Number(`${m[1].replace(/[.,]/g, '') || '0'}.${m[2]}`) : Number(s.replace(/[.,]/g, '') || NaN);
+}
+/** The price of a schema.org offer: Offer, AggregateOffer (lowPrice), or a priceSpecification. */
+function offerPrice(offers) {
+    const list = (Array.isArray(offers) ? offers : offers ? [offers] : []);
+    for (const o of list) {
+        const spec = first(o?.priceSpecification);
+        const raw = o?.price ?? o?.lowPrice ?? spec?.price;
+        const n = typeof raw === 'number' ? raw : amount(String(raw ?? ''));
+        if (raw !== undefined && raw !== null && raw !== '' && Number.isFinite(n))
+            return { price: n, currency: currencyCode(o?.priceCurrency ?? spec?.priceCurrency) };
+        if (o?.offers) {
+            const inner = offerPrice(o.offers);
+            if (inner.price !== null)
+                return inner;
+        }
+    }
+    return { price: null, currency: null };
+}
+/** A product page's schema.org Product (a ProductGroup's first variant), with the breadcrumb trail as category. */
+export function productFromPage(html, url) {
+    const nodes = jsonLd(html);
+    const group = nodes.find((n) => isType(n, 'ProductGroup'));
+    const product = nodes.find((n) => isType(n, 'Product')) ?? group;
+    if (!product)
+        return null;
+    const variant = group ? first(group.hasVariant) : undefined;
+    const offers = offerPrice(product.offers ?? variant?.offers);
+    const name = oneLine(product.name ?? group?.name);
+    const crumbs = nodes.find((n) => isType(n, 'BreadcrumbList'));
+    let trail = [];
+    if (Array.isArray(crumbs?.itemListElement)) {
+        trail = [...crumbs.itemListElement]
+            .sort((a, b) => Number(a?.position ?? 0) - Number(b?.position ?? 0))
+            .map((e) => oneLine(e?.name ?? e?.item?.name))
+            .filter(Boolean);
+        if (trail.length && /^(home|homepage|start|inicio|accueil|startseite)$/i.test(trail[0]))
+            trail.shift();
+        if (trail.length && trail[trail.length - 1] === name)
+            trail.pop();
+    }
+    const slug = new URL(url).pathname.split('/').filter(Boolean).pop() ?? '';
+    return {
+        sku: String(product.sku ?? variant?.sku ?? product.productID ?? product.mpn ?? '').trim() || skuFromHandle(decodeURIComponent(slug).replace(/\.html?$/i, '')),
+        name,
+        description: text(product.description),
+        image_url: imageOf(product.image ?? variant?.image, url),
+        category: category(...trail) ?? (typeof product.category === 'string' ? category(product.category) : undefined),
+        price: offers.price,
+        currency: offers.currency,
+    };
+}
+async function structuredData(web, origin, g) {
+    const { urls, disallowed } = await productUrls(web, origin);
+    if (!urls.length)
+        return { result: disallowed ? 'disallowed' : 'not_found' };
+    g.seen = urls.length;
+    const currencies = new Map();
+    for (const url of urls) {
+        if (g.full())
+            break;
+        let page;
+        try {
+            page = await web.get(url);
+        }
+        catch (e) {
+            if (e instanceof BudgetExceeded)
+                break;
+            continue;
+        }
+        if (!page?.ok)
+            continue;
+        const found = productFromPage(page.text, page.url);
+        if (!found)
+            continue;
+        if (found.currency)
+            currencies.set(found.currency, (currencies.get(found.currency) ?? 0) + 1);
+        g.add(found);
+    }
+    g.currency = [...currencies.entries()].sort((a, b) => b[1] - a[1])[0]?.[0] ?? null;
+    if (g.products.length)
+        return { result: 'products', gathered: g };
+    return { result: web.budgetExhausted ? 'budget_exhausted' : 'no_products' };
+}
+const STRATEGIES = [
+    ['shopify', shopify],
+    ['woocommerce', woocommerce],
+    ['structured_data', structuredData],
+];
+/** The site's products as catalog products in its currency, and how they were found. */
+export async function fromWebsite(c, marketCurrencies, currencyFlag) {
+    const web = new Web();
+    let origin = c.url;
+    // The home page: the site's real origin (after a redirect to www), its title, platform and languages.
+    let home = null;
+    try {
+        home = await web.get(`${origin}/`);
+    }
+    catch (e) {
+        if (e instanceof BudgetExceeded)
+            throw e;
+        unreachable(origin, e);
+    }
+    if (home && home.status === 0)
+        fail('site_unreachable', `${origin} did not answer within ${Math.round(web.timeout / 1000)} s.`, 'Check the address with the human (open it in a browser), and try again later.');
+    if (home?.ok)
+        origin = new URL(home.url).origin;
+    const html = home?.ok ? home.text : '';
+    const title = oneLine(html.match(/<title\b[^>]*>([\s\S]*?)<\/title>/i)?.[1] ?? '') || null;
+    const langs = [html.match(/<html\b[^>]*\blang\s*=\s*["']?([a-zA-Z]{2,3})/)?.[1]?.toLowerCase(), ...hreflangs(html, origin).map((h) => h.language)].filter((l) => !!l);
+    const tried = [];
+    let won = null;
+    for (const [strategy, run] of STRATEGIES) {
+        const g = new Gathered(c.limit);
+        let outcome;
+        try {
+            outcome = await run(web, origin, g);
+        }
+        catch (e) {
+            if (!(e instanceof BudgetExceeded))
+                throw e;
+            outcome = g.products.length ? { result: 'products', gathered: g } : { result: 'budget_exhausted' };
+        }
+        tried.push({ strategy, result: outcome.result });
+        if (outcome.result === 'products') {
+            won = { strategy, gathered: outcome.gathered };
+            break;
+        }
+        if (web.budgetExhausted)
+            break;
+    }
+    if (!won) {
+        const all = tried.length === STRATEGIES.length && tried.every((t) => t.result === 'disallowed');
+        const some = tried.filter((t) => t.result === 'disallowed').map((t) => t.strategy);
+        fail('no_products_found', `Found no products on ${origin}: ${tried.map((t) => `${t.strategy} ${t.result.replace('_', ' ')}`).join(', ')}.`, `${all ? "The site's robots.txt does not let c11r read its products, and c11r does not go around it. " : some.length ? `robots.txt disallows ${some.join(' and ')}. ` : ''}${web.budgetExhausted ? 'The site answered too slowly to finish in 60 s. ' : ''}Ask the human for a product feed or a sheet instead (catalog source merchant or sheets, then c11r catalog pull).`);
+    }
+    const g = won.gathered;
+    const flag = currencyFlag?.trim().toUpperCase();
+    if (flag !== undefined && !/^[A-Z]{3}$/.test(flag))
+        fail('invalid_argument', `--currency is "${currencyFlag}".`, 'Use the ISO 4217 code of the prices on the site, such as EUR.');
+    if (flag && g.currency && flag !== g.currency)
+        fail('currency_mismatch', `The site says its prices are in ${g.currency}, not ${flag}.`, `Leave out --currency: the site's own currency is ${g.currency}.`);
+    const currency = g.currency ?? flag;
+    if (!currency)
+        fail('currency_unknown', `${origin} does not say which currency its prices are in.`, `Ask the human, then run c11r catalog pull --currency <code> (such as EUR). The prices are taken as they are on the site, never converted.`);
+    const others = marketCurrencies.filter((m) => m !== currency);
+    if (marketCurrencies.length && others.length) {
+        fail('currency_mismatch', `The site's prices are in ${currency}; the store's markets sell in ${marketCurrencies.join(', ')}.`, marketCurrencies.includes(currency)
+            ? `A website has one currency: the products would have no ${others.join(', ')} price. Keep only ${currency} markets in store.yml for the demo (ask the human), or add those prices from another source.`
+            : `c11r never converts prices. Ask the human whether the demo store sells in ${currency}: if so, set the market's currency to ${currency} in store.yml, then pull again.`);
+    }
+    let otherCurrency = 0;
+    const products = [];
+    for (const f of g.products) {
+        if (f.currency && f.currency !== currency) {
+            otherCurrency++;
+            continue;
+        }
+        const p = { sku: f.sku, name: f.name, prices: { [currency]: Math.round(f.price * 100) / 100 } };
+        if (f.description)
+            p.description = f.description;
+        if (f.image_url)
+            p.image_url = f.image_url;
+        if (f.category)
+            p.category = f.category;
+        products.push(p);
+    }
+    return {
+        products,
+        report: {
+            strategy: won.strategy,
+            source_site: { url: origin, title, platform: won.strategy === 'structured_data' ? platformOf(html, home?.headers) : won.strategy, product_count_seen: g.seen, currency, languages: [...new Set(langs)] },
+            limit: c.limit,
+            variants_skipped: g.variantsSkipped,
+            robots_skipped: web.robotsSkipped,
+            no_price_skipped: g.noPriceSkipped,
+            duplicates_skipped: g.duplicatesSkipped,
+            other_currency_skipped: otherCurrency,
+            tried,
+            requests: web.requests,
+            budget_exhausted: web.budgetExhausted,
+        },
+    };
+}

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@commercelayer/agent-cli",
-  "version": "0.7.0",
+  "version": "0.8.0",
   "description": "c11r: the Commerce Layer CLI for coding agents. Builds a sandbox store that people and shopping agents can buy from.",
   "homepage": "https://commercelayer.ai",
   "keywords": [