sbuilder-mcp 0.16.1 → 0.17.1

package/CHANGELOG.md

@@ -6,6 +6,23 @@ All notable changes to this project are documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.17.1] - 2026-09-10
+
+### Fixed
+- sb_add and sb_import no longer store a nested node's children twice: a patch batch that gets applied more than once (as every write already is, to validate it before it lands) mutated itself on the first pass by carrying an added node by reference, so the second pass re-inserted its children into a node that already held them.
+- sb_import and sb_import_site now cap a flattened row at 12 columns and treat a grid container as wrapping rather than single-line, so a page whose content wrapper is a CSS grid (a documentation site, for example) no longer imports as one row of hundreds of slivered columns.
+- sb_import and sb_import_site now capture a `<pre>`/`<code>` block as one node instead of one node per syntax-highlighting `<span>`, so a code sample no longer arrives broken into dozens of single-token fragments.
+
+## [0.17.0] - 2026-09-10
+
+### Added
+- sb_import_site reads a whole site from one URL — the publisher's own sitemap first, a bounded link crawl only when there is none — and creates and fills a draft page here for every page it finds, replacing the by-hand loop of one sb_import call per page.
+- sb_import_site reports every repeated path prefix (such as `/products/{slug}`) as one bound template before creating anything, since importing those URLs as static pages would produce a catalogue where nothing is buyable.
+- sb_import_site previews where each page will land, including which URL merges into the site's existing home page and which slug is already taken, and skips a page whose slug collides instead of letting the platform silently rename it.
+
+### Fixed
+- sb_import and sb_import_site now insert imported content into the middle band, before the first global footer, instead of appending it to the end of the page, since appending broke the platform's band-order rule on every page carrying a global footer.
+
 ## [0.16.1] - 2026-09-10
 
 ### Fixed

package/CHANGELOG.vi.md

@@ -6,6 +6,23 @@ Mọi thay đổi đáng chú ý của dự án được ghi lại trong file n
 Định dạng dựa trên [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 và dự án tuân theo [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## [0.17.1] - 2026-09-10
+
+### Fixed
+- sb_add và sb_import giờ không còn lưu nhân đôi các con của một node lồng nhau: một batch patch bị áp dụng nhiều hơn một lần (như mọi lần ghi vẫn vậy, để kiểm tra trước khi áp thật) trước đây tự làm thay đổi chính nó ngay ở lượt đầu vì mang theo một node vừa thêm bằng tham chiếu, khiến lượt thứ hai chèn lại các con vào một node đã sẵn có chúng.
+- sb_import và sb_import_site giờ giới hạn một row được làm phẳng ở tối đa 12 cột và coi container dạng grid là tự xuống dòng thay vì nằm trên một hàng, để một trang có content wrapper là CSS grid (chẳng hạn một trang tài liệu kỹ thuật) không còn bị import thành một row hàng trăm cột mỏng dính.
+- sb_import và sb_import_site giờ lấy một khối `<pre>`/`<code>` thành một node duy nhất thay vì một node cho mỗi `<span>` tô cú pháp, để một đoạn code không còn bị vỡ thành hàng chục mảnh một-token-một-node.
+
+## [0.17.0] - 2026-09-10
+
+### Added
+- sb_import_site giờ đọc toàn bộ một site từ một URL — ưu tiên sitemap của chính nhà xuất bản, chỉ crawl link có giới hạn khi không có sitemap — rồi tạo và điền một draft page ở đây cho từng trang tìm được, thay cho vòng lặp thủ công gọi sb_import từng trang một.
+- sb_import_site giờ báo cáo mọi tiền tố đường dẫn lặp lại (như `/products/{slug}`) là một template gắn với nền tảng trước khi tạo bất cứ gì, vì import các URL đó thành trang tĩnh sẽ tạo ra một catalogue mà không gì có thể mua được.
+- sb_import_site giờ xem trước từng trang sẽ nằm ở đâu, gồm cả URL nào sẽ gộp vào trang chủ hiện có của site và slug nào đã bị chiếm, rồi bỏ qua trang có slug trùng thay vì để nền tảng âm thầm đổi tên nó.
+
+### Fixed
+- sb_import và sb_import_site giờ chèn nội dung import vào giữa các band, trước global footer đầu tiên, thay vì nối vào cuối trang, vì việc nối vào cuối trước đây phá vỡ quy tắc thứ tự band của nền tảng trên mọi trang có global footer.
+
 ## [0.16.1] - 2026-09-10
 
 ### Fixed

package/README.md

@@ -99,6 +99,7 @@ make, because those mean "this person's account".
 | `sb_event` | Give a node a click action — open the cart, go to a page, open a pop-up |
 | `sb_bind` | Bind a node's content to real store data, or make a button add to the cart |
 | `sb_import` | Read a page from any public URL and add its structure and content to the open page as real elements, styled with THIS page's own tokens — a translation, not a clone |
+| `sb_import_site` | Read a WHOLE site from one URL — its sitemap, or the links on that page — and give each page found its own draft page here, built from this site's tokens |
 | `sb_store` | Run a store flow that must happen in a fixed order — the four writes that make a working checkout, or any of the platform's 17 form templates (login, register, forgot, contact, subscribe …) with its own field document |
 | `sb_undo` | Put back what a PUT replaced — the platform has no page history or restore, so this is the only way back |
 

package/README.vi.md

@@ -96,6 +96,7 @@ là "tài khoản của người này".
 | `sb_event` | Gắn click action cho một node — mở giỏ, sang trang, mở pop-up |
 | `sb_bind` | Gắn nội dung một node vào dữ liệu cửa hàng thật, hoặc biến một nút thành nút thêm vào giỏ |
 | `sb_import` | Đọc một trang từ URL công khai bất kỳ và thêm cấu trúc + nội dung của nó vào trang đang mở dưới dạng element thật, mang token của CHÍNH trang này — là dịch lại, không phải sao chép |
+| `sb_import_site` | Đọc CẢ website từ một URL — sitemap của nó, hoặc các link trên trang đó — và tạo cho mỗi trang tìm được một trang nháp riêng ở đây, dựng bằng token của site này |
 | `sb_store` | Chạy một luồng cửa hàng bắt buộc đúng thứ tự — bốn lệnh ghi tạo nên trang thanh toán, hoặc gieo bất kỳ template nào trong 17 form của nền tảng (login, register, forgot, contact, subscribe …) kèm field document của nó |
 | `sb_undo` | Trả lại thứ mà một lệnh PUT đã ghi đè — nền tảng không có lịch sử trang hay restore, nên đây là đường về duy nhất |
 

package/dist/core/patch.js

@@ -78,6 +78,16 @@ function parentOf(state, path) {
  * half-edited with nothing anywhere to say so — which is the exact failure shape
  * this whole repo exists to rule out.
  */
+/**
+ * A patch's value, detached from whatever the caller still holds.
+ *
+ * Primitives are returned as they are — a style key is a string, and cloning
+ * one on every `sb_set` would be pure cost. Only a structure can be aliased,
+ * and only an alias can be mutated behind the batch's back.
+ */
+function clone(value) {
+    return value !== null && typeof value === 'object' ? structuredClone(value) : value;
+}
 export function applyPatches(state, patches) {
     for (const p of patches) {
         if (!isSyncablePatch(p)) {
@@ -86,7 +96,27 @@ export function applyPatches(state, patches) {
         const { holder, key } = parentOf(state, p.path);
         switch (p.op) {
             case 'set':
-                holder[key] = p.value;
+                // THE VALUE IS NEVER ALIASED INTO THE DOCUMENT.
+                //
+                // `addSubtree` builds a node, emits `set nodes/<id>` carrying THAT
+                // object, and then emits `insert` patches that push child ids into
+                // `data.nodes`. Assigning the reference means the first apply MUTATES
+                // the patch's own value, so the batch is no longer the thing it was: a
+                // second apply re-establishes a node that already holds its children and
+                // then inserts them again.
+                //
+                // Applying a batch twice is not hypothetical — `applyAndSave` does it by
+                // design, once through `preview` to judge the write and once for real.
+                // Between v0.16.1 (which introduced that check) and this fix, EVERY
+                // nested `sb_add` stored each child twice, and `sb_import` three times.
+                // It is silent: the tree is well-formed, every id resolves, the save is
+                // accepted, and the page simply renders its content twice. Measured on
+                // a real import — 106 of 231 containers listing one child id three
+                // times.
+                //
+                // A copy makes the batch idempotent for this shape: the re-`set` puts a
+                // pristine node back, and the inserts that follow rebuild the same list.
+                holder[key] = clone(p.value);
                 break;
             case 'unset':
                 delete holder[key];
@@ -95,7 +125,7 @@ export function applyPatches(state, patches) {
                 const arr = holder[key];
                 if (!Array.isArray(arr))
                     break;
-                arr.splice(p.index, 0, p.value);
+                arr.splice(p.index, 0, clone(p.value));
                 break;
             }
             case 'remove': {

package/dist/domains/site/discover.js

@@ -0,0 +1,347 @@
+/**
+ * WHICH PAGES A SITE HAS, starting from one URL.
+ *
+ * `sb_import` reads ONE page, and that is the right shape for "bring this
+ * section over". It is the wrong shape for the thing people actually ask for —
+ * hand over a link and get the site — because the missing half was never the
+ * capture: it was knowing what the pages ARE, and creating one for each.
+ *
+ * Two sources, in that order, because they cost wildly different amounts:
+ *
+ *   1. The site's own sitemap. One HTTP fetch, no browser, and it is the
+ *      publisher's own list rather than a guess — a page nothing links to is in
+ *      it, and a link that goes nowhere is not.
+ *   2. The links on the entry page. A browser navigation each, so it is bounded
+ *      hard, and it is the fallback rather than the default for that reason.
+ *
+ * This module is PURE, for the same reason `importmap.ts` is: the rules about
+ * what counts as a page are the half worth arguing over, and they can be argued
+ * over offline. Fetching belongs to the caller.
+ */
+/** Anything whose extension says it is a file rather than a page. */
+const ASSET = /\.(?:pdf|jpe?g|png|gif|webp|svg|ico|avif|css|js|mjs|json|xml|rss|atom|zip|rar|gz|tgz|tar|mp3|mp4|m4a|webm|mov|avi|woff2?|ttf|otf|eot|docx?|xlsx?|pptx?|csv|txt)$/i;
+/**
+ * Paths that are a SITE'S PLUMBING rather than its content.
+ *
+ * Every one of these is either somebody else's storefront machinery — a cart, a
+ * login, an account page this platform serves at its own fixed path — or an
+ * endless tail (`/tag/`, `/author/`) that would spend the whole page budget on
+ * near-duplicates of pages already taken. A caller who genuinely wants one says
+ * so through `include`, which is checked first.
+ */
+const NOT_CONTENT = [
+    '/wp-admin', '/wp-login', '/wp-json', '/wp-content', '/xmlrpc', '/cdn-cgi',
+    '/feed', '/rss', '/comments',
+    '/cart', '/checkout', '/my-account', '/account', '/login', '/logout',
+    '/register', '/signin', '/sign-in', '/signup', '/sign-up', '/password',
+    '/search', '/wishlist', '/compare',
+    // Trailing slash means "this segment, wherever it appears": a tag archive is
+    // as often /blog/tag/x as /tag/x.
+    '/tag/', '/tags/', '/author/', '/authors/',
+];
+/**
+ * Is this path the site's plumbing rather than one of its pages?
+ *
+ * A SUBSTRING TEST IS THE WRONG TEST, and it drops real pages: `/feedback`
+ * contains `/feed`, `/cartier-watches` contains `/cart`, `/comments-policy`
+ * contains `/comments`. Each of those is an ordinary page a merchant would
+ * expect to see imported, and the loss is reported only as a number.
+ *
+ * So a plain entry matches at a BOUNDARY — the end of the path, a `/`, or a `.`
+ * so `/wp-login.php` is still caught — and an entry written with a trailing
+ * slash matches that segment anywhere.
+ *
+ * ANCHORING IT AT THE START OF THE PATH WAS THE FIRST FIX AND IT WAS HALF ONE.
+ * A locale prefix is the ordinary shape of the sites this tool is pointed at,
+ * and this platform's own market is Vietnamese, so `/en/cart`, `/vi/account` and
+ * `/shop/checkout` all sailed through — each one eating a page slot and handing
+ * the merchant a junk draft. The needle already begins with `/`, so its own left
+ * boundary comes free: scanning anywhere and testing only the RIGHT boundary
+ * catches those without reopening `/cartier-watches`.
+ */
+function isPlumbing(path) {
+    return NOT_CONTENT.some((s) => {
+        if (s.endsWith('/'))
+            return path.includes(s);
+        for (let at = path.indexOf(s); at !== -1; at = path.indexOf(s, at + 1)) {
+            const next = path.charAt(at + s.length);
+            if (next === '' || next === '/' || next === '.')
+                return true;
+        }
+        return false;
+    });
+}
+/**
+ * One canonical spelling of a URL, or null if it is not a page address at all.
+ *
+ * The trailing slash, `index.html` and a fragment are the three ways the same
+ * page arrives under different names, and a crawl that does not fold them
+ * imports the home page four times. The query string is DELIBERATELY kept here
+ * and dropped later, so the drop can be counted and reported rather than
+ * happening invisibly.
+ */
+export function normalizeUrl(raw, base) {
+    let u;
+    try {
+        u = base ? new URL(raw, base) : new URL(raw);
+    }
+    catch {
+        // A NON-HIERARCHICAL BASE THROWS, and the throw is the whole reason this is
+        // wrapped: `new URL('/a', 'data:text/html,…')` is a TypeError, and one bad
+        // href on one page must not end a crawl.
+        return null;
+    }
+    if (u.protocol !== 'http:' && u.protocol !== 'https:')
+        return null;
+    u.hash = '';
+    u.username = '';
+    u.password = '';
+    let path = u.pathname.replace(/\/index\.[a-z]{2,5}$/i, '/');
+    if (path.length > 1)
+        path = path.replace(/\/+$/, '');
+    u.pathname = path === '' ? '/' : path;
+    return u.toString();
+}
+function originOf(url) {
+    try {
+        return new URL(url).origin;
+    }
+    catch {
+        return '';
+    }
+}
+/**
+ * The path, as a human wrote it.
+ *
+ * `URL` percent-encodes `pathname`, so a Vietnamese path comes back as
+ * `/trang-ch%E1%BB%A7` — which turns into `trang-ch-e1-bb-a7` the moment a slug
+ * is derived from it, and makes an `include: ['/tin-tức']` match nothing. The
+ * decode is guarded because a malformed sequence throws, and a path this cannot
+ * read is better matched raw than not at all.
+ */
+function pathOf(url) {
+    let raw;
+    try {
+        raw = new URL(url).pathname;
+    }
+    catch {
+        return '/';
+    }
+    try {
+        return decodeURIComponent(raw);
+    }
+    catch {
+        return raw;
+    }
+}
+/**
+ * The `<loc>` values in a sitemap, and the child sitemaps of an index.
+ *
+ * Read with a regex rather than an XML parser on purpose: the whole grammar
+ * this needs is one element name, the file arrives from a stranger's server,
+ * and a dependency that parses arbitrary XML from an untrusted origin is a
+ * larger surface than the feature is worth. `<sitemapindex>` is the only
+ * distinction that matters — its locs are more sitemaps, not pages.
+ */
+export function sitemapUrls(xml) {
+    const locs = [];
+    for (const m of xml.matchAll(/<loc>\s*([^<]+?)\s*<\/loc>/gi)) {
+        const v = m[1]
+            .replace(/&amp;/g, '&')
+            .replace(/&lt;/g, '<')
+            .replace(/&gt;/g, '>')
+            .replace(/&quot;/g, '"')
+            .replace(/&apos;/g, "'");
+        locs.push(v);
+    }
+    const isIndex = /<sitemapindex[\s>]/i.test(xml);
+    return isIndex ? { pages: [], sitemaps: locs } : { pages: locs, sitemaps: [] };
+}
+/** The `Sitemap:` lines of a robots.txt — where a site says its sitemap really lives. */
+export function robotsSitemaps(txt) {
+    const out = [];
+    for (const line of txt.split(/\r?\n/)) {
+        const m = /^\s*sitemap\s*:\s*(\S+)/i.exec(line);
+        if (m)
+            out.push(m[1]);
+    }
+    return out;
+}
+/** A slug this platform will accept, from a path. The root is the home page. */
+export function slugFor(url, taken) {
+    const path = pathOf(url);
+    const base = path === '/'
+        ? 'home'
+        : path
+            .replace(/\.[a-z0-9]{1,5}$/i, '')
+            .split('/')
+            .filter(Boolean)
+            .join('-');
+    let slug = base
+        .toLowerCase()
+        // NFD does not decompose đ — it is a letter of its own, not a d with a
+        // stroke — and this platform is Vietnamese first, so a path with one in it
+        // would lose the character rather than transliterate it.
+        .replace(/đ/g, 'd')
+        .normalize('NFD')
+        .replace(/[\u0300-\u036f]/g, '')
+        .replace(/[^a-z0-9]+/g, '-')
+        .replace(/^-+|-+$/g, '')
+        .slice(0, 60)
+        .replace(/-+$/g, '');
+    if (!slug)
+        slug = 'page';
+    // A COLLIDING SLUG IS RENAMED BY THE PLATFORM, NOT REFUSED (`uniqueSlug`
+    // suffixes -1, -2 … and its own comment says it never errors), so a duplicate
+    // inside one run would come back under a name the caller never asked for and
+    // every link authored to the requested one would be dead. Settle it here,
+    // where the caller can still see both names in the plan.
+    let out = slug;
+    for (let n = 2; taken.has(out); n += 1)
+        out = `${slug.slice(0, 57)}-${n}`;
+    taken.add(out);
+    return out;
+}
+/** A human name from a slug, for a page whose own `<title>` is not known yet. */
+export function nameFor(slug) {
+    if (slug === 'home')
+        return 'Home';
+    return slug
+        .split('-')
+        .filter(Boolean)
+        .map((w) => w.charAt(0).toUpperCase() + w.slice(1))
+        .join(' ');
+}
+/**
+ * Decide which of the discovered URLs become pages, and in what order.
+ *
+ * ORDER IS PART OF THE ANSWER, not a detail. A sitemap can list five thousand
+ * URLs and the cap will take a dozen; taking the first dozen in file order gives
+ * a site made of whatever the generator happened to emit first, which on a shop
+ * is twelve product pages and no home page. Shallowest first — the root, then
+ * `/about`, then `/blog/a-post` — is the site's own outline, so the cap keeps
+ * the pages a visitor would actually be shown.
+ */
+export function choosePages(entry, urls, opts = {}) {
+    const maxPages = opts.maxPages ?? 12;
+    const include = (opts.include ?? []).map((s) => s.toLowerCase());
+    const exclude = (opts.exclude ?? []).map((s) => s.toLowerCase());
+    const origin = originOf(entry);
+    const skipped = {};
+    const skip = (why) => {
+        skipped[why] = (skipped[why] ?? 0) + 1;
+    };
+    const seen = new Set();
+    const kept = [];
+    // THE ENTRY IS PAGE ONE, whether or not the discovery handed it back. A
+    // sitemap that omits the home page is ordinary — plenty of generators list
+    // only what they manage — and a caller who pasted a link and got a site
+    // without the page they pasted has been given the wrong site. It still passes
+    // through `include` / `exclude` below: forcing it past the caller's own filter
+    // would be a different kind of wrong.
+    for (const f of [{ url: entry, from: 'entry' }, ...urls]) {
+        const norm = normalizeUrl(f.url);
+        if (!norm) {
+            skip('unreadable');
+            continue;
+        }
+        if (originOf(norm) !== origin) {
+            skip('off-site');
+            continue;
+        }
+        const path = pathOf(norm).toLowerCase();
+        const wanted = include.length > 0 ? include.some((s) => path.includes(s)) : null;
+        if (wanted === false) {
+            skip('not-included');
+            continue;
+        }
+        if (exclude.some((s) => path.includes(s))) {
+            skip('excluded');
+            continue;
+        }
+        // AN EXPLICIT include OUTRANKS THE PLUMBING LIST. The list is a heuristic
+        // about a stranger's site, and a caller who names `/account` knows something
+        // this module does not.
+        if (!wanted && norm !== entry) {
+            if (ASSET.test(path)) {
+                skip('asset');
+                continue;
+            }
+            if (isPlumbing(path)) {
+                skip('not-content');
+                continue;
+            }
+        }
+        if (new URL(norm).search) {
+            // A query string is nearly always a filter, a sort or a page number over
+            // content already taken, and following them is how a crawl of a shop
+            // spends twelve pages on the same grid.
+            skip('query');
+            continue;
+        }
+        if (seen.has(norm)) {
+            // A SECOND COPY OF THE ENTRY IS NOT A LOSS. It is injected above and the
+            // discovery hands it back too — a sitemap lists it, a crawl seeds its
+            // frontier with it — so counting that collision reports a page dropped
+            // when none was.
+            if (norm !== entry)
+                skip('duplicate');
+            continue;
+        }
+        seen.add(norm);
+        kept.push({ url: norm, from: norm === entry ? 'entry' : f.from });
+    }
+    // GROUPS ARE COUNTED BEFORE THE CAP, because the whole point of reporting them
+    // is to say what the cap is about to hide.
+    const groups = {};
+    for (const f of kept) {
+        const seg = pathOf(f.url).split('/').filter(Boolean)[0];
+        if (seg)
+            groups[seg] = (groups[seg] ?? 0) + 1;
+    }
+    for (const k of Object.keys(groups))
+        if (groups[k] < 3)
+            delete groups[k];
+    const depthOf = (u) => pathOf(u).split('/').filter(Boolean).length;
+    kept.sort((a, b) => {
+        if (a.url === entry)
+            return -1;
+        if (b.url === entry)
+            return 1;
+        const d = depthOf(a.url) - depthOf(b.url);
+        return d !== 0 ? d : a.url.localeCompare(b.url);
+    });
+    const over = Math.max(0, kept.length - maxPages);
+    if (over > 0)
+        skipped['over-page-limit'] = over;
+    const taken = new Set();
+    const pages = kept.slice(0, maxPages).map((f) => {
+        const slug = slugFor(f.url, taken);
+        return { ...f, slug, name: nameFor(slug), depth: depthOf(f.url) };
+    });
+    return { pages, skipped, groups };
+}
+/**
+ * The filter a link crawl needs, as one function: a raw href in, its canonical
+ * spelling out, or null.
+ *
+ * Shares the tables with `choosePages` on purpose — a crawl that queued the
+ * links `choosePages` is about to throw away would spend its whole navigation
+ * budget on a login page and a PDF. `include` / `exclude` are NOT applied here:
+ * those are the caller's narrowing of the site, and a page excluded from the
+ * import can still be the page that links to one that is not.
+ */
+export function canonFor(entry) {
+    const origin = originOf(entry);
+    return (raw) => {
+        const norm = normalizeUrl(raw, entry);
+        if (!norm || originOf(norm) !== origin)
+            return null;
+        const path = pathOf(norm).toLowerCase();
+        if (ASSET.test(path))
+            return null;
+        if (isPlumbing(path))
+            return null;
+        return norm;
+    };
+}

package/dist/domains/site/traps.js

@@ -56,6 +56,33 @@ export function checkBandOrder(doc) {
     }
     return null;
 }
+/**
+ * WHERE NEW PAGE CONTENT MAY BE ADDED among ROOT's children: before the first
+ * global footer, or at the end when there is none.
+ *
+ * APPENDING TO ROOT IS THE OBVIOUS THING AND IT IS WRONG ON A REAL PAGE. Every
+ * site that has a global footer has one as a ROOT child, so `data.nodes.length`
+ * puts the new section AFTER it — `checkBandOrder` then refuses the save, the
+ * platform would have refused it too, and the caller is told about a band rule
+ * they did not knowingly break. Measured on the one path where it is the
+ * DEFAULT: `sb_import_site` imports the entry URL into the site's existing home
+ * page, which is exactly the page most likely to carry both globals.
+ *
+ * The index is into `data.nodes` RAW, overlays included, because that is what
+ * `addSubtree` takes. Overlays are skipped when deciding, never when counting:
+ * the platform strips them before it checks the bands, so where one sits says
+ * nothing about where content may go.
+ */
+export function middleEnd(doc) {
+    const kids = doc.nodes[doc.root_node_id]?.data?.nodes ?? [];
+    for (let i = 0; i < kids.length; i += 1) {
+        if (isOverlay(doc, kids[i]))
+            continue;
+        if (bandOf(doc, kids[i]) === 'footer')
+            return i;
+    }
+    return kids.length;
+}
 /** Is this node a composed GLOBAL SECTION master — a shared header or footer? */
 export function isGlobal(doc, id) {
     return doc.nodes[id]?.specials?.[SPEC_GLOBAL_ID] !== undefined;

package/dist/tools/importpage.js

@@ -1,30 +1,152 @@
 import { z } from 'zod';
 import { text } from '../mcp/response.js';
-import { capture } from '../vision/capture.js';
+import { capture, captureMany, crawlLinks } from '../vision/capture.js';
 import { uploadMedia } from '../transport/media.js';
 import { addSubtree } from '../domains/site/builder.js';
+import { middleEnd } from '../domains/site/traps.js';
 import { toSpecs, tokensFromPage, imageSources, rehostImages, } from '../domains/site/importmap.js';
+import { canonFor, choosePages, normalizeUrl, robotsSitemaps, sitemapUrls, } from '../domains/site/discover.js';
+import { loadSource } from '../transport/pages.js';
+import { PageDoc } from '../domains/site/document.js';
+import { request } from '../transport/http.js';
+import { siteToken } from './credentialpick.js';
 import { siteFor } from './context.js';
 /**
- * BRING A PAGE FROM ELSEWHERE ONTO THIS SITE.
+ * Read a URL from SOMEBODY ELSE'S ORIGIN.
  *
- * Not a clone, on purpose. The platform HAS an escape hatch that would produce
- * one — `custom-code` embeds raw markup verbatim — and using it would give the
- * merchant a Store Builder page that no inspector can edit, that has no
- * responsive cascade, that binds to nothing, and that carries somebody else's
- * CSS and scripts. Visually closest, structurally a dead end.
+ * NOT through `request()`, and that is the point: every path in that module
+ * attaches a credential, and this one must attach none. A sitemap fetch that
+ * carried `SB_TOKEN` would hand this install's key to a stranger's server
+ * because the caller pasted a link — the same envelope rule this repo keeps for
+ * every other secret, applied to the one call that leaves the platform.
  *
- * So the import is a TRANSLATION: the source's structure and content, rendered
- * with THIS site's own tokens, as real elements the merchant can then edit.
- * Rule 0 of the design skill is the reason — a section that answers the accent,
- * the ink and the radius differently does not read as a new section, it reads as
- * a different website, and importing from elsewhere is the one operation that
- * threatens to do that on purpose.
+ * A failure is an ANSWER, not an error: most sites have no robots.txt, plenty
+ * have no sitemap, and discovery falls through to the link crawl. Nothing here
+ * is worth ending a tool call over.
+ */
+async function fetchForeign(ctx, url) {
+    const f = ctx.fetchImpl ?? fetch;
+    try {
+        const res = await f(url, {
+            signal: AbortSignal.timeout(8_000),
+            redirect: 'follow',
+        });
+        if (!res.ok)
+            return null;
+        const body = await res.text();
+        // A sitemap is text from a stranger. Bounded so a multi-megabyte one costs
+        // a slice rather than the process.
+        return body.length > 5_000_000 ? body.slice(0, 5_000_000) : body;
+    }
+    catch {
+        return null;
+    }
+}
+/**
+ * THE PUBLISHER'S OWN LIST OF ITS PAGES, or null if it does not offer one.
+ *
+ * One fetch and no browser, and it lists pages nothing links to — which is why
+ * it is tried before the crawl rather than after it.
+ */
+export async function fromSitemap(ctx, entry) {
+    const origin = new URL(entry).origin;
+    const declared = new Set();
+    // ROBOTS.TXT IS WHERE A SITE SAYS WHERE ITS SITEMAP REALLY IS, and plenty of
+    // real ones are not at /sitemap.xml — a shop platform names
+    // /sitemap_products_1.xml, a CMS a dated path. Guessing only the default is how
+    // a site with a perfectly good sitemap gets crawled instead.
+    const robots = await fetchForeign(ctx, `${origin}/robots.txt`);
+    if (robots)
+        for (const u of robotsSitemaps(robots))
+            declared.add(u);
+    for (const guess of ['/sitemap.xml', '/sitemap_index.xml', '/sitemap-index.xml']) {
+        declared.add(`${origin}${guess}`);
+    }
+    const pages = new Set();
+    let queue = [...declared].slice(0, 5);
+    // ONE level of index expansion. A sitemap index of indexes exists and is rare;
+    // the bound is what keeps a pathological one from becoming a fetch storm on
+    // somebody else's server.
+    for (let round = 0; round < 2 && queue.length > 0; round += 1) {
+        const next = [];
+        for (const sm of queue.slice(0, 5)) {
+            const xml = await fetchForeign(ctx, sm);
+            if (!xml)
+                continue;
+            const got = sitemapUrls(xml);
+            for (const u of got.pages)
+                pages.add(u);
+            for (const u of got.sitemaps)
+                next.push(u);
+        }
+        queue = next;
+    }
+    // TWO IS THE THRESHOLD, NOT ONE. A sitemap listing only the home page is what a
+    // half-configured generator emits, and taking it would import a one-page site
+    // off a site that has forty.
+    if (pages.size < 2)
+        return null;
+    // THE ENTRY IS NOT PUSHED HERE. `choosePages` guarantees it is page one, so
+    // adding it would arrive as a second copy and be counted as a dropped
+    // duplicate — a phantom loss on every sitemap run.
+    const urls = [];
+    for (const u of pages) {
+        // NORMALIZED, NOT FILTERED. `canonFor` exists to stop a crawl spending a
+        // NAVIGATION on a PDF; a sitemap costs no navigation, so there is nothing to
+        // save by dropping one here — and dropping it here means `choosePages` never
+        // sees it and never counts the reason. Filtering twice and reporting once is
+        // how a caller ends up asking why the cart page vanished.
+        const c = normalizeUrl(u, entry);
+        if (c)
+            urls.push({ url: c, from: 'sitemap' });
+    }
+    return urls;
+}
+/**
+ * THE PAGES THIS SITE ALREADY HAS — read by the preview as well as the run.
  *
- * The tokens are read off the OPEN PAGE rather than asked for, because that is
- * the same thing rule 0 tells a person to do: `sb_node_read` a heading, a
- * button, a section, and reuse those exact values.
+ * The dry run used to make no platform call at all, which made it cheap and
+ * made it LIE: it promised twelve pages on a site where four of those slugs
+ * were taken (each of which the run then skips, because the platform renames a
+ * collision and answers 200) and never said the entry URL was about to be
+ * merged into an existing home page rather than given one of its own. A preview
+ * whose count does not survive contact with the run is not a preview.
  */
+async function existingPages(ctx, siteId) {
+    const listed = (await request({
+        base: ctx.base,
+        method: 'GET',
+        path: `/api/sites/${encodeURIComponent(siteId)}/pages`,
+        token: siteToken(ctx),
+        fetchImpl: ctx.fetchImpl,
+    }));
+    return Array.isArray(listed.pages) ? listed.pages : [];
+}
+/**
+ * WHAT PAGES THIS SITE HAS — the publisher's own answer first, a crawl second.
+ *
+ * The order is about cost, not preference. A sitemap is one fetch and no browser;
+ * a crawl is a browser navigation per page and can only find what the entry page
+ * points at. So the crawl is the fallback, automatically: there is no knob to
+ * force it, because a caller who wants fewer pages than the sitemap offers wants
+ * `include` or `max_pages`, not a slower way to find the same list.
+ */
+async function discoverSite(ctx, entry, opts) {
+    const listed = await fromSitemap(ctx, entry);
+    if (listed)
+        return { source: 'sitemap', urls: listed, titles: new Map(), visited: 0 };
+    const crawled = await crawlLinks(entry, {
+        depth: opts.depth ?? 1,
+        maxVisits: opts.maxVisits ?? 24,
+        canon: canonFor(entry),
+    });
+    return {
+        source: 'links',
+        urls: crawled.urls.map((u) => ({ url: u, from: u === entry ? 'entry' : 'links' })),
+        titles: crawled.titles,
+        visited: crawled.visited,
+    };
+}
 export function registerImportTools(server, ctx, session) {
     server.registerTool('sb_import', {
         description: 'Read a page from any public URL and add its structure and content to the OPEN page as ' +
@@ -133,7 +255,12 @@ export function registerImportTools(server, ctx, session) {
         const all = [];
         const staged = doc.preview([]);
         for (const spec of specs) {
-            const { patches, ids } = addSubtree(staged, staged.doc.root_node_id, spec);
+            // BEFORE THE GLOBAL FOOTER, not after it. Appending to ROOT is the
+            // obvious thing and it breaks trap 3 on every page that has a footer —
+            // the platform refuses the whole save, and the caller is told about a
+            // band rule they did not knowingly break. Recomputed each time because
+            // the last insert moved it.
+            const { patches, ids } = addSubtree(staged, staged.doc.root_node_id, spec, middleEnd(staged.doc));
             staged.apply(patches);
             all.push(...patches);
             added.push(ids[0]);
@@ -160,4 +287,289 @@ export function registerImportTools(server, ctx, session) {
                     : ''),
         });
     });
+    server.registerTool('sb_import_site', {
+        description: 'Read a WHOLE site from one URL — its sitemap, or the links on that page — and give each ' +
+            'page found its own DRAFT page here, built from this site\'s tokens. Not a clone. Dry run ' +
+            'returns the page list before anything is created.',
+        inputSchema: {
+            url: z.string().describe('Any page of the site'),
+            site_id: z.string().optional(),
+            max_pages: z.number().int().min(1).max(60).optional().describe('Default 12'),
+            depth: z.number().int().min(0).max(3).optional().describe('No sitemap: link depth, default 1'),
+            include: z.array(z.string()).optional().describe('Path substrings to keep'),
+            exclude: z.array(z.string()).optional(),
+            max_images: z.number().int().min(0).max(200).optional().describe('Default 24, whole import'),
+            max_nodes: z.number().int().min(1).max(1000).optional().describe('Per page, default 300'),
+            upload_images: z.boolean().optional(),
+            homepage: z.boolean().optional().describe("Entry into this site's home page, default true"),
+            dry_run: z.boolean().optional(),
+        },
+        annotations: { readOnlyHint: false, destructiveHint: false, openWorldHint: true },
+    }, async ({ url, site_id: given, max_pages, depth, include, exclude, max_images, max_nodes, upload_images, homepage, dry_run, }) => {
+        const siteId = siteFor(ctx, given);
+        const entry = normalizeUrl(url);
+        if (!entry) {
+            throw new Error(`sbuilder: "${url}" is not a page address this server can read — an http or https URL is needed.`);
+        }
+        const found = await discoverSite(ctx, entry, {
+            depth,
+            // The crawl may look at more pages than it imports — that is how it finds
+            // the twelfth — but not without bound.
+            maxVisits: Math.max(4, (max_pages ?? 12) * 2),
+        });
+        const chosen = choosePages(entry, found.urls, {
+            maxPages: max_pages,
+            include,
+            exclude,
+        });
+        // A CRAWL ALREADY READ THE TITLE. `nameFor` derives a name from the slug
+        // because a sitemap offers nothing else, but the link crawl opened every
+        // one of these pages to read its links and has the page's own `<title>` —
+        // so the plan can name them the way their author does, before anything is
+        // captured.
+        const plan = {
+            ...chosen,
+            pages: chosen.pages.map((p) => ({ ...p, name: found.titles.get(p.url) || p.name })),
+        };
+        if (plan.pages.length === 0) {
+            throw new Error(`sbuilder: no page worth importing was found from ${entry}. Discovered by ` +
+                `${found.source}; skipped ${JSON.stringify(plan.skipped)}. A site behind a login, or ` +
+                'one whose links are all off-site, reads as empty here.');
+        }
+        // FORTY URLS UNDER ONE PREFIX ARE NOT FORTY PAGES ON THIS PLATFORM.
+        //
+        // They are one entity TEMPLATE plus a catalogue: `/products/x` resolves to
+        // the site's published page of type `product`, bound to the record in the
+        // URL. Importing them as static pages produces a shop where every price is
+        // a literal, nothing is buyable, and `sb_review` reports a missing purchase
+        // action on forty pages at once. Said before anything is created, because
+        // after it the fix is forty deletes.
+        const heavy = Object.entries(plan.groups)
+            .filter(([, n]) => n >= 3)
+            .map(([seg, n]) => `${seg} (${n})`);
+        const templateNote = heavy.length > 0
+            ? `Several URLs share a prefix — ${heavy.join(', ')}. If those are products, ` +
+                'collections or posts, they are ONE template plus real records here, not one page ' +
+                'each: sb_page_create type:"product" (or category/post) seeds the bound page, and ' +
+                'the catalogue comes from the API. Pass exclude to leave them out of the import.'
+            : undefined;
+        // BEST EFFORT IN THE PREVIEW, REQUIRED IN THE RUN.
+        //
+        // The listing is what makes the preview honest — which slugs are taken,
+        // whether the entry merges into an existing home page — but demanding it
+        // would turn "what is on that website?" into a question only a connected
+        // install may ask, and the discovery half needs no credential at all. So a
+        // dry run that cannot read the site says the landing spot is unknown
+        // rather than inventing one; the real run must not guess, and rethrows.
+        let existing = [];
+        let unlistable = '';
+        try {
+            existing = await existingPages(ctx, siteId);
+        }
+        catch (e) {
+            // NAME WHAT FAILED. The bare platform string ("boom", "not found") tells
+            // a caller who asked to import a website nothing about WHICH call broke,
+            // and the one that broke is this site's own page listing — without it the
+            // run cannot tell a free slug from a taken one, or find the home page.
+            if (dry_run === false) {
+                throw new Error(`sbuilder: could not read this site's own pages, so the import cannot tell which ` +
+                    `slugs are free or which page is the home page — nothing was created. ` +
+                    `${e.message.replace(/^sbuilder:\s*/, '')}`);
+            }
+            unlistable = e.message.replace(/^sbuilder:\s*/, '');
+        }
+        const home = existing.find((e) => e.isHomepage === true);
+        const taken = new Set(existing.map((e) => (typeof e.slug === 'string' ? e.slug : '')).filter(Boolean));
+        const lands = (p) => unlistable
+            ? {}
+            : p.url === entry && homepage !== false && home
+                ? { into: 'the existing home page' }
+                : taken.has(p.slug)
+                    ? { conflict: `a page with slug "${p.slug}" already exists — this one is SKIPPED` }
+                    : {};
+        if (dry_run !== false) {
+            return text({
+                dry_run: true,
+                entry,
+                discovered_by: found.source,
+                ...(found.visited ? { pages_read_to_find_them: found.visited } : {}),
+                pages: plan.pages.map((p) => ({ url: p.url, slug: p.slug, name: p.name, ...lands(p) })),
+                ...(Object.keys(plan.skipped).length ? { skipped: plan.skipped } : {}),
+                ...(unlistable
+                    ? {
+                        landing_unknown: `This site's own pages could not be read (${unlistable}), so which of the above ` +
+                            'would merge into an existing home page, and which would collide with a slug ' +
+                            'already taken, is not known yet.',
+                    }
+                    : {}),
+                ...(templateNote ? { entity_pages: templateNote } : {}),
+                note: 'Nothing has been created. Each page above becomes a DRAFT page here, filled with the ' +
+                    "source's structure and text and styled with this site's own tokens — the source's CSS " +
+                    'and layout are not copied. Pass dry_run:false to build them.',
+            });
+        }
+        // THE TOKENS COME FROM THIS SITE, ONCE, FOR EVERY IMPORTED PAGE.
+        //
+        // `sb_import` reads them off the OPEN page, which is right when the import
+        // is one section onto a page that already has a look. Here most of the
+        // target pages do not exist yet and the ones that do are blank, so reading
+        // per page would give the first page element defaults and every later page
+        // the defaults of the blank page before it — rule 0 failing on every page
+        // at once. The open page if there is one, the site's home page otherwise.
+        let tokenDoc = session.peek();
+        if (!tokenDoc && home && typeof home.id === 'string') {
+            try {
+                tokenDoc = PageDoc.from((await loadSource(ctx, siteId, home.id)).document);
+            }
+            catch {
+                // An unreadable home page costs the tokens, not the import: every field
+                // of PageTokens is optional and falls back to the element's own
+                // defaults, which is the same answer an empty target gives.
+            }
+        }
+        const tokens = tokenDoc ? tokensFromPage(tokenDoc.doc) : {};
+        const shots = await captureMany(plan.pages.map((p) => p.url), { maxImages: max_images ?? 24, maxNodes: max_nodes ?? 300 });
+        const byUrl = new Map(shots.map((s) => [s.url, s]));
+        // ONE UPLOAD PER IMAGE FOR THE WHOLE SITE, not per page. A logo, a payment
+        // strip and a footer badge appear on every page of a real site, and
+        // uploading each of them twelve times would fill the merchant's library
+        // with twelve copies and pay twelve round trips for one asset.
+        const budget = max_images ?? 24;
+        const seenSrc = new Set();
+        for (const s of shots) {
+            if (!s.ok)
+                continue;
+            for (const src of imageSources(s.result.sections))
+                seenSrc.add(src);
+        }
+        const wanted = [...seenSrc].slice(0, budget);
+        const overBudget = seenSrc.size - wanted.length;
+        const rehosted = new Map();
+        const failures = new Map();
+        if (upload_images !== false) {
+            for (const src of wanted) {
+                try {
+                    const up = await uploadMedia(ctx, siteId, { url: src });
+                    if (up.url)
+                        rehosted.set(src, up.url);
+                }
+                catch (e) {
+                    const why = e.message.replace(/^sbuilder:\s*/, '').slice(0, 160);
+                    failures.set(why, (failures.get(why) ?? 0) + 1);
+                }
+            }
+        }
+        const failedImages = [...failures.entries()].map(([reason, count]) => ({ reason, count }));
+        // A PAGE THAT FAILS DOES NOT END THE RUN, and this is the one place in the
+        // server where that is the right call. A site import is not atomic and
+        // cannot be — each page is its own create and its own save — so the honest
+        // shape is per-page outcomes. Aborting on the fourth of twelve would leave
+        // three pages built, nine not, and no report saying which.
+        const built = [];
+        const failed = [];
+        let lastOpened = '';
+        for (const p of plan.pages) {
+            const shot = byUrl.get(p.url);
+            if (!shot || !shot.ok) {
+                failed.push({ url: p.url, why: shot ? shot.why : 'was not read' });
+                continue;
+            }
+            try {
+                const sections = rehosted.size > 0 ? rehostImages(shot.result.sections, rehosted) : shot.result.sections;
+                const specs = toSpecs(sections, tokens);
+                if (specs.length === 0) {
+                    failed.push({
+                        url: p.url,
+                        why: `nothing renderable — skipped ${JSON.stringify(shot.result.skipped)}`,
+                    });
+                    continue;
+                }
+                let pageId = '';
+                let into;
+                const isEntry = p.url === entry;
+                if (isEntry && homepage !== false && home && typeof home.id === 'string') {
+                    pageId = home.id;
+                    into = 'the existing home page';
+                }
+                else {
+                    // A COLLIDING SLUG IS RENAMED BY THE PLATFORM, NOT REFUSED, so
+                    // creating over one answers 200 under a name nobody asked for. On a
+                    // second run of this tool that would silently double the site.
+                    if (taken.has(p.slug)) {
+                        failed.push({
+                            url: p.url,
+                            why: `a page with slug "${p.slug}" already exists — left alone, because the ` +
+                                'platform would have stored this one under a different slug and reported success',
+                        });
+                        continue;
+                    }
+                    const made = (await request({
+                        base: ctx.base,
+                        method: 'POST',
+                        path: `/api/sites/${encodeURIComponent(siteId)}/pages`,
+                        token: siteToken(ctx),
+                        // BLANK, DELIBERATELY. `type: "page"` has no seed, and a seeded
+                        // page would mix the platform's own content with the imported
+                        // page's — two headings, two heroes, and no way to tell them apart.
+                        body: { name: shot.result.title || p.name, type: 'page', slug: p.slug },
+                        fetchImpl: ctx.fetchImpl,
+                    }));
+                    if (typeof made.page?.id !== 'string' || !made.page.id) {
+                        failed.push({ url: p.url, why: 'the platform created no page for it' });
+                        continue;
+                    }
+                    pageId = made.page.id;
+                    if (typeof made.page.slug === 'string')
+                        taken.add(made.page.slug);
+                }
+                await session.open(siteId, pageId);
+                const doc = session.current();
+                // STAGED ON A COPY, committed once — the same reason `sb_import` does
+                // it: each section's patches are computed from the tree the last one
+                // left, and applying them to the real document means a refusal halfway
+                // leaves a page half imported with nothing saying which half.
+                const staged = doc.preview([]);
+                const all = [];
+                const added = [];
+                for (const spec of specs) {
+                    const { patches, ids } = addSubtree(staged, staged.doc.root_node_id, spec, middleEnd(staged.doc));
+                    staged.apply(patches);
+                    all.push(...patches);
+                    added.push(ids[0]);
+                }
+                await session.applyAndSave(all);
+                lastOpened = pageId;
+                built.push({
+                    url: p.url,
+                    slug: p.slug,
+                    page_id: pageId,
+                    sections: added.length,
+                    ...(into ? { into } : {}),
+                    ...(Object.keys(shot.result.skipped).length ? { skipped: shot.result.skipped } : {}),
+                });
+            }
+            catch (e) {
+                failed.push({ url: p.url, why: e.message.replace(/^sbuilder:\s*/, '').slice(0, 200) });
+            }
+        }
+        return text({
+            entry,
+            discovered_by: found.source,
+            built,
+            ...(failed.length ? { failed } : {}),
+            ...(Object.keys(plan.skipped).length ? { skipped: plan.skipped } : {}),
+            images: {
+                copied: rehosted.size,
+                ...(failedImages.length ? { failed: failedImages } : {}),
+                ...(overBudget > 0 ? { over_budget: overBudget } : {}),
+            },
+            ...(lastOpened ? { open: lastOpened } : {}),
+            ...(templateNote ? { entity_pages: templateNote } : {}),
+            directive: ctx.notices.once('import_site', 'These pages are DRAFTS: nothing is live until sb_publish. Three things the import ' +
+                "cannot do for you — the source's header and footer were skipped on purpose (this " +
+                'site has its own as globals, and a second menu pointing at somebody else\'s site is ' +
+                'worse than none), no menu links the new pages together, and nothing has been seen at ' +
+                '390px yet. sb_look each page at the three widths before publishing.'),
+        });
+    });
 }

package/dist/tools/page.js

@@ -159,6 +159,18 @@ export class PageSession {
             throw new Error('sbuilder: no page is open — call sb_page_open first');
         return this.doc;
     }
+    /**
+     * The open document, or null.
+     *
+     * `current()` throws, correctly: every editing tool needs a page and the
+     * message names the call that opens one. A site import is the one caller for
+     * which "no page open" is an ordinary answer rather than a mistake — it reads
+     * the design tokens off whatever page is open, and falls back to the site's
+     * home page when the caller has not opened one.
+     */
+    peek() {
+        return this.doc;
+    }
     /**
      * Validate, then save.
      *

package/dist/vision/capture.js

@@ -183,6 +183,25 @@ function capturePage(limits) {
                 taken.nodes++;
                 return [{ kind: 'list', items }];
             }
+            // A CODE BLOCK IS ONE THING, and walking into it produces rubble.
+            //
+            // Every syntax highlighter wraps each token in its own <span>, so the leaf
+            // walk took them one at a time: a twenty-line JSON config arrived as forty
+            // separate text nodes — `{`, `"mcpServers"`, `: {` — each its own block on
+            // its own line. Measured on a real import; it also ate forty of the node
+            // budget to say what one node says.
+            //
+            // Whitespace is collapsed like any other text because this platform has no
+            // code element to preserve it in — the six kinds are what can cross — so
+            // the honest translation is one paragraph the merchant can then restyle,
+            // not a shredded imitation of a listing.
+            if (tag === 'PRE' || tag === 'CODE') {
+                const text = clean(el.textContent);
+                if (!text)
+                    return [];
+                taken.nodes++;
+                return [{ kind: 'text', text }];
+            }
             if (tag === 'P' || tag === 'BLOCKQUOTE') {
                 const text = clean(el.textContent);
                 if (!text)
@@ -197,12 +216,34 @@ function capturePage(limits) {
                     cs.display === 'inline-flex' || cs.display === 'inline-grid';
                 // A ROW is worth keeping; a column is what the page already is, so
                 // wrapping one in a group would add a level that renders identically.
-                const row = cs.display.indexOf('grid') >= 0
-                    ? true
-                    : cs.flexDirection === 'row' || cs.flexDirection === 'row-reverse';
-                if (lays && row && kids.length >= 2) {
+                const grid = cs.display.indexOf('grid') >= 0;
+                const row = grid || cs.flexDirection === 'row' || cs.flexDirection === 'row-reverse';
+                // A ROW OF 279 IS NOT A ROW.
+                //
+                // Measured on a real import of a documentation site: its content wrapper
+                // is a GRID, every grid was read as a single row, and the whole page came
+                // back as one flex row of 279 columns — each column a sliver, every line
+                // of prose broken to one word, and 80 nodes hanging past the viewport.
+                // The page was structurally perfect and visually destroyed.
+                //
+                // A design row is a feature trio, a card shelf, a logo wall: a handful of
+                // columns, chosen. Past that the container is not arranging things side
+                // by side, it is the page's own content column and the browser is
+                // wrapping it — so the honest translation is the stack it already reads
+                // as. Twelve is above any real row seen here and far below a content
+                // grid.
+                const ROW_MAX = 12;
+                if (lays && row && kids.length >= 2 && kids.length <= ROW_MAX) {
                     taken.nodes++;
-                    return [{ kind: 'group', direction: 'row', wrap: cs.flexWrap === 'wrap', children: kids }];
+                    return [{
+                            kind: 'group',
+                            direction: 'row',
+                            // A GRID ALWAYS WRAPS — that is what a grid IS — and `flexWrap` reads
+                            // `nowrap` on one because the property does not apply. Carrying that
+                            // literally gave the columns nowhere to go at any width.
+                            wrap: grid || cs.flexWrap === 'wrap',
+                            children: kids,
+                        }];
                 }
                 return kids;
             }
@@ -302,19 +343,20 @@ function capturePage(limits) {
     return { url: here, title: clean(document.title), sections, skipped };
 }
 /**
- * Open a URL and capture it.
+ * ONE BROWSER FOR THE WHOLE CALL.
  *
- * `load` plus a short settle rather than `networkidle`, for the same reason
- * `shoot.ts` gives: a page with a poller never goes idle, and waiting for that
- * spends the whole budget on a timeout that cannot resolve.
+ * A site import reads a dozen pages, and launching Chrome once per page pays
+ * the launch a dozen times for nothing. It is still a launch of its OWN rather
+ * than `shoot.ts`'s pooled one, for the reason that module gives: an import is
+ * rare, slow and runs untrusted script, and coupling that to the tool a vision
+ * loop calls every few hundred milliseconds is how the fast path gets slow.
+ *
+ * Closed in a `finally`, always. `shoot()` pools for the process lifetime and a
+ * caller that forgets `closeBrowser()` never exits — there is no `beforeExit`
+ * rescue, because an open browser connection is precisely what stops the event
+ * loop draining.
  */
-export async function capture(url, opts = {}) {
-    const limits = {
-        maxSections: opts.maxSections ?? 24,
-        maxImages: opts.maxImages ?? 24,
-        maxTextChars: opts.maxTextChars ?? 1200,
-        maxNodes: opts.maxNodes ?? 400,
-    };
+async function withBrowser(fn) {
     let browser;
     try {
         browser = await chromium.launch({ channel: 'chrome' });
@@ -323,9 +365,24 @@ export async function capture(url, opts = {}) {
         throw new Error('sbuilder: could not start Chrome to read that page. This server uses the SYSTEM ' +
             `browser (playwright-core, channel "chrome") and did not find one: ${e.message}`);
     }
+    try {
+        return await fn(browser);
+    }
+    finally {
+        await browser.close().catch(() => undefined);
+    }
+}
+/**
+ * Open one URL in an existing browser, let it settle, and run one evaluate.
+ *
+ * `load` plus a short settle rather than `networkidle`, for the same reason
+ * `shoot.ts` gives: a page with a poller never goes idle, and waiting for that
+ * spends the whole budget on a timeout that cannot resolve.
+ */
+async function readPage(browser, url, width, work) {
     let page;
     try {
-        page = await browser.newPage({ viewport: { width: opts.width ?? 1440, height: 900 } });
+        page = await browser.newPage({ viewport: { width, height: 900 } });
         await page.goto(url, { waitUntil: 'load', timeout: 30_000 });
         // THE SAME SETTLE `sb_look` USES, not a flat sleep. A fixed 600ms is wrong
         // at both ends: example.com is finished long before it, and a page that
@@ -334,10 +391,128 @@ export async function capture(url, opts = {}) {
         // question actually being asked (has the page stopped changing) and answers
         // when it becomes true, bounded so a page that never settles is still read.
         await settleDom(page);
-        return (await page.evaluate(capturePage, limits));
+        return await work(page);
     }
     finally {
         await page?.close().catch(() => undefined);
-        await browser.close().catch(() => undefined);
     }
 }
+function limitsFrom(opts) {
+    return {
+        maxSections: opts.maxSections ?? 24,
+        maxImages: opts.maxImages ?? 24,
+        maxTextChars: opts.maxTextChars ?? 1200,
+        maxNodes: opts.maxNodes ?? 400,
+    };
+}
+/** Open a URL and capture it. */
+export async function capture(url, opts = {}) {
+    const limits = limitsFrom(opts);
+    return withBrowser((browser) => readPage(browser, url, opts.width ?? 1440, (page) => page.evaluate(capturePage, limits)));
+}
+/**
+ * Capture several pages through one browser.
+ *
+ * A PAGE THAT FAILS MUST NOT END THE RUN. Half the reason to import a site
+ * rather than a page is that the caller does not know what is at each URL: one
+ * of them is behind a login, one 404s, one hangs. Throwing would discard the
+ * eleven that read fine and give the caller nothing to act on, so each outcome
+ * is carried and the tool reports the failures by reason.
+ */
+export async function captureMany(urls, opts = {}) {
+    const limits = limitsFrom(opts);
+    return withBrowser(async (browser) => {
+        const out = [];
+        for (const url of urls) {
+            try {
+                const result = (await readPage(browser, url, opts.width ?? 1440, (page) => page.evaluate(capturePage, limits)));
+                out.push({ url, ok: true, result });
+            }
+            catch (e) {
+                out.push({ url, ok: false, why: e.message.slice(0, 160) });
+            }
+        }
+        return out;
+    });
+}
+/**
+ * Every link on a page, absolute.
+ *
+ * A SECOND, TINY EVALUATE RATHER THAN A FIELD ON `capturePage`. Discovery
+ * visits pages the import may never take — that is what a depth-2 crawl IS —
+ * and running the whole leaf walk on each of them would pay for content that is
+ * thrown away. This asks the one question discovery has.
+ *
+ * Everything it uses is declared INSIDE it: the function is serialized, so a
+ * module-level constant it closes over simply is not there on the other side.
+ */
+function linksOnPage() {
+    const here = location.href;
+    const links = [];
+    const anchors = Array.from(document.querySelectorAll('a[href]'));
+    for (const a of anchors) {
+        const h = a.getAttribute('href');
+        if (!h)
+            continue;
+        try {
+            links.push(new URL(h, here).href);
+        }
+        catch {
+            // `new URL(rel, base)` THROWS on a non-hierarchical base (a data: page).
+            // One unresolvable href must not kill the crawl.
+        }
+    }
+    return { title: document.title, links };
+}
+/**
+ * Walk a site's own links from one entry page, breadth first.
+ *
+ * THE FALLBACK, never the first choice — every step is a browser navigation, so
+ * the sitemap path exists to avoid this entirely. Bounded on both axes: `depth`
+ * limits how far from the entry a page may be, `maxVisits` limits how many
+ * navigations the whole crawl may spend, and the queue is filtered by the
+ * caller's own `keep` so the bound is spent on pages that could actually become
+ * pages.
+ */
+export async function crawlLinks(entry, opts) {
+    const depth = Math.max(0, opts.depth ?? 1);
+    const maxVisits = opts.maxVisits ?? 24;
+    const found = new Set([entry]);
+    const titles = new Map();
+    let visited = 0;
+    await withBrowser(async (browser) => {
+        let frontier = [entry];
+        // NAVIGATE ONLY WHILE THE LINKS CAN STILL BE USED. `depth` is how far from
+        // the entry a discovered page may be, so the pages AT that distance are
+        // results and are never opened: opening them would pay a navigation each for
+        // links the bound has already ruled out. The import pass opens them anyway.
+        for (let level = 0; level < depth && frontier.length > 0; level += 1) {
+            const next = [];
+            for (const url of frontier) {
+                if (visited >= maxVisits)
+                    break;
+                visited += 1;
+                try {
+                    const got = await readPage(browser, url, 1440, (page) => page.evaluate(linksOnPage));
+                    if (got.title)
+                        titles.set(url, got.title);
+                    // A LEVEL BELOW THE LAST IS WALKED FOR ITS LINKS AND NOT QUEUED: at
+                    // `depth` the crawl still wants what that page points at, it just
+                    // must not navigate any further.
+                    for (const raw of got.links) {
+                        const url2 = opts.canon(raw);
+                        if (!url2 || found.has(url2))
+                            continue;
+                        found.add(url2);
+                        next.push(url2);
+                    }
+                }
+                catch {
+                    // A page that will not open contributes nothing and ends nothing.
+                }
+            }
+            frontier = next;
+        }
+    });
+    return { urls: [...found], titles, visited };
+}

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "sbuilder-mcp",
-  "version": "0.16.1",
+  "version": "0.17.1",
   "description": "MCP server that designs and operates a Store Builder site — pages, data, theme and publish — through the platform's own API and live-edit protocol.",
   "mcpName": "io.github.vuluu2k/sbuilder-mcp",
   "type": "module",