@zeropress/build-core 0.7.9 → 0.7.10

package/README.md

@@ -238,7 +238,7 @@ Preview Data `post.excerpt` and `page.excerpt` are authored values. Build Core d
 
 Both fields are available on detail Post/Page contexts. The same precomputed summary is propagated to `posts.items[]`, archive and taxonomy items, Post/Page collection items, collection cursor `prev`/`next` items, and `post.prev`/`post.next`. Themes should use `summary` for optional list/card copy and retain `excerpt` for an explicitly authored detail lede.
 
-Post/Page metadata descriptions use `summary`; ordinary detail routes do not fall back to `site.description`. A Page used as the front page uses `site.description` only when neither an authored nor body-derived summary exists. RSS item descriptions use Post summaries. Native/Pagefind search continues to index the authored excerpt separately and creates a query-specific body excerpt when needed; it does not substitute the general runtime summary. User-defined `meta.description` keys remain ordinary generator/theme metadata and receive no special Build Core interpretation.
+Post/Page metadata descriptions use `summary`; ordinary detail routes do not fall back to `site.description`. A Page used as the front page uses `site.description` only when neither an authored nor body-derived summary exists. RSS item descriptions use Post summaries. When generated, the native search index uses the authored excerpt separately and creates a query-specific body excerpt when needed; it does not substitute the general runtime summary. User-defined `meta.description` keys remain ordinary generator/theme metadata and receive no special Build Core interpretation.
 
 Every ordinary rendered route receives effective feature state on `site`, regardless of whether the corresponding Preview Data preference was omitted:
 
@@ -368,8 +368,81 @@ For ordinary theme-rendered HTML routes, site customization order is determinist
 
 Closing `</head>` and `</body>` tags are matched case-insensitively. If a configured custom HTML slot has no matching closing tag, the build fails with the route, output path, missing tag, and remediation guidance instead of silently dropping the configured content. Themes that consume these slots must render the corresponding closing tag. A `site.front_page.type: "standalone_html"` document remains byte-for-byte independent and does not receive favicon, generator, custom CSS, or custom HTML injection.
 
+### Native search
+
 Native search artifacts are emitted only when preview-data does not set `site.search.enabled: false` and the active theme declares `features.search: true`. When that effective search state is disabled, search widgets are omitted from resolved widget items while their widget areas and non-search siblings remain available. The native adapter tokenizes non-CJK text with `Intl.Segmenter(site.locale)` and falls back to Unicode word matching only when Segmenter is unavailable or fails. CJK runs contribute their full token and unique bigrams while repeated occurrences still contribute term frequency. `search_pagefind.js` is a Pagefind adapter that can replace `search.js` after a post-build Pagefind step.
 
+Pass `options.generateSearchIndex: false` to `buildSite()` or
+`buildSiteFromThemeDir()` to skip native indexing work. Omission or `true`
+generates the index subject to the limits below. This option does not enable
+search on a site or theme that disables it, and it does not delete search
+files: `search.json`, `search.js`, and `search_pagefind.js` are still emitted
+when search is enabled. Explicit and count-based skips avoid extracting
+searchable body text; size checks stop incrementally before serializing an
+oversized record. Normal indexes retain their existing bytes.
+
+The limits use the uncompressed UTF-8 size of the existing two-space-indented
+JSON, including its final newline (1 MiB = 1,048,576 bytes). Searchable records
+are the published, non-delisted posts, pages, and page front page already
+included by the renderer. The front page is counted once.
+
+The first matching rule applies:
+
+| Condition | `search.json` | Search warning |
+| --- | --- | --- |
+| Site or theme search disabled | No search artifacts | None |
+| No searchable records | `[]` followed by a newline | None, even with indexing explicitly skipped |
+| Indexing explicitly skipped | One non-searchable placeholder | `NATIVE_SEARCH_INDEX_SKIPPED` (`requested`) |
+| More than 5,000 searchable records | One non-searchable placeholder | `NATIVE_SEARCH_INDEX_SKIPPED` (`record_limit`) |
+| JSON size at least 5 MiB | One non-searchable placeholder | `NATIVE_SEARCH_INDEX_SKIPPED` (`size_limit`) |
+| JSON size at least 2 MiB and less than 5 MiB | Complete index | `NATIVE_SEARCH_INDEX_LARGE` |
+| JSON size less than 2 MiB | Complete index | None |
+
+Exactly 5,000 records are allowed if the size limit is not reached. Limits
+cannot be bypassed, and no partial index is emitted. Skipping the index does
+not fail the build or change rendered HTML, effective `site.search.enabled`,
+search widgets, or either JavaScript adapter.
+
+When indexing is skipped for a nonempty site, the entire index is replaced by:
+
+```json
+[
+  {
+    "id": "zeropress:search-omitted",
+    "type": "page",
+    "title": "",
+    "url": "/",
+    "excerpt": "",
+    "headings": [],
+    "categories": [],
+    "tags": [],
+    "published_at_iso": "",
+    "updated_at_iso": "",
+    "content_text": ""
+  }
+]
+```
+
+All searchable fields are empty, so the existing native adapter returns no
+results for every query. This record is a placeholder, not a document: an array
+length of one must not be interpreted as one searchable page. A genuinely
+empty site keeps `[]` instead.
+
+The existing WXR starter Pagefind step uses that distinction: a placeholder
+allows it to index the rendered HTML and replace `search.js` with
+`search_pagefind.js`, while `[]` preserves its guard against crawling unmarked
+pages on a site without searchable content. Pagefind does not index the
+placeholder itself. No search manifest or additional output file is needed.
+
+At most one search warning is returned per build. `NATIVE_SEARCH_INDEX_LARGE`
+includes `byteLength` and `recordCount`; `NATIVE_SEARCH_INDEX_SKIPPED` includes
+`reason` and the actual `recordCount`. A size-limit warning does not report an
+exact final size because measurement stops at the limit. Warnings recommend
+an external search index such as Pagefind and state when native search will
+return no results.
+
+### Optional outputs
+
 When archive is disabled or `archive.html` is missing, archive routes and archive widget items are omitted while authored menu items, widget areas, and non-archive siblings remain unchanged. No archive output path is claimed in that state, so content may use `/archive`. Feed behaves the same way for `feed.xml`: a disabled feed neither emits nor claims the file. Enabled feeds add one RSS autodiscovery link to `meta.head_tags` on ordinary theme-rendered routes; `404.html` and standalone front-page HTML do not receive it.
 
 Optional route templates behave as rendering capabilities, not guaranteed outputs:
@@ -391,6 +464,7 @@ Supported options:
 - `favicon`
 - `sitemapStylesheetHref`
 - `generateFeed`
+- `generateSearchIndex`
 - `generateRobotsTxt`
 - `reservedOutputPaths`
 
@@ -398,6 +472,7 @@ Defaults:
 
 - `assetHashing: true`
 - `generateFeed: true`
+- `generateSearchIndex: true` (automatic limits still apply)
 - `generateRobotsTxt: true`
 
 `reservedOutputPaths` is an internal orchestration boundary for callers that already own files in the final output tree, such as a copied public directory. Reserved files are safety-validated and participate in clean-URL alias, exact-path, and file/directory hierarchy collision checks, but Build Core does not write them or include them in the returned file summaries.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@zeropress/build-core",
-  "version": "0.7.9",
+  "version": "0.7.10",
   "description": "Shared ZeroPress build core",
   "type": "module",
   "license": "MIT",

package/src/api/build-site.js

@@ -8,10 +8,12 @@ import { extractHtmlText } from '../render/html-text.js';
 import { createThemeValidationError } from '../theme/format-theme-validation.js';
 import { ZeroPressEngine } from '../render/zeropress-engine.js';
 import { prepareRenderMenus } from '../render/menus.js';
+import { buildSearchIndex } from '../search/build-search-index.js';
 
 const DEFAULT_OPTIONS = {
   assetHashing: true,
   generateFeed: true,
+  generateSearchIndex: true,
   generateRobotsTxt: true,
 };
 
@@ -3156,45 +3158,53 @@ function insertBeforeClosingTag(html, closingTagPattern, content) {
 }
 
 function buildSearchIndexJson(state) {
-  return `${JSON.stringify(buildSearchIndexItems(state), null, 2)}\n`;
-}
+  const documents = [];
+  for (const post of state.renderData.posts) {
+    if (post.status === 'published' && !isDelistedDocument(post)) {
+      documents.push({ type: 'post', document: post });
+    }
+  }
 
-function buildSearchIndexItems(state) {
-  const posts = state.renderData.posts
-    .filter((post) => post.status === 'published' && !isDelistedDocument(post))
-    .map((post) => ({
-      id: `post:${post.slug}`,
-      type: 'post',
-      title: post.title,
-      url: post.url,
-      excerpt: normalizeSearchText(post.excerpt),
-      headings: buildSearchHeadings(post.toc),
-      categories: Array.isArray(post.categories) ? post.categories.map((category) => category.name).filter(Boolean) : [],
-      tags: Array.isArray(post.tags) ? post.tags.map((tag) => tag.name).filter(Boolean) : [],
-      published_at_iso: normalizeIsoTimestamp(post.published_at_iso),
-      updated_at_iso: normalizeIsoTimestamp(post.updated_at_iso),
-      content_text: extractHtmlText(post.html),
-    }));
+  const addPage = (page, url) => {
+    if (page && page.status === 'published' && !isDelistedDocument(page)) {
+      documents.push({
+        type: 'page',
+        document: page,
+        url,
+        referencePath: state.renderData.pageReferencePathByPage.get(page),
+      });
+    }
+  };
+  if (state.renderData.frontPageRoute?.front_page_type === 'page') {
+    addPage(state.renderData.frontPageRoute.page, '/');
+  }
+  for (const page of state.renderData.pages) addPage(page, page.url);
 
-  const frontPagePage = state.renderData.frontPageRoute?.front_page_type === 'page'
-    ? state.renderData.frontPageRoute.page
-    : null;
-  const frontPageItems = frontPagePage && frontPagePage.status === 'published' && !isDelistedDocument(frontPagePage)
-    ? [buildSearchPageItem(
-        frontPagePage,
-        '/',
-        state.renderData.pageReferencePathByPage.get(frontPagePage),
-      )]
-    : [];
-  const pageItems = state.renderData.pages
-    .filter((page) => page.status === 'published' && !isDelistedDocument(page))
-    .map((page) => buildSearchPageItem(
-      page,
-      page.url,
-      state.renderData.pageReferencePathByPage.get(page),
-    ));
+  const result = buildSearchIndex({
+    documents,
+    generateSearchIndex: state.options.generateSearchIndex,
+    createItem: (entry) => entry.type === 'post'
+      ? buildSearchPostItem(entry.document)
+      : buildSearchPageItem(entry.document, entry.url, entry.referencePath),
+  });
+  state.warnings.push(...result.warnings);
+  return result.json;
+}
 
-  return [...posts, ...frontPageItems, ...pageItems];
+function buildSearchPostItem(post) {
+  return {
+    id: `post:${post.slug}`,
+    type: 'post',
+    title: post.title,
+    url: post.url,
+    excerpt: normalizeSearchText(post.excerpt),
+    headings: buildSearchHeadings(post.toc),
+    categories: Array.isArray(post.categories) ? post.categories.map((category) => category.name).filter(Boolean) : [],
+    tags: Array.isArray(post.tags) ? post.tags.map((tag) => tag.name).filter(Boolean) : [],
+    published_at_iso: normalizeIsoTimestamp(post.published_at_iso),
+    updated_at_iso: normalizeIsoTimestamp(post.updated_at_iso),
+    content_text: extractHtmlText(post.html),
+  };
 }
 
 function buildSearchPageItem(page, url, pageReferencePath) {

package/src/index.d.ts

@@ -31,6 +31,7 @@ export interface BuildOptions {
   };
   sitemapStylesheetHref?: string;
   generateFeed?: boolean;
+  generateSearchIndex?: boolean;
   generateRobotsTxt?: boolean;
   reservedOutputPaths?: readonly string[];
 }
@@ -80,7 +81,7 @@ export interface BuildSummaryFile {
   sha256: string;
 }
 
-export interface BuildWarning {
+export interface MenuDepthWarning {
   code: 'MENU_MAX_DEPTH_EXCEEDED';
   message: string;
   menuId: string;
@@ -89,6 +90,25 @@ export interface BuildWarning {
   omittedItems: number;
 }
 
+export interface NativeSearchIndexLargeWarning {
+  code: 'NATIVE_SEARCH_INDEX_LARGE';
+  message: string;
+  byteLength: number;
+  recordCount: number;
+}
+
+export interface NativeSearchIndexSkippedWarning {
+  code: 'NATIVE_SEARCH_INDEX_SKIPPED';
+  message: string;
+  reason: 'requested' | 'record_limit' | 'size_limit';
+  recordCount: number;
+}
+
+export type BuildWarning =
+  | MenuDepthWarning
+  | NativeSearchIndexLargeWarning
+  | NativeSearchIndexSkippedWarning;
+
 export interface BuildSiteResult {
   files: BuildSummaryFile[];
   warnings: BuildWarning[];

package/src/search/build-search-index.js

@@ -0,0 +1,129 @@
+const WARNING_BYTES = 2 * 1024 * 1024;
+const MAX_BYTES = 5 * 1024 * 1024;
+const MAX_RECORDS = 5000;
+
+const OMITTED_INDEX_JSON = `${JSON.stringify([{
+  id: 'zeropress:search-omitted',
+  type: 'page',
+  title: '',
+  url: '/',
+  excerpt: '',
+  headings: [],
+  categories: [],
+  tags: [],
+  published_at_iso: '',
+  updated_at_iso: '',
+  content_text: '',
+}], null, 2)}\n`;
+
+// createItem stays lazy: skip/count limits must not extract searchable body text.
+export function buildSearchIndex({ documents, createItem, generateSearchIndex = true }) {
+  const recordCount = documents.length;
+  if (recordCount === 0) return { json: '[]\n', warnings: [] };
+  if (generateSearchIndex === false) return skippedIndex('requested', recordCount);
+  if (recordCount > MAX_RECORDS) return skippedIndex('record_limit', recordCount);
+
+  const chunks = [];
+  // Opening "[\n" and closing "\n]\n", including the final newline.
+  let byteLength = 5;
+  for (const document of documents) {
+    const item = createItem(document);
+    const separatorBytes = chunks.length === 0 ? 0 : 2;
+    const remaining = MAX_BYTES - byteLength - separatorBytes;
+    const itemBytes = measureItemBytes(item, remaining);
+    if (itemBytes >= remaining) return skippedIndex('size_limit', recordCount);
+
+    // Only serialize a record after proving it fits. Indentation matches the
+    // original JSON.stringify(items, null, 2) output byte for byte.
+    chunks.push('  ' + JSON.stringify(item, null, 2).replaceAll('\n', '\n  '));
+    byteLength += separatorBytes + itemBytes;
+  }
+
+  const warnings = byteLength >= WARNING_BYTES ? [{
+    code: 'NATIVE_SEARCH_INDEX_LARGE',
+    message: `Native search index is ${byteLength} bytes (${recordCount} searchable records). Consider an external search index such as Pagefind for larger sites.`,
+    byteLength,
+    recordCount,
+  }] : [];
+  return { json: `[\n${chunks.join(',\n')}\n]\n`, warnings };
+}
+
+function skippedIndex(reason, recordCount) {
+  const explanation = {
+    requested: 'Native search indexing was skipped by request.',
+    record_limit: `Native search indexing was skipped because it exceeds ${MAX_RECORDS} searchable records.`,
+    size_limit: 'Native search indexing was skipped because the index reached the 5 MiB size limit.',
+  }[reason];
+  return {
+    json: OMITTED_INDEX_JSON,
+    warnings: [{
+      code: 'NATIVE_SEARCH_INDEX_SKIPPED',
+      message: `${explanation} Native search will return no results (${recordCount} searchable records). Use an external search index such as Pagefind.`,
+      reason,
+      recordCount,
+    }],
+  };
+}
+
+// Search records contain only strings and arrays of strings. Count the JSON
+// representation with two-space indentation, stopping at the remaining budget.
+// In particular, never stringify a potentially oversized string to measure it.
+function measureItemBytes(item, limit) {
+  let bytes = 2; // Indent the record inside the outer array.
+  const add = (size) => {
+    bytes += size;
+    return bytes < limit;
+  };
+  const string = (value) => {
+    // UTF-8 JSON is never shorter than its input's UTF-16 length plus quotes.
+    if (value.length + 2 >= limit - bytes || !add(2)) return false;
+    for (let index = 0; index < value.length; index += 1) {
+      const code = value.charCodeAt(index);
+      let size;
+      if (code === 0x22 || code === 0x5c) {
+        size = 2;
+      } else if (code < 0x20) {
+        size = [8, 9, 10, 12, 13].includes(code) ? 2 : 6;
+      } else if (code < 0x80) {
+        size = 1;
+      } else if (code < 0x800) {
+        size = 2;
+      } else if (code >= 0xd800 && code <= 0xdfff) {
+        const next = value.charCodeAt(index + 1);
+        if (code <= 0xdbff && next >= 0xdc00 && next <= 0xdfff) {
+          size = 4;
+          index += 1;
+        } else {
+          size = 6; // Well-formed JSON escapes unpaired UTF-16 surrogates.
+        }
+      } else {
+        size = 3;
+      }
+      if (!add(size)) return false;
+    }
+    return true;
+  };
+
+  if (!add(2)) return limit; // "{\n"
+  const entries = Object.entries(item);
+  for (let index = 0; index < entries.length; index += 1) {
+    const [key, value] = entries[index];
+    if (index > 0 && !add(2)) return limit; // ",\n"
+    if (!add(4) || !string(key) || !add(2)) return limit; // indent, key, ": "
+    if (Array.isArray(value)) {
+      if (value.length === 0) {
+        if (!add(2)) return limit;
+      } else {
+        if (!add(2)) return limit; // "[\n"
+        for (let entry = 0; entry < value.length; entry += 1) {
+          if (entry > 0 && !add(2)) return limit;
+          if (!add(6) || !string(value[entry])) return limit;
+        }
+        if (!add(6)) return limit; // "\n    ]"
+      }
+    } else if (!string(value)) {
+      return limit;
+    }
+  }
+  return add(4) ? bytes : limit; // "\n  }"
+}