astro-indexnow 1.0.0 → 2.0.0

package/README.md

@@ -1,37 +1,36 @@
 # astro-indexnow
 
-An Astro integration that automatically submits **all built pages** to **IndexNow**
-after each build, helping search engines discover updates faster.
+A **production‑grade Astro integration** that automatically submits **only new or changed pages**
+to **IndexNow** after each build.
 
-Designed to follow Astro’s official integration patterns (like `@astrojs/sitemap`):
+This package is designed for **modern CI/CD pipelines**, **Docker-based deployments**, and
+**large static sites**, while remaining fully deterministic and explicit.
 
-- No interactive prompts
-- No automatic file mutation
-- Explicit, config‑driven usage
-- CI‑safe and deterministic
-- Zero runtime impact
+> **Version:** v2.0.0  
+> **Stateful by design. Changed-only submissions. Batch-safe. CI-aware.**
 
 ---
 
 ## What is IndexNow?
 
-[IndexNow](https://www.indexnow.org/) is a protocol supported by search engines
-such as **Bing** and **Yandex** that allows websites to proactively notify them
-when URLs are added or updated.
+[IndexNow](https://www.indexnow.org/) is a protocol supported by search engines such as **Bing**
+and **Yandex** that allows websites to proactively notify search engines when URLs are **added,
+updated, or removed**.
 
-Instead of waiting for crawlers, your site directly tells search engines
-which URLs should be re‑crawled.
+Instead of waiting for crawlers, your site tells search engines exactly which URLs changed.
 
 ---
 
-## Features
+## Key Features
 
 - 🚀 Submits URLs to IndexNow **at build time**
-- 🔁 Submits **all generated pages** every build (simple & reliable)
-- 🔒 No secrets prompted or stored automatically
-- 🧠 Uses Astro’s `site` config as the canonical base URL
-- 🧪 Safe for CI/CD pipelines
-- 🧩 No client‑side or runtime code added
+- 🔍 **Detects changed pages automatically** using HTML hashing
+- 🧠 Stores a small on-disk cache to avoid re-submitting unchanged URLs
+- 📦 **Batches submissions** safely (IndexNow 10,000 URL limit)
+- 🧪 Fully **CI/CD and Docker safe**
+- 🔒 No secrets prompted, stored, or mutated
+- 🧩 No client-side or runtime code added
+- 🧼 Quiet by default, verbose when Astro logging is enabled
 
 ---
 
@@ -49,7 +48,7 @@ npx astro add astro-indexnow
 
 > **Note**  
 > Like other Astro integrations, `astro add` installs the package but does **not**
-> inject configuration values. You must configure it manually (see below).
+> inject configuration values. Configuration is always explicit.
 
 ---
 
@@ -60,8 +59,8 @@ Generate an IndexNow API key here:
 👉 https://www.bing.com/indexnow/getstarted
 
 You will receive:
-- an **API key** (a long string)
-- instructions to create a **key file**
+- an **API key**
+- instructions to create a **key verification file**
 
 ---
 
@@ -79,7 +78,7 @@ If your key is:
 abcd1234
 ```
 
-Create this file:
+Create:
 
 ```
 public/abcd1234.txt
@@ -91,139 +90,169 @@ With **exactly this content**:
 abcd1234
 ```
 
-Astro will automatically copy this file to the build output.
+Astro will automatically copy this file into the build output.
 
-> ⚠️ The plugin does **not** create this file for you.
+> ⚠️ The integration does **not** create this file for you.
 
 ---
 
 ## Step 3 — Configure Astro
 
-Add **both** your site URL and the IndexNow integration
-to `astro.config.mjs`.
-
-### Example
+Add both your site URL and the integration to `astro.config.mjs`.
 
 ```js
-// @ts-check
+// astro.config.mjs
 import { defineConfig } from "astro/config";
 import indexnow from "astro-indexnow";
 
 export default defineConfig({
   site: "https://example.com",
-
   integrations: [
     indexnow({
-      key: "YOUR_INDEXNOW_KEY",
+      key: process.env.INDEXNOW_KEY,
     }),
   ],
 });
 ```
 
-### Using environment variables (recommended)
+Using environment variables is **strongly recommended** for CI/CD.
 
-```js
-indexnow({
-  key: process.env.INDEXNOW_KEY,
-});
-```
+---
+
+## How It Works
+
+On every `astro build`, the integration:
+
+1. Walks the final HTML output directory
+2. Hashes each generated `index.html`
+3. Compares hashes against a stored cache
+4. Detects which URLs are **new or changed**
+5. Submits **only those URLs** to IndexNow
+6. Updates the cache after a successful build
 
-This keeps secrets out of version control and works cleanly in CI.
+This ensures:
+- No duplicate submissions
+- No unnecessary API calls
+- Accurate change detection based on real output
 
 ---
 
-## Configuration Options
+## State & Cache File (Important)
 
-### `key` (required)
+To detect changes between builds, the integration stores a small cache file:
 
-Type: `string`
+```
+.astro-indexnow-cache.json
+```
 
-Your IndexNow API key.
+This file:
+- Is created automatically
+- Contains only URL → hash mappings
+- Contains **no secrets**
+- Is safe to inspect, commit, or delete
+
+Deleting the file simply causes the next build to treat all pages as new.
 
 ---
 
-### `site` (required)
+## CI/CD & Git Deployments (Very Important)
 
-Type: `string`
+### If you deploy via Git (CI/CD):
 
-Set via Astro’s config:
+You **must commit** `.astro-indexnow-cache.json` to your repository.
 
-```js
-defineConfig({
-  site: "https://example.com"
-})
+Why:
+- CI environments are ephemeral
+- Without the cache, every build looks like a first build
+- All URLs would be re-submitted
+
+**Recommended:**
+
+```gitignore
+# DO NOT ignore this file
+!.astro-indexnow-cache.json
 ```
 
-Must begin with `http://` or `https://`.
+### If you deploy on a persistent server:
+
+If your build directory persists between builds (e.g. rsync, SSH, in-place deploy),
+you **should not commit** the cache file. It will persist naturally.
 
 ---
 
-### `enabled` (optional)
+## Logging Behaviour
 
-Type: `boolean`  
-Default: `true`
+- **Info logs**: high-level results (submitted, skipped, disabled)
+- **Debug logs**: cache operations, diffs, batching
+- **Warnings**: network or API failures
 
-Disable IndexNow submissions without removing config:
+Use Astro’s verbose logging to see internal details:
 
-```js
-indexnow({
-  key: "...",
-  enabled: false,
-});
+```bash
+astro build --verbose
 ```
 
 ---
 
-## How It Works
+## Configuration Options
 
-On every `astro build`, the integration:
+### `key` (required)
 
-1. Reads Astro’s build output directory
-2. Detects all generated pages
-3. Converts them to public URLs
-4. Submits them to IndexNow via a single POST request
+Type: `string`
 
-No state is stored and no runtime code is added to your site.
+Your IndexNow API key.
 
 ---
 
-## CI / Deployment
+### `enabled` (optional)
 
-Designed to work cleanly in CI environments.
+Type: `boolean`  
+Default: `true`
 
-Example:
+Set to `false` to disable submissions entirely.
 
-```bash
-INDEXNOW_KEY=your-key astro build
+```js
+indexnow({ enabled: false })
 ```
 
-No prompts. No interactivity. Fully deterministic.
+---
+
+## IndexNow Limits & Batching
+
+- IndexNow allows **up to 10,000 URLs per request**
+- This integration automatically batches submissions
+- Large sites are handled safely without configuration
 
 ---
 
-## Limitations
+## Behavioural Guarantees
 
-- Submissions only run on `astro build`
-- Requires a valid `site` configuration
-- Requires the IndexNow key file to exist
-- Uninstalling the package does not remove config automatically
-  (same behaviour as other Astro integrations)
+- No runtime JavaScript added
+- No mutation of Astro config
+- No prompts or side effects
+- No reliance on Git history
+- Deterministic output-based change detection
 
 ---
 
 ## Philosophy
 
-This integration intentionally mirrors the behaviour of
-official Astro integrations such as:
+This integration follows the same design principles as official Astro integrations
+such as `@astrojs/sitemap`:
+
+- Explicit configuration
+- Predictable behaviour
+- No magic or hidden state
+- Production-first design
+
+---
 
-- `@astrojs/sitemap`
+## Maintainer
 
-It avoids:
-- automatic config mutation
-- secret prompting
-- hidden side effects
+Built and maintained by **Velohost**  
+UK-based, privacy-first infrastructure & developer tooling.
 
-Everything is explicit and predictable.
+https://velohost.co.uk/
 
 ---
 

package/dist/index.d.ts.map

@@ -1 +1 @@
-{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,gBAAgB,EAAE,MAAM,OAAO,CAAC;AAK9C,MAAM,WAAW,eAAe;IAC9B,GAAG,CAAC,EAAE,MAAM,CAAC;IACb,OAAO,CAAC,EAAE,MAAM,CAAC;IACjB,OAAO,CAAC,EAAE,OAAO,CAAC;CACnB;AAED,MAAM,CAAC,OAAO,UAAU,QAAQ,CAC9B,OAAO,GAAE,eAAoB,GAC5B,gBAAgB,CAkHlB"}
+{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA,OAAO,KAAK,EAAE,gBAAgB,EAAE,MAAM,OAAO,CAAC;AAK9C,MAAM,WAAW,eAAe;IAC9B,GAAG,CAAC,EAAE,MAAM,CAAC;IACb,OAAO,CAAC,EAAE,MAAM,CAAC;IACjB,OAAO,CAAC,EAAE,OAAO,CAAC;CACnB;AAED,MAAM,CAAC,OAAO,UAAU,QAAQ,CAC9B,OAAO,GAAE,eAAoB,GAC5B,gBAAgB,CAiMlB"}

package/dist/index.js

@@ -1,86 +1,144 @@
 import fs from "fs";
 import path from "path";
-import { fileURLToPath } from "url";
+import crypto from "crypto";
 export default function indexNow(options = {}) {
     let site = null;
+    const CACHE_FILENAME = ".astro-indexnow-cache.json";
+    const projectRoot = process.cwd();
+    const cachePath = path.join(projectRoot, CACHE_FILENAME);
+    const INDEXNOW_ENDPOINT = "https://api.indexnow.org/indexnow";
+    const INDEXNOW_BATCH_SIZE = 10_000;
+    /* =========================================================
+       Helpers
+       ========================================================= */
+    function ensureCacheFile(logger) {
+        const exists = fs.existsSync(cachePath);
+        logger.debug(`[astro-indexnow] cache exists: ${exists} (${cachePath})`);
+        if (!exists) {
+            logger.debug("[astro-indexnow] creating cache file");
+            fs.writeFileSync(cachePath, "{}", "utf8");
+        }
+    }
+    function hashFile(filePath) {
+        const contents = fs.readFileSync(filePath);
+        const hash = crypto.createHash("sha256");
+        hash.update(contents);
+        return `sha256:${hash.digest("hex")}`;
+    }
+    function loadCache(logger) {
+        logger.debug("[astro-indexnow] loading cache file");
+        try {
+            return JSON.parse(fs.readFileSync(cachePath, "utf8"));
+        }
+        catch {
+            logger.warn("[astro-indexnow] cache file unreadable, resetting");
+            return {};
+        }
+    }
+    function saveCache(logger, data) {
+        logger.debug("[astro-indexnow] writing cache file");
+        fs.writeFileSync(cachePath, JSON.stringify(data, null, 2), "utf8");
+    }
+    function chunk(array, size) {
+        const chunks = [];
+        for (let i = 0; i < array.length; i += size) {
+            chunks.push(array.slice(i, i + size));
+        }
+        return chunks;
+    }
+    /* =========================================================
+       Integration
+       ========================================================= */
     return {
         name: "astro-indexnow",
         hooks: {
-            "astro:config:setup": ({ config }) => {
+            /* -----------------------------------------------
+               Setup
+               ----------------------------------------------- */
+            "astro:config:setup": ({ config, logger }) => {
                 site =
                     options.siteUrl ??
                         (config.site ? config.site.replace(/\/$/, "") : null);
+                logger.debug(`[astro-indexnow] project root: ${projectRoot}`);
+                ensureCacheFile(logger);
             },
+            /* -----------------------------------------------
+               Build done
+               ----------------------------------------------- */
             "astro:build:done": async ({ dir, logger }) => {
                 if (options.enabled === false) {
                     logger.info("[astro-indexnow] disabled");
                     return;
                 }
                 if (!options.key) {
-                    throw new Error("[astro-indexnow] Missing IndexNow key. Provide it in astro.config.mjs.");
+                    throw new Error("[astro-indexnow] Missing IndexNow key");
                 }
                 if (!site) {
-                    throw new Error("[astro-indexnow] Missing site URL. Set `site` in astro.config.mjs or pass `siteUrl`.");
+                    throw new Error("[astro-indexnow] Missing site URL");
                 }
-                const outDir = fileURLToPath(dir);
-                const urls = [];
+                ensureCacheFile(logger);
+                const outDir = new URL(dir).pathname;
+                const previousCache = loadCache(logger);
+                const nextCache = {};
+                const changedUrls = [];
                 function walk(currentDir) {
-                    const entries = fs.readdirSync(currentDir, {
-                        withFileTypes: true,
-                    });
-                    for (const entry of entries) {
+                    for (const entry of fs.readdirSync(currentDir, { withFileTypes: true })) {
                         const fullPath = path.join(currentDir, entry.name);
-                        if (entry.isDirectory()) {
+                        if (entry.isDirectory())
                             walk(fullPath);
-                        }
                         if (entry.isFile() && entry.name === "index.html") {
                             const relativePath = path
                                 .relative(outDir, fullPath)
                                 .replace(/index\.html$/, "")
                                 .replace(/\\/g, "/");
                             const url = site + "/" + relativePath.replace(/^\/+/, "");
-                            urls.push(url);
+                            const hash = hashFile(fullPath);
+                            nextCache[url] = hash;
+                            if (previousCache[url] !== hash) {
+                                changedUrls.push(url);
+                            }
                         }
                     }
                 }
                 walk(outDir);
-                logger.info("[astro-indexnow] detected pages:");
-                for (const url of urls) {
-                    logger.info(` - ${url}`);
+                logger.debug("[astro-indexnow] page diff:");
+                for (const url of Object.keys(nextCache)) {
+                    const state = previousCache[url] === nextCache[url]
+                        ? "unchanged"
+                        : "new/changed";
+                    logger.debug(` - ${url} (${state})`);
                 }
-                if (urls.length === 0) {
-                    logger.warn("[astro-indexnow] no pages detected, skipping submission");
+                if (changedUrls.length === 0) {
+                    logger.info("[astro-indexnow] no changed URLs detected, skipping submission");
+                    saveCache(logger, nextCache);
                     return;
                 }
-                // Warn if key file is missing (do not block)
-                const keyFilePath = path.join(outDir, `${options.key}.txt`);
-                if (!fs.existsSync(keyFilePath)) {
-                    logger.warn(`[astro-indexnow] Key file not found: /${options.key}.txt\n` +
-                        "IndexNow may reject submissions until this file exists.");
-                }
-                // IndexNow submission
-                try {
-                    const response = await fetch("https://api.indexnow.org/indexnow", {
-                        method: "POST",
-                        headers: {
-                            "Content-Type": "application/json",
-                        },
-                        body: JSON.stringify({
-                            host: new URL(site).host,
-                            key: options.key,
-                            keyLocation: `${site}/${options.key}.txt`,
-                            urlList: urls,
-                        }),
-                    });
-                    if (!response.ok) {
-                        logger.warn(`[astro-indexnow] IndexNow request failed (${response.status})`);
-                        return;
+                const batches = chunk(changedUrls, INDEXNOW_BATCH_SIZE);
+                logger.info(`[astro-indexnow] submitting ${changedUrls.length} changed URLs in ${batches.length} batch(es)`);
+                for (let i = 0; i < batches.length; i++) {
+                    const batch = batches[i];
+                    logger.debug(`[astro-indexnow] submitting batch ${i + 1}/${batches.length} (${batch.length} URLs)`);
+                    try {
+                        const response = await fetch(INDEXNOW_ENDPOINT, {
+                            method: "POST",
+                            headers: { "Content-Type": "application/json" },
+                            body: JSON.stringify({
+                                host: new URL(site).host,
+                                key: options.key,
+                                keyLocation: `${site}/${options.key}.txt`,
+                                urlList: batch,
+                            }),
+                        });
+                        if (!response.ok) {
+                            logger.warn(`[astro-indexnow] batch ${i + 1} failed (${response.status})`);
+                        }
+                    }
+                    catch {
+                        logger.warn(`[astro-indexnow] batch ${i + 1} submission failed (network error)`);
                     }
-                    logger.info(`[astro-indexnow] Successfully submitted ${urls.length} URLs to IndexNow`);
-                }
-                catch {
-                    logger.warn("[astro-indexnow] IndexNow submission failed (network error)");
                 }
+                saveCache(logger, nextCache);
+                logger.info(`[astro-indexnow] IndexNow submission complete`);
             },
         },
     };

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "astro-indexnow",
-  "version": "1.0.0",
+  "version": "2.0.0",
   "description": "Astro integration to submit pages to IndexNow automatically after build",
   "type": "module",
   "exports": {
@@ -33,7 +33,7 @@
     "type": "git",
     "url": "https://github.com/velohost/astro-indexnow"
   },
-  "homepage": "https://velohost.co.uk/tools/astro-indexnow",
+  "homepage": "https://velohost.co.uk/astro-indexnow",
   "bugs": {
     "url": "https://github.com/velohost/astro-indexnow/issues"
   },