@tryghost/algolia 0.2.10 → 0.3.0

package/LICENSE

@@ -1,6 +1,6 @@
 MIT License
 
-Copyright (c) 2013-2023 Ghost Foundation
+Copyright (c) 2013-2025 Ghost Foundation
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

package/README.md

@@ -1,70 +1,73 @@
 # Algolia Ghost CLI
 
-CLI tool to initially index the full Ghost post content into an Algolia index.
+`@tryghost/algolia` is a CLI for initially indexing the full published content of a Ghost site in Algolia.
+
+The CLI uses the Ghost 6 Content API. By default, it requests the [maximum 100 posts](https://docs.ghost.org/content-api/parameters#limit) at a time, follows Ghost's [`meta.pagination.next`](https://docs.ghost.org/content-api/pagination) value, and pauses briefly between pages until the full site has been fetched.
 
 ## Install
 
-`npm install @tryghost/algolia --save`
+Add the CLI to a project:
 
-or
+```sh
+npm install @tryghost/algolia
+```
 
-`yarn add @tryghost/algolia`
+or
 
+```sh
+pnpm add @tryghost/algolia
+```
 
 ## Usage
 
-To use the CLI, install the dependencies with `yarn install` or `npm install`.
+Copy [`example.config.json`](example.config.json) to a local file such as `config.json`, then set the Ghost Content API and Algolia credentials. `indexSettings` contains the current defaults and can be customized or removed.
 
-Copy the existing `example.config.json` to e. g. `config.json` and replace the relevant values for Ghost and Algolia.
-`indexSettings` reflects the current default settings and can either be overwritten, or removed from the config file.
+Configuration files contain secrets and should not be committed. Files matching `packages/algolia/config*.json` are ignored by this repository.
 
-To run the batch index, run
+After installing the package in another project, run its binary through that project's package runner:
 
-```bash
-yarn algolia index <pathToConfig> [options]
+```sh
+npx algolia index config.json [options]
 ```
 
-### Caveats
-
-The [Fragmenter](https://github.com/TryGhost/algolia/tree/master/packages/algolia-fragmenter) breaks down large HTML pieces into smaller chunks by its headings. Sometimes the fragment is still too big and Algolia will throw an error listing the post id that caused the large fragment. The post id can be used to get the post slug, which then can be excluded from the batch run like this:
+From this repository, invoke the entrypoint directly:
 
-```bash
-yarn algolia index <pathToConfig> -s post-slug-to-exclude,and-another-post-slug-to-exclude
+```sh
+cd packages/algolia
+node bin/cli.js index config.json [options]
 ```
 
-### Flags
-
-- `pathToConfig`, needs to be the relative (from this package) path to the config JSON file that contains the Ghost and Algolia API keys and settings (see [usage](#usage) above)
+### Options
 
-- `-s, --skip`, takes a comma separated list of post slugs that need to be **excluded** from the index (see [caveats](#caveats) above)
+- `pathToConfig` is the path, relative to the current directory, to the JSON configuration file.
+- `-s, --skip` excludes a comma-separated list of post slugs from the index.
+- `-V, --verbose` enables verbose output.
+- `-l, --limit` makes a single request for 1 to 100 posts instead of fetching every page.
+- `-p, --page` selects the page for that single request and requires `--limit`.
+- `-sjs, --skipjsonslugs` currently controls only a log message about `ignore_slugs`; it does not control exclusion. Configured `ignore_slugs` are always excluded after posts are fetched.
 
-- `-V, --verbose`, switches on verbose mode, but there's not much too see here (yet)
-- `-l, --limit`, limit the amount of posts to receive. Default is 'all'
-- `-p --page`, define the page to fetch posts from. To be used in combination with `limit`.
-- `-sjs --skipjsonslugs`, uses a list of slugs in `config.json` to skip before they're uploaded. This method will request all data from Ghost and skip at the point it would normally upload to Algolia. If you're getting `414 Request-URI Too Large` errors using `-s`, this is the method to use.
+### Large fragments
 
-## Develop
+The [Fragmenter](../algolia-fragmenter/README.md) splits large HTML strings by heading. A fragment can still exceed Algolia's record-size limit; the resulting error includes the post ID. Resolve the ID to a slug and exclude it from the batch:
 
-This is a mono repository, managed with [lerna](https://lernajs.io/).
-
-Follow the instructions for the top-level repo.
-1. `git clone` this repo & `cd` into it as usual
-2. Run `yarn` to install top-level dependencies.
-
-
-## Run
-
-- `yarn dev`
+```sh
+npx algolia index config.json --skip post-slug,another-post-slug
+```
 
+If a long `--skip` filter causes a `414 Request-URI Too Large` response, put the slugs in the configuration file's `ignore_slugs` array. The CLI always applies that array after fetching posts; `--skipjsonslugs` is not required and currently affects logging only.
 
-## Test
+## Development
 
-- `yarn lint` run just eslint
-- `yarn test` run lint and tests
+Install dependencies from the repository root with `pnpm install`. From the root, run this package's tests and ESLint checks with:
 
+```sh
+pnpm --filter @tryghost/algolia test
+```
 
+Run the full monorepo suite with `pnpm test`.
 
+---
 
-# Copyright & License
+## Copyright & License
 
-Copyright (c) 2013-2023 Ghost Foundation - Released under the [MIT license](LICENSE).
+Copyright (c) 2013-2026 Ghost Foundation - Released under the [MIT license](LICENSE). Ghost and the Ghost Logo are trademarks of Ghost Foundation Ltd. Please see our [trademark policy](https://ghost.org/trademark/) for info on acceptable usage.

package/bin/cli.js

@@ -3,6 +3,7 @@ const prettyCLI = require('@tryghost/pretty-cli');
 const ui = require('@tryghost/pretty-cli').ui;
 const fs = require('fs-extra');
 const utils = require('../lib/utils');
+const {fetchPosts} = require('../lib/fetch-posts');
 const GhostContentAPI = require('@tryghost/content-api');
 const transforms = require('@tryghost/algolia-fragmenter');
 const IndexFactory = require('@tryghost/algolia-indexer');
@@ -24,15 +25,28 @@ prettyCLI.command({
             desc: 'Comma separated list of post slugs to exclude from indexing'
         });
         sywac.number('-l --limit', {
-            desc: 'Amount of posts we want to fetch from Ghost'
+            desc: 'Fetch one page containing 1 to 100 posts'
         });
         sywac.number('-p --page', {
-            desc: 'Use page to navigate through posts when setting a limit'
+            desc: 'Select a page; requires --limit'
         });
         sywac.array('-sjs --skipjsonslugs', {
             defaultValue: false,
             desc: 'Exclude post slugs from config JSON file'
         });
+        sywac.check((argv, context) => {
+            if (argv.limit !== undefined && (!Number.isInteger(argv.limit) || argv.limit < 1 || argv.limit > 100)) {
+                context.cliMessage('--limit must be an integer from 1 to 100.');
+            }
+
+            if (argv.page !== undefined && argv.limit === undefined) {
+                context.cliMessage('--page requires --limit.');
+            }
+
+            if (argv.page !== undefined && (!Number.isInteger(argv.page) || argv.page < 1)) {
+                context.cliMessage('--page must be a positive integer.');
+            }
+        });
     },
     run: async (argv) => {
         const mainTimer = Date.now();
@@ -56,31 +70,31 @@ prettyCLI.command({
         // 2. Fetch all posts from the Ghost instance
         try {
             const timer = Date.now();
-            const params = {limit: 'all', include: 'tags,authors'};
+            const fetchOptions = {};
             const ghost = new GhostContentAPI({
                 url: context.ghost.apiUrl,
                 key: context.ghost.apiKey,
-                version: 'canary'
+                version: 'v6.0'
             });
 
             if (argv.skip && argv.skip.length > 0) {
                 const filterSlugs = argv.skip.join(',');
 
-                params.filter = `slug:-[${filterSlugs}]`;
+                fetchOptions.filter = `slug:-[${filterSlugs}]`;
             }
 
-            if (argv.limit) {
-                params.limit = argv.limit;
+            if (argv.limit !== undefined) {
+                fetchOptions.limit = argv.limit;
             }
 
-            ui.log.info(`Fetching ${params.limit} posts from Ghost...`);
+            ui.log.info(`Fetching ${argv.limit === undefined ? 'all' : argv.limit} posts from Ghost...`);
 
-            if (argv.page) {
+            if (argv.page !== undefined) {
                 ui.log.info(`...from page #${argv.page}.`);
-                params.page = argv.page;
+                fetchOptions.page = argv.page;
             }
 
-            context.posts = await ghost.posts.browse(params);
+            context.posts = await fetchPosts(ghost.posts.browse.bind(ghost.posts), fetchOptions);
 
             ui.log.info(`Done fetching posts in ${Date.now() - timer}ms.`);
         } catch (error) {

package/lib/fetch-posts.js

@@ -0,0 +1,73 @@
+const DEFAULT_PAGE_SIZE = 100;
+const PAGE_DELAY_MS = 100;
+
+const waitBetweenPages = () => {
+    return new Promise((resolve) => {
+        setTimeout(resolve, PAGE_DELAY_MS);
+    });
+};
+
+const getNextPage = (posts) => {
+    if (!posts.meta || !posts.meta.pagination || typeof posts.meta.pagination !== 'object') {
+        throw TypeError('Ghost returned posts without pagination metadata.');
+    }
+
+    const nextPage = posts.meta.pagination.next;
+
+    if (nextPage !== null && (!Number.isInteger(nextPage) || nextPage < 1)) {
+        throw TypeError('Ghost returned an invalid next page.');
+    }
+
+    return nextPage;
+};
+
+const validatePosts = (posts) => {
+    if (!Array.isArray(posts)) {
+        throw TypeError('Ghost returned posts in an invalid format.');
+    }
+};
+
+module.exports.fetchPosts = async (browsePosts, options = {}) => {
+    const hasExplicitLimit = options.limit !== undefined;
+    const params = {include: 'tags,authors'};
+
+    if (hasExplicitLimit) {
+        params.limit = options.limit;
+        if (options.page !== undefined) {
+            params.page = options.page;
+        }
+    } else {
+        params.limit = DEFAULT_PAGE_SIZE;
+        params.page = 1;
+    }
+
+    if (options.filter !== undefined) {
+        params.filter = options.filter;
+    }
+
+    const firstPage = await browsePosts(params);
+    validatePosts(firstPage);
+    const posts = [...firstPage];
+
+    if (hasExplicitLimit) {
+        return posts;
+    }
+
+    let nextPage = getNextPage(firstPage);
+    const fetchedPages = new Set([1]);
+
+    while (nextPage !== null) {
+        if (fetchedPages.has(nextPage)) {
+            throw TypeError('Ghost returned a repeated next page.');
+        }
+
+        fetchedPages.add(nextPage);
+        await waitBetweenPages();
+        const page = await browsePosts({...params, page: nextPage});
+        validatePosts(page);
+        posts.push(...page);
+        nextPage = getNextPage(page);
+    }
+
+    return posts;
+};

package/package.json

@@ -1,19 +1,19 @@
 {
   "name": "@tryghost/algolia",
-  "version": "0.2.10",
-  "repository": "https://github.com/TryGhost/algolia/tree/master/packages/algolia",
+  "version": "0.3.0",
+  "repository": {
+    "type": "git",
+    "url": "git+https://github.com/TryGhost/algolia.git",
+    "directory": "packages/algolia"
+  },
   "author": "Ghost Foundation",
   "license": "MIT",
+  "engines": {
+    "node": ">=24"
+  },
   "bin": {
     "algolia": "./bin/cli.js"
   },
-  "scripts": {
-    "dev": "algolia",
-    "algolia": "algolia",
-    "test": "NODE_ENV=testing mocha './test/**/*.test.js'",
-    "lint": "eslint . --ext .js --cache",
-    "posttest": "yarn lint"
-  },
   "files": [
     "bin",
     "index.js",
@@ -22,18 +22,19 @@
   "publishConfig": {
     "access": "public"
   },
-  "devDependencies": {
-    "mocha": "10.2.0",
-    "should": "13.2.3",
-    "sinon": "15.2.0"
-  },
   "dependencies": {
-    "@tryghost/algolia-fragmenter": "^0.2.6",
-    "@tryghost/algolia-indexer": "^0.3.0",
-    "@tryghost/content-api": "1.11.16",
-    "@tryghost/errors": "1.2.26",
-    "@tryghost/pretty-cli": "1.2.38",
-    "fs-extra": "11.1.1"
+    "@tryghost/content-api": "1.12.10",
+    "@tryghost/errors": "1.3.13",
+    "@tryghost/pretty-cli": "1.2.52",
+    "fs-extra": "11.4.0",
+    "@tryghost/algolia-fragmenter": "^0.2.8",
+    "@tryghost/algolia-indexer": "^0.3.2"
   },
-  "gitHead": "0f0d8a416adb0f78fda0ec72f872146406d76eae"
-}
+  "scripts": {
+    "dev": "algolia",
+    "algolia": "algolia",
+    "test": "NODE_ENV=testing vitest run --root .",
+    "lint": "eslint . --ext .js --cache",
+    "posttest": "pnpm lint"
+  }
+}