astro-relatinator 2.1.0 → 3.0.0

package/LICENSE.md

@@ -0,0 +1,9 @@
+MIT License
+
+Copyright (c) 2022 Darko Bozhinovski
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

package/README.md

@@ -1,88 +1,270 @@
-## Astro Integration for Relatinator
+# astro-relatinator
 
-An Astro integration for the [humble library for finding related posts and content](https://github.com/DBozhinovski/relatinator/tree/master/packages/relatinator).
+Find related entries in your Astro content collections.
+
+This package wraps [relatinator](https://github.com/DBozhinovski/relatinator/tree/master/packages/relatinator). It scores entries with BM25 (default) or TF-IDF.
+
+## Compatibility
+
+- Astro 6 and Astro 7. The `astro` peer dependency is `^6.0.0 || ^7.0.0`.
+- Node.js 22.12 or later.
+- Static output and on-demand rendering.
+- The Cloudflare adapter, also when the adapter prerenders pages in workerd.
+- EmDash 1.x (1.0 or later), for `astro-relatinator/emdash` only. The optional `emdash` peer dependency is `^1.0.0`.
+- ES modules only, with TypeScript type declarations.
+
+For the checks behind this list, read [Verification](https://github.com/DBozhinovski/relatinator/blob/master/docs/architecture.md#verification).
 
 ## Installation
 
-Assuming you're running Astro v3 or later, you can install this integration by running:
+```sh
+npx astro add astro-relatinator
+```
+
+You can also install the package without `astro add`:
 
+```sh
+npm install astro-relatinator
 ```
-npx astro add astro-relatinator
+
+The integration is optional. `getRelated()` works without an entry in `integrations`. [How it works](#how-it-works) gives the reason. The `astro add` command adds `relatinator()` to your config. You can keep it or remove it.
+
+```js
+// astro.config.mjs
+import { defineConfig } from "astro/config";
+import relatinator from "astro-relatinator";
+
+export default defineConfig({
+  integrations: [relatinator()],
+});
 ```
 
-OR
+## Usage
+
+Call `getRelated()` in the frontmatter of a page. Give it the collection name and an entry (or the id of an entry). It returns the related entries of the same collection, most related first.
+
+```astro
+---
+// src/pages/posts/[id].astro
+import { getCollection, render } from "astro:content";
+import { getRelated } from "astro-relatinator";
+
+export async function getStaticPaths() {
+  const posts = await getCollection("posts");
+  return posts.map((post) => ({ params: { id: post.id }, props: { post } }));
+}
+
+const { post } = Astro.props;
+const { Content } = await render(post);
+const related = await getRelated("posts", post, { limit: 3 });
+---
+
+<article>
+  <h1>{post.data.title}</h1>
+  <Content />
+</article>
 
+<h2>Related</h2>
+<ul>
+  {related.map((entry) => <li><a href={`/posts/${entry.id}/`}>{entry.data.title}</a></li>)}
+</ul>
 ```
+
+The result has the same type as `getCollection("posts")`, so TypeScript knows the type of `entry.data`.
+
+`getRelated()` throws an error if the collection has no entry with the given id.
+
+## Options
+
+| Option   | Type                  | Default    | Description                                                                 |
+| -------- | --------------------- | ---------- | --------------------------------------------------------------------------- |
+| `limit`  | `number`              | `5`        | The maximum number of related entries.                                      |
+| `method` | `"bm25"` or `"tfidf"` | `"bm25"`   | The scoring method.                                                         |
+| `fields` | `readonly string[]`   | all fields | The `data` keys to read. The body is always read.                           |
+| `text`   | `(entry) => string`   | none       | A function that returns the full text of an entry. It replaces the default. |
+
+The default text of an entry has two parts:
+
+1. All string and number values in `entry.data`, also in arrays and nested objects. `getRelated()` does not read dates, booleans, or `null`. When you set `fields`, it reads only those keys.
+2. The body of the entry (`entry.body`), when the entry has one.
+
+Examples:
+
+```ts
+// Only read the title and tags, plus the body.
+await getRelated("posts", post, { fields: ["title", "tags"] });
+
+// Use your own text for each entry.
+await getRelated("posts", post, {
+  text: (entry) => `${entry.data.title} ${entry.data.tags.join(" ")}`,
+});
+```
+
+## How it works
+
+`getRelated()` reads the collection with `getCollection()` each time that a page calls it. It does not read files when Astro starts, and it does not watch files.
+
+### The model cache
+
+`getRelated()` builds one model for each combination of collection, `method`, and `fields` (or `text`). Later calls use the same model while the collection does not change. Thus a build with many pages builds each model one time.
+
+To find changes, `getRelated()` compares the id and the `digest` of each entry. Entries from the built-in loaders have a digest. If an entry has no digest, it compares the text of the entry.
+
+When you use `text`, it compares the text that your function returns. Your function runs for each entry on each call.
+
+In `astro dev`, the related entries update when you edit a content file. You do not need to restart the server.
+
+### Why the integration is optional
+
+`getRelated()` imports `astro:content`, and only Vite can resolve this module. Astro processes `astro-relatinator` with Vite because the package has an `astro` peer dependency, the `astro` keyword, and a name that starts with `astro-`. Thus `getRelated()` works with or without the integration. For the details, read [The integration is optional](https://github.com/DBozhinovski/relatinator/blob/master/docs/architecture.md#the-integration-is-optional).
+
+## Limitations
+
+- With on-demand rendering, each Node.js process or Cloudflare Workers isolate builds its own model on the first call. The cache is in memory. A new process or isolate starts with an empty cache.
+- Each call reads the whole collection. With `text`, and on EmDash, each call also makes the text of every entry to find changes. The time for a call and for a model build grows with the size of the collection.
+- The cache keeps a model for each set of options that your pages use. It does not remove models.
+- The related entries come from the collection that you give. `getRelated()` does not compare entries across collections.
+- `getEmDashRelated()` ranks only published entries. [Which entries are compared](#which-entries-are-compared) tells how it uses a draft or a preview.
+- The tokenizer is for English text. [How scoring works](https://github.com/DBozhinovski/relatinator/tree/master/packages/relatinator#how-scoring-works) gives the steps.
+
+## EmDash
+
+[EmDash](https://github.com/emdash-cms/emdash) keeps content in a database, not in content collections. For EmDash sites, use `getEmDashRelated()` from `astro-relatinator/emdash`. It uses the same scoring and the same model cache as `getRelated()`.
+
+It needs `emdash` 1.x (1.0 or later). Install the package next to `emdash`:
+
+```sh
 npm install astro-relatinator
 ```
 
-## Configuration
+You do not need the integration in `astro.config.mjs`.
+
+### Usage
 
-The integration takes four input arguments, `paths`, `schema`, `similarityMethod`, and `debug`.
+Give `getEmDashRelated()` the collection slug and an entry (or the id of an entry). It returns the related entries, most related first.
 
-- `paths` is an array of paths to the `.md` or `.mdx` files you want to use for training the model. If you're using the [Content Collections API](https://docs.astro.build/en/guides/content-collections/), you can find these at `./src/collections/<collection-name>/`.
-- `schema` is an array of frontmatter fields you want to use for training the model.
-- `similarityMethod` is the algorithm to use for training the model. Can be `"tfidf"` or `"bm25"`.
-- `debug` is a boolean flag to enable debug mode. It gets VERY verbose, so use it with caution.
+```astro
+---
+// src/pages/posts/[slug].astro
+import { getEmDashEntry } from "emdash";
+import { getEmDashRelated } from "astro-relatinator/emdash";
+
+const { entry: post } = await getEmDashEntry("posts", Astro.params.slug);
+if (!post) return Astro.redirect("/404");
 
-### Example:
+const related = await getEmDashRelated("posts", post, { limit: 3 });
+---
+
+<ul>
+  {related.map((entry) => <li><a href={`/posts/${entry.id}`}>{entry.data.title}</a></li>)}
+</ul>
+```
+
+The result has the same type as the entry that you give. When you give an id, the type of `entry.data` is `Record<string, unknown>`. To set a different type, use `getEmDashRelated<typeof post>("posts", "my-slug")`.
+
+### EmDash options
+
+The options are the same as for `getRelated()`: `limit` (default `5`), `method` (default `"bm25"`), `fields`, and `text`.
+
+The default text of an entry has two parts:
+
+1. The string and number values of the content fields in `entry.data`. A Portable Text value becomes plain text. The plain text contains the text of each block, without keys, styles, and marks. It does not contain other block types, such as images.
+2. The labels of the taxonomy terms of the entry, such as tag and category names.
+
+The default text does not include the values that EmDash adds to each entry. These are `id`, `slug`, `status`, `authorId`, `primaryBylineId`, `createdAt`, `updatedAt`, `publishedAt`, `scheduledAt`, `draftRevisionId`, `liveRevisionId`, `locale`, `translationGroup`, `byline`, `bylines`, and `seo`. The default text also leaves out term ids and term slugs.
+
+When you set `fields`, the default text reads only those keys. It reads all of the values in those keys, including EmDash system values and all of the properties of terms. Portable Text in those keys still becomes plain text. When you set `text`, your function gives all of the text.
 
 ```ts
-export default defineConfig({
-  // ...
-  integrations: [
-    // ...
-    relatinatorIntegration({
-      paths: ["./src/content/posts"], // path yo your content files
-      schema: ["title", "descriptions", "tags"], // frontmatter fields to use for training
-      similarityMethod: "bm25", // use BM25 for training
-      debug: true, // enable debug mode
-    }),
-    // ...
-  ],
+// Only read the title, the excerpt, and the content.
+await getEmDashRelated("posts", post, {
+  fields: ["title", "excerpt", "content"],
 });
 ```
 
-**Note:** the fields you specify in the `schema` array must be present in the frontmatter of the files you're training on. If they're not, the integration will throw an error or lead to unexpected results.
+To convert Portable Text in your own `text` function, use `portableTextToPlainText()`:
 
-## Usage
+```ts
+import {
+  getEmDashRelated,
+  portableTextToPlainText,
+} from "astro-relatinator/emdash";
 
-The integration hooks into the Astro build (or dev) process, and will train the model based on the files you've provided. You can then use the utility functions exported by the package (`BM25Utils` or `TfIdfUtils`, depending on your chosen `similarityMethod`) to find related content and terms.
+await getEmDashRelated("posts", post, {
+  text: (entry) =>
+    `${entry.data.title} ${portableTextToPlainText(entry.data.content)}`,
+});
+```
+
+### Which entries are compared
+
+The model contains only the published entries of the collection. `getEmDashRelated()` reads all pages of the collection with `getEmDashCollection()`.
+
+EmDash uses the slug of an entry as its id. Thus a preview of a published entry has the id of the published entry, unless the draft changes the slug.
+
+- If you give the id of an entry that is not published, `getEmDashRelated()` throws an error.
+- If you give an entry object whose id is in the published entries, `getEmDashRelated()` uses the published text of that id. It does not read the text of the object. For a preview of a published entry, the related entries come from the published text, not from the preview text.
+- If you give an entry object whose id is not in the published entries, `getEmDashRelated()` uses the text of the object. An example is a draft of an entry that was never published. The result does not contain that id.
+- If EmDash returns an error, `getEmDashRelated()` throws an error that names the collection.
+
+### Caching on servers and Workers
 
-### Example:
+EmDash sites render on request. `getEmDashRelated()` reads the collection on each call. It keeps the model until an entry changes. To find changes, it compares the id and the text of each entry. [Limitations](#limitations) tells how the cache works on Node.js servers and on Cloudflare Workers.
+
+## Migrating from 2.x
+
+Version 3.0 removes the old API. Make these changes:
+
+1. Remove all options from `relatinator()` in your Astro config. In a type-checked config (`astro.config.ts`, or a file with `// @ts-check`), the 2.x options cause a type error. In a config without type checks, version 3.0 ignores the options and logs a warning. You can also remove the integration.
+2. Remove `BM25Utils` and `TfIdfUtils` from your pages. Call `getRelated()` instead.
+3. Use entry ids, not titles. Version 2.x used the `title` frontmatter field as the id. `getRelated()` uses `entry.id` and returns entries.
+4. Replace `schema` with `fields`. Replace `similarityMethod` with `method`. The `paths` option has no replacement, because `getRelated()` reads the collection with `getCollection()`.
+5. Note the new default method. Version 2.x used TF-IDF by default. Version 3.0 uses BM25. Set `method: "tfidf"` to keep the old method.
+
+In version 2.x, `findRelated` returns the `title` frontmatter field of each related entry, not the entry id. Thus a lookup by `post.id` finds no entries. This 2.x example finds the entries by title:
 
 ```astro
 ---
 import { getCollection } from "astro:content";
-// Import the utils corresponding to your configured similarityMethod
-// (BM25Utils in this example, matching the config example above)
 import { BM25Utils } from "astro-relatinator";
 
-const blogEntries = await getCollection("posts"); // assuming your collection is called posts
-// let's say we want to compare against the first blog entry
-const first = blogEntries[0];
+const posts = await getCollection("posts");
+const post = posts[0];
+// 2.x returns frontmatter titles, not entry ids.
+const titles = await BM25Utils.findRelated(`${post.data.title} ${post.body}`, post.data.title, 3);
+const related = titles.map((title) => posts.find((p) => p.data.title === title));
+---
+```
+
+In version 3.0, one call replaces this code:
 
-// Concatenate the same fields we used for training and declare how many related posts we want to find
-const related = await BM25Utils.findRelated(`${first.data.title} ${first.data.description} ${first.data.tags.join(' ')} ${first.body}`, first.id, 3)
+```astro
+---
+import { getCollection } from "astro:content";
+import { getRelated } from "astro-relatinator";
 
+const posts = await getCollection("posts");
+const related = await getRelated("posts", posts[0], { limit: 3 });
 ---
+```
 
-<div>
-<!-- render your related posts here -->
-{ related.map((id) => {
-  const post = blogEntries.find((post) => post.id === id);
-  if (!post) return null; // Handle case where post might not be found
-  return (
-    <a href={`/posts/${post.slug}/`}> {/* Adjust href based on your routing */}
-      <h2>{post.data.title}</h2>
-      <p>{post.data.description}</p>
-    </a>
-  );
-}) }
-</div>
+## Development
 
-```
+This package is part of a pnpm workspace. For the workspace commands, read [Development](https://github.com/DBozhinovski/relatinator#development) in the repository README. For the design decisions and the verification, read [docs/architecture.md](https://github.com/DBozhinovski/relatinator/blob/master/docs/architecture.md).
+
+Run these commands from the repository root:
+
+| Command                                        | What it does                                                                                                                 |
+| ---------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------- |
+| `pnpm --filter astro-relatinator build`        | Deletes `dist` and compiles `src` with `tsc`.                                                                                |
+| `pnpm --filter astro-relatinator test`         | Builds `relatinator` in `pretest`. Then it runs the Vitest tests. The tests replace `astro:content` and `emdash` with stubs. |
+| `pnpm --filter astro-relatinator check`        | Builds `relatinator` in `precheck`. Then it type-checks `src` and `test` with `tsc`.                                         |
+| `pnpm --filter astro-relatinator format:check` | Checks the format with Prettier. `format` writes the changes.                                                                |
+
+The tests do not run Astro. To test a change in a real Astro site, build the playground and compare its [expected related lists](https://github.com/DBozhinovski/relatinator/tree/master/packages/playground#expected-related-lists).
+
+The `prepack` script runs `build`. Thus `pnpm pack` and `pnpm publish` always include a new build.
 
-**Note:** The `BM25Utils` and `TfIdfUtils` objects also provide `getTopTermsForId` and `getTopRelatedDocumentsForTerm` functions, similar to the core `relatinator` library.
+## License
 
-**Development Server:** When running `astro dev`, the integration uses debouncing (a small delay) before re-training the model after a markdown file change. This improves performance for larger sites by preventing rapid, successive re-trains during saves.
+MIT

package/dist/emdash-text.d.ts

@@ -0,0 +1,7 @@
+import { type ContentEntry } from "./entry-text.js";
+/**
+ * Default text for an EmDash entry. Portable Text becomes plain text.
+ * Without `fields`, it reads the content fields and the labels of the
+ * taxonomy terms, but not the values that EmDash adds to every entry.
+ */
+export declare function emdashEntryText(entry: ContentEntry, fields?: readonly string[]): string;

package/dist/emdash-text.js

@@ -0,0 +1,45 @@
+import { entryText } from "./entry-text.js";
+import { portableTextValue } from "./portable-text.js";
+// Values that EmDash adds to `entry.data` next to the content fields
+// (emdash 1.2.0: loader.ts INCLUDE_IN_DATA and `slug`, `seo`; query.ts
+// hydrates `byline`, `bylines`, and `terms`).
+const SYSTEM_KEYS = new Set([
+    "id",
+    "slug",
+    "status",
+    "authorId",
+    "primaryBylineId",
+    "createdAt",
+    "updatedAt",
+    "publishedAt",
+    "scheduledAt",
+    "draftRevisionId",
+    "liveRevisionId",
+    "locale",
+    "translationGroup",
+    "byline",
+    "bylines",
+    "seo",
+    "terms",
+]);
+function termLabels(terms) {
+    if (typeof terms !== "object" || terms === null)
+        return [];
+    return Object.values(terms)
+        .flat()
+        .map((term) => term?.label)
+        .filter((label) => typeof label === "string");
+}
+function contentData(data) {
+    const content = Object.fromEntries(Object.entries(data).filter(([key]) => !SYSTEM_KEYS.has(key)));
+    return { ...content, terms: termLabels(data.terms) };
+}
+/**
+ * Default text for an EmDash entry. Portable Text becomes plain text.
+ * Without `fields`, it reads the content fields and the labels of the
+ * taxonomy terms, but not the values that EmDash adds to every entry.
+ */
+export function emdashEntryText(entry, fields) {
+    const data = fields ? entry.data : contentData(entry.data);
+    return entryText({ id: entry.id, data }, fields, portableTextValue);
+}

package/dist/emdash.d.ts

@@ -0,0 +1,17 @@
+import type { ContentEntry } from "./entry-text.js";
+import { portableTextToPlainText } from "./portable-text.js";
+import { type RelatedOptions } from "./related-index.js";
+export { portableTextToPlainText };
+/** The entry type when `getEmDashRelated` gets an id. */
+export interface EmDashEntry {
+    readonly id: string;
+    readonly data: Record<string, unknown>;
+}
+export type GetEmDashRelatedOptions<E> = RelatedOptions<E>;
+/**
+ * Finds the published entries of an EmDash `collection` that are most
+ * related to `entry`, in rank order. The model is built once and reused
+ * until an entry changes.
+ */
+export declare function getEmDashRelated<E extends ContentEntry>(collection: string, entry: E, options?: GetEmDashRelatedOptions<E>): Promise<E[]>;
+export declare function getEmDashRelated<E extends ContentEntry = EmDashEntry>(collection: string, entry: string, options?: GetEmDashRelatedOptions<E>): Promise<E[]>;

package/dist/emdash.js

@@ -0,0 +1,46 @@
+import { emdashEntryText } from "./emdash-text.js";
+import { portableTextToPlainText } from "./portable-text.js";
+import { relatedIndex } from "./related-index.js";
+export { portableTextToPlainText };
+const PAGE_SIZE = 100;
+async function getPublishedEntries(collection) {
+    // Imported on call so that astro.config can import this package.
+    const { getEmDashCollection } = await import("emdash");
+    const entries = [];
+    let cursor;
+    do {
+        const page = await getEmDashCollection(collection, {
+            status: "published",
+            limit: PAGE_SIZE,
+            cursor,
+        });
+        if (page.error) {
+            throw new Error(`astro-relatinator: cannot read EmDash collection "${collection}": ${page.error.message}`, { cause: page.error });
+        }
+        entries.push(...page.entries);
+        cursor = page.nextCursor;
+    } while (cursor);
+    return entries;
+}
+export async function getEmDashRelated(collection, entry, options = {}) {
+    const { text, fields } = options;
+    const textOf = text ?? ((e) => emdashEntryText(e, fields));
+    const index = relatedIndex({
+        key: `emdash\u0000${collection}`,
+        entries: await getPublishedEntries(collection),
+        textOf,
+        // EmDash keeps `updatedAt` when only the terms or bylines of an
+        // entry change, so the version is the text.
+        versionOf: textOf,
+    }, options);
+    return index.entries(rank(index.model, collection, entry, textOf, options.limit));
+}
+function rank(model, collection, entry, textOf, limit) {
+    const id = typeof entry === "string" ? entry : entry.id;
+    if (model.has(id))
+        return model.findRelated(id, { limit });
+    if (typeof entry === "string") {
+        throw new Error(`astro-relatinator: no published entry "${id}" in EmDash collection "${collection}"`);
+    }
+    return model.findRelatedToText(textOf(entry), { limit, exclude: [id] });
+}

package/dist/entry-text.d.ts

@@ -0,0 +1,18 @@
+/** The parts of an Astro or EmDash entry that relatinator reads. */
+export interface ContentEntry {
+    readonly id: string;
+    readonly data: object;
+    readonly body?: string;
+    readonly digest?: string;
+}
+/**
+ * Returns the text of a value that needs special handling, such as
+ * Portable Text, or `undefined` to read the value as plain data.
+ */
+type ValueText = (value: unknown) => string | undefined;
+/**
+ * Default text for an entry: the string and number values of its data
+ * (restricted to `fields` when given), followed by its body.
+ */
+export declare function entryText(entry: ContentEntry, fields?: readonly string[], valueText?: ValueText): string;
+export {};

package/dist/entry-text.js

@@ -0,0 +1,36 @@
+function isPlainObject(value) {
+    if (typeof value !== "object" || value === null)
+        return false;
+    const prototype = Object.getPrototypeOf(value);
+    return prototype === Object.prototype || prototype === null;
+}
+function collectText(value, out, valueText) {
+    const special = valueText?.(value);
+    if (special !== undefined) {
+        out.push(special);
+    }
+    else if (typeof value === "string" || typeof value === "number") {
+        out.push(String(value));
+    }
+    else if (Array.isArray(value)) {
+        for (const item of value)
+            collectText(item, out, valueText);
+    }
+    else if (isPlainObject(value)) {
+        for (const item of Object.values(value))
+            collectText(item, out, valueText);
+    }
+    return out;
+}
+/**
+ * Default text for an entry: the string and number values of its data
+ * (restricted to `fields` when given), followed by its body.
+ */
+export function entryText(entry, fields, valueText) {
+    const data = isPlainObject(entry.data) ? entry.data : {};
+    const values = fields ? fields.map((field) => data[field]) : data;
+    const parts = collectText(values, [], valueText);
+    if (entry.body)
+        parts.push(entry.body);
+    return parts.join(" ");
+}

package/dist/get-related.d.ts

@@ -0,0 +1,9 @@
+import type { CollectionEntry, CollectionKey } from "astro:content";
+import { type RelatedOptions } from "./related-index.js";
+export type GetRelatedOptions<E> = RelatedOptions<E>;
+/**
+ * Finds the entries of `collection` that are most related to `entry`,
+ * in rank order. The model is built once and reused until an entry
+ * changes.
+ */
+export declare function getRelated<C extends CollectionKey>(collection: C, entry: CollectionEntry<C> | string, options?: GetRelatedOptions<CollectionEntry<C>>): Promise<CollectionEntry<C>[]>;

package/dist/get-related.js

@@ -0,0 +1,25 @@
+import { entryText } from "./entry-text.js";
+import { relatedIndex } from "./related-index.js";
+/**
+ * Finds the entries of `collection` that are most related to `entry`,
+ * in rank order. The model is built once and reused until an entry
+ * changes.
+ */
+export async function getRelated(collection, entry, options = {}) {
+    const { text, fields } = options;
+    // Imported on call so that astro.config can import the integration.
+    const { getCollection } = await import("astro:content");
+    const entries = await getCollection(collection);
+    const textOf = text ?? ((e) => entryText(e, fields));
+    const index = relatedIndex({
+        key: String(collection),
+        entries,
+        textOf,
+        versionOf: (e) => (!text && e.digest ? e.digest : textOf(e)),
+    }, options);
+    const id = typeof entry === "string" ? entry : entry.id;
+    if (!index.model.has(id)) {
+        throw new Error(`astro-relatinator: no entry "${id}" in collection "${String(collection)}"`);
+    }
+    return index.entries(index.model.findRelated(id, { limit: options.limit }));
+}

package/dist/index.d.ts

@@ -0,0 +1,2 @@
+export { getRelated, type GetRelatedOptions } from "./get-related.js";
+export { default } from "./integration.js";

package/dist/index.js

@@ -0,0 +1,2 @@
+export { getRelated } from "./get-related.js";
+export { default } from "./integration.js";

package/dist/integration.d.ts

@@ -0,0 +1,6 @@
+import type { AstroIntegration } from "astro";
+/**
+ * Registers astro-relatinator with Astro. It takes no options. A 2.x
+ * options object only produces a warning.
+ */
+export default function relatinator(legacyOptions?: never): AstroIntegration;

package/dist/integration.js

@@ -0,0 +1,16 @@
+const LEGACY_OPTIONS_WARNING = "Integration options are ignored since astro-relatinator 3.0. Call getRelated() in your pages instead. See https://github.com/DBozhinovski/relatinator/tree/master/packages/astro-relatinator#migrating-from-2x";
+/**
+ * Registers astro-relatinator with Astro. It takes no options. A 2.x
+ * options object only produces a warning.
+ */
+export default function relatinator(legacyOptions) {
+    return {
+        name: "astro-relatinator",
+        hooks: {
+            "astro:config:setup": ({ logger }) => {
+                if (legacyOptions !== undefined)
+                    logger.warn(LEGACY_OPTIONS_WARNING);
+            },
+        },
+    };
+}

package/dist/model-cache.d.ts

@@ -0,0 +1,7 @@
+import type { RelatinatorModel } from "relatinator";
+/**
+ * Returns the model stored under `key` while its fingerprint matches.
+ * Otherwise builds a new model and replaces the entry, so the cache
+ * holds at most one model per key.
+ */
+export declare function getCachedModel(key: string, fingerprint: string, build: () => RelatinatorModel): RelatinatorModel;

package/dist/model-cache.js

@@ -0,0 +1,14 @@
+const cache = new Map();
+/**
+ * Returns the model stored under `key` while its fingerprint matches.
+ * Otherwise builds a new model and replaces the entry, so the cache
+ * holds at most one model per key.
+ */
+export function getCachedModel(key, fingerprint, build) {
+    const cached = cache.get(key);
+    if (cached?.fingerprint === fingerprint)
+        return cached.model;
+    const model = build();
+    cache.set(key, { fingerprint, model });
+    return model;
+}

package/dist/portable-text.d.ts

@@ -0,0 +1,8 @@
+/**
+ * Converts Portable Text to plain text: the span text of each `block`,
+ * one block per line. Other block types are skipped. A string is
+ * returned as it is.
+ */
+export declare function portableTextToPlainText(value: unknown): string;
+/** The plain text of a Portable Text value, or `undefined` for other values. */
+export declare function portableTextValue(value: unknown): string | undefined;

package/dist/portable-text.js

@@ -0,0 +1,29 @@
+function blockText({ _type, children }) {
+    if (_type !== "block" || !Array.isArray(children))
+        return undefined;
+    return children
+        .map((span) => typeof span?.text === "string" ? span.text : "")
+        .join("");
+}
+/**
+ * Converts Portable Text to plain text: the span text of each `block`,
+ * one block per line. Other block types are skipped. A string is
+ * returned as it is.
+ */
+export function portableTextToPlainText(value) {
+    if (typeof value === "string")
+        return value;
+    if (!Array.isArray(value))
+        return "";
+    return value
+        .map((block) => blockText(block ?? {}))
+        .filter((text) => text !== undefined)
+        .join("\n");
+}
+/** The plain text of a Portable Text value, or `undefined` for other values. */
+export function portableTextValue(value) {
+    const isPortableText = Array.isArray(value) &&
+        value.length > 0 &&
+        value.every((item) => typeof item?._type === "string");
+    return isPortableText ? portableTextToPlainText(value) : undefined;
+}

package/dist/related-index.d.ts

@@ -0,0 +1,36 @@
+import { type RelatinatorModel, type SimilarityMethod } from "relatinator";
+import type { ContentEntry } from "./entry-text.js";
+export interface RelatedOptions<E> {
+    /** Maximum number of related entries. Default: 5. */
+    limit?: number;
+    /** Scoring method. Default: "bm25". */
+    method?: SimilarityMethod;
+    /**
+     * Data keys to read. Default: all data. The body of an Astro entry is
+     * always read.
+     */
+    fields?: readonly string[];
+    /** Replaces the default text of each entry. */
+    text?: (entry: E) => string;
+}
+/** The entries to rank, and how to read them. */
+export interface Corpus<E> {
+    /** Names the corpus in the model cache. */
+    readonly key: string;
+    readonly entries: readonly E[];
+    readonly textOf: (entry: E) => string;
+    /** A value that changes when the text of the entry changes. */
+    readonly versionOf: (entry: E) => string;
+}
+export interface RelatedIndex<E> {
+    readonly model: RelatinatorModel;
+    /** The entries of the corpus with the given ids, in the same order. */
+    entries(ids: readonly string[]): E[];
+}
+/**
+ * The model of a corpus and a lookup from ids back to its entries. The
+ * model is built once and reused until the version of an entry changes.
+ * The fingerprint tells apart `text` functions that share their source
+ * code.
+ */
+export declare function relatedIndex<E extends ContentEntry>({ key, entries, textOf, versionOf }: Corpus<E>, { method, fields, text }: RelatedOptions<E>): RelatedIndex<E>;

package/dist/related-index.js

@@ -0,0 +1,26 @@
+import { createModel, } from "relatinator";
+import { getCachedModel } from "./model-cache.js";
+/**
+ * The model of a corpus and a lookup from ids back to its entries. The
+ * model is built once and reused until the version of an entry changes.
+ * The fingerprint tells apart `text` functions that share their source
+ * code.
+ */
+export function relatedIndex({ key, entries, textOf, versionOf }, { method = "bm25", fields, text }) {
+    const source = text ? `text:${text}` : `fields:${JSON.stringify(fields)}`;
+    const fingerprint = entries
+        .map((entry) => `${entry.id}\u0000${versionOf(entry)}`)
+        .join("\u0001");
+    const model = getCachedModel(`${key}\u0000${method}\u0000${source}`, fingerprint, () => createModel(entries.map((entry) => ({ id: entry.id, content: textOf(entry) })), { method }));
+    const byId = new Map(entries.map((entry) => [entry.id, entry]));
+    return {
+        model,
+        entries: (ids) => ids.map((id) => {
+            const entry = byId.get(id);
+            if (!entry) {
+                throw new Error(`astro-relatinator: internal error: ranked id "${id}" is not in the corpus`);
+            }
+            return entry;
+        }),
+    };
+}

package/package.json

@@ -1,44 +1,71 @@
 {
   "name": "astro-relatinator",
-  "version": "2.1.0",
-  "description": "Find related content in your Astro website or app",
+  "version": "3.0.0",
+  "description": "Find related entries in your Astro content collections",
   "author": "Darko Bozhinovski <dbozhinovski@gmail.com>",
   "license": "MIT",
   "keywords": [
     "astro",
+    "astro-integration",
+    "emdash",
     "related-content",
     "tfidf",
+    "bm25",
     "relatinator",
     "related",
     "words",
     "posts",
     "articles",
-    "static",
-    "astro-integration"
+    "static"
   ],
-  "files": [
-    "integration.ts"
-  ],
-  "exports": {
-    ".": "./integration.ts"
-  },
   "repository": {
     "type": "git",
     "url": "https://github.com/DBozhinovski/relatinator"
   },
   "type": "module",
-  "devDependencies": {
-    "@types/node": "^22.13.4",
-    "astro": ">=5",
-    "husky": "^9.1.7",
-    "vite": "^6.1.1"
+  "exports": {
+    ".": {
+      "types": "./dist/index.d.ts",
+      "import": "./dist/index.js"
+    },
+    "./emdash": {
+      "types": "./dist/emdash.d.ts",
+      "import": "./dist/emdash.js"
+    }
+  },
+  "files": [
+    "dist"
+  ],
+  "sideEffects": false,
+  "engines": {
+    "node": ">=22.12"
   },
   "dependencies": {
-    "glob": "^11.0.1",
-    "gray-matter": "^4.0.3",
-    "relatinator": "2.1.0"
+    "relatinator": "^3.0.0"
+  },
+  "devDependencies": {
+    "@types/node": "^26.6.5",
+    "astro": "7.3.8",
+    "prettier": "^3.9.10",
+    "typescript": "6.0.3",
+    "vitest": "^5.0.4"
   },
   "peerDependencies": {
-    "astro": ">=5"
+    "astro": "^6.0.0 || ^7.0.0",
+    "emdash": "^1.0.0"
+  },
+  "peerDependenciesMeta": {
+    "emdash": {
+      "optional": true
+    }
+  },
+  "scripts": {
+    "build": "node -e \"require('node:fs').rmSync('dist',{recursive:true,force:true})\" && tsc",
+    "pretest": "pnpm --filter \"astro-relatinator^...\" run build",
+    "test": "vitest run",
+    "precheck": "pnpm --filter \"astro-relatinator^...\" run build",
+    "check": "tsc --noEmit && tsc -p test",
+    "format": "prettier --write .",
+    "format:check": "prettier --check ."
   }
 }

package/integration.ts

@@ -1,170 +0,0 @@
-import type { AstroIntegration } from "astro";
-import path from "node:path";
-import { TfIdfUtils, BM25Utils } from "relatinator";
-import { glob } from "glob";
-import matter from "gray-matter";
-import fs from "node:fs/promises";
-import { RelatinatorDocument } from "relatinator";
-
-// Helper to write debug info to a file
-async function writeDebug(message: string) {
-  console.log("[astro-relatinator]", message);
-}
-
-async function trainModel(
-  similarityMethod: "tfidf" | "bm25",
-  paths: string[],
-  schema: string[],
-  debug = false
-) {
-  const documents: RelatinatorDocument[] = [];
-
-  if (debug) {
-    await writeDebug("[trainModel] Training TF-IDF on existing data...");
-  }
-
-  for (const path of paths) {
-    const markdownFiles = await glob(`${path}/**/*.{md,mdx}`);
-
-    if (debug) {
-      await writeDebug(
-        `[trainModel] Found ${markdownFiles.length} files in ${path}`
-      );
-    }
-
-    for (const file of markdownFiles) {
-      const content = await fs.readFile(file, "utf-8");
-      const parsed = matter(content);
-
-      // Use the document's title as the ID
-      const id = parsed.data.title;
-
-      let documentContent = "";
-      for (const key of schema) {
-        if (parsed.data[key]) {
-          documentContent += " " + stringifyData(parsed.data[key]);
-        }
-      }
-      documentContent += " " + parsed.content;
-
-      if (debug) {
-        await writeDebug(`[trainModel] Adding document with ID: ${id}`);
-      }
-
-      documents.push({
-        id,
-        content: documentContent,
-      });
-    }
-  }
-
-  if (debug) {
-    await writeDebug(
-      `[trainModel] Training with documents: ${documents
-        .map((d) => d.id)
-        .join(", ")}`
-    );
-  }
-
-  // Train the existing instance
-  if (similarityMethod === "tfidf") {
-    TfIdfUtils.train(documents, debug);
-  } else {
-    BM25Utils.train(documents, debug);
-  }
-}
-
-const stringifyData = (data: any): string => {
-  if (typeof data === "object") {
-    if (Array.isArray(data)) {
-      return data.map(stringifyData).join(" ");
-    } else {
-      return Object.values(data).map(stringifyData).join(" ");
-    }
-  } else {
-    return `${data}`;
-  }
-};
-
-const relatinatorIntegration = ({
-  paths,
-  schema,
-  similarityMethod = "tfidf",
-  debug = false,
-}: {
-  paths: string[];
-  schema: string[];
-  similarityMethod?: "tfidf" | "bm25";
-  debug?: boolean;
-}): AstroIntegration => {
-  return {
-    name: "astro-relatinator",
-    hooks: {
-      "astro:server:setup": async ({ server, logger }) => {
-        let debounceTimeout: NodeJS.Timeout | null = null;
-        const debounceMs = 500; // Debounce delay in milliseconds
-        logger.info("relatinator server:setup");
-
-        logger.info(
-          `Training Relatinator using ${similarityMethod} on existing data...`
-        );
-        if (similarityMethod === "tfidf") {
-          await TfIdfUtils.getInstance();
-        } else {
-          await BM25Utils.getInstance();
-        }
-        await trainModel(similarityMethod, paths, schema, debug);
-        logger.info("Initial training done.");
-
-        server.watcher.on("all", (eventName, filePath) => {
-          if (!["add", "change"].includes(eventName)) {
-            return;
-          }
-
-          if (path.extname(filePath).includes("md")) {
-            // Clear existing timer if there is one
-            if (debounceTimeout) {
-              clearTimeout(debounceTimeout);
-            }
-
-            // Set a new timer
-            debounceTimeout = setTimeout(async () => {
-              logger.info(
-                `Debounced: Markdown file changed: ${filePath}; Re-training ${similarityMethod}...`
-              );
-              try {
-                // Reset the instance before re-training
-                if (similarityMethod === "tfidf") {
-                  await TfIdfUtils.resetInstance();
-                } else {
-                  await BM25Utils.resetInstance();
-                }
-                await trainModel(similarityMethod, paths, schema, debug);
-                logger.info("Debounced re-training complete.");
-              } catch (error) {
-                logger.error(`Error during debounced re-training: ${error}`);
-              }
-            }, debounceMs);
-          }
-        });
-      },
-      "astro:build:setup": async ({ logger }) => {
-        logger.info("relatinator build:setup");
-
-        logger.info(
-          `Training Relatinator using ${similarityMethod} on existing data...`
-        );
-        if (similarityMethod === "tfidf") {
-          await TfIdfUtils.getInstance();
-        } else {
-          await BM25Utils.getInstance();
-        }
-        await trainModel(similarityMethod, paths, schema, debug);
-        logger.info(`Training done.`);
-      },
-    },
-  };
-};
-
-export { TfIdfUtils, BM25Utils };
-export default relatinatorIntegration;