@drewswiredin/backstage-plugin-assistants-backend-module-semantic-search 0.2.0 → 0.3.0

package/README.md

@@ -100,12 +100,53 @@ assistants:
 On Azure OpenAI, deploy an embedding model and name the deployment:
 `embeddingModel: azure:<embedding-deployment>`.
 
+### Rank by source
+
+Every source shares one index, so a large one (thousands of tickets that all
+mention the same system) can crowd out the one runbook section that answers
+the question. Name the search document types to weigh them:
+
+```yaml
+assistants:
+  semanticSearch:
+    embeddingModel: openai:text-embedding-3-small
+    maxResults: 8
+    types:
+      software-catalog: { weight: 1.5 }
+      techdocs: { weight: 1.2 }
+      confluence: { weight: 1.0 }
+      jira:
+        weight: 0.5
+        maxResults: 2
+        description: ticket history, decisions and status
+```
+
+The keys are the type names Backstage search uses: `software-catalog`,
+`techdocs`, or a collator's own type. Types left out are copied and ranked
+with weight 1. Set `default: false` to make a source opt-in: the model then
+searches it only when it passes `types: ['jira']`, and the tool description
+says so. Set `enabled: false` to stop copying a source; its chunks are
+deleted on its next search run.
+
 The index fills as search runs: each document type arrives when its collator
 next runs, on the schedule you gave that collator. A source's first run embeds
 all of it; later runs embed only new and changed sections and delete removed
 ones. Changing `embeddingModel` empties the index, and each type refills on its
 next search run.
 
+A type added to `types` takes its name, weight, and metadata on its next run,
+keeping the embeddings it already has; 0.2.0 indexes upgrade the same way. To
+run a collator now rather than wait for its schedule, trigger its search task
+with a user or service token:
+
+```bash
+curl -X POST -H "Authorization: Bearer $TOKEN" \
+  https://backstage.example.com/api/search/.backstage/scheduler/v1/tasks/search_index_jira/trigger
+```
+
+The task id is `search_index_<type>`; the backend logs `Knowledge index
+refreshed` with the type and counts when the copy finishes.
+
 ## Give an assistant the tool
 
 Add `search-knowledge` to the assistant's tools in the assistant editor. It
@@ -149,17 +190,24 @@ document gated by a catalog entity (its `authorization.resourceRef`) keeps that
 entity for the permission check; a document gated by any other kind of
 resource is skipped, since the index can only check entity read permission.
 
-A run does not say which document type it is, so the index recognizes a source
-by its locations: the run joins the stored collection that already holds most
-of its locations, and that collection's chunks the run no longer has are
-deleted. A collection no search run has refreshed in 30 days (a source removed
-from search, or one whose locations all changed at once) is dropped. Each row stores the embedding, a generated full-text vector, and a
-trigram index over its text. The table is a rebuildable copy, not migrated
+Search never tells a decorator which document type it is decorating, so the
+module adds a pass-through decorator to each type named in `types`. Search
+builds a run's pipeline in one pass, so that decorator names the run's type.
+A named type's run is stored under its name; a run of any other type joins
+the stored collection that already holds most of its locations. Chunks the
+run no longer has are deleted. A collection no search run has refreshed in
+30 days (a source removed from search, or an unnamed one whose locations all
+changed at once) is dropped. Each row stores the embedding, a generated
+full-text vector, a trigram index over its text, its type, and the short
+fields its collator attached. The table is a rebuildable copy, not migrated
 data.
 
-**Retrieval.** `search-knowledge` takes the question and up to three alternative
-phrasings and builds one semantic list per phrasing, a keyword list, a rare-word
-list, and a fuzzy list:
+**Retrieval.** `search-knowledge` takes the question, up to three alternative
+phrasings, and optionally the `types` to search (otherwise every type that is
+on by default, plus unnamed sources). It builds one semantic list per
+phrasing, a keyword list, a rare-word list, and a fuzzy list, each keeping
+its best 30 chunks of every type, so a large source cannot fill a list on its
+own:
 
 | List      | Matches                                       | Postgres                                                    |
 | --------- | --------------------------------------------- | ----------------------------------------------------------- |
@@ -171,18 +219,71 @@ list, and a fuzzy list:
 Query words include adjacent pairs written as one word, so "tally up" also
 matches "TallyUp". Reciprocal rank fusion merges the lists with one vote per
 method: the semantic lists share one vote, so extra phrasings broaden recall
-without outvoting an exact match. The rare-word list keeps a name or acronym
-mentioned once in a paragraph about something else from being outranked by
-sections that are only near the question in meaning. The top candidates'
-entities are fetched from the catalog as the user; results for entities the
-user cannot read are dropped, results with no entity are kept, and repeated
-text is shown once.
+without outvoting an exact match. Each chunk's fused score is multiplied by
+its type's weight. The rare-word list keeps a name or acronym mentioned once
+in a paragraph about something else from being outranked by sections that
+are only near the question in meaning. The top candidates' entities are
+fetched from the catalog as the user; results for entities the user cannot
+read are dropped, results with no entity are kept, and repeated text is shown
+once. The ranker then picks the results: by default best first, with at most
+`maxResults` of a capped type while other types can fill the slots.
+
+Each result carries its title, location, type, excerpt, the entity ref for
+catalog and TechDocs results, and `metadata`: the short fields its collator
+attached (strings up to 200 characters, numbers, booleans), such as a
+ticket's status or a page's last update.
+
+## Custom ranking
+
+A backend module can replace the ranker, for example to favour recently
+updated tickets:
+
+```ts
+import { createBackendModule } from '@backstage/backend-plugin-api';
+import { knowledgeRankingExtensionPoint } from '@drewswiredin/backstage-plugin-assistants-backend-module-semantic-search/alpha';
+
+export default createBackendModule({
+  pluginId: 'assistants',
+  moduleId: 'knowledge-recency',
+  register(env) {
+    env.registerInit({
+      deps: { ranking: knowledgeRankingExtensionPoint },
+      async init({ ranking }) {
+        ranking.setRanker(({ candidates, maxResults }) =>
+          candidates
+            .map(candidate => {
+              const updated = Date.parse(String(candidate.metadata?.updated));
+              const ageDays = (Date.now() - updated) / 86_400_000;
+              const boost = ageDays < 90 ? 1.2 : 1;
+              return { candidate, score: candidate.score * boost };
+            })
+            .sort((a, b) => b.score - a.score)
+            .slice(0, maxResults)
+            .map(({ candidate }) => candidate),
+        );
+      },
+    });
+  },
+});
+```
+
+The ranker receives the query, its phrasings, the types searched, and up to
+60 candidates the caller may read. Each candidate has its fused score after
+weights, its rank in each retrieval list, its type, entity ref, and metadata.
+It runs after the permission check, so it cannot surface what the caller
+cannot read; results it returns that were not candidates are dropped.
 
 ## Reference
 
-| Key                                        | Default    | Meaning                                             |
-| ------------------------------------------ | ---------- | --------------------------------------------------- |
-| `assistants.semanticSearch.embeddingModel` | (required) | `<providerId>:<model>` from `assistants.providers`. |
+| Key                                                  | Default    | Meaning                                                                       |
+| ---------------------------------------------------- | ---------- | ----------------------------------------------------------------------------- |
+| `assistants.semanticSearch.embeddingModel`           | (required) | `<providerId>:<model>` from `assistants.providers`.                           |
+| `assistants.semanticSearch.maxResults`               | `8`        | Results per answer unless the model asks for a number (it may ask for 20).    |
+| `assistants.semanticSearch.types.<type>.weight`      | `1`        | Multiplies the type's fused score; `0` never returns it.                      |
+| `assistants.semanticSearch.types.<type>.default`     | `true`     | Searched when the model names no types; `false` makes the type opt-in.        |
+| `assistants.semanticSearch.types.<type>.maxResults`  |            | At most this many of the type's results while other types can fill the slots. |
+| `assistants.semanticSearch.types.<type>.enabled`     | `true`     | `false` stops copying the type and deletes its chunks on its next run.        |
+| `assistants.semanticSearch.types.<type>.description` |            | What the source holds, shown to the model in the tool description.            |
 
 The index refreshes when search does; set each collator's schedule in its own
 config.

package/config.schema.json

@@ -11,6 +11,39 @@
             "embeddingModel": {
               "type": "string",
               "description": "Embedding model in `<providerId>:<model>` format, resolved against `assistants.providers`, e.g. `openai:text-embedding-3-small`. The provider must be `openai`, `azure`, or `openai-compatible`. Changing it rebuilds the index."
+            },
+            "maxResults": {
+              "type": "number",
+              "description": "Results per `search-knowledge` answer unless the caller asks for a number. Defaults to 8."
+            },
+            "types": {
+              "type": "object",
+              "additionalProperties": {
+                "type": "object",
+                "properties": {
+                  "weight": {
+                    "type": "number",
+                    "description": "Multiplies the type's fused score. Defaults to 1; 0 means its chunks are never returned."
+                  },
+                  "default": {
+                    "type": "boolean",
+                    "description": "Searched when the caller names no types. Defaults to true; false makes the type opt-in (the model passes it in `types`)."
+                  },
+                  "maxResults": {
+                    "type": "number",
+                    "description": "At most this many of the type's results in one answer."
+                  },
+                  "enabled": {
+                    "type": "boolean",
+                    "description": "Copied into the index. Defaults to true; false stops copying it and deletes its chunks on its next search run."
+                  },
+                  "description": {
+                    "type": "string",
+                    "description": "What the source holds, shown to the model in the tool description, e.g. \"ticket history, decisions and status\"."
+                  }
+                }
+              },
+              "description": "Settings per Backstage search document type, keyed by the type name search uses (`software-catalog`, `techdocs`, or a collator's own type). Types left out are copied and ranked with the defaults."
             }
           },
           "required": [

package/dist/action.cjs.js

@@ -2,9 +2,11 @@
 
 var ai = require('ai');
 var rank = require('./rank.cjs.js');
+var ranking = require('./ranking.cjs.js');
 
 const CANDIDATES = 30;
-const FUSED = 40;
+const FUSED = 60;
+const MAX_LIMIT = 20;
 const EXCERPT_CHARS = 1500;
 const RARE_MAX_CHUNKS = 3;
 function registerSearchKnowledgeAction({
@@ -12,13 +14,17 @@ function registerSearchKnowledgeAction({
   store,
   catalog,
   auth,
-  embeddingModel
+  embeddingModel,
+  types,
+  maxResults,
+  ranker
 }) {
   actionsRegistry.register({
     name: "search-knowledge",
     title: "Search Knowledge",
     attributes: { destructive: false, readOnly: true, idempotent: true },
-    description: `Search everything the organization's Backstage search indexes (TechDocs, the software catalog, and any other source it collects, such as Confluence or wiki pages) by meaning, not only by exact words. Best for questions in the user's own words: it finds a page even when the question is worded differently from it ("fix errored messages" finds "Retry dead-letter messages") and tolerates misspelled names ("Pilon" finds "Pylon"). It returns the most relevant sections, not every match, and cannot filter by entity kind or source.
+    description: `Search everything the organization's Backstage search indexes (TechDocs, the software catalog, and any other source it collects, such as Confluence or wiki pages) by meaning, not only by exact words. Best for questions in the user's own words: it finds a page even when the question is worded differently from it ("fix errored messages" finds "Retry dead-letter messages") and tolerates misspelled names ("Pilon" finds "Pylon"). It returns the most relevant sections, not every match, and cannot filter by entity kind.
+${sourcesDescription(types)}
 
 How to search the organization's docs and catalog:
 1. Start here, with the user's question.
@@ -27,19 +33,27 @@ How to search the organization's docs and catalog:
 
 Pass the user's question as the "query" parameter. Pass up to three "alternatives": other phrasings, synonyms, or the internal terms the docs might use.
 
-Each result has a title (its heading path), an excerpt, a location, and, for catalog and TechDocs results, the entity ref. A location starting with /docs/ is a TechDocs page: pass it to read-techdocs to read the full page. Any other location is a link to the page or entity; cite it.`,
+Each result has a title (its heading path), an excerpt, a location, its source type, the fields the source attached (such as status, space, or last update: use them to judge how current it is), and, for catalog and TechDocs results, the entity ref. A location starting with /docs/ is a TechDocs page: pass it to read-techdocs to read the full page. Any other location is a link to the page or entity; cite it.`,
     schema: {
       input: (z) => z.object({
         query: z.string().min(1).describe("The question or topic to find"),
         alternatives: z.array(z.string()).max(3).optional().describe("Other phrasings or terms for the same thing"),
-        limit: z.number().int().min(1).max(20).optional().default(8).describe("Number of results to return")
+        types: z.array(z.string()).optional().describe(
+          "Search only these sources (search document types); omit to search the default ones"
+        ),
+        limit: z.number().int().min(1).max(Math.max(MAX_LIMIT, maxResults)).optional().default(maxResults).describe("Number of results to return")
       }),
       output: (z) => z.object({
         results: z.array(
           z.object({
             title: z.string(),
             location: z.string(),
+            type: z.string().optional(),
             entityRef: z.string().optional(),
+            metadata: z.record(
+              z.string(),
+              z.union([z.string(), z.number(), z.boolean()])
+            ).optional(),
             excerpt: z.string()
           })
         )
@@ -51,12 +65,17 @@ Each result has a title (its heading path), an excerpt, a location, and, for cat
           "The knowledge index is still being built; meanwhile use keyword search: the query tool when it is available, otherwise search-techdocs and search-catalog."
         );
       }
+      const filter = ranking.selectTypes(types, await store.types(), input.types);
       const phrasings = [input.query, ...input.alternatives ?? []];
       const words = rank.queryWords(phrasings);
       const joined = rank.joinedWords(phrasings).filter((w) => !words.includes(w));
       const [{ embeddings }, frequencies] = await Promise.all([
         ai.embedMany({ model: embeddingModel, values: phrasings }),
-        store.documentFrequencies([...words, ...joined], RARE_MAX_CHUNKS + 1)
+        store.documentFrequencies(
+          [...words, ...joined],
+          RARE_MAX_CHUNKS + 1,
+          filter
+        )
       ]);
       const known = [...words, ...joined].filter(
         (w) => (frequencies.get(w) ?? 0) > 0
@@ -65,48 +84,80 @@ Each result has a title (its heading path), an excerpt, a location, and, for cat
       const unknown = words.filter((w) => frequencies.get(w) === 0);
       const lists = await Promise.all([
         ...embeddings.map(async (embedding) => ({
-          ids: await store.nearest(embedding, CANDIDATES),
+          name: "semantic",
+          hits: await store.nearest(embedding, CANDIDATES, filter),
           weight: 1 / embeddings.length
         })),
-        store.keyword(known, CANDIDATES).then((ids) => ({ ids, weight: 1 })),
-        store.keyword(rare, CANDIDATES).then((ids) => ({ ids, weight: 1 })),
-        store.fuzzy(unknown, CANDIDATES).then((ids) => ({ ids, weight: 1 }))
+        store.keyword(known, CANDIDATES, filter).then((hits) => ({ name: "keyword", hits, weight: 1 })),
+        store.keyword(rare, CANDIDATES, filter).then((hits) => ({ name: "rareWord", hits, weight: 1 })),
+        store.fuzzy(unknown, CANDIDATES, filter).then((hits) => ({ name: "fuzzy", hits, weight: 1 }))
       ]);
-      const ranked = rank.fuseRankings(lists).slice(0, FUSED);
+      const settingsOf = ranking.typeSettingsOf(types);
+      const fused = rank.fuseRankings(lists, (type) => settingsOf(type).weight).slice(
+        0,
+        FUSED
+      );
       const byId = new Map(
-        (await store.chunks(ranked)).map((chunk) => [chunk.id, chunk])
+        (await store.chunks(fused.map((hit) => hit.id))).map((chunk) => [
+          chunk.id,
+          chunk
+        ])
       );
-      const candidates = ranked.map((id) => byId.get(id)).filter((chunk) => Boolean(chunk));
+      const found = fused.flatMap((hit) => {
+        const chunk = byId.get(hit.id);
+        return chunk ? [{ hit, chunk }] : [];
+      });
       const readable = await readableEntityRefs(catalog, auth, credentials, [
         ...new Set(
-          candidates.flatMap(
-            (chunk) => chunk.entityRef ? [chunk.entityRef] : []
+          found.flatMap(
+            ({ chunk }) => chunk.entityRef ? [chunk.entityRef] : []
           )
         )
       ]);
       const seen = /* @__PURE__ */ new Set();
-      const results = [];
-      for (const chunk of candidates) {
+      const candidates = [];
+      for (const { hit, chunk } of found) {
         const key = `${chunk.location}|${chunk.title}`;
         if (chunk.entityRef && !readable.has(chunk.entityRef) || seen.has(key) || seen.has(chunk.content)) {
           continue;
         }
         seen.add(key);
         seen.add(chunk.content);
-        results.push({
-          title: chunk.title,
-          location: chunk.location,
-          ...chunk.entityRef ? { entityRef: chunk.entityRef } : {},
-          excerpt: chunk.content.slice(0, EXCERPT_CHARS)
-        });
-        if (results.length === input.limit) {
-          break;
-        }
+        candidates.push({ ...chunk, score: hit.score, ranks: hit.ranks });
       }
+      const ids = new Set(candidates.map((candidate) => candidate.id));
+      const ranked = await ranker()({
+        query: input.query,
+        phrasings,
+        types: filter.types,
+        candidates,
+        maxResults: input.limit
+      });
+      const results = ranked.filter((candidate) => ids.delete(candidate.id)).slice(0, input.limit).map((candidate) => ({
+        title: candidate.title,
+        location: candidate.location,
+        ...candidate.type ? { type: candidate.type } : {},
+        ...candidate.entityRef ? { entityRef: candidate.entityRef } : {},
+        ...candidate.metadata ? { metadata: candidate.metadata } : {},
+        excerpt: candidate.content.slice(0, EXCERPT_CHARS)
+      }));
       return { output: { results } };
     }
   });
 }
+function sourcesDescription(types) {
+  const listed = [...types].filter(
+    ([, settings]) => settings.enabled && settings.weight > 0
+  );
+  const lines = listed.map(
+    ([type, settings]) => `- ${type}${settings.default ? "" : ' (only when named in "types")'}${settings.description ? `: ${settings.description}` : ""}`
+  );
+  return [
+    "",
+    `Sources: pass "types" to search only the named search document types; without it the search covers every source except those marked as searched only when named. An unknown name returns the list of types.`,
+    ...lines
+  ].join("\n");
+}
 async function readableEntityRefs(catalog, auth, credentials, entityRefs) {
   if (entityRefs.length === 0) {
     return /* @__PURE__ */ new Set();
@@ -123,4 +174,5 @@ async function readableEntityRefs(catalog, auth, credentials, entityRefs) {
 }
 
 exports.registerSearchKnowledgeAction = registerSearchKnowledgeAction;
+exports.sourcesDescription = sourcesDescription;
 //# sourceMappingURL=action.cjs.js.map

package/dist/action.cjs.js.map

@@ -1 +1 @@
-{"version":3,"file":"action.cjs.js","sources":["../src/action.ts"],"sourcesContent":["import type {\n  AuthService,\n  BackstageCredentials,\n} from '@backstage/backend-plugin-api';\nimport type { ActionsRegistryService } from '@backstage/backend-plugin-api/alpha';\nimport type { CatalogApi } from '@backstage/catalog-client';\nimport { embedMany, type EmbeddingModel } from 'ai';\nimport { fuseRankings, joinedWords, queryWords, type RankedList } from './rank';\nimport type { KnowledgeStore, StoredChunk } from './store';\n\n/** Candidates per retrieval list, before fusion and permission filtering. */\nconst CANDIDATES = 30;\n/** Fused candidates checked for permission; over-fetched so filtering rarely empties the page. */\nconst FUSED = 40;\nconst EXCERPT_CHARS = 1500;\n/** A word in this many chunks or fewer is rare: strong evidence of what the user means. */\nconst RARE_MAX_CHUNKS = 3;\n\nexport interface SearchKnowledgeDeps {\n  actionsRegistry: ActionsRegistryService;\n  store: KnowledgeStore;\n  catalog: CatalogApi;\n  auth: AuthService;\n  embeddingModel: EmbeddingModel;\n}\n\n/**\n * Registers `search-knowledge`: hybrid retrieval over the knowledge index. Each\n * phrasing is embedded and matched by meaning; the query words are matched\n * exactly (full-text, any word) and, for words absent from the index, by\n * near spelling (trigram). Words found in only a few chunks get a list of\n * their own, so a rare name or acronym buried in a paragraph about something\n * else is not outvoted by sections that are merely near the question.\n *\n * The lists are fused by reciprocal rank with one vote per method: the\n * phrasings share the meaning vote, so extra alternatives broaden recall\n * without drowning out exact matches. Results gated by a catalog entity are\n * filtered to entities the caller can read; the rest are visible to every\n * assistant user, as they are in Backstage search. Repeated text is shown\n * once.\n */\nexport function registerSearchKnowledgeAction({\n  actionsRegistry,\n  store,\n  catalog,\n  auth,\n  embeddingModel,\n}: SearchKnowledgeDeps): void {\n  actionsRegistry.register({\n    name: 'search-knowledge',\n    title: 'Search Knowledge',\n    attributes: { destructive: false, readOnly: true, idempotent: true },\n    description: `Search everything the organization's Backstage search indexes (TechDocs, the software catalog, and any other source it collects, such as Confluence or wiki pages) by meaning, not only by exact words. Best for questions in the user's own words: it finds a page even when the question is worded differently from it (\"fix errored messages\" finds \"Retry dead-letter messages\") and tolerates misspelled names (\"Pilon\" finds \"Pylon\"). It returns the most relevant sections, not every match, and cannot filter by entity kind or source.\n\nHow to search the organization's docs and catalog:\n1. Start here, with the user's question.\n2. If the results don't clearly answer, also run a keyword search: the query tool (catalog and TechDocs at once) when it is available, otherwise search-techdocs and search-catalog. Keyword search finds rare names, acronyms and jargon that ranking by meaning can miss. For a bare name, term or acronym, run both searches from the start.\n3. Report something as missing only after both searches come back empty.\n\nPass the user's question as the \"query\" parameter. Pass up to three \"alternatives\": other phrasings, synonyms, or the internal terms the docs might use.\n\nEach result has a title (its heading path), an excerpt, a location, and, for catalog and TechDocs results, the entity ref. A location starting with /docs/ is a TechDocs page: pass it to read-techdocs to read the full page. Any other location is a link to the page or entity; cite it.`,\n    schema: {\n      input: z =>\n        z.object({\n          query: z.string().min(1).describe('The question or topic to find'),\n          alternatives: z\n            .array(z.string())\n            .max(3)\n            .optional()\n            .describe('Other phrasings or terms for the same thing'),\n          limit: z\n            .number()\n            .int()\n            .min(1)\n            .max(20)\n            .optional()\n            .default(8)\n            .describe('Number of results to return'),\n        }),\n      output: z =>\n        z.object({\n          results: z.array(\n            z.object({\n              title: z.string(),\n              location: z.string(),\n              entityRef: z.string().optional(),\n              excerpt: z.string(),\n            }),\n          ),\n        }),\n    },\n    action: async ({ input, credentials }) => {\n      if (!(await store.isReady())) {\n        throw new Error(\n          'The knowledge index is still being built; meanwhile use keyword search: the query tool when it is available, otherwise search-techdocs and search-catalog.',\n        );\n      }\n\n      const phrasings = [input.query, ...(input.alternatives ?? [])];\n      const words = queryWords(phrasings);\n      const joined = joinedWords(phrasings).filter(w => !words.includes(w));\n      const [{ embeddings }, frequencies] = await Promise.all([\n        embedMany({ model: embeddingModel, values: phrasings }),\n        store.documentFrequencies([...words, ...joined], RARE_MAX_CHUNKS + 1),\n      ]);\n      const known = [...words, ...joined].filter(\n        w => (frequencies.get(w) ?? 0) > 0,\n      );\n      const rare = known.filter(w => frequencies.get(w)! <= RARE_MAX_CHUNKS);\n      const unknown = words.filter(w => frequencies.get(w) === 0);\n\n      const lists: RankedList[] = await Promise.all([\n        ...embeddings.map(async embedding => ({\n          ids: await store.nearest(embedding, CANDIDATES),\n          weight: 1 / embeddings.length,\n        })),\n        store.keyword(known, CANDIDATES).then(ids => ({ ids, weight: 1 })),\n        store.keyword(rare, CANDIDATES).then(ids => ({ ids, weight: 1 })),\n        store.fuzzy(unknown, CANDIDATES).then(ids => ({ ids, weight: 1 })),\n      ]);\n\n      const ranked = fuseRankings(lists).slice(0, FUSED);\n      const byId = new Map(\n        (await store.chunks(ranked)).map(chunk => [chunk.id, chunk]),\n      );\n      const candidates = ranked\n        .map(id => byId.get(id))\n        .filter((chunk): chunk is StoredChunk => Boolean(chunk));\n\n      const readable = await readableEntityRefs(catalog, auth, credentials, [\n        ...new Set(\n          candidates.flatMap(chunk =>\n            chunk.entityRef ? [chunk.entityRef] : [],\n          ),\n        ),\n      ]);\n\n      const seen = new Set<string>();\n      const results = [];\n      for (const chunk of candidates) {\n        const key = `${chunk.location}|${chunk.title}`;\n        if (\n          (chunk.entityRef && !readable.has(chunk.entityRef)) ||\n          seen.has(key) ||\n          seen.has(chunk.content)\n        ) {\n          continue;\n        }\n        seen.add(key);\n        seen.add(chunk.content);\n        results.push({\n          title: chunk.title,\n          location: chunk.location,\n          ...(chunk.entityRef ? { entityRef: chunk.entityRef } : {}),\n          excerpt: chunk.content.slice(0, EXCERPT_CHARS),\n        });\n        if (results.length === input.limit) {\n          break;\n        }\n      }\n      return { output: { results } };\n    },\n  });\n}\n\n/**\n * The subset of entity refs the caller can read, asked of the catalog as the\n * caller — so the catalog's own permission policy decides, and a deleted\n * entity's leftover chunks never surface.\n */\nasync function readableEntityRefs(\n  catalog: CatalogApi,\n  auth: AuthService,\n  credentials: BackstageCredentials,\n  entityRefs: string[],\n): Promise<Set<string>> {\n  if (entityRefs.length === 0) {\n    return new Set();\n  }\n  const { token } = await auth.getPluginRequestToken({\n    onBehalfOf: credentials,\n    targetPluginId: 'catalog',\n  });\n  const { items } = await catalog.getEntitiesByRefs(\n    { entityRefs, fields: ['kind', 'metadata.namespace', 'metadata.name'] },\n    { token },\n  );\n  return new Set(entityRefs.filter((_, i) => items[i]));\n}\n"],"names":["queryWords","joinedWords","embedMany","fuseRankings"],"mappings":";;;;;AAWA,MAAM,UAAA,GAAa,EAAA;AAEnB,MAAM,KAAA,GAAQ,EAAA;AACd,MAAM,aAAA,GAAgB,IAAA;AAEtB,MAAM,eAAA,GAAkB,CAAA;AAyBjB,SAAS,6BAAA,CAA8B;AAAA,EAC5C,eAAA;AAAA,EACA,KAAA;AAAA,EACA,OAAA;AAAA,EACA,IAAA;AAAA,EACA;AACF,CAAA,EAA8B;AAC5B,EAAA,eAAA,CAAgB,QAAA,CAAS;AAAA,IACvB,IAAA,EAAM,kBAAA;AAAA,IACN,KAAA,EAAO,kBAAA;AAAA,IACP,YAAY,EAAE,WAAA,EAAa,OAAO,QAAA,EAAU,IAAA,EAAM,YAAY,IAAA,EAAK;AAAA,IACnE,WAAA,EAAa,CAAA;;AAAA;AAAA;AAAA;AAAA;;AAAA;;AAAA,2RAAA,CAAA;AAAA,IAUb,MAAA,EAAQ;AAAA,MACN,KAAA,EAAO,CAAA,CAAA,KACL,CAAA,CAAE,MAAA,CAAO;AAAA,QACP,KAAA,EAAO,EAAE,MAAA,EAAO,CAAE,IAAI,CAAC,CAAA,CAAE,SAAS,+BAA+B,CAAA;AAAA,QACjE,YAAA,EAAc,CAAA,CACX,KAAA,CAAM,CAAA,CAAE,MAAA,EAAQ,CAAA,CAChB,GAAA,CAAI,CAAC,CAAA,CACL,QAAA,EAAS,CACT,SAAS,6CAA6C,CAAA;AAAA,QACzD,OAAO,CAAA,CACJ,MAAA,GACA,GAAA,EAAI,CACJ,IAAI,CAAC,CAAA,CACL,GAAA,CAAI,EAAE,EACN,QAAA,EAAS,CACT,QAAQ,CAAC,CAAA,CACT,SAAS,6BAA6B;AAAA,OAC1C,CAAA;AAAA,MACH,MAAA,EAAQ,CAAA,CAAA,KACN,CAAA,CAAE,MAAA,CAAO;AAAA,QACP,SAAS,CAAA,CAAE,KAAA;AAAA,UACT,EAAE,MAAA,CAAO;AAAA,YACP,KAAA,EAAO,EAAE,MAAA,EAAO;AAAA,YAChB,QAAA,EAAU,EAAE,MAAA,EAAO;AAAA,YACnB,SAAA,EAAW,CAAA,CAAE,MAAA,EAAO,CAAE,QAAA,EAAS;AAAA,YAC/B,OAAA,EAAS,EAAE,MAAA;AAAO,WACnB;AAAA;AACH,OACD;AAAA,KACL;AAAA,IACA,MAAA,EAAQ,OAAO,EAAE,KAAA,EAAO,aAAY,KAAM;AACxC,MAAA,IAAI,CAAE,MAAM,KAAA,CAAM,OAAA,EAAQ,EAAI;AAC5B,QAAA,MAAM,IAAI,KAAA;AAAA,UACR;AAAA,SACF;AAAA,MACF;AAEA,MAAA,MAAM,SAAA,GAAY,CAAC,KAAA,CAAM,KAAA,EAAO,GAAI,KAAA,CAAM,YAAA,IAAgB,EAAG,CAAA;AAC7D,MAAA,MAAM,KAAA,GAAQA,gBAAW,SAAS,CAAA;AAClC,MAAA,MAAM,MAAA,GAASC,gBAAA,CAAY,SAAS,CAAA,CAAE,MAAA,CAAO,OAAK,CAAC,KAAA,CAAM,QAAA,CAAS,CAAC,CAAC,CAAA;AACpE,MAAA,MAAM,CAAC,EAAE,UAAA,EAAW,EAAG,WAAW,CAAA,GAAI,MAAM,QAAQ,GAAA,CAAI;AAAA,QACtDC,aAAU,EAAE,KAAA,EAAO,cAAA,EAAgB,MAAA,EAAQ,WAAW,CAAA;AAAA,QACtD,KAAA,CAAM,oBAAoB,CAAC,GAAG,OAAO,GAAG,MAAM,CAAA,EAAG,eAAA,GAAkB,CAAC;AAAA,OACrE,CAAA;AACD,MAAA,MAAM,QAAQ,CAAC,GAAG,KAAA,EAAO,GAAG,MAAM,CAAA,CAAE,MAAA;AAAA,QAClC,CAAA,CAAA,KAAA,CAAM,WAAA,CAAY,GAAA,CAAI,CAAC,KAAK,CAAA,IAAK;AAAA,OACnC;AACA,MAAA,MAAM,IAAA,GAAO,MAAM,MAAA,CAAO,CAAA,CAAA,KAAK,YAAY,GAAA,CAAI,CAAC,KAAM,eAAe,CAAA;AACrE,MAAA,MAAM,OAAA,GAAU,MAAM,MAAA,CAAO,CAAA,CAAA,KAAK,YAAY,GAAA,CAAI,CAAC,MAAM,CAAC,CAAA;AAE1D,MAAA,MAAM,KAAA,GAAsB,MAAM,OAAA,CAAQ,GAAA,CAAI;AAAA,QAC5C,GAAG,UAAA,CAAW,GAAA,CAAI,OAAM,SAAA,MAAc;AAAA,UACpC,GAAA,EAAK,MAAM,KAAA,CAAM,OAAA,CAAQ,WAAW,UAAU,CAAA;AAAA,UAC9C,MAAA,EAAQ,IAAI,UAAA,CAAW;AAAA,SACzB,CAAE,CAAA;AAAA,QACF,KAAA,CAAM,OAAA,CAAQ,KAAA,EAAO,UAAU,CAAA,CAAE,IAAA,CAAK,CAAA,GAAA,MAAQ,EAAE,GAAA,EAAK,MAAA,EAAQ,CAAA,EAAE,CAAE,CAAA;AAAA,QACjE,KAAA,CAAM,OAAA,CAAQ,IAAA,EAAM,UAAU,CAAA,CAAE,IAAA,CAAK,CAAA,GAAA,MAAQ,EAAE,GAAA,EAAK,MAAA,EAAQ,CAAA,EAAE,CAAE,CAAA;AAAA,QAChE,KAAA,CAAM,KAAA,CAAM,OAAA,EAAS,UAAU,CAAA,CAAE,IAAA,CAAK,CAAA,GAAA,MAAQ,EAAE,GAAA,EAAK,MAAA,EAAQ,CAAA,EAAE,CAAE;AAAA,OAClE,CAAA;AAED,MAAA,MAAM,SAASC,iBAAA,CAAa,KAAK,CAAA,CAAE,KAAA,CAAM,GAAG,KAAK,CAAA;AACjD,MAAA,MAAM,OAAO,IAAI,GAAA;AAAA,QAAA,CACd,MAAM,KAAA,CAAM,MAAA,CAAO,MAAM,CAAA,EAAG,GAAA,CAAI,CAAA,KAAA,KAAS,CAAC,KAAA,CAAM,EAAA,EAAI,KAAK,CAAC;AAAA,OAC7D;AACA,MAAA,MAAM,UAAA,GAAa,MAAA,CAChB,GAAA,CAAI,CAAA,EAAA,KAAM,KAAK,GAAA,CAAI,EAAE,CAAC,CAAA,CACtB,MAAA,CAAO,CAAC,KAAA,KAAgC,OAAA,CAAQ,KAAK,CAAC,CAAA;AAEzD,MAAA,MAAM,QAAA,GAAW,MAAM,kBAAA,CAAmB,OAAA,EAAS,MAAM,WAAA,EAAa;AAAA,QACpE,GAAG,IAAI,GAAA;AAAA,UACL,UAAA,CAAW,OAAA;AAAA,YAAQ,WACjB,KAAA,CAAM,SAAA,GAAY,CAAC,KAAA,CAAM,SAAS,IAAI;AAAC;AACzC;AACF,OACD,CAAA;AAED,MAAA,MAAM,IAAA,uBAAW,GAAA,EAAY;AAC7B,MAAA,MAAM,UAAU,EAAC;AACjB,MAAA,KAAA,MAAW,SAAS,UAAA,EAAY;AAC9B,QAAA,MAAM,MAAM,CAAA,EAAG,KAAA,CAAM,QAAQ,CAAA,CAAA,EAAI,MAAM,KAAK,CAAA,CAAA;AAC5C,QAAA,IACG,MAAM,SAAA,IAAa,CAAC,QAAA,CAAS,GAAA,CAAI,MAAM,SAAS,CAAA,IACjD,IAAA,CAAK,GAAA,CAAI,GAAG,CAAA,IACZ,IAAA,CAAK,GAAA,CAAI,KAAA,CAAM,OAAO,CAAA,EACtB;AACA,UAAA;AAAA,QACF;AACA,QAAA,IAAA,CAAK,IAAI,GAAG,CAAA;AACZ,QAAA,IAAA,CAAK,GAAA,CAAI,MAAM,OAAO,CAAA;AACtB,QAAA,OAAA,CAAQ,IAAA,CAAK;AAAA,UACX,OAAO,KAAA,CAAM,KAAA;AAAA,UACb,UAAU,KAAA,CAAM,QAAA;AAAA,UAChB,GAAI,MAAM,SAAA,GAAY,EAAE,WAAW,KAAA,CAAM,SAAA,KAAc,EAAC;AAAA,UACxD,OAAA,EAAS,KAAA,CAAM,OAAA,CAAQ,KAAA,CAAM,GAAG,aAAa;AAAA,SAC9C,CAAA;AACD,QAAA,IAAI,OAAA,CAAQ,MAAA,KAAW,KAAA,CAAM,KAAA,EAAO;AAClC,UAAA;AAAA,QACF;AAAA,MACF;AACA,MAAA,OAAO,EAAE,MAAA,EAAQ,EAAE,OAAA,EAAQ,EAAE;AAAA,IAC/B;AAAA,GACD,CAAA;AACH;AAOA,eAAe,kBAAA,CACb,OAAA,EACA,IAAA,EACA,WAAA,EACA,UAAA,EACsB;AACtB,EAAA,IAAI,UAAA,CAAW,WAAW,CAAA,EAAG;AAC3B,IAAA,2BAAW,GAAA,EAAI;AAAA,EACjB;AACA,EAAA,MAAM,EAAE,KAAA,EAAM,GAAI,MAAM,KAAK,qBAAA,CAAsB;AAAA,IACjD,UAAA,EAAY,WAAA;AAAA,IACZ,cAAA,EAAgB;AAAA,GACjB,CAAA;AACD,EAAA,MAAM,EAAE,KAAA,EAAM,GAAI,MAAM,OAAA,CAAQ,iBAAA;AAAA,IAC9B,EAAE,UAAA,EAAY,MAAA,EAAQ,CAAC,MAAA,EAAQ,oBAAA,EAAsB,eAAe,CAAA,EAAE;AAAA,IACtE,EAAE,KAAA;AAAM,GACV;AACA,EAAA,OAAO,IAAI,GAAA,CAAI,UAAA,CAAW,MAAA,CAAO,CAAC,GAAG,CAAA,KAAM,KAAA,CAAM,CAAC,CAAC,CAAC,CAAA;AACtD;;;;"}
+{"version":3,"file":"action.cjs.js","sources":["../src/action.ts"],"sourcesContent":["import type {\n  AuthService,\n  BackstageCredentials,\n} from '@backstage/backend-plugin-api';\nimport type { ActionsRegistryService } from '@backstage/backend-plugin-api/alpha';\nimport type { CatalogApi } from '@backstage/catalog-client';\nimport { embedMany, type EmbeddingModel } from 'ai';\nimport type { KnowledgeTypeSettings } from './config';\nimport { fuseRankings, joinedWords, queryWords, type RankedList } from './rank';\nimport {\n  selectTypes,\n  typeSettingsOf,\n  type KnowledgeCandidate,\n  type KnowledgeRanker,\n} from './ranking';\nimport type { KnowledgeStore } from './store';\n\n/** Candidates per type in each retrieval list, before fusion and permission filtering. */\nconst CANDIDATES = 30;\n/** Fused candidates checked for permission and ranked; over-fetched so filtering and type caps rarely empty the page. */\nconst FUSED = 60;\n/** The most results a caller may ask for, unless `maxResults` is configured higher. */\nconst MAX_LIMIT = 20;\nconst EXCERPT_CHARS = 1500;\n/** A word in this many chunks or fewer is rare: strong evidence of what the user means. */\nconst RARE_MAX_CHUNKS = 3;\n\nexport interface SearchKnowledgeDeps {\n  actionsRegistry: ActionsRegistryService;\n  store: KnowledgeStore;\n  catalog: CatalogApi;\n  auth: AuthService;\n  embeddingModel: EmbeddingModel;\n  types: Map<string, KnowledgeTypeSettings>;\n  maxResults: number;\n  /** The ranker in effect: the default, or one a module set. */\n  ranker: () => KnowledgeRanker;\n}\n\n/**\n * Registers `search-knowledge`: hybrid retrieval over the knowledge index. Each\n * phrasing is embedded and matched by meaning; the query words are matched\n * exactly (full-text, any word) and, for words absent from the index, by\n * near spelling (trigram). Words found in only a few chunks get a list of\n * their own, so a rare name or acronym buried in a paragraph about something\n * else is not outvoted by sections that are merely near the question.\n *\n * Each list keeps its best chunks of every document type, so a large source\n * cannot fill a list on its own. The lists are fused by reciprocal rank with\n * one vote per method, each chunk's score multiplied by its type's weight:\n * the phrasings share the meaning vote, so extra alternatives broaden recall\n * without drowning out exact matches. Results gated by a catalog entity are\n * filtered to entities the caller can read; the rest are visible to every\n * assistant user, as they are in Backstage search. Repeated text is shown\n * once, and the ranker picks the results from what remains.\n */\nexport function registerSearchKnowledgeAction({\n  actionsRegistry,\n  store,\n  catalog,\n  auth,\n  embeddingModel,\n  types,\n  maxResults,\n  ranker,\n}: SearchKnowledgeDeps): void {\n  actionsRegistry.register({\n    name: 'search-knowledge',\n    title: 'Search Knowledge',\n    attributes: { destructive: false, readOnly: true, idempotent: true },\n    description: `Search everything the organization's Backstage search indexes (TechDocs, the software catalog, and any other source it collects, such as Confluence or wiki pages) by meaning, not only by exact words. Best for questions in the user's own words: it finds a page even when the question is worded differently from it (\"fix errored messages\" finds \"Retry dead-letter messages\") and tolerates misspelled names (\"Pilon\" finds \"Pylon\"). It returns the most relevant sections, not every match, and cannot filter by entity kind.\n${sourcesDescription(types)}\n\nHow to search the organization's docs and catalog:\n1. Start here, with the user's question.\n2. If the results don't clearly answer, also run a keyword search: the query tool (catalog and TechDocs at once) when it is available, otherwise search-techdocs and search-catalog. Keyword search finds rare names, acronyms and jargon that ranking by meaning can miss. For a bare name, term or acronym, run both searches from the start.\n3. Report something as missing only after both searches come back empty.\n\nPass the user's question as the \"query\" parameter. Pass up to three \"alternatives\": other phrasings, synonyms, or the internal terms the docs might use.\n\nEach result has a title (its heading path), an excerpt, a location, its source type, the fields the source attached (such as status, space, or last update: use them to judge how current it is), and, for catalog and TechDocs results, the entity ref. A location starting with /docs/ is a TechDocs page: pass it to read-techdocs to read the full page. Any other location is a link to the page or entity; cite it.`,\n    schema: {\n      input: z =>\n        z.object({\n          query: z.string().min(1).describe('The question or topic to find'),\n          alternatives: z\n            .array(z.string())\n            .max(3)\n            .optional()\n            .describe('Other phrasings or terms for the same thing'),\n          types: z\n            .array(z.string())\n            .optional()\n            .describe(\n              'Search only these sources (search document types); omit to search the default ones',\n            ),\n          limit: z\n            .number()\n            .int()\n            .min(1)\n            .max(Math.max(MAX_LIMIT, maxResults))\n            .optional()\n            .default(maxResults)\n            .describe('Number of results to return'),\n        }),\n      output: z =>\n        z.object({\n          results: z.array(\n            z.object({\n              title: z.string(),\n              location: z.string(),\n              type: z.string().optional(),\n              entityRef: z.string().optional(),\n              metadata: z\n                .record(\n                  z.string(),\n                  z.union([z.string(), z.number(), z.boolean()]),\n                )\n                .optional(),\n              excerpt: z.string(),\n            }),\n          ),\n        }),\n    },\n    action: async ({ input, credentials }) => {\n      if (!(await store.isReady())) {\n        throw new Error(\n          'The knowledge index is still being built; meanwhile use keyword search: the query tool when it is available, otherwise search-techdocs and search-catalog.',\n        );\n      }\n\n      const filter = selectTypes(types, await store.types(), input.types);\n      const phrasings = [input.query, ...(input.alternatives ?? [])];\n      const words = queryWords(phrasings);\n      const joined = joinedWords(phrasings).filter(w => !words.includes(w));\n      const [{ embeddings }, frequencies] = await Promise.all([\n        embedMany({ model: embeddingModel, values: phrasings }),\n        store.documentFrequencies(\n          [...words, ...joined],\n          RARE_MAX_CHUNKS + 1,\n          filter,\n        ),\n      ]);\n      const known = [...words, ...joined].filter(\n        w => (frequencies.get(w) ?? 0) > 0,\n      );\n      const rare = known.filter(w => frequencies.get(w)! <= RARE_MAX_CHUNKS);\n      const unknown = words.filter(w => frequencies.get(w) === 0);\n\n      const lists: RankedList[] = await Promise.all([\n        ...embeddings.map(async embedding => ({\n          name: 'semantic' as const,\n          hits: await store.nearest(embedding, CANDIDATES, filter),\n          weight: 1 / embeddings.length,\n        })),\n        store\n          .keyword(known, CANDIDATES, filter)\n          .then(hits => ({ name: 'keyword' as const, hits, weight: 1 })),\n        store\n          .keyword(rare, CANDIDATES, filter)\n          .then(hits => ({ name: 'rareWord' as const, hits, weight: 1 })),\n        store\n          .fuzzy(unknown, CANDIDATES, filter)\n          .then(hits => ({ name: 'fuzzy' as const, hits, weight: 1 })),\n      ]);\n\n      const settingsOf = typeSettingsOf(types);\n      const fused = fuseRankings(lists, type => settingsOf(type).weight).slice(\n        0,\n        FUSED,\n      );\n      const byId = new Map(\n        (await store.chunks(fused.map(hit => hit.id))).map(chunk => [\n          chunk.id,\n          chunk,\n        ]),\n      );\n      const found = fused.flatMap(hit => {\n        const chunk = byId.get(hit.id);\n        return chunk ? [{ hit, chunk }] : [];\n      });\n\n      const readable = await readableEntityRefs(catalog, auth, credentials, [\n        ...new Set(\n          found.flatMap(({ chunk }) =>\n            chunk.entityRef ? [chunk.entityRef] : [],\n          ),\n        ),\n      ]);\n\n      const seen = new Set<string>();\n      const candidates: KnowledgeCandidate[] = [];\n      for (const { hit, chunk } of found) {\n        const key = `${chunk.location}|${chunk.title}`;\n        if (\n          (chunk.entityRef && !readable.has(chunk.entityRef)) ||\n          seen.has(key) ||\n          seen.has(chunk.content)\n        ) {\n          continue;\n        }\n        seen.add(key);\n        seen.add(chunk.content);\n        candidates.push({ ...chunk, score: hit.score, ranks: hit.ranks });\n      }\n\n      const ids = new Set(candidates.map(candidate => candidate.id));\n      const ranked = await ranker()({\n        query: input.query,\n        phrasings,\n        types: filter.types,\n        candidates,\n        maxResults: input.limit,\n      });\n      const results = ranked\n        .filter(candidate => ids.delete(candidate.id))\n        .slice(0, input.limit)\n        .map(candidate => ({\n          title: candidate.title,\n          location: candidate.location,\n          ...(candidate.type ? { type: candidate.type } : {}),\n          ...(candidate.entityRef ? { entityRef: candidate.entityRef } : {}),\n          ...(candidate.metadata ? { metadata: candidate.metadata } : {}),\n          excerpt: candidate.content.slice(0, EXCERPT_CHARS),\n        }));\n      return { output: { results } };\n    },\n  });\n}\n\n/**\n * The tool description's paragraph on sources: the configured types, which\n * ones are searched only when named, and what each holds.\n */\nexport function sourcesDescription(\n  types: Map<string, KnowledgeTypeSettings>,\n): string {\n  const listed = [...types].filter(\n    ([, settings]) => settings.enabled && settings.weight > 0,\n  );\n  const lines = listed.map(\n    ([type, settings]) =>\n      `- ${type}${settings.default ? '' : ' (only when named in \"types\")'}${\n        settings.description ? `: ${settings.description}` : ''\n      }`,\n  );\n  return [\n    '',\n    `Sources: pass \"types\" to search only the named search document types; without it the search covers every source except those marked as searched only when named. An unknown name returns the list of types.`,\n    ...lines,\n  ].join('\\n');\n}\n\n/**\n * The subset of entity refs the caller can read, asked of the catalog as the\n * caller — so the catalog's own permission policy decides, and a deleted\n * entity's leftover chunks never surface.\n */\nasync function readableEntityRefs(\n  catalog: CatalogApi,\n  auth: AuthService,\n  credentials: BackstageCredentials,\n  entityRefs: string[],\n): Promise<Set<string>> {\n  if (entityRefs.length === 0) {\n    return new Set();\n  }\n  const { token } = await auth.getPluginRequestToken({\n    onBehalfOf: credentials,\n    targetPluginId: 'catalog',\n  });\n  const { items } = await catalog.getEntitiesByRefs(\n    { entityRefs, fields: ['kind', 'metadata.namespace', 'metadata.name'] },\n    { token },\n  );\n  return new Set(entityRefs.filter((_, i) => items[i]));\n}\n"],"names":["selectTypes","queryWords","joinedWords","embedMany","typeSettingsOf","fuseRankings"],"mappings":";;;;;;AAkBA,MAAM,UAAA,GAAa,EAAA;AAEnB,MAAM,KAAA,GAAQ,EAAA;AAEd,MAAM,SAAA,GAAY,EAAA;AAClB,MAAM,aAAA,GAAgB,IAAA;AAEtB,MAAM,eAAA,GAAkB,CAAA;AA+BjB,SAAS,6BAAA,CAA8B;AAAA,EAC5C,eAAA;AAAA,EACA,KAAA;AAAA,EACA,OAAA;AAAA,EACA,IAAA;AAAA,EACA,cAAA;AAAA,EACA,KAAA;AAAA,EACA,UAAA;AAAA,EACA;AACF,CAAA,EAA8B;AAC5B,EAAA,eAAA,CAAgB,QAAA,CAAS;AAAA,IACvB,IAAA,EAAM,kBAAA;AAAA,IACN,KAAA,EAAO,kBAAA;AAAA,IACP,YAAY,EAAE,WAAA,EAAa,OAAO,QAAA,EAAU,IAAA,EAAM,YAAY,IAAA,EAAK;AAAA,IACnE,WAAA,EAAa,CAAA;AAAA,EACf,kBAAA,CAAmB,KAAK,CAAC;;AAAA;AAAA;AAAA;AAAA;;AAAA;;AAAA,yZAAA,CAAA;AAAA,IAUvB,MAAA,EAAQ;AAAA,MACN,KAAA,EAAO,CAAA,CAAA,KACL,CAAA,CAAE,MAAA,CAAO;AAAA,QACP,KAAA,EAAO,EAAE,MAAA,EAAO,CAAE,IAAI,CAAC,CAAA,CAAE,SAAS,+BAA+B,CAAA;AAAA,QACjE,YAAA,EAAc,CAAA,CACX,KAAA,CAAM,CAAA,CAAE,MAAA,EAAQ,CAAA,CAChB,GAAA,CAAI,CAAC,CAAA,CACL,QAAA,EAAS,CACT,SAAS,6CAA6C,CAAA;AAAA,QACzD,KAAA,EAAO,EACJ,KAAA,CAAM,CAAA,CAAE,QAAQ,CAAA,CAChB,UAAS,CACT,QAAA;AAAA,UACC;AAAA,SACF;AAAA,QACF,KAAA,EAAO,EACJ,MAAA,EAAO,CACP,KAAI,CACJ,GAAA,CAAI,CAAC,CAAA,CACL,GAAA,CAAI,IAAA,CAAK,IAAI,SAAA,EAAW,UAAU,CAAC,CAAA,CACnC,QAAA,GACA,OAAA,CAAQ,UAAU,CAAA,CAClB,QAAA,CAAS,6BAA6B;AAAA,OAC1C,CAAA;AAAA,MACH,MAAA,EAAQ,CAAA,CAAA,KACN,CAAA,CAAE,MAAA,CAAO;AAAA,QACP,SAAS,CAAA,CAAE,KAAA;AAAA,UACT,EAAE,MAAA,CAAO;AAAA,YACP,KAAA,EAAO,EAAE,MAAA,EAAO;AAAA,YAChB,QAAA,EAAU,EAAE,MAAA,EAAO;AAAA,YACnB,IAAA,EAAM,CAAA,CAAE,MAAA,EAAO,CAAE,QAAA,EAAS;AAAA,YAC1B,SAAA,EAAW,CAAA,CAAE,MAAA,EAAO,CAAE,QAAA,EAAS;AAAA,YAC/B,UAAU,CAAA,CACP,MAAA;AAAA,cACC,EAAE,MAAA,EAAO;AAAA,cACT,CAAA,CAAE,KAAA,CAAM,CAAC,CAAA,CAAE,MAAA,EAAO,EAAG,CAAA,CAAE,MAAA,EAAO,EAAG,CAAA,CAAE,OAAA,EAAS,CAAC;AAAA,cAE9C,QAAA,EAAS;AAAA,YACZ,OAAA,EAAS,EAAE,MAAA;AAAO,WACnB;AAAA;AACH,OACD;AAAA,KACL;AAAA,IACA,MAAA,EAAQ,OAAO,EAAE,KAAA,EAAO,aAAY,KAAM;AACxC,MAAA,IAAI,CAAE,MAAM,KAAA,CAAM,OAAA,EAAQ,EAAI;AAC5B,QAAA,MAAM,IAAI,KAAA;AAAA,UACR;AAAA,SACF;AAAA,MACF;AAEA,MAAA,MAAM,MAAA,GAASA,oBAAY,KAAA,EAAO,MAAM,MAAM,KAAA,EAAM,EAAG,MAAM,KAAK,CAAA;AAClE,MAAA,MAAM,SAAA,GAAY,CAAC,KAAA,CAAM,KAAA,EAAO,GAAI,KAAA,CAAM,YAAA,IAAgB,EAAG,CAAA;AAC7D,MAAA,MAAM,KAAA,GAAQC,gBAAW,SAAS,CAAA;AAClC,MAAA,MAAM,MAAA,GAASC,gBAAA,CAAY,SAAS,CAAA,CAAE,MAAA,CAAO,OAAK,CAAC,KAAA,CAAM,QAAA,CAAS,CAAC,CAAC,CAAA;AACpE,MAAA,MAAM,CAAC,EAAE,UAAA,EAAW,EAAG,WAAW,CAAA,GAAI,MAAM,QAAQ,GAAA,CAAI;AAAA,QACtDC,aAAU,EAAE,KAAA,EAAO,cAAA,EAAgB,MAAA,EAAQ,WAAW,CAAA;AAAA,QACtD,KAAA,CAAM,mBAAA;AAAA,UACJ,CAAC,GAAG,KAAA,EAAO,GAAG,MAAM,CAAA;AAAA,UACpB,eAAA,GAAkB,CAAA;AAAA,UAClB;AAAA;AACF,OACD,CAAA;AACD,MAAA,MAAM,QAAQ,CAAC,GAAG,KAAA,EAAO,GAAG,MAAM,CAAA,CAAE,MAAA;AAAA,QAClC,CAAA,CAAA,KAAA,CAAM,WAAA,CAAY,GAAA,CAAI,CAAC,KAAK,CAAA,IAAK;AAAA,OACnC;AACA,MAAA,MAAM,IAAA,GAAO,MAAM,MAAA,CAAO,CAAA,CAAA,KAAK,YAAY,GAAA,CAAI,CAAC,KAAM,eAAe,CAAA;AACrE,MAAA,MAAM,OAAA,GAAU,MAAM,MAAA,CAAO,CAAA,CAAA,KAAK,YAAY,GAAA,CAAI,CAAC,MAAM,CAAC,CAAA;AAE1D,MAAA,MAAM,KAAA,GAAsB,MAAM,OAAA,CAAQ,GAAA,CAAI;AAAA,QAC5C,GAAG,UAAA,CAAW,GAAA,CAAI,OAAM,SAAA,MAAc;AAAA,UACpC,IAAA,EAAM,UAAA;AAAA,UACN,MAAM,MAAM,KAAA,CAAM,OAAA,CAAQ,SAAA,EAAW,YAAY,MAAM,CAAA;AAAA,UACvD,MAAA,EAAQ,IAAI,UAAA,CAAW;AAAA,SACzB,CAAE,CAAA;AAAA,QACF,KAAA,CACG,OAAA,CAAQ,KAAA,EAAO,UAAA,EAAY,MAAM,CAAA,CACjC,IAAA,CAAK,CAAA,IAAA,MAAS,EAAE,IAAA,EAAM,SAAA,EAAoB,IAAA,EAAM,MAAA,EAAQ,GAAE,CAAE,CAAA;AAAA,QAC/D,KAAA,CACG,OAAA,CAAQ,IAAA,EAAM,UAAA,EAAY,MAAM,CAAA,CAChC,IAAA,CAAK,CAAA,IAAA,MAAS,EAAE,IAAA,EAAM,UAAA,EAAqB,IAAA,EAAM,MAAA,EAAQ,GAAE,CAAE,CAAA;AAAA,QAChE,KAAA,CACG,KAAA,CAAM,OAAA,EAAS,UAAA,EAAY,MAAM,CAAA,CACjC,IAAA,CAAK,CAAA,IAAA,MAAS,EAAE,IAAA,EAAM,OAAA,EAAkB,IAAA,EAAM,MAAA,EAAQ,GAAE,CAAE;AAAA,OAC9D,CAAA;AAED,MAAA,MAAM,UAAA,GAAaC,uBAAe,KAAK,CAAA;AACvC,MAAA,MAAM,KAAA,GAAQC,kBAAa,KAAA,EAAO,CAAA,IAAA,KAAQ,WAAW,IAAI,CAAA,CAAE,MAAM,CAAA,CAAE,KAAA;AAAA,QACjE,CAAA;AAAA,QACA;AAAA,OACF;AACA,MAAA,MAAM,OAAO,IAAI,GAAA;AAAA,QAAA,CACd,MAAM,KAAA,CAAM,MAAA,CAAO,KAAA,CAAM,GAAA,CAAI,CAAA,GAAA,KAAO,GAAA,CAAI,EAAE,CAAC,CAAA,EAAG,GAAA,CAAI,CAAA,KAAA,KAAS;AAAA,UAC1D,KAAA,CAAM,EAAA;AAAA,UACN;AAAA,SACD;AAAA,OACH;AACA,MAAA,MAAM,KAAA,GAAQ,KAAA,CAAM,OAAA,CAAQ,CAAA,GAAA,KAAO;AACjC,QAAA,MAAM,KAAA,GAAQ,IAAA,CAAK,GAAA,CAAI,GAAA,CAAI,EAAE,CAAA;AAC7B,QAAA,OAAO,QAAQ,CAAC,EAAE,KAAK,KAAA,EAAO,IAAI,EAAC;AAAA,MACrC,CAAC,CAAA;AAED,MAAA,MAAM,QAAA,GAAW,MAAM,kBAAA,CAAmB,OAAA,EAAS,MAAM,WAAA,EAAa;AAAA,QACpE,GAAG,IAAI,GAAA;AAAA,UACL,KAAA,CAAM,OAAA;AAAA,YAAQ,CAAC,EAAE,KAAA,EAAM,KACrB,KAAA,CAAM,YAAY,CAAC,KAAA,CAAM,SAAS,CAAA,GAAI;AAAC;AACzC;AACF,OACD,CAAA;AAED,MAAA,MAAM,IAAA,uBAAW,GAAA,EAAY;AAC7B,MAAA,MAAM,aAAmC,EAAC;AAC1C,MAAA,KAAA,MAAW,EAAE,GAAA,EAAK,KAAA,EAAM,IAAK,KAAA,EAAO;AAClC,QAAA,MAAM,MAAM,CAAA,EAAG,KAAA,CAAM,QAAQ,CAAA,CAAA,EAAI,MAAM,KAAK,CAAA,CAAA;AAC5C,QAAA,IACG,MAAM,SAAA,IAAa,CAAC,QAAA,CAAS,GAAA,CAAI,MAAM,SAAS,CAAA,IACjD,IAAA,CAAK,GAAA,CAAI,GAAG,CAAA,IACZ,IAAA,CAAK,GAAA,CAAI,KAAA,CAAM,OAAO,CAAA,EACtB;AACA,UAAA;AAAA,QACF;AACA,QAAA,IAAA,CAAK,IAAI,GAAG,CAAA;AACZ,QAAA,IAAA,CAAK,GAAA,CAAI,MAAM,OAAO,CAAA;AACtB,QAAA,UAAA,CAAW,IAAA,CAAK,EAAE,GAAG,KAAA,EAAO,KAAA,EAAO,IAAI,KAAA,EAAO,KAAA,EAAO,GAAA,CAAI,KAAA,EAAO,CAAA;AAAA,MAClE;AAEA,MAAA,MAAM,GAAA,GAAM,IAAI,GAAA,CAAI,UAAA,CAAW,IAAI,CAAA,SAAA,KAAa,SAAA,CAAU,EAAE,CAAC,CAAA;AAC7D,MAAA,MAAM,MAAA,GAAS,MAAM,MAAA,EAAO,CAAE;AAAA,QAC5B,OAAO,KAAA,CAAM,KAAA;AAAA,QACb,SAAA;AAAA,QACA,OAAO,MAAA,CAAO,KAAA;AAAA,QACd,UAAA;AAAA,QACA,YAAY,KAAA,CAAM;AAAA,OACnB,CAAA;AACD,MAAA,MAAM,UAAU,MAAA,CACb,MAAA,CAAO,CAAA,SAAA,KAAa,GAAA,CAAI,OAAO,SAAA,CAAU,EAAE,CAAC,CAAA,CAC5C,MAAM,CAAA,EAAG,KAAA,CAAM,KAAK,CAAA,CACpB,IAAI,CAAA,SAAA,MAAc;AAAA,QACjB,OAAO,SAAA,CAAU,KAAA;AAAA,QACjB,UAAU,SAAA,CAAU,QAAA;AAAA,QACpB,GAAI,UAAU,IAAA,GAAO,EAAE,MAAM,SAAA,CAAU,IAAA,KAAS,EAAC;AAAA,QACjD,GAAI,UAAU,SAAA,GAAY,EAAE,WAAW,SAAA,CAAU,SAAA,KAAc,EAAC;AAAA,QAChE,GAAI,UAAU,QAAA,GAAW,EAAE,UAAU,SAAA,CAAU,QAAA,KAAa,EAAC;AAAA,QAC7D,OAAA,EAAS,SAAA,CAAU,OAAA,CAAQ,KAAA,CAAM,GAAG,aAAa;AAAA,OACnD,CAAE,CAAA;AACJ,MAAA,OAAO,EAAE,MAAA,EAAQ,EAAE,OAAA,EAAQ,EAAE;AAAA,IAC/B;AAAA,GACD,CAAA;AACH;AAMO,SAAS,mBACd,KAAA,EACQ;AACR,EAAA,MAAM,MAAA,GAAS,CAAC,GAAG,KAAK,CAAA,CAAE,MAAA;AAAA,IACxB,CAAC,GAAG,QAAQ,MAAM,QAAA,CAAS,OAAA,IAAW,SAAS,MAAA,GAAS;AAAA,GAC1D;AACA,EAAA,MAAM,QAAQ,MAAA,CAAO,GAAA;AAAA,IACnB,CAAC,CAAC,IAAA,EAAM,QAAQ,CAAA,KACd,CAAA,EAAA,EAAK,IAAI,CAAA,EAAG,QAAA,CAAS,UAAU,EAAA,GAAK,+BAA+B,GACjE,QAAA,CAAS,WAAA,GAAc,KAAK,QAAA,CAAS,WAAW,KAAK,EACvD,CAAA;AAAA,GACJ;AACA,EAAA,OAAO;AAAA,IACL,EAAA;AAAA,IACA,CAAA,2MAAA,CAAA;AAAA,IACA,GAAG;AAAA,GACL,CAAE,KAAK,IAAI,CAAA;AACb;AAOA,eAAe,kBAAA,CACb,OAAA,EACA,IAAA,EACA,WAAA,EACA,UAAA,EACsB;AACtB,EAAA,IAAI,UAAA,CAAW,WAAW,CAAA,EAAG;AAC3B,IAAA,2BAAW,GAAA,EAAI;AAAA,EACjB;AACA,EAAA,MAAM,EAAE,KAAA,EAAM,GAAI,MAAM,KAAK,qBAAA,CAAsB;AAAA,IACjD,UAAA,EAAY,WAAA;AAAA,IACZ,cAAA,EAAgB;AAAA,GACjB,CAAA;AACD,EAAA,MAAM,EAAE,KAAA,EAAM,GAAI,MAAM,OAAA,CAAQ,iBAAA;AAAA,IAC9B,EAAE,UAAA,EAAY,MAAA,EAAQ,CAAC,MAAA,EAAQ,oBAAA,EAAsB,eAAe,CAAA,EAAE;AAAA,IACtE,EAAE,KAAA;AAAM,GACV;AACA,EAAA,OAAO,IAAI,GAAA,CAAI,UAAA,CAAW,MAAA,CAAO,CAAC,GAAG,CAAA,KAAM,KAAA,CAAM,CAAC,CAAC,CAAC,CAAA;AACtD;;;;;"}

package/dist/alpha.cjs.js

@@ -0,0 +1,8 @@
+'use strict';
+
+var ranking = require('./ranking.cjs.js');
+
+
+
+exports.knowledgeRankingExtensionPoint = ranking.knowledgeRankingExtensionPoint;
+//# sourceMappingURL=alpha.cjs.js.map

package/dist/alpha.cjs.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"alpha.cjs.js","sources":[],"sourcesContent":[],"names":[],"mappings":";;;;;;"}

package/dist/alpha.d.ts

@@ -0,0 +1,78 @@
+import * as _backstage_backend_plugin_api from '@backstage/backend-plugin-api';
+
+/**
+ * One `search-knowledge` result candidate, after fusion and the permission
+ * check.
+ *
+ * @alpha
+ */
+interface KnowledgeCandidate {
+    /** The chunk's stable id. */
+    id: string;
+    /** The Backstage search document type; absent for types not named in `assistants.semanticSearch.types`. */
+    type?: string;
+    /** The catalog entity gating it, for catalog and TechDocs results. */
+    entityRef?: string;
+    /** The search result location: a TechDocs path or a link. */
+    location: string;
+    /** Heading path, e.g. `Pylon › Messaging › Dead-letter handling`. */
+    title: string;
+    /** The chunk's text, up to ~1600 characters. */
+    content: string;
+    /** The chunk's 1-based rank in each retrieval list it appears in. */
+    ranks: {
+        semantic?: number;
+        keyword?: number;
+        rareWord?: number;
+        fuzzy?: number;
+    };
+    /** Reciprocal rank fusion score, after the type's configured weight. */
+    score: number;
+    /** The short fields the collator attached (status, space, updated, …). */
+    metadata?: Record<string, string | number | boolean>;
+}
+/**
+ * What a {@link KnowledgeRanker} picks from.
+ *
+ * @alpha
+ */
+interface KnowledgeRankerInput {
+    /** The caller's question. */
+    query: string;
+    /** The query and the caller's alternatives. */
+    phrasings: string[];
+    /** The types searched: those the caller named, or the default ones. */
+    types: string[];
+    /** Candidates the caller may read, best fused score first, with repeated text removed. */
+    candidates: KnowledgeCandidate[];
+    /** How many results the caller gets. */
+    maxResults: number;
+}
+/**
+ * Picks and orders the results from the candidates. Results it returns that
+ * are not among the candidates are dropped, and the list is cut to
+ * `maxResults`.
+ *
+ * @alpha
+ */
+type KnowledgeRanker = (input: KnowledgeRankerInput) => Promise<KnowledgeCandidate[]> | KnowledgeCandidate[];
+/**
+ * Replaces how `search-knowledge` picks its results.
+ *
+ * @alpha
+ */
+interface KnowledgeRankingExtensionPoint {
+    /** Replaces the default ranker (type caps, then `maxResults`). */
+    setRanker(ranker: KnowledgeRanker): void;
+}
+/**
+ * Lets a backend module replace how `search-knowledge` picks its results,
+ * for example to favour recent tickets or the caller's team. The ranker sees
+ * only candidates the caller may read.
+ *
+ * @alpha
+ */
+declare const knowledgeRankingExtensionPoint: _backstage_backend_plugin_api.ExtensionPoint<KnowledgeRankingExtensionPoint>;
+
+export { knowledgeRankingExtensionPoint };
+export type { KnowledgeCandidate, KnowledgeRanker, KnowledgeRankerInput, KnowledgeRankingExtensionPoint };

package/dist/chunk.cjs.js

@@ -5,13 +5,18 @@ var node_crypto = require('node:crypto');
 const MAX_CHARS = 1600;
 const OVERLAP_CHARS = 200;
 function chunkDocument(doc) {
+  const metadataHash = sha256(
+    JSON.stringify(Object.entries(doc.metadata ?? {}).sort())
+  );
   return splitText(doc.text).map((content, index) => ({
     id: sha256(`${doc.location}|${index}`),
     ...doc.entityRef ? { entityRef: doc.entityRef } : {},
     location: doc.location,
     title: doc.title,
     content,
-    hash: sha256(embeddingInput(doc.title, content))
+    hash: sha256(embeddingInput(doc.title, content)),
+    ...doc.metadata ? { metadata: doc.metadata } : {},
+    metadataHash
   }));
 }
 function embeddingInput(title, content) {

package/dist/chunk.cjs.js.map

@@ -1 +1 @@
-{"version":3,"file":"chunk.cjs.js","sources":["../src/chunk.ts"],"sourcesContent":["import { createHash } from 'node:crypto';\nimport type { SourceDocument } from './documents';\n\n/** A stored, embedded unit of a {@link SourceDocument}. */\nexport interface Chunk {\n  id: string;\n  entityRef?: string;\n  location: string;\n  title: string;\n  content: string;\n  /** Hash of what gets embedded; unchanged hash = no re-embedding. */\n  hash: string;\n}\n\nconst MAX_CHARS = 1600;\nconst OVERLAP_CHARS = 200;\n\n/**\n * Splits a document into chunks of at most ~1600 characters on word\n * boundaries, overlapping by ~200 so a sentence cut at a boundary survives in\n * one piece. TechDocs sections are usually one chunk already.\n */\nexport function chunkDocument(doc: SourceDocument): Chunk[] {\n  return splitText(doc.text).map((content, index) => ({\n    id: sha256(`${doc.location}|${index}`),\n    ...(doc.entityRef ? { entityRef: doc.entityRef } : {}),\n    location: doc.location,\n    title: doc.title,\n    content,\n    hash: sha256(embeddingInput(doc.title, content)),\n  }));\n}\n\n/** The text a chunk is embedded as: heading path first, so a terse section keeps its context. */\nexport function embeddingInput(title: string, content: string): string {\n  return `${title}\\n\\n${content}`;\n}\n\nexport function splitText(text: string): string[] {\n  if (text.length <= MAX_CHARS) {\n    return [text];\n  }\n  const pieces: string[] = [];\n  let start = 0;\n  while (start < text.length) {\n    let end = Math.min(start + MAX_CHARS, text.length);\n    if (end < text.length) {\n      const space = text.lastIndexOf(' ', end);\n      if (space > start + MAX_CHARS / 2) {\n        end = space;\n      }\n    }\n    pieces.push(text.slice(start, end).trim());\n    if (end >= text.length) {\n      break;\n    }\n    const restart = text.indexOf(' ', end - OVERLAP_CHARS);\n    start = restart > start && restart < end ? restart + 1 : end;\n  }\n  return pieces.filter(Boolean);\n}\n\nfunction sha256(value: string): string {\n  return createHash('sha256').update(value).digest('hex');\n}\n"],"names":["createHash"],"mappings":";;;;AAcA,MAAM,SAAA,GAAY,IAAA;AAClB,MAAM,aAAA,GAAgB,GAAA;AAOf,SAAS,cAAc,GAAA,EAA8B;AAC1D,EAAA,OAAO,UAAU,GAAA,CAAI,IAAI,EAAE,GAAA,CAAI,CAAC,SAAS,KAAA,MAAW;AAAA,IAClD,IAAI,MAAA,CAAO,CAAA,EAAG,IAAI,QAAQ,CAAA,CAAA,EAAI,KAAK,CAAA,CAAE,CAAA;AAAA,IACrC,GAAI,IAAI,SAAA,GAAY,EAAE,WAAW,GAAA,CAAI,SAAA,KAAc,EAAC;AAAA,IACpD,UAAU,GAAA,CAAI,QAAA;AAAA,IACd,OAAO,GAAA,CAAI,KAAA;AAAA,IACX,OAAA;AAAA,IACA,MAAM,MAAA,CAAO,cAAA,CAAe,GAAA,CAAI,KAAA,EAAO,OAAO,CAAC;AAAA,GACjD,CAAE,CAAA;AACJ;AAGO,SAAS,cAAA,CAAe,OAAe,OAAA,EAAyB;AACrE,EAAA,OAAO,GAAG,KAAK;;AAAA,EAAO,OAAO,CAAA,CAAA;AAC/B;AAEO,SAAS,UAAU,IAAA,EAAwB;AAChD,EAAA,IAAI,IAAA,CAAK,UAAU,SAAA,EAAW;AAC5B,IAAA,OAAO,CAAC,IAAI,CAAA;AAAA,EACd;AACA,EAAA,MAAM,SAAmB,EAAC;AAC1B,EAAA,IAAI,KAAA,GAAQ,CAAA;AACZ,EAAA,OAAO,KAAA,GAAQ,KAAK,MAAA,EAAQ;AAC1B,IAAA,IAAI,MAAM,IAAA,CAAK,GAAA,CAAI,KAAA,GAAQ,SAAA,EAAW,KAAK,MAAM,CAAA;AACjD,IAAA,IAAI,GAAA,GAAM,KAAK,MAAA,EAAQ;AACrB,MAAA,MAAM,KAAA,GAAQ,IAAA,CAAK,WAAA,CAAY,GAAA,EAAK,GAAG,CAAA;AACvC,MAAA,IAAI,KAAA,GAAQ,KAAA,GAAQ,SAAA,GAAY,CAAA,EAAG;AACjC,QAAA,GAAA,GAAM,KAAA;AAAA,MACR;AAAA,IACF;AACA,IAAA,MAAA,CAAO,KAAK,IAAA,CAAK,KAAA,CAAM,OAAO,GAAG,CAAA,CAAE,MAAM,CAAA;AACzC,IAAA,IAAI,GAAA,IAAO,KAAK,MAAA,EAAQ;AACtB,MAAA;AAAA,IACF;AACA,IAAA,MAAM,OAAA,GAAU,IAAA,CAAK,OAAA,CAAQ,GAAA,EAAK,MAAM,aAAa,CAAA;AACrD,IAAA,KAAA,GAAQ,OAAA,GAAU,KAAA,IAAS,OAAA,GAAU,GAAA,GAAM,UAAU,CAAA,GAAI,GAAA;AAAA,EAC3D;AACA,EAAA,OAAO,MAAA,CAAO,OAAO,OAAO,CAAA;AAC9B;AAEA,SAAS,OAAO,KAAA,EAAuB;AACrC,EAAA,OAAOA,uBAAW,QAAQ,CAAA,CAAE,OAAO,KAAK,CAAA,CAAE,OAAO,KAAK,CAAA;AACxD;;;;;;"}
+{"version":3,"file":"chunk.cjs.js","sources":["../src/chunk.ts"],"sourcesContent":["import { createHash } from 'node:crypto';\nimport type { KnowledgeMetadata, SourceDocument } from './documents';\n\n/** A stored, embedded unit of a {@link SourceDocument}. */\nexport interface Chunk {\n  id: string;\n  entityRef?: string;\n  location: string;\n  title: string;\n  content: string;\n  /** Hash of what gets embedded; unchanged hash = no re-embedding. */\n  hash: string;\n  metadata?: KnowledgeMetadata;\n  /** Hash of the metadata; a change alone rewrites the row without re-embedding. */\n  metadataHash: string;\n}\n\nconst MAX_CHARS = 1600;\nconst OVERLAP_CHARS = 200;\n\n/**\n * Splits a document into chunks of at most ~1600 characters on word\n * boundaries, overlapping by ~200 so a sentence cut at a boundary survives in\n * one piece. TechDocs sections are usually one chunk already.\n */\nexport function chunkDocument(doc: SourceDocument): Chunk[] {\n  const metadataHash = sha256(\n    JSON.stringify(Object.entries(doc.metadata ?? {}).sort()),\n  );\n  return splitText(doc.text).map((content, index) => ({\n    id: sha256(`${doc.location}|${index}`),\n    ...(doc.entityRef ? { entityRef: doc.entityRef } : {}),\n    location: doc.location,\n    title: doc.title,\n    content,\n    hash: sha256(embeddingInput(doc.title, content)),\n    ...(doc.metadata ? { metadata: doc.metadata } : {}),\n    metadataHash,\n  }));\n}\n\n/** The text a chunk is embedded as: heading path first, so a terse section keeps its context. */\nexport function embeddingInput(title: string, content: string): string {\n  return `${title}\\n\\n${content}`;\n}\n\nexport function splitText(text: string): string[] {\n  if (text.length <= MAX_CHARS) {\n    return [text];\n  }\n  const pieces: string[] = [];\n  let start = 0;\n  while (start < text.length) {\n    let end = Math.min(start + MAX_CHARS, text.length);\n    if (end < text.length) {\n      const space = text.lastIndexOf(' ', end);\n      if (space > start + MAX_CHARS / 2) {\n        end = space;\n      }\n    }\n    pieces.push(text.slice(start, end).trim());\n    if (end >= text.length) {\n      break;\n    }\n    const restart = text.indexOf(' ', end - OVERLAP_CHARS);\n    start = restart > start && restart < end ? restart + 1 : end;\n  }\n  return pieces.filter(Boolean);\n}\n\nfunction sha256(value: string): string {\n  return createHash('sha256').update(value).digest('hex');\n}\n"],"names":["createHash"],"mappings":";;;;AAiBA,MAAM,SAAA,GAAY,IAAA;AAClB,MAAM,aAAA,GAAgB,GAAA;AAOf,SAAS,cAAc,GAAA,EAA8B;AAC1D,EAAA,MAAM,YAAA,GAAe,MAAA;AAAA,IACnB,IAAA,CAAK,SAAA,CAAU,MAAA,CAAO,OAAA,CAAQ,GAAA,CAAI,YAAY,EAAE,CAAA,CAAE,IAAA,EAAM;AAAA,GAC1D;AACA,EAAA,OAAO,UAAU,GAAA,CAAI,IAAI,EAAE,GAAA,CAAI,CAAC,SAAS,KAAA,MAAW;AAAA,IAClD,IAAI,MAAA,CAAO,CAAA,EAAG,IAAI,QAAQ,CAAA,CAAA,EAAI,KAAK,CAAA,CAAE,CAAA;AAAA,IACrC,GAAI,IAAI,SAAA,GAAY,EAAE,WAAW,GAAA,CAAI,SAAA,KAAc,EAAC;AAAA,IACpD,UAAU,GAAA,CAAI,QAAA;AAAA,IACd,OAAO,GAAA,CAAI,KAAA;AAAA,IACX,OAAA;AAAA,IACA,MAAM,MAAA,CAAO,cAAA,CAAe,GAAA,CAAI,KAAA,EAAO,OAAO,CAAC,CAAA;AAAA,IAC/C,GAAI,IAAI,QAAA,GAAW,EAAE,UAAU,GAAA,CAAI,QAAA,KAAa,EAAC;AAAA,IACjD;AAAA,GACF,CAAE,CAAA;AACJ;AAGO,SAAS,cAAA,CAAe,OAAe,OAAA,EAAyB;AACrE,EAAA,OAAO,GAAG,KAAK;;AAAA,EAAO,OAAO,CAAA,CAAA;AAC/B;AAEO,SAAS,UAAU,IAAA,EAAwB;AAChD,EAAA,IAAI,IAAA,CAAK,UAAU,SAAA,EAAW;AAC5B,IAAA,OAAO,CAAC,IAAI,CAAA;AAAA,EACd;AACA,EAAA,MAAM,SAAmB,EAAC;AAC1B,EAAA,IAAI,KAAA,GAAQ,CAAA;AACZ,EAAA,OAAO,KAAA,GAAQ,KAAK,MAAA,EAAQ;AAC1B,IAAA,IAAI,MAAM,IAAA,CAAK,GAAA,CAAI,KAAA,GAAQ,SAAA,EAAW,KAAK,MAAM,CAAA;AACjD,IAAA,IAAI,GAAA,GAAM,KAAK,MAAA,EAAQ;AACrB,MAAA,MAAM,KAAA,GAAQ,IAAA,CAAK,WAAA,CAAY,GAAA,EAAK,GAAG,CAAA;AACvC,MAAA,IAAI,KAAA,GAAQ,KAAA,GAAQ,SAAA,GAAY,CAAA,EAAG;AACjC,QAAA,GAAA,GAAM,KAAA;AAAA,MACR;AAAA,IACF;AACA,IAAA,MAAA,CAAO,KAAK,IAAA,CAAK,KAAA,CAAM,OAAO,GAAG,CAAA,CAAE,MAAM,CAAA;AACzC,IAAA,IAAI,GAAA,IAAO,KAAK,MAAA,EAAQ;AACtB,MAAA;AAAA,IACF;AACA,IAAA,MAAM,OAAA,GAAU,IAAA,CAAK,OAAA,CAAQ,GAAA,EAAK,MAAM,aAAa,CAAA;AACrD,IAAA,KAAA,GAAQ,OAAA,GAAU,KAAA,IAAS,OAAA,GAAU,GAAA,GAAM,UAAU,CAAA,GAAI,GAAA;AAAA,EAC3D;AACA,EAAA,OAAO,MAAA,CAAO,OAAO,OAAO,CAAA;AAC9B;AAEA,SAAS,OAAO,KAAA,EAAuB;AACrC,EAAA,OAAOA,uBAAW,QAAQ,CAAA,CAAE,OAAO,KAAK,CAAA,CAAE,OAAO,KAAK,CAAA;AACxD;;;;;;"}