@zenera/rag 1.1.9 → 1.1.10

package/README.md

@@ -1,7 +1,8 @@
 # @zenera/rag
 
-**An OpenAPI description, indexed as a graph and searched by meaning — for
-agents that have to call an API they have not read.**
+**A corpus, indexed and searched by meaning — an OpenAPI description as a
+graph, a pile of markdown as quotable passages — for agents that have to work
+with something they have not read.**
 
 [![License: MIT](https://img.shields.io/badge/license-MIT-blue.svg)](https://github.com/andreyryabov/ZeneraNeo/blob/main/LICENSE)
 [![Node](https://img.shields.io/badge/node-%E2%89%A524-brightgreen.svg)](https://nodejs.org)
@@ -11,7 +12,23 @@ agents that have to call an API they have not read.**
 > [`zen`](https://github.com/andreyryabov/ZeneraNeo/blob/main/packages/cli/README.md),
 > which is also where the credentials already are.
 
-## Why
+## Two subjects
+
+A subject is a kind of corpus with its own index format, its own verbs and its
+own flags — not a variation on one command, because what `list` means to an API
+description is not what it means to a folder of notes.
+
+| Subject          | The corpus                | The answer                                     |
+| ---------------- | ------------------------- | ---------------------------------------------- |
+| `zen rag schema` | openapi/swagger documents | the connected piece of the API that matched    |
+| `zen rag docs`   | markdown and plain text   | the passages that matched, quoted with numbers |
+
+Both are built the same way — [LanceDB](https://lancedb.com) for hybrid vector
+
+- full-text retrieval, a manifest that records which embedder made the vectors,
+  and exact commands beside the ranking ones that need no credential at all.
+
+## Schema — an API description as a graph
 
 A large specification does not fit in a prompt, and the parts of it that answer
 a question are scattered: the field is on a schema, the schema is on a request
@@ -117,11 +134,75 @@ zen rag schema search --query - --format ts <<'JSON'
 JSON
 ```
 
+## Docs — markdown as quotable passages
+
+The other subject. Point it at files, directories or globs; `.md`, `.markdown`,
+`.txt` and `.text` are read, hidden directories and `node_modules` are not.
+
+```sh
+zen rag docs index --embedding openai:text-embedding-3-small ./docs
+zen rag docs search "how are rate limits counted"
+```
+
+The answer is the documents themselves — the passages that matched, quoted
+verbatim with their line numbers, and a marker wherever something between two
+of them was left out:
+
+```
+## nsx_4.2.0/api/routing.md — 9 of 148 lines
+
+  5 | ## Rate limits
+  7 | Requests are counted per tenant and rejected past the limit.
+... 12 lines omitted (Retries, Backoff) ...
+ 24 | | route | limit | window |
+ 25 | | --- | --- | --- |
+ 27 | | /api/users | 250 | 1m |
+```
+
+Nobody finds the paragraph they want on the first ask, so **narrowing is the
+interface**, not an afterthought. The second call is the same question inside
+one part of the tree:
+
+```sh
+zen rag docs search --file "nsx_4.2.*/api/**" "rate limit for the users route"
+zen rag docs search --section "Rate limits" --kind table "requests per minute"
+zen rag docs search --mode text "X-RateLimit-Remaining"   # exact wording only
+zen rag docs search --interactive                          # narrow by typing
+```
+
+`--file` is a glob when it has `*` or `?` and a substring otherwise, matched
+against the document's **name**, which is its path relative to the common root
+of everything indexed. That is what keeps
+two releases of the same file apart. `--section` takes a heading title, and
+covers whatever nests inside it. `--kind` takes `paragraph`, `list`, `table`,
+`table_row`, `code`, `frontmatter` or `html`, for when the answer is a table
+and not the prose around it. `-B/-A` widen each passage, `--max-lines` caps the
+whole answer, `--exclude-id` moves on from what was already seen.
+
+Tables are indexed twice over: once as a descriptor carrying the caption and
+the column names, and once per row, with the header row travelling alongside so
+the columns are still named wherever a row lands. A row too wide to be one
+chunk is cut into column groups, with the key column repeated in each.
+
+And beside all that, the exact half — no embedder, no credential, no network:
+
+```sh
+zen rag docs list files                      # every document, and what it holds
+zen rag docs list sections --file "api/**"   # every heading, with its line span
+zen rag docs list tables                     # every table, with its columns
+zen rag docs grep "Retry-After"              # every matching line, and its section
+zen rag docs show api/routing.md --section "Rate limits"
+zen rag docs show api/routing.md --lines 40-80
+```
+
+`grep` reports `found` as the true total even when `--limit` cuts the rows, so
+unlike a search it can answer whether a string appears at all.
+
 ## Which index
 
-Every reading command takes `-d, --dir`. Without one, `$ZEN_SCHEMA_DB` is used
-if it is set; without that, the nearest index to the working directory is found
-and named on stderr as it is used.
+Every reading command takes `-d, --dir`. Without one, `$ZEN_SCHEMA_DB` or
+`$ZEN_DOCS_DB` is used if it is set; without that, the nearest index to the
+working directory is found and named on stderr as it is used.
 
 Nearest means what it says: this directory, then a short way down into it, then
 up a level and again, stopping at your home directory. What is looked for is a
@@ -131,7 +212,10 @@ the same. `schema-db` is only the name a new one is given.
 
 Two indexes the same distance away is a question, not a tie to break, and it is
 refused: the wrong index does not fail, it answers confidently about a
-different API. Name one with `--dir`, or set `ZEN_SCHEMA_DB`.
+different API. Name one with `--dir`, or set the environment variable.
+
+The search is scoped by kind, so a `docs` index and a `schema` index can sit in
+the same tree without either shadowing the other.
 
 ## Commands
 
@@ -147,8 +231,8 @@ zen rag schema stats             What is in an index, and what built it.
 
 Search terms are one flag each — `--all`, `--method`, `--type`, `--input-type`,
 `--output-type`, `--property`, `--input-property`, `--output-property` — shaped
-by `--direction`, `--method-type`, `--limit`, `--max-hops`, `--max-nodes` and
-the four `--exclude-*` filters, and rendered by `--format text | mermaid |
+by `--direction`, `--method-type`, `--limit`, `--max-hops`, `--max-nodes`,
+`--source` and the four `--exclude-*` filters, and rendered by `--format text | mermaid |
 mermaid-flowchart | ts | openapi`. `zen help rag` prints the full table.
 
 `list` and `grep` share `--name`, `--path`, `--regex`, `--case-sensitive`,
@@ -167,6 +251,23 @@ only what was named instead of its neighbourhood.
 zen rag schema grep token --ids-only | xargs zen rag schema show --format ts
 ```
 
+And for documents:
+
+```
+zen rag docs index <path...>     Read the documents and write a searchable index.
+zen rag docs search [text]       Ask it something. --interactive for a prompt.
+zen rag docs list <what>         Every document, section or table. No ranking.
+zen rag docs grep <pattern>      Every matching line, with the section it sits in.
+zen rag docs show <file>         A document, a section of one, or a line range.
+zen rag docs stats               What is in an index, and what built it.
+```
+
+Search takes the question as a bare phrase, narrowed by `--file`,
+`--exclude-file`, `--section`, `--kind` and `--mode`, shaped by `--limit`,
+`-B/--before`, `-A/--after` and `--max-lines`, and moved along by
+`--exclude-id`. `list` and `grep` share `--file`, `--section`, `--regex`,
+`--case-sensitive` and `--limit`.
+
 ## From an agent
 
 ```ts
@@ -201,6 +302,28 @@ need the word explained again, it needs the list of types that have one.
 `trace_api` is the step after both: a field is of no use until the call that
 carries it is known.
 
+Documents come with four, in the group `docs`, selectable as `docs:*`:
+
+```ts
+import { docs } from '@zenera/rag';
+
+const index = await docs.DocsIndex.open('./docs-db', embedder);
+const project = await loadProject('./my-project', { tools: docs.docsTools(index) });
+```
+
+| Tool          | For                                                          |
+| ------------- | ------------------------------------------------------------ |
+| `search_docs` | the passages that match, quoted with their line numbers      |
+| `list_docs`   | the documents, their headings, or their tables — no search   |
+| `grep_docs`   | every matching line, counted in full — no search             |
+| `read_docs`   | a section or a line range, verbatim and with nothing omitted |
+
+Same division, same reason. `search_docs` is the way in when the question is
+vague; `grep_docs` is how "it is not in here" can actually be concluded. Every
+answer carries line numbers and `read_docs` takes them, which is the loop the
+subject exists for: find the passage, read around it, then edit the file the
+passage came from.
+
 ## What an index is
 
 ```
@@ -217,6 +340,21 @@ schema-db/
 The manifest records which embedder made the vectors, and a search with a
 different one is refused rather than answered with noise.
 
+A document index is the same idea with a different middle:
+
+```
+docs-db/
+├── README.md         what this index holds — a live progress report while it builds
+├── manifest.json     written last — its absence means "not indexed"
+├── outline.json      every heading and table, with the lines they cover
+├── sources/          the documents themselves, verbatim — where the quotes come from
+└── lance/            one table: a row per chunk, two texts, one vector
+```
+
+There the copies are not a record but the answer: a search returns line ranges
+and the lines are read back out of `sources/`, so what is quoted is the document
+rather than a reconstruction of it.
+
 Indexing a large document is minutes of silence, so the directory says what is
 happening to it. `README.md` appears first as a progress report — the documents,
 the embedder, the step, how many entities have been embedded of how many, and
@@ -235,7 +373,8 @@ so an index built here is read under a name this machine never sees. `--no-sourc
 leaves the copies out, for an index that will never travel.
 
 The copies are a record, not an input: rebuilding reads the files you name, not
-the ones in `sources/`.
+the ones in `sources/`. A **document** index has no `--no-sources`, because
+there the copies are what every quoted line is read from.
 
 ## Notes
 
@@ -251,6 +390,11 @@ the ones in `sources/`.
 - Filters reaching the store are **closed enums only**. Exclusion lists are
   applied in JavaScript afterwards, so nothing a model wrote ever reaches a SQL
   predicate.
+- A document chunk knows **exactly which lines** of the original it stands for,
+  headings and table headers included. That is what makes an answer quotable,
+  and what lets the next question be phrased in line numbers.
+- Plain text is read as paragraphs and given **no invented headings**: a `.txt`
+  file has one section, which is the document.
 
 ## The rest of the family
 

package/dist/command.js

@@ -1,4 +1,5 @@
 import { bold, cyan, dim, table, usageError, write, } from '@zenera/cli/lib';
+import { command as docs } from "./docs/command.js";
 import { command as schema } from "./schema/command.js";
 // ---------------------------------------------------------------------------
 // zen rag — retrieval, by subject
@@ -18,7 +19,7 @@ import { command as schema } from "./schema/command.js";
 // `help <subject>` is a verb rather than a flag because `--help` never gets
 // here: the frame lifts it out of the arguments and answers with this page.
 // ---------------------------------------------------------------------------
-const SUBJECTS = { schema };
+const SUBJECTS = { schema, docs };
 const USAGE = 'zen rag <subject> <command> [args...]';
 export const command = {
     summary: 'Retrieval over a corpus: index it, then ask it something.',

package/dist/docs/assemble.d.ts

@@ -0,0 +1,52 @@
+import type { DocsIndex, Match } from './search.ts';
+export interface AssembleOptions {
+    /** extra lines quoted before each matching body */
+    before?: number;
+    after?: number;
+    /** a ceiling on the whole answer, so one long section cannot eat it */
+    maxLines?: number;
+    /** two ranges closer than this are joined rather than marked */
+    mergeGap?: number;
+}
+export interface Segment {
+    start: number;
+    end: number;
+    /** the lines themselves, verbatim */
+    lines: string[];
+}
+export interface Omission {
+    start: number;
+    end: number;
+    count: number;
+    /** the headings the skipped lines covered, so the gap has a name */
+    sections: string[];
+}
+export type Piece = ({
+    type: 'segment';
+} & Segment) | ({
+    type: 'omission';
+} & Omission);
+export interface Excerpt {
+    path: string;
+    title: string;
+    score: number;
+    /** what landed in this document, best first */
+    matches: Match[];
+    pieces: Piece[];
+    /** lines actually quoted */
+    shown: number;
+    /** the document's length, so a reader knows what fraction this is */
+    lines: number;
+}
+export interface Assembly {
+    files: Excerpt[];
+    shown: number;
+    /** true when the line budget cut something that had matched */
+    truncated: boolean;
+}
+export declare const DEFAULT_BEFORE = 0;
+export declare const DEFAULT_AFTER = 0;
+export declare const DEFAULT_MAX_LINES = 400;
+export declare const DEFAULT_MERGE_GAP = 3;
+export declare function assemble(index: DocsIndex, matches: readonly Match[], options?: AssembleOptions): Promise<Assembly>;
+//# sourceMappingURL=assemble.d.ts.map

package/dist/docs/assemble.js

@@ -0,0 +1,127 @@
+export const DEFAULT_BEFORE = 0;
+export const DEFAULT_AFTER = 0;
+export const DEFAULT_MAX_LINES = 400;
+export const DEFAULT_MERGE_GAP = 3;
+export async function assemble(index, matches, options = {}) {
+    const before = options.before ?? DEFAULT_BEFORE;
+    const after = options.after ?? DEFAULT_AFTER;
+    const gap = options.mergeGap ?? DEFAULT_MERGE_GAP;
+    let budget = options.maxLines ?? DEFAULT_MAX_LINES;
+    const files = [];
+    let truncated = false;
+    let shown = 0;
+    for (const [path, group] of byFile(matches)) {
+        const outline = index.file(path);
+        const source = await index.lines(path);
+        const total = outline?.lines ?? source.length;
+        const wanted = ranges(lineSet(group, before, after, total), gap);
+        const pieces = [];
+        let quoted = 0;
+        let last;
+        for (const range of wanted) {
+            if (budget <= 0) {
+                truncated = true;
+                break;
+            }
+            const end = Math.min(range.end, range.start + budget - 1);
+            if (last) {
+                pieces.push(omission(last.end + 1, range.start - 1, outline?.headings ?? []));
+            }
+            pieces.push({
+                type: 'segment',
+                start: range.start,
+                end,
+                lines: source.slice(range.start - 1, end),
+            });
+            const count = end - range.start + 1;
+            quoted += count;
+            budget -= count;
+            if (end < range.end) {
+                truncated = true;
+            }
+            last = { start: range.start, end };
+        }
+        if (last && last.end < total) {
+            pieces.push(omission(last.end + 1, total, outline?.headings ?? []));
+        }
+        if (pieces.length === 0) {
+            continue;
+        }
+        shown += quoted;
+        files.push({
+            path,
+            title: outline?.title ?? path,
+            score: group[0].score,
+            matches: group,
+            pieces,
+            shown: quoted,
+            lines: total,
+        });
+    }
+    return { files, shown, truncated };
+}
+// ---------------------------------------------------------------------------
+/** Documents in the order their best match placed, matches within them likewise. */
+function byFile(matches) {
+    const groups = new Map();
+    for (const item of matches) {
+        const group = groups.get(item.path);
+        if (group) {
+            group.push(item);
+        }
+        else {
+            groups.set(item.path, [item]);
+        }
+    }
+    return [...groups.entries()];
+}
+/**
+ * Every line the answer wants. A chunk carries its own render set — its
+ * headings, a table's header row, the line its body started on — and `before`
+ * and `after` widen the body only, so context is padding around the match and
+ * never around a heading quoted from elsewhere in the file.
+ */
+function lineSet(matches, before, after, total) {
+    const wanted = new Set();
+    const add = (line) => {
+        if (line >= 1 && line <= total) {
+            wanted.add(line);
+        }
+    };
+    for (const item of matches) {
+        for (const line of item.lineNumbers) {
+            add(line);
+        }
+        for (let line = item.bodyStart - before; line <= item.bodyEnd + after; line++) {
+            add(line);
+        }
+    }
+    return wanted;
+}
+/** Contiguous runs, with runs closer together than `gap` joined into one. */
+function ranges(wanted, gap) {
+    const sorted = [...wanted].sort((a, b) => a - b);
+    const out = [];
+    for (const line of sorted) {
+        const last = out.at(-1);
+        if (last && line - last.end <= gap + 1) {
+            last.end = line;
+        }
+        else {
+            out.push({ start: line, end: line });
+        }
+    }
+    return out;
+}
+function omission(start, end, headings) {
+    const named = headings.filter((h) => h.line >= start && h.line <= end).map((h) => h.title);
+    return {
+        type: 'omission',
+        start,
+        end,
+        count: end - start + 1,
+        // A gap covering thirty headings is a gap; naming all of them is noise.
+        sections: named.slice(0, 6),
+    };
+}
+//# sourceMappingURL=assemble.js.map

package/dist/docs/build.d.ts

@@ -0,0 +1,34 @@
+import type { Embedder } from '@zenera/neo';
+import { type ChunkOptions } from './chunk.ts';
+import { type Counts, type DocRecord, type Manifest } from './files.ts';
+import { type Corpus } from './load.ts';
+import { type ChunkRecord } from './store.ts';
+export interface BuildOptions {
+    /** files, directories or patterns, as they were named */
+    files: readonly string[];
+    cwd: string;
+    out: string;
+    embedder: Embedder;
+    /** the reference as it was written, which is what a later search will type */
+    embeddingRef?: string;
+    /** told the manifest, so a store can say what wrote it */
+    indexer: string;
+    /** texts sent to the embedder at once */
+    batch?: number;
+    chunk?: ChunkOptions;
+    signal?: AbortSignal;
+    /** what the documents turned out to hold, before a vector has been paid for */
+    onRead?: (summary: BuildSummary) => void;
+    onProgress?: (done: number, total: number) => void;
+}
+export interface BuildSummary {
+    sources: DocRecord[];
+    counts: Counts;
+    skipped: Corpus['skipped'];
+}
+export interface BuildResult {
+    manifest: Manifest;
+    chunks: ChunkRecord[];
+}
+export declare function buildIndex(options: BuildOptions): Promise<BuildResult>;
+//# sourceMappingURL=build.d.ts.map

package/dist/docs/build.js

@@ -0,0 +1,108 @@
+import { beginBuild } from "../common/progress.js";
+import { formatLines } from "./chunk.js";
+import { INDEX_VERSION, SOURCES_DIR, writeIndex, } from "./files.js";
+import { loadDocuments } from "./load.js";
+import { DOCS_REPORT, PHASES } from "./readme.js";
+import { writeChunks } from "./store.js";
+const DEFAULT_BATCH = 96;
+export async function buildIndex(options) {
+    const journal = beginBuild({
+        dir: options.out,
+        files: options.files,
+        embedding: options.embeddingRef ?? options.embedder.id,
+        indexer: options.indexer,
+        phases: PHASES,
+        report: DOCS_REPORT,
+    });
+    try {
+        const corpus = await loadDocuments(options.files, options.cwd, options.chunk);
+        const chunks = recordsOf(corpus);
+        const sources = corpus.docs.map((doc) => ({
+            name: doc.name,
+            file: doc.file,
+            path: `${SOURCES_DIR}/${doc.name}`,
+            sha256: doc.sha256,
+            format: doc.format,
+            title: doc.outline.title,
+            bytes: doc.bytes,
+            lines: doc.outline.lines,
+            sections: doc.outline.headings.length,
+            tables: doc.outline.tables.length,
+            chunks: doc.chunks.length,
+        }));
+        const counts = {
+            documents: sources.length,
+            chunks: chunks.length,
+            lines: sources.reduce((n, s) => n + s.lines, 0),
+            sections: sources.reduce((n, s) => n + s.sections, 0),
+            tables: sources.reduce((n, s) => n + s.tables, 0),
+        };
+        journal.read(counts, chunks.length);
+        options.onRead?.({ sources, counts, skipped: corpus.skipped });
+        journal.phase('embedding');
+        const vectors = await embedAll(chunks, options, journal);
+        journal.phase('writing');
+        const written = await writeChunks(options.out, chunks, vectors);
+        const manifest = {
+            version: INDEX_VERSION,
+            kind: 'docs',
+            createdAt: new Date().toISOString(),
+            indexer: options.indexer,
+            embedding: {
+                ref: options.embeddingRef ?? options.embedder.id,
+                id: options.embedder.id,
+                dimensions: vectors[0]?.length ?? 0,
+            },
+            sources,
+            counts,
+            indexes: { fts: written.fts, vector: written.vector },
+        };
+        const outline = { files: corpus.docs.map((doc) => doc.outline) };
+        const documents = Object.fromEntries(corpus.docs.map((doc) => [doc.name, doc.text]));
+        await writeIndex(options.out, { manifest, outline, documents });
+        journal.finish(manifest);
+        return { manifest, chunks };
+    }
+    catch (err) {
+        journal.fail(err);
+        throw err;
+    }
+}
+/** One row per chunk, with the render set encoded and the document name on it. */
+function recordsOf(corpus) {
+    return corpus.docs.flatMap((doc) => doc.chunks.map((chunk) => ({
+        id: `${doc.name}#c${chunk.index}`,
+        path: doc.name,
+        ordinal: chunk.index,
+        kind: chunk.kind,
+        text: chunk.text,
+        embedText: chunk.embedText,
+        lineSpec: formatLines(chunk.lineNumbers),
+        bodyStart: chunk.bodyStart,
+        bodyEnd: chunk.bodyEnd,
+        structureId: chunk.structureId,
+        structurePath: chunk.structurePath,
+        headings: chunk.headings,
+        tokens: chunk.tokens,
+    })));
+}
+async function embedAll(chunks, options, journal) {
+    const size = options.batch ?? DEFAULT_BATCH;
+    const out = [];
+    for (let at = 0; at < chunks.length; at += size) {
+        const slice = chunks.slice(at, at + size);
+        const response = await options.embedder.embed({
+            input: slice.map((c) => c.embedText),
+            taskType: 'document',
+            signal: options.signal,
+        });
+        if (response.vectors.length !== slice.length) {
+            throw new Error(`${options.embedder.id} answered ${response.vectors.length} vectors for ${slice.length} texts`);
+        }
+        out.push(...response.vectors.map((v) => Float32Array.from(v)));
+        journal.progress(out.length, chunks.length);
+        options.onProgress?.(out.length, chunks.length);
+    }
+    return out;
+}
+//# sourceMappingURL=build.js.map

package/dist/docs/chunk.d.ts

@@ -0,0 +1,73 @@
+import { type ParsedDoc } from './parse.ts';
+/** Soft target, hard ceiling, and the width at which one table row is too wide. */
+export declare const CHUNK_TOKENS = 384;
+export declare const MAX_CHUNK_TOKENS = 512;
+export declare const TABLE_SLICE_TOKENS = 128;
+/** How much of a table its descriptor stands for when no row of it matched. */
+export declare const TABLE_PREVIEW_ROWS = 3;
+/**
+ * And how many rows may share one. The token budget alone would put a narrow
+ * sixteen-row table in a single chunk, so matching one row of it quotes all
+ * sixteen — the rows are independent facts, and a reader asking about one is
+ * not asking about the other fifteen.
+ */
+export declare const TABLE_ROWS_PER_CHUNK = 4;
+/**
+ * Below this a chunk is merged into its neighbour rather than retrieved alone.
+ *
+ * BM25 normalises by document length, so a nine-word chunk that happens to
+ * contain two query terms outscores a real answer that contains them among a
+ * hundred other words. Measured on this repository, a one-line aside about
+ * markdown link syntax took full-text rank 0 for "how to create docs index"
+ * while the vector leg — correctly — put it 185th. It is not that the chunk is
+ * wrong; it is that alone it is not a passage, and a passage is what the
+ * lexical index is scoring.
+ */
+export declare const MIN_CHUNK_TOKENS = 48;
+/**
+ * How much of the block before a chunk may carry for continuity. It is one
+ * line, which is a whisper until the line is the whole of a README's challenge
+ * table on one row: 92kB of badge markup, none of it cited by a line number,
+ * outweighing the seven lines the chunk actually stands for by eighty to one.
+ */
+export declare const CARRY_TOKENS = 32;
+/**
+ * Four characters to a token, which is within about 15% for English prose and
+ * wrong for CJK, for dense numeric cells and for long identifiers. It cannot
+ * cause a request to fail — 512 estimated tokens sits far below any embedding
+ * model's limit, so even a threefold underestimate has headroom. Injectable so
+ * a real tokenizer is a one-line swap if the corpus ever needs one.
+ */
+export declare const tokenCount: (text: string) => number;
+/** The kinds a chunk can be, which is what `--kind` filters on. */
+export declare const CHUNK_KINDS: readonly ["paragraph", "list", "table", "table_row", "code", "frontmatter", "html"];
+export type ChunkKind = (typeof CHUNK_KINDS)[number];
+export interface ChunkOptions {
+    chunkTokens?: number;
+    minChunkTokens?: number;
+    maxChunkTokens?: number;
+    tableSliceTokens?: number;
+    tokenCount?: (text: string) => number;
+}
+export interface Chunk {
+    index: number;
+    kind: ChunkKind;
+    /** the innermost structure node containing the whole body */
+    structureId: string;
+    structurePath: string;
+    /** the breadcrumb, from the document name down to the nearest heading */
+    headings: string;
+    bodyStart: number;
+    bodyEnd: number;
+    /** everything that gets rendered: headings, prelude and body, sorted */
+    lineNumbers: number[];
+    /** the full-text document — wider */
+    text: string;
+    /** the vector's source — tighter */
+    embedText: string;
+    tokens: number;
+}
+export declare function chunkDocument(doc: ParsedDoc, options?: ChunkOptions): Chunk[];
+export declare function formatLines(numbers: readonly number[]): string;
+export declare function parseLines(spec: string): number[];
+//# sourceMappingURL=chunk.d.ts.map