@lloyal-labs/rig 1.2.0

package/README.md

@@ -0,0 +1,188 @@
+# @lloyal-labs/rig
+
+Retrieval-Interleaved Generation for [lloyal-agents](../agents).
+
+```bash
+npm i @lloyal-labs/rig @lloyal-labs/lloyal-agents @lloyal-labs/lloyal.node
+```
+
+## RIG vs RAG
+
+RAG retrieves first, then generates. A retrieval step runs upfront — query the vector DB, get top-k passages, inject them into the prompt, call the model once. The model sees static context. Retrieval and generation are separate phases.
+
+RIG interleaves retrieval and generation inside the decode loop. Agents generate reasoning, decide to search, process results, reason further, fetch a page, form hypotheses from the content, search again with refined queries. Retrieval decisions emerge from ongoing generation — each search query is informed by everything the agent has already discovered.
+
+The difference is observable in tool call inputs. A RAG system constructs search queries from the original user question. A RIG agent constructs queries from hypotheses formed during generation:
+
+```
+grep(/memory leak/)            → 3 matches in 2 files
+read_file(pool.ts L40-80)      → reads allocation logic, spots missing cleanup
+search("resource cleanup on connection close")  → finds teardown handler
+read_file(server.ts L120-155)  → discovers close handler never calls pool.drain()
+grep(/drain|dispose|cleanup/)  → 8 matches, confirms drain exists but is unused
+search("pool drain connection lifecycle interaction")  → targets the gap
+report(findings)
+```
+
+The last search — `"pool drain connection lifecycle interaction"` — is the signature behavior. The agent read the allocation logic, discovered the drain method existed but was never called on connection close, and constructed a search specifically targeting that interaction. This is multi-hop reasoning: not "search and report" but "search, form hypothesis, search for confirmation."
+
+### Why it's emergent
+
+This behavior is not prompted or engineered. It emerges from the concurrency semantics of `lloyal-agents`.
+
+The four-phase tick loop creates a clean decision boundary between each tool call and the next generation step:
+
+1. Agent generates tokens, hits stop token, tool call extracted
+2. Tool executes to completion — agent is suspended
+3. Tool result fully prefilled into the agent's KV cache
+4. Grammar state resets — clean slate for next decision
+5. Agent resumes generating with the complete result as the last thing in context
+
+Step 5 is the critical moment. The model's next-token prediction operates on a context where the tool result is fully present and the grammar is clean. The model makes a fresh decision: call another tool, call the same tool with different arguments, or report findings. This decision is informed by everything the agent has seen — all prior tool results are physically present in the branch's KV cache.
+
+An agent that greps with a narrow pattern and gets 0 matches will broaden the pattern on its next grep — not because it's prompted to retry, but because the 0-match result is in context and the model naturally adjusts. An agent that reads a section and discovers an unexpected connection will construct a search query targeting that specific connection — the read result is in context, and the model forms a hypothesis from it.
+
+Under a concurrent dispatch model where tool results arrive mid-generation, the agent is already producing tokens when results land. The result gets incorporated, but there's no clean pause for hypothesis formation. The observable effect: sequential dispatch produces progressively more specific queries; concurrent dispatch produces variations on the original question.
+
+Depth scales with `maxTurns`. At 2 turns, agents do single-shot retrieval. At 6 turns, agents do 3-4 rounds of iterative refinement. At 20 turns, agents go deep — following citation chains, cross-referencing claims, building evidence maps. The quality difference is in the later tool call inputs.
+
+## Sources
+
+`@lloyal-labs/rig` provides two `Source` implementations (extending the base class from `lloyal-agents`):
+
+**CorpusSource** — local files with grep, semantic search, read_file, and recursive research tools. Agents investigate a knowledge base by pattern matching, reading sections in context, and spawning sub-agents for deeper investigation.
+
+**WebSource** — web search via [Tavily](https://tavily.com), page fetching with attention-based content extraction, and recursive web_research tools. `BufferingFetchPage` wraps fetch results — full content goes to the agent for reasoning, while a parallel buffer stores content for post-research reranking. Content extraction uses `generate({ parent })` to attend over the fetched page and extract summary + links via grammar-constrained generation, then prunes the fork — zero net KV cost per extraction.
+
+Sources are composable. A pipeline can use one source, both, or custom implementations:
+
+```typescript
+import {
+  CorpusSource,
+  WebSource,
+  TavilyProvider,
+  loadResources,
+  chunkResources,
+} from "@lloyal-labs/rig";
+
+const sources = [];
+
+// Local knowledge base
+if (corpusDir) {
+  const resources = loadResources(corpusDir);
+  const chunks = chunkResources(resources);
+  sources.push(new CorpusSource(resources, chunks));
+}
+
+// Web search
+if (process.env.TAVILY_API_KEY) {
+  sources.push(new WebSource(new TavilyProvider()));
+}
+```
+
+When multiple sources are used, they run sequentially — each source gets the full KV budget. After source N completes, its inner branches are pruned and KV is freed for source N+1.
+
+### Bridge
+
+Between sources, a bridge exit gate structures discoveries from the completed source as durable context for the next source's investigation:
+
+```
+Corpus research → Bridge → Web research → Synthesize
+```
+
+The bridge extracts three tiers of discovery:
+
+1. **What was established** — specific data points, study details, statistics, quotes. Evidence preserved verbatim.
+2. **Where evidence is incomplete** — acknowledged limitations, absent study designs, uncertain mechanisms. These are well-researched claims with identified evidence gaps.
+3. **What was not covered** — topics mentioned but not substantiated, or entirely absent.
+
+The distinction between (2) and (3) is critical. A topic with six sections of evidence but no experimental validation is not a gap — it is a well-researched claim with an identified evidence limitation. The bridge flags the limitation, not the topic. This prevents the next source from re-investigating what the previous source already covered, and directs it toward genuine gaps.
+
+Bridge discoveries condition the next source's questions:
+
+```typescript
+activeQuestions = questions.map(
+  (q) => `${q}\n\nPrior research discoveries:\n${discoveries}`,
+);
+```
+
+## Pipeline
+
+A typical RIG pipeline:
+
+```
+Plan → Research → [Bridge →] Synthesize → Eval
+```
+
+**Plan.** Grammar-constrained decomposition of the user query into sub-questions with intent classification (`research` vs `clarify`). If the query is focused enough to investigate directly, produces an empty array (passthrough). Uses `generate()` with a JSON schema grammar — the model outputs structured `{ questions: [{ text, intent }] }` in a single generation pass.
+
+**Research.** Each source's research tool spawns a pool of agents that investigate sub-questions. Agents interleave retrieval and generation — searching, reading, forming hypotheses, searching again. Within each source, all agents run concurrently on shared GPU compute via `useAgentPool`. Sources run sequentially, each getting the full KV budget.
+
+Agents that get cut by context pressure (their tool results exceeded KV headroom) are recovered via scratchpad extraction — `generate({ parent: agent.branch })` with a grammar-constrained reporter prompt attends over the agent's accumulated KV and extracts findings. The agent paid the KV cost of reasoning; the extraction recovers the value.
+
+**Bridge.** Runs between sources when multiple sources are configured. A single agent with report-only tools structures discoveries from the completed source. The bridge output conditions the next source's sub-questions, directing investigation toward gaps rather than re-covering established ground.
+
+**Synthesize.** A synthesis agent integrates findings from all sources into a structured report with source attribution. Research notes provide analytical structure; reranked source passages provide ground truth for citation. The synthesizer cross-references both — using research notes to identify what matters, and source passages for evidence.
+
+**Eval.** Multi-branch semantic comparison via `diverge()`. Fork N branches from a shared frontier, generate independently with the same verify prompt, check convergence. Where branches agree, the model is confident. Where they diverge, the answer needs refinement.
+
+## Tools
+
+### Corpus tools
+
+| Tool           | Description                                                             |
+| -------------- | ----------------------------------------------------------------------- |
+| `SearchTool`   | Semantic search over corpus chunks via reranker scoring                 |
+| `GrepTool`     | Exhaustive regex pattern matching across all files                      |
+| `ReadFileTool` | Read file content at specified line ranges, tracks per-agent read state |
+| `ResearchTool` | Spawn sub-agent pool for deeper investigation of sub-questions          |
+| `ReportTool`   | Terminal tool — agents call this to submit findings                     |
+
+### Web tools
+
+| Tool              | Description                                                     |
+| ----------------- | --------------------------------------------------------------- |
+| `WebSearchTool`   | Web search via configurable provider (Tavily included)          |
+| `FetchPageTool`   | Fetch URL, extract article text via Readability                 |
+| `WebResearchTool` | Spawn sub-agent pool with web tools for recursive investigation |
+
+### Pipeline tools
+
+| Tool                        | Description                                                        |
+| --------------------------- | ------------------------------------------------------------------ |
+| `PlanTool`                  | Grammar-constrained query decomposition with intent classification |
+| `createTools(opts)`         | Build corpus toolkit from resources, chunks, and reranker          |
+| `createReranker(modelPath)` | Semantic reranker for chunk scoring and passage selection          |
+
+## Custom Sources
+
+Extend `Source` from `lloyal-agents` to create custom sources:
+
+```typescript
+import { Source } from "@lloyal-labs/lloyal-agents";
+import type { Tool } from "@lloyal-labs/lloyal-agents";
+
+class DatabaseSource extends Source<SourceContext, Row> {
+  readonly name = "database";
+
+  get researchTool(): Tool {
+    return this._researchTool;
+  }
+
+  *bind(ctx: SourceContext) {
+    // Set up tools with access to ctx.parent (for generate({ parent })),
+    // ctx.reranker, ctx.reporterPrompt, ctx.reportTool
+    this._researchTool = new DatabaseResearchTool(/* ... */);
+  }
+
+  getChunks(): Row[] {
+    return this._results; // buffered for post-research reranking
+  }
+}
+```
+
+The `bind()` lifecycle receives a `SourceContext` with the parent branch (for forking), reranker, reporter prompt, and report tool. Your research tool calls `useAgentPool` or `runAgents` internally — same primitives the built-in sources use.
+
+## License
+
+Apache-2.0

package/dist/index.d.ts

@@ -0,0 +1,19 @@
+/**
+ * Rig — research infrastructure for the lloyal agent pipeline
+ *
+ * Provides source implementations ({@link WebSource}, {@link CorpusSource}),
+ * resource loading/chunking, reranking, and the tool library used by
+ * deep-research harnesses. Sources are composed via the abstract
+ * {@link Source} base class from `@lloyal-labs/lloyal-agents`.
+ *
+ * @packageDocumentation
+ * @category Rig
+ */
+export { createTools, reportTool, ResearchTool, WebSearchTool, TavilyProvider, FetchPageTool, WebResearchTool, PlanTool, } from './tools';
+export type { ResearchToolOpts, WebResearchToolOpts, PlanToolOpts, PlanResult, PlanQuestion, SearchProvider, SearchResult, Reranker, ScoredChunk, ScoredResult, } from './tools';
+export { WebSource, CorpusSource } from './sources';
+export type { SourceContext } from './sources';
+export { loadResources, chunkResources } from './resources';
+export type { Resource, Chunk } from './resources';
+export { createReranker } from './reranker';
+//# sourceMappingURL=index.d.ts.map

package/dist/index.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":"AAAA;;;;;;;;;;GAUG;AAGH,OAAO,EACL,WAAW,EAAE,UAAU,EACvB,YAAY,EAAE,aAAa,EAAE,cAAc,EAAE,aAAa,EAC1D,eAAe,EAAE,QAAQ,GAC1B,MAAM,SAAS,CAAC;AACjB,YAAY,EACV,gBAAgB,EAAE,mBAAmB,EAAE,YAAY,EACnD,UAAU,EAAE,YAAY,EACxB,cAAc,EAAE,YAAY,EAC5B,QAAQ,EAAE,WAAW,EAAE,YAAY,GACpC,MAAM,SAAS,CAAC;AAGjB,OAAO,EAAE,SAAS,EAAE,YAAY,EAAE,MAAM,WAAW,CAAC;AACpD,YAAY,EAAE,aAAa,EAAE,MAAM,WAAW,CAAC;AAG/C,OAAO,EAAE,aAAa,EAAE,cAAc,EAAE,MAAM,aAAa,CAAC;AAC5D,YAAY,EAAE,QAAQ,EAAE,KAAK,EAAE,MAAM,aAAa,CAAC;AAGnD,OAAO,EAAE,cAAc,EAAE,MAAM,YAAY,CAAC"}

package/dist/index.js

@@ -0,0 +1,36 @@
+"use strict";
+/**
+ * Rig — research infrastructure for the lloyal agent pipeline
+ *
+ * Provides source implementations ({@link WebSource}, {@link CorpusSource}),
+ * resource loading/chunking, reranking, and the tool library used by
+ * deep-research harnesses. Sources are composed via the abstract
+ * {@link Source} base class from `@lloyal-labs/lloyal-agents`.
+ *
+ * @packageDocumentation
+ * @category Rig
+ */
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.createReranker = exports.chunkResources = exports.loadResources = exports.CorpusSource = exports.WebSource = exports.PlanTool = exports.WebResearchTool = exports.FetchPageTool = exports.TavilyProvider = exports.WebSearchTool = exports.ResearchTool = exports.reportTool = exports.createTools = void 0;
+// Tools
+var tools_1 = require("./tools");
+Object.defineProperty(exports, "createTools", { enumerable: true, get: function () { return tools_1.createTools; } });
+Object.defineProperty(exports, "reportTool", { enumerable: true, get: function () { return tools_1.reportTool; } });
+Object.defineProperty(exports, "ResearchTool", { enumerable: true, get: function () { return tools_1.ResearchTool; } });
+Object.defineProperty(exports, "WebSearchTool", { enumerable: true, get: function () { return tools_1.WebSearchTool; } });
+Object.defineProperty(exports, "TavilyProvider", { enumerable: true, get: function () { return tools_1.TavilyProvider; } });
+Object.defineProperty(exports, "FetchPageTool", { enumerable: true, get: function () { return tools_1.FetchPageTool; } });
+Object.defineProperty(exports, "WebResearchTool", { enumerable: true, get: function () { return tools_1.WebResearchTool; } });
+Object.defineProperty(exports, "PlanTool", { enumerable: true, get: function () { return tools_1.PlanTool; } });
+// Sources
+var sources_1 = require("./sources");
+Object.defineProperty(exports, "WebSource", { enumerable: true, get: function () { return sources_1.WebSource; } });
+Object.defineProperty(exports, "CorpusSource", { enumerable: true, get: function () { return sources_1.CorpusSource; } });
+// Resources
+var resources_1 = require("./resources");
+Object.defineProperty(exports, "loadResources", { enumerable: true, get: function () { return resources_1.loadResources; } });
+Object.defineProperty(exports, "chunkResources", { enumerable: true, get: function () { return resources_1.chunkResources; } });
+// Reranker
+var reranker_1 = require("./reranker");
+Object.defineProperty(exports, "createReranker", { enumerable: true, get: function () { return reranker_1.createReranker; } });
+//# sourceMappingURL=index.js.map

package/dist/index.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"index.js","sourceRoot":"","sources":["../src/index.ts"],"names":[],"mappings":";AAAA;;;;;;;;;;GAUG;;;AAEH,QAAQ;AACR,iCAIiB;AAHf,oGAAA,WAAW,OAAA;AAAE,mGAAA,UAAU,OAAA;AACvB,qGAAA,YAAY,OAAA;AAAE,sGAAA,aAAa,OAAA;AAAE,uGAAA,cAAc,OAAA;AAAE,sGAAA,aAAa,OAAA;AAC1D,wGAAA,eAAe,OAAA;AAAE,iGAAA,QAAQ,OAAA;AAS3B,UAAU;AACV,qCAAoD;AAA3C,oGAAA,SAAS,OAAA;AAAE,uGAAA,YAAY,OAAA;AAGhC,YAAY;AACZ,yCAA4D;AAAnD,0GAAA,aAAa,OAAA;AAAE,2GAAA,cAAc,OAAA;AAGtC,WAAW;AACX,uCAA4C;AAAnC,0GAAA,cAAc,OAAA"}

package/dist/reranker.d.ts

@@ -0,0 +1,22 @@
+import type { Reranker } from "./tools/types";
+/**
+ * Create a {@link Reranker} backed by a dedicated reranking model context
+ *
+ * Loads a separate model (typically a cross-encoder) into its own KV cache
+ * and exposes `score`, `tokenizeChunks`, and `dispose` methods. The returned
+ * `score` method yields {@link ScoredResult} batches as an async iterable,
+ * mapping raw indices back to the original {@link Chunk} metadata.
+ *
+ * @param modelPath - Absolute path to the reranking model file (GGUF)
+ * @param opts - Optional context sizing overrides
+ * @param opts.nSeqMax - Maximum parallel scoring sequences (default 8)
+ * @param opts.nCtx - Context window size for the reranker model (default 4096)
+ * @returns A ready-to-use reranker instance; call `dispose()` when finished
+ *
+ * @category Rig
+ */
+export declare function createReranker(modelPath: string, opts?: {
+    nSeqMax?: number;
+    nCtx?: number;
+}): Promise<Reranker>;
+//# sourceMappingURL=reranker.d.ts.map

package/dist/reranker.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"reranker.d.ts","sourceRoot":"","sources":["../src/reranker.ts"],"names":[],"mappings":"AAIA,OAAO,KAAK,EAAE,QAAQ,EAAgB,MAAM,eAAe,CAAC;AAE5D;;;;;;;;;;;;;;;GAeG;AACH,wBAAsB,cAAc,CAClC,SAAS,EAAE,MAAM,EACjB,IAAI,CAAC,EAAE;IAAE,OAAO,CAAC,EAAE,MAAM,CAAC;IAAC,IAAI,CAAC,EAAE,MAAM,CAAA;CAAE,GACzC,OAAO,CAAC,QAAQ,CAAC,CA4DnB"}

package/dist/reranker.js

@@ -0,0 +1,76 @@
+"use strict";
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.createReranker = createReranker;
+const lloyal_node_1 = require("@lloyal-labs/lloyal.node");
+const sdk_1 = require("@lloyal-labs/sdk");
+/**
+ * Create a {@link Reranker} backed by a dedicated reranking model context
+ *
+ * Loads a separate model (typically a cross-encoder) into its own KV cache
+ * and exposes `score`, `tokenizeChunks`, and `dispose` methods. The returned
+ * `score` method yields {@link ScoredResult} batches as an async iterable,
+ * mapping raw indices back to the original {@link Chunk} metadata.
+ *
+ * @param modelPath - Absolute path to the reranking model file (GGUF)
+ * @param opts - Optional context sizing overrides
+ * @param opts.nSeqMax - Maximum parallel scoring sequences (default 8)
+ * @param opts.nCtx - Context window size for the reranker model (default 4096)
+ * @returns A ready-to-use reranker instance; call `dispose()` when finished
+ *
+ * @category Rig
+ */
+async function createReranker(modelPath, opts) {
+    const nSeqMax = opts?.nSeqMax ?? 8;
+    const nCtx = opts?.nCtx ?? 4096;
+    const ctx = await (0, lloyal_node_1.createContext)({
+        modelPath,
+        nCtx,
+        nSeqMax,
+        typeK: 'q4_0',
+        typeV: 'q4_0',
+    });
+    const rerank = await sdk_1.Rerank.create(ctx, { nSeqMax, nCtx });
+    return {
+        score(query, chunks) {
+            const inner = rerank.score(query, chunks.map((c) => c.tokens), 10);
+            return {
+                [Symbol.asyncIterator]() {
+                    const it = inner[Symbol.asyncIterator]();
+                    return {
+                        async next() {
+                            const { value, done } = await it.next();
+                            if (done)
+                                return {
+                                    value: undefined,
+                                    done: true,
+                                };
+                            return {
+                                value: {
+                                    filled: value.filled,
+                                    total: value.total,
+                                    results: value.results.map((r) => ({
+                                        file: chunks[r.index].resource,
+                                        heading: chunks[r.index].heading,
+                                        score: r.score,
+                                        startLine: chunks[r.index].startLine,
+                                        endLine: chunks[r.index].endLine,
+                                    })),
+                                },
+                                done: false,
+                            };
+                        },
+                    };
+                },
+            };
+        },
+        async tokenizeChunks(chunks) {
+            for (const chunk of chunks) {
+                chunk.tokens = await rerank.tokenize(chunk.text);
+            }
+        },
+        dispose() {
+            rerank.dispose();
+        },
+    };
+}
+//# sourceMappingURL=reranker.js.map

package/dist/reranker.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"reranker.js","sourceRoot":"","sources":["../src/reranker.ts"],"names":[],"mappings":";;AAsBA,wCA+DC;AArFD,0DAAyD;AACzD,0CAA0C;AAK1C;;;;;;;;;;;;;;;GAeG;AACI,KAAK,UAAU,cAAc,CAClC,SAAiB,EACjB,IAA0C;IAE1C,MAAM,OAAO,GAAG,IAAI,EAAE,OAAO,IAAI,CAAC,CAAC;IACnC,MAAM,IAAI,GAAG,IAAI,EAAE,IAAI,IAAI,IAAI,CAAC;IAChC,MAAM,GAAG,GAAG,MAAM,IAAA,2BAAa,EAAC;QAC9B,SAAS;QACT,IAAI;QACJ,OAAO;QACP,KAAK,EAAE,MAAM;QACb,KAAK,EAAE,MAAM;KACd,CAAC,CAAC;IACH,MAAM,MAAM,GAAG,MAAM,YAAM,CAAC,MAAM,CAAC,GAAgC,EAAE,EAAE,OAAO,EAAE,IAAI,EAAE,CAAC,CAAC;IAExF,OAAO;QACL,KAAK,CAAC,KAAa,EAAE,MAAe;YAClC,MAAM,KAAK,GAAG,MAAM,CAAC,KAAK,CACxB,KAAK,EACL,MAAM,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,MAAM,CAAC,EAC3B,EAAE,CACH,CAAC;YACF,OAAO;gBACL,CAAC,MAAM,CAAC,aAAa,CAAC;oBACpB,MAAM,EAAE,GAAG,KAAK,CAAC,MAAM,CAAC,aAAa,CAAC,EAAE,CAAC;oBACzC,OAAO;wBACL,KAAK,CAAC,IAAI;4BACR,MAAM,EAAE,KAAK,EAAE,IAAI,EAAE,GAAG,MAAM,EAAE,CAAC,IAAI,EAAE,CAAC;4BACxC,IAAI,IAAI;gCACN,OAAO;oCACL,KAAK,EAAE,SAAoC;oCAC3C,IAAI,EAAE,IAAI;iCACX,CAAC;4BACJ,OAAO;gCACL,KAAK,EAAE;oCACL,MAAM,EAAE,KAAK,CAAC,MAAM;oCACpB,KAAK,EAAE,KAAK,CAAC,KAAK;oCAClB,OAAO,EAAE,KAAK,CAAC,OAAO,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC;wCACjC,IAAI,EAAE,MAAM,CAAC,CAAC,CAAC,KAAK,CAAC,CAAC,QAAQ;wCAC9B,OAAO,EAAE,MAAM,CAAC,CAAC,CAAC,KAAK,CAAC,CAAC,OAAO;wCAChC,KAAK,EAAE,CAAC,CAAC,KAAK;wCACd,SAAS,EAAE,MAAM,CAAC,CAAC,CAAC,KAAK,CAAC,CAAC,SAAS;wCACpC,OAAO,EAAE,MAAM,CAAC,CAAC,CAAC,KAAK,CAAC,CAAC,OAAO;qCACjC,CAAC,CAAC;iCACJ;gCACD,IAAI,EAAE,KAAK;6BACZ,CAAC;wBACJ,CAAC;qBACF,CAAC;gBACJ,CAAC;aACF,CAAC;QACJ,CAAC;QAED,KAAK,CAAC,cAAc,CAAC,MAAe;YAClC,KAAK,MAAM,KAAK,IAAI,MAAM,EAAE,CAAC;gBAC3B,KAAK,CAAC,MAAM,GAAG,MAAM,MAAM,CAAC,QAAQ,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC;YACnD,CAAC;QACH,CAAC;QAED,OAAO;YACL,MAAM,CAAC,OAAO,EAAE,CAAC;QACnB,CAAC;KACF,CAAC;AACJ,CAAC"}

package/dist/resources/files.d.ts

@@ -0,0 +1,28 @@
+import type { Resource, Chunk } from './types';
+/**
+ * Load documents from a directory (or single file) into {@link Resource} objects
+ *
+ * If `dir` is a file path, returns a single-element array. If it is a
+ * directory, reads all `.md` files within it. Exits the process with an
+ * error message if the path does not exist or contains no Markdown files.
+ *
+ * @param dir - Absolute path to a directory of `.md` files or a single file
+ * @returns Array of loaded resources with file name and content
+ *
+ * @category Rig
+ */
+export declare function loadResources(dir: string): Resource[];
+/**
+ * Split loaded resources into {@link Chunk} instances for reranking
+ *
+ * Uses native Markdown heading detection (via `parseMarkdown`) to produce
+ * section-level chunks. Falls back to blank-line paragraph splitting for
+ * resources with no headings (or fewer than 10 lines of content).
+ *
+ * @param resources - Resources to chunk (from {@link loadResources})
+ * @returns Flat array of chunks across all resources, ready for {@link Reranker.tokenizeChunks}
+ *
+ * @category Rig
+ */
+export declare function chunkResources(resources: Resource[]): Chunk[];
+//# sourceMappingURL=files.d.ts.map

package/dist/resources/files.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"files.d.ts","sourceRoot":"","sources":["../../src/resources/files.ts"],"names":[],"mappings":"AAGA,OAAO,KAAK,EAAE,QAAQ,EAAE,KAAK,EAAE,MAAM,SAAS,CAAC;AAK/C;;;;;;;;;;;GAWG;AACH,wBAAgB,aAAa,CAAC,GAAG,EAAE,MAAM,GAAG,QAAQ,EAAE,CAkBrD;AA0BD;;;;;;;;;;;GAWG;AACH,wBAAgB,cAAc,CAAC,SAAS,EAAE,QAAQ,EAAE,GAAG,KAAK,EAAE,CAoB7D"}

package/dist/resources/files.js

@@ -0,0 +1,98 @@
+"use strict";
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.loadResources = loadResources;
+exports.chunkResources = chunkResources;
+const fs = require("node:fs");
+const path = require("node:path");
+const lloyal_node_1 = require("@lloyal-labs/lloyal.node");
+const { parseMarkdown } = (0, lloyal_node_1.loadBinary)();
+/**
+ * Load documents from a directory (or single file) into {@link Resource} objects
+ *
+ * If `dir` is a file path, returns a single-element array. If it is a
+ * directory, reads all `.md` files within it. Exits the process with an
+ * error message if the path does not exist or contains no Markdown files.
+ *
+ * @param dir - Absolute path to a directory of `.md` files or a single file
+ * @returns Array of loaded resources with file name and content
+ *
+ * @category Rig
+ */
+function loadResources(dir) {
+    if (!fs.existsSync(dir)) {
+        process.stdout.write(`Error: corpus not found: ${dir}\n`);
+        process.exit(1);
+    }
+    const stat = fs.statSync(dir);
+    if (stat.isFile()) {
+        return [{ name: path.basename(dir), content: fs.readFileSync(dir, 'utf8') }];
+    }
+    const files = fs.readdirSync(dir).filter((f) => f.endsWith('.md'));
+    if (!files.length) {
+        process.stdout.write(`Error: no .md files in: ${dir}\n`);
+        process.exit(1);
+    }
+    return files.map((f) => ({
+        name: f,
+        content: fs.readFileSync(path.join(dir, f), 'utf8'),
+    }));
+}
+/** Split plain text into chunks on blank-line paragraph boundaries */
+function chunkByParagraph(res) {
+    const lines = res.content.split('\n');
+    const chunks = [];
+    let start = 0;
+    for (let i = 0; i <= lines.length; i++) {
+        const blank = i === lines.length || !lines[i].trim();
+        if (blank && i > start) {
+            const text = lines.slice(start, i).join('\n').trim();
+            if (text) {
+                chunks.push({
+                    resource: res.name,
+                    heading: text.slice(0, 60).replace(/\n/g, ' ') + (text.length > 60 ? '\u2026' : ''),
+                    text, tokens: [],
+                    startLine: start + 1,
+                    endLine: i,
+                });
+            }
+        }
+        if (blank)
+            start = i + 1;
+    }
+    return chunks;
+}
+/**
+ * Split loaded resources into {@link Chunk} instances for reranking
+ *
+ * Uses native Markdown heading detection (via `parseMarkdown`) to produce
+ * section-level chunks. Falls back to blank-line paragraph splitting for
+ * resources with no headings (or fewer than 10 lines of content).
+ *
+ * @param resources - Resources to chunk (from {@link loadResources})
+ * @returns Flat array of chunks across all resources, ready for {@link Reranker.tokenizeChunks}
+ *
+ * @category Rig
+ */
+function chunkResources(resources) {
+    const out = [];
+    for (const res of resources) {
+        const sections = parseMarkdown(res.content);
+        // Single section covering the whole file = no headings found -> paragraph split
+        if (sections.length <= 1 && res.content.split('\n').length > 10) {
+            out.push(...chunkByParagraph(res));
+            continue;
+        }
+        const lines = res.content.split('\n');
+        for (const sec of sections) {
+            const text = lines.slice(sec.startLine - 1, sec.endLine).join('\n').trim();
+            if (!text)
+                continue;
+            out.push({
+                resource: res.name, heading: sec.heading || res.name, text, tokens: [],
+                startLine: sec.startLine, endLine: sec.endLine,
+            });
+        }
+    }
+    return out;
+}
+//# sourceMappingURL=files.js.map

package/dist/resources/files.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"files.js","sourceRoot":"","sources":["../../src/resources/files.ts"],"names":[],"mappings":";;AAoBA,sCAkBC;AAsCD,wCAoBC;AAhGD,8BAA8B;AAC9B,kCAAkC;AAClC,0DAAsD;AAItD,MAAM,EAAE,aAAa,EAAE,GAAG,IAAA,wBAAU,GAA2D,CAAC;AAEhG;;;;;;;;;;;GAWG;AACH,SAAgB,aAAa,CAAC,GAAW;IACvC,IAAI,CAAC,EAAE,CAAC,UAAU,CAAC,GAAG,CAAC,EAAE,CAAC;QACxB,OAAO,CAAC,MAAM,CAAC,KAAK,CAAC,4BAA4B,GAAG,IAAI,CAAC,CAAC;QAC1D,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC;IAClB,CAAC;IACD,MAAM,IAAI,GAAG,EAAE,CAAC,QAAQ,CAAC,GAAG,CAAC,CAAC;IAC9B,IAAI,IAAI,CAAC,MAAM,EAAE,EAAE,CAAC;QAClB,OAAO,CAAC,EAAE,IAAI,EAAE,IAAI,CAAC,QAAQ,CAAC,GAAG,CAAC,EAAE,OAAO,EAAE,EAAE,CAAC,YAAY,CAAC,GAAG,EAAE,MAAM,CAAC,EAAE,CAAC,CAAC;IAC/E,CAAC;IACD,MAAM,KAAK,GAAG,EAAE,CAAC,WAAW,CAAC,GAAG,CAAC,CAAC,MAAM,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,CAAC,QAAQ,CAAC,KAAK,CAAC,CAAC,CAAC;IACnE,IAAI,CAAC,KAAK,CAAC,MAAM,EAAE,CAAC;QAClB,OAAO,CAAC,MAAM,CAAC,KAAK,CAAC,2BAA2B,GAAG,IAAI,CAAC,CAAC;QACzD,OAAO,CAAC,IAAI,CAAC,CAAC,CAAC,CAAC;IAClB,CAAC;IACD,OAAO,KAAK,CAAC,GAAG,CAAC,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC;QACvB,IAAI,EAAE,CAAC;QACP,OAAO,EAAE,EAAE,CAAC,YAAY,CAAC,IAAI,CAAC,IAAI,CAAC,GAAG,EAAE,CAAC,CAAC,EAAE,MAAM,CAAC;KACpD,CAAC,CAAC,CAAC;AACN,CAAC;AAED,sEAAsE;AACtE,SAAS,gBAAgB,CAAC,GAAa;IACrC,MAAM,KAAK,GAAG,GAAG,CAAC,OAAO,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC;IACtC,MAAM,MAAM,GAAY,EAAE,CAAC;IAC3B,IAAI,KAAK,GAAG,CAAC,CAAC;IACd,KAAK,IAAI,CAAC,GAAG,CAAC,EAAE,CAAC,IAAI,KAAK,CAAC,MAAM,EAAE,CAAC,EAAE,EAAE,CAAC;QACvC,MAAM,KAAK,GAAG,CAAC,KAAK,KAAK,CAAC,MAAM,IAAI,CAAC,KAAK,CAAC,CAAC,CAAC,CAAC,IAAI,EAAE,CAAC;QACrD,IAAI,KAAK,IAAI,CAAC,GAAG,KAAK,EAAE,CAAC;YACvB,MAAM,IAAI,GAAG,KAAK,CAAC,KAAK,CAAC,KAAK,EAAE,CAAC,CAAC,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC,IAAI,EAAE,CAAC;YACrD,IAAI,IAAI,EAAE,CAAC;gBACT,MAAM,CAAC,IAAI,CAAC;oBACV,QAAQ,EAAE,GAAG,CAAC,IAAI;oBAClB,OAAO,EAAE,IAAI,CAAC,KAAK,CAAC,CAAC,EAAE,EAAE,CAAC,CAAC,OAAO,CAAC,KAAK,EAAE,GAAG,CAAC,GAAG,CAAC,IAAI,CAAC,MAAM,GAAG,EAAE,CAAC,CAAC,CAAC,QAAQ,CAAC,CAAC,CAAC,EAAE,CAAC;oBACnF,IAAI,EAAE,MAAM,EAAE,EAAE;oBAChB,SAAS,EAAE,KAAK,GAAG,CAAC;oBACpB,OAAO,EAAE,CAAC;iBACX,CAAC,CAAC;YACL,CAAC;QACH,CAAC;QACD,IAAI,KAAK;YAAE,KAAK,GAAG,CAAC,GAAG,CAAC,CAAC;IAC3B,CAAC;IACD,OAAO,MAAM,CAAC;AAChB,CAAC;AAED;;;;;;;;;;;GAWG;AACH,SAAgB,cAAc,CAAC,SAAqB;IAClD,MAAM,GAAG,GAAY,EAAE,CAAC;IACxB,KAAK,MAAM,GAAG,IAAI,SAAS,EAAE,CAAC;QAC5B,MAAM,QAAQ,GAAG,aAAa,CAAC,GAAG,CAAC,OAAO,CAAC,CAAC;QAC5C,gFAAgF;QAChF,IAAI,QAAQ,CAAC,MAAM,IAAI,CAAC,IAAI,GAAG,CAAC,OAAO,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC,MAAM,GAAG,EAAE,EAAE,CAAC;YAChE,GAAG,CAAC,IAAI,CAAC,GAAG,gBAAgB,CAAC,GAAG,CAAC,CAAC,CAAC;YACnC,SAAS;QACX,CAAC;QACD,MAAM,KAAK,GAAG,GAAG,CAAC,OAAO,CAAC,KAAK,CAAC,IAAI,CAAC,CAAC;QACtC,KAAK,MAAM,GAAG,IAAI,QAAQ,EAAE,CAAC;YAC3B,MAAM,IAAI,GAAG,KAAK,CAAC,KAAK,CAAC,GAAG,CAAC,SAAS,GAAG,CAAC,EAAE,GAAG,CAAC,OAAO,CAAC,CAAC,IAAI,CAAC,IAAI,CAAC,CAAC,IAAI,EAAE,CAAC;YAC3E,IAAI,CAAC,IAAI;gBAAE,SAAS;YACpB,GAAG,CAAC,IAAI,CAAC;gBACP,QAAQ,EAAE,GAAG,CAAC,IAAI,EAAE,OAAO,EAAE,GAAG,CAAC,OAAO,IAAI,GAAG,CAAC,IAAI,EAAE,IAAI,EAAE,MAAM,EAAE,EAAE;gBACtE,SAAS,EAAE,GAAG,CAAC,SAAS,EAAE,OAAO,EAAE,GAAG,CAAC,OAAO;aAC/C,CAAC,CAAC;QACL,CAAC;IACH,CAAC;IACD,OAAO,GAAG,CAAC;AACb,CAAC"}

package/dist/resources/index.d.ts

@@ -0,0 +1,9 @@
+/**
+ * Resource loading and chunking utilities
+ *
+ * @packageDocumentation
+ * @category Rig
+ */
+export { loadResources, chunkResources } from './files';
+export type { Resource, Chunk } from './types';
+//# sourceMappingURL=index.d.ts.map

package/dist/resources/index.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"index.d.ts","sourceRoot":"","sources":["../../src/resources/index.ts"],"names":[],"mappings":"AAAA;;;;;GAKG;AACH,OAAO,EAAE,aAAa,EAAE,cAAc,EAAE,MAAM,SAAS,CAAC;AACxD,YAAY,EAAE,QAAQ,EAAE,KAAK,EAAE,MAAM,SAAS,CAAC"}

package/dist/resources/index.js

@@ -0,0 +1,13 @@
+"use strict";
+Object.defineProperty(exports, "__esModule", { value: true });
+exports.chunkResources = exports.loadResources = void 0;
+/**
+ * Resource loading and chunking utilities
+ *
+ * @packageDocumentation
+ * @category Rig
+ */
+var files_1 = require("./files");
+Object.defineProperty(exports, "loadResources", { enumerable: true, get: function () { return files_1.loadResources; } });
+Object.defineProperty(exports, "chunkResources", { enumerable: true, get: function () { return files_1.chunkResources; } });
+//# sourceMappingURL=index.js.map

package/dist/resources/index.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"index.js","sourceRoot":"","sources":["../../src/resources/index.ts"],"names":[],"mappings":";;;AAAA;;;;;GAKG;AACH,iCAAwD;AAA/C,sGAAA,aAAa,OAAA;AAAE,uGAAA,cAAc,OAAA"}

package/dist/resources/types.d.ts

@@ -0,0 +1,39 @@
+/**
+ * A loaded document available for search, read, and grep operations
+ *
+ * Represents a single file (typically Markdown) loaded into memory.
+ * Resources are chunked into {@link Chunk} instances for reranking.
+ *
+ * @category Rig
+ */
+export interface Resource {
+    /** File name (basename, not full path) used as the resource identifier */
+    name: string;
+    /** Full text content of the file */
+    content: string;
+}
+/**
+ * A scored passage within a {@link Resource}, used for reranking and retrieval
+ *
+ * Chunks are produced by {@link chunkResources} (section-based for Markdown)
+ * or {@link chunkFetchedPages} (paragraph-based for web content). The
+ * {@link tokens} array is populated lazily by {@link Reranker.tokenizeChunks}
+ * before scoring.
+ *
+ * @category Rig
+ */
+export interface Chunk {
+    /** Resource identifier (file name or URL) this chunk belongs to */
+    resource: string;
+    /** Section heading or auto-generated preview used as a label */
+    heading: string;
+    /** Raw text content of the chunk */
+    text: string;
+    /** Pre-tokenized representation for the reranker — empty until {@link Reranker.tokenizeChunks} runs */
+    tokens: number[];
+    /** First line number (1-based) in the source resource */
+    startLine: number;
+    /** Last line number (1-based) in the source resource */
+    endLine: number;
+}
+//# sourceMappingURL=types.d.ts.map

package/dist/resources/types.d.ts.map

@@ -0,0 +1 @@
+{"version":3,"file":"types.d.ts","sourceRoot":"","sources":["../../src/resources/types.ts"],"names":[],"mappings":"AAAA;;;;;;;GAOG;AACH,MAAM,WAAW,QAAQ;IACvB,0EAA0E;IAC1E,IAAI,EAAE,MAAM,CAAC;IACb,oCAAoC;IACpC,OAAO,EAAE,MAAM,CAAC;CACjB;AAED;;;;;;;;;GASG;AACH,MAAM,WAAW,KAAK;IACpB,mEAAmE;IACnE,QAAQ,EAAE,MAAM,CAAC;IACjB,gEAAgE;IAChE,OAAO,EAAE,MAAM,CAAC;IAChB,oCAAoC;IACpC,IAAI,EAAE,MAAM,CAAC;IACb,uGAAuG;IACvG,MAAM,EAAE,MAAM,EAAE,CAAC;IACjB,yDAAyD;IACzD,SAAS,EAAE,MAAM,CAAC;IAClB,wDAAwD;IACxD,OAAO,EAAE,MAAM,CAAC;CACjB"}

package/dist/resources/types.js

@@ -0,0 +1,3 @@
+"use strict";
+Object.defineProperty(exports, "__esModule", { value: true });
+//# sourceMappingURL=types.js.map

package/dist/resources/types.js.map

@@ -0,0 +1 @@
+{"version":3,"file":"types.js","sourceRoot":"","sources":["../../src/resources/types.ts"],"names":[],"mappings":""}

package/dist/sources/corpus-research.md

@@ -0,0 +1,14 @@
+You are a research assistant analyzing a knowledge base. Your tools:
+- **grep**: regex pattern matching — use for precise, exhaustive retrieval
+- **search**: semantic relevance ranking — use to discover related content
+- **read_file**: read specific line ranges — use to verify and get context
+- **research**: spawn parallel sub-agents that each run their own grep/search/read_file cycle — call with `{"questions": ["q1", "q2", ...]}`
+- **report**: submit your final findings with evidence
+
+Process — follow every step in order:
+1. Grep with short, simple patterns first. Use single keywords or two-word phrases — never combine multiple clauses with `.*`. Run multiple greps if needed.
+2. Use search to discover content that grep may miss (different phrasing, synonyms).
+3. Read every matching line with read_file to verify in context. Do not rely on grep/search summaries alone.
+4. Grep again with a different pattern targeting what you have NOT yet found. This is a completeness check, not confirmation of existing results.
+5. Call research with sub-questions only if your findings point to specific topics you haven't fully investigated.
+6. Report with line numbers and direct quotes as evidence. State what you found and what you checked.