@tangle-network/agent-knowledge 1.10.0 → 1.11.0

package/README.md

@@ -10,10 +10,11 @@ This package turns raw sources and generated markdown knowledge into a versionab
 - [Start here](#start-here) — pick CLI vs programmatic
 - [CLI](#cli) — `init` → `source-add` → `index` → `search` → `lint`
 - [Design](#design) — the invariants (immutable sources, cited claims, deterministic graph)
-- [Agent-Eval integration](#agent-eval-integration) — readiness bundles + release reports
+- [Benchmark harness](#benchmark-harness) — BEIR/MTEB/qrels, RAG answer, hallucination, KB-improvement cases
+- [Agent-Eval integration](#agent-eval-integration) — retrieval eval + readiness bundles + release reports
 - [Memory adapters](#memory-adapters) — generic memory contract + Neo4j Agent Memory bridge
 - [Research loop](#research-loop) — `runKnowledgeResearchLoop` + control-loop adapter
-- [Researcher profile](#researcher-profile) — sandbox `AgentProfile` for `runLoop`
+- [Runtime integration](#runtime-integration) — how agent runners plug into the pure KB loop
 - [Pluggable knowledge sources](#pluggable-knowledge-sources) — live authorities → eval re-runs
 
 ## Install
@@ -29,8 +30,15 @@ Two ways in, depending on what you're doing:
 - **Author / inspect a KB by hand** → the [CLI](#cli) (`init` → `source-add` → `index` → `search` → `lint`). Fastest way to see the shape on disk.
 - **Drive it from an agent** → pick the primitive by intent:
   - *"Does the agent have enough context to run?"* → [`buildEvalKnowledgeBundle`](#agent-eval-integration) (block / ask / acquire before execution).
-  - *"Grow the KB as a researcher"* → [`runKnowledgeResearchLoop`](#research-loop) (deterministic mechanics; your agent owns judgment), [`runTwoAgentResearchLoop`](#two-agent-research-loop) (researcher proposes, verifier checks + fills gaps, offline), or the sandbox [researcher profile](#researcher-profile) for `runLoop`.
-  - *"Spawn one researcher per sub-topic and stop when the KB is ready"* → [`runResearchSupervisor`](#research-supervisor) (a supervisor brain sizes the topology over a `Scope`; LIVE, needs creds).
+  - *"Grow the KB as a researcher"* → [`runKnowledgeResearchLoop`](#research-loop) (deterministic mechanics; your agent owns judgment) or [`runTwoAgentResearchLoop`](#two-agent-research-loop) (researcher proposes, verifier checks + fills gaps).
+  - *"Spawn live agents to improve a KB"* → pass an `updateKnowledge` callback to `improveKnowledgeBase`; runtime-backed supervisors live in `@tangle-network/agent-runtime`.
+  - *"Tune retrieval for a knowledge base"* → `runRetrievalImprovementLoop` in the [Agent-Eval integration](#agent-eval-integration) section.
+  - *"Run an operator-grade KB improvement cycle"* → `improveKnowledgeBase` in the [Agent-Eval integration](#agent-eval-integration) section.
+    It creates a candidate KB, lets agents or deterministic hooks improve it, runs configured evals, can be resumed, and promotes only when the live KB has not changed underneath it.
+  - *"Expose the lower-level RAG lifecycle phases"* → `runRagKnowledgeImprovementLoop` in the [Agent-Eval integration](#agent-eval-integration) section.
+    It exposes retrieval tuning, gap diagnosis, knowledge acquisition/update, answer-quality checks, and promotion as one typed lifecycle.
+  - *"Evaluate RAG answers or a wiki/KB"* → `ragAnswerQualityJudge`, `createRagAnswerQualityHook`, and `scoreKnowledgeBaseIndex` in the [Agent-Eval integration](#agent-eval-integration) section.
+  - *"Run standard RAG/KB benchmarks"* → `runKnowledgeBenchmarkSuite` in the [Benchmark harness](#benchmark-harness) section.
   - *"Does this candidate KB actually improve task success?"* → run an [agent-eval improvement loop](#agent-eval-integration) over KB variants, then `knowledgeReleaseReport` for the promotion decision.
   - *"Keep live authorities fresh"* → [pluggable sources](#pluggable-knowledge-sources) + `detectChanges` → eval re-runs.
 
@@ -91,6 +99,23 @@ from `@tangle-network/agent-knowledge`.
 - `createKnowledgeControlLoopAdapter()` maps those mechanics into
   `agent-eval`'s `runAgentControlLoop()` so products can plug in their own
   proposer, reviewer, and driver policies.
+- `runRagKnowledgeImprovementLoop()` coordinates the whole RAG improvement
+  lifecycle. Retrieval tuning, diagnosis, acquisition, KB update,
+  answer-quality eval, and promotion are separate typed phases so products can
+  plug in browser agents, coding agents, connectors, or deterministic policies
+  without this package hardcoding an agent runner.
+- `improveKnowledgeBase()` wraps that lifecycle with durable candidate state,
+  a per-run lock, resume support, isolated candidate workspaces, KB quality
+  scoring, and conflict-safe promotion.
+  Use it when running agents in loops against a real KB rather than only
+  exposing phase hooks.
+- `evaluateKnowledgeBaseReadiness()` checks one KB root without running an
+  improvement loop: it rebuilds the index, validates/lints it, scores KB
+  quality, and evaluates configured readiness specs.
+- RAG answer evaluation follows the common open-source shape used by Ragas,
+  DeepEval, TruLens, and RAGChecker: context quality, answer relevance,
+  support/faithfulness, citations, abstention, and failure diagnosis.
+  External tools stay pluggable via score normalization and row exporters.
 - Zod schemas define the stable wire shape.
 - Graph/search/lint are deterministic and fast.
 - `searchKnowledge` returns hits with three score fields. `score` and
@@ -101,6 +126,7 @@ from `@tangle-network/agent-knowledge`.
   when comparing against natural confidence thresholds. The normalization is
   within-set ranking, not a cross-query absolute confidence.
 - Release confidence uses `@tangle-network/agent-eval` release gates (`evaluateReleaseConfidence`) instead of reimplementing them.
+- Retrieval eval turns retrieval/RAG configs into `agent-eval` surfaces, auto-searches candidate configs, and scores them against page, source, source-anchor, or source-span targets.
 - `buildEvalKnowledgeBundle()` maps wiki/search evidence into
   `agent-eval` `KnowledgeRequirement`, `KnowledgeBundle`, and
   `KnowledgeReadinessReport` contracts so control loops can block, ask, or
@@ -112,8 +138,214 @@ The `/memory` subpath exports an optional memory adapter contract. Use it to
 bridge episodic or graph-native memory systems into the same source-grounded
 readiness/eval machinery without making `agent-knowledge` own the database.
 
+## Benchmark Harness
+
+Use `runKnowledgeBenchmarkSuite()` when the product goal is to run a fixed RAG/KB benchmark pack and get one report across retrieval, answer quality, hallucination, and candidate-KB improvement cases.
+The module also exports `INDUSTRY_RAG_BENCHMARKS`, a compact manifest for BEIR, MTEB retrieval, MS MARCO, TREC DL, MIRACL, LoTTE, BRIGHT, CRAG, HotpotQA, KILT, RAGTruth, FaithBench, and first-party KB-improvement tasks.
+
+```ts
+import {
+  buildFirstPartyMemoryLifecycleBenchmarkCases,
+  buildIndustryMemoryBenchmarkSmokeCases,
+  buildIndustryRagBenchmarkSmokeCases,
+  buildRetrievalBenchmarkCasesFromQrels,
+  createInMemoryBenchmarkAdapter,
+  createNoopMemoryBenchmarkAdapter,
+  isKnowledgeMemoryBenchmarkCase,
+  parseKnowledgeBenchmarkQrels,
+  respondToIndustryMemoryBenchmarkSmokeCase,
+  respondToIndustryRagBenchmarkSmokeCase,
+  runMemoryAdapterBenchmark,
+  runKnowledgeBenchmarkSuite,
+} from '@tangle-network/agent-knowledge/benchmarks'
+import { createNeo4jAgentMemoryAdapter } from '@tangle-network/agent-knowledge/memory'
+
+await runKnowledgeBenchmarkSuite({
+  cases: buildIndustryRagBenchmarkSmokeCases(),
+  runDir: '.agent-knowledge/benchmark-runs/industry-smoke',
+  respond: respondToIndustryRagBenchmarkSmokeCase,
+})
+
+await runKnowledgeBenchmarkSuite({
+  cases: buildIndustryMemoryBenchmarkSmokeCases(),
+  runDir: '.agent-knowledge/benchmark-runs/memory-smoke',
+  respond: ({ case: testCase }) => {
+    if (!isKnowledgeMemoryBenchmarkCase(testCase)) return { answer: '' }
+    return respondToIndustryMemoryBenchmarkSmokeCase({ case: testCase })
+  },
+})
+
+const memoryRanking = await runMemoryAdapterBenchmark({
+  cases: buildFirstPartyMemoryLifecycleBenchmarkCases(),
+  runDir: '.agent-knowledge/benchmark-runs/memory-ranking',
+  candidates: [
+    { id: 'no-memory', createAdapter: () => createNoopMemoryBenchmarkAdapter() },
+    {
+      id: 'in-memory',
+      createAdapter: () => createInMemoryBenchmarkAdapter(),
+      searchLimit: 1,
+    },
+    {
+      id: 'neo4j-agent-memory',
+      createAdapter: () => createNeo4jAgentMemoryAdapter({ client: neo4jMemoryClient }),
+      searchLimit: 5,
+    },
+  ],
+})
+
+console.log(memoryRanking.rows)
+
+const cases = buildRetrievalBenchmarkCasesFromQrels({
+  benchmarkId: 'beir/nfcorpus',
+  family: 'beir',
+  queries: [{ id: 'q1', text: 'aspirin heart attack prevention', split: 'holdout' }],
+  qrels: parseKnowledgeBenchmarkQrels('q1 0 src-aspirin 1'),
+  targetKind: 'source',
+  k: 10,
+})
+
+const result = await runKnowledgeBenchmarkSuite({
+  cases,
+  runDir: '.agent-knowledge/benchmark-runs/beir-nfcorpus-smoke',
+  respond: async ({ case: testCase }) => {
+    if (testCase.taskKind !== 'retrieval') return { hits: [] }
+    const hits = await retrieveFromYourKb(testCase.query)
+    return { hits, costUsd: 0.001 }
+  },
+})
+
+console.log(result.report.score.mean)
+```
+
+Use `buildRetrievalBenchmarkCasesFromQrels()` for qrels-backed retrieval datasets.
+The smoke pack proves that every declared benchmark family is wired through the runner; full BEIR, MTEB, MS MARCO, TREC DL, MIRACL, LoTTE, BRIGHT, CRAG, HotpotQA, KILT, RAGTruth, and FaithBench runs should pass real dataset rows through the same case shapes.
+Use `KnowledgeAnswerBenchmarkCase` for CRAG/HotpotQA/KILT-style answer checks and RAGTruth/FaithBench-style hallucination checks by encoding required claims, forbidden claims, and expected source IDs.
+Use `taskKind: 'kb-improvement'` when the artifact is candidate KB text produced by `improveKnowledgeBase()`.
+Use `KnowledgeMemoryBenchmarkCase` for memory systems.
+Memory cases carry ordered events plus current required facts, stale forbidden facts, expected event IDs, and expected actor IDs.
+The built-in memory scorer reports current fact recall, stale-memory safety, event recall, and actor recall, which maps to LoCoMo, LongMemEval, Memora, MemoryAgentBench, MemoryBank-style personalization, GroupMemBench, and first-party memory lifecycle checks.
+Use `runMemoryAdapterBenchmark()` to rank real memory systems that implement `AgentMemoryAdapter`.
+This is the path for Neo4j Agent Memory, Mem0, Zep, Letta, Graphiti, a vector store, or a product-owned memory backend.
+
 ## Agent-Eval Integration
 
+Use `ragAnswerQualityJudge` or `createRagAnswerQualityHook` when the product already has answer traces and needs SOTA-style RAG scoring without rebuilding metrics.
+The built-in checks are deterministic and general; pass external scores from Ragas, DeepEval, TruLens, RAGChecker, or a custom evaluator when you want model-based judging.
+
+```ts
+import {
+  createRagAnswerQualityHook,
+  scoreKnowledgeBaseIndex,
+} from '@tangle-network/agent-knowledge'
+
+const evaluateAnswers = createRagAnswerQualityHook({
+  scenarios: answerScenarios,
+  run: async (scenario) => runRagAnswerTrace(scenario),
+  externalEvaluator: async ({ scenario, artifact }) => runRagasOrDeepEval({
+    input: scenario.query,
+    output: artifact.answer,
+    contexts: artifact.contexts,
+  }),
+})
+
+const answerQuality = await evaluateAnswers()
+const kbQuality = scoreKnowledgeBaseIndex(index, {
+  strict: true,
+  minCitationRate: 0.8,
+  maxStaleSourceRate: 0.02,
+})
+```
+
+Use `runRagKnowledgeImprovementLoop` when the product question is broader than retrieval:
+can the system find the gaps, gather or update knowledge, prove generated answers still behave, and decide whether to promote?
+`agent-knowledge` owns the knowledge/eval contract; the caller supplies the research, coding, connector, and answer-eval hooks.
+
+```ts
+import { runRagKnowledgeImprovementLoop } from '@tangle-network/agent-knowledge'
+
+const result = await runRagKnowledgeImprovementLoop({
+  goal: 'Improve the support RAG KB',
+  retrieval: {
+    baseline: { k: 5, hybrid: false },
+    scenarios: trainRetrievalScenarios,
+    holdoutScenarios,
+    index,
+    searchSpace: { k: [5, 10, 20], hybrid: [false, true] },
+    targetRecall: 0.9,
+  },
+  diagnose: async ({ retrieval }) => diagnoseRagGaps(retrieval),
+  acquireKnowledge: async ({ findings }) => researchMissingSources(findings),
+  knowledgeResearch: { root: './kb' },
+  evaluateAnswers,
+  promote: async ({ retrieval, answerQuality }) =>
+    decidePromotion({ retrieval, answerQuality }),
+  requiredPhases: ['retrieval-tuning', 'knowledge-update', 'answer-quality', 'promotion'],
+})
+
+console.log(result.promotion)
+```
+
+Use `improveKnowledgeBase` when a program should own the candidate workspace and promotion mechanics.
+The wrapper composes the same RAG lifecycle, but adds resumable state under `.agent-knowledge/improvements/<runId>/`, a lock lease for parallel operators, and a base-hash check before copying the candidate over the live KB.
+
+```ts
+import { improveKnowledgeBase } from '@tangle-network/agent-knowledge'
+
+const result = await improveKnowledgeBase({
+  root: './kb',
+  goal: 'Improve support refund-policy knowledge',
+  readinessSpecs,
+  retrieval: {
+    baseline: { k: 5 },
+    scenarios: trainRetrievalScenarios,
+    holdoutScenarios,
+    searchSpace: { k: [5, 10, 20] },
+    targetRecall: 0.9,
+  },
+  step: async ({ readiness }) => runResearchAgent({ missing: readiness?.report }),
+  evaluateAnswers,
+  requiredPhases: ['knowledge-update', 'retrieval-tuning', 'answer-quality'],
+})
+
+console.log(result.promoted, result.candidate?.evaluation)
+```
+
+Pass `promote: false` to leave the candidate workspace open for another agent or a human edit.
+Calling `improveKnowledgeBase` again with the same `runId` re-evaluates that candidate and promotes it only if the original live KB hash still matches.
+
+If a required phase is missing its hook, the loop throws.
+That keeps the public API from reporting a fake “RAG improved” result when the caller only wired retrieval or only wired a researcher.
+
+Use retrieval eval when the question is whether a retrieval/RAG config can find the right knowledge before an agent reasons over it.
+The labels should name stable pages, source records, anchors, or source spans, not ephemeral chunk IDs.
+The completion roadmap is in [`docs/eval/rag-eval-roadmap.md`](docs/eval/rag-eval-roadmap.md).
+
+```ts
+import { runRetrievalImprovementLoop } from '@tangle-network/agent-knowledge'
+
+const result = await runRetrievalImprovementLoop({
+  baseline: { k: 5, hybrid: false, reranker: null },
+  scenarios: trainRetrievalScenarios,
+  holdoutScenarios: holdoutRetrievalScenarios,
+  index,
+  searchSpace: {
+    k: [5, 10, 20],
+    hybrid: [false, true],
+    reranker: [null, 'bge-reranker'],
+  },
+  targetRecall: 0.9,
+  deltaThreshold: 0.02,
+  costCeiling: 15,
+  runDir: '.agent-knowledge/retrieval-runs',
+})
+
+console.log(result.winnerConfig)
+```
+
+Pass a custom `retrieve` function to `buildRetrievalEvalDispatch` when the config controls an external vector store, reranker, hybrid search service, or chunker.
+The built-in fallback uses `searchKnowledge` over the local deterministic index.
+Use `buildRetrievalEvalDispatch`, `retrievalRecallJudge`, and `retrievalParameterSweepProposer` directly only when you need custom `agent-eval` wiring.
+
 To answer whether a candidate knowledge base actually improves agent task success, run an `@tangle-network/agent-eval` improvement loop (`runImprovementLoop`) over your KB variants on a real task corpus; each run is scored into a `RunRecord`.
 
 Use `knowledgeReleaseReport()` before promotion: pass the candidate and baseline `RunRecord[]` (plus optional `ReleaseTraceEvidence` and the gate decision) and it folds them into a `ReleaseConfidenceScorecard` and a `KnowledgeRelease` using `agent-eval`'s release gates and `RunRecord` validation.
@@ -308,86 +540,30 @@ await runTwoAgentResearchLoop({
 })
 ```
 
-## Research supervisor
+## Runtime integration
 
-`runResearchSupervisor()` is the LIVE counterpart: a supervisor brain creates the
-topology dynamically — one researcher worker per sub-topic over a `Scope` — and
-stops when the knowledge base is ready. It is a thin wrapper over `supervise()`
-from `@tangle-network/agent-runtime/loops`; it builds nothing new. The worker
-shape is the [researcher profile](#researcher-profile), and the completion oracle
-is `knowledgeReadinessDeliverable` (re-reads the KB from disk and runs the
-readiness gate, so it stops on the real grounded state, not a worker's
-self-report). Needs creds: a supervisor router brain plus a worker backend.
+`agent-knowledge` owns knowledge state and measurement.
+It deliberately does not own an agent runner.
+Live agent orchestration belongs in `@tangle-network/agent-runtime`, which can call `improveKnowledgeBase` by passing an `updateKnowledge` callback:
 
 ```ts
-import { defineReadinessSpec, runResearchSupervisor } from '@tangle-network/agent-knowledge'
+import { improveKnowledgeBase } from '@tangle-network/agent-knowledge'
 
-await runResearchSupervisor({
+await improveKnowledgeBase({
   root: './kb',
   goal: 'Build a grounded onboarding wiki for billing support',
-  readinessSpecs: [defineReadinessSpec({
-    id: 'refund-policy',
-    description: 'Refund policy grounding',
-    query: 'refund policy customer request',
-    requiredFor: ['support-agent'],
-  })],
-  budget: { maxIterations: 12, maxTokens: 200_000, maxUsd: 5 },
-  // WHERE researcher workers run — the real backend seam.
-  backend: {
-    backend: 'router', // or 'sandbox' / 'cli' / 'bridge'
-    routerBaseUrl: process.env.ROUTER_BASE_URL!,
-    routerKey: process.env.ROUTER_KEY!,
-    model: 'your-worker-model',
-  },
-})
-```
-
-## Researcher profile
-
-`@tangle-network/agent-knowledge/profiles` ships a sandbox-SDK
-`AgentProfile` preset for source-grounded research agents. Pairs with
-`runLoop` from `@tangle-network/agent-runtime/loops` — the profile owns
-the prompt + output adapter + validator; the kernel owns iteration,
-concurrency, cost, and trace emission.
-
-```ts
-import { runLoop } from '@tangle-network/agent-runtime/loops'
-import { multiHarnessResearcherFanout } from '@tangle-network/agent-knowledge/profiles'
-
-const research = multiHarnessResearcherFanout({
-  harnesses: ['opencode/zai-coding-plan/glm-5.1', 'claude-code', 'codex'],
-})
-
-const result = await runLoop({
-  driver: research.driver,
-  agentRuns: research.agentRuns,
-  output: research.output,
-  validator: research.validator,
-  task: {
-    question: 'What content does cpg-founder ICP engage with on Twitter?',
-    knowledgeNamespace: 'cust_42',
-    sources: ['twitter', 'web'],
-    maxItems: 20,
-    minConfidence: 0.6,
+  readinessSpecs,
+  updateKnowledge: async ({ goal, findings }) => {
+    // Call your agent runner here, then apply source-backed write blocks.
+    return {
+      applied: true,
+      summary: `updated KB for ${goal} with ${findings.length} finding(s)`,
+    }
   },
-  ctx: { sandboxClient },
 })
-
-if (result.winner?.verdict?.valid) {
-  // result.winner.output.proposedWrites: KnowledgeUpdate[]
-  // The profile does NOT materialize. Decide whether to apply.
-  for (const write of result.winner.output.proposedWrites) {
-    // route through applyKnowledgeWriteBlocks / a KbStore put when ready
-  }
-}
 ```
 
-Three invariants are enforced by the validator:
-
-- **Namespace isolation** — every `KnowledgeItem` + `KnowledgeUpdate`
-  must carry `task.knowledgeNamespace`. Cross-tenant writes hard-fail.
-- **Provenance** — every item carries at least one evidence entry.
-- **Citation density** — quotes-with-source / items >= 0.7 by default.
+This keeps the dependency graph acyclic: `agent-knowledge` depends on `agent-eval`; agent runners depend on `agent-knowledge`, not the reverse.
 
 Validator scoring (default; overridable):
 

package/dist/benchmarks/index.d.ts

@@ -0,0 +1,5 @@
+import '@tangle-network/agent-eval/campaign';
+import '../types-CjqPcTTK.js';
+export { B as BuildRetrievalBenchmarkCasesFromQrelsOptions, I as INDUSTRY_MEMORY_BENCHMARKS, e as INDUSTRY_RAG_BENCHMARKS, K as KnowledgeAnswerBenchmarkCase, f as KnowledgeAnswerBenchmarkTaskKind, g as KnowledgeBenchmarkArtifact, h as KnowledgeBenchmarkCase, i as KnowledgeBenchmarkCaseBase, j as KnowledgeBenchmarkDistribution, k as KnowledgeBenchmarkEvaluation, l as KnowledgeBenchmarkFamily, m as KnowledgeBenchmarkReport, n as KnowledgeBenchmarkResponder, o as KnowledgeBenchmarkScenario, p as KnowledgeBenchmarkSliceSummary, q as KnowledgeBenchmarkSource, r as KnowledgeBenchmarkSpec, s as KnowledgeBenchmarkSplit, t as KnowledgeBenchmarkTaskKind, u as KnowledgeClaimMatcher, v as KnowledgeMemoryBenchmarkCase, w as KnowledgeMemoryBenchmarkTaskKind, x as KnowledgeMemoryEvent, y as KnowledgeMemoryFactMatcher, z as KnowledgeRetrievalBenchmarkCase, A as KnowledgeRetrievalBenchmarkQrel, C as KnowledgeRetrievalBenchmarkQuery, M as MemoryAdapterBenchmarkCandidate, D as MemoryAdapterBenchmarkRankingRow, V as RunKnowledgeBenchmarkSuiteOptions, W as RunKnowledgeBenchmarkSuiteResult, X as RunMemoryAdapterBenchmarkOptions, Y as RunMemoryAdapterBenchmarkResult, Z as buildFirstPartyMemoryLifecycleBenchmarkCases, _ as buildIndustryMemoryBenchmarkSmokeCases, $ as buildIndustryRagBenchmarkSmokeCases, a0 as buildKnowledgeBenchmarkScenarios, a1 as buildRetrievalBenchmarkCasesFromQrels, a4 as createInMemoryBenchmarkAdapter, a5 as createMemoryAdapterBenchmarkResponder, a6 as createNoopMemoryBenchmarkAdapter, a7 as isKnowledgeMemoryBenchmarkCase, a8 as knowledgeBenchmarkJudge, a9 as parseKnowledgeBenchmarkJsonl, aa as parseKnowledgeBenchmarkQrels, ab as renderKnowledgeBenchmarkReportMarkdown, ac as respondToIndustryMemoryBenchmarkSmokeCase, ad as respondToIndustryRagBenchmarkSmokeCase, ai as runKnowledgeBenchmarkSuite, aj as runMemoryAdapterBenchmark, al as scoreKnowledgeBenchmarkArtifact, am as scoreMemoryBenchmarkArtifact, ao as summarizeKnowledgeBenchmarkCampaign } from '../index-Cj5jRXOz.js';
+import '../types-C17sAROL.js';
+import '@tangle-network/agent-eval/rl';

package/dist/benchmarks/index.js

@@ -0,0 +1,52 @@
+import {
+  INDUSTRY_MEMORY_BENCHMARKS,
+  INDUSTRY_RAG_BENCHMARKS,
+  buildFirstPartyMemoryLifecycleBenchmarkCases,
+  buildIndustryMemoryBenchmarkSmokeCases,
+  buildIndustryRagBenchmarkSmokeCases,
+  buildKnowledgeBenchmarkScenarios,
+  buildRetrievalBenchmarkCasesFromQrels,
+  createInMemoryBenchmarkAdapter,
+  createMemoryAdapterBenchmarkResponder,
+  createNoopMemoryBenchmarkAdapter,
+  isKnowledgeMemoryBenchmarkCase,
+  knowledgeBenchmarkJudge,
+  parseKnowledgeBenchmarkJsonl,
+  parseKnowledgeBenchmarkQrels,
+  renderKnowledgeBenchmarkReportMarkdown,
+  respondToIndustryMemoryBenchmarkSmokeCase,
+  respondToIndustryRagBenchmarkSmokeCase,
+  runKnowledgeBenchmarkSuite,
+  runMemoryAdapterBenchmark,
+  scoreKnowledgeBenchmarkArtifact,
+  scoreMemoryBenchmarkArtifact,
+  summarizeKnowledgeBenchmarkCampaign
+} from "../chunk-ULXZI235.js";
+import "../chunk-DQ3PDMDP.js";
+import "../chunk-XVU5FFQA.js";
+import "../chunk-YMKHCTS2.js";
+export {
+  INDUSTRY_MEMORY_BENCHMARKS,
+  INDUSTRY_RAG_BENCHMARKS,
+  buildFirstPartyMemoryLifecycleBenchmarkCases,
+  buildIndustryMemoryBenchmarkSmokeCases,
+  buildIndustryRagBenchmarkSmokeCases,
+  buildKnowledgeBenchmarkScenarios,
+  buildRetrievalBenchmarkCasesFromQrels,
+  createInMemoryBenchmarkAdapter,
+  createMemoryAdapterBenchmarkResponder,
+  createNoopMemoryBenchmarkAdapter,
+  isKnowledgeMemoryBenchmarkCase,
+  knowledgeBenchmarkJudge,
+  parseKnowledgeBenchmarkJsonl,
+  parseKnowledgeBenchmarkQrels,
+  renderKnowledgeBenchmarkReportMarkdown,
+  respondToIndustryMemoryBenchmarkSmokeCase,
+  respondToIndustryRagBenchmarkSmokeCase,
+  runKnowledgeBenchmarkSuite,
+  runMemoryAdapterBenchmark,
+  scoreKnowledgeBenchmarkArtifact,
+  scoreMemoryBenchmarkArtifact,
+  summarizeKnowledgeBenchmarkCampaign
+};
+//# sourceMappingURL=index.js.map

package/dist/chunk-DQ3PDMDP.js

@@ -0,0 +1,115 @@
+// src/search.ts
+var RRF_K = 60;
+var STOP_WORDS = /* @__PURE__ */ new Set([
+  "the",
+  "is",
+  "a",
+  "an",
+  "what",
+  "how",
+  "are",
+  "was",
+  "were",
+  "to",
+  "for",
+  "of",
+  "with",
+  "by",
+  "in",
+  "on",
+  "and"
+]);
+function searchKnowledge(index, query, limit = 10) {
+  const trimmed = query.trim();
+  if (trimmed === "") return [];
+  const tokenRanked = rankByTokens(index.pages, trimmed);
+  const graphRanked = rankByGraph(index.pages, tokenRanked);
+  const scores = reciprocalRankFusion([tokenRanked.map((p) => p.id), graphRanked.map((p) => p.id)]);
+  const byId = new Map(index.pages.map((page) => [page.id, page]));
+  const ranked = [...scores.entries()].map(([id, score]) => ({ page: byId.get(id), score })).filter((item) => Boolean(item.page)).sort((a, b) => b.score - a.score || a.page.path.localeCompare(b.page.path)).slice(0, limit);
+  const topScore = ranked[0]?.score ?? 0;
+  return ranked.map((item, i) => ({
+    page: item.page,
+    score: item.score,
+    rrfScore: item.score,
+    normalizedScore: topScore > 0 ? item.score / topScore : 0,
+    rank: i + 1,
+    snippet: buildSnippet(item.page.text, trimmed),
+    reasons: reasonsFor(item.page, trimmed)
+  }));
+}
+function tokenizeQuery(query) {
+  const raw = query.toLowerCase().split(/[\s,，。！？、；：""''（）()\-_/\\·~～…]+/).filter((token) => token.length > 1 && !STOP_WORDS.has(token));
+  const tokens = [];
+  for (const token of raw) {
+    if (/[\u4e00-\u9fff\u3400-\u4dbf]/.test(token) && token.length > 2) {
+      const chars = [...token];
+      for (let i = 0; i < chars.length - 1; i++) tokens.push(chars[i] + chars[i + 1]);
+      tokens.push(...chars);
+    }
+    tokens.push(token);
+  }
+  return [...new Set(tokens)];
+}
+function reciprocalRankFusion(rankLists, k = RRF_K) {
+  const scores = /* @__PURE__ */ new Map();
+  for (const list of rankLists) {
+    list.forEach((id, idx) => {
+      scores.set(id, (scores.get(id) ?? 0) + 1 / (k + idx + 1));
+    });
+  }
+  return scores;
+}
+function rankByTokens(pages, query) {
+  const tokens = tokenizeQuery(query);
+  const effective = tokens.length > 0 ? tokens : [query.toLowerCase()];
+  return pages.map((page) => ({ page, score: tokenScore(page, query, effective) })).filter((item) => item.score > 0).sort((a, b) => b.score - a.score || a.page.path.localeCompare(b.page.path)).map((item) => item.page);
+}
+function rankByGraph(pages, tokenRanked) {
+  if (tokenRanked.length === 0) return [];
+  const seeds = new Set(tokenRanked.slice(0, 5).map((page) => page.id));
+  return pages.map((page) => ({
+    page,
+    score: page.outLinks.filter((link) => seeds.has(link)).length + page.sourceIds.filter(
+      (source) => tokenRanked.some((seed) => seed.sourceIds.includes(source))
+    ).length
+  })).filter((item) => item.score > 0).sort((a, b) => b.score - a.score || a.page.path.localeCompare(b.page.path)).map((item) => item.page);
+}
+function tokenScore(page, query, tokens) {
+  const title = page.title.toLowerCase();
+  const path = page.path.toLowerCase();
+  const body = page.text.toLowerCase();
+  const phrase = query.toLowerCase();
+  let score = 0;
+  if (path.endsWith(`${phrase}.md`) || title === phrase) score += 200;
+  if (title.includes(phrase)) score += 50;
+  if (body.includes(phrase)) score += 20;
+  for (const token of tokens) {
+    if (title.includes(token)) score += 5;
+    if (body.includes(token)) score += 1;
+    if (path.includes(token)) score += 3;
+  }
+  return score;
+}
+function buildSnippet(text, query) {
+  const compact = text.replace(/\s+/g, " ").trim();
+  const idx = compact.toLowerCase().indexOf(query.toLowerCase());
+  if (idx < 0) return compact.slice(0, 180);
+  return compact.slice(Math.max(0, idx - 80), Math.min(compact.length, idx + query.length + 100));
+}
+function reasonsFor(page, query) {
+  const lower = `${page.title}
+${page.text}`.toLowerCase();
+  const reasons = [];
+  if (lower.includes(query.toLowerCase())) reasons.push("phrase");
+  if (page.sourceIds.length > 0) reasons.push("sourced");
+  if (page.outLinks.length > 0) reasons.push("linked");
+  return reasons;
+}
+
+export {
+  searchKnowledge,
+  tokenizeQuery,
+  reciprocalRankFusion
+};
+//# sourceMappingURL=chunk-DQ3PDMDP.js.map

package/dist/chunk-DQ3PDMDP.js.map

@@ -0,0 +1 @@
+{"version":3,"sources":["../src/search.ts"],"sourcesContent":["import type { KnowledgeIndex, KnowledgePage, KnowledgeSearchResult } from './types'\n\nconst RRF_K = 60\nconst STOP_WORDS = new Set([\n  'the',\n  'is',\n  'a',\n  'an',\n  'what',\n  'how',\n  'are',\n  'was',\n  'were',\n  'to',\n  'for',\n  'of',\n  'with',\n  'by',\n  'in',\n  'on',\n  'and',\n])\n\nexport function searchKnowledge(\n  index: KnowledgeIndex,\n  query: string,\n  limit = 10,\n): KnowledgeSearchResult[] {\n  const trimmed = query.trim()\n  if (trimmed === '') return []\n  const tokenRanked = rankByTokens(index.pages, trimmed)\n  const graphRanked = rankByGraph(index.pages, tokenRanked)\n  const scores = reciprocalRankFusion([tokenRanked.map((p) => p.id), graphRanked.map((p) => p.id)])\n  const byId = new Map(index.pages.map((page) => [page.id, page]))\n\n  const ranked = [...scores.entries()]\n    .map(([id, score]) => ({ page: byId.get(id), score }))\n    .filter((item): item is { page: KnowledgePage; score: number } => Boolean(item.page))\n    .sort((a, b) => b.score - a.score || a.page.path.localeCompare(b.page.path))\n    .slice(0, limit)\n\n  // Normalize against the top hit so callers can compare against natural\n  // [0, 1] thresholds. Raw RRF values are typically ~0.016, which reads as\n  // \"no relevance\" to humans even when the result is the best available.\n  // The top hit becomes 1 by definition; lower-ranked hits scale linearly.\n  const topScore = ranked[0]?.score ?? 0\n\n  return ranked.map((item, i) => ({\n    page: item.page,\n    score: item.score,\n    rrfScore: item.score,\n    normalizedScore: topScore > 0 ? item.score / topScore : 0,\n    rank: i + 1,\n    snippet: buildSnippet(item.page.text, trimmed),\n    reasons: reasonsFor(item.page, trimmed),\n  }))\n}\n\nexport function tokenizeQuery(query: string): string[] {\n  const raw = query\n    .toLowerCase()\n    .split(/[\\s,，。！？、；：\"\"''（）()\\-_/\\\\·~～…]+/)\n    .filter((token) => token.length > 1 && !STOP_WORDS.has(token))\n  const tokens: string[] = []\n  for (const token of raw) {\n    if (/[\\u4e00-\\u9fff\\u3400-\\u4dbf]/.test(token) && token.length > 2) {\n      const chars = [...token]\n      for (let i = 0; i < chars.length - 1; i++) tokens.push(chars[i]! + chars[i + 1]!)\n      tokens.push(...chars)\n    }\n    tokens.push(token)\n  }\n  return [...new Set(tokens)]\n}\n\nexport function reciprocalRankFusion(rankLists: string[][], k = RRF_K): Map<string, number> {\n  const scores = new Map<string, number>()\n  for (const list of rankLists) {\n    list.forEach((id, idx) => {\n      scores.set(id, (scores.get(id) ?? 0) + 1 / (k + idx + 1))\n    })\n  }\n  return scores\n}\n\nfunction rankByTokens(pages: KnowledgePage[], query: string): KnowledgePage[] {\n  const tokens = tokenizeQuery(query)\n  const effective = tokens.length > 0 ? tokens : [query.toLowerCase()]\n  return pages\n    .map((page) => ({ page, score: tokenScore(page, query, effective) }))\n    .filter((item) => item.score > 0)\n    .sort((a, b) => b.score - a.score || a.page.path.localeCompare(b.page.path))\n    .map((item) => item.page)\n}\n\nfunction rankByGraph(pages: KnowledgePage[], tokenRanked: KnowledgePage[]): KnowledgePage[] {\n  if (tokenRanked.length === 0) return []\n  const seeds = new Set(tokenRanked.slice(0, 5).map((page) => page.id))\n  return pages\n    .map((page) => ({\n      page,\n      score:\n        page.outLinks.filter((link) => seeds.has(link)).length +\n        page.sourceIds.filter((source) =>\n          tokenRanked.some((seed) => seed.sourceIds.includes(source)),\n        ).length,\n    }))\n    .filter((item) => item.score > 0)\n    .sort((a, b) => b.score - a.score || a.page.path.localeCompare(b.page.path))\n    .map((item) => item.page)\n}\n\nfunction tokenScore(page: KnowledgePage, query: string, tokens: string[]): number {\n  const title = page.title.toLowerCase()\n  const path = page.path.toLowerCase()\n  const body = page.text.toLowerCase()\n  const phrase = query.toLowerCase()\n  let score = 0\n  if (path.endsWith(`${phrase}.md`) || title === phrase) score += 200\n  if (title.includes(phrase)) score += 50\n  if (body.includes(phrase)) score += 20\n  for (const token of tokens) {\n    if (title.includes(token)) score += 5\n    if (body.includes(token)) score += 1\n    if (path.includes(token)) score += 3\n  }\n  return score\n}\n\nfunction buildSnippet(text: string, query: string): string {\n  const compact = text.replace(/\\s+/g, ' ').trim()\n  const idx = compact.toLowerCase().indexOf(query.toLowerCase())\n  if (idx < 0) return compact.slice(0, 180)\n  return compact.slice(Math.max(0, idx - 80), Math.min(compact.length, idx + query.length + 100))\n}\n\nfunction reasonsFor(page: KnowledgePage, query: string): string[] {\n  const lower = `${page.title}\\n${page.text}`.toLowerCase()\n  const reasons: string[] = []\n  if (lower.includes(query.toLowerCase())) reasons.push('phrase')\n  if (page.sourceIds.length > 0) reasons.push('sourced')\n  if (page.outLinks.length > 0) reasons.push('linked')\n  return reasons\n}\n"],"mappings":";AAEA,IAAM,QAAQ;AACd,IAAM,aAAa,oBAAI,IAAI;AAAA,EACzB;AAAA,EACA;AAAA,EACA;AAAA,EACA;AAAA,EACA;AAAA,EACA;AAAA,EACA;AAAA,EACA;AAAA,EACA;AAAA,EACA;AAAA,EACA;AAAA,EACA;AAAA,EACA;AAAA,EACA;AAAA,EACA;AAAA,EACA;AAAA,EACA;AACF,CAAC;AAEM,SAAS,gBACd,OACA,OACA,QAAQ,IACiB;AACzB,QAAM,UAAU,MAAM,KAAK;AAC3B,MAAI,YAAY,GAAI,QAAO,CAAC;AAC5B,QAAM,cAAc,aAAa,MAAM,OAAO,OAAO;AACrD,QAAM,cAAc,YAAY,MAAM,OAAO,WAAW;AACxD,QAAM,SAAS,qBAAqB,CAAC,YAAY,IAAI,CAAC,MAAM,EAAE,EAAE,GAAG,YAAY,IAAI,CAAC,MAAM,EAAE,EAAE,CAAC,CAAC;AAChG,QAAM,OAAO,IAAI,IAAI,MAAM,MAAM,IAAI,CAAC,SAAS,CAAC,KAAK,IAAI,IAAI,CAAC,CAAC;AAE/D,QAAM,SAAS,CAAC,GAAG,OAAO,QAAQ,CAAC,EAChC,IAAI,CAAC,CAAC,IAAI,KAAK,OAAO,EAAE,MAAM,KAAK,IAAI,EAAE,GAAG,MAAM,EAAE,EACpD,OAAO,CAAC,SAAyD,QAAQ,KAAK,IAAI,CAAC,EACnF,KAAK,CAAC,GAAG,MAAM,EAAE,QAAQ,EAAE,SAAS,EAAE,KAAK,KAAK,cAAc,EAAE,KAAK,IAAI,CAAC,EAC1E,MAAM,GAAG,KAAK;AAMjB,QAAM,WAAW,OAAO,CAAC,GAAG,SAAS;AAErC,SAAO,OAAO,IAAI,CAAC,MAAM,OAAO;AAAA,IAC9B,MAAM,KAAK;AAAA,IACX,OAAO,KAAK;AAAA,IACZ,UAAU,KAAK;AAAA,IACf,iBAAiB,WAAW,IAAI,KAAK,QAAQ,WAAW;AAAA,IACxD,MAAM,IAAI;AAAA,IACV,SAAS,aAAa,KAAK,KAAK,MAAM,OAAO;AAAA,IAC7C,SAAS,WAAW,KAAK,MAAM,OAAO;AAAA,EACxC,EAAE;AACJ;AAEO,SAAS,cAAc,OAAyB;AACrD,QAAM,MAAM,MACT,YAAY,EACZ,MAAM,iCAAiC,EACvC,OAAO,CAAC,UAAU,MAAM,SAAS,KAAK,CAAC,WAAW,IAAI,KAAK,CAAC;AAC/D,QAAM,SAAmB,CAAC;AAC1B,aAAW,SAAS,KAAK;AACvB,QAAI,+BAA+B,KAAK,KAAK,KAAK,MAAM,SAAS,GAAG;AAClE,YAAM,QAAQ,CAAC,GAAG,KAAK;AACvB,eAAS,IAAI,GAAG,IAAI,MAAM,SAAS,GAAG,IAAK,QAAO,KAAK,MAAM,CAAC,IAAK,MAAM,IAAI,CAAC,CAAE;AAChF,aAAO,KAAK,GAAG,KAAK;AAAA,IACtB;AACA,WAAO,KAAK,KAAK;AAAA,EACnB;AACA,SAAO,CAAC,GAAG,IAAI,IAAI,MAAM,CAAC;AAC5B;AAEO,SAAS,qBAAqB,WAAuB,IAAI,OAA4B;AAC1F,QAAM,SAAS,oBAAI,IAAoB;AACvC,aAAW,QAAQ,WAAW;AAC5B,SAAK,QAAQ,CAAC,IAAI,QAAQ;AACxB,aAAO,IAAI,KAAK,OAAO,IAAI,EAAE,KAAK,KAAK,KAAK,IAAI,MAAM,EAAE;AAAA,IAC1D,CAAC;AAAA,EACH;AACA,SAAO;AACT;AAEA,SAAS,aAAa,OAAwB,OAAgC;AAC5E,QAAM,SAAS,cAAc,KAAK;AAClC,QAAM,YAAY,OAAO,SAAS,IAAI,SAAS,CAAC,MAAM,YAAY,CAAC;AACnE,SAAO,MACJ,IAAI,CAAC,UAAU,EAAE,MAAM,OAAO,WAAW,MAAM,OAAO,SAAS,EAAE,EAAE,EACnE,OAAO,CAAC,SAAS,KAAK,QAAQ,CAAC,EAC/B,KAAK,CAAC,GAAG,MAAM,EAAE,QAAQ,EAAE,SAAS,EAAE,KAAK,KAAK,cAAc,EAAE,KAAK,IAAI,CAAC,EAC1E,IAAI,CAAC,SAAS,KAAK,IAAI;AAC5B;AAEA,SAAS,YAAY,OAAwB,aAA+C;AAC1F,MAAI,YAAY,WAAW,EAAG,QAAO,CAAC;AACtC,QAAM,QAAQ,IAAI,IAAI,YAAY,MAAM,GAAG,CAAC,EAAE,IAAI,CAAC,SAAS,KAAK,EAAE,CAAC;AACpE,SAAO,MACJ,IAAI,CAAC,UAAU;AAAA,IACd;AAAA,IACA,OACE,KAAK,SAAS,OAAO,CAAC,SAAS,MAAM,IAAI,IAAI,CAAC,EAAE,SAChD,KAAK,UAAU;AAAA,MAAO,CAAC,WACrB,YAAY,KAAK,CAAC,SAAS,KAAK,UAAU,SAAS,MAAM,CAAC;AAAA,IAC5D,EAAE;AAAA,EACN,EAAE,EACD,OAAO,CAAC,SAAS,KAAK,QAAQ,CAAC,EAC/B,KAAK,CAAC,GAAG,MAAM,EAAE,QAAQ,EAAE,SAAS,EAAE,KAAK,KAAK,cAAc,EAAE,KAAK,IAAI,CAAC,EAC1E,IAAI,CAAC,SAAS,KAAK,IAAI;AAC5B;AAEA,SAAS,WAAW,MAAqB,OAAe,QAA0B;AAChF,QAAM,QAAQ,KAAK,MAAM,YAAY;AACrC,QAAM,OAAO,KAAK,KAAK,YAAY;AACnC,QAAM,OAAO,KAAK,KAAK,YAAY;AACnC,QAAM,SAAS,MAAM,YAAY;AACjC,MAAI,QAAQ;AACZ,MAAI,KAAK,SAAS,GAAG,MAAM,KAAK,KAAK,UAAU,OAAQ,UAAS;AAChE,MAAI,MAAM,SAAS,MAAM,EAAG,UAAS;AACrC,MAAI,KAAK,SAAS,MAAM,EAAG,UAAS;AACpC,aAAW,SAAS,QAAQ;AAC1B,QAAI,MAAM,SAAS,KAAK,EAAG,UAAS;AACpC,QAAI,KAAK,SAAS,KAAK,EAAG,UAAS;AACnC,QAAI,KAAK,SAAS,KAAK,EAAG,UAAS;AAAA,EACrC;AACA,SAAO;AACT;AAEA,SAAS,aAAa,MAAc,OAAuB;AACzD,QAAM,UAAU,KAAK,QAAQ,QAAQ,GAAG,EAAE,KAAK;AAC/C,QAAM,MAAM,QAAQ,YAAY,EAAE,QAAQ,MAAM,YAAY,CAAC;AAC7D,MAAI,MAAM,EAAG,QAAO,QAAQ,MAAM,GAAG,GAAG;AACxC,SAAO,QAAQ,MAAM,KAAK,IAAI,GAAG,MAAM,EAAE,GAAG,KAAK,IAAI,QAAQ,QAAQ,MAAM,MAAM,SAAS,GAAG,CAAC;AAChG;AAEA,SAAS,WAAW,MAAqB,OAAyB;AAChE,QAAM,QAAQ,GAAG,KAAK,KAAK;AAAA,EAAK,KAAK,IAAI,GAAG,YAAY;AACxD,QAAM,UAAoB,CAAC;AAC3B,MAAI,MAAM,SAAS,MAAM,YAAY,CAAC,EAAG,SAAQ,KAAK,QAAQ;AAC9D,MAAI,KAAK,UAAU,SAAS,EAAG,SAAQ,KAAK,SAAS;AACrD,MAAI,KAAK,SAAS,SAAS,EAAG,SAAQ,KAAK,QAAQ;AACnD,SAAO;AACT;","names":[]}