@jcode.labs/ragmir 2.6.0 → 2.7.0

package/examples/document-evidence-benchmark/README.md

@@ -0,0 +1,33 @@
+# Document Evidence Benchmark
+
+Synthetic document benchmark for Ragmir retrieval. It is intentionally safe to commit: the contract,
+RFP, runbook, spec, and legal/tax notes are fictional and contain no customer data.
+
+This benchmark checks two things:
+
+- `recall@k`: each query retrieves the expected source file.
+- exact citations: each query retrieves the expected `relative/path:Lx-Ly#chunkIndex` citation.
+
+The corpus is small on purpose so every source file fits in a single chunk. That keeps the expected
+citations deterministic while exercising the same CLI path used for private dogfooding.
+
+## Run From This Repository Checkout
+
+Build Ragmir once from the repository root:
+
+```bash
+pnpm build
+```
+
+Then run the benchmark from this folder:
+
+```bash
+cd packages/ragmir-core/examples/document-evidence-benchmark
+node ../../dist/cli.js ingest
+node ../../dist/cli.js evaluate --golden golden-queries.json --json
+node ../../dist/cli.js evaluate --golden golden-queries.json --fail-under 1
+```
+
+Use this example as a public-safe template for private document evaluations. Keep real contracts,
+RFPs, runbooks, tax notes, golden queries, and generated JSON reports outside Git or under ignored
+local Ragmir state.

package/examples/document-evidence-benchmark/golden-queries.json

@@ -0,0 +1,35 @@
+{
+  "topK": 3,
+  "queries": [
+    {
+      "id": "contract-data-export-residency",
+      "query": "MSA client data export residency renewal notice DOCBENCH-MSA-001",
+      "expectedPaths": ["raw/contracts/master-services-agreement.md"],
+      "expectedCitations": ["raw/contracts/master-services-agreement.md:L1-L11#0"]
+    },
+    {
+      "id": "rfp-no-hosted-account",
+      "query": "security questionnaire no hosted account no telemetry read focused MCP",
+      "expectedPaths": ["raw/rfp/security-questionnaire.md"],
+      "expectedCitations": ["raw/rfp/security-questionnaire.md:L1-L12#0"]
+    },
+    {
+      "id": "runbook-evidence-pack",
+      "query": "What runbook evidence must be collected after a payment incident?",
+      "expectedPaths": ["raw/runbooks/incident-response-runbook.md"],
+      "expectedCitations": ["raw/runbooks/incident-response-runbook.md:L1-L11#0"]
+    },
+    {
+      "id": "spec-complements-agent-memory",
+      "query": "agent integration spec complements agent memory conversation state",
+      "expectedPaths": ["raw/specs/agent-integration-spec.md"],
+      "expectedCitations": ["raw/specs/agent-integration-spec.md:L1-L11#0"]
+    },
+    {
+      "id": "legal-tax-professional-review",
+      "query": "legal tax residency review professional review filing positions",
+      "expectedPaths": ["raw/legal-tax/residency-review-note.md"],
+      "expectedCitations": ["raw/legal-tax/residency-review-note.md:L1-L10#0"]
+    }
+  ]
+}

package/examples/document-evidence-benchmark/raw/contracts/master-services-agreement.md

@@ -0,0 +1,11 @@
+# Synthetic Master Services Agreement
+
+Agreement ID: DOCBENCH-MSA-001. This fictional contract is for retrieval testing only.
+
+The supplier may process implementation notes, support tickets, and project specifications only from
+the customer-designated workstation. Client data exports must remain in the agreed local evidence
+folder and may not be copied to a hosted account without written approval from the customer owner.
+
+Renewal notice window: either party may stop automatic renewal by sending written notice at least
+60 days before the renewal date. The contract owner must attach the local export residency evidence
+to the renewal review.

package/examples/document-evidence-benchmark/raw/legal-tax/residency-review-note.md

@@ -0,0 +1,10 @@
+# Synthetic Legal And Tax Review Note
+
+Matter: fictional cross-border founder residency review for benchmark testing.
+
+The note compares board minutes, travel calendars, service contracts, and tax residency assumptions.
+It is not legal advice. Professional review is required before relying on residency analysis,
+changing filing positions, or sending a client-facing recommendation.
+
+The cited evidence package must separate facts, assumptions, and uncertainty. Missing documents
+should be listed explicitly instead of filled with model guesses.

package/examples/document-evidence-benchmark/raw/rfp/security-questionnaire.md

@@ -0,0 +1,12 @@
+# Synthetic RFP Security Questionnaire
+
+Question: Does the proposed retrieval layer require a hosted account, product telemetry, or broad
+write access to the repository?
+
+Answer: No hosted account is required. The default mode runs locally with no telemetry. MCP access is
+read-focused: the agent can ask for status, prompt-routing advice, search results, cited answers,
+audits, evaluation results, usage summaries, and security posture, but it does not receive a
+destructive repository tool from Ragmir.
+
+Follow-up: customer-specific evidence stays on disk and is reviewed by a human before an RFP response
+is sent.

package/examples/document-evidence-benchmark/raw/runbooks/incident-response-runbook.md

@@ -0,0 +1,11 @@
+# Synthetic Payment Incident Runbook
+
+Trigger: payment queue latency above the internal threshold for ten minutes.
+
+The on-call engineer opens the local incident template and collects a payment incident evidence pack:
+timeline, customer impact summary, rollback decision, command transcript, and links to cited runbook
+sections. The evidence pack remains under the local Ragmir reports folder until the incident lead
+approves external sharing.
+
+Do not paste raw customer records into an agent prompt. Use cited retrieval snippets and summarize
+uncertainty separately from confirmed evidence.

package/examples/document-evidence-benchmark/raw/specs/agent-integration-spec.md

@@ -0,0 +1,11 @@
+# Synthetic Agent Integration Spec
+
+Goal: let external coding agents use local project evidence without turning Ragmir into another chat
+memory system.
+
+Ragmir complements agent memory. It does not replace the agent conversation state, task plan, or
+native code index. The agent keeps its normal reasoning loop, then calls Ragmir when it needs cited
+evidence from local documents such as contracts, RFPs, runbooks, or specs.
+
+The MCP surface stays read-focused and bounded by topK. This is a security advantage: agents receive
+verifiable passages and audit data without gaining a broad document mutation channel.

package/package.json

@@ -1,6 +1,6 @@
 {
   "name": "@jcode.labs/ragmir",
-  "version": "2.6.0",
+  "version": "2.7.0",
   "description": "Ragmir Core: open-source sovereign local RAG for confidential datasets and AI agents.",
   "type": "module",
   "license": "MIT",
@@ -69,8 +69,8 @@
     "unpdf": "^1.4.0",
     "yaml": "^2.9.0",
     "zod": "^4.1.13",
-    "@jcode.labs/ragmir-chat": "2.6.0",
-    "@jcode.labs/ragmir-tts": "2.6.0"
+    "@jcode.labs/ragmir-chat": "2.7.0",
+    "@jcode.labs/ragmir-tts": "2.7.0"
   },
   "devDependencies": {
     "@types/html-to-text": "^9.0.4",