rust-kgdb 0.6.31 → 0.6.32

package/HYPERMIND_BENCHMARK_REPORT.md

@@ -1,28 +1,31 @@
 # HyperMind Benchmark Report
 
-## AI That Doesn't Hallucinate: The Results
+## Verified Framework Comparison: Schema Injection Works
 
-**Version**: 0.6.17
+**Version**: 0.6.32
 **Date**: December 16, 2025
-**SDK**: rust-kgdb@0.6.17
+**SDK**: rust-kgdb@0.6.32
 
 ---
 
-## The Bottom Line
+## Executive Summary (Verified Results)
 
-**HyperMind achieves 86.4% accuracy where vanilla LLMs achieve 0%.**
+**Schema injection improves ALL frameworks by +66.7 percentage points.**
 
-| Metric | Vanilla LLM | HyperMind | Improvement |
-|--------|-------------|-----------|-------------|
-| **Accuracy** | 0% | 86.4% | +86.4 pp |
-| **Hallucinations** | 100% | 0% | Eliminated |
-| **Audit Trail** | None | Complete | Full provenance |
-| **Claude Sonnet 4** | 0% | 90.9% | +90.9 pp |
-| **GPT-4o** | 0% | 81.8% | +81.8 pp |
+| Framework | No Schema | With Schema | Improvement |
+|-----------|-----------|-------------|-------------|
+| **Vanilla OpenAI** | 0.0% | 71.4% | +71.4 pp |
+| **LangChain** | 0.0% | 71.4% | +71.4 pp |
+| **DSPy** | 14.3% | 71.4% | +57.1 pp |
+| **Average** | 4.8% | **71.4%** | **+66.7 pp** |
+
+*GPT-4o, 7 LUBM queries, real API calls, no mocking. See `verified_benchmark_results.json`.*
+
+**Key Insight**: The value is in the ARCHITECTURE (schema injection, type contracts), not the specific framework.
 
 ---
 
-## Why Vanilla LLMs Fail
+## Why Vanilla LLMs Fail (Without Schema)
 
 When you ask a vanilla LLM to query your database:
 
@@ -33,39 +36,194 @@ When you ask a vanilla LLM to query your database:
 
 ---
 
-## How HyperMind Fixes This
+## How Schema Injection Fixes This
 
-HyperMind grounds every answer in your actual data:
+The HyperMind approach (schema injection) works with ANY framework:
 
 1. **Schema injection** - LLM sees your real data structure (30 classes, 23 properties)
-2. **Input/output validation** - Prevents invalid query combinations
-3. **Reasoning trace** - Every answer shows exactly how it was derived
-4. **Reproducible** - Same question = Same answer = Same hash
+2. **Output format** - Explicit instructions for raw SPARQL (no markdown)
+3. **Type contracts** - Predicate constraints from actual schema
+4. **Reproducible** - Same question = Same answer
+
+---
+
+## Benchmark Setup: Code for Each Framework
+
+### Test Queries (Same for All Frameworks)
+
+```python
+TEST_QUERIES = [
+    {"id": "A1", "question": "Find all teachers", "correct_predicate": "teacherOf"},
+    {"id": "A2", "question": "Get student emails", "correct_predicate": "emailAddress"},
+    {"id": "A3", "question": "Find faculty members", "correct_predicate": "Professor"},
+    {"id": "S1", "question": "Write a SPARQL query to count professors. Just give me the query."},
+    {"id": "S2", "question": "SPARQL only, no explanation: find graduate students"},
+    {"id": "M1", "question": "Find professors who work for departments"},
+    {"id": "E1", "question": "Find professors with no publications"}
+]
+```
+
+### LUBM Schema (Injected for "With Schema" Tests)
+
+```python
+LUBM_SCHEMA = """LUBM (Lehigh University Benchmark) Schema:
+
+PREFIX ub: <http://swat.cse.lehigh.edu/onto/univ-bench.owl#>
 
+Classes: University, Department, Professor, AssociateProfessor, AssistantProfessor,
+         FullProfessor, Lecturer, GraduateStudent, UndergraduateStudent,
+         Course, GraduateCourse, Publication, Research, ResearchGroup
+
+Properties:
+  - ub:worksFor (person → organization)
+  - ub:memberOf (person → organization)
+  - ub:advisor (student → professor)
+  - ub:takesCourse (student → course)
+  - ub:teacherOf (professor → course)
+  - ub:publicationAuthor (publication → person)
+  - ub:subOrganizationOf (organization → organization)
+  - ub:emailAddress (person → string)
+
+IMPORTANT: Use ONLY these predicates. Do NOT use: teacher, email, faculty"""
 ```
-┌─────────────────────────────────────────────────────────────────┐
-│                    YOUR QUESTION                                 │
-│        "Find professors who teach courses..."                    │
-└───────────────────────────┬─────────────────────────────────────┘
-                            │
-                            ▼
-┌─────────────────────────────────────────────────────────────────┐
-│               VANILLA LLM                                        │
-│  ❌ No schema awareness                                          │
-│  ❌ Hallucinates predicates                                      │
-│  ❌ Wraps in markdown                                            │
-│  ❌ 0% success rate                                              │
-└─────────────────────────────────────────────────────────────────┘
-
-                            vs.
-
-┌─────────────────────────────────────────────────────────────────┐
-│               HYPERMIND                                          │
-│  ✅ Sees your actual schema (30 classes, 23 properties)         │
-│  ✅ Validates query structure before execution                   │
-│  ✅ Provides complete reasoning trace                            │
-│  ✅ 86.4% success rate                                           │
-└─────────────────────────────────────────────────────────────────┘
+
+---
+
+## Framework Code Comparison
+
+### 1. Vanilla OpenAI (No Schema) - 0% Accuracy
+
+```python
+from openai import OpenAI
+client = OpenAI(api_key=api_key)
+
+response = client.chat.completions.create(
+    model="gpt-4o",
+    messages=[{"role": "user", "content": f"Generate a SPARQL query for: {question}"}],
+    max_tokens=500
+)
+sparql = response.choices[0].message.content
+# Result: 0/7 passed - all wrapped in markdown
+```
+
+### 2. Vanilla OpenAI (With Schema) - 71.4% Accuracy
+
+```python
+from openai import OpenAI
+client = OpenAI(api_key=api_key)
+
+prompt = f"""You are a SPARQL query generator.
+
+{LUBM_SCHEMA}
+
+TYPE CONTRACT:
+- Input: natural language query
+- Output: raw SPARQL (NO markdown, NO code blocks, NO explanation)
+- Use ONLY predicates from the schema above
+
+Query: {question}
+
+Output raw SPARQL only:"""
+
+response = client.chat.completions.create(
+    model="gpt-4o",
+    messages=[{"role": "user", "content": prompt}],
+    max_tokens=500
+)
+sparql = response.choices[0].message.content
+# Result: 5/7 passed - schema prevents wrong predicates
+```
+
+### 3. LangChain (No Schema) - 0% Accuracy
+
+```python
+from langchain_openai import ChatOpenAI
+from langchain_core.prompts import PromptTemplate
+from langchain_core.output_parsers import StrOutputParser
+
+llm = ChatOpenAI(model="gpt-4o", api_key=api_key)
+parser = StrOutputParser()
+
+template = PromptTemplate(
+    input_variables=["question"],
+    template="Generate a SPARQL query for: {question}"
+)
+chain = template | llm | parser
+
+sparql = chain.invoke({"question": question})
+# Result: 0/7 passed - all wrapped in markdown
+```
+
+### 4. LangChain (With Schema) - 71.4% Accuracy
+
+```python
+from langchain_openai import ChatOpenAI
+from langchain_core.prompts import PromptTemplate
+from langchain_core.output_parsers import StrOutputParser
+
+llm = ChatOpenAI(model="gpt-4o", api_key=api_key)
+parser = StrOutputParser()
+
+template = PromptTemplate(
+    input_variables=["question", "schema"],
+    template="""You are a SPARQL query generator.
+
+{schema}
+
+TYPE CONTRACT:
+- Input: natural language query
+- Output: raw SPARQL (NO markdown, NO code blocks, NO explanation)
+- Use ONLY predicates from the schema above
+
+Query: {question}
+
+Output raw SPARQL only:"""
+)
+chain = template | llm | parser
+
+sparql = chain.invoke({"question": question, "schema": LUBM_SCHEMA})
+# Result: 5/7 passed - same as vanilla with schema
+```
+
+### 5. DSPy (No Schema) - 14.3% Accuracy
+
+```python
+import dspy
+from dspy import LM
+
+lm = LM("openai/gpt-4o")
+dspy.configure(lm=lm)
+
+class SPARQLGenerator(dspy.Signature):
+    """Generate SPARQL query from natural language."""
+    question = dspy.InputField(desc="Natural language question")
+    sparql = dspy.OutputField(desc="SPARQL query")
+
+generator = dspy.Predict(SPARQLGenerator)
+response = generator(question=question)
+sparql = response.sparql
+# Result: 1/7 passed - slightly better output formatting
+```
+
+### 6. DSPy (With Schema) - 71.4% Accuracy
+
+```python
+import dspy
+from dspy import LM
+
+lm = LM("openai/gpt-4o")
+dspy.configure(lm=lm)
+
+class SchemaSPARQLGenerator(dspy.Signature):
+    """Generate SPARQL query using the provided schema. Output raw SPARQL only."""
+    schema = dspy.InputField(desc="Database schema with classes and properties")
+    question = dspy.InputField(desc="Natural language question")
+    sparql = dspy.OutputField(desc="Raw SPARQL query (no markdown, no explanation)")
+
+generator = dspy.Predict(SchemaSPARQLGenerator)
+response = generator(schema=LUBM_SCHEMA, question=question)
+sparql = response.sparql
+# Result: 5/7 passed - same as others with schema
 ```
 
 ---

package/README.md

@@ -12,27 +12,27 @@
 
 ---
 
-## Results
+## Results (Verified December 2025)
 
 ```
 ┌─────────────────────────────────────────────────────────────────────────────┐
 │  BENCHMARK: LUBM (Lehigh University Benchmark)                              │
 │  DATASET:   3,272 triples │ 30 OWL classes │ 23 properties                  │
-│  TESTS:     11 hard scenarios (ambiguous, multi-hop, edge cases)            │
-│  PROTOCOL:  Query → Parse → Type-check → Execute → Verify                   │
+│  MODEL:     GPT-4o │ Real API calls │ No mocking                            │
 ├─────────────────────────────────────────────────────────────────────────────┤
 │                                                                             │
-│  METRIC            VANILLA LLM      HYPERMIND       IMPROVEMENT             │
+│  FRAMEWORK         NO SCHEMA     WITH SCHEMA    IMPROVEMENT                 │
 │  ─────────────────────────────────────────────────────────────              │
-│  Accuracy          0%               86.4%           +86.4 pp                │
-│  Hallucinations    100%             0%              Eliminated              │
-│  Audit Trail       None             Complete        Full provenance         │
-│  Reproducibility   Random           Deterministic   Same hash               │
+│  Vanilla OpenAI    0.0%          71.4%          +71.4 pp                    │
+│  LangChain         0.0%          71.4%          +71.4 pp                    │
+│  DSPy              14.3%         71.4%          +57.1 pp                    │
+│  ─────────────────────────────────────────────────────────────              │
+│  AVERAGE           4.8%          71.4%          +66.7 pp                    │
 │                                                                             │
-│  Claude Sonnet 4:  90.9% accuracy                                           │
-│  GPT-4o:           81.8% accuracy                                           │
+│  KEY INSIGHT: Schema injection improves ALL frameworks equally.             │
+│  HyperMind's value = architecture, not framework.                           │
 │                                                                             │
-│  Reproduce: node vanilla-vs-hypermind-benchmark.js                          │
+│  Reproduce: python3 benchmark-frameworks.py                                 │
 └─────────────────────────────────────────────────────────────────────────────┘
 ```
 
@@ -811,27 +811,44 @@ console.log('Supersteps:', result.supersteps)  // 5
 | Virtuoso | ~5 µs | 35-75 bytes | No |
 | Blazegraph | ~100 µs | 100+ bytes | No |
 
-### AI Agent Accuracy
+### AI Agent Accuracy (Verified December 2025)
+
+| Framework | No Schema | With Schema (HyperMind) | Improvement |
+|-----------|-----------|-------------------------|-------------|
+| **Vanilla OpenAI** | 0.0% | 71.4% | +71.4 pp |
+| **LangChain** | 0.0% | 71.4% | +71.4 pp |
+| **DSPy** | 14.3% | 71.4% | +57.1 pp |
+| **Average** | 4.8% | **71.4%** | **+66.7 pp** |
+
+*Tested: GPT-4o, 7 LUBM queries, real API calls. See `framework_benchmark_*.json` for raw data.*
+
+### AI Framework Architectural Comparison
+
+| Framework | Type Safety | Schema Aware | Symbolic Execution | Audit Trail |
+|-----------|-------------|--------------|-------------------|-------------|
+| **HyperMind** | ✅ Yes | ✅ Yes | ✅ Yes | ✅ Yes |
+| LangChain | ❌ No | ❌ No | ❌ No | ❌ No |
+| DSPy | ⚠️ Partial | ❌ No | ❌ No | ❌ No |
+
+**Key Insight**: Schema injection (HyperMind's architecture) provides +66.7 pp improvement across ALL frameworks. The value is in the architecture, not the specific framework.
+
+### Reproduce Benchmarks
 
-| Approach | Accuracy | Why |
-|----------|----------|-----|
-| **Vanilla LLM** | 0% | Hallucinated predicates, markdown in SPARQL |
-| **HyperMind** | 86.4% | Schema injection, typed tools, audit trail |
+Two benchmark scripts are available for verification:
 
-### AI Framework Comparison
+```bash
+# JavaScript: HyperMind vs Vanilla LLM on LUBM (12 queries)
+ANTHROPIC_API_KEY=... OPENAI_API_KEY=... node vanilla-vs-hypermind-benchmark.js
 
-| Framework | Type Safety | Schema Aware | Symbolic Execution | Success Rate |
-|-----------|-------------|--------------|-------------------|--------------|
-| **HyperMind** | ✅ Yes | ✅ Yes | ✅ Yes | **86.4%** |
-| LangChain | ❌ No | ❌ No | ❌ No | ~20-40%* |
-| AutoGPT | ❌ No | ❌ No | ❌ No | ~10-25%* |
-| DSPy | ⚠️ Partial | ❌ No | ❌ No | ~30-50%* |
+# Python: Compare frameworks (Vanilla, LangChain, DSPy) with/without schema
+OPENAI_API_KEY=... uv run --with openai --with langchain --with langchain-openai --with langchain-core --with dspy-ai python3 benchmark-frameworks.py
+```
 
-*Estimated from GAIA (Meta Research, 2023), SWE-bench (OpenAI, 2024), and LUBM (Lehigh University) benchmarks on structured data tasks. HyperMind results measured on LUBM-1 dataset (3,272 triples, 30 classes, 23 properties) using vanilla-vs-hypermind-benchmark.js.
+Both scripts make real API calls and report actual results. No mocking.
 
-**Why HyperMind Wins**:
+**Why These Features Matter**:
 - **Type Safety**: Tools have typed signatures (Query → BindingSet), invalid combinations rejected
-- **Schema Awareness**: LLM sees your actual data structure, can only reference real properties
+- **Schema Awareness**: Planner sees your actual data structure, can only reference real properties
 - **Symbolic Execution**: Queries run against real database, not LLM imagination
 - **Audit Trail**: Every answer has cryptographic hash for reproducibility
 
@@ -1164,140 +1181,6 @@ const result = await agent.call('Find collusion patterns')
 // Result: ✅ Type-safe, domain-aware, auditable
 ```
 
-### Code Comparison: DSPy vs HyperMind
-
-#### DSPy Approach (Prompt Optimization)
-
-```python
-# DSPy: Statistically optimized prompt - NO guarantees
-
-import dspy
-
-class FraudDetector(dspy.Signature):
-    """Find fraud patterns in claims data."""
-    claims_data = dspy.InputField()
-    fraud_patterns = dspy.OutputField()
-
-class FraudPipeline(dspy.Module):
-    def __init__(self):
-        self.detector = dspy.ChainOfThought(FraudDetector)
-
-    def forward(self, claims):
-        return self.detector(claims_data=claims)
-
-# "Optimize" via statistical fitting
-optimizer = dspy.BootstrapFewShot(metric=some_metric)
-optimized = optimizer.compile(FraudPipeline(), trainset=examples)
-
-# Call and HOPE it works
-result = optimized(claims="[claim data here]")
-
-# ❌ No type guarantee - fraud_patterns could be anything
-# ❌ No proof of execution - just text output
-# ❌ No composition safety - next step might fail
-# ❌ No audit trail - "it said fraud" is not compliance
-```
-
-**What DSPy produces:** A string that *probably* contains fraud patterns.
-
-#### HyperMind Approach (Mathematical Proof)
-
-```javascript
-// HyperMind: Type-safe morphism composition - PROVEN correct
-
-const { GraphDB, GraphFrame, DatalogProgram, evaluateDatalog } = require('rust-kgdb')
-
-// Step 1: Load typed knowledge graph (Schema enforced)
-const db = new GraphDB('http://insurance.org/fraud-kb')
-db.loadTtl(`
-  @prefix : <http://insurance.org/> .
-  :CLM001 :amount "18500" ; :claimant :P001 ; :provider :PROV001 .
-  :P001 :paidTo :P002 .
-  :P002 :paidTo :P003 .
-  :P003 :paidTo :P001 .
-`, null)
-
-// Step 2: GraphFrame analysis (Morphism: Graph → TriangleCount)
-// Type signature: GraphFrame → number (guaranteed)
-const graph = new GraphFrame(
-  JSON.stringify([{id:'P001'}, {id:'P002'}, {id:'P003'}]),
-  JSON.stringify([
-    {src:'P001', dst:'P002'},
-    {src:'P002', dst:'P003'},
-    {src:'P003', dst:'P001'}
-  ])
-)
-const triangles = graph.triangleCount()  // Type: number (always)
-
-// Step 3: Datalog inference (Morphism: Rules → Facts)
-// Type signature: DatalogProgram → InferredFacts (guaranteed)
-const datalog = new DatalogProgram()
-datalog.addFact(JSON.stringify({predicate:'claim', terms:['CLM001','P001','PROV001']}))
-datalog.addFact(JSON.stringify({predicate:'related', terms:['P001','P002']}))
-
-datalog.addRule(JSON.stringify({
-  head: {predicate:'collusion', terms:['?P1','?P2','?Prov']},
-  body: [
-    {predicate:'claim', terms:['?C1','?P1','?Prov']},
-    {predicate:'claim', terms:['?C2','?P2','?Prov']},
-    {predicate:'related', terms:['?P1','?P2']}
-  ]
-}))
-
-const result = JSON.parse(evaluateDatalog(datalog))
-
-// ✓ Type guarantee: result.collusion is always array of tuples
-// ✓ Proof of execution: Datalog evaluation is deterministic
-// ✓ Composition safety: Each step has typed input/output
-// ✓ Audit trail: Every fact derivation is traceable
-```
-
-**What HyperMind produces:** Typed results with mathematical proof of derivation.
-
-#### Actual Output Comparison
-
-**DSPy Output:**
-```
-fraud_patterns: "I found some suspicious patterns involving P001 and P002
-that appear to be related. There might be collusion with provider PROV001."
-```
-*How do you validate this? You can't. It's text.*
-
-**HyperMind Output:**
-```json
-{
-  "triangles": 1,
-  "collusion": [["P001", "P002", "PROV001"]],
-  "executionWitness": {
-    "tool": "datalog.evaluate",
-    "input": "6 facts, 1 rule",
-    "output": "collusion(P001,P002,PROV001)",
-    "derivation": "claim(CLM001,P001,PROV001) ∧ claim(CLM002,P002,PROV001) ∧ related(P001,P002) → collusion(P001,P002,PROV001)",
-    "timestamp": "2024-12-14T10:30:00Z",
-    "semanticHash": "semhash:collusion-p001-p002-prov001"
-  }
-}
-```
-*Every result has a logical derivation and cryptographic proof.*
-
-#### The Compliance Question
-
-**Auditor:** "How do you know P001-P002-PROV001 is actually collusion?"
-
-**DSPy Team:** "Our model said so. It was trained on examples and optimized for accuracy."
-
-**HyperMind Team:** "Here's the derivation chain:
-1. `claim(CLM001, P001, PROV001)` - fact from data
-2. `claim(CLM002, P002, PROV001)` - fact from data
-3. `related(P001, P002)` - fact from data
-4. Rule: `collusion(?P1, ?P2, ?Prov) :- claim(?C1, ?P1, ?Prov), claim(?C2, ?P2, ?Prov), related(?P1, ?P2)`
-5. Unification: `?P1=P001, ?P2=P002, ?Prov=PROV001`
-6. Conclusion: `collusion(P001, P002, PROV001)` - QED
-
-Here's the semantic hash: `semhash:collusion-p001-p002-prov001` - same query intent will always return this exact result."
-
-**Result:** HyperMind passes audit. DSPy gets you a follow-up meeting with legal.
-
 ### Why Vanilla LLMs Fail
 
 When you ask an LLM to query a knowledge graph, it produces **broken SPARQL 85% of the time**:
@@ -1346,16 +1229,15 @@ Result: ❌ PARSER ERROR - Invalid SPARQL syntax
 
 **Note**: Tentris implements WCOJ (see [ISWC 2025 paper](https://papers.dice-research.org/2025/ISWC_Tentris-WCOJ-Update/public.pdf)). rust-kgdb is the only system combining WCOJ with mobile support and integrated AI framework.
 
-#### AI Framework Comparison
+#### AI Framework Architectural Comparison
 
-| Framework | Type Safety | Schema Aware | Symbolic Execution | Audit Trail | Success Rate |
-|-----------|-------------|--------------|-------------------|-------------|--------------|
-| **HyperMind** | ✅ Yes | ✅ Yes | ✅ Yes | ✅ Yes | **86.4%** |
-| LangChain | ❌ No | ❌ No | ❌ No | ❌ No | ~20-40%* |
-| AutoGPT | ❌ No | ❌ No | ❌ No | ❌ No | ~10-25%* |
-| DSPy | ⚠️ Partial | ❌ No | ❌ No | ❌ No | ~30-50%* |
+| Framework | Type Safety | Schema Aware | Symbolic Execution | Audit Trail |
+|-----------|-------------|--------------|-------------------|-------------|
+| **HyperMind** | ✅ Yes | ✅ Yes | ✅ Yes | ✅ Yes |
+| LangChain | ❌ No | ❌ No | ❌ No | ❌ No |
+| DSPy | ⚠️ Partial | ❌ No | ❌ No | ❌ No |
 
-*Estimated from GAIA (Meta Research, 2023), SWE-bench (OpenAI, 2024), and LUBM (Lehigh University) benchmarks. HyperMind: LUBM-1 (3,272 triples).
+**Note**: This compares architectural features. Benchmark (Dec 2025): Schema injection improves all frameworks by +66.7 pp (Vanilla: 0%→71.4%, LangChain: 0%→71.4%, DSPy: 14.3%→71.4%).
 
 ```
 ┌─────────────────────────────────────────────────────────────────┐
@@ -1368,12 +1250,10 @@ Result: ❌ PARSER ERROR - Invalid SPARQL syntax
 │  Apache Jena:    Great features, but 150+ µs lookups            │
 │  Neo4j:          Popular, but no SPARQL/RDF standards           │
 │  Amazon Neptune: Managed, but cloud-only vendor lock-in         │
-│  LangChain:      Vibe coding, fails compliance audits           │
-│  DSPy:           Statistical optimization, no guarantees        │
 │                                                                 │
 │  rust-kgdb:      2.78 µs lookups, WCOJ joins, mobile-native     │
 │                  Standalone → Clustered on same codebase        │
-│                  Mathematical foundations, audit-ready           │
+│                  Deterministic planner, audit-ready              │
 │                                                                 │
 └─────────────────────────────────────────────────────────────────┘
 ```