rust-kgdb 0.6.31 → 0.6.33

package/HYPERMIND_BENCHMARK_REPORT.md

@@ -1,28 +1,31 @@
 # HyperMind Benchmark Report
 
-## AI That Doesn't Hallucinate: The Results
+## Verified Framework Comparison: Schema Injection Works
 
-**Version**: 0.6.17
+**Version**: 0.6.32
 **Date**: December 16, 2025
-**SDK**: rust-kgdb@0.6.17
+**SDK**: rust-kgdb@0.6.32
 
 ---
 
-## The Bottom Line
+## Executive Summary (Verified Results)
 
-**HyperMind achieves 86.4% accuracy where vanilla LLMs achieve 0%.**
+**Schema injection improves ALL frameworks by +66.7 percentage points.**
 
-| Metric | Vanilla LLM | HyperMind | Improvement |
-|--------|-------------|-----------|-------------|
-| **Accuracy** | 0% | 86.4% | +86.4 pp |
-| **Hallucinations** | 100% | 0% | Eliminated |
-| **Audit Trail** | None | Complete | Full provenance |
-| **Claude Sonnet 4** | 0% | 90.9% | +90.9 pp |
-| **GPT-4o** | 0% | 81.8% | +81.8 pp |
+| Framework | No Schema | With Schema | Improvement |
+|-----------|-----------|-------------|-------------|
+| **Vanilla OpenAI** | 0.0% | 71.4% | +71.4 pp |
+| **LangChain** | 0.0% | 71.4% | +71.4 pp |
+| **DSPy** | 14.3% | 71.4% | +57.1 pp |
+| **Average** | 4.8% | **71.4%** | **+66.7 pp** |
+
+*GPT-4o, 7 LUBM queries, real API calls, no mocking. See `verified_benchmark_results.json`.*
+
+**Key Insight**: The value is in the ARCHITECTURE (schema injection, type contracts), not the specific framework.
 
 ---
 
-## Why Vanilla LLMs Fail
+## Why Vanilla LLMs Fail (Without Schema)
 
 When you ask a vanilla LLM to query your database:
 
@@ -33,39 +36,194 @@ When you ask a vanilla LLM to query your database:
 
 ---
 
-## How HyperMind Fixes This
+## How Schema Injection Fixes This
 
-HyperMind grounds every answer in your actual data:
+The HyperMind approach (schema injection) works with ANY framework:
 
 1. **Schema injection** - LLM sees your real data structure (30 classes, 23 properties)
-2. **Input/output validation** - Prevents invalid query combinations
-3. **Reasoning trace** - Every answer shows exactly how it was derived
-4. **Reproducible** - Same question = Same answer = Same hash
+2. **Output format** - Explicit instructions for raw SPARQL (no markdown)
+3. **Type contracts** - Predicate constraints from actual schema
+4. **Reproducible** - Same question = Same answer
+
+---
+
+## Benchmark Setup: Code for Each Framework
+
+### Test Queries (Same for All Frameworks)
+
+```python
+TEST_QUERIES = [
+    {"id": "A1", "question": "Find all teachers", "correct_predicate": "teacherOf"},
+    {"id": "A2", "question": "Get student emails", "correct_predicate": "emailAddress"},
+    {"id": "A3", "question": "Find faculty members", "correct_predicate": "Professor"},
+    {"id": "S1", "question": "Write a SPARQL query to count professors. Just give me the query."},
+    {"id": "S2", "question": "SPARQL only, no explanation: find graduate students"},
+    {"id": "M1", "question": "Find professors who work for departments"},
+    {"id": "E1", "question": "Find professors with no publications"}
+]
+```
+
+### LUBM Schema (Injected for "With Schema" Tests)
+
+```python
+LUBM_SCHEMA = """LUBM (Lehigh University Benchmark) Schema:
+
+PREFIX ub: <http://swat.cse.lehigh.edu/onto/univ-bench.owl#>
 
+Classes: University, Department, Professor, AssociateProfessor, AssistantProfessor,
+         FullProfessor, Lecturer, GraduateStudent, UndergraduateStudent,
+         Course, GraduateCourse, Publication, Research, ResearchGroup
+
+Properties:
+  - ub:worksFor (person → organization)
+  - ub:memberOf (person → organization)
+  - ub:advisor (student → professor)
+  - ub:takesCourse (student → course)
+  - ub:teacherOf (professor → course)
+  - ub:publicationAuthor (publication → person)
+  - ub:subOrganizationOf (organization → organization)
+  - ub:emailAddress (person → string)
+
+IMPORTANT: Use ONLY these predicates. Do NOT use: teacher, email, faculty"""
 ```
-┌─────────────────────────────────────────────────────────────────┐
-│                    YOUR QUESTION                                 │
-│        "Find professors who teach courses..."                    │
-└───────────────────────────┬─────────────────────────────────────┘
-                            │
-                            ▼
-┌─────────────────────────────────────────────────────────────────┐
-│               VANILLA LLM                                        │
-│  ❌ No schema awareness                                          │
-│  ❌ Hallucinates predicates                                      │
-│  ❌ Wraps in markdown                                            │
-│  ❌ 0% success rate                                              │
-└─────────────────────────────────────────────────────────────────┘
-
-                            vs.
-
-┌─────────────────────────────────────────────────────────────────┐
-│               HYPERMIND                                          │
-│  ✅ Sees your actual schema (30 classes, 23 properties)         │
-│  ✅ Validates query structure before execution                   │
-│  ✅ Provides complete reasoning trace                            │
-│  ✅ 86.4% success rate                                           │
-└─────────────────────────────────────────────────────────────────┘
+
+---
+
+## Framework Code Comparison
+
+### 1. Vanilla OpenAI (No Schema) - 0% Accuracy
+
+```python
+from openai import OpenAI
+client = OpenAI(api_key=api_key)
+
+response = client.chat.completions.create(
+    model="gpt-4o",
+    messages=[{"role": "user", "content": f"Generate a SPARQL query for: {question}"}],
+    max_tokens=500
+)
+sparql = response.choices[0].message.content
+# Result: 0/7 passed - all wrapped in markdown
+```
+
+### 2. Vanilla OpenAI (With Schema) - 71.4% Accuracy
+
+```python
+from openai import OpenAI
+client = OpenAI(api_key=api_key)
+
+prompt = f"""You are a SPARQL query generator.
+
+{LUBM_SCHEMA}
+
+TYPE CONTRACT:
+- Input: natural language query
+- Output: raw SPARQL (NO markdown, NO code blocks, NO explanation)
+- Use ONLY predicates from the schema above
+
+Query: {question}
+
+Output raw SPARQL only:"""
+
+response = client.chat.completions.create(
+    model="gpt-4o",
+    messages=[{"role": "user", "content": prompt}],
+    max_tokens=500
+)
+sparql = response.choices[0].message.content
+# Result: 5/7 passed - schema prevents wrong predicates
+```
+
+### 3. LangChain (No Schema) - 0% Accuracy
+
+```python
+from langchain_openai import ChatOpenAI
+from langchain_core.prompts import PromptTemplate
+from langchain_core.output_parsers import StrOutputParser
+
+llm = ChatOpenAI(model="gpt-4o", api_key=api_key)
+parser = StrOutputParser()
+
+template = PromptTemplate(
+    input_variables=["question"],
+    template="Generate a SPARQL query for: {question}"
+)
+chain = template | llm | parser
+
+sparql = chain.invoke({"question": question})
+# Result: 0/7 passed - all wrapped in markdown
+```
+
+### 4. LangChain (With Schema) - 71.4% Accuracy
+
+```python
+from langchain_openai import ChatOpenAI
+from langchain_core.prompts import PromptTemplate
+from langchain_core.output_parsers import StrOutputParser
+
+llm = ChatOpenAI(model="gpt-4o", api_key=api_key)
+parser = StrOutputParser()
+
+template = PromptTemplate(
+    input_variables=["question", "schema"],
+    template="""You are a SPARQL query generator.
+
+{schema}
+
+TYPE CONTRACT:
+- Input: natural language query
+- Output: raw SPARQL (NO markdown, NO code blocks, NO explanation)
+- Use ONLY predicates from the schema above
+
+Query: {question}
+
+Output raw SPARQL only:"""
+)
+chain = template | llm | parser
+
+sparql = chain.invoke({"question": question, "schema": LUBM_SCHEMA})
+# Result: 5/7 passed - same as vanilla with schema
+```
+
+### 5. DSPy (No Schema) - 14.3% Accuracy
+
+```python
+import dspy
+from dspy import LM
+
+lm = LM("openai/gpt-4o")
+dspy.configure(lm=lm)
+
+class SPARQLGenerator(dspy.Signature):
+    """Generate SPARQL query from natural language."""
+    question = dspy.InputField(desc="Natural language question")
+    sparql = dspy.OutputField(desc="SPARQL query")
+
+generator = dspy.Predict(SPARQLGenerator)
+response = generator(question=question)
+sparql = response.sparql
+# Result: 1/7 passed - slightly better output formatting
+```
+
+### 6. DSPy (With Schema) - 71.4% Accuracy
+
+```python
+import dspy
+from dspy import LM
+
+lm = LM("openai/gpt-4o")
+dspy.configure(lm=lm)
+
+class SchemaSPARQLGenerator(dspy.Signature):
+    """Generate SPARQL query using the provided schema. Output raw SPARQL only."""
+    schema = dspy.InputField(desc="Database schema with classes and properties")
+    question = dspy.InputField(desc="Natural language question")
+    sparql = dspy.OutputField(desc="Raw SPARQL query (no markdown, no explanation)")
+
+generator = dspy.Predict(SchemaSPARQLGenerator)
+response = generator(schema=LUBM_SCHEMA, question=question)
+sparql = response.sparql
+# Result: 5/7 passed - same as others with schema
 ```
 
 ---