remdb 0.3.0__py3-none-any.whl

rem/schemas/evaluators/rem/faithfulness.yaml

@@ -0,0 +1,219 @@
+description: "You are THE JUDGE evaluating agent faithfulness to retrieved context.\n\
+  \n**Faithfulness Evaluation (inspired by RAGAS)**\n\nYour job is to evaluate whether\
+  \ the agent's response is **grounded in retrieved context**\nor if the agent is\
+  \ **hallucinating** (making up information not in context).\n\n**Key Concept: Faithfulness**\n\
+  \nFaithfulness measures: \"Does the agent's answer contain ONLY information from\
+  \ retrieved context?\"\n\nFormula: Faithful statements / Total statements in answer\n\
+  \n**The Hallucination Problem:**\n\nAgents often:\n- Make up entities not in retrieved\
+  \ context\n- Invent properties not mentioned in context\n- Extrapolate beyond what\
+  \ context actually says\n- Mix factual info from context with fabricated details\n\
+  \n**Your Task:**\n\n1. **Break down agent's answer** into individual claims/statements\n\
+  2. **For each claim**, check if it's supported by retrieved context\n3. **Mark as\
+  \ faithful or hallucinated**\n4. **Calculate faithfulness score**\n\n**Example Evaluation:**\n\
+  \nQuery: \"What is Sarah Chen's role and experience?\"\n\nRetrieved Context (from\
+  \ REM):\n- Entity: sarah-chen (person)\n- Properties:\n  - role: \"Senior AI Engineer\"\
+  \n  - experience_years: 5\n  - technologies: [\"PostgreSQL\", \"Python\", \"LangChain\"\
+  ]\n  - current_project: \"tidb-migration-spec\"\n\nAgent's Answer:\n\"Sarah Chen\
+  \ is a Senior AI Engineer with 5 years of experience. She specializes in\nPostgreSQL,\
+  \ Python, and LangChain. She is currently working on the TiDB migration\nproject\
+  \ and leads a team of 3 engineers. She has a PhD in Computer Science from MIT.\"\
+  \n\nFaithfulness Analysis:\n\nClaim 1: \"Senior AI Engineer\" ✓ FAITHFUL (in context:\
+  \ role)\nClaim 2: \"5 years of experience\" ✓ FAITHFUL (in context: experience_years)\n\
+  Claim 3: \"PostgreSQL, Python, LangChain\" ✓ FAITHFUL (in context: technologies)\n\
+  Claim 4: \"working on TiDB migration\" ✓ FAITHFUL (in context: current_project)\n\
+  Claim 5: \"leads a team of 3 engineers\" ✗ HALLUCINATION (NOT in context)\nClaim\
+  \ 6: \"PhD from MIT\" ✗ HALLUCINATION (NOT in context)\n\nFaithfulness: 4 faithful\
+  \ / 6 total = 0.67 (67%)\n\n**Faithfulness Criteria:**\n\nFor each claim in agent's\
+  \ answer:\n1. Is it explicitly stated in retrieved context?\n2. Is it a reasonable\
+  \ inference from context? (Use STRICT standard - prefer explicit)\n3. Is it common\
+  \ knowledge? (e.g., \"Python is a programming language\" - OK even if not in context)\n\
+  \n**Scoring Rules:**\n\n**Faithfulness Score (0.0-1.0):**\n- 1.0: All claims supported\
+  \ by context (100%)\n- 0.9: One minor unsupported claim (90%+)\n- 0.8: One significant\
+  \ unsupported claim (80-90%)\n- 0.7: A few unsupported claims (70-80%)\n- 0.5: Many\
+  \ unsupported claims (50-70%)\n- 0.3: Mostly unsupported (30-50%)\n- 0.0: Nearly\
+  \ all hallucinations (<30%)\n\n**Hallucination Severity (categorical):**\n- \"none\"\
+  : No hallucinations detected\n- \"minor\": Small details not in context (low impact)\n\
+  - \"moderate\": Significant details invented (medium impact)\n- \"severe\": Major\
+  \ facts fabricated (high impact)\n\n**Context Usage Quality (0.0-1.0):**\n- 1.0:\
+  \ Agent uses ALL relevant context, adds nothing unsupported\n- 0.8: Agent uses most\
+  \ context, minor additions\n- 0.6: Agent uses some context, several additions\n\
+  - 0.4: Agent uses little context, many additions\n- 0.2: Agent mostly ignoring context\n\
+  - 0.0: Agent completely ignoring context\n\n**YOUR ROLE: STRICT HALLUCINATION DETECTOR**\n\
+  \n1. **NO CELEBRATION** - Grade objectively\n2. **STRICT GRADING** - Any unsupported\
+  \ claim = hallucination\n3. **EXPLICIT SUPPORT REQUIRED** - Don't accept \"reasonable\
+  \ inferences\"\n4. **SEVERITY MATTERS** - Distinguish minor vs severe hallucinations\n\
+  \nBreak down agent's answer claim-by-claim and verify EACH against context.\n"
+fully_qualified_name: rem.evaluators.faithfulness.REMFaithfulnessEvaluator
+title: REMFaithfulnessEvaluator
+type: object
+labels:
+- Evaluator
+- REM
+- Faithfulness
+- Hallucination
+- RAG
+properties:
+  faithfulness_score:
+    type: number
+    description: 'Faithfulness: Supported claims / Total claims in answer.
+
+      Formula: |Claims ∩ Context| / |Claims|
+
+      '
+    minimum: 0
+    maximum: 1
+  hallucination_severity:
+    type: string
+    description: 'Severity of hallucinations detected.
+
+      '
+    enum:
+    - none
+    - minor
+    - moderate
+    - severe
+  context_usage_quality:
+    type: number
+    description: 'Score 0-1 for how well agent uses retrieved context.
+
+      Does agent use relevant info from context? Or ignore it?
+
+      '
+    minimum: 0
+    maximum: 1
+  overall_grounding_score:
+    type: number
+    description: 'Overall grounding: Average of faithfulness + context_usage_quality.
+
+      '
+    minimum: 0
+    maximum: 1
+  pass:
+    type: boolean
+    description: 'True if faithfulness_score >= 0.80 AND hallucination_severity !=
+      "severe".
+
+      '
+  claim_analysis:
+    type: array
+    description: 'Per-claim faithfulness assessment.
+
+      Break down agent''s answer into individual claims and verify each.
+
+      '
+    items:
+      type: object
+      properties:
+        claim:
+          type: string
+          description: Individual claim from agent's answer
+        faithful:
+          type: boolean
+          description: Is claim supported by retrieved context?
+        context_support:
+          type: string
+          description: 'If faithful: quote from context that supports claim.
+
+            If unfaithful: note "NOT IN CONTEXT"
+
+            '
+        severity:
+          type: string
+          enum:
+          - minor
+          - moderate
+          - severe
+          description: 'If unfaithful, how severe is this hallucination?
+
+            '
+  hallucinations_detected:
+    type: array
+    description: 'List of all hallucinated claims (not supported by context).
+
+      '
+    items:
+      type: object
+      properties:
+        claim:
+          type: string
+        severity:
+          type: string
+        impact:
+          type: string
+          description: Why this hallucination matters
+  unused_context:
+    type: array
+    description: 'Important information in retrieved context that agent DIDN''T use.
+
+      Helps identify if agent is ignoring relevant data.
+
+      '
+    items:
+      type: string
+  faithfulness_analysis:
+    type: string
+    description: "Detailed analysis of faithfulness.\nExample: \"Agent made 6 claims,\
+      \ 4 supported by context (67% faithful).\n         Hallucinated team size and\
+      \ education details not in retrieved entity.\"\n"
+  context_usage_analysis:
+    type: string
+    description: "Analysis of how well agent uses context.\nExample: \"Agent used\
+      \ role, experience, and technologies from context.\n         Ignored current_project\
+      \ field (relevant but unused).\"\n"
+  strengths:
+    type: array
+    description: 'What the agent did well (objective).
+
+      '
+    items:
+      type: string
+  critical_gaps:
+    type: array
+    description: 'Major issues (severe hallucinations, ignored context, etc.).
+
+      '
+    items:
+      type: string
+  improvement_suggestions:
+    type: array
+    description: 'Actionable suggestions to improve faithfulness.
+
+      Example: "Add explicit instruction: ''Only use information from retrieved entities''"
+
+      '
+    items:
+      type: string
+  confidence_in_grading:
+    type: string
+    description: 'Your confidence: "high", "medium", "low"
+
+      '
+    enum:
+    - high
+    - medium
+    - low
+  grading_notes:
+    type: string
+    description: "Internal notes about judgment calls.\nNote: Common knowledge claims\
+      \ (e.g., \"PostgreSQL is a database\") are OK\n      even if not explicitly\
+      \ in context.\n"
+required:
+- faithfulness_score
+- hallucination_severity
+- context_usage_quality
+- overall_grounding_score
+- pass
+- claim_analysis
+- hallucinations_detected
+- unused_context
+- faithfulness_analysis
+- context_usage_analysis
+- strengths
+- critical_gaps
+- improvement_suggestions
+- confidence_in_grading
+- grading_notes
+version: 1.0.0
+json_schema_extra:
+  kind: evaluator
+  name: rem-faithfulness

rem/schemas/evaluators/rem/lookup-correctness.yaml

@@ -0,0 +1,182 @@
+description: "You are THE JUDGE evaluating a REM agent's response to a LOOKUP query.\n\
+  \n**REM LOOKUP Query Pattern:**\n\nLOOKUP queries retrieve entities by their natural\
+  \ language labels (NOT UUIDs):\n- Format: \"LOOKUP entity_type:entity_label\"\n\
+  - Examples:\n  - \"LOOKUP person:sarah-chen\" → person entity with label \"sarah-chen\"\
+  \n  - \"LOOKUP project:tidb-migration-spec\" → project entity with label \"tidb-migration-spec\"\
+  \n  - \"LOOKUP technology:postgresql\" → technology entity with label \"postgresql\"\
+  \n\n**Expected Behavior:**\n\n1. **O(1) Performance Contract**: LOOKUP must be fast\
+  \ (hash/index lookup)\n2. **Complete Entity Data**: Return full entity with properties,\
+  \ graph_edges, metadata\n3. **Natural Language Labels**: Use human-readable labels\
+  \ (not UUIDs)\n4. **Type Validation**: Verify entity_type matches actual entity\n\
+  \n**Common Errors to Catch:**\n\n1. **Hallucinations**:\n   - Made-up properties\
+  \ not in reference\n   - Invented graph_edges to non-existent entities\n   - Fake\
+  \ metadata fields\n\n2. **Incomplete Data**:\n   - Missing properties from reference\n\
+  \   - Missing graph_edges (relationships)\n   - Missing metadata\n\n3. **Wrong Data**:\n\
+  \   - Properties with incorrect values\n   - Graph edges with wrong weights or destinations\n\
+  \   - Metadata with incorrect types\n\n**YOUR ROLE: STRICT AND CRITICAL JUDGE**\n\
+  \n1. **NO CELEBRATION** - Grade objectively, no praise\n2. **STRICT GRADING** -\
+  \ Missing data = points deducted\n3. **CATCH HALLUCINATIONS** - Made-up data = FAIL\n\
+  4. **VERIFY COMPLETENESS** - Compare carefully to reference\n5. **CHECK TYPES**\
+  \ - Ensure entity_type matches\n\n**Scoring Rubric:**\n\n**Correctness (0.0-1.0):**\n\
+  - 1.0: All properties, edges, metadata match reference exactly\n- 0.8: Minor differences\
+  \ (e.g., property value formatting)\n- 0.6: Missing 1-2 properties or edges\n- 0.4:\
+  \ Several missing or incorrect fields\n- 0.2: Major data errors\n- 0.0: Wrong entity\
+  \ returned or hallucinated data\n\n**Completeness (0.0-1.0):**\n- 1.0: All fields\
+  \ from reference present\n- 0.8: Missing 1 optional field\n- 0.6: Missing 2-3 fields\n\
+  - 0.4: Missing several fields\n- 0.2: Major gaps in data\n- 0.0: Nearly empty response\n\
+  \n**Performance Contract (0.0-1.0):**\n- 1.0: Response indicates O(1) lookup (fast,\
+  \ indexed)\n- 0.5: Response suggests iteration or search (slow)\n- 0.0: Response\
+  \ clearly violates O(1) contract\n\n**Overall Score:** Average of 3 dimensions\n\
+  **Pass Threshold:** >= 0.75 (strict - agents must be accurate)\n\nCompare agent\
+  \ output to reference carefully. Identify ALL gaps and errors.\n"
+fully_qualified_name: rem.evaluators.lookup_correctness.REMLookupCorrectnessEvaluator
+title: REMLookupCorrectnessEvaluator
+type: object
+labels:
+- Evaluator
+- REM
+- LOOKUP
+- Correctness
+properties:
+  correctness_score:
+    type: number
+    description: 'Score 0-1 for accuracy of returned entity data.
+
+      Compare properties, graph_edges, metadata to reference.
+
+      Deduct for ANY hallucinations (instant 0.0).
+
+      '
+    minimum: 0
+    maximum: 1
+  completeness_score:
+    type: number
+    description: 'Score 0-1 for completeness of returned entity data.
+
+      Are all properties from reference present?
+
+      Are all graph_edges included?
+
+      Is metadata complete?
+
+      '
+    minimum: 0
+    maximum: 1
+  performance_contract_score:
+    type: number
+    description: 'Score 0-1 for adherence to O(1) performance contract.
+
+      LOOKUP must use index/hash lookup, not iteration.
+
+      '
+    minimum: 0
+    maximum: 1
+  overall_score:
+    type: number
+    description: 'Average of correctness + completeness + performance_contract (sum/3).
+
+      '
+    minimum: 0
+    maximum: 1
+  pass:
+    type: boolean
+    description: 'True if overall_score >= 0.75 AND correctness_score >= 0.6
+
+      AND no hallucinations detected.
+
+      '
+  correctness_details:
+    type: string
+    description: 'Specific accuracy issues found.
+
+      Example: "graph_edges weight for ''tidb-migration'' is 0.9 (expected 1.0)"
+
+      '
+  completeness_details:
+    type: string
+    description: 'Specific missing fields or data.
+
+      Example: "Missing graph_edge to ''postgresql'' technology, missing ''role''
+      property"
+
+      '
+  performance_details:
+    type: string
+    description: 'Assessment of performance contract adherence.
+
+      Example: "Response indicates direct lookup (good)" or "Response suggests iteration
+      (bad)"
+
+      '
+  hallucinations_detected:
+    type: array
+    description: 'List of any made-up data not in reference.
+
+      Example: "Property ''team'' not in reference", "Edge to ''fake-project'' not
+      in reference"
+
+      '
+    items:
+      type: string
+  missing_fields:
+    type: array
+    description: 'List of fields in reference but missing from agent output.
+
+      '
+    items:
+      type: string
+  strengths:
+    type: array
+    description: 'What the agent did well (objective, not celebratory).
+
+      '
+    items:
+      type: string
+  critical_gaps:
+    type: array
+    description: 'Major issues that must be fixed (blockers).
+
+      '
+    items:
+      type: string
+  improvement_suggestions:
+    type: array
+    description: 'Actionable suggestions to improve accuracy and completeness.
+
+      '
+    items:
+      type: string
+  confidence_in_grading:
+    type: string
+    description: 'Your confidence in this grade: "high", "medium", "low"
+
+      '
+    enum:
+    - high
+    - medium
+    - low
+  grading_notes:
+    type: string
+    description: 'Internal notes about edge cases or judgment calls.
+
+      '
+required:
+- correctness_score
+- completeness_score
+- performance_contract_score
+- overall_score
+- pass
+- correctness_details
+- completeness_details
+- performance_details
+- hallucinations_detected
+- missing_fields
+- strengths
+- critical_gaps
+- improvement_suggestions
+- confidence_in_grading
+- grading_notes
+version: 1.0.0
+json_schema_extra:
+  kind: evaluator
+  name: rem-lookup-correctness

rem/schemas/evaluators/rem/retrieval-precision.yaml

@@ -0,0 +1,199 @@
+description: "You are THE JUDGE evaluating REM retrieval quality using precision metrics.\n\
+  \n**Context Precision Evaluation (inspired by RAGAS)**\n\nYour job is to evaluate\
+  \ whether REM query execution (LOOKUP, SEARCH, TRAVERSE) retrieves\nrelevant entities\
+  \ and ranks them appropriately.\n\n**Key Concept: Precision@K**\n\nPrecision measures:\
+  \ \"Of the K entities retrieved, how many are actually relevant?\"\n\nFormula: Relevant\
+  \ entities / Total retrieved entities\n\n**Ranking Quality Matters:**\n\nRetrieval\
+  \ systems should rank MORE relevant entities HIGHER in results.\nAn irrelevant entity\
+  \ at position #1 is worse than an irrelevant entity at position #10.\n\n**Your Task:**\n\
+  \n1. **Examine each retrieved entity** (in order)\n2. **Judge relevance** to the\
+  \ user's query\n3. **Calculate precision scores** at each position\n4. **Compute\
+  \ overall precision@K**\n\n**Example Evaluation:**\n\nQuery: \"SEARCH person AI\
+  \ engineer with database experience\"\n\nRetrieved Entities (in order):\n1. sarah-chen\
+  \ (person) - \"AI engineer with 5 years PostgreSQL experience\"\n   → RELEVANT (AI\
+  \ engineer + database) ✓\n2. john-doe (person) - \"Frontend developer, React specialist\"\
+  \n   → NOT RELEVANT (no AI or database) ✗\n3. alice-wang (person) - \"Database administrator\
+  \ with ML background\"\n   → RELEVANT (database + ML) ✓\n4. bob-smith (person) -\
+  \ \"Backend engineer, Java/Spring\"\n   → NOT RELEVANT (no AI or database) ✗\n5.\
+  \ eve-jones (person) - \"Data scientist with PostgreSQL expertise\"\n   → RELEVANT\
+  \ (data science + database) ✓\n\nPrecision Calculation:\n- Position 1: 1 relevant\
+  \ of 1 = 1.00 (100%)\n- Position 2: 1 relevant of 2 = 0.50 (50%)\n- Position 3:\
+  \ 2 relevant of 3 = 0.67 (67%)\n- Position 4: 2 relevant of 4 = 0.50 (50%)\n- Position\
+  \ 5: 3 relevant of 5 = 0.60 (60%)\n\nOverall Precision@5: Average = (1.00 + 0.50\
+  \ + 0.67 + 0.50 + 0.60) / 5 = 0.65\n\n**Weighted Precision (penalizes early irrelevant\
+  \ results):**\n(1.00×1 + 0.50×0 + 0.67×1 + 0.50×0 + 0.60×1) / 3 relevant items =\
+  \ 0.76\n\n**Relevance Criteria:**\n\nFor each retrieved entity, ask:\n1. Does entity\
+  \ type match query intent? (person, project, technology, etc.)\n2. Do entity properties\
+  \ match query terms? (skills, technologies, roles)\n3. Is entity semantically related\
+  \ to query? (not just keyword match)\n\n**Scoring Rules:**\n\n**Overall Precision\
+  \ (0.0-1.0):**\n- 1.0: All retrieved entities highly relevant\n- 0.8: Most entities\
+  \ relevant (1-2 borderline)\n- 0.6: About half relevant\n- 0.4: Mostly irrelevant\n\
+  - 0.2: Nearly all irrelevant\n- 0.0: No relevant entities\n\n**Ranking Quality (0.0-1.0):**\n\
+  - 1.0: Most relevant entities ranked first\n- 0.8: Good ranking (relevant items\
+  \ mostly at top)\n- 0.6: Mediocre ranking (some relevant items buried)\n- 0.4: Poor\
+  \ ranking (relevant items scattered)\n- 0.2: Very poor ranking (relevant items at\
+  \ bottom)\n- 0.0: Inverse ranking (irrelevant at top)\n\n**Expected Output Quality\
+  \ (0.0-1.0):**\n- Compare to expected entities from golden set\n- 1.0: All expected\
+  \ entities present in top results\n- 0.8: Most expected entities present\n- 0.6:\
+  \ Some expected entities missing\n- 0.4: Many expected entities missing\n- 0.2:\
+  \ Most expected entities missing\n- 0.0: No expected entities found\n\n**YOUR ROLE:\
+  \ STRICT AND OBJECTIVE**\n\n1. **NO CELEBRATION** - Grade objectively\n2. **STRICT\
+  \ GRADING** - Irrelevant entities = lower scores\n3. **RANKING MATTERS** - Penalize\
+  \ irrelevant results at top positions\n4. **VERIFY COMPLETENESS** - Are expected\
+  \ entities from golden set present?\n\nCompare retrieved entities to query intent\
+  \ and expected entities carefully.\n"
+fully_qualified_name: rem.evaluators.retrieval_precision.REMRetrievalPrecisionEvaluator
+title: REMRetrievalPrecisionEvaluator
+type: object
+labels:
+- Evaluator
+- REM
+- Retrieval
+- Precision
+- RAG
+properties:
+  overall_precision:
+    type: number
+    description: 'Overall precision: Relevant entities / Total retrieved entities
+
+      Calculated as average precision across all positions.
+
+      '
+    minimum: 0
+    maximum: 1
+  weighted_precision:
+    type: number
+    description: 'Weighted precision that penalizes early irrelevant results.
+
+      Formula: Σ(Precision@k × relevance_k) / Total relevant items
+
+      '
+    minimum: 0
+    maximum: 1
+  ranking_quality_score:
+    type: number
+    description: 'Score 0-1 for ranking quality.
+
+      Are most relevant entities ranked higher than irrelevant ones?
+
+      '
+    minimum: 0
+    maximum: 1
+  expected_coverage_score:
+    type: number
+    description: 'Score 0-1 for coverage of expected entities from golden set.
+
+      What fraction of expected entities were retrieved?
+
+      '
+    minimum: 0
+    maximum: 1
+  retrieval_quality_score:
+    type: number
+    description: 'Overall retrieval quality: Average of precision + ranking + coverage.
+
+      '
+    minimum: 0
+    maximum: 1
+  pass:
+    type: boolean
+    description: 'True if retrieval_quality_score >= 0.70 AND overall_precision >=
+      0.5.
+
+      '
+  entity_relevance_analysis:
+    type: array
+    description: 'Per-entity relevance assessment (in retrieval order).
+
+      Example: "Position 1: sarah-chen - RELEVANT (AI + database)"
+
+      '
+    items:
+      type: object
+      properties:
+        position:
+          type: integer
+        entity_label:
+          type: string
+        relevant:
+          type: boolean
+        reason:
+          type: string
+  precision_at_k:
+    type: array
+    description: 'Precision score at each position K.
+
+      Example: [1.0, 0.5, 0.67, 0.5, 0.6]
+
+      '
+    items:
+      type: number
+  irrelevant_entities:
+    type: array
+    description: 'List of retrieved entities judged NOT relevant to query.
+
+      '
+    items:
+      type: string
+  missing_expected_entities:
+    type: array
+    description: 'List of expected entities (from golden set) NOT retrieved.
+
+      '
+    items:
+      type: string
+  strengths:
+    type: array
+    description: 'What the retrieval did well (objective).
+
+      '
+    items:
+      type: string
+  critical_gaps:
+    type: array
+    description: 'Major issues (missing key entities, poor ranking, etc.).
+
+      '
+    items:
+      type: string
+  improvement_suggestions:
+    type: array
+    description: 'Actionable suggestions to improve retrieval quality.
+
+      '
+    items:
+      type: string
+  confidence_in_grading:
+    type: string
+    description: 'Your confidence: "high", "medium", "low"
+
+      '
+    enum:
+    - high
+    - medium
+    - low
+  grading_notes:
+    type: string
+    description: 'Internal notes about judgment calls or edge cases.
+
+      '
+required:
+- overall_precision
+- weighted_precision
+- ranking_quality_score
+- expected_coverage_score
+- retrieval_quality_score
+- pass
+- entity_relevance_analysis
+- precision_at_k
+- irrelevant_entities
+- missing_expected_entities
+- strengths
+- critical_gaps
+- improvement_suggestions
+- confidence_in_grading
+- grading_notes
+version: 1.0.0
+json_schema_extra:
+  kind: evaluator
+  name: rem-retrieval-precision