remdb 0.3.0__py3-none-any.whl

rem/schemas/evaluators/rem/retrieval-recall.yaml

@@ -0,0 +1,211 @@
+description: "You are THE JUDGE evaluating REM retrieval quality using recall metrics.\n\
+  \n**Context Recall Evaluation (inspired by RAGAS)**\n\nYour job is to evaluate whether\
+  \ REM query execution retrieves ALL relevant entities\nthat should be found for\
+  \ a given query.\n\n**Key Concept: Recall**\n\nRecall measures: \"Of all the relevant\
+  \ entities that SHOULD be retrieved, how many were actually retrieved?\"\n\nFormula:\
+  \ Retrieved relevant entities / Total relevant entities (from golden set)\n\n**The\
+  \ Coverage Problem:**\n\n- **High Precision, Low Recall**: Retrieved entities are\
+  \ relevant, but many are missing\n- **Low Precision, High Recall**: Retrieved many\
+  \ entities, but also grabbed irrelevant ones\n- **Goal**: High precision AND high\
+  \ recall\n\n**Your Task:**\n\n1. **Review expected entities** from golden set (what\
+  \ SHOULD be retrieved)\n2. **Review retrieved entities** from REM query\n3. **Calculate\
+  \ recall** - what fraction of expected entities were found?\n4. **Identify gaps**\
+  \ - which expected entities are missing?\n\n**Example Evaluation:**\n\nQuery: \"\
+  SEARCH person AI engineer with database experience\"\n\nExpected Entities (from\
+  \ golden set):\n- sarah-chen (person) - \"AI engineer with 5 years PostgreSQL experience\"\
+  \n- alice-wang (person) - \"Database administrator with ML background\"\n- eve-jones\
+  \ (person) - \"Data scientist with PostgreSQL expertise\"\n\nRetrieved Entities:\n\
+  - sarah-chen ✓ (found)\n- john-doe (not expected - false positive)\n- alice-wang\
+  \ ✓ (found)\n- bob-smith (not expected - false positive)\n\nRecall Calculation:\n\
+  - Found: sarah-chen, alice-wang (2 entities)\n- Expected: sarah-chen, alice-wang,\
+  \ eve-jones (3 entities)\n- Recall: 2/3 = 0.67 (67%)\n\nMissing: eve-jones (why?\
+  \ Bad embedding? Wrong query parsing?)\n\n**Recall Criteria:**\n\nFor each expected\
+  \ entity from golden set:\n1. Was it retrieved? (present in results)\n2. If not,\
+  \ why might it be missing?\n   - Embedding quality issue?\n   - Query parsing problem?\n\
+  \   - Entity missing from database?\n   - Ranking too low (buried beyond top-K)?\n\
+  \n**Scoring Rules:**\n\n**Recall Score (0.0-1.0):**\n- 1.0: All expected entities\
+  \ retrieved\n- 0.8: Missing 1 expected entity (90%+ recall)\n- 0.6: Missing 2-3\
+  \ expected entities (60-80% recall)\n- 0.4: Missing several expected entities (40-60%\
+  \ recall)\n- 0.2: Missing most expected entities (20-40% recall)\n- 0.0: Missing\
+  \ all expected entities (0% recall)\n\n**Ranking Depth (0.0-1.0):**\n- How deep\
+  \ in results are expected entities found?\n- 1.0: All expected entities in top 3\
+  \ positions\n- 0.8: All expected entities in top 5 positions\n- 0.6: All expected\
+  \ entities in top 10 positions\n- 0.4: Some expected entities beyond position 10\n\
+  - 0.2: Expected entities buried deep in results\n- 0.0: Expected entities not found\
+  \ at all\n\n**Coverage Quality (0.0-1.0):**\n- Balance between recall and precision\n\
+  - 1.0: High recall (>0.9) AND high precision (>0.8)\n- 0.8: Good recall (>0.7) AND\
+  \ good precision (>0.6)\n- 0.6: Moderate recall (>0.5) AND moderate precision (>0.5)\n\
+  - 0.4: Poor recall or precision\n- 0.2: Very poor recall and precision\n- 0.0: Nearly\
+  \ zero recall or precision\n\n**YOUR ROLE: STRICT AND DIAGNOSTIC**\n\n1. **NO CELEBRATION**\
+  \ - Grade objectively\n2. **STRICT GRADING** - Missing entities = lower recall\n\
+  3. **DIAGNOSE GAPS** - Why are expected entities missing?\n4. **RANKING DEPTH**\
+  \ - Are expected entities buried deep?\n\nCompare retrieved entities to expected\
+  \ golden set carefully.\nIdentify ALL missing entities and hypothesize why they're\
+  \ missing.\n"
+fully_qualified_name: rem.evaluators.retrieval_recall.REMRetrievalRecallEvaluator
+title: REMRetrievalRecallEvaluator
+type: object
+labels:
+- Evaluator
+- REM
+- Retrieval
+- Recall
+- RAG
+properties:
+  recall_score:
+    type: number
+    description: 'Recall: Retrieved expected entities / Total expected entities.
+
+      Formula: |Found ∩ Expected| / |Expected|
+
+      '
+    minimum: 0
+    maximum: 1
+  ranking_depth_score:
+    type: number
+    description: 'Score 0-1 for ranking depth of expected entities.
+
+      Are expected entities ranked high (top-K) or buried deep?
+
+      '
+    minimum: 0
+    maximum: 1
+  coverage_quality_score:
+    type: number
+    description: 'Balance between recall and precision.
+
+      Combines recall score with precision context.
+
+      '
+    minimum: 0
+    maximum: 1
+  retrieval_completeness_score:
+    type: number
+    description: 'Overall completeness: Average of recall + ranking_depth + coverage_quality.
+
+      '
+    minimum: 0
+    maximum: 1
+  pass:
+    type: boolean
+    description: 'True if recall_score >= 0.70 AND retrieval_completeness_score >=
+      0.70.
+
+      '
+  expected_entities_found:
+    type: array
+    description: 'List of expected entities that WERE retrieved.
+
+      Include position in results.
+
+      '
+    items:
+      type: object
+      properties:
+        entity_label:
+          type: string
+        position:
+          type: integer
+        notes:
+          type: string
+  missing_expected_entities:
+    type: array
+    description: 'List of expected entities that were NOT retrieved.
+
+      Include hypothesis for why missing.
+
+      '
+    items:
+      type: object
+      properties:
+        entity_label:
+          type: string
+        entity_type:
+          type: string
+        missing_reason_hypothesis:
+          type: string
+          description: "Why might this entity be missing?\nOptions: \"embedding_quality\"\
+            , \"query_parsing\", \"not_in_db\",\n         \"ranking_too_low\", \"\
+            type_filtering\", \"other\"\n"
+  recall_analysis:
+    type: string
+    description: "Detailed analysis of recall performance.\nExample: \"Found 3 of\
+      \ 4 expected entities (75% recall). Missing 'eve-jones'\n         likely due\
+      \ to poor embedding quality - her profile mentions 'data scientist'\n      \
+      \   not 'AI engineer' explicitly.\"\n"
+  ranking_depth_analysis:
+    type: string
+    description: "Analysis of where expected entities appear in results.\nExample:\
+      \ \"Expected entities ranked at positions 1, 3, 8. Position 8 is too deep\n\
+      \         for typical user queries (most users check top 5).\"\n"
+  false_positives:
+    type: array
+    description: 'Entities retrieved but NOT in expected set.
+
+      Note: Not necessarily wrong (golden set may be incomplete).
+
+      '
+    items:
+      type: string
+  strengths:
+    type: array
+    description: 'What the retrieval did well (objective).
+
+      '
+    items:
+      type: string
+  critical_gaps:
+    type: array
+    description: 'Major issues (missing key entities, poor coverage, etc.).
+
+      '
+    items:
+      type: string
+  improvement_suggestions:
+    type: array
+    description: 'Actionable suggestions to improve recall.
+
+      Example: "Improve embeddings for ''data scientist'' → ''AI engineer'' semantic
+      similarity"
+
+      '
+    items:
+      type: string
+  confidence_in_grading:
+    type: string
+    description: 'Your confidence: "high", "medium", "low"
+
+      Note: Low confidence if golden set may be incomplete
+
+      '
+    enum:
+    - high
+    - medium
+    - low
+  grading_notes:
+    type: string
+    description: 'Internal notes about judgment calls.
+
+      Note if golden set seems incomplete (retrieved valid entities not in expected).
+
+      '
+required:
+- recall_score
+- ranking_depth_score
+- coverage_quality_score
+- retrieval_completeness_score
+- pass
+- expected_entities_found
+- missing_expected_entities
+- recall_analysis
+- ranking_depth_analysis
+- false_positives
+- strengths
+- critical_gaps
+- improvement_suggestions
+- confidence_in_grading
+- grading_notes
+version: 1.0.0
+json_schema_extra:
+  kind: evaluator
+  name: rem-retrieval-recall

rem/schemas/evaluators/rem/search-correctness.yaml

@@ -0,0 +1,192 @@
+description: "You are THE JUDGE evaluating a REM agent's response to a SEARCH query.\n\
+  \n**REM SEARCH Query Pattern:**\n\nSEARCH queries perform semantic vector search\
+  \ across entity types:\n- Format: \"SEARCH entity_types query_text\"\n- Examples:\n\
+  \  - \"SEARCH person,project AI engineer with database experience\"\n  - \"SEARCH\
+  \ technology graph database with vector support\"\n  - \"SEARCH document migration\
+  \ planning guide\"\n\n**Expected Behavior:**\n\n1. **Semantic Ranking**: Results\
+  \ ranked by relevance to query\n2. **Type Filtering**: Only return requested entity\
+  \ types\n3. **Top-K Results**: Typically return 5-10 most relevant entities\n4.\
+  \ **Relevance Scores**: Include similarity scores when available\n5. **Entity Labels**:\
+  \ Use natural language labels (not UUIDs)\n\n**Common Errors to Catch:**\n\n1. **Wrong\
+  \ Entity Types**:\n   - Returns person when asked for project\n   - Mixes types\
+  \ when specific type requested\n\n2. **Poor Relevance**:\n   - Returns unrelated\
+  \ entities\n   - Missing obviously relevant entities from reference\n   - Poor ranking\
+  \ (irrelevant results ranked high)\n\n3. **Incomplete Results**:\n   - Returns fewer\
+  \ results than expected\n   - Missing key entities from reference golden set\n\n\
+  4. **Hallucinations**:\n   - Invented entities not in reference\n   - Made-up properties\
+  \ or metadata\n\n**YOUR ROLE: STRICT AND CRITICAL JUDGE**\n\n1. **NO CELEBRATION**\
+  \ - Grade objectively\n2. **STRICT GRADING** - Missing relevant results = points\
+  \ deducted\n3. **CATCH HALLUCINATIONS** - Made-up entities = FAIL\n4. **VERIFY RELEVANCE**\
+  \ - Are results actually related to query?\n5. **CHECK RANKING** - Are most relevant\
+  \ results ranked first?\n\n**Scoring Rubric:**\n\n**Relevance (0.0-1.0):**\n- 1.0:\
+  \ All results highly relevant to query\n- 0.8: Most results relevant, 1-2 borderline\n\
+  - 0.6: Several irrelevant results\n- 0.4: Many irrelevant results\n- 0.2: Mostly\
+  \ irrelevant\n- 0.0: Completely irrelevant or wrong types\n\n**Completeness (0.0-1.0):**\n\
+  - 1.0: All expected entities from reference present\n- 0.8: Missing 1 expected entity\n\
+  - 0.6: Missing 2-3 expected entities\n- 0.4: Missing several expected entities\n\
+  - 0.2: Missing most expected entities\n- 0.0: Missing all expected entities\n\n\
+  **Ranking Quality (0.0-1.0):**\n- 1.0: Most relevant results ranked first\n- 0.8:\
+  \ Good ranking with minor issues\n- 0.6: Mediocre ranking (some relevant buried)\n\
+  - 0.4: Poor ranking\n- 0.2: Very poor ranking\n- 0.0: No discernible ranking logic\n\
+  \n**Overall Score:** Average of 3 dimensions\n**Pass Threshold:** >= 0.70 (slightly\
+  \ lower than LOOKUP - semantic matching is harder)\n\nCompare agent results to reference\
+  \ golden set. Check relevance, completeness, ranking.\n"
+fully_qualified_name: rem.evaluators.search_correctness.REMSearchCorrectnessEvaluator
+title: REMSearchCorrectnessEvaluator
+type: object
+labels:
+- Evaluator
+- REM
+- SEARCH
+- Correctness
+- Semantic
+properties:
+  relevance_score:
+    type: number
+    description: 'Score 0-1 for relevance of returned entities to query.
+
+      Are results semantically related to query text?
+
+      Are entity types correct?
+
+      '
+    minimum: 0
+    maximum: 1
+  completeness_score:
+    type: number
+    description: 'Score 0-1 for completeness compared to reference.
+
+      Are all expected entities from reference present?
+
+      Are key relevant entities included?
+
+      '
+    minimum: 0
+    maximum: 1
+  ranking_quality_score:
+    type: number
+    description: 'Score 0-1 for ranking quality.
+
+      Are most relevant results ranked first?
+
+      Is there clear relevance ordering?
+
+      '
+    minimum: 0
+    maximum: 1
+  overall_score:
+    type: number
+    description: 'Average of relevance + completeness + ranking_quality (sum/3).
+
+      '
+    minimum: 0
+    maximum: 1
+  pass:
+    type: boolean
+    description: 'True if overall_score >= 0.70 AND relevance_score >= 0.5
+
+      AND no hallucinated entities detected.
+
+      '
+  relevance_details:
+    type: string
+    description: 'Assessment of result relevance to query.
+
+      Example: "First 3 results highly relevant, last 2 borderline"
+
+      '
+  completeness_details:
+    type: string
+    description: 'Comparison to reference golden set.
+
+      Example: "Missing ''sarah-chen'' person entity expected in top results"
+
+      '
+  ranking_details:
+    type: string
+    description: 'Assessment of ranking quality.
+
+      Example: "Most relevant entity ranked #3 (should be #1)"
+
+      '
+  hallucinations_detected:
+    type: array
+    description: 'List of entities in results but not in reference.
+
+      May not be errors (new data) but flag for review.
+
+      '
+    items:
+      type: string
+  missing_expected_entities:
+    type: array
+    description: 'List of entities in reference but missing from results.
+
+      '
+    items:
+      type: string
+  irrelevant_results:
+    type: array
+    description: 'List of results that don''t match query intent.
+
+      '
+    items:
+      type: string
+  strengths:
+    type: array
+    description: 'What the search did well (objective).
+
+      '
+    items:
+      type: string
+  critical_gaps:
+    type: array
+    description: 'Major issues (missing key results, wrong types, etc.).
+
+      '
+    items:
+      type: string
+  improvement_suggestions:
+    type: array
+    description: 'Actionable suggestions to improve search quality.
+
+      '
+    items:
+      type: string
+  confidence_in_grading:
+    type: string
+    description: 'Your confidence: "high", "medium", "low"
+
+      (Semantic matching is subjective - lower confidence OK)
+
+      '
+    enum:
+    - high
+    - medium
+    - low
+  grading_notes:
+    type: string
+    description: 'Internal notes about judgment calls or edge cases.
+
+      '
+required:
+- relevance_score
+- completeness_score
+- ranking_quality_score
+- overall_score
+- pass
+- relevance_details
+- completeness_details
+- ranking_details
+- hallucinations_detected
+- missing_expected_entities
+- irrelevant_results
+- strengths
+- critical_gaps
+- improvement_suggestions
+- confidence_in_grading
+- grading_notes
+version: 1.0.0
+json_schema_extra:
+  kind: evaluator
+  name: rem-search-correctness

rem/services/__init__.py

@@ -0,0 +1,16 @@
+"""
+REM Services
+
+Service layer for REM system operations:
+- PostgresService: PostgreSQL/CloudNativePG database operations
+- RemService: REM query execution and graph operations
+
+For file/S3 operations, use rem.services.fs instead:
+    from rem.services.fs import FS, S3Provider
+"""
+
+from .fs.service import FileSystemService
+from .postgres import PostgresService
+from .rem import RemService
+
+__all__ = ["PostgresService", "RemService", "FileSystemService"]