neotoma 0.3.0 → 0.3.2

package/docs/migration/llm_extraction_removal.md

@@ -1,387 +0,0 @@
-# LLM Extraction Removal — Migration to Rule-Based Extraction
-
-**⚠️ OUTDATED DOCUMENT:** This document describes a historical migration and is no longer accurate. The architecture has evolved (v0.2.0+) to allow AI interpretation for unstructured files via the interpretation service with full auditability and system-level idempotence. See `docs/architecture/determinism.md` for current extraction approach.
-
-## Purpose
-Documents the **historical removal of LLM-based extraction** from early prototypes and replacement with the interpretation service pattern that provides AI interpretation with auditability and idempotence guarantees.
-## Why LLM Extraction Was Initially Implemented
-### Original Rationale
-LLM extraction was initially implemented for:
-1. **Flexibility:** Handle diverse document formats without explicit rules
-2. **Field extraction accuracy:** LLM could "understand" context and extract fields intelligently
-3. **Rapid prototyping:** Faster initial implementation than building comprehensive regex libraries
-### Problems Discovered
-1. **Nondeterminism:** Same file → different extractions across runs (unacceptable per manifest)
-2. **Cost:** API costs for every file upload (unsustainable at scale)
-3. **Latency:** LLM API calls add 2-5s per document (poor UX)
-4. **Architectural violation:** Violates NEOTOMA_MANIFEST.md section 5.1 (Determinism Above All)
-5. **Testing impossibility:** Cannot write deterministic tests for LLM outputs
-6. **Trust erosion:** Users cannot trust extraction results if they change on re-upload
-## Why Rule-Based Extraction is Required for MVP
-### Architectural Alignment
-`docs/NEOTOMA_MANIFEST.md` section 5.1:
-> **Determinism Above All**
->
-> A given input MUST always produce identical output:
->
-> - Same raw_text → same extracted_fields
-> - Same entity name → same entity_id
-> - Same date field → same event_id
-> - Same query + same DB state → same search order
-> - Same file uploaded twice → same record (deduplicated)
->
-> **No randomness. No nondeterminism. No LLM extraction (MVP).**
-### MVP Success Criteria
-MVP requires:
-- **Determinism:** Same file uploaded 100 times → 100 identical extractions
-- **Testability:** Unit tests can verify extraction output
-- **Performance:** Fast, local extraction (no API calls)
-- **Cost:** Zero per-document API costs
-- **Trust:** Users can rely on consistent behavior
-### Post-MVP Path
-LLM extraction **may** be added post-MVP with:
-- Deterministic fallback for ambiguous cases
-- Caching to ensure same input → same output within session
-- Explicit user opt-in ("Use AI-assisted extraction?")
-- A/B testing to measure accuracy vs determinism tradeoffs
-**Current Approach (v0.2.0+):** AI interpretation for unstructured files via interpretation service (auditable, idempotent).
-## Migration Path
-### Phase 1: Remove LLM Code (FU-100)
-**Target Files:**
-- `src/services/file_analysis.ts` (primary extraction service)
-- Any other files calling `openai.chat.completions.create()`
-**Actions:**
-1. Remove all OpenAI client initialization
-2. Remove all `openai.chat.completions.create()` calls
-3. Remove LLM prompt templates
-4. Remove any LLM-dependent logic
-**Verification:**
-```bash
-# Must return zero results after removal
-grep -r "openai.chat.completions.create" src/services/
-grep -r "OpenAI" src/services/ | grep -v "// OpenAI API (removed)" # Comments OK
-```
-### Phase 2: Implement Multi-Pattern Schema Detection
-**Pattern:** Documents must match **2 or more patterns** for a given type to be classified as that type.
-**Rationale:** Multi-pattern matching reduces false positives while maintaining high accuracy.
-**Implementation Location:** `src/services/file_analysis.ts` (or dedicated schema detection service)
-**Example Implementation:**
-```typescript
-// Complete detection patterns from docs/subsystems/record_types.md section 7
-const SCHEMA_DETECTION_PATTERNS = {
-  invoice: [
-    /invoice\s*#?\s*:?\s*([A-Z0-9-]+)/i,
-    /bill\s*to:/i,
-    /amount\s*due:/i,
-    /invoice\s*date/i,
-    /payment\s*terms/i,
-  ],
-  receipt: [
-    /receipt/i,
-    /thank\s*you\s*for\s*your\s*purchase/i,
-    /items?\s*purchased/i,
-    /total\s*amount/i,
-    /payment\s*method/i,
-  ],
-  // ... (complete patterns in record_types.md)
-};
-function detectSchemaType(rawText: string): string {
-  const matchCounts: Record<string, number> = {};
-  // Count pattern matches for each type
-  for (const [type, patterns] of Object.entries(SCHEMA_DETECTION_PATTERNS)) {
-    matchCounts[type] = patterns.filter((pattern) =>
-      pattern.test(rawText)
-    ).length;
-  }
-  // Find types with 2+ matches
-  const candidates = Object.entries(matchCounts)
-    .filter(([_, count]) => count >= 2)
-    .sort(([_, countA], [__, countB]) => countB - countA);
-  // Return type with most matches, or fallback to 'document'
-  if (candidates.length > 0) {
-    return candidates[0][0];
-  }
-  return "document"; // Generic fallback
-}
-```
-**Testing:**
-```typescript
-describe("detectSchemaType", () => {
-  it("detects invoice with 2+ pattern matches", () => {
-    const text = `
-      INVOICE #INV-2024-001
-      Bill To: Customer Corp
-      Amount Due: $1,500.00
-    `;
-    // Run 100 times, verify determinism
-    for (let i = 0; i < 100; i++) {
-      expect(detectSchemaType(text)).toBe("invoice");
-    }
-  });
-  it("falls back to document when < 2 patterns match", () => {
-    const text = "This is a generic document.";
-    expect(detectSchemaType(text)).toBe("document");
-  });
-});
-```
-### Phase 3: Implement Rule-Based Field Extraction
-**Pattern:** Use regex and parsing for field extraction per schema type.
-**Implementation:** Per-type extraction functions (see `docs/subsystems/record_types.md` section 4)
-**Example: Invoice Field Extraction**
-```typescript
-interface InvoiceFields {
-  invoice_number: string;
-  amount: number;
-  currency: string;
-  date_issued: string; // ISO 8601
-  vendor_name?: string;
-  customer_name?: string;
-}
-function extractInvoiceFields(rawText: string): InvoiceFields {
-  const patterns = {
-    invoice_number: /invoice\s*#?\s*:?\s*([A-Z0-9-]+)/i,
-    amount: /(?:amount|total)[\s:]*\$?\s*([\d,]+\.?\d*)/i,
-    date_issued: /(?:date|issued)[\s:]*(\d{1,2}[-/]\d{1,2}[-/]\d{2,4})/i,
-    vendor_name: /(?:from|vendor|seller)[\s:]*([A-Za-z0-9\s&.,]+)/i,
-  };
-  const fields: Partial<InvoiceFields> = {};
-  // Extract invoice_number (required)
-  const invoiceMatch = rawText.match(patterns.invoice_number);
-  if (!invoiceMatch) {
-    throw new Error("Missing required field: invoice_number");
-  }
-  fields.invoice_number = invoiceMatch[1];
-  // Extract amount (required)
-  const amountMatch = rawText.match(patterns.amount);
-  if (!amountMatch) {
-    throw new Error("Missing required field: amount");
-  }
-  fields.amount = parseFloat(amountMatch[1].replace(/,/g, ""));
-  // Default currency (could be extracted with pattern)
-  fields.currency = "USD";
-  // Extract date (required)
-  const dateMatch = rawText.match(patterns.date_issued);
-  if (!dateMatch) {
-    throw new Error("Missing required field: date_issued");
-  }
-  fields.date_issued = parseDateToISO8601(dateMatch[1]);
-  // Extract vendor_name (optional)
-  const vendorMatch = rawText.match(patterns.vendor_name);
-  if (vendorMatch) {
-    fields.vendor_name = vendorMatch[1].trim();
-  }
-  return fields as InvoiceFields;
-}
-function parseDateToISO8601(dateString: string): string {
-  // Parse various date formats deterministically
-  // Return ISO 8601 format: YYYY-MM-DDTHH:MM:SSZ
-  // Implementation details omitted for brevity
-}
-```
-**Testing:**
-```typescript
-describe("extractInvoiceFields", () => {
-  const sampleInvoice = `
-    INVOICE #INV-2024-001
-    From: Acme Corp
-    Bill To: Customer Corp
-    Amount Due: $1,500.00
-    Invoice Date: 01/15/2024
-  `;
-  it("extracts all fields deterministically", () => {
-    const expected = {
-      invoice_number: "INV-2024-001",
-      amount: 1500.0,
-      currency: "USD",
-      date_issued: "2024-01-15T00:00:00Z",
-      vendor_name: "Acme Corp",
-    };
-    // Run 100 times, verify determinism
-    for (let i = 0; i < 100; i++) {
-      expect(extractInvoiceFields(sampleInvoice)).toEqual(expected);
-    }
-  });
-  it("throws error when required fields missing", () => {
-    const incomplete = "Just some text";
-    expect(() => extractInvoiceFields(incomplete)).toThrow(
-      "Missing required field"
-    );
-  });
-});
-```
-### Phase 4: Update Fallback Logic
-**Old Behavior (with LLM):**
-- LLM attempts to classify unknown documents
-- Custom types may be generated (e.g., "ResearchPaper", "Memo")
-**New Behavior (rule-based):**
-- Multi-pattern matching for known types
-- Fallback to `'document'` (generic) if < 2 patterns match
-- No custom types (only canonical types from `record_types.md`)
-**Implementation:**
-```typescript
-function normalizeRecordType(detectedType: string): string {
-  const CANONICAL_TYPES = [
-    "invoice",
-    "receipt",
-    "transaction",
-    "statement",
-    "account",
-    "note",
-    "document",
-    "message",
-    "task",
-    "project",
-    "event",
-    "contact",
-    "dataset",
-    "contract",
-    "travel_document",
-    "identity_document",
-  ];
-  if (CANONICAL_TYPES.includes(detectedType)) {
-    return detectedType;
-  }
-  // Fallback to generic document
-  return "document";
-}
-```
-### Phase 5: Comprehensive Testing
-**Determinism Tests (Critical):**
-```typescript
-describe("End-to-End Determinism", () => {
-  const testFiles = [
-    "./fixtures/invoice-sample-001.pdf",
-    "./fixtures/receipt-sample-001.jpg",
-    "./fixtures/contract-sample-001.pdf",
-  ];
-  for (const file of testFiles) {
-    it(`produces identical extraction for ${file} across 100 runs`, async () => {
-      const results = [];
-      for (let i = 0; i < 100; i++) {
-        const result = await ingestFile(file);
-        results.push(result);
-      }
-      // Verify all results identical
-      const firstResult = JSON.stringify(results[0]);
-      for (const result of results) {
-        expect(JSON.stringify(result)).toBe(firstResult);
-      }
-    });
-  }
-});
-```
-**Coverage Tests:**
-```typescript
-describe("Schema Detection Coverage", () => {
-  // Test all application types
-  const typeFixtures = {
-    invoice: "./fixtures/invoice-sample-001.pdf",
-    receipt: "./fixtures/receipt-sample-001.jpg",
-    transaction: "./fixtures/transaction-sample-001.pdf",
-    // ... (complete list)
-  };
-  for (const [type, fixture] of Object.entries(typeFixtures)) {
-    it(`correctly detects type: ${type}`, async () => {
-      const result = await ingestFile(fixture);
-      expect(result.type).toBe(type);
-    });
-  }
-  it("falls back to document for unrecognized files", async () => {
-    const result = await ingestFile("./fixtures/unknown-document.pdf");
-    expect(result.type).toBe("document");
-  });
-});
-```
-## Migration Checklist
-**Pre-Migration:**
-- [ ] Review `docs/subsystems/record_types.md` (complete type catalog)
-- [ ] Review `docs/subsystems/ingestion/ingestion.md` section 5 (detection patterns)
-- [ ] Create test fixtures for all application types
-- [ ] Document expected extraction output per fixture
-**Implementation:**
-- [ ] Remove all LLM code (`grep -r "openai.chat" src/` returns zero)
-- [ ] Implement multi-pattern schema detection (2+ patterns required)
-- [ ] Implement rule-based field extraction per type (all types from `record_types.md`)
-- [ ] Update fallback logic (unrecognized → `'document'`)
-- [ ] Remove custom type sanitization
-**Testing:**
-- [ ] Determinism tests pass (100 runs → 100 identical results)
-- [ ] Coverage tests pass (all application types detected correctly)
-- [ ] Fallback tests pass (unrecognized → `'document'`)
-- [ ] Performance tests pass (extraction < 10s P95)
-- [ ] Integration tests pass (end-to-end ingestion pipeline)
-**Verification:**
-- [ ] Zero LLM API calls in codebase
-- [ ] All extraction outputs deterministic
-- [ ] All types match `record_types.md` catalog
-- [ ] MVP launch criteria met
-## Rollback Plan (If Needed)
-**Note:** Rollback **NOT RECOMMENDED** — rule-based extraction is required for MVP.
-However, if critical issues arise:
-1. **Preserve LLM code in Git history:**
-   - Tag commit before LLM removal: `git tag pre-llm-removal`
-   - LLM code can be restored from history if needed
-2. **Revert Process:**
-   ```bash
-   git revert <commit-hash-of-llm-removal>
-   npm test
-   ```
-3. **Post-Revert Actions:**
-   - Add deterministic caching to LLM calls (cache results by content hash)
-   - Add explicit user opt-in for LLM extraction
-   - Mark as experimental/post-MVP feature
-   - Continue working on rule-based extraction for MVP path
-**Recommendation:** Do NOT rollback. Fix forward with improved rule-based patterns.
-## Post-MVP: Hybrid Approach
-After MVP launch, consider **hybrid approach**:
-1. **Primary:** Rule-based extraction (default, deterministic)
-2. **Fallback:** LLM-assisted extraction for ambiguous cases (user opt-in)
-3. **Caching:** Cache LLM results by content hash for determinism
-4. **Validation:** Compare LLM output vs rule-based output, flag discrepancies
-5. **User Control:** Allow users to choose extraction method per document type
-**Criteria for LLM Re-Introduction:**
-- Deterministic caching implemented (content hash → LLM result mapping)
-- User opt-in required (no surprise nondeterminism)
-- A/B testing shows accuracy improvement > 10% vs rule-based
-- Cost model validated (LLM cost per document acceptable at scale)
-## References
-- `docs/NEOTOMA_MANIFEST.md` section 5.1 — Determinism Above All
-- `docs/subsystems/record_types.md` — Complete type catalog, extraction rules
-- `docs/subsystems/ingestion/ingestion.md` section 5 — Schema detection patterns
-- `docs/specs/MVP_EXECUTION_PLAN.md` FU-100 — Implementation plan
-- `docs/architecture/determinism.md` — Determinism requirements
-## Agent Instructions
-### When to Load This Document
-Load `docs/migration/llm_extraction_removal.md` when:
-- Implementing FU-100 (LLM removal)
-- Modifying schema detection logic
-- Modifying field extraction logic
-- Debugging extraction nondeterminism
-- Planning post-MVP LLM re-introduction
-### Required Co-Loaded Documents
-- `docs/NEOTOMA_MANIFEST.md` (determinism requirements)
-- `docs/subsystems/record_types.md` (type catalog, extraction rules)
-- `docs/subsystems/ingestion/ingestion.md` (ingestion pipeline)
-- `docs/specs/MVP_EXECUTION_PLAN.md` (FU-100 details)
-### Constraints Agents Must Enforce
-1. **Zero LLM calls in MVP:** Code MUST NOT call `openai.chat.completions.create()` or equivalent
-2. **Determinism required:** Same file → same extraction, always (100 runs → 100 identical results)
-3. **Multi-pattern matching:** Schema detection MUST use 2+ pattern matches for non-fallback types
-4. **Canonical types only:** Extracted types MUST match `record_types.md` application types
-5. **Fallback correctness:** Unrecognized documents → `type = 'document'` (not custom types)
-### Forbidden Patterns
-- Using LLM for extraction in MVP
-- Adding nondeterministic logic to extraction
-- Generating custom types not in `record_types.md`
-- Skipping determinism tests (100-run tests required)
-- Caching LLM results without explicit user opt-in (post-MVP only)
-### Validation Checklist
-- [ ] Zero LLM calls in codebase (`grep -r "openai.chat"` returns zero)
-- [ ] Determinism tests pass (100 runs → identical results)
-- [ ] Multi-pattern matching implemented (2+ patterns required)
-- [ ] Canonical types only (all match `record_types.md`)
-- [ ] Fallback to `document` for unrecognized types
-- [ ] Performance < 10s P95 (same as or better than LLM version)
-- [ ] All application types have test fixtures and expected outputs
-- [ ] MVP launch criteria met
-**END OF MIGRATION GUIDE**

package/docs/migration/llm_interpretation_docs_update.md

@@ -1,130 +0,0 @@
-# LLM Interpretation Documentation Update
-
-**Date:** 2026-01-19  
-**Purpose:** Fix documentation inconsistencies regarding LLM interpretation for unstructured data storage
-
-## Summary
-
-Updated documentation to reflect that AI interpretation is allowed for unstructured files (PDFs, images) with full auditability and system-level idempotence. Removed outdated "rule-based only" constraints that conflicted with the current architecture (v0.2.0+).
-
-## Key Changes
-
-### Architectural Clarification
-
-**Previous (Incorrect):** "No LLM extraction (MVP constraint; rule-based only)"
-
-**Current (Correct):** 
-- AI interpretation for unstructured files (PDFs, images) via interpretation service
-- Auditable (interpretation config logged: model, temperature, prompt_hash, code_version)
-- System-level idempotence enforced (canonicalization + hashing ensures same source + config → same final state)
-- Rule-based extraction available for structured data from agents (deterministic)
-
-### Distinction: Replay Determinism vs. Idempotence
-
-- **Replay Determinism:** Same input → identical output (byte-for-byte) — NOT achievable with LLMs
-- **Idempotence:** Same operation → same final state (no duplicates) — ACHIEVED via post-processing
-- AI interpretation is **idempotent** but not **replay-deterministic**
-- See `docs/architecture/determinism.md` Section 1.4 for details
-
-## Files Updated
-
-### Core Specifications
-1. **`docs/specs/MVP_OVERVIEW.md`**
-   - Updated extraction approach description
-   - Changed "Rule-based extraction only" to "AI interpretation for unstructured files"
-   - Updated key differentiators section
-   - Removed "LLM-based extraction" from "Not in MVP" list
-
-2. **`docs/subsystems/ingestion/ingestion.md`**
-   - Updated Section 5.1: Changed "MVP Extraction Approach: Rule-Based Only" to current architecture
-   - Added references to interpretation service
-   - Updated constraints and validation checklist
-   - Clarified rule-based patterns are for structured data
-
-3. **`docs/foundation/philosophy.md`**
-   - Removed "No LLM extraction (MVP constraint)" from MUST NOT list
-   - Added "No replay-deterministic claims for AI interpretation"
-   - Updated to reflect auditability requirement
-
-4. **`docs/foundation/agent_instructions_rules.mdc`**
-   - Updated absolute constraints (Section 23.2)
-   - Updated code generation rules (Section 23.4)
-   - Updated forbidden patterns
-   - Updated validation checklist
-   - Added AI interpretation requirements with audit trail
-
-5. **`.cursor/rules/foundation_agent_instructions_rules.mdc`**
-   - Synced with updated source file
-
-### MVP Documentation
-6. **`docs/specs/MVP_FEATURE_UNITS.md`**
-   - Updated FU-100 extraction approach
-   - Changed constraints to require interpretation service
-
-7. **`docs/specs/FUNCTIONAL_REQUIREMENTS.md`**
-   - Updated extraction requirement
-   - Changed validation checklist
-
-8. **`docs/releases/v1.0.0/release_plan.md`**
-   - Updated technical requirements
-   - Replaced "no LLM extraction" with "AI interpretation with auditability"
-
-9. **`docs/releases/v1.0.0/acceptance_criteria.md`**
-   - Updated Truth Layer criteria
-   - Changed validation from "no OpenAI calls" to "interpretation config logged"
-
-### Supporting Documentation
-10. **`docs/subsystems/record_types.md`**
-    - Updated extraction approach references
-    - Changed to "idempotent extraction"
-
-11. **`docs/migration/llm_extraction_removal.md`**
-    - Added prominent warning that document is outdated
-    - Clarified this describes historical migration
-    - Referenced current architecture docs
-
-## What Was NOT Changed
-
-- Architecture documents (`docs/architecture/determinism.md`, `docs/releases/v0.2.0/release_plan.md`) already correctly describe AI interpretation
-- Implementation code (`src/services/interpretation.ts`, `src/services/llm_extraction.ts`) already implements AI interpretation correctly
-- MCP spec (`docs/specs/MCP_SPEC.md`) already correctly shows `interpret: true` parameter
-
-## Remaining Inconsistencies (Low Priority)
-
-The following files still contain "rule-based only" or "no LLM" references but are lower priority:
-
-- v0.1.0 release docs (historical, not updated)
-- v0.2.3 schema extension docs (note AI interpretation not used for agent-created entities)
-- Schema expansion docs (notes pattern detection is non-LLM)
-- Implementation overwrite guide (may be obsolete)
-- Various test/compliance reports (historical snapshots)
-
-These can be updated if they cause confusion, but are less critical since they're either:
-- Historical release documentation
-- Specific edge cases (agent-created data, schema expansion)
-- Archived compliance reports
-
-## Validation
-
-To verify the updates are correct:
-
-1. Read `docs/architecture/determinism.md` Section 1.2-1.4 for canonical definition
-2. Read `docs/releases/v0.2.0/release_plan.md` Section 5 for interpretation approach
-3. Check `src/services/interpretation.ts` for implementation
-4. Review `docs/specs/MCP_SPEC.md` for `interpret` parameter usage
-
-## References
-
-- **Authoritative Source:** `docs/architecture/determinism.md` — Complete determinism doctrine
-- **Implementation:** `src/services/interpretation.ts` — Interpretation service with LLM
-- **Architecture:** `docs/architecture/sources_first_ingestion_final.md` — Sources-first with interpretation runs
-- **Release Context:** `docs/releases/v0.2.0/release_plan.md` — v0.2.0 introduced interpretation service
-
-## Next Steps
-
-If further documentation inconsistencies are discovered:
-1. Search for "rule-based only", "no LLM", "LLM extraction" in docs
-2. Check if reference is to:
-   - **Historical constraint** → Add note referencing current architecture
-   - **Current requirement** → Update to reflect AI interpretation with auditability
-   - **Edge case** (agent-created data, schema expansion) → Leave as-is or clarify scope

package/docs/observability/logging.md

@@ -1,50 +0,0 @@
-# Neotoma Logging Standard
-*(Log Levels, Schema, and PII Restrictions)*
-## Log Levels
-| Level | When to Use |
-|-------|-------------|
-| `error` | Failures requiring attention |
-| `warn` | Degraded state, retry attempted |
-| `info` | Normal operations (ingestion, search) |
-| `debug` | Detailed debugging (dev only) |
-## Log Schema
-```typescript
-interface LogEntry {
-  level: 'error' | 'warn' | 'info' | 'debug';
-  message: string;
-  timestamp: string;                   // ISO 8601
-  trace_id?: string;
-  fields: Record<string, any>;         // NO PII
-}
-```
-**Example:**
-```json
-{
-  "level": "info",
-  "message": "Record uploaded",
-  "timestamp": "2024-01-15T10:30:00Z",
-  "trace_id": "abc123",
-  "fields": {
-    "record_id": "rec_xyz",
-    "type": "FinancialRecord",
-    "file_size_bytes": 1048576
-  }
-}
-```
-## PII Restrictions
-**MUST NOT Log:**
-- PII from `properties` (names, SSN, addresses)
-- Full `raw_text`
-- Auth tokens
-**MAY Log:**
-- Record IDs
-- Schema types
-- Error codes
-- Performance metrics
-## Agent Instructions
-Load when adding logging or debugging.
-Required co-loaded: `docs/subsystems/privacy.md`
-Constraints:
-- MUST NOT log PII
-- MUST include trace_id
-- MUST use structured logging

package/docs/observability/metrics_standard.md

@@ -1,38 +0,0 @@
-# Neotoma Metrics Standard
-*(Metrics Naming, Types, and Collection Rules)*
-## Metric Naming
-**Pattern:** `neotoma_{subsystem}_{operation}_{unit}`
-**Examples:**
-- `neotoma_record_upload_total` (counter)
-- `neotoma_record_upload_duration_ms` (histogram)
-- `neotoma_search_results_count` (histogram)
-## Metric Types
-### Counter
-**When:** Monotonically increasing values
-**Examples:**
-- `neotoma_record_upload_total{status="success"}`
-- `neotoma_extraction_errors_total{error_code="OCR_FAILED"}`
-### Gauge
-**When:** Values that go up and down
-**Examples:**
-- `neotoma_active_ingestions` (current active)
-- `neotoma_db_connection_pool_size`
-### Histogram
-**When:** Distribution of values
-**Examples:**
-- `neotoma_record_upload_duration_ms` (buckets: 100, 500, 1000, 5000)
-- `neotoma_search_duration_ms`
-## Required Metrics
-| Metric | Type | Labels | Description |
-|--------|------|--------|-------------|
-| `neotoma_record_upload_total` | Counter | `status` | Total uploads |
-| `neotoma_record_upload_duration_ms` | Histogram | - | Upload duration |
-| `neotoma_extraction_errors_total` | Counter | `error_code` | Extraction errors |
-| `neotoma_search_duration_ms` | Histogram | - | Search duration |
-## Agent Instructions
-Load when adding metrics or planning observability for Feature Units.
-Required co-loaded: `docs/subsystems/events.md`, `docs/observability/logging.md`
-Constraints:
-- MUST use standard naming pattern
-- MUST NOT include PII in labels
-- MUST document metric meaning

package/docs/observability/tracing.md

@@ -1,41 +0,0 @@
-# Neotoma Tracing Standard
-*(Distributed Tracing Spans and Context Propagation)*
-## Span Naming
-**Pattern:** `{subsystem}.{operation}`
-**Examples:**
-- `ingestion.ingest_file`
-- `extraction.extract_fields`
-- `search.execute_query`
-## Span Attributes
-```typescript
-{
-  "span_name": "ingestion.ingest_file",
-  "start_time": "2024-01-15T10:30:00.000Z",
-  "duration_ms": 1234,
-  "attributes": {
-    "file_size_bytes": 1048576,
-    "schema_type": "FinancialRecord",
-    "record_id": "rec_xyz"
-  }
-}
-```
-## Trace Propagation
-**Pattern:** `trace_id` passed through all layers.
-```typescript
-async function ingestFile(file: File, traceId: string): Promise<Record> {
-  const span = startSpan('ingestion.ingest_file', { trace_id: traceId });
-  
-  try {
-    // ... ingestion logic ...
-    return record;
-  } finally {
-    span.end();
-  }
-}
-```
-## Agent Instructions
-Load when adding tracing or debugging distributed flows.
-Required co-loaded: `docs/observability/logging.md`
-Constraints:
-- MUST propagate trace_id
-- MUST NOT include PII in attributes

package/docs/operations/debugging/README.md

@@ -1,11 +0,0 @@
-# Debugging and Error Management
-
-This directory contains documentation for debugging and error management in the Neotoma project.
-
-## Documentation
-
-- **[Error Debug Instructions](error_debug_instructions.md)** - Workflow for processing errors in the error queue
-
-## Related
-
-- **Operations**: `docs/operations/troubleshooting.md` - General troubleshooting guide