neotoma 0.3.0 → 0.3.2

package/docs/NEOTOMA_MANIFEST.md

@@ -1,152 +0,0 @@
----
-version: "1.0.0"
-last_updated: "2026-02-02"
-status: "canonical"
----
-
-# Neotoma Manifest
-## Scope
-This manifest defines the canonical architectural principles, product context, and non negotiable invariants for Neotoma. It does not define implementation details, APIs, or feature unit plans.
-
-## Purpose
-Provide a single root of truth for Neotoma. All documentation, code, and tests must remain consistent with this manifest.
-
-## Scope
-This document covers:
-- Truth Layer identity and boundaries
-- Determinism and immutability requirements
-- Schema first processing constraints
-- Entity and timeline doctrine
-- Product positioning and target users
-It does not cover:
-- Implementation details in services or UI
-- Release planning and execution schedules
-- Feature unit templates or manifests
-
-## Invariants
-1. Neotoma is a Truth Layer. It does not implement strategy or execution logic.
-2. Determinism is mandatory. The same input must produce the same output.
-3. Truth is immutable. Raw text, schema type, and extracted fields do not change after storage.
-4. Provenance is required. Every field must trace to a source and rule.
-5. Schema first processing is required. Extraction derives from schema definitions.
-6. Explicit user control is required. No background ingestion or hidden data collection.
-7. Privacy is required. No PII in logs and no hidden access paths.
-
-## Definitions
-- **Truth Layer**: The canonical storage and computation layer for structured personal data.
-- **Observation**: A source specific fact extracted from a payload or agent submission.
-- **Entity**: A canonical representation of a real world object with a deterministic ID.
-- **Snapshot**: A deterministic reducer output that represents current truth.
-- **Provenance**: Trace data that links fields to source, rule, and timestamp.
-
-## Architecture overview
-Neotoma sits below the Strategy and Execution layers.
-
-```mermaid
-flowchart TD
-    ExecutionLayer[ExecutionLayer] --> DomainEvents[DomainEvents]
-    DomainEvents --> Reducers[Reducers]
-    Reducers --> TruthLayer[TruthLayer]
-    StrategyLayer[StrategyLayer] --> TruthLayer
-```
-
-## 5. Core principles and constraints
-
-### 5.1 Determinism above all
-- Same input must yield the same output.
-- No random IDs, no nondeterministic ordering, no time based IDs.
-- LLM interpretation is auditable but not replay deterministic.
-
-### 5.2 Explicit user control
-- Only user provided data is ingested.
-- No background scanning or implicit ingestion.
-
-### 5.3 Truth Layer boundaries
-- No inference, prediction, or strategy logic in the Truth Layer.
-- No execution side effects. All mutations are via explicit actions.
-
-### 5.4 Truth is immutable
-- `raw_text`, `schema_type`, and `extracted_fields` do not change after storage.
-- Corrections create new observations with higher priority.
-
-### 5.5 Schema first processing
-- All extraction flows from schema definitions and rules.
-- No freeform or semantic inference for schema assignment.
-
-### 5.6 No synthetic data
-- No guessed fields, inferred entities, or hallucinated relationships.
-
-### 5.7 Full provenance
-- Every field must trace to a source and rule.
-
-## 12. Data model commitments
-Neotoma uses a Source, Observation, Entity, Snapshot model. See `docs/foundation/data_models.md` for details.
-
-### 12.2 Schema expectations
-- JSON schema definable
-- Versioned and backward compatible
-- Deterministic extraction rules
-- Additive evolution only
-
-## 14. Ingestion and extraction constraints
-- All ingestion is explicit.
-- All extraction is rule based.
-
-### 14.4 Multi pattern matching
-Schema detection must use two or more patterns per type. See `docs/subsystems/record_types.md`.
-
-## 15. Entity resolution doctrine
-- Entity IDs are hash based and deterministic.
-- Same canonical name yields the same entity ID.
-- Entities are never inferred beyond extracted fields.
-
-## 16. Timeline and events doctrine
-- Events derive only from extracted date fields.
-- Events use deterministic IDs and stable ordering.
-- Events are immutable after creation.
-
-## Product positioning
-Neotoma is a deterministic, privacy first, cross platform Truth Layer for AI tools. It is not a note app or a strategy engine. See `docs/foundation/product_positioning.md`.
-
-## Examples
-**Deterministic entity ID**
-```
-ent_{sha256(entity_type + ":" + normalized_name)}
-```
-
-**Explicit correction flow**
-```
-source -> observation(priority=100) -> correction(priority=1000) -> snapshot
-```
-
-## Testing requirements
-1. Determinism tests for IDs, reducers, and ordering.
-2. Idempotency tests for ingestion and corrections.
-3. Provenance tests that verify field level traceability.
-
-## Agent Instructions
-### When to Load This Document
-Load this manifest before any code or documentation work.
-
-### Required Co-Loaded Documents
-- `docs/context/index_rules.mdc`
-- `docs/foundation/core_identity.md`
-- `docs/foundation/philosophy.md`
-- `docs/architecture/determinism.md`
-
-### Constraints Agents Must Enforce
-1. No strategy or execution logic in the Truth Layer.
-2. No nondeterministic logic in core components.
-3. No schema mutation after assignment.
-4. All outputs trace to provenance.
-
-### Forbidden Patterns
-- LLM based field extraction in MVP
-- Semantic search in MVP
-- Implicit ingestion or hidden data collection
-
-### Validation Checklist
-- [ ] Truth Layer boundaries respected
-- [ ] No nondeterministic logic introduced
-- [ ] Schema first constraints enforced
-- [ ] Provenance preserved

package/docs/api/rest_api.md

@@ -1,466 +0,0 @@
-# REST API Reference
-
-_(HTTP Endpoints for Frontend and External Clients)_
-
-## Scope
-
-This document covers:
-
-- All HTTP endpoints and their parameters
-- Request/response formats
-- Authentication requirements
-- Error handling
-- Rate limiting
-  This document does NOT cover:
-- MCP tools (see `docs/specs/MCP_SPEC.md`)
-- WebSocket connections
-- Internal service-to-service communication
-
-## Base URL
-
-**Development:**
-
-```
-http://localhost:8080
-```
-
-**Production:**
-
-```
-https://neotoma.fly.dev
-```
-
-## Authentication
-
-All endpoints (except `/health` and `/openapi.yaml`) require Bearer token authentication:
-
-```http
-Authorization: Bearer <ACTIONS_BEARER_TOKEN>
-```
-
-**Token Configuration:**
-
-- Set via `ACTIONS_BEARER_TOKEN` environment variable
-- Use strong random token in production
-- Never commit tokens to git
-
-## MCP OAuth Endpoints
-
-OAuth endpoints for MCP client authentication (no bearer token required for public endpoints).
-
-### Initiate OAuth Flow
-
-Start OAuth authorization flow for MCP client.
-
-**Endpoint:** `POST /api/mcp/oauth/initiate`
-
-**Request:**
-
-```json
-{
-  "connection_id": "cursor-2025-01-21-abc123",
-  "client_name": "Cursor"
-}
-```
-
-**Parameters:**
-
-- `connection_id`: Unique identifier for this connection (required)
-- `client_name`: Name of MCP client (optional, e.g., "Cursor", "Claude Code")
-
-**Response:** `200 OK`
-
-```json
-{
-  "auth_url": "https://your-server/api/mcp/oauth/authorize?...",
-  "connection_id": "cursor-2025-01-21-abc123",
-  "expires_at": "2025-01-21T10:10:00Z"
-}
-```
-
-**Errors:**
-
-- `400`: Missing or invalid connection_id
-- `500`: Failed to create OAuth state
-
-### OAuth Callback
-
-Handles OAuth authorization callback.
-
-**Endpoint:** `GET /api/mcp/oauth/callback`
-
-**Query Parameters:**
-
-- `code`: Authorization code from OAuth provider (required)
-- `state`: OAuth state token (required)
-
-**Response:** Redirects to frontend with connection status
-
-- Success: `{FRONTEND_URL}/mcp-setup?connection_id={id}&status=success`
-- Error: `{FRONTEND_URL}/mcp-setup?status=error&error={message}`
-
-**Errors:**
-
-- `400`: Missing code or state
-- `500`: Token exchange or storage failed
-
-### Get Connection Status
-
-Check status of MCP OAuth connection.
-
-**Endpoint:** `GET /api/mcp/oauth/status`
-
-**Query Parameters:**
-
-- `connection_id`: Connection identifier (required)
-
-**Response:** `200 OK`
-
-```json
-{
-  "status": "active",
-  "connection_id": "cursor-2025-01-21-abc123"
-}
-```
-
-**Status Values:**
-
-- `pending`: OAuth flow initiated but not completed
-- `active`: Connection active and ready to use
-- `expired`: Connection revoked or OAuth flow timed out
-
-**Errors:**
-
-- `400`: Missing connection_id
-- `500`: Failed to query status
-
-### List MCP Connections
-
-List user's active MCP OAuth connections (authenticated).
-
-**Endpoint:** `GET /api/mcp/oauth/connections`
-
-**Headers:**
-
-```http
-Authorization: Bearer <SESSION_TOKEN>
-```
-
-**Response:** `200 OK`
-
-```json
-{
-  "connections": [
-    {
-      "connectionId": "cursor-2025-01-21-abc123",
-      "clientName": "Cursor",
-      "createdAt": "2025-01-21T10:00:00Z",
-      "lastUsedAt": "2025-01-21T11:30:00Z"
-    }
-  ]
-}
-```
-
-**Errors:**
-
-- `401`: Missing or invalid bearer token
-- `500`: Failed to fetch connections
-
-### Revoke MCP Connection
-
-Revoke an OAuth connection (authenticated).
-
-**Endpoint:** `DELETE /api/mcp/oauth/connections/:connection_id`
-
-**Headers:**
-
-```http
-Authorization: Bearer <SESSION_TOKEN>
-```
-
-**Response:** `200 OK`
-
-```json
-{
-  "success": true
-}
-```
-
-**Errors:**
-
-- `401`: Missing or invalid bearer token
-- `500`: Failed to revoke connection
-
-## Core Record Operations
-
-**Use MCP actions instead:**
-
-- `store()` - For [storing](#storing) [source](#source) (unstructured files or structured data)
-- `correct()` - For corrections
-- `merge_entities()` - For entity merging
-- `retrieve_entities()` - For querying entities
-- See `docs/specs/MCP_SPEC.md` for complete MCP action reference
-
-### Retrieve Records
-
-Query records with filters and search.
-**Endpoint:** `POST /retrieve_records`
-**Request:**
-
-```json
-{
-  "type": "note",
-  "properties": {
-    "tags": ["important"]
-  },
-  "search": ["meeting", "notes"],
-  "search_mode": "both",
-  "limit": 20,
-  "offset": 0,
-  "include_total_count": true
-}
-```
-
-**Response:** `200 OK`
-
-```json
-{
-  "records": [
-    {
-      "id": "uuid-1",
-      "type": "note",
-      "properties": { "text": "Meeting notes", "tags": ["important"] }
-    }
-  ],
-  "total_count": 42
-}
-```
-
-**Parameters:**
-
-- `type`: Filter by record type (optional)
-- `properties`: Filter by property values (optional, JSON object)
-- `search`: Array of search terms (optional)
-- `search_mode`: `"keyword"`, `"semantic"`, or `"both"` (default: `"both"`)
-- `limit`: Maximum results (default: 100, max: 1000)
-- `offset`: Pagination offset (default: 0)
-- `include_total_count`: Include total count in response (default: false)
-  **Errors:**
-- `400`: Invalid request (invalid search_mode)
-- `401`: Missing or invalid bearer token
-
-## File Operations
-
-### Upload File
-
-Upload a file and optionally create/update a record.
-**Endpoint:** `POST /upload_file`
-**Content-Type:** `multipart/form-data`
-**Form Fields:**
-
-- `file`: File to upload (required)
-- `record_id`: Existing record ID to attach file to (optional)
-- `properties`: JSON string of properties (optional)
-- `bucket`: Storage bucket name (optional, default: `files`)
-  **Request Example:**
-
-```bash
-curl -X POST http://localhost:8080/upload_file \
-  -H "Authorization: Bearer $TOKEN" \
-  -F "file=@document.pdf" \
-  -F "properties={\"title\":\"My Document\"}"
-```
-
-**Response:** `201 Created` or `200 OK`
-
-```json
-{
-  "id": "uuid-here",
-  "type": "pdf_document",
-  "properties": {
-    "title": "My Document",
-    "file_name": "document.pdf",
-    "file_size": 102400
-  },
-  "file_urls": ["/api/files/uuid/document.pdf"],
-  "created_at": "2024-01-01T12:00:00Z"
-}
-```
-
-**Behavior:**
-
-- If `record_id` provided: Attach file to existing record
-- If `record_id` omitted: Auto-analyze file and create new record
-- CSV files: Create one record per row
-  **Errors:**
-- `400`: Invalid file (too large, unsupported type)
-- `401`: Missing or invalid bearer token
-- `413`: File too large (>50MB)
-
-### Get File URL
-
-Generate signed URL for file access.
-**Endpoint:** `GET /get_file_url`
-**Query Parameters:**
-
-- `file_path`: Path to file in storage (required)
-- `expires_in`: URL expiration in seconds (optional, default: 3600)
-  **Request:**
-
-```http
-GET /get_file_url?file_path=files/uuid/document.pdf&expires_in=7200
-Authorization: Bearer <token>
-```
-
-**Response:** `200 OK`
-
-```json
-{
-  "url": "/api/files/uuid/document.pdf?token=...",
-  "expires_at": "2024-01-01T14:00:00Z"
-}
-```
-
-**Errors:**
-
-- `400`: Invalid request (missing `file_path`)
-- `404`: File not found
-- `401`: Missing or invalid bearer token
-
-## Utility Endpoints
-
-### Health Check
-
-Check if server is running.
-**Endpoint:** `GET /health`
-**Request:**
-
-```http
-GET /health
-```
-
-**Response:** `200 OK`
-
-```json
-{
-  "status": "ok",
-  "timestamp": "2024-01-01T12:00:00Z"
-}
-```
-
-**No authentication required.**
-
-### OpenAPI Specification
-
-Get OpenAPI spec for API documentation.
-**Endpoint:** `GET /openapi.yaml`
-**Request:**
-
-```http
-GET /openapi.yaml
-```
-
-**Response:** `200 OK`
-
-```yaml
-openapi: 3.0.0
-info:
-  title: Neotoma API
-  version: 1.0.0
-...
-```
-
-**No authentication required.**
-
-## Error Responses
-
-All errors follow this format:
-
-```json
-{
-  "error_code": "INGESTION_FILE_TOO_LARGE",
-  "message": "File exceeds maximum size of 50MB",
-  "details": {
-    "file_size": 52428800,
-    "max_size": 52428800
-  },
-  "trace_id": "trace-uuid",
-  "timestamp": "2024-01-01T12:00:00Z"
-}
-```
-
-**HTTP Status Codes:**
-
-- `200`: Success
-- `201`: Created
-- `400`: Bad Request (client error)
-- `401`: Unauthorized (missing/invalid token)
-- `403`: Forbidden (insufficient permissions)
-- `404`: Not Found
-- `409`: Conflict (duplicate record)
-- `413`: Payload Too Large
-- `429`: Too Many Requests (rate limited)
-- `500`: Internal Server Error
-- `503`: Service Unavailable
-  See `docs/reference/error_codes.md` for complete error code reference.
-
-## Rate Limiting
-
-**Current Limits (MVP):**
-
-- No explicit rate limiting (post-MVP feature)
-- Recommended: <100 requests/second per client
-  **Future Limits:**
-- Per-user rate limits
-- Per-endpoint rate limits
-- Rate limit headers in responses
-
-## Pagination
-
-Use `limit` and `offset` for pagination:
-
-```json
-{
-  "limit": 20,
-  "offset": 0
-}
-```
-
-**Best Practices:**
-
-- Use `limit` ≤ 100 for optimal performance
-- Use `include_total_count: true` for UI pagination
-- Implement cursor-based pagination (post-MVP)
-
-## Agent Instructions
-
-### When to Load This Document
-
-Load when:
-
-- Implementing frontend API calls
-- Integrating external clients
-- Debugging API issues
-- Understanding request/response formats
-
-### Required Co-Loaded Documents
-
-- `docs/specs/MCP_SPEC.md` — MCP tool equivalents
-- `docs/reference/error_codes.md` — Error code reference
-- `docs/subsystems/errors.md` — Error handling patterns
-
-### Constraints Agents Must Enforce
-
-1. **Always include Authorization header** — Except `/health` and `/openapi.yaml`
-2. **Validate request format** — Check required fields
-3. **Handle errors gracefully** — Check error_code, not just HTTP status
-4. **Respect rate limits** — Implement exponential backoff
-5. **Use pagination** — Don't fetch all records at once
-
-### Forbidden Patterns
-
-- Calling endpoints without authentication (except health/openapi)
-- Ignoring error responses
-- Fetching unlimited records (use pagination)
-- Hardcoding bearer tokens in frontend code
-- Bypassing rate limits

package/docs/architecture/agentic_portfolio_overview.md

@@ -1,99 +0,0 @@
-# Agentic Portfolio Overview
-## Purpose
-The **Agentic Portfolio** is an example instance of the **Strategy Layer** in Neotoma's layered architecture.
-It demonstrates how the Strategy Layer (General Strategy Engine) operates as pure cognition: reading world state, evaluating priorities and constraints, and outputting Decisions + Commands.
-The Agentic Portfolio encodes what _should_ happen to a user's financial position over time, in deterministic, inspectable, agent-operable form.
-It is the canonical representation of **intent**, **strategy**, **constraints**, and **allowable transformations** of a portfolio.
-The Agentic Portfolio is not an executor.
-It defines strategy-as-data and exposes it so that agents—including the Agentic Wallet—can compute compliant actions.
-## Role in Neotoma Architecture
-Neotoma = Truth Layer (event-sourced, reducer-driven)
-Strategy Layer = Pure Cognition (Agentic Portfolio is an example instance)
-Execution Layer = Pure Effect (Agentic Wallet is part of this layer)
-### Strategy Layer Responsibilities (Agentic Portfolio as Example)
-As part of the Strategy Layer, Agentic Portfolio:
-- **Reads current world state** from Neotoma (read-only)
-- **Evaluates priorities, constraints, risk, commitments, time, and financial posture**
-- **Plans what should happen next**
-- **Outputs Decisions + Commands** (intents)
-- **NEVER mutates truth or performs side effects**
-Strategy is pure cognition:
-- State in → Decisions out
-- Declarative, testable, deterministic
-### Core Responsibilities
-- Define **target allocations**
-- Encode **risk constraints** and **boundaries**
-- Define **permissible transformations** (rebalance, accumulate, reduce, roll, unwind)
-- Define **rebalance triggers** and deviation thresholds
-- Define **long-horizon strategy trajectories**
-- Define **scenario-driven rules** (e.g., max exposure to chain X when volatility > Y)
-- Provide deterministic, immutable portfolios that multiple agents can reason over identically
-- Evaluate multi-domain reasoning (financial flows, portfolio management, tax & entity logic)
-- Enforce constraint & value enforcement (budgets, time blocks, autonomy caps, risk tolerances)
-### Explicit "What Should Happen" Decision-Making
-The Agentic Portfolio answers:
-- What should my exposures be?
-- When should they shift?
-- Under what conditions does rebalancing occur?
-- How should allocations evolve as markets change?
-- What assets are allowable or forbidden?
-It does **not** decide _how_ to achieve these transitions on-chain (that's the Execution Layer's responsibility).
-It does **not** perform side effects or mutate truth (all truth updates flow through Domain Events → Reducers).
-## Data Model (High-Level)
-Core objects:
-- **Portfolio State**
-  - asset list
-  - quantities
-  - valuations
-  - metadata
-- **Strategy Graph**
-  - target allocations
-  - constraints
-  - optimization preferences
-- **Transformation Rules**
-  - rebalance rules
-  - drift thresholds
-  - allowable transformation classes
-- **Trajectory Plans**
-  - multi-step objectives
-  - long-horizon allocation envelopes
-All fields are deterministic, versioned, and validated by the Neotoma Truth Layer.
-## Relationship with Execution Layer (Agentic Wallet + Domain Agents)
-- The Agentic Portfolio (Strategy Layer) defines the _goal state_ and _rules_.
-- The Execution Layer (including Agentic Wallet) takes Commands and executes them via adapters.
-- The Execution Layer reads Commands from Strategy Layer; Strategy Layer never reads or depends on Execution Layer.
-- The Execution Layer emits Domain Events (e.g., TRADE_EXECUTED, PAYMENT_INITIATED) describing what happened.
-- Domain Events flow through Reducers → Updated world state in Neotoma.
-**Portfolio = Strategy Layer (pure cognition)**
-**Wallet = Execution Layer (pure effect)**
-### Event Flow
-```
-Agentic Portfolio (Strategy Layer)
-  ↓ Reads world state
-  ↓ Evaluates priorities/constraints
-  ↓ Outputs Decisions + Commands
-  ↓
-Execution Layer (Agentic Wallet + Domain Agents)
-  ↓ Takes Commands
-  ↓ Performs side effects via adapters
-  ↓ Emits Domain Events
-  ↓
-Neotoma (Truth Layer)
-  ↓ Reducers process Domain Events
-  ↓ Updated world state
-  ↓
-Next Strategy Tick
-```
-## Agent Instructions
-- Agents may query the strategy graph and compute next eligible transformations.
-- Agents must not mutate the portfolio directly—only propose transformations compliant with rules.
-- The portfolio state may be recomputed deterministically and compared against proposed changes.
-- All strategic outputs must remain inspectable and fully explainable.
-- **Critical:** Agentic Portfolio (Strategy Layer) MUST NOT perform side effects or mutate truth. All truth updates flow through Domain Events → Reducers.
-- Strategy Layer functions are pure: State in → Decisions out. No file I/O, no external API calls, no state mutations.
-## Example Use Cases
-- Long-term BTC accumulation with dynamic rebalance rules
-- Structured risk reduction strategies
-- Multi-asset yield optimization under stable constraints
-- RWA allocation boundaries and rebalance trajectories
-- Personalized risk envelopes encoded as deterministic rules