proofnest 0.2.0__py3-none-any.whl → 0.2.1__py3-none-any.whl

.planning/research/ARCHITECTURE.md

@@ -1,585 +0,0 @@
-# Architecture Patterns: PROOFNEST Trust/Proof Infrastructure
-
-**Domain:** Trust infrastructure for AI decisions
-**Researched:** 2026-02-03
-**Focus:** Integration patterns for existing Python+Go architecture
-
----
-
-## Executive Summary
-
-PROOFNEST has a solid dual-layer architecture: **Python Core** (FastAPI REST) for proof primitives and **Go Blockchain** (HotStuff-2 BFT) for immutable anchoring. The integration challenge is not redesign but **bridging** these layers for:
-
-1. Enterprise API patterns (multi-tenant, rate-limited)
-2. AI agent authentication (machine-to-machine, delegation)
-3. Real-time Hub (WebSocket, event-driven feeds)
-4. Scaling proof verification (batch, Merkle trees)
-
-**Key Recommendation:** Build an **API Gateway layer** between clients and the Python Core, not a replacement.
-
----
-
-## Existing Architecture (DO NOT REDESIGN)
-
-### Current Component Map
-
-```
-                                   CLIENTS
-                                      |
-                    +----------------+----------------+
-                    |                                 |
-              [Enterprise]                    [AI-Human Hub]
-                    |                                 |
-                    v                                 v
-+================================================================+
-|                      API GATEWAY (NEW)                          |
-|  - Multi-tenant routing                                         |
-|  - Rate limiting per tenant                                     |
-|  - AI agent authentication                                      |
-|  - WebSocket upgrade for Hub                                    |
-+================================================================+
-                    |
-                    v
-+================================================================+
-|              PYTHON CORE (proofnest_core)                       |
-|  Port: 8200                                                     |
-|  - FastAPI REST API                                             |
-|  - PROOF primitive (claim + evidence + identity + signature)    |
-|  - DID:PN registry with key rotation                            |
-|  - Consensus engine (in-memory validators)                      |
-|  - SQLite persistence                                           |
-|  - Bitcoin anchor service (mock)                                |
-|  - Blockchain client (JSON-RPC to Go chain)                     |
-+================================================================+
-                    |
-                    v
-+================================================================+
-|              GO BLOCKCHAIN (proofnest_v2/chain)                 |
-|  Port: 26657 (JSON-RPC)                                         |
-|  - HotStuff-2 BFT consensus (~36K LOC)                          |
-|  - Dilithium3 post-quantum signatures                           |
-|  - libp2p P2P with Kademlia DHT                                 |
-|  - TxType.PROOF for proof submissions                           |
-|  - ~2s block time, ~4s finality                                 |
-|  - Go SDK for direct integration                                |
-+================================================================+
-```
-
-### Current Data Flow
-
-```
-1. PROOF Creation:
-   Client -> POST /v1/proofs -> Python Core -> SQLite
-                                     |
-                                     v (if consensus)
-                              Validators -> Vote -> QuorumCertificate
-                                     |
-                                     v (if anchoring)
-                              Go Blockchain (TxType.PROOF)
-
-2. PROOF Verification:
-   Client -> GET /v1/proofs/{id}/verify -> Python Core
-                                                |
-                                                v
-                                         Go Blockchain (verify_proof_on_chain)
-```
-
-### Current Authentication
-
-- **X-API-Key header** or **api_key query param**
-- Public endpoints: `/`, `/health`, `/v1/proofs/{id}`, `/v1/proofs/{id}/verify`, `/v1/chain/status`
-- Protected endpoints: All POST operations (create identity, proof, validator, vote)
-
----
-
-## Integration Architecture Recommendations
-
-### Layer 1: API Gateway (HIGH PRIORITY)
-
-**Why:** Current Python Core lacks multi-tenancy, rate limiting, and AI-specific auth flows.
-
-**Recommendation:** Deploy **Kong Gateway** or **Traefik** in front of Python Core.
-
-| Pattern | Implementation | Rationale |
-|---------|----------------|-----------|
-| Multi-tenant routing | `X-Tenant-ID` header | Isolate enterprise customers |
-| Rate limiting | Per-tenant, per-endpoint | Prevent abuse, fair usage |
-| AI agent auth | OAuth2 Client Credentials | Machine-to-machine without humans |
-| Request logging | Tenant + agent context | Audit trail for enterprise |
-
-**Component Boundaries:**
-
-| Component | Responsibility | Communicates With |
-|-----------|----------------|-------------------|
-| API Gateway | Auth, rate limit, routing | Clients, Python Core |
-| Python Core | PROOF lifecycle, DID registry | API Gateway, Go Blockchain |
-| Go Blockchain | Immutable anchoring, consensus | Python Core (RPC) |
-
-**Configuration Pattern:**
-
-```yaml
-# Kong declarative config example
-services:
-  - name: proofnest-core
-    url: http://localhost:8200
-    routes:
-      - name: enterprise-route
-        paths: ["/v1"]
-        headers:
-          X-Tenant-ID: ["~*"]  # Required header
-    plugins:
-      - name: rate-limiting
-        config:
-          minute: 100
-          policy: redis
-          identifier: header
-          header_name: X-Tenant-ID
-      - name: oauth2
-        config:
-          scopes_required: [proof:create, identity:read]
-```
-
-**Confidence:** HIGH (Kong and Traefik are proven for this pattern)
-
-**Sources:**
-- [Kong Multi-Tenancy Guide](https://konghq.com/blog/enterprise/multi-tenancy)
-- [Azure API Management Multitenant](https://learn.microsoft.com/en-us/azure/architecture/guide/multitenant/service/api-management)
-
----
-
-### Layer 2: AI Agent Authentication
-
-**Why:** Traditional OAuth2 assumes human in the loop. AI agents need machine-to-machine (M2M) flows with delegation semantics.
-
-**Recommendation:** Implement **OAuth2 Client Credentials + Agent-ID Tokens**
-
-**Authentication Flow:**
-
-```
-1. AGENT REGISTRATION (one-time):
-   Agent Owner -> POST /v1/agents/register
-   {
-     "agent_name": "my-ai-agent",
-     "owner_did": "did:pn:human:abc123",
-     "capabilities": ["proof:create", "proof:read"],
-     "max_proof_rate": 100  // per minute
-   }
-   Response: { "client_id": "...", "client_secret": "..." }
-
-2. TOKEN REQUEST (per-session):
-   Agent -> POST /oauth/token
-   {
-     "grant_type": "client_credentials",
-     "client_id": "...",
-     "client_secret": "...",
-     "scope": "proof:create proof:read"
-   }
-   Response: {
-     "access_token": "...",  // JWT with agent_id, owner_did, scopes
-     "token_type": "Bearer",
-     "expires_in": 3600
-   }
-
-3. API CALLS (authenticated):
-   Agent -> POST /v1/proofs
-   Headers:
-     Authorization: Bearer <access_token>
-     X-Agent-ID: <agent_did>  // Agent's DID:PN identity
-```
-
-**Agent Identity Lifecycle:**
-
-```
-1. CREATION:
-   - Owner (human/org) registers agent
-   - Agent gets DID:PN identity (did:pn:agent:xxx)
-   - Agent gets OAuth2 credentials (client_id/secret)
-
-2. OPERATION:
-   - Agent authenticates via Client Credentials
-   - Agent creates PROOFs as issuer
-   - All PROOFs link back to agent DID
-
-3. DELEGATION (optional):
-   - Human delegates authority to agent
-   - Agent acts "on behalf of" human
-   - PROOF includes both agent_did AND delegator_did
-
-4. REVOCATION:
-   - Owner revokes agent credentials
-   - All future auth fails
-   - Historical PROOFs remain valid (immutable)
-```
-
-**Key Design Decisions:**
-
-| Decision | Choice | Why |
-|----------|--------|-----|
-| Agent identity | DID:PN (same as humans) | Unified identity model |
-| Auth mechanism | OAuth2 Client Credentials | Industry standard for M2M |
-| Token format | JWT with custom claims | Stateless verification |
-| Delegation | PROOF of type DELEGATION | Auditable on-chain |
-
-**Confidence:** HIGH (OAuth2 M2M is well-established, Agent-ID is emerging standard)
-
-**Sources:**
-- [Stytch: Agent-to-Agent OAuth Guide](https://stytch.com/blog/agent-to-agent-oauth-guide/)
-- [Microsoft: OAuth Must Evolve for AI Agents](https://techcommunity.microsoft.com/blog/microsoft-entra-blog/the-future-of-ai-agents%E2%80%94and-why-oauth-must-evolve/3827391)
-- [AWS Cognito: Empower AI Agents](https://aws.amazon.com/blogs/security/empower-ai-agents-with-user-context-using-amazon-cognito/)
-
----
-
-### Layer 3: AI-Human Hub Architecture
-
-**Why:** Hub needs real-time feeds, threading, reactions - not request-response.
-
-**Recommendation:** Event-driven architecture with **WebSocket + Redis Pub/Sub + Kafka (optional)**
-
-**Hub Data Model (extends PROOF):**
-
-```
-POST (thread starter):
-  - proof_type: CLAIM
-  - claim: "The AI decision was correct"
-  - metadata: {
-      "hub_type": "post",
-      "visibility": "public" | "tenant" | "private",
-      "attachments": ["proof:xxx", "proof:yyy"]
-    }
-
-REPLY (thread response):
-  - proof_type: CLAIM
-  - previous_proof: <parent_post_id>  // Thread link
-  - metadata: {
-      "hub_type": "reply",
-      "thread_root": <root_post_id>
-    }
-
-REACTION (lightweight signal):
-  - proof_type: ATTESTATION
-  - subject: <target_post_did>
-  - claim: "thumbs_up" | "verified" | "disputed"
-  - metadata: {
-      "hub_type": "reaction"
-    }
-```
-
-**Real-Time Architecture:**
-
-```
-+------------------+     +-----------------+     +------------------+
-|    WebSocket     |     |   Redis Pub/Sub |     |   Python Core    |
-|    Endpoint      |<--->|                 |<--->|   (PROOF CRUD)   |
-+------------------+     +-----------------+     +------------------+
-        ^                        ^                        |
-        |                        |                        v
-        |                        |               +------------------+
-   [Browsers]               [Multiple           |   SQLite/Postgres |
-   [Mobile Apps]             WS Servers]        +------------------+
-   [AI Agents]                                           |
-                                                         v
-                                                +------------------+
-                                                |   Go Blockchain  |
-                                                +------------------+
-```
-
-**Component Flow:**
-
-```
-1. CLIENT CONNECTS:
-   Client -> WebSocket /ws/hub?token=<jwt>
-   Server -> Validate JWT, extract tenant_id, agent_id
-   Server -> Subscribe to Redis channels:
-            - hub:<tenant_id>:all
-            - hub:<tenant_id>:thread:<thread_id>  (if watching thread)
-            - hub:agent:<agent_id>:notifications
-
-2. POST CREATION:
-   Client -> WS message: {"type": "create_post", "data": {...}}
-   Server -> Create PROOF via Python Core
-   Server -> PUBLISH to Redis: hub:<tenant_id>:all
-   All subscribers -> Receive real-time update
-
-3. FEED DELIVERY:
-   Redis subscriber -> WS message: {"type": "new_post", "data": {...}}
-   Client -> Renders in feed
-```
-
-**WebSocket Message Protocol:**
-
-```json
-// Client -> Server
-{"type": "subscribe", "channel": "hub:tenant123:all"}
-{"type": "create_post", "data": {"claim": "...", "visibility": "public"}}
-{"type": "create_reply", "parent_id": "proof:xxx", "data": {...}}
-{"type": "react", "target_id": "proof:xxx", "reaction": "verified"}
-
-// Server -> Client
-{"type": "subscribed", "channel": "hub:tenant123:all"}
-{"type": "new_post", "data": {"proof_id": "...", "issuer": "...", ...}}
-{"type": "new_reply", "thread_id": "...", "data": {...}}
-{"type": "new_reaction", "target_id": "...", "reaction": "...", "count": 42}
-```
-
-**Scaling Considerations:**
-
-| At Scale | Solution |
-|----------|----------|
-| 100 users | Single WS server, in-memory pub/sub |
-| 10K users | Redis Pub/Sub, multiple WS servers |
-| 1M users | Kafka for durability, dedicated feed service |
-
-**Confidence:** MEDIUM-HIGH (Pattern is proven, specifics need validation)
-
-**Sources:**
-- [InfoQ: Serverless WebSockets for Real-Time Messaging](https://www.infoq.com/articles/serverless-websockets-realtime-messaging/)
-- [Kafka + WebSockets + React](https://medium.com/@akshat.available/real-time-event-driven-architecture-with-kafka-websockets-and-react-b4698361e68a)
-
----
-
-### Layer 4: Proof Verification Scaling
-
-**Why:** As PROOF volume grows, verification becomes bottleneck.
-
-**Recommendation:** Implement **batch verification with Merkle proofs**
-
-**Current Flow (per-proof):**
-
-```
-Verify(proof_id):
-  1. Fetch PROOF from SQLite
-  2. Verify signature (Dilithium3 - slow!)
-  3. Verify hash matches
-  4. Optionally: verify_proof_on_chain (RPC to Go)
-```
-
-**Scaled Flow (batch):**
-
-```
-BatchVerify([proof_id1, proof_id2, ...]):
-  1. Fetch PROOFs from SQLite (batch query)
-  2. Build Merkle tree of proof_hashes
-  3. Verify Merkle root against chain (single RPC)
-  4. Return: { verified: true, merkle_root: "...", proofs: [...] }
-```
-
-**Merkle Tree Integration:**
-
-```
-PROOF Submission:
-  1. Collect PROOFs in mempool (Python Core)
-  2. Every N seconds OR M proofs: create batch
-  3. Build Merkle tree: root = hash(hash(p1) + hash(p2) + ...)
-  4. Submit batch to Go Blockchain (single TX with merkle_root)
-  5. Store merkle_proof for each PROOF
-
-PROOF Verification:
-  1. Client requests verify(proof_id)
-  2. Lookup PROOF and its merkle_proof
-  3. Verify merkle_proof against on-chain root
-  4. O(log n) verification instead of O(1) chain RPC
-```
-
-**Database Schema Extension:**
-
-```sql
--- Add to proofs table
-ALTER TABLE proofs ADD COLUMN batch_id TEXT;
-ALTER TABLE proofs ADD COLUMN merkle_index INTEGER;
-ALTER TABLE proofs ADD COLUMN merkle_proof JSON;  -- Array of sibling hashes
-
--- New batches table
-CREATE TABLE proof_batches (
-    batch_id TEXT PRIMARY KEY,
-    merkle_root TEXT NOT NULL,
-    chain_tx_hash TEXT,
-    chain_height INTEGER,
-    proof_count INTEGER,
-    created_at INTEGER,
-    confirmed_at INTEGER
-);
-```
-
-**Performance Expectations:**
-
-| Metric | Per-Proof | Batched (1000 proofs) |
-|--------|-----------|----------------------|
-| Chain RPCs | 1 per proof | 1 per batch |
-| Verification time | ~50ms | ~5ms per proof |
-| Storage overhead | 0 | ~1KB merkle_proof |
-
-**Confidence:** HIGH (Merkle trees are standard, pattern from rollups)
-
-**Sources:**
-- [Reilabs: Scaling Sparse Merkle Trees](https://reilabs.io/blog/scaling-sparse-merkle-trees-to-billions-of-keys-with-largesmt/)
-- [Designing Blockchain: Merkle Trees and State Verification](https://prodsens.live/2026/01/15/designing-blockchain-4-merkle-trees-and-state-verification/)
-
----
-
-## Build Order (Dependencies)
-
-**Phase 1: Foundation (Week 1-2)**
-
-```
-1. API Gateway deployment
-   - Kong or Traefik in front of Python Core
-   - Basic rate limiting
-   - X-Tenant-ID routing
-
-   Dependencies: None
-   Blocks: Everything else
-
-2. Database migration (SQLite -> PostgreSQL)
-   - Production persistence
-   - Connection pooling
-
-   Dependencies: None
-   Blocks: Hub, Scaling
-```
-
-**Phase 2: AI Agent Auth (Week 3-4)**
-
-```
-3. OAuth2 Client Credentials
-   - Token endpoint in Python Core
-   - JWT generation/validation
-
-   Dependencies: API Gateway
-
-4. Agent identity lifecycle
-   - Agent registration endpoint
-   - Agent DID:PN creation
-   - Delegation PROOF type
-
-   Dependencies: OAuth2
-```
-
-**Phase 3: Hub Core (Week 5-6)**
-
-```
-5. WebSocket endpoint
-   - Connection handling
-   - JWT validation on connect
-   - Basic subscribe/publish
-
-   Dependencies: Agent Auth
-
-6. Redis Pub/Sub integration
-   - Cross-server broadcasting
-   - Channel management
-
-   Dependencies: WebSocket endpoint
-
-7. Hub PROOF extensions
-   - Post/Reply/Reaction types
-   - Thread linking
-   - Visibility controls
-
-   Dependencies: Redis Pub/Sub
-```
-
-**Phase 4: Scaling (Week 7-8)**
-
-```
-8. Batch verification
-   - Merkle tree implementation
-   - Batch submission to chain
-   - Merkle proof storage
-
-   Dependencies: Hub Core
-
-9. Performance optimization
-   - Connection pooling
-   - Query optimization
-   - Caching layer
-
-   Dependencies: Batch verification
-```
-
----
-
-## Anti-Patterns to Avoid
-
-### Anti-Pattern 1: Bypassing Python Core
-
-**What:** Clients connect directly to Go Blockchain for speed
-**Why bad:** Loses PROOF lifecycle, consensus, DID validation
-**Instead:** All PROOF operations through Python Core; Go is storage layer
-
-### Anti-Pattern 2: Stateful WebSocket Servers
-
-**What:** Storing user state in WS server memory
-**Why bad:** Can't scale horizontally, single point of failure
-**Instead:** Redis for all shared state, WS servers are stateless
-
-### Anti-Pattern 3: Synchronous Chain Verification
-
-**What:** Calling Go Blockchain RPC in every API request
-**Why bad:** 50ms+ latency per call, creates bottleneck
-**Instead:** Batch verification, Merkle proofs, async confirmation
-
-### Anti-Pattern 4: API Key Per-Agent
-
-**What:** Generating static API keys for each AI agent
-**Why bad:** No expiration, no scope, hard to revoke
-**Instead:** OAuth2 tokens with expiration, scopes, revocation
-
----
-
-## Technology Decisions Summary
-
-| Layer | Recommended | Alternatives | Why |
-|-------|-------------|--------------|-----|
-| API Gateway | Kong | Traefik, NGINX+ | Best multi-tenant support |
-| AI Auth | OAuth2 + JWT | PASETO, mTLS | Industry standard, tooling |
-| Real-time | WebSocket + Redis | SSE, gRPC streams | Bidirectional, proven |
-| Message bus | Redis Pub/Sub | Kafka | Simpler ops, sufficient scale |
-| Batch verification | Custom Merkle | Hyperledger | Control, no extra deps |
-| Database | PostgreSQL | Keep SQLite | Production-grade |
-
----
-
-## Confidence Assessment
-
-| Area | Level | Reason |
-|------|-------|--------|
-| API Gateway patterns | HIGH | Standard enterprise pattern, proven tools |
-| AI Agent auth | HIGH | OAuth2 M2M is well-documented, emerging standards |
-| Hub real-time | MEDIUM-HIGH | Pattern proven, specifics need tuning |
-| Merkle batch verification | HIGH | Standard blockchain scaling technique |
-| Overall integration | MEDIUM-HIGH | Existing code is clean, integration points clear |
-
----
-
-## Open Questions for Phase-Specific Research
-
-1. **Hub thread depth:** How deep can thread nesting go before UX degrades?
-2. **Batch size optimization:** What's optimal batch size for Merkle trees?
-3. **Agent rate limits:** Per-agent or per-owner limits for enterprise?
-4. **Cross-tenant visibility:** Can agents from tenant A see tenant B posts?
-5. **Offline agents:** How to handle proofs from agents that later go offline?
-
----
-
-## Sources
-
-**Multi-Tenant API Patterns:**
-- [Kong Multi-Tenancy Guide](https://konghq.com/blog/enterprise/multi-tenancy)
-- [Azure API Management Multitenant](https://learn.microsoft.com/en-us/azure/architecture/guide/multitenant/service/api-management)
-- [AWS Multi-Tenant APIs](https://aws.amazon.com/blogs/compute/managing-multi-tenant-apis-using-amazon-api-gateway/)
-
-**AI Agent Authentication:**
-- [Stytch: Agent-to-Agent OAuth](https://stytch.com/blog/agent-to-agent-oauth-guide/)
-- [Microsoft: OAuth Evolution for AI](https://techcommunity.microsoft.com/blog/microsoft-entra-blog/the-future-of-ai-agents%E2%80%94and-why-oauth-must-evolve/3827391)
-- [WorkOS: Securing AI Agents](https://workos.com/blog/securing-ai-agents)
-- [Composio: AI Agent Infrastructure Guide](https://composio.dev/blog/secure-ai-agent-infrastructure-guide)
-
-**Real-Time Architecture:**
-- [InfoQ: Serverless WebSockets](https://www.infoq.com/articles/serverless-websockets-realtime-messaging/)
-- [Kafka + WebSockets + React](https://medium.com/@akshat.available/real-time-event-driven-architecture-with-kafka-websockets-and-react-b4698361e68a)
-- [Event Sourcing with CQRS and WebSockets](https://resources.fenergo.com/engineering-at-fenergo/working-with-event-sourcing-cqrs-and-web-sockets-on-aws)
-
-**Merkle Tree Scaling:**
-- [Reilabs: LargeSMT](https://reilabs.io/blog/scaling-sparse-merkle-trees-to-billions-of-keys-with-largesmt/)
-- [Designing Blockchain: Merkle Trees](https://prodsens.live/2026/01/15/designing-blockchain-4-merkle-trees-and-state-verification/)
-- [Adaptive Merkle Trees (ScienceDirect)](https://www.sciencedirect.com/science/article/pii/S2542660524002567)