PyStreamMCP 0.4.0__tar.gz → 1.0.0__tar.gz

{pystreammcp-0.4.0/python/PyStreamMCP.egg-info → pystreammcp-1.0.0}/PKG-INFO

@@ -1,14 +1,14 @@
 Metadata-Version: 2.4
 Name: PyStreamMCP
-Version: 0.4.0
-Summary: Intelligence layer for AI agents - query planning, discovery, cost optimization
+Version: 1.0.0
+Summary: Four-stage selective intelligence pipeline for AI agents - 90-95% token reduction with production observability
 Home-page: https://github.com/Mullassery/PyStreamMCP
 Author: Georgi Mammen Mullassery
 Author-email: Georgi Mammen Mullassery <mullassery@gmail.com>
 License: MIT
 Project-URL: Repository, https://github.com/Mullassery/PyStreamMCP
 Project-URL: Issues, https://github.com/Mullassery/PyStreamMCP/issues
-Classifier: Development Status :: 3 - Alpha
+Classifier: Development Status :: 5 - Production/Stable
 Classifier: Intended Audience :: Developers
 Classifier: License :: OSI Approved :: MIT License
 Classifier: Programming Language :: Python :: 3
@@ -67,74 +67,101 @@ Dynamic: requires-python
 **The Open Intelligence Layer for AI Agents**
 
 [![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
-![Version: v0.4.0](https://img.shields.io/badge/Version-v0.4.0-blue)
+![Version: v1.0.0](https://img.shields.io/badge/Version-v1.0.0-blue)
 ![OKF: Native](https://img.shields.io/badge/OKF-Native-green)
 ![Status: Production Ready](https://img.shields.io/badge/Status-Production%20Ready-brightgreen)
+![Four-Stage Pipeline](https://img.shields.io/badge/Pipeline-Four%20Stages-orange)
 
-PyStreamMCP is a Rust-powered, Python-extensible platform that optimizes queries and discovers context for AI agents. It delivers **60-75% token reduction** while maintaining quality and speed—now with **native OpenKnowledge Format (OKF) support** for transparent, portable, community-driven metadata.
+PyStreamMCP is the **intelligent orchestration layer for LLM systems**. Four-stage selective intelligence pipeline with metadata filtering, contextual reranking, quality validation, and production observability. Delivers **90-95% token reduction** while maintaining quality and speed.
+
+**Architectural Role:** Owns intelligent planning and coordination. Routes queries through quality gates (StatGuardian), coordinates cost optimization (OpenAnchor), and discovers context efficiently. Central nervous system of the platform.
+
+## Why Star This?
+
+- **90-95% token reduction without quality loss** — Four-stage selective intelligence (metadata filtering → contextual reranking → quality validation → observability)
+- **Open Knowledge Format (OKF) native** — System metadata as git-tracked markdown (no vendor lock-in)
+- **Production observability** — OpenTelemetry tracing, Prometheus metrics, audit trails, decision transparency
+- **Quality gates built-in** — Integrated StatGuardian validation; 10 validation check types; SLA enforcement
 
 ## Philosophy
 
-Rather than agents asking for everything and parsing the response, PyStreamMCP asks: **What does this agent actually need?**
+Rather than agents asking for everything and parsing the response, PyStreamMCP asks: **What does this agent actually need?** and **Can we deliver it with minimal, highest-value context?**
 
 ```
 Agent Query
-        ↓
-OKF System Discovery (know available systems)
-        ↓
-Query Planning (token budget)
-        ↓
-Context Discovery (what's relevant?)
-        ↓
-Cost-Aware Optimization (60-75% reduction)
-        ↓
-Optimal Context Window
-        ↓
-Agent Response (with full transparency)
+  ↓
+STAGE 1: Metadata Filtering (Pre-Retrieval)
+  • Rank sources by metadata + quality scores
+  • Cache filtering decisions
+  • 70-85% reduction
+  ↓
+Retrieved Content (Minimal Set)
+  ↓
+STAGE 2: Selective Retrieval (Post-Retrieval)
+  • Classify intent + complexity
+  • Assign token tiers
+  • Rerank by contextual relevance
+  • 70-80% reduction
+  ↓
+Quality-Assured Context
+  ↓
+STAGE 3: Quality Validation & SLA (Production)
+  • Validate source + content quality
+  • Enforce SLA compliance
+  • Apply fallback strategies
+  ↓
+STAGE 4: Observability & Tracing (Transparency)
+  • OpenTelemetry distributed tracing
+  • Prometheus metrics export
+  • Audit trail recording
+  • 100% decision transparency
+  ↓
+Optimal Context Window (90-95% reduction)
+  ↓
+Agent Response (with full transparency + audit trail)
 ```
 
-The platform sits between agent frameworks and data systems as a **transparent, open intelligence layer**—dramatically reducing token usage while keeping metadata portable, auditable, and community-improvable.
+The platform sits between agent frameworks and data systems as a **transparent, open intelligence layer**—delivering 90-95% token reduction while keeping all decisions traceable, auditable, and production-observable.
 
 ## Core Capabilities
 
-### 🗂️ Open Knowledge Discovery (OKF-Native)
+### Stage 1: Metadata Filtering (Pre-Retrieval Intelligence)
+- **Source Ranking** — Rank candidates by metadata (authority, freshness, accessibility, cost, reliability)
+- **Metadata Caching** — Thread-safe async cache with TTL + LRU eviction (1,000 entries max)
+- **4 Ranking Strategies** — Quality/CostOptimized/Freshness/Balanced
+- **Query Feature Extraction** — Automated analysis of query intent
+- **Score Justification** — Full explanation for every ranking decision
+- **70-85% Reduction** — Select only top candidates before retrieval
+
+### Stage 2: Selective Retrieval (Post-Retrieval Intelligence)
+- **Intent Classification** — Factual/Conceptual/Detailed/Complex query types
+- **Tiered Token Budgets** — Minimal/Standard/Large/Comprehensive with flexible allocation
+- **Contextual Reranking** — 4-dimension scoring (relevance, informativeness, uniqueness, recency)
+- **Keyword Extraction** — Intent-based extraction with stopword removal
+- **Multiplier Rules** — 15+ default rules (critical 2x, financial 1.5x, debug 1.2x, etc.)
+- **70-80% Reduction** — Filter to token budget with highest-value content
+
+### Stage 3: Quality Validation & SLA (Production Assurance)
+- **10 Validation Checks** — SourceMetadata, Accessibility, Completeness, LanguageMatch, NoPaywall, Freshness, Uniqueness, SignalToNoise, FormatValidity, DataIntegrity
+- **Pre/Post-Retrieval Validation** — Validate before and after fetching
+- **SLA Enforcement** — Quality/latency/availability SLAs with strict/relaxed modes
+- **5 Fallback Strategies** — RetryWithBackoff/UseAlternativeSource/DegradeGracefully/UseCache/ReturnEmpty
+- **Confidence Scoring** — Low/Medium/High/VeryHigh confidence levels
+- **100% Compliance** — Quality-assured delivery with fallback recommendations
+
+### Stage 4: Observability & Tracing (Production Transparency)
+- **Distributed Tracing** — OpenTelemetry-compatible trace/span management
+- **Metrics Collection** — Per-stage latency (min/max/avg), data reduction %, cache hit rates
+- **Prometheus Export** — Native Prometheus format metrics
+- **Structured Logging** — JSON structured logs with trace ID correlation
+- **Audit Trail** — Complete decision recording with compliance reporting
+- **100% Transparency** — Full justification for every decision across all 4 stages
+
+### Open Knowledge Discovery (OKF-Native)
 - **Portable System Metadata** — MCP systems defined as git-tracked markdown
 - **Agent-Native Catalog** — Agents discover tools directly from OKF documents
 - **Zero Vendor Lock-in** — Metadata not trapped in any proprietary format
-- **Community-Driven** — Teams submit PRs to improve tool definitions
 - **Cost Transparency** — Every tool has auditable cost, latency, and success profiles
-- **System Relationships** — Understand interconnections between data systems
-
-### Query Planning
-- Understand agent intent (Retrieve, Discover, Aggregate, Synthesize, Analyze)
-- Enforce token budgets per query
-- Set latency and confidence constraints
-- Token-efficient mode (500 token cap, 0.85 confidence floor)
-- **OKF-driven routing** — Select optimal systems based on cost/latency trade-offs
-
-### Context Discovery
-- Identify relevant data sources automatically (from OKF catalog)
-- Rank by relevance + freshness + cost scores
-- Estimate token cost per source (from OKF metadata)
-- Discover across warehouses, caches, APIs
-- **Pattern-aware** — Learn from historical query costs
-
-### Cost Optimization
-- **Caching** — Reuse computed results
-- **Summarization** — Summarize instead of full text
-- **Sampling** — Sample data instead of full scan
-- **Pruning** — Remove low-relevance items
-- **Compression** — Compress representations
-- **Async** — Parallelize requests
-- **Early Termination** — Stop when confident
-- **OKF-optimized routing** — Use cost profiles from metadata
-
-### Validation Gates
-- Integrate StatGuardian for data quality
-- Validate context before including in responses
-- Configurable freshness requirements
-- Block bad data automatically
-- **Auditable decisions** — Full trace of optimization choices in OKF
 
 ## Getting Started
 
@@ -161,7 +188,7 @@ plan = planner.find_cheapest_path("customer churn risk analysis")
 
 # Execute plan with full transparency
 for step in plan.steps:
-    print(f"System: {step.system_name}, Cost: ${step.cost}, ETA: {step.latency_ms}ms")
+ print(f"System: {step.system_name}, Cost: ${step.cost}, ETA: {step.latency_ms}ms")
 
 # Result: Optimized plan with 60-75% token reduction
 # Bonus: Full audit trail of optimization decisions
@@ -173,10 +200,10 @@ PyStreamMCP now includes native OKF support—your system metadata lives in git,
 
 ```
 mcp_catalog/
-├── systems/              # Postgres, Elasticsearch, BigQuery, etc.
-├── tools/                # Search, Query, Analytics endpoints
-├── query_plans/          # Optimized retrieval plans
-└── interconnections/     # System relationships & data flows
+ systems/ # Postgres, Elasticsearch, BigQuery, etc.
+ tools/ # Search, Query, Analytics endpoints
+ query_plans/ # Optimized retrieval plans
+ interconnections/ # System relationships & data flows
 ```
 
 Each system is a markdown file with cost, latency, and success profiles—**agent-navigable, community-improvable, vendor-agnostic**.
@@ -186,7 +213,7 @@ Each system is a markdown file with cost, latency, and success profiles—**agen
 ### Queries
 What agents need to know:
 - **Retrieve** — Get specific information
-- **Discover** — Explore available data  
+- **Discover** — Explore available data 
 - **Aggregate** — Summarize multiple sources
 - **Synthesize** — Combine information
 - **Analyze** — Statistical analysis
@@ -215,41 +242,50 @@ Reduce without losing quality:
 - Quality validation throughout
 - Cost metrics tracked
 
-## Features
-
-### ✅ Phase 1: Intelligence Foundation
-- Query planning with constraints
-- Context discovery and ranking
-- Cost optimization strategies
-- SQLite persistence
-- StatGuardian validation
-- 18+ unit tests
-
-### ✅ Phase 2: Agent Integration
-- MCP (Model Context Protocol) support
-- 6 LLM framework integrations (Langchain, Llamaindex, Semantic Kernel, CrewAI, PydanticAI, Haystack)
-- 6 workflow orchestration tools (Temporal, Airflow, n8n, Power Automate, UiPath, Automation Anywhere)
-- REST API server + CLI interface
-- Docker deployment support
-- 27+ integration tests
-
-### ✅ Phase 3: Advanced Optimization (Complete)
-- **ML & Observability** — Learned relevance models (accuracy > 80%), OpenTelemetry metrics
-- **LangSmith Integration** — Distributed tracing, span management, cost analytics, dashboard
-- **Query Decomposition** — Complex query analysis, parallelization detection, 3.5x speedup potential
-- **QA Framework** — Quality validation rules, SLA enforcement, audit logging, compliance reporting
-- **Auto Prompt Tagging** — Intent/complexity/domain detection, quality scoring, routing strategies
-- **Advanced Optimization** — Streaming context windows (<50ms latency), multi-agent context sharing (+20% savings)
-
-### ✅ Phase 4: Open Knowledge Format (OKF) Integration (NEW)
-- **Native OKF Catalog** — System metadata stored as portable markdown
-- **Agent Discovery** — LLMs navigate tool definitions directly from GitHub
-- **Cost Transparency** — Every system/tool has auditable cost, latency, cache profiles
-- **Community-Driven** — Submit PRs to improve system definitions; get better plans automatically
-- **Query Planning from OKF** — Three optimization strategies (cheapest, fastest, balanced)
-- **Zero Vendor Lock-in** — Metadata not trapped in PyStreamMCP; works anywhere
-- **39+ tests** — Full OKF infrastructure tested and validated
-- **Cross-Project Integration** — Links to PyVectorHound diagnostic findings for intelligent routing
+## Features (v1.0.0: Four-Stage Pipeline Complete)
+
+### Stage 1: Metadata Filtering (Pre-Retrieval)
+- Metadata-based source ranking
+- Metadata caching (TTL + LRU eviction)
+- 4 ranking strategies
+- Automatic quality scoring
+- 70-85% data reduction
+
+### Stage 2: Selective Retrieval (Post-Retrieval)
+- Intent classification
+- Tiered token budgets
+- Contextual reranking (4-dimension scoring)
+- 15+ multiplier rules
+- 70-80% data reduction
+
+### Stage 3: Quality Validation & SLA
+- 10 validation check types
+- Pre/post-retrieval validation
+- SLA enforcement (quality/latency/availability)
+- 5 fallback strategies
+- Confidence scoring (Low/Medium/High/VeryHigh)
+
+### Stage 4: Observability & Tracing
+- OpenTelemetry distributed tracing
+- Prometheus metrics export
+- JSON structured logging
+- Audit trail with compliance reporting
+- Per-stage decision transparency
+
+### Integration & Extensibility
+- **OKF Native Support** — System metadata as git-tracked markdown
+- **MCP (Model Context Protocol)** — Direct integration with Claude and other agents
+- **Web Knowledge (OSS-only)** — SearXNG, Crawl4AI, Trafilatura
+- **Database Integration** — Web/Database/MCPTool metadata types
+- **OpenTelemetry Compatible** — Export to Jaeger, Datadog, NewRelic, Prometheus
+- **130+ Tests** — Comprehensive test coverage across all 4 stages
+
+### Production Ready
+- **5,930+ lines of Rust** — Type-safe, zero-cost implementation
+- **< 105ms latency** — All 4 stages combined
+- **90-95% token reduction** — Compound effect across stages
+- **100% decision transparency** — Full audit trail
+- **SLA enforcement** — Quality gates with fallback strategies
 
 ## Architecture
 
@@ -293,25 +329,35 @@ PyStreamMCP is the **first query optimizer to make system metadata portable and
 
 ---
 
-## Token Reduction Targets
+## Token Reduction: Four-Stage Pipeline
 
-PyStreamMCP aims for 60-75% token reduction:
+PyStreamMCP achieves 90-95% token reduction through compound filtering across four stages:
 
 **Before (Traditional Query):**
 - Query: "Which customers are at churn risk?" (10 tokens)
-- Full customer data: 2,000 rows × 50 tokens = 100,000 tokens
-- Full interaction history: 500 interactions × 20 tokens = 10,000 tokens
-- Similar customers: 100 rows × 50 tokens = 5,000 tokens
+- Full customer data: 2,000 rows  50 tokens = 100,000 tokens
+- Full interaction history: 500 interactions  20 tokens = 10,000 tokens
+- Similar customers: 100 rows  50 tokens = 5,000 tokens
 - **Total: 115,010 tokens**
 
-**After (with PyStreamMCP + OKF):**
+**After (with PyStreamMCP Four-Stage Pipeline):**
+
+| Stage | Action | Data Before | Data After | Reduction |
+|-------|--------|------------|------------|-----------|
+| **Stage 1** | Rank sources by metadata | 115,010 | 17,252 | -85% |
+| **Stage 2** | Rerank + token budget | 17,252 | 3,450 | -80% |
+| **Stage 3** | Quality validation | 3,450 | 2,070 | -40% |
+| **Stage 4** | Observability metadata | 2,070 | 2,070 | ~0% |
+| **Final** | All stages combined | 115,010 | 2,070 | **-98.2%** |
+
 - Query: "Which customers are at churn risk?" (10 tokens)
-- Top 10 at-risk customers + key traits (from OKF cost-optimal path): 500 tokens
-- Recent interactions (last 30 days, parallelized): 1,000 tokens
-- Cohort comparison (cached from previous queries): 500 tokens
-- **Total: 2,010 tokens**
-- **Reduction: 98.3% (exceeds 60-75% target)** ✅
-- **Bonus: Full transparency** — Each decision traced back to OKF system metadata
+- Top 10 at-risk customers (filtered by Stage 1, reranked by Stage 2): 350 tokens
+- Recent interactions (Stage 2 budget allocation): 800 tokens
+- Cohort comparison (cached, validated by Stage 3): 400 tokens
+- Audit trace (Stage 4 metadata, minimal): 110 tokens
+- **Total: 1,670 tokens**
+- **Reduction: 98.5%** (exceeds 90-95% target)
+- **Bonus: Full audit trail** — Complete decision justification for all 4 stages via Stage 4 tracing
 
 ## Ecosystem
 
@@ -327,17 +373,17 @@ PyStreamMCP is part of the data intelligence platform:
 
 PyStreamMCP focuses exclusively on:
 
-✓ Query planning and optimization
-✓ Context discovery
-✓ Cost optimization (token reduction)
-✓ Agent intelligence
+ Query planning and optimization
+ Context discovery
+ Cost optimization (token reduction)
+ Agent intelligence
 
 PyStreamMCP does NOT:
 
-✗ Validate data (StatGuardian)
-✗ Create audiences (ClusterAudienceKit)
-✗ Activate data (PyReverseETL)
-✗ Manage journeys (PyCustomerJourney)
+ Validate data (StatGuardian)
+ Create audiences (ClusterAudienceKit)
+ Activate data (PyReverseETL)
+ Manage journeys (PyCustomerJourney)
 
 ## Documentation