ico-cache 1.0.0__tar.gz

ico_cache-1.0.0/LICENSE

@@ -0,0 +1 @@
+MIT License

ico_cache-1.0.0/PKG-INFO

@@ -0,0 +1,92 @@
+Metadata-Version: 2.4
+Name: ico-cache
+Version: 1.0.0
+Summary: ICO-Cache: Generalized LLM Semantic Cache
+Author: Dev
+Requires-Python: >=3.11
+Description-Content-Type: text/markdown
+License-File: LICENSE
+Requires-Dist: fastapi<1.0.0,>=0.110.0
+Requires-Dist: uvicorn<1.0.0,>=0.28.0
+Requires-Dist: qdrant-client<2.0.0,>=1.8.0
+Requires-Dist: redis<6.0.0,>=5.0.0
+Requires-Dist: fastembed<1.0.0,>=0.2.0
+Requires-Dist: pydantic<3.0.0,>=2.6.0
+Requires-Dist: pydantic-settings<3.0.0,>=2.2.0
+Requires-Dist: PyPDF2<4.0.0,>=3.0.0
+Requires-Dist: beautifulsoup4<5.0.0,>=4.12.0
+Requires-Dist: lancedb<1.0.0,>=0.5.0
+Requires-Dist: structlog<26.0.0,>=24.1.0
+Requires-Dist: litellm>=1.40.0
+Requires-Dist: prometheus-client<1.0.0,>=0.20.0
+Requires-Dist: opentelemetry-api>=1.20.0
+Requires-Dist: opentelemetry-sdk>=1.20.0
+Provides-Extra: loaders
+Requires-Dist: pymupdf>=1.24.0; extra == "loaders"
+Requires-Dist: tree-sitter>=0.23.0; extra == "loaders"
+Requires-Dist: tree-sitter-javascript>=0.23.0; extra == "loaders"
+Requires-Dist: tree-sitter-go>=0.23.0; extra == "loaders"
+Requires-Dist: pdf2image>=1.16.0; extra == "loaders"
+Requires-Dist: pytesseract>=0.3.10; extra == "loaders"
+Requires-Dist: Pillow>=10.0.0; extra == "loaders"
+Requires-Dist: odfpy>=1.4.1; extra == "loaders"
+Requires-Dist: python-docx>=1.1.0; extra == "loaders"
+Requires-Dist: openpyxl>=3.1.0; extra == "loaders"
+Requires-Dist: python-pptx>=0.6.23; extra == "loaders"
+Provides-Extra: observability
+Requires-Dist: opentelemetry-exporter-otlp-proto-http>=1.20.0; extra == "observability"
+Requires-Dist: langfuse>=2.0.0; extra == "observability"
+Provides-Extra: dev
+Requires-Dist: ico-cache[loaders,observability]; extra == "dev"
+Requires-Dist: pytest>=8.0.0; extra == "dev"
+Requires-Dist: pytest-asyncio>=1.0.0; extra == "dev"
+Requires-Dist: ruff>=0.4.0; extra == "dev"
+Requires-Dist: mypy>=1.8.0; extra == "dev"
+Dynamic: license-file
+
+# ICO-Cache: Generalized LLM Semantic Cache
+
+ICO-Cache is a high-performance, multi-layered semantic caching engine for Large Language Models. It minimizes redundant LLM calls by precisely matching queries contextually and semantically, while strictly guarding against false hits through centralized entity, topic, and quarter metadata filtering.
+
+## Quickstart (Zero-Infra Embedded Mode)
+
+```python
+from ico_cache import ICOCache, ICOConfig
+from ico_cache.backends.vector.lancedb_store import LanceDBStore
+from ico_cache.backends.exact.sqlite_store import SQLiteStore
+from ico_cache.backends.embedding.fastembed_embedder import FastEmbedder
+
+# Initialize with embedded databases
+engine = ICOCache(
+    embedder=FastEmbedder(),
+    vector_store=LanceDBStore(uri="./lancedb"),
+    exact_store=SQLiteStore(db_path="cache.db")
+)
+
+# Resolve query
+result = await engine.resolve("What is Apple's revenue?", meta={"entity": "AAPL", "topic": "revenue"})
+if result["source"] == "MISS":
+    # Generate and ingest...
+```
+
+## Performance & Reliability (Actual Measured Benchmarks)
+
+Results from the `verify_all_5` suite run:
+
+| Metric | Result | Detail |
+|--------|--------|--------|
+| **False-Hit Rate** | **0%** (0 / 100) | Tested on challenging near-miss negatives (entity/quarter/topic swaps). |
+| **True-Hit Rate (L3)** | **100%** (10 / 10) | Correctly fires on same-context paraphrased queries. |
+| **Hit Latency** | **~46ms** | 100 isolated pairs evaluated in 4.6s. |
+| **Concurrency Guard** | **0 dupes** | Tested at 20, 50, and 100 concurrent identical queries (0 lock errors, max 1 generation). |
+
+## Architecture: Embedded vs Server Mode
+
+| Feature | Embedded Mode | Server Mode |
+|---------|---------------|-------------|
+| **Vector Store** | LanceDB (Local) | Qdrant (Docker) |
+| **Exact Store** | SQLite (Local) | Redis (Docker) |
+| **Setup Complexity** | Zero (just `pip install`) | Requires Docker/Compose |
+| **Best For** | Prototyping, small scripts, CLI apps | Production, multi-tenant APIs, heavy concurrency |
+
+For Server Mode, see `docker-compose.yml` and `api/server.py`.

ico_cache-1.0.0/README.md

@@ -0,0 +1,46 @@
+# ICO-Cache: Generalized LLM Semantic Cache
+
+ICO-Cache is a high-performance, multi-layered semantic caching engine for Large Language Models. It minimizes redundant LLM calls by precisely matching queries contextually and semantically, while strictly guarding against false hits through centralized entity, topic, and quarter metadata filtering.
+
+## Quickstart (Zero-Infra Embedded Mode)
+
+```python
+from ico_cache import ICOCache, ICOConfig
+from ico_cache.backends.vector.lancedb_store import LanceDBStore
+from ico_cache.backends.exact.sqlite_store import SQLiteStore
+from ico_cache.backends.embedding.fastembed_embedder import FastEmbedder
+
+# Initialize with embedded databases
+engine = ICOCache(
+    embedder=FastEmbedder(),
+    vector_store=LanceDBStore(uri="./lancedb"),
+    exact_store=SQLiteStore(db_path="cache.db")
+)
+
+# Resolve query
+result = await engine.resolve("What is Apple's revenue?", meta={"entity": "AAPL", "topic": "revenue"})
+if result["source"] == "MISS":
+    # Generate and ingest...
+```
+
+## Performance & Reliability (Actual Measured Benchmarks)
+
+Results from the `verify_all_5` suite run:
+
+| Metric | Result | Detail |
+|--------|--------|--------|
+| **False-Hit Rate** | **0%** (0 / 100) | Tested on challenging near-miss negatives (entity/quarter/topic swaps). |
+| **True-Hit Rate (L3)** | **100%** (10 / 10) | Correctly fires on same-context paraphrased queries. |
+| **Hit Latency** | **~46ms** | 100 isolated pairs evaluated in 4.6s. |
+| **Concurrency Guard** | **0 dupes** | Tested at 20, 50, and 100 concurrent identical queries (0 lock errors, max 1 generation). |
+
+## Architecture: Embedded vs Server Mode
+
+| Feature | Embedded Mode | Server Mode |
+|---------|---------------|-------------|
+| **Vector Store** | LanceDB (Local) | Qdrant (Docker) |
+| **Exact Store** | SQLite (Local) | Redis (Docker) |
+| **Setup Complexity** | Zero (just `pip install`) | Requires Docker/Compose |
+| **Best For** | Prototyping, small scripts, CLI apps | Production, multi-tenant APIs, heavy concurrency |
+
+For Server Mode, see `docker-compose.yml` and `api/server.py`.

ico_cache-1.0.0/pyproject.toml

@@ -0,0 +1,63 @@
+[build-system]
+requires = ["setuptools>=61.0"]
+build-backend = "setuptools.build_meta"
+
+[project]
+name = "ico-cache"
+version = "1.0.0"
+description = "ICO-Cache: Generalized LLM Semantic Cache"
+authors = [{name = "Dev"}]
+readme = "README.md"
+requires-python = ">=3.11"
+dependencies = [
+    "fastapi>=0.110.0,<1.0.0",
+    "uvicorn>=0.28.0,<1.0.0",
+    "qdrant-client>=1.8.0,<2.0.0",
+    "redis>=5.0.0,<6.0.0",
+    "fastembed>=0.2.0,<1.0.0",
+    "pydantic>=2.6.0,<3.0.0",
+    "pydantic-settings>=2.2.0,<3.0.0",
+    "PyPDF2>=3.0.0,<4.0.0",
+    "beautifulsoup4>=4.12.0,<5.0.0",
+    "lancedb>=0.5.0,<1.0.0",
+    "structlog>=24.1.0,<26.0.0",
+    "litellm>=1.40.0",
+    "prometheus-client>=0.20.0,<1.0.0",
+    "opentelemetry-api>=1.20.0",
+    "opentelemetry-sdk>=1.20.0",
+]
+
+[project.optional-dependencies]
+# Document format support (used by loaders and their tests)
+loaders = [
+    "pymupdf>=1.24.0",
+    "tree-sitter>=0.23.0",
+    "tree-sitter-javascript>=0.23.0",
+    "tree-sitter-go>=0.23.0",
+    "pdf2image>=1.16.0",
+    "pytesseract>=0.3.10",
+    "Pillow>=10.0.0",
+    "odfpy>=1.4.1",
+    "python-docx>=1.1.0",
+    "openpyxl>=3.1.0",
+    "python-pptx>=0.6.23",
+]
+# Tracing/metrics exporters
+observability = [
+    "opentelemetry-exporter-otlp-proto-http>=1.20.0",
+    "langfuse>=2.0.0",
+]
+# Full dev/test environment
+dev = [
+    "ico-cache[loaders,observability]",
+    "pytest>=8.0.0",
+    "pytest-asyncio>=1.0.0",
+    "ruff>=0.4.0",
+    "mypy>=1.8.0",
+]
+
+[tool.setuptools.packages.find]
+where = ["src"]
+
+[tool.setuptools.package-data]
+ico_cache = ["py.typed"]

ico_cache-1.0.0/setup.cfg

@@ -0,0 +1,4 @@
+[egg_info]
+tag_build = 
+tag_date = 0
+

ico_cache-1.0.0/src/ico_cache/__init__.py

@@ -0,0 +1,7 @@
+from .core.cache_engine import CacheEngine, CacheEngine as ICOCache
+from .core.config import ICOConfig
+from .loaders.auto_loader import ingest, AutoLoader
+
+__version__ = "0.1.0"
+
+__all__ = ["CacheEngine", "ICOCache", "ICOConfig", "ingest", "AutoLoader", "__version__"]

ico_cache-1.0.0/src/ico_cache/async_ingest.py

@@ -0,0 +1,207 @@
+import asyncio
+import logging
+import os
+import threading
+import time
+import uuid
+from typing import Dict, Optional
+from pydantic import BaseModel, Field
+
+from .loaders.auto_loader import AutoLoader
+from .core.cache_engine import CacheEngine
+from .core.metadata_guard import MetadataSchema
+
+logger = logging.getLogger("ico_cache.async_ingest")
+
+
+class IngestionJob(BaseModel):
+    job_id: str
+    tenant_id: str
+    file_path: str
+    file_size_bytes: int
+    status: str = "queued"  # queued, processing, completed, failed
+    is_async: bool = False
+    chunks_total: int = 0
+    chunks_processed: int = 0
+    created_at: float = Field(default_factory=time.time)
+    completed_at: Optional[float] = None
+    elapsed_s: float = 0.0
+    error: Optional[str] = None
+
+
+class IngestionJobManager:
+    """
+    Manages synchronous and asynchronous document ingestion.
+    Files exceeding async_threshold_bytes (default 1MB) are ingested
+    asynchronously in background tasks with queryable job status.
+    """
+
+    def __init__(
+        self,
+        async_threshold_bytes: int = 1 * 1024 * 1024,
+        job_ttl_seconds: int = 3600,
+        max_jobs: int = 10000,
+    ):
+        self.async_threshold_bytes = async_threshold_bytes
+        self.job_ttl_seconds = job_ttl_seconds
+        self.max_jobs = max_jobs
+        self.jobs: Dict[str, IngestionJob] = {}
+        self._lock = threading.Lock()
+
+    def _cleanup_expired_unlocked(self, now: Optional[float] = None) -> int:
+        """Evicts completed/failed jobs older than job_ttl_seconds, and enforces max_jobs cap."""
+        ts = now if now is not None else time.time()
+        to_delete = []
+
+        # 1. Evict by TTL
+        for jid, job in self.jobs.items():
+            if job.status in ["completed", "failed"] and job.completed_at:
+                if (ts - job.completed_at) > self.job_ttl_seconds:
+                    to_delete.append(jid)
+
+        for jid in to_delete:
+            del self.jobs[jid]
+
+        # 2. Evict oldest finished jobs if exceeding capacity cap
+        if len(self.jobs) > self.max_jobs:
+            finished = sorted(
+                [j for j in self.jobs.values() if j.status in ["completed", "failed"]],
+                key=lambda x: x.completed_at or 0.0,
+            )
+            excess = len(self.jobs) - self.max_jobs
+            for j in finished[:excess]:
+                if j.job_id in self.jobs:
+                    del self.jobs[j.job_id]
+                    to_delete.append(j.job_id)
+
+        return len(to_delete)
+
+    def cleanup_expired_jobs(self, now: Optional[float] = None) -> int:
+        """Public thread-safe cleanup method."""
+        with self._lock:
+            return self._cleanup_expired_unlocked(now=now)
+
+    def get_job(self, job_id: str) -> Optional[IngestionJob]:
+        with self._lock:
+            self._cleanup_expired_unlocked()
+            job = self.jobs.get(job_id)
+            if job and job.status == "processing":
+                job.elapsed_s = time.time() - job.created_at
+            return job
+
+    def list_jobs(self, tenant_id: Optional[str] = None) -> Dict[str, IngestionJob]:
+        with self._lock:
+            self._cleanup_expired_unlocked()
+            if tenant_id:
+                return {k: v for k, v in self.jobs.items() if v.tenant_id == tenant_id}
+            return dict(self.jobs)
+
+    def submit_ingest(
+        self,
+        file_path: str,
+        tenant_id: str,
+        cache_engine: CacheEngine,
+        schema: Optional[MetadataSchema] = None,
+        force_async: Optional[bool] = None,
+    ) -> IngestionJob:
+        if not os.path.exists(file_path):
+            raise FileNotFoundError(f"File not found: {file_path}")
+
+        file_size = os.path.getsize(file_path)
+        job_id = str(uuid.uuid4())
+        should_async = force_async if force_async is not None else (file_size >= self.async_threshold_bytes)
+
+        job = IngestionJob(
+            job_id=job_id,
+            tenant_id=tenant_id,
+            file_path=file_path,
+            file_size_bytes=file_size,
+            status="processing",
+            is_async=should_async,
+            created_at=time.time(),
+        )
+
+        with self._lock:
+            self.jobs[job_id] = job
+
+        if should_async:
+            # Dispatch background worker thread
+            worker = threading.Thread(
+                target=self._run_ingest,
+                args=(job_id, file_path, tenant_id, cache_engine, schema),
+                daemon=True,
+            )
+            worker.start()
+            return job
+        else:
+            # Run synchronously in an isolated thread to protect caller's event loop
+            worker = threading.Thread(
+                target=self._run_ingest,
+                args=(job_id, file_path, tenant_id, cache_engine, schema),
+                daemon=True,
+            )
+            worker.start()
+            worker.join()
+            with self._lock:
+                return self.jobs[job_id]
+
+    def _run_ingest(
+        self,
+        job_id: str,
+        file_path: str,
+        tenant_id: str,
+        cache_engine: CacheEngine,
+        schema: Optional[MetadataSchema] = None,
+    ):
+        start_t = time.time()
+        try:
+            loader = AutoLoader(schema=schema)
+            chunks = loader.load(file_path)
+
+            with self._lock:
+                job = self.jobs[job_id]
+                job.chunks_total = len(chunks)
+
+            loop = asyncio.new_event_loop()
+            asyncio.set_event_loop(loop)
+
+            try:
+                for idx, chunk in enumerate(chunks):
+                    loader_kind = getattr(chunk, "loader_type", None) or (chunk.metadata.get("loader") if isinstance(chunk.metadata, dict) else "text")
+                    dummy_resp = {
+                        "content": chunk.text[:200],
+                        "source": chunk.source_file,
+                        "section": chunk.page_or_section,
+                        "loader_type": loader_kind,
+                        "extraction_method": chunk.metadata.get("extraction_method", "direct") if isinstance(chunk.metadata, dict) else "direct",
+                        "metadata": chunk.metadata if isinstance(chunk.metadata, dict) else {},
+                    }
+                    cache_engine.set_l1(chunk.text[:100], dummy_resp, meta=chunk.metadata, tenant_id=tenant_id)
+                    loop.run_until_complete(
+                        cache_engine.async_write_l2(
+                            chunk.text[:100], dummy_resp, meta=chunk.metadata, tenant_id=tenant_id
+                        )
+                    )
+                    with self._lock:
+                        job.chunks_processed = idx + 1
+            finally:
+                loop.close()
+
+            with self._lock:
+                job = self.jobs[job_id]
+                job.status = "completed"
+                job.completed_at = time.time()
+                job.elapsed_s = job.completed_at - start_t
+                logger.info(f"Ingestion job {job_id} completed: {job.chunks_processed} chunks in {job.elapsed_s:.2f}s")
+
+        except Exception as e:
+            with self._lock:
+                job = self.jobs[job_id]
+                job.status = "failed"
+                job.completed_at = time.time()
+                job.elapsed_s = job.completed_at - start_t
+                job.error = str(e)
+            logger.error(f"Ingestion job {job_id} failed: {e}", exc_info=True)
+
+
+job_manager = IngestionJobManager()

ico_cache-1.0.0/src/ico_cache/backends/base.py

@@ -0,0 +1,47 @@
+from abc import ABC, abstractmethod
+from typing import List, Any, Optional
+
+class BaseEmbedder(ABC):
+    @abstractmethod
+    def embed(self, text: str) -> List[float]:
+        pass
+
+class BaseVectorStore(ABC):
+    @abstractmethod
+    async def insert(self, collection: str, id: int, vector: Any, payload: dict):
+        pass
+
+    @abstractmethod
+    async def search(self, collection: str, vector: Any, query_filter: Any, limit: int, score_threshold: float, using: Optional[str] = None, **kwargs: Any) -> List[Any]:
+        pass
+
+    @abstractmethod
+    async def delete(self, collection: str, id: int):
+        pass
+
+    @abstractmethod
+    def collection_exists(self, collection: str) -> bool:
+        pass
+
+    @abstractmethod
+    def create_collection(self, collection: str, config: Any):
+        pass
+
+    def delete_collection(self, collection: str):
+        pass
+
+    async def delete_matching(self, collection: str, filter_dict: Optional[dict] = None) -> int:
+        return 0
+
+class BaseExactStore(ABC):
+    @abstractmethod
+    def get(self, key: str) -> Optional[bytes]:
+        pass
+
+    @abstractmethod
+    def set(self, key: str, value: bytes, ex: Optional[int] = None, nx: bool = False):
+        pass
+
+    def delete_prefix(self, prefix: str) -> int:
+        return 0
+

ico_cache-1.0.0/src/ico_cache/backends/embedding/fastembed_embedder.py

@@ -0,0 +1,10 @@
+from fastembed import TextEmbedding
+from typing import List
+from ..base import BaseEmbedder
+
+class FastEmbedder(BaseEmbedder):
+    def __init__(self, model_name: str = "BAAI/bge-small-en-v1.5"):
+        self.model = TextEmbedding(model_name=model_name)
+
+    def embed(self, text: str) -> List[float]:
+        return list(self.model.embed([text]))[0].tolist()

ico_cache-1.0.0/src/ico_cache/backends/exact/redis_store.py

@@ -0,0 +1,42 @@
+import redis
+from typing import Optional
+from ..base import BaseExactStore
+
+class RedisStore(BaseExactStore):
+    def __init__(self, host: str = "localhost", port: int = 6379, password: str = None):
+        self.r = redis.Redis(host=host, port=port, password=password)
+
+    @property
+    def client(self):
+        return self.r
+
+    def get(self, key: str) -> Optional[bytes]:
+        return self.r.get(key)
+
+    def set(self, key: str, value: bytes, ex: Optional[int] = None, nx: bool = False) -> bool:
+        # nx=True only writes when the key is absent (conditional write).
+        result = self.r.set(key, value, ex=ex, nx=nx)
+        return result is not None
+
+    def delete(self, key: str) -> bool:
+        return bool(self.r.delete(key))
+
+    def delete_prefix(self, prefix: str) -> int:
+        cursor = 0
+        deleted = 0
+        match_pattern = f"{prefix}*"
+        while True:
+            cursor, keys = self.r.scan(cursor=cursor, match=match_pattern, count=100)  # type: ignore[misc]
+            if keys:
+                deleted += self.r.delete(*keys)  # type: ignore[operator]
+            if cursor == 0:
+                break
+        return deleted
+
+    def xadd(self, stream: str, fields: dict) -> str:
+        res = self.r.xadd(stream, fields)
+        return res.decode() if isinstance(res, bytes) else str(res)
+
+    def xread(self, streams: dict, count: Optional[int] = None, block: Optional[int] = None):
+        return self.r.xread(streams, count=count, block=block)
+

ico_cache-1.0.0/src/ico_cache/backends/exact/sqlite_store.py

@@ -0,0 +1,43 @@
+import sqlite3
+from typing import Optional
+from ..base import BaseExactStore
+
+class SQLiteStore(BaseExactStore):
+    def __init__(self, db_path: str = "cache.db"):
+        self.db_path = db_path
+        self._init_db()
+
+    def _init_db(self):
+        with sqlite3.connect(self.db_path) as conn:
+            conn.execute('''CREATE TABLE IF NOT EXISTS cache
+                            (key TEXT PRIMARY KEY, value BLOB)''')
+
+    def get(self, key: str) -> Optional[bytes]:
+        with sqlite3.connect(self.db_path) as conn:
+            cursor = conn.cursor()
+            cursor.execute("SELECT value FROM cache WHERE key=?", (key,))
+            row = cursor.fetchone()
+            if row:
+                return row[0]
+            return None
+
+    def set(self, key: str, value: bytes, ex: Optional[int] = None, nx: bool = False) -> bool:
+        # Note: SQLite store doesn't support TTL out of the box in this simple
+        # implementation. nx=True performs a conditional (insert-if-absent) write.
+        verb = "INSERT OR IGNORE" if nx else "INSERT OR REPLACE"
+        with sqlite3.connect(self.db_path) as conn:
+            cur = conn.execute(f"{verb} INTO cache (key, value) VALUES (?, ?)", (key, value))
+            if nx:
+                return cur.rowcount > 0
+            return True
+
+    def delete(self, key: str) -> bool:
+        with sqlite3.connect(self.db_path) as conn:
+            cur = conn.execute("DELETE FROM cache WHERE key = ?", (key,))
+            return cur.rowcount > 0
+
+    def delete_prefix(self, prefix: str) -> int:
+        with sqlite3.connect(self.db_path) as conn:
+            cur = conn.execute("DELETE FROM cache WHERE key LIKE ?", (f"{prefix}%",))
+            return cur.rowcount
+