memoryagent-lib 0.1.1__tar.gz

memoryagent_lib-0.1.1/PKG-INFO

@@ -0,0 +1,186 @@
+Metadata-Version: 2.4
+Name: memoryagent-lib
+Version: 0.1.1
+Summary: Add your description here
+Author-email: Jiawei Zheng <jw.zhengai@gmail.com>
+License: MIT
+Project-URL: Homepage, https://github.com/jia-wei-zheng/MemoryAgent
+Project-URL: Repository, https://github.com/jia-wei-zheng/MemoryAgent
+Requires-Python: >=3.10
+Description-Content-Type: text/markdown
+Requires-Dist: python-dotenv>=0.9.9
+Requires-Dist: openai>=2.16.0
+Requires-Dist: pydantic>=2.0
+Requires-Dist: sqlite-vec>=0.1.6
+
+# 
+
+<div align="center">
+  <img src="https://raw.githubusercontent.com/jia-wei-zheng/MemoryAgent/refs/heads/master/memoryagent_logo.jpg?token=GHSAT0AAAAAADSXDGNDXIWZF42E6S3D2XPG2MDPSDQ" alt="MemoryAgent" width="500">
+  <h1>MemoryAgent: An Open, Modular Memory Framework for Agents (Beta)</h1>
+</div>
+
+MemoryAgent is a reusable memory framework for LLM-based agent systems. It provides tiered memory (working, episodic, semantic, perceptual), hot/cold storage, archive indexing, confidence-based retrieval escalation, and optional local vector search via sqlite-vec.
+
+## Highlights
+- **Tiered memory**: working (TTL), episodic, semantic, perceptual
+- **Storage tiers**: hot metadata (SQLite + sqlite-vec), cold archive (filesystem), archive index (vector index)
+- **Memory retrieval pipeline**: hot -> archive -> cold hydration with rerank + context packaging
+- **Local mode**: SQLite + sqlite-vec (optional) + filesystem
+- **Async-friendly** with sync convenience methods
+
+## Project Layout
+```
+memoryagent/
+  config.py            # Default system settings and retrieval thresholds
+  models.py            # Pydantic data models for memory items, queries, bundles
+  system.py            # MemorySystem entry point and wiring
+  retrieval.py         # Retrieval orchestration and reranking
+  confidence.py        # Confidence scoring components
+  policy.py            # Conversation + routing policies
+  indexers.py          # Episodic/semantic/perceptual indexers
+  workers.py           # Consolidation, archiving, rehydration, compaction
+  storage/
+    base.py            # Storage adapter interfaces
+    in_memory.py       # Simple in-memory vector + graph stores
+    local_disk.py      # SQLite metadata/features + sqlite-vec + file object store
+  examples/
+    minimal.py         # Basic usage example
+    openai_agent.py    # CLI OpenAI agent with memory retrieval
+    memory_api_server.py # Local API for memory + chat
+    memory_viz.html    # Web UI for chat + memory visualization
+```
+
+## Installation
+Python 3.10+ required.
+
+Development (sync deps from `uv.lock`):
+```bash
+uv sync
+```
+
+Use as a dependency:
+```bash
+uv add memoryagent-lib
+# or
+pip install memoryagent-lib
+```
+
+Optional extras:
+```bash
+uv add openai sqlite-vec
+# or
+pip install openai sqlite-vec
+```
+
+## Quick Start
+```python
+from memoryagent import MemoryEvent, MemorySystem
+
+memory = MemorySystem()
+owner = "user-001"
+
+memory.write(
+    MemoryEvent(
+        content="User prefers concise summaries about climate policy.",
+        type="semantic",
+        owner=owner,
+        tags=["preference", "summary"],
+        confidence=0.7,
+        stability=0.8,
+    )
+)
+
+bundle = memory.retrieve("What policy topics did we cover?", owner=owner)
+print(bundle.confidence.total)
+for block in bundle.blocks:
+    print(block.text)
+
+memory.flush(owner)
+```
+
+## Enable sqlite-vec (Local Vector Search)
+```python
+from memoryagent import MemorySystem, MemorySystemConfig
+
+config = MemorySystemConfig(
+    use_sqlite_vec=True,
+    vector_dim=1536,  # match your embedding model
+)
+
+memory = MemorySystem(config=config)
+```
+
+If sqlite-vec cannot be auto-loaded, set an explicit path:
+```python
+from pathlib import Path
+from memoryagent import MemorySystemConfig
+
+config = MemorySystemConfig(
+    use_sqlite_vec=True,
+    vector_dim=1536,
+    sqlite_vec_extension_path=Path("/path/to/sqlite_vec.dylib"),
+)
+```
+
+## Policies
+### Conversation storage policy
+`HeuristicMemoryPolicy` decides whether a turn should be stored and whether it becomes episodic or semantic memory.
+
+### Routing policy
+`MemoryRoutingPolicy` decides where a memory should be written:
+- **Hot** metadata store
+- **Vector index**
+- **Feature store** (perceptual)
+- **Cold** archive (via workers)
+
+## Background Workers
+- `ConsolidationWorker`: working → episodic/semantic
+- `ArchiverWorker`: hot → cold + archive index
+- `RehydratorWorker`: cold → hot (based on access)
+- `Compactor`: cleanup/TTL
+
+## Examples
+### OpenAI Agent (CLI)
+```bash
+python -m memoryagent.examples.openai_agent
+```
+- Uses OpenAI responses + embeddings.
+- Stores session transcript as a single working memory item.
+
+### Memory Visualization + API
+Start the API server:
+```bash
+python -m memoryagent.examples.memory_api_server
+```
+Open in browser:
+```
+http://127.0.0.1:8000/memory_viz.html
+```
+
+An example (System records semantic memory and updating working memory): 
+
+![Screenshot](https://raw.githubusercontent.com/jia-wei-zheng/MemoryAgent/master/Memory%20Agent%20_%20Live%20Console.jpeg?token=GHSAT0AAAAAADSXDGNDWOWFKKV777VMZM6U2MDPSTQ)
+
+
+The page calls:
+- `GET /api/memory?owner=user-001`
+- `POST /api/chat`
+
+## Data Stores
+- **Hot metadata**: `.memoryagent_hot.sqlite`
+- **Vector index**: `.memoryagent_vectors.sqlite` (sqlite-vec)
+- **Features**: `.memoryagent_features.sqlite`
+- **Cold archive**: `.memoryagent_cold/records/<owner>/YYYY/MM/DD/daily_notes.json`
+
+## Configuration
+See `memoryagent/config.py` for defaults:
+- `working_ttl_seconds`
+- `retrieval_plan` thresholds and budgets
+- `use_sqlite_vec`, `vector_dim`, `sqlite_vec_extension_path`
+
+## Notes
+- Working memory is stored as a single session transcript (updated each turn).
+- Episodic/semantic memories are candidates for cold archive.
+
+## License

memoryagent_lib-0.1.1/README.md

@@ -0,0 +1,171 @@
+# 
+
+<div align="center">
+  <img src="https://raw.githubusercontent.com/jia-wei-zheng/MemoryAgent/refs/heads/master/memoryagent_logo.jpg?token=GHSAT0AAAAAADSXDGNDXIWZF42E6S3D2XPG2MDPSDQ" alt="MemoryAgent" width="500">
+  <h1>MemoryAgent: An Open, Modular Memory Framework for Agents (Beta)</h1>
+</div>
+
+MemoryAgent is a reusable memory framework for LLM-based agent systems. It provides tiered memory (working, episodic, semantic, perceptual), hot/cold storage, archive indexing, confidence-based retrieval escalation, and optional local vector search via sqlite-vec.
+
+## Highlights
+- **Tiered memory**: working (TTL), episodic, semantic, perceptual
+- **Storage tiers**: hot metadata (SQLite + sqlite-vec), cold archive (filesystem), archive index (vector index)
+- **Memory retrieval pipeline**: hot -> archive -> cold hydration with rerank + context packaging
+- **Local mode**: SQLite + sqlite-vec (optional) + filesystem
+- **Async-friendly** with sync convenience methods
+
+## Project Layout
+```
+memoryagent/
+  config.py            # Default system settings and retrieval thresholds
+  models.py            # Pydantic data models for memory items, queries, bundles
+  system.py            # MemorySystem entry point and wiring
+  retrieval.py         # Retrieval orchestration and reranking
+  confidence.py        # Confidence scoring components
+  policy.py            # Conversation + routing policies
+  indexers.py          # Episodic/semantic/perceptual indexers
+  workers.py           # Consolidation, archiving, rehydration, compaction
+  storage/
+    base.py            # Storage adapter interfaces
+    in_memory.py       # Simple in-memory vector + graph stores
+    local_disk.py      # SQLite metadata/features + sqlite-vec + file object store
+  examples/
+    minimal.py         # Basic usage example
+    openai_agent.py    # CLI OpenAI agent with memory retrieval
+    memory_api_server.py # Local API for memory + chat
+    memory_viz.html    # Web UI for chat + memory visualization
+```
+
+## Installation
+Python 3.10+ required.
+
+Development (sync deps from `uv.lock`):
+```bash
+uv sync
+```
+
+Use as a dependency:
+```bash
+uv add memoryagent-lib
+# or
+pip install memoryagent-lib
+```
+
+Optional extras:
+```bash
+uv add openai sqlite-vec
+# or
+pip install openai sqlite-vec
+```
+
+## Quick Start
+```python
+from memoryagent import MemoryEvent, MemorySystem
+
+memory = MemorySystem()
+owner = "user-001"
+
+memory.write(
+    MemoryEvent(
+        content="User prefers concise summaries about climate policy.",
+        type="semantic",
+        owner=owner,
+        tags=["preference", "summary"],
+        confidence=0.7,
+        stability=0.8,
+    )
+)
+
+bundle = memory.retrieve("What policy topics did we cover?", owner=owner)
+print(bundle.confidence.total)
+for block in bundle.blocks:
+    print(block.text)
+
+memory.flush(owner)
+```
+
+## Enable sqlite-vec (Local Vector Search)
+```python
+from memoryagent import MemorySystem, MemorySystemConfig
+
+config = MemorySystemConfig(
+    use_sqlite_vec=True,
+    vector_dim=1536,  # match your embedding model
+)
+
+memory = MemorySystem(config=config)
+```
+
+If sqlite-vec cannot be auto-loaded, set an explicit path:
+```python
+from pathlib import Path
+from memoryagent import MemorySystemConfig
+
+config = MemorySystemConfig(
+    use_sqlite_vec=True,
+    vector_dim=1536,
+    sqlite_vec_extension_path=Path("/path/to/sqlite_vec.dylib"),
+)
+```
+
+## Policies
+### Conversation storage policy
+`HeuristicMemoryPolicy` decides whether a turn should be stored and whether it becomes episodic or semantic memory.
+
+### Routing policy
+`MemoryRoutingPolicy` decides where a memory should be written:
+- **Hot** metadata store
+- **Vector index**
+- **Feature store** (perceptual)
+- **Cold** archive (via workers)
+
+## Background Workers
+- `ConsolidationWorker`: working → episodic/semantic
+- `ArchiverWorker`: hot → cold + archive index
+- `RehydratorWorker`: cold → hot (based on access)
+- `Compactor`: cleanup/TTL
+
+## Examples
+### OpenAI Agent (CLI)
+```bash
+python -m memoryagent.examples.openai_agent
+```
+- Uses OpenAI responses + embeddings.
+- Stores session transcript as a single working memory item.
+
+### Memory Visualization + API
+Start the API server:
+```bash
+python -m memoryagent.examples.memory_api_server
+```
+Open in browser:
+```
+http://127.0.0.1:8000/memory_viz.html
+```
+
+An example (System records semantic memory and updating working memory): 
+
+![Screenshot](https://raw.githubusercontent.com/jia-wei-zheng/MemoryAgent/master/Memory%20Agent%20_%20Live%20Console.jpeg?token=GHSAT0AAAAAADSXDGNDWOWFKKV777VMZM6U2MDPSTQ)
+
+
+The page calls:
+- `GET /api/memory?owner=user-001`
+- `POST /api/chat`
+
+## Data Stores
+- **Hot metadata**: `.memoryagent_hot.sqlite`
+- **Vector index**: `.memoryagent_vectors.sqlite` (sqlite-vec)
+- **Features**: `.memoryagent_features.sqlite`
+- **Cold archive**: `.memoryagent_cold/records/<owner>/YYYY/MM/DD/daily_notes.json`
+
+## Configuration
+See `memoryagent/config.py` for defaults:
+- `working_ttl_seconds`
+- `retrieval_plan` thresholds and budgets
+- `use_sqlite_vec`, `vector_dim`, `sqlite_vec_extension_path`
+
+## Notes
+- Working memory is stored as a single session transcript (updated each turn).
+- Episodic/semantic memories are candidates for cold archive.
+
+## License

memoryagent_lib-0.1.1/memoryagent/__init__.py

@@ -0,0 +1,35 @@
+from memoryagent.config import MemorySystemConfig
+from memoryagent.models import (
+    ConfidenceReport,
+    MemoryBlock,
+    MemoryEvent,
+    MemoryItem,
+    MemoryQuery,
+    MemoryType,
+    RetrievalPlan,
+)
+from memoryagent.policy import (
+    ConversationMemoryPolicy,
+    HeuristicMemoryPolicy,
+    MemoryDecision,
+    MemoryRoutingPolicy,
+    RoutingDecision,
+)
+from memoryagent.system import MemorySystem
+
+__all__ = [
+    "MemorySystem",
+    "MemorySystemConfig",
+    "MemoryEvent",
+    "MemoryItem",
+    "MemoryQuery",
+    "MemoryType",
+    "MemoryBlock",
+    "RetrievalPlan",
+    "ConfidenceReport",
+    "ConversationMemoryPolicy",
+    "HeuristicMemoryPolicy",
+    "MemoryDecision",
+    "MemoryRoutingPolicy",
+    "RoutingDecision",
+]

memoryagent_lib-0.1.1/memoryagent/confidence.py

@@ -0,0 +1,82 @@
+from __future__ import annotations
+
+from datetime import datetime, timezone
+from typing import List
+
+from memoryagent.models import ConfidenceReport, MemoryQuery, ScoredMemory
+from memoryagent.utils import clamp, safe_div, unique_tokens
+
+
+def _semantic_relevance(results: List[ScoredMemory]) -> float:
+    if not results:
+        return 0.0
+    top_scores = [r.score for r in results[:5]]
+    return sum(top_scores) / len(top_scores)
+
+
+def _coverage(query: MemoryQuery, results: List[ScoredMemory]) -> float:
+    query_tokens = unique_tokens(query.text)
+    if not query_tokens:
+        return 0.0
+    covered = set()
+    for item in results[:5]:
+        covered |= unique_tokens(item.item.text())
+    return safe_div(len(query_tokens & covered), len(query_tokens))
+
+
+def _temporal_fit(results: List[ScoredMemory]) -> float:
+    if not results:
+        return 0.0
+    now = datetime.now(timezone.utc)
+    scores = []
+    for item in results[:5]:
+        age_days = max(0.0, (now - item.item.created_at).total_seconds() / 86400)
+        scores.append(1.0 / (1.0 + age_days))
+    return sum(scores) / len(scores)
+
+
+def _authority(results: List[ScoredMemory]) -> float:
+    if not results:
+        return 0.0
+    scores = [0.5 * r.item.authority + 0.5 * r.item.stability for r in results[:5]]
+    return sum(scores) / len(scores)
+
+
+def _consistency(results: List[ScoredMemory]) -> float:
+    if len(results) < 2:
+        return 0.5
+    tag_sets = [set(r.item.tags) for r in results[:5] if r.item.tags]
+    if not tag_sets:
+        return 0.4
+    overlap = set.intersection(*tag_sets) if len(tag_sets) > 1 else tag_sets[0]
+    union = set.union(*tag_sets) if len(tag_sets) > 1 else tag_sets[0]
+    return safe_div(len(overlap), len(union))
+
+
+def evaluate_confidence(query: MemoryQuery, results: List[ScoredMemory]) -> ConfidenceReport:
+    semantic = _semantic_relevance(results)
+    coverage = _coverage(query, results)
+    temporal = _temporal_fit(results)
+    authority = _authority(results)
+    consistency = _consistency(results)
+
+    total = clamp(0.35 * semantic + 0.2 * coverage + 0.2 * temporal + 0.15 * authority + 0.1 * consistency)
+
+    if total >= 0.75:
+        recommendation = "accept"
+    elif total >= 0.6:
+        recommendation = "escalate_archive"
+    elif total >= 0.45:
+        recommendation = "fetch_cold"
+    else:
+        recommendation = "uncertain"
+
+    return ConfidenceReport(
+        total=total,
+        semantic_relevance=semantic,
+        coverage=coverage,
+        temporal_fit=temporal,
+        authority=authority,
+        consistency=consistency,
+        recommendation=recommendation,
+    )

memoryagent_lib-0.1.1/memoryagent/config.py

@@ -0,0 +1,35 @@
+from __future__ import annotations
+
+from pathlib import Path
+from typing import Optional
+
+from pydantic import BaseModel, Field
+
+from memoryagent.models import RetrievalPlan
+
+
+class ConsolidationConfig(BaseModel):
+    archive_on_flush: bool = True
+    semantic_min_count: int = 2
+    perceptual_summary_limit: int = 5
+
+
+class MemorySystemConfig(BaseModel):
+    """System-wide configuration with sane local defaults."""
+
+    working_ttl_seconds: int = 3600
+    retrieval_plan: RetrievalPlan = Field(default_factory=RetrievalPlan)
+    consolidation: ConsolidationConfig = Field(default_factory=ConsolidationConfig)
+    cold_store_path: Path = Field(default_factory=lambda: Path(".memoryagent_cold"))
+    metadata_db_path: Path = Field(default_factory=lambda: Path(".memoryagent_hot.sqlite"))
+    feature_db_path: Path = Field(default_factory=lambda: Path(".memoryagent_features.sqlite"))
+    vector_db_path: Path = Field(default_factory=lambda: Path(".memoryagent_vectors.sqlite"))
+    vector_dim: int = 384
+    use_sqlite_vec: bool = False
+    sqlite_vec_extension_path: Optional[Path] = None
+    archive_index_path: Optional[Path] = None
+
+    def resolved_archive_path(self) -> Path:
+        if self.archive_index_path is not None:
+            return self.archive_index_path
+        return self.cold_store_path / "archive_index.json"

memoryagent_lib-0.1.1/memoryagent/consolidation.py

@@ -0,0 +1,5 @@
+from __future__ import annotations
+
+from memoryagent.workers import ConsolidationWorker
+
+__all__ = ["ConsolidationWorker"]

memoryagent_lib-0.1.1/memoryagent/examples/export_memory.py

@@ -0,0 +1,110 @@
+from __future__ import annotations
+
+import json
+import sqlite3
+from pathlib import Path
+from typing import Any, Dict, List
+
+ROOT = Path(__file__).resolve().parents[2]
+COLD_ROOT = ROOT / ".memoryagent_cold"
+HOT_DB = ROOT / ".memoryagent_hot.sqlite"
+FEATURE_DB = ROOT / ".memoryagent_features.sqlite"
+ARCHIVE_INDEX = COLD_ROOT / "archive_index.json"
+
+
+def load_hot() -> List[Dict[str, Any]]:
+    if not HOT_DB.exists():
+        return []
+    with sqlite3.connect(HOT_DB) as conn:
+        rows = conn.execute(
+            "SELECT id, type, owner, summary, content_json, tags_json, created_at, updated_at, last_accessed, tier, pointer_json, ttl_seconds, confidence, authority, stability FROM memory_items"
+        ).fetchall()
+    items = []
+    for row in rows:
+        (
+            item_id,
+            item_type,
+            owner,
+            summary,
+            content_json,
+            tags_json,
+            created_at,
+            updated_at,
+            last_accessed,
+            tier,
+            pointer_json,
+            ttl_seconds,
+            confidence,
+            authority,
+            stability,
+        ) = row
+        items.append(
+            {
+                "id": item_id,
+                "type": item_type,
+                "owner": owner,
+                "summary": summary,
+                "content": json.loads(content_json) if content_json else None,
+                "tags": json.loads(tags_json) if tags_json else [],
+                "created_at": created_at,
+                "updated_at": updated_at,
+                "last_accessed": last_accessed,
+                "tier": tier,
+                "pointer": json.loads(pointer_json) if pointer_json else {},
+                "ttl_seconds": ttl_seconds,
+                "confidence": confidence,
+                "authority": authority,
+                "stability": stability,
+            }
+        )
+    return items
+
+
+def load_features() -> List[Dict[str, Any]]:
+    if not FEATURE_DB.exists():
+        return []
+    with sqlite3.connect(FEATURE_DB) as conn:
+        rows = conn.execute("SELECT owner, created_at, payload_json FROM features").fetchall()
+    features = []
+    for owner, created_at, payload_json in rows:
+        features.append(
+            {
+                "owner": owner,
+                "created_at": created_at,
+                "payload": json.loads(payload_json),
+            }
+        )
+    return features
+
+
+def load_cold_records() -> List[Dict[str, Any]]:
+    if not COLD_ROOT.exists():
+        return []
+    records = []
+    records_root = COLD_ROOT / "records"
+    if not records_root.exists():
+        return []
+    for path in records_root.rglob("*.json"):
+        try:
+            records.append({"path": str(path.relative_to(ROOT)), "payload": json.loads(path.read_text())})
+        except Exception:
+            continue
+    return records
+
+
+def load_archive_index() -> Dict[str, Any]:
+    if not ARCHIVE_INDEX.exists():
+        return {}
+    return json.loads(ARCHIVE_INDEX.read_text())
+
+
+def get_memory_payload() -> Dict[str, Any]:
+    return {
+        "hot_items": load_hot(),
+        "features": load_features(),
+        "cold_records": load_cold_records(),
+        "archive_index": load_archive_index(),
+    }
+
+
+__all__ = ["get_memory_payload"]