comet-memory 0.1.0__tar.gz

comet_memory-0.1.0/PKG-INFO

@@ -0,0 +1,215 @@
+Metadata-Version: 2.4
+Name: comet-memory
+Version: 0.1.0
+Summary: CoMeT — Cognitive Memory Tree: Hierarchical memory system for LLM agents
+Author: Dirac-Robot
+License: MIT
+Keywords: memory,llm,agent,rag,cognitive
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Programming Language :: Python :: 3.12
+Requires-Python: >=3.12
+Description-Content-Type: text/markdown
+Requires-Dist: ato>=2.2.0
+Requires-Dist: chromadb>=1.4.1
+Requires-Dist: langchain>=1.2.2
+Requires-Dist: langchain-openai>=1.1.7
+Requires-Dist: langgraph>=1.0.5
+Requires-Dist: loguru>=0.7.3
+Requires-Dist: openai>=2.16.0
+Requires-Dist: pydantic>=2.12.5
+
+# ☄️ CoMeT — Cognitive Memory Tree
+
+**Lossless structured memory for AI agents.**
+
+CoMeT compresses long conversations into a navigable tree of memory nodes.  
+Unlike naive summarization that loses details, CoMeT preserves raw data behind structured summaries — agents read summaries first, then drill into raw data only when needed.
+
+## Architecture
+
+```
+User Input
+    │
+    ▼
+┌─────────┐    SLM (fast)     ┌───────────┐
+│  Sensor  │ ───────────────▶ │ L1 Buffer │
+└─────────┘   entity/intent   └─────┬─────┘
+                                    │ cognitive load trigger
+                                    ▼
+                              ┌───────────┐
+                              │ Compacter │  LLM (slow)
+                              └─────┬─────┘
+                                    │ summary + trigger + recall_mode + tags
+                                    ▼
+                         ┌──────────┴──────────┐
+                         │                     │
+                   ┌───────────┐        ┌─────────────┐
+                   │   Store   │        │ VectorIndex │  ChromaDB
+                   │  depth 0-2│        │ (dual-path) │  summary + trigger
+                   └───────────┘        └──────┬──────┘
+                                               │ semantic search
+                                               ▼
+                                         ┌───────────┐
+                                         │ Retriever │  RRF fusion
+                                         └───────────┘
+```
+
+### Dual-Speed Layer
+- **Fast (Sensor)**: SLM extracts entities/intent per turn, detects topic shifts via cognitive load assessment
+- **Slow (Compacter)**: Main LLM structures accumulated L1 buffer into `MemoryNode` with summary, trigger, recall mode, and topic tags
+
+### Dynamic Resolution (depth 0 → 1 → 2)
+
+| Depth | Content | Use Case |
+|-------|---------|----------|
+| 0 | Summary + Trigger | Agent's initial context window |
+| 1 | + Topic tags + Links | Navigation / node selection |
+| 2 | Full raw data + Links | Fact retrieval |
+
+### Recall Mode
+
+Each memory node is classified by `recall_mode` at compaction time:
+
+| Mode | Behavior | Examples |
+|------|----------|----------|
+| `passive` | Always included in context window | User identity, persistent preferences |
+| `active` | Retrieved on-demand via semantic search | Factual details, decisions, events |
+| `both` | Always in context + searchable via RAG | Core constraints with retrievable details |
+
+### Dual-Path RAG Retrieval
+
+CoMeT embeds both `summary` (what the node contains) and `trigger` (when to recall it) into separate vector collections. At query time:
+
+1. **QueryAnalyzer** decomposes the query into `semantic_query` + `search_intent`
+2. **Summary path**: matches what the information is about
+3. **Trigger path**: matches when the information would be needed
+4. **ScoreFusion** (Reciprocal Rank Fusion): merges results from both paths
+
+Triggers are written from the **LLM's perspective** (`"내가 ~정보가 필요할 때"`) rather than user-centric, enabling broader semantic matching even without explicit user requests.
+
+### Topic-Aware Auto-Linking
+Nodes share a global topic tag set. The compacter reuses existing tags when possible, enabling automatic bidirectional linking between related nodes across different conversation segments.
+
+## Benchmark (52 turns, 5 conversations, 10 questions)
+
+| Method | Context Cost | Accuracy |
+|--------|-------------|----------|
+| Full Context Injection | 5,198 chars (100%) | 10/10 |
+| **CoMeT** | **1,397 chars (27%)** | **9/10** |
+| Naive Summary | 1,179 chars (23%) | 1/10 |
+
+- CoMeT uses **27% of the tokens** while retaining **90% accuracy**
+- 6/10 questions required **link traversal** (agent read 2-3 nodes)
+- Cross-topic questions: CoMeT 5/5 vs Naive 0/5
+
+## Quick Start
+
+### Session Memory (within a conversation)
+
+```python
+from comet import CoMeT, scope
+
+@scope
+def main(config):
+    memo = CoMeT(config)
+
+    # Add conversation turns
+    memo.add("B200 4대로 월드모델 학습 가능할까?")
+    memo.add("2B면 충분하고 커봐야 8B")
+    memo.add("DPO 데이터는 negative를 syntax error로 구성했어")
+
+    # Force compact remaining buffer
+    memo.force_compact()
+
+    # Navigation
+    for node in memo.list_memories():
+        print(memo.read_memory(node['node_id'], depth=0))
+
+    # Agent tools (LangChain compatible)
+    tools = memo.get_tools()
+    # → get_memory_index, read_memory_node, search_memory
+
+main()
+```
+
+### Cross-Session RAG Retrieval
+
+```python
+from comet import CoMeT, scope
+
+@scope
+def main(config):
+    config.retrieval.vector_db_path = './memory_store/vectors'
+
+    memo = CoMeT(config)
+
+    # Ingest turns (auto-indexed to VectorIndex on compaction)
+    memo.add("JWT 액세스 토큰 만료는 15분, 리프레시는 7일로 설정")
+    memo.force_compact()
+
+    # Semantic retrieval across all sessions
+    results = memo.retrieve("토큰 만료 설정이 어떻게 되어있어?")
+    for r in results:
+        print(f"[{r.node.node_id}] score={r.relevance_score:.4f}")
+        print(f"  {r.node.summary}")
+
+    # Agent tools include retrieve_memory when retrieval is configured
+    tools = memo.get_tools()
+    # → get_memory_index, read_memory_node, search_memory, retrieve_memory
+
+main()
+```
+
+## Configuration ([ato](https://github.com/Dirac-Robot/ato))
+
+```python
+# comet/config.py
+@scope.observe(default=True)
+def default(config):
+    config.slm_model = 'gpt-4o-mini'
+    config.main_model = 'gpt-4o'
+    config.compacting.load_threshold = 3
+    config.compacting.max_l1_buffer = 5
+
+    # RAG retrieval (enabled when retrieval block exists)
+    config.retrieval.embedding_model = 'text-embedding-3-small'
+    config.retrieval.vector_backend = 'chroma'
+    config.retrieval.vector_db_path = './memory_store/vectors'
+    config.retrieval.top_k = 5
+
+@scope.observe()
+def local_slm(config):
+    config.slm_model = 'ollama/gemma3:4b'
+
+@scope.observe()
+def aggressive(config):
+    config.compacting.load_threshold = 2
+    config.compacting.max_l1_buffer = 3
+```
+
+```bash
+# Use default
+python main.py
+
+# Local SLM + aggressive compacting
+python main.py local_slm aggressive
+```
+
+## Project Structure
+
+```
+comet/
+├── orchestrator.py    # CoMeT main class
+├── sensor.py          # L1 extraction + cognitive load (SLM)
+├── compacter.py       # L1→L2 structuring + auto-linking (LLM)
+├── storage.py         # JSON key-value store + navigation
+├── schemas.py         # MemoryNode, L1Memory, CognitiveLoad, RetrievalResult
+├── config.py          # ato scope configuration
+├── vector_index.py    # ChromaDB dual-collection vector store
+├── retriever.py       # QueryAnalyzer + ScoreFusion + Retriever
+└── templates/
+    ├── compacting.txt      # Memory structuring prompt
+    └── query_analysis.txt  # Query decomposition prompt
+```

comet_memory-0.1.0/README.md

@@ -0,0 +1,193 @@
+# ☄️ CoMeT — Cognitive Memory Tree
+
+**Lossless structured memory for AI agents.**
+
+CoMeT compresses long conversations into a navigable tree of memory nodes.  
+Unlike naive summarization that loses details, CoMeT preserves raw data behind structured summaries — agents read summaries first, then drill into raw data only when needed.
+
+## Architecture
+
+```
+User Input
+    │
+    ▼
+┌─────────┐    SLM (fast)     ┌───────────┐
+│  Sensor  │ ───────────────▶ │ L1 Buffer │
+└─────────┘   entity/intent   └─────┬─────┘
+                                    │ cognitive load trigger
+                                    ▼
+                              ┌───────────┐
+                              │ Compacter │  LLM (slow)
+                              └─────┬─────┘
+                                    │ summary + trigger + recall_mode + tags
+                                    ▼
+                         ┌──────────┴──────────┐
+                         │                     │
+                   ┌───────────┐        ┌─────────────┐
+                   │   Store   │        │ VectorIndex │  ChromaDB
+                   │  depth 0-2│        │ (dual-path) │  summary + trigger
+                   └───────────┘        └──────┬──────┘
+                                               │ semantic search
+                                               ▼
+                                         ┌───────────┐
+                                         │ Retriever │  RRF fusion
+                                         └───────────┘
+```
+
+### Dual-Speed Layer
+- **Fast (Sensor)**: SLM extracts entities/intent per turn, detects topic shifts via cognitive load assessment
+- **Slow (Compacter)**: Main LLM structures accumulated L1 buffer into `MemoryNode` with summary, trigger, recall mode, and topic tags
+
+### Dynamic Resolution (depth 0 → 1 → 2)
+
+| Depth | Content | Use Case |
+|-------|---------|----------|
+| 0 | Summary + Trigger | Agent's initial context window |
+| 1 | + Topic tags + Links | Navigation / node selection |
+| 2 | Full raw data + Links | Fact retrieval |
+
+### Recall Mode
+
+Each memory node is classified by `recall_mode` at compaction time:
+
+| Mode | Behavior | Examples |
+|------|----------|----------|
+| `passive` | Always included in context window | User identity, persistent preferences |
+| `active` | Retrieved on-demand via semantic search | Factual details, decisions, events |
+| `both` | Always in context + searchable via RAG | Core constraints with retrievable details |
+
+### Dual-Path RAG Retrieval
+
+CoMeT embeds both `summary` (what the node contains) and `trigger` (when to recall it) into separate vector collections. At query time:
+
+1. **QueryAnalyzer** decomposes the query into `semantic_query` + `search_intent`
+2. **Summary path**: matches what the information is about
+3. **Trigger path**: matches when the information would be needed
+4. **ScoreFusion** (Reciprocal Rank Fusion): merges results from both paths
+
+Triggers are written from the **LLM's perspective** (`"내가 ~정보가 필요할 때"`) rather than user-centric, enabling broader semantic matching even without explicit user requests.
+
+### Topic-Aware Auto-Linking
+Nodes share a global topic tag set. The compacter reuses existing tags when possible, enabling automatic bidirectional linking between related nodes across different conversation segments.
+
+## Benchmark (52 turns, 5 conversations, 10 questions)
+
+| Method | Context Cost | Accuracy |
+|--------|-------------|----------|
+| Full Context Injection | 5,198 chars (100%) | 10/10 |
+| **CoMeT** | **1,397 chars (27%)** | **9/10** |
+| Naive Summary | 1,179 chars (23%) | 1/10 |
+
+- CoMeT uses **27% of the tokens** while retaining **90% accuracy**
+- 6/10 questions required **link traversal** (agent read 2-3 nodes)
+- Cross-topic questions: CoMeT 5/5 vs Naive 0/5
+
+## Quick Start
+
+### Session Memory (within a conversation)
+
+```python
+from comet import CoMeT, scope
+
+@scope
+def main(config):
+    memo = CoMeT(config)
+
+    # Add conversation turns
+    memo.add("B200 4대로 월드모델 학습 가능할까?")
+    memo.add("2B면 충분하고 커봐야 8B")
+    memo.add("DPO 데이터는 negative를 syntax error로 구성했어")
+
+    # Force compact remaining buffer
+    memo.force_compact()
+
+    # Navigation
+    for node in memo.list_memories():
+        print(memo.read_memory(node['node_id'], depth=0))
+
+    # Agent tools (LangChain compatible)
+    tools = memo.get_tools()
+    # → get_memory_index, read_memory_node, search_memory
+
+main()
+```
+
+### Cross-Session RAG Retrieval
+
+```python
+from comet import CoMeT, scope
+
+@scope
+def main(config):
+    config.retrieval.vector_db_path = './memory_store/vectors'
+
+    memo = CoMeT(config)
+
+    # Ingest turns (auto-indexed to VectorIndex on compaction)
+    memo.add("JWT 액세스 토큰 만료는 15분, 리프레시는 7일로 설정")
+    memo.force_compact()
+
+    # Semantic retrieval across all sessions
+    results = memo.retrieve("토큰 만료 설정이 어떻게 되어있어?")
+    for r in results:
+        print(f"[{r.node.node_id}] score={r.relevance_score:.4f}")
+        print(f"  {r.node.summary}")
+
+    # Agent tools include retrieve_memory when retrieval is configured
+    tools = memo.get_tools()
+    # → get_memory_index, read_memory_node, search_memory, retrieve_memory
+
+main()
+```
+
+## Configuration ([ato](https://github.com/Dirac-Robot/ato))
+
+```python
+# comet/config.py
+@scope.observe(default=True)
+def default(config):
+    config.slm_model = 'gpt-4o-mini'
+    config.main_model = 'gpt-4o'
+    config.compacting.load_threshold = 3
+    config.compacting.max_l1_buffer = 5
+
+    # RAG retrieval (enabled when retrieval block exists)
+    config.retrieval.embedding_model = 'text-embedding-3-small'
+    config.retrieval.vector_backend = 'chroma'
+    config.retrieval.vector_db_path = './memory_store/vectors'
+    config.retrieval.top_k = 5
+
+@scope.observe()
+def local_slm(config):
+    config.slm_model = 'ollama/gemma3:4b'
+
+@scope.observe()
+def aggressive(config):
+    config.compacting.load_threshold = 2
+    config.compacting.max_l1_buffer = 3
+```
+
+```bash
+# Use default
+python main.py
+
+# Local SLM + aggressive compacting
+python main.py local_slm aggressive
+```
+
+## Project Structure
+
+```
+comet/
+├── orchestrator.py    # CoMeT main class
+├── sensor.py          # L1 extraction + cognitive load (SLM)
+├── compacter.py       # L1→L2 structuring + auto-linking (LLM)
+├── storage.py         # JSON key-value store + navigation
+├── schemas.py         # MemoryNode, L1Memory, CognitiveLoad, RetrievalResult
+├── config.py          # ato scope configuration
+├── vector_index.py    # ChromaDB dual-collection vector store
+├── retriever.py       # QueryAnalyzer + ScoreFusion + Retriever
+└── templates/
+    ├── compacting.txt      # Memory structuring prompt
+    └── query_analysis.txt  # Query decomposition prompt
+```

comet_memory-0.1.0/comet/__init__.py

@@ -0,0 +1,27 @@
+"""CoMeT: Cognitive Memory OS - Dynamic Resolution Memory System"""
+from comet.schemas import MemoryNode, CognitiveLoad, CoMeTState, L1Memory, RetrievalResult
+from comet.sensor import CognitiveSensor
+from comet.compacter import MemoryCompacter
+from comet.storage import MemoryStore
+from comet.vector_index import VectorIndex
+from comet.retriever import Retriever
+from comet.consolidator import Consolidator
+from comet.orchestrator import CoMeT, MessageInput
+from comet.config import scope
+
+__all__ = [
+    'CoMeT',
+    'MemoryNode',
+    'CognitiveLoad',
+    'CoMeTState',
+    'L1Memory',
+    'RetrievalResult',
+    'CognitiveSensor',
+    'MemoryCompacter',
+    'MemoryStore',
+    'VectorIndex',
+    'Retriever',
+    'Consolidator',
+    'scope',
+]
+