agent-os-kernel 1.1.0__py3-none-any.whl → 1.2.0__py3-none-any.whl

modules/emk/RELEASE_NOTES.md

@@ -0,0 +1,82 @@
+# Release v0.1.0 - Initial Public Release
+
+**EMK (Episodic Memory Kernel)** - An immutable, append-only ledger of agent experiences for AI systems.
+
+## 🎯 Highlights
+
+- **Immutable Storage**: Episodes follow the Goal → Action → Result → Reflection pattern
+- **Pluggable Backends**: FileAdapter (JSONL) and ChromaDBAdapter (vector search)
+- **Minimal Dependencies**: Core requires only `pydantic` and `numpy`
+- **Research Ready**: Reproducible benchmarks, paper structure, HF Hub integration
+
+## 📦 Installation
+
+```bash
+# Basic installation
+pip install emk
+
+# With vector search support
+pip install emk[chromadb]
+
+# With Hugging Face Hub integration
+pip install emk[huggingface]
+
+# Everything
+pip install emk[all]
+```
+
+## 🚀 Quick Start
+
+```python
+from emk import Episode, FileAdapter
+
+store = FileAdapter("memories.jsonl")
+
+episode = Episode(
+    goal="Retrieve user data",
+    action="Query database",
+    result="Success",
+    reflection="Efficient query"
+)
+
+episode_id = store.store(episode)
+```
+
+## 📊 Performance (1000 episodes, seed=42)
+
+| Operation | Latency | Throughput |
+|-----------|---------|------------|
+| Episode Creation | 0.036ms | 27,694 ops/sec |
+| Storage Write | 1.53ms | 652 ops/sec |
+| Retrieval | 25.8ms | 39 ops/sec |
+| Indexer | 0.088ms | 11,346 ops/sec |
+
+## ✨ Features
+
+### Core
+- `Episode` - Immutable data structure with SHA-256 content-addressed IDs
+- `FileAdapter` - JSONL-based local storage with metadata filtering
+- `ChromaDBAdapter` - Vector similarity search (optional dependency)
+- `Indexer` - Tag extraction and search text generation
+
+### New in this release
+- **GitHub Actions CI/CD** - Multi-platform testing, PyPI publishing with OIDC
+- **Experiments** - Reproducible benchmark runner with fixed seeds
+- **Hugging Face Integration** - Upload/download episodes to HF Hub
+- **Research Paper** - Whitepaper structure for academic publication
+- **PEP 561** - Typed package support with `py.typed` marker
+
+## 📚 Documentation
+
+- [README](https://github.com/imran-siddique/emk#readme)
+- [Contributing Guide](https://github.com/imran-siddique/emk/blob/main/CONTRIBUTING.md)
+- [Changelog](https://github.com/imran-siddique/emk/blob/main/CHANGELOG.md)
+
+## 🔒 Security
+
+- CodeQL scanning on all PRs
+- Security policy in [SECURITY.md](https://github.com/imran-siddique/emk/blob/main/SECURITY.md)
+
+---
+
+**Full Changelog**: https://github.com/imran-siddique/emk/commits/v0.1.0

modules/emk/SECURITY.md

@@ -0,0 +1,52 @@
+# Security Policy
+
+## Supported Versions
+
+| Version | Supported          |
+| ------- | ------------------ |
+| 0.1.x   | :white_check_mark: |
+
+## Reporting a Vulnerability
+
+If you discover a security vulnerability in EMK, please report it responsibly:
+
+1. **Do NOT** open a public GitHub issue for security vulnerabilities
+2. Email the maintainer directly at: imran-siddique@users.noreply.github.com
+3. Include:
+   - Description of the vulnerability
+   - Steps to reproduce
+   - Potential impact
+   - Suggested fix (if any)
+
+### What to Expect
+
+- **Acknowledgment**: Within 48 hours
+- **Initial Assessment**: Within 1 week
+- **Resolution Timeline**: Depends on severity
+  - Critical: 24-72 hours
+  - High: 1 week
+  - Medium: 2 weeks
+  - Low: Next release
+
+### Security Best Practices
+
+When using EMK:
+
+1. **Validate Input**: Always validate episode content before storage
+2. **Access Control**: Implement proper access controls at the application layer
+3. **Data Sensitivity**: Do not store sensitive information (passwords, tokens) in episodes
+4. **File Permissions**: Ensure storage files have appropriate permissions
+5. **Dependencies**: Keep EMK and its dependencies updated
+
+## Security Measures in EMK
+
+- **Immutability**: Episodes cannot be modified after creation
+- **Content Hashing**: SHA-256 based episode IDs for integrity verification
+- **Minimal Dependencies**: Reduced attack surface
+- **CodeQL Scanning**: Automated security analysis on all PRs
+
+## Past Security Fixes
+
+### v0.1.0
+- Fixed `tempfile.mktemp()` usage vulnerability
+- Fixed potential metadata mutation issues

modules/emk/codecov.yml

@@ -0,0 +1,39 @@
+# Codecov Configuration
+# https://docs.codecov.com/docs/codecov-yaml
+
+coverage:
+  precision: 2
+  round: down
+  range: "70...100"
+  
+  status:
+    project:
+      default:
+        target: 70%
+        threshold: 2%
+    patch:
+      default:
+        target: 80%
+        threshold: 5%
+
+parsers:
+  gcov:
+    branch_detection:
+      conditional: yes
+      loop: yes
+      method: no
+      macro: no
+
+comment:
+  layout: "reach,diff,flags,files,footer"
+  behavior: default
+  require_changes: no
+  require_base: no
+  require_head: yes
+
+ignore:
+  - "tests/**/*"
+  - "experiments/**/*"
+  - "examples/**/*"
+  - "paper/**/*"
+  - "setup.py"

modules/emk/docs/MEMORY_MANAGEMENT.md

@@ -0,0 +1,285 @@
+# Memory Management Features
+
+This document describes the memory decay and negative memory features in EMK.
+
+## Overview
+
+EMK now includes two powerful memory management features:
+
+1. **Memory Decay & Compression (Sleep Cycle)**: Automatically summarize old episodes into semantic rules and compress storage
+2. **Negative Memory (Anti-Patterns)**: Explicitly track failures to avoid repeating mistakes
+
+These features address the core problem: **Infinite memory leads to infinite retrieval costs and confusing context.**
+
+## Feature 1: Memory Decay & Compression
+
+### The Problem
+
+As agents accumulate more and more episodes, several issues arise:
+- Storage costs increase linearly
+- Retrieval becomes slower and more expensive
+- Context windows become bloated with redundant information
+- Signal-to-noise ratio decreases
+
+### The Solution: Sleep Cycle
+
+The `MemoryCompressor` implements a "sleep cycle" utility that:
+1. Identifies old episodes based on a configurable age threshold
+2. Summarizes batches of episodes into semantic rules
+3. Stores compressed knowledge in a separate semantic rules store
+4. Optionally archives/deletes raw episodes (future enhancement)
+
+### Usage
+
+```python
+from emk import Episode, FileAdapter, MemoryCompressor
+
+# Initialize storage
+store = FileAdapter("episodes.jsonl")
+compressor = MemoryCompressor(
+    store=store,
+    age_threshold_days=30,      # Episodes older than 30 days
+    compression_batch_size=50,   # Compress 50 episodes at a time
+    rules_filepath="semantic_rules.jsonl"
+)
+
+# Run compression cycle (dry run first)
+dry_result = compressor.compress_old_episodes(dry_run=True)
+print(f"Would compress {dry_result['compressed_count']} episodes")
+
+# Run actual compression
+result = compressor.compress_old_episodes(dry_run=False)
+print(f"Compressed {result['compressed_count']} episodes into {result['rules_created']} rules")
+
+# Retrieve semantic rules
+rules = compressor.retrieve_rules(limit=10)
+for rule in rules:
+    print(f"Rule: {rule.rule}")
+    print(f"Confidence: {rule.confidence}")
+    print(f"Source episodes: {len(rule.source_episode_ids)}")
+```
+
+### Custom Summarization
+
+You can provide your own summarization function:
+
+```python
+def custom_summarizer(episodes):
+    """Custom logic to summarize episodes."""
+    # Your AI model or logic here
+    return "Custom summary based on episodes"
+
+compressor.compress_old_episodes(summarizer=custom_summarizer)
+```
+
+### Benefits
+
+- **Reduced Storage**: Compress 100 episodes into 1 semantic rule
+- **Faster Retrieval**: Query compressed knowledge instead of raw logs
+- **Preserved Context**: Keep insights without overwhelming detail
+- **Scale by Subtraction**: No need for infinite context windows
+
+## Feature 2: Negative Memory (Anti-Patterns)
+
+### The Problem
+
+Traditional memory systems only store what works. Agents repeatedly try the same failing approaches, wasting time and resources.
+
+### The Solution: Explicit Failure Tracking
+
+EMK now explicitly tracks failures as "DO NOT TOUCH" patterns:
+- Mark episodes as failures with reasons
+- Query for both successes AND failures
+- Prune search space by avoiding known failures
+
+### Usage
+
+#### Marking Failures
+
+```python
+from emk import Episode, FileAdapter
+
+store = FileAdapter("episodes.jsonl")
+
+# Method 1: Mark during creation
+failed_episode = Episode(
+    goal="Connect to external API",
+    action="GET https://api.example.com/data",
+    result="Connection timeout",
+    reflection="Network request failed",
+    metadata={"is_failure": True, "failure_reason": "Connection timeout"}
+)
+store.store(failed_episode)
+
+# Method 2: Mark existing episode (creates new immutable instance)
+episode = Episode(
+    goal="Deploy application",
+    action="Run deployment script",
+    result="Failed",
+    reflection="Container failed to start"
+)
+failed = episode.mark_as_failure(reason="Docker image not found")
+store.store(failed)
+```
+
+#### Querying Patterns
+
+```python
+# Get only successful patterns
+successes = store.retrieve_successes(limit=10)
+
+# Get only failures (anti-patterns)
+failures = store.retrieve_failures(limit=10)
+
+# Get both for comprehensive analysis
+patterns = store.retrieve_with_anti_patterns(limit=10)
+print(f"Successes: {len(patterns['successes'])}")
+print(f"Failures: {len(patterns['failures'])}")
+
+# What WORKS
+for ep in patterns['successes']:
+    print(f"✓ {ep.goal} → {ep.result}")
+
+# What DOESN'T work (DO NOT TOUCH)
+for ep in patterns['failures']:
+    print(f"✗ {ep.goal} → {ep.result}")
+    print(f"  Reason: {ep.metadata['failure_reason']}")
+```
+
+### Benefits
+
+- **Avoid Repetition**: Don't try the same failing approach twice
+- **Faster Learning**: Learn from both successes and failures
+- **Pruned Search Space**: Eliminate known dead-ends instantly
+- **Explicit Anti-Patterns**: "DO NOT TOUCH" vectors prevent mistakes
+
+## Combined Power
+
+Using both features together:
+
+```python
+from emk import Episode, FileAdapter, MemoryCompressor
+
+store = FileAdapter("episodes.jsonl")
+compressor = MemoryCompressor(store, age_threshold_days=30)
+
+# Store episodes (mix of success and failure)
+for i in range(100):
+    episode = Episode(...)
+    if should_fail(episode):
+        episode = episode.mark_as_failure(reason="...")
+    store.store(episode)
+
+# Compress old episodes into rules
+result = compressor.compress_old_episodes()
+print(f"Compressed {result['compressed_count']} episodes")
+
+# Retrieve compressed knowledge
+rules = compressor.retrieve_rules()
+for rule in rules:
+    if rule.metadata.get('failure_count', 0) > 0:
+        print(f"⚠️  Rule includes failures: {rule.rule}")
+    else:
+        print(f"✓ Successful pattern: {rule.rule}")
+
+# Query for current patterns (not compressed yet)
+patterns = store.retrieve_with_anti_patterns(limit=10)
+print(f"Current successes: {len(patterns['successes'])}")
+print(f"Current failures: {len(patterns['failures'])}")
+```
+
+## API Reference
+
+### SemanticRule
+
+Immutable compressed knowledge from multiple episodes.
+
+**Fields:**
+- `rule` (str): The compressed semantic knowledge
+- `source_episode_ids` (List[str]): IDs of episodes that contributed
+- `created_at` (datetime): When the rule was created
+- `context` (Optional[str]): Context about when/how this rule applies
+- `confidence` (float): Confidence score (0.0 to 1.0)
+- `metadata` (Dict): Additional context or tags
+- `rule_id` (str): Unique hash-based identifier (auto-generated)
+
+### MemoryCompressor
+
+**Constructor:**
+```python
+MemoryCompressor(
+    store: VectorStoreAdapter,
+    age_threshold_days: int = 30,
+    compression_batch_size: int = 50,
+    rules_filepath: Optional[str] = None
+)
+```
+
+**Methods:**
+- `identify_old_episodes(episodes)`: Filter episodes by age
+- `summarize_episodes(episodes, summarizer=None)`: Create semantic rule
+- `store_rule(rule)`: Store semantic rule
+- `retrieve_rules(filters=None, limit=100)`: Query semantic rules
+- `compress_old_episodes(summarizer=None, dry_run=False)`: Execute compression cycle
+
+### Episode Extensions
+
+**New Methods:**
+- `is_failure()`: Check if episode is marked as a failure
+- `mark_as_failure(reason=None)`: Create new episode marked as failure (immutable)
+
+### VectorStoreAdapter Extensions
+
+**New Methods:**
+- `retrieve_failures(query_embedding=None, filters=None, limit=10)`: Get only failures
+- `retrieve_successes(query_embedding=None, filters=None, limit=10)`: Get only successes
+- `retrieve_with_anti_patterns(query_embedding=None, filters=None, limit=10, include_failures=True)`: Get both
+
+## Best Practices
+
+1. **Regular Compression**: Run sleep cycles periodically (e.g., daily or weekly)
+2. **Failure Documentation**: Always include `failure_reason` when marking failures
+3. **Custom Summarizers**: Use LLMs for high-quality summarization
+4. **Incremental Compression**: Start with small batch sizes and adjust
+5. **Dry Runs**: Test compression with `dry_run=True` before actual compression
+6. **Query Both**: Always check both successes and failures before taking action
+
+## Performance Considerations
+
+- **Memory Decay**: Reduces storage by 50-100x depending on compression ratio
+- **Anti-Patterns**: No performance overhead - uses metadata filtering
+- **Retrieval**: Semantic rules are 10-100x faster to retrieve than raw episodes
+- **Batch Size**: Larger batches = fewer rules but less granular compression
+
+## Future Enhancements
+
+Potential future improvements:
+- Automatic archival/deletion of compressed episodes
+- Vector embeddings for semantic rules
+- Time-based rule decay (rules can expire too)
+- Hierarchical compression (compress rules into meta-rules)
+- Integration with LLM-based summarization
+- Rule merging and deduplication
+
+## Examples
+
+See `examples/memory_features_demo.py` for a complete working example demonstrating both features.
+
+## Testing
+
+Run the test suite:
+
+```bash
+# Test memory decay
+pytest tests/test_sleep_cycle.py -v
+
+# Test negative memory
+pytest tests/test_negative_memory.py -v
+pytest tests/test_store_anti_patterns.py -v
+
+# Test semantic rules
+pytest tests/test_semantic_rules.py -v
+
+# Run all tests
+pytest tests/ -v
+```

modules/emk/emk/__init__.py

@@ -0,0 +1,106 @@
+"""
+emk - Episodic Memory Kernel.
+
+An immutable, append-only ledger of agent experiences for AI systems.
+
+This package provides a Layer 1 primitive for storing agent experiences
+as structured episodes following the pattern: Goal → Action → Result → Reflection.
+
+Example:
+    >>> from emk import Episode, FileAdapter
+    >>> store = FileAdapter("memories.jsonl")
+    >>> episode = Episode(
+    ...     goal="Retrieve user data",
+    ...     action="Query database",
+    ...     result="Success",
+    ...     reflection="Efficient query"
+    ... )
+    >>> episode_id = store.store(episode)
+
+Attributes:
+    __version__: Package version string.
+    __author__: Package author.
+    __license__: Package license.
+
+See Also:
+    - GitHub: https://github.com/imran-siddique/emk
+    - Documentation: https://github.com/imran-siddique/emk#readme
+"""
+
+from typing import TYPE_CHECKING, List
+
+__version__ = "0.1.0"
+__author__ = "Imran Siddique"
+__license__ = "MIT"
+
+# Core exports - always available
+from emk.schema import Episode, SemanticRule
+from emk.store import VectorStoreAdapter, FileAdapter
+from emk.indexer import Indexer
+from emk.sleep_cycle import MemoryCompressor
+
+# Define explicit public API
+__all__: List[str] = [
+    # Metadata
+    "__version__",
+    "__author__",
+    "__license__",
+    # Core classes
+    "Episode",
+    "SemanticRule",
+    "VectorStoreAdapter", 
+    "FileAdapter",
+    "Indexer",
+    "MemoryCompressor",
+]
+
+# Optional ChromaDB adapter - only import if chromadb is installed
+try:
+    from emk.store import ChromaDBAdapter
+    __all__.append("ChromaDBAdapter")
+except ImportError:
+    # ChromaDB not installed, ChromaDBAdapter will not be available
+    if TYPE_CHECKING:
+        from emk.store import ChromaDBAdapter  # noqa: F401
+
+# Optional Hugging Face utilities - only import if huggingface_hub is installed
+try:
+    from emk.hf_utils import (
+        upload_episodes_to_hub,
+        download_episodes_from_hub,
+        push_experiment_results,
+    )
+    __all__.extend([
+        "upload_episodes_to_hub",
+        "download_episodes_from_hub", 
+        "push_experiment_results",
+    ])
+except ImportError:
+    # huggingface_hub not installed
+    pass
+
+
+def get_version_info() -> dict:
+    """
+    Get detailed version information about the emk package.
+    
+    Returns:
+        dict: A dictionary containing version, author, license, and
+            available optional features.
+    
+    Example:
+        >>> import emk
+        >>> info = emk.get_version_info()
+        >>> print(info['version'])
+        '0.1.0'
+    """
+    features = {
+        "chromadb": "ChromaDBAdapter" in __all__,
+        "huggingface": "upload_episodes_to_hub" in __all__,
+    }
+    return {
+        "version": __version__,
+        "author": __author__,
+        "license": __license__,
+        "features": features,
+    }