omem-os 0.0.1__tar.gz

omem_os-0.0.1/.cursorrules

@@ -0,0 +1,38 @@
+# OMem Cursor Rules
+
+You are an expert AI assistant helping developers build and optimize **OMem**—a fast, intelligent memory system for AI agents. 
+
+## 1. Project Context
+- **Objective**: Provide AI agents with human-like memory (context-aware, automatically scoring importance, consolidating generic information, and forgetting irrelevant data).
+- **Core Languages**: Python (3.9+) and Rust.
+- **Primary Dependencies**: `sentence-transformers`, `faiss-cpu`, `sqlite3`, `pydantic`.
+
+## 2. Architecture & Design
+OMem consists of three main tiers:
+1. **API Layer (`omem/api.py`)**: Unified entry mechanism for the user (`OMem` class).
+2. **Logic & Engine (`omem/core/`)**: 
+   - `brain/`: Advanced cognitive functions (dreaming/consolidation, importance scoring, conflict resolution, reflection).
+   - `engine/`: Execution pathways (ingestion/adding, RAG retrieval).
+3. **Performance Core (`rust/`)**: Rust-based engine compiled via PyO3 to `omem_rust`. Handles batched SIMD hybrid scoring (vector + keyword + recency + importance).
+
+## 3. Interaction with LLMs and AI Agents
+OMem includes a **Model Context Protocol (MCP)** server natively.
+- **Entry point**: `omem serve` or `omem.integrations.mcp_server`.
+- If a user asks to integrate Claude Desktop, Cursor MCP, or an agent (like CrewAI / LangChain), point them to `omem/integrations/`.
+
+## 4. Coding Standards (Python)
+- Use **strict type hints** (`Dict`, `List`, `Optional`, etc.) for all function signatures.
+- Document any complex logic or reasoning within docstrings using clean markdown.
+- Treat `core/brain` operations as modular heuristics. If you add a feature, ensure it does not slow down the synchronous `.add` or `.recall` loops unnecessarily.
+
+## 5. Coding Standards (Rust)
+- Any operations processing entire arrays of memories or computing heavy distance metrics MUST go in the Rust core.
+- Keep the boundary between Python and Rust thin (pass primitive arrays or simple structs).
+- Remember to instruct the user to run `pip install -e .` if Rust files are modified.
+
+## 6. Testing
+- `pytest tests/ -v` handles the test suite. 
+- When changing cognitive layers or adding new integrations, provide test cases simulating agent flows.
+
+## 7. Formatting & Brand Tone
+- Maintain a highly professional but "cute and approachable" brand tone in user-facing documentation. Avoid overly academic jargon without practical examples.

omem_os-0.0.1/.dockerignore

@@ -0,0 +1,66 @@
+# Python
+__pycache__/
+*.py[cod]
+*$py.class
+*.so
+.Python
+*.egg-info/
+dist/
+build/
+.eggs/
+
+# Virtual environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# IDEs
+.vscode/
+.idea/
+*.swp
+*.swo
+*~
+
+# Testing
+.pytest_cache/
+.coverage
+htmlcov/
+.tox/
+
+# Git
+.git/
+.gitignore
+.gitattributes
+
+# Documentation
+*.md
+docs/
+
+# CI/CD
+.github/
+
+# Rust build artifacts (we copy only the compiled libs)
+rust/target/debug/
+rust/target/release/*.d
+rust/target/release/*.rlib
+rust/Cargo.lock
+
+# Data and logs
+*.db
+*.db-shm
+*.db-wal
+logs/
+data/
+
+# OS
+.DS_Store
+Thumbs.db
+
+# Development
+examples/
+tests/test_*.json
+benchmarks/

omem_os-0.0.1/.env.example

@@ -0,0 +1,38 @@
+# OMem Configuration
+
+# Database Configuration
+# Location where OMem stores memory data
+# Default: ~/.omem/brain.db
+OMEM_DB_PATH=~/.omem/brain.db
+
+# Embedding Model Configuration
+# Sentence-transformer model used for generating embeddings
+# Options: all-MiniLM-L6-v2, all-mpnet-base-v2, paraphrase-multilingual-MiniLM-L12-v2
+# Default: all-MiniLM-L6-v2
+OMEM_MODEL=all-MiniLM-L6-v2
+
+# Performance Configuration
+# LRU cache size for hot memories (in number of entries)
+# Higher values consume more memory but improve retrieval speed
+# Default: 128000
+OMEM_CACHE_SIZE=128000
+
+# Database connection pool size for concurrent operations
+# Increase for high-concurrency workloads
+# Default: 5
+OMEM_POOL_SIZE=5
+
+# Logging Configuration
+# Log level for OMem operations
+# Options: DEBUG, INFO, WARNING, ERROR, CRITICAL
+# Default: INFO
+OMEM_LOG_LEVEL=INFO
+
+# MCP Server Configuration (optional)
+# Port for MCP server when running `omem serve`
+# Default: 3000
+OMEM_MCP_PORT=3000
+
+# Host for MCP server
+# Default: localhost
+OMEM_MCP_HOST=localhost

omem_os-0.0.1/.github/workflows/ci.yml

@@ -0,0 +1,36 @@
+name: CI
+
+on:
+  push:
+    branches: [ "dev", "staging", "prod", "main", "master" ]
+  pull_request:
+    branches: [ "dev", "staging", "prod", "main", "master" ]
+
+jobs:
+  test:
+    name: Test and Lint
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.11"
+          
+      - name: Install Rust
+        uses: dtolnay/rust-toolchain@stable
+        
+      - name: Install Dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install -e ".[dev]"
+          pip install ruff
+          
+      - name: Lint with Ruff
+        run: |
+          ruff check .
+          
+      - name: Test with pytest
+        run: |
+          pytest tests/ -v

omem_os-0.0.1/.github/workflows/publish.yml

@@ -0,0 +1,104 @@
+name: Publish to PyPI
+
+on:
+  release:
+    types: [published]
+
+jobs:
+  build_wheels:
+    name: Build wheels on ${{ matrix.os }}
+    runs-on: ${{ matrix.os }}
+    strategy:
+      matrix:
+        os: [ubuntu-latest, windows-latest, macos-latest]
+
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+
+      - name: Extract release version
+        shell: bash
+        run: echo "RELEASE_VERSION=${GITHUB_REF_NAME#v}" >> $GITHUB_ENV
+
+      - name: Set up QEMU
+        if: runner.os == 'Linux'
+        uses: docker/setup-qemu-action@v3
+        with:
+          platforms: arm64
+
+      - name: Install Rust toolchain
+        uses: dtolnay/rust-toolchain@stable
+
+      - name: Add macOS cross-compilation targets
+        if: runner.os == 'macOS'
+        run: |
+          rustup target add x86_64-apple-darwin
+          rustup target add aarch64-apple-darwin
+
+      - name: Build wheels
+        uses: pypa/cibuildwheel@v2.17.0
+        env:
+          CIBW_BUILD: "cp39-* cp310-* cp311-* cp312-* cp313-*"
+          CIBW_ARCHS_LINUX: "x86_64 aarch64"
+          CIBW_ARCHS_MACOS: "x86_64 arm64"
+          CIBW_BEFORE_ALL_LINUX: "curl https://sh.rustup.rs -sSf | sh -s -- --default-toolchain stable -y"
+          CIBW_ENVIRONMENT: >-
+            PATH="$PATH:$HOME/.cargo/bin"
+            SETUPTOOLS_SCM_PRETEND_VERSION="${{ env.RELEASE_VERSION }}"
+          CIBW_ENVIRONMENT_MACOS: >-
+            PATH="$PATH:$HOME/.cargo/bin"
+            SETUPTOOLS_SCM_PRETEND_VERSION="${{ env.RELEASE_VERSION }}"
+          CIBW_ENVIRONMENT_WINDOWS: >-
+            SETUPTOOLS_SCM_PRETEND_VERSION="${{ env.RELEASE_VERSION }}"
+          CIBW_SKIP: "*-musllinux* pp*"
+          CIBW_BUILD_VERBOSITY: 1
+
+      - uses: actions/upload-artifact@v4
+        with:
+          name: cibw-wheels-${{ matrix.os }}-${{ strategy.job-index }}
+          path: ./wheelhouse/*.whl
+
+  build_sdist:
+    name: Build source distribution
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+        with:
+          fetch-depth: 0
+
+      - name: Extract release version
+        shell: bash
+        run: echo "RELEASE_VERSION=${GITHUB_REF_NAME#v}" >> $GITHUB_ENV
+
+      - name: Install Rust toolchain
+        uses: dtolnay/rust-toolchain@stable
+
+      - name: Build sdist
+        run: pipx run build --sdist
+        env:
+          SETUPTOOLS_SCM_PRETEND_VERSION: ${{ env.RELEASE_VERSION }}
+
+      - uses: actions/upload-artifact@v4
+        with:
+          name: cibw-sdist
+          path: dist/*.tar.gz
+
+  release:
+    name: Publish to PyPI
+    needs: [build_wheels, build_sdist]
+    runs-on: ubuntu-latest
+    environment: pypi
+    permissions:
+      id-token: write
+    steps:
+      - uses: actions/download-artifact@v4
+        with:
+          pattern: cibw-*
+          path: dist
+          merge-multiple: true
+
+      - name: Publish to PyPI
+        uses: pypa/gh-action-pypi-publish@release/v1
+        with:
+          skip_existing: true

omem_os-0.0.1/.gitignore

@@ -0,0 +1,46 @@
+# Byte-compiled
+__pycache__/
+*.py[cod]
+*$py.class
+
+# Distribution
+dist/
+build/
+*.egg-info/
+*.egg
+
+# Virtual environments
+.venv/
+venv/
+env/
+
+# IDE
+.idea/
+.vscode/
+*.swp
+*.swo
+
+# OS
+.DS_Store
+Thumbs.db
+
+# Data
+*.db
+data/
+
+# Testing
+.pytest_cache/
+htmlcov/
+.coverage
+
+# Rust build artifacts
+rust/target/
+
+# Linter cache
+.ruff_cache/
+
+# IDE / local config
+.claude/settings.local.json
+
+# Documentation
+*.md

omem_os-0.0.1/.pre-commit-config.yaml

@@ -0,0 +1,45 @@
+repos:
+  - repo: https://github.com/pre-commit/pre-commit-hooks
+    rev: v4.5.0
+    hooks:
+      - id: trailing-whitespace
+        exclude: ^tests/
+      - id: end-of-file-fixer
+        exclude: ^tests/
+      - id: check-yaml
+      - id: check-added-large-files
+        args: ['--maxkb=1000']
+      - id: check-json
+      - id: check-toml
+      - id: check-merge-conflict
+      - id: detect-private-key
+
+  - repo: https://github.com/astral-sh/ruff-pre-commit
+    rev: v0.3.0
+    hooks:
+      - id: ruff
+        args: [--fix, --exit-non-zero-on-fix]
+        exclude: ^(tests/|benchmarks/)
+
+  - repo: https://github.com/psf/black
+    rev: 24.2.0
+    hooks:
+      - id: black
+        language_version: python3.9
+        exclude: ^(tests/|benchmarks/)
+
+  - repo: https://github.com/pre-commit/mirrors-mypy
+    rev: v1.8.0
+    hooks:
+      - id: mypy
+        additional_dependencies: [types-all]
+        exclude: ^(tests/|benchmarks/|examples/)
+        args: [--ignore-missing-imports, --no-strict-optional]
+
+  - repo: local
+    hooks:
+      - id: no-commit-secrets
+        name: check for secrets in commit
+        entry: bash -c 'if git diff --cached | grep -iE "(api_key|secret_key|password|token|bearer)" | grep -v "OMEM_"; then exit 1; fi'
+        language: system
+        pass_filenames: false

omem_os-0.0.1/DEVELOPER.md

@@ -0,0 +1,313 @@
+> **Technical documentation for extending, integrating, and contributing to OMem.**
+> 
+> *Version: 0.0.1 (Pre-Alpha)*
+
+---
+
+## Table of Contents
+
+1. [System Architecture](#system-architecture)
+2. [Programmatic Usage (How to use the "bits")](#programmatic-usage-how-to-use-the-bits)
+   - [Initialization & Backends](#initialization--backends)
+   - [Adding Memories (Types & Importance)](#adding-memories-types--importance)
+   - [Recalling Memories (Hybrid RAG)](#recalling-memories-hybrid-rag)
+   - [Memory Maintenance (Sleep & TMS)](#memory-maintenance-sleep--tms)
+   - [Graph-RAG Explained](#graph-rag-explained)
+   - [Agent Integrations (MCP & LangChain)](#agent-integrations-mcp--langchain)
+3. [Contributor Guide](#contributor-guide)
+   - [Environment Setup](#environment-setup)
+   - [Branching & PR Workflow](#branching--pr-workflow)
+   - [Testing Standard](#testing-standard)
+   - [Benchmarking](#benchmarking)
+   - [Code Style & Linting](#code-style--linting)
+   - [Adding New Backends](#adding-new-backends)
+4. [CLI Reference](#cli-reference)
+5. [Context FAQ](#memory-layer-faq--does-it-bloat-context)
+
+---
+
+## System Architecture
+
+OMem is organized into highly optimized, modular components:
+
+### 1. Interface Layer (`omem/api.py`)
+Provides the unified entry point for application integration (`class OMem`). It handles request normalization, sensible defaults, and orchestrates interaction between the Brain and Engine layers.
+
+### 2. Retrieval Layer (Rust Core)
+Based in `omem/core/retrieval/` and utilizing the `omem_rust` native extension.
+- **Hybrid Scoring**: SIMD-accelerated ranking combining vector similarity, keyword matching, recency, and importance in a single pass.
+- **Optimized Retrieval**: Built on FAISS HNSW. Thread-safe management of embeddings.
+
+### 3. Logic Layer (The "Brain")
+Based in `omem/core/brain/`. Implementation of core cognitive functions:
+- **Consolidation (`dream.py`)**: Clustering and summarization for memory synthesis.
+- **Truth Maintenance (`tms.py`)**: Detects and resolves conflicting information.
+- **Importance (`importance.py`)**: Heuristic scoring for memory prioritization.
+- **Security (`secrets.py`)**: Automated sensitivity detection.
+
+### 4. Engine Layer
+Based in `omem/core/engine/`. Manages memory lifecycle and state:
+- **Ingestion Pipeline (`add.py`)**: Validates, embeds, classifies, and indexes raw data.
+- **Context Retrieval (`rag.py`)**: Executes complex, multi-signal retrieval strategies.
+- **State Management (`lifecycle.py`)**: Handles archiving, pruning, and snapshotting.
+
+---
+
+## Programmatic Usage (How to use the "bits")
+
+If you are building an AI application, you will interact primarily with the `OMem` class in Python.
+
+### Initialization & Backends
+
+OMem supports multiple storage backends depending on your deployment needs.
+
+```python
+from omem import OMem
+
+# 1. Local SQLite (Default, best for most single-machine agents)
+# Automatically creates ~/.omem/brain.db
+brain = OMem()
+
+# 2. In-Memory (Best for unit tests or highly ephemeral tasks)
+test_brain = OMem(backend="memory")
+
+# 3. PostgreSQL (Best for production, distributed multi-agent systems)
+pg_brain = OMem(
+    backend="postgres",
+    db_path="postgresql://user:pass@localhost:5432/omem"
+)
+```
+
+### Adding Memories (Types & Importance)
+
+OMem automatically classifies what you input, but you can explicitly define importance and type to control how the agent remembers it.
+
+```python
+from omem.types import MemoryType
+
+# Automatic (OMem detects this is a 'DECISION' and boosts importance based on keywords)
+brain.add("We decided to migrate from REST to GraphQL due to over-fetching.")
+
+# Explicit Control
+brain.add(
+    "Critical API key rotation policy: Must rotate every 30 days.",
+    importance=0.95,                  # 0.0 to 1.0 (forces it to stay in memory longer)
+    mem_type=MemoryType.DECISION      # Explicitly categorize the memory
+)
+
+# Procedural Tool Snippets (For MCP / Agent tool calling)
+brain.add(
+    "To deploy, run 'aws ecs update-service --cluster prod'",
+    mem_type=MemoryType.PROCEDURAL
+)
+```
+
+### Recalling Memories (Hybrid RAG)
+
+Retrieval in OMem is not just vector search. It uses 4 signals (Vector + Keyword + Recency + Importance). 
+
+```python
+# Standard recall (Returns top 5 matches as a combined string)
+context = brain.recall("What is our API rotation policy?")
+
+# Advanced Recall (Returns raw Memory objects for custom formatting)
+raw_memories = brain.recall(
+    "deployment steps",
+    k=3,                          # Limit to 3 results
+    context_type="PROCEDURAL",    # Only look for procedural memories
+    time_range="recent"           # Only look at recent memories
+)
+
+for mem in raw_memories:
+    print(f"[{mem.type}] {mem.content} (Score: {mem.score})")
+```
+
+**Debugging Retrieval**: If you want to see *why* OMem picked a memory, use `inspect()`:
+```python
+for result in brain.inspect("GraphQL"):
+    print(result.explain())
+    # Output: vector=0.88, keyword=0.45, recency=0.99, importance=1.2x boost -> Final: 0.92
+```
+
+### Memory Maintenance (Sleep & TMS)
+
+Agents, like humans, need to sleep to consolidate memories and remove garbage.
+
+```python
+# Run the full sleep cycle:
+# 1. Deduplicates repetitive memories
+# 2. Forgets low-importance, old memories
+# 3. Reflects on recent events to create high-level insights
+stats = brain.sleep()
+print(f"Forgotten: {stats['forgotten']}, Consolidated: {stats['consolidated']}")
+
+# Truth Maintenance System (TMS) - Resolving Conflicts
+brain.add("Python version is 3.9")
+brain.add("Python version is 3.11") # OMem detects this conflicts with 3.9
+
+# Explicitly resolve the conflict (archives 3.9, keeps 3.11)
+brain.resolve_conflict("Python version")
+```
+
+### Graph-RAG Explained
+
+When you `add()` a memory, OMem runs a fast NER (Named Entity Recognition) pass to extract entities (e.g., "Python", "GraphQL", "AWS") and builds a graph edge between them.
+
+During `recall()`, if `graph_boost=True` (which is default), OMem will:
+1. Find the top vector matches.
+2. Look at the entities in those matches.
+3. Traverse the graph to find *connected* memories that might not share the exact semantic meaning but are highly relevant structurally.
+4. Inject them into the result set.
+
+### Agent Integrations (MCP & LangChain)
+
+**Model Context Protocol (MCP)**
+To expose OMem natively to Claude Desktop or Cursor, simply run the server:
+```python
+from omem.integrations.mcp_server import serve_mcp
+serve_mcp() # Or use CLI: omem serve
+```
+
+**LangChain Wrapper**
+```python
+from omem.integrations.langchain import OMemRetriever
+retriever = OMemRetriever(omem_instance=brain)
+# Pass retriever to your LangChain agents
+```
+
+---
+
+## Contributor Guide
+
+We welcome contributions! OMem is built to be fast, typed, and fully tested.
+
+### Environment Setup
+
+1. **Clone & Virtual Env**:
+   ```bash
+   git clone https://github.com/mohitkumarrajbadi/omem
+   cd omem
+   python3 -m venv .venv
+   source .venv/bin/activate
+   ```
+2. **Install with Dev Dependencies & Rust Extensions**:
+   ```bash
+   # Note: You must have Rust installed (curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh)
+   pip install -e ".[dev]"
+   ```
+3. **Verify**:
+   ```bash
+   pytest tests/ -v
+   ```
+
+> **macOS / Anaconda users:** If you experience FAISS/OpenMP crashes, add this to your environment:
+> `export KMP_DUPLICATE_LIB_OK=TRUE`
+> To speed up local runs without network checks: `export HF_HUB_OFFLINE=1`
+
+### Branching & PR Workflow
+
+1. **`dev` (Default)**: All active development happens here. PRs target `dev`.
+2. **`staging`**: Pre-release integration testing.
+3. **`main`/`prod`**: Stable releases (`1.0.0`, etc.). Current line is pre-alpha (`0.0.x`).
+
+**To contribute:**
+1. Checkout `dev`: `git checkout dev`
+2. Create feature branch: `git checkout -b feat/my-new-feature`
+3. Commit your changes.
+4. Open a Pull Request targeting the `dev` branch.
+
+### Testing Standard
+
+- **Coverage**: All new core logic in `omem/core/` must have unit tests.
+- **Location**: Place tests in the `tests/` directory matching the module structure (e.g., `tests/test_tms.py`).
+- **Command**: Run `pytest tests/` before submitting a PR.
+- **No API Calls**: Tests should mock LLM or external API calls to remain fast and deterministic.
+
+### Benchmarking
+
+Because OMem competes directly with vector DBs on speed, performance regressions are strictly monitored.
+If you modify the `rust/` core, `omem/core/retrieval/`, or `omem/core/engine/add.py`, you **must** run the benchmark:
+
+```bash
+python benchmarks/competitor.py
+# Or using the CLI:
+omem benchmark --n 10000
+```
+Ensure that `add()` operations remain under `5ms` and `RAG` latency remains under `30ms`.
+
+### Code Style & Linting
+
+- **Typing**: Strict type hinting is enforced (`def add(text: str, importance: float = 0.5) -> str:`).
+- **Format**: We use standard Python formatting. In the future, we will enforce `ruff`.
+- **Comments**: Explain *why* complex heuristic logic exists, especially in the `brain/` directory.
+
+### Adding New Backends
+
+To add a new storage backend (e.g., Redis, MongoDB):
+1. Create `omem/backends/redis.py`.
+2. Inherit from `omem.backends.base.StorageBackend`.
+3. Implement the required interface (`save`, `load`, `delete`, `query`).
+4. Register the backend in `omem/api.py` inside the `_initialize_backend()` factory.
+
+---
+
+## CLI Reference
+
+The `omem` CLI is the primary interface for managing the memory system. Install it with `pip install omem-os`.
+
+### Setup
+| Command | Description |
+|---|---|
+| `omem init` | Initialize memory system at `~/.omem/brain.db` |
+| `omem init --db-path ./custom.db` | Use a custom database path |
+| `omem health` | Health check — exits `0` if OK, `1` if error |
+
+### Writing & Reading
+```bash
+omem add "content" -i 0.9 -n myproject -t DECISION
+omem search "query" -k 10 -n myproject -c architecture -t recent
+omem list -n myproject -l 50
+omem inspect "query" # Debug retrieval scoring
+```
+
+### Maintenance & Integrations
+```bash
+omem maintain              # full cycle: compress + reflect + forget + dream
+omem export -f json -o dump.json
+omem load dump.json
+omem serve                 # start MCP stdio server
+omem dashboard             # launch web memory dashboard (port 7900)
+```
+
+---
+
+## Memory Layer FAQ — Does It Bloat Context?
+
+A common concern when connecting OMem to AI agents is:
+> *"Won't injecting memory into every prompt increase context size and add latency?"*
+
+**No. OMem is a retrieval layer, not an injection layer.** 
+
+### ❌ What people assume (wrong)
+```
+Every agent turn → dump ALL stored memories into context
+256 memories × ~100 tokens = 25,600 tokens injected every request
+→ Blown context window + high cost + slow responses
+```
+
+### ✅ What actually happens
+```
+Every agent turn → semantic search returns only TOP 3–5 relevant memories
+256 memories stored, ~5 retrieved → ~200–500 tokens injected
+→ Lean, precise, always relevant context
+```
+
+### Direct comparison
+| | Without OMem | With OMem |
+|---|---|---|
+| Context per turn | Full conversation history (grows unboundedly) | 3–5 recalled memories (~300 tokens) |
+| Cross-session memory | ❌ Starts from zero each session | ✅ Persistent across sessions, projects, restarts |
+| Token cost | High — re-reads everything | Low — retrieves only what's relevant |
+| Latency added | 0ms (no memory) | <1ms (FAISS search) + ~10ms (embedding) |
+
+**Connecting OMem to an agent makes it smarter with *less* context, not slower with more.**