memsearch 0.1.0__tar.gz

memsearch-0.1.0/.github/workflows/docs.yml

@@ -0,0 +1,27 @@
+name: Deploy docs
+
+on:
+  push:
+    branches: [main]
+    paths:
+      - 'docs/**'
+      - 'mkdocs.yml'
+      - '.github/workflows/docs.yml'
+  workflow_dispatch:
+
+permissions:
+  contents: write
+
+jobs:
+  deploy:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: '3.12'
+
+      - run: pip install mkdocs-material mkdocs-terminal pymdown-extensions
+
+      - run: mkdocs gh-deploy --force

memsearch-0.1.0/.github/workflows/release.yml

@@ -0,0 +1,40 @@
+# Publish to PyPI when a version tag is pushed
+#
+# Usage:
+#   git tag v0.1.0  # Must match the version in pyproject.toml
+#   git push --tags
+
+name: Publish Python Package to PyPI
+
+on:
+  push:
+    tags:
+      - "v*"
+
+jobs:
+  publish:
+    name: Publish to PyPI
+    runs-on: ubuntu-latest
+    environment: pypi
+
+    permissions:
+      id-token: write
+      contents: read
+
+    steps:
+      - name: Checkout code
+        uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: "3.10"
+
+      - name: Install build tools
+        run: python -m pip install build
+
+      - name: Build package
+        run: python -m build
+
+      - name: Publish to PyPI
+        uses: pypa/gh-action-pypi-publish@release/v1

memsearch-0.1.0/.gitignore

@@ -0,0 +1,11 @@
+__pycache__/
+*.py[cod]
+*.egg-info/
+dist/
+build/
+.venv/
+*.db
+.pytest_cache/
+.ruff_cache/
+.memsearch/
+CONTEXT.md

memsearch-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 zhangchen
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

memsearch-0.1.0/PKG-INFO

@@ -0,0 +1,466 @@
+Metadata-Version: 2.4
+Name: memsearch
+Version: 0.1.0
+Summary: Semantic memory search for markdown knowledge bases
+License-Expression: MIT
+License-File: LICENSE
+Requires-Python: >=3.11
+Requires-Dist: click>=8.1
+Requires-Dist: openai>=1.0
+Requires-Dist: pymilvus[milvus-lite]>=2.5.0
+Requires-Dist: setuptools<75
+Requires-Dist: tomli-w>=1.0
+Requires-Dist: watchdog>=4.0
+Provides-Extra: all
+Requires-Dist: anthropic>=0.40; extra == 'all'
+Requires-Dist: google-genai>=1.0; extra == 'all'
+Requires-Dist: ollama>=0.4; extra == 'all'
+Requires-Dist: sentence-transformers>=3.0; extra == 'all'
+Requires-Dist: voyageai>=0.3; extra == 'all'
+Provides-Extra: anthropic
+Requires-Dist: anthropic>=0.40; extra == 'anthropic'
+Provides-Extra: google
+Requires-Dist: google-genai>=1.0; extra == 'google'
+Provides-Extra: local
+Requires-Dist: sentence-transformers>=3.0; extra == 'local'
+Provides-Extra: ollama
+Requires-Dist: ollama>=0.4; extra == 'ollama'
+Provides-Extra: voyage
+Requires-Dist: voyageai>=0.3; extra == 'voyage'
+Description-Content-Type: text/markdown
+
+<p align="center">
+  <img src="assets/logo-icon.jpg" alt="memsearch" width="120">
+</p>
+
+<h1 align="center">memsearch</h1>
+
+<p align="center">
+  <strong><a href="https://github.com/openclaw/openclaw">OpenClaw</a>'s memory, everywhere.</strong>
+</p>
+
+> 💡 **memsearch extracts [OpenClaw](https://github.com/openclaw/openclaw)'s memory system into a standalone library** — same markdown-first architecture, same chunking, same chunk ID format. Pluggable into *any* agent framework, backed by [Milvus](https://milvus.io/) (local Milvus Lite → Milvus Server → Zilliz Cloud). See it in action with the included **[Claude Code plugin](ccplugin/README.md)**.
+
+### ✨ Why memsearch?
+
+- 🦞 **OpenClaw's memory, everywhere** — OpenClaw has one of the best memory designs in open-source AI: **markdown as the single source of truth** — simple, human-readable, `git`-friendly, zero vendor lock-in
+- ⚡ **Smart dedup** — SHA-256 content hashing means unchanged content is never re-embedded
+- 🔄 **Live sync** — File watcher auto-indexes on changes, deletes stale chunks when files are removed
+- 🧹 **Memory flush** — LLM-powered summarization compresses old memories, just like OpenClaw's flush cycle
+- 🧩 **Claude Code plugin included** — A real-world example: **[ccplugin/](ccplugin/README.md)** gives Claude persistent memory across sessions with zero config
+
+## 🔍 How It Works
+
+**Markdown is the source of truth** — the vector store is just a derived index, rebuildable anytime.
+
+```
+  ┌─── Search ─────────────────────────────────────────────────────────┐
+  │                                                                    │
+  │  "how to configure Redis?"                                         │
+  │        │                                                           │
+  │        ▼                                                           │
+  │   ┌──────────┐     ┌─────────────────┐     ┌──────────────────┐   │
+  │   │  Embed   │────▶│ Cosine similarity│────▶│ Top-K results    │   │
+  │   │  query   │     │ (Milvus)        │     │ with source info │   │
+  │   └──────────┘     └─────────────────┘     └──────────────────┘   │
+  │                                                                    │
+  └────────────────────────────────────────────────────────────────────┘
+
+  ┌─── Ingest ─────────────────────────────────────────────────────────┐
+  │                                                                    │
+  │  MEMORY.md                                                         │
+  │  memory/2026-02-09.md     ┌──────────┐     ┌────────────────┐     │
+  │  memory/2026-02-08.md ───▶│ Chunker  │────▶│ Dedup          │     │
+  │                           │(heading, │     │(chunk_hash PK) │     │
+  │                           │paragraph)│     └───────┬────────┘     │
+  │                           └──────────┘             │              │
+  │                                             new chunks only       │
+  │                                                    ▼              │
+  │                                            ┌──────────────┐       │
+  │                                            │  Embed &     │       │
+  │                                            │  Milvus upsert│      │
+  │                                            └──────────────┘       │
+  │                                                                    │
+  └────────────────────────────────────────────────────────────────────┘
+
+  ┌─── Watch ──────────────────────────────────────────────────────────┐
+  │  File watcher (1500ms debounce) ──▶ auto re-index / delete stale  │
+  └────────────────────────────────────────────────────────────────────┘
+
+  ┌─── Flush ──────────────────────────────────────────────────────────┐
+  │  Retrieve chunks ──▶ LLM summarize ──▶ write memory/YYYY-MM-DD.md │
+  └────────────────────────────────────────────────────────────────────┘
+```
+
+🔒 The entire pipeline runs locally by default — your data never leaves your machine unless you choose a remote Milvus backend or a cloud embedding provider.
+
+## 🧩 Claude Code Plugin
+
+memsearch ships with a **[Claude Code plugin](ccplugin/README.md)** — a real-world example of OpenClaw's memory running outside OpenClaw. It gives Claude **automatic persistent memory** across sessions: every session is summarized to markdown, every prompt triggers a semantic search, and a background watcher keeps the index in sync. No commands to learn, no manual saving — just install and go.
+
+```bash
+# Install memsearch, then launch Claude with the plugin
+pip install memsearch
+claude --plugin-dir ./ccplugin
+```
+
+```
+  Session start ──▶ start memsearch watch (singleton) ──▶ inject recent memories
+                           │
+  User prompt ──▶ memsearch search ──▶ inject relevant memories
+                           │
+  Claude stops ──▶ haiku summary ──▶ write .memsearch/memory/YYYY-MM-DD.md
+                           │                                │
+  Session end ──▶ stop watch              watch auto-indexes ◀┘
+```
+
+Under the hood: 4 shell hooks + 1 watch process, all calling the `memsearch` CLI. Memories are transparent `.md` files — human-readable, git-friendly, rebuildable. See **[ccplugin/README.md](ccplugin/README.md)** for the full architecture, hook details, progressive disclosure model, and comparison with claude-mem.
+
+## 📦 Installation
+
+```bash
+pip install memsearch
+```
+
+### Additional embedding providers
+
+```bash
+pip install "memsearch[google]"      # Google Gemini
+pip install "memsearch[voyage]"      # Voyage AI
+pip install "memsearch[ollama]"      # Ollama (local)
+pip install "memsearch[local]"       # sentence-transformers (local, no API key)
+pip install "memsearch[all]"         # Everything
+```
+
+## 🐍 Python API — Build an Agent with Memory
+
+The example below shows a complete agent loop with memory: save knowledge to markdown, index it, and recall it later via semantic search.
+
+```python
+import asyncio
+from datetime import date
+from pathlib import Path
+from openai import OpenAI
+from memsearch import MemSearch
+
+MEMORY_DIR = "./memory"
+llm = OpenAI()                                        # your LLM client
+ms = MemSearch(paths=[MEMORY_DIR])                    # memsearch handles the rest
+
+def save_memory(content: str):
+    """Append a note to today's memory log (OpenClaw-style daily markdown)."""
+    p = Path(MEMORY_DIR) / f"{date.today()}.md"
+    p.parent.mkdir(parents=True, exist_ok=True)
+    with open(p, "a") as f:
+        f.write(f"\n{content}\n")
+
+async def agent_chat(user_input: str) -> str:
+    # 1. Recall — search past memories for relevant context
+    memories = await ms.search(user_input, top_k=3)
+    context = "\n".join(f"- {m['content'][:200]}" for m in memories)
+
+    # 2. Think — call LLM with memory context
+    resp = llm.chat.completions.create(
+        model="gpt-4o-mini",
+        messages=[
+            {"role": "system", "content": f"You have these memories:\n{context}"},
+            {"role": "user", "content": user_input},
+        ],
+    )
+    answer = resp.choices[0].message.content
+
+    # 3. Remember — save this exchange and index it
+    save_memory(f"## {user_input}\n{answer}")
+    await ms.index()
+
+    return answer
+
+async def main():
+    # Seed some knowledge
+    save_memory("## Team\n- Alice: frontend lead\n- Bob: backend lead")
+    save_memory("## Decision\nWe chose Redis for caching over Memcached.")
+    await ms.index()
+
+    # Agent can now recall those memories
+    print(await agent_chat("Who is our frontend lead?"))
+    print(await agent_chat("What caching solution did we pick?"))
+
+asyncio.run(main())
+```
+
+<details>
+<summary>💜 <b>Anthropic Claude example</b> — click to expand</summary>
+
+```bash
+pip install memsearch anthropic
+```
+
+```python
+import asyncio
+from datetime import date
+from pathlib import Path
+from anthropic import Anthropic
+from memsearch import MemSearch
+
+MEMORY_DIR = "./memory"
+llm = Anthropic()
+ms = MemSearch(paths=[MEMORY_DIR])
+
+def save_memory(content: str):
+    p = Path(MEMORY_DIR) / f"{date.today()}.md"
+    p.parent.mkdir(parents=True, exist_ok=True)
+    with open(p, "a") as f:
+        f.write(f"\n{content}\n")
+
+async def agent_chat(user_input: str) -> str:
+    # 1. Recall
+    memories = await ms.search(user_input, top_k=3)
+    context = "\n".join(f"- {m['content'][:200]}" for m in memories)
+
+    # 2. Think — call Claude with memory context
+    resp = llm.messages.create(
+        model="claude-sonnet-4-5-20250929",
+        max_tokens=1024,
+        system=f"You have these memories:\n{context}",
+        messages=[{"role": "user", "content": user_input}],
+    )
+    answer = resp.content[0].text
+
+    # 3. Remember
+    save_memory(f"## {user_input}\n{answer}")
+    await ms.index()
+    return answer
+
+async def main():
+    save_memory("## Team\n- Alice: frontend lead\n- Bob: backend lead")
+    await ms.index()
+    print(await agent_chat("Who is our frontend lead?"))
+
+asyncio.run(main())
+```
+
+</details>
+
+<details>
+<summary>🦙 <b>Ollama (fully local, no API key)</b> — click to expand</summary>
+
+```bash
+pip install "memsearch[ollama]"
+ollama pull nomic-embed-text          # embedding model
+ollama pull llama3.2                  # chat model
+```
+
+```python
+import asyncio
+from datetime import date
+from pathlib import Path
+from ollama import chat
+from memsearch import MemSearch
+
+MEMORY_DIR = "./memory"
+ms = MemSearch(paths=[MEMORY_DIR], embedding_provider="ollama")
+
+def save_memory(content: str):
+    p = Path(MEMORY_DIR) / f"{date.today()}.md"
+    p.parent.mkdir(parents=True, exist_ok=True)
+    with open(p, "a") as f:
+        f.write(f"\n{content}\n")
+
+async def agent_chat(user_input: str) -> str:
+    # 1. Recall
+    memories = await ms.search(user_input, top_k=3)
+    context = "\n".join(f"- {m['content'][:200]}" for m in memories)
+
+    # 2. Think — call Ollama locally
+    resp = chat(
+        model="llama3.2",
+        messages=[
+            {"role": "system", "content": f"You have these memories:\n{context}"},
+            {"role": "user", "content": user_input},
+        ],
+    )
+    answer = resp.message.content
+
+    # 3. Remember
+    save_memory(f"## {user_input}\n{answer}")
+    await ms.index()
+    return answer
+
+async def main():
+    save_memory("## Team\n- Alice: frontend lead\n- Bob: backend lead")
+    await ms.index()
+    print(await agent_chat("Who is our frontend lead?"))
+
+asyncio.run(main())
+```
+
+</details>
+
+### 🗄️ Milvus Backend Configuration
+
+memsearch supports three Milvus deployment modes — just change `milvus_uri` and `milvus_token`:
+
+#### 1. Milvus Lite (default — zero config, local file)
+
+```python
+ms = MemSearch(
+    paths=["./docs/"],
+    milvus_uri="~/.memsearch/milvus.db",    # local file, no server needed
+)
+```
+
+No server to install. Data is stored in a single `.db` file. Perfect for personal use, single-agent setups, and development.
+
+#### 2. Milvus Server (self-hosted)
+
+```python
+ms = MemSearch(
+    paths=["./docs/"],
+    milvus_uri="http://localhost:19530",     # your Milvus server
+    milvus_token="root:Milvus",              # default credentials, change in production
+)
+```
+
+Deploy via Docker (`docker compose`) or Kubernetes. Ideal for multi-agent workloads and team environments where you need a shared, always-on vector store.
+
+#### 3. Zilliz Cloud (fully managed)
+
+```python
+ms = MemSearch(
+    paths=["./docs/"],
+    milvus_uri="https://in03-xxx.api.gcp-us-west1.zillizcloud.com",
+    milvus_token="your-api-key",
+)
+```
+
+Zero-ops, auto-scaling managed service. Get your free cluster at [cloud.zilliz.com](https://cloud.zilliz.com). Great for production deployments and when you don't want to manage infrastructure.
+
+## 🖥️ CLI Usage
+
+### Index markdown files
+
+```bash
+# Index one or more directories / files
+memsearch index ./docs/ ./notes/
+
+# Use a different embedding provider
+memsearch index ./docs/ --provider google
+
+# Force re-index everything
+memsearch index ./docs/ --force
+
+# Use a remote Milvus server
+memsearch index ./docs/ --milvus-uri http://localhost:19530 --milvus-token root:Milvus
+```
+
+### Search
+
+```bash
+memsearch search "how to configure Redis caching"
+
+# Return more results
+memsearch search "authentication flow" --top-k 10
+
+# JSON output (for piping to other tools)
+memsearch search "error handling" --json-output
+```
+
+### Watch for changes
+
+```bash
+# Auto-index on file changes (Ctrl+C to stop)
+memsearch watch ./docs/ ./notes/
+
+# Custom debounce interval
+memsearch watch ./docs/ --debounce-ms 3000
+```
+
+### Flush (compress memories)
+
+Summarize indexed chunks into a condensed memory using an LLM:
+
+```bash
+memsearch flush
+
+# Use a specific LLM
+memsearch flush --llm-provider anthropic
+memsearch flush --llm-provider gemini
+
+# Only flush chunks from a specific source
+memsearch flush --source ./docs/old-notes.md
+```
+
+### Configuration management
+
+```bash
+memsearch config init               # Interactive wizard
+memsearch config set milvus.uri http://localhost:19530
+memsearch config get milvus.uri
+memsearch config list --resolved    # Show merged config from all sources
+memsearch config list --global      # Show ~/.memsearch/config.toml only
+memsearch config list --project     # Show .memsearch.toml only
+```
+
+### Manage
+
+```bash
+memsearch stats    # Show index statistics
+memsearch reset    # Drop all indexed data (with confirmation)
+```
+
+## ⚙️ Configuration
+
+memsearch uses a layered configuration system.  Settings are resolved in priority order (lowest → highest):
+
+1. **Built-in defaults**
+2. **Global config** — `~/.memsearch/config.toml`
+3. **Project config** — `.memsearch.toml` (in your working directory)
+4. **Environment variables** — `MEMSEARCH_SECTION_FIELD` (e.g. `MEMSEARCH_MILVUS_URI`)
+5. **CLI flags** — `--milvus-uri`, `--provider`, etc.
+
+### API keys
+
+API keys for embedding and LLM providers are read from standard environment variables:
+
+```bash
+# Embedding providers (set the one you use)
+export OPENAI_API_KEY="sk-..."
+export OPENAI_BASE_URL="https://..."   # optional, for proxies / Azure
+export GOOGLE_API_KEY="..."
+export VOYAGE_API_KEY="..."
+
+# LLM for flush/summarization (set the one you use)
+export ANTHROPIC_API_KEY="..."         # for flush with Anthropic
+```
+
+## 🔌 Embedding Providers
+
+| Provider | Install | Env Var | Default Model |
+|----------|---------|---------|---------------|
+| OpenAI | `memsearch` (included) | `OPENAI_API_KEY` | `text-embedding-3-small` |
+| Google | `memsearch[google]` | `GOOGLE_API_KEY` | `text-embedding-004` |
+| Voyage | `memsearch[voyage]` | `VOYAGE_API_KEY` | `voyage-3-lite` |
+| Ollama | `memsearch[ollama]` | `OLLAMA_HOST` (optional) | `nomic-embed-text` |
+| Local | `memsearch[local]` | — | `all-MiniLM-L6-v2` |
+
+## 🐾 OpenClaw Compatibility
+
+memsearch is designed to be a drop-in memory backend for projects following [OpenClaw's memory architecture](https://github.com/openclaw/openclaw):
+
+| Feature | OpenClaw | memsearch |
+|---------|----------|-----------|
+| Memory layout | `MEMORY.md` + `memory/YYYY-MM-DD.md` | ✅ Same |
+| Chunk ID format | `hash(source:startLine:endLine:contentHash:model)` | ✅ Same |
+| Dedup strategy | Content-hash primary key | ✅ Same |
+| Flush target | Append to daily markdown log | ✅ Same |
+| Source of truth | Markdown files (vector DB is derived) | ✅ Same |
+| File watch debounce | 1500ms | ✅ Same default |
+| Vector backend | Built-in | Milvus (Lite / Server / Zilliz Cloud) |
+| Embedding providers | Built-in | Pluggable (OpenAI, Google, Voyage, Ollama, local) |
+| Packaging | Part of OpenClaw monorepo | Standalone `pip install` |
+
+If you're already using OpenClaw's memory directory layout, just point memsearch at it — no migration needed.
+
+## 📄 License
+
+MIT