memstem 0.17.0__tar.gz

memstem-0.17.0/.github/ISSUE_TEMPLATE/adapter_request.md

@@ -0,0 +1,31 @@
+---
+name: Adapter request
+about: Request an adapter for a new AI system
+title: '[ADAPTER] '
+labels: adapter
+assignees: ''
+---
+
+## Target system
+
+Which AI system / tool?
+
+## Memory storage location
+
+Where does it store sessions / memories on disk? (path patterns, file formats)
+
+## File format
+
+JSONL, markdown, sqlite, other?
+
+## Update frequency
+
+How often does the system write?
+
+## Existing schema (if any)
+
+```
+paste a sample
+```
+
+## Why this matters to you

memstem-0.17.0/.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,32 @@
+---
+name: Bug report
+about: Something isn't working
+title: '[BUG] '
+labels: bug
+assignees: ''
+---
+
+## Description
+
+What happened? What did you expect to happen?
+
+## Steps to reproduce
+
+1. ...
+2. ...
+3. ...
+
+## Environment
+
+- Memstem version:
+- Python version:
+- OS:
+- AI clients connected (Claude Code / OpenClaw / other):
+
+## Logs
+
+```
+paste relevant log output here
+```
+
+## Additional context

memstem-0.17.0/.github/ISSUE_TEMPLATE/config.yml

@@ -0,0 +1,5 @@
+blank_issues_enabled: false
+contact_links:
+  - name: Discussions
+    url: https://github.com/memstem/memstem/discussions
+    about: Ask a question or discuss an idea

memstem-0.17.0/.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,19 @@
+---
+name: Feature request
+about: Suggest a new capability
+title: '[FEAT] '
+labels: enhancement
+assignees: ''
+---
+
+## Problem
+
+What pain are you trying to solve?
+
+## Proposed solution
+
+What would you like to see?
+
+## Alternatives considered
+
+## Additional context

memstem-0.17.0/.github/dependabot.yml

@@ -0,0 +1,23 @@
+version: 2
+updates:
+  - package-ecosystem: "pip"
+    directory: "/"
+    schedule:
+      interval: "weekly"
+    open-pull-requests-limit: 5
+    labels:
+      - "dependencies"
+    commit-message:
+      prefix: "deps"
+      include: "scope"
+
+  - package-ecosystem: "github-actions"
+    directory: "/"
+    schedule:
+      interval: "monthly"
+    labels:
+      - "dependencies"
+      - "ci"
+    commit-message:
+      prefix: "ci"
+      include: "scope"

memstem-0.17.0/.github/pull_request_template.md

@@ -0,0 +1,27 @@
+## Summary
+
+What does this PR do?
+
+## Why
+
+What problem does it solve?
+
+## Changes
+
+- Change 1
+- Change 2
+
+## Testing
+
+How did you verify this works?
+
+- [ ] Tests added / updated
+- [ ] Lint passes (`ruff check`, `ruff format --check`)
+- [ ] Type check passes (`mypy src/`)
+- [ ] Documentation updated (if user-facing change)
+
+## Architecture decision record
+
+If this changes storage layout, search ranking, the adapter interface, or any cross-cutting design — link or include the ADR.
+
+## Notes for reviewer

memstem-0.17.0/.github/workflows/ci.yml

@@ -0,0 +1,72 @@
+name: CI
+
+on:
+  push:
+    branches: [main]
+  pull_request:
+    branches: [main]
+
+jobs:
+  lint:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v6
+      - uses: actions/setup-python@v6
+        with:
+          python-version: "3.11"
+      - name: Install dependencies
+        run: |
+          pip install -e ".[dev]"
+      - name: Ruff check
+        run: ruff check .
+      - name: Ruff format check
+        run: ruff format --check .
+      - name: Mypy
+        run: mypy src/ tests/ scripts/
+
+  test:
+    runs-on: ${{ matrix.os }}
+    continue-on-error: ${{ matrix.experimental }}
+    strategy:
+      fail-fast: false
+      matrix:
+        os: [ubuntu-latest]
+        python-version: ["3.11", "3.12"]
+        experimental: [false]
+        include:
+          # macOS is supported for v0.1 manual installs but not gated in CI:
+          # actions/setup-python's macOS builds ship Python without
+          # `enable_load_extension`, which sqlite-vec requires. The hosted
+          # runner image is the constraint here, not Memstem itself — users
+          # with `brew install python@3.11` are fine. Marked experimental so
+          # failures surface in CI without blocking merge until the runner
+          # image catches up.
+          - os: macos-latest
+            python-version: "3.11"
+            experimental: true
+          - os: macos-latest
+            python-version: "3.12"
+            experimental: true
+          # Windows isn't supported in v0.1 (use WSL2). We run it in CI anyway
+          # for visibility, marked experimental so failures don't block merge.
+          - os: windows-latest
+            python-version: "3.11"
+            experimental: true
+          - os: windows-latest
+            python-version: "3.12"
+            experimental: true
+    steps:
+      - uses: actions/checkout@v6
+      - uses: actions/setup-python@v6
+        with:
+          python-version: ${{ matrix.python-version }}
+      - name: Install dependencies
+        run: |
+          pip install -e ".[dev]"
+      - name: Run tests
+        run: pytest --cov-report=xml
+      - name: Upload coverage
+        uses: codecov/codecov-action@v6
+        if: matrix.os == 'ubuntu-latest' && matrix.python-version == '3.11'
+        with:
+          fail_ci_if_error: false

memstem-0.17.0/.github/workflows/release.yml

@@ -0,0 +1,85 @@
+name: Release
+
+on:
+  push:
+    tags:
+      - "v*.*.*"
+  # Manual publishing escape hatch — used for the first PyPI upload (the
+  # version was already tagged before the pypi job existed) and for re-runs
+  # after a transient publish failure. Skips the GitHub-release job.
+  workflow_dispatch:
+
+permissions:
+  contents: write
+
+jobs:
+  release:
+    name: Publish GitHub release from CHANGELOG
+    if: github.event_name == 'push'
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Extract CHANGELOG section
+        run: |
+          VERSION="${GITHUB_REF_NAME#v}"
+          awk -v ver="$VERSION" \
+            '$0 ~ "^## \\["ver"\\]" {p=1; next} /^## \[/ {p=0} p' \
+            CHANGELOG.md > release-notes.md
+          if ! grep -q '[^[:space:]]' release-notes.md; then
+            echo "::error::No CHANGELOG.md section found for ${VERSION} — add one before tagging." >&2
+            exit 1
+          fi
+
+      - name: Build release title
+        run: |
+          # Title is "vX.Y.Z — <summary paragraph>" when the section opens with
+          # prose before the first ### heading; bare "vX.Y.Z" otherwise.
+          SUMMARY=$(awk '/^#/{exit} NF{printf "%s ", $0}' release-notes.md \
+            | sed 's/ $//' | cut -c1-120)
+          if [ -n "$SUMMARY" ]; then
+            echo "RELEASE_TITLE=${GITHUB_REF_NAME} — ${SUMMARY}" >> "$GITHUB_ENV"
+          else
+            echo "RELEASE_TITLE=${GITHUB_REF_NAME}" >> "$GITHUB_ENV"
+          fi
+
+      - name: Create release
+        env:
+          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+        run: |
+          gh release create "$GITHUB_REF_NAME" \
+            --title "$RELEASE_TITLE" \
+            --notes-file release-notes.md \
+            --verify-tag
+
+  pypi:
+    name: Publish to PyPI
+    runs-on: ubuntu-latest
+    environment: pypi
+    permissions:
+      id-token: write   # OIDC for PyPI Trusted Publishing — no stored token
+      contents: read
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: "3.12"
+
+      - name: Verify tag matches pyproject version
+        if: github.event_name == 'push'
+        run: |
+          TAG_VERSION="${GITHUB_REF_NAME#v}"
+          PY_VERSION=$(python -c "import tomllib; print(tomllib.load(open('pyproject.toml','rb'))['project']['version'])")
+          if [ "$TAG_VERSION" != "$PY_VERSION" ]; then
+            echo "::error::Tag ${GITHUB_REF_NAME} does not match pyproject.toml version ${PY_VERSION}" >&2
+            exit 1
+          fi
+
+      - name: Build sdist and wheel
+        run: |
+          python -m pip install --quiet build
+          python -m build
+
+      - name: Publish
+        uses: pypa/gh-action-pypi-publish@release/v1

memstem-0.17.0/.gitignore

@@ -0,0 +1,86 @@
+# Python
+__pycache__/
+*.py[cod]
+*$py.class
+*.so
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+share/python-wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+MANIFEST
+
+# Virtual environments
+.env
+.venv
+env/
+venv/
+ENV/
+env.bak/
+venv.bak/
+
+# Testing
+htmlcov/
+.tox/
+.nox/
+.coverage
+.coverage.*
+.cache
+nosetests.xml
+coverage.xml
+*.cover
+*.py,cover
+.hypothesis/
+.pytest_cache/
+cover/
+
+# Tools
+.mypy_cache/
+.ruff_cache/
+.pyre/
+.pytype/
+.claude/
+
+# Memstem-specific
+*.db
+*.db-journal
+index.db
+embeddings/
+.memstem/
+test-vaults/
+test-data/
+
+# IDE
+.vscode/
+.idea/
+*.swp
+*.swo
+*~
+.DS_Store
+
+# Logs
+*.log
+logs/
+
+# Local secrets
+.env.local
+.env.*.local
+
+# Local working docs (session handoffs — never published)
+HANDOFF.md
+
+# Internal planning docs (kept locally, not published)
+PLAN.md
+RECALL-PLAN.md

memstem-0.17.0/.pre-commit-config.yaml

@@ -0,0 +1,36 @@
+repos:
+  - repo: https://github.com/pre-commit/pre-commit-hooks
+    rev: v4.6.0
+    hooks:
+      - id: trailing-whitespace
+      - id: end-of-file-fixer
+      - id: check-yaml
+      - id: check-added-large-files
+        args: ['--maxkb=500']
+      - id: check-merge-conflict
+      - id: detect-private-key
+
+  - repo: https://github.com/astral-sh/ruff-pre-commit
+    rev: v0.15.12
+    hooks:
+      - id: ruff
+        args: [--fix]
+      - id: ruff-format
+
+  - repo: https://github.com/pre-commit/mirrors-mypy
+    rev: v1.20.2
+    hooks:
+      - id: mypy
+        additional_dependencies:
+          - pydantic>=2.0.0
+          - pytest>=8.0.0
+          - watchdog>=4.0.0
+          - sqlite-vec>=0.1.0
+          - python-frontmatter>=1.0.0
+          - httpx>=0.27.0
+          - typer>=0.12.0
+          - mcp>=1.0.0
+          - pyyaml>=6.0
+          - aiofiles>=23.0.0
+          - fastapi>=0.110.0
+          - uvicorn>=0.30.0

memstem-0.17.0/.python-version

@@ -0,0 +1 @@
+3.11

memstem-0.17.0/ARCHITECTURE.md

@@ -0,0 +1,175 @@
+# Architecture
+
+## Goals
+
+1. **Single canonical store** for memories and skills shared across multiple AI clients
+2. **Immunity to upgrade churn** in any individual AI client
+3. **Sub-second ingestion** of new content from any source
+4. **Hybrid search** — keyword + semantic — with no remote API dependencies (optional)
+5. **Human-readable** canonical layer (markdown + frontmatter)
+6. **MCP-native** integration
+
+## Layered design
+
+```
+┌──────────────────────────────────────────────────────────┐
+│                    AI Clients                            │
+│  Claude Code  │  OpenClaw  │  Codex  │  Cursor  │  ...   │
+└──────────────────────────────────────────────────────────┘
+                          ▲ MCP / HTTP
+                          │
+┌──────────────────────────────────────────────────────────┐
+│                  Memstem Daemon                          │
+│  ┌─────────────┐  ┌──────────────┐  ┌────────────────┐  │
+│  │   Servers   │  │   Hygiene    │  │   Adapters     │  │
+│  │  MCP / HTTP │  │ Worker (bg)  │  │ Watchers (bg)  │  │
+│  └─────────────┘  └──────────────┘  └────────────────┘  │
+│         ▲                ▲                  ▲           │
+│         └────────────────┼──────────────────┘           │
+│                          │                              │
+│  ┌─────────────────────────────────────────────────┐    │
+│  │            Index (SQLite + FTS5 + vec)          │    │
+│  └─────────────────────────────────────────────────┘    │
+│                          ▲                              │
+│  ┌─────────────────────────────────────────────────┐    │
+│  │      Canonical Storage (markdown vault)         │    │
+│  │  memories/  skills/  sessions/  daily/          │    │
+│  └─────────────────────────────────────────────────┘    │
+└──────────────────────────────────────────────────────────┘
+                          ▲
+                          │ inotify watches
+┌──────────────────────────────────────────────────────────┐
+│              External AI Filesystems                     │
+│  ~/.claude/projects/  ~/<agent>/memory/  ~/.codex/sessions/   │
+└──────────────────────────────────────────────────────────┘
+```
+
+## Two-layer storage
+
+### Canonical: markdown files in a structured tree
+
+```
+~/memstem-vault/
+├── memories/
+│   ├── people/
+│   │   └── brad-besner.md
+│   ├── decisions/
+│   │   └── 2026-04-25-deploy-via-cloudflare.md
+│   └── facts/
+│       └── port-assignments.md
+├── skills/
+│   ├── deploy-to-kinsta.md
+│   └── send-telegram.md
+├── sessions/
+│   └── 2026-04-25/
+│       ├── claude-code-abc123.md
+│       └── openclaw-def456.md
+├── daily/
+│   └── 2026-04-25.md
+└── _meta/
+    ├── taxonomy.md
+    └── config.yaml
+```
+
+Every file has YAML frontmatter (see [frontmatter-spec.md](./docs/frontmatter-spec.md)). Files survive any version of any tool, are diffable, git-friendly, and openable in any markdown editor.
+
+### Index: SQLite with FTS5 + sqlite-vec
+
+A single `index.db` file rebuilt from the canonical store at any time. FTS5 provides BM25 keyword retrieval; sqlite-vec provides cosine similarity over embeddings. Hybrid queries use Reciprocal Rank Fusion (RRF) to merge.
+
+If the index is ever corrupted, lost, or becomes incompatible: `memstem reindex` rebuilds it from the canonical files. The truth is never at risk.
+
+## Pull-based ingestion
+
+Each adapter is a small module that knows how to read one external AI's filesystem.
+
+- **Claude Code adapter**: watches `~/.claude/projects/*/sessions/*.jsonl` with `inotify`, extracts user/assistant turns, dedupes, writes a clean memory record into `memories/sessions/`.
+- **OpenClaw adapter**: watches each configured agent workspace’s `memory/`, `skills/`, and daily logs. Mirrors directly into the canonical vault.
+- **Codex adapter** (planned): watches `~/.codex/sessions/`.
+- **Cursor adapter** (planned): watches Cursor's local memory store.
+- **Generic file adapter** (planned): watches an arbitrary directory provided by the user.
+
+Adapters emit normalized memory records. Storage and indexing are downstream — adapters never touch the index directly.
+
+A 5-minute reconciliation pass runs alongside `inotify` to catch anything missed (crashes, race conditions, files moved out-of-band).
+
+## Hybrid search
+
+Every query goes through three stages:
+
+1. **Embed the query** using the configured embedding backend. Memstem ships four pluggable implementations: `OllamaEmbedder` (default, local, `nomic-embed-text` 768d), `OpenAIEmbedder` (with `base_url` knob for OpenAI-compatible providers), `GeminiEmbedder` (`gemini-embedding-2-preview` default with Matryoshka support, so 768d Ollama indexes can switch over without reindexing), and `VoyageEmbedder` (Anthropic's recommended partner). Backend and dimensions are configured in `_meta/config.yaml`; see ADR 0009.
+2. **Run two retrievals in parallel**:
+   - FTS5 BM25 over the markdown body + frontmatter tags
+   - sqlite-vec cosine similarity over chunk embeddings
+3. **Merge with RRF** (k=60 default): combined ranking is the normalized inverse-rank sum from both retrievers.
+
+Optional third signal (planned): entity-link retrieval over wikilinks (`[[Entity]]`) extracted at ingest time.
+
+Optional fourth signal (planned): recency + importance score from the hygiene worker.
+
+## Hygiene worker
+
+Runs in a background thread, processing the canonical store at a low priority:
+
+- **Dedup**: pairs with cosine similarity > 0.95 are merged; the higher-importance record wins, the duplicate becomes a redirect.
+- **Decay**: importance score decays over time; bursts of recall raise it.
+- **Bi-temporal validity** (planned): when a fact contradicts an existing one, the old one gets `valid_to: <date>` rather than being deleted.
+
+The hygiene package also hosts two CLI-driven derived-record writers
+(see ADR 0020 and ADR 0021) that turn raw transcripts and project
+session sets into retrieval-shaped records:
+
+- **Session distillation** (`memstem hygiene distill-sessions`): one
+  `type: distillation` per meaningful session, link-back via
+  frontmatter. Importance seeded above raw sessions so the existing
+  multiplier surfaces them on close ties.
+- **Project records** (`memstem hygiene project-records`): one
+  `type: project` per Claude Code project tag with ≥2 sessions,
+  preferring linked distillations as input. Hand-edited records
+  carrying `manual: true` are protected from regeneration.
+
+Both run via the same pluggable `Summarizer` abstraction
+(`core/summarizer.py`) — NoOp default, OpenAI / Ollama opt-in. Their
+outputs are **derivative artifacts with mandatory provenance**, which
+is the boundary that keeps them inside the read-files-from-disk
+architecture.
+
+> Note: MemStem does **not** author skills. Each AI generates skills its
+> own way (Claude Code, Codex, Hermes, OpenClaw all have their own
+> conventions); MemStem reads `SKILL.md` files from disk and indexes
+> them. Distillations and project records are explicitly different —
+> they summarize source records with mandatory link-back rather than
+> producing standalone knowledge claims. See
+> [ADR 0019](decisions/0019-no-skill-authoring.md) for the boundary
+> rule and [ADR 0020](decisions/0020-session-distillation-writer.md) /
+> [ADR 0021](decisions/0021-project-records.md) for the
+> distillation / project-record designs.
+
+## API surface
+
+**MCP tools** (primary):
+
+- `memstem_search(query, limit=10, types=[...]) -> [Result]`
+- `memstem_get(id_or_path) -> Memory`
+- `memstem_list_skills(scope=...) -> [Skill]`
+- `memstem_get_skill(name) -> Skill`
+- `memstem_upsert(content, frontmatter) -> Memory` (write path)
+
+**HTTP API** (secondary): same shape under `/api/v1/`.
+
+**Anthropic memory-tool adapter** (planned flagship feature): implements `BetaAbstractMemoryTool` so Claude Code's official memory tool routes natively into Memstem.
+
+See [docs/mcp-api.md](./docs/mcp-api.md) for full tool definitions.
+
+## Configuration
+
+A single `~/memstem-vault/_meta/config.yaml` file controls the daemon. See [docs/configuration.md](./docs/configuration.md) for the full schema (forthcoming).
+
+## What Memstem is not
+
+- Not a chat memory layer (that's mem0's space)
+- Not a graph database (Graphiti, Letta)
+- Not a managed cloud service (yet — see [ROADMAP.md](./ROADMAP.md))
+- Not a wiki engine or note-taking app
+
+Memstem is **infrastructure for AI agents to share knowledge without coupling to each other**.