embedsync 0.4.0__tar.gz

embedsync-0.4.0/.env.example

@@ -0,0 +1,2 @@
+EMBEDSYNC_STATE_DB=.embedsync/state.db
+EMBEDSYNC_LOG_LEVEL=INFO

embedsync-0.4.0/.github/workflows/ci.yml

@@ -0,0 +1,11 @@
+name: CI
+on: [push, pull_request]
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+      - uses: actions/setup-python@v5
+        with: {python-version: "3.12"}
+      - run: pip install -e ".[dev]"
+      - run: pytest tests/ -v

embedsync-0.4.0/.gitignore

@@ -0,0 +1,5 @@
+__pycache__/
+.venv/
+.pytest_cache/
+.embedsync/
+dist/

embedsync-0.4.0/CHANGELOG.md

@@ -0,0 +1,29 @@
+# Changelog
+
+## [0.4.0] - 2026-09-14
+
+### Added
+- Ollama embedder via `--embedder ollama` or `--embedder ollama:<model>` (default model `nomic-embed-text`)
+
+## [0.3.0] - 2026-08-19
+
+### Added
+- Chunk-level re-embed: unchanged chunks are skipped on document edit
+- SQLite `chunks` table in the state store
+
+## [0.2.0] - 2026-08-19
+
+### Added
+- Hash embedder for offline/CI runs (`--embedder hash`)
+- Paragraph chunking with stable-ish chunk IDs
+- JSONL destination (`--destination jsonl:/path`)
+- Unchanged documents write 0 embeddings on the second run
+
+### Notes
+- Real pgvector/Qdrant destinations are still open
+
+## [0.1.0] - 2026-08-18
+
+### Added
+- Local markdown source, SQLite state store, sync plan engine
+- Memory destination stub and dry-run mode

embedsync-0.4.0/CONTRIBUTING.md

@@ -0,0 +1,30 @@
+# Contributing
+
+## Running tests
+
+Prefer Docker Compose:
+
+```bash
+docker compose run --rm test
+docker compose run --rm plan
+```
+
+Locally:
+
+```bash
+pip install -e ".[dev]"
+pytest tests/ -v
+embedsync plan examples/docs --state-db /tmp/embedsync-demo.db
+embedsync run examples/docs --dry-run --state-db /tmp/embedsync-demo.db
+```
+
+## Pull requests
+
+- Keep embedder changes covered by unit tests (mock Ollama where needed)
+- Update README/CHANGELOG for new destinations or CLI flags
+- Prefer small PRs with a clear test plan
+
+## Commit style
+
+- Imperative subject line; mention the user-facing why when relevant
+- Do not add AI co-author trailers (e.g. Co-authored-by: Cursor) to commits.

embedsync-0.4.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 embedsync contributors
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

embedsync-0.4.0/PKG-INFO

@@ -0,0 +1,128 @@
+Metadata-Version: 2.5
+Name: embedsync
+Version: 0.4.0
+Summary: Incremental synchronization between source documents and vector indexes
+Project-URL: Homepage, https://github.com/yashshah9/embedsync
+Project-URL: Repository, https://github.com/yashshah9/embedsync
+Project-URL: Issues, https://github.com/yashshah9/embedsync/issues
+Author-email: Yash Shah <yash376351@gmail.com>
+License-Expression: MIT
+License-File: LICENSE
+Requires-Python: >=3.11
+Requires-Dist: click>=8.1
+Requires-Dist: pydantic-settings>=2.2
+Requires-Dist: pydantic>=2.6
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: rich>=13.7
+Requires-Dist: structlog>=24.1
+Provides-Extra: dev
+Requires-Dist: mypy>=1.9; extra == 'dev'
+Requires-Dist: pytest>=8.0; extra == 'dev'
+Requires-Dist: ruff>=0.4; extra == 'dev'
+Description-Content-Type: text/markdown
+
+# embedsync
+
+Incremental synchronization between **source documents** and **vector indexes** — detect changes, re-embed only deltas, and delete stale chunks.
+
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
+[![Python 3.11+](https://img.shields.io/badge/python-3.11+-blue.svg)](https://www.python.org/downloads/)
+[![CI](https://github.com/yashshah9/embedsync/actions/workflows/ci.yml/badge.svg)](https://github.com/yashshah9/embedsync/actions/workflows/ci.yml)
+
+> **Status:** v0.4 — hash + Ollama embedders, paragraph chunks, JSONL destination, chunk-level re-embed.
+
+## 60-second try
+
+```bash
+docker compose run --rm plan  # plan sync for examples/docs
+docker compose run --rm test  # pytest
+```
+
+## Why this vs alternatives
+
+| Approach | Strength | Gap |
+|----------|----------|-----|
+| **embedsync** | Content-hash deltas + pluggable embedders | Destinations still local (JSONL/memory) |
+| Full re-embed pipelines | Simple mentally | Expensive; misses deletes |
+| Framework ingestion (e.g. LlamaIndex) | Rich connectors | Change detection is DIY |
+| One-off sync scripts | Fits one repo | No shared plan/state model |
+
+## Problem
+
+RAG indexes rot when documents change. Full re-embeds are expensive and miss deletes. Every team rebuilds change detection from scratch.
+
+## Key features (v0.4)
+
+- Content-hash change detection per document
+- Sync plan: add / update / delete actions
+- Hash embedder for offline/CI (`--embedder hash`)
+- Ollama embedder (`--embedder ollama` or `ollama:nomic-embed-text`)
+- JSONL or in-memory destination
+- Unchanged docs/chunks skip re-embedding on the next run
+
+## Architecture
+
+```
+embedsync run ./docs
+    ├── LocalFileSource
+    ├── StateStore (SQLite)
+    ├── plan_sync() → diff
+    └── Destination (Memory → pgvector next)
+```
+
+## Installation
+
+```bash
+pip install embedsync
+pip install -e ".[dev]"
+```
+
+## Usage
+
+```bash
+embedsync health
+embedsync plan examples/docs --state-db /tmp/embedsync-demo.db
+embedsync run examples/docs --dry-run --state-db /tmp/embedsync-demo.db
+embedsync run examples/docs --embedder hash --destination memory --state-db /tmp/embedsync-demo.db
+embedsync run examples/docs --embedder hash --destination jsonl:/tmp/index.jsonl
+# Requires a running Ollama with an embedding model:
+embedsync run examples/docs --embedder ollama --destination jsonl:/tmp/index.jsonl
+embedsync run examples/docs --embedder ollama:nomic-embed-text --destination memory
+```
+
+## Docker
+
+```bash
+docker compose run --rm test
+docker compose run --rm plan
+```
+
+## Configuration
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `EMBEDSYNC_STATE_DB` | `.embedsync/state.db` | State database path |
+| `EMBEDSYNC_LOG_LEVEL` | `INFO` | Log level |
+
+Ollama uses `OLLAMA_HOST` when set (otherwise the embedder default host).
+
+## Roadmap
+
+- [x] Pluggable embedder protocol + hash backend
+- [x] JSONL destination (local stand-in)
+- [x] Chunk-level stable IDs across edits
+- [x] Ollama embedder (`--embedder ollama`)
+- [ ] pgvector and Qdrant destinations
+- [ ] Notion and sitemap sources
+
+## License
+
+MIT
+
+## Known limitations (v0.4)
+
+- Hash embeddings are not semantic — use `--embedder ollama` for local semantic vectors
+- JSONL is not a vector DB
+- Local markdown files only
+- Re-runs reuse `.embedsync/state.db`; pass `--state-db` for an isolated plan
+- Ollama must already be running and have the embedding model pulled

embedsync-0.4.0/README.md

@@ -0,0 +1,105 @@
+# embedsync
+
+Incremental synchronization between **source documents** and **vector indexes** — detect changes, re-embed only deltas, and delete stale chunks.
+
+[![License: MIT](https://img.shields.io/badge/License-MIT-yellow.svg)](LICENSE)
+[![Python 3.11+](https://img.shields.io/badge/python-3.11+-blue.svg)](https://www.python.org/downloads/)
+[![CI](https://github.com/yashshah9/embedsync/actions/workflows/ci.yml/badge.svg)](https://github.com/yashshah9/embedsync/actions/workflows/ci.yml)
+
+> **Status:** v0.4 — hash + Ollama embedders, paragraph chunks, JSONL destination, chunk-level re-embed.
+
+## 60-second try
+
+```bash
+docker compose run --rm plan  # plan sync for examples/docs
+docker compose run --rm test  # pytest
+```
+
+## Why this vs alternatives
+
+| Approach | Strength | Gap |
+|----------|----------|-----|
+| **embedsync** | Content-hash deltas + pluggable embedders | Destinations still local (JSONL/memory) |
+| Full re-embed pipelines | Simple mentally | Expensive; misses deletes |
+| Framework ingestion (e.g. LlamaIndex) | Rich connectors | Change detection is DIY |
+| One-off sync scripts | Fits one repo | No shared plan/state model |
+
+## Problem
+
+RAG indexes rot when documents change. Full re-embeds are expensive and miss deletes. Every team rebuilds change detection from scratch.
+
+## Key features (v0.4)
+
+- Content-hash change detection per document
+- Sync plan: add / update / delete actions
+- Hash embedder for offline/CI (`--embedder hash`)
+- Ollama embedder (`--embedder ollama` or `ollama:nomic-embed-text`)
+- JSONL or in-memory destination
+- Unchanged docs/chunks skip re-embedding on the next run
+
+## Architecture
+
+```
+embedsync run ./docs
+    ├── LocalFileSource
+    ├── StateStore (SQLite)
+    ├── plan_sync() → diff
+    └── Destination (Memory → pgvector next)
+```
+
+## Installation
+
+```bash
+pip install embedsync
+pip install -e ".[dev]"
+```
+
+## Usage
+
+```bash
+embedsync health
+embedsync plan examples/docs --state-db /tmp/embedsync-demo.db
+embedsync run examples/docs --dry-run --state-db /tmp/embedsync-demo.db
+embedsync run examples/docs --embedder hash --destination memory --state-db /tmp/embedsync-demo.db
+embedsync run examples/docs --embedder hash --destination jsonl:/tmp/index.jsonl
+# Requires a running Ollama with an embedding model:
+embedsync run examples/docs --embedder ollama --destination jsonl:/tmp/index.jsonl
+embedsync run examples/docs --embedder ollama:nomic-embed-text --destination memory
+```
+
+## Docker
+
+```bash
+docker compose run --rm test
+docker compose run --rm plan
+```
+
+## Configuration
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `EMBEDSYNC_STATE_DB` | `.embedsync/state.db` | State database path |
+| `EMBEDSYNC_LOG_LEVEL` | `INFO` | Log level |
+
+Ollama uses `OLLAMA_HOST` when set (otherwise the embedder default host).
+
+## Roadmap
+
+- [x] Pluggable embedder protocol + hash backend
+- [x] JSONL destination (local stand-in)
+- [x] Chunk-level stable IDs across edits
+- [x] Ollama embedder (`--embedder ollama`)
+- [ ] pgvector and Qdrant destinations
+- [ ] Notion and sitemap sources
+
+## License
+
+MIT
+
+## Known limitations (v0.4)
+
+- Hash embeddings are not semantic — use `--embedder ollama` for local semantic vectors
+- JSONL is not a vector DB
+- Local markdown files only
+- Re-runs reuse `.embedsync/state.db`; pass `--state-db` for an isolated plan
+- Ollama must already be running and have the embedding model pulled

embedsync-0.4.0/SECURITY.md

@@ -0,0 +1,14 @@
+# Security Policy
+
+## Reporting a vulnerability
+
+Email **yash376351@gmail.com** with the repo name, a short description, and steps to reproduce. Please do not open a public issue for exploitable findings until we have had a reasonable chance to respond.
+
+## Threat model (honest)
+
+embedsync reads local documents, stores sync state in SQLite, and may call a local Ollama (or other configured) embedder.
+
+- It is **not** a vector database security boundary.
+- Document content is sent to the embedder you select (`hash` stays local; `ollama` hits your Ollama host).
+- State DBs and JSONL destinations may contain document text and embeddings — treat them as sensitive if the source corpus is.
+- There is no multi-tenant isolation; run one sync workspace per trust domain.

embedsync-0.4.0/compose.yaml

@@ -0,0 +1,11 @@
+services:
+  test:
+    build: {context: ., dockerfile: docker/Dockerfile}
+    volumes: [".:/app"]
+    working_dir: /app
+    command: ["pytest", "tests/", "-v"]
+  plan:
+    build: {context: ., dockerfile: docker/Dockerfile}
+    volumes: [".:/app"]
+    working_dir: /app
+    command: ["embedsync", "plan", "examples/docs", "--state-db", "/tmp/embedsync-demo.db"]

embedsync-0.4.0/docker/Dockerfile

@@ -0,0 +1,8 @@
+FROM python:3.12-slim AS dev
+WORKDIR /app
+COPY pyproject.toml README.md ./
+COPY src ./src
+COPY examples ./examples
+COPY tests ./tests
+RUN pip install --no-cache-dir -e ".[dev]"
+CMD ["pytest", "tests/", "-v"]

embedsync-0.4.0/examples/docs/api.md

@@ -0,0 +1,3 @@
+# API Reference
+
+Endpoint documentation goes here.

embedsync-0.4.0/examples/docs/getting-started.md

@@ -0,0 +1,5 @@
+# Getting Started
+
+Welcome to our docs.
+
+This is a sample document that will be indexed.

embedsync-0.4.0/plans/README.md

@@ -0,0 +1,6 @@
+# embedsync — Planning Documents
+
+| Document | Status | Description |
+|----------|--------|-------------|
+| [v0.1-completed-foundation.md](./v0.1-completed-foundation.md) | **Completed** | Initial foundation release |
+| [v0.2-next-phase-roadmap.md](./v0.2-next-phase-roadmap.md) | **In Progress** | v0.2.0 shipped hash embedder + JSONL; pgvector still open |

embedsync-0.4.0/plans/v0.1-completed-foundation.md

@@ -0,0 +1,220 @@
+# v0.1 Completed Foundation Plan — embedsync
+
+**Status:** Completed  
+**Version:** 0.1.0  
+**Completed:** 2026-08-18  
+**Owner:** Yash Shah  
+
+---
+
+## 1. Executive Summary
+
+Deliver a **minimal viable open-source foundation** for incremental document-to-vector-index synchronization. v0.1 focuses on content-hash change detection, add/update/delete sync planning, SQLite state persistence, local markdown sources, and an in-memory destination with dry-run CLI — zero embedding API calls, zero vector DB infrastructure.
+
+---
+
+## 2. Problem Statement
+
+- RAG indexes rot when source documents change; full re-embeds are expensive and miss deletes
+- Every team rebuilds change detection, state tracking, and delta sync from scratch
+- Existing tools either full-reindex or require heavyweight orchestration
+- No standard CLI for "diff documents against last sync state and plan vector index mutations"
+
+---
+
+## 3. Goals (v0.1)
+
+| # | Goal | Status |
+|---|------|--------|
+| G1 | Provide CLI installable Python package | Done |
+| G2 | Content-hash change detection per document | Done |
+| G3 | Sync plan: add / update / delete actions | Done |
+| G4 | SQLite state store (no extra infrastructure) | Done |
+| G5 | Local markdown file source | Done |
+| G6 | In-memory destination for testing and dry-run | Done |
+| G7 | `--dry-run` mode on `run` command | Done |
+| G8 | Docker-based dev and test environment | Done |
+| G9 | Contributor-ready repo (README, CI, LICENSE) | Done |
+
+## 4. Non-Goals (v0.1)
+
+- Real vector database destinations (pgvector, Qdrant)
+- Embedding API calls or pluggable embedders
+- Chunk-level stable IDs across document edits
+- Notion, sitemap, or remote document sources
+- Chunk-level diff (document-level hash only)
+- PyPI publish (deferred to v0.2)
+- Web UI or hosted sync dashboard
+
+---
+
+## 5. Architecture Delivered
+
+```
+CLI (Click + Rich)
+  ├── health              → version + sanity check
+  ├── plan <source_dir>   → plan_sync() → Rich table (dry)
+  └── run <source_dir>    → execute_sync() with --dry-run flag
+
+Sync Engine
+  ├── plan_sync()         → diff source vs state → SyncPlan
+  └── execute_sync()      → apply actions to destination + update state
+
+Sources
+  └── LocalFileSource     → glob **/*.md, relative path as doc_id
+
+State
+  └── StateStore (SQLite) → documents(doc_id, content_hash, chunk_count)
+
+Destinations
+  └── MemoryDestination   → in-memory dict for testing/dry-run
+
+Config
+  └── Settings (pydantic-settings) → EMBEDSYNC_* env vars
+```
+
+### Sync Flow
+
+```
+LocalFileSource.list_documents()
+    └── for each doc:
+            ├── content_hash(SHA-256) vs StateStore.get()
+            ├── new doc        → ADD
+            ├── hash changed   → UPDATE
+            └── missing in source → DELETE (from state)
+                    └── SyncPlan → Destination.apply() → StateStore upsert/delete
+```
+
+### Technology Choices
+
+| Layer | Choice | Rationale |
+|-------|--------|-----------|
+| Language | Python 3.11+ | ML/RAG ecosystem familiarity |
+| CLI | Click | Mature, composable subcommands |
+| Terminal UX | Rich | Colored ADD/UPDATE/DELETE plan table |
+| State | SQLite (stdlib) | Zero infra; portable single file |
+| Hashing | SHA-256 (stdlib) | Fast, deterministic content fingerprint |
+| Config | pydantic-settings | Env-based settings with `.env` support |
+| Logging | structlog | Structured logs per sync action |
+| Packaging | hatchling | Modern pyproject.toml builds |
+| Quality | ruff, mypy, pytest | Standard Python OSS toolchain |
+
+---
+
+## 6. Deliverables Completed
+
+### 6.1 Source Code
+
+- [x] `src/embedsync/cli.py` — `health`, `plan`, `run` commands
+- [x] `src/embedsync/sync/engine.py` — `plan_sync()`, `execute_sync()`, `SyncPlan`
+- [x] `src/embedsync/state/store.py` — `StateStore`, `DocumentState`, `content_hash()`
+- [x] `src/embedsync/sources/local.py` — `LocalFileSource`, `SourceDocument`
+- [x] `src/embedsync/destinations/memory.py` — `MemoryDestination`, `SyncAction`
+- [x] `src/embedsync/config.py` — `Settings` with env prefix
+- [x] `src/embedsync/__init__.py` — package version
+
+### 6.2 Sync Actions
+
+| Action | Trigger | Destination Effect | State Effect |
+|--------|---------|-------------------|--------------|
+| ADD | doc_id not in state | Index doc (chunk count recorded) | upsert hash + chunk_count |
+| UPDATE | content_hash changed | Re-index doc | upsert new hash |
+| DELETE | doc_id in state but not in source | Remove from index | delete row |
+
+### 6.3 State Schema (SQLite)
+
+| Column | Type | Description |
+|--------|------|-------------|
+| `doc_id` | TEXT PRIMARY KEY | Relative path from source root |
+| `content_hash` | TEXT NOT NULL | SHA-256 hex digest of document content |
+| `chunk_count` | INTEGER | Heuristic chunk count (not real embedding yet) |
+
+### 6.4 Chunk Heuristic (v0.1)
+
+| Parameter | Value | Notes |
+|-----------|-------|-------|
+| Chunk size | 500 chars | Fixed heuristic; `ponytail:` comment in engine |
+| Formula | `max(1, len(content) // 500)` | Placeholder until real chunker in v0.2 |
+
+### 6.5 Examples & Fixtures
+
+- [x] `examples/docs/getting-started.md` — sample document
+- [x] `examples/docs/api.md` — sample document
+
+### 6.6 Tests
+
+- [x] `tests/test_engine.py` — plan detects 2 new docs; run indexes 2 docs
+- [x] All tests passing in Docker (Python 3.12)
+
+### 6.7 DevOps & Documentation
+
+- [x] `docker/Dockerfile` — multi-stage dev/runtime
+- [x] `compose.yaml` — `test` and `plan` services
+- [x] `.github/workflows/ci.yml` — ruff, mypy, pytest
+- [x] `README.md`, `CHANGELOG.md`, `LICENSE`
+- [x] `pyproject.toml` — hatchling build, `[project.scripts]` entry point
+
+---
+
+## 7. CLI Reference (v0.1)
+
+| Command | Description | Key Options |
+|---------|-------------|-------------|
+| `embedsync health` | Print version and OK status | — |
+| `embedsync plan <dir>` | Show ADD/UPDATE/DELETE plan without applying | `--state-db` |
+| `embedsync run <dir>` | Execute sync plan against destination | `--dry-run`, `--state-db` |
+
+### Configuration
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `EMBEDSYNC_STATE_DB` | `.embedsync/state.db` | SQLite state database path |
+| `EMBEDSYNC_LOG_LEVEL` | `INFO` | Log level |
+| `EMBEDSYNC_DRY_RUN` | `false` | Default dry-run (CLI flag overrides) |
+
+---
+
+## 8. Acceptance Criteria (Met)
+
+| # | Criterion | Verification |
+|---|-----------|--------------|
+| AC1 | `embedsync plan examples/docs` shows 2 ADD actions on first run | Manual + Docker `plan` |
+| AC2 | `embedsync run examples/docs` indexes 2 documents in MemoryDestination | test_engine.py |
+| AC3 | Second run with no changes produces 0 actions | Manual |
+| AC4 | Content edit triggers UPDATE on subsequent plan | Manual |
+| AC5 | Deleted file triggers DELETE on subsequent plan | Manual |
+| AC6 | `--dry-run` applies no state or destination mutations | Manual |
+| AC7 | `docker compose run --rm test` passes all tests | CI-ready |
+| AC8 | No secrets or credentials in repository | .gitignore |
+
+---
+
+## 9. Known Limitations (v0.1)
+
+- Local markdown files only (`**/*.md` glob)
+- In-memory destination — no real vector DB writes
+- Document-level hash only — any edit re-indexes entire document
+- Naive fixed-size chunk count heuristic — no real chunking or embedding
+- No embedding API calls — chunk_count is a placeholder integer
+- No concurrent sync or locking — single-process CLI only
+- No Notion, sitemap, or HTTP sources
+
+---
+
+## 10. Metrics at Completion
+
+| Metric | Value |
+|--------|-------|
+| Source files | 7 Python modules |
+| Test cases | 2 |
+| CLI commands | 3 (`health`, `plan`, `run`) |
+| Sync action types | 3 (add, update, delete) |
+| Source types | 1 (local markdown) |
+| Destination types | 1 (in-memory) |
+| Docker services | 2 (`test`, `plan`) |
+
+---
+
+## 11. Sign-Off
+
+v0.1 foundation is **complete and shippable** as an alpha OSS project. The sync engine, SQLite state, and dry-run CLI provide immediate value for planning document-to-index deltas without embedding cost. pgvector/Qdrant destinations, pluggable embedders, stable chunk IDs, and remote sources are deferred to v0.2 per roadmap.