docsif 0.1.0__tar.gz

docsif-0.1.0/.github/workflows/docs.yml

@@ -0,0 +1,38 @@
+name: Docs Validation
+
+on:
+  pull_request:
+    paths:
+      - 'docs/**'
+      - 'README.md'
+      - 'src/sif/cli/**'
+      - 'src/sif/config/**'
+      - 'scripts/generate_cli_ref.py'
+      - 'scripts/generate_config_ref.py'
+      - 'tests/test_docs.py'
+  push:
+    branches: [main]
+
+jobs:
+  docs-test:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@v4
+
+      - uses: actions/setup-python@v5
+        with:
+          python-version: '3.11'
+
+      - name: Install dependencies
+        run: |
+          pip install --upgrade pip
+          pip install -e ".[dev]"
+
+      - name: Run docs tests
+        run: pytest tests/test_docs.py -v
+
+      - name: Verify generation scripts work
+        run: |
+          python scripts/generate_cli_ref.py
+          python scripts/generate_config_ref.py
+          git diff --exit-code docs/cli-reference.md docs/configuration.md || true

docsif-0.1.0/.gitignore

@@ -0,0 +1,46 @@
+# Python
+__pycache__/
+*.py[cod]
+*$py.class
+*.so
+.Python
+build/
+develop-eggs/
+dist/
+downloads/
+eggs/
+.eggs/
+lib/
+lib64/
+parts/
+sdist/
+var/
+wheels/
+*.egg-info/
+.installed.cfg
+*.egg
+
+# Virtual environments
+.venv/
+venv/
+ENV/
+env/
+
+# IDE
+.vscode/
+.idea/
+*.swp
+*.swo
+*~
+
+# Testing
+.coverage
+coverage.xml
+htmlcov/
+.pytest_cache/
+
+# OS
+.DS_Store
+Thumbs.db
+/models/
+.claude/

docsif-0.1.0/.planning/.continue-here.md

@@ -0,0 +1,70 @@
+---
+context: default
+phase: e2e-verification
+status: in_progress
+last_updated: "2026-04-27T12:53:33Z"
+---
+
+# BLOCKING CONSTRAINTS — Read Before Anything Else
+
+- [x] **CONSTRAINT: SQLite DDL triggers implicit commit** — Creating tables inside a transaction that later rolls back will NOT roll back the DDL. Structural mitigation: never assert DDL rollback in SQLite tests.
+- [x] **CONSTRAINT: macOS load_extension("vec0") fails** — Use `sqlite_vec.load(conn)` from the Python package instead.
+- [x] **CONSTRAINT: Qwen3-Reranker-0.6B is NOT a CrossEncoder** — It requires `AutoModelForCausalLM` with yes/no token logit prediction.
+- [x] **CONSTRAINT: Stored document paths are full resolved paths** — FIXED: `get_by_filename()` added as fallback when path lookup fails.
+
+<current_state>
+All 3 E2E bugs fixed and verified. MPS (Apple GPU) support added to embedder. Automated E2E passed 13/13. User paused to run manual E2E test against real Obsidian vault. All changes uncommitted.
+</current_state>
+
+<completed_work>
+
+1. Fixed ~300+ ruff issues (UP045, UP006, SIM108, B904, PLC0415, complexity, PERF401, etc.)
+2. Fixed datetime.utcnow() → datetime.now(timezone.utc) across all src and test files
+3. Fixed 9 PytestReturnNotNoneWarning in test_sif_complete.py
+4. All 378 tests pass, 0 warnings
+5. Fixed Bug 1 & 2: `get_by_filename()` fallback in get_cmd and multi_get_cmd
+6. Fixed Bug 3: `_row_to_collection` now reads document_count/chunk_count from DB
+7. Added `mark_indexed()` method to Collection dataclass
+8. Added MPS (Apple GPU) device detection: CUDA > MPS > CPU
+9. Automated E2E verification: 13/13 tests passed
+10. Manual test plan written at `.planning/E2E-manual-test-plan.md`
+</completed_work>
+
+<remaining_work>
+
+- **Commit all changes** — ~55 files modified, nothing committed yet
+- **Run manual E2E test** — test plan at `.planning/E2E-manual-test-plan.md` against `/Users/forrest/SynologyDrive/Obsidian/ScholarCopilot-Dev/raw/markdown` (307 papers, 266MB)
+- **Re-run embed with MPS** — `sif index embed -c scholar-test` should now use MPS instead of CPU
+</remaining_work>
+
+<decisions_made>
+
+- Added noqa for Click-inherent complexity (structural, not fixable)
+- Added noqa for intentional lazy imports (ML libraries)
+- Used lambda wrappers for datetime in field default_factory
+- MPS device detection: CUDA > MPS > CPU for both SentenceTransformerEmbedder and ModelScopeEmbedder
+- Get by filename: path lookup first, then filename column fallback
+- Collection stats: _row_to_collection must read all DB columns
+</decisions_made>
+
+<blockers>
+None.
+</blockers>
+
+## Required Reading (in order)
+
+1. `.planning/HANDOFF.json` — structured machine-readable state
+2. `.planning/E2E-manual-test-plan.md` — manual test plan for real Obsidian vault
+3. `.planning/E2E-UAT.md` — automated E2E results (13/13 passed)
+4. `.planning/debug/e2e-get-and-stats-bugs.md` — debug session with root cause analysis
+
+## Infrastructure State
+- SIF CLI installed and working (`sif --version` → 0.1.0)
+- Test DB at `/tmp/sif-e2e-test.sqlite` cleaned up
+- MPS available: confirmed (`torch.backends.mps.is_available()` → True)
+- Model cached: Qwen/Qwen3-Embedding-0.6B at `/Users/forrest/Library/Caches/sif/`
+- User may have a partial embed in progress for `scholar-test` collection
+
+<next_action>
+Commit all changes first. Then user runs manual E2E test plan: re-index and embed with MPS enabled for real Obsidian vault.
+</next_action>

docsif-0.1.0/.planning/E2E-UAT.md

@@ -0,0 +1,90 @@
+---
+status: complete
+phase: E2E
+source: ROADMAP.md (all phases)
+started: "2026-04-27T12:00:00Z"
+updated: "2026-04-27T12:30:00Z"
+---
+
+## Current Test
+
+[testing complete]
+
+## Tests
+
+### 1. Cold Start Smoke Test
+expected: sif --version prints version string. sif status runs without error on a fresh temp database (no prior state).
+result: pass
+note: sif --version prints "sif, version 0.1.0". sif status on empty DB says "No index found" gracefully.
+
+### 2. Collection Add and List
+expected: sif collection add creates a collection. sif collection list shows it.
+result: pass
+note: Used `sif collection add /tmp/sif-e2e-test-docs -n test-e2e`. Collection listed with path, document count, default flag.
+
+### 3. Index Documents into Collection
+expected: sif index update indexes 3 documents with progress output.
+result: pass
+note: `sif index update -c test-e2e` found 3 files, indexed all: Added: 3, Updated: 0, Removed: 0.
+
+### 4. Collection Show Stats (Bug Fix Verified)
+expected: sif collection show test-e2e displays Documents: 3 (not 0).
+result: pass
+note: Shows Documents: 3, Last Indexed timestamp. The _row_to_collection bug fix works.
+
+### 5. Get Document by Full Path
+expected: sif get get /full/path/python-basics.md retrieves full content.
+result: pass
+note: Displays title, path, collection, and full markdown content with code blocks.
+
+### 6. Get Document by Short Filename (Bug Fix Verified)
+expected: sif get get python-basics.md retrieves document by filename only.
+result: pass
+note: get_by_filename() fallback works. Both short and full path return the same document.
+
+### 7. Multi-Get by Filename
+expected: sif get multi-get retrieves matching documents.
+result: pass
+note: `sif get multi-get "python-basics.md"` found 1 matching document.
+
+### 8. BM25 Keyword Search
+expected: sif search search "embedding vector" finds research-notes.md.
+result: pass
+note: research-notes.md found with score 0.5352.
+
+### 9. Search JSON Output
+expected: sif search search with --json outputs valid JSON.
+result: pass
+note: Valid JSON array with document_id, title, path, score, highlights fields.
+
+### 10. List Documents (ls)
+expected: sif ls test-e2e shows indexed documents.
+result: pass
+note: Shows tree: test-e2e / private / tmp / sif-e2e-test-docs / {3 .md files}.
+
+### 11. Context Add and List
+expected: sif context add collection test-e2e "..." adds context. sif context list shows it.
+result: pass
+note: Context added and listed with type=collection, target, and content preview.
+
+### 12. Status Command
+expected: sif status shows database path, collection count, document count.
+result: pass
+note: Shows Index Path, Collections: 1, Documents: 3, Chunks: 0, Contexts: 1.
+
+### 13. Collection Remove Cleanup
+expected: sif collection remove test-e2e removes collection. list shows empty.
+result: pass
+note: Requires confirmation (y/N). After removal, "No collections found."
+
+## Summary
+
+total: 13
+passed: 13
+issues: 0
+pending: 0
+skipped: 0
+
+## Gaps
+
+[none]

docsif-0.1.0/.planning/E2E-manual-test-plan.md

@@ -0,0 +1,249 @@
+# SIF Manual Test Plan
+
+**Test Directory:** `/Users/forrest/SynologyDrive/Obsidian/ScholarCopilot-Dev/raw/markdown`
+**Scale:** 307 papers, 266MB
+**Use a separate database** to avoid polluting daily index.
+
+## Setup
+
+```bash
+export SIF_DB_PATH=/tmp/sif-scholar-test.sqlite
+```
+
+## Test 1: Create Collection & Index
+
+```bash
+# Create collection
+sif collection add /Users/forrest/SynologyDrive/Obsidian/ScholarCopilot-Dev/raw/markdown -n scholar-test -d "学术论文测试集"
+
+# Index (observe speed and output)
+sif index update -c scholar-test
+```
+
+**Check:** Index completes without errors, shows Added: ~307
+
+---
+
+## Test 2: Collection Stats
+
+```bash
+sif collection show scholar-test
+sif status
+```
+
+**Check:** Documents count is correct (~307), NOT 0
+
+---
+
+## Test 3: Generate Embeddings (Vectorization)
+
+```bash
+# Generate vector embeddings for all indexed documents
+# Default model: Qwen/Qwen3-Embedding-0.6B (1024 dim, via modelscope)
+# This step is slow — expect several minutes for 307 papers
+sif index embed -c scholar-test
+
+# (Optional) Force re-embed with a specific model type
+# sif index embed -c scholar-test --model-type sentence_transformers --model "all-MiniLM-L6-v2"
+
+# (Optional) Force re-embed all (even already embedded)
+# sif index embed -c scholar-test -f
+```
+
+**Check:** Embedding completes without errors. Output shows progress and count of embedded chunks.
+
+**Note:** First run will download the model (~1.2GB for Qwen3-Embedding-0.6B). Subsequent runs use cache. If model download fails, try `--model-type sentence_transformers` as fallback.
+
+---
+
+## Test 4: Verify Vector Index
+
+```bash
+# Check chunks were created
+sif collection show scholar-test
+
+# Status should show Chunks > 0
+sif status
+```
+
+**Check:** Chunks count > 0, indicating vectors were stored in sqlite-vec
+
+---
+
+## Test 5: Vector Search
+
+```bash
+# Pure vector similarity search
+sif search vsearch "how does chromosome folding affect gene regulation"
+
+# Another semantic query (no exact keyword match needed)
+sif search vsearch "methods for detecting drug side effects in children"
+
+# With limit and JSON output
+sif search vsearch "immune system antibody response" -n 5 --json | python -m json.tool | head -30
+```
+
+**Check:** Returns semantically relevant papers even without exact keyword overlap. Scores and results differ from BM25 (Test 7).
+
+---
+
+## Test 6: Hybrid Search (BM25 + Vector + RRF)
+
+```bash
+# Hybrid search combines BM25 and vector via Reciprocal Rank Fusion
+sif search query "gene expression regulation"
+
+# Compare with pure BM25
+sif search search "gene expression regulation"
+
+# Compare with pure vector
+sif search vsearch "gene expression regulation"
+
+# Hybrid with explain to see score breakdown
+sif search query "gene expression regulation" --explain -n 3
+```
+
+**Check:** Hybrid results should rank differently from pure BM25 or pure vector. `--explain` shows BM25 score, vector score, and final RRF fused score.
+
+---
+
+## Test 8: Get by Filename
+
+```bash
+# Short filename (PMID number)
+sif get get 30522467.md
+
+# Another one
+sif get get 11847345.md
+```
+
+**Check:** Can retrieve paper content via short filename
+
+---
+
+## Test 9: BM25 Keyword Search
+
+```bash
+# English keywords
+sif search search "chromosome conformation"
+
+# Medical keywords
+sif search search "poisoning paediatric"
+
+# Cross-domain
+sif search search "antibody immune"
+```
+
+**Check:** Returns relevant papers, scores reasonable, highlights meaningful
+
+---
+
+## Test 10: Search Output Formats
+
+```bash
+# JSON
+sif search search "chromosome" --json | python -m json.tool | head -20
+
+# CSV
+sif search search "chromosome" --csv | head -5
+
+# Markdown
+sif search search "chromosome" --md | head -15
+```
+
+**Check:** JSON valid, CSV has headers, Markdown readable
+
+---
+
+## Test 11: ls File Tree
+
+```bash
+sif ls scholar-test
+```
+
+**Check:** Shows 1/2/3/4 subdirectory structure with paper files
+
+---
+
+## Test 12: Multi-get Batch Retrieval
+
+```bash
+# Glob pattern with byte limit
+sif get multi-get "*.md" -b 500
+```
+
+**Check:** Returns multiple documents (truncated to 500 bytes each)
+
+---
+
+## Test 13: Search with Filters
+
+```bash
+# Limit results
+sif search search "DNA" -n 3
+
+# Minimum score filter
+sif search search "DNA" --min-score 0.3
+
+# Explain mode
+sif search search "DNA" --explain
+```
+
+**Check:** `-n 3` returns exactly 3, `--min-score` filters low scores, `--explain` shows score breakdown
+
+---
+
+## Test 14: Context Enhancement
+
+```bash
+# Add collection-level context
+sif context add collection scholar-test "学术论文集合，包含生物医学研究论文，按 PMID 编号组织"
+
+# Add path-level context
+sif context add path /Users/forrest/SynologyDrive/Obsidian/ScholarCopilot-Dev/raw/markdown/3 "Group 3 论文"
+
+# Verify
+sif context list
+
+# Check context in search results
+sif search search "chromosome" --full
+```
+
+**Check:** Search results include context_description field
+
+---
+
+## Test 15: Full Content Search
+
+```bash
+sif search search "Saccharomyces cerevisiae" --full -n 1
+```
+
+**Check:** Returns full paper content (or large excerpt)
+
+---
+
+## Test 16: Cleanup
+
+```bash
+# Remove test collection
+echo "y" | sif collection remove scholar-test
+
+# Confirm removed
+sif collection list
+
+# Clean up
+unset SIF_DB_PATH
+rm -f /tmp/sif-scholar-test.sqlite
+```
+
+---
+
+## Key Focus Areas
+
+- Indexing speed and memory usage for 307 papers
+- Short filename lookup works with PMID numbers (pure digits)
+- Embedding generation speed (first run downloads ~1.2GB model)
+- Vector search returns semantically relevant results (not just keyword match)
+- Hybrid search (BM25 + Vector + RRF) produces better ranking than either alone
+- Search performance and result truncation for long documents

docsif-0.1.0/.planning/HANDOFF.json

@@ -0,0 +1,44 @@
+{
+  "version": "1.0",
+  "timestamp": "2026-04-27T12:53:33Z",
+  "phase": "E2E",
+  "phase_name": "end-to-end-verification",
+  "phase_dir": ".planning",
+  "plan": 0,
+  "task": 4,
+  "total_tasks": 5,
+  "status": "paused",
+  "completed_tasks": [
+    {"id": 1, "name": "Fix all ruff lint errors and warnings", "status": "done", "commit": "pending"},
+    {"id": 2, "name": "Fix all pytest warnings", "status": "done", "commit": "pending"},
+    {"id": 3, "name": "Fix 3 E2E bugs (get-by-filename, multi-get, collection stats)", "status": "done", "commit": "pending"},
+    {"id": 4, "name": "E2E verification (automated)", "status": "done", "progress": "13/13 tests passed against temp DB"}
+  ],
+  "remaining_tasks": [
+    {"id": 5, "name": "Commit all changes and run manual E2E test plan", "status": "in_progress", "progress": "Manual test plan written at .planning/E2E-manual-test-plan.md, user about to run it against real Obsidian vault"}
+  ],
+  "blockers": [],
+  "human_actions_pending": [
+    {"action": "Run manual E2E test plan against /Users/forrest/SynologyDrive/Obsidian/ScholarCopilot-Dev/raw/markdown", "context": "Automated E2E passed but user wants to verify with real data (307 papers, 266MB)", "blocking": false},
+    {"action": "Re-run sif index embed with MPS (GPU) after fix", "context": "MPS device support added to embedder.py, current embed may be running on CPU", "blocking": false}
+  ],
+  "decisions": [
+    {"decision": "Added MPS (Apple GPU) device detection to SentenceTransformerEmbedder and ModelScopeEmbedder", "rationale": "Code only checked CUDA, Mac users with Apple Silicon get stuck on CPU. Device selection now: CUDA > MPS > CPU", "phase": "E2E"},
+    {"decision": "Added get_by_filename() to DocumentRepository with fallback in get_cmd/multi_get_cmd", "rationale": "Users expect `sif get readme.md` to work, not just full paths", "phase": "E2E"},
+    {"decision": "Fixed _row_to_collection to include document_count and chunk_count from DB rows", "rationale": "Collection stats always showed 0 because fields were silently dropped during row-to-object conversion", "phase": "E2E"},
+    {"decision": "Added mark_indexed() method to Collection dataclass", "rationale": "DocumentIndexer called nonexistent method — latent bug not caught by tests", "phase": "E2E"}
+  ],
+  "uncommitted_files": [
+    "src/sif/database/repositories.py",
+    "src/sif/cli/commands/get.py",
+    "src/sif/core/models.py",
+    "src/sif/embedding/embedder.py",
+    "tests/unit/cli/test_get.py",
+    "~50 other source/test files with ruff fixes",
+    ".planning/E2E-UAT.md",
+    ".planning/E2E-manual-test-plan.md",
+    ".planning/debug/e2e-get-and-stats-bugs.md"
+  ],
+  "next_action": "Commit all changes, then user runs manual E2E test plan against real Obsidian vault",
+  "context_notes": "Session covered: (1) ~300+ ruff lint fixes across all source and test files, (2) 3 bug fixes from E2E testing (get-by-filename, collection stats, mark_indexed), (3) MPS GPU support for Mac, (4) automated E2E verification passed 13/13 tests. User is about to run manual testing against their real Obsidian vault with 307 academic papers. They started sif index embed but it was running on CPU — MPS fix just applied, needs re-run."
+}

docsif-0.1.0/.planning/PROJECT.md

@@ -0,0 +1,80 @@
+# DocSift
+
+## What This Is
+
+DocSift 是 [qmd](https://github.com/tobi/qmd) 的 Python 重构版本，一个面向个人知识库的**本地混合搜索引擎**。它索引 Markdown 笔记、文档和会议纪要，支持关键词搜索（BM25）、语义向量搜索（Vector Search），以及基于 LLM 的重排序（Rerank）和查询扩展。同时提供 CLI 工具和 MCP（Model Context Protocol）服务器两种使用方式，完全在本地运行，无需外部 API。
+
+## Core Value
+
+用户可以在自己的笔记和文档库中，用自然语言快速、准确地找到需要的信息——无论关键词是否匹配。
+
+## Requirements
+
+### Validated
+
+- ✓ 集合（Collection）的增删改查 — 现有
+- ✓ 基于文件扫描的文档索引（含 checksum 去重） — 现有
+- ✓ SQLite FTS5 全文搜索（BM25） — 现有
+- ✓ 基础混合搜索（BM25 + Vector + RRF） — 现有
+- ✓ MCP 服务器骨架（stdio + HTTP 双传输） — 现有
+- ✓ 多格式输出（rich table / JSON / CSV / Markdown / XML） — 现有
+- ✓ 文档片段（chunk）和嵌入存储 — 现有
+- ✓ 修复现有 bug 和占位实现（stub）：硬编码模型路径、随机 embedding fallback、MCP 连接安全等 — Validated in Phase 01
+- ✓ 依赖声明完整性：补全 `pyproject.toml` 中缺失的运行时依赖 — Validated in Phase 01
+
+### Active
+
+- ✓ 补齐缺失的 CLI 命令：multi-get、ls、update-cmd、pull — Validated in Phase 02
+- [ ] 补齐缺失的 CLI 命令：bench
+- [ ] 实现 LLM 重排序（rerank）和查询扩展（query expansion），达到 qmd `query` 命令的搜索质量
+- [ ] 统一 MCP 服务器实现：整合 legacy `mcp/` 和 refactored `mcp_server/`
+- [ ] 嵌入模型灵活可配置：支持本地 GGUF（llama-cpp-python）、sentence-transformers，以及可选的 OpenAI 兼容 API
+
+### Out of Scope
+
+- 多用户/权限系统 — 本地单用户工具，不需要
+- 实时协作/同步 — 超出 qmd 原有范围
+- Web UI — qmd 也没有，仅 CLI + MCP
+- 非 Markdown 格式的复杂解析（如 PDF、Word）—— 先聚焦文本和代码文件
+
+## Context
+
+- 原始项目 qmd 使用 TypeScript / Bun 开发，依赖 `node-llama-cpp` 加载 GGUF 模型进行本地 embedding 和 rerank。
+- 当前 docsift 代码库已完成大部分核心架构（分层设计、Repository 模式、插件化 Chunker / Searcher），但存在多处 stub、bug 和未实现的 CLI 命令。
+- 目标是**功能对等**的 Python 版本，同时改进架构（更好的模块化、可测试性）。
+- 需要保持与 qmd 的 CLI 命令和 MCP 工具接口兼容，方便现有用户迁移。
+
+## Constraints
+
+- **Tech Stack**: Python 3.9+、SQLite（FTS5 + 可选 sqlite-vec）、Click CLI、Pydantic Settings
+- **Local-First**: 默认所有模型本地运行，不依赖外部 API；API 调用仅作为可选配置
+- **Compatibility**: CLI 接口和 MCP Tool Schema 尽量与 qmd 保持一致
+- **Build**: hatchling 构建后端，pytest 测试
+
+## Key Decisions
+
+| Decision | Rationale | Outcome |
+|----------|-----------|---------|
+| 完全复刻 qmd 功能 | 用户明确要求功能对等 | — In Progress |
+| 本地模型 + API 切换双支持 | 提升灵活性和可访问性 | — In Progress |
+| 优先修复 bug/stub，再补 CLI，再做 rerank/MCP | 先让现有代码可用，再扩展功能 | Phase 01 Complete |
+
+## Evolution
+
+This document evolves at phase transitions and milestone boundaries.
+
+**After each phase transition** (via `/gsd-transition`):
+1. Requirements invalidated? → Move to Out of Scope with reason
+2. Requirements validated? → Move to Validated with phase reference
+3. New requirements emerged? → Add to Active
+4. Decisions to log? → Add to Key Decisions
+5. "What This Is" still accurate? → Update if drifted
+
+**After each milestone** (via `/gsd-complete-milestone`):
+1. Full review of all sections
+2. Core Value check — still the right priority?
+3. Audit Out of Scope — reasons still valid?
+4. Update Context with current state
+
+---
+*Last updated: 2026-04-16 after Phase 02 completion*