desksearch 0.1.0__tar.gz

desksearch-0.1.0/.github/ISSUE_TEMPLATE/bug_report.md

@@ -0,0 +1,42 @@
+---
+name: Bug Report
+about: Report a bug to help us improve DeskSearch
+title: '[Bug] '
+labels: bug
+assignees: ''
+---
+
+## Describe the Bug
+
+A clear and concise description of what the bug is.
+
+## Steps to Reproduce
+
+1. Run `desksearch ...`
+2. Search for '...'
+3. See error
+
+## Expected Behavior
+
+What you expected to happen.
+
+## Actual Behavior
+
+What actually happened.
+
+## Environment
+
+- **OS**: [e.g., macOS 14.2, Ubuntu 22.04, Windows 11]
+- **Python version**: [e.g., 3.12.1]
+- **DeskSearch version**: [e.g., 0.1.0]
+- **Install method**: [pip / source / desktop app]
+
+## Logs / Error Output
+
+```
+Paste any error messages or logs here
+```
+
+## Additional Context
+
+Add any other context, screenshots, or file samples (if relevant) here.

desksearch-0.1.0/.github/ISSUE_TEMPLATE/feature_request.md

@@ -0,0 +1,23 @@
+---
+name: Feature Request
+about: Suggest a new feature or improvement
+title: '[Feature] '
+labels: enhancement
+assignees: ''
+---
+
+## Problem
+
+What problem does this feature solve? What are you trying to do that you can't do today?
+
+## Proposed Solution
+
+Describe the solution you'd like. How would it work from a user's perspective?
+
+## Alternatives Considered
+
+Any alternative solutions or features you've considered.
+
+## Additional Context
+
+Add any other context, mockups, or examples here.

desksearch-0.1.0/.github/workflows/build-release.yml

@@ -0,0 +1,91 @@
+name: Build & Release
+
+on:
+  push:
+    tags:
+      - 'v*'
+
+permissions:
+  contents: write
+
+jobs:
+  build:
+    strategy:
+      matrix:
+        include:
+          - os: macos-latest
+            platform: mac
+            artifact: "dist/electron/DeskSearch-*.dmg"
+          - os: windows-latest
+            platform: win
+            artifact: "dist/electron/DeskSearch Setup *.exe"
+          - os: ubuntu-latest
+            platform: linux
+            artifact: "dist/electron/DeskSearch-*.AppImage"
+
+    runs-on: ${{ matrix.os }}
+
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: '3.12'
+
+      - name: Set up Node.js
+        uses: actions/setup-node@v4
+        with:
+          node-version: '22'
+
+      - name: Install Python dependencies
+        run: |
+          pip install -e ".[dev]"
+          pip install pyinstaller
+
+      - name: Build frontend
+        working-directory: src/ui
+        run: |
+          npm ci
+          npm run build
+
+      - name: Bundle backend with PyInstaller
+        run: |
+          pyinstaller --onedir --name desksearch-backend \
+            --hidden-import desksearch --hidden-import desksearch.api \
+            --hidden-import desksearch.api.server --hidden-import desksearch.api.routes \
+            --hidden-import desksearch.api.schemas --hidden-import desksearch.core \
+            --hidden-import desksearch.core.search --hidden-import desksearch.core.bm25 \
+            --hidden-import desksearch.core.dense --hidden-import desksearch.core.fusion \
+            --hidden-import desksearch.core.snippets --hidden-import desksearch.indexer \
+            --hidden-import desksearch.indexer.pipeline --hidden-import desksearch.indexer.parsers \
+            --hidden-import desksearch.indexer.chunker --hidden-import desksearch.indexer.embedder \
+            --hidden-import desksearch.indexer.store --hidden-import desksearch.indexer.watcher \
+            --hidden-import desksearch.config --hidden-import desksearch.onboarding \
+            --hidden-import desksearch.daemon --hidden-import desksearch.plugins \
+            --hidden-import uvicorn --hidden-import fastapi --hidden-import tantivy \
+            --hidden-import faiss --hidden-import onnxruntime --hidden-import tokenizers \
+            --hidden-import huggingface_hub \
+            --collect-all desksearch --collect-all tantivy --collect-all onnxruntime \
+            --collect-all tokenizers \
+            --add-data "src/ui/dist:ui/dist" \
+            --noconfirm \
+            src/desksearch/__main__.py
+
+      - name: Build Electron app
+        working-directory: electron
+        run: |
+          npm ci
+          npm run dist:${{ matrix.platform }}
+
+      - name: Upload artifact
+        uses: actions/upload-artifact@v4
+        with:
+          name: desksearch-${{ matrix.platform }}
+          path: ${{ matrix.artifact }}
+
+      - name: Upload to Release
+        if: startsWith(github.ref, 'refs/tags/')
+        uses: softprops/action-gh-release@v2
+        with:
+          files: ${{ matrix.artifact }}

desksearch-0.1.0/.gitignore

@@ -0,0 +1,30 @@
+# Python
+.venv/
+__pycache__/
+*.pyc
+*.egg-info/
+dist/
+build/
+.pytest_cache/
+*.so
+
+# Node
+node_modules/
+
+# IDE
+.idea/
+.vscode/
+*.swp
+*.swo
+
+# OS
+.DS_Store
+Thumbs.db
+
+# LocalSearch data
+.localsearch/
+
+# Job search / personal
+job-search/
+arxiv-daily/
+OVERNIGHT_PLAN.md

desksearch-0.1.0/ARCHITECTURE.md

@@ -0,0 +1,197 @@
+# DeskSearch — Private Semantic Search Engine for Your Files
+
+## Vision
+A fast, beautiful, private semantic search engine that runs entirely on your laptop.
+Index everything — documents, emails, notes, code, images — and find anything by asking in natural language.
+"Perplexity for your own files."
+
+## Architecture Overview
+
+```
+┌─────────────────────────────────────────────────┐
+│                  Web UI (React)                  │
+│   Search bar → Results with snippets & sources   │
+│   File preview │ Filters │ Collections           │
+└──────────────────────┬──────────────────────────┘
+                       │ HTTP/WebSocket
+┌──────────────────────▼──────────────────────────┐
+│              FastAPI Backend (Python)             │
+│   /search  /index  /status  /settings            │
+│   Query understanding → Hybrid retrieval →       │
+│   Reranking → Snippet extraction                 │
+└──────┬───────────┬───────────┬──────────────────┘
+       │           │           │
+┌──────▼──┐ ┌─────▼────┐ ┌───▼──────────────────┐
+│ BM25    │ │  Dense   │ │   Metadata Store     │
+│ Index   │ │  Index   │ │   (SQLite)           │
+│(tantivy)│ │ (FAISS/  │ │   file paths, dates, │
+│         │ │  usearch)│ │   types, thumbnails  │
+└─────────┘ └──────────┘ └──────────────────────┘
+       ▲           ▲
+       │           │
+┌──────┴───────────┴──────────────────────────────┐
+│              Indexing Pipeline                    │
+│   File watcher (watchdog) → Parser → Chunker →   │
+│   Embedder (local model) → Index writer          │
+│                                                  │
+│   Parsers: PDF(marker) │ DOCX │ TXT │ MD │ Code │
+│            Images(OCR) │ Email(.eml) │ HTML      │
+└─────────────────────────────────────────────────┘
+```
+
+## Tech Stack
+
+### Core (Python)
+- **FastAPI** — async API server
+- **SQLite + FTS5** — metadata store + full-text fallback
+- **tantivy-py** — fast BM25 index (Rust-based, way faster than Whoosh)
+- **FAISS or usearch** — dense vector index
+- **sentence-transformers** — local embedding model (all-MiniLM-L6-v2 for MVP, upgrade later)
+- **watchdog** — filesystem watcher for live indexing
+
+### Document Parsing
+- **marker** — PDF → markdown (best quality)
+- **python-docx** — Word documents
+- **python-pptx** — PowerPoint
+- **openpyxl** — Excel
+- **beautifulsoup4** — HTML/emails
+- **Pillow + pytesseract** — OCR for images/screenshots
+- **tree-sitter** — code files (syntax-aware chunking)
+
+### UI (React + Vite)
+- **React 18** with TypeScript
+- **Tailwind CSS** — styling
+- **Vite** — build tool
+- Clean, minimal Perplexity-inspired design
+- File preview panel
+- Search filters (date, file type, folder)
+
+### Packaging
+- **pip install desksearch** — Python package
+- **brew install desksearch** — macOS (later)
+- Single command to start: `desksearch serve`
+- Auto-indexes ~/Documents, ~/Desktop, ~/Downloads by default
+
+## Key Design Decisions
+
+1. **100% Local** — No cloud, no API keys needed for MVP. Embedding model runs locally.
+2. **Hybrid Search** — BM25 + dense embeddings + reciprocal rank fusion. This is where Dylan's expertise shines.
+3. **Incremental Indexing** — File watcher detects changes, only re-indexes modified files.
+4. **Chunk with Context** — Each chunk stores parent document reference for full-context answers.
+5. **Fast Startup** — Index persists on disk. Startup = load index + start server. Should be <2 seconds.
+
+## MVP Scope (v0.1)
+
+### Must Have
+- [ ] Index text files: .txt, .md, .pdf, .docx
+- [ ] Hybrid search: BM25 + dense embeddings
+- [ ] Reciprocal rank fusion for combining results
+- [ ] Web UI with search bar and results
+- [ ] File snippets with highlighted matches
+- [ ] Click result → open file in system default app
+- [ ] CLI: `desksearch index <path>` and `desksearch serve`
+- [ ] Incremental indexing (only new/changed files)
+- [ ] Basic filters: file type, date range
+
+### Nice to Have (v0.2)
+- [ ] Image OCR indexing
+- [ ] Code-aware indexing (tree-sitter)
+- [ ] Answer generation (LLM summarizes top results)
+- [ ] Email indexing (.eml, .mbox)
+- [ ] Collections / saved searches
+- [ ] System tray app (background daemon)
+
+### Future (v1.0)
+- [ ] macOS native app
+- [ ] Browser extension (index bookmarks)
+- [ ] Slack/Discord/Gmail integrations (premium)
+- [ ] Team/shared search (premium)
+- [ ] Mobile companion app
+
+## Agent Assignments
+
+### Agent 1: Core Search Engine (src/core/)
+- Hybrid retrieval: BM25 (tantivy) + dense (FAISS)
+- Reciprocal rank fusion
+- Query processing
+- Result ranking and snippet extraction
+- Tests
+
+### Agent 2: Indexing Pipeline (src/indexer/)
+- File discovery and watching
+- Document parsing (PDF, DOCX, TXT, MD, code)
+- Chunking strategies
+- Embedding generation
+- SQLite metadata store
+- Incremental index updates
+- Tests
+
+### Agent 3: API Server (src/api/)
+- FastAPI endpoints: /search, /index, /status, /settings
+- WebSocket for live indexing progress
+- CORS for UI
+- Settings management
+- Tests
+
+### Agent 4: Web UI (src/ui/)
+- React + Vite + Tailwind
+- Search interface (Perplexity-inspired)
+- Results display with snippets, file icons, dates
+- File type/date filters
+- File preview panel
+- Settings page (indexed folders, reindex trigger)
+
+## File Structure
+```
+desksearch/
+├── pyproject.toml
+├── README.md
+├── ARCHITECTURE.md
+├── src/
+│   ├── __init__.py
+│   ├── __main__.py          # CLI entry point
+│   ├── config.py            # Settings/configuration
+│   ├── core/
+│   │   ├── __init__.py
+│   │   ├── search.py        # Hybrid search engine
+│   │   ├── bm25.py          # Tantivy BM25 wrapper
+│   │   ├── dense.py         # FAISS/usearch dense index
+│   │   ├── fusion.py        # Reciprocal rank fusion
+│   │   └── snippets.py      # Snippet extraction & highlighting
+│   ├── indexer/
+│   │   ├── __init__.py
+│   │   ├── pipeline.py      # Main indexing pipeline
+│   │   ├── watcher.py       # Filesystem watcher
+│   │   ├── parsers.py       # Document parsers
+│   │   ├── chunker.py       # Text chunking
+│   │   ├── embedder.py      # Local embedding model
+│   │   └── store.py         # SQLite metadata store
+│   ├── api/
+│   │   ├── __init__.py
+│   │   ├── server.py        # FastAPI app
+│   │   ├── routes.py        # API endpoints
+│   │   └── schemas.py       # Pydantic models
+│   └── ui/                  # React app (built separately)
+│       ├── package.json
+│       ├── vite.config.ts
+│       ├── index.html
+│       ├── src/
+│       │   ├── App.tsx
+│       │   ├── components/
+│       │   │   ├── SearchBar.tsx
+│       │   │   ├── ResultsList.tsx
+│       │   │   ├── ResultCard.tsx
+│       │   │   ├── FilePreview.tsx
+│       │   │   └── Filters.tsx
+│       │   ├── hooks/
+│       │   │   └── useSearch.ts
+│       │   └── styles/
+│       │       └── globals.css
+│       └── tailwind.config.js
+├── tests/
+│   ├── test_search.py
+│   ├── test_indexer.py
+│   └── test_api.py
+└── data/                    # Default index storage
+    └── .gitkeep
+```

desksearch-0.1.0/CONTRIBUTING.md

@@ -0,0 +1,122 @@
+# Contributing to DeskSearch
+
+Thanks for your interest in contributing! This guide will help you get started.
+
+## Getting Started
+
+1. **Fork** the repository on GitHub
+2. **Clone** your fork locally:
+   ```bash
+   git clone https://github.com/YOUR_USERNAME/desksearch.git
+   cd desksearch
+   ```
+3. **Install** development dependencies:
+   ```bash
+   pip install -e ".[dev]"
+   ```
+4. **Create a branch** for your change:
+   ```bash
+   git checkout -b feature/my-feature
+   ```
+
+## Development Setup
+
+### Backend (Python)
+
+```bash
+# Install in development mode
+pip install -e ".[dev]"
+
+# Run tests
+pytest
+
+# Run a specific test
+pytest tests/test_search.py -v
+```
+
+### Frontend (Web UI)
+
+```bash
+cd src/ui
+npm install
+npm run dev    # Start dev server with hot reload
+npm run build  # Production build
+```
+
+## Making Changes
+
+### Code Style
+
+- **Python**: Follow PEP 8. Use type hints where practical.
+- **TypeScript/React**: Follow the existing patterns in `src/ui/`.
+- Keep changes focused — one feature or fix per PR.
+
+### Testing
+
+- Add tests for new functionality
+- Ensure all existing tests pass: `pytest`
+- Test across file formats if touching parsing/indexing
+
+### Commit Messages
+
+Use clear, descriptive commit messages:
+
+```
+feat: add EPUB parser plugin
+fix: handle empty PDF pages in parser
+docs: update CLI reference with daemon commands
+perf: batch embedding for 3x indexing speedup
+refactor: extract snippet highlighting into module
+```
+
+Prefixes: `feat`, `fix`, `docs`, `perf`, `refactor`, `test`, `chore`
+
+## Pull Request Process
+
+1. **Update documentation** if your change affects user-facing behavior
+2. **Add tests** for new functionality
+3. **Run the test suite** and make sure everything passes
+4. **Push** to your fork and open a Pull Request
+5. **Describe your changes** — what and why, not just how
+6. **Link related issues** if applicable
+
+### PR Title Format
+
+```
+feat: add support for PPTX files
+fix: search crash when index is empty
+```
+
+## Reporting Bugs
+
+Open an issue using the [bug report template](.github/ISSUE_TEMPLATE/bug_report.md) and include:
+
+- Steps to reproduce
+- Expected vs actual behavior
+- Python version and OS
+- Error messages or logs
+
+## Requesting Features
+
+Open an issue using the [feature request template](.github/ISSUE_TEMPLATE/feature_request.md) and describe:
+
+- The problem you're trying to solve
+- Your proposed solution
+- Any alternatives you've considered
+
+## Architecture Overview
+
+```
+src/desksearch/
+├── __main__.py          # CLI entry point (Click)
+├── config.py            # Configuration (Pydantic)
+├── api/                 # FastAPI web server
+├── core/                # Search engines (BM25, dense, fusion, snippets)
+├── indexer/             # Parsing, chunking, embedding, pipeline
+├── plugins/             # Plugin system (base classes + loader)
+└── daemon/              # Background service, tray, autostart
+```
+
+## License
+
+By contributing, you agree that your contributions will be licensed under the [MIT License](LICENSE).

desksearch-0.1.0/DESIGN_DECISIONS.md

@@ -0,0 +1,116 @@
+# Design Decisions — Performance & UX
+
+## 1. Search Must Be INSTANT (<50ms)
+
+### Problem
+Users expect Google-speed search. Loading a sentence-transformer model takes 2-5 seconds. Embedding a query takes 50-200ms. This is too slow for keystroke-by-keystroke search.
+
+### Solution: Dual-Mode Search
+- **Phase 1 (instant, <10ms):** BM25 only via tantivy (Rust-based, sub-millisecond). Results appear as you type.
+- **Phase 2 (background, <200ms):** Dense embedding search runs in background, results merge in via RRF. UI smoothly updates.
+- **Startup:** Tantivy index loads in <100ms from memory-mapped files. Dense index loads async in background.
+- **Model pre-loading:** Embedding model loaded once at startup, stays in memory. First search may have 1-2s delay for model load, then instant.
+
+### Alternative Considered
+- Pre-compute query embeddings for common terms → too much storage, stale
+- Use ONNX runtime instead of PyTorch → 3-5x faster inference, should implement in v0.2
+
+## 2. App-Like Experience
+
+### Problem
+`pip install` + `desksearch serve` is developer-friendly but not consumer-friendly. Regular users want to download an app.
+
+### Solution: Progressive Packaging
+- **v0.1 (now):** pip install + CLI. Target: developers and power users.
+- **v0.2:** System tray app using `pystray`. Runs in background, menubar icon, opens web UI in browser.
+- **v0.3:** Electron or Tauri desktop app. Single .dmg/.exe download. Bundles Python runtime + models.
+- **v1.0:** Native macOS app (Swift) / Windows app. Proper file system integration.
+
+### For MVP: System Tray Daemon
+```
+desksearch install-daemon  # Creates LaunchAgent (macOS) or service (Linux/Windows)
+                           # App starts on login, sits in system tray
+                           # Click tray icon → opens search in browser
+                           # Keyboard shortcut: Cmd+Shift+Space → instant search
+```
+
+## 3. Search Quality (This is the differentiator)
+
+### Why Our Search Is Better Than Spotlight/Windows Search
+1. **Hybrid retrieval:** BM25 catches exact terms, dense catches semantic meaning. Neither alone is sufficient.
+2. **Reciprocal Rank Fusion:** Principled way to combine two ranking signals (Dylan's IR expertise).
+3. **Smart chunking:** Paragraph-aware, respects document structure. Not blind character splits.
+4. **Query understanding:** Detect query type (keyword vs question vs phrase) and adjust retrieval strategy.
+5. **Snippet quality:** Show the MOST RELEVANT passage, not just the first occurrence.
+
+### Ranking Pipeline (v0.2+)
+```
+Query → Query Analysis → BM25 Search + Dense Search → RRF Fusion → Reranker (cross-encoder) → Results
+```
+Adding a cross-encoder reranker (e.g., ms-marco-MiniLM) on top-20 fused results would dramatically improve precision. This is standard in production search but NO local search tool does it.
+
+## 4. Indexing Must Be Background & Non-Intrusive
+
+### Requirements
+- First index: scan + parse + embed all files. May take 5-30 min depending on corpus size.
+- Incremental: file watcher detects changes, re-indexes only modified files. <1s per file.
+- CPU usage: cap embedding at 50% CPU. Don't make the laptop fan spin up.
+- Disk usage: index should be <10% of original corpus size.
+
+### Implementation
+- Use ThreadPoolExecutor with max_workers=2 for embedding (limits CPU)
+- Batch embed: collect 32 chunks, embed in one call (much faster than 1-by-1)
+- Progress reporting via WebSocket to UI
+- Pause/resume indexing
+
+## 5. Embedding Model Choice
+
+### For MVP: all-MiniLM-L6-v2
+- 80MB model, loads fast
+- 384-dim embeddings (small index)
+- Good quality for general text
+- Runs on CPU in ~5ms per query
+
+### For v0.2: User-selectable
+- Option to use larger models (e5-large, bge-base) for better quality
+- ONNX runtime for 3-5x speedup
+- Apple Silicon MPS acceleration
+
+### For v1.0: Custom model
+- Train a model specifically optimized for local file search
+- Matryoshka embeddings (Dylan's Starbucks work) for flexible dim/precision tradeoff
+- This is the ultimate differentiator — a model trained FOR this exact use case
+
+## 6. Global Keyboard Shortcut
+
+Critical for adoption. Users should be able to:
+- Press `Cmd+Shift+Space` (configurable) from anywhere
+- Search bar pops up immediately
+- Type query → instant results
+- Press Enter → open file
+- Press Escape → dismiss
+
+Implementation: Requires native integration.
+- macOS: NSEvent global monitor (Swift helper or pyobjc)
+- Linux: X11/Wayland hotkey binding
+- Windows: RegisterHotKey
+
+## 7. File Type Priority Ranking
+
+Not all files are equal. A matching PDF in ~/Documents is more relevant than a .pyc in node_modules.
+
+Default boosts:
+- Documents (.pdf, .docx, .md): 1.5x
+- Notes/text (.txt, .org): 1.3x
+- Code (.py, .js): 1.0x
+- Config (.json, .yaml, .toml): 0.8x
+- Recent files: 1.2x boost for files modified in last 7 days
+- Frequently opened: track open count, boost popular files
+
+## 8. Privacy & Security
+
+- ALL processing local. No network calls except to download embedding model (one-time).
+- Index stored in ~/.desksearch/ with user-only permissions (0700).
+- No telemetry, no analytics, no phone home.
+- Option to encrypt index at rest (v0.3).
+- Excluded paths: .ssh, .gnupg, .env files with secrets auto-excluded.

desksearch-0.1.0/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Shuai Wang
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.