lybrary 0.1.0__tar.gz

lybrary-0.1.0/.gitignore

@@ -0,0 +1,47 @@
+# Python
+__pycache__/
+*.py[cod]
+*.pyo
+*.pyd
+*.egg
+*.egg-info/
+dist/
+build/
+.eggs/
+wheels/
+
+# Virtual environments
+.venv/
+venv/
+env/
+ENV/
+
+# Testing & coverage
+.pytest_cache/
+.coverage
+.coverage.*
+htmlcov/
+*.xml
+
+# Type checking
+.mypy_cache/
+.pyright/
+
+# lybrary runtime data — never commit a repo's index
+.lybrary/
+
+# IDE
+.idea/
+.vscode/
+*.swp
+*.swo
+*.DS_Store
+
+# Distribution / packaging
+MANIFEST
+pip-wheel-metadata/
+share/python-wheels/
+*.dist-info/
+
+# Logs
+*.log

lybrary-0.1.0/CHANGELOG.md

@@ -0,0 +1,50 @@
+# Changelog
+
+All notable changes to lybrary are documented here.
+Format follows [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
+
+---
+
+## [0.1.0] — 2026-08-07
+
+### Added
+- AST-aware chunking (cAST-style) via tree-sitter for Python, JavaScript,
+  TypeScript, TSX, Go, Rust, Java, C, and C++.
+- Incremental indexing via xxhash content-hash tracking (`file_hashes.json`).
+- LanceDB-backed vector store with MiniLM-L6-v2 embeddings
+  (sentence-transformers, fully local).
+- Background daemon with watchdog file watcher and configurable debounce.
+- CLI commands: `init`, `start`, `stop`, `status`, `index`, `query`, `logs`,
+  `mcp`, `version`.
+- MCP server (`lybrary mcp`) exposing three tools over stdio transport:
+  - `memory_query` — semantic search over the indexed code memory.
+  - `memory_status` — daemon/index health check.
+  - `memory_update` — incremental or full re-index, optionally scoped to
+    specific files.
+- `py.typed` marker for PEP 561 compliance.
+
+### Fixed
+- TSX files now use the tsx tree-sitter grammar instead of being silently
+  downgraded to the TypeScript grammar.
+- Nested definitions (methods inside classes) are no longer emitted twice.
+- Overlapping byte ranges are properly merged before computing leftover
+  module-level content.
+- Chunk IDs include the byte offset, preventing collisions for same-named
+  symbols at the same line.
+- `_sanitize_name` strips characters that would corrupt chunk IDs or
+  LanceDB filter strings.
+- `delete_path` escapes double-quotes in paths to prevent filter injection.
+- `search_vector` no longer returns the raw 384-float vector column to callers.
+- Full re-index uses `truncate()` (drop + recreate) instead of iterating
+  `all_paths()`, eliminating stale row leakage.
+- File-move/rename events are handled correctly: the new path is indexed and
+  the old path is deleted from the store.
+- `_update_file` reads the file exactly once, eliminating the race condition
+  between chunk generation and hash storage.
+- Windows daemon now spawns a detached subprocess and writes a PID file so
+  that `lybrary stop` and `lybrary status` work on Windows.
+- Score formula uses proper cosine similarity (`1 - d²/2`) for unit-normalised
+  vectors instead of the previous `1/(1+d)` approximation.
+- Unused `ThreadPoolExecutor` import removed from `walker.py`.
+- Dead `merge_siblings` config option removed from `ChunkConfig`.
+- Embedder normalises the device string before using it as an `lru_cache` key.

lybrary-0.1.0/PKG-INFO

@@ -0,0 +1,161 @@
+Metadata-Version: 2.4
+Name: lybrary
+Version: 0.1.0
+Summary: Living structure-aware code memory for AI coding agents
+Author: lybrary contributors
+License-Expression: MIT
+Keywords: agent,ast,code,indexing,mcp,memory,rag
+Classifier: Development Status :: 3 - Alpha
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Scientific/Engineering :: Artificial Intelligence
+Classifier: Topic :: Software Development :: Libraries
+Classifier: Typing :: Typed
+Requires-Python: >=3.11
+Requires-Dist: fastembed<2.0.0,>=0.4.0
+Requires-Dist: mcp<2.0.0,>=1.0.0
+Requires-Dist: numpy<3.0.0,>=1.26.0
+Requires-Dist: pathspec<1.0.0,>=0.12.0
+Requires-Dist: pydantic-settings<3.0.0,>=2.2.0
+Requires-Dist: pydantic<3.0.0,>=2.6.0
+Requires-Dist: rich<14.0.0,>=13.7.0
+Requires-Dist: tomli-w<2.0.0,>=1.0.0
+Requires-Dist: tree-sitter-c<1.0.0,>=0.21.0
+Requires-Dist: tree-sitter-cpp<1.0.0,>=0.22.0
+Requires-Dist: tree-sitter-go<1.0.0,>=0.21.0
+Requires-Dist: tree-sitter-java<1.0.0,>=0.21.0
+Requires-Dist: tree-sitter-javascript<1.0.0,>=0.21.0
+Requires-Dist: tree-sitter-python<1.0.0,>=0.21.0
+Requires-Dist: tree-sitter-rust<1.0.0,>=0.21.0
+Requires-Dist: tree-sitter-typescript<1.0.0,>=0.21.0
+Requires-Dist: tree-sitter<0.26.0,>=0.24.0
+Requires-Dist: typer[all]<1.0.0,>=0.12.0
+Requires-Dist: watchdog<6.0.0,>=4.0.0
+Requires-Dist: xxhash<4.0.0,>=3.4.0
+Provides-Extra: dev
+Requires-Dist: pytest-asyncio<1.0.0,>=0.23.0; extra == 'dev'
+Requires-Dist: pytest<9.0.0,>=8.0.0; extra == 'dev'
+Requires-Dist: ruff<1.0.0,>=0.4.0; extra == 'dev'
+Description-Content-Type: text/markdown
+
+# lybrary
+
+**Living structure-aware code memory for AI coding agents.**
+
+`lybrary` indexes your repository with real AST boundaries (functions, classes, methods), keeps the index fresh via a background daemon, and gives AI agents a high-signal memory they can query instead of reading raw files.
+
+This reduces token burn and hallucinations compared to pure agentic search or flat embedding RAG.
+
+## Why lybrary?
+
+Current IDE agents either:
+
+- Re-explore the repo with grep/read every session (expensive, amnesiac), or
+- Use flat semantic search that doesn't understand code structure.
+
+`lybrary` sits in the middle:
+
+- **AST-aware chunks** (cAST-style) — never splits a function in half
+- **Persistent daemon** that auto-updates on every file change
+- **Hybrid retrieval** (vector search + token-budget packing)
+- **Agent-first design** — agents query memory instead of walking the tree
+- **MCP server** — works natively with Kiro, Cursor, Claude Desktop, Windsurf, and any MCP-compatible IDE
+- **Fully local** — no cloud, no API keys, embeddings run on your machine
+
+## Quick start
+
+```bash
+pip install lybrary
+
+cd /path/to/your/repo
+lybrary init
+lybrary start          # builds index + starts background daemon
+lybrary query "authentication flow"
+```
+
+After `lybrary start`, the daemon keeps running even if you close the terminal. File changes are picked up automatically and only the affected chunks are re-indexed.
+
+## MCP integration (AI IDEs)
+
+Add this to your MCP config (works with Kiro, Cursor, Claude Desktop, Windsurf):
+
+```json
+{
+  "mcpServers": {
+    "lybrary": {
+      "command": "lybrary",
+      "args": ["mcp"]
+    }
+  }
+}
+```
+
+The agent then has three tools available:
+
+| Tool | What it does |
+|---|---|
+| `memory_query` | Semantic search — returns ranked code chunks with full source, file path, and line numbers |
+| `memory_status` | Reports daemon state, chunk count, and tracked files |
+| `memory_update` | Triggers incremental or full re-index, optionally scoped to specific files |
+
+**Agents should call `memory_query` before reading any files.** This replaces multi-file reads with a single targeted query, cutting token usage by 80–90% on large codebases.
+
+## CLI
+
+| Command            | Description                                      |
+|--------------------|--------------------------------------------------|
+| `lybrary init`     | Create `.lybrary/` and default config            |
+| `lybrary start`    | Index (if needed) + start persistent daemon      |
+| `lybrary stop`     | Stop the daemon                                  |
+| `lybrary status`   | Show running state, chunk count, tracked files   |
+| `lybrary index`    | Force (re)index                                  |
+| `lybrary query`    | Semantic search over the memory                  |
+| `lybrary logs`     | View / follow daemon log                         |
+| `lybrary mcp`      | Start MCP server (stdio transport)               |
+
+## How chunking works
+
+1. Detect language from file extension
+2. Parse with **tree-sitter**
+3. Extract definition nodes (functions, classes, methods, interfaces, impls…)
+4. Container nodes (classes, impls) emit as a chunk **and** recurse so nested methods get their own chunks
+5. Attach rich metadata + a context header used for embedding
+6. Fall back to careful line-based splitting only when a grammar is missing
+
+Supported languages: Python, JavaScript, TypeScript, TSX, Go, Rust, Java, C, C++.
+
+## Architecture (v0.1)
+
+```
+.lybrary/
+├── config.toml
+├── index.db          # SQLite: chunks + vectors (numpy float32 blobs)
+├── file_hashes.json  # content-hash map for incremental updates
+├── daemon.pid
+└── daemon.log
+```
+
+- **Indexer**: tree-sitter → AST chunks → local embeddings (fastembed / all-MiniLM-L6-v2, ONNX Runtime)
+- **Store**: SQLite + numpy (cosine similarity via batched dot product)
+- **Daemon**: watchdog file watcher + debounce + incremental re-chunk/embed
+- **Query**: vector search + token-budget packing
+- **MCP**: FastMCP server over stdio
+
+## Roadmap
+
+- [x] AST chunker (multi-language, cAST-style)
+- [x] Incremental indexing via content hashes
+- [x] Background daemon + file watcher (Windows + Unix)
+- [x] CLI (`init/start/stop/status/index/query/logs/mcp`)
+- [x] MCP server (`memory_query`, `memory_status`, `memory_update`)
+- [ ] Call/import graph + expansion
+- [ ] Hierarchical file/package summaries
+- [ ] Cross-session decision memory
+- [ ] systemd/launchd user service helper
+
+## License
+
+MIT

lybrary-0.1.0/README.md

@@ -0,0 +1,118 @@
+# lybrary
+
+**Living structure-aware code memory for AI coding agents.**
+
+`lybrary` indexes your repository with real AST boundaries (functions, classes, methods), keeps the index fresh via a background daemon, and gives AI agents a high-signal memory they can query instead of reading raw files.
+
+This reduces token burn and hallucinations compared to pure agentic search or flat embedding RAG.
+
+## Why lybrary?
+
+Current IDE agents either:
+
+- Re-explore the repo with grep/read every session (expensive, amnesiac), or
+- Use flat semantic search that doesn't understand code structure.
+
+`lybrary` sits in the middle:
+
+- **AST-aware chunks** (cAST-style) — never splits a function in half
+- **Persistent daemon** that auto-updates on every file change
+- **Hybrid retrieval** (vector search + token-budget packing)
+- **Agent-first design** — agents query memory instead of walking the tree
+- **MCP server** — works natively with Kiro, Cursor, Claude Desktop, Windsurf, and any MCP-compatible IDE
+- **Fully local** — no cloud, no API keys, embeddings run on your machine
+
+## Quick start
+
+```bash
+pip install lybrary
+
+cd /path/to/your/repo
+lybrary init
+lybrary start          # builds index + starts background daemon
+lybrary query "authentication flow"
+```
+
+After `lybrary start`, the daemon keeps running even if you close the terminal. File changes are picked up automatically and only the affected chunks are re-indexed.
+
+## MCP integration (AI IDEs)
+
+Add this to your MCP config (works with Kiro, Cursor, Claude Desktop, Windsurf):
+
+```json
+{
+  "mcpServers": {
+    "lybrary": {
+      "command": "lybrary",
+      "args": ["mcp"]
+    }
+  }
+}
+```
+
+The agent then has three tools available:
+
+| Tool | What it does |
+|---|---|
+| `memory_query` | Semantic search — returns ranked code chunks with full source, file path, and line numbers |
+| `memory_status` | Reports daemon state, chunk count, and tracked files |
+| `memory_update` | Triggers incremental or full re-index, optionally scoped to specific files |
+
+**Agents should call `memory_query` before reading any files.** This replaces multi-file reads with a single targeted query, cutting token usage by 80–90% on large codebases.
+
+## CLI
+
+| Command            | Description                                      |
+|--------------------|--------------------------------------------------|
+| `lybrary init`     | Create `.lybrary/` and default config            |
+| `lybrary start`    | Index (if needed) + start persistent daemon      |
+| `lybrary stop`     | Stop the daemon                                  |
+| `lybrary status`   | Show running state, chunk count, tracked files   |
+| `lybrary index`    | Force (re)index                                  |
+| `lybrary query`    | Semantic search over the memory                  |
+| `lybrary logs`     | View / follow daemon log                         |
+| `lybrary mcp`      | Start MCP server (stdio transport)               |
+
+## How chunking works
+
+1. Detect language from file extension
+2. Parse with **tree-sitter**
+3. Extract definition nodes (functions, classes, methods, interfaces, impls…)
+4. Container nodes (classes, impls) emit as a chunk **and** recurse so nested methods get their own chunks
+5. Attach rich metadata + a context header used for embedding
+6. Fall back to careful line-based splitting only when a grammar is missing
+
+Supported languages: Python, JavaScript, TypeScript, TSX, Go, Rust, Java, C, C++.
+
+## Architecture (v0.1)
+
+```
+.lybrary/
+├── config.toml
+├── index.db          # SQLite: chunks + vectors (numpy float32 blobs)
+├── file_hashes.json  # content-hash map for incremental updates
+├── daemon.pid
+└── daemon.log
+```
+
+- **Indexer**: tree-sitter → AST chunks → local embeddings (fastembed / all-MiniLM-L6-v2, ONNX Runtime)
+- **Store**: SQLite + numpy (cosine similarity via batched dot product)
+- **Daemon**: watchdog file watcher + debounce + incremental re-chunk/embed
+- **Query**: vector search + token-budget packing
+- **MCP**: FastMCP server over stdio
+
+## Roadmap
+
+- [x] AST chunker (multi-language, cAST-style)
+- [x] Incremental indexing via content hashes
+- [x] Background daemon + file watcher (Windows + Unix)
+- [x] CLI (`init/start/stop/status/index/query/logs/mcp`)
+- [x] MCP server (`memory_query`, `memory_status`, `memory_update`)
+- [ ] Call/import graph + expansion
+- [ ] Hierarchical file/package summaries
+- [ ] Cross-session decision memory
+- [ ] systemd/launchd user service helper
+
+## License
+
+MIT

lybrary-0.1.0/lybrary/__init__.py

@@ -0,0 +1,3 @@
+"""lybrary — Living structure-aware code memory for AI coding agents."""
+
+__version__ = "0.1.0"

lybrary-0.1.0/lybrary/_daemon_worker.py

@@ -0,0 +1,39 @@
+"""Entry point for the Windows background daemon subprocess.
+
+Invoked as:  python -m lybrary._daemon_worker <root>
+
+Writes its own PID to the pid file (overwriting the one set by the parent
+just before spawn, in case they differ), then starts the file watcher.
+"""
+
+from __future__ import annotations
+
+import os
+import sys
+from pathlib import Path
+
+
+def main() -> None:
+    if len(sys.argv) < 2:
+        print("Usage: python -m lybrary._daemon_worker <repo_root>", file=sys.stderr)
+        sys.exit(1)
+
+    root = Path(sys.argv[1]).resolve()
+
+    from lybrary.config import LybraryConfig, get_lybrary_dir
+
+    pid_file = get_lybrary_dir(root) / "daemon.pid"
+    # Overwrite with our actual PID (the parent wrote its own PID first).
+    pid_file.write_text(str(os.getpid()))
+
+    config = LybraryConfig.load(root)
+
+    print(f"lybrary daemon watching {root}", flush=True)
+
+    from lybrary.watcher import start_watching
+
+    start_watching(root, config)
+
+
+if __name__ == "__main__":
+    main()

lybrary-0.1.0/lybrary/cli.py

@@ -0,0 +1,247 @@
+"""lybrary CLI — living code memory for AI agents."""
+
+from __future__ import annotations
+
+from pathlib import Path
+from typing import Optional
+
+import typer
+from rich.console import Console
+from rich.panel import Panel
+from rich.table import Table
+
+from lybrary import __version__
+from lybrary.config import LybraryConfig, find_repo_root, get_lybrary_dir
+
+app = typer.Typer(
+    name="lybrary",
+    help="Living structure-aware code memory for AI coding agents.",
+    no_args_is_help=True,
+    rich_markup_mode="rich",
+)
+console = Console()
+
+
+def _root(path: Optional[Path] = None) -> Path:
+    return find_repo_root(path)
+
+
+@app.command()
+def init(
+    path: Optional[Path] = typer.Argument(None, help="Repository path (default: cwd)"),
+) -> None:
+    """Initialize lybrary in a repository."""
+    root = _root(path)
+    ly_dir = get_lybrary_dir(root)
+    ly_dir.mkdir(parents=True, exist_ok=True)
+
+    config = LybraryConfig()
+    config.save(root)
+
+    # Create a basic ignore file if missing
+    ignore = ly_dir / "ignore"
+    if not ignore.exists():
+        ignore.write_text(
+            "\n".join(
+                [
+                    "# Extra ignore patterns (gitignore syntax)",
+                    "*.min.js",
+                    "*.map",
+                    "dist/",
+                    "build/",
+                    "",
+                ]
+            )
+        )
+
+    console.print(
+        Panel.fit(
+            f"[bold green]lybrary initialized[/bold green]\n\n"
+            f"Root: [cyan]{root}[/cyan]\n"
+            f"Data: [cyan]{ly_dir}[/cyan]\n\n"
+            f"Next: [bold]lybrary start[/bold]",
+            title="lybrary",
+        )
+    )
+
+
+@app.command()
+def start(
+    path: Optional[Path] = typer.Argument(None, help="Repository path"),
+    foreground: bool = typer.Option(False, "--foreground", "-f", help="Run in foreground"),
+) -> None:
+    """Start the lybrary daemon (indexes if needed, then watches for changes)."""
+    root = _root(path)
+    ly_dir = get_lybrary_dir(root)
+
+    if not ly_dir.exists():
+        console.print("[yellow]No .lybrary/ found. Running init first...[/yellow]")
+        init(path)
+
+    from lybrary.daemon import start_daemon
+
+    start_daemon(root, foreground=foreground)
+
+
+@app.command()
+def stop(
+    path: Optional[Path] = typer.Argument(None, help="Repository path"),
+) -> None:
+    """Stop the running lybrary daemon."""
+    root = _root(path)
+    from lybrary.daemon import stop_daemon
+
+    stop_daemon(root)
+
+
+@app.command()
+def status(
+    path: Optional[Path] = typer.Argument(None, help="Repository path"),
+) -> None:
+    """Show daemon and index status."""
+    root = _root(path)
+    from lybrary.daemon import daemon_status
+
+    info = daemon_status(root)
+
+    table = Table(title="lybrary status", show_header=False)
+    table.add_column("Key", style="bold")
+    table.add_column("Value")
+
+    for k, v in info.items():
+        table.add_row(k, str(v))
+
+    console.print(table)
+
+
+@app.command()
+def index(
+    path: Optional[Path] = typer.Argument(None, help="Repository path"),
+    full: bool = typer.Option(False, "--full", help="Force full re-index"),
+) -> None:
+    """Index (or re-index) the repository."""
+    root = _root(path)
+    ly_dir = get_lybrary_dir(root)
+    if not ly_dir.exists():
+        init(path)
+
+    from lybrary.indexer.store import IndexStore
+    from lybrary.indexer.walker import index_repository
+
+    config = LybraryConfig.load(root)
+    store = IndexStore(root)
+
+    console.print(f"[bold]Indexing[/bold] [cyan]{root}[/cyan] ...")
+    stats = index_repository(root, store, config, full=full)
+
+    console.print(
+        Panel.fit(
+            f"[green]Index complete[/green]\n\n"
+            f"Files: {stats.get('files', 0)}\n"
+            f"Chunks: {stats.get('chunks', 0)}\n"
+            f"Added/updated: {stats.get('updated', 0)}",
+            title="lybrary index",
+        )
+    )
+
+
+@app.command()
+def query(
+    q: str = typer.Argument(..., help="Search query"),
+    path: Optional[Path] = typer.Option(None, "--path", "-p", help="Repository path"),
+    limit: int = typer.Option(8, "--limit", "-n", help="Max results"),
+    max_tokens: int = typer.Option(6000, "--max-tokens", help="Token budget for context"),
+) -> None:
+    """Search the code memory."""
+    root = _root(path)
+    from lybrary.query import search
+
+    results = search(root, q, limit=limit, max_tokens=max_tokens)
+
+    if not results:
+        console.print("[yellow]No results.[/yellow]")
+        return
+
+    for i, r in enumerate(results, 1):
+        console.print(
+            Panel(
+                f"[dim]{r.get('path')} :: {r.get('symbol', '')}[/dim]\n\n"
+                f"{r.get('content', '')[:800]}{'...' if len(r.get('content', '')) > 800 else ''}",
+                title=f"[{i}] {r.get('score', 0):.3f}",
+                border_style="blue",
+            )
+        )
+
+
+@app.command()
+def logs(
+    path: Optional[Path] = typer.Argument(None, help="Repository path"),
+    follow: bool = typer.Option(False, "--follow", "-f", help="Follow log output"),
+) -> None:
+    """Show daemon logs."""
+    root = _root(path)
+    log_file = get_lybrary_dir(root) / "daemon.log"
+    if not log_file.exists():
+        console.print("[yellow]No log file yet.[/yellow]")
+        raise typer.Exit(1)
+
+    if follow:
+        import time
+
+        with open(log_file) as f:
+            f.seek(0, 2)
+            while True:
+                line = f.readline()
+                if line:
+                    console.print(line.rstrip())
+                else:
+                    time.sleep(0.3)
+    else:
+        console.print(log_file.read_text()[-8000:])
+
+
+@app.command()
+def mcp(
+    repo_root: Optional[Path] = typer.Option(
+        None, "--root", "-r", help="Repository root (default: auto-detect)"
+    ),
+) -> None:
+    """Start the MCP server (stdio transport) for AI IDE integration.
+
+    Configure your IDE / MCP host to run:  lybrary mcp
+
+    Kiro / Cursor / Claude Desktop example config:
+      {
+        "mcpServers": {
+          "lybrary": {
+            "command": "lybrary",
+            "args": ["mcp"]
+          }
+        }
+      }
+    """
+    try:
+        from lybrary.mcp.server import mcp as _mcp_server  # noqa: F401
+    except ImportError:
+        console.print(
+            "[red]MCP dependencies not installed.[/red]\n"
+            "Run: [bold]pip install 'lybrary[mcp]'[/bold]"
+        )
+        raise typer.Exit(1)
+
+    if repo_root:
+        import os
+        os.chdir(repo_root)
+
+    from lybrary.mcp.server import run
+    run()
+
+
+@app.command()
+def version() -> None:
+    """Show version."""
+    console.print(f"lybrary {__version__}")
+
+
+if __name__ == "__main__":
+    app()