agent-library 0.7.0__py3-none-any.whl

agent_library-0.7.0.dist-info/METADATA

@@ -0,0 +1,201 @@
+Metadata-Version: 2.4
+Name: agent-library
+Version: 0.7.0
+Summary: Markdown document management system with vector and full-text search
+Author-email: "Arcade.dev" <dev@arcade.dev>
+License-Expression: MIT
+License-File: LICENSE
+Keywords: ai,embeddings,markdown,mcp,search,vector
+Classifier: Development Status :: 4 - Beta
+Classifier: Environment :: Console
+Classifier: Intended Audience :: Developers
+Classifier: License :: OSI Approved :: MIT License
+Classifier: Operating System :: OS Independent
+Classifier: Programming Language :: Python :: 3
+Classifier: Programming Language :: Python :: 3.10
+Classifier: Programming Language :: Python :: 3.11
+Classifier: Programming Language :: Python :: 3.12
+Classifier: Topic :: Software Development :: Libraries :: Python Modules
+Classifier: Topic :: Text Processing :: Indexing
+Requires-Python: >=3.10
+Requires-Dist: arcade-mcp-server>=0.1.0
+Requires-Dist: numpy>=1.24.0
+Requires-Dist: python-frontmatter>=1.0.0
+Requires-Dist: pyyaml>=6.0
+Requires-Dist: rich>=13.0.0
+Requires-Dist: sentence-transformers>=2.2.0
+Requires-Dist: sqlite-vec>=0.1.0
+Requires-Dist: typer[all]>=0.9.0
+Provides-Extra: dev
+Requires-Dist: mypy>=1.5.1; extra == 'dev'
+Requires-Dist: pre-commit>=3.4.0; extra == 'dev'
+Requires-Dist: pytest-asyncio>=0.25.0; extra == 'dev'
+Requires-Dist: pytest-cov>=4.0.0; extra == 'dev'
+Requires-Dist: pytest>=8.3.0; extra == 'dev'
+Requires-Dist: ruff>=0.7.4; extra == 'dev'
+Requires-Dist: types-pyyaml>=6.0.1; extra == 'dev'
+Provides-Extra: evals
+Requires-Dist: arcade-mcp[evals]>=1.0.0; extra == 'evals'
+Description-Content-Type: text/markdown
+
+# Librarian
+
+A personal knowledge library for AI agents, built on [Arcade](https://arcade.dev) for the Model Context Protocol (MCP).
+
+## Overview
+
+Librarian provides AI agents with persistent storage for text, documents, and knowledge. Agents can store information and retrieve it later through semantic and keyword search, maintaining context across conversations.
+
+```mermaid
+graph LR
+    A[Agent Stores Info] --> B[Parser]
+    B --> C[Chunker]
+    C --> D[Embedder]
+    D --> E[(SQLite + vec)]
+    F[Agent Queries] --> G[Hybrid Search]
+    E --> G
+    G --> H[Relevant Context]
+```
+
+## Features
+
+- Persistent knowledge storage for AI agents
+- SQLite storage with `sqlite-vec` for vector search
+- Full-text search using FTS5 with BM25 ranking
+- Hybrid search combining semantic and keyword matching
+- Max Marginal Relevance (MMR) for diverse results
+- Configurable embedding models (local or OpenAI-compatible API)
+- Header-aware text chunking with overlap
+- Time-bounded search filters
+- CLI and MCP server interfaces
+
+## Installation
+
+```bash
+git clone https://github.com/ArcadeAI/librarian.git
+cd librarian
+./setup.sh
+```
+
+Or install manually:
+
+```bash
+uv pip install -e ".[dev]"
+```
+
+## CLI Usage
+
+```bash
+# Add files to the library
+libr add ~/notes
+
+# Search the library
+libr search "machine learning concepts"
+
+# List sources
+libr list
+
+# View library statistics
+libr index
+
+# Rebuild the index
+libr index build
+```
+
+## MCP Server
+
+Start the server for AI assistant integration:
+
+```bash
+# stdio transport (Claude Desktop, CLI)
+libr serve stdio
+
+# HTTP transport (Cursor, VS Code)
+libr serve http --port 8000
+```
+
+See the [Arcade MCP documentation](https://docs.arcade.dev) for integration details.
+
+### Available Tools
+
+| Tool | Description |
+|------|-------------|
+| `Librarian_SearchLibrary` | Search the library with hybrid vector + keyword search |
+| `Librarian_SemanticSearchLibrary` | Find content by meaning (semantic similarity) |
+| `Librarian_KeywordSearchLibrary` | Find content by exact keywords |
+| `Librarian_SearchLibraryByDates` | Search within a date range |
+| `Librarian_AddToLibrary` | Store new content in the library |
+| `Librarian_UpdateLibraryDoc` | Update existing content |
+| `Librarian_ReadFromLibrary` | Read full document content |
+| `Librarian_RemoveFromLibrary` | Remove content from the library |
+| `Librarian_ListLibraryContents` | List all stored content |
+| `Librarian_IndexDirectoryToLibrary` | Bulk import files |
+| `Librarian_GetLibrarySources` | List sources with document/chunk counts |
+| `Librarian_GetLibraryStats` | Overall library statistics |
+
+## Configuration
+
+Set via environment variables:
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `DOCUMENTS_PATH` | `./documents` | Root directory for files |
+| `DATABASE_PATH` | `~/.librarian/index.db` | SQLite database location |
+| `EMBEDDING_PROVIDER` | `openai` | `local` or `openai` |
+| `EMBEDDING_MODEL` | `all-MiniLM-L6-v2` | Local model name |
+| `OPENAI_API_BASE` | `http://localhost:7171/v1` | OpenAI-compatible API URL |
+| `OPENAI_EMBEDDING_MODEL` | `qwen3-embedding-06b` | API model name |
+| `CHUNK_SIZE` | `512` | Max characters per chunk |
+| `CHUNK_OVERLAP` | `50` | Overlap between chunks |
+| `SEARCH_LIMIT` | `10` | Default results limit |
+| `MMR_LAMBDA` | `0.5` | MMR diversity (0=diverse, 1=relevant) |
+| `HYBRID_ALPHA` | `0.7` | Vector vs keyword weight (1=vector only) |
+
+## Project Structure
+
+```
+librarian/
+├── cli.py           # Command-line interface
+├── server.py        # MCP server and tool definitions
+├── config.py        # Configuration management
+├── indexing.py      # Document indexing service
+├── types.py         # Shared type definitions
+├── storage/
+│   ├── database.py  # SQLite operations
+│   ├── vector_store.py  # sqlite-vec search
+│   └── fts_store.py     # FTS5 search
+├── processing/
+│   ├── embed/       # Embedding providers
+│   ├── parsers/     # Document parsers
+│   └── transform/   # Text chunking
+├── retrieval/
+│   └── search.py    # Hybrid search + MMR
+└── utils/
+    └── timeframe.py # Time filter utilities
+```
+
+## Development
+
+```bash
+make install    # Install dependencies
+make test       # Run tests
+make lint       # Run linter
+make format     # Format code
+make typecheck  # Type checking
+make check      # All checks
+make evals      # Run evaluations
+```
+
+## Resources
+
+- [Arcade.dev](https://arcade.dev) - Build AI-native applications
+- [Arcade Documentation](https://docs.arcade.dev) - Integration guides and API reference
+
+## License
+
+MIT License - see [LICENSE](LICENSE) for details.
+
+## Contact
+
+- Email: <contact@arcade.dev>
+- Website: [arcade.dev](https://arcade.dev)

agent_library-0.7.0.dist-info/RECORD

@@ -0,0 +1,36 @@
+librarian/.env.example,sha256=osYXUMrQ9obx7S8gf-XgwJ5YcvIkTr2suxHwOhIhDQ8,318
+librarian/__init__.py,sha256=YNH8ZTOY-3NBfvvv4V5kImTzvLiA4F-gTgqr5sDkejg,1172
+librarian/cli.py,sha256=xXpOoYGnsewqHRj765NKNEIat2l_LJ0oUALN_OlyrvM,40672
+librarian/config.py,sha256=PvCx0XD4sE5nZQs40pqmfshSv1McT9hMJhbTagcEXj8,4219
+librarian/indexing.py,sha256=Dt2t7YhCy4cJ0aIzCpMcXOykbhu_XOq29MmKOSK5028,4691
+librarian/server.py,sha256=fJkzmPox6FGIa3q3jI22w5kHKKpe1o5sAwELnM-Y980,39255
+librarian/types.py,sha256=_kVn7m9AdaL1aAus4aFLddrYsY6qDmCPFxuN4-pkV7Y,3958
+librarian/processing/__init__.py,sha256=LhZ20oWS4RnbOZIxgt-qyKz7riO32mLsiiYovCQo55Y,519
+librarian/processing/manager.py,sha256=WuQkkut-wUExXDG4Dukz9ucW1yfkpfv7tMlsluvmvHo,7854
+librarian/processing/embed/__init__.py,sha256=YGKjFrRBMJ6QwfGZyyuu9RI3hpqljGR6P6g6lk0PqtU,5181
+librarian/processing/embed/base.py,sha256=tB9gEFpSC4gKjV0OBWhSc2MqK_dSPMWvns_XDnjIG8s,2374
+librarian/processing/embed/local.py,sha256=U2grCyLvUFF3AhpznZ1ab4K36Ez2j0ESQCjTWj9q8mg,5850
+librarian/processing/embed/openai.py,sha256=8IAF7OTkGstp7hxN1HCqDOeP3Dy4euDEE03beZNXB40,8169
+librarian/processing/parsers/__init__.py,sha256=lRAcqtFHqt5DUUzERIwqVryQr4HyqKB_ascaEAPGsYo,433
+librarian/processing/parsers/base.py,sha256=w24W6L6Hnhf-5z0KQyiutKQTSaruPG6RPZF6Bk2NXX8,2062
+librarian/processing/parsers/md.py,sha256=njUWb2AGxJC4bgY8KNQajN73lrkzAFKZW1oNGdWr1Yo,7240
+librarian/processing/parsers/obsidian.py,sha256=cyZgN-eWJRLw8RVPYFLvEkXuD2jUytcRp6hXrMU3lDc,5400
+librarian/processing/transform/__init__.py,sha256=iMuiKGNG7y3Fh_AhI5lwk_OzyQUs38HuzfQTuCLWZfo,217
+librarian/processing/transform/chunker.py,sha256=2cRzuBiZmHL2dRLuXH6gtUPjBNlhjSEvcXUGvleri5s,12233
+librarian/providers/__init__.py,sha256=TUkDXQzW-lvzbqtTHcT8L9JAEoyRRX4ySpAA6Pu_ToM,779
+librarian/providers/anthropic.py,sha256=AlPD-wNLRihUowt6vUZIDMOdZMAQpMVMWi16Wc7gS9A,6709
+librarian/providers/base.py,sha256=VgVQPPBtMld5fX4LBSD-PZEwGjOLb2C_XmeRInKZnQg,3232
+librarian/providers/oai.py,sha256=iRbijYFvV8TyrE6LBZYtQ41KeeGUttGm2c5JbP15UHM,5770
+librarian/retrieval/__init__.py,sha256=w9IOWBp11l-6e8EPsZTVCDkY0NybJ7ge6w_SveVhEiI,257
+librarian/retrieval/search.py,sha256=AL4igvl9czDlzgX5GFK57ZnKfleAsWwvgKBElUnC8u0,11268
+librarian/storage/__init__.py,sha256=b0n6Y1xynDOtQPORvKq0IIFHjr5NCYTuhdPiPYJuG_o,590
+librarian/storage/database.py,sha256=N6TTUo0p-gjyQLeiRN2tmCx4c3QE5SYP_zZ8ao7S7sE,21159
+librarian/storage/fts_store.py,sha256=jTHjkbbQeI9w_IQ3nnTuaoiasqKLgAW5OOwjGwxXKnk,6983
+librarian/storage/vector_store.py,sha256=rzL2VhVRf7m2d6nzNiry1UBqbP1ZuMEE3fFGWTvHUEE,6524
+librarian/utils/__init__.py,sha256=IsdXj8h6TRyhd9e1PFY87TrWC6CYJq6bj4piIup1wGs,274
+librarian/utils/timeframe.py,sha256=GSVQWCdLo9YyTj2RNwvVJHikd9rkWbhUS-6EPwOTY78,5224
+agent_library-0.7.0.dist-info/METADATA,sha256=c2HDX-Ys2INKSxziNOnymiiJZju81tNqnknh7MnLpjA,6369
+agent_library-0.7.0.dist-info/WHEEL,sha256=WLgqFyCfm_KASv4WHyYy0P3pM_m7J5L9k2skdKLirC8,87
+agent_library-0.7.0.dist-info/entry_points.txt,sha256=cX5fyMxssr_Bd_F_YLjAex6dC9-HfH93Wf5u3XiCNvY,73
+agent_library-0.7.0.dist-info/licenses/LICENSE,sha256=_ehAoqrx6bKWEjcst2Q4SOkLBZyYDBIShk9xeCMRaeU,1064
+agent_library-0.7.0.dist-info/RECORD,,

agent_library-0.7.0.dist-info/WHEEL

@@ -0,0 +1,4 @@
+Wheel-Version: 1.0
+Generator: hatchling 1.28.0
+Root-Is-Purelib: true
+Tag: py3-none-any

agent_library-0.7.0.dist-info/entry_points.txt

@@ -0,0 +1,3 @@
+[console_scripts]
+libr = librarian.cli:app
+librarian = librarian.cli:app

agent_library-0.7.0.dist-info/licenses/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2024, spartee
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

librarian/.env.example

@@ -0,0 +1,11 @@
+# Arcade Context Environment Configuration
+# Copy this file to .env and update the values
+
+# Path to your Obsidian vault directory
+OBSIDIAN_VAULT_PATH="/path/to/your/obsidian/vault"
+
+# Index configuration (optional)
+INDEX_POLL_INTERVAL=60
+INDEX_START_DELAY=5
+INDEX_STORAGE_PATH="~/.arcade/obsidian"
+INDEX_NAME="index"

librarian/__init__.py

@@ -0,0 +1,52 @@
+"""
+Librarian - Markdown document management system.
+
+A complete system for maintaining, indexing, ingesting, and retrieving
+markdown documents through the Model Context Protocol (MCP).
+
+Features:
+- SQLite-based storage with vector search (sqlite-vec)
+- Full-text search (FTS5) with BM25 ranking
+- Hybrid search combining vector and keyword search
+- Max Marginal Relevance (MMR) for diverse results
+- Configurable embedding providers (local or OpenAI)
+- Intelligent text chunking with overlap
+- Time-bounded search with timeframe filters
+
+Usage:
+    # For MCP server
+    from librarian.server import app
+
+    # For processing
+    from librarian.processing import ProcessingManager
+
+    # For CLI
+    librarian --help
+"""
+
+from librarian.config import (
+    CHUNK_OVERLAP,
+    CHUNK_SIZE,
+    DATABASE_PATH,
+    DOCUMENTS_PATH,
+    EMBEDDING_MODEL,
+    EMBEDDING_PROVIDER,
+    HYBRID_ALPHA,
+    MMR_LAMBDA,
+    SEARCH_LIMIT,
+)
+
+__version__ = "0.6.0"
+
+__all__ = [
+    "CHUNK_OVERLAP",
+    "CHUNK_SIZE",
+    "DATABASE_PATH",
+    "DOCUMENTS_PATH",
+    "EMBEDDING_MODEL",
+    "EMBEDDING_PROVIDER",
+    "HYBRID_ALPHA",
+    "MMR_LAMBDA",
+    "SEARCH_LIMIT",
+    "__version__",
+]