fastref 0.1.0__tar.gz

fastref-0.1.0/PKG-INFO

@@ -0,0 +1,230 @@
+Metadata-Version: 2.4
+Name: fastref
+Version: 0.1.0
+Summary: Fast offline documentation reference desk and skill generator for AI agents
+Keywords: ai-agents,documentation,offline-docs,sqlite-fts5,skills,bm25
+Author: Rafael Darder
+Author-email: Rafael Darder <darder@gmail.com>
+License-Expression: MIT
+Requires-Dist: litert-lm-api>=0.18.0
+Requires-Python: >=3.12
+Description-Content-Type: text/markdown
+
+# fastref 📚⚡
+
+> Fast offline documentation reference desk and skill generator for AI coding agents.
+
+`fastref` gives AI coding agents an instant, local offline reference desk. It indexes documentation from any tool, framework, or site into a high-performance **SQLite FTS5 database with BM25 ranking**, and automatically generates self-contained **Agent Skills** (under `~/.agents/skills/` or `./.agents/skills/`).
+
+Coding agents can query authoritative documentation and code examples locally in sub-milliseconds without hallucinating APIs, wasting web search latency, or blowing context windows.
+
+---
+
+## The Ingestion Pattern
+
+Instead of fragile ad-hoc scrapers, `fastref` unifies documentation ingestion around two core patterns:
+
+1. **Harvest & Assemble (`fastref harvest`)**: For tools and websites that publish an `llms.txt` file (or website root). `fastref` spiders and crawls linked markdown documentation, resolves relative references, checks external domain boundaries, and bundles pages into clean, structured documentation chunks.
+2. **Fetch & Index Structured JSON (`fastref add`)**: For tools and documentation systems that generate machine-readable search indexes (such as Fuse.js, Algolia, or MkDocs-style search JSON endpoints). `fastref` ingests both flat arrays and nested section trees directly.
+
+---
+
+## Features
+
+- **Dual Ingestion Engine**:
+  - **`llms.txt` Spider & Harvester**: Discovers and indexes documentation hierarchies from `llms.txt` files or site roots, featuring domain boundary controls, crawl depth limits, and link preview (`--dry-run`).
+  - **Structured JSON Ingestor**: Automatically normalizes both flat lists and nested hierarchical search indexes into a unified schema.
+- **Hybrid Search (BM25 + Dense Vectors)**:
+  - **Lexical BM25**: Native SQLite FTS5 for exact keywords, function names, and prefix matching.
+  - **On-Device Semantic Vectors**: Google's **LiteRT-LM** with **EmbeddingGemma 2 Text-270M** (quantized to 157 MB, 256d normalized Matryoshka embeddings) for conceptual retrieval without external APIs or background daemons.
+  - **Reciprocal Rank Fusion (RRF)**: Merges lexical and semantic rankings for optimal recall and precision.
+- **Zero-Daemon Architecture**: In-process inference starts in ~180ms, eliminating background sockets, daemons, and orphaned processes.
+- **Prompt Injection & SSRF Hardened**: Defends against indirect prompt injection by stripping invisible unicode/HTML comments, sanitizing skill frontmatter, blocking private/metadata network IPs, and framing retrieved docs in untrusted reference envelopes.
+- **Automatic Agent Skill Generation**: Creates clean `SKILL.md` definitions with security boundary notices and automatically bridges them into active agent harnesses (`~/.agents/skills/`, `~/.gemini/config/skills/`, `~/.claude/skills/`, `~/.config/goose/skills/`).
+- **Harness-Agnostic**: Compatible with Antigravity, Claude Code, Cursor, Codex, Goose, Aider, and any agent that supports shell commands or standard skill directories.
+- **XDG-Compliant Caching**: Keeps database artifacts neatly organized in `$XDG_CACHE_HOME/fastref` (`~/.cache/fastref`).
+
+---
+
+## Installation
+
+```bash
+# Using pip
+pip install fastref
+
+# Or with uv
+uv tool install fastref
+```
+
+---
+
+## Quick Start
+
+### 1. Ingest Documentation & Generate a Skill
+
+#### Pattern A: Harvest from `llms.txt` or Site Root
+
+```bash
+# Harvest from an llms.txt index
+fastref harvest mytool https://example.com/llms.txt
+
+# Or point directly to the website root (auto-detects /llms.txt)
+fastref harvest mytool https://example.com
+
+# Preview discovered links and external domains without fetching
+fastref harvest mytool https://example.com/llms.txt --dry-run
+```
+
+#### Pattern B: Ingest Pre-Built Structured Search JSON
+
+```bash
+# Index structured search index JSON directly from a URL or local file
+fastref add mytool https://example.com/search_index.json
+fastref add mytool ./path/to/search_index.json
+```
+
+By default, both commands:
+1. Compile an SQLite FTS5 database to `~/.cache/fastref/<name>.db`.
+2. Archive the raw manifest/index to `~/.agents/skills/<name>/resources/raw_index.json`.
+3. Write `~/.agents/skills/<name>/SKILL.md` with trigger rules for AI agents.
+4. Auto-bridge the skill into installed agent environments (`~/.gemini/config/skills/`, `~/.claude/skills/`, `~/.config/goose/skills/`).
+
+### 2. Search Documentation
+
+`fastref search` provides three adaptive profiles:
+- **`find`** (default when vectors are enabled): Returns the top focused code & API snippets (default limit: 3) without catalog clutter.
+- **`directory`** (default when vectors are disabled): Returns a catalog of matching topic titles (capped at 20) plus top snippets.
+- **`lucky`**: Prints the top matching document in full, followed by 5 related topic links.
+
+```bash
+# Focused search (find profile: top snippets, hybrid RRF when vectors enabled)
+fastref search mytool "authentication"
+fastref search mytool "middleware" --limit 5
+
+# Directory profile: view matching topic catalog + top snippets
+fastref search mytool "jwt_validation" --directory
+
+# Lucky profile: print full content of top match + related topic links
+fastref search mytool "installation" --lucky
+
+# Show full content of all matches directly
+fastref search mytool "config" --full
+
+# Retrieve full documentation by exact URL or topic ID
+fastref get mytool "/docs/auth#jwt"
+fastref get mytool "/docs/users" "/docs/roles"
+```
+
+### 3. Optional: Enable On-Device Vector & Hybrid Search
+
+Vector features are disabled by default to keep the CLI lightweight and zero-noise. When enabled, `fastref` uses Google's **LiteRT-LM** and downloads the **EmbeddingGemma 2 Text-270M** model (157 MB).
+
+```bash
+# Enable vector features (downloads the model artifact on first run)
+fastref vector enable
+
+# Backfill vector embeddings for an already indexed library
+fastref vector index mytool
+
+# Check vector status and indexed libraries
+fastref vector status
+
+# When vectors are enabled, fastref search automatically defaults to hybrid RRF:
+fastref search mytool "how to handle webhooks asynchronously"
+
+# Or query explicitly via semantic or lexical mode:
+fastref search mytool "reactive state management" --semantic
+fastref search mytool "on_mount_hook" --lexical
+
+# Disable vector features at any time
+fastref vector disable
+```
+
+### 4. Explore Documentation Structure
+
+```bash
+# View table of contents chapters and topic counts
+fastref toc mytool
+
+# Filter chapters matching a keyword
+fastref toc mytool auth
+```
+
+### 5. Manage Indexed Documentation
+
+```bash
+# List all indexed tools, chunk counts, vector status, and database sizes
+fastref list
+
+# Remove an indexed tool (both cache database and skill folder)
+fastref remove mytool
+```
+
+
+---
+
+## How Agents Use `fastref`
+
+When `fastref` generates a skill, it writes a clean `SKILL.md` into your agent's skill directory. When you ask your agent to implement a feature or debug code using an indexed tool:
+
+1. **Discovery**: The agent runs `fastref search <name> "<query>"` to locate exact components, function signatures, or concepts.
+2. **Context-Efficient Retrieval**: The agent inspects targeted snippets or pulls complete topic documentation via `--lucky` or `fastref get <name> <url>`, without loading megabytes of irrelevant HTML into context.
+3. **Accurate Implementation**: The agent codes against authoritative local documentation and tested code examples.
+
+---
+
+## CLI Reference
+
+### `fastref harvest <name> <source>`
+Harvest documentation from an `llms.txt` file or website root:
+- `<name>`: Identifier for the tool or library.
+- `<source>`: URL or local path to `llms.txt` (or website root URL).
+- `--max-docs <N>`: Maximum documents to spider and index (default: 60).
+- `--max-depth <D>`: Maximum spider crawl depth (default: 2).
+- `--dry-run`: Preview discovered links and domain categorization without fetching or indexing.
+- `--allow-urls <prefixes>`: Comma-separated URL or domain prefixes to allow (e.g. `https://raw.githubusercontent.com/...`).
+- `--allow-all-urls`: Allow fetching documentation from any external URL or domain.
+- `--allow-remote`: Allow network fetching when source is a local file.
+- `--allow-private-ips`: Allow fetching from private, loopback, or local network IPs (e.g. localhost during dev).
+- `-p`, `--project`: Install the skill into the local project directory (`./.agents/skills/`) instead of global user skills.
+- `--skills-dir <dir>`: Custom destination directory for the generated skill.
+- `--cache-dir <dir>`: Custom destination directory for the SQLite database.
+- `--description <desc>`: Custom description for the YAML frontmatter in `SKILL.md`.
+- `--no-skill`: Only build the SQLite index without generating an agent skill.
+
+### `fastref add <name> <source>`
+Index documentation from a pre-built machine-readable search index JSON:
+- `<name>`: Identifier for the tool or library.
+- `<source>`: HTTP/HTTPS URL or local path to `search_index.json`.
+- `--allow-private-ips`: Allow fetching from private, loopback, or local network IPs.
+- `-p`, `--project`: Install the skill into the local project directory (`./.agents/skills/`) instead of global user skills.
+- `--skills-dir <dir>`: Custom destination directory for the generated skill.
+- `--cache-dir <dir>`: Custom destination directory for the SQLite database.
+- `--description <desc>`: Custom description for the YAML frontmatter in `SKILL.md`.
+- `--no-skill`: Only build the SQLite index without generating an agent skill.
+
+### `fastref search <name> <query>`
+Search documentation with BM25 relevance ranking:
+- `<name>`: Identifier for the indexed tool.
+- `<query>`: Search query (supports SQLite FTS5 match expressions).
+- `--lucky`: Print the entire top match immediately and list remaining candidate snippets.
+- `--full`: Include full topic content for all matches directly.
+- `--limit <N>`: Maximum results to return (default: 8).
+
+### `fastref get <name> <url1> [url2 ...]`
+Fetch and display the full documentation text for one or more documents by URL or topic ID.
+
+### `fastref toc <name> [filter]`
+Display Table of Contents chapters, topic counts, and sample titles, optionally filtered by keyword.
+
+### `fastref list`
+List all indexed tools, their chunk counts, and database sizes.
+
+### `fastref remove <name>`
+Remove an indexed tool's SQLite database cache and associated skill directories.
+
+---
+
+## License
+
+MIT License.

fastref-0.1.0/README.md

@@ -0,0 +1,218 @@
+# fastref 📚⚡
+
+> Fast offline documentation reference desk and skill generator for AI coding agents.
+
+`fastref` gives AI coding agents an instant, local offline reference desk. It indexes documentation from any tool, framework, or site into a high-performance **SQLite FTS5 database with BM25 ranking**, and automatically generates self-contained **Agent Skills** (under `~/.agents/skills/` or `./.agents/skills/`).
+
+Coding agents can query authoritative documentation and code examples locally in sub-milliseconds without hallucinating APIs, wasting web search latency, or blowing context windows.
+
+---
+
+## The Ingestion Pattern
+
+Instead of fragile ad-hoc scrapers, `fastref` unifies documentation ingestion around two core patterns:
+
+1. **Harvest & Assemble (`fastref harvest`)**: For tools and websites that publish an `llms.txt` file (or website root). `fastref` spiders and crawls linked markdown documentation, resolves relative references, checks external domain boundaries, and bundles pages into clean, structured documentation chunks.
+2. **Fetch & Index Structured JSON (`fastref add`)**: For tools and documentation systems that generate machine-readable search indexes (such as Fuse.js, Algolia, or MkDocs-style search JSON endpoints). `fastref` ingests both flat arrays and nested section trees directly.
+
+---
+
+## Features
+
+- **Dual Ingestion Engine**:
+  - **`llms.txt` Spider & Harvester**: Discovers and indexes documentation hierarchies from `llms.txt` files or site roots, featuring domain boundary controls, crawl depth limits, and link preview (`--dry-run`).
+  - **Structured JSON Ingestor**: Automatically normalizes both flat lists and nested hierarchical search indexes into a unified schema.
+- **Hybrid Search (BM25 + Dense Vectors)**:
+  - **Lexical BM25**: Native SQLite FTS5 for exact keywords, function names, and prefix matching.
+  - **On-Device Semantic Vectors**: Google's **LiteRT-LM** with **EmbeddingGemma 2 Text-270M** (quantized to 157 MB, 256d normalized Matryoshka embeddings) for conceptual retrieval without external APIs or background daemons.
+  - **Reciprocal Rank Fusion (RRF)**: Merges lexical and semantic rankings for optimal recall and precision.
+- **Zero-Daemon Architecture**: In-process inference starts in ~180ms, eliminating background sockets, daemons, and orphaned processes.
+- **Prompt Injection & SSRF Hardened**: Defends against indirect prompt injection by stripping invisible unicode/HTML comments, sanitizing skill frontmatter, blocking private/metadata network IPs, and framing retrieved docs in untrusted reference envelopes.
+- **Automatic Agent Skill Generation**: Creates clean `SKILL.md` definitions with security boundary notices and automatically bridges them into active agent harnesses (`~/.agents/skills/`, `~/.gemini/config/skills/`, `~/.claude/skills/`, `~/.config/goose/skills/`).
+- **Harness-Agnostic**: Compatible with Antigravity, Claude Code, Cursor, Codex, Goose, Aider, and any agent that supports shell commands or standard skill directories.
+- **XDG-Compliant Caching**: Keeps database artifacts neatly organized in `$XDG_CACHE_HOME/fastref` (`~/.cache/fastref`).
+
+---
+
+## Installation
+
+```bash
+# Using pip
+pip install fastref
+
+# Or with uv
+uv tool install fastref
+```
+
+---
+
+## Quick Start
+
+### 1. Ingest Documentation & Generate a Skill
+
+#### Pattern A: Harvest from `llms.txt` or Site Root
+
+```bash
+# Harvest from an llms.txt index
+fastref harvest mytool https://example.com/llms.txt
+
+# Or point directly to the website root (auto-detects /llms.txt)
+fastref harvest mytool https://example.com
+
+# Preview discovered links and external domains without fetching
+fastref harvest mytool https://example.com/llms.txt --dry-run
+```
+
+#### Pattern B: Ingest Pre-Built Structured Search JSON
+
+```bash
+# Index structured search index JSON directly from a URL or local file
+fastref add mytool https://example.com/search_index.json
+fastref add mytool ./path/to/search_index.json
+```
+
+By default, both commands:
+1. Compile an SQLite FTS5 database to `~/.cache/fastref/<name>.db`.
+2. Archive the raw manifest/index to `~/.agents/skills/<name>/resources/raw_index.json`.
+3. Write `~/.agents/skills/<name>/SKILL.md` with trigger rules for AI agents.
+4. Auto-bridge the skill into installed agent environments (`~/.gemini/config/skills/`, `~/.claude/skills/`, `~/.config/goose/skills/`).
+
+### 2. Search Documentation
+
+`fastref search` provides three adaptive profiles:
+- **`find`** (default when vectors are enabled): Returns the top focused code & API snippets (default limit: 3) without catalog clutter.
+- **`directory`** (default when vectors are disabled): Returns a catalog of matching topic titles (capped at 20) plus top snippets.
+- **`lucky`**: Prints the top matching document in full, followed by 5 related topic links.
+
+```bash
+# Focused search (find profile: top snippets, hybrid RRF when vectors enabled)
+fastref search mytool "authentication"
+fastref search mytool "middleware" --limit 5
+
+# Directory profile: view matching topic catalog + top snippets
+fastref search mytool "jwt_validation" --directory
+
+# Lucky profile: print full content of top match + related topic links
+fastref search mytool "installation" --lucky
+
+# Show full content of all matches directly
+fastref search mytool "config" --full
+
+# Retrieve full documentation by exact URL or topic ID
+fastref get mytool "/docs/auth#jwt"
+fastref get mytool "/docs/users" "/docs/roles"
+```
+
+### 3. Optional: Enable On-Device Vector & Hybrid Search
+
+Vector features are disabled by default to keep the CLI lightweight and zero-noise. When enabled, `fastref` uses Google's **LiteRT-LM** and downloads the **EmbeddingGemma 2 Text-270M** model (157 MB).
+
+```bash
+# Enable vector features (downloads the model artifact on first run)
+fastref vector enable
+
+# Backfill vector embeddings for an already indexed library
+fastref vector index mytool
+
+# Check vector status and indexed libraries
+fastref vector status
+
+# When vectors are enabled, fastref search automatically defaults to hybrid RRF:
+fastref search mytool "how to handle webhooks asynchronously"
+
+# Or query explicitly via semantic or lexical mode:
+fastref search mytool "reactive state management" --semantic
+fastref search mytool "on_mount_hook" --lexical
+
+# Disable vector features at any time
+fastref vector disable
+```
+
+### 4. Explore Documentation Structure
+
+```bash
+# View table of contents chapters and topic counts
+fastref toc mytool
+
+# Filter chapters matching a keyword
+fastref toc mytool auth
+```
+
+### 5. Manage Indexed Documentation
+
+```bash
+# List all indexed tools, chunk counts, vector status, and database sizes
+fastref list
+
+# Remove an indexed tool (both cache database and skill folder)
+fastref remove mytool
+```
+
+
+---
+
+## How Agents Use `fastref`
+
+When `fastref` generates a skill, it writes a clean `SKILL.md` into your agent's skill directory. When you ask your agent to implement a feature or debug code using an indexed tool:
+
+1. **Discovery**: The agent runs `fastref search <name> "<query>"` to locate exact components, function signatures, or concepts.
+2. **Context-Efficient Retrieval**: The agent inspects targeted snippets or pulls complete topic documentation via `--lucky` or `fastref get <name> <url>`, without loading megabytes of irrelevant HTML into context.
+3. **Accurate Implementation**: The agent codes against authoritative local documentation and tested code examples.
+
+---
+
+## CLI Reference
+
+### `fastref harvest <name> <source>`
+Harvest documentation from an `llms.txt` file or website root:
+- `<name>`: Identifier for the tool or library.
+- `<source>`: URL or local path to `llms.txt` (or website root URL).
+- `--max-docs <N>`: Maximum documents to spider and index (default: 60).
+- `--max-depth <D>`: Maximum spider crawl depth (default: 2).
+- `--dry-run`: Preview discovered links and domain categorization without fetching or indexing.
+- `--allow-urls <prefixes>`: Comma-separated URL or domain prefixes to allow (e.g. `https://raw.githubusercontent.com/...`).
+- `--allow-all-urls`: Allow fetching documentation from any external URL or domain.
+- `--allow-remote`: Allow network fetching when source is a local file.
+- `--allow-private-ips`: Allow fetching from private, loopback, or local network IPs (e.g. localhost during dev).
+- `-p`, `--project`: Install the skill into the local project directory (`./.agents/skills/`) instead of global user skills.
+- `--skills-dir <dir>`: Custom destination directory for the generated skill.
+- `--cache-dir <dir>`: Custom destination directory for the SQLite database.
+- `--description <desc>`: Custom description for the YAML frontmatter in `SKILL.md`.
+- `--no-skill`: Only build the SQLite index without generating an agent skill.
+
+### `fastref add <name> <source>`
+Index documentation from a pre-built machine-readable search index JSON:
+- `<name>`: Identifier for the tool or library.
+- `<source>`: HTTP/HTTPS URL or local path to `search_index.json`.
+- `--allow-private-ips`: Allow fetching from private, loopback, or local network IPs.
+- `-p`, `--project`: Install the skill into the local project directory (`./.agents/skills/`) instead of global user skills.
+- `--skills-dir <dir>`: Custom destination directory for the generated skill.
+- `--cache-dir <dir>`: Custom destination directory for the SQLite database.
+- `--description <desc>`: Custom description for the YAML frontmatter in `SKILL.md`.
+- `--no-skill`: Only build the SQLite index without generating an agent skill.
+
+### `fastref search <name> <query>`
+Search documentation with BM25 relevance ranking:
+- `<name>`: Identifier for the indexed tool.
+- `<query>`: Search query (supports SQLite FTS5 match expressions).
+- `--lucky`: Print the entire top match immediately and list remaining candidate snippets.
+- `--full`: Include full topic content for all matches directly.
+- `--limit <N>`: Maximum results to return (default: 8).
+
+### `fastref get <name> <url1> [url2 ...]`
+Fetch and display the full documentation text for one or more documents by URL or topic ID.
+
+### `fastref toc <name> [filter]`
+Display Table of Contents chapters, topic counts, and sample titles, optionally filtered by keyword.
+
+### `fastref list`
+List all indexed tools, their chunk counts, and database sizes.
+
+### `fastref remove <name>`
+Remove an indexed tool's SQLite database cache and associated skill directories.
+
+---
+
+## License
+
+MIT License.

fastref-0.1.0/pyproject.toml

@@ -0,0 +1,33 @@
+[project]
+name = "fastref"
+version = "0.1.0"
+description = "Fast offline documentation reference desk and skill generator for AI agents"
+readme = "README.md"
+license = "MIT"
+keywords = [
+    "ai-agents",
+    "documentation",
+    "offline-docs",
+    "sqlite-fts5",
+    "skills",
+    "bm25",
+]
+requires-python = ">=3.12"
+dependencies = ["litert-lm-api>=0.18.0"]
+
+[[project.authors]]
+name = "Rafael Darder"
+email = "darder@gmail.com"
+
+[project.scripts]
+fastref = "fastref:main"
+
+[build-system]
+requires = ["uv_build>=0.12.5,<0.13.0"]
+build-backend = "uv_build"
+
+[dependency-groups]
+dev = [
+    "pytest>=9.1.1",
+    "ruff>=0.16.9",
+]

fastref-0.1.0/pyproject.toml.orig

@@ -0,0 +1,28 @@
+[project]
+name = "fastref"
+version = "0.1.0"
+description = "Fast offline documentation reference desk and skill generator for AI agents"
+readme = "README.md"
+license = "MIT"
+keywords = ["ai-agents", "documentation", "offline-docs", "sqlite-fts5", "skills", "bm25"]
+
+authors = [
+    { name = "Rafael Darder", email = "darder@gmail.com" }
+]
+requires-python = ">=3.12"
+dependencies = [
+    "litert-lm-api>=0.18.0",
+]
+
+[project.scripts]
+fastref = "fastref:main"
+
+[build-system]
+requires = ["uv_build>=0.12.5,<0.13.0"]
+build-backend = "uv_build"
+
+[dependency-groups]
+dev = [
+    "pytest>=9.1.1",
+    "ruff>=0.16.9",
+]

fastref-0.1.0/src/fastref/__init__.py

@@ -0,0 +1,6 @@
+"""fastref: Fast offline documentation reference desk and skill generator for AI agents."""
+
+from fastref.cli import main
+
+__version__ = "0.1.0"
+__all__ = ["__version__", "main"]