ragnar-cli 0.1.0.pre.3 → 0.1.0.pre.5

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 6b9a7fdbf0345f1c111f8028f8b881d8014a55226cfe3d02f6a76fd6cd9b213c
-  data.tar.gz: 2341e27f16b442c0631876303e0da5141438559ef6685e2c15514cd18416d99c
+  metadata.gz: 06c692710e5d5deb8cf5b8122050968deb459ad6feebd5bbdeabf63679a04d7c
+  data.tar.gz: dc8784c1b36aec7c473b9f93cc1929bdbd1d75ecec18d727c4e7892aeea6df30
 SHA512:
-  metadata.gz: a87f39a5dfd246732be4e24b19aba8b49a7a735f78d825d0241f04b0b776fc8b23c15f0b3488416dedfba37d9027ff6442f38ab2aad43bb79395e0c769247275
-  data.tar.gz: 00d7533c2e16b57da59786a840f1b653cfe472c63168a6e003abb31a79b3f6e0f056fdabe32f4b4566ea682d229e3d0710167358ab7c78fd18c497690d9a3675
+  metadata.gz: cd6e88640e7629725fef32214f088d11362ddbcbbf52a90aa6d315ecc0b08c0d8f487501fd802221c9b647f1e22b55aa91a97f8eefa7046b38c2c49eff3d1bd3
+  data.tar.gz: b58174568e4b76d6cce4aeb19e7674cddb9f8c11c0c6e44a1da0e9f545263a3874b0f99cc16fdc3723e516ffb7c4ffee6660540c891cd595b805ded4f3ff0a49

data/README.md

@@ -2,6 +2,10 @@
 
 A complete Ruby implementation of Retrieval-Augmented Generation (RAG) pipeline using native Ruby ML/NLP gems.
 
+<p align="center">
+  <img src="/docs/assets/screenshot.png" alt="ragnar TUI" width="600">
+</p>
+
 ## Overview
 
 Ragnar provides a production-ready RAG pipeline for Ruby applications, integrating:
@@ -124,14 +128,14 @@ flowchart TB
 ### As a Gem
 
 ```bash
-gem install ragnar
+gem install ragnar-cli
 ```
 
 ### From Source
 
 ```bash
-git clone https://github.com/yourusername/ragnar.git
-cd ragnar
+git clone https://github.com/scientist-labs/ragnar-cli.git
+cd ragnar-cli
 bundle install
 gem build ragnar.gemspec
 gem install ./ragnar-*.gem
@@ -151,21 +155,70 @@ ragnar index ./documents \
   --chunk-overlap 100
 ```
 
-### 2. Train UMAP (Optional)
+### 2. Interactive Mode (TUI)
+
+Running `ragnar` with no arguments launches an interactive TUI powered by [ratatui](https://ratatui.rs/):
+
+```bash
+# Launch the TUI (default when no command given)
+ragnar
+
+# Or explicitly
+ragnar interactive
+```
+
+The TUI provides:
+- **Auto-completion** for commands and options
+- **Persistent history** across sessions
+- **Live output** — see indexing progress, query results, and topic analysis inline
+- **All CLI commands** available via `/command` syntax (e.g., `/index .`, `/umap train`, `/query "my question"`)
+- **`/verbose`** — toggle verbose mode to see query pipeline details (retrieval, reranking, context)
+- **`/profile`** — list or switch LLM profiles mid-session
+
+### 3. Train UMAP (Optional)
 
 Reduce embedding dimensions for faster search:
 
 ```bash
 # Train UMAP model (auto-adjusts parameters based on data)
-ragnar train-umap \
+ragnar umap train \
   --n-components 50 \
   --n-neighbors 15
 
 # Apply to all embeddings
-ragnar apply-umap
+ragnar umap apply
 ```
 
-### 3. Query the System
+### 4. Extract Topics
+
+Perform topic modeling to discover themes in your indexed documents:
+
+```bash
+# Basic topic extraction (requires minimum 20-30 indexed documents)
+ragnar topics
+
+# Adjust clustering parameters for smaller datasets
+ragnar topics --min-cluster-size 3  # Allow smaller topics
+ragnar topics --min-samples 2       # Less strict density requirements
+
+# Export visualizations
+ragnar topics --export html  # Interactive D3.js visualization
+ragnar topics --export json  # JSON data for further processing
+
+# Verbose mode for debugging
+ragnar topics --verbose
+```
+
+**Note**: Topic modeling requires sufficient documents to identify meaningful patterns. For best results:
+- Index at least 20-30 documents (ideally 50+)
+- Ensure documents cover diverse topics
+- Documents should be substantial (50+ words each)
+
+The HTML export includes:
+- **Topic Bubbles**: Interactive bubble chart showing topic sizes and coherence
+- **Embedding Scatter Plot**: Visualization of all documents in embedding space, colored by cluster
+
+### 5. Query the System
 
 ```bash
 # Basic query
@@ -197,7 +250,7 @@ When using `--verbose` or `-v`, you'll see:
 6. **Response Generation**: The final LLM prompt and response
 7. **Final Results**: Confidence score and source attribution
 
-### 4. Check Statistics
+### 6. Check Statistics
 
 ```bash
 ragnar stats
@@ -231,30 +284,142 @@ ragnar stats
 
 ## Configuration
 
-### Default Settings
+Ragnar uses a flexible YAML-based configuration system that allows you to customize all aspects of the RAG pipeline.
 
-```ruby
-DEFAULT_DB_PATH = "ragnar_database"
-DEFAULT_CHUNK_SIZE = 512
-DEFAULT_CHUNK_OVERLAP = 50
-DEFAULT_EMBEDDING_MODEL = "jinaai/jina-embeddings-v2-base-en"
+### Configuration File
+
+Ragnar looks for configuration files in the following order:
+1. `.ragnar.yml` in the current directory
+2. `.ragnarrc.yml` in the current directory  
+3. `ragnar.yml` in the current directory
+4. `.ragnar.yml` in your home directory
+5. Built-in defaults
+
+Generate a configuration file:
+```bash
+# Create local config (in current directory)
+ragnar init-config
+
+# Create global config (in home directory)
+ragnar init-config --global
+
+# Force overwrite existing config
+ragnar init-config --force
 ```
 
+### Configuration Options
+
+Example `.ragnar.yml` file:
+
+```yaml
+# Storage paths (all support ~ expansion)
+storage:
+  database_path: "~/.cache/ragnar/database"
+  models_dir: "~/.cache/ragnar/models"
+  history_file: "~/.cache/ragnar/history"
+
+# Embedding configuration
+embeddings:
+  model: jinaai/jina-embeddings-v2-base-en
+  chunk_size: 512
+  chunk_overlap: 50
+
+# UMAP dimensionality reduction
+umap:
+  reduced_dimensions: 64
+  n_neighbors: 15
+  min_dist: 0.1
+  model_filename: umap_model.bin
+
+# LLM profiles — switch between local and cloud models
+llm:
+  default_profile: red_candle
+  profiles:
+    red_candle:
+      provider: red_candle
+      model: MaziyarPanahi/Qwen3-4B-GGUF
+    opus:
+      provider: anthropic
+      model: claude-opus-4-6
+      api_key: sk-ant-...           # or set ANTHROPIC_API_KEY env var
+    sonnet:
+      provider: anthropic
+      model: claude-sonnet-4-6
+    ollama:
+      provider: ollama
+      model: llama3.1:8b
+
+# Query processing
+query:
+  top_k: 3                          # Number of documents to retrieve
+  enable_query_rewriting: true       # Use LLM to improve queries
+  enable_reranking: true             # Cross-encoder reranking (disable for small corpora)
+  reranker_model: BAAI/bge-reranker-base  # Reranker model
+
+# Interactive mode
+interactive:
+  prompt: 'ragnar> '
+  quiet_mode: true
+
+# Output settings
+output:
+  show_progress: true
+```
+
+### LLM Profiles
+
+Profiles let you switch between LLM providers without editing config. Use local models for development and cloud models for production quality:
+
+```bash
+# Use a specific profile for a single command
+ragnar --profile opus query "What is our security policy?"
+
+# In TUI mode, switch profiles mid-session
+ragnar> /profile           # List available profiles
+ragnar> /profile opus      # Switch to Opus
+ragnar> /profile red_candle  # Switch back to local
+```
+
+Ragnar supports any [RubyLLM](https://rubyllm.com/) provider: `red_candle` (local), `anthropic`, `openai`, `ollama`, and more. API keys can be set per-profile in the config or via environment variables (`ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, etc.).
+
+### Viewing Configuration
+
+```bash
+ragnar config     # Show all settings including active profile
+ragnar model      # Show LLM model information
+ragnar profile    # List all LLM profiles
+```
+
+In interactive mode (launch with `ragnar`):
+```
+ragnar> /config    # Show configuration
+ragnar> /profile   # List profiles
+```
+
+### Environment Variables
+
+Configuration values can be overridden with environment variables:
+- `XDG_CACHE_HOME` - Override default cache directory (~/.cache)
+- `ANTHROPIC_API_KEY` - Anthropic API key (used by anthropic profiles)
+- `OPENAI_API_KEY` - OpenAI API key (used by openai profiles)
+
 ### Supported Models
 
-**Embedding Models** (via red-candle):
-- jinaai/jina-embeddings-v2-base-en
-- BAAI/bge-base-en-v1.5
-- sentence-transformers/all-MiniLM-L6-v2
+**LLM Providers** (via RubyLLM):
+- `red_candle` — Local GGUF models (default): `MaziyarPanahi/Qwen3-4B-GGUF`, `MaziyarPanahi/Qwen3-8B-GGUF`
+- `anthropic` — Claude models: `claude-opus-4-6`, `claude-sonnet-4-6`
+- `openai` — GPT models: `gpt-4o`, `gpt-4o-mini`
+- `ollama` — Local Ollama models: `llama3.1:8b`, `mistral:7b`
+- Any other [RubyLLM provider](https://rubyllm.com/providers/)
 
-**LLM Models** (via red-candle):
-- Qwen/Qwen2.5-1.5B-Instruct
-- microsoft/phi-2
-- TinyLlama/TinyLlama-1.1B-Chat-v1.0
+**Embedding Models** (via red-candle):
+- `jinaai/jina-embeddings-v2-base-en` (default, 768 dimensions)
+- `BAAI/bge-base-en-v1.5`
+- `sentence-transformers/all-MiniLM-L6-v2`
 
-**Reranker Models** (via red-candle):
-- BAAI/bge-reranker-base
-- cross-encoder/ms-marco-MiniLM-L-6-v2
+**Reranker Models** (via red-candle, configurable):
+- `BAAI/bge-reranker-base` (default, XLM-RoBERTa)
+- `cross-encoder/ms-marco-MiniLM-L-12-v2` (smaller, BERT-based)
 
 ## Advanced Usage
 
@@ -284,6 +449,60 @@ puts result[:answer]
 puts "Confidence: #{result[:confidence]}%"
 ```
 
+### Topic Modeling
+
+Extract topics from your indexed documents:
+
+```ruby
+# Example with sufficient documents for clustering (minimum ~20-30 needed)
+documents = [
+  # Finance cluster
+  "Federal Reserve raises interest rates to combat inflation",
+  "Stock markets rally on positive earnings reports",
+  "Cryptocurrency markets show increased volatility",
+  "Corporate bonds yield higher returns this quarter",
+  "Central banks coordinate global monetary policy",
+
+  # Technology cluster
+  "AI breakthrough in natural language processing announced",
+  "Machine learning transforms healthcare diagnostics",
+  "Cloud computing adoption accelerates in enterprises",
+  "Quantum computing reaches new error correction milestone",
+  "Open source frameworks receive major updates",
+
+  # Healthcare cluster
+  "Clinical trials show promise for cancer immunotherapy",
+  "Telemedicine reshapes patient care delivery models",
+  "Gene editing advances treatment for rare diseases",
+  "Mental health awareness campaigns gain momentum",
+  "mRNA vaccine technology platform expands",
+
+  # Add more documents for better clustering...
+  # See TOPIC_MODELING_EXAMPLE.md for complete example
+]
+
+# Extract topics using Topical
+database = Ragnar::Database.new("ragnar_database")
+docs = database.get_all_documents_with_embeddings
+
+embeddings = docs.map { |d| d[:embedding] }
+texts = docs.map { |d| d[:chunk_text] }
+
+topics = Topical.extract(
+  embeddings: embeddings,
+  documents: texts,
+  min_topic_size: 3  # Minimum docs per topic
+)
+
+topics.each do |topic|
+  puts "Topic: #{topic.label}"
+  puts "Terms: #{topic.terms.join(', ')}"
+  puts "Size: #{topic.size} documents\n\n"
+end
+```
+
+For a complete working example with 40+ documents, see [TOPIC_MODELING_EXAMPLE.md](TOPIC_MODELING_EXAMPLE.md).
+
 ### Custom Chunking Strategies
 
 ```ruby
@@ -420,20 +639,9 @@ MIT License - see LICENSE file for details
 
 This project integrates several excellent Ruby gems:
 - [red-candle](https://github.com/assaydepot/red-candle) - Ruby ML/LLM toolkit
-- [lancelot](https://github.com/cpetersen/lancelot) - Lance database bindings
-- [clusterkit](https://github.com/cpetersen/clusterkit) - UMAP and clustering implementation
-- [parsekit](https://github.com/cpetersen/parsekit) - Content extraction
+- [lancelot](https://github.com/scientist-labs/lancelot) - Lance database bindings
+- [clusterkit](https://github.com/scientist-labs/clusterkit) - UMAP and clustering implementation
+- [thor-interactive](https://github.com/scientist-labs/thor-interactive) - Interactive TUI for Thor CLIs
+- [ratatui_ruby](https://github.com/nicholasgasior/ratatui-ruby) - Ratatui terminal UI bindings
+- [parsekit](https://github.com/scientist-labs/parsekit) - Content extraction
 - [baran](https://github.com/moeki0/baran) - Text splitting utilities
-
-## Roadmap
-
-- [ ] Add support for PDF and HTML documents
-- [ ] Implement incremental indexing
-- [ ] Add conversation memory for multi-turn queries
-- [ ] Support for hybrid search (vector + keyword)
-- [ ] Web UI for interactive queries
-- [ ] Docker containerization
-- [ ] Performance benchmarking suite
-- [ ] Support for multiple embedding models simultaneously
-- [ ] Query result caching
-- [ ] Automatic index optimization