leann 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: a59608a07bbd0265d0d049085ddfa8fb5e560464a1ab943baca3b9562e821a6c
+  data.tar.gz: de9180940fdab66734ebbab598e0a0a3d75e201b649c38530e5a3268017c5ddd
+SHA512:
+  metadata.gz: 3fd7a484b92c4246fedc25f07692c3174043019706aa8b99e5e840f3d0daa43afe75542d71dd65c924cc3763669bc67adb9b8ea4ee67ac56b84c1b75cc61b11d
+  data.tar.gz: 9da1019a629a361a74986e8088f2c19d2e5f9b34cb6345d15e57a30184022c64cd299f6c93e9276b92b8c952378868cd633bdddb97065fb2ea865bca28557424

data/LICENSE.txt

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Chris Hasiński
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,375 @@
+# LEANN
+
+**Lightweight vector search with 85-96% storage savings.**
+
+LEANN stores only the graph structure, not the embeddings. During search, it recomputes embeddings on-the-fly for visited nodes only. This achieves massive storage reduction while maintaining search quality.
+
+```ruby
+# Build an index - embeddings are computed but NOT stored
+Leann.build("knowledge_base") do
+  add "LEANN saves 85-96% storage compared to traditional vector databases."
+  add "It uses graph-only storage with on-demand embedding recomputation."
+  add_directory "docs/", extensions: [".md", ".txt"]
+end
+
+# Search semantically
+results = Leann.search("knowledge_base", "storage efficiency")
+puts results.first.text
+```
+
+## Storage Benchmark
+
+At **1 million documents**:
+
+| Embedding Model | Traditional HNSW | LEANN | Savings |
+|-----------------|------------------|-------|---------:|
+| Ollama (384 dim) | 1.67 GB | 251.8 MB | **85%** |
+| OpenAI (1536 dim) | 5.96 GB | 251.8 MB | **96%** |
+
+Full benchmark across scales:
+
+| Documents | Traditional HNSW | LEANN | Savings |
+|----------:|------------------|-------|---------:|
+| 1,000 | 1.7 MB | 257 KB | **85%** |
+| 10,000 | 17 MB | 2.5 MB | **85%** |
+| 100,000 | 171 MB | 25 MB | **85%** |
+| 1,000,000 | 1.67 GB | 252 MB | **85%** |
+
+*Measured with Ollama all-minilm (384 dimensions). OpenAI models achieve ~96% savings due to larger embeddings.*
+
+## How It Works
+
+**Traditional vector databases:**
+```
+[Documents] + [Embeddings] + [Index] → Large storage (1.67 GB for 1M docs)
+```
+
+**LEANN:**
+```
+[Documents] + [Graph-only] → Tiny storage (252 MB for 1M docs)
+                ↓
+        Embeddings recomputed on-the-fly during search
+```
+
+The trade-off: search is slightly slower (requires API calls to recompute embeddings for visited nodes), but storage is dramatically smaller.
+
+## Quick Start
+
+### 1. Install
+
+```ruby
+gem 'leann'
+gem 'ruby_llm'  # Recommended - LEANN uses RubyLLM for embeddings automatically
+```
+
+### 2. Configure
+
+```ruby
+# If you have RubyLLM, just configure it - LEANN uses it automatically
+RubyLLM.configure do |config|
+  config.openai_api_key = ENV["OPENAI_API_KEY"]
+end
+
+# Or configure LEANN directly (without RubyLLM)
+Leann.configure do |config|
+  config.embedding_provider = :openai  # or :ollama
+  config.openai_api_key = ENV["OPENAI_API_KEY"]
+end
+```
+
+### 3. Build & Search
+
+```ruby
+# Index some documents
+Leann.build("my_index") do
+  add "Ruby is a dynamic programming language."
+  add "Rails is a web framework written in Ruby."
+  add "Sinatra is a lightweight Ruby web framework."
+end
+
+# Search
+results = Leann.search("my_index", "web frameworks")
+results.each { |r| puts "#{r.score.round(3)}: #{r.text}" }
+```
+
+## Real-World Use Cases
+
+### Documentation Search
+
+```ruby
+# Index your documentation
+Leann.build("docs") do
+  add_directory "docs/", extensions: [".md"]
+  add_directory "guides/", extensions: [".md"]
+end
+
+# Search API
+get "/search" do
+  results = Leann.search("docs", params[:q], limit: 10)
+  json results.map(&:to_h)
+end
+```
+
+### Code Search
+
+```ruby
+# Index your codebase
+Leann.build("codebase", embedding: :ollama, model: "nomic-embed-text") do
+  add_directory "app/", extensions: [".rb"]
+  add_directory "lib/", extensions: [".rb"]
+end
+
+# Find relevant code
+results = Leann.search("codebase", "user authentication", limit: 5)
+```
+
+### Local-First with Ollama
+
+```ruby
+# No API keys needed - runs entirely local
+Leann.configure do |config|
+  config.embedding_provider = :ollama
+end
+
+Leann.build("local_index") do
+  add "Your private data stays on your machine."
+end
+
+results = Leann.search("local_index", "privacy")
+```
+
+## RubyLLM Integration
+
+LEANN works seamlessly with [RubyLLM](https://github.com/crmne/ruby_llm):
+
+- **Embeddings**: If RubyLLM is present, LEANN uses it automatically for embeddings
+- **RAG**: Use LEANN as a tool for retrieval-augmented generation
+
+### Embeddings via RubyLLM
+
+```ruby
+# Just configure RubyLLM - LEANN detects and uses it automatically
+RubyLLM.configure do |config|
+  config.openai_api_key = ENV["OPENAI_API_KEY"]
+end
+
+# Build uses RubyLLM.embed under the hood
+Leann.build("docs") do
+  add "Your documents here..."
+end
+```
+
+### RAG with Search Tool
+
+```ruby
+require "leann"
+require "leann/ruby_llm/search"
+
+# Build your index
+Leann.build("knowledge_base") do
+  add "LEANN saves 85-96% storage compared to traditional vector databases."
+  add "It uses graph-only storage for massive space savings."
+end
+
+# Create a RubyLLM chat with LEANN as a tool
+chat = RubyLLM.chat(model: "gpt-4o")
+  .with_tool(Leann::RubyLLM::Search.new("knowledge_base"))
+
+# The LLM will automatically search when needed
+answer = chat.ask("How much storage does LEANN save?")
+puts answer
+# => "Based on my search, LEANN saves 85-96% storage compared to traditional vector databases..."
+```
+
+### Multiple Indexes
+
+```ruby
+require "leann/ruby_llm/search"
+
+docs_tool = Leann::RubyLLM::Search.new("docs", name: "search_docs")
+code_tool = Leann::RubyLLM::Search.new("codebase", name: "search_code")
+
+chat = RubyLLM.chat(model: "gpt-4o")
+  .with_tools(docs_tool, code_tool)
+
+chat.ask("How does authentication work in this project?")
+# => LLM searches both indexes and synthesizes an answer
+```
+
+## API Reference
+
+### Building Indexes
+
+```ruby
+# DSL style (recommended)
+Leann.build("name") do
+  add "text", key: "value"           # Add text with metadata
+  add_file "path/to/file.txt"        # Add file contents
+  add_directory "docs/"              # Add all files from directory
+end
+
+# Programmatic style
+builder = Leann::Builder.new("name", embedding: :openai)
+builder.add("text one")
+builder.add("text two", category: "example")
+builder.save
+```
+
+### Searching
+
+```ruby
+# Simple search
+results = Leann.search("index", "query")
+
+# With options
+results = Leann.search("index", "query",
+  limit: 10,                    # Max results
+  threshold: 0.7,               # Min similarity
+  filters: { category: "docs" } # Metadata filter
+)
+
+# Working with results
+results.each do |r|
+  puts r.id           # Document ID
+  puts r.text         # Document text
+  puts r.score        # Similarity score (0-1)
+  puts r.metadata     # Metadata hash
+end
+
+puts results.combined_text  # All texts joined
+```
+
+### Configuration
+
+```ruby
+Leann.configure do |config|
+  # Embedding provider (defaults to :ruby_llm if available, otherwise :openai)
+  config.embedding_provider = :ruby_llm  # :ruby_llm, :openai, :ollama, or :fastembed
+
+  # Provider-specific settings (only needed if not using RubyLLM)
+  config.openai_api_key = "sk-..."
+  config.ollama_host = "http://localhost:11434"
+  config.default_embedding_model = "text-embedding-3-small"
+
+  # Index settings
+  config.hnsw_m = 16                         # Graph connectivity
+  config.hnsw_ef_construction = 200          # Build quality
+end
+```
+
+## Rails Integration
+
+LEANN provides ActiveRecord integration for storing indexes in your database instead of files.
+
+### Setup
+
+```bash
+# Generate the migration
+rails generate leann:install
+rails db:migrate
+```
+
+This creates two tables:
+- `leann_indexes` - Stores index metadata and graph configuration
+- `leann_passages` - Stores documents with their text, metadata, and graph neighbors
+
+### Usage
+
+```ruby
+require "leann/rails"
+
+# Build an index (stored in database)
+Leann::Rails.build("products") do
+  add "Red running shoes for athletes", category: "shoes", price: 89.99
+  add "Blue denim jeans, slim fit", category: "pants", price: 59.99
+  add "White cotton t-shirt", category: "tops", price: 24.99
+end
+
+# Search
+results = Leann::Rails.search("products", "comfortable footwear")
+results.each { |r| puts "#{r.score.round(3)}: #{r.text}" }
+
+# Other operations
+Leann::Rails.exists?("products")  # => true
+Leann::Rails.list                  # => ["products"]
+Leann::Rails.delete("products")    # => true
+```
+
+### In Controllers
+
+```ruby
+class SearchController < ApplicationController
+  def index
+    @results = Leann::Rails.search("products", params[:q], limit: 10)
+  end
+end
+```
+
+### With ActiveRecord Models
+
+```ruby
+# Direct access to index records
+index = Leann::Rails::Index.find_by(name: "products")
+index.document_count  # => 3
+index.search("shoes") # Search directly on the index
+
+# Access passages
+index.passages.each do |passage|
+  puts passage.text
+  puts passage.metadata
+end
+```
+
+### Benefits of Database Storage
+
+- **Transactions**: Index updates are ACID-compliant
+- **Backups**: Indexes are included in database backups
+- **Scaling**: Use read replicas for search-heavy workloads
+- **Deployment**: No need to manage separate index files
+
+## Embedding Providers
+
+| Provider | Setup | Best For |
+|----------|-------|----------|
+| **RubyLLM** (default) | `gem 'ruby_llm'` | Unified API, multiple backends |
+| **OpenAI** | `OPENAI_API_KEY` | Direct API access |
+| **Ollama** | [Install Ollama](https://ollama.com) | Local, privacy-first |
+| **FastEmbed** | `gem 'fastembed'` | Fast local, no server needed |
+
+When RubyLLM is present, LEANN uses it automatically. This gives you access to all embedding providers RubyLLM supports (OpenAI, Ollama, and more) through a single configuration.
+
+### FastEmbed (Local, Serverless)
+
+FastEmbed provides fast local embeddings using ONNX Runtime - no API keys or running servers required:
+
+```ruby
+gem 'fastembed'
+```
+
+```ruby
+Leann.configure do |config|
+  config.embedding_provider = :fastembed
+end
+
+# Or specify model explicitly
+Leann.build("index", embedding: :fastembed, model: "BAAI/bge-small-en-v1.5") do
+  add "Your documents here..."
+end
+```
+
+**Supported models:**
+- `BAAI/bge-small-en-v1.5` (384 dim, default) - Fast English
+- `BAAI/bge-base-en-v1.5` (768 dim) - Higher accuracy English
+- `intfloat/multilingual-e5-small` (384 dim) - 100+ languages
+- `nomic-ai/nomic-embed-text-v1.5` (768 dim) - Long context (8192 tokens)
+
+## Requirements
+
+- Ruby 3.0+
+- RubyLLM gem (recommended) OR direct API access:
+  - OpenAI API key (for cloud embeddings)
+  - Ollama running locally (for local embeddings)
+
+## License
+
+MIT

data/exe/leann

@@ -0,0 +1,167 @@
+#!/usr/bin/env ruby
+# frozen_string_literal: true
+
+require "leann"
+
+# Simple CLI for LEANN
+class CLI
+  COMMANDS = {
+    "build" => "Build an index from files",
+    "search" => "Search an index",
+    "ask" => "Ask a question (RAG)",
+    "info" => "Show index information",
+    "list" => "List all indexes",
+    "delete" => "Delete an index",
+    "help" => "Show this help"
+  }.freeze
+
+  def run(args)
+    command = args.shift || "help"
+
+    case command
+    when "build"
+      build(args)
+    when "search"
+      search(args)
+    when "ask"
+      ask(args)
+    when "info"
+      info(args)
+    when "list"
+      list_indexes
+    when "delete"
+      delete(args)
+    when "help", "--help", "-h"
+      help
+    when "--version", "-v"
+      puts "leann #{Leann::VERSION}"
+    else
+      puts "Unknown command: #{command}"
+      puts "Run 'leann help' for usage."
+      exit 1
+    end
+  end
+
+  private
+
+  def build(args)
+    if args.empty?
+      puts "Usage: leann build <name> <directory>"
+      puts "Example: leann build docs ./docs"
+      exit 1
+    end
+
+    name = args.shift
+    directory = args.shift || "."
+    extensions = %w[.md .txt .rb .py .js .ts]
+
+    puts "Building index '#{name}' from #{directory}..."
+
+    Leann.build(name) do
+      add_directory directory, extensions: extensions
+    end
+
+    puts "Done!"
+  end
+
+  def search(args)
+    if args.size < 2
+      puts "Usage: leann search <index> <query>"
+      puts "Example: leann search docs 'authentication'"
+      exit 1
+    end
+
+    name = args.shift
+    query = args.join(" ")
+
+    results = Leann.search(name, query, limit: 5)
+
+    if results.empty?
+      puts "No results found."
+    else
+      puts results.to_s
+    end
+  end
+
+  def ask(args)
+    if args.size < 2
+      puts "Usage: leann ask <index> <question>"
+      puts "Example: leann ask docs 'How do I authenticate?'"
+      exit 1
+    end
+
+    name = args.shift
+    question = args.join(" ")
+
+    puts "Searching for context..."
+    answer = Leann.ask(name, question)
+    puts
+    puts answer
+  end
+
+  def info(args)
+    if args.empty?
+      puts "Usage: leann info <index>"
+      exit 1
+    end
+
+    name = args.first
+    index = Leann.open(name)
+    puts index.to_s
+  end
+
+  def list_indexes
+    indexes = Leann.list
+    if indexes.empty?
+      puts "No indexes found."
+    else
+      puts "Indexes:"
+      indexes.each { |i| puts "  - #{i}" }
+    end
+  end
+
+  def delete(args)
+    if args.empty?
+      puts "Usage: leann delete <index>"
+      exit 1
+    end
+
+    name = args.first
+    print "Delete index '#{name}'? [y/N] "
+    confirm = $stdin.gets.strip.downcase
+
+    if confirm == "y"
+      Leann.delete(name)
+      puts "Deleted."
+    else
+      puts "Cancelled."
+    end
+  end
+
+  def help
+    puts <<~HELP
+      LEANN - Lightweight vector search and RAG for Ruby
+
+      Usage: leann <command> [options]
+
+      Commands:
+    HELP
+
+    COMMANDS.each do |cmd, desc|
+      puts "  #{cmd.ljust(10)} #{desc}"
+    end
+
+    puts <<~EXAMPLES
+
+      Examples:
+        leann build docs ./documentation
+        leann search docs "authentication"
+        leann ask docs "How do I set up OAuth?"
+        leann info docs
+        leann list
+        leann delete docs
+    EXAMPLES
+  end
+end
+
+CLI.new.run(ARGV)

data/lib/generators/leann/install/install_generator.rb

@@ -0,0 +1,51 @@
+# frozen_string_literal: true
+
+require "rails/generators"
+require "rails/generators/active_record"
+
+module Leann
+  module Generators
+    class InstallGenerator < ::Rails::Generators::Base
+      include ::Rails::Generators::Migration
+
+      source_root File.expand_path("templates", __dir__)
+
+      desc "Creates LEANN database tables for storing vector indexes"
+
+      def self.next_migration_number(dirname)
+        ::ActiveRecord::Generators::Base.next_migration_number(dirname)
+      end
+
+      def create_migration_file
+        migration_template(
+          "migration.rb.erb",
+          "db/migrate/create_leann_tables.rb"
+        )
+      end
+
+      def show_instructions
+        say ""
+        say "LEANN tables will be created!", :green
+        say ""
+        say "Next steps:"
+        say "  1. Run migrations: rails db:migrate"
+        say "  2. Configure LEANN in an initializer:"
+        say ""
+        say "     # config/initializers/leann.rb"
+        say "     Leann.configure do |config|"
+        say "       config.embedding_provider = :openai"
+        say "       config.openai_api_key = ENV['OPENAI_API_KEY']"
+        say "     end"
+        say ""
+        say "  3. Build and search indexes:"
+        say ""
+        say "     Leann::Rails.build('products') do"
+        say "       add 'Red running shoes', category: 'shoes'"
+        say "     end"
+        say ""
+        say "     results = Leann::Rails.search('products', 'comfortable footwear')"
+        say ""
+      end
+    end
+  end
+end

data/lib/generators/leann/install/templates/migration.rb.erb

@@ -0,0 +1,28 @@
+# frozen_string_literal: true
+
+class CreateLeannTables < ActiveRecord::Migration<%= "[#{ActiveRecord::Migration.current_version}]" rescue "[7.0]" %>
+  def change
+    create_table :leann_indexes do |t|
+      t.string :name, null: false, index: { unique: true }
+      t.string :embedding_provider, null: false
+      t.string :embedding_model
+      t.integer :dimensions, null: false
+      t.text :config  # JSON: stores hnsw_m, hnsw_ef_construction, entry_point_id, max_level
+
+      t.timestamps
+    end
+
+    create_table :leann_passages do |t|
+      t.references :leann_index, null: false, foreign_key: { to_table: :leann_indexes, on_delete: :cascade }
+      t.string :external_id, null: false
+      t.text :text, null: false
+      t.text :metadata  # JSON: stores user-defined metadata
+      t.text :neighbors # JSON: stores neighbor IDs for graph traversal
+
+      t.timestamps
+    end
+
+    add_index :leann_passages, [:leann_index_id, :external_id], unique: true
+    add_index :leann_passages, :external_id
+  end
+end

data/lib/leann/backend/base.rb

@@ -0,0 +1,51 @@
+# frozen_string_literal: true
+
+module Leann
+  module Backend
+    # Base class for vector storage backends
+    #
+    # Subclasses must implement:
+    # - #build(embeddings, ids, path) - build index from embeddings
+    # - #search(query_embedding, limit:) - search for nearest neighbors
+    # - .load(path) - load existing index
+    #
+    class Base
+      # @return [Integer] Embedding dimensions
+      attr_reader :dimensions
+
+      # @param dimensions [Integer]
+      def initialize(dimensions:)
+        @dimensions = dimensions
+      end
+
+      # Build an index from embeddings
+      #
+      # @param embeddings [Array<Array<Float>>]
+      # @param ids [Array<String>]
+      # @param path [String] Index path
+      # @raise [NotImplementedError]
+      def build(embeddings, ids, path)
+        raise NotImplementedError, "Subclasses must implement #build"
+      end
+
+      # Search for nearest neighbors
+      #
+      # @param query_embedding [Array<Float>]
+      # @param limit [Integer]
+      # @return [Array<Array(String, Float)>] Array of [id, score] pairs
+      # @raise [NotImplementedError]
+      def search(query_embedding, limit:)
+        raise NotImplementedError, "Subclasses must implement #search"
+      end
+
+      # Load an existing index
+      #
+      # @param path [String]
+      # @return [Base]
+      # @raise [NotImplementedError]
+      def self.load(path)
+        raise NotImplementedError, "Subclasses must implement .load"
+      end
+    end
+  end
+end