rag_embeddings 0.2.0 → 0.2.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 3bebe9dc8527ed47e0d7b48534911e97ee7549bb73763c1dd59db063e41558c8
-  data.tar.gz: fdb272ee4dd12f52f33fb2cd33b9ff22f5207911c68c09b976af59346f666039
+  metadata.gz: '0065892eb9dc58605d7a30a62de23dfc4a7609b7590cbcac70fc378109846024'
+  data.tar.gz: 143f9878b807ff6ad6d9db3011ece7cdb3c733af7c4cdb81373166d6e6c70f2f
 SHA512:
-  metadata.gz: a106f044b23d4438110516ee57dc6079b177861c8e09da4eeffc7e842c3aa9d506b96b6141d6e033a74de1ca61ecf1e53490682f906e14915d76fd5fe4d81103
-  data.tar.gz: 6be68129ca5338a99f3d55816cbb06785cff2530e39784d5202c4602bb73aec783e5180b7cf09a1f1ef962bb5d6a2070af3db080f677369b49a5463d7c466be3
+  metadata.gz: 46acb57d8f5467bafb2b48b594ab308a09097ed323a7958284615714616e01eecd0717256212df3372be2a0b34ae46315315d27cc323fbc2b2362dc56a71f47c
+  data.tar.gz: ca063e0963c0b95f95f5b611efdd239d13e8db6a566fb8882cb33eb08372c38af832000555420e82a11acce2865a23debcb4b5d60bb1847acff3f8c433584963

data/README.md

@@ -22,42 +22,50 @@
 
 ---
 
-## 🔧 Installation
+## 🌍 Real-world Use Cases
 
-Add to your Gemfile:
+- **Question Answering over Documents:** Instantly search and retrieve the most relevant document snippets from thousands of articles, FAQs, or customer support logs in your Ruby app.
+- **Semantic Search for E-commerce:** Power product search with semantic understanding, returning items similar in meaning, not just keywords.
+- **Personalized Recommendations:** Find related content (articles, products, videos) by comparing user preferences and content embeddings.
+- **Knowledge Base Augmentation:** Use with OpenAI or Ollama to enhance chatbots, letting them ground answers in your company’s internal documentation or wiki.
+- **Fast Prototyping for AI Products:** Effortlessly build MVPs for RAG-enabled chatbots, semantic search tools, and AI-driven discovery apps—all in native Ruby.
 
-```ruby
-gem "rag_embeddings"
-gem "langchainrb"
-gem "faraday"
-gem "sqlite3"
-```
+---
 
-bundle install
-rake compile
+## 👷 Requirements
 
-(Requires a working C compiler!)
+- Ruby >= 3.3
+- `langchainrb` (for embedding)
+- At the moment `ollama` is used as LLM so it must be active and working, although there are some workarounds
+- `sqlite3` (for storage)
 
-## 🏁 Running the test suite
+## 🔧 Installation
 
-To run all specs (RSpec required):
+Requires a working C compiler in order to build the native extension
+
+`gem install rag_embeddings`
+
+If you'd rather install it using bundler, add a line for it in your Gemfile (but set the require option to false, as it is a standalone tool):
+
+```ruby
+gem "rag_embeddings", require: false
+```
 
-`bundle exec rspec`
 
 ## 🧪 Practical examples
 
 ### 1. Generate an embedding from text
 
 ```ruby
-text = "Hello world, this is RAG!"
-embedding = RagEmbeddings.embed(text)
+require "rag_embeddings"
+embedding = RagEmbeddings.embed("Hello world, this is RAG!")
 # embedding is a float array
 ```
 
 The default model is llama3.2 but you can set another one (reload the console as the llm is memoized):
 
 ```ruby
-embedding = RagEmbeddings.embed(text, model: 'qwen3:0.6b')
+embedding = RagEmbeddings.embed("Hello world, this is RAG!", model: 'qwen3:0.6b')
 ````
 
 ### 2. Create a C embedding object
@@ -94,46 +102,197 @@ result = db.top_k_similar("Hello!", k: 1)
 puts "Most similar text: #{result.first[1]}, score: #{result.first[2]}"
 ```
 
+### 5. Batch-index a folder of documents
+
+```ruby
+# load all .txt files
+files = Dir["./docs/*.txt"].map { |f| [File.basename(f), File.read(f)] }
+
+db = RagEmbeddings::Database.new("knowledge_base.db")
+files.each do |name, text|
+  vector = RagEmbeddings.embed(text)
+  db.insert(name, vector)
+end
+
+puts "Indexed #{files.size} documents."
+```
+
+### 6. Simple Retrieval-Augmented Generation (RAG) loop
+
+```ruby
+require "openai"        # or your favorite LLM client
+
+# 1) build or open your vector store
+db = RagEmbeddings::Database.new("knowledge_base.db")
+
+# 2) embed your user question
+client      = OpenAI::Client.new(api_key: ENV.fetch("OPENAI_API_KEY"))
+q_embedding = client.embeddings(
+  parameters: {
+    model: "text-embedding-ada-002",
+    input: "What are the benefits of retrieval-augmented generation?"
+  }
+).dig("data", 0, "embedding")
+
+# 3) retrieve top-3 relevant passages
+results = db.top_k_similar(q_embedding, k: 3)
+
+# 4) build a prompt for your LLM
+context = results.map { |id, text, score| text }.join("\n\n---\n\n")
+prompt  = <<~PROMPT
+  You are an expert.  
+  Use the following context to answer the question:
+
+  CONTEXT:
+  #{context}
+
+  QUESTION:
+  What are the benefits of retrieval-augmented generation?
+PROMPT
+
+# 5) call the LLM for final answer
+response = client.chat(
+  parameters: {
+    model: "gpt-4o",
+    messages: [{ role: "user", content: prompt }]
+  }
+)
+puts response.dig("choices", 0, "message", "content")
+
+```
+
+### 7. In-memory store for fast prototyping
+
+```ruby
+# use SQLite :memory: for ephemeral experiments
+db = RagEmbeddings::Database.new(":memory:")
+
+# insert & search exactly as with a file-backed DB
+db.insert("Quick test", RagEmbeddings.embed("Quick test"))
+db.top_k_similar("Test", k: 1)
+```
+
+---
+
 ## 🏗️ How it works
 
-- Embeddings are managed as dynamic C objects for efficiency (variable dimension).
-- The only correct way to construct an embedding object is using .from_array.
-- Langchainrb integration lets you easily change the embedding provider (Ollama, OpenAI, etc).
-- Storage uses local SQLite with embeddings as BLOB, for maximum portability and simplicity.
+**rag_embeddings** combines the simplicity of Ruby with the performance of C to deliver fast vector operations for RAG applications.
+
+### Architecture Overview
+
+The library uses a **hybrid memory-storage approach**:
+
+1. **In-Memory Processing**: All vector operations (cosine similarity calculations, embedding manipulations) happen entirely in memory using optimized C code
+2. **Persistent Storage**: SQLite serves as a simple, portable storage layer for embeddings and associated text
+3. **Dynamic C Objects**: Embeddings are managed as native C structures with automatic memory management
+
+### Key Components
+
+**C Extension (`embedding.c`)**
+- Handles all computationally intensive operations
+- Manages dynamic vector dimensions (adapts to any LLM output size)
+- Performs cosine similarity calculations with optimized algorithms
+- Ensures memory-safe operations with proper garbage collection integration
+
+**Ruby Interface**
+- Provides an intuitive API for vector operations
+- Integrates seamlessly with LLM providers via langchainrb
+- Handles database operations and query orchestration
+
+**SQLite Storage**
+- Stores embeddings as BLOBs alongside their associated text
+- Provides persistent storage without requiring external databases
+- Supports both file-based and in-memory (`:memory:`) databases
+- Enables portable, self-contained applications
+
+### Processing Flow
+
+1. **Text → Embedding**: Generate vectors using your preferred LLM (Ollama, OpenAI, etc.)
+2. **Memory Allocation**: Create C embedding objects with `Embedding.from_array()`
+3. **Storage**: Persist embeddings and text to SQLite for later retrieval
+4. **Query Processing**:
+    - Load query embedding into memory
+    - Compare against stored embeddings using fast C-based cosine similarity
+    - Return top-K most similar results ranked by similarity score
+
+### Why This Design?
+
+**Performance**: Critical operations run in optimized C code, delivering significant speed improvements over pure Ruby implementations.
+
+**Memory Efficiency**: While embeddings are stored in SQLite, all vector computations happen in memory, avoiding I/O bottlenecks during similarity calculations.
+
+**Simplicity**: SQLite eliminates the need for complex vector database setups while maintaining good performance for moderate-scale applications.
+
+**Portability**: The entire knowledge base fits in a single SQLite file, making deployment and backup trivial.
+
+### Performance Characteristics
+
+- **Embedding creation**: ~82ms for 10,000 operations
+- **Cosine similarity**: ~107ms for 10,000 calculations
+- **Memory usage**: ~34MB for 10,000 embeddings
+- **Scalability**: Suitable for thousands to tens of thousands of vectors
+
+For applications requiring millions of vectors, consider specialized vector databases (Faiss, sqlite-vss) while using this library for prototyping and smaller-scale production use.
 
 ## 🎛️ Customization
 
 - Embedding provider: switch model/provider in engine.rb (Ollama, OpenAI, etc)
 - Database: set the SQLite file path as desired
 
-## 🔢 Embeddings dimension
+If you need to customize the c part (`ext/rag_embeddings/embedding.c`), recompile it with:
+
+`rake compile`
 
-The size of embeddings is dynamic and fits with what the LLM provides.
+---
+
+## 🏁 Running the test suite
+
+To run all specs (RSpec required):
+
+`bundle exec rspec`
 
 ## ⚡️ Performance
 
-Embedding creation (10000 times): 82 ms
-Cosine similarity (10000 times): 107 ms
-RSS: 186.7 MB
-.
-Memory usage delta: 33.97 MB for 10000 embeddings
-.
+`bundle exec rspec spec/performance_spec.rb`
 
-Finished in 0.42577 seconds (files took 0.06832 seconds to load)
-2 examples, 0 failures
+You'll get something like this in random order:
 
-## 👷 Requirements
+```bash
+Performance test with embedding size: 768
+Embedding creation (10000 times): 19 ms
+Cosine similarity (10000 times): 27 ms
+RSS: 132.3 MB
 
-- Ruby >= 3.3
-- langchainrb (for embedding)
-- sqlite3 (for storage)
-- A working C compiler
+Memory usage test with embedding size: 768
+Memory usage delta: 3.72 MB for 10000 embeddings
 
-## 📑 Notes
 
-- Always create embeddings with .from_array
-- All memory management is idiomatic and safe
-- For millions of vectors, consider vector DBs (Faiss, sqlite-vss, etc.)
+Performance test with embedding size: 2048
+Embedding creation (10000 times): 69 ms
+Cosine similarity (10000 times): 73 ms
+RSS: 170.08 MB
+
+Memory usage test with embedding size: 2048
+Memory usage delta: 25.11 MB for 10000 embeddings
+
+
+Performance test with embedding size: 3072
+Embedding creation (10000 times): 98 ms
+Cosine similarity (10000 times): 112 ms
+RSS: 232.97 MB
+
+Memory usage test with embedding size: 3072
+Memory usage delta: 60.5 MB for 10000 embeddings
+
+
+Performance test with embedding size: 4096
+Embedding creation (10000 times): 96 ms
+Cosine similarity (10000 times): 140 ms
+RSS: 275.2 MB
+
+Memory usage test with embedding size: 4096
+Memory usage delta: 92.41 MB for 10000 embeddings
+```
 
 ## 📬 Contact & Issues
 Open an issue or contact the maintainer for questions, suggestions, or bugs.

data/Rakefile

@@ -1,5 +1,8 @@
 task :compile do
   Dir.chdir("ext/rag_embeddings") do
+    # Delete embedding.so or embedding.o
+    # Delete embedding.bundle and the folder embedding.bundle.*
+    FileUtils.rm_rf(Dir["embedding.so", "embedding.o", "embedding.bundle", "embedding.bundle.*"])
     ruby "extconf.rb"
     system("make")
   end

data/lib/rag_embeddings/version.rb

@@ -1,3 +1,3 @@
 module RagEmbeddings
-  VERSION = "0.2.0"
+  VERSION = "0.2.2".freeze
 end

data/lib/rag_embeddings.rb

@@ -1,4 +1,8 @@
 require_relative "rag_embeddings/version"
 require_relative "rag_embeddings/engine"
 require_relative "rag_embeddings/database"
-require_relative "../ext/rag_embeddings/embedding" # Loads the compiled C extension
+
+# Loads the compiled C extension
+require "rag_embeddings/embedding"
+
+require "faraday"

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: rag_embeddings
 version: !ruby/object:Gem::Version
-  version: 0.2.0
+  version: 0.2.2
 platform: ruby
 authors:
 - Marco Mastrodonato
@@ -51,6 +51,76 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
+- !ruby/object:Gem::Dependency
+  name: rake
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: rspec
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: rubocop
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: dotenv
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: debug
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
 description: Manage AI vector embeddings in C with Ruby integration
 email:
 - m.mastrodonato@gmail.com
@@ -77,6 +147,7 @@ metadata:
 rdoc_options: []
 require_paths:
 - lib
+- ext
 required_ruby_version: !ruby/object:Gem::Requirement
   requirements:
   - - ">="