ask-rag 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 908700d8186c0433071a87f8b6f28ec3d60f857ef9bd0a5cf474a1804f888251
+  data.tar.gz: 9abbec0e77ea2ca596b065c67af63d92ea373da22dd33f418442119a0d32defd
+SHA512:
+  metadata.gz: 20d21031b52e4b0594f5485bbd8e709a71fa716115676ad493ebee998085e4ce8a378d4023a94b3c9587221837c4fcd18488c09c270c306714b4ff16ab9e7b1c
+  data.tar.gz: 2b2a31e86ae1d5adceb08782d186db4347b57aa1b15e899e8210c20dd65732c3ba7e32ff79b1109e8fe958b77ddd8e2a91361f0c389c14769a70f0bbb2bad323

data/CHANGELOG.md

@@ -0,0 +1,31 @@
+## [0.1.0] — 2026-07-26
+
+### Added
+
+- **Initial release** — RAG pipeline for the ask-rb ecosystem.
+
+- **`Ask::Document`** — text+metadata value object (re-exported from ask-core 0.6.0).
+
+- **Loaders** — load files into documents:
+  - `Ask::RAG::Loader::Text` — plain text files
+  - `Ask::RAG::Loader::Markdown` — Markdown files
+  - `Ask::RAG::Loader::CSV` — CSV files (one document per row, configurable content/metadata columns)
+  - `Ask::RAG::Loader::HTML` — HTML files with Nokogiri (removes script, style, nav elements)
+  - `Ask::RAG::Loader::PDF` — PDF files via pdf-reader (one document per page)
+
+- **Text Splitters** — split documents into smaller chunks:
+  - `Ask::RAG::TextSplitter::RecursiveCharacter` — recursively splits by `\n\n`, `\n`, `.`, ` `, character
+  - `Ask::RAG::TextSplitter::Markdown` — splits by headers, preserves header hierarchy
+
+- **Vector Stores** — store and search document embeddings:
+  - `Ask::RAG::VectorStore::InMemory` — pure Ruby cosine similarity, zero dependencies
+  - `Ask::RAG::VectorStore::PGVector` — PostgreSQL + pgvector adapter (optional, loaded if gem present)
+
+- **High-level Query API** — `Ask::RAG::Query.query` for one-shot retrieve → prompt → answer
+
+- **Design**:
+  - HTML and PDF loaders auto-activate when their dependency gems are installed
+  - PGVector adapter auto-activates when the pgvector gem is available
+  - All loaders provide both `load` (eager) and `lazy_load` (enumerator) interfaces
+  - All splitters implement `split_text` (raw text) and `split_documents` (Document arrays)
+  - Scores returned in document metadata as `:score`

data/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Kaka Ruto
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,197 @@
+# ask-rag
+
+**RAG pipeline for the ask-rb ecosystem.** Load documents, split them into chunks, store embeddings, and retrieve relevant context — all from Ruby.
+
+```ruby
+gem "ask-rag"
+```
+
+## Quick Start
+
+```ruby
+require "ask-rag"
+
+# 1. Load and split
+loader = Ask::RAG::Loader::Text.new
+docs = loader.load("README.md")
+
+splitter = Ask::RAG::TextSplitter::RecursiveCharacter.new(
+  chunk_size: 500, chunk_overlap: 50
+)
+chunks = splitter.split_documents(docs)
+
+# 2. Embed and store
+store = Ask::RAG::VectorStore::InMemory.new
+store.add(chunks, model: "text-embedding-3-small")
+
+# 3. Search
+results = store.similarity_search("How do I install ask-rag?", limit: 3)
+results.each do |doc|
+  puts "[#{doc.metadata[:score].round(3)}] #{doc.content[0..80]}..."
+end
+
+# 4. Or one-shot RAG query
+answer = Ask::RAG::Query.query(
+  store: store,
+  question: "What are the dependencies?",
+  model: "gpt-4o"
+)
+puts answer.content
+```
+
+## Components
+
+### Loaders
+
+Load files into `Ask::Document` objects (text + metadata).
+
+| Loader | Format | Dependency |
+|---|---|---|
+| `Ask::RAG::Loader::Text` | Plain text | None (stdlib) |
+| `Ask::RAG::Loader::Markdown` | Markdown | None (stdlib) |
+| `Ask::RAG::Loader::CSV` | CSV (1 doc/row) | None (`csv` gem) |
+| `Ask::RAG::Loader::HTML` | HTML (strips script/style/nav) | `nokogiri` |
+| `Ask::RAG::Loader::PDF` | PDF (1 doc/page) | `pdf-reader` |
+
+Loaders without their dependency gem installed are skipped silently — install the gem when you need the format.
+
+```ruby
+loader = Ask::RAG::Loader::PDF.new
+docs = loader.load("manual.pdf")
+docs.length      # => number of pages
+docs[0].metadata # => { source: "manual.pdf", page: 1, format: "pdf" }
+```
+
+### Text Splitters
+
+Split documents into smaller, semantically coherent chunks.
+
+| Splitter | Strategy | Best for |
+|---|---|---|
+| `RecursiveCharacter` | Descending separators (`\n\n`, `\n`, `. `, ` `, char) | General text |
+| `Markdown` | Split on headers, preserve hierarchy | Markdown docs |
+
+```ruby
+splitter = Ask::RAG::TextSplitter::RecursiveCharacter.new(
+  chunk_size: 1000,
+  chunk_overlap: 200
+)
+chunks = splitter.split_documents(docs)
+# metadata[:chunk] tracks the chunk index per source document
+
+splitter = Ask::RAG::TextSplitter::Markdown.new
+chunks = splitter.split_text(markdown_content)
+# Each chunk prepends header hierarchy: "Introduction > Background\n\ncontent..."
+```
+
+### Vector Stores
+
+Store embeddings and search by similarity.
+
+| Store | Description | Dependencies |
+|---|---|---|
+| `InMemory` | Pure Ruby cosine similarity. Zero deps. | None |
+| `PGVector` (auto-loaded) | PostgreSQL + pgvector extension | `pgvector`, `activerecord` |
+
+```ruby
+# InMemory — no database needed
+store = Ask::RAG::VectorStore::InMemory.new
+store.add(documents, model: "text-embedding-3-small")
+results = store.similarity_search("query", limit: 5)
+
+# PGVector — for production Rails apps
+store = Ask::RAG::VectorStore::PGVector.new(table_name: :embeddings)
+store.add(documents, model: "text-embedding-3-small")
+```
+
+Search results have their similarity score in `metadata[:score]`:
+
+```ruby
+results = store.similarity_search("authentication", limit: 3)
+results.each do |doc|
+  puts "#{doc.metadata[:score]} — #{doc.content[0..60]}"
+end
+```
+
+### High-Level RAG Query
+
+Retrieve + prompt + answer in one call:
+
+```ruby
+answer = Ask::RAG::Query.query(
+  store: store,
+  question: "What is the default timeout configuration?",
+  model: "gpt-4o"
+)
+
+puts answer.content           # The LLM's answer
+puts answer.metadata[:sources]  # ["config.md", "settings.md"]
+```
+
+## Pipeline (End-to-End)
+
+```ruby
+# Load
+loader = Ask::RAG::Loader::PDF.new
+docs = loader.load("user_manual.pdf")
+
+# Split
+splitter = Ask::RAG::TextSplitter::RecursiveCharacter.new(
+  chunk_size: 500, chunk_overlap: 50
+)
+chunks = splitter.split_documents(docs)
+
+# Store
+store = Ask::RAG::VectorStore::InMemory.new
+store.add(chunks, model: "text-embedding-3-small")
+
+# Ask
+answer = Ask::RAG::Query.query(
+  store: store,
+  question: "How do I reset my password?",
+  model: "gpt-4o"
+)
+puts answer.content
+```
+
+## Integration with ask-agent
+
+```ruby
+require "ask-rag"
+require "ask-agent"
+
+# Index your documentation
+store = Ask::RAG::VectorStore::InMemory.new
+loader = Ask::RAG::Loader::Markdown.new
+splitter = Ask::RAG::TextSplitter::RecursiveCharacter.new(chunk_size: 1000)
+docs = splitter.split_documents(loader.load("docs/"))
+store.add(docs, model: "text-embedding-3-small")
+
+# Give the agent a search tool
+class SearchDocs < Ask::Tool
+  description "Search documentation for relevant information"
+  param :query, type: :string, desc: "Search query"
+
+  def execute(query:)
+    store.similarity_search(query, limit: 3).map do |doc|
+      { content: doc.content[0..500], score: doc.metadata[:score] }
+    end
+  end
+end
+
+session = Ask::Agent::Session.new(
+  model: "gpt-4o",
+  tools: [SearchDocs]
+)
+session.run("How do I configure authentication?")
+```
+
+## Development
+
+```bash
+bundle exec rake test
+```
+
+## License
+
+MIT

data/lib/ask/rag/document.rb

@@ -0,0 +1,9 @@
+# frozen_string_literal: true
+
+module Ask
+  module RAG
+    # Re-export Ask::Document so it's accessible as Ask::RAG::Document too,
+    # for users who prefer to namespace everything under Ask::RAG.
+    Document = Ask::Document unless const_defined?(:Document)
+  end
+end

data/lib/ask/rag/loader/base.rb

@@ -0,0 +1,27 @@
+# frozen_string_literal: true
+
+module Ask
+  module RAG
+    module Loader
+      class Base
+        # Load a file and return an array of Ask::Document objects.
+        # Subclasses implement #load to produce documents from a specific format.
+        #
+        # @param path [String] path to the file
+        # @return [Array<Ask::Document>]
+        def load(path)
+          raise NotImplementedError, "#{self.class} must implement #load"
+        end
+
+        # Load a file and return a lazy enumerator of documents.
+        # Override in subclasses that support streaming.
+        #
+        # @param path [String] path to the file
+        # @return [Enumerator<Ask::Document>]
+        def lazy_load(path)
+          load(path).each
+        end
+      end
+    end
+  end
+end

data/lib/ask/rag/loader/csv.rb

@@ -0,0 +1,64 @@
+# frozen_string_literal: true
+
+require "csv"
+
+module Ask
+  module RAG
+    module Loader
+      # Loads a CSV file, producing one document per row.
+      #
+      # Each document's content is the row's cell values joined by newlines.
+      # Metadata includes the source path and any identifying columns.
+      class CSV < Base
+        # @param path [String] path to the CSV file
+        # @param content_columns [Array<String>, nil] columns to include in content
+        #   (nil = all columns)
+        # @param metadata_columns [Array<String>, nil] columns to include in metadata
+        #   (nil = all columns except content_columns)
+        # @param separator [String] column separator (default: ",")
+        def initialize(content_columns: nil, metadata_columns: nil, separator: ",")
+          @content_columns = content_columns
+          @metadata_columns = metadata_columns
+          @separator = separator
+        end
+
+        def load(path)
+          docs = []
+          options = { headers: true, col_sep: @separator }
+
+          ::CSV.foreach(path, **options) do |row|
+            content = build_content(row)
+            metadata = build_metadata(row, path)
+            docs << Ask::Document.new(content: content, metadata: metadata)
+          end
+
+          docs
+        end
+
+        private
+
+        def build_content(row)
+          if @content_columns
+            @content_columns.map { |col| row[col] }.compact.join("\n")
+          else
+            row.fields.join("\n")
+          end
+        end
+
+        def build_metadata(row, path)
+          meta = { source: path }
+          if @metadata_columns
+            @metadata_columns.each do |col|
+              meta[col] = row[col] if row[col]
+            end
+          elsif @content_columns
+            row.headers.each do |h|
+              meta[h] = row[h] unless @content_columns.include?(h)
+            end
+          end
+          meta
+        end
+      end
+    end
+  end
+end

data/lib/ask/rag/loader/html.rb

@@ -0,0 +1,33 @@
+# frozen_string_literal: true
+
+module Ask
+  module RAG
+    module Loader
+      # Loads an HTML file and extracts text content using Nokogiri.
+      # Strips script, style, and nav elements by default.
+      class HTML < Base
+        # @param elements_to_remove [Array<String>] CSS selectors to strip
+        def initialize(elements_to_remove: %w[script style nav footer header])
+          @elements_to_remove = elements_to_remove
+        end
+
+        def load(path)
+          content = File.read(path)
+          doc = Nokogiri::HTML.parse(content)
+
+          @elements_to_remove.each do |selector|
+            doc.css(selector).each(&:remove)
+          end
+
+          text = doc.at_xpath("//body")&.text || doc.text
+          text = text.gsub(/\s+/, " ").strip
+
+          [Ask::Document.new(
+            content: text,
+            metadata: { source: path, format: "html" }
+          )]
+        end
+      end
+    end
+  end
+end

data/lib/ask/rag/loader/markdown.rb

@@ -0,0 +1,19 @@
+# frozen_string_literal: true
+
+module Ask
+  module RAG
+    module Loader
+      # Loads a Markdown file as a single document.
+      # Use MarkdownTextSplitter afterward to split by headers.
+      class Markdown < Base
+        def load(path)
+          content = File.read(path)
+          [Ask::Document.new(
+            content: content,
+            metadata: { source: path, format: "markdown" }
+          )]
+        end
+      end
+    end
+  end
+end

data/lib/ask/rag/loader/pdf.rb

@@ -0,0 +1,21 @@
+# frozen_string_literal: true
+
+module Ask
+  module RAG
+    module Loader
+      # Loads a PDF file using the pdf-reader gem, producing one document
+      # per page. Each document's metadata includes the page number.
+      class PDF < Base
+        def load(path)
+          reader = PDF::Reader.new(path)
+          reader.pages.map.with_index(1) do |page, idx|
+            Ask::Document.new(
+              content: page.text,
+              metadata: { source: path, page: idx, format: "pdf" }
+            )
+          end
+        end
+      end
+    end
+  end
+end

data/lib/ask/rag/loader/text.rb

@@ -0,0 +1,18 @@
+# frozen_string_literal: true
+
+module Ask
+  module RAG
+    module Loader
+      # Loads a plain text file as a single document.
+      class Text < Base
+        def load(path)
+          content = File.read(path)
+          [Ask::Document.new(
+            content: content,
+            metadata: { source: path }
+          )]
+        end
+      end
+    end
+  end
+end

data/lib/ask/rag/query.rb

@@ -0,0 +1,97 @@
+# frozen_string_literal: true
+
+module Ask
+  module RAG
+    # High-level RAG query API.
+    #
+    # Retrieves relevant documents from a vector store and uses an LLM to
+    # answer a question based on those documents.
+    #
+    # @example
+    #   answer = Ask::RAG.query(
+    #     store: my_vector_store,
+    #     question: "What is the default timeout?",
+    #     model: "gpt-4o"
+    #   )
+    #   puts answer.content
+    #
+    module Query
+      module_function
+
+      # Perform a RAG query: retrieve relevant documents, build a prompt,
+      # and ask an LLM for an answer based on those documents.
+      #
+      # @param store [Ask::RAG::VectorStore] the vector store to search
+      # @param question [String] the user's question
+      # @param model [String] LLM model to use for answering
+      # @param limit [Integer] number of documents to retrieve (default: 5)
+      # @param system_prompt [String, nil] custom system prompt override
+      # @param provider [Symbol, nil] optional provider override
+      # @return [Ask::Document, nil] the answer with metadata including sources
+      def query(store:, question:, model: nil, limit: 5, system_prompt: nil, provider: nil)
+        documents = store.similarity_search(question, limit: limit)
+
+        return nil if documents.empty?
+
+        context = documents.map.with_index(1) do |doc, i|
+          "[#{i}] #{doc.content}\n"
+        end.join("\n")
+
+        prompt = system_prompt || build_default_prompt
+        full_prompt = "#{prompt}\n\nContext:\n#{context}\n\nQuestion: #{question}"
+
+        answer = ask_llm(full_prompt, model: model, provider: provider)
+
+        Ask::Document.new(
+          content: answer,
+          metadata: {
+            sources: documents.map { |d| d.metadata[:source] }.compact.uniq,
+            model: model
+          }
+        )
+      end
+
+      # Generate an answer using the LLM.
+      def ask_llm(prompt, model: nil, provider: nil) # rubocop:disable Metrics/MethodLength
+        require "ask/agent" unless defined?(Ask::Agent)
+
+        session_opts = { tools: [] }
+        session_opts[:model] = model if model
+        session_opts[:provider] = provider if provider
+
+        session = Ask::Agent::Session.new(**session_opts)
+        response = session.run(prompt)
+        response.to_s
+      rescue NameError
+        # Fallback: use basic Ask::Provider directly if ask-agent isn't available
+        fallback_ask(prompt, model)
+      end
+
+      def build_default_prompt
+        "You are a helpful assistant. Answer the question based on the provided context. " \
+        "If the context doesn't contain enough information to answer, say so."
+      end
+
+      def fallback_ask(prompt, model)
+        model ||= "gpt-4o"
+        info = Ask::ModelCatalog.find(model)
+        klass = Ask::Provider.resolve(info.provider)
+
+        config = Object.new
+        klass.configuration_requirements.each do |req|
+          val = ENV["#{req.to_s.upcase}"]
+          config.define_singleton_method(req) { val } if val
+        end
+
+        provider = klass.new(config)
+        conv = Ask::Conversation.new
+        conv.system("Answer based on the context provided.")
+        conv.user(prompt)
+        response = provider.chat(conv.to_a, model: model)
+        response.content.to_s
+      end
+
+      private_class_method :build_default_prompt, :fallback_ask
+    end
+  end
+end

data/lib/ask/rag/text_splitter/base.rb

@@ -0,0 +1,48 @@
+# frozen_string_literal: true
+
+module Ask
+  module RAG
+    module TextSplitter
+      class Base
+        # @param chunk_size [Integer] maximum size of each chunk
+        # @param chunk_overlap [Integer] overlap between consecutive chunks
+        # @param length_function [#call] function that measures text length (default: :length)
+        def initialize(chunk_size: 1000, chunk_overlap: 200, length_function: :length.to_proc)
+          @chunk_size = chunk_size
+          @chunk_overlap = chunk_overlap
+          @length_function = length_function
+        end
+
+        # Split a single string into chunks.
+        # @param text [String] text to split
+        # @return [Array<String>] text chunks
+        def split_text(text)
+          raise NotImplementedError
+        end
+
+        # Split documents into smaller documents.
+        # @param documents [Array<Ask::Document>] documents to split
+        # @return [Array<Ask::Document>] split documents with metadata preserved
+        def split_documents(documents)
+          documents.flat_map do |doc|
+            chunks = split_text(doc.content)
+            chunks.map.with_index do |chunk, idx|
+              Ask::Document.new(
+                content: chunk,
+                metadata: doc.metadata.merge(chunk: idx)
+              )
+            end
+          end
+        end
+
+        private
+
+        attr_reader :chunk_size, :chunk_overlap, :length_function
+
+        def measure(text)
+          length_function.call(text)
+        end
+      end
+    end
+  end
+end

data/lib/ask/rag/text_splitter/markdown.rb

@@ -0,0 +1,66 @@
+# frozen_string_literal: true
+
+module Ask
+  module RAG
+    module TextSplitter
+      # Splits Markdown text by headers, preserving header hierarchy in metadata.
+      #
+      # Each chunk starts at a header and includes all content until the next
+      # header at the same or higher level. Header context is tracked in
+      # metadata so retrieval results know which section they came from.
+      #
+      # @example
+      #   splitter = Ask::RAG::TextSplitter::Markdown.new
+      #   chunks = splitter.split_documents(docs)
+      #   chunks[0].metadata[:headers] # => { h1: "Introduction", h2: "Background" }
+      #
+      class Markdown < Base
+        HEADER_PATTERN = /^(#{'#'}{1,6})\s+(.+)$/
+
+        def split_text(text)
+          chunks = []
+          current_header = {}
+          current_lines = []
+
+          text.each_line do |line|
+            if (match = line.match(HEADER_PATTERN))
+              # Save previous section
+              if current_lines.any?
+                chunks << build_chunk(current_lines.join, current_header)
+              end
+
+              level = match[1].length
+              heading_text = match[2].strip
+
+              # Clear lower-level headers
+              current_header.select! { |k, _| HEADER_LEVELS[k] && HEADER_LEVELS[k] < level }
+              current_header["h#{level}".to_sym] = heading_text
+
+              current_lines = []
+            else
+              current_lines << line
+            end
+          end
+
+          # Last section
+          chunks << build_chunk(current_lines.join, current_header) if current_lines.any?
+          chunks
+        end
+
+        private
+
+        HEADER_LEVELS = { h1: 1, h2: 2, h3: 3, h4: 4, h5: 5, h6: 6 }.freeze
+
+        def build_chunk(text, headers)
+          cleaned = text.strip
+          return nil if cleaned.empty?
+
+          # Include header text at the top for context
+          header_line = headers.values.join(" > ")
+          content = header_line.empty? ? cleaned : "#{header_line}\n\n#{cleaned}"
+          content
+        end
+      end
+    end
+  end
+end

data/lib/ask/rag/text_splitter/recursive_character.rb

@@ -0,0 +1,101 @@
+# frozen_string_literal: true
+
+module Ask
+  module RAG
+    module TextSplitter
+      # Splits text recursively by descending separator length.
+      #
+      # The default separators try paragraph breaks, then line breaks,
+      # then sentences, then words, then characters. This produces
+      # semantically coherent chunks that fit within chunk_size.
+      #
+      # @example
+      #   splitter = Ask::RAG::TextSplitter::RecursiveCharacter.new(
+      #     chunk_size: 500, chunk_overlap: 50
+      #   )
+      #   chunks = splitter.split_text(long_text)
+      #
+      class RecursiveCharacter < Base
+        # @param separators [Array<String>] ordered list of separators to try
+        # @param chunk_size [Integer] max chunk size
+        # @param chunk_overlap [Integer] overlap between chunks
+        def initialize(separators: ["\n\n", "\n", ". ", " ", ""], **options)
+          @separators = separators
+          super(**options)
+        end
+
+        def split_text(text)
+          splits = split_with_separators(text, @separators)
+          merge_splits(splits)
+        end
+
+        private
+
+        def split_with_separators(text, seps)
+          separator = seps.first || ""
+          splits = if separator.empty?
+            text.chars.map { |c| [c, separator] }
+          else
+            text.split(separator).map { |s| [s, separator] }
+          end
+
+          # Recursively split any piece that's still too long
+          splits = splits.flat_map do |piece, sep|
+            remaining = seps[1..]
+            if remaining.any? && piece.length > chunk_size * 1.5
+              split_with_separators(piece, remaining)
+            else
+              [[piece, sep]]
+            end
+          end
+
+          splits
+        end
+
+        def merge_splits(splits)
+          chunks = []
+          current = +""
+          current_size = 0
+
+          splits.each do |piece, _separator|
+            piece_size = measure(piece)
+
+            if current_size + piece_size <= chunk_size || current.empty?
+              current << piece
+              current_size += piece_size
+            else
+              # Finalize current chunk
+              chunks << current.dup
+
+              # Build overlap from the end of the current chunk
+              overlap = build_overlap(current)
+              current = overlap.dup
+              current_size = measure(current)
+              current << piece
+              current_size += piece_size
+            end
+          end
+
+          chunks << current unless current.empty?
+          chunks
+        end
+
+        def build_overlap(text)
+          return +"" if chunk_overlap <= 0
+
+          chars = text.chars
+          overlap_chars = []
+          overlap_len = 0
+
+          chars.reverse_each do |c|
+            break if overlap_len >= chunk_overlap
+            overlap_chars.unshift(c)
+            overlap_len += measure(c)
+          end
+
+          overlap_chars.join
+        end
+      end
+    end
+  end
+end

data/lib/ask/rag/vector_store/base.rb

@@ -0,0 +1,54 @@
+# frozen_string_literal: true
+
+module Ask
+  module RAG
+    # Abstract base class for vector stores.
+    #
+    # Vector stores store embedded documents and provide similarity search.
+    # Subclasses implement the storage and retrieval logic for specific
+    # backends (InMemory, PGVector, Chroma, etc.).
+    #
+    # @example
+    #   store = Ask::RAG::VectorStore::InMemory.new
+    #   store.add(documents, model: "text-embedding-3-small")
+    #   results = store.similarity_search("query text", limit: 5)
+    #
+    class VectorStore
+      # Add documents to the store. They are embedded using the given model
+      # via ask-llm-providers or a custom embedding function.
+      #
+      # @param documents [Array<Ask::Document>] documents to add
+      # @param model [String] embedding model name (e.g. "text-embedding-3-small")
+      # @param batch_size [Integer] number of documents to embed per API call
+      # @return [Array<String>] IDs of the added documents
+      def add(documents, model:, batch_size: 20)
+        raise NotImplementedError
+      end
+
+      # Search for documents similar to the given query text.
+      #
+      # @param query [String] the query text
+      # @param limit [Integer] maximum number of results
+      # @return [Array<Ask::Document>] documents with a +score+ attribute added
+      def similarity_search(query, limit: 10)
+        raise NotImplementedError
+      end
+
+      # Remove documents by ID.
+      # @param ids [Array<String>] document IDs to remove
+      def delete(ids)
+        raise NotImplementedError
+      end
+
+      # Remove all documents from the store.
+      def clear
+        raise NotImplementedError
+      end
+
+      # @return [Integer] number of documents in the store
+      def size
+        raise NotImplementedError
+      end
+    end
+  end
+end

data/lib/ask/rag/vector_store/in_memory.rb

@@ -0,0 +1,172 @@
+# frozen_string_literal: true
+
+module Ask
+  module RAG
+    class VectorStore
+      # In-memory vector store using cosine similarity.
+      #
+      # Stores all documents and vectors in a Ruby Hash. Uses pure Ruby for
+      # cosine similarity — no external dependencies. Suitable for
+      # development, testing, and small-scale prototypes.
+      #
+      # @example
+      #   store = Ask::RAG::VectorStore::InMemory.new
+      #   store.add(chunks, model: "text-embedding-3-small")
+      #   results = store.similarity_search("query", limit: 5)
+      #
+      class InMemory < VectorStore
+        Entry = Struct.new(:id, :document, :vector, keyword_init: true)
+
+        def initialize
+          @entries = {}
+          @mutex = Mutex.new
+        end
+
+        def add(documents, model:, batch_size: 20)
+          ids = documents.map { |d| d.id || SecureRandom.uuid }
+          texts = documents.map(&:content)
+
+          # Embed using ask-llm-providers
+          provider = resolve_embedding_provider(model)
+          raw_vectors = provider.embed(texts, model: model)
+
+          vectors = normalize_vectors(raw_vectors, texts)
+
+          @mutex.synchronize do
+            documents.each_with_index do |doc, idx|
+              @entries[ids[idx]] = Entry.new(
+                id: ids[idx],
+                document: doc,
+                vector: vectors[idx]
+              )
+            end
+          end
+
+          ids
+        end
+
+        def similarity_search(query, limit: 10)
+          query_vector = embed_query(query)
+
+          scored = @mutex.synchronize do
+            @entries.map do |_id, entry|
+              score = cosine_similarity(query_vector, entry.vector)
+              [entry, score]
+            end
+          end
+
+          scored.sort_by! { |_, score| -score }
+          scored.first(limit).map do |entry, score|
+            doc = entry.document
+            Ask::Document.new(
+              content: doc.content,
+              metadata: doc.metadata.merge(score: score),
+              id: doc.id
+            )
+          end
+        end
+
+        def delete(ids)
+          @mutex.synchronize do
+            ids.each { |id| @entries.delete(id) }
+          end
+        end
+
+        def clear
+          @mutex.synchronize { @entries.clear }
+        end
+
+        def size
+          @mutex.synchronize { @entries.size }
+        end
+
+        private
+
+        def resolve_embedding_provider(model)
+          info = Ask::ModelCatalog.find(model)
+          provider_class = Ask::Provider.resolve(info.provider)
+          config = build_provider_config(provider_class)
+          provider_class.new(config)
+        rescue Ask::ModelNotFound, Ask::UnknownProvider
+          # Fallback: try to use OpenAI-compatible directly
+          Ask::Provider.resolve(:openai).new(
+            build_provider_config(Ask::Provider.resolve(:openai))
+          )
+        end
+
+        def build_provider_config(provider_class)
+          config = Object.new
+          provider_class.configuration_requirements.each do |req|
+            val = ENV["#{req.to_s.upcase}"] || ENV["#{provider_class.slug.upcase}_#{req.to_s.upcase}"]
+            config.define_singleton_method(req) { val } if val
+          end
+          provider_class.configuration_options.each do |opt|
+            val = ENV["#{opt.to_s.upcase}"] || ENV["#{provider_class.slug.upcase}_#{opt.to_s.upcase}"]
+            config.define_singleton_method(opt) { val } if val
+          end
+          config
+        end
+
+        def embed_query(query)
+          provider = resolve_embedding_provider("text-embedding-3-small")
+          raw = provider.embed(query, model: "text-embedding-3-small")
+
+          if raw.is_a?(Ask::Result)
+            raw = raw.output
+          end
+
+          Array(raw).flatten.map(&:to_f)
+        rescue StandardError
+          # Return a zero vector as fallback (will produce no meaningful matches)
+          [0.0] * 256
+        end
+
+        def normalize_vectors(raw_vectors, texts)
+          result = if raw_vectors.is_a?(Ask::Result)
+            raw_vectors.output
+          else
+            raw_vectors
+          end
+
+          result = Array(result)
+
+          # If the provider returned a flat vector for a single document
+          if result.length == 1 && texts.length == 1
+            [Array(result.first).map(&:to_f)]
+          elsif result.first.is_a?(Array)
+            result.map { |v| Array(v).map(&:to_f) }
+          else
+            # Flattened — try to split by dimensions
+            dims = infer_dimensions(result)
+            result.each_slice(dims).to_a
+          end
+        end
+
+        def infer_dimensions(flat)
+          # Common embedding dimensions
+          [3072, 1536, 1024, 768, 512, 384, 256].each do |d|
+            return d if flat.length % d == 0 && flat.length / d > 0
+          end
+          flat.length
+        end
+
+        def cosine_similarity(a, b)
+          return 0.0 if a.empty? || b.empty? || a.length != b.length
+
+          dot = 0.0
+          norm_a = 0.0
+          norm_b = 0.0
+
+          a.zip(b).each do |ai, bi|
+            dot += ai * bi
+            norm_a += ai * ai
+            norm_b += bi * bi
+          end
+
+          denom = Math.sqrt(norm_a) * Math.sqrt(norm_b)
+          denom > 0 ? dot / denom : 0.0
+        end
+      end
+    end
+  end
+end

data/lib/ask/rag/vector_store/pgvector.rb

@@ -0,0 +1,145 @@
+# frozen_string_literal: true
+
+module Ask
+  module RAG
+    class VectorStore
+      # PGVector-backed vector store using PostgreSQL + the pgvector extension.
+      #
+      # Stores document text, metadata, and embeddings in a PostgreSQL table.
+      # Requires the `pgvector` gem and an ActiveRecord connection.
+      #
+      # @example Setup
+      #   # In your migration:
+      #   #   create_table :documents do |t|
+      #   #     t.text :content
+      #   #     t.jsonb :metadata
+      #   #     t.vector :embedding, limit: 1536
+      #   #   end
+      #   #   add_index :documents, :embedding, using: :ivfflat, opclass: :vector_cosine_ops
+      #
+      # @example Usage
+      #   store = Ask::RAG::VectorStore::PGVector.new(
+      #     table_name: :documents,
+      #     embedding_column: :embedding
+      #   )
+      #   store.add(chunks, model: "text-embedding-3-small")
+      #   results = store.similarity_search("query", limit: 5)
+      #
+      class PGVector < VectorStore
+        # @param table_name [Symbol] the ActiveRecord table name
+        # @param content_column [Symbol] column storing text content (default: :content)
+        # @param metadata_column [Symbol] column storing JSON metadata (default: :metadata)
+        # @param embedding_column [Symbol] column storing the vector (default: :embedding)
+        # @param model_class [Class, nil] ActiveRecord model class (default: inferred from table_name)
+        def initialize(table_name:, content_column: :content, metadata_column: :metadata,
+                       embedding_column: :embedding, model_class: nil)
+          @table_name = table_name.to_s
+          @content_column = content_column.to_s
+          @metadata_column = metadata_column.to_s
+          @embedding_column = embedding_column.to_s
+          @model_class = model_class || infer_model_class
+        end
+
+        def add(documents, model:, batch_size: 20)
+          ids = []
+          model_class = @model_class
+
+          documents.each_slice(batch_size) do |batch|
+            texts = batch.map(&:content)
+            vectors = embed_texts(texts, model)
+
+            batch.each_with_index do |doc, idx|
+              record = model_class.create!(
+                @content_column => doc.content,
+                @metadata_column => doc.metadata,
+                @embedding_column => vectors[idx]
+              )
+              ids << record.id.to_s
+            end
+          end
+
+          ids
+        end
+
+        def similarity_search(query, limit: 10)
+          query_vector = embed_query(query)
+          column = "#{@table_name}.#{@embedding_column}"
+          model_class = @model_class
+
+          records = model_class
+            .select("#{@table_name}.*, 1 - (#{column} <=> '#{query_vector.to_s}') AS score")
+            .where("#{column} IS NOT NULL")
+            .order(Arel.sql("#{column} <=> '#{query_vector.to_s}'"))
+            .limit(limit)
+
+          records.map do |record|
+            doc = Ask::Document.new(
+              content: record.send(@content_column),
+              metadata: (record.send(@metadata_column) || {}).merge(db_id: record.id)
+            )
+            doc.define_singleton_method(:score) { record.score }
+            doc
+          end
+        end
+
+        def delete(ids)
+          @model_class.where(id: ids).delete_all
+        end
+
+        def clear
+          @model_class.delete_all
+        end
+
+        def size
+          @model_class.count
+        end
+
+        private
+
+        def infer_model_class
+          klass = Class.new(ActiveRecord::Base) # rubocop:disable Rails/ApplicationRecord
+          klass.table_name = @table_name
+          klass
+        end
+
+        def embed_texts(texts, model)
+          provider = resolve_provider(model)
+          raw = provider.embed(texts, model: model)
+          raw = raw.output if raw.is_a?(Ask::Result)
+          Array(raw).map { |v| Array(v).map(&:to_f) }
+        end
+
+        def embed_query(query)
+          provider = resolve_provider("text-embedding-3-small")
+          raw = provider.embed(query, model: "text-embedding-3-small")
+          raw = raw.output if raw.is_a?(Ask::Result)
+          Array(raw).flatten.map(&:to_f)
+        rescue StandardError
+          [0.0] * 1536
+        end
+
+        def resolve_provider(model)
+          info = Ask::ModelCatalog.find(model)
+          klass = Ask::Provider.resolve(info.provider)
+          config = build_config(klass)
+          klass.new(config)
+        rescue Ask::ModelNotFound, Ask::UnknownProvider
+          Ask::Provider.resolve(:openai).new(build_config(Ask::Provider.resolve(:openai)))
+        end
+
+        def build_config(klass)
+          config = Object.new
+          klass.configuration_requirements.each do |req|
+            val = ENV["#{req.to_s.upcase}"]
+            config.define_singleton_method(req) { val } if val
+          end
+          klass.configuration_options.each do |opt|
+            val = ENV["#{opt.to_s.upcase}"]
+            config.define_singleton_method(opt) { val } if val
+          end
+          config
+        end
+      end
+    end
+  end
+end

data/lib/ask/rag/version.rb

@@ -0,0 +1,7 @@
+# frozen_string_literal: true
+
+module Ask
+  module RAG
+    VERSION = "0.1.0"
+  end
+end

data/lib/ask/rag.rb

@@ -0,0 +1,41 @@
+# frozen_string_literal: true
+
+require_relative "rag/version"
+require_relative "rag/document"
+
+# Loaders
+require_relative "rag/loader/base"
+require_relative "rag/loader/text"
+require_relative "rag/loader/markdown"
+require_relative "rag/loader/csv"
+
+# Optional loaders (loaded only when their dependency gem is available)
+begin
+  require "nokogiri"
+  require_relative "rag/loader/html"
+rescue LoadError # rubocop:disable Lint/SuppressedException
+end
+
+begin
+  require "pdf-reader"
+  require_relative "rag/loader/pdf"
+rescue LoadError # rubocop:disable Lint/SuppressedException
+end
+
+# Text splitters
+require_relative "rag/text_splitter/base"
+require_relative "rag/text_splitter/recursive_character"
+require_relative "rag/text_splitter/markdown"
+
+# Vector stores
+require_relative "rag/vector_store/base"
+require_relative "rag/vector_store/in_memory"
+
+# Optional: PGVector
+begin
+  require "pgvector"
+  require_relative "rag/vector_store/pgvector"
+rescue LoadError # rubocop:disable Lint/SuppressedException
+end
+
+require_relative "rag/query"

data/lib/ask-rag.rb

@@ -0,0 +1,4 @@
+# frozen_string_literal: true
+
+require "ask"
+require_relative "ask/rag"

metadata

@@ -0,0 +1,148 @@
+--- !ruby/object:Gem::Specification
+name: ask-rag
+version: !ruby/object:Gem::Version
+  version: 0.1.0
+platform: ruby
+authors:
+- Kaka Ruto
+bindir: bin
+cert_chain: []
+date: 1980-01-02 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: ask-core
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0.6'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0.6'
+- !ruby/object:Gem::Dependency
+  name: csv
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: minitest
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '5.25'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '5.25'
+- !ruby/object:Gem::Dependency
+  name: mocha
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '3.1'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '3.1'
+- !ruby/object:Gem::Dependency
+  name: rake
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '13.0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '13.0'
+- !ruby/object:Gem::Dependency
+  name: reline
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+description: Document loaders, text splitters, vector stores, and retrieval queries
+  for building RAG applications with ask-rb.
+email:
+- kaka@myrrlabs.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- CHANGELOG.md
+- LICENSE
+- README.md
+- lib/ask-rag.rb
+- lib/ask/rag.rb
+- lib/ask/rag/document.rb
+- lib/ask/rag/loader/base.rb
+- lib/ask/rag/loader/csv.rb
+- lib/ask/rag/loader/html.rb
+- lib/ask/rag/loader/markdown.rb
+- lib/ask/rag/loader/pdf.rb
+- lib/ask/rag/loader/text.rb
+- lib/ask/rag/query.rb
+- lib/ask/rag/text_splitter/base.rb
+- lib/ask/rag/text_splitter/markdown.rb
+- lib/ask/rag/text_splitter/recursive_character.rb
+- lib/ask/rag/vector_store/base.rb
+- lib/ask/rag/vector_store/in_memory.rb
+- lib/ask/rag/vector_store/pgvector.rb
+- lib/ask/rag/version.rb
+homepage: https://github.com/ask-rb/ask-rag
+licenses:
+- MIT
+metadata:
+  homepage_uri: https://github.com/ask-rb/ask-rag
+  source_code_uri: https://github.com/ask-rb/ask-rag
+  changelog_uri: https://github.com/ask-rb/ask-rag/blob/master/CHANGELOG.md
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '3.2'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubygems_version: 4.0.3
+specification_version: 4
+summary: RAG pipeline for the ask-rb ecosystem
+test_files: []