ask-rag 0.2.2 → 0.4.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 85bc75fa9d57ddf8d6ca80d7baa63c283223f6a0fcccf3303d117857f27eb736
-  data.tar.gz: 8982a83aca2b04308bdf9f885457b5ec2944cd553b2ec4d184005705894aa300
+  metadata.gz: c1e704da2a70f456b0b4e155cc5530a6f5e701eec62831c6425187119ba04e73
+  data.tar.gz: 618753ff1f58592c41b7cf5a9f0369f12d6b460a4f6e865abdd77e4d197fbab1
 SHA512:
-  metadata.gz: b55d1f856729e6b3bccd35ec5589dc55afe289474ebebc333d1229cb7c40cc405194522770a13f2fb1db4edaca42c0f9b6344ec2936446d65ddd2aefb8233229
-  data.tar.gz: de95270aa7090a81fa0d5563d1dba63d57b9dd2b793a55442674ebdc953b951d95b6d33b558b15af137d0bb58f2f81bcf3f29a43e9616614b827fa5a62f2d3df
+  metadata.gz: 6554f213b35c30724385afcb8a95a266f834f141783cf2cd50cd333db6835e338b9f90dad9f1b5ce56e87c62804df3879ba175191319ba06be06c32531ea1295
+  data.tar.gz: e601fe5825391db348453562504be11f8ae42bd95c5a83a2fc166cee418b9e0140f8ef13a2dcd135376e0af9d3efb64cbb78724a1026ef8d284c866a086de486

data/CHANGELOG.md

@@ -1,3 +1,120 @@
+## [0.4.0] — 2026-09-11
+
+### Added
+
+- **Rails support, inside this gem.** There is no `ask-rag-rails` wrapper
+  and there will not be one: Rails integration ships here, the way Kamal
+  ships deployment tooling that happens to work well in Rails. Three
+  pieces, all optional, all inert outside Rails:
+  - `Ask::RAG.configure` — global defaults for the embedding model, the
+    chat model, and the PGVector table and columns, so an app configures
+    once instead of threading the same arguments through every call. Every
+    setting remains overridable per call.
+  - An in-gem `Railtie` exposing `config.ask_rag` (set it in
+    `config/application.rb` like anything else). It loads only where
+    `rails/railtie` exists; `railties` stays a development dependency,
+    never a runtime one.
+  - An `ask_rag:install` generator (`bin/rails generate ask_rag:install`)
+    that writes the pgvector migration and a commented initializer — the
+    `kamal init` equivalent.
+- **API keys resolve from Rails credentials.** Provider keys are read from
+  `Rails.application.credentials` first (`ask.openai_api_key`, falling
+  back to `llm.openai_api_key`), then the same environment variables
+  plain Ruby uses. Non-Rails apps skip the lookup entirely; nothing about
+  the ENV behaviour changed.
+- **`model:` is now optional everywhere.** `add`, `embed_texts`,
+  `Query.query`, and `Query.retrieve` fall back to the configured models,
+  which default to the previous hardcoded values — existing calls behave
+  exactly as before.
+- **`PGVector.new` takes no arguments.** Table, columns, and text search
+  configuration default to the configured values, which match the table
+  the install generator migrates (`ask_rag_embeddings`). The previous
+  required `table_name:` keyword now has a default, so existing calls
+  that pass it explicitly are unaffected.
+- **A helpful error when ActiveRecord is missing.** Building a `PGVector`
+  without the ORM raises `EmbeddingError` telling you which gem to add,
+  instead of a `NameError` deep in the stack. The `neighbor` and
+  `pgvector` client gems load lazily inside the constructor, so
+  InMemory-only apps never boot the database stack.
+- **The generated migration is tested for real.** The suite runs the
+  template against live Postgres and searches through the resulting
+  table, so the install path cannot rot.
+
+## [0.3.0] — 2026-09-11
+
+### Fixed
+
+- **`InMemory` ignored your query.** `embed_query` took no argument and
+  embedded the literal string `"query"`, so every search returned the
+  documents nearest that word — the same ranking for every question. This
+  was the same defect fixed in `PGVector` in 0.2.1; the fix had never
+  crossed over, and the suite stubbed `embed_query` in every search test, so
+  it stayed green. `similarity_search` now embeds the text the caller
+  passed. If you were relying on the old behaviour, nothing will look
+  familiar: results are now ranked by your query.
+- **`InMemory` results carried no id.** `add` returned generated UUIDs while
+  results carried `document.id` (usually `nil`), so the documented
+  retrieve-then-delete loop could not be closed. Results now carry the id
+  the document was stored under.
+- **`PGVector` ignored `mmr:`.** The argument was accepted and silently
+  dropped. It now applies Max Marginal Relevance, using the stored vectors
+  for the pairwise similarity.
+- **`PGVector` interpolated vectors into SQL.** The embedding is
+  API-provided and was concatenated into `select`, `order`, and `where`
+  clauses. It is now bound as a parameter with a `::vector` cast. The
+  full-text query likewise goes through `websearch_to_tsquery` as a bound
+  parameter.
+- **`PGVector#delete_by` deleted nothing.** It called `filter_scope` with
+  the filter where the relation belonged.
+- **`PGVector` returned string metadata keys.** Rows read back as
+  `metadata["source"]` where the loaders, `InMemory`, and the documented
+  examples all use `:source`. Metadata is symbolized on the way out so the
+  two stores are interchangeable.
+- Embedding failures now raise `EmbeddingError`. A provider returning an
+  `Ask::Result.failure` was coerced into floats, and a batch whose vector
+  count did not match its text count was silently truncated or misaligned.
+
+### Added
+
+- **`hybrid_search` on both stores.** Dense plus lexical retrieval, fused
+  by Reciprocal Rank Fusion. Dense retrieval alone cannot find an error
+  code, a method name, or an identifier that is out of the embedding
+  model's vocabulary; lexical alone cannot find a paraphrase. A document
+  both halves find outranks one only a single half found. `InMemory` uses
+  BM25; `PGVector` uses Postgres full-text search.
+- **Stopword handling in `InMemory`'s lexical half.** A question made only
+  of stopwords ("how do I do it") matched every document, and a real
+  question ("how do I deploy to kubernetes") matched unrelated pages, on the
+  strength of "how" and "to" alone. IDF does not suppress those at the
+  corpus sizes this gem serves. Terms are now dropped before scoring, and
+  tokenization splits identifiers the way Postgres does, so the two stores
+  agree on what matches.
+- **`min_score:`** on `similarity_search`, `similarity_search_by_vector`,
+  and `hybrid_search`. Everything below the threshold is dropped rather
+  than the nearest-but-wrong documents being returned to fill `limit`. This
+  is the difference between "no relevant context" and five confident
+  passages that do not answer the question.
+- **`delete_by(filter)`** on both stores, for clearing the chunks of a
+  source that has been edited or removed.
+- **Stable document ids** (`Ask::RAG::DocumentId`). Documents without an
+  id get one derived from source, chunk index, and content, so re-indexing
+  the same pipeline output updates in place instead of duplicating the
+  corpus and re-embedding it. `add` is now an upsert on both stores.
+- **`Query.retrieve`**, exposing retrieval without calling an LLM, plus
+  `hybrid:` and `min_score:` on `Query.query`.
+- **PGVector tests against a live Postgres**, including the SQL paths above.
+  They skip when no database is reachable. Set `ASK_RAG_TEST_DATABASE_URL`
+  (default `postgres://localhost/ask_rag_test`) to run them.
+- `Ask::RAG::Embeddings`, the shared embedding path both stores use, so a
+  store implements storage rather than an embeddings client.
+
+### Changed
+
+- `similarity_search_by_vector` accepts `mmr:` and `diversity_bonus:`,
+  matching `similarity_search`.
+- The `chunks.embedding` column's vector dimension is no longer assumed to
+  be 1536 in tests; the store reads whatever the column holds.
+
 ## [0.2.2] — 2026-08-03
 
 ### Fixed

data/README.md

@@ -20,7 +20,7 @@ splitter = Ask::RAG::TextSplitter::RecursiveCharacter.new(
 )
 chunks = splitter.split_documents(docs)
 
-# 2. Embed and store
+# 2. Embed and store (re-running this replaces, never duplicates)
 store = Ask::RAG::VectorStore::InMemory.new
 store.add(chunks, model: "text-embedding-3-small")
 
@@ -39,6 +39,124 @@ answer = Ask::RAG::Query.query(
 puts answer.content
 ```
 
+## Rails
+
+Rails support lives in this gem — there is no separate wrapper gem. Add
+`ask-rag` to your Gemfile and run the installer:
+
+```bash
+bin/rails generate ask_rag:install
+bin/rails db:migrate
+```
+
+The generator writes a migration for an `ask_rag_embeddings` table
+(content, metadata, embedding, plus the HNSW, full-text, and upsert
+indexes the stores assume) and a commented initializer. Two more lines
+in your Gemfile and credentials complete the setup:
+
+```ruby
+gem "ask-rag"
+gem "neighbor" # registers the vector column type with ActiveRecord
+```
+
+```yaml
+# config/credentials.yml.enc (bin/rails credentials:edit)
+ask:
+  openai_api_key: sk-...
+```
+
+That is the whole integration. `config.ask_rag` (or
+`Ask::RAG.configure`) sets the embedding model, the chat model, and the
+store's table and columns once; provider keys resolve from credentials
+first and `ENV` second; and `PGVector.new` with no arguments connects to
+the migrated table:
+
+```ruby
+store = Ask::RAG::VectorStore::PGVector.new
+store.add(chunks) # model: defaults to the configured embedding model
+
+answer = Ask::RAG::Query.query(store: store, question: "How do I reset my password?")
+# model: defaults to the configured chat model
+```
+
+Everything stays overridable per call, and everything outside Rails
+works exactly as before — the Railtie loads only where `rails/railtie`
+exists, and `railties` is a development dependency, never a runtime one.
+
+## Retrieval
+
+Two ways to search, and the difference matters:
+
+```ruby
+# Dense only: finds paraphrases, misses exact terms.
+store.similarity_search("ERR_4021", limit: 5)
+
+# Hybrid: dense + keyword, fused with Reciprocal Rank Fusion. Use this
+# for anything a person or an agent will read and act on.
+store.hybrid_search("ERR_4021", limit: 5)
+```
+
+Dense retrieval is bad at strings that the embedding model has no concept
+of: error codes, SKUs, method names, identifiers, rare proper nouns. A
+question containing `ERR_4021` will not retrieve the page that documents
+it, however good the model is. Lexical retrieval alone has the opposite
+problem — it cannot match "how do I reset my password" to "changing your
+credentials". `hybrid_search` runs both and merges the ranked lists, so a
+document **both** halves find outranks one only a single half found. That
+agreement is the strongest relevance signal available without a reranker.
+
+### Requiring a relevance floor
+
+By default you get `limit` results whether or not any of them are relevant.
+For a retrieval-augmented answer that is dangerous: five irrelevant chunks
+look exactly like five relevant ones to the model reading them.
+
+```ruby
+results = store.hybrid_search("what is the refund policy?", limit: 5, min_score: 0.35)
+# => [] when nothing clears the bar
+```
+
+An empty array is a legitimate answer — "this corpus does not cover that" —
+and is far more useful than the nearest five documents. For cosine
+similarity, 0.3–0.4 is a reasonable starting floor; there is no universal
+number, so measure it against your own corpus.
+
+### Diversifying results
+
+```ruby
+results = store.similarity_search(
+  "database configuration",
+  limit: 5,
+  mmr: true,
+  diversity_bonus: 0.5  # 0 = pure relevance, 1 = pure diversity
+)
+```
+
+Plain top-k returns five paraphrases of the same paragraph, because
+near-duplicate chunks all score highly. Max Marginal Relevance trades a
+little relevance for coverage, which matters most when retrieval works
+*well*. Results carry `:mmr_score` alongside `:score`.
+
+### Re-indexing
+
+`add` is an upsert. Documents without an id are given a stable one derived
+from their source, chunk index, and content, so:
+
+```ruby
+store.add(chunks, model: "text-embedding-3-small")  # first run
+store.add(chunks, model: "text-embedding-3-small")  # no-op, not a duplicate
+```
+
+When a source changes, its content hash changes and new ids appear. Clear
+the old rows first:
+
+```ruby
+store.delete_by(source: "docs/config.md")
+store.add(reloaded_chunks, model: "text-embedding-3-small")
+```
+
+Pass an explicit `id:` on the `Ask::Document` to control identity yourself.
+
 ## Components
 
 ### Loaders
@@ -50,6 +168,7 @@ Load files into `Ask::Document` objects (text + metadata).
 | `Ask::RAG::Loader::Text` | Plain text | None (stdlib) |
 | `Ask::RAG::Loader::Markdown` | Markdown | None (stdlib) |
 | `Ask::RAG::Loader::CSV` | CSV (1 doc/row) | None (`csv` gem) |
+| `Ask::RAG::Loader::JSON` | JSON (1 doc/object) | None (stdlib) |
 | `Ask::RAG::Loader::HTML` | HTML (strips script/style/nav) | `nokogiri` |
 | `Ask::RAG::Loader::PDF` | PDF (1 doc/page) | `pdf-reader` |
 | `Ask::RAG::Loader::Directory` | Auto-detects file types by extension | None (uses other loaders internally) |
@@ -94,14 +213,18 @@ Store embeddings and search by similarity.
 | `InMemory` | Pure Ruby cosine similarity. Zero deps. | None |
 | `PGVector` (auto-loaded) | PostgreSQL + pgvector extension | `pgvector`, `activerecord` |
 
+Both stores implement the same contract — including `hybrid_search`,
+`min_score`, MMR, and upsert semantics — so switching between them is a
+constructor change, not a rewrite.
+
 ```ruby
 # InMemory — no database needed
 store = Ask::RAG::VectorStore::InMemory.new
 store.add(documents, model: "text-embedding-3-small")
 results = store.similarity_search("query", limit: 5)
 
-# PGVector — for production Rails apps
-store = Ask::RAG::VectorStore::PGVector.new(table_name: :embeddings)
+# PGVector — for production Rails apps (see Rails above)
+store = Ask::RAG::VectorStore::PGVector.new
 store.add(documents, model: "text-embedding-3-small")
 ```
 
@@ -114,15 +237,44 @@ results.each do |doc|
 end
 ```
 
+#### PGVector schema
+
+```ruby
+create_table :ask_rag_embeddings do |t|
+  t.text :content
+  t.jsonb :metadata
+  t.vector :embedding, limit: 1536
+end
+add_index :ask_rag_embeddings, :embedding, using: :hnsw, opclass: :vector_cosine_ops
+
+# Makes hybrid_search's keyword half fast rather than a sequential scan:
+add_index :ask_rag_embeddings, "(to_tsvector('english', content))", using: :gin
+
+# Makes `add` an upsert at the database level too:
+add_index :ask_rag_embeddings, "(metadata ->> 'id')", unique: true
+```
+
+This is what `bin/rails generate ask_rag:install` writes for you.
+
+Document identity lives in `metadata ->> 'id'`, not in the table's primary
+key — the table is the store's, the document id is yours. Correctness does
+not depend on the indexes; speed does.
+
+The text search configuration defaults to `"english"` and is settable via
+`text_search_config:`.
+
 ### Metadata filtering
 
-Filter results by document metadata:
+Filter results by document metadata. The same `filter:` applies to
+`similarity_search`, `hybrid_search`, and `delete_by`:
 
 ```ruby
 results = store.similarity_search(
   "authentication",
   filter: { source: "api_docs.md", version: "2.0" }
 )
+
+store.delete_by(source: "api_docs.md")
 ```
 
 ### MMR (diversified search)
@@ -154,6 +306,23 @@ puts answer.content           # The LLM's answer
 puts answer.metadata[:sources]  # ["config.md", "settings.md"]
 ```
 
+`query` returns `nil` when nothing is retrieved, so an unanswerable
+question is distinguishable from a model failure. Add `hybrid: true` and
+`min_score:` to retrieve the way the section above recommends:
+
+```ruby
+answer = Ask::RAG::Query.query(
+  store: store,
+  question: "What is the default timeout configuration?",
+  model: "gpt-4o",
+  hybrid: true,
+  min_score: 0.35
+)
+```
+
+To inspect what would be retrieved without calling an LLM, use
+`Ask::RAG::Query.retrieve(store, question, hybrid: true, min_score: 0.35)`.
+
 ## Pipeline (End-to-End)
 
 ```ruby
@@ -175,7 +344,8 @@ store.add(chunks, model: "text-embedding-3-small")
 answer = Ask::RAG::Query.query(
   store: store,
   question: "How do I reset my password?",
-  model: "gpt-4o"
+  model: "gpt-4o",
+  hybrid: true
 )
 puts answer.content
 ```
@@ -199,8 +369,11 @@ class SearchDocs < Ask::Tool
   param :query, type: :string, desc: "Search query"
 
   def execute(query:)
-    store.similarity_search(query, limit: 3).map do |doc|
-      { content: doc.content[0..500], score: doc.metadata[:score] }
+    # An agent acts on what it retrieves, so give it the hybrid path and a
+    # floor: "nothing relevant" is a better answer than the nearest five
+    # documents, which the agent cannot tell apart from real matches.
+    store.hybrid_search(query, limit: 3, min_score: 0.35).map do |doc|
+      { content: doc.content[0..500], score: doc.metadata[:rrf_score] }
     end
   end
 end
@@ -218,6 +391,16 @@ session.run("How do I configure authentication?")
 bundle exec rake test
 ```
 
+The PGVector store is tested against a live Postgres; those tests skip when
+no database is reachable. To run them:
+
+```bash
+createdb ask_rag_test
+bundle exec rake test
+# or point elsewhere:
+ASK_RAG_TEST_DATABASE_URL=postgres://localhost/my_test_db bundle exec rake test
+```
+
 ## License
 
 MIT

data/lib/ask/rag/configuration.rb

@@ -0,0 +1,65 @@
+# frozen_string_literal: true
+
+module Ask
+  module RAG
+    # Global configuration for Ask::RAG.
+    #
+    # Plain Ruby objects read these as defaults, so a Rails app configures
+    # once — in an initializer, via `config.ask_rag`, or through credentials
+    # — instead of threading the same arguments through every call site.
+    # Every setting remains overridable per call.
+    #
+    # @example
+    #   Ask::RAG.configure do |config|
+    #     config.embedding_model = "text-embedding-3-small"
+    #     config.chat_model = "gpt-4o"
+    #     config.table_name = :embeddings
+    #   end
+    class Configuration
+      attr_accessor :embedding_model, :chat_model, :table_name, :content_column,
+        :metadata_column, :embedding_column, :text_search_config, :embedding_dimensions
+
+      def initialize
+        @embedding_model = "text-embedding-3-small"
+        @chat_model = nil
+        # Namespaced so the table never collides with the host app's own
+        # tables — in a Rails database this table lives alongside the
+        # app's, and a bare `embeddings` name is asking for a collision.
+        @table_name = :ask_rag_embeddings
+        @content_column = :content
+        @metadata_column = :metadata
+        @embedding_column = :embedding
+        @text_search_config = "english"
+        @embedding_dimensions = 1536
+      end
+
+      # Apply an ActiveSupport::OrderedOptions (or plain Hash), ignoring
+      # unknown keys so an app's `config.ask_rag` stays forward-compatible.
+      def apply_ordered_options(options)
+        options.each_pair do |key, value|
+          setter = :"#{key}="
+          public_send(setter, value) if respond_to?(setter)
+        end
+        self
+      end
+    end
+
+    class << self
+      # @return [Ask::RAG::Configuration] the global configuration
+      def configuration
+        @configuration ||= Configuration.new
+      end
+
+      # Configure Ask::RAG globally.
+      # @yield [Ask::RAG::Configuration]
+      def configure
+        yield configuration
+      end
+
+      # Reset the global configuration to its defaults. Mainly for tests.
+      def reset_configuration!
+        @configuration = Configuration.new
+      end
+    end
+  end
+end

data/lib/ask/rag/credentials.rb

@@ -0,0 +1,69 @@
+# frozen_string_literal: true
+
+module Ask
+  module RAG
+    # API key resolution that works the Rails way with zero configuration.
+    #
+    # Provider keys resolve from +Rails.application.credentials+ first (under
+    # +ask+ or +llm+), then fall back to the same environment variables plain
+    # Ruby already uses — +OPENAI_API_KEY+ namespaced first, then bare
+    # +API_KEY+. Non-Rails apps skip the credentials lookup entirely.
+    #
+    # @example credentials (config/credentials.yml.enc)
+    #   ask:
+    #     openai_api_key: sk-...
+    module Credentials
+      class << self
+        # @param slug [String] provider slug, e.g. "openai"
+        # @param option [Symbol, String] configuration key, e.g. :api_key
+        # @return [String, nil] the first value found
+        def fetch(slug, option)
+          rails_credential(slug, option) || env_fallback(slug, option)
+        end
+
+        # @return [true, false] whether the app boots with Rails credentials
+        def rails_credentials_available?
+          defined?(::Rails) && ::Rails.respond_to?(:application) &&
+            !::Rails.application.nil? && ::Rails.application.respond_to?(:credentials) &&
+            !::Rails.application.credentials.nil?
+        rescue StandardError
+          false
+        end
+
+        private
+
+        # Credentials live one nesting down under either +ask+ or +llm+:
+        # `ask.openai_api_key` reads as "the ask ecosystem's OpenAI key"
+        # without colliding with a top-level `openai_api_key` the app
+        # itself might define.
+        def rails_credential(slug, option)
+          return nil unless rails_credentials_available?
+
+          credentials = ::Rails.application.credentials
+          key = :"#{slug}_#{option}"
+          value = dig_credentials(credentials, :ask, key) ||
+            dig_credentials(credentials, :llm, key) ||
+            dig_credentials(credentials, key)
+          value&.to_s
+        rescue StandardError
+          nil
+        end
+
+        def dig_credentials(credentials, *keys)
+          keys.reduce(credentials) do |node, key|
+            break nil if node.nil?
+            if node.respond_to?(:dig)
+              node.dig(key)
+            elsif node.is_a?(Hash)
+              node[key] || node[key.to_s]
+            end
+          end
+        end
+
+        def env_fallback(slug, option)
+          ENV["#{slug.to_s.upcase}_#{option.to_s.upcase}"] || ENV[option.to_s.upcase]
+        end
+      end
+    end
+  end
+end

data/lib/ask/rag/document_id.rb

@@ -0,0 +1,54 @@
+# frozen_string_literal: true
+
+require "digest"
+
+module Ask
+  module RAG
+    # Stable identity for a document that arrives without one.
+    #
+    # Re-indexing is the common case, not the exception: a folder gets
+    # re-crawled, a job is retried, a developer runs the same script twice.
+    # If every `add` mints a fresh id, each of those runs duplicates the
+    # corpus and re-embeds it, and `delete` has nothing stable to target.
+    #
+    # So identity is derived from what a document *is* rather than when it
+    # was added. Chunks carry metadata[:source] and metadata[:chunk] from
+    # the loaders and splitters, and the content hash covers edits: an
+    # unchanged file reproduces its ids and `add` overwrites in place, while
+    # an edited file yields new ids and the stale chunks are what
+    # `delete_by(source: ...)` is for.
+    #
+    # Documents with no source fall back to content alone, which dedupes
+    # identical text. Pass an explicit +id+ to override all of this.
+    module DocumentId
+      LENGTH = 32
+
+      class << self
+        # @param document [Ask::Document]
+        # @return [String] the document's own id, or a stable derived one
+        def for(document)
+          document.id || derived(document)
+        end
+
+        # @param document [Ask::Document]
+        # @return [String] a content-addressed id
+        def derived(document)
+          Digest::SHA256.hexdigest(identity_parts(document).join("\u0000"))[0, LENGTH]
+        end
+
+        private
+
+        # Metadata survives JSON round-trips in the loaders, so a key can be
+        # a symbol or a string depending on which loader produced it.
+        def identity_parts(document)
+          metadata = document.metadata
+          [
+            metadata[:source] || metadata["source"],
+            metadata[:chunk] || metadata["chunk"],
+            document.content
+          ]
+        end
+      end
+    end
+  end
+end