ruby_llm-voyage 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 8ff013c0d7cc147977b07d6bd229fe1417390abf857b3252cb9721969609c1bf
+  data.tar.gz: 9e6075faf6732a6dcb9f68a03f25a7302435f4fa12b89a253999d09ec8e98549
+SHA512:
+  metadata.gz: ce3de774db04508201dcf029283c28d3c3c9dc1de6286cbf7dab209743f46692af300516439c23ba4bb49a9e654ee123479eeef29ff6ec7dfa35edaa57899f40
+  data.tar.gz: e0ba556b4ffa536441a2760c2190dec86ccc80894772702a5ca187f2db753241192e26a180ac9005530f59d1f0465e5b78f2596c98454cec991f944144673f6f

data/CHANGELOG.md

@@ -0,0 +1,45 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+
+## [0.1.0] - 2026-07-11
+
+### Added
+
+- Voyage text embedding provider for RubyLLM 1.16, covering single and batch
+  inputs, query and document input types, truncation control, flexible
+  dimensions, quantized output types, and Base64 decoding.
+- `RubyLLM::Voyage.embed_query` and `RubyLLM::Voyage.embed_documents` for the
+  two sides of retrieval, with `input_type` fixed per method.
+- `RubyLLM::Voyage.embed` facade for request-level Voyage options, falling
+  back to `config.default_embedding_model` when `model:` is omitted.
+- Reranking via `RubyLLM::Voyage.rerank`, with `Reranking#reorder` to map
+  results back onto the original candidate collection.
+- Contextualized chunk embeddings via
+  `RubyLLM::Voyage.contextualized_embed`, including server-side auto
+  chunking.
+- Files API support: upload, retrieve, list, download, and bulk delete, with
+  `find_file` and `download_file` aliases matching RubyLLM's provider file
+  naming. Downloads follow signed-URL redirects without forwarding the API
+  key and reject non-HTTPS redirects.
+- Public `VectorDecoder` for decoding Base64 embeddings out of batch result
+  files.
+- Configurable request defaults: `voyage_default_input_type`,
+  `voyage_default_truncation`, `voyage_default_output_dimension`,
+  `voyage_default_output_dtype`, and `voyage_default_encoding_format`.
+- Fail-fast `ArgumentError` for nil or empty inputs before any network
+  request.
+- Instrumentation events for embeddings, reranking, and contextualized
+  embeddings with result, response model, and token usage.
+- Offline unit and HTTP contract tests, opt-in live API tests, and CI
+  coverage for Ruby 3.1 through 4.0.
+- Trusted-publishing release workflow, SHA-pinned GitHub Actions, and
+  contributor, security, and release documentation.
+
+[Unreleased]: https://github.com/dinsley/ruby_llm-voyage/compare/v0.1.0...HEAD
+[0.1.0]: https://github.com/dinsley/ruby_llm-voyage/releases/tag/v0.1.0

data/CONTRIBUTING.md

@@ -0,0 +1,72 @@
+# Contributing
+
+Thanks for helping improve `ruby_llm-voyage`.
+
+## Development setup
+
+The project supports Ruby 3.1.3 through Ruby 4.0 and RubyLLM 1.x beginning
+with 1.16.
+
+```sh
+git clone https://github.com/dinsley/ruby_llm-voyage.git
+cd ruby_llm-voyage
+bin/setup
+bundle exec rake test
+bundle exec rubocop
+bundle exec rake yard
+bundle exec rake rdoc
+```
+
+`bin/console` starts an IRB session with the gem loaded.
+
+The default suite is network-isolated with WebMock. New behavior should have
+offline coverage and must not require a Voyage API key.
+
+## Live API tests
+
+Live tests are manual, billable, and never run in CI:
+
+```sh
+VOYAGE_API_KEY=your-key bundle exec rake test:live
+```
+
+The live suite covers single and batch embedding, context-based defaults,
+input types, retrieval ranking, every flexible dimension and output dtype,
+Base64 decoding, `provider_options` precedence, reranking, and contextualized
+embeddings. WebMock allows connections only to `api.voyageai.com`, so an
+accidental request to any other host still fails.
+
+The suite defaults to `voyage-4-lite`, which supports the complete
+dimension/dtype matrix. Environment variables adjust what runs:
+
+```sh
+# Use a different primary model (must support flexible dimensions and dtypes)
+VOYAGE_LIVE_MODEL=voyage-4 bundle exec rake test:live
+
+# Smoke-test additional model families with their defaults
+VOYAGE_LIVE_MODELS=voyage-4-large,voyage-code-3 bundle exec rake test:live
+
+# Include the mutating Files API lifecycle test (uploads and deletes a file)
+VOYAGE_LIVE_FILES=1 bundle exec rake test:live
+```
+
+Use short inputs and the default `voyage-4-lite` model unless a test
+specifically requires another model. Never commit credentials, captured
+authorization headers, or API responses containing private input.
+
+## Pull requests
+
+- Add or update tests for behavior changes.
+- Update `README.md` for user-facing changes.
+- Add an entry under `Unreleased` in `CHANGELOG.md`.
+- Run `bundle exec rake` (tests and RuboCop) and
+  `gem build ruby_llm-voyage.gemspec`.
+- Add YARD/RDoc comments for public API changes and verify both documentation
+  tasks complete without warnings.
+- Keep live tests separate from the default suite.
+
+## Compatibility changes
+
+Changes to the minimum Ruby or RubyLLM version are user-facing compatibility
+changes. Update the gemspec, CI matrix, the requirements note at the top of
+the README, and the changelog together.

data/LICENSE.txt

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Daniel Insley
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,389 @@
+# ruby_llm-voyage
+
+[![CI](https://github.com/dinsley/ruby_llm-voyage/actions/workflows/ci.yml/badge.svg)](https://github.com/dinsley/ruby_llm-voyage/actions/workflows/ci.yml)
+[![Gem Version](https://badge.fury.io/rb/ruby_llm-voyage.svg)](https://rubygems.org/gems/ruby_llm-voyage)
+
+[Voyage AI](https://docs.voyageai.com/) builds embedding and reranking models
+for search and retrieval. This gem adds them to
+[RubyLLM](https://rubyllm.com/): document and query embeddings, reranking,
+contextualized chunk embeddings, and the Files API.
+
+Requires Ruby 3.1.3+ and RubyLLM 1.16+.
+
+## Installation
+
+Add the gem to your Gemfile:
+
+```ruby
+gem 'ruby_llm-voyage'
+```
+
+Then configure your API key (created in the [Voyage dashboard](https://dashboard.voyageai.com/)):
+
+```ruby
+RubyLLM.configure do |config|
+  config.voyage_api_key = ENV.fetch('VOYAGE_API_KEY')
+end
+```
+
+## Your first embedding
+
+Voyage works through the standard RubyLLM embedding interface:
+
+```ruby
+embedding = RubyLLM.embed(
+  'Ruby is optimized for programmer happiness.',
+  model: 'voyage-4',
+  provider: :voyage
+)
+
+embedding.vectors      # => [0.018, -0.032, ...]
+embedding.input_tokens # => 9
+```
+
+Pass an array to embed a batch in one request, and `dimensions:` to control vector size:
+
+```ruby
+RubyLLM.embed(%w[Ruby Rails], model: 'voyage-4-lite', provider: :voyage, dimensions: 512)
+```
+
+Voyage tunes vectors differently for the text you store (`document`) and the text people search with (`query`), and using
+both sides improves retrieval quality. The gem gives each side its own method:
+
+```ruby
+RubyLLM::Voyage.embed_documents(['Reset your password from account settings.'])
+RubyLLM::Voyage.embed_query('how do I reset my password')
+```
+
+Both return the same `RubyLLM::Embedding` object and use `config.default_embedding_model` when `model:` is omitted.
+
+For the remaining options — truncation, quantized output, an explicit `nil` input type — use `RubyLLM::Voyage.embed`:
+
+```ruby
+embedding = RubyLLM::Voyage.embed(
+  'How do I reset my password?',
+  model: 'voyage-4-lite',
+  input_type: 'query',
+  truncation: false,
+  dimensions: 512
+)
+```
+
+## Semantic search in Rails
+
+Here's a complete search feature for a help center, using [Neighbor](https://github.com/ankane/neighbor) and PostgreSQL's pgvector
+extension.
+
+Set up the gems and configure the embedding defaults once:
+
+```ruby
+# Gemfile
+gem 'ruby_llm-voyage'
+gem 'neighbor'
+```
+
+```ruby
+# config/initializers/ruby_llm.rb
+RubyLLM.configure do |config|
+  config.voyage_api_key = Rails.credentials.voyage_api_key
+  config.default_embedding_model = 'voyage-4-lite'
+  config.voyage_default_output_dimension = 512
+end
+```
+
+Then the vector column:
+
+```sh
+bin/rails generate neighbor:vector
+bin/rails db:migrate
+```
+
+```ruby
+class AddEmbeddingToArticles < ActiveRecord::Migration[8.0]
+  def change
+    add_column :articles, :embedding, :vector, limit: 512
+
+    add_column :articles, :embedding_model, :string   # Identifies rows that need re-embedding when you upgrade models
+    add_column :articles, :embedding_digest, :string  # Digest of the embedded data to track changes to the field
+    add_column :articles, :embedded_at, :datetime     # Identifies when the data last had its embedding created
+
+    add_index :articles, :embedding, using: :hnsw, opclass: :vector_cosine_ops
+    add_index :articles, :embedding_model
+  end
+end
+```
+
+Embed in the background as articles are created or updated:
+
+```ruby
+class Article < ApplicationRecord
+  has_neighbors :embedding
+
+  after_save_commit -> { EmbedArticleJob.perform_later(self) }, if: :saved_change_to_content?
+
+  def refresh_embedding!
+    digest = Digest::SHA256.hexdigest(content)
+    return if embedding_digest == digest
+
+    result = RubyLLM::Voyage.embed_documents(content)
+
+    update_columns(
+      embedding: result.vectors,
+      embedding_model: result.model,
+      embedding_digest: digest,
+      embedded_at: Time.zone.current
+    )
+  end
+end
+
+class EmbedArticleJob < ApplicationJob
+  def perform(article) = article.refresh_embedding!
+end
+```
+
+Then search with a query embedding:
+
+```ruby
+class ArticlesController < ApplicationController
+  def search
+    query_vector = RubyLLM::Voyage.embed_query(params[:q]).vectors
+
+    @articles = Article.nearest_neighbors(
+      :embedding, query_vector, distance: 'cosine'
+    ).first(10)
+  end
+end
+```
+
+### Embedding an entire table
+
+The per-record job above handles ongoing writes, but backfilling existing data one row at a time wastes
+API calls. Embed whole tables in batches instead — each batch is one request, and Voyage accepts up to
+1,000 texts per request ([current limits](https://docs.voyageai.com/reference/embeddings-api)):
+
+```ruby
+class Article < ApplicationRecord
+  def self.embed_missing!(batch_size: 100)
+    where(embedding: nil).find_in_batches(batch_size: batch_size) do |batch|
+      result = RubyLLM::Voyage.embed_documents(batch.map(&:content))
+
+      batch.zip(result.vectors) do |(article, vector)|
+        article.update_columns(
+          embedding: vector,
+          embedding_model: result.model,
+          embedding_digest: Digest::SHA256.hexdigest(article.content),
+          embedded_at: Time.current
+        )
+      end
+    end
+  end
+end
+
+Article.embed_missing!
+```
+
+The provenance columns make model upgrades a repeat of the same procedure:
+clear the vectors built with the old model, backfill, and switch the query
+side once nothing is left to embed.
+
+```ruby
+Article.where.not(embedding_model: 'voyage-4').update_all(embedding: nil)
+Article.embed_missing!
+```
+
+## Better results with reranking
+
+Vector search is fast but approximate. When result order matters, fetch a generous candidate
+set with vector search, then let a [Voyage reranker](https://docs.voyageai.com/docs/reranker) put
+the best matches first:
+
+```ruby
+question = 'How do I configure SAML SSO?'
+query_vector = RubyLLM::Voyage.embed_query(question).vectors
+
+candidates = Article.nearest_neighbors(
+  :embedding, query_vector, distance: 'cosine'
+).first(25)
+
+reranking = RubyLLM::Voyage.rerank(
+  query: question,
+  documents: candidates.map(&:content),
+  model: 'rerank-2.5-lite',
+  top_k: 5
+)
+
+best_articles = reranking.reorder(candidates) # the 5 most relevant records
+```
+
+`reorder` maps the reranked positions back onto the collection you passed as
+`documents:`. Each result also carries the original `index`, a
+`relevance_score`, and (with `return_documents: true`) the document text.
+
+Rerankers score every candidate against the query, so rerank a small candidate set rather than the whole corpus.
+
+## Embedding long documents with context
+
+Chunking a long document strips each chunk of its surroundings: a handbook chunk that reads "this does not apply during the probation period" no longer
+says which policy it belongs to. Voyage's [contextualized models](https://docs.voyageai.com/docs/contextualized-chunk-embeddings) embed each chunk with
+information from the rest of its document.
+
+Pass pre-chunked documents as nested arrays:
+
+```ruby
+result = RubyLLM::Voyage.contextualized_embed(
+  [
+    ['Vacation policy overview', 'Accrual rates by tenure', 'Carryover rules'],
+    ['Remote work policy', 'Equipment stipend', 'Home office requirements']
+  ],
+  model: 'voyage-context-4',
+  input_type: :document,
+  dimensions: 512
+)
+
+result.results.each do |document|
+  document.chunks.each do |chunk|
+    # document.index, chunk.index, chunk.embedding
+  end
+end
+```
+
+Or let Voyage do the chunking server-side:
+
+```ruby
+result = RubyLLM::Voyage.contextualized_embed(
+  [handbook.full_text],
+  model: 'voyage-context-4',
+  input_type: :document,
+  dimensions: 512,
+  auto_chunk: true,
+  chunk_size: 512,
+  chunk_overlap: 0
+)
+
+result.results.first.chunks.each do |chunk|
+  HandbookChunk.create!(
+    position: chunk.index,
+    content: chunk.text,   # the exact text this vector represents
+    embedding: chunk.embedding
+  )
+end
+```
+
+With `auto_chunk`, store `chunk.text` alongside each vector, and query the
+index with the same contextualized model.
+
+## Giving a chatbot search
+
+A RubyLLM tool can embed the model's search query with Voyage and pull
+matching articles out of your database:
+
+```ruby
+class SearchKnowledgeBase < RubyLLM::Tool
+  desc 'Search the help center for articles relevant to a question'
+
+  param :query, desc: 'A concise search query'
+
+  def execute(query:)
+    query_vector = RubyLLM::Voyage.embed_query(query).vectors
+
+    Article.nearest_neighbors(:embedding, query_vector, distance: 'cosine')
+           .first(5)
+           .map { |article| { id: article.id, content: article.content } }
+           .to_json
+  end
+end
+
+chat = RubyLLM.chat(model: 'claude-sonnet-4-5')
+chat.with_instructions('Answer using the knowledge base. Cite article IDs.')
+chat.with_tool(SearchKnowledgeBase)
+
+chat.ask('How do I set up single sign-on?')
+```
+
+If articles are scoped per account, filter the `nearest_neighbors` query by the current user's permissions before returning results to the model.
+
+## Batch workflows with the Files API
+
+Voyage's asynchronous [batch workflows](https://docs.voyageai.com/docs/batch-inference) exchange JSONL files. This gem covers the full file lifecycle:
+
+```ruby
+uploaded = RubyLLM::Voyage.upload_file('requests.jsonl')
+
+RubyLLM::Voyage.retrieve_file(uploaded.id)   # metadata
+RubyLLM::Voyage.list_files(purpose: 'batch') # cursor-paginated listing
+RubyLLM::Voyage.file_content(uploaded.id)    # raw JSONL back out
+
+RubyLLM::Voyage.delete_files([uploaded.id])
+```
+
+`find_file` and `download_file` are aliases of `retrieve_file` and `file_content`, matching RubyLLM's own file-method naming.
+
+Downloads follow Voyage's redirect to a signed storage URL without forwarding your API key, and refuse non-HTTPS redirects.
+
+## Configuration reference
+
+Every Voyage option can be set globally, per `RubyLLM.context`, or per
+request:
+
+```ruby
+RubyLLM.configure do |config|
+  config.voyage_api_key = ENV.fetch('VOYAGE_API_KEY')
+  config.default_embedding_model = 'voyage-4-lite'  # when model: is omitted
+  config.voyage_default_output_dimension = 512      # when dimensions: is omitted
+  config.voyage_default_input_type = 'document'
+  config.voyage_default_truncation = false          # error instead of truncating
+  config.voyage_default_output_dtype = 'float'
+  config.voyage_default_encoding_format = 'base64'
+  config.voyage_api_base = 'https://api.voyageai.com/v1'
+end
+```
+
+Request-level options on `RubyLLM::Voyage.embed` (and on `embed_query` /
+`embed_documents`, which fix `input_type` for you):
+
+| Option | Voyage field | Values |
+| --- | --- | --- |
+| `model:` | `model` | Any Voyage embedding model ID |
+| `input_type:` | `input_type` | `query`, `document`, or `nil` |
+| `truncation:` | `truncation` | `true` or `false` |
+| `dimensions:` | `output_dimension` | Model-dependent |
+| `output_dtype:` | `output_dtype` | `float`, `int8`, `uint8`, `binary`, `ubinary` |
+| `encoding_format:` | `encoding_format` | `base64` or `nil` |
+| `provider_options:` | merged into request | Fields the gem doesn't have a keyword for yet |
+
+## Not supported
+
+- **Multimodal embeddings** (`voyage-multimodal-3.5`) — text inputs only.
+- **Batch jobs** — the Files API is covered, but creating and managing
+  asynchronous batch jobs is not.
+- **Token counting** — Voyage's official SDKs tokenize locally; this gem
+  reports `input_tokens` from API responses instead.
+- **Request splitting** — one call sends one API request. Collections over
+  Voyage's 1,000-text limit must be batched by the caller, as in the
+  backfill example above.
+
+New Voyage request fields usually work before the gem names them: pass them through `provider_options:`. For anything else, open an [issue](https://github.com/dinsley/ruby_llm-voyage/issues).
+
+## Development
+
+```sh
+bin/setup
+bundle exec rake test     # offline suite, no API key needed
+bundle exec rubocop
+```
+
+An opt-in live suite runs against the real Voyage API:
+
+```sh
+VOYAGE_API_KEY=your-key bundle exec rake test:live
+```
+
+See [CONTRIBUTING.md](https://github.com/dinsley/ruby_llm-voyage/blob/main/CONTRIBUTING.md)
+for development guidelines and live test options, and
+[docs/RELEASING.md](https://github.com/dinsley/ruby_llm-voyage/blob/main/docs/RELEASING.md)
+for the release process.
+
+## License
+
+The gem is available as open source under the terms of the
+[MIT License](https://github.com/dinsley/ruby_llm-voyage/blob/main/LICENSE.txt).

data/SECURITY.md

@@ -0,0 +1,31 @@
+# Security Policy
+
+## Supported versions
+
+Security fixes are provided for the latest released version of
+`ruby_llm-voyage`.
+
+## Reporting a vulnerability
+
+Please do not open a public issue for a suspected vulnerability. Use
+[GitHub private vulnerability reporting](https://github.com/dinsley/ruby_llm-voyage/security/advisories/new)
+to report it privately.
+
+Include:
+
+- The affected version
+- Reproduction steps or a minimal proof of concept
+- The potential impact
+- Any suggested mitigation
+
+Do not include real Voyage API keys, customer embedding inputs, or production
+responses. You should receive an acknowledgement within seven days.
+
+## Credential and data handling
+
+- Store `VOYAGE_API_KEY` in a secret manager or environment variable.
+- Never expose the API key to browsers or other untrusted clients.
+- Only configure `voyage_api_base` from trusted application configuration.
+- Treat embedding inputs and vectors as potentially sensitive data.
+- Review RubyLLM instrumentation exporters before sending event payloads to a
+  third party; events can contain embedding text or reranking queries.

data/docs/RELEASING.md

@@ -0,0 +1,91 @@
+# Releasing
+
+Releases are published by the `Release` GitHub Actions workflow using
+[RubyGems trusted publishing](https://guides.rubygems.org/trusted-publishing/).
+No RubyGems API key exists anywhere; the workflow authenticates with an OIDC
+token scoped to this repository, the `release.yml` workflow file, and the
+`release` environment.
+
+## One-time setup
+
+Already done for this repository, listed for reference:
+
+- A trusted publisher on rubygems.org for `ruby_llm-voyage` pointing at
+  `dinsley/ruby_llm-voyage`, workflow `release.yml`, environment `release`.
+  (The first release used a pending publisher, which expires after about
+  12 hours if unused — re-register if it lapses before the first tag.)
+- Private vulnerability reporting and Dependabot security updates enabled.
+- Default branch protected, requiring the `CI` workflow to pass.
+
+Optional hardening: add required reviewers to the `release` GitHub
+environment so tag pushes pause for approval before publishing.
+
+## Prepare the release
+
+1. Confirm CI passes on every supported Ruby version.
+2. Run the offline release checks:
+
+   ```sh
+   bundle exec rake
+   bundle exec rake yard
+   bundle exec rake rdoc
+   gem build ruby_llm-voyage.gemspec
+   gem specification ruby_llm-voyage-*.gem
+   ```
+
+3. Optionally run the billable live suite:
+
+   ```sh
+   VOYAGE_API_KEY=your-key bundle exec rake test:live
+   ```
+
+4. Update `RubyLLM::Voyage::VERSION` in
+   `lib/ruby_llm/voyage/version.rb`.
+5. Move the `CHANGELOG.md` entries under `Unreleased` into a dated version
+   section and update its comparison links.
+6. Commit the release changes and open a pull request.
+
+## Publish
+
+After the release commit is merged:
+
+```sh
+git switch main
+git pull --ff-only
+git tag -s vVERSION -m "ruby_llm-voyage VERSION"
+git push origin vVERSION
+```
+
+The tag push triggers the `Release` workflow, which runs the test suite and
+publishes the gem to RubyGems with provenance attestations. Watch it at:
+
+```sh
+gh run watch
+```
+
+Then create a GitHub release from the tag using the matching changelog
+section as the release notes:
+
+```sh
+gh release create vVERSION --title "ruby_llm-voyage VERSION" --notes-file <(changelog section)
+```
+
+## Verify
+
+```sh
+gem install ruby_llm-voyage -v VERSION
+ruby -e 'require "ruby_llm-voyage"; puts RubyLLM::Voyage::VERSION'
+```
+
+Confirm that the RubyGems page links to the expected source, changelog,
+documentation, and issue tracker. Do not reuse or move a published version
+tag; release a new patch version for corrections.
+
+## Manual fallback
+
+If the workflow is unavailable, a gem owner with MFA can publish manually:
+
+```sh
+bundle exec rake build
+gem push pkg/ruby_llm-voyage-VERSION.gem
+```