ruby_llm-infinite_context 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: f3b5d8dd372848f3ae7d47169dc6b2aae612c7ea8642ce02e77dbe3e885f82e7
+  data.tar.gz: 77b06484bf8102c34a1180429dbe6edbf80fa2e9d43bc2567d406a1f9b1015da
+SHA512:
+  metadata.gz: afc5f0a653883822ba4b47c57eec3e96e2db5ce7c3ac8bd7d95baf542341b7d555177ad1442f4b3ae69fd3bd877549850dce6d371aa1e406b3926b1ff086832c
+  data.tar.gz: 4b5ae78486a80e821b2dbb43968155f338623b4af356621caacd28a81ebb6ffa3cdd5c48f63a53da7cd0ba3fdf185616ece50bbf2c25f7b0cd54a80be66fba44

data/CHANGELOG.md

@@ -0,0 +1,51 @@
+# Changelog
+
+All notable changes to this gem. The format follows
+[Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and the gem follows
+[Semantic Versioning](https://semver.org/).
+
+## [0.1.0] - not yet released
+
+First version. The gem is not on rubygems.org yet.
+
+### Added
+
+- The log, the summary tree and the saved views: `ric_entries`, `ric_nodes`, `ric_views`,
+  `ric_zoom_events` and `ric_trees` (json columns and row locks, so SQLite and Postgres both
+  work). Install with `rails g ruby_llm:infinite_context:install`.
+- The compactor, with up to 5 tries to fit a node under 512 bytes, and the pump that builds
+  the tree in parallel (`Pump::Runner`, `PumpJob` for ActiveJob).
+- Saved views that fit a byte budget the caller picks, with the sawtooth merge. Budgets are
+  bytes everywhere; `tokens:` and `20_000.tokens` (after
+  `using RubyLLM::InfiniteContext::Tokens`) convert once at the API edge at 4 bytes per token.
+- The `zoom`, `read` and `date` tools, with a repeat-zoom guard and a per-turn cap.
+- Counters: `dig_count`, `last_dug_at` and the zoom-event table. No rewrites yet.
+- Tool call and result are summarized as a pair.
+- Prompt-cache marks for providers with explicit breakpoints (Anthropic).
+- `RubyLLM::InfiniteContext.attach` (also `Manager.attach`) and the `Model` concern with
+  `has_infinite_context`.
+- `provider:`, `protocol:`, `assume_model_exists:` and `context:` pass-through for the chats the
+  gem builds (`chat_options:`, `compactor_options:`, `compactor_chat_options`). RubyLLM 2.1.1
+  sends an OpenAI-provider chat to the Responses API by default; Fireworks and other
+  OpenAI-compatible services need `protocol: :chat_completions`. See the README.
+- `BatchImport` for compressing an existing long thread at batch prices with `RubyLLM::Batch`.
+- A public, synthetic benchmark in `bench/` (not part of the gem package), and a spend guard
+  for paid benchmark rounds. Paid runs are disabled in this release; the guard has only been
+  tested over a fake network.
+
+### Differences to know about
+
+- `has_infinite_context` is a plain concern: the model needs `acts_as_chat` as well. The record
+  names the model, and each turn builds a fresh chat from it. The record's own message history
+  is not sent; the tree is the memory.
+- The `20_000.tokens` literal needs `using RubyLLM::InfiniteContext::Tokens`. The `tokens:`
+  keyword does not.
+- The agent tools are `zoom`, `read` and `date`.
+- The action on repeated digs ships as `:none` only. The counters are there so a later release
+  can decide from data.
+
+### Known limits
+
+- Token budgets use a rough 4 bytes per token. For exact budgets, pass bytes.
+- Nothing in this release has been run against a live model provider by its authors' test
+  suite; the suite uses a fake network.

data/LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2026 Christian Genco
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,198 @@
+# ruby_llm-infinite_context
+
+Endless memory for [RubyLLM](https://github.com/crmne/ruby_llm) 2.x chats and agents.
+
+Every message is logged. A cheap model compresses the log into a binary tree of one-line
+summaries. The chat sees a saved "view" of the tree that fits a budget you pick (for example
+20,000 tokens): recent messages one per line, older ones in progressively coarser lines. The
+agent gets tools to zoom into any line, down to the original message.
+
+## Status
+
+0.1.0 is the first version and has not been released yet. It has been tested end to end on three
+real models: DeepSeek V4.1 Flash (11 of 12 questions right, 33 zooms), Claude Haiku 4.5
+(10 of 12 right, 31 zooms) and GLM-5.3 Flash (10 of 12 right, 26 zooms), each as both
+compactor and reader, on a small synthetic log (14 entries, 12 questions). That is a smoke
+test that the whole path works, not a benchmark.
+Benchmark results (quality and cost across models and view budgets) come later; the benchmark
+itself is in `bench/`.
+
+## Credits
+
+This gem is based on **OptChat / UniiChat**, Victor Taelin's design for one endless chat whose
+log is summarized into a binary tree:
+
+- OptChat/UniiChat gist: https://gist.github.com/VictorTaelin/91837951a5ce5b38f341ec1ba1df6449
+- [pi-optchat](https://github.com/jonaslsaa/pi-optchat) (Jonas, MIT), a Pi extension that
+  implements the gist, and its npm package
+  [pi-optchat](https://www.npmjs.com/package/pi-optchat).
+
+The design ideas come from the gist and pi-optchat; the prompts in this gem are our own words.
+See `THIRD_PARTY_NOTICES.md`.
+
+## Install
+
+```ruby
+gem "ruby_llm-infinite_context"
+```
+
+Then create the tables:
+
+```
+rails g ruby_llm:infinite_context:install
+rails db:migrate
+```
+
+## Use
+
+### With a chat
+
+```ruby
+chat = RubyLLM.chat(model: "claude-sonnet-5-5")
+memory = RubyLLM::InfiniteContext.attach(chat, key: "conv-1",
+                                         compactor: "claude-haiku-4-5",
+                                         tokens: 20_000)
+memory.turn("...")
+memory.tree.view_lines(budget: 80_000)   # the saved view, "id+n|text" lines
+memory.tree.stats.hot_nodes(limit: 10)   # the most opened lines
+```
+
+One `Chat` per turn; the tree is the only memory between turns. Each turn logs the user's
+message and the reply, shows the view as instructions, hands the agent `zoom`, `read` and
+`date` tools, and then pumps the compactor for nodes that are not built yet.
+
+### Fireworks and other OpenAI-compatible services
+
+**RubyLLM 2.1.1 sends an OpenAI-provider chat to the Responses API by default**
+(`POST {base}/responses`, with `store: false`). Many OpenAI-compatible services only offer
+chat completions (`POST {base}/chat/completions`). Fireworks offers both, but its Responses
+API has storing features, so use chat completions there. For any of these, build the chat
+with `protocol: :chat_completions`, for both your chat and the compactor:
+
+```ruby
+fireworks = RubyLLM.context do |config|
+  config.openai_api_key = Rails.application.credentials.fireworks_api_key
+  config.openai_api_base = "https://api.fireworks.ai/inference/v1"
+end
+
+wire = { provider: :openai, protocol: :chat_completions,
+         assume_model_exists: true, context: fireworks }
+
+memory = RubyLLM::InfiniteContext.attach(
+  "accounts/fireworks/models/deepseek-v4p1-flash",    # a model NAME: the gem builds the chat
+  key: "conv-1",
+  compactor: "accounts/fireworks/models/glm-5p3-flash",
+  chat_options: wire,        # how the gem builds your chat from the name
+  compactor_options: wire    # how it builds the compactor's chats
+)
+```
+
+The options are the ones `RubyLLM.chat` takes to pick the wire: `provider:`, `protocol:`,
+`assume_model_exists:` and `context:`. An unknown key raises, so a typo such as `protcol:`
+cannot quietly pick the wrong API. The same `chat_options:` is accepted by
+`Compactor.new`, `BatchImport.new` and `Pump::Runner.new`. If you pass a ready `Chat` (or a
+lambda that returns one), you build it yourself:
+`RubyLLM.chat(model: ..., provider: :openai, protocol: :chat_completions, ...)`; only the
+compactor then needs `compactor_options:`.
+
+The pump that runs inside an ActiveJob (`pump: :job`, `PumpJob`) cannot carry per-call
+options, so set the compactor's default once at boot:
+
+```ruby
+RubyLLM::InfiniteContext.compactor_chat_options = wire
+```
+
+For a Rails model, see below: set `protocol`, `provider` and `context` on the record, and
+pass `compactor_options:` to `has_infinite_context`.
+
+### Tokens or bytes
+
+Every budget inside the gem is UTF-8 bytes. You can ask in tokens: pass `tokens:`
+(`tokens: 20_000`), or write `budget: 20_000.tokens` after
+`using RubyLLM::InfiniteContext::Tokens`. Both convert once at the API edge with a rough
+estimator of 4 bytes per token, which is right for English prose; text with code or
+non-ASCII runs heavier. For exact budgets, pass `budget:` in bytes.
+
+### With a Rails model
+
+```ruby
+class Conversation < ApplicationRecord
+  acts_as_chat
+  include RubyLLM::InfiniteContext::Model
+  has_infinite_context compactor: "claude-haiku-4-5"
+end
+
+memory = conversation.with_infinite_context(tokens: 20_000)
+memory.turn("...")
+```
+
+The record names the model. Each turn builds a fresh RubyLLM chat from the record's
+`model_id`, `provider`, `protocol`, `assume_model_exists` and `context` (the attributes
+`acts_as_chat` gives it). The record's own message history is not sent: the tree is the
+memory between turns, so the turn does not grow with the thread. For Fireworks, set
+`conversation.protocol = :chat_completions` (the attributes `protocol`, `assume_model_exists`
+and `context` are not saved; set them after loading the record) and use
+`has_infinite_context compactor: "...", compactor_options: wire`.
+
+### Bulk import of an existing thread
+
+```ruby
+importer = RubyLLM::InfiniteContext::BatchImport.new(tree)
+submission = importer.submit(messages.map { |m| { role: m.role, content: m.content } })
+# persist submission.id; later:
+batch = RubyLLM::Batch.find(submission.id, provider: submission.provider)
+importer.collect!(submission, batch) if batch.refresh.complete?
+```
+
+Compresses an existing long thread at batch prices with `RubyLLM::Batch` (`ask_later`).
+Call `submit` again after `collect!` when the thread is longer than one batch covers:
+later levels become ready as their children finish.
+
+## How it works
+
+- **Log** (`ric_entries`): append-only, never updated or deleted.
+- **Tree** (`ric_nodes`): node `(level, i)` covers entries `[i·2^level, (i+1)·2^level)`.
+  A source that fits in 512 bytes is its own node with no model call; otherwise the compactor
+  model compresses or merges it, with the view as context, up to 5 tries to fit under 512
+  bytes (the shortest try wins). Built once, never recomputed.
+- **Views** (`ric_views`): one saved view per budget. The view only appends at the end and
+  merges the most due adjacent pair whose parent is built; it never splits, so older stretches
+  sit in coarser and coarser lines (the sawtooth). A saved view is never rebuilt from the log.
+  The compactor works against its own 24 KB view, separate from whatever the consumer sees.
+- **Zoom** (`ric_zoom_events`): `zoom(id, n)` opens a line into the two lines it was built
+  from; `zoom(id, 1)` shows the message, paged when long (`read(id, page)`). Opening a line
+  the agent already opened this turn returns "you already opened this line; its text is
+  above" instead of the text again, and a per-turn cap (12 by default) bounds runaway zooming.
+- **Counters, not rewrites**: nodes count `dig_count` and `last_dug_at`; zoom events record
+  the leaf a dig reached, the steps it cost, and the turn. No parent rewrite, no fold boost:
+  the shipped action on repeated digs is `:none` (do nothing) until data supports one.
+- **Tool pairs**: a tool result is summarized with its call as context, so the pair survives
+  as one fact.
+- **Prompt caching** (`CacheMark`): for providers that accept explicit cache breakpoints
+  (Anthropic), the view is sent as fixed blocks of 8 parts with one mark on the part the next
+  messages will not change. Other providers rely on implicit prefix caching.
+
+## SQLite and Postgres
+
+The tables use `json` columns and row locks, not `jsonb` or advisory locks, so both work.
+
+## Benchmarks
+
+`bench/` holds a public, synthetic benchmark: a generator that writes agent sessions with
+known gold facts, and a runner that scores compactor quality (fit rate, ruler-copy rate,
+retention up the tree) and end-to-end accuracy at several view budgets against cut-off and
+rolling-compaction baselines, with cost in USD. `bench/dry_run.rb` exercises the whole
+machinery with no API calls. No client data is used anywhere in the corpus. See
+`bench/README.md` to reproduce.
+
+## Development
+
+```
+bin/setup
+rake test
+rake standard
+```
+
+## License
+
+MIT. See `THIRD_PARTY_NOTICES.md` for third-party credits.

data/THIRD_PARTY_NOTICES.md

@@ -0,0 +1,52 @@
+# Third-party notices
+
+This gem includes no third-party code, but its design is derived from these projects:
+
+## OptChat / UniiChat (Victor Taelin)
+
+The core design — an append-only message log summarized into a binary tree of one-line
+summaries, a byte-budget view over that tree, and tools to zoom from a summary line down to
+the original messages — comes from Victor Taelin's "OptChat" gist (titled "UniiChat").
+
+- https://gist.github.com/VictorTaelin/91837951a5ce5b38f341ec1ba1df6449
+
+The prompts in this gem are original text written for this gem; they implement the same ideas
+(message kinds, the value order of what to keep, the byte limit, tagging items with their
+kind) but are not copied from the gist.
+
+## pi-optchat (Jonas) — MIT
+
+An independent implementation of the same gist as a Pi extension, used as the up-to-date
+reference for behavior. Its third-party notices file is the model for this file.
+
+- https://github.com/jonaslsaa/pi-optchat
+- https://www.npmjs.com/package/pi-optchat
+
+Its license text:
+
+```
+MIT License
+
+Copyright (c) 2026 Jonas Silva
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.
+```
+
+(License text from pi-optchat's LICENSE file, read from the published npm tarball on
+2026-11-09.)

data/lib/generators/ruby_llm/infinite_context/install/install_generator.rb

@@ -0,0 +1,34 @@
+# frozen_string_literal: true
+
+require "rails/generators"
+require "rails/generators/migration"
+require "rails/generators/active_record"
+
+module RubyLLM
+  module InfiniteContext
+    module Generators
+      # `rails g ruby_llm:infinite_context:install` copies the migration
+      # that creates the ric_ tables.
+      class InstallGenerator < ::Rails::Generators::Base
+        include ::Rails::Generators::Migration
+
+        # Set by hand: Rails would derive "ruby_l_l_m:..." from the class
+        # name (RubyLLM), and `rails g ruby_llm:infinite_context:install`
+        # would not be found.
+        namespace "ruby_llm:infinite_context:install"
+
+        source_root File.expand_path("templates", __dir__)
+        desc "Creates the ric_ tables for ruby_llm-infinite_context"
+
+        # Rails::Generators::Migration leaves this to the including class.
+        def self.next_migration_number(dir)
+          ::ActiveRecord::Generators::Base.next_migration_number(dir)
+        end
+
+        def copy_migration
+          migration_template "migration.rb.tt", "db/migrate/create_ruby_llm_infinite_context_tables.rb"
+        end
+      end
+    end
+  end
+end

data/lib/generators/ruby_llm/infinite_context/install/templates/migration.rb.tt

@@ -0,0 +1,7 @@
+# frozen_string_literal: true
+
+class CreateRubyLlmInfiniteContextTables < ActiveRecord::Migration[<%= ActiveRecord::Migration.current_version %>]
+  def change
+    RubyLLM::InfiniteContext::Schema.migration(self)
+  end
+end

data/lib/ruby_llm/infinite_context/batch_import.rb

@@ -0,0 +1,163 @@
+# frozen_string_literal: true
+
+module RubyLLM
+  module InfiniteContext
+    # Bulk import: compress an existing long thread with RubyLLM::Batch.
+    # Appends the thread to the log, then submits one batched request per
+    # node that needs a model call (long entries and merges).
+    #
+    #   messages = existing_thread.map { |m| { role: m.role, content: m.content } }
+    #   submission = RubyLLM::InfiniteContext::BatchImport.new(tree).submit(messages)
+    #   submission.id    # persist this
+    #   ...
+    #   batch = RubyLLM::Batch.find(submission.id, provider: submission.provider)
+    #   importer.collect!(submission, batch) if batch.refresh.complete?
+    class BatchImport
+      include TreeMath
+
+      # nodes: the [level, i] pairs in the batch; id: the provider batch id;
+      # provider: the provider name, for RubyLLM::Batch.find.
+      Submission = Data.define(:nodes, :id, :provider)
+
+      # chat_options: see Compactor (provider:, protocol:, ...).
+      def initialize(tree, model: nil, chat_factory: nil, batch_class: nil, chat_options: nil)
+        @tree = tree
+        @model = model || tree.compactor_model
+        @chat_options = chat_options && ChatOptions.normalize(chat_options)
+        @chat_factory = chat_factory
+        @batch_class = batch_class || (RubyLLM::Batch if defined?(RubyLLM::Batch))
+      end
+
+      # Appends `messages` (role/content pairs) to the log and submits the
+      # ready nodes as one batch. Returns nil when nothing needs a model
+      # call. When RubyLLM::Batch is unavailable, use Pump::Runner instead.
+      def submit(messages)
+        import_messages(messages)
+        chats = []
+        nodes = []
+        batchable_nodes.each do |level, i|
+          prompt = prompt_for(level, i)
+          next unless prompt
+
+          chat = new_chat
+          chat.ask_later(prompt)
+          chats << chat
+          nodes << [level, i]
+        end
+        return nil if chats.empty?
+
+        raise Error, "RubyLLM::Batch is not available" unless @batch_class
+
+        batch = @batch_class.submit(chats)
+        Submission.new(nodes:, id: batch.id, provider: provider_name(batch))
+      end
+
+      # Stores the batch's answers on their nodes. Validates the whole batch
+      # FIRST: a missing, empty or short result fails the whole collect and
+      # stores nothing, so a partial batch never half-populates the tree.
+      def collect!(submission, batch)
+        raise Error, "batch is not complete (#{batch.status})" unless batch.complete?
+
+        messages = batch.messages
+        problems = []
+        if messages.size != submission.nodes.size
+          problems << "expected #{submission.nodes.size} results, got #{messages.size}"
+        end
+        messages.zip(submission.nodes).each_with_index do |(message, node), i|
+          text = message&.content.to_s.strip
+          problems << "result #{i} for node #{TreeMath.label(*node)} is missing or empty" if text.empty?
+        end
+        return 0 if submission.nodes.empty?
+        raise Error, "batch results incomplete, nothing stored: #{problems.join("; ")}" unless problems.empty?
+
+        stored = 0
+        messages.zip(submission.nodes).each do |message, node|
+          @tree.store_node!(node[0], node[1], message.content.to_s.strip, tries: 1)
+          stored += 1
+        end
+        stored
+      end
+
+      private
+
+      def import_messages(messages)
+        messages.each do |m|
+          role = (m[:role] || m["role"] || :user).to_s
+          kind = if role == "assistant"
+            :assistant
+          else
+            (if role == "tool"
+               :echo
+             else
+               ((role == "tool_call") ? :tool : role)
+             end)
+          end
+          kind = :user unless KINDS.include?(kind.to_s)
+          @tree.append(kind, (m[:content] || m["content"]).to_s)
+        end
+      end
+
+      def total = @tree.entry_count
+
+      # The nodes that can be batched now: long level-0 entries, then merges
+      # whose children are built. Unlike the pump's rule, a level-0 node does
+      # not wait for the lines before it: in a batch its context is
+      # best-effort. Call `submit` again after `collect!` for the next wave.
+      def batchable_nodes
+        total = @tree.entry_count
+        sizes = @tree.node_sizes
+        out = []
+        total.times do |i|
+          out << [0, i] if !sizes.key?([0, i]) && TreeMath.bytes(@tree.entry(i).text) > NODE
+        end
+        l = 1
+        while (1 << l) <= total
+          (0...(total >> l)).each do |i|
+            next if sizes.key?([l, i])
+            next unless sizes.key?([l - 1, 2 * i]) && sizes.key?([l - 1, 2 * i + 1])
+
+            joined = "#{@tree.node_text(l - 1, 2 * i)}\n#{@tree.node_text(l - 1, 2 * i + 1)}"
+            out << [l, i] if TreeMath.bytes(joined) > NODE
+          end
+          l += 1
+        end
+        out
+      end
+
+      # The batched prompt for one node, or nil when no model call is needed
+      # (a short entry is its own node; a joined pair under NODE is a node).
+      def prompt_for(level, i)
+        if level.zero?
+          entry = @tree.entry(i)
+          return nil if TreeMath.bytes(entry.text) <= NODE
+
+          Compactor.compress_prompt(entry.kind, entry.text, call: @tree.paired_call(i)&.text)
+        else
+          a = @tree.node_text(level - 1, 2 * i)
+          b = @tree.node_text(level - 1, 2 * i + 1)
+          return nil unless a && b
+
+          joined = "#{a}\n#{b}"
+          return nil if TreeMath.bytes(joined) <= NODE
+
+          Compactor.merge_prompt(a, b)
+        end
+      end
+
+      def new_chat
+        chat = if @chat_factory
+          @chat_factory.call
+        else
+          ChatOptions.build(@model, @chat_options || InfiniteContext.compactor_chat_options)
+        end
+        chat.with_instructions(Prompts::COMPACT_SYSTEM)
+      end
+
+      def provider_name(batch)
+        batch.provider&.respond_to?(:slug) ? batch.provider.slug : nil
+      rescue
+        nil
+      end
+    end
+  end
+end

data/lib/ruby_llm/infinite_context/cache_mark.rb

@@ -0,0 +1,76 @@
+# frozen_string_literal: true
+
+module RubyLLM
+  module InfiniteContext
+    # How to send the view so that consecutive turns read it from the
+    # provider's prompt cache. Only providers that accept explicit cache
+    # breakpoints use this (Anthropic); others rely on implicit prefix
+    # caching, and some (OpenAI) reject explicit breakpoints outright.
+    #
+    # A cache entry matches on the exact block layout, so:
+    # - The view goes as blocks of GRID parts each (a fixed grid by part
+    #   index). Where two views agree, their blocks are byte-identical.
+    # - The mark goes on the last grid cut inside the part of the view that
+    #   the next messages will not change. The fold is deterministic, so we
+    #   run it HORIZON messages ahead (with lines of a typical size) and
+    #   stop before the first part it would merge. Under budget that is the
+    #   whole view.
+    class CacheMark
+      GRID = 8
+      HORIZON = 8 # messages ahead: about two turns
+
+      # cuts: part indexes where blocks end (the last block runs to the end);
+      # mark: the cut whose block gets the cache mark, or nil.
+      Plan = Data.define(:cuts, :mark)
+
+      # parts: the view's parts; sizes: [level, i] => bytes of built nodes;
+      # previous_mark: the parts prefix the previous turn marked.
+      def self.plan(parts:, total:, sizes:, previous_mark: [], budget: DEFAULT_VIEW_BUDGET, horizon: HORIZON, grid: GRID, recall: nil)
+        cuts = (grid...parts.size).step(grid).to_a
+        stable = stable_prefix(parts, total, sizes, budget, horizon, recall)
+        mark = cuts.select { |c| c <= stable }.max
+        kept = previous_mark.size
+        mark = kept if kept.positive? && cuts.include?(kept) && (mark.nil? || kept > mark) && parts.first(kept) == previous_mark
+        Plan.new(cuts:, mark:)
+      end
+
+      # How many leading parts stay the same after `horizon` more messages.
+      def self.stable_prefix(parts, total, sizes, budget, horizon, recall = nil)
+        return 0 if parts.empty?
+
+        typical = typical_line(parts, sizes)
+        guess = lambda do |l, i|
+          next sizes[[l, i]] if sizes.key?([l, i])
+          next typical if l.zero? && i >= total # a message that does not exist yet
+
+          a = sizes[[l - 1, 2 * i]]
+          b = sizes[[l - 1, 2 * i + 1]]
+          [a + b, NODE].min if a && b # a parent the pump will build soon
+        end
+        fold = Fold.new(parts:, budget:, recall:, &guess)
+        horizon.times { |h| fold.append(total + h).fit(total + h + 1) }
+        parts.zip(fold.parts).take_while { |a, b| a == b }.size
+      end
+
+      def self.typical_line(parts, sizes)
+        recent = parts.last(50).filter_map { |l, i| sizes[[l, i]] if l.zero? }
+        recent.empty? ? NODE / 2 : recent.sum / recent.size
+      end
+
+      Block = Data.define(:text, :mark)
+
+      # The view as text blocks: "<chat>\n" opens the first block, every line
+      # ends with "\n", and "</chat>" is a block of its own, so the text of a
+      # block never depends on where the view ends.
+      def self.blocks(lines, plan)
+        bounds = [0, *plan.cuts, lines.size].uniq
+        pieces = bounds.each_cons(2).map do |a, b|
+          text = lines[a...b].map { |l| "#{l}\n" }.join
+          Block.new(text: a.zero? ? "<chat>\n#{text}" : text, mark: b == plan.mark)
+        end
+        pieces = [Block.new(text: "<chat>\n", mark: false)] if pieces.empty?
+        [*pieces, Block.new(text: "</chat>", mark: false)]
+      end
+    end
+  end
+end

data/lib/ruby_llm/infinite_context/chat_options.rb

@@ -0,0 +1,61 @@
+# frozen_string_literal: true
+
+module RubyLLM
+  module InfiniteContext
+    # How the gem builds the chats it creates itself: the compactor's chats
+    # (the pump, BatchImport) and a chat that `attach` builds from a model
+    # name. The options are the ones RubyLLM.chat takes for picking the wire:
+    #
+    #   provider:             :openai
+    #   protocol:             :chat_completions
+    #   assume_model_exists:  true
+    #   context:              a RubyLLM context (its own keys and base URL)
+    #
+    # RubyLLM 2.1.1 sends an OpenAI-provider chat to the Responses API
+    # (POST {base}/responses) by default. Fireworks and most other
+    # OpenAI-compatible services only offer chat completions, so pass
+    # `protocol: :chat_completions` for them. See the README.
+    module ChatOptions
+      KEYS = %i[provider protocol assume_model_exists context].freeze
+
+      module_function
+
+      # Returns a frozen copy of `options` with symbol keys. Unknown keys raise
+      # (a typo such as `protcol:` would silently pick the wrong API).
+      def normalize(options)
+        options = (options || {}).to_h.transform_keys(&:to_sym)
+        unknown = options.keys - KEYS
+        unless unknown.empty?
+          raise ArgumentError, "unknown chat option(s) #{unknown.inspect}; known: #{KEYS.inspect}"
+        end
+
+        options.freeze
+      end
+
+      # RubyLLM.chat (or context.chat) with the options passed through.
+      def build(model, options = nil)
+        options = normalize(options).dup
+        context = options.delete(:context)
+        (context || RubyLLM).chat(model: model, **options)
+      end
+    end
+
+    class << self
+      # Default options for the compactor's chats when a Compactor,
+      # BatchImport or the pump is built without its own `chat_options:`.
+      # The pump's default builder and PumpJob (ActiveJob) have nowhere else
+      # to read them from, so set this once at boot for those paths:
+      #
+      #   RubyLLM::InfiniteContext.compactor_chat_options =
+      #     {provider: :openai, protocol: :chat_completions, assume_model_exists: true}
+      def compactor_chat_options = @compactor_chat_options ||= {}.freeze
+
+      def compactor_chat_options=(options)
+        @compactor_chat_options = ChatOptions.normalize(options)
+      end
+
+      # Attaches a chat to a memory tree; see Manager.attach.
+      def attach(...) = Manager.attach(...)
+    end
+  end
+end