truffler 0.1.1 → 0.1.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 78736a9bfd83f38c6623b52904121d19a15c33f9d53bad271c1021968749c11d
-  data.tar.gz: 68f40b4143278a1ab48102aaa5d0146fc00c71885f332180dacdd2495e6e0a7d
+  metadata.gz: 99d7f61714bf653e012b5cea1f9d57cfbde9d390aae048ba0d52387292a37159
+  data.tar.gz: b9826342d2552dae2b1cd9d671ac4d4d448f5fcbe3822b21e5788b7d272365b0
 SHA512:
-  metadata.gz: 2b98ccc3b180c487994c17eb6ed194e300ecace2948b549952a7b731e9bd5364c74a59656e93d8235bd6499aa41cb0926ea6ccf80acb9eed69e4946f5219b133
-  data.tar.gz: 860b3932fa42544edd3bee2ecf65ec3e90ae0aa8d9d9c6d8c1ea073dd9e5a0b7d7e4af78ab28dcd469aced26a47aa73e91b148878b98ea999fe8cf7cee5b3e5a
+  metadata.gz: 414ddacdd2e195a077792a36b3c1e8050a8e086a99ccb7ffed065776826f6af2c83da47835ce36d663db166b664e2559a01f2f92dcc6c493e120bbcd60ae89f8
+  data.tar.gz: 272d2d762fdcd5e0f6a6d0cec08ca332619a14e6ab8b3b94a5f2670e16091bf0c75395866a5a77ede42e7d22483a95252cbf6b0cee3565cacfb91d2ae5c17569

data/CHANGELOG.md

@@ -1,5 +1,15 @@
 # Changelog
 
+## [0.1.2]
+
+Fixes from happyhappy's production `churn_risk` backfill and its 0.1.1 upgrade.
+
+- `rake truffler:backfill` waits out budget denials instead of ending on `budget_denied`. It backs off 1 s, doubling to 30 s (longer when the budget's new `Decision#retry_after` hint says so), and retries from the same cursor until the backfill completes or reaches the spend cap. `MAX_DURATION=<seconds>` stops it with `paused` and the cursor. While waiting it prints counts, spend, and the cursor, never record text. In code: `Labeling::Backfill#run(wait: true, max_duration:, sleeper:, clock:, progress:)`. `BackfillJob` reschedules denials with the same backoff (it used to wait a fixed 30 s), counting consecutive denials in its arguments.
+- The backfill spend cap holds across runs. Spend is recorded per model and app-wide vocabulary version in a new `truffler_backfill_spends` table. Each Jev request reserves its estimate against the cap in SQL, so rerunning the rake task or overlapping `BackfillJob` chains can no longer spend more than `backfill_spend_cap` for one vocabulary version. Before, overlapping chains could spend up to twice the cap. A vocabulary change starts a new ledger. `truffler:status` prints the spend for the current version, and `RESET_SPEND=1` zeroes it before a backfill. Lens backfills still charge only their lens row.
+- Upgrading from 0.1.1: run `bin/rails generate truffler:upgrade && bin/rails db:migrate` to add `truffler_backfill_spends`. Until you do, backfills log one warning and cap spend per run as in 0.1.1. Fresh installs get the table from `truffler:install`.
+- Time phrases now include hours ("last 3 hours", "past hour", "last hour"), "past week", "past month", and "past/last N months" as rolling windows ending now; "this/last week" and "this/last month" keep their calendar meaning. Chips and suppression are unchanged.
+- A query word now names a choice option by the words of its display name or description (minus stopwords), not only its key, so "spiral" names option `p_17` described as "Spiral writing tool" when the encoding applies it. The request state's label vocabulary adds `option_names` (option key to display name) next to `options`.
+
 ## [0.1.1]
 
 Fixes from the first host integration (happyhappy).

data/README.md

@@ -34,12 +34,23 @@ bin/rails db:migrate
 
 The generator writes three files:
 
-- A migration that creates `truffler_labels`, `truffler_record_states`, `truffler_embeddings`, `truffler_query_misses`, `truffler_lenses`, and `truffler_lens_versions`.
+- A migration that creates `truffler_labels`, `truffler_record_states`, `truffler_embeddings`, `truffler_query_misses`, `truffler_lenses`, `truffler_lens_versions`, and `truffler_backfill_spends`.
 - `config/initializers/truffler.rb`.
 - `app/channels/truffler_channel.rb`.
 
 It takes two options. `--record-id-type=bigint|integer|string|uuid` must match your primary keys. `--vector-dimensions=N` stores embeddings in a pgvector column of that width; it needs Postgres and the `neighbor` gem, which provides the `t.vector` column type.
 
+### Upgrading from 0.1.1
+
+0.1.2 adds the `truffler_backfill_spends` table, which keeps the backfill spend cap across runs. Add it with:
+
+```bash
+bin/rails generate truffler:upgrade
+bin/rails db:migrate
+```
+
+The migration creates only that table and skips it if it already exists. Until you run it, backfills log one warning and cap spend per run, as 0.1.1 did.
+
 Truffler digests user keys and query misses with `secret_key_base`. Rails supplies it automatically; outside Rails, set `config.secret_key_base`.
 
 ## Declaring a model
@@ -172,9 +183,9 @@ result = Email.truffler(params[:q], tenant: Current.account.id, scope: Current.a
 
 A keystroke search makes no network call. It reads the query encoding and query vector from the cache. When the cache misses, it enqueues `EncodeQueryJob`, and the next keystroke or reload picks up the result.
 
-Query encoding sends Jev the label vocabulary (each label's description and a choice label's option names) next to the query, and asks for each label whether the query filters on it, prefers it, or ignores it, plus the role of each word. A choice label's option question also offers `Truffler::NO_OPTION` (`"truffler:none"`), meaning the query names none of its options, so a host option literally called `none` stays filterable. Word roles are then checked locally: a word that names a label the query applies (its key, a word of its key, or the chosen option, ignoring case and plurals, or a word of four letters or more sharing its first three letters, so "angry" names `anger`) counts as naming the label, and common stopwords are dropped. When the encoding applies a label filter, the filter decides which records match and keyword hits only rank them; without one, the remaining keywords must match.
+Query encoding sends Jev the label vocabulary (each label's description and a choice label's option names) next to the query, and asks for each label whether the query filters on it, prefers it, or ignores it, plus the role of each word. A choice label's option question also offers `Truffler::NO_OPTION` (`"truffler:none"`), meaning the query names none of its options, so a host option literally called `none` stays filterable. Word roles are then checked locally: a word that names a label the query applies (its key, a word of its key, the chosen option, or a word of that option's display name or description, ignoring case and plurals, or a word of four letters or more sharing its first three letters, so "angry" names `anger`) counts as naming the label, and common stopwords are dropped. When the encoding applies a label filter, the filter decides which records match and keyword hits only rank them; without one, the remaining keywords must match.
 
-Time phrases are handled in Ruby and never asked of Jev: `today`, `yesterday`, `this week`, `last week`, `this month`, `last month`, `past|last N days|weeks`, and `since monday` through `since sunday`. The first one in a query limits results to records whose `arrived_at` column falls in that window, its words are not keywords, and it shows as a chip `{key: "time", label: "time", kind: :time, name: "This week"}`. Pass `"time"` in `suppressed:` to drop it. Weeks start on `Date.beginning_of_week`, and the window is computed from `Time.current` (or a `clock:` callable passed to the search, for tests).
+Time phrases are handled in Ruby and never asked of Jev: `today`, `yesterday`, `this week`, `last week`, `this month`, `last month`, `past|last N hours|days|weeks|months`, `last hour`, `past hour`, `past week`, `past month`, and `since monday` through `since sunday`. "Past/last N units" is a rolling window ending now; "this/last week" and "this/last month" are calendar windows. The first one in a query limits results to records whose `arrived_at` column falls in that window, its words are not keywords, and it shows as a chip `{key: "time", label: "time", kind: :time, name: "This week"}`. Pass `"time"` in `suppressed:` to drop it. Weeks start on `Date.beginning_of_week`, and the window is computed from `Time.current` (or a `clock:` callable passed to the search, for tests).
 
 The returned `Truffler::Search::Result` exposes:
 
@@ -327,7 +338,7 @@ Set these in `Truffler.configure do |config| ... end`.
 | `max_field_chars`, `request_token_budget`, `max_questions_per_request` | 4,000, 48,000, 200 | Request packing limits. |
 | `queue_name` | `:default` | Queue for every Truffler job. |
 | `cost_per_million_tokens` | 0.042 | Jev input price, used in usage events and estimates. |
-| `backfill_spend_cap` | 5.0 | Dollar cap per backfill run (`BackfillJob` chains, `ResumeJob` backfills, and `truffler:backfill`). `nil` disables it; for the rake task, `SPEND_CAP=none` does. Supplied labels cost nothing and are still written once it is reached. |
+| `backfill_spend_cap` | 5.0 | Dollar cap on backfill spend per model and vocabulary version, shared by `BackfillJob` chains, `ResumeJob` backfills, and `truffler:backfill` runs (see Backfill). `nil` disables it; for the rake task, `SPEND_CAP=none` does. Supplied labels cost nothing and are still written once it is reached. |
 | `resume_pending_after` | 5 minutes | How long before `ResumeJob` treats work as stuck. |
 | `embedder` | `Embeddings::RubyLLMEmbedder.new` | Any `Embeddings::Embedder` subclass. The default calls `RubyLLM.embed`, which works on ruby_llm 1.x and 2. |
 | `embedding_cost_per_million_tokens` | 0.02 | Embedding price. |
@@ -353,10 +364,10 @@ Truffler enqueues most of its own jobs. Run a worker for `config.queue_name` and
 | `Truffler::Jobs::ResumeJob` | Every few minutes. Requeues failed and stuck labeling after an outage or a crashed worker, and enqueues up to 1,000 missing or stale embeddings per model per hour. |
 | `Truffler::Jobs::PruneQueryMissesJob` | Daily. Enforces `miss_retention`. |
 | `Truffler::Jobs::ExpireLensesJob` | Daily. Expires lenses unused for `lenses.expire_after`. |
-| `bin/rails "truffler:backfill[Email]"` (`SPEND_CAP=20`, or `none`; default `backfill_spend_cap`) or `Truffler::Jobs::BackfillJob.perform_later("Email")` | After adopting Truffler, changing a declaration, or changing the model pin. |
+| `bin/rails "truffler:backfill[Email]"` (see Backfill) or `Truffler::Jobs::BackfillJob.perform_later("Email")` | After adopting Truffler, changing a declaration, or changing the model pin. |
 | `Truffler::Embeddings::Backfill.new(Email).enqueue` | After enabling embeddings or changing the embedding model, width, or fields, to re-embed everything now instead of through the `ResumeJob` sweep. It enqueues 1,000 jobs at a time; pass `limit:` to cap the total. |
 
-`bin/rails "truffler:status[Email]"` prints labeling counts. `bin/rails "truffler:suggestions[Email]"` prints candidate questions drawn from logged query misses.
+`bin/rails "truffler:status[Email]"` prints labeling counts and the backfill spend for the current vocabulary version. `bin/rails "truffler:suggestions[Email]"` prints candidate questions drawn from logged query misses.
 
 With Solid Queue, for example:
 
@@ -373,6 +384,18 @@ truffler_expire_lenses:
   schedule: every day at 3am
 ```
 
+### Backfill
+
+`bin/rails "truffler:backfill[Email]"` labels missing, stale, and failed records inline, newest first, at backfill priority. Backfill has the lowest share of the request budget, so the task waits whenever the budget turns it away. It backs off 1 s, doubling to 30 s (or longer if the budget says a slot frees later), and resumes from the same cursor. It ends when every record is current (`complete`) or the spend cap is reached (`spend_cap_reached`). While it waits it prints the records labeled, the spend, and the cursor, never record text. Environment variables:
+
+| Variable | Effect |
+|---|---|
+| `SPEND_CAP=20` or `none` | Overrides `backfill_spend_cap` for this run. |
+| `MAX_DURATION=600` | Stops after that many seconds with `paused` and the cursor. Rerun to continue. |
+| `RESET_SPEND=1` | Zeroes the spend recorded for the current vocabulary version before starting. |
+
+The spend cap holds across runs. Truffler records backfill spend per model and app-wide vocabulary version in `truffler_backfill_spends`, and reserves each request's estimate against the cap in SQL. A rerun, a `ResumeJob` backfill, and overlapping `BackfillJob` chains all draw on the same total, so together they stop at `backfill_spend_cap`. Changing the vocabulary (a reworded question, a new label, or an activated lens) starts a new total. Lens backfills are capped separately by each lens's `spend_cap_usd`. `BackfillJob` reschedules itself after a denial with the same backoff. In code, `Truffler::Labeling::Backfill.new(Email).run(wait: true, max_duration: 600)` does what the task does.
+
 ## Privacy
 
 These guarantees hold for every model, and matter most for models that use Active Record encryption:

data/lib/generators/truffler/install/templates/migration.rb.tt

@@ -105,5 +105,15 @@ class CreateTrufflerTables < ActiveRecord::Migration<%= migration_version %>
     end
     add_index :truffler_lens_versions, [:lens_id, :number], unique: true,
       name: "index_truffler_lens_versions_on_lens_and_number"
+
+    create_table :truffler_backfill_spends do |t|
+      t.string :record_type, null: false
+      t.string :vocabulary_version, null: false
+      t.float :spent_usd, null: false, default: 0.0
+      t.integer :requests, null: false, default: 0
+      t.timestamps
+    end
+    add_index :truffler_backfill_spends, [:record_type, :vocabulary_version], unique: true,
+      name: "index_truffler_backfill_spends_on_ledger"
   end
 end

data/lib/generators/truffler/upgrade/templates/backfill_spends_migration.rb.tt

@@ -0,0 +1,19 @@
+class CreateTrufflerBackfillSpends < ActiveRecord::Migration<%= migration_version %>
+  def up
+    return if table_exists?(:truffler_backfill_spends)
+
+    create_table :truffler_backfill_spends do |t|
+      t.string :record_type, null: false
+      t.string :vocabulary_version, null: false
+      t.float :spent_usd, null: false, default: 0.0
+      t.integer :requests, null: false, default: 0
+      t.timestamps
+    end
+    add_index :truffler_backfill_spends, [:record_type, :vocabulary_version], unique: true,
+      name: "index_truffler_backfill_spends_on_ledger"
+  end
+
+  def down
+    drop_table :truffler_backfill_spends, if_exists: true
+  end
+end

data/lib/generators/truffler/upgrade/upgrade_generator.rb

@@ -0,0 +1,24 @@
+require "rails/generators"
+require "rails/generators/active_record"
+
+module Truffler
+  module Generators
+    # Adds the tables a newer Truffler needs to an app installed with an older
+    # one. Each migration skips a table that already exists.
+    class UpgradeGenerator < Rails::Generators::Base
+      include ActiveRecord::Generators::Migration
+
+      source_root File.expand_path("templates", __dir__)
+
+      def create_backfill_spends_migration
+        migration_template "backfill_spends_migration.rb.tt", File.join(db_migrate_path, "create_truffler_backfill_spends.rb")
+      end
+
+      private
+
+      def migration_version
+        "[#{ActiveRecord::Migration.current_version}]"
+      end
+    end
+  end
+end

data/lib/tasks/truffler.rake

@@ -17,12 +17,34 @@ namespace :truffler do
     end
   end
 
-  desc "Backfill stale, missing, and failed labels for a model (SPEND_CAP=dollars or none; default config.backfill_spend_cap)"
+  resolve_max_duration = lambda do |value|
+    next if value.to_s.strip.empty?
+
+    seconds = Float(value, exception: false)
+    abort "MAX_DURATION must be a number of seconds, got #{value.inspect}" unless seconds&.positive?
+    seconds
+  end
+
+  describe_cursor = ->(cursor) { cursor.nil? ? "none" : cursor }
+
+  desc "Backfill stale, missing, and failed labels for a model, waiting out budget denials " \
+    "(SPEND_CAP=dollars or none; default config.backfill_spend_cap; MAX_DURATION=seconds; RESET_SPEND=1 for a fresh spend ledger)"
   task :backfill, [ :model ] => :setup do |_, args|
     model = resolve_model.call(args[:model])
     spend_cap = resolve_spend_cap.call(ENV.fetch("SPEND_CAP", nil))
-    result = Truffler::Labeling::Backfill.new(model, spend_cap: spend_cap).run
-    puts "#{model.name}: #{result.status}, #{result.labeled} labeled in #{result.requests} requests, $#{format('%.6f', result.cost)}"
+    max_duration = resolve_max_duration.call(ENV.fetch("MAX_DURATION", nil))
+    if ENV.fetch("RESET_SPEND", nil) == "1"
+      Truffler::Labeling::Backfill.reset_spend!(model)
+      puts "#{model.name}: fresh spend ledger for the current vocabulary version"
+    end
+    progress = lambda do |so_far, delay|
+      puts "#{model.name}: waiting #{format('%.1f', delay)}s for backfill budget (#{so_far.labeled} labeled, " \
+        "$#{format('%.6f', so_far.cost)} spent, cursor #{describe_cursor.call(so_far.cursor)})"
+    end
+    result = Truffler::Labeling::Backfill.new(model, spend_cap: spend_cap).run(wait: true, max_duration: max_duration, progress: progress)
+    summary = "#{model.name}: #{result.status}, #{result.labeled} labeled in #{result.requests} requests, $#{format('%.6f', result.cost)}"
+    summary += ", cursor #{describe_cursor.call(result.cursor)}" if result.status == :paused
+    puts summary
   end
 
   desc "Print a model's labeling counts by status and staleness"
@@ -30,5 +52,13 @@ namespace :truffler do
     model = resolve_model.call(args[:model])
     puts model.name
     Truffler::Labeling::Backfill.status(model).each { |key, count| puts format("  %-9s %d", key, count) }
+    if Truffler::Records::BackfillSpend.available?
+      ledger = Truffler::Labeling::Backfill.spend(model)
+      cap = Truffler.config.backfill_spend_cap
+      puts format("  %-9s $%.6f in %d requests (vocabulary %s, cap %s)", "spent", ledger&.spent_usd.to_f, ledger&.requests.to_i,
+        Truffler::Labeling::Backfill.ledger_version(model).first(12), cap ? format("$%.2f", cap) : "none")
+    else
+      puts format("  %-9s %s", "spent", "not tracked across runs; run bin/rails g truffler:upgrade && bin/rails db:migrate")
+    end
   end
 end

data/lib/truffler/budget.rb

@@ -8,12 +8,17 @@ module Truffler
   #
   # Outcomes: :granted takes a slot; :demoted means a tenant is over its live
   # cap and its records should wait at backfill priority (no slot is taken);
-  # :denied means skip, pause, or reschedule. Only live callers wait, up to
-  # max_wait. A cache that cannot count (the null store) never blocks.
+  # :denied means skip, pause, or reschedule, with `retry_after` seconds until
+  # the denying window rolls over. Only live callers wait, up to max_wait. A
+  # cache that cannot count (the null store) never blocks.
   class Budget
     PRIORITIES = %i[live encode rerank backfill].freeze
 
-    Decision = Data.define(:outcome, :priority, :reason) do
+    Decision = Data.define(:outcome, :priority, :reason, :retry_after) do
+      def initialize(outcome:, priority:, reason:, retry_after: nil)
+        super
+      end
+
       def granted? = outcome == :granted
       def demoted? = outcome == :demoted
       def denied? = outcome == :denied
@@ -38,9 +43,9 @@ module Truffler
         return Decision.new(:demoted, :backfill, :tenant_cap) unless take(tenant_counter(tenant_key), records, config.tenant_live_cap, taken)
       end
       if (cap = config.user_caps[priority]) && user_key
-        return deny(priority, :user_cap, taken) unless take(user_counter(priority, user_key), 1, cap, taken)
+        return deny(priority, :user_cap, taken, until_next_minute) unless take(user_counter(priority, user_key), 1, cap, taken)
       end
-      return deny(priority, :exhausted, taken) unless take_second(priority)
+      return deny(priority, :exhausted, taken, until_next_second) unless take_second(priority)
 
       Decision.new(:granted, priority, nil)
     end
@@ -54,7 +59,7 @@ module Truffler
 
       cap = config.user_caps[priority]
       return Decision.new(:granted, priority, nil) unless cap && user_key
-      return deny(priority, :user_cap, []) unless take(user_counter(priority, user_key), 1, cap)
+      return deny(priority, :user_cap, [], until_next_minute) unless take(user_counter(priority, user_key), 1, cap)
 
       Decision.new(:granted, priority, nil)
     end
@@ -97,10 +102,20 @@ module Truffler
       end
     end
 
-    def deny(priority, reason, taken)
+    def deny(priority, reason, taken, retry_after)
       taken.each { |key, amount| @cache.decrement(key, amount, expires_in: 2.minutes) }
       Instrumentation.instrument(:budget_denied, priority: priority)
-      Decision.new(:denied, priority, reason)
+      Decision.new(:denied, priority, reason, retry_after)
+    end
+
+    def until_next_second
+      now = @clock.call
+      now.floor + 1 - now
+    end
+
+    def until_next_minute
+      now = @clock.call
+      (minute + 1) * 60 - now
     end
 
     def tenant_counter(tenant_key)

data/lib/truffler/errors.rb

@@ -7,7 +7,17 @@ module Truffler
   class LiveCallInTest < Error; end
   class IncompleteAnswers < Error; end
   class CassetteMiss < Error; end
-  class BudgetExhausted < Error; end
+
+  # Carries the budget's hint of how many seconds until a retry can succeed.
+  class BudgetExhausted < Error
+    attr_reader :retry_after
+
+    def initialize(message = nil, retry_after: nil)
+      @retry_after = retry_after
+      super(message)
+    end
+  end
+
   class TenantMismatch < Error; end
   class NotAuthorized < Error; end
   class InvalidLens < Error; end

data/lib/truffler/jobs/backfill_job.rb

@@ -2,20 +2,21 @@ module Truffler
   module Jobs
     # Backfills one model's stale, missing, failed, and demoted labels at
     # backfill priority. Arguments are the record type, the id cursor, the
-    # spend so far, the cap, and the retry attempt, never record text. A
-    # budget denial or a page limit reschedules the job from its cursor; the
-    # spend cap ends it. A Jev error reschedules it with backoff and the spend
+    # spend so far, the cap, the retry attempt, and the count of budget
+    # denials in a row, never record text. A budget denial reschedules the job
+    # from its cursor after the same backoff a waiting Labeling::Backfill uses
+    # (reset whenever a run lands work); a page limit reschedules it at once;
+    # the spend cap ends it. A Jev error reschedules it with backoff and the spend
     # already made, so retries cannot push past the cap; after MAX_ATTEMPTS
     # the released rows wait for the ResumeJob sweep.
     class BackfillJob < ActiveJob::Base
       MAX_PAGES = 20
       MAX_ATTEMPTS = 10
-      BUDGET_WAIT = 30.seconds
 
       queue_as { Truffler.config.queue_name }
 
       def perform(record_type, cursor: nil, spent: 0.0, spend_cap: Truffler.config.backfill_spend_cap, max_pages: MAX_PAGES,
-        attempt: 0)
+        attempt: 0, denials: 0)
         model = record_type.safe_constantize
         return unless model.respond_to?(:truffler_definition) && model.truffler_definition
 
@@ -25,7 +26,7 @@ module Truffler
 
         follow_up = { cursor: result.cursor, spent: spent + result.cost, spend_cap: spend_cap, max_pages: max_pages }
         case result.status
-        when :budget_denied then self.class.set(wait: BUDGET_WAIT).perform_later(record_type, **follow_up)
+        when :budget_denied then retry_after_denial(record_type, follow_up, result, denials)
         when :paused then self.class.perform_later(record_type, **follow_up)
         when :client_error then retry_later(record_type, follow_up, attempt + 1)
         when :complete, :spend_cap_reached then nil
@@ -35,6 +36,12 @@ module Truffler
 
       private
 
+      def retry_after_denial(record_type, follow_up, result, denials)
+        denials = 0 if result.labeled.positive? || result.requests.positive?
+        wait = Labeling::Backfill.backoff(denials, result.retry_after)
+        self.class.set(wait: wait).perform_later(record_type, **follow_up, denials: denials + 1)
+      end
+
       def retry_later(record_type, follow_up, attempt)
         return if attempt >= MAX_ATTEMPTS
 

data/lib/truffler/labeling/backfill.rb

@@ -11,39 +11,84 @@ module Truffler
     # done, and records already current are skipped, so a rerun never asks
     # Jev about them again. A spend cap stops the run before a request would
     # exceed it; host-supplied labels cost nothing, so they are still written
-    # once the cap is reached. A Jev error releases the claimed rows and ends
+    # once the cap is reached. The cap counts everything spent under the
+    # model's current app-wide vocabulary version, kept in the
+    # truffler_backfill_spends ledger, so reruns and overlapping jobs share
+    # it; without that table it falls back to this run plus `spent:`. A Jev error releases the claimed rows and ends
     # the run with `:client_error`, so the caller keeps the spend metered so far.
+    #
+    # A budget denial ends the run with `:budget_denied`, unless the run
+    # waits: then it backs off (see .backoff) and retries from the same
+    # cursor until it completes, reaches the spend cap, or runs out of
+    # `max_duration` seconds, which pauses it with the cursor to resume from.
     class Backfill
-      Result = Data.define(:status, :labeled, :requests, :cost, :cursor)
+      Result = Data.define(:status, :labeled, :requests, :cost, :cursor, :retry_after)
+
+      INITIAL_BACKOFF = 1.0
+      MAX_BACKOFF = 30.0
+
+      class_attribute :sleeper, default: ->(seconds) { sleep(seconds) }
+      class_attribute :clock, default: -> { Process.clock_gettime(Process::CLOCK_MONOTONIC) }
 
       STATES = Records::RecordState.table_name
 
       class SpendCapReached < StandardError; end
 
       # Wraps the client to meter spend per request and refuse a request whose
-      # estimated cost would push spend past the cap.
+      # estimated cost would push spend past the cap. With a ledger the
+      # estimate is reserved in SQL before the request and settled to the
+      # reported cost after it, so concurrent meters on one ledger never
+      # both take the last of the cap.
       class SpendMeter
-        attr_reader :spent, :requests
+        attr_reader :requests, :cost
 
-        def initialize(client, cap:, spent:, config: Truffler.config)
+        def initialize(client, cap:, spent:, ledger: nil, config: Truffler.config)
           @client = client
           @cap = cap
           @spent = spent.to_f
+          @ledger = ledger
           @requests = 0
+          @cost = 0.0
           @config = config
         end
 
-        def ask(state:, questions:, **options)
-          raise SpendCapReached if @cap && @spent + estimate(state, questions) > @cap
+        def spent
+          @ledger ? @ledger.total : @spent
+        end
 
-          answers = @client.ask(state: state, questions: questions, **options)
-          @requests += 1
-          @spent += answers.usage&.cost.to_f
+        def ask(state:, questions:, **options)
+          estimate = estimate(state, questions)
+          reserve(estimate)
+          answers = nil
+          begin
+            answers = @client.ask(state: state, questions: questions, **options)
+          ensure
+            @ledger&.settle(-estimate, requests: 0) unless answers
+          end
+          record(answers.usage&.cost.to_f, estimate)
           answers
         end
 
         private
 
+        def record(cost, estimate)
+          @requests += 1
+          @cost += cost
+          if @ledger
+            @ledger.settle(cost - estimate)
+          else
+            @spent += cost
+          end
+        end
+
+        def reserve(estimate)
+          if @ledger
+            raise SpendCapReached unless @ledger.reserve(estimate, @cap)
+          elsif @cap && @spent + estimate > @cap
+            raise SpendCapReached
+          end
+        end
+
         def estimate(state, questions)
           @config.cost_for(Tokens.estimate({ state: state, questions: questions }))
         end
@@ -53,6 +98,34 @@ module Truffler
         new(model).status
       end
 
+      # The ledger row for the model's current vocabulary version, or nil
+      # when nothing was spent yet or the ledger table is missing.
+      def self.spend(model)
+        return unless Records::BackfillSpend.available?
+
+        Records::BackfillSpend.for_model(model).find_by(vocabulary_version: ledger_version(model))
+      end
+
+      # Zeroes the current vocabulary version's ledger in place, so a chain
+      # still running keeps its row and continues against the fresh total.
+      def self.reset_spend!(model)
+        return unless Records::BackfillSpend.available?
+
+        Records::BackfillSpend.for_model(model).where(vocabulary_version: ledger_version(model))
+          .update_all(spent_usd: 0.0, requests: 0, updated_at: Time.current)
+      end
+
+      def self.ledger_version(model)
+        model.truffler_definition.vocabulary.version(all_users: true)
+      end
+
+      # Seconds to wait after `denials` consecutive budget denials with no
+      # work in between: 1, 2, 4, ... capped at MAX_BACKOFF, or the budget's
+      # retry hint when that is longer.
+      def self.backoff(denials, retry_after = nil)
+        [ [ INITIAL_BACKOFF * (2**denials), MAX_BACKOFF ].min, retry_after.to_f ].max
+      end
+
       attr_reader :model, :batch_size, :page_size
 
       def initialize(model, spend_cap: Truffler.config.backfill_spend_cap, batch_size: Truffler.config.batch_size,
@@ -62,32 +135,35 @@ module Truffler
         @batch_size = batch_size
         @page_size = page_size || batch_size * 5
         @cursor = cursor
-        @meter = SpendMeter.new(client, cap: spend_cap, spent: spent)
+        @spend_cap = spend_cap
+        @spent = spent
+        @client = client
         @budget = budget
         @versions = {}
       end
 
-      def run(max_pages: nil)
+      # `progress` is called with the result so far and the delay before each
+      # wait; it carries counts, cost, and the cursor, never record text.
+      def run(max_pages: nil, wait: false, max_duration: nil, sleeper: self.class.sleeper, clock: self.class.clock,
+        progress: nil)
         @labeled = 0
-        @started_spent = @meter.spent
-        cursor = @cursor
-        pages = 0
+        @started_cost = meter.cost
+        @pages = 0
+        deadline = max_duration && clock.call + max_duration
+        denials = 0
 
         loop do
-          scanned, rows = page(cursor)
-          return result(:complete, nil) if scanned.empty?
-          return result(:paused, cursor) if max_pages && pages >= max_pages
+          before = [ @labeled, meter.requests ]
+          status = sweep(max_pages, deadline, clock)
+          return result(status) unless wait && status == :budget_denied
 
-          rows.group_by(&:last).each do |tenant_key, tenant_rows|
-            tenant_rows.map(&:first).each_slice(batch_size) do |ids|
-              stop = label(ids, tenant_key)
-              return result(stop, cursor) if stop
-            end
-          end
+          denials = 0 unless before == [ @labeled, meter.requests ]
+          delay = self.class.backoff(denials, @retry_after)
+          return result(:paused) if deadline && clock.call + delay > deadline
 
-          cursor = scanned.last
-          pages += 1
-          return result(:complete, nil) if scanned.size < page_size
+          progress&.call(result(status), delay)
+          sleeper.call(delay)
+          denials += 1
         end
       end
 
@@ -118,13 +194,47 @@ module Truffler
         @queue ||= Queue.new(model)
       end
 
+      # Spend carried in with `spent:` is ignored when the ledger holds it.
+      def meter
+        @meter ||= begin
+          ledger = Records::BackfillSpend.ledger(model, version_for(nil)) if Records::BackfillSpend.available?
+          SpendMeter.new(@client, cap: @spend_cap, spent: ledger ? 0.0 : @spent, ledger: ledger)
+        end
+      end
+
       def version_for(tenant_key)
         @versions[tenant_key] ||= definition.vocabulary.version(tenant_key: tenant_key, all_users: true)
       end
 
-      def result(status, cursor)
-        Result.new(status: status, labeled: @labeled, requests: @meter.requests, cost: @meter.spent - @started_spent,
-          cursor: cursor)
+      def result(status)
+        Result.new(status: status, labeled: @labeled, requests: meter.requests, cost: meter.cost - @started_cost,
+          cursor: @cursor, retry_after: (@retry_after if status == :budget_denied))
+      end
+
+      # Walks pages below @cursor until done or stopped, returning the status.
+      def sweep(max_pages, deadline, clock)
+        @retry_after = nil
+        loop do
+          scanned, rows = page(@cursor)
+          return complete if scanned.empty?
+          return :paused if (max_pages && @pages >= max_pages) || (deadline && clock.call >= deadline)
+
+          rows.group_by(&:last).each do |tenant_key, tenant_rows|
+            tenant_rows.map(&:first).each_slice(batch_size) do |ids|
+              stop = label(ids, tenant_key)
+              return stop if stop
+            end
+          end
+
+          @cursor = scanned.last
+          @pages += 1
+          return complete if scanned.size < page_size
+        end
+      end
+
+      def complete
+        @cursor = nil
+        :complete
       end
 
       # Returns the scanned ids (for the cursor) and the [id, tenant_key] rows
@@ -173,10 +283,14 @@ module Truffler
         return if claimed.empty?
 
         begin
-          Labeler.new(model, client: @meter, budget: @budget).label(claimed, priority: :backfill)
-        rescue BudgetExhausted, SpendCapReached => error
+          Labeler.new(model, client: meter, budget: @budget).label(claimed, priority: :backfill)
+        rescue SpendCapReached
+          queue.demote(claimed)
+          return :spend_cap_reached
+        rescue BudgetExhausted => error
           queue.demote(claimed)
-          return error.is_a?(SpendCapReached) ? :spend_cap_reached : :budget_denied
+          @retry_after = error.retry_after
+          return :budget_denied
         rescue ClientError, IncompleteAnswers => error
           queue.release(claimed, error)
           return :client_error

data/lib/truffler/labeling/labeler.rb

@@ -50,7 +50,7 @@ module Truffler
             retry_failed_supplied(supplier.failed_ids, states_by_id)
             return Result.new(labeled: current.size, requests: index, cost: cost, demoted: true)
           end
-          raise BudgetExhausted, "no Jev budget for #{priority} labeling" if decision.denied?
+          raise BudgetExhausted.new("no Jev budget for #{priority} labeling", retry_after: decision.retry_after) if decision.denied?
 
           answers = client.ask(state: request.state, questions: request.questions, priority: decision.priority)
           cost += answers.usage&.cost.to_f

data/lib/truffler/query_encoding/encoder.rb

@@ -8,12 +8,14 @@ module Truffler
     # never asked (R18). Query text travels only in `state` ("query" and
     # "tokens"); a token question names its word by position, `tokens[n]`,
     # so searcher text never lands in an instruction (R8). The label
-    # vocabulary rides along in `state["labels"]` so a token question can
-    # tell a word that names a label from text to match.
+    # vocabulary, with choice option display names, rides along in
+    # `state["labels"]` so a token question can tell a word that names a
+    # label from text to match.
     #
     # Jev's word roles are then reconciled locally: a keyword that names a
-    # label the query applies (its key, a word of its key, or the chosen
-    # option, ignoring case and plurals, or sharing its first three letters) becomes a label term, and a common
+    # label the query applies (its key, a word of its key, the chosen option,
+    # or a word of that option's display name, ignoring case and plurals, or
+    # sharing its first three letters) becomes a label term, and a common
     # stopword becomes filler.
     #
     # Answers become a `Search::Encoding` with the KTD20 intent vector: boost
@@ -131,7 +133,7 @@ module Truffler
             boosts[key] = intent[key] = label.boost || DEFAULT_BOOST
           else next
           end
-          terms.concat(label_terms(key))
+          terms.concat(label_terms(key, option_name(label, key, tenant_key)))
         end
 
         query = Search::Query.new(request.state["query"])
@@ -189,7 +191,12 @@ module Truffler
       def vocabulary_state(labels, tenant_key)
         labels.transform_values do |label|
           entry = { "description" => label.description }
-          entry["options"] = label.options(tenant_key).keys if label.type == :choice
+          if label.type == :choice
+            options = label.options(tenant_key)
+            entry["options"] = options.keys
+            names = options.compact
+            entry["option_names"] = names if names.any?
+          end
           entry
         end
       end
@@ -212,13 +219,22 @@ module Truffler
       end
 
       # "category:billing" names "category" and "billing"; "needs_action"
-      # names "needs_action", "needs", and "action".
+      # names "needs_action", "needs", and "action". An option's display
+      # name adds its words minus stopwords: "p_17" shown as "Spiral writing
+      # tool" also names "spiral", "writing", and "tool".
       STEM = 3
 
-      def label_terms(storage_key)
+      def label_terms(storage_key, option_name = nil)
         label, option = Search::Encoding.split_key(storage_key)
-        [ label.split(":").last, option ].compact.flat_map { |name| [ name.downcase, *name.downcase.split(/[^\p{Alnum}]+/) ] }
-          .reject(&:empty?).map(&:singularize)
+        keys = [ label.split(":").last, option ].compact.flat_map { |name| [ name.downcase, *name.downcase.split(/[^\p{Alnum}]+/) ] }
+        words = option_name.to_s.downcase.split(/[^\p{Alnum}]+/).reject { |word| STOPWORDS.include?(word) }
+        (keys + words).reject(&:empty?).map(&:singularize)
+      end
+
+      def option_name(label, storage_key, tenant_key)
+        return unless label.type == :choice
+
+        label.options(tenant_key)[Search::Encoding.split_key(storage_key).last]
       end
 
       # "angry" names "anger": the same word ignoring plurals, or two words of

data/lib/truffler/records/backfill_spend.rb

@@ -0,0 +1,51 @@
+module Truffler
+  module Records
+    # The backfill spend ledger: one row per model and app-wide vocabulary
+    # version, so a spend cap holds across runs, reruns, and overlapping
+    # BackfillJob chains. Spend is reserved and settled in SQL, never read,
+    # added to, and written back.
+    class BackfillSpend < ActiveRecord::Base
+      self.table_name = "truffler_backfill_spends"
+
+      scope :for_model, ->(model) { where(record_type: model.polymorphic_name) }
+
+      # False on apps that upgraded the gem without running
+      # `rails g truffler:upgrade`; warns once per process.
+      def self.available?
+        return true if connection.data_source_exists?(table_name)
+
+        warn_missing
+        false
+      end
+
+      def self.ledger(model, version)
+        create_or_find_by!(record_type: model.polymorphic_name, vocabulary_version: version)
+      end
+
+      def self.warn_missing
+        return if @missing_warned
+
+        @missing_warned = true
+        Truffler.config.logger.warn("[truffler] #{table_name} is missing, so backfill spend caps apply per run only. " \
+          "Run `bin/rails g truffler:upgrade && bin/rails db:migrate`.")
+      end
+      private_class_method :warn_missing
+
+      # Adds `amount` unless that would pass `cap`; true when reserved.
+      def reserve(amount, cap)
+        scope = self.class.where(id: id)
+        scope = scope.where("spent_usd + ? <= ?", amount, cap) if cap
+        scope.update_all([ "spent_usd = spent_usd + ?, updated_at = ?", amount, Time.current ]) == 1
+      end
+
+      def settle(amount, requests: 1)
+        self.class.where(id: id)
+          .update_all([ "spent_usd = spent_usd + ?, requests = requests + ?, updated_at = ?", amount, requests, Time.current ])
+      end
+
+      def total
+        self.class.where(id: id).pick(:spent_usd).to_f
+      end
+    end
+  end
+end

data/lib/truffler/search/time_phrase.rb

@@ -6,8 +6,11 @@ module Truffler
     # to `[from, to]`, where `to` is nil for "until now".
     #
     #   today, yesterday, this week, last week, this month, last month,
-    #   past|last N day(s)|week(s), since monday..sunday
+    #   past|last N hour(s)|day(s)|week(s)|month(s), past hour|week|month,
+    #   last hour, since monday..sunday
     #
+    # "past/last N units" and "past week" are rolling: they end now and
+    # start N units back. "this/last week/month" are calendar windows.
     # Weeks start on `Date.beginning_of_week` (Monday by default). Only the
     # first phrase in a query counts; a quoted phrase is exact text.
     TimePhrase = Data.define(:name, :positions, :resolver) do
@@ -36,17 +39,30 @@ module Truffler
         end
       end
 
+      UNITS = %i[hours days weeks months].to_h { |unit| [ unit.to_s, unit ] }
+        .merge(%i[hour day week month].to_h { |unit| [ unit.to_s, :"#{unit}s" ] }).freeze
+
       def self.rolling(tokens, start)
         return unless %w[past last].include?(tokens[start]) && tokens[start + 1].to_s.match?(/\A\d{1,3}\z/)
 
         count = Integer(tokens[start + 1], 10)
-        unit = { "day" => :days, "days" => :days, "week" => :weeks, "weeks" => :weeks }[tokens[start + 2]]
+        unit = UNITS[tokens[start + 2]]
         return unless unit && count.positive?
 
         name = "Last #{count} #{count == 1 ? unit.to_s.singularize : unit}"
         new(name: name, positions: [ start, start + 1, start + 2 ], resolver: ->(now) { [ now - count.public_send(unit), nil ] })
       end
 
+      # "past hour", "past week", "past month", "last hour": one unit back
+      # from now. "last week" and "last month" stay calendar phrases.
+      def self.rolling_unit(tokens, start)
+        unit = tokens[start + 1]
+        return unless (tokens[start] == "past" && %w[hour week month].include?(unit)) || tokens[start, 2] == %w[last hour]
+
+        new(name: "#{tokens[start].capitalize} #{unit}", positions: [ start, start + 1 ],
+          resolver: ->(now) { [ now - 1.public_send(unit), nil ] })
+      end
+
       def self.since(tokens, start)
         wday = Date::DAYNAMES.map(&:downcase).index(tokens[start + 1]) if tokens[start] == "since"
         return unless wday
@@ -57,6 +73,7 @@ module Truffler
 
       PATTERNS = [
         method(:rolling),
+        method(:rolling_unit),
         method(:since),
         fixed(%w[today], "Today") { |now| [ now.beginning_of_day, nil ] },
         fixed(%w[yesterday], "Yesterday") { |now| [ now.yesterday.beginning_of_day, now.beginning_of_day ] },

data/lib/truffler/version.rb

@@ -1,3 +1,3 @@
 module Truffler
-  VERSION = "0.1.1"
+  VERSION = "0.1.2"
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: truffler
 version: !ruby/object:Gem::Version
-  version: 0.1.1
+  version: 0.1.2
 platform: ruby
 authors:
 - Kieran Klaassen
@@ -198,6 +198,8 @@ files:
 - lib/generators/truffler/install/templates/channel.rb.tt
 - lib/generators/truffler/install/templates/initializer.rb.tt
 - lib/generators/truffler/install/templates/migration.rb.tt
+- lib/generators/truffler/upgrade/templates/backfill_spends_migration.rb.tt
+- lib/generators/truffler/upgrade/upgrade_generator.rb
 - lib/tasks/truffler.rake
 - lib/tasks/truffler/bench.rake
 - lib/tasks/truffler/suggestions.rake
@@ -283,6 +285,7 @@ files:
 - lib/truffler/query_encoding/prefetch.rb
 - lib/truffler/questions.rb
 - lib/truffler/railtie.rb
+- lib/truffler/records/backfill_spend.rb
 - lib/truffler/records/embedding.rb
 - lib/truffler/records/label.rb
 - lib/truffler/records/query_miss.rb