truffler 0.1.3 → 0.1.4

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: bc0514a875fdc157249448de1514bb62560348ab4dfba22c75e3d0b0137b9072
-  data.tar.gz: 93b65a47fb89826536cdf03bd68efd824193488d20675789776400de2f19fb7a
+  metadata.gz: 3be74707295fff06a13ad0b12bbfb2d876f851893f0fff5f3f49b14c368bdb9b
+  data.tar.gz: 59a2e7d7e57669e8ffa35eadf656cc4e52fbae6cc47798f9ae4d01a43f870109
 SHA512:
-  metadata.gz: 4cbcba6f0deaee8afecbbe3e72988a175422b6ce42ba0f3d42b6d30b77f4736f28f485c21e15aac956fb31c4853724c30402872c52f8e5aff44acec36c0850a1
-  data.tar.gz: df78717442f56984544151b2962f3b78d724e001b6111d2ea92c9fd41903cb53c57117f85bac698bac4726b66000608b6a5132352eaa3c0fa30c688c893c1d04
+  metadata.gz: e71fd94ea16d17b9ca2e2f219591b51dfc94ab82f749bd21d5d277389039e6fa30f3c29130dd07c2b8aecd588abe0e1d91e8c6e70b66d199b485a77a90cc0a59
+  data.tar.gz: b3ea709b2d26f377b3c7731e7cc020d5249780bfbc756ad1f01dde3506a542f80b34bc7e8846387527d9311cbbe0015c826ea90916fdd251fdaa596efef123c6

data/CHANGELOG.md

@@ -1,5 +1,16 @@
 # Changelog
 
+## [0.1.4]
+
+Fixes from happyhappy production.
+
+- Generic filler nouns are no longer required keywords. "customers in the last 3 hours" returned nothing while "in the last 3 hours" returned the records. New `config.filler_words` (default: customer, customers, people, person, user, users, message, messages, email, emails, item, items, stuff, thing, things; replace or extend it, matched ignoring plurals). One rule now serves both the encoder's word reconciliation and the cold-cache keywords: stopwords and filler words are dropped unless that would leave the search with no keyword, no applied label, and no time phrase, so a lone "customers" still searches text and "customers refund this week" requires only "refund".
+- `rake truffler:status` with no model prints the status of every registered Truffler model instead of " is not a Truffler model". An unknown model name still aborts with a message.
+- A choice option may be `{ description:, search: }`: the long description stays what Jev labels with, and the short search text drives query-word matching and the request state's `option_names`. Plain descriptions and per-tenant callables keep working, and callables may return the new shape. Labeling fingerprints now cover only what Jev labels with: asked labels are unchanged (search texts are not part of the question), and a supplied label's fingerprint digests its type, option keys, legend, and `version:` but no longer its descriptions. Descriptions and search texts go into a separate encoding version that keys the query-encoding cache, so rewording re-encodes queries without staling labels.
+- Removing a chip gives its words back to the keywords. The cached encoding records which applied labels each label-term word named (by token position, no query text); once every one of them is removed, the word is a keyword again, so "urgent refunds" without the urgent chip matches both words. A word that named a label or option key only by shared prefix adds a soft keyword score (a quarter of the keyword weight) and is never required.
+- Upgrading from 0.1.3: supplied (`from:`) labels get a new fingerprint, so their stored rows and record states read stale once. `rake truffler:backfill` (or the next watched change) rewrites them at no Jev cost; the new vocabulary version also starts a fresh backfill spend ledger. Query encodings cached by 0.1.3 miss once and are re-encoded.
+- Removing the chip that justified dropping a filler word (the time range, or the last applied label) brings the filler noun back as a keyword, cached or cold. When nothing anchors a search and only droppable words are left, filler nouns stay keywords and only pure stopwords are dropped ("the customers" searches "customers").
+
 ## [0.1.3]
 
 - Display-name and description words of an applied choice option now match query words exactly (ignoring case and plurals) instead of by shared three-letter prefix. Under a filter, ordinary search words such as "email", "inbox" and "summary" (with Cora applied) or "chat" and "change" (with billing applied) no longer become label terms and keep ranking results. Label keys and option keys still match by shared prefix, so "angry" names `anger`.

data/README.md

@@ -92,6 +92,7 @@ Here is what each option does:
 
 - `filter_at` makes a label a hard filter at that probability when the query asks for it. `boost` is the label's weight when the query prefers it. `filter_weight:` (default 0) is how much intent weight a filter adds on top.
 - Choice `options:` may be a callable of the tenant key, which gives each tenant its own vocabulary. A tenant it gives no options (`{}` or nil) simply lacks the label: it is neither asked nor encoded there. The option name `truffler:none` is reserved (see `Truffler::NO_OPTION` below).
+- An option's value may be `{ description: "...", search: "..." }` instead of a plain description, e.g. `options: { "prod-a1" => { description: "Cora, the AI email assistant that drafts replies...", search: "Cora email assistant" } }`. Jev labels records with the description; query words are matched against the short search text (else the description), and the request state's `option_names` carries it. A per-tenant callable may return the same shape. Editing a search text never stales labels or triggers a backfill; it only changes the query-encoding cache key, so queries are re-encoded.
 - `watch :column, ...` relabels every label when one of those columns changes. Saving a record only relabels on columns that changed, so a `reads` field backed by a method (a conversation built from messages, say) needs the columns it is built from in `watch`.
 - `keyword` also accepts a single callable, `->(scope, tokens) { relation }`.
 - `embeddings column: :my_vector` searches a vector column you maintain yourself. Truffler never writes it.
@@ -103,7 +104,7 @@ Here is what each option does:
 |---|---|---|
 | `question:` | asked labels (required); optional with `from:` | What Jev is asked about each record. |
 | `criteria:` | `:noul` | `{ true => "...", false => "..." }` guidance for Jev. |
-| `options:` | `:choice` (required) | Option names, `{ option => description }`, or a callable of the tenant key. |
+| `options:` | `:choice` (required) | Option names, `{ option => description }`, `{ option => { description:, search: } }`, or a callable of the tenant key returning any of these. |
 | `legend:` | `:score` (required) | Two or more ordered levels, as an array or `{ index => description }`. |
 | `filter_at:`, `boost:`, `filter_weight:` | all | Filter threshold, boost weight, and the intent weight a filter adds. |
 | `description:` | all | The label's wording in query encoding. Defaults to the question, then to the key. |
@@ -158,7 +159,7 @@ To keep them fresh:
 - Call `record.truffler_refresh_labels!` when the answers change somewhere truffler cannot see, such as the job that runs your classifier after insert. It rewrites the record's supplied labels immediately, with no Jev call.
 - Bump `version:` when the logic behind `from:` changes. That changes the vocabulary version, so `rake truffler:backfill` rewrites the label for every record, again at no cost.
 
-A supplied label's fingerprint digests its type, options, legend, description, and `version:`, never the Jev model, so changing `config.model` does not rewrite supplied labels.
+A supplied label's fingerprint digests its type, option keys, legend, and `version:`, never the Jev model or its wording, so changing `config.model`, `description:`, or an option's description or search text does not rewrite supplied labels. Wording changes only re-key the query-encoding cache.
 
 ## Clients
 
@@ -183,7 +184,9 @@ result = Email.truffler(params[:q], tenant: Current.account.id, scope: Current.a
 
 A keystroke search makes no network call. It reads the query encoding and query vector from the cache. When the cache misses, it enqueues `EncodeQueryJob`, and the next keystroke or reload picks up the result.
 
-Query encoding sends Jev the label vocabulary (each label's description and a choice label's option names) next to the query, and asks for each label whether the query filters on it, prefers it, or ignores it, plus the role of each word. A choice label's option question also offers `Truffler::NO_OPTION` (`"truffler:none"`), meaning the query names none of its options, so a host option literally called `none` stays filterable. Word roles are then checked locally: a word that names a label the query applies (its key, a word of its key, or the chosen option key, ignoring case and plurals or sharing their first three letters when both words have four letters or more, so "angry" names `anger`; or, matched exactly, a word of that option's display name or description) counts as naming the label, and common stopwords are dropped. When the encoding applies a label filter, the filter decides which records match and keyword hits only rank them; without one, the remaining keywords must match.
+Query encoding sends Jev the label vocabulary (each label's description and a choice label's option names) next to the query, and asks for each label whether the query filters on it, prefers it, or ignores it, plus the role of each word. A choice label's option question also offers `Truffler::NO_OPTION` (`"truffler:none"`), meaning the query names none of its options, so a host option literally called `none` stays filterable. Word roles are then checked locally: a word that names a label the query applies (its key, a word of its key, or the chosen option key, ignoring case and plurals or sharing their first three letters when both words have four letters or more, so "angry" names `anger`; or, matched exactly, a word of that option's display name or description) counts as naming the label, and common stopwords and `config.filler_words` (generic nouns such as "customers", "users", "emails") are dropped. They are kept only when dropping them would leave no keyword, no applied label, and no time phrase, so "customers in the last 3 hours" lists the window's records while a lone "customers" still searches text; the same rule applies before the encoding is cached. When the encoding applies a label filter, the filter decides which records match and keyword hits only rank them; without one, the remaining keywords must match.
+
+When the searcher removes a chip, the words that named only that label (or only labels that are now all removed) become keywords again, so removing the `urgent` chip from "urgent refunds" searches for both words. A word that named a label or option key only by shared prefix ("urgently" for `urgent`) also adds a small keyword score (a quarter of the keyword weight) while its label applies, ranking records whose text contains it higher without requiring it.
 
 Time phrases are handled in Ruby and never asked of Jev: `today`, `yesterday`, `this week`, `last week`, `this month`, `last month`, `past|last N hours|days|weeks|months`, `last hour`, `past hour`, `past week`, `past month`, and `since monday` through `since sunday`. "Past/last N units" is a rolling window ending now; "this/last week" and "this/last month" are calendar windows. The first one in a query limits results to records whose `arrived_at` column falls in that window, its words are not keywords, and it shows as a chip `{key: "time", label: "time", kind: :time, name: "This week"}`. Pass `"time"` in `suppressed:` to drop it. Weeks start on `Date.beginning_of_week`, and the window is computed from `Time.current` (or a `clock:` callable passed to the search, for tests).
 
@@ -343,6 +346,7 @@ Set these in `Truffler.configure do |config| ... end`.
 | `embedder` | `Embeddings::RubyLLMEmbedder.new` | Any `Embeddings::Embedder` subclass. The default calls `RubyLLM.embed`, which works on ruby_llm 1.x and 2. |
 | `embedding_cost_per_million_tokens` | 0.02 | Embedding price. |
 | `vector_store` | `:auto` | `:neighbor` (pgvector `<=>`, or sqlite-vec `vec_distance_cosine` when you load the extension), `:ruby` (exact cosine in Ruby), or `:auto` (neighbor when available, otherwise Ruby). A model declaring `embeddings column:` always reads its own column. |
+| `filler_words` | `customer(s) people person user(s) message(s) email(s) item(s) stuff thing(s)` | Generic nouns never required as keywords on their own (matched ignoring plurals). Replace the list or extend it (`config.filler_words += %w[ticket]`). |
 | `encoding_prefetch` | `QueryEncoding::Prefetch.new` | Cache-miss hook, called as `call(model, query, cache_key:, tenant_key:, user_key:)`. |
 | `encoding_deadline` | 1.0 | Seconds a Smart run waits for an in-flight query encoding. |
 | `rerank_depth`, `rerank_chunk_size`, `rerank_max_field_chars` | 30, 10, 1,200 | Candidates reranked, candidates per Jev request, characters per field sent. |
@@ -367,7 +371,7 @@ Truffler enqueues most of its own jobs. Run a worker for `config.queue_name` and
 | `bin/rails "truffler:backfill[Email]"` (see Backfill) or `Truffler::Jobs::BackfillJob.perform_later("Email")` | After adopting Truffler, changing a declaration, or changing the model pin. |
 | `Truffler::Embeddings::Backfill.new(Email).enqueue` | After enabling embeddings or changing the embedding model, width, or fields, to re-embed everything now instead of through the `ResumeJob` sweep. It enqueues 1,000 jobs at a time; pass `limit:` to cap the total. |
 
-`bin/rails "truffler:status[Email]"` prints labeling counts and the backfill spend for the current vocabulary version. `bin/rails "truffler:suggestions[Email]"` prints candidate questions drawn from logged query misses.
+`bin/rails "truffler:status[Email]"` prints labeling counts and the backfill spend for the current vocabulary version; `bin/rails truffler:status` prints them for every registered Truffler model. `bin/rails "truffler:suggestions[Email]"` prints candidate questions drawn from logged query misses.
 
 With Solid Queue, for example:
 

data/bench/cassettes/1037e0b63ba790c02d14d0ddfc18b68151c11d03ce88d764658044b6bab2e55b.json

@@ -0,0 +1 @@
+{"answers":{"c001__relevance":{"noul":0.0,"type":"noul"},"c002__relevance":{"noul":0.0,"type":"noul"},"c003__relevance":{"noul":0.0,"type":"noul"},"c004__relevance":{"noul":0.0,"type":"noul"},"c005__relevance":{"noul":0.0,"type":"noul"}},"model":"synthetic-jev","request_hash":"1037e0b63ba790c02d14d0ddfc18b68151c11d03ce88d764658044b6bab2e55b","usage":{"input_tokens":873}}

data/bench/cassettes/19bba849a846d30a69dc385a337d67704fbe5194506fd3c914d3270d86495fde.json

@@ -0,0 +1 @@
+{"answers":{"c001__relevance":{"noul":0.0,"type":"noul"},"c002__relevance":{"noul":0.0,"type":"noul"},"c003__relevance":{"noul":0.0,"type":"noul"},"c004__relevance":{"noul":0.0,"type":"noul"},"c005__relevance":{"noul":0.0,"type":"noul"},"c006__relevance":{"noul":0.0,"type":"noul"},"c007__relevance":{"noul":0.0,"type":"noul"}},"model":"synthetic-jev","request_hash":"19bba849a846d30a69dc385a337d67704fbe5194506fd3c914d3270d86495fde","usage":{"input_tokens":1164}}

data/bench/cassettes/3768847dc57df8bf2328771e7763c0b8bd013a167244e7849967dac144420601.json

@@ -0,0 +1 @@
+{"answers":{"c001__relevance":{"noul":0.0,"type":"noul"},"c002__relevance":{"noul":0.0,"type":"noul"}},"model":"synthetic-jev","request_hash":"3768847dc57df8bf2328771e7763c0b8bd013a167244e7849967dac144420601","usage":{"input_tokens":417}}

data/lib/tasks/truffler.rake

@@ -47,9 +47,7 @@ namespace :truffler do
     puts summary
   end
 
-  desc "Print a model's labeling counts by status and staleness"
-  task :status, [ :model ] => :setup do |_, args|
-    model = resolve_model.call(args[:model])
+  print_status = lambda do |model|
     puts model.name
     Truffler::Labeling::Backfill.status(model).each { |key, count| puts format("  %-9s %d", key, count) }
     if Truffler::Records::BackfillSpend.available?
@@ -61,4 +59,16 @@ namespace :truffler do
       puts format("  %-9s %s", "spent", "not tracked across runs; run bin/rails g truffler:upgrade && bin/rails db:migrate")
     end
   end
+
+  desc "Print a model's labeling counts by status and staleness (every registered Truffler model when none is named)"
+  task :status, [ :model ] => :setup do |_, args|
+    if args[:model].to_s.strip.empty?
+      Rails.application.eager_load! if defined?(Rails.application) && Rails.application
+      models = Truffler.registry.models.select { |model| model.try(:truffler_definition) }.sort_by(&:name)
+      abort "No Truffler models are registered" if models.empty?
+      models.each(&print_status)
+    else
+      print_status.call(resolve_model.call(args[:model]))
+    end
+  end
 end

data/lib/truffler/configuration.rb

@@ -17,6 +17,8 @@ module Truffler
     attr_reader :lenses
     attr_accessor :encoding_deadline, :rerank_depth, :rerank_chunk_size, :rerank_max_field_chars, :smart_thresholds,
       :smart_run_ttl, :smart_candidate_pool, :broadcaster
+    # Generic nouns that are never required keywords on their own (Search::Filler).
+    attr_accessor :filler_words
 
     def initialize(env: ENV)
       @model = "jev-latest"
@@ -50,6 +52,7 @@ module Truffler
       @smart_run_ttl = 15.minutes
       @smart_candidate_pool = 200
       @broadcaster = nil
+      @filler_words = Search::Filler::DEFAULT_WORDS.dup
     end
 
     def client

data/lib/truffler/label_definition.rb

@@ -3,7 +3,9 @@ module Truffler
   # normalized position, and choices one row per option ("label:option") with
   # that option's probability. Choice options may be a callable of the tenant
   # key, which makes the vocabulary per-tenant; a tenant it gives no options
-  # (`{}` or nil) simply does not have the label.
+  # (`{}` or nil) simply does not have the label. An option's value is its
+  # description, or `{ description:, search: }` to give query encoding a
+  # short search text apart from the classifier description.
   #
   # A label with `from:` is supplied by the host: its answer is read from the
   # record in the shape Jev answers normalize to and Jev is never asked. Its
@@ -13,6 +15,7 @@ module Truffler
   class LabelDefinition
     TYPES = %i[noul choice score].freeze
     KEY = /\A[a-z][a-z0-9_]*\z/
+    OPTION_FIELDS = %i[description search].freeze
 
     attr_reader :key, :type, :instructions, :filter_at, :boost, :watch, :version
     # Intent weight a filter decision adds to the KTD20 query vector (default 0).
@@ -49,13 +52,23 @@ module Truffler
       @options.respond_to?(:call)
     end
 
+    # {option => description}: the classifier wording Jev labels with. An
+    # option given as `{ description:, search: }` contributes its description.
     def options(tenant_key = nil)
-      options = per_tenant? ? @options.call(tenant_key) : @options
-      options = Array(options).to_h { |option| [ option, nil ] } unless options.is_a?(Hash)
-      options = options.to_h { |option, description| [ option.to_s, description ] }
-      raise DefinitionError, "#{key}: the option name #{NO_OPTION} is reserved" if options.key?(NO_OPTION)
+      option_entries(tenant_key).transform_values { |entry| entry[:description] }
+    end
 
-      options
+    # {option => name} for query words: the option's short search text, else
+    # its description; options with neither are left out.
+    def option_names(tenant_key = nil)
+      option_entries(tenant_key).transform_values { |entry| entry[:search].presence || entry[:description] }.compact
+    end
+
+    # What query encoding reads beyond the labeling fingerprint: descriptions
+    # and search texts. Digested into the encoding cache key only, so
+    # rewording never stales stored labels.
+    def encoding_wording(tenant_key = nil)
+      { description: description, options: (option_entries(tenant_key) if type == :choice) }
     end
 
     # False for a per-tenant choice with no options for this tenant.
@@ -75,11 +88,11 @@ module Truffler
     end
 
     # Stored supplied values change with their shape and version, never with
-    # the Jev model. The description and option wording are digested too
-    # because query encoding asks about them.
+    # the Jev model or the wording: descriptions and search texts reach only
+    # query encoding (see encoding_wording).
     def supplied_fingerprint(tenant_key = nil)
-      Canonical.digest(supplied: true, type: type, options: (options(tenant_key) if type == :choice),
-        levels: (levels if type == :score), description: description, version: version)
+      Canonical.digest(supplied: true, type: type, options: (options(tenant_key).keys if type == :choice),
+        levels: (levels if type == :score), version: version)
     end
 
     # {storage_key => value} read from the record, or nil when the host has
@@ -106,6 +119,25 @@ module Truffler
 
     private
 
+    def option_entries(tenant_key)
+      options = per_tenant? ? @options.call(tenant_key) : @options
+      options = Array(options).to_h { |option| [ option, nil ] } unless options.is_a?(Hash)
+      options = options.to_h { |option, value| [ option.to_s, option_entry(option, value) ] }
+      raise DefinitionError, "#{key}: the option name #{NO_OPTION} is reserved" if options.key?(NO_OPTION)
+
+      options
+    end
+
+    def option_entry(option, value)
+      return { description: value, search: nil } unless value.is_a?(Hash)
+
+      entry = value.to_h.symbolize_keys
+      unknown = entry.keys - OPTION_FIELDS
+      raise DefinitionError, "#{key}: option #{option} takes description: and search:, not #{unknown.join(', ')}" if unknown.any?
+
+      { description: entry[:description], search: entry[:search] }
+    end
+
     def levels
       Array(@legend).size
     end

data/lib/truffler/lenses/lens_label.rb

@@ -29,6 +29,13 @@ module Truffler
         @question["criteria"].to_h.transform_keys(&:to_s)
       end
 
+      def option_names(tenant_key = nil)
+        options(tenant_key).compact
+      end
+
+      # The lens question, wording included, is already in its fingerprint.
+      def encoding_wording(_tenant_key = nil) = nil
+
       def storage_keys(tenant_key = nil)
         type == :choice ? options(tenant_key).keys.map { |option| "#{key}:#{option}" } : [ key ]
       end

data/lib/truffler/query_encoding/encoder.rb

@@ -16,7 +16,7 @@ module Truffler
     # label the query applies (its key, a word of its key, the chosen option,
     # or a word of that option's display name, ignoring case and plurals, or
     # sharing its first three letters) becomes a label term, and a common
-    # stopword becomes filler.
+    # stopword or `config.filler_words` noun becomes filler.
     #
     # Answers become a `Search::Encoding` with the KTD20 intent vector: boost
     # gives the declared boost, filter narrows and adds `filter_weight`
@@ -35,10 +35,7 @@ module Truffler
         "filler" => "A word that carries no meaning for the search"
       }.freeze
       NO_OPTION = Truffler::NO_OPTION
-      STOPWORDS = %w[
-        a about all an and any are at be by for from have i in is it me my now of on or our please so some that the their
-        them there they this to up us was we what when where which who why with you your
-      ].to_set.freeze
+      STOPWORDS = Search::Filler::STOPWORDS
 
       Request = Data.define(:state, :questions, :token_ids, :exact_tokens, :unasked_tokens)
 
@@ -120,8 +117,7 @@ module Truffler
         filters = {}
         boosts = {}
         intent = {}
-        stems = []
-        terms = []
+        names = {}
         labels(model, tenant_key, user_key).each_value do |label|
           key = storage_key(label, answers, tenant_key)
           next unless key
@@ -134,16 +130,17 @@ module Truffler
             boosts[key] = intent[key] = label.boost || DEFAULT_BOOST
           else next
           end
-          stems.concat(label_terms(key))
-          terms.concat(name_terms(option_name(label, key, tenant_key)))
+          names[key] = [ label_terms(key), name_terms(option_name(label, key, tenant_key)) ]
         end
 
         query = Search::Query.new(request.state["query"])
-        roles = reconcile(query, request.token_ids.transform_values { |id| answers.choice(id) }, stems.uniq, terms.uniq)
-        keyword_tokens = query.tokens.each_index.filter_map { |position| query.tokens[position] if roles[position] == "keyword" }
-        label_term_tokens = query.tokens.each_index.filter_map { |position| query.tokens[position] if roles[position] == "label_term" }
-        Search::Encoding.new(filters: filters, boosts: boosts, intent_vector: intent, keyword_tokens: keyword_tokens,
-          label_term_tokens: label_term_tokens)
+        roles, sources, soft = reconcile(query, request.token_ids.transform_values { |id| answers.choice(id) }, names)
+        tokens = ->(positions) { positions.map { |position| query.tokens[position] } }
+        Search::Encoding.new(filters: filters, boosts: boosts, intent_vector: intent,
+          keyword_tokens: tokens.call(roles.keys.select { |position| roles[position] == "keyword" }),
+          label_term_tokens: tokens.call(sources.keys), soft_keyword_tokens: tokens.call(soft).uniq,
+          filler_tokens: tokens.call(roles.keys.select { |position| roles[position] == "filler" }).uniq,
+          label_term_sources: sources.group_by { |position, _| query.tokens[position] }.transform_values { |pairs| pairs.flat_map(&:last).uniq })
       end
 
       private
@@ -194,35 +191,43 @@ module Truffler
         labels.transform_values do |label|
           entry = { "description" => label.description }
           if label.type == :choice
-            options = label.options(tenant_key)
-            entry["options"] = options.keys
-            names = options.compact
+            entry["options"] = label.options(tenant_key).keys
+            names = label.option_names(tenant_key)
             entry["option_names"] = names if names.any?
           end
           entry
         end
       end
 
-      # {position => role} for every token. Time phrase words are "time";
-      # exact tokens and unasked words are keywords; a keyword naming an
-      # applied label becomes a label term; stopwords become filler unless
-      # they are all that would be left of an encoding that applies nothing.
-      def reconcile(query, answered, stems, terms)
+      # Returns `[roles, sources, soft]`: {position => role} for every token,
+      # {label-term position => applied storage keys it names}, and the
+      # label-term positions that named a key only by prefix. Time phrase
+      # words are "time"; exact tokens and unasked words are keywords; a
+      # keyword naming an applied label becomes a label term; stopwords and
+      # filler words become filler unless they are all that would be left of
+      # an encoding that applies no label and no time range (Search::Filler).
+      # A word Jev called a label term that names no applied label locally
+      # is sourced to every applied label.
+      def reconcile(query, answered, names)
         roles = query.tokens.each_index.to_h do |position|
           [ position, query.time_position?(position) ? "time" : answered.fetch(position, "keyword") ]
         end
+        matches = roles.keys.to_h { |position| [ position, roles[position] == "time" ? {} : label_matches(query.tokens[position], names) ] }
         words = roles.keys.select { |position| roles[position] == "keyword" && !query.exact_tokens.include?(query.tokens[position]) }
-        words.each { |position| roles[position] = "label_term" if names_label?(query.tokens[position], stems, terms) }
-        stopwords = words.select { |position| roles[position] == "keyword" && STOPWORDS.include?(query.tokens[position]) }
-        return roles if stems.empty? && roles.values.count("keyword") == stopwords.size
-
-        stopwords.each { |position| roles[position] = "filler" }
-        roles
+        words.each { |position| roles[position] = "label_term" if matches[position].any? }
+        filler = words.select { |position| roles[position] == "keyword" && Search::Filler.word?(query.tokens[position]) }
+        Search::Filler.drop(filler, keyword_count: roles.values.count("keyword"), anchored: names.any? || !query.time_phrase.nil?,
+          stopword: ->(position) { Search::Filler.stopword?(query.tokens[position]) })
+          .each { |position| roles[position] = "filler" }
+        label_terms = roles.keys.select { |position| roles[position] == "label_term" }
+        sources = label_terms.to_h { |position| [ position, matches[position].keys.presence || names.keys ] }
+        soft = (words & label_terms).select { |position| matches[position].values.all?(:prefix) }
+        [ roles, sources, soft ]
       end
 
       # "category:billing" names "category" and "billing"; "needs_action"
       # names "needs_action", "needs", and "action". These key terms also
-      # match by shared stem (see names_label?).
+      # match by shared stem (see label_matches).
       STEM = 3
 
       def label_terms(storage_key)
@@ -231,8 +236,9 @@ module Truffler
           .reject(&:empty?).map(&:singularize)
       end
 
-      # An option's display name adds its words minus stopwords: "p_17" shown
-      # as "Spiral writing tool" names "spiral", "writing", and "tool". These
+      # An option's display name (its search text, else its description) adds
+      # its words minus stopwords: "p_17" shown as "Spiral writing tool" names
+      # "spiral", "writing", and "tool". These
       # match exactly only: descriptions are prose, and a stem match on them
       # would swallow ordinary search words ("chat" against "charge").
       def name_terms(option_name)
@@ -242,16 +248,23 @@ module Truffler
       def option_name(label, storage_key, tenant_key)
         return unless label.type == :choice
 
-        label.options(tenant_key)[Search::Encoding.split_key(storage_key).last]
+        label.option_names(tenant_key)[Search::Encoding.split_key(storage_key).last]
       end
 
-      # "angry" names "anger": the same word ignoring plurals, or, against a
-      # label or option key, two words of four letters or more that share
-      # their first three letters. Display-name words match exactly.
-      def names_label?(word, stems, terms)
+      # {storage key => :exact or :prefix} for the applied labels `word`
+      # names. "angry" names "anger": the same word ignoring plurals is
+      # :exact; against a label or option key, two words of four letters or
+      # more that share their first three letters are :prefix. Display-name
+      # words match exactly only.
+      def label_matches(word, names)
         word = word.singularize
-        terms.include?(word) ||
-          stems.any? { |term| term == word || (word.length > STEM && term.length > STEM && word[0, STEM] == term[0, STEM]) }
+        names.each_with_object({}) do |(key, (stems, terms)), matches|
+          if terms.include?(word) || stems.include?(word)
+            matches[key] = :exact
+          elsif word.length > STEM && stems.any? { |term| term.length > STEM && word[0, STEM] == term[0, STEM] }
+            matches[key] = :prefix
+          end
+        end
       end
 
       def intent_instructions(label)

data/lib/truffler/search/encoding.rb

@@ -10,11 +10,20 @@ module Truffler
     #   for the label term SUM(weight * value). Defaults to `boosts`; the
     #   encoder decides whether filtered labels also carry weight here.
     # - `keyword_tokens`: query tokens for the keyword source; nil means
-    #   every token that is not a label term.
+    #   every token that is not a label term or filler.
     # - `label_term_tokens`: tokens that named a label rather than a keyword.
+    # - `label_term_sources`: label-term token => the applied storage keys it
+    #   named. Once the searcher removes every one of them, the token is a
+    #   keyword again.
+    # - `soft_keyword_tokens`: label terms that named a label or option key
+    #   only by shared prefix ("urgently" for urgent). They add a small
+    #   keyword score and are never required.
+    # - `filler_tokens`: stopwords and filler words the encoder dropped.
+    #   Removing the chips that justified the drop brings filler nouns back.
     # - `time`: the query's `TimeRange`, resolved at search time and never
     #   cached, since "today" moves.
-    Encoding = Data.define(:filters, :boosts, :intent_vector, :keyword_tokens, :label_term_tokens, :time) do
+    Encoding = Data.define(:filters, :boosts, :intent_vector, :keyword_tokens, :label_term_tokens, :label_term_sources,
+      :soft_keyword_tokens, :filler_tokens, :time) do
       def self.load(value, query)
         return value if value.is_a?(self)
         return if value.nil?
@@ -22,15 +31,22 @@ module Truffler
         tokens = query.tokens
         new(filters: value["filters"], boosts: value["boosts"], intent_vector: value["intent_vector"],
           keyword_tokens: value["keyword_positions"]&.map { |position| tokens[position] }&.compact,
-          label_term_tokens: Array(value["label_term_positions"]).filter_map { |position| tokens[position] })
+          label_term_tokens: Array(value["label_term_positions"]).filter_map { |position| tokens[position] },
+          label_term_sources: Array(value["label_term_sources"]).filter_map { |position, keys| [ tokens[position], keys ] if tokens[position] },
+          soft_keyword_tokens: Array(value["soft_keyword_positions"]).filter_map { |position| tokens[position] },
+          filler_tokens: Array(value["filler_positions"]).filter_map { |position| tokens[position] })
       end
 
-      def initialize(filters: {}, boosts: {}, intent_vector: nil, keyword_tokens: nil, label_term_tokens: [], time: nil)
+      def initialize(filters: {}, boosts: {}, intent_vector: nil, keyword_tokens: nil, label_term_tokens: [], label_term_sources: {},
+        soft_keyword_tokens: [], filler_tokens: [], time: nil)
         filters = weights(filters)
         boosts = weights(boosts)
         intent_vector = weights(intent_vector || boosts).reject { |_, weight| weight.zero? }
+        sources = label_term_sources.to_h { |token, keys| [ token.to_s, Array(keys).map(&:to_s).freeze ] }
         super(filters: filters.freeze, boosts: boosts.freeze, intent_vector: intent_vector.freeze,
-          keyword_tokens: keyword_tokens&.map(&:to_s)&.freeze, label_term_tokens: Array(label_term_tokens).map(&:to_s).freeze, time: time)
+          keyword_tokens: keyword_tokens&.map(&:to_s)&.freeze, label_term_tokens: Array(label_term_tokens).map(&:to_s).freeze,
+          label_term_sources: sources.freeze, soft_keyword_tokens: Array(soft_keyword_tokens).map(&:to_s).freeze,
+          filler_tokens: Array(filler_tokens).map(&:to_s).freeze, time: time)
       end
 
       def empty?
@@ -38,14 +54,22 @@ module Truffler
       end
 
       # The encoding minus the chips the searcher removed (R20), matched by
-      # storage key or by label key; "time" removes the time range.
+      # storage key or by label key; "time" removes the time range. A label
+      # term whose every source label is gone becomes a keyword again.
       def without(suppressed)
         suppressed = Array(suppressed).map(&:to_s).to_set
         return self if suppressed.empty?
 
         keep = ->(key, _) { !suppressed.include?(key) && !suppressed.include?(self.class.split_key(key).first) }
-        with(filters: filters.select(&keep), boosts: boosts.select(&keep), intent_vector: intent_vector.select(&keep),
-          time: (time unless suppressed.include?(TimeRange.key)))
+        kept = { filters: filters.select(&keep), boosts: boosts.select(&keep), intent_vector: intent_vector.select(&keep) }
+        applied = kept.values.flat_map(&:keys).to_set
+        freed = label_term_sources.select { |_, keys| keys.any? && keys.none? { |key| applied.include?(key) } }.keys
+        kept_time = (time unless suppressed.include?(TimeRange.key))
+        keywords = keyword_tokens && (keyword_tokens + freed).uniq
+        keywords = Filler.keywords(keywords + filler_tokens, anchored: false) if keywords && applied.empty? && kept_time.nil?
+        with(**kept, time: kept_time, label_term_tokens: label_term_tokens - freed,
+          label_term_sources: label_term_sources.except(*freed), soft_keyword_tokens: soft_keyword_tokens - freed,
+          keyword_tokens: keywords)
       end
 
       # Splits a storage key into its label key and choice option. Lens keys
@@ -60,8 +84,11 @@ module Truffler
         end
       end
 
+      # Without encoder decisions (a cold cache), every search token that is
+      # not a label term, minus filler words (see Filler).
       def keywords(query)
-        keyword_tokens || (query.search_tokens - label_term_tokens)
+        keyword_tokens ||
+          Filler.keywords(query.search_tokens - label_term_tokens, anchored: !empty? || !time.nil?, exact: query.exact_tokens)
       end
 
       # The cache form: decisions plus token positions in the normalized
@@ -70,7 +97,12 @@ module Truffler
       def dump(query)
         { "filters" => filters, "boosts" => boosts, "intent_vector" => intent_vector,
           "keyword_positions" => keyword_tokens && positions(query, keyword_tokens),
-          "label_term_positions" => positions(query, label_term_tokens) }
+          "label_term_positions" => positions(query, label_term_tokens),
+          "label_term_sources" => query.tokens.each_index.filter_map do |position|
+            [ position, label_term_sources[query.tokens[position]] ] if label_term_sources.key?(query.tokens[position])
+          end,
+          "soft_keyword_positions" => positions(query, soft_keyword_tokens),
+          "filler_positions" => positions(query, filler_tokens) }
       end
 
       private

data/lib/truffler/search/encoding_cache.rb

@@ -1,8 +1,9 @@
 module Truffler
   module Search
     # Where keystroke search finds query encodings and query vectors (R12,
-    # R15). Keys digest the model, the normalized query, and the vocabulary
-    # version, plus the tenant when the vocabulary has per-tenant choices.
+    # R15). Keys digest the model, the normalized query, and the vocabulary's
+    # encoding version (labeling fingerprints plus descriptions and option
+    # search texts), plus the tenant when the vocabulary has per-tenant choices.
     # Values hold decisions and floats, never query text. Encoding keys use
     # the searcher's vocabulary version, which holds the lenses that searcher
     # can see (KTD21), so a personal lens's encoding is never shared; query
@@ -60,7 +61,7 @@ module Truffler
       def digest(model, query, tenant_key, user_key)
         definition = model.truffler_definition
         tenant = tenant_key&.to_s if definition.per_tenant_vocabulary?
-        version = definition.vocabulary.version(tenant_key: tenant_key&.to_s, user_key: user_key)
+        version = definition.vocabulary.encoding_version(tenant_key: tenant_key&.to_s, user_key: user_key)
         Canonical.digest(record_type: model.polymorphic_name, query: query.normalized, vocabulary_version: version, tenant_key: tenant)
       end
     end

data/lib/truffler/search/filler.rb

@@ -0,0 +1,48 @@
+module Truffler
+  module Search
+    # Words that carry no search meaning on their own: common stopwords and
+    # `config.filler_words` (generic nouns such as "customers" or "emails",
+    # matched ignoring plurals). One rule serves the encoder's reconcile and
+    # the cold-cache keywords: filler is dropped as a keyword unless dropping
+    # it would leave the search with no keyword, no applied label, and no
+    # time range, so a lone "customers" still searches text.
+    module Filler
+      STOPWORDS = %w[
+        a about all an and any are at be by for from have i in is it me my now of on or our please so some that the their
+        them there they this to up us was we what when where which who why with you your
+      ].to_set.freeze
+
+      DEFAULT_WORDS = %w[customer customers people person user users message messages email emails item items stuff thing things].freeze
+
+      module_function
+
+      def stopword?(word)
+        STOPWORDS.include?(word.to_s.downcase)
+      end
+
+      def word?(word, filler_words: Truffler.config.filler_words)
+        word = word.to_s.downcase
+        stopword?(word) || Array(filler_words).any? { |filler| filler.to_s.downcase.singularize == word.singularize }
+      end
+
+      # The subset of `candidates` (droppable keywords) to drop, given how many
+      # keywords there are and whether a label or time range is applied. When
+      # nothing anchors the search and only droppable words are left, the
+      # filler nouns stay as keywords and only pure stopwords go ("customers
+      # in the" searches "customers"); if every word is a stopword, all stay.
+      def drop(candidates, keyword_count:, anchored:, stopword: ->(candidate) { stopword?(candidate) })
+        return candidates if anchored || candidates.size < keyword_count
+
+        stopwords = candidates.select(&stopword)
+        stopwords.size == candidates.size ? [] : stopwords
+      end
+
+      # `tokens` minus filler words, or only minus stopwords when nothing else
+      # would anchor the search. Exact tokens (a quoted "the") are never filler.
+      def keywords(tokens, anchored:, exact: [])
+        dropped = drop(tokens.select { |token| !exact.include?(token) && word?(token) }, keyword_count: tokens.size, anchored: anchored)
+        tokens - dropped
+      end
+    end
+  end
+end

data/lib/truffler/search/sql.rb

@@ -5,12 +5,15 @@ module Truffler
     #   w_label * SUM(weight * value) over the intent's nonzero label keys
     #   + w_text * text similarity (inline SQL, or the store's top-K CASE)
     #   + w_keyword * keyword hit + w_exact * exact-source hit
+    #   + SOFT_KEYWORD * w_keyword * soft keyword hit
     #
     # A weighted dot product, not cosine: cosine would divide out magnitude
     # and let a record high on unrelated labels outrank the one the query
     # asked for. Hard filters are EXISTS subqueries that run before scoring.
     class Sql
       LABELS = "truffler_labels".freeze
+      # Share of the keyword weight a soft keyword hit adds (see Encoding).
+      SOFT_KEYWORD = 0.25
 
       attr_reader :model, :tenant_key, :query, :encoding, :vector, :weights
 
@@ -130,20 +133,28 @@ module Truffler
       def keyword_sql
         return @keyword_sql if defined?(@keyword_sql)
 
-        @keyword_sql = keywords.empty? ? nil : keyword_condition(definition.keyword)
+        @keyword_sql = keywords.empty? ? nil : keyword_condition(definition.keyword, keywords)
       end
 
-      def keyword_condition(source)
+      # Any soft keyword hit; soft keywords rank and never narrow.
+      def soft_keyword_sql
+        return @soft_keyword_sql if defined?(@soft_keyword_sql)
+
+        conditions = (encoding.soft_keyword_tokens - keywords).uniq.filter_map { |token| keyword_condition(definition.keyword, [ token ]) }
+        @soft_keyword_sql = conditions.empty? ? nil : conditions.map { |condition| "(#{condition})" }.join(" OR ")
+      end
+
+      def keyword_condition(source, tokens)
         case source
         when Array
           return if source.empty?
 
-          keywords.map do |token|
+          tokens.map do |token|
             pattern = quote("%#{model.sanitize_sql_like(token)}%")
             "(#{source.map { |name| "LOWER(#{column(name)}) LIKE #{pattern} ESCAPE '\\'" }.join(' OR ')})"
           end.join(" AND ")
         when nil then nil
-        else membership_sql(source.call(tenant_scope, keywords))
+        else membership_sql(source.call(tenant_scope, tokens))
         end
       end
 
@@ -189,12 +200,19 @@ module Truffler
         terms = {
           truffler_label_score: (label_score_sql && "#{Float(weights[:label])} * #{label_score_sql}"),
           truffler_text_score: (text_score_sql && "#{Float(weights[:text])} * #{text_score_sql}"),
-          truffler_keyword_score: (keyword_sql && "#{Float(weights[:keyword])} * (CASE WHEN #{keyword_sql} THEN 1.0 ELSE 0.0 END)"),
+          truffler_keyword_score: keyword_score_sql,
           truffler_exact_score: (exact_sql && "#{Float(weights[:exact])} * (CASE WHEN #{exact_sql} THEN 1.0 ELSE 0.0 END)")
         }.compact
         { truffler_score: terms.any? ? terms.values.map { |term| "(#{term})" }.join(" + ") : "0.0", **terms }
       end
 
+      def keyword_score_sql
+        weight = Float(weights[:keyword])
+        terms = [ (keyword_sql && "#{weight} * (CASE WHEN #{keyword_sql} THEN 1.0 ELSE 0.0 END)"),
+          (soft_keyword_sql && "#{weight * SOFT_KEYWORD} * (CASE WHEN #{soft_keyword_sql} THEN 1.0 ELSE 0.0 END)") ].compact
+        terms.map { |term| "(#{term})" }.join(" + ") if terms.any?
+      end
+
       def ordering
         order_column, direction = definition.order
         [ Arel.sql("truffler_score DESC"), (Arel.sql("#{column(order_column)} #{direction == :asc ? 'ASC' : 'DESC'}") if order_column),

data/lib/truffler/version.rb

@@ -1,3 +1,3 @@
 module Truffler
-  VERSION = "0.1.3"
+  VERSION = "0.1.4"
 end

data/lib/truffler/vocabulary.rb

@@ -1,9 +1,9 @@
 module Truffler
   # A label's fingerprint is the SHA-256 of its canonical question plus the
   # pinned Jev model; a host-supplied label's is its supplied fingerprint
-  # (type, options, version), which no Jev model change touches. The
-  # vocabulary version digests every fingerprint. A stored label is stale
-  # when its fingerprint differs from the current one.
+  # (type, option keys, version), which no Jev model change or rewording
+  # touches. The vocabulary version digests every fingerprint. A stored
+  # label is stale when its fingerprint differs from the current one.
   #
   # A scope's vocabulary also holds its active lens labels (KTD21), so a lens
   # changes the version only where it applies. user_key is the searcher key
@@ -39,6 +39,16 @@ module Truffler
       Canonical.digest(fingerprints(tenant_key: tenant_key, user_key: user_key, all_users: all_users))
     end
 
+    # The version query encodings are cached under: the labeling version plus
+    # a digest of the wording only query encoding reads (label descriptions,
+    # option descriptions and search texts), which labeling never sees for
+    # supplied labels and never sees for search texts.
+    def encoding_version(tenant_key: nil, user_key: nil)
+      labels = labels_for(tenant_key: tenant_key, user_key: user_key)
+      Canonical.digest(version: Canonical.digest(labels.transform_values { |label| fingerprint_of(label, tenant_key) }),
+        wording: labels.transform_values { |label| label.encoding_wording(tenant_key) }.compact)
+    end
+
     private
 
     def fingerprint_of(label, tenant_key)

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: truffler
 version: !ruby/object:Gem::Version
-  version: 0.1.3
+  version: 0.1.4
 platform: ruby
 authors:
 - Kieran Klaassen
@@ -102,12 +102,14 @@ files:
 - bench/cassettes/0858c5006c0bfac3e970ddaff3c139f6c110dbe3d8539fd95263706c637cdc8e.json
 - bench/cassettes/0c0eb7c27d76bd0b9263690a63cd92d6b68c305826318555fd5b084a09970b78.json
 - bench/cassettes/0fada0c0a103882e2895f7f455d73d2cbe2b4a81bf743bb19b2b4f6e0130126b.json
+- bench/cassettes/1037e0b63ba790c02d14d0ddfc18b68151c11d03ce88d764658044b6bab2e55b.json
 - bench/cassettes/1117d79ac7c43b21beaf75e510461f91cd4ecc3cdbbe461f1794644af4aaf8f3.json
 - bench/cassettes/11b7444a074cca4d0ef1535db0cf0edefe4c34eac4be3c593c9c2f33d3b038b8.json
 - bench/cassettes/11d00371d1b993675be00463159164536f1f8ed90a48652b0935eb9c36f8b684.json
 - bench/cassettes/1456ebe12616ce961d68a14eada62778a7271ebd3d1b4cb6eefd6857651c7afc.json
 - bench/cassettes/14c3693f64b0e9ea7e470277fccae6ab501d404185eec5bb9074ef4b75c572c0.json
 - bench/cassettes/14fe9f6d8055d41f5d970dc32ab640c213ea8782d365c82d32b9884ba4f28c67.json
+- bench/cassettes/19bba849a846d30a69dc385a337d67704fbe5194506fd3c914d3270d86495fde.json
 - bench/cassettes/1a1d8557e4698426c2467aa573a240357715d3d9e3fa77bf066672a8b6bf9a86.json
 - bench/cassettes/1b5a8aaaf98305560588246cd51f53c6c185a951f8018c26bf588954d9de1cea.json
 - bench/cassettes/1fe596be747e33e9c7e4d7c50de32613957ac66bee43679b4bed96c551b1fd0a.json
@@ -116,6 +118,7 @@ files:
 - bench/cassettes/2f6576325dfab696cec8d8b35fa89c29960bee15d10dab31c91cef507f4842be.json
 - bench/cassettes/33ec27d22c0f63dbbe93008f50af07fa8667a5c806a9fbfc7897683915412b40.json
 - bench/cassettes/351f745b0c47e0cf3272b0fea7fc4941b6aa733f013c55a9a19707b542543200.json
+- bench/cassettes/3768847dc57df8bf2328771e7763c0b8bd013a167244e7849967dac144420601.json
 - bench/cassettes/37d974fdc0c1876413111c244d363c8d2306e3229fec668c158b5c924ffdb983.json
 - bench/cassettes/39524467463dd612cbf03e314b9cfd8cbe1c5f06969da2289359cf7cf83de5da.json
 - bench/cassettes/3f2c758c111ab7f1163ba579f2329718afe15620507d356d3ebe2a053df237ea.json
@@ -294,6 +297,7 @@ files:
 - lib/truffler/registry.rb
 - lib/truffler/search/encoding.rb
 - lib/truffler/search/encoding_cache.rb
+- lib/truffler/search/filler.rb
 - lib/truffler/search/keystroke.rb
 - lib/truffler/search/query.rb
 - lib/truffler/search/result.rb