ask-tokens-rails 0.2.2 → 0.4.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 8670efe472c23b2b9f14c0fe8fe4a941adbb77d5c7e1d0dfa32150c1fdad8049
-  data.tar.gz: d47f088143f600f04d2f89d0a58046f7160f28b841cfef73da3c9616656e8e24
+  metadata.gz: fe4e15b2393197cf4983d5db195ed7e4d513de3593b553b4359f4a5abd114883
+  data.tar.gz: 56f42f53d6899ea2c7fdba9d8e664cf083398abe0b510f207663e817564387b2
 SHA512:
-  metadata.gz: 994761b2ad77166078a7750c92194bd217a67fb4280bbf51bb5e43ec97af10fc1bf0e8781529237b6689f43d74857d95235c689927fdb63dee3ae8f87a436676
-  data.tar.gz: a192e5b37cbf5b800b39701d1ab33f390f8b1cfbbce9a20f2e6d65e37a111750e1adb8d01a0f6283b56677b7bfb0859437745ec63b088ba4d2c87a9b83d73e38
+  metadata.gz: bcb48f78eed19e4777b2236b7b0a98a53645173a122845db5f8b26c35c7e9f004f98766889b2b366d94814820e1d7d3cb5fdec9d68081aa28c2e3912ec518b14
+  data.tar.gz: 2f5d0a2ae26cd3997b2964c4e6d804b757382b5db26af238ae2a5cd809e616e3c16d8bfece4446bf289c17d5bca56a0853ee197c0f2ea1b69cfe4b41526b4e80

data/CHANGELOG.md

@@ -1,3 +1,81 @@
+## [0.4.0] — 2026-09-14
+
+### Added
+
+- **Billing-detail columns on the ledger.** `model_id`, `provider`,
+  `input_tokens`, `output_tokens`, `cached_tokens`, `llm_cost_usd`, and
+  `multiplier` — already lifted out of `metadata` by the ActiveRecord store —
+  are now in the install migration, so ledgers can be summed and grouped
+  (model spend, margin) without JSON casts. The columns are optional: the
+  store intersects the extraction keys with the table's actual columns, so
+  installs created before this release keep the detail in `metadata`.
+  Existing installs can add the columns with the equivalent migration.
+- **`credits` scope and `debit?`/`credit?` predicates** on the ledger model,
+  matching the `debits`/`grants` scopes.
+- **`has_token_wallet` macro.** Including `Ask::Tokens::Rails::HasTokenWallet`
+  still works; the railtie also makes the documented `has_token_wallet` macro
+  available on every model.
+- **`expires_at` index** on the transactions table, for the sweep job.
+
+### Fixed
+
+- **`SweepExpiredTokensJob` never loaded and had a broken query.** The job is
+  now required by the railtie, and its grant lookup no longer hardcodes the
+  legacy `token_transactions` table or a nonexistent `source_transaction_id`
+  column (the source grant lives in `metadata`).
+- **Expiry no longer eats a later top-up.** The sweep replays the wallet's
+  ledger first-to-expire-first and removes only each expired grant's unspent
+  remainder, instead of capping at whatever the wallet balance happens to be.
+  Repeated runs are idempotent.
+
+## [0.3.0] — 2026-09-14
+
+### Changed
+
+
+
+- **Default table names are now gem-prefixed: `ask_tokens_wallets` and
+
+  `ask_tokens_transactions`.** They can no longer collide with an app's own
+
+  ledger, so no app configuration is needed. To keep the old names (or adopt
+
+  existing tables), configure them:
+
+
+
+  ```ruby
+
+  Ask::Tokens::Rails.configure do |config|
+
+    config.wallets_table = "token_wallets"
+
+    config.transactions_table = "token_transactions"
+
+  end
+
+  ```
+
+
+
+  Existing installs must either set the configuration above or rename their
+
+  tables in a migration.
+
+
+
+### Added
+
+
+
+- `Ask::Tokens::Rails.configure` and `reset_config!` — configurable wallet
+
+  and transaction table names, honoured by the models and the install
+
+  generator.
+
+
+
 # Changelog
 
 ## [0.1.0] - 2026-08-19

data/README.md

@@ -1,22 +1,31 @@
-# ask-token-usage-rails
+# ask-tokens-rails
 
-ActiveRecord persistence for the [ask-token-usage](https://github.com/ask-rb/ask-token-usage) wallet engine.
+ActiveRecord persistence for the [ask-tokens](https://github.com/ask-rb/ask-tokens) wallet engine.
 
-Ships an ActiveRecord store adapter, a `has_token_wallet` concern, install generator with migrations, and an expiry sweep job.
+Ships an ActiveRecord store adapter, a `has_token_wallet` macro, install generator with migrations, and an expiry sweep job.
 
 ## Installation
 
 ```ruby
 # Gemfile
-gem "ask-token-usage-rails"
+gem "ask-tokens-rails"
 ```
 
 ```sh
 bundle install
-rails g ask_token_usage:install
+rails g ask_tokens:install
 rails db:migrate
 ```
 
+Table names default to `ask_tokens_wallets` and `ask_tokens_transactions`. To adopt existing tables:
+
+```ruby
+Ask::Tokens::Rails.configure do |config|
+  config.wallets_table = "token_wallets"
+  config.transactions_table = "token_transactions"
+end
+```
+
 ## Usage
 
 ```ruby
@@ -35,19 +44,25 @@ end
 # Check balance
 user.token_balance   # => 9_998
 
+# Read the ledger
+user.token_transactions.credits
+user.token_transactions.debits.since(30.days.ago)
+
 # Use the PORO wallet directly
 wallet = user.ask_token_wallet
 wallet.entries
 wallet.used_since(30.days.ago)
 ```
 
+Pass `metadata: {model_id:, provider:, input_tokens:, output_tokens:, cached_tokens:, llm_cost_usd:, multiplier:}` on a grant/deduction and the store lifts those keys into the ledger's billing columns, so model spend and margin can be summed without JSON casts.
+
 ## Scheduled jobs
 
-`SweepExpiredTokensJob` expires grant entries whose `expires_at` has passed.
+`SweepExpiredTokensJob` expires grant entries whose `expires_at` has passed, removing only each grant's unspent remainder (first-to-expire-first allocation). Add an `expires_at` index if your install predates migration column v0.4.0.
 
 ```yaml
 # config/recurring.yml (Solid Queue)
-ask_token_usage_sweep:
-  class: Ask::TokenUsage::Rails::SweepExpiredTokensJob
-  schedule: "every 1 hour"
+ask_tokens_sweep:
+  class: Ask::Tokens::Rails::SweepExpiredTokensJob
+  schedule: every hour
 ```

data/lib/ask/tokens/rails/concerns/has_token_wallet.rb

@@ -5,6 +5,14 @@ require "active_support/concern"
 module Ask
   module Tokens
     module Rails
+      # Included into ActiveRecord::Base by the railtie, so any model can say
+      # `has_token_wallet` — the macro documented in the README.
+      module TokenWalletOwner
+        def has_token_wallet
+          include Ask::Tokens::Rails::HasTokenWallet
+        end
+      end
+
       # Include this concern in any ActiveRecord model to give it a token
       # wallet backed by ask-tokens-rails.
       #

data/lib/ask/tokens/rails/configuration.rb

@@ -0,0 +1,38 @@
+# frozen_string_literal: true
+
+module Ask
+  module Tokens
+    module Rails
+      # Table names are prefixed with the gem's name by default, so they can
+      # never collide with an app's own ledger — no configuration needed.
+      # Override only to adopt existing tables:
+      #
+      #   Ask::Tokens::Rails.configure do |config|
+      #     config.wallets_table = "token_wallets"
+      #     config.transactions_table = "token_transactions"
+      #   end
+      class Configuration
+        attr_accessor :wallets_table, :transactions_table
+
+        def initialize
+          @wallets_table = "ask_tokens_wallets"
+          @transactions_table = "ask_tokens_transactions"
+        end
+      end
+
+      class << self
+        def config
+          @config ||= Configuration.new
+        end
+
+        def configure
+          yield config
+        end
+
+        def reset_config!
+          @config = Configuration.new
+        end
+      end
+    end
+  end
+end

data/lib/ask/tokens/rails/jobs/sweep_expired_tokens_job.rb

@@ -8,67 +8,109 @@ module Ask
       # Expire grant entries whose +expires_at+ has passed. Scheduled as a
       # recurring job (Solid Queue cron, Sidekiq Cron, etc.):
       #
-      #   # config/recurring.yml
+      #   # config/recurring.yml (Solid Queue)
       #   ask_tokens_sweep:
       #     class: Ask::Tokens::Rails::SweepExpiredTokensJob
-      #     schedule: "every 1 hour"
+      #     schedule: every hour
       #
-      # For each expired grant, the actual amount removed is capped to the
-      # wallet's current balance (myrr approximation — correct for most
-      # grant-heavy usage patterns; perfect accuracy requires FIFO allocation
-      # which is a future option).
+      # Grants are consumed first-to-expire-first: a debit draws from the
+      # grant that will expire soonest (non-expiring grants last). Replaying
+      # the wallet's ledger therefore tells us exactly how much of each
+      # expired grant is left unspent — only that remainder is removed, so
+      # expiry can never eat a later top-up. Recorded expiry rows count as
+      # consumption of their source grant, which makes repeated runs
+      # idempotent.
       #
       class SweepExpiredTokensJob < ActiveJob::Base
         queue_as :default
 
         def perform(now: Time.current)
-          Ask::Tokens::TokenTransaction
+          wallet_ids = Ask::Tokens::TokenTransaction
             .grants
             .where("expires_at IS NOT NULL AND expires_at <= ?", now)
-            .where.not(entry_type: "expiry")
-            .where("id NOT IN (SELECT source_transaction_id FROM token_transactions WHERE entry_type = ?)", "expiry")
-            .find_each do |grant|
-              sweep_grant(grant)
-            end
+            .distinct
+            .pluck(:token_wallet_id)
+
+          Ask::Tokens::TokenWallet.where(id: wallet_ids).find_each do |wallet|
+            sweep_wallet(wallet, now: now)
+          end
         end
 
         private
 
-        def sweep_grant(grant)
-          wallet = grant.token_wallet
-          return unless wallet
-
+        def sweep_wallet(wallet, now:)
           wallet.with_lock do
-            remaining = grant.amount - already_expired_amount(grant)
-            return if remaining <= 0
-
-            actual = [remaining, wallet.balance].min
-            return if actual <= 0
-
-            Ask::Tokens::TokenTransaction.create!(
-              token_wallet_id: wallet.id,
-              entry_type: "expiry",
-              amount: -actual,
-              reason: "token_expiry",
-              balance: wallet.balance - actual,
-              metadata: {
-                source_transaction_id: grant.id,
-                expired_amount: remaining,
-                actual_expired: actual
-              },
-              created_at: now
-            )
-
-            wallet.update_column(:balance, wallet.balance - actual)
+            expired = expired_remainders(wallet, now: now)
+            return if expired.empty?
+
+            balance = wallet.balance
+            expired.each do |grant, remaining|
+              actual = [remaining, balance].min
+              next if actual <= 0
+
+              Ask::Tokens::TokenTransaction.create!(
+                token_wallet_id: wallet.id,
+                entry_type: "expiry",
+                amount: -actual,
+                reason: "token_expiry",
+                balance: balance - actual,
+                metadata: {
+                  source_transaction_id: grant.id,
+                  expired_amount: remaining,
+                  actual_expired: actual
+                },
+                created_at: now
+              )
+              balance -= actual
+            end
+
+            wallet.update_column(:balance, balance)
           end
         end
 
-        def already_expired_amount(grant)
-          Ask::Tokens::TokenTransaction
-            .where(token_wallet_id: grant.token_wallet_id, entry_type: "expiry")
-            .where("metadata->>'source_transaction_id' = ?", grant.id.to_s)
-            .sum(:amount)
-            .abs
+        # [[grant, unspent_remainder], ...] for every grant whose window has
+        # closed and that still has tokens left after replaying the wallet's
+        # ledger.
+        def expired_remainders(wallet, now:)
+          rows = Ask::Tokens::TokenTransaction
+            .where(token_wallet_id: wallet.id)
+            .order(:created_at, :id)
+            .to_a
+
+          lots = rows
+            .select { |row| row.entry_type == "grant" }
+            .sort_by { |grant| [grant.expires_at ? 0 : 1, grant.expires_at || grant.created_at, grant.id] }
+            .map { |grant| {grant: grant, remaining: grant.amount} }
+          by_grant_id = lots.index_by { |lot| lot[:grant].id }
+
+          rows.each do |row|
+            case row.entry_type
+            when "debit"
+              consume(lots, -row.amount)
+            when "expiry"
+              lot = by_grant_id[row.metadata["source_transaction_id"].to_i]
+              consume_lot(lot, -row.amount) if lot
+            end
+          end
+
+          lots
+            .select { |lot| lot[:remaining] > 0 && lot[:grant].expires_at && lot[:grant].expires_at <= now }
+            .map { |lot| [lot[:grant], lot[:remaining]] }
+        end
+
+        def consume(lots, amount)
+          lots.each do |lot|
+            amount = consume_lot(lot, amount)
+            break if amount <= 0
+          end
+        end
+
+        def consume_lot(lot, amount)
+          return amount if lot.nil? || amount <= 0
+
+          taken = [lot[:remaining], amount].min
+          lot[:remaining] -= taken
+          amount - taken
         end
       end
     end

data/lib/ask/tokens/rails/models/token_transaction.rb

@@ -1,13 +1,16 @@
 # frozen_string_literal: true
 
 require "active_record"
+require_relative "../configuration"
 
 module Ask
   module Tokens
     # Append-only ledger row. Every grant, debit, adjustment, or expiry is
     # written here as an immutable record. Never updated or deleted.
     class TokenTransaction < ::ActiveRecord::Base
-      self.table_name = "token_transactions"
+      def self.table_name
+        Ask::Tokens::Rails.config.transactions_table
+      end
 
       belongs_to :token_wallet, class_name: "Ask::Tokens::TokenWallet",
                                 foreign_key: :token_wallet_id,
@@ -22,9 +25,19 @@ module Ask
       scope :debits, -> { where(entry_type: "debit") }
       scope :adjustments, -> { where(entry_type: "adjustment") }
       scope :expiries, -> { where(entry_type: "expiry") }
+      # Everything that adds tokens to a wallet (grants and adjustments).
+      scope :credits, -> { where(entry_type: %w[grant adjustment]) }
       scope :since, ->(time) { where("created_at >= ?", time) }
       scope :newest_first, -> { order(created_at: :desc) }
 
+      def debit?
+        entry_type == "debit"
+      end
+
+      def credit?
+        %w[grant adjustment].include?(entry_type)
+      end
+
       validate :immutable_after_creation, on: :update
 
       private

data/lib/ask/tokens/rails/models/token_wallet.rb

@@ -1,12 +1,15 @@
 # frozen_string_literal: true
 
 require "active_record"
+require_relative "../configuration"
 
 module Ask
   module Tokens
     # Polymorphic wallet record. One per owner. Holds the cached balance.
     class TokenWallet < ::ActiveRecord::Base
-      self.table_name = "token_wallets"
+      def self.table_name
+        Ask::Tokens::Rails.config.wallets_table
+      end
 
       has_many :token_transactions, class_name: "Ask::Tokens::TokenTransaction",
                                    foreign_key: :token_wallet_id,

data/lib/ask/tokens/rails/railtie.rb

@@ -11,11 +11,18 @@ module Ask
           require "ask/tokens/rails/models/token_transaction"
           require "ask/tokens/rails/stores/active_record_store"
           require "ask/tokens/rails/concerns/has_token_wallet"
+          require "ask/tokens/rails/jobs/sweep_expired_tokens_job"
 
           Ask::Tokens.configure do |c|
             c.store = Ask::Tokens::Rails::ActiveRecordStore.new
           end
         end
+
+        initializer "ask_tokens.has_token_wallet_macro" do
+          ActiveSupport.on_load(:active_record) do
+            extend Ask::Tokens::Rails::TokenWalletOwner
+          end
+        end
       end
     end
   end

data/lib/ask/tokens/rails/stores/active_record_store.rb

@@ -94,10 +94,14 @@ module Ask
         # Extract well-known metadata keys into their matching model columns.
         # Only writes keys that exist in metadata, so grants/debits without
         # LLM context are unaffected. Symbol and string keys are both handled.
+        # Intersected with the table's actual columns so installs created
+        # before the billing columns existed keep the detail in metadata.
         COLUMN_KEYS = %i[model_id provider input_tokens output_tokens cached_tokens llm_cost_usd multiplier].freeze
 
         def extract_columns_from_metadata(meta)
-          COLUMN_KEYS.each_with_object({}) do |key, hash|
+          keys = COLUMN_KEYS & Ask::Tokens::TokenTransaction.column_names.map(&:to_sym)
+
+          keys.each_with_object({}) do |key, hash|
             value = meta[key] || meta[key.to_s]
             hash[key] = value if value
           end

data/lib/ask/tokens/rails/version.rb

@@ -3,7 +3,7 @@
 module Ask
   module Tokens
     module Rails
-      VERSION = "0.2.2"
+      VERSION = "0.4.0"
     end
   end
 end

data/lib/ask-tokens-rails.rb

@@ -2,4 +2,5 @@
 
 require "ask-tokens"
 require "ask/tokens/rails/version"
+require "ask/tokens/rails/configuration"
 require "ask/tokens/rails/railtie"

data/lib/generators/ask_tokens/install/install_generator.rb

@@ -1,5 +1,6 @@
 # frozen_string_literal: true
 
+require "ask-tokens-rails"
 require "rails/generators"
 require "rails/generators/migration"
 

data/lib/generators/ask_tokens/install/templates/migration.rb

@@ -1,25 +1,37 @@
 # frozen_string_literal: true
 
-class CreateTokenWallets < ActiveRecord::Migration[7.0]
+class CreateAskTokensTables < ActiveRecord::Migration[7.0]
   def change
-    create_table :token_wallets do |t|
+    create_table :<%= Ask::Tokens::Rails.config.wallets_table %> do |t|
       t.references :owner, polymorphic: true, null: false
       t.bigint :balance, null: false, default: 0
       t.timestamps
     end
-    add_index :token_wallets, %i[owner_type owner_id], unique: true, name: "idx_token_wallets_owner"
+    add_index :<%= Ask::Tokens::Rails.config.wallets_table %>, %i[owner_type owner_id], unique: true,
+      name: "idx_<%= Ask::Tokens::Rails.config.wallets_table %>_owner"
 
-    create_table :token_transactions do |t|
-      t.references :token_wallet, null: false, foreign_key: true
+    create_table :<%= Ask::Tokens::Rails.config.transactions_table %> do |t|
+      t.references :token_wallet, null: false, foreign_key: {to_table: Ask::Tokens::Rails.config.wallets_table}
       t.string :entry_type, null: false
       t.bigint :amount, null: false
       t.string :reason, null: false
       t.jsonb :metadata, null: false, default: {}
       t.bigint :balance, null: false
       t.datetime :expires_at
+      # Billing detail the ActiveRecord store lifts out of :metadata when
+      # present. Columns exist so ledgers can be summed and grouped (model
+      # spend, margin) without JSON casts.
+      t.string :model_id
+      t.string :provider
+      t.bigint :input_tokens
+      t.bigint :output_tokens
+      t.bigint :cached_tokens
+      t.decimal :llm_cost_usd, precision: 12, scale: 8
+      t.decimal :multiplier, precision: 6, scale: 4, default: 1.0, null: false
       t.datetime :created_at, null: false
     end
-    add_index :token_transactions, %i[token_wallet_id created_at]
-    add_index :token_transactions, :entry_type
+    add_index :<%= Ask::Tokens::Rails.config.transactions_table %>, %i[token_wallet_id created_at]
+    add_index :<%= Ask::Tokens::Rails.config.transactions_table %>, :entry_type
+    add_index :<%= Ask::Tokens::Rails.config.transactions_table %>, :expires_at
   end
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: ask-tokens-rails
 version: !ruby/object:Gem::Version
-  version: 0.2.2
+  version: 0.4.0
 platform: ruby
 authors:
 - Kaka Ruto
@@ -93,6 +93,7 @@ files:
 - README.md
 - lib/ask-tokens-rails.rb
 - lib/ask/tokens/rails/concerns/has_token_wallet.rb
+- lib/ask/tokens/rails/configuration.rb
 - lib/ask/tokens/rails/jobs/sweep_expired_tokens_job.rb
 - lib/ask/tokens/rails/models/token_transaction.rb
 - lib/ask/tokens/rails/models/token_wallet.rb