llm_cost_tracker 0.6.0 → 0.6.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: fa3f705baf280c2c2239b2dab3522fe7db9b60e26060b00fc08dcc039117da83
-  data.tar.gz: ea34bdad7cb0d7c9fb3233b40b609cea1361ded833ad130c4a7a7ce559b34758
+  metadata.gz: 6942269ea8e3ce5f22bb30a1dc6f6c73ee056b33f4f5d7697d1749ce5742ca93
+  data.tar.gz: 1b0712cfcbea56ee0e03dd1cb6c400969fc485c5648f75616140a62f6e82fc02
 SHA512:
-  metadata.gz: 03c55866b522b36b0728f73fd0ae0075e0a42faa6f80c429865f120d2c597e0b0996665b6a82528c566a0f8554d7636dfd03d9ab600441dafd5a4f0233d3f56b
-  data.tar.gz: af01c4554912d80276bf54ed97aa379c2eaed791fa357ca135aa008fdcbdd41e365c236ac74c9ceb1982c0fbbf2538bc569df1690acb4b5758895dd48c4497b5
+  metadata.gz: 9793cc211dd08669deecf4a1d6a242ec4f2fff3fc41922472d158002731f4db40d618163d6fb054eeb36cae4e8aa708a0055a0fe1820faf88081b26de407107a
+  data.tar.gz: 05ed21f8309b16475b4ca7a0a938c999336f0ebb09baf897225422c69dd8e7e81f73ef01e69ff2d4c80d392f49230c212a87ec5c925c6e242b24942c5369f37f

data/CHANGELOG.md

@@ -4,6 +4,12 @@ Format: [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). Versioning: [S
 
 ## [Unreleased]
 
+## [0.6.1] - 2026-04-29
+
+### Fixed
+
+- Exclude repository documentation from the published gem package.
+
 ## [0.6.0] - 2026-04-29
 
 ### Added

data/lib/llm_cost_tracker/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module LlmCostTracker
-  VERSION = "0.6.0"
+  VERSION = "0.6.1"
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: llm_cost_tracker
 version: !ruby/object:Gem::Version
-  version: 0.6.0
+  version: 0.6.1
 platform: ruby
 authors:
 - Sergii Khomenko
@@ -283,23 +283,6 @@ files:
 - app/views/llm_cost_tracker/tags/index.html.erb
 - app/views/llm_cost_tracker/tags/show.html.erb
 - config/routes.rb
-- docs/architecture.md
-- docs/budgets.md
-- docs/configuration.md
-- docs/cookbook.md
-- docs/dashboard-overview.png
-- docs/dashboard.md
-- docs/extending.md
-- docs/operations.md
-- docs/pricing.md
-- docs/querying.md
-- docs/streaming.md
-- docs/technical/README.md
-- docs/technical/data-flow.md
-- docs/technical/extension-points.md
-- docs/technical/module-map.md
-- docs/technical/operational-notes.md
-- docs/upgrading.md
 - lib/llm_cost_tracker.rb
 - lib/llm_cost_tracker/active_record_adapter.rb
 - lib/llm_cost_tracker/assets.rb

data/docs/architecture.md

@@ -1,28 +0,0 @@
-# Architecture
-
-LLM Cost Tracker is a provider-agnostic billing ledger. Core code should model durable billing concepts, not the naming quirks of one provider or one model family.
-
-Core vocabulary belongs in provider-neutral terms:
-
-- `input_tokens`
-- `cache_read_input_tokens`
-- `cache_write_input_tokens`
-- `output_tokens`
-- `hidden_output_tokens`
-- `pricing_mode`
-- `provider_response_id`
-
-Provider-specific names belong only at ingestion boundaries: parsers and stream adapters. Those adapters translate raw fields into the canonical ledger vocabulary before data reaches `Tracker`, `Pricing`, storage, dashboard services, or reports.
-
-Pricing logic should prefer generic mechanisms over provider branches. Use provider/model price entries only for lookup and rate selection. Use `pricing_mode` plus mode-prefixed price keys for alternate billing modes instead of adding model-specific conditionals.
-
-Tags remain the extension point for app-specific attribution such as tenant, user, feature, trace, job, workflow, or agent session. Do not promote those dimensions into first-class columns unless the ledger itself needs them for provider-agnostic billing behavior.
-
-Hot-path guardrails must not aggregate over the growing call ledger. ActiveRecord period budgets should read maintained rows in `llm_cost_tracker_period_totals`; dashboard analytics may run grouped queries because they are user-initiated reporting paths. Do not add dashboard-only aggregate tables until bounded indexed reads from `llm_api_calls` are no longer enough for the supported date range.
-
-## Technical Docs
-
-- [Module map](technical/module-map.md)
-- [Data flow](technical/data-flow.md)
-- [Extension points](technical/extension-points.md)
-- [Operational notes](technical/operational-notes.md)

data/docs/budgets.md

@@ -1,45 +0,0 @@
-# Budgets and Guardrails
-
-Budgets are safety rails for a Rails app using LLMs in production. They are not
-invoice reconciliation and they are not a transactional quota system.
-
-The full behavior reference is moving here from the README: monthly, daily, and
-per-call budgets; notification payloads; preflight behavior; and failure modes.
-
-## Canonical Sources
-
-Until this page is expanded, use:
-
-- [Budgets](../README.md#budgets)
-- [Known limitations](../README.md#known-limitations)
-- [Operations](operations.md)
-
-## Behaviors
-
-- `:notify`: call `on_budget_exceeded` after a priced event crosses a limit.
-- `:raise`: record the event, then raise `BudgetExceededError`.
-- `:block_requests`: preflight future calls when stored period totals are
-  already over budget.
-
-```ruby
-config.monthly_budget = 500.00
-config.daily_budget = 50.00
-config.per_call_budget = 2.00
-config.budget_exceeded_behavior = :block_requests
-```
-
-`:block_requests` needs ActiveRecord storage for shared period totals. Under
-concurrency it stops the next request after overspend is visible; it does not
-make provider spend transactional.
-
-## Error Payload
-
-`BudgetExceededError` exposes:
-
-- `budget_type`
-- `total`
-- `budget`
-- `monthly_total`
-- `daily_total`
-- `call_cost`
-- `last_event`

data/docs/configuration.md

@@ -1,65 +0,0 @@
-# Configuration
-
-Configuration is the contract between the host app and the ledger: where events
-go, which integrations are enabled, how attribution is attached, and how the app
-reacts when storage, pricing, or budgets need attention.
-
-The full option reference is moving here from the README. Until that migration is
-complete, the README anchors below remain canonical.
-
-## Canonical Sources
-
-Until this page is expanded, use:
-
-- [Quickstart](../README.md#quickstart)
-- [Capturing calls](../README.md#capturing-calls)
-- [Tags](../README.md#tags-who-burned-this-money)
-- [Pricing](../README.md#pricing)
-- [Budgets](../README.md#budgets)
-
-## Scope
-
-This page is scoped to:
-
-- `storage_backend`: `:log`, `:active_record`, and `:custom`; ActiveRecord capture uses a durable inbox when the ingestion migration is present
-- `default_tags`: static tags and per-request callable tags
-- `instrument`: RubyLLM and official SDK integrations
-- `prices_file` and `pricing_overrides`
-- `monthly_budget`, `daily_budget`, and `per_call_budget`
-- `budget_exceeded_behavior`
-- `storage_error_behavior`
-- `unknown_pricing_behavior`
-- `openai_compatible_providers`
-- `report_tag_breakdowns`
-
-## Minimal Production Config
-
-```ruby
-LlmCostTracker.configure do |config|
-  config.storage_backend = :active_record
-  config.default_tags = -> { { environment: Rails.env } }
-  config.prices_file = Rails.root.join("config/llm_cost_tracker_prices.yml")
-  config.instrument :openai
-end
-```
-
-Keep configuration at boot. Mutable shared settings are frozen after
-`configure` returns so request-time code cannot silently change global tracking
-behavior.
-
-Enabled SDK integrations are fail-fast. The client gem must be loaded, meet the
-minimum supported version, and expose the expected classes and methods before
-LLM Cost Tracker installs its wrapper.
-
-## Capture Verification
-
-After boot, run:
-
-```bash
-bin/rails llm_cost_tracker:verify_capture
-```
-
-For ActiveRecord storage, the task records a synthetic manual event inside a
-rollback and checks that notifications and persistence both work. For log and
-custom storage, it reports the configured capture path without invoking external
-sinks.

data/docs/cookbook.md

@@ -1,185 +0,0 @@
-# Cookbook
-
-Short integration recipes for common Ruby clients. Prefer SDK integrations or middleware. Use `track` and `track_stream` only as fallback helpers for unsupported clients.
-
-| Client | Best path | Why |
-|---|---|---|
-| RubyLLM | `config.instrument :ruby_llm` | The integration wraps RubyLLM's provider layer without adding a third-party instrumentation gem. |
-| Official `openai` gem | `config.instrument :openai` | The integration wraps SDK resource methods without changing call sites. |
-| Official `anthropic` gem | `config.instrument :anthropic` | The integration records returned message usage without changing call sites. |
-| `ruby-openai` | Faraday middleware | The client is built on Faraday and accepts middleware via the constructor block. |
-| OpenAI-compatible proxy | Faraday middleware | Use `ruby-openai` or a direct Faraday client against the proxy host. |
-| Custom Faraday client | Faraday middleware | The middleware can parse known provider responses automatically. |
-| Other clients | Adapter first, fallback helpers second | Add a stable integration instead of scattering per-call ledger code. |
-
-## RubyLLM
-
-Enable the integration once, then keep normal RubyLLM calls unchanged.
-
-```ruby
-LlmCostTracker.configure do |config|
-  config.instrument :ruby_llm
-end
-
-LlmCostTracker.with_tags(feature: "support_chat") do
-  RubyLLM.chat.ask("Hello")
-  RubyLLM.embed("text to embed")
-end
-```
-
-The RubyLLM integration supports `ruby_llm >= 1.14.1` and checks RubyLLM's provider contract at boot. Chat, embedding, and transcription calls are captured. Image generation, moderation, and tool execution are not recorded as separate ledger rows.
-
-## Official OpenAI SDK
-
-Enable the integration once, then keep normal `openai` gem calls unchanged.
-
-```ruby
-LlmCostTracker.configure do |config|
-  config.instrument :openai
-end
-
-client = OpenAI::Client.new(api_key: ENV["OPENAI_API_KEY"])
-
-client.responses.create(model: "gpt-4o", input: "Hello")
-client.chat.completions.create(
-  model: "gpt-4o",
-  messages: [{ role: "user", content: "Hello" }]
-)
-
-client.responses.stream(model: "gpt-4o", input: "Hello").each do |event|
-  puts event.type
-end
-
-client.responses.stream_raw(model: "gpt-4o", input: "Hello").each do |event|
-  puts event.type
-end
-
-client.chat.completions.stream_raw(
-  model: "gpt-4o",
-  messages: [{ role: "user", content: "Hello" }],
-  stream_options: { include_usage: true }
-).each do |event|
-  puts event
-end
-```
-
-The OpenAI SDK integration supports `openai >= 0.59.0`. Streaming calls are recorded after the returned stream is consumed. Chat Completions streams need `stream_options: { include_usage: true }` for final usage.
-
-## Official Anthropic SDK
-
-Enable the integration once, then keep normal `anthropic` gem calls unchanged.
-
-```ruby
-LlmCostTracker.configure do |config|
-  config.instrument :anthropic
-end
-
-client = Anthropic::Client.new(api_key: ENV["ANTHROPIC_API_KEY"])
-
-client.messages.create(
-  max_tokens: 1024,
-  model: "claude-sonnet-4-5-20250929",
-  messages: [{ role: "user", content: "Hello" }]
-)
-
-client.messages.stream(
-  max_tokens: 1024,
-  model: "claude-sonnet-4-5-20250929",
-  messages: [{ role: "user", content: "Hello" }]
-).each do |event|
-  puts event.type
-end
-
-client.messages.stream_raw(
-  max_tokens: 1024,
-  model: "claude-sonnet-4-5-20250929",
-  messages: [{ role: "user", content: "Hello" }]
-).each do |event|
-  puts event.type
-end
-```
-
-The Anthropic SDK integration supports `anthropic >= 1.36.0`. Streaming calls are recorded after the returned stream is consumed.
-
-## ruby-openai
-
-`ruby-openai` is a community client that occupies the same `OpenAI::Client` constant as the official gem; only one of the two can be loaded. `config.instrument :openai` is for the official gem. For `ruby-openai`, attach the Faraday middleware via the constructor block:
-
-```ruby
-client = OpenAI::Client.new(access_token: ENV["OPENAI_API_KEY"]) do |f|
-  f.use :llm_cost_tracker, tags: { feature: "chat" }
-end
-
-client.chat(
-  parameters: {
-    model: "gpt-4o",
-    messages: [{ role: "user", content: "Hello" }],
-    stream: proc { |chunk, _bytesize| puts chunk.dig("choices", 0, "delta", "content") },
-    stream_options: { include_usage: true }
-  }
-)
-```
-
-Use the constructor block on every client you build, or wrap client creation in your own factory.
-
-## Azure OpenAI
-
-Azure's v1 API works with OpenAI-compatible HTTP shapes, but pricing and deployment names are yours. Register the Azure host, use the Faraday middleware path, and keep Azure-specific prices in `prices_file` or `pricing_overrides`.
-
-```ruby
-LlmCostTracker.configure do |config|
-  config.openai_compatible_providers["my-resource.openai.azure.com"] = "azure_openai"
-end
-
-conn = Faraday.new(url: "https://my-resource.openai.azure.com") do |f|
-  f.use :llm_cost_tracker, tags: { feature: "chat" }
-  f.request :json
-  f.response :json
-  f.adapter Faraday.default_adapter
-end
-
-conn.post(
-  "/openai/v1/responses",
-  { model: "gpt-4o-prod", input: "Hello" },
-  { "api-key" => ENV.fetch("AZURE_OPENAI_API_KEY") }
-)
-```
-
-## Gemini API
-
-Google's official Gemini SDKs do not include Ruby. Use a Faraday client against the REST API so the Gemini parser can capture usage automatically.
-
-```ruby
-conn = Faraday.new(url: "https://generativelanguage.googleapis.com") do |f|
-  f.use :llm_cost_tracker, tags: { feature: "chat" }
-  f.request :json
-  f.response :json
-  f.adapter Faraday.default_adapter
-end
-
-conn.post(
-  "/v1beta/models/gemini-2.5-flash:generateContent?key=#{ENV.fetch("GOOGLE_API_KEY")}",
-  { contents: [{ role: "user", parts: [{ text: "Hello" }] }] }
-)
-```
-
-## LiteLLM proxy
-
-LiteLLM Proxy speaks an OpenAI-compatible HTTP shape, so register the proxy host once and keep using the normal middleware path.
-
-```ruby
-LlmCostTracker.configure do |config|
-  config.openai_compatible_providers["proxy.internal.example"] = "litellm"
-end
-
-client = OpenAI::Client.new(
-  access_token: ENV["LITELLM_API_KEY"],
-  uri_base: "https://proxy.internal.example"
-) do |f|
-  f.use :llm_cost_tracker, tags: { gateway: "litellm" }
-end
-
-client.chat(parameters: { model: "openai/gpt-5-mini", messages: [{ role: "user", content: "Hello" }] })
-```
-
-If your proxy exposes custom model IDs or discounts, add them in `prices_file` or `pricing_overrides`.

data/docs/dashboard.md

@@ -1,38 +0,0 @@
-# Dashboard
-
-The dashboard is a Rails Engine for humans reviewing spend, attribution, and data
-quality. It is server-rendered ERB, has no JavaScript bundle, and reads from the
-host app's `llm_api_calls` table.
-
-The detailed dashboard guide is moving here from the README: mounting, route
-constraints, authentication examples, page map, and operational notes.
-
-## Canonical Sources
-
-Until this page is expanded, use:
-
-- [Dashboard](../README.md#dashboard)
-- [Privacy](../README.md#privacy)
-- [Operations](operations.md)
-
-## Mounting
-
-```ruby
-mount LlmCostTracker::Engine => "/llm-costs"
-```
-
-Use `storage_backend = :active_record` for apps that mount the dashboard.
-
-## Pages
-
-- Overview: spend trend, budget status, anomaly banner, provider rollup, top models
-- Models: spend and usage by provider and model
-- Calls: filterable call ledger with CSV export
-- Tags: tag keys and tag value breakdowns
-- Data Quality: unknown pricing, untagged calls, missing latency, incomplete streams
-
-## Authentication
-
-The gem does not ship dashboard auth. Mount the engine behind the host app's
-existing authentication layer: Devise, basic auth, Cloudflare Access, or your own
-constraints.

data/docs/extending.md

@@ -1,32 +0,0 @@
-# Extending LLM Cost Tracker
-
-Extensions belong at clear boundaries: parsers for response shapes, integrations
-for SDK hooks, pricing files for rates, and custom storage for apps that own
-persistence themselves.
-
-The practical extension guide is moving here from the README. The lower-level
-contracts already live in the technical extension reference.
-
-## Canonical Sources
-
-Until this page is expanded, use:
-
-- [Capturing calls](../README.md#capturing-calls)
-- [Pricing](pricing.md)
-- [Technical extension points](technical/extension-points.md)
-
-## Extension Points
-
-- Custom parser: translate a provider response into `ParsedUsage`.
-- OpenAI-compatible host: register the host-to-provider mapping.
-- Custom storage: receive the canonical `Event` and write it elsewhere.
-- Notifications subscriber: observe `llm_request.llm_cost_tracker`.
-- Local price file: model gateway IDs, contract rates, or unsupported models.
-
-## Parser Boundary
-
-A parser matches request URLs, translates provider response shapes into
-`ParsedUsage`, and returns `nil` when the response is outside its contract.
-
-Do provider-specific translation at this boundary. Keep `Tracker`, storage,
-dashboard, and pricing in canonical ledger terms.

data/docs/operations.md

@@ -1,44 +0,0 @@
-# Operations
-
-Production use is mostly about choosing the right storage backend, keeping the
-database healthy, and understanding where the gem is intentionally best effort.
-
-The operational guide is moving here from the README: retention, tag storage,
-thread safety, connection pools, and deployment notes.
-
-## Canonical Sources
-
-Until this page is expanded, use:
-
-- [Privacy](../README.md#privacy)
-- [Known limitations](../README.md#known-limitations)
-- [Technical operational notes](technical/operational-notes.md)
-
-## Production Defaults
-
-- Use `storage_backend = :active_record` for the shared ledger, dashboard, and
-  cross-process budget guardrails.
-- Size the ActiveRecord connection pool for your app plus ledger writes.
-- Keep `storage_error_behavior = :warn` unless losing the LLM response is better
-  than losing the ledger event.
-- Treat `:block_requests` as a guardrail, not a hard quota.
-- Keep `default_tags` callables fast and thread-safe.
-
-## Retention
-
-Retention is explicit. Use the prune task when the ledger should not grow
-forever:
-
-```bash
-DAYS=90 bin/rails llm_cost_tracker:prune
-```
-
-When ActiveRecord period rollups are installed, pruning decrements the
-affected daily and monthly buckets in the same batch transaction as the ledger
-delete.
-
-## Data Shape
-
-Tags are JSONB with a GIN index on PostgreSQL and JSON text elsewhere. The
-dashboard and query helpers work across supported adapters, but PostgreSQL is the
-strongest path for large tag-heavy ledgers.

data/docs/pricing.md

@@ -1,94 +0,0 @@
-# Pricing and Price Refresh
-
-LLM Cost Tracker prices calls locally from recorded usage and a versioned price
-registry. Providers usually return token counts, not a stable per-request price,
-so the gem stores the calculated cost with each ledger row.
-
-The full pricing reference is moving here from the README: registry shape,
-refresh tasks, precedence, provider-qualified keys, and mode-specific rates.
-
-## Canonical Sources
-
-Until this page is expanded, use:
-
-- [Pricing](../README.md#pricing)
-- [Supported providers](../README.md#supported-providers)
-- [Known limitations](../README.md#known-limitations)
-
-## Registry Rules
-
-- Built-in prices live in `lib/llm_cost_tracker/prices.json`.
-- Local snapshots live wherever `config.prices_file` points.
-- Precedence is `pricing_overrides`, then `prices_file`, then bundled prices.
-- Provider-qualified keys like `openai/gpt-4o-mini` win over model-only keys.
-- Historical rows keep the cost calculated when the call was recorded.
-
-## Refresh Commands
-
-```bash
-bin/rails generate llm_cost_tracker:prices
-bin/rails llm_cost_tracker:prices:refresh
-bin/rails llm_cost_tracker:prices:check
-PROVIDER=openai MODEL=gpt-4o bin/rails llm_cost_tracker:prices:explain
-```
-
-The refresh task reads the maintained LLM Cost Tracker snapshot and writes to
-`ENV["OUTPUT"]`, then `config.prices_file`, then
-`config/llm_cost_tracker_prices.yml`.
-
-## Price Fields
-
-Base fields:
-
-- `input`
-- `output`
-- `cache_read_input`
-- `cache_write_input`
-
-Mode-prefixed fields use the same base terms:
-
-- `batch_input`
-- `batch_output`
-- `priority_input`
-- `batch_cache_read_input`
-
-## Pricing Modes
-
-Pass `pricing_mode: :batch` when usage came from a provider batch job or another
-discounted mode:
-
-```ruby
-LlmCostTracker.track(
-  provider: "openai",
-  model: "gpt-4o",
-  input_tokens: 1_000_000,
-  output_tokens: 250_000,
-  pricing_mode: :batch,
-  feature: "offline_eval"
-)
-```
-
-The calculator uses `batch_input`, `batch_output`, and other matching
-mode-prefixed fields when present, then falls back to the base field for missing
-mode-specific rates.
-
-## Price Explain
-
-Use `prices:explain` when Data Quality shows unknown pricing or a local override
-does not behave as expected:
-
-```bash
-PROVIDER=openai MODEL=gpt-4o PRICING_MODE=batch bin/rails llm_cost_tracker:prices:explain
-```
-
-Optional token env vars let the command check the exact buckets that a call used:
-
-```bash
-PROVIDER=custom MODEL=gateway-model INPUT_TOKENS=1000 OUTPUT_TOKENS=200 CACHE_READ_INPUT_TOKENS=50 bin/rails llm_cost_tracker:prices:explain
-```
-
-The command reports the matched source, matched key, match strategy, effective
-rates, and any missing rate needed to price the event.
-
-Provider-specific pricing pages belong in scrapers and snapshots. Runtime
-pricing should stay in canonical billing terms.

data/docs/querying.md

@@ -1,36 +0,0 @@
-# Querying and Reports
-
-Once calls are in `llm_api_calls`, the host app owns the data. Query it from a
-console, a scheduled job, your admin UI, or the mounted dashboard.
-
-The full querying reference is moving here from the README: ActiveRecord scopes,
-reporting helpers, tag breakdowns, and SQL-side grouping patterns.
-
-## Canonical Sources
-
-Until this page is expanded, use:
-
-- [Querying](../README.md#querying)
-- [Dashboard](dashboard.md)
-- [Operations](operations.md)
-
-## Common Queries
-
-```ruby
-LlmCostTracker::LlmApiCall.today.total_cost
-LlmCostTracker::LlmApiCall.this_month.cost_by_model
-LlmCostTracker::LlmApiCall.this_month.cost_by_provider
-LlmCostTracker::LlmApiCall.this_month.cost_by_tag("feature")
-LlmCostTracker::LlmApiCall.by_tags(user_id: 42, feature: "chat").this_month.total_cost
-LlmCostTracker::LlmApiCall.daily_costs(days: 7)
-```
-
-## Report Task
-
-```bash
-bin/rails llm_cost_tracker:report
-DAYS=7 bin/rails llm_cost_tracker:report
-```
-
-This page is scoped to cost scopes, tag grouping, period grouping, latency
-helpers, unknown-pricing queries, and report tag breakdowns.

data/docs/streaming.md

@@ -1,70 +0,0 @@
-# Streaming Capture
-
-Streaming calls should appear in the ledger instead of disappearing into a live
-callback. LLM Cost Tracker records them when the provider emits final usage or
-when the app supplies explicit totals.
-
-The full streaming reference is moving here from the README: Faraday streaming,
-`track_stream`, provider response IDs, final usage events, and data-quality
-states.
-
-## Canonical Sources
-
-Until this page is expanded, use:
-
-- [Capturing calls](../README.md#capturing-calls)
-- [Known limitations](../README.md#known-limitations)
-- [Cookbook](cookbook.md)
-
-## Faraday Path
-
-The middleware tees Faraday's `on_data` callback, keeps chunks flowing to the
-caller, and records usage when the response completes.
-
-OpenAI streams need final usage:
-
-```ruby
-stream_options: { include_usage: true }
-```
-
-Anthropic and Gemini are parsed from their provider stream event shapes when
-usage is present.
-
-## SDK Path
-
-Official OpenAI and Anthropic SDK streams are captured when `config.instrument`
-is enabled for the provider. The returned stream object is preserved, and usage
-is recorded after the stream is consumed.
-
-```ruby
-config.instrument :openai
-config.instrument :anthropic
-```
-
-Captured SDK helpers:
-
-- OpenAI `responses.stream`, `responses.stream_raw`, `responses.retrieve_streaming`, and `chat.completions.stream_raw`.
-- Anthropic `messages.stream` and `messages.stream_raw`.
-
-OpenAI Chat Completions streams need final usage:
-
-```ruby
-stream_options: { include_usage: true }
-```
-
-## Manual Path
-
-```ruby
-LlmCostTracker.track_stream(provider: "openai", model: "gpt-4o") do |stream|
-  my_client.stream(...) { |event| stream.event(event.to_h) }
-end
-```
-
-If the client already knows totals, skip provider event parsing:
-
-```ruby
-stream.usage(input_tokens: 120, output_tokens: 45)
-```
-
-Missing final usage is stored with `usage_source: "unknown"` so the Data Quality
-page can surface it.

data/docs/technical/README.md

@@ -1,10 +0,0 @@
-# Technical Documentation
-
-These files describe the internal module boundaries for LLM Cost Tracker.
-
-- [Module map](module-map.md)
-- [Data flow](data-flow.md)
-- [Extension points](extension-points.md)
-- [Operational notes](operational-notes.md)
-
-The main rule is simple: provider-specific API shapes stop at ingestion boundaries. The ledger, storage, budgets, dashboard, and reports work with canonical billing concepts.

data/docs/technical/data-flow.md

@@ -1,70 +0,0 @@
-# Data Flow
-
-This is the normal path from an application LLM call to stored ledger data.
-
-## Faraday Requests
-
-1. The host app sends an HTTP request through Faraday.
-2. `LlmCostTracker::Middleware::Faraday` checks whether a parser matches the request URL.
-3. For non-streaming responses, the middleware passes request and response data to the parser.
-4. For streaming responses, the middleware tees `on_data`, collects stream events, and parses final usage when the stream completes.
-5. The parser returns `ParsedUsage` with canonical fields.
-6. `Tracker.record` prices and persists the event.
-
-## SDK Integrations
-
-1. The host app enables an integration with `config.instrument`.
-2. `LlmCostTracker::Integrations` checks the SDK version, target classes, and target methods once at install time.
-3. `LlmCostTracker::Integrations` prepends a narrow wrapper to supported SDK resource methods.
-4. The host app keeps calling the provider SDK normally.
-5. The wrapper measures latency, extracts usage from the SDK response object, and sends canonical fields to `Tracker.record`.
-6. If an explicitly enabled SDK is not loaded or does not satisfy the install contract, boot raises before the app silently misses usage.
-
-## Explicit Tracking
-
-1. The host app calls `LlmCostTracker.track` with known usage totals, or `LlmCostTracker.track_stream` with stream events.
-2. `track` sends manual totals directly to `Tracker.record`.
-3. `track_stream` uses `StreamCollector`, then parser lookup by provider when events need parsing.
-4. `Tracker.record` prices and persists the event.
-
-## Canonical Event Build
-
-`Tracker.record` performs the central normalization step:
-
-1. Blank model identifiers become `unknown`.
-2. Input, output, cache-read, cache-write, hidden-output, and pricing-mode values are extracted from metadata.
-3. `Pricing.cost_for` calculates a `Cost` object or returns `nil` for unknown pricing.
-4. Tags are merged from `with_tags`, `default_tags`, middleware tags, and explicit metadata.
-5. An `Event` is created and emitted through `ActiveSupport::Notifications`.
-6. The configured storage backend receives the event.
-7. Budget checks run unless storage explicitly returns `false`.
-
-## ActiveRecord Storage
-
-1. `Storage::ActiveRecordInbox.save` writes a compact durable event row when the ingestion tables are present.
-2. `Storage::ActiveRecordIngestor` claims retryable inbox rows through a database lease and writes batches into `llm_api_calls`.
-3. `Storage::ActiveRecordStore.insert_many` converts tags for JSON or text storage and writes optional fields only when their columns exist.
-4. The call rows, period rollup updates, and inbox deletes happen in one transaction.
-5. `ActiveRecordRollups.increment_many!` updates daily and monthly totals only for rows inserted by the batch.
-6. Budget reads use period totals plus pending inbox totals when available.
-
-The inbox write is the durability boundary. Ledger freshness is eventually consistent unless the caller explicitly waits with `LlmCostTracker.flush!`.
-
-## Dashboard Reads
-
-1. Controllers build a filtered `LlmApiCall` scope.
-2. Dashboard services run targeted aggregate queries.
-3. Helpers render filters, charts, pagination, CSV links, and numeric formatting.
-4. Views render plain ERB with the engine CSS asset.
-
-Dashboard reads do not mutate ledger state. They can be heavier than request-time code, but they still need explicit grouping and indexes.
-
-## Pricing Refresh
-
-1. `llm_cost_tracker:prices:refresh` chooses `ENV["OUTPUT"]`, then `config.prices_file`, then `config/llm_cost_tracker_prices.yml`.
-2. `PriceSync::Fetcher` fetches the maintained LLM Cost Tracker price snapshot.
-3. `PriceSync` validates schema compatibility, gem-version compatibility, and model price shape.
-4. `RegistryWriter` writes a local JSON or YAML registry.
-5. Runtime pricing reloads the local file when its mtime changes.
-
-The gem never fetches pricing from the network during normal request tracking.

data/docs/technical/extension-points.md

@@ -1,111 +0,0 @@
-# Extension Points
-
-Extensions should plug into existing provider-agnostic boundaries. If a new feature needs a provider-specific branch outside ingestion code, revisit the design first.
-
-## Custom Parsers
-
-Use parser registration when a provider or gateway has a response shape the built-ins do not cover.
-
-Expected parser contract:
-
-- `match?(url)` detects supported request URLs.
-- `parse(request_url, request_body, response_status, response_body)` returns `ParsedUsage` or `nil`.
-- `parse_stream(request_url, request_body, response_status, events)` returns `ParsedUsage` or `nil`.
-- `streaming_request?(request_url, request_body)` detects streaming requests when the provider does not use a simple `stream: true` field.
-- `provider_names` returns provider names that can be used by `track_stream(provider: ...)`.
-
-Use `Parsers::Base` helpers for URL matching and stream-event extraction. Use `Parsers::OpenaiUsage` only for OpenAI-shaped usage hashes.
-
-## SDK Integrations
-
-Use SDK integrations when a popular Ruby client does not expose a Faraday middleware stack but returns stable usage objects. RubyLLM and the official `openai` and `anthropic` gems qualify. Faraday-based clients that expose a middleware hook, such as `ruby-openai`'s constructor block, are covered by the Faraday middleware instead. Clients with no stable hook must use the explicit `track` / `track_stream` fallback until an integration exists.
-
-Expected integration contract:
-
-- no hard dependency on the provider SDK
-- fail-fast boot when an explicitly enabled SDK is missing or below the minimum supported version
-- install-time checks for the target classes and methods
-- idempotent `Module#prepend` around narrow resource methods
-- no tracking when the integration is not enabled in configuration
-- canonical usage fields passed to `Tracker.record`
-
-SDK integrations belong under `LlmCostTracker::Integrations`. Do not put SDK object-shape handling in parsers, storage, or pricing.
-
-External integrations can register an adapter with
-`LlmCostTracker::Integrations.register(:name, adapter)`. The adapter must
-respond to `install` and `status`, and enabled names are still selected through
-`config.instrument`.
-
-## OpenAI-Compatible Gateways
-
-Use `config.openai_compatible_providers` when a gateway speaks the OpenAI request and response shape.
-
-This is for shape compatibility, not pricing. Gateway-specific model IDs or discounts belong in `prices_file` or `pricing_overrides`.
-
-## Prices
-
-Use `config.prices_file` for the app's source-controlled price snapshot.
-
-Use `config.pricing_overrides` for urgent or environment-specific overrides that are easier to keep in Ruby.
-
-Supported canonical keys:
-
-- `input`
-- `output`
-- `cache_read_input`
-- `cache_write_input`
-- `batch_input`
-- `batch_output`
-- mode-prefixed keys such as `priority_input` or `batch_cache_read_input`
-
-Provider-specific pricing details must be translated before they reach runtime pricing.
-
-## Tags
-
-Tags are the extension point for application attribution:
-
-- tenant
-- user
-- feature
-- trace
-- job
-- workflow
-- agent session
-
-Use `config.default_tags`, middleware `tags:`, explicit metadata, and `LlmCostTracker.with_tags`. Do not add first-class columns for app dimensions unless the ledger needs that field for provider-agnostic billing behavior.
-
-## Storage
-
-Use `storage_backend = :custom` only when the host app needs to own persistence completely.
-
-Custom storage receives a canonical `Event`. Returning `false` tells the tracker not to run budget checks for that event.
-
-ActiveRecord storage is the production path for dashboards and cross-process budgets.
-
-Storage adapters can register with
-`LlmCostTracker::Storage.register(:name, backend)`. A backend must respond to
-`save(event)` and may expose `verify` for capture diagnostics.
-
-## Dashboard
-
-Dashboard additions should be read-only services under `app/services/llm_cost_tracker/dashboard`.
-
-Keep controller actions thin:
-
-- parse params
-- build filtered scope
-- call services
-- render views
-
-Keep view logic in helpers when it is reused across pages. Do not add JavaScript for dashboard behavior.
-
-## Generators
-
-Generators are installation contracts. New generator behavior should be:
-
-- additive when possible
-- idempotent where Rails generator APIs allow it
-- explicit about destructive or table-rewriting operations
-- covered by generator template specs
-
-Fresh install templates and upgrade generators should stay aligned. If a fresh install gains a column or index, the upgrade path needs a generator unless the next release intentionally makes a breaking install path.

data/docs/technical/module-map.md

@@ -1,197 +0,0 @@
-# Module Map
-
-LLM Cost Tracker is organized around a small set of durable responsibilities. File layout does not need to mirror these modules perfectly, but new code should fit one of these boundaries.
-
-## Public API and Configuration
-
-Primary files:
-
-- `lib/llm_cost_tracker.rb`
-- `lib/llm_cost_tracker/configuration.rb`
-- `lib/llm_cost_tracker/tag_context.rb`
-- `lib/llm_cost_tracker/doctor.rb`
-- `lib/llm_cost_tracker/logging.rb`
-- `lib/llm_cost_tracker/errors.rb`
-
-Responsibilities:
-
-- Expose `configure`, `track`, `track_stream`, `with_tags`, and `enforce_budget!`.
-- Keep configuration immutable after `configure` returns.
-- Merge scoped tags and default tags without leaking state across threads.
-- Report installation and pricing health through `llm_cost_tracker:doctor`.
-
-This module should stay small. It can orchestrate other modules, but it should not contain provider parsing, SQL details, dashboard aggregation, or pricing-source logic.
-
-## SDK Integrations
-
-Primary files:
-
-- `lib/llm_cost_tracker/integrations/*`
-
-Responsibilities:
-
-- Add optional instrumentation for Ruby SDKs without adding provider SDK dependencies.
-- Install narrow, idempotent `Module#prepend` wrappers around stable SDK resource methods.
-- Extract SDK response objects into canonical usage fields.
-- Keep SDK-specific object handling out of `Tracker` and storage.
-
-Integrations are for Ruby SDK object shapes. Parsers are for HTTP and stream payload shapes.
-
-## Ingestion
-
-Primary files:
-
-- `lib/llm_cost_tracker/middleware/faraday.rb`
-- `lib/llm_cost_tracker/stream_collector.rb`
-- `lib/llm_cost_tracker/parsed_usage.rb`
-- `lib/llm_cost_tracker/request_url.rb`
-- `lib/llm_cost_tracker/parsers/*`
-
-Responsibilities:
-
-- Detect supported LLM HTTP requests.
-- Parse provider responses and stream events into `ParsedUsage`.
-- Translate provider-specific fields into canonical usage fields.
-- Preserve app streaming behavior while teeing events for tracking.
-
-Provider-specific code belongs here. The output boundary is `ParsedUsage`, not raw provider JSON.
-
-## Canonical Ledger
-
-Primary files:
-
-- `lib/llm_cost_tracker/tracker.rb`
-- `lib/llm_cost_tracker/event.rb`
-- `lib/llm_cost_tracker/event_metadata.rb`
-- `lib/llm_cost_tracker/usage_breakdown.rb`
-- `lib/llm_cost_tracker/cost.rb`
-- `lib/llm_cost_tracker/unknown_pricing.rb`
-
-Responsibilities:
-
-- Normalize provider, model, usage, tags, latency, streaming flags, and response IDs.
-- Price canonical usage through `Pricing`.
-- Emit `ActiveSupport::Notifications`.
-- Persist the event through the configured storage backend.
-- Run budget checks after successful storage.
-
-This module must remain provider-agnostic. It should never branch on a specific provider model family.
-
-## Pricing
-
-Primary files:
-
-- `lib/llm_cost_tracker/pricing.rb`
-- `lib/llm_cost_tracker/price_registry.rb`
-- `lib/llm_cost_tracker/price_freshness.rb`
-- `lib/llm_cost_tracker/prices.json`
-- `lib/llm_cost_tracker/price_sync/*`
-- `lib/tasks/llm_cost_tracker.rake`
-
-Responsibilities:
-
-- Load bundled prices, local price snapshots, and Ruby overrides.
-- Apply pricing precedence: `pricing_overrides`, `prices_file`, bundled prices.
-- Calculate costs from canonical usage fields.
-- Update local snapshots from the maintained LLM Cost Tracker price registry.
-- Validate snapshot schema compatibility, gem-version compatibility, and price entry shape.
-
-Pricing refresh must not perform boot-time or request-time network work. Runtime pricing uses bundled prices, local files, and in-memory caches.
-
-## Storage
-
-Primary files:
-
-- `lib/llm_cost_tracker/llm_api_call.rb`
-- `lib/llm_cost_tracker/period_total.rb`
-- `lib/llm_cost_tracker/llm_api_call_metrics.rb`
-- `lib/llm_cost_tracker/storage/active_record_store.rb`
-- `lib/llm_cost_tracker/storage/active_record_rollups.rb`
-- `lib/llm_cost_tracker/storage/registry.rb`
-- `lib/llm_cost_tracker/tags_column.rb`
-- `lib/llm_cost_tracker/tag_key.rb`
-- `lib/llm_cost_tracker/tag_sql.rb`
-- `lib/llm_cost_tracker/tag_query.rb`
-- `lib/llm_cost_tracker/tag_accessors.rb`
-- `lib/llm_cost_tracker/period_grouping.rb`
-
-Responsibilities:
-
-- Persist canonical events into ActiveRecord.
-- Hide database-specific tag storage differences.
-- Maintain period rollups for hot-path budget reads.
-- Provide safe scopes for filters, periods, tags, unknown pricing, and reports.
-
-Storage can know about database adapters and optional columns. It should not parse provider responses or fetch price data.
-
-## Budgets and Retention
-
-Primary files:
-
-- `lib/llm_cost_tracker/budget.rb`
-- `lib/llm_cost_tracker/retention.rb`
-- `lib/llm_cost_tracker/storage/active_record_rollups.rb`
-
-Responsibilities:
-
-- Enforce monthly, daily, and per-call guardrails.
-- Support preflight blocking where ActiveRecord rollups are available.
-- Prune old ledger rows in batches.
-- Keep budget checks bounded by maintained aggregates, not by full ledger scans.
-
-Budget behavior is part of the hot path. Any change here must be measured against per-request overhead.
-
-## Dashboard and Reporting
-
-Primary files:
-
-- `lib/llm_cost_tracker/report*.rb`
-- `app/controllers/llm_cost_tracker/*`
-- `app/services/llm_cost_tracker/dashboard/*`
-- `app/helpers/llm_cost_tracker/*`
-- `app/views/llm_cost_tracker/*`
-- `app/assets/llm_cost_tracker/application.css`
-
-Responsibilities:
-
-- Render server-side dashboard pages.
-- Aggregate spend, calls, providers, models, tags, latency, and data quality.
-- Export filtered calls as CSV.
-- Keep dashboard queries explicit and indexed.
-
-Dashboard code may run grouped SQL because it is user-initiated reporting. It must stay server-rendered and must not introduce a JavaScript bundle.
-
-## Rails Integration and Generators
-
-Primary files:
-
-- `lib/llm_cost_tracker/railtie.rb`
-- `lib/llm_cost_tracker/engine.rb`
-- `lib/llm_cost_tracker/assets.rb`
-- `lib/llm_cost_tracker/generators/llm_cost_tracker/*`
-- `config/routes.rb`
-
-Responsibilities:
-
-- Register rake tasks and Faraday middleware.
-- Mount the isolated Rails engine.
-- Generate migrations, initializer, dashboard route, and local price snapshots.
-- Serve dashboard CSS as a fingerprinted engine asset.
-
-Generator templates are public installation contracts. Treat them like API.
-
-## Test Suites
-
-Primary files:
-
-- `spec/llm_cost_tracker/*`
-- `spec/llm_cost_tracker/engine/*`
-- `spec/llm_cost_tracker/dashboard/*`
-- `spec/fixtures/pricing/*`
-- `spec/support/*`
-
-Responsibilities:
-
-- Cover canonical behavior, parser boundaries, pricing precedence, storage rollups, dashboard rendering, generators, and concurrency.
-- Keep request specs plain and stable.
-- Run through `bin/check` before release work or commits that touch code.

data/docs/technical/operational-notes.md

@@ -1,97 +0,0 @@
-# Operational Notes
-
-This file describes runtime constraints that should shape implementation decisions.
-
-## Hot Paths
-
-Hot-path code includes:
-
-- Faraday middleware request and response handling
-- stream collection
-- `Tracker.record`
-- `Pricing.cost_for`
-- ActiveRecord event persistence
-- budget checks
-
-Hot-path code must avoid:
-
-- network calls
-- per-event schema discovery beyond memoized checks
-- full ledger aggregation
-- unbounded stream buffers
-- N+1 queries
-- price-refresh work
-
-## Pricing Freshness
-
-Runtime pricing is local:
-
-1. Ruby overrides
-2. configured local price snapshot
-3. bundled prices
-
-Price update tasks are operational tooling. They can fetch the maintained LLM Cost Tracker price snapshot because the operator runs them intentionally. Request tracking must never depend on live provider pricing pages.
-
-## Budget Reads
-
-Monthly and daily budgets should read `llm_cost_tracker_period_totals` when the table exists and add pending `llm_cost_tracker_inbox_events` totals while durable ingestion is enabled. Falling back to summing `llm_api_calls` is an upgrade compatibility path, not the preferred production path.
-
-The stored period total and pending inbox total should be read in one database statement so request-time budget checks do not undercount during the inbox-to-ledger handoff.
-
-Per-call budgets are checked from the current event only.
-
-## Durable Ingestion
-
-Inbox writes inside an open caller transaction need a separate database connection to survive caller rollbacks. If the pool cannot provide one, storage should fail honestly through `storage_error_behavior` instead of writing into the caller transaction and pretending the event is durable.
-
-The ingestor is database-leased and database-polled, with an opportunistic local wake after a successful inbox insert. The wake only reduces freshness latency in the process that wrote the row; correctness still comes from the shared database lease, retryable row locks, and adaptive polling across Puma, Sidekiq, Unicorn, deploy restarts, and multi-process hosts.
-
-Freshness and durability are separate concerns. If the writing process exits before its local ingestor drains the row, another process can pick it up on a later poll; budget reads include pending inbox totals and operators can call `LlmCostTracker.flush!` when they need the ledger drained before continuing.
-
-The ingestor should check for claimable rows before acquiring the leader lease. Empty queues should not create steady lease-table writes across an idle fleet.
-
-Batch size is a conservative internal constant. Do not expose it as a configuration knob until production measurements show that a supported workload needs tuning; more knobs make installations harder to reason about.
-
-Ingestors should claim only retryable rows. Rows that keep failing after the retry cap stay in `llm_cost_tracker_inbox_events` with `last_error` for operator inspection and must not block healthy rows behind them.
-
-Process shutdown should stop the local ingestor thread without forcing every exiting process to drain the shared inbox. Operators can call `LlmCostTracker.flush!` when they intentionally want to wait for the durable inbox to drain.
-
-## Retention
-
-Retention may delete old `llm_api_calls`. Period rollups are the durable budget aggregate. Any migration or refactor that changes rollups must preserve the meaning of retained totals or clearly document a breaking change.
-
-## Optional Columns
-
-The gem supports upgrade paths where older apps may not have every column yet. Optional column checks must be memoized and refreshed when ActiveRecord column information is reset.
-
-Do not put table or column checks directly in loops that run for every event without caching.
-
-## Dashboard Queries
-
-Dashboard queries can aggregate because they are user-initiated. They should still use:
-
-- filtered scopes
-- bounded pagination
-- database-side grouping
-- indexes that match common filters
-- single aggregate queries for related counters
-
-Avoid loading ledger rows into Ruby just to count, sum, group, or sort.
-
-The dashboard is not the center of the storage design. Prefer bounded ranges, existing ledger indexes, pagination, and database-side aggregates over new dashboard-specific tables. Add a summary table only when a measured supported dashboard query cannot be made acceptable with the existing ledger and period totals.
-
-## Streaming
-
-Streaming capture must keep the host app's stream behavior intact.
-
-The middleware should collect enough data to parse final usage while bounding memory. When usage never arrives or capture overflows, record an unknown-usage event so Data Quality can surface the gap.
-
-## Release Checks
-
-Run `bin/check` before committing code changes intended for release. It includes full RuboCop, full RSpec, project coverage, and patch coverage for the current diff.
-
-Project coverage defaults to the Codecov target. Patch coverage defaults to 95% so local checks stay stricter than Codecov parser differences. Thresholds can be adjusted locally with `PROJECT_COVERAGE_MIN`, `PATCH_COVERAGE_MIN`, or `COVERAGE_BASE`.
-
-For the closest match to the Codecov upload job, run `BUNDLE_GEMFILE=gemfiles/rails_8_1.gemfile bin/check`.
-
-Docs-only changes do not require the full suite, but any code, generator, migration, parser, pricing, dashboard, or storage change does.

data/docs/upgrading.md

@@ -1,47 +0,0 @@
-# Upgrading
-
-LLM Cost Tracker is still moving quickly, so upgrades should be explicit:
-inspect the changelog, run doctor, and apply only the generators your schema is
-missing.
-
-The version-by-version upgrade guide is moving here from the README.
-
-## Canonical Sources
-
-Until this page is expanded, use:
-
-- [Changelog](../CHANGELOG.md)
-- [Quickstart](../README.md#quickstart)
-- [Operations](operations.md)
-
-## Schema Generators
-
-Existing installs can add newer optional columns through focused generators:
-
-```bash
-bin/rails generate llm_cost_tracker:add_period_totals
-bin/rails generate llm_cost_tracker:add_ingestion
-bin/rails generate llm_cost_tracker:add_streaming
-bin/rails generate llm_cost_tracker:add_provider_response_id
-bin/rails generate llm_cost_tracker:add_usage_breakdown
-bin/rails generate llm_cost_tracker:upgrade_tags_to_jsonb
-bin/rails generate llm_cost_tracker:upgrade_cost_precision
-bin/rails generate llm_cost_tracker:add_latency_ms
-bin/rails db:migrate
-bin/rails llm_cost_tracker:doctor
-```
-
-On PostgreSQL, `upgrade_tags_to_jsonb` rewrites `llm_api_calls`. For large
-tables, run it during a maintenance window or replace it with a two-phase
-backfill.
-
-## Upgrade Habit
-
-Run:
-
-```bash
-bin/rails llm_cost_tracker:doctor
-```
-
-Doctor tells you which optional columns and production-hardening pieces are still
-missing.