llm_cost_tracker 0.1.4 → 0.2.0.alpha2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +58 -91
- data/PLAN_0.2.md +488 -0
- data/README.md +140 -320
- data/app/controllers/llm_cost_tracker/application_controller.rb +42 -0
- data/app/controllers/llm_cost_tracker/calls_controller.rb +77 -0
- data/app/controllers/llm_cost_tracker/dashboard_controller.rb +54 -0
- data/app/controllers/llm_cost_tracker/data_quality_controller.rb +10 -0
- data/app/controllers/llm_cost_tracker/models_controller.rb +12 -0
- data/app/controllers/llm_cost_tracker/tags_controller.rb +21 -0
- data/app/helpers/llm_cost_tracker/application_helper.rb +113 -0
- data/app/services/llm_cost_tracker/dashboard/data_quality.rb +38 -0
- data/app/services/llm_cost_tracker/dashboard/filter.rb +109 -0
- data/app/services/llm_cost_tracker/dashboard/overview_stats.rb +87 -0
- data/app/services/llm_cost_tracker/dashboard/provider_breakdown.rb +44 -0
- data/app/services/llm_cost_tracker/dashboard/tag_breakdown.rb +58 -0
- data/app/services/llm_cost_tracker/dashboard/tag_key_explorer.rb +125 -0
- data/app/services/llm_cost_tracker/dashboard/time_series.rb +44 -0
- data/app/services/llm_cost_tracker/dashboard/top_models.rb +89 -0
- data/app/services/llm_cost_tracker/pagination.rb +59 -0
- data/app/views/layouts/llm_cost_tracker/application.html.erb +342 -0
- data/app/views/llm_cost_tracker/calls/index.html.erb +127 -0
- data/app/views/llm_cost_tracker/calls/show.html.erb +67 -0
- data/app/views/llm_cost_tracker/dashboard/index.html.erb +145 -0
- data/app/views/llm_cost_tracker/data_quality/index.html.erb +110 -0
- data/app/views/llm_cost_tracker/errors/database.html.erb +8 -0
- data/app/views/llm_cost_tracker/errors/invalid_filter.html.erb +4 -0
- data/app/views/llm_cost_tracker/errors/not_found.html.erb +5 -0
- data/app/views/llm_cost_tracker/models/index.html.erb +95 -0
- data/app/views/llm_cost_tracker/shared/_bar.html.erb +5 -0
- data/app/views/llm_cost_tracker/shared/setup_required.html.erb +6 -0
- data/app/views/llm_cost_tracker/tags/index.html.erb +34 -0
- data/app/views/llm_cost_tracker/tags/show.html.erb +69 -0
- data/config/routes.rb +10 -0
- data/lib/llm_cost_tracker/budget.rb +16 -38
- data/lib/llm_cost_tracker/configuration.rb +3 -1
- data/lib/llm_cost_tracker/cost.rb +1 -3
- data/lib/llm_cost_tracker/engine.rb +13 -0
- data/lib/llm_cost_tracker/engine_compatibility.rb +15 -0
- data/lib/llm_cost_tracker/errors.rb +2 -0
- data/lib/llm_cost_tracker/event.rb +1 -3
- data/lib/llm_cost_tracker/event_metadata.rb +9 -18
- data/lib/llm_cost_tracker/llm_api_call.rb +4 -17
- data/lib/llm_cost_tracker/middleware/faraday.rb +4 -4
- data/lib/llm_cost_tracker/parsed_usage.rb +5 -9
- data/lib/llm_cost_tracker/parsers/anthropic.rb +4 -5
- data/lib/llm_cost_tracker/parsers/base.rb +3 -8
- data/lib/llm_cost_tracker/parsers/gemini.rb +3 -3
- data/lib/llm_cost_tracker/parsers/openai_usage.rb +3 -3
- data/lib/llm_cost_tracker/parsers/registry.rb +5 -12
- data/lib/llm_cost_tracker/period_grouping.rb +68 -0
- data/lib/llm_cost_tracker/price_registry.rb +22 -30
- data/lib/llm_cost_tracker/pricing.rb +10 -19
- data/lib/llm_cost_tracker/report.rb +4 -4
- data/lib/llm_cost_tracker/report_data.rb +21 -24
- data/lib/llm_cost_tracker/report_formatter.rb +4 -2
- data/lib/llm_cost_tracker/storage/active_record_store.rb +1 -3
- data/lib/llm_cost_tracker/tag_key.rb +16 -0
- data/lib/llm_cost_tracker/tracker.rb +35 -1
- data/lib/llm_cost_tracker/version.rb +1 -1
- data/lib/llm_cost_tracker.rb +3 -6
- data/llm_cost_tracker.gemspec +13 -9
- metadata +91 -20
- data/.rubocop.yml +0 -44
- data/lib/llm_cost_tracker/storage/active_record_backend.rb +0 -19
- data/lib/llm_cost_tracker/storage/backends.rb +0 -26
- data/lib/llm_cost_tracker/storage/custom_backend.rb +0 -16
- data/lib/llm_cost_tracker/storage/log_backend.rb +0 -28
- data/lib/llm_cost_tracker/value_object.rb +0 -45
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: '043255892f8db58b53a84d26d25ee09561b8349b72f463429f0f31691c6449a5'
|
|
4
|
+
data.tar.gz: a4577ed5935b1f65e85c4d73700bde447715875d40846d1bf6a8575b0588e2fc
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 0a7d8c3454cb88b93da2ef83d4b8cda330060200e3cb5073017f2d9c927d2d6e6ceef6e8d3cbc29120d8e44d955c8c1800126271ebb92f981253f9315d9abde8
|
|
7
|
+
data.tar.gz: 239c293ddf252d5933adf294329f20b9f5df2e1322bb5413aa4392b43077f26385ad89bbf382f786e1bc79a94252b47ed157d19464973beb4467a312cef368eb
|
data/CHANGELOG.md
CHANGED
|
@@ -1,133 +1,100 @@
|
|
|
1
1
|
# Changelog
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
Format: [Keep a Changelog](https://keepachangelog.com/en/1.1.0/). Versioning: [SemVer](https://semver.org/spec/v2.0.0.html).
|
|
4
4
|
|
|
5
|
-
|
|
6
|
-
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
5
|
+
## [0.2.0.alpha1, 0.2.0.alpha2] - 2026-04-20
|
|
7
6
|
|
|
8
|
-
|
|
9
|
-
|
|
10
|
-
### Breaking Changes
|
|
7
|
+
### Breaking
|
|
11
8
|
|
|
12
|
-
-
|
|
13
|
-
|
|
14
|
-
-
|
|
15
|
-
|
|
16
|
-
-
|
|
17
|
-
`LlmApiCall.cost_by_tag("feature")`.
|
|
9
|
+
- Require Ruby 3.3+ (was 3.1), Rails/ActiveRecord 7.1+ (was 7.0), Faraday 2.0+ (was 1.0).
|
|
10
|
+
- `Event`, `Cost`, and `ParsedUsage` are plain `Data.define` value objects; use method access (`event.cost.total_cost`) instead of Hash lookups. `ActiveSupport::Notifications` payloads are unchanged.
|
|
11
|
+
- Rename `LlmCostTracker::InvalidFilter` → `InvalidFilterError`.
|
|
12
|
+
- Drop `LlmApiCall.by_provider` / `by_model` scopes — use `where(provider:)` / `where(model:)`.
|
|
13
|
+
- `ReportData` no longer hardcodes a `"feature"` tag breakdown. Configure `config.report_tag_breakdowns = %w[feature env]` (or pass `tag_breakdowns:` to `ReportData.build` / `Report.generate`). Default is empty.
|
|
18
14
|
|
|
19
15
|
### Added
|
|
20
16
|
|
|
21
|
-
-
|
|
22
|
-
-
|
|
17
|
+
- `LlmApiCall.group_by_period(:day/:month)` — SQL-side period grouping.
|
|
18
|
+
- Opt-in `LlmCostTracker::Engine` dashboard (Rails 7.1+): overview with delta-vs-previous-period, provider rollup, models, filterable call list with CSV export and outlier sort modes, call details, tag key explorer, per-key tag breakdown, data quality. PostgreSQL/SQLite use adapter-specific SQL; MySQL falls back to an in-Ruby scan capped at 50k rows. Core middleware still works without Rails.
|
|
23
19
|
|
|
24
|
-
## [0.1.
|
|
20
|
+
## [0.1.4] - 2026-04-18
|
|
25
21
|
|
|
26
|
-
###
|
|
22
|
+
### Breaking
|
|
27
23
|
|
|
28
|
-
|
|
24
|
+
- Drop `LlmApiCall.by_user` / `by_feature` scopes and `LlmApiCall#user_id` / `#feature` accessors. Use `by_tag("user_id", id)` / `by_tag("feature", name)` or `by_tags(...)`; read stored tags via `parsed_tags[...]`.
|
|
25
|
+
- Drop `ReportData#cost_by_feature` — use `cost_by_tags.fetch("feature")` or `LlmApiCall.cost_by_tag("feature")`.
|
|
29
26
|
|
|
30
|
-
|
|
27
|
+
### Added
|
|
31
28
|
|
|
32
|
-
|
|
29
|
+
- `group_by_tag(key)` / `cost_by_tag(key)` SQL aggregations across any tag key.
|
|
30
|
+
- Generic tag breakdowns in reports.
|
|
33
31
|
|
|
32
|
+
## [0.1.3] - 2026-04-18
|
|
33
|
+
|
|
34
|
+
### Fixed / Changed
|
|
35
|
+
|
|
36
|
+
- Mutex-guard `PriceRegistry.file_prices` and `Pricing.sorted_price_keys` memoization.
|
|
34
37
|
- Warn on unknown keys in local prices files.
|
|
35
|
-
- Add `llm_cost_tracker:prices` generator for creating a local price override template.
|
|
36
38
|
- Document that budget guardrails skip events with unknown pricing.
|
|
37
39
|
|
|
38
|
-
|
|
40
|
+
### Added
|
|
39
41
|
|
|
40
|
-
-
|
|
41
|
-
-
|
|
42
|
-
-
|
|
42
|
+
- `llm_cost_tracker:prices` generator for a local price override template.
|
|
43
|
+
- Callable Faraday `tags:` for per-request Rails attribution via `Current`.
|
|
44
|
+
- `llm_cost_tracker:report` rake task.
|
|
43
45
|
|
|
44
|
-
|
|
46
|
+
### Internal
|
|
45
47
|
|
|
46
|
-
- Extract `Logging`
|
|
47
|
-
- Extract `TagQuery`, `TagsColumn`, and `TagAccessors` helpers from `LlmApiCall`.
|
|
48
|
-
- Introduce typed `Cost`, `Event`, and `ParsedUsage` value objects while preserving hash-like access.
|
|
49
|
-
- Move storage dispatch into dedicated backend objects with a uniform save contract.
|
|
50
|
-
- Split `Report` into `ReportData` and `ReportFormatter`.
|
|
51
|
-
- Use `OpenaiUsage` composition for OpenAI-compatible providers instead of parser inheritance.
|
|
52
|
-
- Move config enum validation into `Configuration` setters.
|
|
53
|
-
- Memoize the merged built-in/file/override prices table.
|
|
54
|
-
- Restrict the Gemini parser to `generateContent` and `streamGenerateContent` paths.
|
|
48
|
+
- Extract `Logging`, `TagQuery`, `TagsColumn`, `TagAccessors` helpers; `Cost`, `Event`, `ParsedUsage` value objects; storage backend objects; split `Report` into data + formatter; `OpenaiUsage` composition for OpenAI-compatible providers; move enum validation into `Configuration`; memoize merged prices table; restrict Gemini parser to `generateContent` / `streamGenerateContent`.
|
|
55
49
|
|
|
56
50
|
## [0.1.2] - 2026-04-18
|
|
57
51
|
|
|
58
52
|
### Added
|
|
59
53
|
|
|
60
|
-
- Auto-detect OpenRouter and DeepSeek as OpenAI-compatible
|
|
61
|
-
-
|
|
62
|
-
-
|
|
63
|
-
-
|
|
64
|
-
-
|
|
65
|
-
-
|
|
66
|
-
-
|
|
67
|
-
-
|
|
68
|
-
-
|
|
69
|
-
- Add `latency_ms` tracking for Faraday calls, manual tracking, notifications, and ActiveRecord storage.
|
|
70
|
-
- Add `with_latency`, `average_latency_ms`, `latency_by_model`, and `latency_by_provider`.
|
|
71
|
-
- Use PostgreSQL `jsonb` storage for tags in newly generated migrations.
|
|
72
|
-
- Add a GIN index on `llm_api_calls.tags` for PostgreSQL installs.
|
|
73
|
-
- Add adapter-aware `by_tag` querying with JSONB containment on PostgreSQL and text fallback elsewhere.
|
|
74
|
-
- Add `by_tags`, `by_user`, and `by_feature` scopes for common attribution queries.
|
|
75
|
-
- Add `llm_cost_tracker:upgrade_tags_to_jsonb` generator for existing PostgreSQL installs.
|
|
76
|
-
- Add `llm_cost_tracker:upgrade_cost_precision` generator for widening stored cost columns.
|
|
77
|
-
- Add `llm_cost_tracker:add_latency_ms` generator for existing installs.
|
|
54
|
+
- Auto-detect OpenRouter and DeepSeek as OpenAI-compatible.
|
|
55
|
+
- `openai_compatible_providers` config for private gateways.
|
|
56
|
+
- `BudgetExceededError` + `budget_exceeded_behavior` (`:notify`, `:raise`, `:block_requests`). `:block_requests` is best-effort under concurrency.
|
|
57
|
+
- `StorageError` + `storage_error_behavior`; `UnknownPricingError` + `unknown_pricing_behavior`.
|
|
58
|
+
- Built-in `prices.json` registry with metadata and source URLs; `prices_file` for local JSON/YAML overrides.
|
|
59
|
+
- `with_cost`, `without_cost`, `unknown_pricing` scopes.
|
|
60
|
+
- `latency_ms` tracking end-to-end; `with_latency`, `average_latency_ms`, `latency_by_model`, `latency_by_provider`.
|
|
61
|
+
- `jsonb` tags + GIN index on PostgreSQL in new migrations; adapter-aware `by_tag` (JSONB containment on PG, text fallback elsewhere); `by_tags` / `by_user` / `by_feature`.
|
|
62
|
+
- Generators: `upgrade_tags_to_jsonb`, `upgrade_cost_precision`, `add_latency_ms`.
|
|
78
63
|
|
|
79
64
|
### Changed
|
|
80
65
|
|
|
81
|
-
-
|
|
82
|
-
-
|
|
83
|
-
-
|
|
84
|
-
-
|
|
85
|
-
-
|
|
86
|
-
-
|
|
87
|
-
- Count Gemini `thoughtsTokenCount` as output tokens for better thinking-mode cost estimates.
|
|
88
|
-
- Warn when Faraday exposes an unreadable streaming/SSE response body.
|
|
89
|
-
- Document tag storage behavior, budget guardrail limits, known limitations, common tag scopes, and upgrade flows.
|
|
90
|
-
- Clarify that budget errors raised after a response occur after the event has been recorded.
|
|
91
|
-
- Route custom storage exceptions that inherit from `LlmCostTracker::Error` through `storage_error_behavior`.
|
|
66
|
+
- Tags stored as Hash for JSON-backed columns, JSON text for fallback.
|
|
67
|
+
- Normalize provider-prefixed model IDs (e.g. `openai/gpt-4o-mini`) for price lookup.
|
|
68
|
+
- Widen generated cost columns to `precision: 20, scale: 8`.
|
|
69
|
+
- Count Gemini `thoughtsTokenCount` as output tokens.
|
|
70
|
+
- Warn on unreadable streaming/SSE response bodies.
|
|
71
|
+
- Route storage exceptions inheriting from `LlmCostTracker::Error` through `storage_error_behavior`.
|
|
92
72
|
|
|
93
73
|
## [0.1.1] - 2026-04-17
|
|
94
74
|
|
|
95
75
|
### Fixed
|
|
96
76
|
|
|
97
|
-
- Lazy-load ActiveRecord storage so
|
|
98
|
-
-
|
|
99
|
-
- Track OpenAI Responses API
|
|
100
|
-
- Parse
|
|
101
|
-
-
|
|
102
|
-
- Parse Gemini cached content token usage when present.
|
|
103
|
-
- Store ActiveRecord tag values as strings so `by_tag("user_id", "42")` works for numeric IDs.
|
|
77
|
+
- Lazy-load ActiveRecord storage so `:active_record` persists events reliably.
|
|
78
|
+
- Stop double-counting the latest event in monthly budget callbacks.
|
|
79
|
+
- Track OpenAI Responses API (`/v1/responses`).
|
|
80
|
+
- Parse cached/cache-read/cache-creation token details across OpenAI, Anthropic, Gemini.
|
|
81
|
+
- Store tag values as strings so `by_tag("user_id", "42")` matches numeric IDs.
|
|
104
82
|
|
|
105
83
|
### Changed
|
|
106
84
|
|
|
107
|
-
- Refresh built-in pricing for current OpenAI, Anthropic,
|
|
108
|
-
-
|
|
109
|
-
- Tighten OpenAI URL matching to supported endpoint families
|
|
110
|
-
- Reposition README around self-hosted Rails/Ruby cost tracking for Faraday-based clients.
|
|
85
|
+
- Refresh built-in pricing for current OpenAI, Anthropic, Gemini models.
|
|
86
|
+
- Cache-aware cost fields (cached input, cache reads, cache creation).
|
|
87
|
+
- Tighten OpenAI URL matching to supported endpoint families.
|
|
111
88
|
|
|
112
89
|
### Added
|
|
113
90
|
|
|
114
|
-
-
|
|
115
|
-
- Add RuboCop configuration, rake task, and CI lint step.
|
|
116
|
-
- Require MFA metadata for RubyGems publishing.
|
|
91
|
+
- ActiveRecord integration specs; RuboCop config + CI lint step; RubyGems MFA metadata.
|
|
117
92
|
|
|
118
93
|
## [0.1.0] - 2026-04-16
|
|
119
94
|
|
|
120
|
-
|
|
121
|
-
|
|
122
|
-
-
|
|
123
|
-
-
|
|
124
|
-
-
|
|
125
|
-
-
|
|
126
|
-
- ActiveSupport::Notifications integration
|
|
127
|
-
- ActiveRecord storage backend with scopes and aggregations
|
|
128
|
-
- Manual `LlmCostTracker.track()` for non-Faraday clients
|
|
129
|
-
- Per-user / per-feature tagging
|
|
130
|
-
- Monthly budget alerts with configurable callbacks
|
|
131
|
-
- Rails generator: `rails generate llm_cost_tracker:install`
|
|
132
|
-
- Custom storage backend support
|
|
133
|
-
- Pricing overrides via configuration
|
|
95
|
+
- Faraday middleware for LLM call interception.
|
|
96
|
+
- Parsers: OpenAI, Anthropic, Gemini. Built-in pricing for 20+ models with fuzzy matching.
|
|
97
|
+
- `ActiveSupport::Notifications` integration; ActiveRecord backend with scopes and aggregations.
|
|
98
|
+
- Manual `LlmCostTracker.track(...)` for non-Faraday clients.
|
|
99
|
+
- Per-user / per-feature tagging; monthly budget alerts with configurable callbacks.
|
|
100
|
+
- `rails generate llm_cost_tracker:install`; custom storage backend; pricing overrides.
|
data/PLAN_0.2.md
ADDED
|
@@ -0,0 +1,488 @@
|
|
|
1
|
+
# llm_cost_tracker v0.2.0 Plan
|
|
2
|
+
|
|
3
|
+
## Goal
|
|
4
|
+
|
|
5
|
+
Ship an opt-in Rails dashboard engine:
|
|
6
|
+
|
|
7
|
+
```ruby
|
|
8
|
+
require "llm_cost_tracker/engine"
|
|
9
|
+
|
|
10
|
+
mount LlmCostTracker::Engine => "/llm-costs"
|
|
11
|
+
```
|
|
12
|
+
|
|
13
|
+
The dashboard is read-only and shows where a Rails app spends money on LLM APIs:
|
|
14
|
+
|
|
15
|
+
- spend over time
|
|
16
|
+
- top models
|
|
17
|
+
- recent calls
|
|
18
|
+
- call details
|
|
19
|
+
- tag breakdowns such as `feature`
|
|
20
|
+
- optional monthly budget status
|
|
21
|
+
|
|
22
|
+
The first milestone is a useful Overview page that can be used as the README screenshot.
|
|
23
|
+
|
|
24
|
+
## Non-Goals
|
|
25
|
+
|
|
26
|
+
These are intentionally out of scope for v0.2.0:
|
|
27
|
+
|
|
28
|
+
- built-in authorization
|
|
29
|
+
- CSV or JSON export
|
|
30
|
+
- alerts, webhooks, or Slack notifications
|
|
31
|
+
- JavaScript charts or live updates
|
|
32
|
+
- editing settings from the dashboard
|
|
33
|
+
- persistent budget models
|
|
34
|
+
- `/budgets`
|
|
35
|
+
- `/tags` index or automatic tag-key discovery
|
|
36
|
+
- timezone-aware SQL bucketing
|
|
37
|
+
- hourly period grouping
|
|
38
|
+
- SaaS or remote sync
|
|
39
|
+
|
|
40
|
+
## Compatibility Decisions
|
|
41
|
+
|
|
42
|
+
### Core Gem
|
|
43
|
+
|
|
44
|
+
The core gem remains lightweight and usable outside Rails:
|
|
45
|
+
|
|
46
|
+
- keep `activesupport >= 7.0, < 9.0`
|
|
47
|
+
- do not add `railties` as a runtime dependency
|
|
48
|
+
- keep middleware-only usage working for plain Faraday, Sinatra, Hanami, and service objects
|
|
49
|
+
|
|
50
|
+
### Dashboard Engine
|
|
51
|
+
|
|
52
|
+
The Engine is Rails-only and opt-in:
|
|
53
|
+
|
|
54
|
+
- users must `require "llm_cost_tracker/engine"`
|
|
55
|
+
- Engine requires Rails 7.1+
|
|
56
|
+
- the Rails version guard must run at top level in `engine.rb`, before defining `class Engine < ::Rails::Engine`
|
|
57
|
+
- `railties` is a development/test dependency only
|
|
58
|
+
|
|
59
|
+
Example guard:
|
|
60
|
+
|
|
61
|
+
```ruby
|
|
62
|
+
unless Gem::Version.new(Rails.version) >= Gem::Version.new("7.1.0")
|
|
63
|
+
raise LlmCostTracker::Error, "LlmCostTracker::Engine requires Rails 7.1+"
|
|
64
|
+
end
|
|
65
|
+
```
|
|
66
|
+
|
|
67
|
+
## Phase 1: Public Query Primitive
|
|
68
|
+
|
|
69
|
+
Add SQL-side period grouping to `LlmCostTracker::LlmApiCall`.
|
|
70
|
+
|
|
71
|
+
### API
|
|
72
|
+
|
|
73
|
+
```ruby
|
|
74
|
+
LlmCostTracker::LlmApiCall.group_by_period(:day).sum(:total_cost)
|
|
75
|
+
LlmCostTracker::LlmApiCall.group_by_period(:month).count
|
|
76
|
+
LlmCostTracker::LlmApiCall.group_by_period(:day, column: :created_at)
|
|
77
|
+
```
|
|
78
|
+
|
|
79
|
+
Default column: `:tracked_at`.
|
|
80
|
+
|
|
81
|
+
### Supported Periods
|
|
82
|
+
|
|
83
|
+
Only:
|
|
84
|
+
|
|
85
|
+
- `:day`
|
|
86
|
+
- `:month`
|
|
87
|
+
|
|
88
|
+
Do not add `:hour` or `:week` in v0.2.0.
|
|
89
|
+
|
|
90
|
+
### Return Keys
|
|
91
|
+
|
|
92
|
+
Return string keys across all adapters:
|
|
93
|
+
|
|
94
|
+
```ruby
|
|
95
|
+
:day # "2026-04-18"
|
|
96
|
+
:month # "2026-04"
|
|
97
|
+
```
|
|
98
|
+
|
|
99
|
+
### SQL Strategy
|
|
100
|
+
|
|
101
|
+
- PostgreSQL: `TO_CHAR(DATE_TRUNC(...), ...)`
|
|
102
|
+
- MySQL: `DATE_FORMAT(...)`
|
|
103
|
+
- SQLite: `strftime(...)`
|
|
104
|
+
|
|
105
|
+
### Validation
|
|
106
|
+
|
|
107
|
+
- period must be whitelisted
|
|
108
|
+
- column must exist in `column_names`
|
|
109
|
+
- invalid period or column raises `ArgumentError`
|
|
110
|
+
- no `time_zone:` parameter in v0.2.0
|
|
111
|
+
|
|
112
|
+
### Tests
|
|
113
|
+
|
|
114
|
+
- real SQLite grouping specs
|
|
115
|
+
- generated SQL expression specs for PostgreSQL/MySQL if those adapters are not in CI
|
|
116
|
+
- injection tests for period and column
|
|
117
|
+
- composition test:
|
|
118
|
+
|
|
119
|
+
```ruby
|
|
120
|
+
LlmCostTracker::LlmApiCall
|
|
121
|
+
.this_month
|
|
122
|
+
.where(provider: "openai")
|
|
123
|
+
.group_by_period(:day)
|
|
124
|
+
.sum(:total_cost)
|
|
125
|
+
```
|
|
126
|
+
|
|
127
|
+
## Phase 2: Engine Skeleton
|
|
128
|
+
|
|
129
|
+
Add the minimal Rails Engine structure:
|
|
130
|
+
|
|
131
|
+
```text
|
|
132
|
+
lib/llm_cost_tracker/engine.rb
|
|
133
|
+
app/controllers/llm_cost_tracker/application_controller.rb
|
|
134
|
+
app/controllers/llm_cost_tracker/dashboard_controller.rb
|
|
135
|
+
app/views/layouts/llm_cost_tracker/application.html.erb
|
|
136
|
+
config/routes.rb
|
|
137
|
+
spec/dummy/
|
|
138
|
+
```
|
|
139
|
+
|
|
140
|
+
### Requirements
|
|
141
|
+
|
|
142
|
+
- isolated namespace: `isolate_namespace LlmCostTracker`
|
|
143
|
+
- no automatic loading for non-Rails users
|
|
144
|
+
- no asset pipeline dependency
|
|
145
|
+
- no JavaScript
|
|
146
|
+
- inline CSS in the Engine layout
|
|
147
|
+
- all CSS classes prefixed with `.lct-`
|
|
148
|
+
- minimal `spec/dummy` Rails app for Engine request specs
|
|
149
|
+
|
|
150
|
+
### Acceptance
|
|
151
|
+
|
|
152
|
+
- dummy app mounts the Engine at `/llm-costs`
|
|
153
|
+
- `GET /llm-costs` returns 200
|
|
154
|
+
- empty database shows an intentional empty state
|
|
155
|
+
- missing `llm_api_calls` table shows a friendly setup error
|
|
156
|
+
|
|
157
|
+
## Phase 3: Dashboard Data Layer
|
|
158
|
+
|
|
159
|
+
Use plain Ruby objects under:
|
|
160
|
+
|
|
161
|
+
```text
|
|
162
|
+
app/services/llm_cost_tracker/dashboard/
|
|
163
|
+
```
|
|
164
|
+
|
|
165
|
+
### Filter
|
|
166
|
+
|
|
167
|
+
Parses dashboard params and returns an ActiveRecord relation.
|
|
168
|
+
|
|
169
|
+
Supported params:
|
|
170
|
+
|
|
171
|
+
- `from`
|
|
172
|
+
- `to`
|
|
173
|
+
- `provider`
|
|
174
|
+
- `model`
|
|
175
|
+
- `tag[key]=value`
|
|
176
|
+
|
|
177
|
+
Rules:
|
|
178
|
+
|
|
179
|
+
- parse `from` and `to` with `Date.iso8601`
|
|
180
|
+
- ignore invalid dates
|
|
181
|
+
- use AR placeholders for provider/model
|
|
182
|
+
- parse all `params[:tag]` keys as a hash
|
|
183
|
+
- pass multi-key tag filters to `by_tags`
|
|
184
|
+
- validate tag keys with the same whitelist as `group_by_tag`
|
|
185
|
+
|
|
186
|
+
Example:
|
|
187
|
+
|
|
188
|
+
```text
|
|
189
|
+
?tag[feature]=chat&tag[user_id]=42
|
|
190
|
+
```
|
|
191
|
+
|
|
192
|
+
becomes:
|
|
193
|
+
|
|
194
|
+
```ruby
|
|
195
|
+
scope.by_tags("feature" => "chat", "user_id" => "42")
|
|
196
|
+
```
|
|
197
|
+
|
|
198
|
+
### Page
|
|
199
|
+
|
|
200
|
+
Use a small immutable object, not a large service:
|
|
201
|
+
|
|
202
|
+
```ruby
|
|
203
|
+
Dashboard::Page = Data.define(:page, :per)
|
|
204
|
+
```
|
|
205
|
+
|
|
206
|
+
Rules:
|
|
207
|
+
|
|
208
|
+
- `page` minimum: 1
|
|
209
|
+
- `per` default: 50
|
|
210
|
+
- `per` maximum: 200
|
|
211
|
+
- expose `limit`, `offset`, `prev_page?`, and `next_page?`
|
|
212
|
+
|
|
213
|
+
### OverviewStats
|
|
214
|
+
|
|
215
|
+
Compute:
|
|
216
|
+
|
|
217
|
+
- total spend
|
|
218
|
+
- total calls
|
|
219
|
+
- average cost per call
|
|
220
|
+
- average latency only if `latency_ms` exists
|
|
221
|
+
- monthly budget status only if `LlmCostTracker.configuration.monthly_budget` is set
|
|
222
|
+
|
|
223
|
+
Do not compute `known_pricing_rate` in v0.2.0.
|
|
224
|
+
|
|
225
|
+
### TimeSeries
|
|
226
|
+
|
|
227
|
+
Uses `group_by_period(:day).sum(:total_cost)`.
|
|
228
|
+
|
|
229
|
+
Rules:
|
|
230
|
+
|
|
231
|
+
- default range: last 30 days
|
|
232
|
+
- fill missing days with zero
|
|
233
|
+
- output array:
|
|
234
|
+
|
|
235
|
+
```ruby
|
|
236
|
+
[{ label: "2026-04-01", cost: 0.0 }]
|
|
237
|
+
```
|
|
238
|
+
|
|
239
|
+
### TopModels
|
|
240
|
+
|
|
241
|
+
Compute:
|
|
242
|
+
|
|
243
|
+
- provider
|
|
244
|
+
- model
|
|
245
|
+
- calls
|
|
246
|
+
- total cost
|
|
247
|
+
- average cost per call
|
|
248
|
+
- input tokens
|
|
249
|
+
- output tokens
|
|
250
|
+
- average latency if available
|
|
251
|
+
|
|
252
|
+
### TopTags
|
|
253
|
+
|
|
254
|
+
Only for configured keys.
|
|
255
|
+
|
|
256
|
+
Default keys:
|
|
257
|
+
|
|
258
|
+
```ruby
|
|
259
|
+
["feature"]
|
|
260
|
+
```
|
|
261
|
+
|
|
262
|
+
No automatic tag-key discovery in v0.2.0.
|
|
263
|
+
|
|
264
|
+
## Phase 4: Dashboard Pages
|
|
265
|
+
|
|
266
|
+
### Overview: `GET /`
|
|
267
|
+
|
|
268
|
+
The main screenshot-worthy page.
|
|
269
|
+
|
|
270
|
+
Show:
|
|
271
|
+
|
|
272
|
+
- total spend
|
|
273
|
+
- total calls
|
|
274
|
+
- average cost per call
|
|
275
|
+
- average latency if available
|
|
276
|
+
- monthly budget status if configured
|
|
277
|
+
- daily spend table with CSS bars
|
|
278
|
+
- top 5 models
|
|
279
|
+
- cost by `feature` tag if data exists
|
|
280
|
+
|
|
281
|
+
Budget wording:
|
|
282
|
+
|
|
283
|
+
> Soft monthly limit. Blocking is not atomic under concurrency.
|
|
284
|
+
|
|
285
|
+
### Calls Index: `GET /calls`
|
|
286
|
+
|
|
287
|
+
Filters:
|
|
288
|
+
|
|
289
|
+
- `from`
|
|
290
|
+
- `to`
|
|
291
|
+
- `provider`
|
|
292
|
+
- `model`
|
|
293
|
+
- `tag[key]=value`
|
|
294
|
+
- `page`
|
|
295
|
+
- `per`
|
|
296
|
+
|
|
297
|
+
Columns:
|
|
298
|
+
|
|
299
|
+
- created_at
|
|
300
|
+
- provider
|
|
301
|
+
- model
|
|
302
|
+
- input tokens
|
|
303
|
+
- output tokens
|
|
304
|
+
- cache tokens if present
|
|
305
|
+
- total cost
|
|
306
|
+
- latency if available
|
|
307
|
+
- tags summary
|
|
308
|
+
- details link
|
|
309
|
+
|
|
310
|
+
Rules:
|
|
311
|
+
|
|
312
|
+
- newest first
|
|
313
|
+
- max `per=200`
|
|
314
|
+
- one count query and one paginated query
|
|
315
|
+
- no N+1 behavior
|
|
316
|
+
|
|
317
|
+
### Call Details: `GET /calls/:id`
|
|
318
|
+
|
|
319
|
+
Show:
|
|
320
|
+
|
|
321
|
+
- all stored columns
|
|
322
|
+
- token breakdown
|
|
323
|
+
- cost breakdown
|
|
324
|
+
- latency if available
|
|
325
|
+
- tags as pretty JSON
|
|
326
|
+
- metadata if present
|
|
327
|
+
|
|
328
|
+
Missing records render a friendly 404 inside the Engine layout.
|
|
329
|
+
|
|
330
|
+
### Models: `GET /models`
|
|
331
|
+
|
|
332
|
+
Show rows grouped by provider/model:
|
|
333
|
+
|
|
334
|
+
- provider
|
|
335
|
+
- model
|
|
336
|
+
- calls
|
|
337
|
+
- total cost
|
|
338
|
+
- average cost per call
|
|
339
|
+
- input tokens
|
|
340
|
+
- output tokens
|
|
341
|
+
- average latency if available
|
|
342
|
+
|
|
343
|
+
Default sort: total cost descending.
|
|
344
|
+
|
|
345
|
+
### Tag Breakdown: `GET /tags/:key`
|
|
346
|
+
|
|
347
|
+
Validate `key` using the same tag-key whitelist as `group_by_tag`.
|
|
348
|
+
|
|
349
|
+
Show:
|
|
350
|
+
|
|
351
|
+
- tag value
|
|
352
|
+
- calls
|
|
353
|
+
- total cost
|
|
354
|
+
- average cost per call
|
|
355
|
+
|
|
356
|
+
Invalid keys render a friendly 400.
|
|
357
|
+
|
|
358
|
+
Do not add `/tags` index in v0.2.0.
|
|
359
|
+
|
|
360
|
+
## Graceful Degradation
|
|
361
|
+
|
|
362
|
+
The dashboard must not explode when optional pieces are missing.
|
|
363
|
+
|
|
364
|
+
Handle:
|
|
365
|
+
|
|
366
|
+
- empty database
|
|
367
|
+
- missing `llm_api_calls` table
|
|
368
|
+
- missing `latency_ms`
|
|
369
|
+
- text `tags` fallback instead of JSONB
|
|
370
|
+
- malformed legacy tag JSON
|
|
371
|
+
- unknown-pricing rows with `total_cost = nil`
|
|
372
|
+
- no configured monthly budget
|
|
373
|
+
|
|
374
|
+
## Security
|
|
375
|
+
|
|
376
|
+
- dashboard is read-only
|
|
377
|
+
- routes are GET-only
|
|
378
|
+
- keep `protect_from_forgery with: :exception`
|
|
379
|
+
- no built-in auth
|
|
380
|
+
- README must show Basic Auth and Devise examples
|
|
381
|
+
- validate every param through `Dashboard::Filter`
|
|
382
|
+
- do not log full tags or request details from the Engine
|
|
383
|
+
|
|
384
|
+
README warning:
|
|
385
|
+
|
|
386
|
+
> Do not expose this dashboard publicly. Tags may contain internal user, tenant, or feature identifiers.
|
|
387
|
+
|
|
388
|
+
## Styling
|
|
389
|
+
|
|
390
|
+
- ERB templates
|
|
391
|
+
- no JS
|
|
392
|
+
- no external fonts
|
|
393
|
+
- no external CSS
|
|
394
|
+
- no Chart.js
|
|
395
|
+
- no Tailwind dependency
|
|
396
|
+
- inline `<style>` in layout
|
|
397
|
+
- `.lct-` class prefix
|
|
398
|
+
- CSS bar charts through inline custom properties
|
|
399
|
+
|
|
400
|
+
Example:
|
|
401
|
+
|
|
402
|
+
```html
|
|
403
|
+
<div class="lct-bar" style="--lct-width: 73%"></div>
|
|
404
|
+
```
|
|
405
|
+
|
|
406
|
+
## Testing Plan
|
|
407
|
+
|
|
408
|
+
### Core Specs
|
|
409
|
+
|
|
410
|
+
- `group_by_period(:day)`
|
|
411
|
+
- `group_by_period(:month)`
|
|
412
|
+
- invalid period
|
|
413
|
+
- invalid column
|
|
414
|
+
- composition with existing scopes
|
|
415
|
+
|
|
416
|
+
### Engine Request Specs
|
|
417
|
+
|
|
418
|
+
- `/` empty DB
|
|
419
|
+
- `/` seeded DB
|
|
420
|
+
- `/calls`
|
|
421
|
+
- `/calls` filters narrow results
|
|
422
|
+
- `/calls/:id`
|
|
423
|
+
- `/calls/:missing`
|
|
424
|
+
- `/models`
|
|
425
|
+
- `/tags/feature`
|
|
426
|
+
- invalid tag key returns 400
|
|
427
|
+
- missing table renders setup error
|
|
428
|
+
|
|
429
|
+
### Service Specs
|
|
430
|
+
|
|
431
|
+
- `Dashboard::Filter`
|
|
432
|
+
- `Dashboard::Page`
|
|
433
|
+
- `Dashboard::TimeSeries`
|
|
434
|
+
- `Dashboard::OverviewStats`
|
|
435
|
+
- `Dashboard::TopModels`
|
|
436
|
+
- `Dashboard::TopTags`
|
|
437
|
+
|
|
438
|
+
### CI
|
|
439
|
+
|
|
440
|
+
- Ruby 3.3, 3.4
|
|
441
|
+
- Rails Engine specs on Rails 7.1, 7.2, 8.0
|
|
442
|
+
- SQLite default
|
|
443
|
+
- PostgreSQL job for period grouping and tag grouping
|
|
444
|
+
- MySQL best-effort only; not required for v0.2.0 CI
|
|
445
|
+
|
|
446
|
+
## Documentation
|
|
447
|
+
|
|
448
|
+
README additions:
|
|
449
|
+
|
|
450
|
+
- Dashboard quick start
|
|
451
|
+
- mount snippet
|
|
452
|
+
- Basic Auth snippet
|
|
453
|
+
- Devise snippet
|
|
454
|
+
- warning not to expose dashboard publicly
|
|
455
|
+
- note that Engine requires Rails 7.1+
|
|
456
|
+
- note that core middleware remains usable without Rails
|
|
457
|
+
|
|
458
|
+
## Release Strategy
|
|
459
|
+
|
|
460
|
+
1. Work on `codex/engine-dashboard`.
|
|
461
|
+
2. Implement `group_by_period`.
|
|
462
|
+
3. Add Engine skeleton.
|
|
463
|
+
4. Build Overview first.
|
|
464
|
+
5. Build Calls index.
|
|
465
|
+
6. Build Call details.
|
|
466
|
+
7. Build Models.
|
|
467
|
+
8. Build `/tags/:key`.
|
|
468
|
+
9. Dogfood in a real Rails app.
|
|
469
|
+
10. Release `0.2.0.alpha1`.
|
|
470
|
+
11. Fix issues from real usage.
|
|
471
|
+
12. Release `0.2.0.rc1`.
|
|
472
|
+
13. Release final `0.2.0`.
|
|
473
|
+
|
|
474
|
+
## v0.2.0 Acceptance Criteria
|
|
475
|
+
|
|
476
|
+
The release is ready when:
|
|
477
|
+
|
|
478
|
+
- `group_by_period(:day/:month)` is tested and documented
|
|
479
|
+
- Engine is opt-in and requires Rails 7.1+
|
|
480
|
+
- non-Rails core usage does not gain Rails runtime dependencies
|
|
481
|
+
- `/llm-costs` renders without JS or asset pipeline dependencies
|
|
482
|
+
- empty DB and missing table states are friendly
|
|
483
|
+
- seeded dashboard shows useful spend data
|
|
484
|
+
- filters cannot SQL-inject
|
|
485
|
+
- Overview is good enough for README screenshot
|
|
486
|
+
- README includes auth snippets
|
|
487
|
+
- full test suite and RuboCop pass
|
|
488
|
+
- alpha has been tested in one real Rails app
|