ruby_llm-monitoring 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 8823877c782a9d6e67c1aa975c2ed5f64312735141d456e7408d03b7fee0356f
+  data.tar.gz: 9568944cf43cbd83acbb4a455e240f7241515b74f38290547847e6cabe0b8a9d
+SHA512:
+  metadata.gz: b53443b86fca0140c277a54f9738cd100886b409b3aecf594626223d2dfcdd97d77518494b1e7f40696190b4d7a988fe3b2142f4f2d3dcf0f2f9b3930ceb256a
+  data.tar.gz: 36fcc184799318b6c70d2233cf5ecb598d52449bf7cbee71d7ec6f1d79949aca5c5f1482ee42b99f364c7a502de59fed09e64dbed240523fbe9d8e57429e089c

data/README.md

@@ -0,0 +1,179 @@
+# RubyLLM::Monitoring
+
+Monitor your LLM usage within your Rails application.
+
+## Installation
+
+> [!NOTE]
+> This engine relies on [RubyLLM](https://github.com/crmne/ruby_llm). Make sure you have it installed and configured.
+
+Add this line to your application's Gemfile:
+
+```ruby
+gem "ruby_llm-monitoring"
+```
+
+And then execute:
+
+```bash
+$ bundle
+```
+
+To copy and migrate RubyLLM::Monitoring's migrations, run:
+
+```
+$ rails ruby_llm_monitoring:install:migrations db:migrate
+```
+
+And then mount the engine in your `config/routes.rb`:
+
+```ruby
+Rails.application.routes.draw do
+  # ...
+
+  mount RubyLLM::Monitoring::Engine, at: "/monitoring"
+end
+```
+
+Now you should be able to browse to `/monitoring` and monitor your LLM usage.
+
+![metrics](./assets/metrics.png)
+![alerts](./assets/alerts.png)
+
+### Authentication and authorization
+
+RubyLLM::Monitoring leaves authentication and authorization to the user. If no authentication is enforced, `/monitoring` will be available to everyone.
+
+To enforce authentication, you can use route [constraints](https://guides.rubyonrails.org/routing.html#advanced-constraints), or set up a HTTP Basic auth middleware.
+
+For example, if you're using devise, you can do this:
+
+```ruby
+# config/routes.rb
+authenticate :user do
+  mount RubyLLM::Monitoring::Engine, at: "/monitoring"
+end
+```
+
+See more examples [here](https://github.com/heartcombo/devise/wiki/How-To%3A-Define-resource-actions-that-require-authentication-using-routes.rb).
+
+However, if you're using Rails' default authentication generator, or an authentication solution that doesn't provide constraints, you need to roll out your own solution:
+
+```ruby
+# config/routes.rb
+constraints ->(request) { Constraints::Auth.authenticated?(request) } do
+  mount RubyLLM::Monitoring::Engine, at: "/monitoring"
+end
+
+# lib/constraints/auth.rb
+class Constraints::Auth
+  def self.authenticated?(request)
+    cookies = ActionDispatch::Cookies::CookieJar.build(request, request.cookies)
+
+    Session.find_by id: cookies.signed[:session_id]
+  end
+end
+```
+
+You can also set up a HTTP Basic auth middleware in the engine:
+
+```ruby
+# config/initializers/ruby_llm-monitoring.rb
+RubyLLM::Monitoring::Engine.middleware.use(Rack::Auth::Basic) do |username, password|
+  ActiveSupport::SecurityUtils.secure_compare(Rails.application.credentials.ruby_llm_monitoring_username, username) &
+    ActiveSupport::SecurityUtils.secure_compare(Rails.application.credentials.ruby_llm_monitoring_password, password)
+end
+```
+
+## Alerts
+
+RubyLLM::Monitoring can send alerts when certain conditions are met. Useful for monitoring cost, errors, etc.
+
+### Configuration
+
+In `config/initializers/ruby_llm_monitoring.rb`, you can set the notification channels and alert rules:
+
+```ruby
+RubyLLM::Monitoring.channels = {
+  email: { to: "team@example.com" },
+  slack: { webhook_url: ENV["SLACK_WEBHOOK_URL"] },
+}
+
+# Default cooldown between repeated alerts (optional, defaults to 5 minutes)
+RubyLLM::Monitoring.alert_cooldown = 15.minutes
+
+RubyLLM::Monitoring.alert_rules += [{
+  time_range: -> { 1.hour.ago.. },
+  rule: ->(events) { events.where.not(exception_class: nil).count > 10 },
+  channels: [:slack],
+  message: { text: "More than 10 errors in the last hour" }
+}, {
+  time_range: -> { Time.current.at_beginning_of_month.. },
+  rule: ->(events) { events.sum(:cost) >= 500 },
+  channels: [:email, :slack],
+  message: { text: "More than $500 spent this month" }
+}]
+```
+
+### Rule options
+
+| Option       | Required | Description                                                                 |
+| ------------ | -------- | --------------------------------------------------------------------------- |
+| `time_range` | Yes      | Lambda returning a range for filtering events (e.g., `-> { 1.hour.ago.. }`) |
+| `rule`       | Yes      | Lambda receiving events scope, returns true to trigger alert                |
+| `channels`   | Yes      | Array of channel names to notify                                            |
+| `message`    | Yes      | Hash with `:text` key for the alert message                                 |
+| `id`         | No       | Identifier for debugging and cooldown tracking                              |
+| `cooldown`   | No       | Override default cooldown for this rule                                     |
+
+### Built-in channels
+
+#### Slack
+
+```ruby
+RubyLLM::Monitoring.channels = {
+  slack: {
+    webhook_url: ENV["SLACK_WEBHOOK_URL"]
+  }
+}
+```
+
+#### Email
+
+```ruby
+RubyLLM::Monitoring.channels = {
+  email: {
+    to: "team@example.com",
+    from: "alerts@example.com",  # optional
+    subject: "LLM Alert"         # optional
+  }
+}
+```
+
+### Custom channels
+
+Register custom notification channels:
+
+```ruby
+class PagerDutyChannel < RubyLLM::Monitoring::Channels::Base
+  def self.deliver(message, config)
+    # Your implementation
+    # message[:text] contains the alert text
+    # config contains channel configuration
+  end
+end
+
+RubyLLM::Monitoring.channel_registry.register(:pagerduty, PagerDutyChannel)
+
+RubyLLM::Monitoring.channels = {
+  pagerduty: { api_key: ENV["PAGERDUTY_API_KEY"] }
+}
+```
+
+## Contributing
+
+You can open an issue or a PR in GitHub.
+
+## License
+
+The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).

data/Rakefile

@@ -0,0 +1,8 @@
+require "bundler/setup"
+
+APP_RAKEFILE = File.expand_path("test/dummy/Rakefile", __dir__)
+load "rails/tasks/engine.rake"
+
+load "rails/tasks/statistics.rake"
+
+require "bundler/gem_tasks"

data/app/assets/stylesheets/ruby_llm/monitoring/application.css

@@ -0,0 +1,15 @@
+/*
+ * This is a manifest file that'll be compiled into application.css, which will include all the files
+ * listed below.
+ *
+ * Any CSS and SCSS file within this directory, lib/assets/stylesheets, vendor/assets/stylesheets,
+ * or any plugin's vendor/assets/stylesheets directory can be referenced here using a relative path.
+ *
+ * You're free to add application-wide styles to this file and they'll appear at the bottom of the
+ * compiled file so the styles you add here take precedence over styles defined in any other CSS/SCSS
+ * files in this directory. Styles in this file should be added after the last require_* statement.
+ * It is generally better to create a new file per style scope.
+ *
+ *= require_tree .
+ *= require_self
+ */