ruby_llm-evals 0.0.1 → 0.1.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 85722601415fbabae3058f547a2cb814db4b6c4895bbc31d5eaac0fd6af68fb9
-  data.tar.gz: 732663d65dfb1707a1bcc4188ed0f5d98ac062fba7f02ae018037d8b7e0129c1
+  metadata.gz: 3ac750c3e2f75518591afb49f7914ba0512f1c8837f76af0d111d1e20707f127
+  data.tar.gz: 1430a5bbf5ccaa37bd86fb0065ed472ab331ca3ff5ec78a00687d5b8c0998b8c
 SHA512:
-  metadata.gz: 32246c8e3d33f6230893a295c4a7dc347d81634a52d83b607b455c1522a55609ad22d866b636cbde7838fe54eb58ae8e7dbc29b98853aeba7859488de021a097
-  data.tar.gz: efa95f8b3f32cdde9b1fe40e649c9c028d73f85a31024a219903104dba0032a4ec47fd991c42a7bf0496d2325609ba3d0c3151472f6ba52e3a2c5b9bbf32a7df
+  metadata.gz: 2ae0ec74f8d651e3f55559767c5757a2b6680d8eb8fbd65a80696236c7e6104e089540c37ece8671b5150b22823f4aafd4ad6183d9584ffb95c9e28fc7f72f4c
+  data.tar.gz: 8b751eb6510b03aa7929a446b8435942e206814dd0c71edf4985bd46ff12a58f742b28bd31c251cdb49d38f6915330a204d7c582618e8d01bec4a684ab54ce2e

data/README.md

@@ -1,10 +1,12 @@
-# RubyLlm::Evals
-Short description and motivation.
+# RubyLLM::Evals
 
-## Usage
-How to use my plugin.
+Test, compare, and improve your LLM prompts within your Rails application.
 
 ## Installation
+
+> [!NOTE]
+> This engine relies on ActiveJob, ActiveStorage, and [RubyLLM](https://github.com/crmne/ruby_llm). Make sure you have them installed and configured.
+
 Add this line to your application's Gemfile:
 
 ```ruby
@@ -12,17 +14,187 @@ gem "ruby_llm-evals"
 ```
 
 And then execute:
+
 ```bash
 $ bundle
 ```
 
-Or install it yourself as:
-```bash
-$ gem install ruby_llm-evals
+To copy and migrate RubyLLM::Evals's migrations, run:
+
+```
+$ rails ruby_llm_evals:install:migrations db:migrate
+```
+
+And then mount the engine in your `config/routes.rb`:
+
+```ruby
+Rails.application.routes.draw do
+  # ...
+
+  mount RubyLLM::Evals::Engine, at: "/evals"
+end
+```
+
+Now you should be able to browse to `/evals` and create, test, compare, and improve your LLM prompts. Continue reading to see how a typical workflow looks like, and how you can leverage your app's data to add samples to your prompts.
+
+![prompts](./assets/prompts.png)
+![runs](./assets/runs.png)
+![run](./assets/run.png)
+
+### Authentication and authorization
+
+RubyLLM::Evals leaves authentication and authorization to the user. If no authentication is enforced, `/evals` will be available to everyone.
+
+To enforce authentication, you can use route [constraints](https://guides.rubyonrails.org/routing.html#advanced-constraints), or set up a HTTP Basic auth middleware.
+
+For example, if you're using devise, you can do this:
+
+```ruby
+# config/routes.rb
+authenticate :user do
+  mount RubyLLM::Evals::Engine, at: "/evals"
+end
+```
+
+See more examples [here](https://github.com/heartcombo/devise/wiki/How-To%3A-Define-resource-actions-that-require-authentication-using-routes.rb).
+
+However, if you're using Rails' default authentication generator, or an authentication solution that doesn't provide constraints, you need to roll out your own solution:
+
+```ruby
+# config/routes.rb
+constraints ->(request) { Constraints::Auth.authenticated?(request) } do
+  mount RubyLLM::Evals::Engine, at: "/evals"
+end
+
+# lib/constraints/auth.rb
+class Constraints::Auth
+  def self.authenticated?(request)
+    cookies = ActionDispatch::Cookies::CookieJar.build(request, request.cookies)
+
+    Session.find_by id: cookies.signed[:session_id]
+  end
+end
+```
+
+You can also set up a HTTP Basic auth middleware in the engine:
+
+```ruby
+# config/initializers/ruby_llm-evals.rb
+RubyLLM::Evals::Engine.middleware.use(Rack::Auth::Basic) do |username, password|
+  ActiveSupport::SecurityUtils.secure_compare(Rails.application.credentials.ruby_llm_evals_username, username) &
+    ActiveSupport::SecurityUtils.secure_compare(Rails.application.credentials.ruby_llm_evals_password, password)
+end
+```
+
+## Usage
+
+### Workflow
+
+A typical workflow looks like this:
+
+#### Create a prompt
+
+A prompt represents an LLM prompt template with:
+
+* Provider: see [available providers](https://github.com/crmne/ruby_llm/tree/main/lib/ruby_llm/providers)
+* Model: see [available models](https://github.com/crmne/ruby_llm/blob/main/lib/ruby_llm/models.json). In case you're selecting a local provider (eg. Ollama), you can enter the model name in a text field.
+* Instructions: optional, the system prompt.
+* Message: message template.
+* Temperature: optional, controls randomness (0.0 to 1.0). Lower values make output more focused and deterministic.
+* Params: optional, additional provider-specific parameters as JSON (e.g., `{"max_tokens": 1000}`).
+* Tools: optional, array of tool class names that the LLM can use (e.g., `["Weather", "Calculator"]`). See how tools are defined in [RubyLLM](https://rubyllm.com/tools/).
+* Schema: optional, a Ruby class name (e.g., `User`) to structure the LLM's response, or use "other" to provide a custom JSON schema in the Schema Other field. See [RubyLLM structured output](https://rubyllm.com/structured-output/).
+
+Both the instructions and the message template can contain liquid tags that will be rendered at runtime. To add variables, enclose them with braces. Eg: `{{name}}`.
+
+> [!NOTE]
+> In order to use a provider, you must have it configured in `config/initializers/ruby_llm.rb` as explained [here](https://rubyllm.com/configuration/#provider-configuration)
+
+#### Add samples
+
+When creating/editing a prompt you can add samples, where you can define:
+
+* Variables: a JSON that contains the values to use when executing the prompt. Eg: `{ "name": "Patricio" }`
+* Eval type: the evaluation criteria: exact match, contains, regex, or human review.
+* Expected output: optional if the eval type is `human`
+* Files: optional attachments.
+
+#### Run evaluations
+
+Once you have a prompt with its examples you can run the evaluations. This will enqueue a job that will create an run and run each sample with the current prompt configuration.
+
+The run will save the current prompt configuration for later analysis, such as the current provider/model, instructions, messages, variables, etc.
+
+#### Analyze the results
+
+You can view the accuracy, cost, and duration of the entire run and each individual prompt execution.
+
+If you chose the human review eval type, it's now that you can review if an eval passed or not.
+
+### Beyond a typical workflow
+
+#### Using your data to create prompts/samples
+
+Suppose you want to categorize images. You can have a prompt (eg. `image-categorization`) and then add your data to the eval set:
+
+```ruby
+prompt = RubyLLM::Evals::Prompt.find_by slug: "image-categorization"
+
+Image.where(category: nil).take(50).each do |image|
+  sample = prompt.samples.create eval_type: :human_judge
+  sample.files.attach image.attachment.blob
+end
+```
+
+Then you can iterate over the prompt trying to find the best configuration possible.
+
+#### Using the prompt
+
+Once you've tested and refined your prompt, you can use it in your application code.
+
+Execute prompts by their slug to get a response object with content and metadata:
+
+```ruby
+# Simple execution without variables
+response = RubyLLM::Evals::Prompt.execute("image-categorization")
+response.content  # => "landscape"
+
+# With variables
+response = RubyLLM::Evals::Prompt.execute(
+  "text-summarization",
+  variables: { "text" => "Long article content here..." }
+)
+response.content  # => "Brief summary of the article"
+
+# With file attachments
+response = RubyLLM::Evals::Prompt.execute(
+  "image-categorization",
+  files: [image.attachment.blob]
+)
+response.content  # => "person"
+
+# Access token counts and metadata
+response = RubyLLM::Evals::Prompt.execute(
+  "sentiment-analysis",
+  variables: { "text" => "I love this product!" }
+)
+response.content        # => "positive"
+response.input_tokens   # => 25
+response.output_tokens  # => 3
+```
+
+You can also execute a prompt directly on a Prompt instance:
+
+```ruby
+prompt = RubyLLM::Evals::Prompt.find_by(slug: "sentiment-analysis")
+response = prompt.execute(variables: { "text" => "I love this product!" })
+response.content  # => "positive"
 ```
 
 ## Contributing
-Contribution directions go here.
+
+You can open an issue or a PR in GitHub.
 
 ## License
+
 The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).

data/Rakefile

@@ -3,6 +3,4 @@ require "bundler/setup"
 APP_RAKEFILE = File.expand_path("test/dummy/Rakefile", __dir__)
 load "rails/tasks/engine.rake"
 
-load "rails/tasks/statistics.rake"
-
 require "bundler/gem_tasks"

data/app/assets/stylesheets/ruby_llm/evals/application.css

@@ -0,0 +1,15 @@
+/*
+ * This is a manifest file that'll be compiled into application.css, which will include all the files
+ * listed below.
+ *
+ * Any CSS and SCSS file within this directory, lib/assets/stylesheets, vendor/assets/stylesheets,
+ * or any plugin's vendor/assets/stylesheets directory can be referenced here using a relative path.
+ *
+ * You're free to add application-wide styles to this file and they'll appear at the bottom of the
+ * compiled file so the styles you add here take precedence over styles defined in any other CSS/SCSS
+ * files in this directory. Styles in this file should be added after the last require_* statement.
+ * It is generally better to create a new file per style scope.
+ *
+ *= require_tree .
+ *= require_self
+ */