langsmithrb_rails 0.1.0 → 0.1.1

data/README.md

@@ -0,0 +1,268 @@
+# Langsmith Rails
+
+LangsmithrbRails provides seamless integration with [LangSmith](https://smith.langchain.com/) for your Rails applications. LangSmith is a platform for debugging, testing, evaluating, and monitoring LLM applications.
+
+This gem makes it easy to add LangSmith tracing to your Rails application, allowing you to monitor and debug your LLM operations. It provides a comprehensive set of features including request tracing, PII redaction, local buffering, evaluation frameworks, and more.
+
+## Features
+
+- 🔍 **Request Tracing**: Automatically trace HTTP requests with middleware
+- 🧩 **Service & Job Tracing**: Trace your services and background jobs
+- 🔒 **PII Redaction**: Automatically redact sensitive information
+- 💾 **Local Buffering**: Store traces locally and send them in batches
+- 📊 **Evaluations**: Evaluate LLM responses with customizable criteria
+- 🔄 **CI Integration**: Run evaluations in your CI pipeline
+- 🚀 **Demo Application**: Get started quickly with a sample application
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+```ruby
+gem 'langsmithrb_rails'
+```
+
+And then execute:
+
+```bash
+$ bundle install
+```
+
+Or install it yourself as:
+
+```bash
+$ gem install langsmithrb_rails
+```
+
+## Setup
+
+After installing the gem, run the install generator to set up LangSmith in your Rails application:
+
+```bash
+$ rails generate langsmithrb_rails:install
+```
+
+This will:
+
+1. Create a LangSmith initializer at `config/initializers/langsmith.rb`
+2. Generate a YAML configuration file at `config/langsmith.yml`
+3. Display post-install instructions
+
+## Configuration
+
+LangSmith is configured through environment variables and a YAML configuration file:
+
+```
+# Required
+LANGSMITH_API_KEY=your_api_key
+
+# Optional
+LANGSMITH_PROJECT=your_project_name
+LANGSMITH_SAMPLING=1.0  # Sampling rate (0.0 to 1.0)
+LANGSMITH_ENABLED=true  # Enable/disable tracing
+```
+
+You can get your API key from [LangSmith](https://smith.langchain.com/).
+
+The YAML configuration file (`config/langsmith.yml`) allows for environment-specific settings:
+
+```yaml
+default: &default
+  api_key: <%= ENV.fetch("LANGSMITH_API_KEY", nil) %>
+  project_name: <%= ENV.fetch("LANGSMITH_PROJECT", nil) %>
+  sampling_rate: <%= ENV.fetch("LANGSMITH_SAMPLING", 1.0).to_f %>
+  enabled: <%= ENV.fetch("LANGSMITH_ENABLED", "true") == "true" %>
+  redact_pii: true
+  timeout: 5
+
+development:
+  <<: *default
+
+test:
+  <<: *default
+
+production:
+  <<: *default
+  sampling_rate: <%= ENV.fetch("LANGSMITH_SAMPLING", 0.1).to_f %>
+```
+
+You can also configure LangSmith programmatically in your initializer:
+
+```ruby
+LangsmithrbRails.configure do |config|
+  config.enabled = true # Enable LangSmith tracing
+  config.api_key = "your_api_key" # Your LangSmith API key
+  config.project_name = "your_project" # Optional project name
+  config.sampling_rate = 0.1 # Sample 10% of traces
+  config.redact_pii = true # Redact PII from traces
+end
+```
+
+## Generators
+
+LangsmithrbRails provides several generators to help you set up different features:
+
+### Install Generator
+
+```bash
+$ rails generate langsmithrb_rails:install
+```
+
+Sets up the basic configuration for LangSmith.
+
+### Tracing Generator
+
+```bash
+$ rails generate langsmithrb_rails:tracing
+```
+
+Adds middleware and concerns for request-level tracing:
+
+- Creates a middleware for tracing HTTP requests
+- Adds a concern for tracing services
+- Adds a concern for tracing background jobs
+- Updates your application configuration
+
+### Buffer Generator
+
+```bash
+$ rails generate langsmithrb_rails:buffer
+```
+
+Sets up local buffering for traces:
+
+- Creates a migration for the buffer table
+- Adds a model for the buffer
+- Adds a job for flushing the buffer
+- Adds rake tasks for manual flushing
+
+### Evals Generator
+
+```bash
+$ rails generate langsmithrb_rails:evals
+```
+
+Adds an evaluation framework for LLM responses:
+
+- Creates sample datasets for evaluation
+- Adds evaluation checks (correctness, LLM-graded)
+- Adds evaluation targets (HTTP, Ruby)
+- Adds rake tasks for running evaluations
+
+### CI Generator
+
+```bash
+$ rails generate langsmithrb_rails:ci
+```
+
+Sets up CI integration for LangSmith evaluations:
+
+- Creates a GitHub Actions workflow
+- Adds a script for generating evaluation summaries
+- Configures PR comments with evaluation results
+
+### Privacy Generator
+
+```bash
+$ rails generate langsmithrb_rails:privacy
+```
+
+Adds privacy features for LangSmith traces:
+
+- Creates a custom redactor for PII
+- Adds a privacy initializer
+- Generates a privacy configuration file
+
+### Demo Generator
+
+```bash
+$ rails generate langsmithrb_rails:demo
+```
+
+Adds a demo application with LangSmith tracing:
+
+- Creates a chat interface with LLM integration
+- Adds a service for interacting with LLMs
+- Configures tracing for all LLM operations
+
+## Usage
+
+### Basic Tracing
+
+Once configured, you can trace your code using the `LangsmithTraced` concern:
+
+```ruby
+class MyService
+  include LangsmithTraced
+  
+  def process(input)
+    langsmith_trace("my_operation", inputs: { input: input }) do |run|
+      # Your code here
+      result = do_something(input)
+      
+      # Record the output
+      run.outputs = { result: result }
+      
+      result
+    end
+  end
+end
+```
+
+### Tracing Background Jobs
+
+For background jobs, use the `LangsmithTracedJob` concern:
+
+```ruby
+class MyJob < ApplicationJob
+  include LangsmithTracedJob
+  
+  def perform(args)
+    # Job is automatically traced
+    # ...
+  end
+end
+```
+
+### Local Buffering
+
+If you've set up buffering, traces will be stored locally and sent in batches. You can manually flush the buffer:
+
+```bash
+$ rails langsmith:flush
+```
+
+### Running Evaluations
+
+To run an evaluation:
+
+```bash
+$ rails langsmith:eval[sample,http,my_experiment]
+```
+
+Where:
+- `sample` is the dataset name
+- `http` is the target name
+- `my_experiment` is the experiment name
+
+### Comparing Experiments
+
+To compare two experiments:
+
+```bash
+$ rails langsmith:compare[exp_a,exp_b]
+```
+
+## Development
+
+After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
+
+To install this gem onto your local machine, run `bundle exec rake install`.
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/protocolgrid/langsmithrb_rails.
+
+## License
+
+The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).

data/Rakefile

@@ -0,0 +1,10 @@
+# frozen_string_literal: true
+
+require "bundler/gem_tasks"
+require "rspec/core/rake_task"
+require "rubocop/rake_task"
+
+RSpec::Core::RakeTask.new(:spec)
+RuboCop::RakeTask.new
+
+task default: [:spec, :rubocop]

data/langsmithrb_rails.gemspec

@@ -0,0 +1,45 @@
+# frozen_string_literal: true
+
+require_relative "lib/langsmithrb_rails/version"
+
+Gem::Specification.new do |spec|
+  spec.name = "langsmithrb_rails"
+  spec.version = LangsmithrbRails::VERSION
+  spec.authors = ["Protocol Grid"]
+  spec.email = ["info@protocolgrid.com"]
+
+  spec.summary = "Rails integration for LangSmith tracing and monitoring"
+  spec.description = "Seamlessly integrate LangSmith tracing and monitoring into your Rails applications"
+  spec.homepage = "https://github.com/cdaviis/langsmithrb_rails"
+  spec.license = "MIT"
+  spec.required_ruby_version = ">= 2.7.0"
+
+  spec.metadata["homepage_uri"] = spec.homepage
+  spec.metadata["source_code_uri"] = "https://github.com/cdaviis/langsmithrb_rails"
+  spec.metadata["changelog_uri"] = "https://github.com/cdaviis/langsmithrb_rails/blob/main/CHANGELOG.md"
+
+  # Specify which files should be added to the gem when it is released.
+  # The `git ls-files -z` loads the files in the RubyGem that have been added into git.
+  spec.files = Dir.chdir(__dir__) do
+    `git ls-files -z`.split("\x0").reject do |f|
+      (File.expand_path(f) == __FILE__) || f.start_with?(*%w[bin/ test/ spec/ features/ .git .circleci appveyor])
+    end
+  end
+  spec.bindir = "exe"
+  spec.executables = spec.files.grep(%r{\Aexe/}) { |f| File.basename(f) }
+  spec.require_paths = ["lib"]
+
+  # Dependencies
+  spec.add_dependency "langsmithrb", "~> 0.1.0"
+  spec.add_dependency "rails", ">= 6.0.0"
+  
+  # Development dependencies
+  spec.add_development_dependency "rspec", "~> 3.0"
+  spec.add_development_dependency "rspec-rails", "~> 6.0"
+  spec.add_development_dependency "generator_spec", "~> 0.9"
+  spec.add_development_dependency "yard", "~> 0.9"
+  spec.add_development_dependency "rubocop", "~> 1.50"
+  spec.add_development_dependency "rubocop-rspec", "~> 2.22"
+  spec.add_development_dependency "rubocop-rails", "~> 2.19"
+  spec.add_development_dependency "simplecov", "~> 0.22.0"
+end

data/lib/generators/langsmithrb_rails/buffer/buffer_generator.rb

@@ -0,0 +1,94 @@
+# frozen_string_literal: true
+
+module LangsmithrbRails
+  module Generators
+    # Generator for adding LangSmith buffer to Rails applications
+    class BufferGenerator < Rails::Generators::Base
+      source_root File.expand_path("templates", __dir__)
+      
+      desc "Adds LangSmith buffer to your Rails application"
+      
+      def create_migration
+        timestamp = Time.now.utc.strftime("%Y%m%d%H%M%S")
+        template "migration.rb", "db/migrate/#{timestamp}_create_langsmith_run_buffers.rb"
+      end
+      
+      def create_model
+        template "langsmith_run_buffer.rb", "app/models/langsmith_run_buffer.rb"
+      end
+      
+      def create_job
+        template "flush_buffer_job.rb", "app/jobs/langsmith/flush_buffer_job.rb"
+      end
+      
+      def create_rake_task
+        template "langsmith.rake", "lib/tasks/langsmith.rake"
+      end
+      
+      def update_config
+        initializer_path = "config/initializers/langsmith.rb"
+        
+        if File.exist?(initializer_path)
+          buffer_config = "\n# Enable buffer for LangSmith traces\nLangsmithrbRails::Config.set(:use_buffer, true)\n"
+          
+          # Check if buffer config is already present
+          if File.read(initializer_path).include?("use_buffer")
+            say_status :skip, "Buffer config already present in initializer", :yellow
+          else
+            append_to_file initializer_path, buffer_config
+            say_status :append, "Added buffer config to initializer", :green
+          end
+        else
+          say_status :error, "Could not find langsmith initializer", :red
+          say_status :create, "Creating initializer with buffer config", :green
+          
+          create_file initializer_path do
+            <<~RUBY
+              # frozen_string_literal: true
+              
+              # LangSmith Rails Initializer
+              
+              # Configure LangSmith Rails
+              LangsmithrbRails.configure do |config|
+                # Whether LangSmith tracing is enabled
+                config.enabled = ENV["LANGSMITH_API_KEY"].present?
+                
+                # LangSmith API key
+                config.api_key = ENV["LANGSMITH_API_KEY"]
+                
+                # LangSmith project name
+                config.project_name = ENV["LANGSMITH_PROJECT"]
+              end
+              
+              # Enable buffer for LangSmith traces
+              LangsmithrbRails::Config.set(:use_buffer, true)
+            RUBY
+          end
+        end
+      end
+      
+      def create_directories
+        empty_directory "app/jobs/langsmith"
+      end
+      
+      def display_post_install_message
+        say "\n"
+        say "LangSmith buffer has been added to your Rails application! 🎉", :green
+        say "\n"
+        say "Next steps:", :yellow
+        say "  1. Run the migration to create the buffer table:", :yellow
+        say "     bin/rails db:migrate", :yellow
+        say "  2. Set up a job to flush the buffer periodically:", :yellow
+        say "     # In config/sidekiq.yml or equivalent", :yellow
+        say "     :schedule:", :yellow
+        say "       langsmith_flush_buffer:", :yellow
+        say "         cron: '*/5 * * * *'  # Every 5 minutes", :yellow
+        say "         class: LangsmithFlushBufferJob", :yellow
+        say "\n"
+        say "You can also flush the buffer manually:", :yellow
+        say "  bin/rails langsmith:flush", :yellow
+        say "\n"
+      end
+    end
+  end
+end

data/lib/generators/langsmithrb_rails/buffer/templates/create_langsmith_run_buffers.rb

@@ -0,0 +1,29 @@
+# frozen_string_literal: true
+
+# Migration for creating the langsmith_run_buffers table
+class CreateLangsmithRunBuffers < ActiveRecord::Migration[7.0]
+  def change
+    create_table :langsmith_run_buffers do |t|
+      t.string :name, null: false
+      t.string :run_type, null: false
+      t.string :status, null: false, default: "pending"
+      t.string :request_id
+      t.string :user_ref
+      t.string :run_id
+      t.string :parent_run_id
+      t.datetime :started_at
+      t.datetime :ended_at
+      t.json :meta, default: {}
+      t.json :payload, null: false
+      t.text :error
+      t.integer :retry_count, default: 0
+
+      t.timestamps
+    end
+    
+    add_index :langsmith_run_buffers, :status
+    add_index :langsmith_run_buffers, :run_id
+    add_index :langsmith_run_buffers, :request_id
+    add_index :langsmith_run_buffers, :retry_count
+  end
+end

data/lib/generators/langsmithrb_rails/buffer/templates/flush_buffer_job.rb

@@ -0,0 +1,40 @@
+# frozen_string_literal: true
+
+module Langsmith
+  # Job for flushing the LangSmith buffer
+  class FlushBufferJob < ApplicationJob
+    queue_as :langsmith
+
+    # Maximum number of traces to process in a single job
+    BATCH_SIZE = 100
+
+    # Process the job
+    # @param retry_failed [Boolean] Whether to retry failed traces
+    def perform(retry_failed: false)
+      # Get pending traces
+      traces = LangsmithRunBuffer.pending.limit(BATCH_SIZE)
+      process_traces(traces)
+
+      # Get failed traces if requested
+      if retry_failed
+        traces = LangsmithRunBuffer.ready_to_retry.limit(BATCH_SIZE)
+        process_traces(traces)
+      end
+    end
+
+    private
+
+    # Process a batch of traces
+    # @param traces [ActiveRecord::Relation] Traces to process
+    def process_traces(traces)
+      traces.find_each do |trace|
+        begin
+          trace.send_to_langsmith
+        rescue => e
+          trace.mark_failed(e.message)
+          Rails.logger.error("Failed to send trace #{trace.id}: #{e.message}")
+        end
+      end
+    end
+  end
+end

data/lib/generators/langsmithrb_rails/buffer/templates/langsmith.rake

@@ -0,0 +1,71 @@
+# frozen_string_literal: true
+
+namespace :langsmith do
+  desc "Flush the LangSmith buffer"
+  task flush: :environment do
+    puts "Flushing LangSmith buffer..."
+    
+    # Get counts before
+    pending_count = LangsmithRunBuffer.pending.count
+    failed_count = LangsmithRunBuffer.ready_to_retry.count
+    
+    # Process pending traces
+    Langsmith::FlushBufferJob.new.perform(retry_failed: true)
+    
+    # Get counts after
+    pending_after = LangsmithRunBuffer.pending.count
+    failed_after = LangsmithRunBuffer.ready_to_retry.count
+    sent_count = pending_count + failed_count - pending_after - failed_after
+    
+    puts "Sent #{sent_count} traces to LangSmith"
+    puts "#{pending_after} traces still pending"
+    puts "#{failed_after} traces failed"
+  end
+  
+  desc "Replay traces from a file"
+  task :replay, [:file] => :environment do |_t, args|
+    file_path = args[:file]
+    
+    unless file_path && File.exist?(file_path)
+      puts "Error: File not found or not specified"
+      puts "Usage: rake langsmith:replay[path/to/file.json]"
+      exit 1
+    end
+    
+    puts "Replaying traces from #{file_path}..."
+    
+    # Load the file
+    traces = JSON.parse(File.read(file_path))
+    
+    # Create buffer entries
+    traces.each do |trace|
+      LangsmithRunBuffer.create!(
+        name: trace["name"] || "replayed_trace",
+        run_type: trace["run_type"] || "llm",
+        status: "pending",
+        request_id: trace["request_id"],
+        user_ref: trace["user_ref"],
+        run_id: trace["run_id"],
+        parent_run_id: trace["parent_run_id"],
+        started_at: trace["started_at"],
+        ended_at: trace["ended_at"],
+        meta: trace["meta"] || {},
+        payload: trace,
+        error: nil,
+        retry_count: 0
+      )
+    end
+    
+    puts "Created #{traces.size} buffer entries"
+    puts "Run rake langsmith:flush to send them to LangSmith"
+  end
+  
+  desc "Clean up old sent traces"
+  task cleanup: :environment do
+    # Delete traces that were sent more than 7 days ago
+    cutoff_date = 7.days.ago
+    count = LangsmithRunBuffer.sent.where("updated_at < ?", cutoff_date).delete_all
+    
+    puts "Deleted #{count} old traces from the buffer"
+  end
+end

data/lib/generators/langsmithrb_rails/buffer/templates/langsmith_run_buffer.rb

@@ -0,0 +1,70 @@
+# frozen_string_literal: true
+
+# Model for buffering LangSmith traces
+class LangsmithRunBuffer < ApplicationRecord
+  # Validations
+  validates :name, presence: true
+  validates :run_type, presence: true
+  validates :status, presence: true
+  
+  # Scopes
+  scope :pending, -> { where(status: "pending") }
+  scope :failed, -> { where(status: "failed") }
+  scope :sent, -> { where(status: "sent") }
+  scope :ready_to_retry, -> { where(status: "failed").where("retry_count < ?", 5) }
+  
+  # Send the trace to LangSmith
+  # @return [Boolean] Whether the trace was sent successfully
+  def send_to_langsmith
+    client = LangsmithrbRails::Client.new
+    
+    if run_id.present?
+      # This is an update operation
+      response = client.update_run(run_id, payload)
+    else
+      # This is a create operation
+      response = client.create_run(payload)
+      
+      # Store the run ID if available
+      if response&.dig(:status) == 200 && response&.dig(:body, "id").present?
+        update(run_id: response.dig(:body, "id"))
+      end
+    end
+    
+    if response&.dig(:status) == 200
+      update(status: "sent")
+      true
+    else
+      increment!(:retry_count)
+      update(
+        status: "failed", 
+        error: response&.dig(:error) || "Unknown error"
+      )
+      false
+    end
+  rescue => e
+    increment!(:retry_count)
+    update(
+      status: "failed", 
+      error: e.message
+    )
+    false
+  end
+  
+  # Mark the trace as failed
+  # @param error [String] Error message
+  # @return [Boolean] Whether the update was successful
+  def mark_failed(error)
+    increment!(:retry_count)
+    update(
+      status: "failed", 
+      error: error
+    )
+  end
+  
+  # Mark the trace as sent
+  # @return [Boolean] Whether the update was successful
+  def mark_sent
+    update(status: "sent")
+  end
+end

data/lib/generators/langsmithrb_rails/buffer/templates/migration.rb

@@ -0,0 +1,28 @@
+# frozen_string_literal: true
+
+class CreateLangsmithRunBuffers < ActiveRecord::Migration[6.0]
+  def change
+    create_table :langsmith_run_buffers do |t|
+      t.string :name, null: false
+      t.string :run_type, null: false
+      t.string :status, null: false, default: "pending"
+      t.string :request_id
+      t.string :user_ref
+      t.string :run_id
+      t.string :parent_run_id
+      t.datetime :started_at
+      t.datetime :ended_at
+      t.jsonb :meta
+      t.jsonb :payload
+      t.text :error
+      t.integer :retry_count, null: false, default: 0
+      
+      t.timestamps
+    end
+    
+    add_index :langsmith_run_buffers, :status
+    add_index :langsmith_run_buffers, :request_id
+    add_index :langsmith_run_buffers, :run_id
+    add_index :langsmith_run_buffers, :parent_run_id
+  end
+end

data/lib/generators/langsmithrb_rails/ci/ci_generator.rb

@@ -0,0 +1,37 @@
+# frozen_string_literal: true
+
+module LangsmithrbRails
+  module Generators
+    # Generator for adding CI integration for LangSmith evaluations
+    class CiGenerator < Rails::Generators::Base
+      source_root File.expand_path("templates", __dir__)
+      
+      desc "Adds CI integration for LangSmith evaluations"
+      
+      def create_github_workflow
+        empty_directory ".github/workflows"
+        template "langsmith-evals.yml", ".github/workflows/langsmith-evals.yml"
+      end
+      
+      def create_ci_script
+        empty_directory "script/ci"
+        template "langsmith_export_summary.rb", "script/ci/langsmith_export_summary.rb"
+        chmod "script/ci/langsmith_export_summary.rb", 0755
+      end
+      
+      def display_post_install_message
+        say "\n"
+        say "LangSmith CI integration has been added to your Rails application! 🎉", :green
+        say "\n"
+        say "This adds:", :yellow
+        say "  1. GitHub Actions workflow for running evaluations", :yellow
+        say "  2. Script for generating evaluation summaries for PR comments", :yellow
+        say "\n"
+        say "To customize the CI configuration:", :yellow
+        say "  1. Edit .github/workflows/langsmith-evals.yml", :yellow
+        say "  2. Configure the evaluation dataset and target in the workflow", :yellow
+        say "\n"
+      end
+    end
+  end
+end