langsmithrb_rails 0.1.0 → 0.3.0

data/README.md

@@ -0,0 +1,421 @@
+# Langsmith Rails
+
+LangsmithrbRails provides seamless integration with [LangSmith](https://smith.langchain.com/) for your Rails applications. LangSmith is a platform for debugging, testing, evaluating, and monitoring LLM applications.
+
+This gem makes it easy to add LangSmith tracing to your Rails application, allowing you to monitor and debug your LLM operations. It provides a comprehensive set of features including request tracing, PII redaction, local buffering, evaluation frameworks, and more.
+
+## Features
+
+- 🔍 **Request Tracing**: Automatically trace HTTP requests with middleware
+- 🧩 **Service & Job Tracing**: Trace your services and background jobs
+- 🔒 **PII Redaction**: Automatically redact sensitive information
+- 💾 **Local Buffering**: Store traces locally and send them in batches
+- 📊 **Evaluations**: Evaluate LLM responses with customizable criteria
+- 🔄 **CI Integration**: Run evaluations in your CI pipeline
+- 🚀 **Demo Application**: Get started quickly with a sample application
+- 🔌 **LLM Provider Wrappers**: Trace OpenAI, Anthropic, and other LLM providers
+- 🌳 **Advanced Tracing**: Hierarchical run trees for complex workflows
+- 📡 **OpenTelemetry Integration**: Distributed tracing with OpenTelemetry
+- 🧪 **Enhanced Evaluation Framework**: String and LLM-based evaluators
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+```ruby
+gem 'langsmithrb_rails'
+```
+
+And then execute:
+
+```bash
+$ bundle install
+```
+
+Or install it yourself as:
+
+```bash
+$ gem install langsmithrb_rails
+```
+
+## Setup
+
+After installing the gem, run the install generator to set up LangSmith in your Rails application:
+
+```bash
+$ rails generate langsmithrb_rails:install
+```
+
+This will:
+
+1. Create a LangSmith initializer at `config/initializers/langsmith.rb`
+2. Generate a YAML configuration file at `config/langsmith.yml`
+3. Display post-install instructions
+
+## Configuration
+
+LangSmith is configured through environment variables and a YAML configuration file:
+
+```
+# Required
+LANGSMITH_API_KEY=your_api_key
+
+# Optional
+LANGSMITH_PROJECT=your_project_name
+LANGSMITH_SAMPLING=1.0  # Sampling rate (0.0 to 1.0)
+LANGSMITH_ENABLED=true  # Enable/disable tracing
+```
+
+You can get your API key from [LangSmith](https://smith.langchain.com/).
+
+The YAML configuration file (`config/langsmith.yml`) allows for environment-specific settings:
+
+```yaml
+default: &default
+  api_key: <%= ENV.fetch("LANGSMITH_API_KEY", nil) %>
+  project_name: <%= ENV.fetch("LANGSMITH_PROJECT", nil) %>
+  sampling_rate: <%= ENV.fetch("LANGSMITH_SAMPLING", 1.0).to_f %>
+  enabled: <%= ENV.fetch("LANGSMITH_ENABLED", "true") == "true" %>
+  redact_pii: true
+  timeout: 5
+
+development:
+  <<: *default
+
+test:
+  <<: *default
+
+production:
+  <<: *default
+  sampling_rate: <%= ENV.fetch("LANGSMITH_SAMPLING", 0.1).to_f %>
+```
+
+You can also configure LangSmith programmatically in your initializer:
+
+```ruby
+LangsmithrbRails.configure do |config|
+  config.enabled = true # Enable LangSmith tracing
+  config.api_key = "your_api_key" # Your LangSmith API key
+  config.project_name = "your_project" # Optional project name
+  config.sampling_rate = 0.1 # Sample 10% of traces
+  config.redact_pii = true # Redact PII from traces
+  
+  # Advanced tracing options
+  config.trace_all = true # Trace all operations
+  config.trace_level = "info" # Trace level (debug, info, warn, error, fatal)
+  
+  # OpenTelemetry options
+  config.otel_enabled = false # Enable OpenTelemetry integration
+  config.otel_service_name = "my-rails-app" # Service name for OpenTelemetry
+  
+  # Evaluation options
+  config.evaluation_enabled = true # Enable evaluation framework
+  
+  # Logging options
+  config.log_level = "info" # Log level (debug, info, warn, error, fatal)
+  config.log_to_stdout = true # Log to STDOUT
+end
+```
+
+## Generators
+
+LangsmithrbRails provides several generators to help you set up different features:
+
+### Install Generator
+
+```bash
+$ rails generate langsmithrb_rails:install
+```
+
+Sets up the basic configuration for LangSmith.
+
+### Tracing Generator
+
+```bash
+$ rails generate langsmithrb_rails:tracing
+```
+
+Adds middleware and concerns for request-level tracing:
+
+- Creates a middleware for tracing HTTP requests
+- Adds a concern for tracing services
+- Adds a concern for tracing background jobs
+- Updates your application configuration
+
+### Buffer Generator
+
+```bash
+$ rails generate langsmithrb_rails:buffer
+```
+
+Sets up local buffering for traces:
+
+- Creates a migration for the buffer table
+- Adds a model for the buffer
+- Adds a job for flushing the buffer
+- Adds rake tasks for manual flushing
+
+### Evals Generator
+
+```bash
+$ rails generate langsmithrb_rails:evals
+```
+
+Adds an evaluation framework for LLM responses:
+
+- Creates sample datasets for evaluation
+- Adds evaluation checks (correctness, LLM-graded)
+- Adds evaluation targets (HTTP, Ruby)
+- Adds rake tasks for running evaluations
+
+### CI Generator
+
+```bash
+$ rails generate langsmithrb_rails:ci
+```
+
+Sets up CI integration for LangSmith evaluations:
+
+- Creates a GitHub Actions workflow
+- Adds a script for generating evaluation summaries
+- Configures PR comments with evaluation results
+
+### Privacy Generator
+
+```bash
+$ rails generate langsmithrb_rails:privacy
+```
+
+Adds privacy features for LangSmith traces:
+
+- Creates a custom redactor for PII
+- Adds a privacy initializer
+- Generates a privacy configuration file
+
+### Demo Generator
+
+```bash
+$ rails generate langsmithrb_rails:demo
+```
+
+Adds a demo application with LangSmith tracing:
+
+- Creates a chat interface with LLM integration
+- Adds a service for interacting with LLMs
+- Configures tracing for all LLM operations
+
+## Usage
+
+### Basic Tracing
+
+Once configured, you can trace your code using the `LangsmithTraced` concern:
+
+```ruby
+class MyService
+  include LangsmithTraced
+  
+  def process(input)
+    langsmith_trace("my_operation", inputs: { input: input }) do |run|
+      # Your code here
+      result = do_something(input)
+      
+      # Record the output
+      run.outputs = { result: result }
+      
+      result
+    end
+  end
+end
+```
+
+### LLM Provider Wrappers
+
+You can wrap LLM providers to automatically trace all operations:
+
+```ruby
+# Wrap an OpenAI client
+require "openai"
+openai_client = OpenAI::Client.new(access_token: ENV["OPENAI_API_KEY"])
+traced_client = LangsmithrbRails.wrap(openai_client, provider: :openai, project_name: "my-project")
+
+# Use the traced client normally
+response = traced_client.chat(
+  parameters: {
+    model: "gpt-4",
+    messages: [{ role: "user", content: "Hello, world!" }]
+  }
+)
+
+# Wrap an Anthropic client
+require "anthropic"
+anthropic_client = Anthropic::Client.new(api_key: ENV["ANTHROPIC_API_KEY"])
+traced_client = LangsmithrbRails.wrap(anthropic_client, provider: :anthropic)
+
+# Use the traced client normally
+response = traced_client.messages(
+  model: "claude-3-opus-20240229",
+  max_tokens: 1024,
+  messages: [{ role: "user", content: "Hello, world!" }]
+)
+
+# Wrap any LLM client
+custom_client = MyCustomLLM.new
+traced_client = LangsmithrbRails.wrap(custom_client, provider: :llm)
+
+# Make any method traceable
+LangsmithrbRails.traceable(object, :method_name, run_name: "my-run")
+```
+
+### Tracing Background Jobs
+
+For background jobs, use the `LangsmithTracedJob` concern:
+
+```ruby
+class MyJob < ApplicationJob
+  include LangsmithTracedJob
+  
+  def perform(args)
+    # Job is automatically traced
+    # ...
+  end
+end
+```
+
+### Advanced Tracing with Run Trees
+
+For complex workflows, use run trees to create hierarchical traces:
+
+```ruby
+LangsmithrbRails.run(name: "parent-operation", run_type: "chain", inputs: { query: "Hello" }) do |parent_run|
+  # Do some work in the parent run
+  intermediate_result = process_query(parent_run.inputs[:query])
+  
+  # Create a child run
+  child_run = parent_run.create_child(
+    name: "child-operation",
+    run_type: "llm",
+    inputs: { prompt: intermediate_result }
+  )
+  
+  # Do some work in the child run
+  result = call_llm(child_run.inputs[:prompt])
+  
+  # End the child run with outputs
+  child_run.end(outputs: { response: result })
+  
+  # Return the final result
+  result
+end
+```
+
+### Local Buffering
+
+If you've set up buffering, traces will be stored locally and sent in batches. You can manually flush the buffer:
+
+```bash
+$ rails langsmith:flush
+```
+
+### OpenTelemetry Integration
+
+You can enable OpenTelemetry integration for distributed tracing:
+
+```ruby
+# Initialize OpenTelemetry
+LangsmithrbRails::OTEL.init(
+  service_name: "my-rails-app",
+  service_version: "1.0.0"
+)
+
+# Trace an operation with OpenTelemetry
+LangsmithrbRails::OTEL.trace("my-operation", attributes: { key: "value" }) do
+  # Your code here
+  result = do_something()
+  result
+end
+
+# Trace an LLM operation
+LangsmithrbRails::OTEL.trace_llm(
+  "llm-operation",
+  inputs: { prompt: "Hello, world!" },
+  run_type: "llm",
+  project_name: "my-project",
+  tags: ["tag1", "tag2"]
+) do
+  # Your LLM code here
+  response = llm.generate("Hello, world!")
+  response
+end
+```
+
+### Running Evaluations
+
+To run an evaluation:
+
+```bash
+$ rails langsmith:eval[sample,http,my_experiment]
+```
+
+Where:
+- `sample` is the dataset name
+- `http` is the target name
+- `my_experiment` is the experiment name
+
+### Enhanced Evaluation Framework
+
+Use the enhanced evaluation framework to evaluate LLM responses:
+
+```ruby
+# Create a string evaluator
+string_evaluator = LangsmithrbRails.evaluator(
+  :string,
+  match_type: :exact,
+  case_sensitive: false,
+  project_name: "my-project"
+)
+
+# Evaluate a prediction against a reference
+result = string_evaluator.evaluate("Hello, world!", "hello, world!")
+# => { score: 1.0, metadata: { match: true, match_type: "exact", case_sensitive: false } }
+
+# Create an LLM-based evaluator
+llm_client = OpenAI::Client.new(access_token: ENV["OPENAI_API_KEY"])
+llm_evaluator = LangsmithrbRails.evaluator(
+  :llm,
+  llm: llm_client,
+  criteria: "Evaluate the response for accuracy and relevance.",
+  project_name: "my-project"
+)
+
+# Evaluate a run
+result = llm_evaluator.evaluate_run("run-id")
+
+# Evaluate a dataset
+results = LangsmithrbRails::Evaluation.evaluate_dataset(
+  "dataset-id",
+  [string_evaluator, llm_evaluator],
+  experiment_name: "my-experiment"
+)
+```
+
+### Comparing Experiments
+
+To compare two experiments:
+
+```bash
+$ rails langsmith:compare[exp_a,exp_b]
+```
+
+## Development
+
+After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
+
+To install this gem onto your local machine, run `bundle exec rake install`.
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/protocolgrid/langsmithrb_rails.
+
+## License
+
+The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).

data/Rakefile

@@ -0,0 +1,10 @@
+# frozen_string_literal: true
+
+require "bundler/gem_tasks"
+require "rspec/core/rake_task"
+require "rubocop/rake_task"
+
+RSpec::Core::RakeTask.new(:spec)
+RuboCop::RakeTask.new
+
+task default: [:spec, :rubocop]

data/langsmithrb_rails.gemspec

@@ -0,0 +1,45 @@
+# frozen_string_literal: true
+
+require_relative "lib/langsmithrb_rails/version"
+
+Gem::Specification.new do |spec|
+  spec.name = "langsmithrb_rails"
+  spec.version = LangsmithrbRails::VERSION
+  spec.authors = ["Protocol Grid"]
+  spec.email = ["info@protocolgrid.com"]
+
+  spec.summary = "Rails integration for LangSmith tracing and monitoring"
+  spec.description = "Seamlessly integrate LangSmith tracing and monitoring into your Rails applications"
+  spec.homepage = "https://github.com/cdaviis/langsmithrb_rails"
+  spec.license = "MIT"
+  spec.required_ruby_version = ">= 2.7.0"
+
+  spec.metadata["homepage_uri"] = spec.homepage
+  spec.metadata["source_code_uri"] = "https://github.com/cdaviis/langsmithrb_rails"
+  spec.metadata["changelog_uri"] = "https://github.com/cdaviis/langsmithrb_rails/blob/main/CHANGELOG.md"
+
+  # Specify which files should be added to the gem when it is released.
+  # The `git ls-files -z` loads the files in the RubyGem that have been added into git.
+  spec.files = Dir.chdir(__dir__) do
+    `git ls-files -z`.split("\x0").reject do |f|
+      (File.expand_path(f) == __FILE__) || f.start_with?(*%w[bin/ test/ spec/ features/ .git .circleci appveyor])
+    end
+  end
+  spec.bindir = "exe"
+  spec.executables = spec.files.grep(%r{\Aexe/}) { |f| File.basename(f) }
+  spec.require_paths = ["lib"]
+
+  # Dependencies
+  spec.add_dependency "langsmithrb", "~> 0.1.0"
+  spec.add_dependency "rails", ">= 6.0.0"
+  
+  # Development dependencies
+  spec.add_development_dependency "rspec", "~> 3.0"
+  spec.add_development_dependency "rspec-rails", "~> 6.0"
+  spec.add_development_dependency "generator_spec", "~> 0.9"
+  spec.add_development_dependency "yard", "~> 0.9"
+  spec.add_development_dependency "rubocop", "~> 1.50"
+  spec.add_development_dependency "rubocop-rspec", "~> 2.22"
+  spec.add_development_dependency "rubocop-rails", "~> 2.19"
+  spec.add_development_dependency "simplecov", "~> 0.22.0"
+end

data/lib/generators/langsmithrb_rails/buffer/buffer_generator.rb

@@ -0,0 +1,94 @@
+# frozen_string_literal: true
+
+module LangsmithrbRails
+  module Generators
+    # Generator for adding LangSmith buffer to Rails applications
+    class BufferGenerator < Rails::Generators::Base
+      source_root File.expand_path("templates", __dir__)
+      
+      desc "Adds LangSmith buffer to your Rails application"
+      
+      def create_migration
+        timestamp = Time.now.utc.strftime("%Y%m%d%H%M%S")
+        template "migration.rb", "db/migrate/#{timestamp}_create_langsmith_run_buffers.rb"
+      end
+      
+      def create_model
+        template "langsmith_run_buffer.rb", "app/models/langsmith_run_buffer.rb"
+      end
+      
+      def create_job
+        template "flush_buffer_job.rb", "app/jobs/langsmith/flush_buffer_job.rb"
+      end
+      
+      def create_rake_task
+        template "langsmith.rake", "lib/tasks/langsmith.rake"
+      end
+      
+      def update_config
+        initializer_path = "config/initializers/langsmith.rb"
+        
+        if File.exist?(initializer_path)
+          buffer_config = "\n# Enable buffer for LangSmith traces\nLangsmithrbRails::Config.set(:use_buffer, true)\n"
+          
+          # Check if buffer config is already present
+          if File.read(initializer_path).include?("use_buffer")
+            say_status :skip, "Buffer config already present in initializer", :yellow
+          else
+            append_to_file initializer_path, buffer_config
+            say_status :append, "Added buffer config to initializer", :green
+          end
+        else
+          say_status :error, "Could not find langsmith initializer", :red
+          say_status :create, "Creating initializer with buffer config", :green
+          
+          create_file initializer_path do
+            <<~RUBY
+              # frozen_string_literal: true
+              
+              # LangSmith Rails Initializer
+              
+              # Configure LangSmith Rails
+              LangsmithrbRails.configure do |config|
+                # Whether LangSmith tracing is enabled
+                config.enabled = ENV["LANGSMITH_API_KEY"].present?
+                
+                # LangSmith API key
+                config.api_key = ENV["LANGSMITH_API_KEY"]
+                
+                # LangSmith project name
+                config.project_name = ENV["LANGSMITH_PROJECT"]
+              end
+              
+              # Enable buffer for LangSmith traces
+              LangsmithrbRails::Config.set(:use_buffer, true)
+            RUBY
+          end
+        end
+      end
+      
+      def create_directories
+        empty_directory "app/jobs/langsmith"
+      end
+      
+      def display_post_install_message
+        say "\n"
+        say "LangSmith buffer has been added to your Rails application! 🎉", :green
+        say "\n"
+        say "Next steps:", :yellow
+        say "  1. Run the migration to create the buffer table:", :yellow
+        say "     bin/rails db:migrate", :yellow
+        say "  2. Set up a job to flush the buffer periodically:", :yellow
+        say "     # In config/sidekiq.yml or equivalent", :yellow
+        say "     :schedule:", :yellow
+        say "       langsmith_flush_buffer:", :yellow
+        say "         cron: '*/5 * * * *'  # Every 5 minutes", :yellow
+        say "         class: LangsmithFlushBufferJob", :yellow
+        say "\n"
+        say "You can also flush the buffer manually:", :yellow
+        say "  bin/rails langsmith:flush", :yellow
+        say "\n"
+      end
+    end
+  end
+end

data/lib/generators/langsmithrb_rails/buffer/templates/create_langsmith_run_buffers.rb

@@ -0,0 +1,29 @@
+# frozen_string_literal: true
+
+# Migration for creating the langsmith_run_buffers table
+class CreateLangsmithRunBuffers < ActiveRecord::Migration[7.0]
+  def change
+    create_table :langsmith_run_buffers do |t|
+      t.string :name, null: false
+      t.string :run_type, null: false
+      t.string :status, null: false, default: "pending"
+      t.string :request_id
+      t.string :user_ref
+      t.string :run_id
+      t.string :parent_run_id
+      t.datetime :started_at
+      t.datetime :ended_at
+      t.json :meta, default: {}
+      t.json :payload, null: false
+      t.text :error
+      t.integer :retry_count, default: 0
+
+      t.timestamps
+    end
+    
+    add_index :langsmith_run_buffers, :status
+    add_index :langsmith_run_buffers, :run_id
+    add_index :langsmith_run_buffers, :request_id
+    add_index :langsmith_run_buffers, :retry_count
+  end
+end

data/lib/generators/langsmithrb_rails/buffer/templates/flush_buffer_job.rb

@@ -0,0 +1,40 @@
+# frozen_string_literal: true
+
+module Langsmith
+  # Job for flushing the LangSmith buffer
+  class FlushBufferJob < ApplicationJob
+    queue_as :langsmith
+
+    # Maximum number of traces to process in a single job
+    BATCH_SIZE = 100
+
+    # Process the job
+    # @param retry_failed [Boolean] Whether to retry failed traces
+    def perform(retry_failed: false)
+      # Get pending traces
+      traces = LangsmithRunBuffer.pending.limit(BATCH_SIZE)
+      process_traces(traces)
+
+      # Get failed traces if requested
+      if retry_failed
+        traces = LangsmithRunBuffer.ready_to_retry.limit(BATCH_SIZE)
+        process_traces(traces)
+      end
+    end
+
+    private
+
+    # Process a batch of traces
+    # @param traces [ActiveRecord::Relation] Traces to process
+    def process_traces(traces)
+      traces.find_each do |trace|
+        begin
+          trace.send_to_langsmith
+        rescue => e
+          trace.mark_failed(e.message)
+          Rails.logger.error("Failed to send trace #{trace.id}: #{e.message}")
+        end
+      end
+    end
+  end
+end