langsmithrb 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 690abea56aa77b88aacff5ec66936e3e9a7dcb2c72b3aadf5a88fa4d45d469d6
+  data.tar.gz: 7ee637182e89738d54366520fe536891c3add80c0239b0841862ac7204635cc8
+SHA512:
+  metadata.gz: cc8d65163bd826bc11aabc3bb84a4962c65b21985cc875885d44507fb901cdd5f9dc8cd9aac0dcd21ac932e8d38f3464daa88314666dfcc37fc90ab0a52f0e58
+  data.tar.gz: 9fd9a7069982e9ad8001b37e227f0a5c6a6615e9af7adb7c4522150554765734bb83e25af6feb39f49ae58de98e01b4f4ca14c801848a803943a457230f10435

data/README.md

@@ -0,0 +1,626 @@
+# 💎🔗 LangSmith for Ruby
+
+⚡ The fastest way to integrate LangSmith tracing and evaluations into your Ruby applications ⚡
+
+This gem provides a comprehensive Ruby client for [LangSmith](https://smith.langchain.com/), a platform for debugging, testing, evaluating, and monitoring LLM applications.
+
+[![Tests status](https://github.com/cdaviis/langsmithrb/actions/workflows/ci.yml/badge.svg?branch=main)](https://github.com/cdaviis/langsmithrb/actions)
+[![Gem Version](https://badge.fury.io/rb/langsmithrb.svg)](https://badge.fury.io/rb/langsmithrb)
+[![License](https://img.shields.io/badge/license-MIT-green.svg)](https://github.com/cdaviis/langsmithrb/blob/main/LICENSE.txt)
+
+## Use Cases
+
+- Trace and monitor LLM applications
+- Create and manage datasets for testing
+- Run evaluations on LLM outputs
+- Collect and analyze feedback on LLM responses
+- Manage teams, webhooks, and API keys
+- Perform bulk operations on runs
+- Access analytics for projects and datasets
+- Integrate with your Ruby applications
+
+## Table of Contents
+
+- [Installation](#installation)
+- [Configuration](#configuration)
+- [Usage](#usage)
+  - [Client](#client)
+  - [Runs](#runs)
+  - [Projects](#projects)
+  - [Datasets](#datasets)
+  - [Evaluations](#evaluations)
+  - [Feedback](#feedback)
+  - [Tracer Sessions](#tracer-sessions)
+  - [Organizations](#organizations)
+  - [API Keys](#api-keys)
+  - [Tenants](#tenants)
+  - [Tags](#tags)
+  - [Prompts](#prompts)
+  - [Annotation Queues](#annotation-queues)
+  - [Feedback Configurations](#feedback-configurations)
+  - [Analytics](#analytics)
+  - [Comments](#comments)
+  - [Events](#events)
+  - [Settings](#settings)
+  - [Bulk Operations](#bulk-operations)
+  - [Webhooks](#webhooks)
+  - [Team Management](#team-management)
+- [Error Handling](#error-handling)
+- [Development](#development)
+- [Contributing](#contributing)
+- [License](#license)
+
+## Dependencies
+
+* Ruby 3.1+
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+```ruby
+gem 'langsmithrb'
+```
+
+And then execute:
+
+```bash
+$ bundle install
+```
+
+Or install it yourself as:
+
+```bash
+$ gem install langsmithrb
+```
+
+## Configuration
+
+You'll need a LangSmith API key to use this gem. You can get one by signing up at [LangSmith](https://smith.langchain.com/).
+
+There are several ways to configure the LangSmith client:
+
+### Environment Variables
+
+Set the following environment variables:
+
+```bash
+LANGSMITH_API_KEY=your_api_key
+LANGSMITH_API_URL=https://api.smith.langchain.com  # Optional, this is the default
+```
+
+### Global Configuration
+
+```ruby
+require 'langsmith'
+
+Langsmith.api_key = "your_api_key"
+Langsmith.api_url = "https://api.smith.langchain.com"  # Optional
+```
+
+### Client Instance Configuration
+
+```ruby
+require 'langsmith'
+
+client = Langsmith::Client.new(
+  api_key: "your_api_key",
+  api_url: "https://api.smith.langchain.com"  # Optional
+)
+```
+
+## Usage
+
+### Client
+
+The `Langsmith::Client` class is the main interface for interacting with the LangSmith API:
+
+```ruby
+require 'langsmith'
+
+# Initialize the client
+client = Langsmith::Client.new(api_key: "your_api_key")
+
+# Create a new project
+project = client.create_project(name: "my-project", description: "My awesome LLM project")
+
+# List all projects
+projects = client.list_projects
+```
+
+### Runs
+
+Runs represent individual executions of LLM calls, chains, or tools:
+
+```ruby
+# Create a new run
+run = client.create_run(
+  name: "llm-call",
+  run_type: "llm",
+  project_name: "my-project",
+  inputs: { prompt: "Tell me a joke about Ruby programming" }
+)
+
+# Update a run with outputs
+run.end(outputs: { response: "Why did the Ruby developer go broke? Because he used all his gems!" })
+
+# Or update with an error
+run.end(error: "API connection failed")
+
+# Get a run by ID
+run = client.get_run(run_id: "run_123")
+
+# List runs in a project
+runs = client.list_runs(project_name: "my-project", limit: 10)
+```
+
+### Projects
+
+Projects help you organize related runs:
+
+```ruby
+# Create a new project
+project = client.create_project(name: "my-project", description: "My awesome LLM project")
+
+# Get a project by name
+project = client.get_project(name: "my-project")
+
+# List all projects
+projects = client.list_projects(limit: 10)
+
+# Create a run in a specific project
+run = project.create_run(
+  name: "llm-call",
+  run_type: "llm",
+  inputs: { prompt: "Tell me a joke about Ruby programming" }
+)
+
+# List runs in a project
+runs = project.list_runs(limit: 10)
+```
+
+### Datasets
+
+Datasets allow you to store examples for testing and evaluation:
+
+```ruby
+# Create a new dataset
+dataset = client.create_dataset(name: "qa-dataset", description: "Question-answering examples")
+
+# Add examples to the dataset
+dataset.create_example(
+  inputs: { question: "What is Ruby?" },
+  outputs: { answer: "Ruby is a dynamic, open source programming language with a focus on simplicity and productivity." }
+)
+
+# List examples in the dataset
+examples = dataset.list_examples(limit: 10)
+```
+
+### Evaluations
+
+Evaluations help you assess the quality of your LLM outputs:
+
+```ruby
+# Create an evaluation run
+evaluation = dataset.create_evaluation_run(
+  evaluator_name: "correctness",
+  run_ids: ["run_123", "run_456"]
+)
+
+# Check if the evaluation is complete
+evaluation.completed?
+
+# Wait for the evaluation to complete
+evaluation.wait_for_completion(timeout: 300)
+
+# Get the evaluation results
+results = evaluation.results
+```
+
+### Feedback
+
+Feedback allows you to collect human or automated assessments of runs:
+
+```ruby
+# Add feedback to a run
+feedback = client.create_feedback(
+  run_id: "run_123",
+  key: "helpfulness",
+  score: 0.95,
+  comment: "Very helpful response!"
+)
+
+# Get feedback for a run
+feedback_list = client.get_feedback(run_id: "run_123")
+
+# Add feedback directly to a run
+run.add_feedback(key: "correctness", score: 0.8, comment: "Mostly correct")
+
+# Get feedback for a run
+feedback_list = run.get_feedback
+```
+
+### Tracer Sessions
+
+Tracer sessions help you organize and manage traces of your LLM applications:
+
+```ruby
+# Create a new tracer session
+tracer_session = client.create_tracer_session(
+  name: "my-tracer-session",
+  description: "Tracing my LLM application"
+)
+
+# Get a tracer session by ID
+tracer_session = client.get_tracer_session(tracer_session_id: "ts_123")
+
+# List tracer sessions
+tracer_sessions = client.list_tracer_sessions(limit: 10)
+
+# Update a tracer session
+updated_session = client.update_tracer_session(
+  tracer_session_id: "ts_123",
+  name: "updated-tracer-session",
+  description: "Updated description"
+)
+
+# Delete a tracer session
+client.delete_tracer_session(tracer_session_id: "ts_123")
+```
+
+### Organizations
+
+Manage organizations in LangSmith:
+
+```ruby
+# Get the current organization
+org = client.get_current_organization
+
+# List organizations
+orgs = client.list_organizations
+
+# Get an organization by ID
+org = client.get_organization(organization_id: "org_123")
+```
+
+### API Keys
+
+Manage API keys for accessing LangSmith:
+
+```ruby
+# List API keys
+keys = client.list_api_keys
+
+# Create a new API key
+new_key = client.create_api_key(name: "my-api-key")
+
+# Delete an API key
+client.delete_api_key(api_key_id: "key_123")
+```
+
+### Tenants
+
+Manage tenants within your organization:
+
+```ruby
+# Get a tenant by ID
+tenant = client.get_tenant(tenant_id: "tenant_123")
+
+# List tenants
+tenants = client.list_tenants(limit: 10)
+
+# Create a new tenant
+new_tenant = client.create_tenant(
+  name: "my-tenant",
+  description: "My team's tenant"
+)
+
+# Update a tenant
+updated_tenant = client.update_tenant(
+  tenant_id: "tenant_123",
+  name: "updated-tenant",
+  description: "Updated description"
+)
+```
+
+### Tags
+
+Manage tags for organizing runs and other resources:
+
+```ruby
+# Create a new tag
+tag = client.create_tag(name: "important")
+
+# Get a tag by ID
+tag = client.get_tag(tag_id: "tag_123")
+
+# List tags
+tags = client.list_tags(limit: 10)
+
+# Delete a tag
+client.delete_tag(tag_id: "tag_123")
+```
+
+### Prompts
+
+Manage prompt templates in LangSmith:
+
+```ruby
+# Create a new prompt
+prompt = client.create_prompt(
+  name: "qa-prompt",
+  prompt_template: "Answer the following question: {{question}}",
+  description: "A simple Q&A prompt template"
+)
+
+# Get a prompt by ID
+prompt = client.get_prompt(prompt_id: "prompt_123")
+
+# List prompts
+prompts = client.list_prompts(limit: 10)
+
+# Update a prompt
+updated_prompt = client.update_prompt(
+  prompt_id: "prompt_123",
+  name: "updated-prompt",
+  prompt_template: "Please answer this question: {{question}}",
+  description: "Updated description"
+)
+
+# Delete a prompt
+client.delete_prompt(prompt_id: "prompt_123")
+```
+
+### Annotation Queues
+
+Manage annotation queues for human feedback:
+
+```ruby
+# Create a new annotation queue
+queue = client.create_annotation_queue(
+  name: "feedback-queue",
+  description: "Queue for human feedback"
+)
+
+# Get an annotation queue by ID
+queue = client.get_annotation_queue(annotation_queue_id: "queue_123")
+
+# List annotation queues
+queues = client.list_annotation_queues(limit: 10)
+
+# Update an annotation queue
+updated_queue = client.update_annotation_queue(
+  annotation_queue_id: "queue_123",
+  name: "updated-queue",
+  description: "Updated description"
+)
+
+# Delete an annotation queue
+client.delete_annotation_queue(annotation_queue_id: "queue_123")
+```
+
+### Feedback Configurations
+
+Manage feedback configurations for collecting structured feedback:
+
+```ruby
+# Create a new feedback configuration
+config = client.create_feedback_configuration(
+  name: "helpfulness",
+  description: "Measure helpfulness of responses",
+  criteria: "How helpful was this response?"
+)
+
+# Get a feedback configuration by ID
+config = client.get_feedback_configuration(feedback_configuration_id: "config_123")
+
+# List feedback configurations
+configs = client.list_feedback_configurations(limit: 10)
+
+# Update a feedback configuration
+updated_config = client.update_feedback_configuration(
+  feedback_configuration_id: "config_123",
+  name: "updated-config",
+  description: "Updated description",
+  criteria: "Updated criteria"
+)
+
+# Delete a feedback configuration
+client.delete_feedback_configuration(feedback_configuration_id: "config_123")
+```
+
+### Analytics
+
+Access analytics data for projects and datasets:
+
+```ruby
+# Get project analytics
+analytics = client.get_project_analytics(
+  project_id: "proj_123",
+  start_time: Time.now - 86400, # 24 hours ago
+  end_time: Time.now,
+  group_by: "run_type",
+  metrics: ["latency", "token_usage"]
+)
+
+# Get dataset analytics
+analytics = client.get_dataset_analytics(
+  dataset_id: "ds_123",
+  start_time: Time.now - 86400, # 24 hours ago
+  end_time: Time.now,
+  group_by: "example_id"
+)
+```
+
+### Comments
+
+Manage comments on runs:
+
+```ruby
+# Create a comment on a run
+comment = client.create_run_comment(
+  run_id: "run_123",
+  comment: "This run looks promising",
+  metadata: { source: "code review" }
+)
+
+# List comments for a run
+comments = client.list_run_comments(run_id: "run_123")
+
+# Delete a comment
+client.delete_run_comment(run_id: "run_123", comment_id: "comment_123")
+```
+
+### Events
+
+Manage events in LangSmith:
+
+```ruby
+# Create an event
+event = client.create_event(
+  event_type: "user_feedback",
+  payload: { score: 0.9, comment: "Great response!" }
+)
+
+# List events
+events = client.list_events(
+  event_type: "user_feedback",
+  start_time: Time.now - 86400, # 24 hours ago
+  end_time: Time.now
+)
+```
+
+### Settings
+
+Manage LangSmith settings:
+
+```ruby
+# Get settings
+settings = client.get_settings
+
+# Update settings
+updated_settings = client.update_settings(
+  default_project_id: "proj_123",
+  default_dataset_id: "ds_123"
+)
+```
+
+### Bulk Operations
+
+Perform bulk operations on runs:
+
+```ruby
+# Bulk tag runs
+result = client.bulk_tag_runs(
+  run_ids: ["run_123", "run_456", "run_789"],
+  tag_ids: ["tag_123", "tag_456"]
+)
+
+# Bulk untag runs
+result = client.bulk_untag_runs(
+  run_ids: ["run_123", "run_456"],
+  tag_ids: ["tag_123"]
+)
+
+# Bulk delete runs
+result = client.bulk_delete_runs(
+  run_ids: ["run_123", "run_456", "run_789"]
+)
+```
+
+### Webhooks
+
+Manage webhooks for LangSmith events:
+
+```ruby
+# Create a webhook
+webhook = client.create_webhook(
+  url: "https://example.com/webhook",
+  event_types: ["run.created", "run.updated"],
+  metadata: { description: "Notify our system when runs are created or updated" }
+)
+
+# Get a webhook by ID
+webhook = client.get_webhook(webhook_id: "webhook_123")
+
+# List webhooks
+webhooks = client.list_webhooks(limit: 10)
+
+# Update a webhook
+updated_webhook = client.update_webhook(
+  webhook_id: "webhook_123",
+  url: "https://updated-example.com/webhook",
+  event_types: ["run.created", "run.updated", "run.deleted"],
+  metadata: { description: "Updated webhook configuration" }
+)
+
+# Delete a webhook
+client.delete_webhook(webhook_id: "webhook_123")
+```
+
+### Team Management
+
+Manage team members in your organization:
+
+```ruby
+# List team members
+members = client.list_team_members(organization_id: "org_123", limit: 10)
+
+# Invite a team member
+invite = client.invite_team_member(
+  organization_id: "org_123",
+  email: "new-member@example.com",
+  role: "member"
+)
+
+# Remove a team member
+client.remove_team_member(
+  organization_id: "org_123",
+  user_id: "user_123"
+)
+```
+
+## Error Handling
+
+The LangSmith Ruby gem provides robust error handling through several error classes:
+
+```ruby
+require 'langsmith'
+
+begin
+  client = Langsmith::Client.new(api_key: "invalid_key")
+  client.list_projects
+rescue Langsmith::Errors::AuthenticationError => e
+  puts "Authentication failed: #{e.message}"
+rescue Langsmith::Errors::ResourceNotFoundError => e
+  puts "Resource not found: #{e.message}"
+rescue Langsmith::Errors::APIError => e
+  puts "API error: #{e.message}"
+rescue StandardError => e
+  puts "Unexpected error: #{e.message}"
+end
+```
+
+The gem defines the following error classes:
+
+- `Langsmith::Errors::BaseError` - Base class for all LangSmith errors
+- `Langsmith::Errors::AuthenticationError` - Raised when authentication fails (e.g., invalid API key)
+- `Langsmith::Errors::ResourceNotFoundError` - Raised when a requested resource is not found
+- `Langsmith::Errors::APIError` - Raised for other API errors with status codes outside the 200-299 range
+
+Each error includes the response body to help diagnose the issue.
+
+## Development
+
+After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
+
+To install this gem onto your local machine, run `bundle exec rake install`.
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/cdaviis/langsmithrb. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [code of conduct](https://github.com/cdaviis/langsmithrb/blob/main/CODE_OF_CONDUCT.md).
+
+## License
+
+The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).

data/Rakefile

@@ -0,0 +1,8 @@
+# frozen_string_literal: true
+
+require "bundler/gem_tasks"
+require "rspec/core/rake_task"
+
+RSpec::Core::RakeTask.new(:spec)
+
+task default: :spec