exaonruby 1.0.0 → 1.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 2da95920bde51eb7724ef7c1cfa01d25906d4cb04cde2d2dc9d1055ec1267f9e
-  data.tar.gz: 190c39f5437c045f60067856033ad54330ce05425dd0edea79cb0fa74fed989f
+  metadata.gz: f7fa267a3b26d29aff9a6e5886cd43568ad0820aad17e54c8760141bbe208aa8
+  data.tar.gz: 2a1823d83853927fee270c1070b4ead7801d1d2df2d6e5c8769d644cefb31459
 SHA512:
-  metadata.gz: dea6072b3c1b149c869a4d76854cd36a3aa1a18241cbdea6a105f1331e2fb297723def00c59a30e13c4c044d93095b03b1bd7594f9dc9815e8c20b3bc59c3546
-  data.tar.gz: 12db1ecb3e8b3528bcaf74c03ea9a259136d794a9c68125562d69fe1d031808582f140cb2e50eb06b2faf3b9ec476c2eefc56809fe013ad3439d63fba35a7776
+  metadata.gz: d128dce03cfee863a805039fcb4e9d4b12b93bff213ae0e03be21702040422a284864910df6916183c6aca5b59a91811d4cc3589c348733c31b441eee46db518
+  data.tar.gz: 87fb73c8641ef0a78af2f426d0a9499a934d4149ac6c482acab4a688575b87449a8d4c1feb23d46cf9f0b9c7853c787cf052acda30367c9c6fa8f38946559c47

data/CHANGELOG.md

@@ -0,0 +1,134 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [1.2.0] - 2025-12-18
+
+### Added
+
+- **Request Logging Middleware** - Pretty HTTP request/response logging
+  - Configurable log levels and body logging
+  - Sensitive data filtering (API keys, passwords)
+  - Request timing with status indicators (✓ ⚠ ✗)
+
+- **Rate Limiter Middleware** - Client-side rate limiting
+  - Token bucket algorithm for smooth rate control
+  - Configurable requests per second and burst size
+  - Thread-safe implementation
+
+- **Response Cache Middleware** - Reduce API costs
+  - `Exa::Cache::MemoryStore` for single-process apps
+  - `Exa::Cache::RedisStore` for distributed caching
+  - Configurable TTL and cacheable endpoints
+
+- **OpenTelemetry Instrumentation** - Distributed tracing
+  - Automatic span creation for all requests
+  - HTTP attributes and status codes
+  - Error recording
+
+- **Parallel Requests Utility** - Batch operations
+  - `Exa::Utils::Parallel.map` for concurrent requests
+  - Configurable concurrency limits
+  - Multiple error handling strategies
+
+- **Rails Integration** - First-class Rails support
+  - Railtie for automatic configuration
+  - `rails g exa:install` generator
+  - ActiveJob adapter for async research
+  - ActionCable integration for streaming
+  - Rake tasks (`rails exa:verify`, `rails exa:config`)
+
+### Changed
+
+- Added `concurrent-ruby` as a core dependency for parallel requests
+
+## [1.1.0] - 2025-12-18
+
+### Added
+
+- **SSE Streaming** - Real-time streaming for Answer and Research APIs
+  - `Exa::Utils::SSEClient.stream_answer` - Stream answer tokens as they're generated
+  - `Exa::Utils::SSEClient.stream_research` - Stream research progress and output
+  - Automatic reconnection and error handling
+  - Ping/heartbeat support
+
+- **Sorbet Type Definitions** - Optional static type checking
+  - `Exa::Types` module with T::Struct definitions for all API types
+  - Optional dependency on `sorbet-runtime`
+  - Full type coverage for Search, Answer, Research, Websets, Monitors, Imports, Webhooks, Events
+
+## [1.0.0] - 2025-12-18
+
+### Added
+
+- **Search API**
+  - Neural, auto, fast, and deep search types
+  - Content extraction with text, highlights, and summaries
+  - Domain and date filtering
+  - Category filtering (company, person, research paper, etc.)
+
+- **Contents API**
+  - Fetch full page contents from URLs
+  - Livecrawl support (never, fallback, preferred, always)
+  - Subpage extraction
+
+- **Find Similar API**
+  - Discover semantically similar pages
+  - Exclude source URL options
+
+- **Answer API**
+  - LLM-powered question answering
+  - Citations with source URLs
+  - Search options integration
+
+- **Research API**
+  - Async research task creation
+  - Multiple models (exa-research-fast, exa-research, exa-research-pro)
+  - Structured output schema support
+  - Task polling and cancellation
+
+- **Websets API**
+  - Full CRUD operations for Websets
+  - Item management (list, get, delete)
+  - Search operations (create, get, cancel)
+  - Enrichment operations (create, list, get, delete)
+
+- **Monitors API**
+  - Create automated search/refresh schedules
+  - Cron expression support
+  - Monitor run tracking
+
+- **Imports API**
+  - CSV upload with presigned URLs
+  - Entity type configuration
+  - Import status tracking
+
+- **Webhooks API**
+  - Create webhook subscriptions
+  - Event type filtering
+  - Webhook attempt tracking
+
+- **Events API**
+  - List and filter events
+  - Event type helpers
+
+- **CLI**
+  - Beautiful colorful command-line interface
+  - Search, answer, similar, research commands
+  - Websets management subcommands
+  - JSON output option
+
+- **n8n/Zapier Integration**
+  - Webhook signature verification
+  - HMAC-SHA256 with timing attack prevention
+  - Timestamp validation for replay attack prevention
+  - Framework-agnostic header parsing
+
+- **Core Features**
+  - Automatic retry with exponential backoff
+  - Comprehensive error hierarchy
+  - Rate limit handling with retry_after
+  - YARD documentation on all public methods

data/README.md

@@ -1,6 +1,8 @@
 # Exa Ruby
 
-A production-ready Ruby gem wrapper for the [Exa.ai](https://exa.ai) API, providing intelligent web search, content extraction, and structured data collection capabilities.
+A Ruby gem wrapper for the [Exa.ai](https://exa.ai) API, providing intelligent web search, content extraction, and structured data collection capabilities.
+
+Want to say thanks? Click the ⭐ at the top of the page.
 
 ## Features
 
@@ -13,10 +15,17 @@ A production-ready Ruby gem wrapper for the [Exa.ai](https://exa.ai) API, provid
 - **Monitors**: Automated scheduled searches and content refresh
 - **Imports**: Upload CSV data into Websets
 - **Webhooks & Events**: Real-time notifications for Websets activity
+- **SSE Streaming**: Real-time token streaming for Answer and Research APIs
+- **Request Logging**: Pretty HTTP logging with timing and status indicators
+- **Rate Limiting**: Client-side token bucket rate limiting
+- **Response Caching**: Memory and Redis caching to reduce API costs
+- **Instrumentation**: OpenTelemetry distributed tracing
+- **Parallel Requests**: Concurrent batch operations
+- **Rails Integration**: Railtie, generator, ActiveJob, ActionCable
+- **Sorbet Types**: Optional T::Struct type definitions
 - **Beautiful CLI**: Colorful command-line interface
 - **n8n/Zapier Integration**: Webhook signature verification utilities
 - **Automatic Retries**: Built-in retry logic for transient failures
-- **Rate Limit Handling**: Proper error handling with retry information
 - **Type Documentation**: Comprehensive YARD documentation
 
 ## Installation
@@ -593,12 +602,85 @@ exa search "AI news" --json
 exa version
 ```
 
+## SSE Streaming
+
+Stream tokens in real-time for Answer and Research APIs:
+
+```ruby
+# Stream an answer with real-time token output
+Exa::Utils::SSEClient.stream_answer(
+  api_key: ENV["EXA_API_KEY"],
+  query: "What is quantum computing?"
+) do |event|
+  case event[:type]
+  when :token
+    print event[:data]  # Print each token as it arrives
+  when :citation
+    puts "\nSource: #{event[:data][:url]}"
+  when :done
+    puts "\n\nComplete!"
+  when :error
+    puts "Error: #{event[:data]}"
+  end
+end
+
+# Stream research progress
+Exa::Utils::SSEClient.stream_research(
+  api_key: ENV["EXA_API_KEY"],
+  instructions: "Research latest AI developments"
+) do |event|
+  case event[:type]
+  when :progress
+    puts "Progress: #{event[:data][:percent]}%"
+  when :output
+    puts event[:data]
+  end
+end
+
+# Instance-based streaming
+streamer = Exa::Utils::SSEClient.new(api_key: ENV["EXA_API_KEY"])
+streamer.answer("What is GPT-4?") { |e| print e[:data] if e[:type] == :token }
+```
+
+## Sorbet Type Definitions
+
+Optional static type checking with Sorbet:
+
+```ruby
+# Install sorbet-runtime for type definitions
+# gem install sorbet-runtime
+
+require 'exa'
+
+# Types are available when sorbet-runtime is installed
+params = Exa::Types::SearchParams.new(
+  query: "AI research",
+  type: "neural",
+  num_results: 10,
+  text: true
+)
+
+# Type definitions for all API responses
+# Exa::Types::SearchResultData
+# Exa::Types::AnswerResponseData
+# Exa::Types::ResearchTaskData
+# Exa::Types::WebsetData
+# Exa::Types::MonitorData
+# Exa::Types::ImportData
+# Exa::Types::WebhookData
+# Exa::Types::EventData
+```
+
 ## Requirements
 
 - Ruby >= 3.1
 - faraday >= 2.0
 - faraday-retry >= 2.0
 - thor >= 1.0
+- concurrent-ruby >= 1.2
+- sorbet-runtime >= 0.5 (optional, for type definitions)
+- opentelemetry-sdk (optional, for instrumentation)
+- redis (optional, for distributed caching)
 
 ## Development
 
@@ -611,4 +693,3 @@ The gem is available as open source under the terms of the [MIT License](https:/
 ## Contributing
 
 Bug reports and pull requests are welcome on GitHub at https://github.com/exa-labs/exa-ruby.
-

data/exaonruby.gemspec

@@ -8,10 +8,12 @@ Gem::Specification.new do |spec|
   spec.authors = ["tigel-agm"]
   spec.email = []
 
-  spec.summary = "Complete Ruby client for the Exa.ai API with beautiful CLI"
+  spec.summary = "Complete Ruby client for the Exa.ai API with CLI, middleware, and Rails integration"
   spec.description = "A production-ready Ruby gem wrapper for the Exa.ai Search and Websets APIs. " \
                      "Features neural search, LLM-powered answers, async research tasks, " \
-                     "Websets management (monitors, imports, webhooks), and a beautiful CLI. " \
+                     "Websets management (monitors, imports, webhooks), SSE streaming, " \
+                     "request logging, rate limiting, response caching, OpenTelemetry instrumentation, " \
+                     "parallel requests, Rails integration, Sorbet types, and a beautiful CLI. " \
                      "Includes n8n/Zapier webhook signature verification utilities."
   spec.homepage = "https://github.com/tigel-agm/exaonruby"
   spec.license = "MIT"
@@ -24,14 +26,20 @@ Gem::Specification.new do |spec|
   spec.metadata["rubygems_mfa_required"] = "true"
 
   # Include all lib files explicitly since we may not have git
-  spec.files = Dir.glob("{lib,exe}/**/*") + %w[LICENSE.txt README.md]
+  spec.files = Dir.glob("{lib,exe}/**/*") + %w[LICENSE.txt README.md CHANGELOG.md]
   spec.files += Dir.glob("*.gemspec")
 
   spec.bindir = "exe"
   spec.executables = ["exa"]
   spec.require_paths = ["lib"]
 
+  # Core dependencies
   spec.add_dependency "faraday", ">= 2.0", "< 3.0"
   spec.add_dependency "faraday-retry", ">= 2.0", "< 3.0"
   spec.add_dependency "thor", ">= 1.0", "< 3.0"
+  spec.add_dependency "concurrent-ruby", ">= 1.2", "< 2.0"
+
+  # Optional: Sorbet types (install sorbet-runtime for type checking)
+  # Optional: OpenTelemetry (install opentelemetry-sdk for instrumentation)
+  # Optional: Redis (install redis gem for distributed caching)
 end

data/lib/exa/middleware/instrumentation.rb

@@ -0,0 +1,97 @@
+# frozen_string_literal: true
+
+# typed: strict
+
+module Exa
+  module Middleware
+    # OpenTelemetry instrumentation middleware for distributed tracing
+    #
+    # Integrates with OpenTelemetry for comprehensive request tracing.
+    # Requires opentelemetry-sdk gem to be installed.
+    #
+    # @example Enable instrumentation
+    #   require 'opentelemetry/sdk'
+    #   OpenTelemetry::SDK.configure
+    #
+    #   client = Exa::Client.new(api_key: key) do |config|
+    #     config.instrumentation = true
+    #   end
+    class Instrumentation < Faraday::Middleware
+      # @param app [Faraday::Middleware] Next middleware
+      # @param tracer_name [String] Name for the tracer
+      def initialize(app, tracer_name: "exa.api")
+        super(app)
+        @tracer_name = tracer_name
+        @tracer = nil
+      end
+
+      def call(env)
+        return @app.call(env) unless otel_available?
+
+        tracer.in_span(span_name(env), attributes: span_attributes(env), kind: :client) do |span|
+          begin
+            response = @app.call(env)
+            
+            response.on_complete do |response_env|
+              span.set_attribute("http.status_code", response_env[:status])
+              span.status = otel_status(response_env[:status])
+            end
+            
+            response
+          rescue StandardError => e
+            span.record_exception(e)
+            span.status = OpenTelemetry::Trace::Status.error(e.message)
+            raise
+          end
+        end
+      end
+
+      private
+
+      def otel_available?
+        defined?(OpenTelemetry) && OpenTelemetry.tracer_provider
+      rescue StandardError
+        false
+      end
+
+      def tracer
+        @tracer ||= OpenTelemetry.tracer_provider.tracer(@tracer_name)
+      end
+
+      def span_name(env)
+        method = env[:method].to_s.upcase
+        path = env[:url].path.split("/").reject(&:empty?).first || "request"
+        "Exa #{method} /#{path}"
+      end
+
+      def span_attributes(env)
+        {
+          "http.method" => env[:method].to_s.upcase,
+          "http.url" => sanitize_url(env[:url].to_s),
+          "http.target" => env[:url].path,
+          "http.host" => env[:url].host,
+          "http.scheme" => env[:url].scheme,
+          "net.peer.name" => env[:url].host,
+          "net.peer.port" => env[:url].port
+        }
+      end
+
+      def sanitize_url(url)
+        url.gsub(/api_key=[^&]+/, "api_key=[REDACTED]")
+      end
+
+      def otel_status(status_code)
+        case status_code
+        when 200..299
+          OpenTelemetry::Trace::Status.ok
+        when 400..599
+          OpenTelemetry::Trace::Status.error("HTTP #{status_code}")
+        else
+          OpenTelemetry::Trace::Status.unset
+        end
+      end
+    end
+
+    Faraday::Middleware.register_middleware(exa_instrumentation: Instrumentation)
+  end
+end

data/lib/exa/middleware/rate_limiter.rb

@@ -0,0 +1,72 @@
+# frozen_string_literal: true
+
+# typed: strict
+
+module Exa
+  module Middleware
+    # Client-side rate limiter using token bucket algorithm
+    #
+    # Prevents 429 errors by limiting requests before they're sent.
+    # Uses a thread-safe token bucket with configurable rates.
+    #
+    # @example Enable rate limiting
+    #   client = Exa::Client.new(api_key: key) do |config|
+    #     config.rate_limit = 10         # requests per second
+    #     config.rate_limit_burst = 20   # max burst size
+    #   end
+    #
+    # @example Lower limits for safety
+    #   client = Exa::Client.new(api_key: key) do |config|
+    #     config.rate_limit = 5          # conservative 5 req/sec
+    #   end
+    class RateLimiter < Faraday::Middleware
+      # @param app [Faraday::Middleware] Next middleware
+      # @param rate [Float] Tokens per second (requests per second)
+      # @param burst [Integer] Maximum tokens (burst capacity)
+      def initialize(app, rate: 10.0, burst: 20)
+        super(app)
+        @rate = rate.to_f
+        @burst = burst
+        @tokens = burst.to_f
+        @last_time = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+        @mutex = Mutex.new
+      end
+
+      def call(env)
+        wait_for_token
+        @app.call(env)
+      end
+
+      private
+
+      # Wait until a token is available
+      def wait_for_token
+        @mutex.synchronize do
+          refill_tokens
+          
+          if @tokens < 1.0
+            # Calculate wait time needed
+            wait_time = (1.0 - @tokens) / @rate
+            sleep(wait_time) if wait_time > 0
+            refill_tokens
+          end
+
+          @tokens -= 1.0
+        end
+      end
+
+      # Refill tokens based on time elapsed
+      def refill_tokens
+        now = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+        elapsed = now - @last_time
+        @last_time = now
+
+        # Add tokens based on elapsed time
+        @tokens = [@tokens + (elapsed * @rate), @burst.to_f].min
+      end
+    end
+
+    # Faraday middleware registration
+    Faraday::Middleware.register_middleware(exa_rate_limiter: RateLimiter)
+  end
+end

data/lib/exa/middleware/request_logger.rb

@@ -0,0 +1,170 @@
+# frozen_string_literal: true
+
+# typed: strict
+
+require "logger"
+require "json"
+
+module Exa
+  module Middleware
+    # Pretty request/response logging middleware for Faraday
+    #
+    # Logs all HTTP requests and responses with timing, status codes, and
+    # optional body logging. Useful for debugging and monitoring API usage.
+    #
+    # @example Enable logging on client
+    #   client = Exa::Client.new(api_key: key) do |config|
+    #     config.logger = Logger.new($stdout)
+    #     config.log_level = :info
+    #     config.log_bodies = true
+    #   end
+    #
+    # @example Custom logger
+    #   client = Exa::Client.new(api_key: key) do |config|
+    #     config.logger = Rails.logger
+    #     config.log_level = :debug
+    #   end
+    class RequestLogger < Faraday::Middleware
+      DEFAULT_OPTIONS = {
+        log_level: :info,
+        log_headers: false,
+        log_bodies: false,
+        filter_keys: %w[api_key x-api-key authorization password secret token].freeze,
+        max_body_size: 2000
+      }.freeze
+
+      # @param app [Faraday::Middleware] Next middleware in chain
+      # @param logger [Logger] Logger instance
+      # @param options [Hash] Configuration options
+      def initialize(app, logger: nil, **options)
+        super(app)
+        @logger = logger || default_logger
+        @options = DEFAULT_OPTIONS.merge(options)
+      end
+
+      def call(env)
+        start_time = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+        request_id = generate_request_id
+
+        log_request(env, request_id)
+
+        @app.call(env).on_complete do |response_env|
+          elapsed = ((Process.clock_gettime(Process::CLOCK_MONOTONIC) - start_time) * 1000).round(2)
+          log_response(response_env, request_id, elapsed)
+        end
+      rescue StandardError => e
+        elapsed = ((Process.clock_gettime(Process::CLOCK_MONOTONIC) - start_time) * 1000).round(2)
+        log_error(env, request_id, elapsed, e)
+        raise
+      end
+
+      private
+
+      def log_request(env, request_id)
+        method = env[:method].to_s.upcase
+        url = env[:url].to_s.gsub(/api_key=[^&]+/, "api_key=[FILTERED]")
+
+        message = "[Exa] #{request_id} → #{method} #{url}"
+
+        if @options[:log_headers]
+          headers = filter_sensitive(env[:request_headers] || {})
+          message += "\n  Headers: #{headers.to_json}"
+        end
+
+        if @options[:log_bodies] && env[:body]
+          body = truncate_body(filter_body(env[:body]))
+          message += "\n  Body: #{body}"
+        end
+
+        log(message)
+      end
+
+      def log_response(env, request_id, elapsed)
+        status = env[:status]
+        status_emoji = status_indicator(status)
+
+        message = "[Exa] #{request_id} ← #{status_emoji} #{status} (#{elapsed}ms)"
+
+        if @options[:log_bodies] && env[:body]
+          body = truncate_body(env[:body].to_s)
+          message += "\n  Body: #{body}"
+        end
+
+        log(message)
+      end
+
+      def log_error(env, request_id, elapsed, error)
+        message = "[Exa] #{request_id} ✗ ERROR after #{elapsed}ms: #{error.class}: #{error.message}"
+        log(message, :error)
+      end
+
+      def log(message, level = nil)
+        level ||= @options[:log_level]
+
+        case level
+        when :debug
+          @logger.debug(message)
+        when :info
+          @logger.info(message)
+        when :warn
+          @logger.warn(message)
+        when :error
+          @logger.error(message)
+        else
+          @logger.info(message)
+        end
+      end
+
+      def status_indicator(status)
+        case status
+        when 200..299 then "✓"
+        when 300..399 then "↻"
+        when 400..499 then "⚠"
+        when 500..599 then "✗"
+        else "?"
+        end
+      end
+
+      def generate_request_id
+        format("%06x", rand(0xFFFFFF))
+      end
+
+      def filter_sensitive(hash)
+        hash.each_with_object({}) do |(key, value), result|
+          if @options[:filter_keys].any? { |k| key.to_s.downcase.include?(k.downcase) }
+            result[key] = "[FILTERED]"
+          else
+            result[key] = value
+          end
+        end
+      end
+
+      def filter_body(body)
+        return body unless body.is_a?(Hash)
+
+        body.each_with_object({}) do |(key, value), result|
+          if @options[:filter_keys].any? { |k| key.to_s.downcase.include?(k.downcase) }
+            result[key] = "[FILTERED]"
+          else
+            result[key] = value
+          end
+        end.to_json
+      end
+
+      def truncate_body(body)
+        return body if body.length <= @options[:max_body_size]
+
+        "#{body[0, @options[:max_body_size]]}... (truncated)"
+      end
+
+      def default_logger
+        logger = Logger.new($stdout)
+        logger.formatter = proc { |_, _, _, msg| "#{msg}\n" }
+        logger
+      end
+    end
+
+    # Register middleware with Faraday
+    Faraday::Middleware.register_middleware(exa_logger: RequestLogger)
+  end
+end