exa-ai-ruby 1.0.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 61569eb14ed0573b4f252e3e7279320bd716ea2d044c5af4c830f33c536608da
+  data.tar.gz: 60b89c6b10b38756628c1d0bbefee6bd907309a946d2eac795b0749c456c2d61
+SHA512:
+  metadata.gz: 6fdd59ce00cfd0b46024db0bff5750770249b06ccd59dc792cc8d46037262bfe2ef864f29981d1f5028ca48e98be4f232b246b6b26edc764c7bbc17ea521c8cb
+  data.tar.gz: 3955736a25363da2379351f8514603509ee976d5130d230de601cd063e185b3c2b1e9143e032cc3d8a5ee7fde9ff688854a55331f6b379db1e40133f9511c3a9

data/CHANGELOG.md

@@ -0,0 +1,4 @@
+# Changelog
+
+## [0.1.0] - 2025-10-25
+- Initial gem scaffolding.

data/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2025 Vicente
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,247 @@
+# Exa Ruby Client
+
+> Typed, Sorbet-friendly Ruby bindings for the Exa API, inspired by `openai-ruby` and aligned with Exa’s OpenAPI specs.
+
+This README doubles as the canonical “llms-full” reference for the project: it documents the architecture, the current API surface, and the functional-programming patterns we’re porting from OpenAI’s client. If you’re building the Exa Ruby SDK—or just trying to understand how to extend it—start here.
+
+---
+
+## Table of Contents
+
+1. [Project Goals](#project-goals)
+2. [Environment & Installation](#environment--installation)
+3. [Client Architecture Overview](#client-architecture-overview)
+4. [Typed Resources & Usage Examples](#typed-resources--usage-examples)
+   - [Search stack](#search-stack)
+   - [Research](#research)
+   - [Websets (core + items + enrichments + monitors)](#websets-core--items--enrichments--monitors)
+   - [Events, Imports, Webhooks](#events-imports-webhooks)
+5. [Structured Output via Sorbet + dspy-schema](#structured-output-via-sorbet--dspy-schema)
+6. [Streaming & Transport Helpers](#streaming--transport-helpers)
+7. [Testing & TDD Plan](#testing--tdd-plan)
+8. [Roadmap / TODOs](#roadmap--todos)
+
+---
+
+## Project Goals
+
+- Mirror `openai-ruby` ergonomics so Sorbet-aware developers get typed resources, model structs, and helpers out of the box.
+- Port over OpenAI’s functional patterns: request structs, transport abstraction, streaming/pagination utilities, structured-output DSL.
+- Understand the entire Exa API surface (search, contents, answers, research, websets, monitors, imports, events, webhooks, etc.) and encode it via Sorbet types generated from `openapi-spec/`.
+- Bake Sorbet-generated JSON Schemas directly into v1 using the published `dspy-schema` gem—structured outputs should accept Sorbet types, not free-form hashes.
+
+See `docs/architecture.md` for deep-dive notes, mermaid diagrams, and highlights from `openai-ruby`, `exa-py`, and `exa-js`.
+
+---
+
+## Environment & Installation
+
+```
+$ git clone https://github.com/vicentereig/exa-ruby
+$ cd exa-ruby
+$ rbenv install 3.4.5   # .ruby-version already pins this
+$ bundle install
+
+# or install from RubyGems (gem name: exa-ai-ruby)
+$ gem install exa-ai-ruby
+```
+
+Runtime dependencies:
+- `sorbet-runtime` – typed structs/enums and runtime assertions.
+- `connection_pool` – `Net::HTTP` pooling in `PooledNetRequester`.
+- `dspy-schema` – converts Sorbet types to JSON Schema (structured output support).
+
+Set the API key via `EXA_API_KEY` or pass `api_key:` when instantiating `Exa::Client`.
+
+---
+
+## Client Architecture Overview
+
+```ruby
+require "exa"
+
+client = Exa::Client.new(
+  api_key: ENV.fetch("EXA_API_KEY"),
+  base_url: ENV["EXA_BASE_URL"] || "https://api.exa.ai",
+  timeout: 120,
+  max_retries: 2
+)
+```
+
+- `Exa::Client` inherits from `Exa::Internal::Transport::BaseClient`, giving us:
+  - Header normalization + auth injection (`x-api-key`).
+  - Retry/backoff logic with HTTP status checks.
+  - Streaming support that returns `Exa::Internal::Transport::Stream`.
+- Request payloads are Sorbet structs under `Exa::Types::*`, serialized via `Exa::Types::Serializer`, which camelizes keys and auto-converts Sorbet schemas (see [Structured Output](#structured-output-via-sorbet--dspy-schema)).
+- Response models live in `lib/exa/responses/*`. Whenever an endpoint returns typed data the resource sets `response_model:` so the client converts the JSON hash into Sorbet structs (e.g., `Exa::Responses::SearchResponse`, `Webset`, `Research`, etc.).
+- Transport stack:
+  - `PooledNetRequester` manages per-origin `Net::HTTP` pools via `connection_pool`.
+  - Responses stream through fused enumerators so we can decode JSON/JSONL/SSE lazily and ensure sockets are closed once consumers finish iterating.
+
+---
+
+## Typed Resources & Usage Examples
+
+### Search stack
+
+```ruby
+resp = client.search.search(
+  query: "latest reasoning LLM papers",
+  num_results: 5,
+  text: {max_characters: 1_000}
+)
+resp.results.each { puts "#{_1.title} – #{_1.url}" }
+
+contents = client.search.contents(urls: ["https://exa.ai"], text: true)
+
+# Structured answer with typed search options + Sorbet schema
+class AnswerShape < T::Struct
+  const :headline, String
+  const :key_points, T::Array[String]
+end
+
+answer = client.search.answer(
+  query: "Summarize robotics grant funding",
+  search_options: {num_results: 3, type: Exa::Types::SearchType::Deep},
+  summary: {schema: AnswerShape}
+)
+puts answer.raw # Hash with schema-validated payload
+```
+
+Covers `/search`, `/contents`, `/findSimilar`, and `/answer` with typed request structs (`Exa::Types::SearchRequest`, etc.) and typed responses (`Exa::Responses::SearchResponse`, `FindSimilarResponse`, `ContentsResponse`).
+
+### Research
+
+```ruby
+class ResearchShape < T::Struct
+  const :organization, String
+  const :funding_rounds, T::Array[String]
+end
+
+research = client.research.create(
+  instructions: "Map frontier labs & their funders",
+  output_schema: ResearchShape
+)
+
+# Polling
+details = client.research.get(research.id)
+puts details.status # pending/running/completed
+
+# Streaming (Server-Sent Events)
+client.research.get(research.id, stream: true).each_event_json do |event|
+  puts "[#{event[:event]}] #{event[:data]}"
+end
+
+# Cancel
+client.research.cancel(research.id)
+```
+
+Responses use `Exa::Responses::Research` and `ResearchListResponse`, which preserve raw payloads plus typed attributes (status, operations, events, output hashes, etc.). Streaming helpers (`each_event`, `each_event_json`) live on `Exa::Internal::Transport::Stream`.
+
+### Websets (core + items + enrichments + monitors)
+
+```ruby
+webset = client.websets.create(name: "Competitive Intelligence")
+webset = client.websets.update(webset.id, title: "Updated title")
+ListResp = client.websets.list(limit: 10)
+
+# Items
+items = client.websets.items.list(webset.id, limit: 5)
+item = client.websets.items.retrieve(webset.id, items.data.first.id)
+client.websets.items.delete(webset.id, item.id)
+
+# Enrichments
+enrichment = client.websets.enrichments.create(
+  webset.id,
+  description: "Company revenue information",
+  format: "text"
+)
+client.websets.enrichments.update(webset.id, enrichment.id, description: "Updated task")
+client.websets.enrichments.cancel(webset.id, enrichment.id)
+
+# Monitors
+monitor = client.websets.monitors.create(name: "Daily digest")
+runs = client.websets.monitors.runs_list(monitor.id)
+```
+
+Typed responses:
+- `Exa::Responses::Webset`, `WebsetListResponse`
+- `WebsetItem`, `WebsetItemListResponse`
+- `WebsetEnrichment`
+- `Monitor`, `MonitorRun`, etc.
+
+### Events, Imports, Webhooks
+
+```ruby
+events = client.events.list(types: ["webset.created"])
+event = client.events.retrieve(events.data.first.id)
+
+import = client.imports.create(source: {...})
+imports = client.imports.list(limit: 10)
+
+webhook = client.webhooks.create(
+  url: "https://example.com/hooks",
+  events: ["webset.completed"]
+)
+attempts = client.webhooks.attempts(webhook.id, limit: 5)
+```
+
+Every call returns typed structs (`Exa::Responses::Event`, `Import`, `Webhook`, etc.) so consumers get predictable Sorbet shapes.
+
+---
+
+## Structured Output via Sorbet + dspy-schema
+
+`dspy-schema`’s Sorbet converter is bundled so any Sorbet `T::Struct`, `T::Enum`, or `T.type_alias` can be dropped into a request payload and automatically serialized to JSON Schema. This powers `summary: {schema: ...}` and `research.output_schema`, letting the API validate outputs against your Sorbet model.
+
+Key points:
+- `Exa::Types::Schema.to_json_schema(SomeStruct)` calls `DSPy::TypeSystem::SorbetJsonSchema`.
+- `Exa::Types::Serializer` detects Sorbet classes/aliases before serializing request payloads.
+- Tests in `test/types/serializer_test.rb` ensure schema conversion works end-to-end.
+
+---
+
+## Streaming & Transport Helpers
+
+- `Exa::Internal::Transport::Stream` (returned when `stream: true`) exposes:
+  - `each` – raw chunk iteration.
+  - `each_line` – line-by-line iteration with automatic closing.
+  - `each_json_line(symbolize: true)` – NDJSON helper.
+  - `each_event` / `each_event_json` – SSE decoding with automatic JSON parsing.
+- `Exa::Internal::Util` utilities:
+  - `decode_content` auto-detects JSON/JSONL/SSE vs binary bodies.
+  - `decode_lines` + `decode_sse` implement fused enumerators so sockets close exactly once.
+- `PooledNetRequester` calibrates socket timeouts per request deadline and reuses connections via `connection_pool`.
+
+See `test/transport/stream_test.rb` for examples.
+
+---
+
+## Testing & TDD Plan
+
+Run the suite:
+
+```
+RBENV_VERSION=3.4.5 ~/.rbenv/shims/bundle exec rake test
+```
+
+Current coverage includes:
+- Resource tests for search, research, websets (core/items/enrichments/monitors), imports, events, webhooks, etc., using `TestSupport::FakeRequester`.
+- Type serialization tests ensuring camelCase conversion + schema inference.
+- Streaming helper tests verifying SSE/JSONL decoding.
+
+Future tests:
+- End-to-end HTTP tests once a real transport target is wired (probably using recorded fixtures but not VCR).
+- Schema-specific validations once JSON Schema generation is extended to all endpoints.
+
+---
+
+## Roadmap / TODOs
+
+1. **Typed coverage for the remaining OpenAPI endpoints** – webhooks attempts detail, events schema typing, structured outputs for additional research events, etc.
+2. **Code generation from OpenAPI specs** – automate request/response struct creation so spec updates can be pulled regularly.
+3. **Transport polish** – retries with `Retry-After`, idempotency keys, better error envelopes mirroring Exa’s payloads.
+4. **README “llms-full” expansion** – continue updating this document as new features land so it remains the single source of truth.
+5. **Release packaging** – ensure `.gem` builds exclude dev artifacts (`pkg/`, `.idea/`, etc.) and publish initial versions to RubyGems.
+
+Have ideas or find gaps? Open an issue or PR in `vicentereig/exa-ruby`—contributions welcome!***

data/lib/exa/client.rb

@@ -0,0 +1,33 @@
+# frozen_string_literal: true
+
+module Exa
+  class Client < Exa::Internal::Transport::BaseClient
+    DEFAULT_BASE_URL = "https://api.exa.ai"
+
+    attr_reader :api_key, :search, :research, :websets, :events, :webhooks, :imports
+
+    def initialize(
+      api_key: ENV["EXA_API_KEY"],
+      base_url: ENV["EXA_BASE_URL"] || DEFAULT_BASE_URL,
+      **opts
+    )
+      raise Exa::Errors::ConfigurationError, "api_key is required" if api_key.nil? || api_key.empty?
+
+      @api_key = api_key
+      super(base_url: base_url, **opts)
+
+      @search = Exa::Resources::Search.new(client: self)
+      @research = Exa::Resources::Research.new(client: self)
+      @websets = Exa::Resources::Websets.new(client: self)
+      @events = Exa::Resources::Events.new(client: self)
+      @webhooks = Exa::Resources::Webhooks.new(client: self)
+      @imports = Exa::Resources::Imports.new(client: self)
+    end
+
+    private
+
+    def auth_headers
+      {"x-api-key" => api_key}
+    end
+  end
+end

data/lib/exa/errors.rb

@@ -0,0 +1,34 @@
+# frozen_string_literal: true
+
+module Exa
+  module Errors
+    class Error < StandardError
+      attr_reader :url, :status, :headers, :body
+
+      def initialize(message = nil, url: nil, status: nil, headers: nil, body: nil)
+        super(message)
+        @url = url
+        @status = status
+        @headers = headers
+        @body = body
+      end
+    end
+
+    class ConfigurationError < Error; end
+    class APIError < Error; end
+
+    class APIStatusError < APIError
+      def self.raise!(url:, status:, headers:, body:)
+        message = "Exa API responded with status #{status}"
+        raise new(message, url: url, status: status, headers: headers, body: body)
+      end
+    end
+
+    class APIConnectionError < APIError; end
+    class APITimeoutError < APIError; end
+  end
+end
+
+module Exa
+  Error = Exa::Errors::Error
+end

data/lib/exa/internal/transport/base_client.rb

@@ -0,0 +1,171 @@
+# frozen_string_literal: true
+
+require "uri"
+require "cgi"
+require "json"
+
+require_relative "../util"
+require_relative "pooled_net_requester"
+require_relative "stream"
+require "exa/errors"
+
+module Exa
+  module Internal
+    module Transport
+      class BaseClient
+        PLATFORM_HEADERS = {
+          "x-stainless-lang" => "ruby",
+          "x-stainless-runtime" => RUBY_ENGINE,
+          "x-stainless-runtime-version" => RUBY_ENGINE_VERSION
+        }.freeze
+
+        DEFAULT_MAX_RETRIES = 2
+        DEFAULT_TIMEOUT = 120.0
+        DEFAULT_INITIAL_RETRY_DELAY = 0.5
+        DEFAULT_MAX_RETRY_DELAY = 8.0
+
+        attr_reader :base_url, :timeout, :max_retries, :initial_retry_delay, :max_retry_delay, :headers, :requester
+
+        def initialize(
+          base_url:,
+          timeout: DEFAULT_TIMEOUT,
+          max_retries: DEFAULT_MAX_RETRIES,
+          initial_retry_delay: DEFAULT_INITIAL_RETRY_DELAY,
+          max_retry_delay: DEFAULT_MAX_RETRY_DELAY,
+          headers: {},
+          requester: Exa::Internal::Transport::PooledNetRequester.new
+        )
+          @base_url = URI(base_url)
+          @timeout = timeout
+          @max_retries = max_retries
+          @initial_retry_delay = initial_retry_delay
+          @max_retry_delay = max_retry_delay
+          @headers = headers || {}
+          @requester = requester
+        end
+
+        def request(method:, path:, query: nil, headers: nil, body: nil, unwrap: nil, stream: false, response_model: nil)
+          req = build_request(
+            method: method,
+            path: Array(path).join("/"),
+            query: query,
+            headers: headers,
+            body: body
+          )
+
+          _, response, stream_enum = send_request(req)
+          parsed_headers = Exa::Internal::Util.normalized_headers(response.each_header.to_h)
+
+          if stream
+            Exa::Internal::Transport::Stream.new(headers: parsed_headers, stream: stream_enum)
+          else
+            decoded = Exa::Internal::Util.decode_content(parsed_headers, stream: stream_enum)
+            coerced = coerce_response(response_model, decoded)
+            unwrap ? dig(coerced, unwrap) : coerced
+          end
+        end
+
+        private
+
+        def normalize_path(path)
+          segments = Array(path).flat_map do |segment|
+            next [] if segment.nil?
+            segment.to_s.split("/")
+          end
+          cleaned = segments.reject(&:empty?)
+          return "" if cleaned.empty?
+          cleaned.join("/")
+        end
+
+        def build_request(method:, path:, query:, headers:, body:)
+          normalized_path = normalize_path(path)
+          url = @base_url + normalized_path
+          url.query = Exa::Internal::Util.build_query(query)
+
+          header_overrides = headers ? headers.each_with_object({}) { |(k, v), acc| acc[k] = v unless v.nil? } : {}
+          final_headers = PLATFORM_HEADERS.merge(default_headers).merge(header_overrides)
+
+          payload = case body
+                    when nil
+                      nil
+                    when String
+                      final_headers["content-type"] ||= "application/json"
+                      body
+                    else
+                      final_headers["content-type"] ||= "application/json"
+                      JSON.generate(body)
+                    end
+
+          {
+            method: method,
+            url: url,
+            headers: final_headers,
+            body: payload,
+            deadline: Process.clock_gettime(Process::CLOCK_MONOTONIC) + timeout,
+            max_retries: max_retries
+          }
+        end
+
+        def default_headers
+          merged = headers.merge(auth_headers)
+          merged.each_with_object({}) do |(k, v), acc|
+            next if v.nil?
+            acc[k] = v
+          end
+        end
+
+        def auth_headers
+          {}
+        end
+
+        def send_request(request, retry_count: 0)
+          status, response, body_enum = @requester.execute(request)
+          if should_retry?(status) && retry_count < max_retries
+            sleep(retry_delay(retry_count))
+            return send_request(request, retry_count: retry_count + 1)
+          end
+
+          if status >= 400
+            decoded_headers = Exa::Internal::Util.normalized_headers(response.each_header.to_h)
+            payload = Exa::Internal::Util.decode_content(decoded_headers, stream: body_enum)
+            Exa::Errors::APIStatusError.raise!(
+              url: request[:url],
+              status: status,
+              headers: decoded_headers,
+              body: payload
+            )
+          end
+
+          [status, response, body_enum]
+        rescue Exa::Errors::APITimeoutError, Exa::Errors::APIConnectionError, Exa::Errors::APIStatusError
+          raise
+        rescue Timeout::Error
+          raise Exa::Errors::APITimeoutError.new("Request timed out", url: request[:url])
+        rescue StandardError => e
+          raise Exa::Errors::APIConnectionError.new(e.message, url: request[:url])
+        end
+
+        def should_retry?(status)
+          [408, 409, 429].include?(status) || status >= 500
+        end
+
+        def retry_delay(retry_count)
+          delay = initial_retry_delay * (2**retry_count)
+          [delay, max_retry_delay].min
+        end
+
+        def dig(obj, path)
+          Array(path).reduce(obj) do |memo, key|
+            memo.is_a?(Hash) ? memo[key] : nil
+          end
+        end
+
+        def coerce_response(model, data)
+          return data unless model
+          return model.from_hash(data) if model.respond_to?(:from_hash)
+          model.new(data)
+        end
+      end
+    end
+  end
+end

data/lib/exa/internal/transport/pooled_net_requester.rb

@@ -0,0 +1,113 @@
+# frozen_string_literal: true
+
+require "net/http"
+require "uri"
+require "connection_pool"
+require "etc"
+
+module Exa
+  module Internal
+    module Transport
+      class PooledNetRequester
+        KEEP_ALIVE_TIMEOUT = 30
+        DEFAULT_MAX_CONNECTIONS = [Etc.nprocessors, 99].max
+
+        def initialize(size: DEFAULT_MAX_CONNECTIONS)
+          @size = size
+          @mutex = Mutex.new
+          @pools = {}
+        end
+
+        def execute(request)
+          url = request.fetch(:url)
+          deadline = request.fetch(:deadline)
+
+          pool_for(url).with(timeout: remaining(deadline)) do |conn|
+            req, body_closer = build_request(request) do
+              calibrate_socket_timeout(conn, deadline)
+            end
+
+            calibrate_socket_timeout(conn, deadline)
+            start_connection(conn)
+            calibrate_socket_timeout(conn, deadline)
+
+            enum = Enumerator.new do |y|
+              conn.request(req) do |resp|
+                y << [req, resp]
+                resp.read_body do |chunk|
+                  y << chunk.force_encoding(Encoding::BINARY)
+                end
+              end
+            rescue Timeout::Error
+              raise Exa::Errors::APITimeoutError.new("Request timed out", url: url)
+            rescue StandardError => e
+              raise Exa::Errors::APIConnectionError.new(e.message, url: url)
+            ensure
+              body_closer&.call
+            end
+
+            _, response = enum.next
+            body = Exa::Internal::Util.fused_enum(enum)
+            [Integer(response.code), response, body]
+          end
+        end
+
+        private
+
+        def pool_for(url)
+          origin = uri_origin(url)
+          @mutex.synchronize do
+            @pools[origin] ||= ConnectionPool.new(size: @size) { build_connection(url) }
+          end
+        end
+
+        def build_connection(url)
+          conn = Net::HTTP.new(url.host, url.port)
+          conn.use_ssl = url.scheme == "https"
+          conn.keep_alive_timeout = KEEP_ALIVE_TIMEOUT
+          conn
+        end
+
+        def start_connection(conn)
+          return if conn.started?
+          conn.start
+        end
+
+        def calibrate_socket_timeout(conn, deadline)
+          remaining = remaining(deadline)
+          conn.open_timeout = remaining
+          conn.read_timeout = remaining
+          conn.write_timeout = remaining if conn.respond_to?(:write_timeout=)
+        end
+
+        def remaining(deadline)
+          [deadline - Process.clock_gettime(Process::CLOCK_MONOTONIC), 0.001].max
+        end
+
+        def build_request(request)
+          method, url, headers, body = request.values_at(:method, :url, :headers, :body)
+          req = Net::HTTPGenericRequest.new(method.to_s.upcase, !body.nil?, method != :head, URI(url.to_s))
+          headers.each { req[_1] = _2 }
+
+          case body
+          when nil
+          when String
+            req.body = body
+            req["content-length"] = body.bytesize.to_s
+          when StringIO
+            req.body_stream = body
+            req["content-length"] = body.size.to_s
+          else
+            req.body = body
+          end
+
+          [req, req.body_stream&.method(:close)]
+        end
+
+        def uri_origin(uri)
+          [uri.scheme, uri.host, uri.port].join(":")
+        end
+      end
+    end
+  end
+end