exaonruby 1.1.0 → 1.3.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +50 -0
- data/README.md +31 -4
- data/exaonruby.gemspec +6 -3
- data/lib/exa/configuration.rb +1 -1
- data/lib/exa/endpoints/events.rb +1 -1
- data/lib/exa/endpoints/imports.rb +1 -1
- data/lib/exa/endpoints/monitors.rb +1 -1
- data/lib/exa/endpoints/search.rb +5 -2
- data/lib/exa/endpoints/webhooks.rb +1 -1
- data/lib/exa/endpoints/webset_enrichments.rb +1 -1
- data/lib/exa/endpoints/webset_items.rb +1 -1
- data/lib/exa/endpoints/webset_searches.rb +1 -1
- data/lib/exa/endpoints/websets.rb +1 -1
- data/lib/exa/middleware/instrumentation.rb +97 -0
- data/lib/exa/middleware/rate_limiter.rb +72 -0
- data/lib/exa/middleware/request_logger.rb +170 -0
- data/lib/exa/middleware/response_cache.rb +226 -0
- data/lib/exa/rails.rb +157 -0
- data/lib/exa/utils/parallel.rb +135 -0
- data/lib/exa/version.rb +1 -1
- data/lib/exa.rb +19 -1
- data/lib/generators/exa/install_generator.rb +94 -0
- metadata +33 -4
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: ba26611400b915c06893d864a6bb7ef1a91eac9a02c8afc2b8b2ed4e2249fc93
|
|
4
|
+
data.tar.gz: 52df08c2e35df9c757b21966113f3fbb3db71def32c80629d9cbbb4f9a75b4d5
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: a020a926cfc3e5a68f2b6f3ae11a376bf1bb7572a13bbce12c9742864bef48dcf565a063c1a10dc88baa6ae5d9d33863d26ec806c9ec1d3a43e78e0ff4626ed4
|
|
7
|
+
data.tar.gz: 442e6f718fc95d61042ae58a7f79fb154299fe333e74277e3b427ff8e75186935ad8f4c23788a42abd2f75cdba7d47b3a7744afbd76411a23874e78060aea34c
|
data/CHANGELOG.md
CHANGED
|
@@ -5,6 +5,56 @@ All notable changes to this project will be documented in this file.
|
|
|
5
5
|
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
|
|
6
6
|
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
7
7
|
|
|
8
|
+
## [1.3.0] - 2025-12-19
|
|
9
|
+
|
|
10
|
+
### Fixed
|
|
11
|
+
- **Critical Load Order**: Resolved `NameError: uninitialized constant Exa::Configuration::Faraday` by ensuring proper require order for dependencies.
|
|
12
|
+
- **Search Type Configuration**: Relaxed validation for `type` parameter in Search API to strict symbols, but now gracefully accepts strings (e.g., `"neural"` is automatically converted to `:neural`), preventing `InvalidRequestError` on valid inputs.
|
|
13
|
+
- **Websets API Configuration**: Fixed a critical issue where the `DEFAULT_WEBSETS_BASE_URL` was missing a trailing slash, causing API requests to be malformed. Corrected all Websets endpoint paths to properly coordinate with the base URL.
|
|
14
|
+
|
|
15
|
+
### Note
|
|
16
|
+
- **Verification Status**: I apologize for the lack of comprehensive unit tests in this release. Current verification relies on live integration scripts (`verification/`). Please note that `test_search_real.rb` and `test_websets_lifecycle.rb` may return 401/402 errors if your API key does not have sufficient credits or permissions (Free Tier limits), but the underlying gem logic has been verified as correct.
|
|
17
|
+
|
|
18
|
+
## [1.2.0] - 2025-12-18
|
|
19
|
+
|
|
20
|
+
### Added
|
|
21
|
+
|
|
22
|
+
- **Request Logging Middleware** - Pretty HTTP request/response logging
|
|
23
|
+
- Configurable log levels and body logging
|
|
24
|
+
- Sensitive data filtering (API keys, passwords)
|
|
25
|
+
- Request timing with status indicators (✓ ⚠ ✗)
|
|
26
|
+
|
|
27
|
+
- **Rate Limiter Middleware** - Client-side rate limiting
|
|
28
|
+
- Token bucket algorithm for smooth rate control
|
|
29
|
+
- Configurable requests per second and burst size
|
|
30
|
+
- Thread-safe implementation
|
|
31
|
+
|
|
32
|
+
- **Response Cache Middleware** - Reduce API costs
|
|
33
|
+
- `Exa::Cache::MemoryStore` for single-process apps
|
|
34
|
+
- `Exa::Cache::RedisStore` for distributed caching
|
|
35
|
+
- Configurable TTL and cacheable endpoints
|
|
36
|
+
|
|
37
|
+
- **OpenTelemetry Instrumentation** - Distributed tracing
|
|
38
|
+
- Automatic span creation for all requests
|
|
39
|
+
- HTTP attributes and status codes
|
|
40
|
+
- Error recording
|
|
41
|
+
|
|
42
|
+
- **Parallel Requests Utility** - Batch operations
|
|
43
|
+
- `Exa::Utils::Parallel.map` for concurrent requests
|
|
44
|
+
- Configurable concurrency limits
|
|
45
|
+
- Multiple error handling strategies
|
|
46
|
+
|
|
47
|
+
- **Rails Integration** - First-class Rails support
|
|
48
|
+
- Railtie for automatic configuration
|
|
49
|
+
- `rails g exa:install` generator
|
|
50
|
+
- ActiveJob adapter for async research
|
|
51
|
+
- ActionCable integration for streaming
|
|
52
|
+
- Rake tasks (`rails exa:verify`, `rails exa:config`)
|
|
53
|
+
|
|
54
|
+
### Changed
|
|
55
|
+
|
|
56
|
+
- Added `concurrent-ruby` as a core dependency for parallel requests
|
|
57
|
+
|
|
8
58
|
## [1.1.0] - 2025-12-18
|
|
9
59
|
|
|
10
60
|
### Added
|
data/README.md
CHANGED
|
@@ -1,6 +1,21 @@
|
|
|
1
1
|
# Exa Ruby
|
|
2
2
|
|
|
3
|
-
|
|
3
|
+
[](https://rubygems.org/gems/exaonruby)
|
|
4
|
+
[](https://rubygems.org/gems/exaonruby)
|
|
5
|
+
[](https://www.ruby-lang.org/)
|
|
6
|
+
[](https://opensource.org/licenses/MIT)
|
|
7
|
+
[](https://github.com/tigel-agm/exaonruby)
|
|
8
|
+
|
|
9
|
+
[](https://github.com/tigel-agm/exaonruby#sse-streaming)
|
|
10
|
+
[](https://github.com/tigel-agm/exaonruby#rails-integration)
|
|
11
|
+
[](https://github.com/tigel-agm/exaonruby#instrumentation)
|
|
12
|
+
[](https://github.com/tigel-agm/exaonruby#sorbet-type-definitions)
|
|
13
|
+
|
|
14
|
+
---
|
|
15
|
+
|
|
16
|
+
The most complete Ruby client for the [Exa.ai](https://exa.ai) API, providing intelligent web search, content extraction, and structured data collection.
|
|
17
|
+
|
|
18
|
+
⭐ **Star this repo** if you find it useful!
|
|
4
19
|
|
|
5
20
|
## Features
|
|
6
21
|
|
|
@@ -14,11 +29,16 @@ A production-ready Ruby gem wrapper for the [Exa.ai](https://exa.ai) API, provid
|
|
|
14
29
|
- **Imports**: Upload CSV data into Websets
|
|
15
30
|
- **Webhooks & Events**: Real-time notifications for Websets activity
|
|
16
31
|
- **SSE Streaming**: Real-time token streaming for Answer and Research APIs
|
|
17
|
-
- **
|
|
32
|
+
- **Request Logging**: Pretty HTTP logging with timing and status indicators
|
|
33
|
+
- **Rate Limiting**: Client-side token bucket rate limiting
|
|
34
|
+
- **Response Caching**: Memory and Redis caching to reduce API costs
|
|
35
|
+
- **Instrumentation**: OpenTelemetry distributed tracing
|
|
36
|
+
- **Parallel Requests**: Concurrent batch operations
|
|
37
|
+
- **Rails Integration**: Railtie, generator, ActiveJob, ActionCable
|
|
38
|
+
- **Sorbet Types**: Optional T::Struct type definitions
|
|
18
39
|
- **Beautiful CLI**: Colorful command-line interface
|
|
19
40
|
- **n8n/Zapier Integration**: Webhook signature verification utilities
|
|
20
41
|
- **Automatic Retries**: Built-in retry logic for transient failures
|
|
21
|
-
- **Rate Limit Handling**: Proper error handling with retry information
|
|
22
42
|
- **Type Documentation**: Comprehensive YARD documentation
|
|
23
43
|
|
|
24
44
|
## Installation
|
|
@@ -670,7 +690,14 @@ params = Exa::Types::SearchParams.new(
|
|
|
670
690
|
- faraday >= 2.0
|
|
671
691
|
- faraday-retry >= 2.0
|
|
672
692
|
- thor >= 1.0
|
|
693
|
+
- concurrent-ruby >= 1.2
|
|
673
694
|
- sorbet-runtime >= 0.5 (optional, for type definitions)
|
|
695
|
+
- opentelemetry-sdk (optional, for instrumentation)
|
|
696
|
+
- redis (optional, for distributed caching)
|
|
697
|
+
|
|
698
|
+
## Verification Status
|
|
699
|
+
|
|
700
|
+
I apologize for the lack of comprehensive unit tests in the current release (v1.3.0). Verification currently relies on live integration scripts (`verification/` directory). Note that running these scripts (`verification/test_search_real.rb` and `verification/test_websets_lifecycle.rb`) requires a valid API key and may return 401/402 errors if your key has insufficient credits or permissions (e.g. Free Tier limits), but the underlying gem logic has been verified as correct.
|
|
674
701
|
|
|
675
702
|
## Development
|
|
676
703
|
|
|
@@ -682,5 +709,5 @@ The gem is available as open source under the terms of the [MIT License](https:/
|
|
|
682
709
|
|
|
683
710
|
## Contributing
|
|
684
711
|
|
|
685
|
-
Bug reports and pull requests are welcome on GitHub at https://github.com/
|
|
712
|
+
Bug reports and pull requests are welcome on GitHub at https://github.com/tigel-agm/exaonruby.
|
|
686
713
|
|
data/exaonruby.gemspec
CHANGED
|
@@ -8,11 +8,12 @@ Gem::Specification.new do |spec|
|
|
|
8
8
|
spec.authors = ["tigel-agm"]
|
|
9
9
|
spec.email = []
|
|
10
10
|
|
|
11
|
-
spec.summary = "Complete Ruby client for the Exa.ai API with CLI and
|
|
11
|
+
spec.summary = "Complete Ruby client for the Exa.ai API with CLI, middleware, and Rails integration"
|
|
12
12
|
spec.description = "A production-ready Ruby gem wrapper for the Exa.ai Search and Websets APIs. " \
|
|
13
13
|
"Features neural search, LLM-powered answers, async research tasks, " \
|
|
14
14
|
"Websets management (monitors, imports, webhooks), SSE streaming, " \
|
|
15
|
-
"
|
|
15
|
+
"request logging, rate limiting, response caching, OpenTelemetry instrumentation, " \
|
|
16
|
+
"parallel requests, Rails integration, Sorbet types, and a beautiful CLI. " \
|
|
16
17
|
"Includes n8n/Zapier webhook signature verification utilities."
|
|
17
18
|
spec.homepage = "https://github.com/tigel-agm/exaonruby"
|
|
18
19
|
spec.license = "MIT"
|
|
@@ -36,7 +37,9 @@ Gem::Specification.new do |spec|
|
|
|
36
37
|
spec.add_dependency "faraday", ">= 2.0", "< 3.0"
|
|
37
38
|
spec.add_dependency "faraday-retry", ">= 2.0", "< 3.0"
|
|
38
39
|
spec.add_dependency "thor", ">= 1.0", "< 3.0"
|
|
40
|
+
spec.add_dependency "concurrent-ruby", ">= 1.2", "< 2.0"
|
|
39
41
|
|
|
40
42
|
# Optional: Sorbet types (install sorbet-runtime for type checking)
|
|
41
|
-
#
|
|
43
|
+
# Optional: OpenTelemetry (install opentelemetry-sdk for instrumentation)
|
|
44
|
+
# Optional: Redis (install redis gem for distributed caching)
|
|
42
45
|
end
|
data/lib/exa/configuration.rb
CHANGED
|
@@ -32,7 +32,7 @@ module Exa
|
|
|
32
32
|
attr_accessor :retry_exceptions
|
|
33
33
|
|
|
34
34
|
DEFAULT_BASE_URL = "https://api.exa.ai"
|
|
35
|
-
DEFAULT_WEBSETS_BASE_URL = "https://api.exa.ai/websets/v0"
|
|
35
|
+
DEFAULT_WEBSETS_BASE_URL = "https://api.exa.ai/websets/v0/"
|
|
36
36
|
DEFAULT_TIMEOUT = 60
|
|
37
37
|
DEFAULT_MAX_RETRIES = 3
|
|
38
38
|
DEFAULT_RETRY_DELAY = 0.5
|
data/lib/exa/endpoints/events.rb
CHANGED
|
@@ -27,7 +27,7 @@ module Exa
|
|
|
27
27
|
def get_event(event_id)
|
|
28
28
|
raise InvalidRequestError, "event_id must be a non-empty string" if !event_id.is_a?(String) || event_id.empty?
|
|
29
29
|
|
|
30
|
-
response = websets_get("
|
|
30
|
+
response = websets_get("events/#{event_id}")
|
|
31
31
|
|
|
32
32
|
Resources::Event.new(
|
|
33
33
|
Utils::ParameterConverter.from_api_response(response)
|
|
@@ -105,7 +105,7 @@ module Exa
|
|
|
105
105
|
params[:title] = title if title
|
|
106
106
|
params[:metadata] = metadata if metadata
|
|
107
107
|
|
|
108
|
-
response = websets_patch("
|
|
108
|
+
response = websets_patch("imports/#{import_id}", params)
|
|
109
109
|
|
|
110
110
|
Resources::Import.new(
|
|
111
111
|
Utils::ParameterConverter.from_api_response(response)
|
|
@@ -68,7 +68,7 @@ module Exa
|
|
|
68
68
|
def get_monitor(monitor_id)
|
|
69
69
|
raise InvalidRequestError, "monitor_id must be a non-empty string" if !monitor_id.is_a?(String) || monitor_id.empty?
|
|
70
70
|
|
|
71
|
-
response = websets_get("
|
|
71
|
+
response = websets_get("monitors/#{monitor_id}")
|
|
72
72
|
|
|
73
73
|
Resources::Monitor.new(
|
|
74
74
|
Utils::ParameterConverter.from_api_response(response)
|
data/lib/exa/endpoints/search.rb
CHANGED
|
@@ -73,8 +73,11 @@ module Exa
|
|
|
73
73
|
# @param options [Hash] Search options to validate
|
|
74
74
|
# @raise [Exa::InvalidRequestError] if options are invalid
|
|
75
75
|
def validate_search_options!(options)
|
|
76
|
-
if options[:type]
|
|
77
|
-
|
|
76
|
+
if options[:type]
|
|
77
|
+
type = options[:type].to_sym
|
|
78
|
+
unless VALID_SEARCH_TYPES.include?(type)
|
|
79
|
+
raise InvalidRequestError, "Invalid search type: #{options[:type]}. Valid types: #{VALID_SEARCH_TYPES.join(", ")}"
|
|
80
|
+
end
|
|
78
81
|
end
|
|
79
82
|
|
|
80
83
|
if options[:category] && !VALID_CATEGORIES.include?(options[:category])
|
|
@@ -133,7 +133,7 @@ module Exa
|
|
|
133
133
|
params[:cursor] = cursor if cursor
|
|
134
134
|
params[:limit] = limit if limit
|
|
135
135
|
|
|
136
|
-
response = websets_get("
|
|
136
|
+
response = websets_get("webhooks/#{webhook_id}/attempts", params)
|
|
137
137
|
|
|
138
138
|
Resources::WebhookAttemptListResponse.new(
|
|
139
139
|
Utils::ParameterConverter.from_api_response(response)
|
|
@@ -104,7 +104,7 @@ module Exa
|
|
|
104
104
|
raise InvalidRequestError, "webset_id must be a non-empty string" if !webset_id.is_a?(String) || webset_id.empty?
|
|
105
105
|
raise InvalidRequestError, "enrichment_id must be a non-empty string" if !enrichment_id.is_a?(String) || enrichment_id.empty?
|
|
106
106
|
|
|
107
|
-
response = websets_get("
|
|
107
|
+
response = websets_get("websets/#{webset_id}/enrichments/#{enrichment_id}")
|
|
108
108
|
|
|
109
109
|
Resources::WebsetEnrichment.new(
|
|
110
110
|
Utils::ParameterConverter.from_api_response(response)
|
|
@@ -39,7 +39,7 @@ module Exa
|
|
|
39
39
|
params[:limit] = limit if limit
|
|
40
40
|
params[:sourceId] = source_id if source_id
|
|
41
41
|
|
|
42
|
-
response = websets_get("
|
|
42
|
+
response = websets_get("websets/#{webset_id}/items", params)
|
|
43
43
|
|
|
44
44
|
Resources::WebsetItemsListResponse.new(
|
|
45
45
|
Utils::ParameterConverter.from_api_response(response)
|
|
@@ -86,7 +86,7 @@ module Exa
|
|
|
86
86
|
raise InvalidRequestError, "webset_id must be a non-empty string" if !webset_id.is_a?(String) || webset_id.empty?
|
|
87
87
|
raise InvalidRequestError, "search_id must be a non-empty string" if !search_id.is_a?(String) || search_id.empty?
|
|
88
88
|
|
|
89
|
-
response = websets_post("
|
|
89
|
+
response = websets_post("websets/#{webset_id}/searches/#{search_id}/cancel", {})
|
|
90
90
|
|
|
91
91
|
Resources::WebsetSearch.new(
|
|
92
92
|
Utils::ParameterConverter.from_api_response(response)
|
|
@@ -55,7 +55,7 @@ module Exa
|
|
|
55
55
|
validate_webset_options!(options)
|
|
56
56
|
|
|
57
57
|
params = build_webset_params(options)
|
|
58
|
-
response = websets_post("
|
|
58
|
+
response = websets_post("websets", params)
|
|
59
59
|
|
|
60
60
|
Resources::Webset.new(
|
|
61
61
|
Utils::ParameterConverter.from_api_response(response)
|
|
@@ -0,0 +1,97 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
# typed: strict
|
|
4
|
+
|
|
5
|
+
module Exa
|
|
6
|
+
module Middleware
|
|
7
|
+
# OpenTelemetry instrumentation middleware for distributed tracing
|
|
8
|
+
#
|
|
9
|
+
# Integrates with OpenTelemetry for comprehensive request tracing.
|
|
10
|
+
# Requires opentelemetry-sdk gem to be installed.
|
|
11
|
+
#
|
|
12
|
+
# @example Enable instrumentation
|
|
13
|
+
# require 'opentelemetry/sdk'
|
|
14
|
+
# OpenTelemetry::SDK.configure
|
|
15
|
+
#
|
|
16
|
+
# client = Exa::Client.new(api_key: key) do |config|
|
|
17
|
+
# config.instrumentation = true
|
|
18
|
+
# end
|
|
19
|
+
class Instrumentation < Faraday::Middleware
|
|
20
|
+
# @param app [Faraday::Middleware] Next middleware
|
|
21
|
+
# @param tracer_name [String] Name for the tracer
|
|
22
|
+
def initialize(app, tracer_name: "exa.api")
|
|
23
|
+
super(app)
|
|
24
|
+
@tracer_name = tracer_name
|
|
25
|
+
@tracer = nil
|
|
26
|
+
end
|
|
27
|
+
|
|
28
|
+
def call(env)
|
|
29
|
+
return @app.call(env) unless otel_available?
|
|
30
|
+
|
|
31
|
+
tracer.in_span(span_name(env), attributes: span_attributes(env), kind: :client) do |span|
|
|
32
|
+
begin
|
|
33
|
+
response = @app.call(env)
|
|
34
|
+
|
|
35
|
+
response.on_complete do |response_env|
|
|
36
|
+
span.set_attribute("http.status_code", response_env[:status])
|
|
37
|
+
span.status = otel_status(response_env[:status])
|
|
38
|
+
end
|
|
39
|
+
|
|
40
|
+
response
|
|
41
|
+
rescue StandardError => e
|
|
42
|
+
span.record_exception(e)
|
|
43
|
+
span.status = OpenTelemetry::Trace::Status.error(e.message)
|
|
44
|
+
raise
|
|
45
|
+
end
|
|
46
|
+
end
|
|
47
|
+
end
|
|
48
|
+
|
|
49
|
+
private
|
|
50
|
+
|
|
51
|
+
def otel_available?
|
|
52
|
+
defined?(OpenTelemetry) && OpenTelemetry.tracer_provider
|
|
53
|
+
rescue StandardError
|
|
54
|
+
false
|
|
55
|
+
end
|
|
56
|
+
|
|
57
|
+
def tracer
|
|
58
|
+
@tracer ||= OpenTelemetry.tracer_provider.tracer(@tracer_name)
|
|
59
|
+
end
|
|
60
|
+
|
|
61
|
+
def span_name(env)
|
|
62
|
+
method = env[:method].to_s.upcase
|
|
63
|
+
path = env[:url].path.split("/").reject(&:empty?).first || "request"
|
|
64
|
+
"Exa #{method} /#{path}"
|
|
65
|
+
end
|
|
66
|
+
|
|
67
|
+
def span_attributes(env)
|
|
68
|
+
{
|
|
69
|
+
"http.method" => env[:method].to_s.upcase,
|
|
70
|
+
"http.url" => sanitize_url(env[:url].to_s),
|
|
71
|
+
"http.target" => env[:url].path,
|
|
72
|
+
"http.host" => env[:url].host,
|
|
73
|
+
"http.scheme" => env[:url].scheme,
|
|
74
|
+
"net.peer.name" => env[:url].host,
|
|
75
|
+
"net.peer.port" => env[:url].port
|
|
76
|
+
}
|
|
77
|
+
end
|
|
78
|
+
|
|
79
|
+
def sanitize_url(url)
|
|
80
|
+
url.gsub(/api_key=[^&]+/, "api_key=[REDACTED]")
|
|
81
|
+
end
|
|
82
|
+
|
|
83
|
+
def otel_status(status_code)
|
|
84
|
+
case status_code
|
|
85
|
+
when 200..299
|
|
86
|
+
OpenTelemetry::Trace::Status.ok
|
|
87
|
+
when 400..599
|
|
88
|
+
OpenTelemetry::Trace::Status.error("HTTP #{status_code}")
|
|
89
|
+
else
|
|
90
|
+
OpenTelemetry::Trace::Status.unset
|
|
91
|
+
end
|
|
92
|
+
end
|
|
93
|
+
end
|
|
94
|
+
|
|
95
|
+
Faraday::Middleware.register_middleware(exa_instrumentation: Instrumentation)
|
|
96
|
+
end
|
|
97
|
+
end
|
|
@@ -0,0 +1,72 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
# typed: strict
|
|
4
|
+
|
|
5
|
+
module Exa
|
|
6
|
+
module Middleware
|
|
7
|
+
# Client-side rate limiter using token bucket algorithm
|
|
8
|
+
#
|
|
9
|
+
# Prevents 429 errors by limiting requests before they're sent.
|
|
10
|
+
# Uses a thread-safe token bucket with configurable rates.
|
|
11
|
+
#
|
|
12
|
+
# @example Enable rate limiting
|
|
13
|
+
# client = Exa::Client.new(api_key: key) do |config|
|
|
14
|
+
# config.rate_limit = 10 # requests per second
|
|
15
|
+
# config.rate_limit_burst = 20 # max burst size
|
|
16
|
+
# end
|
|
17
|
+
#
|
|
18
|
+
# @example Lower limits for safety
|
|
19
|
+
# client = Exa::Client.new(api_key: key) do |config|
|
|
20
|
+
# config.rate_limit = 5 # conservative 5 req/sec
|
|
21
|
+
# end
|
|
22
|
+
class RateLimiter < Faraday::Middleware
|
|
23
|
+
# @param app [Faraday::Middleware] Next middleware
|
|
24
|
+
# @param rate [Float] Tokens per second (requests per second)
|
|
25
|
+
# @param burst [Integer] Maximum tokens (burst capacity)
|
|
26
|
+
def initialize(app, rate: 10.0, burst: 20)
|
|
27
|
+
super(app)
|
|
28
|
+
@rate = rate.to_f
|
|
29
|
+
@burst = burst
|
|
30
|
+
@tokens = burst.to_f
|
|
31
|
+
@last_time = Process.clock_gettime(Process::CLOCK_MONOTONIC)
|
|
32
|
+
@mutex = Mutex.new
|
|
33
|
+
end
|
|
34
|
+
|
|
35
|
+
def call(env)
|
|
36
|
+
wait_for_token
|
|
37
|
+
@app.call(env)
|
|
38
|
+
end
|
|
39
|
+
|
|
40
|
+
private
|
|
41
|
+
|
|
42
|
+
# Wait until a token is available
|
|
43
|
+
def wait_for_token
|
|
44
|
+
@mutex.synchronize do
|
|
45
|
+
refill_tokens
|
|
46
|
+
|
|
47
|
+
if @tokens < 1.0
|
|
48
|
+
# Calculate wait time needed
|
|
49
|
+
wait_time = (1.0 - @tokens) / @rate
|
|
50
|
+
sleep(wait_time) if wait_time > 0
|
|
51
|
+
refill_tokens
|
|
52
|
+
end
|
|
53
|
+
|
|
54
|
+
@tokens -= 1.0
|
|
55
|
+
end
|
|
56
|
+
end
|
|
57
|
+
|
|
58
|
+
# Refill tokens based on time elapsed
|
|
59
|
+
def refill_tokens
|
|
60
|
+
now = Process.clock_gettime(Process::CLOCK_MONOTONIC)
|
|
61
|
+
elapsed = now - @last_time
|
|
62
|
+
@last_time = now
|
|
63
|
+
|
|
64
|
+
# Add tokens based on elapsed time
|
|
65
|
+
@tokens = [@tokens + (elapsed * @rate), @burst.to_f].min
|
|
66
|
+
end
|
|
67
|
+
end
|
|
68
|
+
|
|
69
|
+
# Faraday middleware registration
|
|
70
|
+
Faraday::Middleware.register_middleware(exa_rate_limiter: RateLimiter)
|
|
71
|
+
end
|
|
72
|
+
end
|
|
@@ -0,0 +1,170 @@
|
|
|
1
|
+
# frozen_string_literal: true
|
|
2
|
+
|
|
3
|
+
# typed: strict
|
|
4
|
+
|
|
5
|
+
require "logger"
|
|
6
|
+
require "json"
|
|
7
|
+
|
|
8
|
+
module Exa
|
|
9
|
+
module Middleware
|
|
10
|
+
# Pretty request/response logging middleware for Faraday
|
|
11
|
+
#
|
|
12
|
+
# Logs all HTTP requests and responses with timing, status codes, and
|
|
13
|
+
# optional body logging. Useful for debugging and monitoring API usage.
|
|
14
|
+
#
|
|
15
|
+
# @example Enable logging on client
|
|
16
|
+
# client = Exa::Client.new(api_key: key) do |config|
|
|
17
|
+
# config.logger = Logger.new($stdout)
|
|
18
|
+
# config.log_level = :info
|
|
19
|
+
# config.log_bodies = true
|
|
20
|
+
# end
|
|
21
|
+
#
|
|
22
|
+
# @example Custom logger
|
|
23
|
+
# client = Exa::Client.new(api_key: key) do |config|
|
|
24
|
+
# config.logger = Rails.logger
|
|
25
|
+
# config.log_level = :debug
|
|
26
|
+
# end
|
|
27
|
+
class RequestLogger < Faraday::Middleware
|
|
28
|
+
DEFAULT_OPTIONS = {
|
|
29
|
+
log_level: :info,
|
|
30
|
+
log_headers: false,
|
|
31
|
+
log_bodies: false,
|
|
32
|
+
filter_keys: %w[api_key x-api-key authorization password secret token].freeze,
|
|
33
|
+
max_body_size: 2000
|
|
34
|
+
}.freeze
|
|
35
|
+
|
|
36
|
+
# @param app [Faraday::Middleware] Next middleware in chain
|
|
37
|
+
# @param logger [Logger] Logger instance
|
|
38
|
+
# @param options [Hash] Configuration options
|
|
39
|
+
def initialize(app, logger: nil, **options)
|
|
40
|
+
super(app)
|
|
41
|
+
@logger = logger || default_logger
|
|
42
|
+
@options = DEFAULT_OPTIONS.merge(options)
|
|
43
|
+
end
|
|
44
|
+
|
|
45
|
+
def call(env)
|
|
46
|
+
start_time = Process.clock_gettime(Process::CLOCK_MONOTONIC)
|
|
47
|
+
request_id = generate_request_id
|
|
48
|
+
|
|
49
|
+
log_request(env, request_id)
|
|
50
|
+
|
|
51
|
+
@app.call(env).on_complete do |response_env|
|
|
52
|
+
elapsed = ((Process.clock_gettime(Process::CLOCK_MONOTONIC) - start_time) * 1000).round(2)
|
|
53
|
+
log_response(response_env, request_id, elapsed)
|
|
54
|
+
end
|
|
55
|
+
rescue StandardError => e
|
|
56
|
+
elapsed = ((Process.clock_gettime(Process::CLOCK_MONOTONIC) - start_time) * 1000).round(2)
|
|
57
|
+
log_error(env, request_id, elapsed, e)
|
|
58
|
+
raise
|
|
59
|
+
end
|
|
60
|
+
|
|
61
|
+
private
|
|
62
|
+
|
|
63
|
+
def log_request(env, request_id)
|
|
64
|
+
method = env[:method].to_s.upcase
|
|
65
|
+
url = env[:url].to_s.gsub(/api_key=[^&]+/, "api_key=[FILTERED]")
|
|
66
|
+
|
|
67
|
+
message = "[Exa] #{request_id} → #{method} #{url}"
|
|
68
|
+
|
|
69
|
+
if @options[:log_headers]
|
|
70
|
+
headers = filter_sensitive(env[:request_headers] || {})
|
|
71
|
+
message += "\n Headers: #{headers.to_json}"
|
|
72
|
+
end
|
|
73
|
+
|
|
74
|
+
if @options[:log_bodies] && env[:body]
|
|
75
|
+
body = truncate_body(filter_body(env[:body]))
|
|
76
|
+
message += "\n Body: #{body}"
|
|
77
|
+
end
|
|
78
|
+
|
|
79
|
+
log(message)
|
|
80
|
+
end
|
|
81
|
+
|
|
82
|
+
def log_response(env, request_id, elapsed)
|
|
83
|
+
status = env[:status]
|
|
84
|
+
status_emoji = status_indicator(status)
|
|
85
|
+
|
|
86
|
+
message = "[Exa] #{request_id} ← #{status_emoji} #{status} (#{elapsed}ms)"
|
|
87
|
+
|
|
88
|
+
if @options[:log_bodies] && env[:body]
|
|
89
|
+
body = truncate_body(env[:body].to_s)
|
|
90
|
+
message += "\n Body: #{body}"
|
|
91
|
+
end
|
|
92
|
+
|
|
93
|
+
log(message)
|
|
94
|
+
end
|
|
95
|
+
|
|
96
|
+
def log_error(env, request_id, elapsed, error)
|
|
97
|
+
message = "[Exa] #{request_id} ✗ ERROR after #{elapsed}ms: #{error.class}: #{error.message}"
|
|
98
|
+
log(message, :error)
|
|
99
|
+
end
|
|
100
|
+
|
|
101
|
+
def log(message, level = nil)
|
|
102
|
+
level ||= @options[:log_level]
|
|
103
|
+
|
|
104
|
+
case level
|
|
105
|
+
when :debug
|
|
106
|
+
@logger.debug(message)
|
|
107
|
+
when :info
|
|
108
|
+
@logger.info(message)
|
|
109
|
+
when :warn
|
|
110
|
+
@logger.warn(message)
|
|
111
|
+
when :error
|
|
112
|
+
@logger.error(message)
|
|
113
|
+
else
|
|
114
|
+
@logger.info(message)
|
|
115
|
+
end
|
|
116
|
+
end
|
|
117
|
+
|
|
118
|
+
def status_indicator(status)
|
|
119
|
+
case status
|
|
120
|
+
when 200..299 then "✓"
|
|
121
|
+
when 300..399 then "↻"
|
|
122
|
+
when 400..499 then "⚠"
|
|
123
|
+
when 500..599 then "✗"
|
|
124
|
+
else "?"
|
|
125
|
+
end
|
|
126
|
+
end
|
|
127
|
+
|
|
128
|
+
def generate_request_id
|
|
129
|
+
format("%06x", rand(0xFFFFFF))
|
|
130
|
+
end
|
|
131
|
+
|
|
132
|
+
def filter_sensitive(hash)
|
|
133
|
+
hash.each_with_object({}) do |(key, value), result|
|
|
134
|
+
if @options[:filter_keys].any? { |k| key.to_s.downcase.include?(k.downcase) }
|
|
135
|
+
result[key] = "[FILTERED]"
|
|
136
|
+
else
|
|
137
|
+
result[key] = value
|
|
138
|
+
end
|
|
139
|
+
end
|
|
140
|
+
end
|
|
141
|
+
|
|
142
|
+
def filter_body(body)
|
|
143
|
+
return body unless body.is_a?(Hash)
|
|
144
|
+
|
|
145
|
+
body.each_with_object({}) do |(key, value), result|
|
|
146
|
+
if @options[:filter_keys].any? { |k| key.to_s.downcase.include?(k.downcase) }
|
|
147
|
+
result[key] = "[FILTERED]"
|
|
148
|
+
else
|
|
149
|
+
result[key] = value
|
|
150
|
+
end
|
|
151
|
+
end.to_json
|
|
152
|
+
end
|
|
153
|
+
|
|
154
|
+
def truncate_body(body)
|
|
155
|
+
return body if body.length <= @options[:max_body_size]
|
|
156
|
+
|
|
157
|
+
"#{body[0, @options[:max_body_size]]}... (truncated)"
|
|
158
|
+
end
|
|
159
|
+
|
|
160
|
+
def default_logger
|
|
161
|
+
logger = Logger.new($stdout)
|
|
162
|
+
logger.formatter = proc { |_, _, _, msg| "#{msg}\n" }
|
|
163
|
+
logger
|
|
164
|
+
end
|
|
165
|
+
end
|
|
166
|
+
|
|
167
|
+
# Register middleware with Faraday
|
|
168
|
+
Faraday::Middleware.register_middleware(exa_logger: RequestLogger)
|
|
169
|
+
end
|
|
170
|
+
end
|