brightdata 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 22a0b2f8657a178d69eb1c6c3220db2d086425af48c8388b68b2633d6d9ff194
+  data.tar.gz: cd967fbe25a97ff0eb0cad7b1319a830c98684d406bac3add844dc075b3a1ab2
+SHA512:
+  metadata.gz: bec94e0b1b6d38365cd5f0d77579159ef0b3bbe359378c2f4f76a0bf984b52be62f010c608421b59a39d63cb17ce5a004cd7e440b2e87e5797c5a7d0913f94c1
+  data.tar.gz: 204b8987e39b9284fb956373a75f349b8ddcaccb62a6a917ef3a99de4942135486354bb8f90ec247c9ea2374632c408597a50b78bba8d0522f6d5eab493c55d2

data/CHANGELOG.md

@@ -0,0 +1,38 @@
+# Changelog
+
+All notable changes to this project will be documented in this file.
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+## [Unreleased]
+
+## [0.1.0] - 2026-05-28
+
+Initial public release.
+
+### Added
+- `BrightData::Client` with configurable `api_token`, `base_url`, and `logger`.
+- LinkedIn endpoints under `client.linkedin`:
+  - `profiles` and `companies` (collect by URL)
+  - `jobs.collect_by_url`, `jobs.discover_by_url`, `jobs.discover_by_keyword`
+  - `posts.collect_by_url`, `posts.discover_by_url`,
+    `posts.discover_by_profile_url`, `posts.discover_by_company_url`
+  - `people.discover_new_profiles`
+- Every endpoint exposes both `#scrape` (synchronous, parsed results) and
+  `#trigger` (asynchronous, returns a `Snapshot`).
+- `BrightData::Snapshot#wait` polling with configurable `timeout` and
+  `poll_interval`, returning a `BrightData::Result` (`SimpleResult::Success` or
+  `Failure`).
+- Typed `Data`-backed result objects (`Profile`, `Company`, `Job`, `Post`,
+  `DiscoveredProfile`) and input objects (`JobKeywordInput`,
+  `PeopleDiscoverInput`, and `*UrlInput` variants), each exposing typed readers
+  plus `#raw`.
+- Error hierarchy rooted at `BrightData::Error`: `ConfigurationError`,
+  `ArgumentError`, `AuthError`, `HTTPError`, `RateLimitError` (`#retry_after`),
+  `ServerError`, `ScrapeTimeoutError` (`#snapshot`).
+- Mutable `BrightData::Datasets::LINKEDIN` registry for overriding or adding
+  dataset IDs at runtime.
+- Optional request/response tracing via `BRIGHTDATA_LIVE`, recorded under
+  `tmp/live/`.
+- `llm.md` — single-file, LLM-friendly reference generated from YARD docs via
+  `bin/generate_llm.rb` and `bin/prepare_release`.

data/LICENSE.txt

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Lucian Ghinda
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,149 @@
+# brightdata
+
+A typed, ergonomic Ruby client for [Bright Data](https://brightdata.com)'s
+Datasets v3 scraper APIs. Version 0.1.0 ships the LinkedIn endpoints.
+
+## Installation
+
+Add it to your Gemfile:
+
+```ruby
+gem "brightdata"
+```
+
+Then run `bundle install`, or install it directly:
+
+```sh
+gem install brightdata
+```
+
+Requires Ruby 3.4.4 or newer.
+
+## Configuration
+
+Create a client with your Bright Data API token:
+
+```ruby
+client = BrightData::Client.new(api_token: ENV.fetch("BRIGHTDATA_API_TOKEN"))
+```
+
+Optional keyword arguments:
+
+- `base_url:` — override the API host (defaults to `https://api.brightdata.com`).
+- `logger:` — a `Logger` that receives one `debug` line per request.
+
+## Synchronous vs. asynchronous
+
+Every endpoint exposes two methods:
+
+- `#scrape(...)` runs synchronously and returns parsed, typed results. Bright
+  Data caps synchronous scrapes at 60 seconds; if a job exceeds that,
+  `scrape` raises `BrightData::ScrapeTimeoutError`, which carries a resumable
+  snapshot (`error.snapshot`).
+- `#trigger(...)` starts an asynchronous collection and returns a
+  `BrightData::Snapshot` you poll with `#wait`.
+
+```ruby
+# Synchronous
+profiles = client.linkedin.profiles.scrape(
+  urls: ["https://www.linkedin.com/in/example/"]
+)
+profiles.first.name # => "Example Person"
+
+# Asynchronous
+snapshot = client.linkedin.profiles.trigger(
+  urls: ["https://www.linkedin.com/in/example/"]
+)
+result = snapshot.wait # blocks, polling progress until ready/failed/timeout
+if result.success?
+  result.payload # => Array<BrightData::LinkedIn::Types::Profile>
+else
+  result.error   # => raw failure payload from Bright Data
+end
+```
+
+`Snapshot#wait` accepts `timeout:` (default 300s) and `poll_interval:`
+(default 5s), and raises `BrightData::ScrapeTimeoutError` if the deadline
+passes before the snapshot reaches a terminal state.
+
+## LinkedIn endpoints
+
+| Call | Argument | Returns |
+| --- | --- | --- |
+| `linkedin.profiles` | `urls:` | `Types::Profile` |
+| `linkedin.companies` | `urls:` | `Types::Company` |
+| `linkedin.jobs.collect_by_url` | `urls:` | `Types::Job` |
+| `linkedin.jobs.discover_by_url` | `urls:` | `Types::Job` |
+| `linkedin.jobs.discover_by_keyword` | `queries:` (`Types::JobKeywordInput`) | `Types::Job` |
+| `linkedin.posts.collect_by_url` | `urls:` | `Types::Post` |
+| `linkedin.posts.discover_by_url` | `urls:` | `Types::Post` |
+| `linkedin.posts.discover_by_profile_url` | `profile_urls:` | `Types::Post` |
+| `linkedin.posts.discover_by_company_url` | `company_urls:` | `Types::Post` |
+| `linkedin.people.discover_new_profiles` | `queries:` (`Types::PeopleDiscoverInput`) | `Types::DiscoveredProfile` |
+
+### Discovery by keyword
+
+```ruby
+query = BrightData::LinkedIn::Types::JobKeywordInput.new(
+  location: "New York",
+  keyword: "ruby",
+  country: nil, time_range: nil, job_type: nil, experience_level: nil,
+  remote: nil, company: nil, selective_search: nil,
+  jobs_to_not_include: nil, location_radius: nil
+)
+
+jobs = client.linkedin.jobs.discover_by_keyword.scrape(queries: [query])
+```
+
+`nil` fields are omitted from the request payload.
+
+## Result types
+
+Results are immutable `Data` value objects (`Types::Profile`, `Types::Company`,
+`Types::Job`, `Types::Post`, `Types::DiscoveredProfile`). Each exposes typed
+readers for the common fields plus `#raw`, the full parsed response hash, so you
+can reach fields the gem does not yet type:
+
+```ruby
+profile = profiles.first
+profile.name        # typed reader
+profile.raw[:posts] # anything not yet typed
+```
+
+## Error handling
+
+All errors inherit from `BrightData::Error`:
+
+- `BrightData::ConfigurationError` — blank API token.
+- `BrightData::ArgumentError` — bad argument shape (note: not Ruby's
+  `::ArgumentError`).
+- `BrightData::AuthError` — 401/403 from the API.
+- `BrightData::RateLimitError` — 429; exposes `#retry_after`.
+- `BrightData::ServerError` — 5xx.
+- `BrightData::HTTPError` — other transport failures and timeouts.
+- `BrightData::ScrapeTimeoutError` — synchronous scrape exceeded the 60s cap;
+  recover via `error.snapshot.wait`.
+
+```ruby
+begin
+  client.linkedin.profiles.scrape(urls: urls)
+rescue BrightData::ScrapeTimeoutError => e
+  e.snapshot.wait # fall back to async polling
+rescue BrightData::RateLimitError => e
+  sleep(e.retry_after || 5)
+  retry
+rescue BrightData::Error => e
+  warn "Bright Data request failed: #{e.message}"
+end
+```
+
+## Documentation for AI agents
+
+[`llm.md`](llm.md) is a single-file, LLM-friendly reference generated from the
+gem's YARD documentation. Point a coding assistant at it for the full API
+surface and usage examples. Regenerate it with `bin/prepare_release` (or
+`bundle exec yardoc --format=markdown && bin/generate_llm.rb`).
+
+## License
+
+Released under the [MIT License](LICENSE.txt).

data/lib/brightdata/client.rb

@@ -0,0 +1,25 @@
+# frozen_string_literal: true
+
+module BrightData
+  # Top-level Bright Data API client.
+  #
+  # @example
+  #   client = BrightData::Client.new(api_token: ENV.fetch("BRIGHTDATA_API_TOKEN"))
+  #   client.linkedin
+  class Client
+    # @return [BrightData::HTTP] underlying HTTP wrapper
+    attr_reader :http
+
+    # @return [BrightData::LinkedIn::Namespace] LinkedIn endpoint namespace
+    attr_reader :linkedin
+
+    # @param api_token [String] Bright Data API token
+    # @param base_url [String] override Bright Data API base URL
+    # @param logger [Logger, nil] optional logger for request tracing
+    # @raise [BrightData::ConfigurationError] if `api_token` is nil or empty
+    def initialize(api_token:, base_url: BrightData::HTTP::BASE_URL, logger: nil)
+      @http = HTTP.new(api_token:, base_url:, logger:)
+      @linkedin = LinkedIn::Namespace.new(http: @http)
+    end
+  end
+end

data/lib/brightdata/datasets.rb

@@ -0,0 +1,31 @@
+# frozen_string_literal: true
+
+module BrightData
+  # Registry of Bright Data dataset IDs keyed by symbolic endpoint name.
+  module Datasets
+    # @return [Hash{Symbol=>String}] LinkedIn dataset IDs
+    LINKEDIN = { # rubocop:disable Style/MutableConstant -- intentionally mutable so callers can register or override dataset IDs
+      profiles_collect_by_url: "gd_l1viktl72bvl7bjuj0",
+      companies_collect_by_url: "gd_l1vikfnt1wgvvqz95w",
+      jobs_collect_by_url: "gd_lpfll7v5hcqtkxl6l",
+      jobs_discover_by_url: "gd_lpfll7v5hcqtkxl6l",
+      jobs_discover_by_keyword: "gd_lpfll7v5hcqtkxl6l",
+      posts_collect_by_url: "gd_lyy3tktm25m4avu764",
+      posts_discover_by_profile_url: "gd_lyy3tktm25m4avu764",
+      posts_discover_by_url: "gd_lyy3tktm25m4avu764",
+      posts_discover_by_company_url: "gd_lyy3tktm25m4avu764",
+      people_discover_new_profiles: "gd_m8d03he47z8nwb5xc"
+    }
+
+    # Fetch a LinkedIn dataset ID.
+    #
+    # @param key [Symbol] symbolic endpoint name
+    # @return [String] dataset ID
+    # @raise [BrightData::ArgumentError] if key is unknown
+    def self.id_for(key)
+      LINKEDIN.fetch(key) do
+        raise ::BrightData::ArgumentError, "Unknown LinkedIn dataset key: #{key.inspect}"
+      end
+    end
+  end
+end

data/lib/brightdata/errors.rb

@@ -0,0 +1,101 @@
+# frozen_string_literal: true
+
+module BrightData
+  # Base error class. Rescue this to catch any BrightData failure.
+  class Error < StandardError; end
+
+  # Raised when the gem is misconfigured, for example when `api_token` is blank.
+  class ConfigurationError < Error; end
+
+  # Raised when a caller passes an invalid argument shape or value.
+  #
+  # This class intentionally inherits from {BrightData::Error}, not from
+  # `::ArgumentError`. Use `rescue BrightData::ArgumentError` or
+  # `rescue BrightData::Error`.
+  class ArgumentError < Error; end
+
+  # Raised on 401/403 responses from the Bright Data API.
+  class AuthError < Error; end
+
+  # Base class for transport-level errors.
+  class HTTPError < Error
+    # @return [Integer, nil] HTTP status, or nil if the request never completed
+    attr_reader :status
+
+    # @return [String, nil] raw response body
+    attr_reader :body
+
+    # @return [Net::HTTPResponse, nil] raw Net::HTTP response
+    attr_reader :response
+
+    # @param message [String] human-readable error message
+    # @param status [Integer, nil] HTTP status, or nil if unavailable
+    # @param body [String, nil] raw response body
+    # @param response [Net::HTTPResponse, nil] raw Net::HTTP response
+    def initialize(message, status: nil, body: nil, response: nil)
+      super(message)
+      @status = status
+      @body = body
+      @response = response
+    end
+  end
+
+  # Raised on 429 Too Many Requests responses.
+  class RateLimitError < HTTPError
+    # @return [Integer, nil] value of the `Retry-After` header in seconds, or nil if absent
+    attr_reader :retry_after
+
+    # @param message [String] human-readable error message
+    # @param status [Integer] HTTP status
+    # @param body [String, nil] raw response body
+    # @param response [Net::HTTPResponse, nil] raw Net::HTTP response
+    # @param retry_after [Integer, nil] retry delay in seconds
+    def initialize(message, status: 429, body: nil, response: nil, retry_after: nil)
+      super(message, status:, body:, response:)
+      @retry_after = retry_after
+    end
+  end
+
+  # Raised on 5xx responses from the Bright Data API.
+  class ServerError < HTTPError; end
+
+  # Raised when `/scrape` exceeds Bright Data's 60-second synchronous cap.
+  class ScrapeTimeoutError < Error
+    # @return [String] snapshot ID returned by Bright Data
+    attr_reader :snapshot_id
+
+    # @return [BrightData::Snapshot, nil] resumable snapshot returned by Bright Data
+    attr_reader :snapshot
+
+    # @param message [String] human-readable error message
+    # @param snapshot_id [String] snapshot ID returned by Bright Data
+    # @param snapshot [BrightData::Snapshot, nil] snapshot object that can be waited on
+    def initialize(message, snapshot_id:, snapshot: nil)
+      super(message)
+      @snapshot_id = snapshot_id
+      @snapshot = snapshot
+    end
+  end
+
+  # Raised by explicit exception-based snapshot failure flows.
+  #
+  # `Snapshot#wait` normally returns `SimpleResult::Failure` on failed
+  # snapshots. This class is reserved for callers who opt into exception
+  # semantics in future API versions.
+  class SnapshotFailedError < Error
+    # @return [String] snapshot ID
+    attr_reader :snapshot_id
+
+    # @return [Hash, nil] failure details returned by Bright Data
+    attr_reader :details
+
+    # @param message [String] human-readable error message
+    # @param snapshot_id [String] snapshot ID
+    # @param details [Hash, nil] failure details returned by Bright Data
+    def initialize(message, snapshot_id:, details: nil)
+      super(message)
+      @snapshot_id = snapshot_id
+      @details = details
+    end
+  end
+end

data/lib/brightdata/http.rb

@@ -0,0 +1,195 @@
+# frozen_string_literal: true
+
+require "json"
+require "net/http"
+require "openssl"
+require "uri"
+
+module BrightData
+  # Thin Net::HTTP wrapper. Single point of egress for the gem.
+  #
+  # Optional request tracing lives in {LiveTrace}, kept out of this class so the
+  # request path stays a small, readable Net::HTTP shim.
+  class HTTP # rubocop:disable Metrics/ClassLength -- deliberately one cohesive HTTP shim; splitting it would scatter the request path
+    # @return [String] default Bright Data API base URL
+    BASE_URL = "https://api.brightdata.com"
+
+    # Must be greater than Bright Data's `/scrape` 60-second API cap.
+    # @return [Integer] default socket read timeout in seconds
+    DEFAULT_TIMEOUT = 90
+
+    # @param api_token [String] Bright Data API token
+    # @param base_url [String] override for testing
+    # @param open_timeout [Integer] TCP open timeout in seconds
+    # @param read_timeout [Integer] socket read timeout in seconds
+    # @param logger [Logger, nil] optional logger for debug request traces
+    # @raise [BrightData::ConfigurationError] if `api_token` is nil or empty
+    def initialize(api_token:, base_url: BASE_URL, open_timeout: 10, read_timeout: DEFAULT_TIMEOUT, logger: nil)
+      raise ConfigurationError, "api_token is required" if api_token.nil? || api_token.empty?
+
+      @api_token = api_token
+      @base_url = base_url
+      @open_timeout = open_timeout
+      @read_timeout = read_timeout
+      @logger = logger
+    end
+
+    # POST a JSON body.
+    #
+    # @param path [String] API path
+    # @param query [Hash] query string params
+    # @param body [Hash, nil] JSON body to encode
+    # @return [Hash, Array, nil] parsed JSON body
+    # @raise [BrightData::AuthError, BrightData::RateLimitError, BrightData::ServerError, BrightData::HTTPError]
+    def post(path:, query: {}, body: nil)
+      request(method: :post, path:, query:, body:)
+    end
+
+    # GET a path.
+    #
+    # @param path [String] API path
+    # @param query [Hash] query string params
+    # @return [Hash, Array, nil] parsed JSON body
+    # @raise [BrightData::AuthError, BrightData::RateLimitError, BrightData::ServerError, BrightData::HTTPError]
+    def get(path:, query: {})
+      request(method: :get, path:, query:)
+    end
+
+    private
+
+    def request(method:, path:, query: {}, body: nil)
+      uri = build_uri(path:, query:)
+      trace = LiveTrace.for(LiveTrace::Request.new(method:, path:, query:, body:, uri:))
+
+      response, duration = timed_perform(uri:, req: build_request(method:, uri:, body:))
+      trace.record_response(response:, duration:)
+
+      log_request(method:, path:, status: response.code.to_i, duration:)
+
+      handle_response(response)
+    rescue StandardError => e
+      trace&.record_error(e)
+      raise
+    end
+
+    def timed_perform(uri:, req:)
+      started = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+
+      response = perform(uri:, req:)
+
+      [response, Process.clock_gettime(Process::CLOCK_MONOTONIC) - started]
+    end
+
+    def build_uri(path:, query:)
+      uri = URI.parse("#{@base_url}#{path}")
+      uri.query = URI.encode_www_form(query) unless query.empty?
+
+      uri
+    end
+
+    def build_request(method:, uri:, body:)
+      klass = if method == :post
+                Net::HTTP::Post
+              else
+                Net::HTTP::Get
+              end
+      req = klass.new(uri)
+
+      apply_headers(req)
+      req.body = JSON.generate(body) if body
+
+      req
+    end
+
+    def apply_headers(req)
+      req["Authorization"] = "Bearer #{@api_token}"
+      req["Content-Type"] = "application/json"
+      req["Accept"] = "application/json"
+    end
+
+    def perform(uri:, req:) # rubocop:disable Metrics/MethodLength -- This is a simple method, the lines are from the Net::HTTP params
+      Net::HTTP.start(
+        uri.host,
+        uri.port,
+        use_ssl: uri.scheme == "https",
+        open_timeout: @open_timeout,
+        read_timeout: @read_timeout
+      ) do |http|
+        http.request(req)
+      end
+    rescue Net::OpenTimeout, Net::ReadTimeout => e
+      raise HTTPError.new("Timeout: #{e.message}", status: nil, body: nil)
+    rescue SocketError, SystemCallError, OpenSSL::SSL::SSLError, IOError => e
+      raise HTTPError.new("Connection failed: #{e.message}", status: nil, body: nil)
+    end
+
+    def handle_response(response)
+      status = response.code.to_i
+      return parse_body(response) if (200..299).cover?(status)
+
+      raise error_for(response, status)
+    end
+
+    # Factory mapping an error status to its exception. A dispatch at one level
+    # of abstraction, so it stays a single `case`.
+    def error_for(response, status) # rubocop:disable Metrics/MethodLength -- A case should be together all the time
+      body = response.body.to_s
+      case status
+      when 401, 403
+        AuthError.new("Bright Data API rejected the token (status #{status}): #{body}")
+      when 429
+        RateLimitError.new(
+          "Bright Data rate limit hit (status 429)",
+          status:,
+          body:,
+          response:,
+          retry_after: parse_retry_after(response)
+        )
+      when 500..599
+        ServerError.new("Bright Data server error (status #{status})", status:, body:, response:)
+      else
+        HTTPError.new("Unexpected HTTP status #{status}", status:, body:, response:)
+      end
+    end
+
+    def parse_body(response)
+      text = response.body.to_s
+      return nil if text.empty?
+
+      JSON.parse(text, symbolize_names: true)
+    rescue JSON::ParserError => e
+      return parse_json_lines(text, response:) if json_lines?(text, response:)
+
+      raise HTTPError.new("Invalid JSON response: #{e.message}", status: response.code.to_i, body: text, response:)
+    end
+
+    def json_lines?(text, response:)
+      response["content-type"].to_s.include?("jsonl") || text.lines.count > 1
+    end
+
+    def parse_json_lines(text, response:)
+      text.each_line.filter_map do |line|
+        stripped = line.strip
+        next if stripped.empty?
+
+        JSON.parse(stripped, symbolize_names: true)
+      end
+    rescue JSON::ParserError => e
+      raise HTTPError.new("Invalid JSONL response: #{e.message}", status: response.code.to_i, body: text, response:)
+    end
+
+    def parse_retry_after(response)
+      raw = response["retry-after"] || response["Retry-After"]
+      Integer(raw, 10) if raw&.match?(/\A\d+\z/)
+    end
+
+    def log_request(method:, path:, status:, duration:)
+      return unless @logger
+
+      @logger.debug do
+        format("[brightdata] %<method>s %<path>s -> %<status>d in %<duration>.3fs",
+               method: method.upcase, path:, status:, duration:)
+      end
+    end
+  end
+end

data/lib/brightdata/linkedin/companies.rb

@@ -0,0 +1,32 @@
+# frozen_string_literal: true
+
+module BrightData
+  module LinkedIn
+    # `client.linkedin.companies` endpoint family for LinkedIn companies by URL.
+    #
+    # @example Trigger an async collection
+    #   snapshot = client.linkedin.companies.trigger(urls: ["https://www.linkedin.com/company/example/"])
+    # @example Scrape synchronously
+    #   companies = client.linkedin.companies.scrape(urls: ["https://www.linkedin.com/company/example/"])
+    #
+    # @!method trigger(urls:)
+    #   @param urls [Array<String>] LinkedIn company URLs
+    #   @return [BrightData::Snapshot]
+    #   @raise [BrightData::ArgumentError] if `urls` is not an Array
+    # @!method scrape(urls:)
+    #   @param urls [Array<String>] LinkedIn company URLs
+    #   @return [Array<BrightData::LinkedIn::Types::Company>]
+    #   @raise [BrightData::ArgumentError] if `urls` is not an Array
+    #   @raise [BrightData::ScrapeTimeoutError] when results exceed Bright Data's synchronous cap
+    class Companies
+      include Endpoint
+
+      endpoint(
+        dataset_key: :companies_collect_by_url,
+        input: Types::CompanyUrlInput,
+        result: Types::Company,
+        param: :urls
+      )
+    end
+  end
+end