urlpipe 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: be7381c8419e7711e1e5506df18decd8adf8a9174678a4bf6b14a30343cffe47
+  data.tar.gz: 496d714f333fb7d6a983e943a531a330b8b185064eae7ab3d933c10a19098fdf
+SHA512:
+  metadata.gz: 54769706ad4bea7fddb97c0190793713011e3dc3d71023ec90d24ee63ea2810f0430120b037d73ed33edab04a3ae7bb8809dca722120c17083e3246f39842128
+  data.tar.gz: 1e8522f8eb99c9ffc2e982ef34412a738d9c5f53dcb0a25ab1e0e906be135567c60cf4913fce9cbf831c36f8ae41fbf9487b59a07cc8978e60e3f85f6d25fb58

data/CHANGELOG.md

@@ -0,0 +1,13 @@
+# Changelog
+
+## 0.1.0
+
+The first release.
+
+- `Urlpipe::Client` with one method per operation: `markdown`, `html`, `summarize`, `screenshot`, `meta`, `keywords`, `console`, `lighthouse` and `scrape`, plus `result` and `wait` for tokens.
+- Calls are synchronous by default. An analysis that outlives the API's 60-second sync window is polled until it lands.
+- `Urlpipe::Response` with the result typed per operation, the token, labels and the parsed metadata headers.
+- Screenshots decoded to bytes, with the MIME type and the key-free `result_url`.
+- Automatic retries with an `Idempotency-Key` per call, so a retry never runs or bills the work twice.
+- A typed error for every documented API error.
+- `Urlpipe::Webhook.verify` for signed webhook deliveries.

data/LICENSE

@@ -0,0 +1,21 @@
+MIT License
+
+Copyright (c) 2026 Aliat Partner S.L.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,245 @@
+# urlpipe
+
+Turn any URL into clean Markdown, rendered HTML, a full-page screenshot, metadata, a summary, keywords, console errors or a Lighthouse audit, from Ruby. Pages are rendered in real Chrome, so JavaScript-heavy sites come back complete.
+
+This is the official Ruby client for [URLpipe](https://urlpipe.dev). It has no runtime dependencies and runs on Ruby 3.1 and later.
+
+## Install
+
+```sh
+bundle add urlpipe
+```
+
+or `gem install urlpipe`.
+
+## Quickstart
+
+Get an API key from a project in the [dashboard](https://urlpipe.dev) (the Free plan includes 1,000 credits a month, no card) and put it in `URLPIPE_API_KEY`:
+
+```ruby
+require "urlpipe"
+
+client = Urlpipe::Client.new # reads ENV["URLPIPE_API_KEY"]
+
+response = client.markdown("https://example.com")
+puts response.data           # "# Example Domain\n\n..."
+response.meta.quota.remaining # => 943
+```
+
+Every call waits for the result and hands it back in `response.data`. Cached results and failed calls cost nothing.
+
+## The client
+
+```ruby
+Urlpipe::Client.new(
+  api_key: nil,          # falls back to ENV["URLPIPE_API_KEY"]; raises Urlpipe::ConfigurationError if neither is set
+  base_url: "https://urlpipe.dev",
+  timeout: 90,           # seconds per HTTP request; the API holds a sync request up to 60 s
+  max_retries: 2,        # see "Retries and idempotency"
+  wait_timeout: 300,     # how long a long analysis is polled before Urlpipe::WaitTimeoutError
+  poll_interval: 2       # seconds between polls of GET /result/:token
+)
+```
+
+A client holds no connection state, so one instance can be shared across threads.
+
+## Operations
+
+Every method takes the URL first. These options work on all of them and are sent only when you give them:
+
+| Option | What it does |
+| --- | --- |
+| `sync:` | `true` by default: the call returns the result. `false` returns straight away with a token (see [Async](#async-and-wait)). |
+| `max_age:` | How fresh a cached result must be: seconds (`3600`) or a duration (`"3 days"`). `0` skips the cache. |
+| `labels:` | Your own ids for the request, e.g. `{ client: "acme" }`. Returned with the result and the webhook. |
+| `residential:` | `true` fetches the page from a home broadband address. |
+| `report_to:` | A webhook URL the result is POSTed to (async calls). |
+| `page_options:` | `wait_for_selector`, `delay`, `block_ads`, `block_cookie_banners`, `remove_selectors`. Not on `lighthouse`. |
+| `idempotency_key:` | Sent as the `Idempotency-Key` header. |
+| `extra:` | A Hash merged into the request body, for API options this version doesn't name yet. |
+
+Option Hashes go to the API as given; it validates them and tells you what to change.
+
+### markdown, html, summarize
+
+`data` is a String.
+
+```ruby
+client.markdown("https://example.com", page_options: { block_cookie_banners: true }).data
+client.html("https://example.com").data      # the rendered document
+client.summarize("https://example.com").data # a Markdown summary
+```
+
+### screenshot
+
+`data` is a `Urlpipe::Screenshot`: the decoded image bytes, its `mime_type` (`image/png`, `image/jpeg` or `image/webp`) and `result_url`, a link to the image that needs no API key.
+
+```ruby
+shot = client.screenshot("https://example.com",
+                         screenshot_options: { viewport_width: 390, format: "webp" }).data
+shot.save("example.webp")
+shot.mime_type  # => "image/webp"
+shot.result_url # => "https://..." - put it straight in an <img> tag
+```
+
+### meta
+
+`data` is a Hash with `title`, `description`, `language`, `main_image_url`, `favicon_url`, `author_name`, `feed_url`, `publication_date` and `additional_author_information`. Any of them can be `nil`.
+
+```ruby
+client.meta("https://example.com").data["title"] # => "Example Domain"
+```
+
+### keywords
+
+`data` is an Array of Strings, most relevant first.
+
+```ruby
+client.keywords("https://example.com").data # => ["example domain", "documentation", ...]
+```
+
+### console
+
+`data` is an Array of `{"type" => "error" | "warning" | "exception", "text" => "..."}`.
+
+```ruby
+client.console("https://example.com").data.select { |entry| entry["type"] == "exception" }
+```
+
+### lighthouse
+
+`data` is the audit as a Hash. `device:` is `"mobile"` (the default) or `"desktop"`; `include_audits: true` adds the full audits object.
+
+```ruby
+report = client.lighthouse("https://example.com", device: "desktop").data
+report.dig("categories", "performance", "score")
+```
+
+### scrape
+
+Several operations off one page visit, so you get them all much sooner. `data` is `{"url" => ..., "operations" => {"markdown" => {"success", "result", "error", "cached"}, ...}}`. Each operation succeeds or fails on its own. A screenshot inside a scrape stays Base64, as the API returns it.
+
+```ruby
+page = client.scrape("https://example.com", %w[markdown meta screenshot],
+                     screenshot_options: { format: "jpeg" }).data
+page.dig("operations", "meta", "result", "title")
+```
+
+## The Response
+
+Every method returns a `Urlpipe::Response`:
+
+| Field | |
+| --- | --- |
+| `status` | `"completed"`, `"accepted"` (an async request was accepted) or `"processing"` (still running). Also `completed?`, `accepted?`, `processing?`. |
+| `data` | The result when completed, `nil` otherwise. |
+| `token` | The request's result token. Fetching the result again with it is free for 30 days. |
+| `labels` | The labels the request was made with; `{}` when it had none. |
+| `meta` | `cache` (`"hit"`, `"miss"`, `"partial"`), `cache_age`, `processing_time_ms`, `quota` (`cost`, `limit`, `remaining`, `overage`, `resets_at`), `concurrency_limit`, `result_url`, `idempotent_replayed`. |
+
+A value the API didn't send is `nil`; `idempotent_replayed` is always `true` or `false`. On an unlimited plan `quota.limit`, `quota.remaining` and `concurrency_limit` are the String `"unlimited"`.
+
+## Long analyses
+
+A sync request is held for up to 60 seconds. When an analysis takes longer (a Lighthouse audit, a slow site), the client polls `GET /result/:token` every 2 seconds and returns the result when it lands, so your code sees one call. After `wait_timeout` it raises `Urlpipe::WaitTimeoutError`, whose `token` you can collect later: the analysis keeps running.
+
+## Async and wait
+
+With `sync: false` the call returns at once with a token, and the work continues in the background. Collect it with `wait`, or with a webhook:
+
+```ruby
+accepted = client.lighthouse("https://example.com", sync: false)
+accepted.status # => "accepted"
+
+report = client.wait(accepted.token, operation: "lighthouse").data
+```
+
+`wait(token, operation: nil, timeout: wait_timeout, interval: poll_interval)` polls until the result is ready. `result(token, operation: nil)` checks once and returns a `"processing"` Response while it is still running.
+
+The API doesn't say which operation produced a token, so pass `operation:` to get `data` typed the way that method returns it. Without it JSON comes back parsed and text stays a String, which means a screenshot stays Base64: `client.wait(token, operation: "screenshot")` decodes it into a `Urlpipe::Screenshot`.
+
+## Webhooks
+
+Turn on webhook signing for the project (Settings → Webhook Signing) and verify each delivery before trusting it. `Urlpipe::Webhook.verify` needs no client:
+
+```ruby
+# app/controllers/urlpipe_webhooks_controller.rb
+class UrlpipeWebhooksController < ActionController::API
+  def create
+    payload = Urlpipe::Webhook.verify(request.raw_post, request.headers,
+                                      ENV.fetch("URLPIPE_WEBHOOK_SECRET"))
+    ProcessResultJob.perform_later(payload["token"])
+    head :ok
+  rescue Urlpipe::WebhookVerificationError
+    head :unauthorized
+  end
+end
+```
+
+`verify(raw_body, headers, secret, tolerance: 300)` returns the payload Hash (`token`, `operation`, `labels`, `success`, `result`, `result_url`, `error`, `meta`), or raises `Urlpipe::WebhookVerificationError` saying why. It accepts any signature in the header that matches, so deliveries keep verifying through a secret rotation, and it refuses a timestamp more than `tolerance` seconds from now.
+
+Pass the **raw** request body, exactly the bytes received: `request.raw_post` in Rails, `request.body.read` in Rack. Parsed and re-serialized JSON has different bytes and will not verify. `headers` can be Rails' `request.headers`, a Rack env or a plain Hash.
+
+Make the handler idempotent on `token`: a delivery can be retried.
+
+## Errors
+
+Everything raised is a `Urlpipe::Error`, with `status`, `code` (the API's error code, when there is one), `message`, `body` (parsed JSON, or the raw text) and `token`.
+
+| Error | When |
+| --- | --- |
+| `Urlpipe::AuthenticationError` | 401: the API key is missing or not an active project key. |
+| `Urlpipe::EmailUnverifiedError` | 403: confirm the email address on the account. |
+| `Urlpipe::InvalidRequestError` | 422 with a code: `invalid_url`, `invalid_max_age`, `invalid_options`, `invalid_labels`, `invalid_idempotency_key`, `idempotency_key_reused`, or a `report_to` the API won't deliver to. |
+| `Urlpipe::AnalysisFailedError` | 422: the page couldn't be analysed; the message says why. A scrape where every operation failed keeps the scrape on `body`. |
+| `Urlpipe::QuotaExceededError` | 429 on the Free plan: `limit`, `used`, `needed`, `resets_at`. |
+| `Urlpipe::ConcurrencyLimitError` | 429: `limit` requests already `running`. |
+| `Urlpipe::RateLimitedError` | 429: sending too fast; `retry_after` seconds. |
+| `Urlpipe::NotFoundError` | 404: no result for this token in this project. |
+| `Urlpipe::StaleResultError` | 410: the result is past the 30-day window. |
+| `Urlpipe::ServerError` | Any other 5xx. |
+| `Urlpipe::ConnectionError` | The API couldn't be reached, or a request outlived `timeout`. |
+| `Urlpipe::WaitTimeoutError` | `wait_timeout` passed; `token` is still collectable. |
+| `Urlpipe::ConfigurationError` | No API key. |
+| `Urlpipe::WebhookVerificationError` | A webhook delivery didn't verify. |
+
+```ruby
+begin
+  client.markdown(url)
+rescue Urlpipe::AnalysisFailedError => e
+  logger.info("Skipped #{url}: #{e.message}") # "The requested page was not found."
+rescue Urlpipe::QuotaExceededError => e
+  retry_at(e.resets_at)
+end
+```
+
+## Retries and idempotency
+
+The client retries, up to `max_retries` times, what a retry can fix: connection errors, 500/502/503, `rate_limited` (after `Retry-After`, capped at 60 s) and `concurrency_limit` (after 1 s, 2 s, ...). It never retries 401, 403, 404, 410, 422 or `quota_exceeded`.
+
+Every analysis request that may be retried carries an `Idempotency-Key`: yours if you pass `idempotency_key:`, otherwise a fresh UUID for that call, reused on each of its retries. The API answers a repeated key with the first request's token and result, so a retry after a dropped connection never runs or bills the work twice. Pass your own key (a job id, say) to get the same guarantee across process restarts:
+
+```ruby
+client.markdown(url, idempotency_key: "import-#{job.id}")
+```
+
+Set `max_retries: 0` to turn retries off; no key is generated then.
+
+## Links
+
+- API docs: https://urlpipe.dev/docs
+- Pricing: https://urlpipe.dev/pricing
+- MCP server, to give an AI assistant the same tools: https://github.com/URLpipe/mcp
+
+## Development
+
+```sh
+bundle install
+bundle exec rake test
+```
+
+The tests run against a local stub server and never touch the network. Set `URLPIPE_API_KEY` to also run one live smoke test against the real API.
+
+## License
+
+MIT. Copyright (c) 2026 Aliat Partner S.L.

data/lib/urlpipe/client.rb

@@ -0,0 +1,347 @@
+# frozen_string_literal: true
+
+require "json"
+require "net/http"
+require "openssl"
+require "securerandom"
+require "uri"
+
+module Urlpipe
+  # A client for the URLpipe API. Every analysis method takes the URL first
+  # and returns a Response. Calls are synchronous by default: the method
+  # returns once the result is ready, however long the analysis takes.
+  #
+  #   client = Urlpipe::Client.new(api_key: "...")
+  #   client.markdown("https://example.com").data # => "# Example Domain..."
+  class Client
+    DEFAULT_BASE_URL = "https://urlpipe.dev"
+    USER_AGENT = "urlpipe-ruby/#{VERSION}".freeze
+
+    TEXT_OPERATIONS = %w[markdown html summarize].freeze
+    JSON_OPERATIONS = %w[meta keywords console lighthouse scrape].freeze
+
+    RETRYABLE_STATUSES = [500, 502, 503].freeze
+    MAX_RETRY_DELAY = 60
+
+    NETWORK_ERRORS = [
+      SocketError, SystemCallError, IOError, Timeout::Error,
+      Net::OpenTimeout, Net::ReadTimeout, Net::WriteTimeout,
+      Net::HTTPBadResponse, OpenSSL::SSL::SSLError
+    ].freeze
+
+    # The raw HTTP exchange, before it becomes a Response or an error.
+    RawResponse = Struct.new(:status, :headers, :text, keyword_init: true) do
+      def json? = headers["content-type"].to_s.include?("json")
+    end
+    private_constant :RawResponse
+
+    attr_reader :base_url, :timeout, :max_retries, :wait_timeout, :poll_interval
+
+    # - +api_key+: a project API key; falls back to ENV["URLPIPE_API_KEY"]
+    # - +timeout+: seconds per HTTP request (the API holds a sync request up
+    #   to 60 s, so keep it above that)
+    # - +max_retries+: automatic retries on connection errors, 500/502/503,
+    #   rate_limited and concurrency_limit
+    # - +wait_timeout+: how long a sync call or +wait+ keeps polling a long
+    #   analysis before raising WaitTimeoutError
+    # - +poll_interval+: seconds between polls of GET /result/:token
+    def initialize(api_key: nil, base_url: DEFAULT_BASE_URL, timeout: 90, max_retries: 2,
+                   wait_timeout: 300, poll_interval: 2)
+      api_key = ENV.fetch("URLPIPE_API_KEY", nil) if api_key.nil? || api_key.to_s.strip.empty?
+      if api_key.nil? || api_key.strip.empty?
+        raise ConfigurationError,
+              "Pass api_key: to Urlpipe::Client.new, or set URLPIPE_API_KEY. " \
+              "Each project's API key is in the URLpipe dashboard."
+      end
+
+      @api_key = api_key.strip
+      @base_url = base_url.to_s.sub(%r{/+\z}, "")
+      @base_uri = URI(@base_url)
+      @timeout = timeout
+      @max_retries = Integer(max_retries)
+      @wait_timeout = wait_timeout
+      @poll_interval = poll_interval
+    end
+
+    def inspect
+      "#<#{self.class.name} base_url=#{@base_url.inspect} timeout=#{@timeout} " \
+        "max_retries=#{@max_retries} wait_timeout=#{@wait_timeout}>"
+    end
+
+    # The page as Markdown. +data+ is a String.
+    def markdown(url, sync: true, max_age: nil, labels: nil, residential: nil, report_to: nil,
+                 page_options: nil, idempotency_key: nil, extra: {})
+      analyse("markdown", url, { sync:, max_age:, labels:, residential:, report_to:, page_options: },
+              idempotency_key, extra)
+    end
+
+    # The rendered HTML document. +data+ is a String.
+    def html(url, sync: true, max_age: nil, labels: nil, residential: nil, report_to: nil,
+             page_options: nil, idempotency_key: nil, extra: {})
+      analyse("html", url, { sync:, max_age:, labels:, residential:, report_to:, page_options: },
+              idempotency_key, extra)
+    end
+
+    # A summary of the page, as Markdown. +data+ is a String.
+    def summarize(url, sync: true, max_age: nil, labels: nil, residential: nil, report_to: nil,
+                  page_options: nil, idempotency_key: nil, extra: {})
+      analyse("summarize", url, { sync:, max_age:, labels:, residential:, report_to:, page_options: },
+              idempotency_key, extra)
+    end
+
+    # A full-page screenshot. +data+ is a Screenshot (decoded bytes,
+    # +mime_type+, +result_url+).
+    def screenshot(url, sync: true, max_age: nil, labels: nil, residential: nil, report_to: nil,
+                   page_options: nil, screenshot_options: nil, idempotency_key: nil, extra: {})
+      analyse("screenshot", url,
+              { sync:, max_age:, labels:, residential:, report_to:, page_options:, screenshot_options: },
+              idempotency_key, extra)
+    end
+
+    # Page metadata. +data+ is a Hash: title, description, language,
+    # main_image_url, favicon_url, author_name, feed_url, publication_date,
+    # additional_author_information.
+    def meta(url, sync: true, max_age: nil, labels: nil, residential: nil, report_to: nil,
+             page_options: nil, idempotency_key: nil, extra: {})
+      analyse("meta", url, { sync:, max_age:, labels:, residential:, report_to:, page_options: },
+              idempotency_key, extra)
+    end
+
+    # The page's keywords, most relevant first. +data+ is an Array of Strings.
+    def keywords(url, sync: true, max_age: nil, labels: nil, residential: nil, report_to: nil,
+                 page_options: nil, idempotency_key: nil, extra: {})
+      analyse("keywords", url, { sync:, max_age:, labels:, residential:, report_to:, page_options: },
+              idempotency_key, extra)
+    end
+
+    # Console errors, warnings and uncaught exceptions. +data+ is an Array of
+    # Hashes with "type" ("error", "warning" or "exception") and "text".
+    def console(url, sync: true, max_age: nil, labels: nil, residential: nil, report_to: nil,
+                page_options: nil, idempotency_key: nil, extra: {})
+      analyse("console", url, { sync:, max_age:, labels:, residential:, report_to:, page_options: },
+              idempotency_key, extra)
+    end
+
+    # A Lighthouse audit. +data+ is a Hash. +device+ is "mobile" (the API's
+    # default) or "desktop".
+    def lighthouse(url, sync: true, max_age: nil, labels: nil, residential: nil, report_to: nil,
+                   device: nil, include_audits: nil, idempotency_key: nil, extra: {})
+      analyse("lighthouse", url, { sync:, max_age:, labels:, residential:, report_to:, device:, include_audits: },
+              idempotency_key, extra)
+    end
+
+    # Several operations off one page visit. +operations+ is an Array such as
+    # ["markdown", "meta"]. +data+ is a Hash:
+    # {"url" => ..., "operations" => {"markdown" => {"success", "result", "error", "cached"}}}.
+    # A screenshot inside a scrape stays Base64, as the API returns it.
+    def scrape(url, operations, sync: true, max_age: nil, labels: nil, residential: nil, report_to: nil,
+               page_options: nil, device: nil, include_audits: nil, screenshot_options: nil,
+               idempotency_key: nil, extra: {})
+      analyse("scrape", url,
+              { operations:, sync:, max_age:, labels:, residential:, report_to:, page_options:,
+                device:, include_audits:, screenshot_options: },
+              idempotency_key, extra)
+    end
+
+    # One GET /result/:token. Returns a completed Response (200) or a
+    # processing one (202); raises for everything else.
+    #
+    # +operation+ names the operation that produced the token, so +data+ is
+    # typed as that operation returns it. Without it, JSON stays parsed JSON
+    # and text stays a String (a screenshot stays Base64: pass
+    # <tt>operation: "screenshot"</tt> to get a Screenshot).
+    def result(token, operation: nil)
+      raw = execute(:get, "/result/#{URI.encode_www_form_component(token)}")
+
+      case raw.status
+      when 200 then completed(raw, operation&.to_s)
+      when 202 then pending(raw, "processing", fallback_token: token)
+      else
+        if processing_timeout?(raw)
+          pending(raw, "processing", fallback_token: token)
+        else
+          raise error_for(raw)
+        end
+      end
+    end
+
+    # Polls GET /result/:token every +interval+ seconds until the result is
+    # ready, then returns the completed Response. Raises WaitTimeoutError
+    # after +timeout+ seconds, or the API's error if the analysis failed.
+    def wait(token, operation: nil, timeout: @wait_timeout, interval: @poll_interval)
+      deadline = monotonic_now + timeout
+
+      loop do
+        response = result(token, operation: operation)
+        return response if response.completed?
+
+        if monotonic_now + interval > deadline
+          raise WaitTimeoutError.new(
+            "The result for #{token} was not ready within #{timeout} seconds. " \
+            "It keeps running: collect it later with client.wait(#{token.inspect}).",
+            token: token
+          )
+        end
+
+        sleep(interval)
+      end
+    end
+
+    private
+
+    def analyse(operation, url, options, idempotency_key, extra)
+      body = { "url" => url }
+      options.each { |key, value| body[key.to_s] = value unless value.nil? }
+      (extra || {}).each { |key, value| body[key.to_s] = value }
+
+      idempotency_key ||= SecureRandom.uuid if @max_retries.positive?
+      raw = execute(:post, "/#{operation}", body: body, idempotency_key: idempotency_key)
+
+      if raw.status == 200
+        sync?(body["sync"]) ? completed(raw, operation) : pending(raw, "accepted")
+      elsif processing_timeout?(raw) && (token = json_body(raw)["token"])
+        wait(token, operation: operation)
+      else
+        raise error_for(raw)
+      end
+    end
+
+    def sync?(value) = [true, "true"].include?(value)
+
+    def completed(raw, operation)
+      Response.new(
+        status: "completed",
+        data: typed_data(raw, operation),
+        token: raw.headers["x-result-token"],
+        labels: header_labels(raw) || {},
+        meta: Meta.from_headers(raw.headers)
+      )
+    end
+
+    def pending(raw, status, fallback_token: nil)
+      body = json_body(raw)
+      Response.new(
+        status: status,
+        data: nil,
+        token: raw.headers["x-result-token"] || body["token"] || fallback_token,
+        labels: header_labels(raw) || (body["labels"].is_a?(Hash) ? body["labels"] : {}),
+        meta: Meta.from_headers(raw.headers)
+      )
+    end
+
+    def typed_data(raw, operation)
+      if TEXT_OPERATIONS.include?(operation)
+        raw.text
+      elsif operation == "screenshot"
+        Screenshot.from_base64(raw.text, result_url: raw.headers["x-result-url"])
+      elsif JSON_OPERATIONS.include?(operation) || (operation.nil? && raw.json?)
+        parse_result_json(raw)
+      else
+        raw.text
+      end
+    end
+
+    def parse_result_json(raw)
+      JSON.parse(raw.text)
+    rescue JSON::ParserError => e
+      raise Error.new("The API answered #{raw.status} with a body that is not valid JSON: #{e.message}",
+                      status: raw.status, body: raw.text)
+    end
+
+    def header_labels(raw)
+      value = raw.headers["x-labels"]
+      return nil if value.nil? || value.strip.empty?
+
+      JSON.parse(value)
+    rescue JSON::ParserError
+      nil
+    end
+
+    def json_body(raw)
+      parsed = ErrorFactory.parse_json(raw.text)
+      parsed.is_a?(Hash) ? parsed : {}
+    end
+
+    def processing_timeout?(raw)
+      raw.status == 504 && json_body(raw)["error"] == "processing_timeout"
+    end
+
+    def error_for(raw)
+      ErrorFactory.build(status: raw.status, text: raw.text, headers: raw.headers)
+    end
+
+    # Sends the request, retrying what is safe to retry. The same
+    # Idempotency-Key goes out on every attempt, so a retry can never run or
+    # bill the work twice.
+    def execute(method, path, body: nil, idempotency_key: nil)
+      attempt = 0
+
+      loop do
+        begin
+          raw = perform(method, path, body, idempotency_key)
+        rescue ConnectionError
+          raise if attempt >= @max_retries
+
+          sleep(backoff(attempt))
+          attempt += 1
+          next
+        end
+
+        delay = retry_delay(raw, attempt)
+        return raw if delay.nil? || attempt >= @max_retries
+
+        sleep(delay)
+        attempt += 1
+      end
+    end
+
+    def retry_delay(raw, attempt)
+      return backoff(attempt) if RETRYABLE_STATUSES.include?(raw.status)
+      return nil unless raw.status == 429
+
+      case json_body(raw)["error"]
+      when "rate_limited"
+        retry_after = ErrorFactory.retry_after(raw.headers, json_body(raw))
+        [retry_after || 1, MAX_RETRY_DELAY].min
+      when "concurrency_limit"
+        backoff(attempt)
+      end
+    end
+
+    def backoff(attempt) = [2**attempt, MAX_RETRY_DELAY].min
+
+    def perform(method, path, body, idempotency_key)
+      uri = @base_uri.dup
+      uri.path = "#{@base_uri.path}#{path}"
+
+      request = method == :post ? Net::HTTP::Post.new(uri) : Net::HTTP::Get.new(uri)
+      request["Authorization"] = "Bearer #{@api_key}"
+      request["User-Agent"] = USER_AGENT
+      request["Accept"] = "*/*"
+      request["Idempotency-Key"] = idempotency_key if idempotency_key
+      if body
+        request["Content-Type"] = "application/json"
+        request.body = JSON.generate(body)
+      end
+
+      response = http_for(uri).start { |http| http.request(request) }
+      headers = response.each_header.to_h { |name, value| [name.downcase, value] }
+      text = (response.body || +"").dup.force_encoding(Encoding::UTF_8)
+      RawResponse.new(status: response.code.to_i, headers: headers, text: text)
+    rescue *NETWORK_ERRORS => e
+      raise ConnectionError, "Could not reach #{@base_url}: #{e.class}: #{e.message}"
+    end
+
+    def http_for(uri)
+      http = Net::HTTP.new(uri.host, uri.port)
+      http.use_ssl = uri.scheme == "https"
+      http.open_timeout = @timeout
+      http.read_timeout = @timeout
+      http.write_timeout = @timeout
+      http.max_retries = 0
+      http
+    end
+
+    def monotonic_now = Process.clock_gettime(Process::CLOCK_MONOTONIC)
+  end
+end

data/lib/urlpipe/errors.rb

@@ -0,0 +1,205 @@
+# frozen_string_literal: true
+
+require "json"
+
+module Urlpipe
+  # Base class of every error this library raises.
+  #
+  # - +status+: the HTTP status, when the error came from a response
+  # - +code+: the body's +error+ value when it is a machine-readable code
+  #   (+"invalid_url"+, +"quota_exceeded"+, ...)
+  # - +body+: the parsed JSON body, or the raw text when it was not JSON
+  # - +token+: the result token, when the body carried one
+  class Error < StandardError
+    attr_reader :status, :code, :body, :token
+
+    def initialize(message = nil, status: nil, code: nil, body: nil, token: nil)
+      super(message)
+      @status = status
+      @code = code
+      @body = body
+      @token = token
+    end
+  end
+
+  # No API key was given and URLPIPE_API_KEY is not set.
+  class ConfigurationError < Error; end
+
+  # 401: the API key is missing or not an active project key.
+  class AuthenticationError < Error; end
+
+  # 403 email_unverified: the key is valid, but the account's email address
+  # has not been confirmed yet.
+  class EmailUnverifiedError < Error; end
+
+  # 422 with a parameter code: invalid_url, invalid_max_age, invalid_options,
+  # invalid_labels, invalid_idempotency_key, idempotency_key_reused, or a
+  # report_to the API will not deliver to.
+  class InvalidRequestError < Error; end
+
+  # 422 whose error is a sentence: the page could not be analysed. The
+  # sentence is the message. A /scrape where every operation failed keeps the
+  # whole scrape object on +body+.
+  class AnalysisFailedError < Error; end
+
+  # 429 quota_exceeded (Free plan only): the month's credits are spent.
+  class QuotaExceededError < Error
+    attr_reader :limit, :used, :needed, :resets_at
+
+    def initialize(message = nil, limit: nil, used: nil, needed: nil, resets_at: nil, **rest)
+      super(message, **rest)
+      @limit = limit
+      @used = used
+      @needed = needed
+      @resets_at = resets_at
+    end
+  end
+
+  # 429 concurrency_limit: too many of your requests are already running.
+  class ConcurrencyLimitError < Error
+    attr_reader :limit, :running
+
+    def initialize(message = nil, limit: nil, running: nil, **rest)
+      super(message, **rest)
+      @limit = limit
+      @running = running
+    end
+  end
+
+  # 429 rate_limited: sending too fast. +retry_after+ is in seconds.
+  class RateLimitedError < Error
+    attr_reader :retry_after
+
+    def initialize(message = nil, retry_after: nil, **rest)
+      super(message, **rest)
+      @retry_after = retry_after
+    end
+  end
+
+  # 404 not_found: no result for this token under your project.
+  class NotFoundError < Error; end
+
+  # 410 stale: the result is older than the 30-day retention window.
+  class StaleResultError < Error; end
+
+  # A 5xx response.
+  class ServerError < Error; end
+
+  # The API could not be reached: DNS, refused or dropped connection, TLS,
+  # or a request that outlived +timeout+.
+  class ConnectionError < Error; end
+
+  # +wait+ (or a sync call that timed out on the server) ran out of
+  # +wait_timeout+. The analysis may still finish: keep +token+.
+  class WaitTimeoutError < Error; end
+
+  # A webhook delivery failed signature verification. The message says why.
+  class WebhookVerificationError < Error; end
+
+  # Builds the right error from an HTTP error response.
+  module ErrorFactory
+    CODE_PATTERN = /\A[a-z][a-z0-9_]*\z/
+    INVALID_REQUEST_CODES = %w[
+      invalid_url invalid_max_age invalid_options invalid_labels
+      invalid_idempotency_key idempotency_key_reused
+    ].freeze
+
+    DEFAULT_MESSAGES = {
+      AuthenticationError => "The API key is missing or not an active project key. " \
+                             "Copy your project's key from the dashboard.",
+      EmailUnverifiedError => "Confirm the email address on this account before using the API. " \
+                              "The link is in the email we sent when you signed up.",
+      NotFoundError => "No result for this token under this project.",
+      StaleResultError => "This result is older than the 30-day retention window. Run a new analysis."
+    }.freeze
+
+    module_function
+
+    def build(status:, text:, headers: {})
+      parsed = parse_json(text)
+      hash = parsed.is_a?(Hash) ? parsed : {}
+      error_value = hash["error"].is_a?(String) ? hash["error"] : nil
+      code = error_value if error_value&.match?(CODE_PATTERN)
+
+      klass, extra = classify(status, code, error_value, hash, headers)
+      message = message_for(klass, hash, error_value, code, status, text)
+
+      klass.new(
+        message,
+        status: status,
+        code: code,
+        body: parsed.nil? ? text : parsed,
+        token: hash["token"].is_a?(String) ? hash["token"] : headers["x-result-token"],
+        **extra
+      )
+    end
+
+    def parse_json(text)
+      return nil if text.nil? || text.strip.empty?
+
+      JSON.parse(text)
+    rescue JSON::ParserError
+      nil
+    end
+
+    def classify(status, code, error_value, hash, headers)
+      case status
+      when 401 then [AuthenticationError, {}]
+      when 403 then [code == "email_unverified" ? EmailUnverifiedError : Error, {}]
+      when 404 then [NotFoundError, {}]
+      when 410 then [StaleResultError, {}]
+      when 422
+        [invalid_request?(error_value) ? InvalidRequestError : AnalysisFailedError, {}]
+      when 429 then classify_429(code, hash, headers)
+      when 500..599 then [ServerError, {}]
+      else [Error, {}]
+      end
+    end
+
+    def classify_429(code, hash, headers)
+      case code
+      when "quota_exceeded"
+        [QuotaExceededError,
+         { limit: hash["limit"], used: hash["used"], needed: hash["needed"], resets_at: hash["resets_at"] }]
+      when "concurrency_limit"
+        [ConcurrencyLimitError, { limit: hash["limit"], running: hash["running"] }]
+      when "rate_limited"
+        [RateLimitedError, { retry_after: retry_after(headers, hash) }]
+      else
+        [Error, {}]
+      end
+    end
+
+    def invalid_request?(error_value)
+      return false unless error_value
+
+      INVALID_REQUEST_CODES.include?(error_value) || error_value.start_with?("report_to")
+    end
+
+    def retry_after(headers, hash)
+      value = headers["retry-after"]
+      return Integer(value, 10) if value.is_a?(String) && value.strip.match?(/\A\d+\z/)
+
+      hash["retry_after"].is_a?(Integer) ? hash["retry_after"] : nil
+    end
+
+    def message_for(klass, hash, error_value, code, status, text)
+      return hash["message"] if hash["message"].is_a?(String) && !hash["message"].empty?
+      return error_value if error_value && !error_value.empty? && code.nil?
+      return scrape_failure_message(hash) if klass == AnalysisFailedError && hash["operations"].is_a?(Hash)
+      return DEFAULT_MESSAGES[klass] if DEFAULT_MESSAGES.key?(klass)
+      return "The API refused the request with HTTP #{status} (#{code})." if code
+
+      snippet = text.to_s.strip[0, 200]
+      snippet.empty? ? "HTTP #{status}" : "HTTP #{status}: #{snippet}"
+    end
+
+    def scrape_failure_message(hash)
+      details = hash["operations"].map do |name, entry|
+        error = entry.is_a?(Hash) ? entry["error"] : nil
+        error ? "#{name}: #{error}" : name
+      end
+      "Every operation failed: #{details.join("; ")}"
+    end
+  end
+end

data/lib/urlpipe/response.rb

@@ -0,0 +1,68 @@
+# frozen_string_literal: true
+
+require "json"
+
+module Urlpipe
+  # What an analysis cost and what is left of the month's allowance.
+  # +limit+ and +remaining+ are Integers, or the String "unlimited".
+  Quota = Struct.new(:cost, :limit, :remaining, :overage, :resets_at, keyword_init: true)
+
+  # The response's metadata headers, parsed. A header that was not sent is
+  # +nil+ (+idempotent_replayed+ is always true or false).
+  #
+  # - +cache+: "hit", "miss" or (for /scrape) "partial"
+  # - +cache_age+: seconds, when served from cache
+  # - +processing_time_ms+: once the work has finished
+  # - +quota+: a Quota
+  # - +concurrency_limit+: Integer or "unlimited"
+  # - +result_url+: screenshots only, a link to the image that needs no key
+  # - +idempotent_replayed+: +true+ when this answers an earlier request with
+  #   the same Idempotency-Key, +false+ otherwise
+  Meta = Struct.new(
+    :cache, :cache_age, :processing_time_ms, :quota, :concurrency_limit,
+    :result_url, :idempotent_replayed,
+    keyword_init: true
+  ) do
+    # +headers+ is a Hash of lowercase header names to values.
+    def self.from_headers(headers)
+      new(
+        cache: headers["x-cache"],
+        cache_age: integer_or_nil(headers["x-cache-age"]),
+        processing_time_ms: integer_or_nil(headers["x-processing-time-ms"]),
+        quota: Quota.new(
+          cost: integer_or_given(headers["x-quota-cost"]),
+          limit: integer_or_given(headers["x-quota-limit"]),
+          remaining: integer_or_given(headers["x-quota-remaining"]),
+          overage: integer_or_given(headers["x-quota-overage"]),
+          resets_at: headers["x-quota-reset"]
+        ),
+        concurrency_limit: integer_or_given(headers["x-concurrency-limit"]),
+        result_url: headers["x-result-url"],
+        idempotent_replayed: headers["idempotent-replayed"].to_s.strip.casecmp?("true")
+      )
+    end
+
+    def self.integer_or_nil(value)
+      value = value&.strip
+      value&.match?(/\A-?\d+\z/) ? Integer(value, 10) : nil
+    end
+
+    def self.integer_or_given(value)
+      integer_or_nil(value) || value&.strip
+    end
+  end
+
+  # What every method returns.
+  #
+  # - +status+: "completed", "accepted" (an async request was accepted) or
+  #   "processing" (GET /result/:token is still working on it)
+  # - +data+: the result when completed, +nil+ otherwise
+  # - +token+: the request's result token
+  # - +labels+: the labels the request was made with; {} when it had none
+  # - +meta+: a Meta
+  Response = Struct.new(:status, :data, :token, :labels, :meta, keyword_init: true) do
+    def completed? = status == "completed"
+    def accepted? = status == "accepted"
+    def processing? = status == "processing"
+  end
+end

data/lib/urlpipe/screenshot.rb

@@ -0,0 +1,45 @@
+# frozen_string_literal: true
+
+module Urlpipe
+  # A decoded screenshot: +data+ is the image as a binary String,
+  # +mime_type+ is "image/png", "image/jpeg" or "image/webp", and
+  # +result_url+ is a link to the same image that needs no API key.
+  class Screenshot
+    attr_reader :data, :mime_type, :result_url
+
+    # Builds a Screenshot from the API's Base64 body.
+    def self.from_base64(text, result_url: nil)
+      new(text.to_s.unpack1("m"), result_url: result_url)
+    end
+
+    def self.detect_mime_type(bytes)
+      if bytes.start_with?("\x89PNG\r\n\x1A\n".b)
+        "image/png"
+      elsif bytes.start_with?("\xFF\xD8\xFF".b)
+        "image/jpeg"
+      elsif bytes.byteslice(0, 4) == "RIFF".b && bytes.byteslice(8, 4) == "WEBP".b
+        "image/webp"
+      else
+        "image/png"
+      end
+    end
+
+    def initialize(data, result_url: nil, mime_type: nil)
+      @data = data.b
+      @mime_type = mime_type || self.class.detect_mime_type(@data)
+      @result_url = result_url
+    end
+
+    # Writes the image to +path+ and returns the path.
+    def save(path)
+      File.binwrite(path, data)
+      path
+    end
+
+    def bytesize = data.bytesize
+
+    def inspect
+      "#<#{self.class.name} #{mime_type} #{bytesize} bytes result_url=#{result_url.inspect}>"
+    end
+  end
+end

data/lib/urlpipe/version.rb

@@ -0,0 +1,5 @@
+# frozen_string_literal: true
+
+module Urlpipe
+  VERSION = "0.1.0"
+end

data/lib/urlpipe/webhook.rb

@@ -0,0 +1,118 @@
+# frozen_string_literal: true
+
+require "json"
+require "openssl"
+
+module Urlpipe
+  # Verifies signed webhook deliveries. No Client needed.
+  #
+  #   # Rails
+  #   payload = Urlpipe::Webhook.verify(request.raw_post, request.headers,
+  #                                     ENV["URLPIPE_WEBHOOK_SECRET"])
+  #   payload["token"] # => "0Zx3...9aQ"
+  #
+  # Give it the RAW request body, exactly the bytes received. Parsed and
+  # re-serialized JSON has different bytes, and the signature will not match.
+  module Webhook
+    TIMESTAMP_HEADER = "X-URLpipe-Timestamp"
+    SIGNATURE_HEADER = "X-URLpipe-Signature"
+    DEFAULT_TOLERANCE = 300
+
+    module_function
+
+    # Checks the delivery's signature and freshness and returns the parsed
+    # payload, a Hash with "token", "operation", "labels", "success",
+    # "result", "result_url", "error" and "meta".
+    #
+    # - +raw_body+: the request body as received (a String, or an IO)
+    # - +headers+: the request headers. Accepts a plain Hash, Rails'
+    #   +request.headers+, a Rack env (+HTTP_X_URLPIPE_SIGNATURE+) or any
+    #   object with +[]+; names are matched case-insensitively.
+    # - +secret+: the project's signing secret (+whsec_...+), used as-is
+    # - +tolerance+: seconds a timestamp may be off from now, either way
+    #
+    # Raises WebhookVerificationError with the reason when the delivery
+    # does not verify.
+    def verify(raw_body, headers, secret, tolerance: DEFAULT_TOLERANCE)
+      raise WebhookVerificationError, "No webhook signing secret was given." if secret.to_s.empty?
+
+      body = read_body(raw_body)
+      timestamp = header(headers, TIMESTAMP_HEADER).to_s.strip
+      signature = header(headers, SIGNATURE_HEADER).to_s.strip
+
+      raise WebhookVerificationError, "The #{SIGNATURE_HEADER} header is missing." if signature.empty?
+      raise WebhookVerificationError, "The #{TIMESTAMP_HEADER} header is missing." if timestamp.empty?
+      unless timestamp.match?(/\A\d+\z/)
+        raise WebhookVerificationError, "The #{TIMESTAMP_HEADER} header is not a Unix timestamp."
+      end
+
+      age = Time.now.to_i - Integer(timestamp, 10)
+      if age.abs > tolerance
+        raise WebhookVerificationError,
+              "The delivery's timestamp is #{age.abs} seconds #{age.negative? ? "ahead" : "old"}, " \
+              "outside the #{tolerance}-second tolerance."
+      end
+
+      expected = OpenSSL::HMAC.hexdigest("SHA256", secret.to_s, "#{timestamp}.".b + body)
+      unless candidates(signature).any? { |candidate| OpenSSL.secure_compare(candidate, expected) }
+        raise WebhookVerificationError, "No v1 signature in #{SIGNATURE_HEADER} matches this body and secret."
+      end
+
+      parse(body)
+    end
+
+    def read_body(raw_body)
+      body = raw_body.respond_to?(:read) ? raw_body.read.tap { raw_body.rewind if raw_body.respond_to?(:rewind) } : raw_body
+      body.to_s.b
+    end
+
+    # The v1 values of a "v1=<hex>,v1=<hex>" header; other schemes are ignored.
+    def candidates(signature)
+      signature.split(",").filter_map do |part|
+        scheme, value = part.strip.split("=", 2)
+        value.strip.downcase if scheme == "v1" && value
+      end
+    end
+
+    def parse(body)
+      payload = JSON.parse(body.dup.force_encoding(Encoding::UTF_8))
+      raise WebhookVerificationError, "The delivery body is not a JSON object." unless payload.is_a?(Hash)
+
+      payload
+    rescue JSON::ParserError => e
+      raise WebhookVerificationError, "The delivery body is not valid JSON: #{e.message}"
+    end
+
+    def header(headers, name)
+      normalized = normalize(name)
+
+      if headers.respond_to?(:[])
+        [name, name.downcase, "HTTP_#{normalized}"].each do |key|
+          value = safe_lookup(headers, key)
+          return header_value(value) unless value.nil?
+        end
+      end
+
+      return nil unless headers.respond_to?(:each)
+
+      headers.each do |key, value|
+        return header_value(value) if normalize(key.to_s) == normalized
+      end
+      nil
+    end
+
+    def normalize(key)
+      key.upcase.tr("-", "_").delete_prefix("HTTP_")
+    end
+
+    def safe_lookup(headers, key)
+      headers[key]
+    rescue StandardError
+      nil
+    end
+
+    def header_value(value)
+      value.is_a?(Array) ? value.join(",") : value.to_s
+    end
+  end
+end

data/lib/urlpipe.rb

@@ -0,0 +1,17 @@
+# frozen_string_literal: true
+
+require_relative "urlpipe/version"
+require_relative "urlpipe/errors"
+require_relative "urlpipe/response"
+require_relative "urlpipe/screenshot"
+require_relative "urlpipe/client"
+require_relative "urlpipe/webhook"
+
+# Ruby client for the URLpipe API (https://urlpipe.dev): turn a URL into
+# Markdown, rendered HTML, a screenshot, metadata, a summary, keywords,
+# console errors or a Lighthouse audit.
+#
+#   client = Urlpipe::Client.new # reads URLPIPE_API_KEY
+#   client.markdown("https://example.com").data
+module Urlpipe
+end

metadata

@@ -0,0 +1,60 @@
+--- !ruby/object:Gem::Specification
+name: urlpipe
+version: !ruby/object:Gem::Version
+  version: 0.1.0
+platform: ruby
+authors:
+- Aliat Partner S.L.
+bindir: bin
+cert_chain: []
+date: 1980-01-02 00:00:00.000000000 Z
+dependencies: []
+description: Official Ruby client for URLpipe. Get a page's Markdown, rendered HTML,
+  a full-page screenshot, metadata, a summary, keywords, console errors or a Lighthouse
+  audit, rendered in real Chrome. Handles long analyses, retries with idempotency
+  keys, and webhook signature verification. No runtime dependencies.
+email:
+- contact@urlpipe.dev
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- CHANGELOG.md
+- LICENSE
+- README.md
+- lib/urlpipe.rb
+- lib/urlpipe/client.rb
+- lib/urlpipe/errors.rb
+- lib/urlpipe/response.rb
+- lib/urlpipe/screenshot.rb
+- lib/urlpipe/version.rb
+- lib/urlpipe/webhook.rb
+homepage: https://urlpipe.dev
+licenses:
+- MIT
+metadata:
+  homepage_uri: https://urlpipe.dev
+  documentation_uri: https://urlpipe.dev/docs
+  source_code_uri: https://github.com/URLpipe/urlpipe-ruby
+  changelog_uri: https://github.com/URLpipe/urlpipe-ruby/blob/main/CHANGELOG.md
+  bug_tracker_uri: https://github.com/URLpipe/urlpipe-ruby/issues
+  rubygems_mfa_required: 'true'
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '3.1'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubygems_version: 4.0.20
+specification_version: 4
+summary: 'Ruby client for the URLpipe API: turn a URL into Markdown, HTML, a screenshot
+  and more.'
+test_files: []