x-streams 1.0.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 351c36af5826dec08a5ca7c014b7b34d18062bbe1de19b0a9593290c075e7f7d
+  data.tar.gz: c3473196d06dad56b10422b60859074fde027611fda1c99038e4461d2422487e
+SHA512:
+  metadata.gz: f1570d57cec22227dc0ba01f18649c5874aca5422ea10953415379a17029c13d5e8a5b166c3eb3a9bdade21c899c028b0ee1427067690d7bce9d4a20e2b924f1
+  data.tar.gz: f05079b5e9b1783711f7938d4f5347376205ec7208b703d6d20617d32fe38c12716922459845cfa9c8f00cf88f5e297ce096581c7d77db42a04b69c42eb6f405

data/.yardopts

@@ -0,0 +1,8 @@
+--markup markdown
+--readme README.md
+--hide-api private
+--embed-mixins
+lib/**/*.rb
+-
+CHANGELOG.md
+LICENSE.txt

data/CHANGELOG.md

@@ -0,0 +1,99 @@
+# Changelog
+
+All notable changes to `x-streams` will be documented in this file.
+
+The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
+and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
+
+`x-streams` is released in lockstep with the other gems of the [x-ruby](https://github.com/sferik/x-ruby) repository, at one version across `x-core`, `x-uploads`, `x-streams`, `x-resources`, and `x`. This file holds the changes to the streams; [the changelog of the repository](https://github.com/sferik/x-ruby/blob/main/CHANGELOG.md) holds the changes to every gem.
+
+## [1.0.0] - 2026-10-06
+
+The first release of `x-streams`, which 1.0.0 split out of the `x` gem. The entries below are the changes since `x` 0.19, the last release before the split; see [UPGRADING.md](https://github.com/sferik/x-ruby/blob/main/UPGRADING.md) for the changes that code written for 0.19 needs. It requires Ruby 3.4 or later.
+
+### Added
+* Split the `x` gem into five gems released in lockstep, at one version
+  * `x-core` (the HTTP client), `x-uploads` (uploads), `x-streams` (streams and their rules), `x-resources` (resources).
+  * `x` is a meta-gem that depends on all four and mixes their object, upload, and streaming methods into `X::Client`.
+  * `x-core` declares `X::Error`, the base class of every error the gems raise.
+  * Public classes sit directly under `X`; `X::Resources::Error`, `X::Uploads::Error`, and `X::Streams::Error` catch one gem's.
+  * `x` depends on exactly its own version of the other four; `x-uploads`, `x-streams`, and `x-resources` each depend on `x-core` with `>= 1.0.0, < 2`, so a later 1.x of `x-core` installs beside them, and never an earlier one.
+* Stream with `X::StreamingClient`, which `X::Client#streaming` builds from a client
+  * It shares the client's credentials, base URL, parsing classes, and `on_response`, and keeps its own settings for life.
+  * `X::Client#streaming` takes `read_timeout:`, `max_reconnects:`, and `on_reconnect:`; for other settings, stream from `client.with(...)`.
+  * `stream` raises `ArgumentError` when given no block, before it opens the stream.
+* Give a client `streaming` with `X::Streams::API`, which `x` includes into `X::Client`
+  * With `x-core` alone, call `X::Client.include(X::Streams::API)`, or build one with `X::StreamingClient.new(client)`.
+  * A streaming client opens each stream with `X::Client#get_stream` of `x-core`.
+* Stop the streams of a streaming client from any thread, or the trap of a signal, with `X::StreamingClient#stop`
+  * Each stops the next time it waits on the API, at once for an idle stream or one waiting to reconnect.
+  * A stream a Fiber scheduler runs, as in an `Async` task, stops once its block next returns, at the next keep-alive, or within a second as it waits to reconnect.
+  * A stopped streaming client stays stopped: a stream asked of it later returns nil at once, without a request.
+  * Each stopped `stream` call returns nil, `stop` returns nil, and `stopped?` tells a stopped streaming client.
+  * A block, `on_response`, or `on_reconnect` running at that moment runs to its end first.
+  * `X::Client#streaming` builds a new streaming client each time, so keep the one you stop in a variable.
+* Stop a stream from its block: `break` stops it and returns its value, and `throw` unwinds past it
+  * An error raised by the block, by `on_response`, or by the `object_class` stops the stream and reaches the caller.
+  * That includes `StopIteration`, and a `JSON::ParserError`, which is not taken for a line that is not JSON.
+* Read and change the rules of the filtered stream with `rules`, `add_rules`, and `delete_rules` on `X::StreamingClient`
+  * They authenticate as the app, as a stream does; `rules` takes `params:` and reads every page.
+  * A rule to add is an `X::StreamRule`, a Hash of a `value` and `tag`, or a String; anything else raises `ArgumentError`.
+  * A rule is deleted by its `id` (from an `X::StreamRule`, a Hash, an Integer, or an `X::MatchingRule`, and nothing else that answers `id`) or by its value.
+  * So `delete_rules(post.matching_rules)` deletes the rules a post of `x-resources` matched.
+  * `dry_run: true` has the API check the rules and change none; no rules send no request.
+  * `add_rules` returns the rules added; `delete_rules` returns the number deleted.
+  * A rule the app already has, a `DuplicateRule`, raises nothing and is not among the rules returned; `rules` reads it, with its `id` and `tag`.
+* Yield each problem the API reports of the rules it did not add or delete as an `X::Problem`, or raise `X::RulesRejected` without a block
+  * A block is yielded a `DuplicateRule` too; without one, only the rules X rejected raise, so `add_rules` can run at every boot.
+  * A rule X rejects, such as an invalid one, may keep it from adding any, and it reports that one alone; the rules returned say which were added.
+  * `X::RulesRejected` holds the `problems`, and what the method would have returned as `added` or `deleted_count`.
+* Add `X::StreamRule`, a frozen value of a rule's `value`, `tag`, and `id`, an Integer or nil for a rule to add
+  * An `id` is an Integer that is not negative or a String of digits alone; anything else raises `ArgumentError`.
+  * It compares and hashes by all three, reads as a Hash with `to_h`, and matches a pattern of them.
+  * It writes itself with Marshal and YAML in a versioned format that every 1.x release reads back.
+  * A format it does not read raises `X::UnsupportedFormat`.
+* Stream with app-only authentication from a client that signs with OAuth 1.0a, since the stream endpoints refuse it
+  * An OAuth 2.0 user client without app credentials streams, and reads and changes rules, as the user.
+  * X refuses that with 403, which raises `X::Forbidden`, not `X::UnsupportedOperation` before connecting.
+* Raise `X::StreamError`, an `X::Streams::Error`, for a stream line that holds errors and no data
+  * It holds the line's `problems`, reads the stream's `http_method` and `uri`, and names them in its message.
+  * A line of `operational-disconnect`s alone reconnects, then raises once no reconnects are left; others raise at once.
+  * `X::Problem#disconnect?` tells an `operational-disconnect` apart, and `on_response` is passed the line first.
+* Reconnect a stream that ends, drops, or delivers a line that is not JSON, backing off as X recommends
+  * It reconnects up to `max_reconnects` times in a row, unlimited by default.
+  * An object or keep-alive read from a connection open for a minute resets the count and the backoff; an earlier one resets neither.
+  * So a stream whose connections each deliver an object and drop backs off, and runs out of reconnects, rather than reconnect at once without end.
+  * A line that is not JSON raises `X::InvalidResponse`, and an ended stream `X::NetworkError`, once no reconnects are left.
+  * A rate limit backs off from a minute, doubling each attempt, or waits for a later reset.
+  * A wait past `max_rate_limit_wait`, or the project's usage cap, raises `X::TooManyRequests` at once.
+  * A server error, a 408, or a 409 backs off from 5 seconds, doubling up to 320 seconds, or waits longer for its `Retry-After`.
+  * A `Retry-After` past `max_rate_limit_wait` raises the error at once.
+  * What arrived of a line the stream ended within is dropped, neither parsed nor passed to `on_response`.
+  * An error `on_response` raises for a failed response stops the stream and reaches the caller, as one for an object does.
+  * A certificate that does not verify raises its `X::NetworkError` at once, since it would not verify on the next attempt.
+* Report each reconnect of a stream to `on_reconnect:`, a callable passed the error and the seconds it waits
+  * It is passed an error every time, never nil, and is called with those two arguments and no others throughout 1.x.
+  * It can call `stop` to give up, as on a host that never resolves, and the stream returns nil.
+  * An error it raises stops the stream and reaches the caller.
+* Read a stream with the `read_timeout` of the streaming client, 30 seconds by default
+* Raise `ArgumentError` from `X::Client#streaming` and `X::StreamingClient.new` for an invalid setting
+  * `max_reconnects` must be an Integer of at least 0, or `Float::INFINITY`.
+  * `read_timeout` must be a finite number of seconds of at least 25, or nil, since X sends a keep-alive every 20 seconds.
+  * `stream` raises it too for an invalid `array_class`, `object_class`, or endpoint, before it opens the stream.
+* Rescue the errors of `x-streams`, `X::StreamError` and `X::RulesRejected`, with `X::Streams::Error`, an `X::Error`
+* Add `X::Streams.gem_version`, which returns `X::Streams::VERSION` as a `Gem::Version`
+* Add `X::StreamingClient::DEFAULT_READ_TIMEOUT` and `X::StreamingClient::DEFAULT_MAX_RECONNECTS`
+* Build `X::StreamError` and `X::RulesRejected` with public constructors, so code that rescues one can be tested
+  * Each takes an optional message and `problems:`, beside `http_method:` and `uri:`, or `added:` and `deleted_count:`.
+  * `raise X::StreamError, "dropped"` raises it as any other exception is raised.
+* Raise `TypeError` from `Marshal.dump`, `YAML.dump`, `as_json`, and `to_json` of a streaming client
+  * They would write the credentials of its client in the clear.
+* Name the internals of `x-streams` under `X::Streams`, as private constants marked `@api private`
+  * They are `StreamParser`, `ReconnectHandler`, `StreamRules`, `Validator`, `Stopper`, and `CallbackError`.
+* Ship this changelog with `x-streams`, which the `changelog_uri` of its gemspec names
+* Ship a `.yardopts` with `x-streams`, so its documentation on rubydoc.info leaves out the private API
+
+### Changed
+* Move `stream` from `X::Client` to `X::StreamingClient`, so that the client carries no streaming settings
+
+[1.0.0]: https://github.com/sferik/x-ruby/releases/tag/v1.0.0

data/LICENSE.txt

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2023 Erik Berlin
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,61 @@
+# x-streams
+
+Streaming for the [`x` gem](https://github.com/sferik/x-ruby), built on the HTTP client in [`x-core`](https://github.com/sferik/x-ruby/tree/main/x-core).
+
+* `X::StreamingClient` reads the sample and filtered streams of the X API, a post at a time, and reconnects a stream that drops, backing off as X recommends.
+* It reads, adds, and deletes the rules of the filtered stream, each an `X::StreamRule`.
+
+Its only dependency is `x-core`. Installing [`x`](https://rubygems.org/gems/x) installs this gem too, and `require "x"` gives every `X::Client` its `streaming` method.
+
+## Installation
+
+`x-streams` requires Ruby 3.4 or later.
+
+    bundle add x-streams
+
+## Usage
+
+```ruby
+require "x/core"
+require "x/streams"
+
+X::Client.include(X::Streams::API)
+
+client = X::Client.new(bearer_token: "INSERT YOUR BEARER TOKEN HERE")
+
+# A rule the app already has is not added again, and raises nothing, so the rules an app needs are added each time
+# it boots; the rules returned are those added, and rules reads them all, each with its id and tag
+client.streaming.add_rules([{value: "ruby -is:retweet", tag: "ruby"}, "crystal"])
+client.streaming.rules # => [#<X::StreamRule id=1 value="ruby -is:retweet" tag="ruby">, ...]
+client.streaming.stream("tweets/search/stream") { |post| puts post["data"]["text"] }
+
+# A stream runs until its block, or stop, stops it: break to stop it and return a value, throw to unwind further out, or
+# raise to stop it with an error the caller sees
+first = client.streaming.stream("tweets/search/stream") { |post| break post }
+```
+
+Another thread, or the trap of a signal, stops every stream a streaming client runs with `stop`, and each stream it stops returns nil. A stream that a Fiber scheduler runs, as in an `Async` task, is stopped once its block next returns, at the next keep-alive, within 20 seconds, or within a second as it waits to reconnect. A stopped streaming client stays stopped, as `stopped?` tells: a stream asked of it later returns nil without a request. Keep the streaming client you stop in a variable, since each call to `client.streaming` builds a new one.
+
+`X::Client.include(X::Streams::API)` gives a client `streaming`, as `x` does, and a type checker needs the include declared too, as the `sig/x.rbs` of `x` declares it, with `class X::Client` and `include X::Streams::API` in a signature of your own; without the include, build a streaming client of a client with `X::StreamingClient.new(client)`, which takes the same `read_timeout:`, `max_reconnects:`, and `on_reconnect:`, a callable passed the error and the wait of each reconnect.
+
+A streaming client opens each stream with `X::Client#get_stream` of `x-core`, the public method of a client that sends a GET whose body is read as it arrives, from an app-only copy of the client whose `read_timeout` is the stream's (a client that authenticates with OAuth 2.0 as a user and holds no credentials of the app streams as the user, which X refuses with `X::Forbidden`), so a stream carries the credentials, headers, and proxy of the client as any request does. It reads each line of the stream into the object it delivers, passes each to the client's `on_response` as an `X::Response`, and reconnects after a dropped connection, a server error, a rate limit, or an `operational-disconnect`, as the [`x` README](https://github.com/sferik/x-ruby#streaming) describes.
+
+A stream reconnects at once the first time it drops, then waits longer before each reconnect in a row, up to `max_reconnects` of them. An object or keep-alive read from a connection that has been open for a minute starts the waits and the count over; one read from a younger connection starts neither over, so a stream whose connections each deliver something and drop backs off, and runs out of reconnects, rather than reconnect at once without end.
+
+`X::StreamError`, which a line of a stream that holds errors and no data raises, and `X::RulesRejected`, which `add_rules` and `delete_rules` raise for the rules the API rejected when no block takes them (a rule the app already has is not one of them: `add_rules` raises nothing for it, yields its `DuplicateRule` to a block, and returns the rules it added alone, so read a rule the app already had, with its `id` and `tag`, with `rules`), descend from `X::Streams::Error`, which descends from `X::Error`, so `rescue X::Streams::Error` catches the errors x-streams raises of its own, and `rescue X::Error` every failure of a stream.
+
+With `x-resources` loaded as well, a stream builds each post as an `X::Post` given `object_class: X::Post`, whose `matching_rules` are the rules of the filtered stream it matched.
+
+## Development
+
+This gem has its own `Gemfile`, `Steepfile`, signatures, test suite, and mutation config. It uses the `x-core` in this repository and does not load `x-resources` or `x-uploads`:
+
+    bundle install
+    bundle exec rake test
+    bundle exec rake mutant
+    bundle exec rake steep
+    bundle exec rake yardstick
+
+## License
+
+The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).

data/lib/x/streams/api.rb

@@ -0,0 +1,41 @@
+# frozen_string_literal: true
+
+require_relative "streaming_client"
+
+module X
+  module Streams
+    # The streaming method mixed into a client, which builds a streaming client of the client
+    #
+    # The x gem includes it into X::Client. With x-core and x-streams alone, include it yourself:
+    # X::Client.include(X::Streams::API), or build a streaming client of a client with X::StreamingClient.new. It
+    # passes the object it is included into to the streaming client as its client, which is an X::Client, so it belongs
+    # in X::Client or a subclass.
+    #
+    # @api public
+    module API
+      # A client for the streaming endpoints, which reads and reconnects differently
+      #
+      # Each call builds a new streaming client, so the one a stream runs on is kept in a variable to stop it with
+      # {StreamingClient#stop}, and a streaming client that was stopped, which stays stopped, is replaced by another.
+      #
+      # @api public
+      # @param read_timeout [Integer, Float, nil] the timeout for reading from a stream in seconds, as
+      #   {StreamingClient#initialize} takes it
+      # @param max_reconnects [Integer, Float] the maximum number of times in a row to reconnect a stream that drops, as
+      #   {StreamingClient#initialize} takes it
+      # @param on_reconnect [#call, nil] a callable passed the error that dropped a stream and the seconds it waits
+      #   before each reconnect, as {StreamingClient#initialize} takes it, or nil for none
+      # @return [StreamingClient] a streaming client that shares this client's credentials and settings
+      # @raise [ArgumentError] if the read timeout is neither a finite number of seconds of at least 25 nor nil, the
+      #   maximum number of reconnects is neither a count nor Float::INFINITY, or on_reconnect neither responds to call
+      #   nor is nil
+      # @example Stream filtered posts, giving up after five reconnects in a row
+      #   client.streaming(max_reconnects: 5).stream("tweets/search/stream") { |post| puts post }
+      # @example Log each reconnect of a stream
+      #   client.streaming(on_reconnect: ->(error, wait) { logger.warn("#{error.message}; reconnecting in #{wait}s") })
+      def streaming(read_timeout: StreamingClient::DEFAULT_READ_TIMEOUT, max_reconnects: StreamingClient::DEFAULT_MAX_RECONNECTS, on_reconnect: nil)
+        StreamingClient.new(_ = self, read_timeout:, max_reconnects:, on_reconnect:)
+      end
+    end
+  end
+end

data/lib/x/streams/callback_error.rb

@@ -0,0 +1,53 @@
+# frozen_string_literal: true
+
+module X
+  module Streams
+    # Wraps an error a callback of a stream raised, so that it is not taken for an error of the stream
+    #
+    # A stream runs on_response, the from_response of its object_class, and its block inside the request that reads
+    # it, where an IOError or a SystemCallError a callback raised would be taken for a connection that dropped, and an
+    # X::ServiceUnavailable for the response of the stream, and the stream reconnected after it. So a callback's error
+    # is wrapped in this, which neither X::Client#get_stream nor the reconnects of a stream take for one of theirs, and
+    # raised as it was once the stream is left.
+    #
+    # Internal to x-streams: StreamParser tags the errors of a callback with it, as StreamingClient tags those of the
+    # on_response its stream client is passed a failed response with, and ReconnectHandler raises the error it holds
+    # in its place.
+    #
+    # @api private
+    class CallbackError < StandardError
+      # The error the callback raised
+      # @api private
+      # @return [StandardError] the error
+      attr_reader :error
+
+      # A callable that tags the error another raises, or nil for no callable
+      #
+      # X::Client#get_stream tags the error of the on_response it passes a failed response as its own, and raises it
+      # as it was, so a callable whose error is not to be taken for one of the stream raises this itself.
+      #
+      # @api private
+      # @param callable [#call, nil] the callable
+      # @return [Proc, nil] the callable that tags its errors, or nil if it is nil
+      # @example Tag the errors of the on_response of a client
+      #   X::Streams::CallbackError.tagging(client.on_response)
+      def self.tagging(callable)
+        callable && lambda do |*arguments|
+          callable.call(*arguments)
+        rescue => e
+          raise new(e)
+        end
+      end
+
+      # Initialize a new CallbackError
+      # @api private
+      # @param error [StandardError] the error the callback raised
+      # @return [CallbackError] a new instance
+      def initialize(error)
+        super
+        @error = error
+      end
+    end
+    private_constant :CallbackError
+  end
+end

data/lib/x/streams/error.rb

@@ -0,0 +1,34 @@
+# frozen_string_literal: true
+
+require "x/core"
+
+module X
+  module Streams
+    # Base error class for the failures of a stream, which every error x-streams raises of its own descends from
+    #
+    # It descends from X::Error, so that rescuing the errors of the X API catches the failure of a stream too. The
+    # errors that descend from it are named directly under X, as the errors of x-core are, so that this is the one name
+    # under X::Streams a rescue reaches for.
+    #
+    # It catches the errors x-streams raises of its own: a line of a stream that held errors and no data, and rules
+    # of the filtered stream the API left unchanged. It does not catch the X::Error of a request the API refused, or
+    # that got no response, such as the X::NetworkError of a stream that dropped with no reconnects left, nor the
+    # ArgumentError of a mistake in the arguments of a call. Rescue X::Error to catch every failure of a stream.
+    #
+    # @api public
+    class Error < X::Error
+      private
+
+      # The title and detail of each problem, separated by commas
+      #
+      # It is the message of an error built of problems without a message of its own.
+      #
+      # @api private
+      # @param problems [Array<Problem>] the problems
+      # @return [String, nil] the title and detail of each problem, or nil for none
+      def describe(problems)
+        problems.map { |problem| [problem.title, problem.detail || problem.message].compact.join(": ") }.join(", ") unless problems.empty?
+      end
+    end
+  end
+end