tapfall 0.0.1

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 10d16e103fb9bb828f86ba932076d45f3f7d5e89b81eceb3f60292dfc5605aa7
+  data.tar.gz: 3087c8bbc196e557f046aaef5bfc48ac49220d2efc835996319b3b74db352aa7
+SHA512:
+  metadata.gz: e18feb0d610d2d6e16edcb6e68fd6f8e8a949117e609e519501b4b2299aa0f2b721e7b008f8f4be5f3fdb1239f43ae8a5ce7fa3cc2d6991262fa99917e7b21d2
+  data.tar.gz: 91557b3f07d953e622ad190c24c0bf471ece14184e004731f30eaa104a36f78121704ee1e38b0191ec7d3e5d02549f383300002d5bb11efeaed48363c0537b3a

data/CHANGELOG.md

@@ -0,0 +1,3 @@
+## [0.0.1] - 2025-12-22
+
+- first working version, with streaming from Tap, support for ack and admin password options, and calling two HTTP endpoints

data/LICENSE.txt

@@ -0,0 +1,21 @@
+The zlib License
+
+Copyright (c) 2025 Jakub Suder
+
+This software is provided 'as-is', without any express or implied
+warranty.  In no event will the authors be held liable for any damages
+arising from the use of this software.
+
+Permission is granted to anyone to use this software for any purpose,
+including commercial applications, and to alter it and redistribute it
+freely, subject to the following restrictions:
+
+1. The origin of this software must not be misrepresented; you must not
+   claim that you wrote the original software. If you use this software
+   in a product, an acknowledgment in the product documentation would be
+   appreciated but is not required.
+
+2. Altered source versions must be plainly marked as such, and must not be
+   misrepresented as being the original software.
+
+3. This notice may not be removed or altered from any source distribution.

data/README.md

@@ -0,0 +1,209 @@
+# Tapfall
+
+A Ruby gem for ingesting ATProto repository data from a [Tap](https://github.com/bluesky-social/indigo/tree/main/cmd/tap) service (extension of the [Skyfall](https://tangled.org/mackuba.eu/skyfall) gem).
+
+> [!NOTE]
+> Part of ATProto Ruby SDK: [ruby.sdk.blue](https://ruby.sdk.blue)
+
+
+## What does it do
+
+[Tap](https://github.com/bluesky-social/indigo/tree/main/cmd/tap) is a [tool made by Bluesky](https://docs.bsky.app/blog/introducing-tap), which combines a firehose client/adapter with a JSON output like [Jetstream](https://github.com/bluesky-social/jetstream) and a repository/PDS crawler and importer. It can be useful if you're building some kind of backend app that needs to import and store some set of records from the Atmosphere. It's meant to be run locally on your app's server, with only your app connecting to it, and it simplifies a lot of code for you if you need to both import & backfill existing records of some kind and also stream any new ones that are added after you start the import.
+
+Basically, before Tap, your app code needed to:
+
+1) Connect to a relay or Jetstream
+2) Filter only records of selected kinds
+3) Find out which repos on which PDSes have records that are relevant to you
+4) Connect to those PDSes and get those records via listRecords or getRepo
+5) Handle possible duplicates between imported repo and the firehose
+
+If you only want some kinds of records from some specific repos, that still leaves you with both a firehose client and a repo importer and merging the results from both somehow.
+
+With Tap, you only need to:
+
+1) Run Tap, passing it a list of record types to sync in command line parameters or in env
+2) Connect to the Tap stream on localhost
+3) Save everything coming from the stream
+
+So instead of two ways of importing the records, you only have one and it's the much less involved one. And this library also handles a lot of this for you.
+
+**Tapfall** is an extension of [Skyfall](https://tangled.org/mackuba.eu/skyfall), which is a gem for streaming records from a relay/PDS firehose or Jetstream, and it adds support for the event format used by Tap and for some additional HTTP APIs it provides.
+
+
+## Installation
+
+Add this to your `Gemfile`:
+
+    gem 'tapfall'
+
+
+## Usage
+
+Create a `Tapfall::Stream` object, specifying the address of the Tap service websocket:
+
+```rb
+require 'tapfall'
+
+tap = Tapfall::Stream.new('ws://localhost:2480')
+```
+
+You can also just pass a hostname, but then it's interpreted as HTTPS/WSS, which might not be what you want.
+
+Next, set up event listeners to handle incoming messages and get notified of errors. Here are all the available listeners (you will need at least `on_message`):
+
+```rb
+# this gives you a parsed message object, one of subclasses of Tapfall::TapMessage
+tap.on_message { |msg| p msg }
+
+# lifecycle events
+tap.on_connecting { |url| puts "Connecting to #{url}..." }
+tap.on_connect { puts "Connected" }
+tap.on_disconnect { puts "Disconnected" }
+tap.on_reconnect { puts "Connection lost, trying to reconnect..." }
+tap.on_timeout { puts "Connection stalled, triggering a reconnect..." }
+
+# handling errors (there's a default error handler that does exactly this)
+tap.on_error { |e| puts "ERROR: #{e}" }
+```
+
+You can also call these as setters accepting a `Proc` – e.g. to disable default error handling, you can do:
+
+```rb
+tap.on_error = nil
+```
+
+When you're ready, open the connection by calling `connect`:
+
+```rb
+tap.connect
+```
+
+The `#connect` method blocks until the connection is explicitly closed with `#disconnect` from an event or interrupt handler. Tapfall & Skyfall use [EventMachine](https://github.com/eventmachine/eventmachine) under the hood, so in order to run some things in parallel, you can use e.g. `EM::PeriodicTimer`.
+
+Tapfall also supports Skyfall's `on_raw_message` handler version, but only if you use Tap in "disable acks" mode (see below), which is not recommended beyond testing, unless you're doing the acks yourself. (This is because Tapfall needs to parse the message into a JSON form in order to get the `id` of the event to send the "ack".)
+
+> [!NOTE]
+> Unlike standard firehose and Jetstream, Tap streams don't have a cursor that you store and pass when reconnecting. It's meant to be used only by one client, and it tracks internally itself which events have been sent to you and which weren't. You can think about it this way: it's not a public service like Jetstream that you can share with others, it's a microservice that you run as a component of your app.
+
+
+### Acks
+
+Tap by default runs in a mode where it expects the client to send back an "ack" after receiving and processing each event. When it gets the ack, it marks the event as processed and will not send it again. If you don't send an ack, it tries to retransmit the event after a moment.
+
+You can also run it with acks disabled, by passing a `--disable-acks` option or `TAP_DISABLE_ACKS=true` env var, in which case it will assume an event has been processed as soon as it's sent to you. This is not recommended to do in production, since if your process crashes during an event processing loop, that event will be lost (and you can't ask for an earlier cursor because there's no cursor).
+
+Tapfall handles the acks for you automatically. If you want it to not send acks, pass an `:ack => false` option to the constructor:
+
+```rb
+tap = Tapfall::Stream.new(server, { ack: false })
+```
+
+### Password-protected access
+
+Tap also lets you set an admin password, which you can set with the `--admin-password` option or `TAP_ADMIN_PASSWORD` env var. This locks the stream and the API behind HTTP Basic auth (with the user `admin`). Pass the password to Tapfall constructor like this:
+
+```rb
+tap = Tapfall::Stream.new(server { admin_password: 'abracadabra' })
+```
+
+### Processing messages
+
+Each message passed to `on_message` is an instance of a subclass of `Tapfall::TapMessage`. The main event type is `Tapfall::RecordMessage`, which includes a record operation; you will also receive `Tapfall::IdentityMessage` events, which provide info about an account change like changed handle or migration to a new PDS. `UnknownMessage` might be sent if new unrecognized message types are sent in the future.
+
+All message types share these properties:
+
+- `type` (symbol) – the message type identifier, e.g. `:record`
+- `id` (integer), aliased as `seq` – a sequential index of the message
+
+The `:record` messages have an `operations` method, which includes an array of add/remove/edit `Operation`s done on some records. Currently Tap event format only includes one single record operation in each event, but it's returned as an array here for symmetry with the `Skyfall::Firehose` stream version.
+
+An `Operation` has such fields (also matching the API of `Skyfall::Firehose::Operation` and `Skyfall::Jetstream::Operation`):
+
+- `repo` or `did` (string) – DID of the repository (user account)
+- `collection` (string) – name of the collection / record type, e.g. `app.bsky.feed.post` for posts
+- `type` (symbol) – short name of the collection, e.g. `:bsky_post`
+- `rkey` (string) – identifier of a record in a collection
+- `path` (string) – the path part of the at:// URI – collection name + ID (rkey) of the item
+- `uri` (string) – the complete at:// URI
+- `action` (symbol) – `:create`, `:update` or `:delete`
+- `cid` (CID) – CID of the operation/record (`nil` for delete operations)
+- `live?` (boolean) – true if the record was received from the firehose, false if it was backfilled from the repo
+
+Create and update operations will also have an attached record (JSON object) with details of the post, like etc. The record data is currently available as a Ruby hash via `raw_record` property (custom types may be added in future).
+
+So for example, in order to filter only "create post" operations and print their details, you can do something like this:
+
+```rb
+tap.on_message do |m|
+  next if m.type != :record
+
+  m.operations.each do |op|
+    next unless op.action == :create && op.type == :bsky_post
+
+    puts "#{op.repo}:"
+    puts op.raw_record['text']
+    puts
+  end
+end
+```
+
+
+### Note on custom lexicons
+
+Note that the `Operation` objects have two properties that tell you the kind of record they're about: `#collection`, which is a string containing the official name of the collection/lexicon, e.g. `"app.bsky.feed.post"`; and `#type`, which is a symbol meant to save you some typing, e.g. `:bsky_post`.
+
+When Tapfall receives a message about a record type that's not on the list, whether in the `app.bsky` namespace or not, the operation `type` will be `:unknown`, while the `collection` will be the original string. So if an app like e.g. "Skygram" appears with a `zz.skygram.*` namespace that lets you share photos on ATProto, the operations will have a type `:unknown` and collection names like `zz.skygram.feed.photo`, and you can check the `collection` field for record types known to you and process them in some appropriate way, even if Tapfall doesn't recognize the record type.
+
+Do not however check if such operations have a `type` equal to `:unknown` first – just ignore the type and only check the `collection` string. The reason is that some next version might start recognizing those records and add a new `type` value for them like e.g. `:skygram_photo`, and then they won't match your condition anymore.
+
+
+### Reconnection logic
+
+See the section in the [Skyfall readme](https://tangled.org/mackuba.eu/skyfall#reconnection-logic) about the options for handling reconnecting to a flaky firehose – but since in this case the Tap service will run under your control, likely on the same machine, this might not be as useful in practice.
+
+
+## HTTP API
+
+Apart from the `/channel` websocket endpoint, Tap also has [a few other endpoints](https://github.com/bluesky-social/indigo/tree/main/cmd/tap#http-api) for adding/removing repos and checking various stats.
+
+You can call all the below methods either on the `Tapfall::Stream` instance you use for connecting to the websocket, or on a separate `Tapfall::API` object if you prefer.
+
+Currently implemented endpoints:
+
+### /repos/add
+
+Tap can work in three possible ways regarding the subset of repos it tracks:
+
+1) `TAP_FULL_NETWORK`, when it tracks *all repos everywhere*
+2) `TAP_SIGNAL_COLLECTION`, when it finds and tracks all repos that have some specific types of records you're interested in
+3) default mode, when it only tracks repos you've added manually
+
+In that third mode, use this to add repos to the tracking list:
+
+```rb
+@tap.add_repo('did:plc:uh4errluyq5thgszrwwrtpuq')
+
+# or:
+@tap.add_repos(did_list)
+```
+
+### /repos/remove
+
+To remove repos from the list, use:
+
+```rb
+@tap.remove_repo('did:plc:uh4errluyq5thgszrwwrtpuq')
+
+# or:
+@tap.remove_repos(did_list)
+```
+
+
+## Credits
+
+Copyright © 2025 Kuba Suder ([@mackuba.eu](https://bsky.app/profile/did:plc:oio4hkxaop4ao4wz2pp3f4cr)).
+
+The code is available under the terms of the [zlib license](https://choosealicense.com/licenses/zlib/) (permissive, similar to MIT).
+
+Bug reports and pull requests are welcome 😎

data/lib/tapfall/api.rb

@@ -0,0 +1,78 @@
+require 'json'
+require 'net/http'
+require 'uri'
+
+module Tapfall
+  class API
+    def initialize(server, options = {})
+      @root_url = build_root_url(server)
+      @options = options
+    end
+
+    def add_repo(did)
+      add_repos([did])
+    end
+
+    def add_repos(dids)
+      post_request('/repos/add', { dids: dids })
+    end
+
+    def remove_repo(did)
+      remove_repos([did])
+    end
+
+    def remove_repos(dids)
+      post_request('/repos/remove', { dids: dids })
+    end
+
+    private
+
+    def build_root_url(server)
+      if !server.is_a?(String)
+        raise ArgumentError, "Server parameter should be a string"
+      end
+
+      if server.include?('://')
+        uri = URI(server)
+
+        if uri.scheme != 'http' && uri.scheme != 'https'
+          raise ArgumentError, "Server parameter should be a hostname or a http:// or https:// URL"
+        elsif uri.path != ''
+          raise ArgumentError, "Server URL should only include a hostname, without path"
+        end
+
+        uri.to_s
+      else
+        server = "https://#{server}"
+        uri = URI(server) # raises if invalid
+        server
+      end
+    end
+
+    def post_request(path, json_data)
+      uri = URI(@root_url + path)
+
+      request = Net::HTTP::Post.new(uri)
+      request.body = JSON.generate(json_data)
+      request.content_type = "application/json"
+
+      if @options[:admin_password]
+        request.basic_auth('admin', @options[:admin_password])
+      end
+
+      response = Net::HTTP.start(uri.hostname, uri.port, :use_ssl => (uri.scheme == 'https')) do |http|
+        http.request(request)
+      end
+
+      status = response.code.to_i
+      message = response.message
+      response_body = (response.content_type == 'application/json') ? JSON.parse(response.body) : response.body
+
+      if response.is_a?(Net::HTTPSuccess)
+        response_body
+      else
+        raise BadResponseError.new(status, message, response_body)
+      end
+    end
+  end
+end

data/lib/tapfall/errors.rb

@@ -0,0 +1,32 @@
+module Tapfall
+  class BadResponseError < StandardError
+    attr_reader :status, :data
+
+    def initialize(status, status_message, data)
+      @status = status
+      @data = data
+
+      message = if error_message
+        "#{status} #{status_message}: #{error_message}"
+      else
+        "#{status} #{status_message}"
+      end
+
+      super(message)
+    end
+
+    def error_type
+      @data['error'] if @data.is_a?(Hash)
+    end
+
+    def error_message
+      @data['message'] if @data.is_a?(Hash)
+    end
+  end
+
+  class ConfigError < StandardError
+  end
+
+  class DecodeError < StandardError
+  end
+end

data/lib/tapfall/messages/identity_message.rb

@@ -0,0 +1,30 @@
+require_relative '../errors'
+require_relative 'operation'
+require_relative 'tap_message'
+
+module Tapfall
+  class IdentityMessage < TapMessage
+    def initialize(json)
+      raise DecodeError.new("Missing event details") if json['identity'].nil?
+      @identity = json['identity']
+
+      super
+    end
+
+    def did
+      @identity['did']
+    end
+
+    def handle
+      @identity['handle']
+    end
+
+    def active?
+      @identity['isActive']
+    end
+
+    def status
+      @identity['status']&.to_sym
+    end
+  end
+end

data/lib/tapfall/messages/operation.rb

@@ -0,0 +1,56 @@
+require 'skyfall/cid'
+require 'skyfall/collection'
+
+module Tapfall
+  class Operation
+    def initialize(json)
+      @json = json
+    end
+
+    def live?
+      @json['live']
+    end
+
+    def did
+      @json['did']
+    end
+
+    alias repo did
+
+    def rev
+      @json['rev']
+    end
+
+    def path
+      @json['collection'] + '/' + @json['rkey']
+    end
+
+    def action
+      @json['action'].to_sym
+    end
+
+    def collection
+      @json['collection']
+    end
+
+    def rkey
+      @json['rkey']
+    end
+
+    def uri
+      "at://#{repo}/#{collection}/#{rkey}"
+    end
+
+    def cid
+      @cid ||= @json['cid'] && Skyfall::CID.from_json(@json['cid'])
+    end
+
+    def raw_record
+      @json['record']
+    end
+
+    def type
+      Skyfall::Collection.short_code(collection)
+    end
+  end
+end

data/lib/tapfall/messages/record_message.rb

@@ -0,0 +1,15 @@
+require_relative '../errors'
+require_relative 'operation'
+
+module Tapfall
+  class RecordMessage < TapMessage
+    def initialize(json)
+      raise DecodeError.new("Missing record details") if json['record'].nil?
+      super
+    end
+
+    def operations
+      @operations ||= [Operation.new(json['record'])]
+    end
+  end
+end

data/lib/tapfall/messages/tap_message.rb

@@ -0,0 +1,44 @@
+require 'json'
+require_relative '../errors'
+
+module Tapfall
+  class TapMessage
+    attr_reader :id, :type
+    alias seq id
+
+    # :nodoc: - consider this as semi-private API
+    attr_reader :json
+
+    def self.new(data)
+      require_relative 'identity_message'
+      require_relative 'record_message'
+      require_relative 'unknown_message'
+
+      json = JSON.parse(data)
+
+      message_class = case json['type']
+        when 'record'   then RecordMessage
+        when 'identity' then IdentityMessage
+        else UnknownMessage
+      end
+
+      message = message_class.allocate
+      message.send(:initialize, json)
+      message
+    end
+
+    def initialize(json)
+      @json = json
+      @type = @json['type'].to_sym
+      @id = @json['id']
+    end
+
+    def unknown?
+      self.is_a?(UnknownMessage)
+    end
+
+    def operations
+      []
+    end
+  end
+end

data/lib/tapfall/messages/unknown_message.rb

@@ -0,0 +1,4 @@
+module Tapfall
+  class UnknownMessage < TapMessage
+  end
+end

data/lib/tapfall/stream.rb

@@ -0,0 +1,101 @@
+require 'forwardable'
+require 'skyfall/stream'
+
+require_relative 'api'
+require_relative 'errors'
+require_relative 'messages/tap_message'
+require_relative 'version'
+
+module Tapfall
+  class Tapfall::Stream < Skyfall::Stream
+    extend Forwardable
+
+    def_delegators :@api, :add_repo, :add_repos, :remove_repo, :remove_repos
+
+    def initialize(server, options = {})
+      super(server)
+
+      @options = options
+      @root_url = ensure_empty_path(@root_url)
+      @ack = true unless options[:ack] == false
+      @password = options[:admin_password]
+
+      @api = build_api
+    end
+
+    def connect
+      if @ack && @handlers[:message].nil?
+        raise ConfigError, "The on(:message) handler must be set unless :ack => false option is passed"
+      end
+
+      super
+    end
+
+    def handle_message(packet)
+      data = packet.data
+      @handlers[:raw_message]&.call(data)
+
+      if @handlers[:message]
+        tap_message = TapMessage.new(data)
+        @handlers[:message].call(tap_message)
+        send_ack(tap_message) if @ack
+      end
+    end
+
+    def send_ack(msg)
+      json = %({"type":"ack","id":#{msg.id}})
+      send_data(json)
+    end
+
+    private
+
+    # TMP
+    def send_data(data)
+      @ws.send(data)
+    end
+
+    def build_websocket_client(url)
+      Faye::WebSocket::Client.new(url, nil, { headers: { 'User-Agent' => user_agent }.merge(request_headers) })
+    end
+
+    def ensure_empty_path(url)
+      url = url.chomp('/')
+
+      if URI(url).path != ''
+        raise ArgumentError, "Server URL should only include a hostname, without any path"
+      end
+
+      url
+    end
+    # END
+
+    def basic_auth(account, password)
+      'Basic ' + ["#{account}:#{password}"].pack('m0')
+    end
+
+    def request_headers
+      if @password
+        { 'Authorization' => basic_auth('admin', @password) }
+      else
+        {}
+      end
+    end
+
+    def build_websocket_url
+      @root_url + "/channel"
+    end
+
+    def build_api_url
+      if @root_url.start_with?('ws://')
+        @root_url.gsub(/^ws:/, 'http:')
+      else
+        @root_url.gsub(/^wss:/, 'https:')
+      end
+    end
+
+    def build_api
+      api_url = build_api_url
+      API.new(api_url, { admin_password: @password })
+    end
+  end
+end

data/lib/tapfall/version.rb

@@ -0,0 +1,5 @@
+# frozen_string_literal: true
+
+module Tapfall
+  VERSION = '0.0.1'
+end

data/lib/tapfall.rb

@@ -0,0 +1,7 @@
+# frozen_string_literal: true
+
+require_relative 'tapfall/stream'
+require_relative 'tapfall/version'
+
+module Tapfall
+end

data/sig/tapfall.rbs

@@ -0,0 +1,4 @@
+module Tapfall
+  VERSION: String
+  # See the writing guide of rbs: https://github.com/ruby/rbs#guides
+end

metadata

@@ -0,0 +1,71 @@
+--- !ruby/object:Gem::Specification
+name: tapfall
+version: !ruby/object:Gem::Version
+  version: 0.0.1
+platform: ruby
+authors:
+- Kuba Suder
+bindir: bin
+cert_chain: []
+date: 1980-01-02 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: skyfall
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.6'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.6'
+email:
+- jakub.suder@gmail.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- CHANGELOG.md
+- LICENSE.txt
+- README.md
+- lib/tapfall.rb
+- lib/tapfall/api.rb
+- lib/tapfall/errors.rb
+- lib/tapfall/messages/identity_message.rb
+- lib/tapfall/messages/operation.rb
+- lib/tapfall/messages/record_message.rb
+- lib/tapfall/messages/tap_message.rb
+- lib/tapfall/messages/unknown_message.rb
+- lib/tapfall/stream.rb
+- lib/tapfall/version.rb
+- sig/tapfall.rbs
+homepage: https://ruby.sdk.blue
+licenses:
+- Zlib
+metadata:
+  bug_tracker_uri: https://tangled.org/mackuba.eu/tapfall/issues
+  changelog_uri: https://tangled.org/mackuba.eu/tapfall/blob/master/CHANGELOG.md
+  source_code_uri: https://tangled.org/mackuba.eu/tapfall
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: 3.2.0
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubygems_version: 4.0.1
+specification_version: 4
+summary: A gem for ingesting ATProto repository data from a Tap service (extension
+  of the Skyfall gem)
+test_files: []