skyfall 0.3.0 → 0.4.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 34e2166dc123a74140ce5cae67d3c000c1f88972d15cbe8a2550bde11ba4034d
-  data.tar.gz: 334f4b8b0ddce03b2258a0266e01e08cac9528924ac13380e91b66a0bb1217b8
+  metadata.gz: db537537fcd4e38f184c6cdea3f51ba1b8554848d4c3cd1de20f63ca2d956fc2
+  data.tar.gz: 34c51f6c8c0152589562bf0a3b3c5246cd8a1e95664f1128c86bd17c6a313d59
 SHA512:
-  metadata.gz: f56ab132941d80a577e5f3ba630b58d6f99548d0cde398fdb4f15e7c24d9557c3b0243d64a3940c18bf370afb219df3282c935300fc0359b6b0e00915b7002ca
-  data.tar.gz: 4bd7672a9450d8b1ec80ec679b69e6ba4242c4a6554865e296e7a347769795239b99a95539fe0b637407c2736b0716dfc8d1c3a9dea2d8d7a9654b47158262c9
+  metadata.gz: 11477293a1bc0377ef5d9eaa68cc2d374f76ac4d942006572e8b9e1668cb8f54a582abdeda81b65c1d72f6452ba2cad1153c403027ba5d7efbfa093624713105
+  data.tar.gz: 8d86b1f71a4fd5fa7f01077fefe8d8e292dc6b5007b79ee4d1ed1343f887c4104c0480b297e7ce4be61db0cce65211a568b642411fa5c3e7f62c4c91a4fe0f7b

data/CHANGELOG.md

@@ -1,3 +1,17 @@
+## [0.4.0] - 2024-09-23
+
+- (re)added a "hearbeat" feature (removed earlier in 0.2.0) to fix the occasional issue when the websocket stops receiving data, but doesn't disconnect (not enabled by default, turn it on by setting `check_heartbeat` to true)
+- added a way to set the user agent sent when connecting using the `user_agent` field (default is `"Skyfall/#{version}"`)
+- added `app.bsky.feed.postgate` record type
+
+## [0.3.1] - 2024-06-28
+
+- added `app.bsky.graph.starterpack` and `chat.bsky.actor.declaration` record types
+- added `#account` event type (`AccountMessage`)
+- added `handle` field to `IdentityMessage`
+- fixed param validation on `Stream` initialization
+- reverted the change that added Ruby stdlib dependencies explicitly to the gemspec, since this causes more problems than it's worth - only `base64` is left there, since it's the one now required to be listed
+
 ## [0.3.0] - 2024-03-21
 
 - added support for labeller firehose, served by labeller services at the `com.atproto.label.subscribeLabels` endpoint (aliased as `:subscribe_labels`)

data/LICENSE.txt

@@ -1,6 +1,6 @@
 The zlib License
 
-Copyright (c) 2023 Jakub Suder
+Copyright (c) 2023-2024 Jakub Suder
 
 This software is provided 'as-is', without any express or implied
 warranty.  In no event will the authors be held liable for any damages

data/README.md

@@ -1,6 +1,9 @@
 # Skyfall
 
-🌤 A Ruby gem for streaming data from the Bluesky/AtProto firehose 🦋
+A Ruby gem for streaming data from the Bluesky/AtProto firehose 🦋
+
+> [!NOTE]
+> ATProto Ruby gems collection: [skyfall](https://github.com/mackuba/skyfall) | [blue_factory](https://github.com/mackuba/blue_factory) | [minisky](https://github.com/mackuba/minisky) | [didkit](https://github.com/mackuba/didkit)
 
 
 ## What does it do
@@ -120,9 +123,36 @@ When Skyfall receives a message about a record type that's not on the list, whet
 Do not however check if such operations have a `type` equal to `:unknown` first - just ignore the type and only check the `collection` string. The reason is that some next version of Skyfall might start recognizing those records and add a new `type` value for them like e.g. `:skygram_photo`, and then they won't match your condition anymore.
 
 
+## Configuration
+
+### User agent
+
+`Skyfall::Stream` sends a user agent header when making a connection. This is set by default to `"Skyfall/0.x.y"`, but it's recommended that you override it using the `user_agent` field to something that identifies your app and its author – this will let the owner of the server you're connecting to know who to contact in case the client is causing some problems.
+
+You can also append your user agent info to the default value like this:
+
+```rb
+sky.user_agent = "NewsBot (@news.bot) #{sky.default_user_agent}"
+```
+
+### Heartbeat and reconnecting
+
+Occasionally, especially during times of very heavy traffic, the websocket can get into a stuck state where it stops receiving any data, but doesn't disconnect and just hangs like this forever. To work around this, there is a "heartbeat" feature which starts a background timer, which periodically checks how much time has passed since the last received event, and if the time exceeds a set limit, it manually disconnects and reconnects the stream.
+
+The option is not enabled by default, because there are some firehoses which will not be sending events often, possibly only once in a while – e.g. labellers and independent PDS firehoses – and in this case we don't want any heartbeat since it will be completely normal not to have any events for a long time. It's not really possible to detect easily if we're connecting to a full network relay or one of those, so in order to avoid false alarms, you need to enable this manually using the `check_heartbeat` property.
+
+You can also change the `heartbeat_interval`, i.e. how often the timer is triggered (default: 10s), and the `heartbeat_timeout`, i.e. the amount of time passed without events when it reconnects (default: 5 min):
+
+```rb
+sky.check_heartbeat = true
+sky.heartbeat_interval = 5
+sky.heartbeat_timeout = 120
+```
+
+
 ## Credits
 
-Copyright © 2023 Kuba Suder ([@mackuba.eu](https://bsky.app/profile/mackuba.eu)).
+Copyright © 2024 Kuba Suder ([@mackuba.eu](https://bsky.app/profile/mackuba.eu)).
 
 The code is available under the terms of the [zlib license](https://choosealicense.com/licenses/zlib/) (permissive, similar to MIT).
 

data/example/follower_tracker.rb

@@ -0,0 +1,84 @@
+#!/usr/bin/env ruby
+
+# Example: track when people follow and unfollow your account.
+
+# load skyfall from a local folder - you normally won't need this
+$LOAD_PATH.unshift(File.expand_path('../lib', __dir__))
+
+require 'json'
+require 'open-uri'
+require 'skyfall'
+
+$monitored_did = ARGV[0]
+
+if $monitored_did.to_s.empty?
+  puts "Usage: #{$PROGRAM_NAME} <monitored_did>"
+  exit 1
+elsif ARGV[0] !~ /^did:plc:[a-z0-9]{24}$/
+  puts "Not a valid DID: #{$monitored_did}"
+  exit 1
+end
+
+sky = Skyfall::Stream.new('bsky.network', :subscribe_repos)
+
+sky.on_connect { puts "Connected, monitoring #{$monitored_did}" }
+sky.on_disconnect { puts "Disconnected" }
+sky.on_reconnect { puts "Reconnecting..." }
+sky.on_error { |e| puts "ERROR: #{e}" }
+
+sky.on_message do |msg|
+  # we're only interested in repo commit messages
+  next if msg.type != :commit
+
+  msg.operations.each do |op|
+    next if op.action != :create
+
+    begin
+      case op.type
+      when :bsky_block
+        process_block(msg, op)
+      when :bsky_listitem
+        process_list_item(msg, op)
+      end
+    rescue StandardError => e
+      puts "Error: #{e}"
+    end
+  end
+end
+
+def process_block(msg, op)
+  if op.raw_record['subject'] == $monitored_did
+    owner_handle = get_user_handle(op.repo)
+    puts "@#{owner_handle} has blocked you! (#{msg.time.getlocal})"
+  end
+end
+
+def process_list_item(msg, op)
+  if op.raw_record['subject'] == $monitored_did
+    owner_handle = get_user_handle(op.repo)
+
+    list_uri = op.raw_record['list']
+    list_name = get_list_name(list_uri)
+
+    puts "@#{owner_handle} has added you to list \"#{list_name}\" (#{msg.time.getlocal})"
+  end
+end
+
+def get_user_handle(did)
+  url = "https://plc.directory/#{did}"
+  json = JSON.parse(URI.open(url).read)
+  json['alsoKnownAs'][0].gsub('at://', '')
+end
+
+def get_list_name(list_uri)
+  repo, type, rkey = list_uri.gsub('at://', '').split('/')
+  url = "https://bsky.social/xrpc/com.atproto.repo.getRecord?repo=#{repo}&collection=#{type}&rkey=#{rkey}"
+
+  json = JSON.parse(URI.open(url).read)
+  json['value']['name']
+end
+
+# close the connection cleanly on Ctrl+C
+trap("SIGINT") { sky.disconnect }
+
+sky.connect

data/lib/skyfall/collection.rb

@@ -1,16 +1,20 @@
 module Skyfall
   module Collection
-    BSKY_PROFILE    = "app.bsky.actor.profile"
-    BSKY_FEED       = "app.bsky.feed.generator"
-    BSKY_LIKE       = "app.bsky.feed.like"
-    BSKY_POST       = "app.bsky.feed.post"
-    BSKY_REPOST     = "app.bsky.feed.repost"
-    BSKY_THREADGATE = "app.bsky.feed.threadgate"
-    BSKY_BLOCK      = "app.bsky.graph.block"
-    BSKY_FOLLOW     = "app.bsky.graph.follow"
-    BSKY_LIST       = "app.bsky.graph.list"
-    BSKY_LISTBLOCK  = "app.bsky.graph.listblock"
-    BSKY_LISTITEM   = "app.bsky.graph.listitem"
-    BSKY_LABELER    = "app.bsky.labeler.service"
+    BSKY_PROFILE     = "app.bsky.actor.profile"
+    BSKY_FEED        = "app.bsky.feed.generator"
+    BSKY_LIKE        = "app.bsky.feed.like"
+    BSKY_POST        = "app.bsky.feed.post"
+    BSKY_POSTGATE    = "app.bsky.feed.postgate"
+    BSKY_REPOST      = "app.bsky.feed.repost"
+    BSKY_THREADGATE  = "app.bsky.feed.threadgate"
+    BSKY_BLOCK       = "app.bsky.graph.block"
+    BSKY_FOLLOW      = "app.bsky.graph.follow"
+    BSKY_LIST        = "app.bsky.graph.list"
+    BSKY_LISTBLOCK   = "app.bsky.graph.listblock"
+    BSKY_LISTITEM    = "app.bsky.graph.listitem"
+    BSKY_STARTERPACK = "app.bsky.graph.starterpack"
+    BSKY_LABELER     = "app.bsky.labeler.service"
+
+    BSKY_CHAT_DECLARATION = "chat.bsky.actor.declaration"
   end
 end

data/lib/skyfall/messages/account_message.rb

@@ -0,0 +1,11 @@
+module Skyfall
+  class AccountMessage < WebsocketMessage
+    def active?
+      @data_object['active']
+    end
+
+    def status
+      @data_object['status']&.to_sym
+    end
+  end
+end

data/lib/skyfall/messages/handle_message.rb

@@ -1,4 +1,9 @@
 module Skyfall
+
+  #
+  # Note: this event type is deprecated and will stop being emitted at some point.
+  # You should instead listen for 'identity' events (Skyfall::IdentityMessage).
+  #
   class HandleMessage < WebsocketMessage
     def handle
       @data_object['handle']

data/lib/skyfall/messages/identity_message.rb

@@ -1,4 +1,7 @@
 module Skyfall
   class IdentityMessage < WebsocketMessage
+    def handle
+      @data_object['handle']
+    end
   end
 end

data/lib/skyfall/messages/tombstone_message.rb

@@ -1,4 +1,9 @@
 module Skyfall
+
+  #
+  # Note: this event type is deprecated and will stop being emitted at some point.
+  # You should instead listen for 'account' events (Skyfall::AccountMessage).
+  #
   class TombstoneMessage < WebsocketMessage
   end
 end

data/lib/skyfall/messages/websocket_message.rb

@@ -8,6 +8,7 @@ module Skyfall
   class WebsocketMessage
     using Skyfall::Extensions
 
+    require_relative 'account_message'
     require_relative 'commit_message'
     require_relative 'handle_message'
     require_relative 'identity_message'
@@ -25,6 +26,7 @@ module Skyfall
       type_object, data_object = decode_cbor_objects(data)
 
       message_class = case type_object['t']
+        when '#account' then AccountMessage
         when '#commit' then CommitMessage
         when '#handle' then HandleMessage
         when '#identity' then IdentityMessage

data/lib/skyfall/operation.rb

@@ -43,18 +43,21 @@ module Skyfall
 
     def type
       case collection
-      when Collection::BSKY_BLOCK      then :bsky_block
-      when Collection::BSKY_FEED       then :bsky_feed
-      when Collection::BSKY_FOLLOW     then :bsky_follow
-      when Collection::BSKY_LABELER    then :bsky_labeler
-      when Collection::BSKY_LIKE       then :bsky_like
-      when Collection::BSKY_LIST       then :bsky_list
-      when Collection::BSKY_LISTBLOCK  then :bsky_listblock
-      when Collection::BSKY_LISTITEM   then :bsky_listitem
-      when Collection::BSKY_POST       then :bsky_post
-      when Collection::BSKY_PROFILE    then :bsky_profile
-      when Collection::BSKY_REPOST     then :bsky_repost
-      when Collection::BSKY_THREADGATE then :bsky_threadgate
+      when Collection::BSKY_BLOCK       then :bsky_block
+      when Collection::BSKY_FEED        then :bsky_feed
+      when Collection::BSKY_FOLLOW      then :bsky_follow
+      when Collection::BSKY_LABELER     then :bsky_labeler
+      when Collection::BSKY_LIKE        then :bsky_like
+      when Collection::BSKY_LIST        then :bsky_list
+      when Collection::BSKY_LISTBLOCK   then :bsky_listblock
+      when Collection::BSKY_LISTITEM    then :bsky_listitem
+      when Collection::BSKY_POST        then :bsky_post
+      when Collection::BSKY_POSTGATE    then :bsky_postgate
+      when Collection::BSKY_PROFILE     then :bsky_profile
+      when Collection::BSKY_REPOST      then :bsky_repost
+      when Collection::BSKY_STARTERPACK then :bsky_starterpack
+      when Collection::BSKY_THREADGATE  then :bsky_threadgate
+      when Collection::BSKY_CHAT_DECLARATION then :bsky_chat_declaration
       else :unknown
       end
     end

data/lib/skyfall/stream.rb

@@ -14,11 +14,12 @@ module Skyfall
       :subscribe_labels => SUBSCRIBE_LABELS
     }
 
-    EVENTS = %w(message raw_message connecting connect disconnect reconnect error)
+    EVENTS = %w(message raw_message connecting connect disconnect reconnect error timeout)
 
     MAX_RECONNECT_INTERVAL = 300
 
-    attr_accessor :heartbeat_timeout, :heartbeat_interval, :cursor, :auto_reconnect
+    attr_accessor :cursor, :auto_reconnect, :last_update, :user_agent
+    attr_accessor :heartbeat_timeout, :heartbeat_interval, :check_heartbeat
 
     def initialize(server, endpoint, cursor = nil)
       @endpoint = check_endpoint(endpoint)
@@ -26,7 +27,12 @@ module Skyfall
       @cursor = check_cursor(cursor)
       @handlers = {}
       @auto_reconnect = true
+      @check_heartbeat = false
       @connection_attempts = 0
+      @heartbeat_interval = 10
+      @heartbeat_timeout = 300
+      @last_update = nil
+      @user_agent = default_user_agent
 
       @handlers[:error] = proc { |e| puts "ERROR: #{e}" }
     end
@@ -47,15 +53,18 @@ module Skyfall
           @handlers[:error]&.call(e)
         end
 
-        @ws = Faye::WebSocket::Client.new(url)
+        @ws = Faye::WebSocket::Client.new(url, nil, { headers: { 'User-Agent' => user_agent }})
 
         @ws.on(:open) do |e|
           @handlers[:connect]&.call
+          @last_update = Time.now
+          start_heartbeat_timer
         end
 
         @ws.on(:message) do |msg|
           @reconnecting = false
           @connection_attempts = 0
+          @last_update = Time.now
 
           data = msg.data.pack('C*')
           @handlers[:raw_message]&.call(data)
@@ -85,6 +94,7 @@ module Skyfall
               connect
             end
           else
+            stop_heartbeat_timer
             @engines_on = false
             @handlers[:disconnect]&.call
             EM.stop_event_loop unless @ws
@@ -110,6 +120,40 @@ module Skyfall
 
     alias close disconnect
 
+    def default_user_agent
+      "Skyfall/#{Skyfall::VERSION}"
+    end
+
+    def check_heartbeat=(value)
+      @check_heartbeat = value
+
+      if @check_heartbeat && @engines_on && @ws && !@heartbeat_timer
+        start_heartbeat_timer
+      elsif !@check_heartbeat && @heartbeat_timer
+        stop_heartbeat_timer
+      end
+    end
+
+    def start_heartbeat_timer
+      return if !@check_heartbeat || @heartbeat_interval.to_f <= 0 || @heartbeat_timeout.to_f <= 0
+      return if @heartbeat_timer
+
+      @heartbeat_timer = EM::PeriodicTimer.new(@heartbeat_interval) do
+        next if @ws.nil? || @heartbeat_timeout.to_f <= 0
+        time_passed = Time.now - @last_update
+
+        if time_passed > @heartbeat_timeout
+          @handlers[:timeout]&.call
+          reconnect
+        end
+      end
+    end
+
+    def stop_heartbeat_timer
+      @heartbeat_timer&.cancel
+      @heartbeat_timer = nil
+    end
+
     EVENTS.each do |event|
       define_method "on_#{event}" do |&block|
         @handlers[event.to_sym] = block

data/lib/skyfall/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Skyfall
-  VERSION = "0.3.0"
+  VERSION = "0.4.0"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: skyfall
 version: !ruby/object:Gem::Version
-  version: 0.3.0
+  version: 0.4.0
 platform: ruby
 authors:
 - Kuba Suder
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2024-03-21 00:00:00.000000000 Z
+date: 2024-09-23 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: base32
@@ -30,6 +30,20 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: 0.3.4
+- !ruby/object:Gem::Dependency
+  name: base64
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.1'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.1'
 - !ruby/object:Gem::Dependency
   name: cbor
   requirement: !ruby/object:Gem::Requirement
@@ -84,62 +98,6 @@ dependencies:
     - - "~>"
       - !ruby/object:Gem::Version
         version: '0.11'
-- !ruby/object:Gem::Dependency
-  name: base64
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - "~>"
-      - !ruby/object:Gem::Version
-        version: '0.1'
-  type: :runtime
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - "~>"
-      - !ruby/object:Gem::Version
-        version: '0.1'
-- !ruby/object:Gem::Dependency
-  name: stringio
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - "~>"
-      - !ruby/object:Gem::Version
-        version: '3.0'
-  type: :runtime
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - "~>"
-      - !ruby/object:Gem::Version
-        version: '3.0'
-- !ruby/object:Gem::Dependency
-  name: time
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - "~>"
-      - !ruby/object:Gem::Version
-        version: '0.3'
-  type: :runtime
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - "~>"
-      - !ruby/object:Gem::Version
-        version: '0.3'
-- !ruby/object:Gem::Dependency
-  name: uri
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - "~>"
-      - !ruby/object:Gem::Version
-        version: '0.13'
-  type: :runtime
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - "~>"
-      - !ruby/object:Gem::Version
-        version: '0.13'
 description: "\n    Skyfall is a Ruby library for connecting to the \"firehose\" of
   the Bluesky social network, i.e. a websocket which\n    streams all new posts and
   everything else happening on the Bluesky network in real time. The code connects
@@ -156,6 +114,7 @@ files:
 - LICENSE.txt
 - README.md
 - example/block_tracker.rb
+- example/follower_tracker.rb
 - example/monitor_phrases.rb
 - example/print_all_posts.rb
 - example/push_notifications.rb
@@ -166,6 +125,7 @@ files:
 - lib/skyfall/errors.rb
 - lib/skyfall/extensions.rb
 - lib/skyfall/label.rb
+- lib/skyfall/messages/account_message.rb
 - lib/skyfall/messages/commit_message.rb
 - lib/skyfall/messages/handle_message.rb
 - lib/skyfall/messages/identity_message.rb