skyfall 0.0.1 → 0.1.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 677e5fd4ee93debfd3495c7c3504642c603eeda97ddba4273d1848d3240892f6
-  data.tar.gz: 0db72ec396022df1fb970a614bf217c2f20a04d843e6558eabc39dd74ddb8979
+  metadata.gz: 4045bf9dcd6a2ed5078cb557a64dc67c9e3d3c829a2909d9451c72c89f591c42
+  data.tar.gz: 1b2813d54fbd7cdd34334404f1e9607d05bbf923584009eedd0b630f8182fd7b
 SHA512:
-  metadata.gz: dc9c6ea0447c1f4310c1772ee093e88f98666ee925f6b81219c60b09a8a761f6c05a9a56449af57d270109148c3fd1a4d1420ceb8cd9a39f27d74d21c430e9cf
-  data.tar.gz: db9097d342156c20e043b59da778fc11cb2d9281fa2c3c432744b6b4fa81e0a505dda6a4670357bdbfb435f87e7436a77fb80953a74bb0c40a7a3df457a5259e
+  metadata.gz: 5c439017a0d277eb20977600847e74c8d52570ebb4eed3db5c39624d573b44dcc7b2404154131cf828f15df97b132a2b16de2eb608b6ed3a64e52c5a8f80b1e1
+  data.tar.gz: feadd80a468759c3a1a62b634e22ac4aa950bad708ffed6d7ed77b575a21c8841c01a0dc145da039f36203be0f92695d251f76e41de395d2296b4a1afd0e87fe

data/CHANGELOG.md

@@ -1,5 +1,10 @@
-## [Unreleased]
+## [0.1.0] - 2023-06-01
 
-## [0.1.0] - 2023-05-31
+- connecting to the firehose websocket
 
-- Initial release
+## [0.0.1] - 2023-05-31
+
+Initial release:
+
+- parsing CBOR objects from a websocket message
+- parsing CIDs, CAR archives and operation details from the objects

data/LICENSE.txt

@@ -0,0 +1,22 @@
+The zlib License
+
+Copyright (c) 2023 Jakub Suder
+
+This software is provided 'as-is', without any express or implied
+warranty.  In no event will the authors be held liable for any damages
+arising from the use of this software.
+
+Permission is granted to anyone to use this software for any purpose,
+including commercial applications, and to alter it and redistribute it
+freely, subject to the following restrictions:
+
+1. The origin of this software must not be misrepresented; you must not
+   claim that you wrote the original software. If you use this software
+   in a product, an acknowledgment in the product documentation would be
+   appreciated but is not required.
+
+2. Altered source versions must be plainly marked as such, and must not be
+   misrepresented as being the original software.
+
+3. This notice may not be removed or altered from any source distribution.
+

data/README.md

@@ -1,35 +1,95 @@
 # Skyfall
 
-TODO: Delete this and the text below, and describe your gem
+🌤 A Ruby gem for streaming data from the Bluesky/AtProto firehose 🦋
 
-Welcome to your new gem! In this directory, you'll find the files you need to be able to package up your Ruby library into a gem. Put your Ruby code in the file `lib/skyfall`. To experiment with that code, run `bin/console` for an interactive prompt.
 
-## Installation
+## What does it do
 
-TODO: Replace `UPDATE_WITH_YOUR_GEM_NAME_PRIOR_TO_RELEASE_TO_RUBYGEMS_ORG` with your gem name right after releasing it to RubyGems.org. Please do not do it earlier due to security reasons. Alternatively, replace this section with instructions to install your gem from git if you don't plan to release to RubyGems.org.
+Skyfall is a Ruby library for connecting to the *"firehose"* of the Bluesky social network, i.e. a websocket which
+streams all new posts and everything else happening on the Bluesky network in real time. The code connects to the
+websocket endpoint, decodes the messages which are encoded in some binary formats like DAG-CBOR, and returns the data as Ruby objects, which you can filter and save to some kind of database (e.g. in order to create a custom feed).
 
-Install the gem and add to the application's Gemfile by executing:
 
-    $ bundle add UPDATE_WITH_YOUR_GEM_NAME_PRIOR_TO_RELEASE_TO_RUBYGEMS_ORG
+## Installation
 
-If bundler is not being used to manage dependencies, install the gem by executing:
+    gem install skyfall
 
-    $ gem install UPDATE_WITH_YOUR_GEM_NAME_PRIOR_TO_RELEASE_TO_RUBYGEMS_ORG
 
 ## Usage
 
-TODO: Write usage instructions here
+Start a connection to the firehose by creating a `Skyfall::Stream` object, passing the server hostname and endpoint name:
+
+```rb
+require 'skyfall'
+
+sky = Skyfall::Stream.new('bsky.social', :subscribe_repos)
+```
+
+Add event listeners to handle incoming messages and get notified of errors:
+
+```rb
+sky.on_connect { puts "Connected" }
+sky.on_disconnect { puts "Disconnected" }
+
+sky.on_message { |m| p m }
+sky.on_error { |e| puts "ERROR: #{e}" }
+```
+
+When you're ready, open the connection by calling `connect`:
+
+```rb
+sky.connect
+```
+
+The connection is started asynchronously on a separate thread. If you're running this code in a simple script (and not as a part of a server), call e.g. `sleep` without arguments or `loop { STDIN.read }` to prevent the script for exiting while the connection is open.
+
+
+### Processing messages
+
+Each message passed to `on_message` is an instance of the `WebsocketMessage` class and has such properties:
+
+- `type` (symbol) - usually `:commit`
+- `seq` (sequential number)
+- `time` (Time)
+- `repo` (string) - DID of the repository (user account)
+- `commit` - CID
+- `operations` - list of operations (usually one)
+
+Operations are objects of type `Operation` and have such properties:
+
+- `repo` (string) - DID of the repository (user account)
+- `collection` (string) - name of the relevant collection in the repository, e.g. `app.bsky.feed.post` for posts
+- `path` (string) - the path part of the at:// URI - collection name + ID (rkey) of the item
+- `action` (symbol) - `:create`, `:update` or `:delete`
+- `uri` (string) - the at:// URI
+- `type` (symbol) - short name of the collection, e.g. `:bsky_post`
+- `cid` - CID
+
+Most operations will also have an attached record (JSON object) with details of the post, like etc. The record data is currently available as a Ruby hash via `raw_record` property (custom types will be added in a later version).
+
+So for example, in order to filter only "create post" operations and print their details, you can do something like this:
+
+```rb
+sky.on_message do |m|
+  next if m.type != :commit
+
+  m.operations.each do |op|
+    next unless op.action == :create && op.type == :bsky_post
 
-## Development
+    puts "#{op.repo}:"
+    puts op.raw_record['text']
+    puts
+  end
+end
+```
 
-After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
+See complete example in [example/firehose.rb](https://github.com/mackuba/skyfall/blob/master/example/firehose.rb).
 
-To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and the created tag, and push the `.gem` file to [rubygems.org](https://rubygems.org).
 
-## Contributing
+## Credits
 
-Bug reports and pull requests are welcome on GitHub at https://github.com/[USERNAME]/skyfall.
+Copyright © 2023 Kuba Suder ([@mackuba.eu](https://bsky.app/profile/mackuba.eu)).
 
-## License
+The code is available under the terms of the [zlib license](https://choosealicense.com/licenses/zlib/) (permissive, similar to MIT).
 
-The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).
+Bug reports and pull requests are welcome 😎

data/example/firehose.rb

@@ -0,0 +1,27 @@
+#!/usr/bin/env ruby
+
+lib = File.expand_path('../../lib', __FILE__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+
+require 'skyfall'
+
+sky = Skyfall::Stream.new('bsky.social', :subscribe_repos)
+
+sky.on_message do |m|
+  next if m.type != :commit
+
+  m.operations.each do |op|
+    next unless op.action == :create && op.type == :bsky_post
+
+    puts "#{op.repo}:"
+    puts op.raw_record['text']
+    puts
+  end
+end
+
+sky.on_connect { puts "Connected" }
+sky.on_disconnect { puts "Disconnected" }
+sky.on_error { |e| puts "ERROR: #{e}" }
+
+sky.connect
+sleep

data/lib/skyfall/operation.rb

@@ -2,7 +2,7 @@ require_relative 'collection'
 
 module Skyfall
   class Operation
-    attr_reader :repo, :path, :action, :cid, :record
+    attr_reader :repo, :path, :action, :cid
 
     def initialize(repo, path, action, cid, record)
       @repo = repo
@@ -12,6 +12,10 @@ module Skyfall
       @record = record
     end
 
+    def raw_record
+      @record
+    end
+
     def uri
       "at://#{repo}/#{path}"
     end

data/lib/skyfall/stream.rb

@@ -0,0 +1,101 @@
+require_relative 'websocket_message'
+require 'websocket-client-simple'
+
+module Skyfall
+  class Stream
+    SUBSCRIBE_REPOS = "com.atproto.sync.subscribeRepos"
+
+    NAMED_ENDPOINTS = {
+      :subscribe_repos => SUBSCRIBE_REPOS
+    }
+
+    def initialize(server, endpoint)
+      @endpoint = check_endpoint(endpoint)
+      @server = check_hostname(server)
+      @handlers = {}
+    end
+
+    def connect
+      return if @websocket
+
+      url = "wss://#{@server}/xrpc/#{@endpoint}"
+      handlers = @handlers
+
+      @websocket = WebSocket::Client::Simple.connect(url) do |ws|
+        ws.on :message do |msg|
+          handlers[:raw_message]&.call(msg.data)
+
+          if handlers[:message]
+            atp_message = Skyfall::WebsocketMessage.new(msg.data)
+            handlers[:message].call(atp_message)
+          end
+        end
+
+        ws.on :open do
+          handlers[:connect]&.call
+        end
+
+        ws.on :close do |e|
+          handlers[:disconnect]&.call(e)
+        end
+
+        ws.on :error do |e|
+          handlers[:error]&.call(e)
+        end
+      end
+    end
+
+    def disconnect
+      return unless @websocket
+
+      @websocket.close
+      @websocket = nil
+    end
+
+    alias close disconnect
+
+    def on_message(&block)
+      @handlers[:message] = block
+    end
+
+    def on_raw_message(&block)
+      @handlers[:raw_message] = block
+    end
+
+    def on_connect(&block)
+      @handlers[:connect] = block
+    end
+
+    def on_disconnect(&block)
+      @handlers[:disconnect] = block
+    end
+
+    def on_error(&block)
+      @handlers[:error] = block
+    end
+
+
+    private
+
+    def check_endpoint(endpoint)
+      if endpoint.is_a?(String)
+        raise ArgumentError("Invalid endpoint name: #{endpoint}") if endpoint.strip.empty? || !endpoint.include?('.')
+      elsif endpoint.is_a?(Symbol)
+        raise ArgumentError("Unknown endpoint: #{endpoint}") if NAMED_ENDPOINTS[endpoint].nil?
+        endpoint = NAMED_ENDPOINTS[endpoint]
+      else
+        raise ArgumentError("Endpoint should be a string or a symbol")
+      end
+    end
+
+    def check_hostname(server)
+      if server.is_a?(String)
+        raise ArgumentError("Invalid server name: #{server}") if server.strip.empty? || server.include?('/')
+      else
+        raise ArgumentError("Server name should be a string")
+      end
+
+      server
+    end
+  end
+end

data/lib/skyfall/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module Skyfall
-  VERSION = "0.0.1"
+  VERSION = "0.1.0"
 end

data/lib/skyfall/websocket_message.rb

@@ -11,6 +11,7 @@ module Skyfall
   class WebsocketMessage
     using Skyfall::Extensions
 
+    attr_reader :type_object, :data_object
     attr_reader :type, :repo, :time, :seq, :commit, :blocks, :operations
 
     def initialize(data)
@@ -25,6 +26,7 @@ module Skyfall
       raise UnsupportedError.new("Unexpected CBOR object: #{@type_object}") unless @type_object['op'] == 1
 
       @type = @type_object['t'][1..-1].to_sym
+      @operations = []
 
       @repo = @data_object['repo']
       @time = Time.parse(@data_object['time'])
@@ -44,5 +46,11 @@ module Skyfall
         Operation.new(@repo, path, action, cid, record)
       }
     end
+
+    def inspect
+      keys = instance_variables - [:@type_object, :@data_object, :@blocks]
+      vars = keys.map { |v| "#{v}=#{instance_variable_get(v).inspect}" }.join(", ")
+      "#<#{self.class}:0x#{object_id} #{vars}>"
+    end
   end
 end

data/lib/skyfall.rb

@@ -1,4 +1,5 @@
 # frozen_string_literal: true
 
+require_relative 'skyfall/stream'
 require_relative 'skyfall/websocket_message'
 require_relative 'skyfall/version'

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: skyfall
 version: !ruby/object:Gem::Version
-  version: 0.0.1
+  version: 0.1.0
 platform: ruby
 authors:
 - Kuba Suder
@@ -10,10 +10,33 @@ bindir: bin
 cert_chain: []
 date: 2023-06-01 00:00:00.000000000 Z
 dependencies:
+- !ruby/object:Gem::Dependency
+  name: base32
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.3'
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 0.3.4
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.3'
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 0.3.4
 - !ruby/object:Gem::Dependency
   name: cbor
   requirement: !ruby/object:Gem::Requirement
     requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.5'
     - - ">="
       - !ruby/object:Gem::Version
         version: 0.5.9.6
@@ -21,37 +44,48 @@ dependencies:
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.5'
     - - ">="
       - !ruby/object:Gem::Version
         version: 0.5.9.6
 - !ruby/object:Gem::Dependency
-  name: base32
+  name: websocket-client-simple
   requirement: !ruby/object:Gem::Requirement
     requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.6'
     - - ">="
       - !ruby/object:Gem::Version
-        version: 0.3.4
+        version: 0.6.1
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.6'
     - - ">="
       - !ruby/object:Gem::Version
-        version: 0.3.4
-description:
+        version: 0.6.1
+description: "\n    Skyfall is a Ruby library for connecting to the \"firehose\" of
+  the Bluesky social network, i.e. a websocket which\n    streams all new posts and
+  everything else happening on the Bluesky network in real time. The code connects
+  to the\n    websocket endpoint, decodes the messages which are encoded in some binary
+  formats, and returns the data as Ruby\n    objects, which you can filter and save
+  to some kind of database (e.g. in order to create a custom feed).\n  "
 email:
 - jakub.suder@gmail.com
 executables: []
 extensions: []
 extra_rdoc_files: []
 files:
-- ".rspec"
 - CHANGELOG.md
-- Gemfile
-- Gemfile.lock
-- MIT-LICENSE.txt
+- LICENSE.txt
 - README.md
-- Rakefile
+- example/firehose.rb
 - lib/skyfall.rb
 - lib/skyfall/car_archive.rb
 - lib/skyfall/cid.rb
@@ -59,14 +93,17 @@ files:
 - lib/skyfall/errors.rb
 - lib/skyfall/extensions.rb
 - lib/skyfall/operation.rb
+- lib/skyfall/stream.rb
 - lib/skyfall/version.rb
 - lib/skyfall/websocket_message.rb
 - sig/skyfall.rbs
-- skyfall.gemspec
 homepage: https://github.com/mackuba/skyfall
 licenses:
-- MIT
-metadata: {}
+- Zlib
+metadata:
+  bug_tracker_uri: https://github.com/mackuba/skyfall/issues
+  changelog_uri: https://github.com/mackuba/skyfall/blob/master/CHANGELOG.md
+  source_code_uri: https://github.com/mackuba/skyfall
 post_install_message:
 rdoc_options: []
 require_paths:

data/.rspec

@@ -1,3 +0,0 @@
---format documentation
---color
---require spec_helper

data/Gemfile

@@ -1,10 +0,0 @@
-# frozen_string_literal: true
-
-source "https://rubygems.org"
-
-# Specify your gem's dependencies in skyfall.gemspec
-gemspec
-
-gem "rake", "~> 13.0"
-
-gem "rspec", "~> 3.0"

data/Gemfile.lock

@@ -1,38 +0,0 @@
-PATH
-  remote: .
-  specs:
-    skyfall (0.1.0)
-      base32 (>= 0.3.4)
-      cbor (>= 0.5.9.6)
-
-GEM
-  remote: https://rubygems.org/
-  specs:
-    base32 (0.3.4)
-    cbor (0.5.9.6)
-    diff-lcs (1.5.0)
-    rake (13.0.6)
-    rspec (3.12.0)
-      rspec-core (~> 3.12.0)
-      rspec-expectations (~> 3.12.0)
-      rspec-mocks (~> 3.12.0)
-    rspec-core (3.12.2)
-      rspec-support (~> 3.12.0)
-    rspec-expectations (3.12.3)
-      diff-lcs (>= 1.2.0, < 2.0)
-      rspec-support (~> 3.12.0)
-    rspec-mocks (3.12.5)
-      diff-lcs (>= 1.2.0, < 2.0)
-      rspec-support (~> 3.12.0)
-    rspec-support (3.12.0)
-
-PLATFORMS
-  arm64-darwin-21
-
-DEPENDENCIES
-  rake (~> 13.0)
-  rspec (~> 3.0)
-  skyfall!
-
-BUNDLED WITH
-   2.4.13

data/MIT-LICENSE.txt

@@ -1,21 +0,0 @@
-The MIT License (MIT)
-
-Copyright (c) 2023 Kuba Suder
-
-Permission is hereby granted, free of charge, to any person obtaining a copy
-of this software and associated documentation files (the "Software"), to deal
-in the Software without restriction, including without limitation the rights
-to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-copies of the Software, and to permit persons to whom the Software is
-furnished to do so, subject to the following conditions:
-
-The above copyright notice and this permission notice shall be included in
-all copies or substantial portions of the Software.
-
-THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
-THE SOFTWARE.

data/Rakefile

@@ -1,8 +0,0 @@
-# frozen_string_literal: true
-
-require "bundler/gem_tasks"
-require "rspec/core/rake_task"
-
-RSpec::Core::RakeTask.new(:spec)
-
-task default: :spec

data/skyfall.gemspec

@@ -1,35 +0,0 @@
-# frozen_string_literal: true
-
-require_relative "lib/skyfall/version"
-
-Gem::Specification.new do |spec|
-  spec.name = "skyfall"
-  spec.version = Skyfall::VERSION
-  spec.authors = ["Kuba Suder"]
-  spec.email = ["jakub.suder@gmail.com"]
-
-  spec.summary = "A Ruby gem for streaming data from the Bluesky/AtProto firehose"
-  spec.homepage = "https://github.com/mackuba/skyfall"
-
-  # spec.description = "TODO: Write a longer description or delete this line."
-
-  spec.license = "MIT"
-  spec.required_ruby_version = ">= 2.6.0"
-
-  # spec.metadata["homepage_uri"] = spec.homepage
-  # spec.metadata["source_code_uri"] = "TODO: Put your gem's public repo URL here."
-  # spec.metadata["changelog_uri"] = "TODO: Put your gem's CHANGELOG.md URL here."
-
-  # Specify which files should be added to the gem when it is released.
-  # The `git ls-files -z` loads the files in the RubyGem that have been added into git.
-  spec.files = Dir.chdir(__dir__) do
-    `git ls-files -z`.split("\x0").reject do |f|
-      (File.expand_path(f) == __FILE__) || f.start_with?(*%w[bin/ test/ spec/ features/ .git .circleci appveyor])
-    end
-  end
-
-  spec.require_paths = ["lib"]
-
-  spec.add_dependency 'cbor', '>= 0.5.9.6'
-  spec.add_dependency 'base32', '>= 0.3.4'
-end