skyfall 0.4.0 → 0.5.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: db537537fcd4e38f184c6cdea3f51ba1b8554848d4c3cd1de20f63ca2d956fc2
-  data.tar.gz: 34c51f6c8c0152589562bf0a3b3c5246cd8a1e95664f1128c86bd17c6a313d59
+  metadata.gz: d272120c19a97451a6df364e1a7dbbb8191dc0247fd929129ecab0cf05902599
+  data.tar.gz: 242ebbbbf44aee36a883c2fc78be327d29c14fa27168fc1eac66ef38ad16cc16
 SHA512:
-  metadata.gz: 11477293a1bc0377ef5d9eaa68cc2d374f76ac4d942006572e8b9e1668cb8f54a582abdeda81b65c1d72f6452ba2cad1153c403027ba5d7efbfa093624713105
-  data.tar.gz: 8d86b1f71a4fd5fa7f01077fefe8d8e292dc6b5007b79ee4d1ed1343f887c4104c0480b297e7ce4be61db0cce65211a568b642411fa5c3e7f62c4c91a4fe0f7b
+  metadata.gz: cebb5837133466638a7009d52be516af961644c55f91fe3f3e9a6fe120bc8875f614bbf849cea02fe23b81f3f1f4652226214d375adff12198faf3dd81c1bedd
+  data.tar.gz: f1c8c48d58c344a67f4df27afc792b6d38272ecb0aefdf7cf4336269290e36c788d356547c2af0520a9d11664e62fb872157da88951cb669f7081bb0414cc029

data/CHANGELOG.md

@@ -1,3 +1,22 @@
+## [0.5.0] - 2024-11-15
+
+Jetstream support! You can now connect to [Jetstream](https://github.com/bluesky-social/jetstream) sources using `Skyfall::Jetstream` (see readme).
+
+This required some breaking changes in the existing API:
+
+- `Skyfall::Stream` has been renamed to `Skyfall::Firehose`, `Skyfall::Stream` is now a base class of both `Firehose` and `Jetstream`; the existing `Skyfall::Stream` constructor works for now but will be removed soon
+- `Skyfall::WebsocketMessage` and its subclasses have been separated into two parallel families under `Skyfall::Firehose` and `Skyfall::Jetstream`, with the base classes just named `Message`
+- same thing happened with `Skyfall::Operation`
+- `data_object` and `type_object` properties in `WebsocketMessage` are considered semi-private API now ("nodoc")
+
+In most cases, you should only need to update the `Skyfall::Stream` class name in the constructor. If you've referenced message classes like `Skyfall::CommitMessage` directly, it's probably better to just check the `#type` property instead.
+
+Also, small change to the user agent API: `Skyfall::Stream` now has an additional metod `version_string`, which will always return `Skyfall/0.x.y` - it's recommended to use that instead of `default_user_agent` to build your own user agent string that includes the library version. `default_user_agent` now passes through to `version_string`, but it could be changed in future to return something else.
+
+## [0.4.1] - 2024-10-04
+
+- performance fix - don't decode CAR sections which aren't needed, which is most of them; this cuts the amount of memory that GC has to free up by about one third, and should speed up processing by around ~10%
+
 ## [0.4.0] - 2024-09-23
 
 - (re)added a "hearbeat" feature (removed earlier in 0.2.0) to fix the occasional issue when the websocket stops receiving data, but doesn't disconnect (not enabled by default, turn it on by setting `check_heartbeat` to true)

data/README.md

@@ -1,6 +1,6 @@
 # Skyfall
 
-A Ruby gem for streaming data from the Bluesky/AtProto firehose 🦋
+A Ruby gem for streaming data from the Bluesky/ATProto firehose 🦋
 
 > [!NOTE]
 > ATProto Ruby gems collection: [skyfall](https://github.com/mackuba/skyfall) | [blue_factory](https://github.com/mackuba/blue_factory) | [minisky](https://github.com/mackuba/minisky) | [didkit](https://github.com/mackuba/didkit)
@@ -8,57 +8,140 @@ A Ruby gem for streaming data from the Bluesky/AtProto firehose 🦋
 
 ## What does it do
 
-Skyfall is a Ruby library for connecting to the *"firehose"* of the Bluesky social network, i.e. a websocket which
-streams all new posts and everything else happening on the Bluesky network in real time. The code connects to the
-websocket endpoint, decodes the messages which are encoded in some binary formats like DAG-CBOR, and returns the data as Ruby objects, which you can filter and save to some kind of database (e.g. in order to create a custom feed).
+Skyfall is a Ruby library for connecting to the *"[firehose](https://atproto.com/specs/event-stream)"* of the Bluesky social network, i.e. a websocket which streams all new posts and everything else happening on the Bluesky network in real time. The code connects to the websocket endpoint, decodes the messages which are encoded in some binary formats like DAG-CBOR, and returns the data as Ruby objects, which you can filter and save to some kind of database (e.g. in order to create a custom feed).
+
+Since version 0.5, Skyfall also supports connecting to [Jetstream](https://github.com/bluesky-social/jetstream/) sources, which serve the same kind of stream, but as JSON messages instead of CBOR.
 
 
 ## Installation
 
+From the command line:
+
     gem install skyfall
 
+Or, add this to your `Gemfile`:
+
+    gem 'skyfall', '~> 0.5'
+
 
 ## Usage
 
-Start a connection to the firehose by creating a `Skyfall::Stream` object, passing the server hostname and endpoint name:
+### Standard ATProto firehose
+
+To connect to the firehose, start by creating a `Skyfall::Firehose` object, specifying the server hostname and endpoint name:
 
 ```rb
 require 'skyfall'
 
-sky = Skyfall::Stream.new('bsky.network', :subscribe_repos)
+sky = Skyfall::Firehose.new('bsky.network', :subscribe_repos)
 ```
 
-Add event listeners to handle incoming messages and get notified of errors:
+The server name can be just a hostname, or a full URL with a `ws:` or `wss:` scheme, which is useful if you want to use a non-encrypted websocket connection, e.g. `"ws://localhost:8000"`. The endpoint can be either a full NSID string like `"com.atproto.sync.subscribeRepos"`, or one of the defined symbol shortcuts - you will almost always want to pass `:subscribe_repos` here.
+
+Next, set up event listeners to handle incoming messages and get notified of errors. Here are all the available listeners (you will need at least either `on_message` or `on_raw_message`):
 
 ```rb
+# this gives you a parsed message object, one of subclasses of Skyfall::Firehose::Message
+sky.on_message { |msg| p msg }
+
+# this gives you raw binary data as received from the websocket
+sky.on_raw_message { |data| p data }
+
+# lifecycle events
+sky.on_connecting { |url| puts "Connecting to #{url}..." }
 sky.on_connect { puts "Connected" }
 sky.on_disconnect { puts "Disconnected" }
+sky.on_reconnect { puts "Connection lost, trying to reconnect..." }
+sky.on_timeout { puts "Connection stalled, triggering a reconnect..." }
 
-sky.on_message { |m| p m }
+# handling errors (there's a default error handler that does exactly this)
 sky.on_error { |e| puts "ERROR: #{e}" }
 ```
 
+You can also call these as setters accepting a `Proc` - e.g. to disable default error handling, you can do:
+
+```rb
+sky.on_error = nil
+```
+
 When you're ready, open the connection by calling `connect`:
 
 ```rb
 sky.connect
 ```
 
+The `#connect` method blocks until the connection is explicitly closed with `#disconnect` from an event or interrupt handler. Skyfall uses [EventMachine](https://github.com/eventmachine/eventmachine) under the hood, so in order to run some things in parallel, you can use e.g. `EM::PeriodicTimer`.
+
+
+### Using a Jetstream source
+
+Alternatively, you can connect to a [Jetstream](https://github.com/bluesky-social/jetstream/) server. Jetstream is a firehose proxy that lets you stream data as simple JSON instead, which uses much less bandwidth, and allows you to pick only a subset of events that you're interested in, e.g. only posts or only from specific accounts. (See the [configuration section](#jetstream-filters) for more info on Jetstream filtering.)
+
+Jetstream connections are made using a `Skyfall::Jetstream` instance, which has more or less the same API as `Skyfall::Firehose`, so it should be possible to switch between those by just changing the line that creates the client instance:
+
+```rb
+sky = Skyfall::Jetstream.new('jetstream1.us-east.bsky.network')
+
+sky.on_message { |msg| ... }
+sky.on_error { |e| ... }
+sky.on_connect { ... }
+...
+
+sky.connect
+```
+
+### Cursors
+
+ATProto websocket endpoints implement a "*cursor*" feature to help you make sure that you don't miss anything if your connection is down for a bit (because of a network issue, server restart, deploy etc.). Each message includes a `seq` field, which is the sequence number of the event. You can keep track of the last seq you've seen, and when you reconnect, you pass that number as a cursor parameter - the server will then "replay" all events you might have missed since that last one. (The `bsky.network` Relay firehose currently has a buffer of about 72 hours, though that's not something required by specification.)
+
+To use a cursor when connecting to the firehose, pass it as the third parameter to `Skyfall::Firehose`. You should then regularly save the `seq` of the last event to some permanent storage, and then load it from there when reconnecting.
+
+A full-network firehose sends many hundreds of events per second, so depending on your use case, it might be enough if you save it every n events (e.g. every 100 or 1000) and on clean shutdown:
+
+```rb
+cursor = load_cursor
+
+sky = Skyfall::Firehose.new('bsky.network', :subscribe_repos, cursor)
+sky.on_message do |msg|
+  save_cursor(msg.seq) if msg.seq % 1000 == 0
+  process_message(msg)
+end
+```
+
+Jetstream has a similar mechanism, except the cursor is the event's timestamp in Unix time microseconds instead of just a number incrementing by 1. For `Skyfall::Jetstream`, pass the cursor as a key in an options hash:
+
+```rb
+cursor = load_cursor
+
+sky = Skyfall::Jetstream.new('jetstream1.us-east.bsky.network', { cursor: cursor })
+sky.on_message do |msg|
+  save_cursor(msg.seq)
+  process_message(msg)
+end
+```
+
 
 ### Processing messages
 
-Each message passed to `on_message` is an instance of a subclass of `WebsocketMessage`, depending on the message type. The supported message types are:
+Each message passed to `on_message` is an instance of a subclass of either `Skyfall::Firehose::Message` or `Skyfall::Jetstream::Message`, depending on the selected source. The supported message types are:
 
 - `CommitMessage` (`#commit`) - represents a change in a user's repo; most messages are of this type
-- `HandleMessage` (`#handle`) - when a different handle is assigned to a user's DID
-- `TombstoneMessage` (`#tombstone`) - when an account is deleted
+- `IdentityMessage` (`#identity`) - notifies about a change in user's DID document, e.g. a handle change or a migration to a new PDS
+- `AccountMessage` (`#account`) - notifies about a change of an account's status (de/activation, suspension, deletion)
+- `HandleMessage` (`#handle` - deprecated) - when a different handle is assigned to a user's DID
+- `TombstoneMessage` (`#tombstone` - deprecated) - when an account is deleted
+- `LabelsMessage` (`#labels`) - only used in `subscribe_labels` endpoint
 - `InfoMessage` (`#info`) - a protocol error message, e.g. about an invalid cursor parameter
 - `UnknownMessage` is used for other unrecognized message types
 
-All message objects have the following properties:
+`#handle` and `#tombstone` events are considered deprecated, replaced by `#identity` and `#account` respectively. They are still being emitted at the moment (in parallel with the newer event types), but they might stop being sent at any moment, so it's recommended that you don't rely on those.
+
+`Skyfall::Firehose::Message` and `Skyfall::Jetstream::Message` variants of message classes should have more or less the same interface, except when a given field is not included in one of the formats.
+
+All message objects have the following shared properties:
 
 - `type` (symbol) - the message type identifier, e.g. `:commit`
-- `seq` (integer) - a sequential index of the message
+- `seq` (integer) - a sequential index of the message; Jetstream messages instead have a `time_us` value, which is a Unix timestamp in microseconds (also aliased as `seq` for compatibility)
 - `repo` or `did` (string) - DID of the repository (user account)
 - `time` (Time) - timestamp of the described action
 
@@ -67,13 +150,17 @@ All properties except `type` may be nil for some message types that aren't relat
 Commit messages additionally have:
 
 - `commit` - CID of the commit
-- `prev` - CID of the previous commit in that repo
 - `operations` - list of operations (usually one)
 
-Handle messages additionally have:
+Handle and Identity messages additionally have:
 
 - `handle` - the new handle assigned to the DID
 
+Account messages additionally have:
+
+- `active?` - whether the account is active, or inactive for any reason
+- `status` - if not active, shows the status of the account (`:deactivated`, `:deleted`, `:takendown`)
+
 Info messages additionally have:
 
 - `name` - identifier of the message/error
@@ -82,7 +169,7 @@ Info messages additionally have:
 
 ### Commit operations
 
-Operations are objects of type `Operation` and have such properties:
+Operations are objects of type `Skyfall::Firehose::Operation` or `Skyfall::Jetstream::Operation` and have such properties:
 
 - `repo` or `did` (string) - DID of the repository (user account)
 - `collection` (string) - name of the relevant collection in the repository, e.g. `app.bsky.feed.post` for posts
@@ -91,7 +178,7 @@ Operations are objects of type `Operation` and have such properties:
 - `path` (string) - the path part of the at:// URI - collection name + ID (rkey) of the item
 - `uri` (string) - the complete at:// URI
 - `action` (symbol) - `:create`, `:update` or `:delete`
-- `cid` - CID of the operation/record (`nil` for delete operations)
+- `cid` (CID) - CID of the operation/record (`nil` for delete operations)
 
 Create and update operations will also have an attached record (JSON object) with details of the post, like etc. The record data is currently available as a Ruby hash via `raw_record` property (custom types will be added in future).
 
@@ -114,34 +201,33 @@ end
 For more examples, see the [example](https://github.com/mackuba/skyfall/blob/master/example) folder or the [bluesky-feeds-rb](https://github.com/mackuba/bluesky-feeds-rb/blob/master/app/firehose_stream.rb) project, which implements a feed generator service.
 
 
-### Custom lexicons
+### Note on custom lexicons
 
-A note on custom lexicons: the `Skyfall::Operation` objects have two properties that tell you the kind of record they're about: `#collection`, which is a string containing the official name of the collection/lexicon, e.g. `"app.bsky.feed.post"`; and `#type`, which is a symbol meant to save you some typing, e.g. `:bsky_post`.
+Note that the `Operation` objects have two properties that tell you the kind of record they're about: `#collection`, which is a string containing the official name of the collection/lexicon, e.g. `"app.bsky.feed.post"`; and `#type`, which is a symbol meant to save you some typing, e.g. `:bsky_post`.
 
 When Skyfall receives a message about a record type that's not on the list, whether in the `app.bsky` namespace or not, the operation `type` will be `:unknown`, while the `collection` will be the original string. So if an app like e.g. "Skygram" appears with a `zz.skygram.*` namespace that lets you share photos on ATProto, the operations will have a type `:unknown` and collection names like `zz.skygram.feed.photo`, and you can check the `collection` field for record types known to you and process them in some appropriate way, even if Skyfall doesn't recognize the record type.
 
 Do not however check if such operations have a `type` equal to `:unknown` first - just ignore the type and only check the `collection` string. The reason is that some next version of Skyfall might start recognizing those records and add a new `type` value for them like e.g. `:skygram_photo`, and then they won't match your condition anymore.
 
 
-## Configuration
+## Reconnection logic
 
-### User agent
+In a perfect world, the websocket would never disconnect until you disconnect it, but unfortunately we don't live in a perfect world. The socket sometimes disconnects or stops responding, and Skyfall has some built-in protections to make sure it can operate without much oversight.
 
-`Skyfall::Stream` sends a user agent header when making a connection. This is set by default to `"Skyfall/0.x.y"`, but it's recommended that you override it using the `user_agent` field to something that identifies your app and its author – this will let the owner of the server you're connecting to know who to contact in case the client is causing some problems.
 
-You can also append your user agent info to the default value like this:
+### Broken connections
 
-```rb
-sky.user_agent = "NewsBot (@news.bot) #{sky.default_user_agent}"
-```
+If the connection is randomly closed for some reason, Skyfall will by default try to reconnect automatically. If the reconnection fails (e.g. because the network is down), it will wait with an [exponential backoff](https://en.wikipedia.org/wiki/Exponential_backoff) up to 5 minute intervals and keep retrying forever until it connects again. The `on_reconnect` callback is triggered when the connection is closed (before the wait delay). This mechanism should generally solve most of the problem.
+
+The auto reconnecting feature is enabled by default, but you can turn it off by setting `auto_reconnect` to `false`.
 
-### Heartbeat and reconnecting
+### Stalled connections & heartbeat
 
 Occasionally, especially during times of very heavy traffic, the websocket can get into a stuck state where it stops receiving any data, but doesn't disconnect and just hangs like this forever. To work around this, there is a "heartbeat" feature which starts a background timer, which periodically checks how much time has passed since the last received event, and if the time exceeds a set limit, it manually disconnects and reconnects the stream.
 
-The option is not enabled by default, because there are some firehoses which will not be sending events often, possibly only once in a while – e.g. labellers and independent PDS firehoses – and in this case we don't want any heartbeat since it will be completely normal not to have any events for a long time. It's not really possible to detect easily if we're connecting to a full network relay or one of those, so in order to avoid false alarms, you need to enable this manually using the `check_heartbeat` property.
+This feature is not enabled by default, because there are some firehoses which will not be sending events often, possibly only once in a while – e.g. labellers and independent PDS firehoses – and in this case we don't want any heartbeat since it will be completely normal not to have any events for a long time. It's not really possible to detect easily if we're connecting to a full network relay or one of those, so in order to avoid false alarms, you need to enable this manually using the `check_heartbeat` property.
 
-You can also change the `heartbeat_interval`, i.e. how often the timer is triggered (default: 10s), and the `heartbeat_timeout`, i.e. the amount of time passed without events when it reconnects (default: 5 min):
+You can also change the `heartbeat_interval`, i.e. how often the timer is triggered (default: 10s), and the `heartbeat_timeout`, i.e. the amount of time passed without events needed to cause a reconnect (default: 5 min):
 
 ```rb
 sky.check_heartbeat = true
@@ -149,6 +235,73 @@ sky.heartbeat_interval = 5
 sky.heartbeat_timeout = 120
 ```
 
+### Cursors when reconnecting
+
+Skyfall keeps track of the last event's `seq` internally in the `cursor` property, so if the client reconnects for whatever reason, it will automatically use the latest cursor in the URL.
+
+> [!NOTE]
+> This only happens if you use the `on_message` callback and not `on_raw_message`, since the event is not parsed from binary data into a `Message` object if you use `on_raw_message`, so Skyfall won't have access to the `seq` field then.
+
+
+## Streaming from labellers
+
+Apart from `subscribe_repos`, there is a second endpoint `subscribe_labels`, which is used to stream labels from [labellers](https://atproto.com/specs/label) (ATProto moderation services). This endpoint only sends `#labels` events (and possibly `#info`).
+
+To connect to a labeller, pass `:subscribe_labels` as the endpoint name to `Skyfall::Firehose`. The `on_message` callback will get called with `Skyfall::Firehose::LabelsMessage` events, each of which includes one or more labels as `Skyfall::Label`:
+
+```rb
+cursor = load_cursor(service)
+sky = Skyfall::Firehose.new(service, :subscribe_labels, cursor)
+sky.on_message do |msg|
+  if msg.type == :labels
+    msg.labels.each do |l|
+      puts "[#{l.created_at}] #{l.subject} => #{l.value}"
+    end
+  end
+end
+```
+
+See [ATProto label docs](https://atproto.com/specs/label) for info on what fields are included with each label - `Skyfall::Label` includes properties with these original names, and also more friendly aliases for each (e.g. `value` instead of `val`).
+
+
+## Other configuration
+
+### User agent
+
+Skyfall sends a user agent header when making a connection. This is set by default to `"Skyfall/0.x.y"`, but it's recommended that you override it using the `user_agent` field to something that identifies your app and its author – this will let the owner of the server you're connecting to know who to contact in case the client is causing some problems.
+
+You can also append your user agent info to the default value like this:
+
+```rb
+sky.user_agent = "NewsBot (@news.bot) #{sky.version_string}"
+```
+
+### Jetstream filters
+
+Jetstream allows you to specify [filters](https://github.com/bluesky-social/jetstream?tab=readme-ov-file#consuming-jetstream) of collection types and/or tracked DIDs when you connect, so it will send you only the events you're interested in. You can e.g. ask only for posts and ignore likes, or only profile events and ignore everything else, or only listen for posts from a few specific accounts.
+
+To use these filters, pass the "wantedCollections" and/or "wantedDids" parameters in the options hash when initializing `Skyfall::Jetstream`. You can use the original JavaScript param names, or a more Ruby-like snake_case form:
+
+```rb
+sky = Skyfall::Jetstream.new('jetstream1.us-east.bsky.network', {
+  wanted_collections: 'app.bsky.feed.post',
+  wanted_dids: @dids
+})
+```
+
+For collections, you can also use the symbol codes used in `Operation#type`, e.g. `:bsky_post`:
+
+```rb
+sky = Skyfall::Jetstream.new('jetstream1.us-east.bsky.network', {
+  wanted_collections: [:bsky_post]
+})
+```
+
+See [Jetstream docs](https://github.com/bluesky-social/jetstream?tab=readme-ov-file#consuming-jetstream) for more info on available filters.
+
+> [!NOTE]
+> The `compress` and `requireHello` options (and zstd compression) are not available at the moment. Also the "subscriber sourced messages" aren't implemented yet.
+
 
 ## Credits
 

data/example/block_tracker.rb

@@ -19,7 +19,7 @@ elsif ARGV[0] !~ /^did:plc:[a-z0-9]{24}$/
   exit 1
 end
 
-sky = Skyfall::Stream.new('bsky.network', :subscribe_repos)
+sky = Skyfall::Firehose.new('bsky.network', :subscribe_repos)
 
 sky.on_connect { puts "Connected, monitoring #{$monitored_did}" }
 sky.on_disconnect { puts "Disconnected" }

data/example/{monitor_phrases.rb → jet_monitor_phrases.rb}

@@ -1,7 +1,7 @@
 #!/usr/bin/env ruby
 
 # Example: monitor new posts for mentions of one or more words or phrases (e.g. anyone mentioning your name or the name
-# of your company, project etc.).
+# of your company, project etc.). This example uses a Jetstream connection.
 
 # load skyfall from a local folder - you normally won't need this
 $LOAD_PATH.unshift(File.expand_path('../lib', __dir__))
@@ -17,7 +17,8 @@ if terms.empty?
   exit 1
 end
 
-sky = Skyfall::Stream.new('bsky.network', :subscribe_repos)
+# tell Jetstream to send us only post records
+sky = Skyfall::Jetstream.new('jetstream1.us-east.bsky.network', { wanted_collections: [:bsky_post] })
 
 sky.on_message do |msg|
   # we're only interested in repo commit messages

data/example/print_all_posts.rb

@@ -7,7 +7,7 @@ $LOAD_PATH.unshift(File.expand_path('../lib', __dir__))
 
 require 'skyfall'
 
-sky = Skyfall::Stream.new('bsky.network', :subscribe_repos)
+sky = Skyfall::Firehose.new('bsky.network', :subscribe_repos)
 
 sky.on_message do |msg|
   # we're only interested in repo commit messages

data/example/push_notifications.rb

@@ -45,7 +45,7 @@ class NotificationEngine
   end
 
   def connect
-    @sky = Skyfall::Stream.new('bsky.network', :subscribe_repos)
+    @sky = Skyfall::Firehose.new('bsky.network', :subscribe_repos)
 
     @sky.on_connect { puts "Connected, monitoring #{@user_did}" }
     @sky.on_disconnect { puts "Disconnected" }

data/lib/skyfall/car_archive.rb

@@ -11,11 +11,15 @@ require 'stringio'
 
 module Skyfall
   class CarSection
-    attr_reader :cid, :body
+    attr_reader :cid
 
-    def initialize(cid, body)
+    def initialize(cid, body_data)
       @cid = cid
-      @body = body
+      @body_data = body_data
+    end
+
+    def body
+      @body ||= CarArchive.convert_data(CBOR.decode(@body_data))
     end
   end
 
@@ -37,9 +41,7 @@ module Skyfall
       section && section.body
     end
 
-    private
-
-    def convert_data(object)
+    def self.convert_data(object)
       if object.is_a?(Hash)
         object.each do |k, v|
           if v.is_a?(Hash) || v.is_a?(Array)
@@ -65,14 +67,16 @@ module Skyfall
       end
     end
 
-    def make_cid_link(cid)
+    def self.make_cid_link(cid)
       { '$link' => CID.from_cbor_tag(cid) }
     end
 
-    def make_bytes(data)
+    def self.make_bytes(data)
       { '$bytes' => Base64.encode64(data).chomp.gsub(/=+$/, '') }
     end
 
+    private
+
     def read_header(buffer)
       len = buffer.read_varint
 
@@ -112,10 +116,7 @@ module Skyfall
       cid = CID.new(prefix + cid_data)
 
       body_data = sbuffer.read
-      body = CBOR.decode(body_data)
-      convert_data(body)
-
-      @sections << CarSection.new(cid, body)
+      @sections << CarSection.new(cid, body_data)
     end
   end
 end

data/lib/skyfall/collection.rb

@@ -16,5 +16,31 @@ module Skyfall
     BSKY_LABELER     = "app.bsky.labeler.service"
 
     BSKY_CHAT_DECLARATION = "chat.bsky.actor.declaration"
+
+    SHORT_CODES = {
+      BSKY_BLOCK       => :bsky_block,
+      BSKY_FEED        => :bsky_feed,
+      BSKY_FOLLOW      => :bsky_follow,
+      BSKY_LABELER     => :bsky_labeler,
+      BSKY_LIKE        => :bsky_like,
+      BSKY_LIST        => :bsky_list,
+      BSKY_LISTBLOCK   => :bsky_listblock,
+      BSKY_LISTITEM    => :bsky_listitem,
+      BSKY_POST        => :bsky_post,
+      BSKY_POSTGATE    => :bsky_postgate,
+      BSKY_PROFILE     => :bsky_profile,
+      BSKY_REPOST      => :bsky_repost,
+      BSKY_STARTERPACK => :bsky_starterpack,
+      BSKY_THREADGATE  => :bsky_threadgate,
+      BSKY_CHAT_DECLARATION => :bsky_chat_declaration,
+    }
+
+    def self.short_code(collection)
+      SHORT_CODES[collection] || :unknown
+    end
+
+    def self.from_short_code(code)
+      SHORT_CODES.detect { |k, v| v == code }&.first
+    end
   end
 end

data/lib/skyfall/{messages → firehose}/account_message.rb

@@ -1,5 +1,7 @@
+require_relative '../firehose'
+
 module Skyfall
-  class AccountMessage < WebsocketMessage
+  class Firehose::AccountMessage < Firehose::Message
     def active?
       @data_object['active']
     end

data/lib/skyfall/{messages → firehose}/commit_message.rb

@@ -1,14 +1,16 @@
 require_relative '../car_archive'
 require_relative '../cid'
-require_relative '../operation'
+require_relative '../firehose'
+require_relative 'operation'
 
 module Skyfall
-  class CommitMessage < WebsocketMessage
+  class Firehose::CommitMessage < Firehose::Message
     def commit
       @commit ||= @data_object['commit'] && CID.from_cbor_tag(@data_object['commit'])
     end
 
     def prev
+      STDERR.puts "Warning: `prev` property has been deprecated and will be removed in a future version."
       @prev ||= @data_object['prev'] && CID.from_cbor_tag(@data_object['prev'])
     end
 
@@ -17,7 +19,11 @@ module Skyfall
     end
 
     def operations
-      @operations ||= @data_object['ops'].map { |op| Operation.new(self, op) }
+      @operations ||= @data_object['ops'].map { |op| Firehose::Operation.new(self, op) }
+    end
+
+    def raw_record_for_operation(op)
+      op.cid && blocks.section_with_cid(op.cid)
     end
   end
 end

data/lib/skyfall/firehose/handle_message.rb

@@ -0,0 +1,14 @@
+require_relative '../firehose'
+
+module Skyfall
+
+  #
+  # Note: this event type is deprecated and will stop being emitted at some point.
+  # You should instead listen for 'identity' events (Skyfall::Firehose::IdentityMessage).
+  #
+  class Firehose::HandleMessage < Firehose::Message
+    def handle
+      @data_object['handle']
+    end
+  end
+end

data/lib/skyfall/firehose/identity_message.rb

@@ -0,0 +1,9 @@
+require_relative '../firehose'
+
+module Skyfall
+  class Firehose::IdentityMessage < Firehose::Message
+    def handle
+      @data_object['handle']
+    end
+  end
+end

data/lib/skyfall/{messages → firehose}/info_message.rb

@@ -1,5 +1,7 @@
+require_relative '../firehose'
+
 module Skyfall
-  class InfoMessage < WebsocketMessage
+  class Firehose::InfoMessage < Firehose::Message
     attr_reader :name, :message
 
     OUTDATED_CURSOR = "OutdatedCursor"

data/lib/skyfall/{messages → firehose}/labels_message.rb

@@ -1,8 +1,8 @@
-require_relative 'websocket_message'
+require_relative '../firehose'
 require_relative '../label'
 
 module Skyfall
-  class LabelsMessage
+  class Firehose::LabelsMessage
     using Skyfall::Extensions
 
     attr_reader :type_object, :data_object

data/lib/skyfall/{messages/websocket_message.rb → firehose/message.rb}

@@ -1,11 +1,12 @@
 require_relative '../errors'
 require_relative '../extensions'
+require_relative '../firehose'
 
 require 'cbor'
 require 'time'
 
 module Skyfall
-  class WebsocketMessage
+  class Firehose::Message
     using Skyfall::Extensions
 
     require_relative 'account_message'
@@ -17,23 +18,24 @@ module Skyfall
     require_relative 'tombstone_message'
     require_relative 'unknown_message'
 
-    attr_reader :type_object, :data_object
     attr_reader :type, :did, :seq
-
     alias repo did
 
+    # :nodoc: - consider this as semi-private API
+    attr_reader :type_object, :data_object
+
     def self.new(data)
       type_object, data_object = decode_cbor_objects(data)
 
       message_class = case type_object['t']
-        when '#account' then AccountMessage
-        when '#commit' then CommitMessage
-        when '#handle' then HandleMessage
-        when '#identity' then IdentityMessage
-        when '#info' then InfoMessage
-        when '#labels' then LabelsMessage
-        when '#tombstone' then TombstoneMessage
-        else UnknownMessage
+        when '#account'   then Firehose::AccountMessage
+        when '#commit'    then Firehose::CommitMessage
+        when '#handle'    then Firehose::HandleMessage
+        when '#identity'  then Firehose::IdentityMessage
+        when '#info'      then Firehose::InfoMessage
+        when '#labels'    then Firehose::LabelsMessage
+        when '#tombstone' then Firehose::TombstoneMessage
+        else Firehose::UnknownMessage
       end
 
       message = message_class.allocate