bad_pigeon 0.1.0 → 0.1.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 383a6c54b644b03f93d3b6c67ca811045938f8bf0de44feaf3c5f7344838eaf8
-  data.tar.gz: b69a8b24eeb45f7e3761309aa993620ed845a6625ad8fbaae1114c398578e17c
+  metadata.gz: 342303be8c8d86ebacfe718abecf55e553e1e9528540bac71d82d7f5729e6f84
+  data.tar.gz: cae73cef84dd9267c5cd79946afc129ccd93c8dcae62127e8c24b4ee88a8c6e6
 SHA512:
-  metadata.gz: 35ccd0f949bca44ce2b0374f1ba185f7addedb35c2b574e3bedfbccf9c7487ca80a3d418f0e9cd27c8501ee25c874d564d77f1959e1a8e9a4925707673861be9
-  data.tar.gz: 1af6038dafd3046b4f3af1cfcb739dd1df4428d527b0eea1ad986ef913711e0b67d9ce1ea8148d3f4d58b0d73b940e93409887f604c526c8c49c655d81608b5a
+  metadata.gz: d648e2cafc7d9f6b71650dfe283a57dc2a569d9b6da42938b38050559b8c0af0a53917624eb200febdc7a2522bcdfe1242b2645b11532dfcfddf81f408f3ba70
+  data.tar.gz: f30740c2a5422ada1760dbb46ab66868095546781843d75d3c3f4b1d232994c928835ca78e8812451f71e822503cd0dd2f96cd47f4045c655ee2539d35c242de

data/CHANGELOG.md

@@ -1,5 +1,14 @@
-## [Unreleased]
+## [0.1.1] - 2023-06-20
 
-## [0.1.0] - 2023-06-18
+- timeline parsing improvements
+- added some helper methods
+- fix for HAR exports from Firefox/Chrome
 
-- Initial release
+## [0.1.0] - 2023-06-19
+
+First working proof of concept version:
+
+- parsing a HAR archive
+- parsing user, list and home timelines from archive requests
+- exporting tweet JSON data
+- command line script

data/README.md

@@ -1,31 +1,93 @@
 # BadPigeon
 
-TODO: Delete this and the text below, and describe your gem
+A tool for exporting tweet data from Twitter by parsing GraphQL fetch requests made by the Twitter website.
 
-Welcome to your new gem! In this directory, you'll find the files you need to be able to package up your Ruby library into a gem. Put your Ruby code in the file `lib/bad_pigeon`. To experiment with that code, run `bin/console` for an interactive prompt.
+<p>
+  <img src="https://github.com/mackuba/bad_pigeon/assets/28465/99c3eee1-1fab-41be-a909-6b53d141b7db" width="600"><br>
+  <i>Photo by Martin Vorel, <a href="https://libreshot.com">libreshot.com</a></i>
+</p>
 
-## Installation
 
-TODO: Replace `UPDATE_WITH_YOUR_GEM_NAME_PRIOR_TO_RELEASE_TO_RUBYGEMS_ORG` with your gem name right after releasing it to RubyGems.org. Please do not do it earlier due to security reasons. Alternatively, replace this section with instructions to install your gem from git if you don't plan to release to RubyGems.org.
+## What is this about?
 
-Install the gem and add to the application's Gemfile by executing:
+**Problem:** You were running some kind of project that used Twitter API to load tweets from some number of feeds and process them in some way - for archiving, research, statistics, whatever. Now the free API access has been shut down, all your API keys have been revoked and your project doesn't work anymore ☹️
 
-    $ bundle add UPDATE_WITH_YOUR_GEM_NAME_PRIOR_TO_RELEASE_TO_RUBYGEMS_ORG
+**Solution 1:** sign up for paid access and pay more than all your streaming, media, internet, mobile and app subscriptions combined every month just to fetch some tweets 🤑💰💰💰
 
-If bundler is not being used to manage dependencies, install the gem by executing:
+**Solution 2:** go the Chad Scraper route and scrape the data from the website with some scripts, playing a cat and mouse game and worrying that your account and/or IP will be blocked 😬
 
-    $ gem install UPDATE_WITH_YOUR_GEM_NAME_PRIOR_TO_RELEASE_TO_RUBYGEMS_ORG
+**Solution 3:** passively record the requests that the Twitter frontend is making to the API using Safari Web Inspector, then use some Ruby code to extract any data you want from the saved JSON responses 🤔
 
-## Usage
 
-TODO: Write usage instructions here
+## How it works:
 
-## Development
+1. Open the Twitter website in a browser (preferably Safari or Firefox).
+2. Open the Web Inspector / Developer Tools on the Network tab.
+    - in Safari, make sure the "Export" button is not grayed out; if it is, reload the page first
+3. Scroll through some timelines (home, lists etc.) to make sure everything you want to save has been loaded.
+4. In the Network tab list, type "graphql" to the filter bar - only those requests are parsed, so no point making the archive larger than necessary.
+    - it seems that Chrome-based browsers always export all requests to the archive, so the file size gets into tens of megabytes very quickly - so it's better to use Safari or Firefox, which only export requests matching the filter
+5. Click "Export" and save the requests to a "HAR" archive file.
+    - in Safari, the button is in the top-right corner of the Network tab
+    - in Firefox, click the "gear" button in the top-right corner and choose "Save All As HAR"
+    - in Chrome, click the down arrow button at the end of the top toolbar
+6. Feed the archive file to the Bad Pigeon (the Ruby code or the command line tool).
+7. Profit 👍
 
-After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
+Note: one obvious drawback of this method is that the request recording part is somewhat manual, so it's (probably) not possible to completely automate it so that it runs on a server somewhere, unattended. However, it should be enough if you're ok with having to remember to periodically browse through a few timelines, save the export and run a script on it.
 
-To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and the created tag, and push the `.gem` file to [rubygems.org](https://rubygems.org).
 
-## Contributing
+## Stability warning ⚠️
 
-Bug reports and pull requests are welcome on GitHub at https://github.com/[USERNAME]/bad_pigeon.
+This is a very early version of this tool. The API \*will\* change between versions, possibly even between point releases. Don't be surprised if something breaks.
+
+
+## How to use:
+
+To install the tool, run:
+
+```
+gem install bad_pigeon
+```
+
+The [`TweetExtractor`](https://github.com/mackuba/bad_pigeon/blob/docs/lib/bad_pigeon/tweet_extractor.rb) class is the entry point. Pass the contents of the `.har` file to the `#get_tweets_from_har` method to get an array of `Tweet` objects parsed from the whole archive:
+
+```rb
+require 'bad_pigeon'
+
+data = File.read(path_to_har)
+extractor = BadPigeon::TweetExtractor.new
+tweets = extractor.get_tweets_from_har(data)
+
+tweets.sort_by(&:created_at).reverse.each do |tweet|
+  puts "#{tweet.created_at} @#{tweet.user.screen_name}: \"#{tweet.text}\""
+end
+```
+
+The `Tweet` class is meant to be API compatible with the one from the popular [twitter gem](https://github.com/sferik/twitter/), so you should be able to use it as a drop-in replacement if your project used that library (although only some subset of properties will work right now - please [report issues](https://github.com/mackuba/bad_pigeon/issues) for any missing ones).
+
+
+### Command line
+
+The gem also installs a command-line script `pigeon`. You can pass it the archive file and get a JSON array of tweet data on the output:
+
+```
+pigeon < tweets.har > tweets.json
+```
+
+At the moment this is the only thing it does. There will be some options in the future to e.g. filter the tweets only from some sources and so on. The format that it exports the tweets in is also meant to match the hashes returned from the `#attrs` method in the `Tweet` class in the [twitter gem](https://github.com/sferik/twitter/).
+
+
+## Credits
+
+Copyright © 2023 Kuba Suder ([@mackuba.eu](https://bsky.app/profile/mackuba.eu)).
+
+The code is available under the terms of the [zlib license](https://choosealicense.com/licenses/zlib/) (permissive, similar to MIT).
+
+Bug reports and pull requests are welcome 😎 (note: if you're having problems parsing some tweets, please send me links to some examples of specific tweets that are making it fail).
+
+---
+
+#### Why *bad* pigeon?
+
+Because pigeons are generally bad :<

data/lib/bad_pigeon/elements/timeline_entry.rb

@@ -27,12 +27,16 @@ module BadPigeon
       @json['clientEventInfo'] && @json['clientEventInfo']['component']
     end
 
+    def all_tweets
+      items.map(&:tweet).compact
+    end
+
     def items
       case self.type
       when Type::ITEM
-        item_from_content(@json['itemContent'])
+        [item_from_content(@json['itemContent'])].compact
       when Type::MODULE
-        @json['items'].map { |i| item_from_content(i['item']['itemContent']) }
+        @json['items'].map { |i| item_from_content(i['item']['itemContent']) }.compact
       when Type::CURSOR
         []
       else
@@ -44,12 +48,12 @@ module BadPigeon
     def item_from_content(item_content)
       case item_content['itemType']
       when 'TimelineTweet'
-        [TimelineTweet.new(item_content)]
+        TimelineTweet.new(item_content)
       when 'TimelineUser'
-        []
+        nil
       else
         assert("Unknown itemContent type: #{item_content['itemType']}")
-        []
+        nil
       end
     end
   end

data/lib/bad_pigeon/elements/timeline_tweet.rb

@@ -28,7 +28,7 @@ module BadPigeon
     end
 
     def tweet
-      Tweet.new(tweet_data)
+      tweet_data && Tweet.new(tweet_data)
     end
   end
 end

data/lib/bad_pigeon/har/har_archive.rb

@@ -7,8 +7,12 @@ module BadPigeon
       @json = JSON.parse(data)
     end
 
-    def entries
+    def requests
       @json['log']['entries'].map { |j| HARRequest.new(j) }
     end
+
+    def inspect
+      to_s
+    end
   end
 end

data/lib/bad_pigeon/har/har_request.rb

@@ -1,3 +1,4 @@
+require 'addressable/uri'
 require 'json'
 
 module BadPigeon
@@ -18,6 +19,19 @@ module BadPigeon
       url.start_with?('https://api.twitter.com/graphql/') || url.start_with?('https://twitter.com/i/api/graphql/')
     end
 
+    def includes_tweet_data?
+      graphql_endpoint? && method == :get && status == 200 && has_json_response?
+    end
+
+    def endpoint_name
+      Addressable::URI.parse(url).path.split('/').last
+    end
+
+    def params
+      vars = Addressable::URI.parse(url).query_values['variables']
+      vars && JSON.parse(vars) || {}
+    end
+
     def status
       @json['response']['status']
     end
@@ -27,7 +41,7 @@ module BadPigeon
     end
 
     def has_json_response?
-      mime_type == 'application/json'
+      mime_type.gsub(/;.*/, '').strip == 'application/json'
     end
 
     def response_body
@@ -37,5 +51,11 @@ module BadPigeon
     def response_json
       response_body && JSON.parse(response_body)
     end
+
+    def inspect
+      keys = [:method, :url, :status]
+      vars = keys.map { |k| "#{k}=#{self.send(k).inspect}" }.join(", ")
+      "#<#{self.class}:0x#{object_id} #{vars}>"
+    end
   end
 end

data/lib/bad_pigeon/models/tweet.rb

@@ -73,17 +73,27 @@ module BadPigeon
     def attrs
       user = json['core']['user_results']['result']
 
-      fields = legacy.merge({
+      fields = {
         id: id,
         source: json['source'],
         text: text,
         truncated: false,
-      }).reject { |k, v| ['retweeted_status_result', 'quoted_status_result'].include?(k) }
+      }
 
-      user_fields = user['legacy'].merge({
+      legacy.each do |k, v|
+        next if ['retweeted_status_result', 'quoted_status_result'].include?(k)
+        fields[k.to_sym] = v
+      end
+
+      user_fields = {
         id: user['rest_id'].to_i,
         id_str: user['rest_id'],
-      }).reject { |k, v| k =~ /^profile_\w+_extensions/ }
+      }
+
+      user['legacy'].each do |k, v|
+        next if k =~ /^profile_\w+_extensions/
+        user_fields[k.to_sym] = v
+      end
 
       fields[:user] = StrictHash[user_fields]
 

data/lib/bad_pigeon/timelines.rb

@@ -3,8 +3,25 @@ Dir[File.join(__dir__, 'timelines', '*.rb')].each { |f| require(f) }
 module BadPigeon
   TIMELINE_TYPES = {
     'UserTweets' => UserTimeline,
+    'UserMedia' => UserTimeline,
     'HomeLatestTimeline' => HomeTimeline,
     'HomeTimeline' => HomeTimeline,
-    'ListLatestTweetsTimeline' => ListTimeline
+    'ListLatestTweetsTimeline' => ListTimeline,
+
+    # ignored requests:
+    'AudioSpaceById' => nil,
+    'CommunitiesTabBarItemQuery' => nil,
+    'DataSaverMode' => nil,
+    'GetUserClaims' => nil,
+    'ListByRestId' => nil,
+    'ListMembers' => nil,
+    'ListPins' => nil,
+    'ListSubscribers' => nil,
+    'ListsManagementPageTimeline' => nil,
+    'ProfileSpotlightsQuery' => nil,
+    'UserByRestId' => nil,
+    'UserByScreenName' => nil,
+    'Viewer' => nil,
+    'getAltTextPromptPreference' => nil,
   }
 end

data/lib/bad_pigeon/tweet_extractor.rb

@@ -15,23 +15,21 @@ module BadPigeon
 
     def get_tweets_from_har(har_data)
       archive = HARArchive.new(har_data)
+      requests = archive.requests.select(&:includes_tweet_data?)
 
-      requests = archive.entries.select { |e|
-        e.graphql_endpoint? && e.method == :get && e.status == 200 && e.has_json_response?
-      }
-
-      entries = requests.map { |e|
-        endpoint = URI(e.url).path.split('/').last
+      timeline_entries = requests.map { |e| timeline_entries_from_request(e) }.flatten
+      timeline_entries.select { |e| @filter.include_entry?(e) }.map(&:all_tweets).flatten
+    end
 
-        if timeline_class = TIMELINE_TYPES[endpoint]
-          timeline_class.new(e.response_json).instructions.map(&:entries)
-        elsif !TIMELINE_TYPES.has_key?(endpoint)
-          debug "Unknown endpoint: #{endpoint}"
-          []
-        end
-      }.flatten
+    def timeline_entries_from_request(request)
+      endpoint = request.endpoint_name
 
-      entries.select { |e| @filter.include_entry?(e) }.map(&:items).flatten.map(&:tweet).compact
+      if timeline_class = TIMELINE_TYPES[endpoint]
+        timeline_class.new(request.response_json).instructions.map(&:entries)
+      else
+        debug "Unknown endpoint: #{endpoint}" unless TIMELINE_TYPES.has_key?(endpoint)
+        []
+      end
     end
   end
 end

data/lib/bad_pigeon/version.rb

@@ -1,5 +1,5 @@
 # frozen_string_literal: true
 
 module BadPigeon
-  VERSION = "0.1.0"
+  VERSION = "0.1.1"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: bad_pigeon
 version: !ruby/object:Gem::Version
-  version: 0.1.0
+  version: 0.1.1
 platform: ruby
 authors:
 - Kuba Suder
 autorequire:
 bindir: exe
 cert_chain: []
-date: 2023-06-19 00:00:00.000000000 Z
+date: 2023-06-20 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: addressable
@@ -25,10 +25,10 @@ dependencies:
       - !ruby/object:Gem::Version
         version: '2.8'
 description: "\n    BadPigeon is a Ruby gem that allows you to extract tweet data
-  from the XHR requests that the Twitter.com frontend\n    website does in user's
-  browser. The requests need to be saved into a \"HAR\" archive file from the browser's
-  web\n    inspector tool and then that file is fed into either the appropriate Ruby
-  class or the `pigeon` command line tool.\n    \n    The tool intents to be API compatible
+  from the XHR requests that the Twitter frontend\n    website does in user's browser.
+  The requests need to be saved into a \"HAR\" archive file from the browser's web\n
+  \   inspector tool and then that file is fed into either the appropriate Ruby class
+  or the `pigeon` command line tool.\n\n    The tool intents to be API compatible
   with the popular `twitter` gem and generate the same kind of tweet JSON\n    structure
   as is read and exported by that library.\n  "
 email: