twitterscraper-ruby 0.6.0 → 0.7.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: fe6db831d59218f3e701e0211487d79ef2354524610f8338a1a17e4cc426e437
-  data.tar.gz: 34cd890b8d2837bcacb3ab7b03fb43845294eecb9d7b0c4891c048eacedbe233
+  metadata.gz: 9cfd03782734642da8ac29839788f142399d2a3f4ec601e8b6f47ae1ca38c17f
+  data.tar.gz: 07a398e51fd2fbdc735ae27008d9a23e97dc390632179738045db4c81bd4fcad
 SHA512:
-  metadata.gz: 6c16c89ca290cc3c9ed5fd245c5aa26e5386c95011cfa14277e774e860359495cafec1624fba0af55de98ebbb34abb599e75e210bdbb18b3b11e49bc1527b643
-  data.tar.gz: d54e25e0294eddf8226c0e27a1d46c6128e9066c9d04edc429b382498c0de1af7ccf2e5c1333ad2031210bbefd7f9b7edc73b9a0e79ab2bd3673674b2e648f3c
+  metadata.gz: 6f417fe3379a3d9d134c308a9ea9d4e01b458018c9c5a3f8508a85e7f5890d01991838cfcabe87b8246f69edf4458c66d17924359798017907862071353f643d
+  data.tar.gz: 758bcb55ded936c3696f99647f64bc9921386b3cb0c783c218510c0e36991ae6b95a9d08fa071e02072c8b727bbadb6674ceeb19a74e356a842d62c1ec4c038f

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    twitterscraper-ruby (0.6.0)
+    twitterscraper-ruby (0.7.0)
       nokogiri
       parallel
 

data/README.md

@@ -1,46 +1,127 @@
 # twitterscraper-ruby
 
-Welcome to your new gem! In this directory, you'll find the files you need to be able to package up your Ruby library into a gem. Put your Ruby code in the file `lib/twitterscraper/ruby`. To experiment with that code, run `bin/console` for an interactive prompt.
+[![Gem Version](https://badge.fury.io/rb/twitterscraper-ruby.svg)](https://badge.fury.io/rb/twitterscraper-ruby)
 
-TODO: Delete this and the text above, and describe your gem
+A gem to scrape https://twitter.com/search. This gem is inspired by [taspinar/twitterscraper](https://github.com/taspinar/twitterscraper).
 
-## Installation
 
-Add this line to your application's Gemfile:
+## Twitter Search API vs. twitterscraper-ruby
 
-```ruby
-gem 'twitterscraper-ruby'
-```
+### Twitter Search API
+
+- The number of tweets: 180 - 450 requests/15 minutes (18,000 - 45,000 tweets/15 minutes)
+- The time window: the past 7 days
+
+### twitterscraper-ruby
+
+- The number of tweets: Unlimited
+- The time window: from 2006-3-21 to today
 
-And then execute:
 
-    $ bundle install
+## Installation
 
-Or install it yourself as:
+First install the library:
 
-    $ gem install twitterscraper-ruby
+```shell script
+$ gem install twitterscraper-ruby
+````
+    
 
 ## Usage
 
+Command-line interface:
+
+```shell script
+$ twitterscraper --query KEYWORD --start_date 2020-06-01 --end_date 2020-06-30 --lang ja \
+      --limit 100 --threads 10 --proxy --output output.json
+```
+
+From Within Ruby:
+
 ```ruby
 require 'twitterscraper'
+
+options = {
+  start_date: '2020-06-01',
+  end_date:   '2020-06-30',
+  lang:       'ja',
+  limit:      100,
+  threads:    10,
+  proxy:      true
+}
+
+client = Twitterscraper::Client.new
+tweets = client.query_tweets(KEYWORD, options)
+
+tweets.each do |tweet|
+  puts tweet.tweet_id
+  puts tweet.text
+  puts tweet.created_at
+  puts tweet.tweet_url
+end
 ```
 
-## Development
 
-After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake test` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
+## Examples
+
+```shell script
+$ twitterscraper --query twitter --limit 1000
+$ cat tweets.json | jq . | less
+```
+
+```json
+[
+  {
+    "screen_name": "@screenname",
+    "name": "name",
+    "user_id": 1194529546483000000,
+    "tweet_id": 1282659891992000000,
+    "tweet_url": "https://twitter.com/screenname/status/1282659891992000000",
+    "created_at": "2020-07-13 12:00:00 +0000",
+    "text": "Thanks Twitter!"
+  },
+  ...
+]
+```
+
+## Attributes
+
+### Tweet
+
+- tweet_id
+- text
+- user_id
+- screen_name
+- name
+- tweet_url
+- created_at
+
+
+## CLI Options
+
+| Option | Description | Default |
+| ------------- | ------------- | ------------- |
+| `-h`, `--help` | This option displays a summary of twitterscraper. | |
+| `--query` | Specify a keyword used during the search. | |
+| `--start_date` | Set the date from which twitterscraper-ruby should start scraping for your query. | |
+| `--end_date` | Set the enddate which twitterscraper-ruby should use to stop scraping for your query. | |
+| `--lang` | Retrieve tweets written in a specific language. | |
+| `--limit` | Stop scraping when *at least* the number of tweets indicated with --limit is scraped. | 100 |
+| `--threads` | Set the number of threads twitterscraper-ruby should initiate while scraping for your query. | 2 |
+| `--proxy` | Scrape https://twitter.com/search via proxies. | false |
+| `--output` | The name of the output file. | tweets.json |
 
-To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
 
 ## Contributing
 
-Bug reports and pull requests are welcome on GitHub at https://github.com/[USERNAME]/twitterscraper-ruby. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [code of conduct](https://github.com/[USERNAME]/twitterscraper-ruby/blob/master/CODE_OF_CONDUCT.md).
+Bug reports and pull requests are welcome on GitHub at https://github.com/ts-3156/twitterscraper-ruby. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [code of conduct](https://github.com/ts-3156/twitterscraper-ruby/blob/master/CODE_OF_CONDUCT.md).
 
 
 ## License
 
 The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).
 
+
 ## Code of Conduct
 
-Everyone interacting in the Twitterscraper::Ruby project's codebases, issue trackers, chat rooms and mailing lists is expected to follow the [code of conduct](https://github.com/[USERNAME]/twitterscraper-ruby/blob/master/CODE_OF_CONDUCT.md).
+Everyone interacting in the twitterscraper-ruby project's codebases, issue trackers, chat rooms and mailing lists is expected to follow the [code of conduct](https://github.com/ts-3156/twitterscraper-ruby/blob/master/CODE_OF_CONDUCT.md).

data/lib/twitterscraper/cli.rb

@@ -15,7 +15,6 @@ module Twitterscraper
       print_help || return if print_help?
       print_version || return if print_version?
 
-      client = Twitterscraper::Client.new
       query_options = {
           start_date: options['start_date'],
           end_date: options['end_date'],
@@ -24,6 +23,7 @@ module Twitterscraper
           threads: options['threads'],
           proxy: options['proxy']
       }
+      client = Twitterscraper::Client.new
       tweets = client.query_tweets(options['query'], query_options)
       File.write(options['output'], generate_json(tweets))
     end

data/lib/twitterscraper/query.rb

@@ -3,6 +3,7 @@ require 'net/http'
 require 'nokogiri'
 require 'date'
 require 'json'
+require 'erb'
 require 'parallel'
 
 module Twitterscraper
@@ -41,7 +42,8 @@ module Twitterscraper
       end
     end
 
-    def get_single_page(url, headers, proxies, timeout = 10, retries = 30)
+    def get_single_page(url, headers, proxies, timeout = 6, retries = 30)
+      return nil if stop_requested?
       Twitterscraper::Http.get(url, headers, proxies.sample, timeout)
     rescue => e
       logger.debug "query_single_page: #{e.inspect}"
@@ -54,6 +56,8 @@ module Twitterscraper
     end
 
     def parse_single_page(text, html = true)
+      return [nil, nil] if text.nil? || text == ''
+
       if html
         json_resp = nil
         items_html = text
@@ -68,12 +72,14 @@ module Twitterscraper
 
     def query_single_page(query, lang, pos, from_user = false, headers: [], proxies: [])
       logger.info("Querying #{query}")
-      query = query.gsub(' ', '%20').gsub('#', '%23').gsub(':', '%3A').gsub('&', '%26')
+      query = ERB::Util.url_encode(query)
 
       url = build_query_url(query, lang, pos, from_user)
       logger.debug("Scraping tweets from #{url}")
 
       response = get_single_page(url, headers, proxies)
+      return [], nil if response.nil?
+
       html, json_resp = parse_single_page(response, pos.nil?)
 
       tweets = Tweet.from_html(html)
@@ -91,55 +97,112 @@ module Twitterscraper
       end
     end
 
-    def query_tweets(query, start_date: nil, end_date: nil, lang: '', limit: 100, threads: 2, proxy: false)
-      start_date = start_date ? Date.parse(start_date) : Date.parse('2006-3-21')
-      end_date = end_date ? Date.parse(end_date) : Date.today
-      if start_date == end_date
-        raise 'Please specify different values for :start_date and :end_date.'
-      elsif start_date > end_date
-        raise ':start_date must occur before :end_date.'
+    OLDEST_DATE = Date.parse('2006-3-21')
+
+    def validate_options!(query, start_date:, end_date:, lang:, limit:, threads:, proxy:)
+      if query.nil? || query == ''
+        raise 'Please specify a search query.'
       end
 
-      proxies = proxy ? Twitterscraper::Proxy::Pool.new : []
+      if ERB::Util.url_encode(query).length >= 500
+        raise ':query must be a UTF-8, URL-encoded search query of 500 characters maximum, including operators.'
+      end
 
-      date_range = start_date.upto(end_date - 1)
-      queries = date_range.map { |date| query + " since:#{date} until:#{date + 1}" }
-      threads = queries.size if threads > queries.size
-      logger.info("Threads #{threads}")
+      if start_date && end_date
+        if start_date == end_date
+          raise 'Please specify different values for :start_date and :end_date.'
+        elsif start_date > end_date
+          raise ':start_date must occur before :end_date.'
+        end
+      end
 
-      headers = {'User-Agent': USER_AGENT_LIST.sample, 'X-Requested-With': 'XMLHttpRequest'}
-      logger.info("Headers #{headers}")
+      if start_date
+        if start_date < OLDEST_DATE
+          raise ":start_date must be greater than or equal to #{OLDEST_DATE}"
+        end
+      end
 
-      all_tweets = []
-      mutex = Mutex.new
+      if end_date
+        today = Date.today
+        if end_date > Date.today
+          raise ":end_date must be less than or equal to today(#{today})"
+        end
+      end
+    end
 
-      Parallel.each(queries, in_threads: threads) do |query|
+    def build_queries(query, start_date, end_date)
+      if start_date && end_date
+        date_range = start_date.upto(end_date - 1)
+        date_range.map { |date| query + " since:#{date} until:#{date + 1}" }
+      elsif start_date
+        [query + " since:#{start_date}"]
+      elsif end_date
+        [query + " until:#{end_date}"]
+      else
+        [query]
+      end
+    end
 
-        pos = nil
+    def main_loop(query, lang, limit, headers, proxies)
+      pos = nil
 
-        while true
-          new_tweets, new_pos = query_single_page(query, lang, pos, headers: headers, proxies: proxies)
-          unless new_tweets.empty?
-            mutex.synchronize {
-              all_tweets.concat(new_tweets)
-              all_tweets.uniq! { |t| t.tweet_id }
-            }
-          end
-          logger.info("Got #{new_tweets.size} tweets (total #{all_tweets.size}) worker=#{Parallel.worker_number}")
+      while true
+        new_tweets, new_pos = query_single_page(query, lang, pos, headers: headers, proxies: proxies)
+        unless new_tweets.empty?
+          @mutex.synchronize {
+            @all_tweets.concat(new_tweets)
+            @all_tweets.uniq! { |t| t.tweet_id }
+          }
+        end
+        logger.info("Got #{new_tweets.size} tweets (total #{@all_tweets.size})")
 
-          break unless new_pos
-          break if all_tweets.size >= limit
+        break unless new_pos
+        break if @all_tweets.size >= limit
 
-          pos = new_pos
-        end
+        pos = new_pos
+      end
+
+      if @all_tweets.size >= limit
+        logger.info("Limit reached #{@all_tweets.size}")
+        @stop_requested = true
+      end
+    end
 
-        if all_tweets.size >= limit
-          logger.info("Reached limit #{all_tweets.size}")
-          raise Parallel::Break
+    def stop_requested?
+      @stop_requested
+    end
+
+    def query_tweets(query, start_date: nil, end_date: nil, lang: '', limit: 100, threads: 2, proxy: false)
+      start_date = Date.parse(start_date) if start_date && start_date.is_a?(String)
+      end_date = Date.parse(end_date) if end_date && end_date.is_a?(String)
+      queries = build_queries(query, start_date, end_date)
+      threads = queries.size if threads > queries.size
+      proxies = proxy ? Twitterscraper::Proxy::Pool.new : []
+
+      validate_options!(queries[0], start_date: start_date, end_date: end_date, lang: lang, limit: limit, threads: threads, proxy: proxy)
+
+      logger.info("The number of threads #{threads}")
+
+      headers = {'User-Agent': USER_AGENT_LIST.sample, 'X-Requested-With': 'XMLHttpRequest'}
+      logger.info("Headers #{headers}")
+
+      @all_tweets = []
+      @mutex = Mutex.new
+      @stop_requested = false
+
+      if threads > 1
+        Parallel.each(queries, in_threads: threads) do |query|
+          main_loop(query, lang, limit, headers, proxies)
+          raise Parallel::Break if stop_requested?
+        end
+      else
+        queries.each do |query|
+          main_loop(query, lang, limit, headers, proxies)
+          break if stop_requested?
         end
       end
 
-      all_tweets.sort_by { |tweet| -tweet.created_at.to_i }
+      @all_tweets.sort_by { |tweet| -tweet.created_at.to_i }
     end
   end
 end

data/lib/version.rb

@@ -1,3 +1,3 @@
 module Twitterscraper
-  VERSION = '0.6.0'
+  VERSION = '0.7.0'
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: twitterscraper-ruby
 version: !ruby/object:Gem::Version
-  version: 0.6.0
+  version: 0.7.0
 platform: ruby
 authors:
 - ts-3156