twitterscraper-ruby 0.5.0 → 0.10.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: e6701ff59f3eb13db9e3b2d024ea983264b528194d64f4f03a95f3576338ed77
-  data.tar.gz: 74106816dd406ef1b355b4d4fc94b1baf4509465f6d5bf1ea8f7c654e518eec0
+  metadata.gz: c2429cf6172b5f19caede64ac35f5c796a7c8a67e76fff8dd2f08677fb15406b
+  data.tar.gz: 0f32ca6b559a18c4e3aac3205f6503149e372d4d7d1976b1e83db26036d9ff17
 SHA512:
-  metadata.gz: 9710fb74c90dcbc17a22dd613cfe4dce75106951f1e55cd9cfa94a825ecf0b6773a2851ff1cca842f83b5207d3744ac63bcce29031061b0a0ac84cc12d62b8a3
-  data.tar.gz: f0e7cd90ecb773a1837be9245b83f51d60f25d48cab716e3584a8d3e1b6f0fe4951eadaf034850599205209d2bf8d7cd4ebfda97f9f862a528c393a2f81887a7
+  metadata.gz: a36ce6c91a363b64b36deeb3abbaaaebb725f3449f280b70be92532497a94dc5915ba449926acfacfc0d852d52471d258d41140a8891e64b6040bf262d0c347f
+  data.tar.gz: a737c7db151190a1493b1a2a92bea304cfcf7512b2ee03fc13c6f25794f5dc727fe548e52cb39eccc2a63261fee0d58fc005920a0e7cd7650d20600e184d79cb

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    twitterscraper-ruby (0.5.0)
+    twitterscraper-ruby (0.10.0)
       nokogiri
       parallel
 

data/README.md

@@ -1,46 +1,162 @@
 # twitterscraper-ruby
 
-Welcome to your new gem! In this directory, you'll find the files you need to be able to package up your Ruby library into a gem. Put your Ruby code in the file `lib/twitterscraper/ruby`. To experiment with that code, run `bin/console` for an interactive prompt.
+[![Gem Version](https://badge.fury.io/rb/twitterscraper-ruby.svg)](https://badge.fury.io/rb/twitterscraper-ruby)
 
-TODO: Delete this and the text above, and describe your gem
+A gem to scrape https://twitter.com/search. This gem is inspired by [taspinar/twitterscraper](https://github.com/taspinar/twitterscraper).
 
-## Installation
 
-Add this line to your application's Gemfile:
+## Twitter Search API vs. twitterscraper-ruby
 
-```ruby
-gem 'twitterscraper-ruby'
-```
+### Twitter Search API
 
-And then execute:
+- The number of tweets: 180 - 450 requests/15 minutes (18,000 - 45,000 tweets/15 minutes)
+- The time window: the past 7 days
 
-    $ bundle install
+### twitterscraper-ruby
 
-Or install it yourself as:
+- The number of tweets: Unlimited
+- The time window: from 2006-3-21 to today
 
-    $ gem install twitterscraper-ruby
+
+## Installation
+
+First install the library:
+
+```shell script
+$ gem install twitterscraper-ruby
+````
+    
 
 ## Usage
 
+Command-line interface:
+
+```shell script
+$ twitterscraper --query KEYWORD --start_date 2020-06-01 --end_date 2020-06-30 --lang ja \
+      --limit 100 --threads 10 --proxy --output output.json
+```
+
+From Within Ruby:
+
 ```ruby
 require 'twitterscraper'
+
+options = {
+  start_date: '2020-06-01',
+  end_date:   '2020-06-30',
+  lang:       'ja',
+  limit:      100,
+  threads:    10,
+  proxy:      true
+}
+
+client = Twitterscraper::Client.new
+tweets = client.query_tweets(KEYWORD, options)
+
+tweets.each do |tweet|
+  puts tweet.tweet_id
+  puts tweet.text
+  puts tweet.tweet_url
+  puts tweet.created_at
+
+  hash = tweet.attrs
+  puts hash.keys
+end
 ```
 
-## Development
 
-After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake test` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
+## Attributes
+
+### Tweet
+
+- screen_name
+- name
+- user_id
+- tweet_id
+- text
+- links
+- hashtags
+- image_urls
+- video_url
+- has_media
+- likes
+- retweets
+- replies
+- is_replied
+- is_reply_to
+- parent_tweet_id
+- reply_to_users
+- tweet_url
+- created_at
+
+
+## Search operators
+
+| Operator | Finds Tweets... |
+| ------------- | ------------- |
+| watching now | containing both "watching" and "now". This is the default operator. |
+| "happy hour" | containing the exact phrase "happy hour". |
+| love OR hate | containing either "love" or "hate" (or both). |
+| beer -root | containing "beer" but not "root". |
+| #haiku | containing the hashtag "haiku". |
+| from:interior | sent from Twitter account "interior". |
+| to:NASA | a Tweet authored in reply to Twitter account "NASA". |
+| @NASA | mentioning Twitter account "NASA". |
+| puppy filter:media | containing "puppy" and an image or video. |
+| puppy -filter:retweets | containing "puppy", filtering out retweets |
+| superhero since:2015-12-21 | containing "superhero" and sent since date "2015-12-21" (year-month-day). |
+| puppy until:2015-12-21 | containing "puppy" and sent before the date "2015-12-21". |
+
+Search operators documentation is in [Standard search operators](https://developer.twitter.com/en/docs/tweets/rules-and-filtering/overview/standard-operators).
+
+
+## Examples
+
+```shell script
+$ twitterscraper --query twitter --limit 1000
+$ cat tweets.json | jq . | less
+```
+
+```json
+[
+  {
+    "screen_name": "@screenname",
+    "name": "name",
+    "user_id": 1194529546483000000,
+    "tweet_id": 1282659891992000000,
+    "tweet_url": "https://twitter.com/screenname/status/1282659891992000000",
+    "created_at": "2020-07-13 12:00:00 +0000",
+    "text": "Thanks Twitter!"
+  }
+]
+```
+
+## CLI Options
+
+| Option | Description | Default |
+| ------------- | ------------- | ------------- |
+| `-h`, `--help` | This option displays a summary of twitterscraper. | |
+| `--query` | Specify a keyword used during the search. | |
+| `--start_date` | Set the date from which twitterscraper-ruby should start scraping for your query. | |
+| `--end_date` | Set the enddate which twitterscraper-ruby should use to stop scraping for your query. | |
+| `--lang` | Retrieve tweets written in a specific language. | |
+| `--limit` | Stop scraping when *at least* the number of tweets indicated with --limit is scraped. | 100 |
+| `--threads` | Set the number of threads twitterscraper-ruby should initiate while scraping for your query. | 2 |
+| `--proxy` | Scrape https://twitter.com/search via proxies. | false |
+| `--format` | The format of the output. | json |
+| `--output` | The name of the output file. | tweets.json |
 
-To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
 
 ## Contributing
 
-Bug reports and pull requests are welcome on GitHub at https://github.com/[USERNAME]/twitterscraper-ruby. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [code of conduct](https://github.com/[USERNAME]/twitterscraper-ruby/blob/master/CODE_OF_CONDUCT.md).
+Bug reports and pull requests are welcome on GitHub at https://github.com/ts-3156/twitterscraper-ruby. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [code of conduct](https://github.com/ts-3156/twitterscraper-ruby/blob/master/CODE_OF_CONDUCT.md).
 
 
 ## License
 
 The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).
 
+
 ## Code of Conduct
 
-Everyone interacting in the Twitterscraper::Ruby project's codebases, issue trackers, chat rooms and mailing lists is expected to follow the [code of conduct](https://github.com/[USERNAME]/twitterscraper-ruby/blob/master/CODE_OF_CONDUCT.md).
+Everyone interacting in the twitterscraper-ruby project's codebases, issue trackers, chat rooms and mailing lists is expected to follow the [code of conduct](https://github.com/ts-3156/twitterscraper-ruby/blob/master/CODE_OF_CONDUCT.md).

data/lib/twitterscraper.rb

@@ -5,14 +5,14 @@ require 'twitterscraper/lang'
 require 'twitterscraper/query'
 require 'twitterscraper/client'
 require 'twitterscraper/tweet'
+require 'twitterscraper/template'
 require 'version'
 
 module Twitterscraper
   class Error < StandardError; end
-  # Your code goes here...
 
   def self.logger
-    @logger ||= ::Logger.new(STDOUT)
+    @logger ||= ::Logger.new(STDOUT, level: ::Logger::INFO)
   end
 
   def self.logger=(logger)

data/lib/twitterscraper/cli.rb

@@ -8,18 +8,36 @@ module Twitterscraper
   class Cli
     def parse
       @options = parse_options(ARGV)
+      initialize_logger
     end
 
     def run
+      print_help || return if print_help?
+      print_version || return if print_version?
+
+      query_options = {
+          start_date: options['start_date'],
+          end_date: options['end_date'],
+          lang: options['lang'],
+          limit: options['limit'],
+          threads: options['threads'],
+          proxy: options['proxy']
+      }
       client = Twitterscraper::Client.new
-      limit = options['limit'] ? options['limit'].to_i : 100
-      threads = options['threads'] ? options['threads'].to_i : 2
-      tweets = client.query_tweets(options['query'], limit: limit, threads: threads, start_date: options['start_date'], end_date: options['end_date'])
-      File.write('tweets.json', generate_json(tweets))
+      tweets = client.query_tweets(options['query'], query_options)
+      export(tweets) unless tweets.empty?
     end
 
-    def options
-      @options
+    def export(tweets)
+      write_json = lambda { File.write(options['output'], generate_json(tweets)) }
+
+      if options['format'] == 'json'
+        write_json.call
+      elsif options['format'] == 'html'
+        File.write('tweets.html', Template.tweets_embedded_html(tweets))
+      else
+        write_json.call
+      end
     end
 
     def generate_json(tweets)
@@ -30,16 +48,59 @@ module Twitterscraper
       end
     end
 
+    def options
+      @options
+    end
+
     def parse_options(argv)
-      argv.getopts(
+      options = argv.getopts(
           'h',
+          'help',
+          'v',
+          'version',
           'query:',
-          'limit:',
           'start_date:',
           'end_date:',
+          'lang:',
+          'limit:',
           'threads:',
+          'output:',
+          'format:',
+          'proxy',
           'pretty',
+          'verbose',
       )
+
+      options['lang'] ||= ''
+      options['limit'] = (options['limit'] || 100).to_i
+      options['threads'] = (options['threads'] || 2).to_i
+      options['format'] ||= 'json'
+      options['output'] ||= "tweets.#{options['format']}"
+
+      options
+    end
+
+    def initialize_logger
+      Twitterscraper.logger.level = ::Logger::DEBUG if options['verbose']
+    end
+
+    def print_help?
+      options['h'] || options['help']
+    end
+
+    def print_help
+      puts <<~'SHELL'
+        Usage:
+          twitterscraper --query KEYWORD --limit 100 --threads 10 --start_date 2020-07-01 --end_date 2020-07-10 --lang ja --proxy --output output.json
+      SHELL
+    end
+
+    def print_version?
+      options['v'] || options['version']
+    end
+
+    def print_version
+      puts "twitterscraper-#{Twitterscraper::VERSION}"
     end
   end
 end

data/lib/twitterscraper/proxy.rb

@@ -6,9 +6,9 @@ module Twitterscraper
     class RetryExhausted < StandardError
     end
 
-    class Result
-      def initialize(items)
-        @items = items
+    class Pool
+      def initialize
+        @items = Proxy.get_proxies
         @cur_index = 0
       end
 
@@ -31,7 +31,6 @@ module Twitterscraper
       def reload
         @items = Proxy.get_proxies
         @cur_index = 0
-        Twitterscraper.logger.debug "Reload #{proxies.size} proxies"
       end
     end
 
@@ -46,13 +45,14 @@ module Twitterscraper
 
       table.xpath('tbody/tr').each do |tr|
         cells = tr.xpath('td')
-        ip, port, https = [0, 1, 6].map { |i| cells[i].text.strip }
+        ip, port, anonymity, https = [0, 1, 4, 6].map { |i| cells[i].text.strip }
+        next unless ['elite proxy', 'anonymous'].include?(anonymity)
         next if https == 'no'
         proxies << ip + ':' + port
       end
 
       Twitterscraper.logger.debug "Fetch #{proxies.size} proxies"
-      Result.new(proxies.shuffle)
+      proxies.shuffle
     rescue => e
       if (retries -= 1) > 0
         retry

data/lib/twitterscraper/query.rb

@@ -3,6 +3,7 @@ require 'net/http'
 require 'nokogiri'
 require 'date'
 require 'json'
+require 'erb'
 require 'parallel'
 
 module Twitterscraper
@@ -41,7 +42,8 @@ module Twitterscraper
       end
     end
 
-    def get_single_page(url, headers, proxies, timeout = 10, retries = 30)
+    def get_single_page(url, headers, proxies, timeout = 6, retries = 30)
+      return nil if stop_requested?
       Twitterscraper::Http.get(url, headers, proxies.sample, timeout)
     rescue => e
       logger.debug "query_single_page: #{e.inspect}"
@@ -54,6 +56,8 @@ module Twitterscraper
     end
 
     def parse_single_page(text, html = true)
+      return [nil, nil] if text.nil? || text == ''
+
       if html
         json_resp = nil
         items_html = text
@@ -68,12 +72,14 @@ module Twitterscraper
 
     def query_single_page(query, lang, pos, from_user = false, headers: [], proxies: [])
       logger.info("Querying #{query}")
-      query = query.gsub(' ', '%20').gsub('#', '%23').gsub(':', '%3A').gsub('&', '%26')
+      query = ERB::Util.url_encode(query)
 
       url = build_query_url(query, lang, pos, from_user)
       logger.debug("Scraping tweets from #{url}")
 
       response = get_single_page(url, headers, proxies)
+      return [], nil if response.nil?
+
       html, json_resp = parse_single_page(response, pos.nil?)
 
       tweets = Tweet.from_html(html)
@@ -91,54 +97,112 @@ module Twitterscraper
       end
     end
 
-    def query_tweets(query, start_date: nil, end_date: nil, limit: 100, threads: 2, lang: '')
-      start_date = start_date ? Date.parse(start_date) : Date.parse('2006-3-21')
-      end_date = end_date ? Date.parse(end_date) : Date.today
-      if start_date == end_date
-        raise 'Please specify different values for :start_date and :end_date.'
-      elsif start_date > end_date
-        raise ':start_date must occur before :end_date.'
-      end
+    OLDEST_DATE = Date.parse('2006-03-21')
 
-      proxies = Twitterscraper::Proxy.get_proxies
+    def validate_options!(query, start_date:, end_date:, lang:, limit:, threads:, proxy:)
+      if query.nil? || query == ''
+        raise 'Please specify a search query.'
+      end
 
-      date_range = start_date.upto(end_date - 1)
-      queries = date_range.map { |date| query + " since:#{date} until:#{date + 1}" }
-      threads = queries.size if threads > queries.size
-      logger.info("Threads #{threads}")
+      if ERB::Util.url_encode(query).length >= 500
+        raise ':query must be a UTF-8, URL-encoded search query of 500 characters maximum, including operators.'
+      end
 
-      all_tweets = []
-      mutex = Mutex.new
+      if start_date && end_date
+        if start_date == end_date
+          raise 'Please specify different values for :start_date and :end_date.'
+        elsif start_date > end_date
+          raise ':start_date must occur before :end_date.'
+        end
+      end
 
-      Parallel.each(queries, in_threads: threads) do |query|
-        headers = {'User-Agent': USER_AGENT_LIST.sample, 'X-Requested-With': 'XMLHttpRequest'}
-        logger.info("Headers #{headers}")
+      if start_date
+        if start_date < OLDEST_DATE
+          raise ":start_date must be greater than or equal to #{OLDEST_DATE}"
+        end
+      end
 
-        pos = nil
+      if end_date
+        today = Date.today
+        if end_date > Date.today
+          raise ":end_date must be less than or equal to today(#{today})"
+        end
+      end
+    end
 
-        while true
-          new_tweets, new_pos = query_single_page(query, lang, pos, headers: headers, proxies: proxies)
-          unless new_tweets.empty?
-            mutex.synchronize {
-              all_tweets.concat(new_tweets)
-              all_tweets.uniq! { |t| t.tweet_id }
-            }
-          end
-          logger.info("Got #{new_tweets.size} tweets (total #{all_tweets.size}) worker=#{Parallel.worker_number}")
+    def build_queries(query, start_date, end_date)
+      if start_date && end_date
+        date_range = start_date.upto(end_date - 1)
+        date_range.map { |date| query + " since:#{date} until:#{date + 1}" }
+      elsif start_date
+        [query + " since:#{start_date}"]
+      elsif end_date
+        [query + " until:#{end_date}"]
+      else
+        [query]
+      end
+    end
 
-          break unless new_pos
-          break if all_tweets.size >= limit
+    def main_loop(query, lang, limit, headers, proxies)
+      pos = nil
 
-          pos = new_pos
+      while true
+        new_tweets, new_pos = query_single_page(query, lang, pos, headers: headers, proxies: proxies)
+        unless new_tweets.empty?
+          @mutex.synchronize {
+            @all_tweets.concat(new_tweets)
+            @all_tweets.uniq! { |t| t.tweet_id }
+          }
         end
+        logger.info("Got #{new_tweets.size} tweets (total #{@all_tweets.size})")
 
-        if all_tweets.size >= limit
-          logger.info("Reached limit #{all_tweets.size}")
-          raise Parallel::Break
+        break unless new_pos
+        break if @all_tweets.size >= limit
+
+        pos = new_pos
+      end
+
+      if @all_tweets.size >= limit
+        logger.info("Limit reached #{@all_tweets.size}")
+        @stop_requested = true
+      end
+    end
+
+    def stop_requested?
+      @stop_requested
+    end
+
+    def query_tweets(query, start_date: nil, end_date: nil, lang: '', limit: 100, threads: 2, proxy: false)
+      start_date = Date.parse(start_date) if start_date && start_date.is_a?(String)
+      end_date = Date.parse(end_date) if end_date && end_date.is_a?(String)
+      queries = build_queries(query, start_date, end_date)
+      threads = queries.size if threads > queries.size
+      proxies = proxy ? Twitterscraper::Proxy::Pool.new : []
+
+      validate_options!(queries[0], start_date: start_date, end_date: end_date, lang: lang, limit: limit, threads: threads, proxy: proxy)
+
+      logger.info("The number of threads #{threads}")
+
+      headers = {'User-Agent': USER_AGENT_LIST.sample, 'X-Requested-With': 'XMLHttpRequest'}
+      logger.info("Headers #{headers}")
+
+      @all_tweets = []
+      @mutex = Mutex.new
+      @stop_requested = false
+
+      if threads > 1
+        Parallel.each(queries, in_threads: threads) do |query|
+          main_loop(query, lang, limit, headers, proxies)
+          raise Parallel::Break if stop_requested?
+        end
+      else
+        queries.each do |query|
+          main_loop(query, lang, limit, headers, proxies)
+          break if stop_requested?
         end
       end
 
-      all_tweets.sort_by { |tweet| -tweet.created_at.to_i }
+      @all_tweets.sort_by { |tweet| -tweet.created_at.to_i }
     end
   end
 end

data/lib/twitterscraper/template.rb

@@ -0,0 +1,48 @@
+module Twitterscraper
+  module Template
+    module_function
+
+    def tweets_embedded_html(tweets)
+      tweets_html = tweets.map { |t| EMBED_TWEET_HTML.sub('__TWEET_URL__', t.tweet_url) }
+      EMBED_TWEETS_HTML.sub('__TWEETS__', tweets_html.join)
+    end
+
+    EMBED_TWEET_HTML = <<~'HTML'
+      <blockquote class="twitter-tweet">
+        <a href="__TWEET_URL__"></a>
+      </blockquote>
+    HTML
+
+    EMBED_TWEETS_HTML = <<~'HTML'
+      <html>
+        <head>
+          <style type=text/css>
+            .twitter-tweet {
+              margin: 30px auto 0 auto !important;
+            }
+          </style>
+          <script>
+            window.twttr = (function(d, s, id) {
+              var js, fjs = d.getElementsByTagName(s)[0], t = window.twttr || {};
+              if (d.getElementById(id)) return t;
+              js = d.createElement(s);
+              js.id = id;
+              js.src = "https://platform.twitter.com/widgets.js";
+              fjs.parentNode.insertBefore(js, fjs);
+
+              t._e = [];
+              t.ready = function(f) {
+                  t._e.push(f);
+              };
+
+              return t;
+            }(document, "script", "twitter-wjs"));
+          </script>
+        </head>
+        <body>
+          __TWEETS__
+        </body>
+      </html>
+    HTML
+  end
+end

data/lib/twitterscraper/tweet.rb

@@ -2,7 +2,28 @@ require 'time'
 
 module Twitterscraper
   class Tweet
-    KEYS = [:screen_name, :name, :user_id, :tweet_id, :tweet_url, :created_at, :text]
+    KEYS = [
+        :screen_name,
+        :name,
+        :user_id,
+        :tweet_id,
+        :text,
+        :links,
+        :hashtags,
+        :image_urls,
+        :video_url,
+        :has_media,
+        :likes,
+        :retweets,
+        :replies,
+        :is_replied,
+        :is_reply_to,
+        :parent_tweet_id,
+        :reply_to_users,
+        :tweet_url,
+        :timestamp,
+        :created_at,
+    ]
     attr_reader *KEYS
 
     def initialize(attrs)
@@ -11,10 +32,14 @@ module Twitterscraper
       end
     end
 
-    def to_json(options = {})
+    def attrs
       KEYS.map do |key|
         [key, send(key)]
-      end.to_h.to_json
+      end.to_h
+    end
+
+    def to_json(options = {})
+      attrs.to_json
     end
 
     class << self
@@ -31,15 +56,51 @@ module Twitterscraper
 
       def from_tweet_html(html)
         inner_html = Nokogiri::HTML(html.inner_html)
+        tweet_id = html.attr('data-tweet-id').to_i
+        text = inner_html.xpath("//div[@class[contains(., 'js-tweet-text-container')]]/p[@class[contains(., 'js-tweet-text')]]").first.text
+        links = inner_html.xpath("//a[@class[contains(., 'twitter-timeline-link')]]").map { |elem| elem.attr('data-expanded-url') }.select { |link| link && !link.include?('pic.twitter') }
+        image_urls = inner_html.xpath("//div[@class[contains(., 'AdaptiveMedia-photoContainer')]]").map { |elem| elem.attr('data-image-url') }
+        video_url = inner_html.xpath("//div[@class[contains(., 'PlayableMedia-container')]]/a").map { |elem| elem.attr('href') }[0]
+        has_media = !image_urls.empty? || (video_url && !video_url.empty?)
+
+        actions = inner_html.xpath("//div[@class[contains(., 'ProfileTweet-actionCountList')]]")
+        likes = actions.xpath("//span[@class[contains(., 'ProfileTweet-action--favorite')]]/span[@class[contains(., 'ProfileTweet-actionCount')]]").first.attr('data-tweet-stat-count').to_i || 0
+        retweets = actions.xpath("//span[@class[contains(., 'ProfileTweet-action--retweet')]]/span[@class[contains(., 'ProfileTweet-actionCount')]]").first.attr('data-tweet-stat-count').to_i || 0
+        replies = actions.xpath("//span[@class[contains(., 'ProfileTweet-action--reply u-hiddenVisually')]]/span[@class[contains(., 'ProfileTweet-actionCount')]]").first.attr('data-tweet-stat-count').to_i || 0
+        is_replied = replies != 0
+
+        parent_tweet_id = inner_html.xpath('//*[@data-conversation-id]').first.attr('data-conversation-id').to_i
+        if tweet_id == parent_tweet_id
+          is_reply_to = false
+          parent_tweet_id = nil
+          reply_to_users = []
+        else
+          is_reply_to = true
+          reply_to_users = inner_html.xpath("//div[@class[contains(., 'ReplyingToContextBelowAuthor')]]/a").map { |user| {screen_name: user.text.delete_prefix('@'), user_id: user.attr('data-user-id')} }
+        end
+
         timestamp = inner_html.xpath("//span[@class[contains(., 'js-short-timestamp')]]").first.attr('data-time').to_i
         new(
             screen_name: html.attr('data-screen-name'),
             name: html.attr('data-name'),
             user_id: html.attr('data-user-id').to_i,
-            tweet_id: html.attr('data-tweet-id').to_i,
+            tweet_id: tweet_id,
+            text: text,
+            links: links,
+            hashtags: text.scan(/#\w+/).map { |tag| tag.delete_prefix('#') },
+            image_urls: image_urls,
+            video_url: video_url,
+            has_media: has_media,
+            likes: likes,
+            retweets: retweets,
+            replies: replies,
+            is_replied: is_replied,
+            is_reply_to: is_reply_to,
+            parent_tweet_id: parent_tweet_id,
+            reply_to_users: reply_to_users,
             tweet_url: 'https://twitter.com' + html.attr('data-permalink-path'),
+            timestamp: timestamp,
             created_at: Time.at(timestamp, in: '+00:00'),
-            text: inner_html.xpath("//div[@class[contains(., 'js-tweet-text-container')]]/p[@class[contains(., 'js-tweet-text')]]").first.text,
         )
       end
     end

data/lib/version.rb

@@ -1,3 +1,3 @@
 module Twitterscraper
-  VERSION = '0.5.0'
+  VERSION = '0.10.0'
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: twitterscraper-ruby
 version: !ruby/object:Gem::Version
-  version: 0.5.0
+  version: 0.10.0
 platform: ruby
 authors:
 - ts-3156
@@ -68,6 +68,7 @@ files:
 - lib/twitterscraper/logger.rb
 - lib/twitterscraper/proxy.rb
 - lib/twitterscraper/query.rb
+- lib/twitterscraper/template.rb
 - lib/twitterscraper/tweet.rb
 - lib/version.rb
 - twitterscraper-ruby.gemspec