rbitter 0.0.8

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 4f1a1e7507e94a8e8fa04802986c0c01d9277e5b
+  data.tar.gz: 410d9d4a7107d40394706f01a2f2af34b482a5b1
+SHA512:
+  metadata.gz: 7f58e87ad4cb1ddd5f31d4fe1d9f0672aaf1e48d65630b16a019f33e8b8d034f9d7ec727fce4f2e8c3d3d1154ada8845b421d0d9efba2ba2ddb7d3ccb9cb9348
+  data.tar.gz: 8331188a07b56e71855333703197016e89f634efd006e4ec5709abaf116a07afc9388ad7685f6b0bb0abda3d4b154f50f348b007f77584ad438d88ddcefa1aa5

data/.gitignore

@@ -0,0 +1,21 @@
+/.bundle/
+/.yardoc
+/Gemfile.lock
+/_yardoc/
+/coverage/
+/doc/
+/pkg/
+/spec/reports/
+/tmp/
+/build/
+/vendor/
+*.bundle
+*.so
+*.o
+*.a
+mkmf.log
+*.gem
+*.json
+*.sqlite
+.rbitter
+.rbitter/*

data/.rspec

@@ -0,0 +1,2 @@
+--color
+--require spec_helper

data/.travis.yml

@@ -0,0 +1,9 @@
+language: ruby
+rvm:
+  - 1.9.3
+  - 2.1.0
+  - 2.2.0
+  - ruby-head
+  - jruby-19mode
+  - jruby-head
+  - rbx-2

data/Gemfile

@@ -0,0 +1,11 @@
+source "https://rubygems.org"
+
+group :test do
+  gem 'rspec', '~>3.0'
+  gem 'simplecov', :require => false, :group => :test
+end
+
+gem "bundler"
+gem 'jruby-openssl', platforms: :jruby
+
+gemspec

data/LICENSE.txt

@@ -0,0 +1,22 @@
+Copyright (c) 2015 Nidev Plontra
+
+MIT License
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,65 @@
+# Rbitter #
+[![Build Status](https://travis-ci.org/nidev/rbitter.svg?branch=master)](https://travis-ci.org/nidev/rbitter)
+
+Rbitter is a Twitter streaming client specialized in tweet archiving with remote access via XMLRPC, which is written in Ruby.
+
+You can save all tweets appeared on timeline and watch them later.
+
+**Rbitter can be built into a Ruby gem by typing 'gem build rbitter.gemspec'. It is not yet uploaded due to testing.**
+
+## Requirements ##
+
+'bundle update' may install every gem Rbitter needs.
+
+Ruby 2.0.0 or above
+Sqlite3
+Mysql(MariaDB)
+
+## Configuration ##
+You can simply manipulate default configuration file by:
+
+```bash
+$ rbitter configure
+```
+
+Put your customized config.json to one of below locations.
+
+1. $HOME/config.json
+2. $HOME/.rbitter/config.json
+3. ./config.json (current folder)
+4. ./.rbitter/config.json (current folder)
+
+For location #3 and #4, they're referred when #1 and #2 are not available. In those cases, *rbitter* must be executed from the directory where config.json exists.
+
+## Set up and run ##
+1. With config.json,
+2. Get Twitter token and copy them properly.
+2. Choose preferred ActiveRecord backend.
+3. Modify default location where Twitter images are downloaded.
+4. Configure XMLRPC server
+5. Start by typing:
+
+```bash
+$ rbitter serve
+```
+## XMLRPC API ##
+
+See XMLRPC.md
+
+## TODO ##
+* Streaming Client
+  * Saving JSON dump
+* XMLRPC
+  * Receiving direct messages
+
+## Issue report or feature request ##
+It's recommended to open an issue even it seemed too small. A small flaw may result in instability or bad situation. So every feature requests/bug reports are welcomed.
+
+Please attach stack trace, Ruby version, and detail description.
+
+## Disclaimer ##
+Rbitter is intended for personal usage. Archived data should not be shared over Internet. Please keep them secure and safe, and protect privacy.
+
+Using sqlite3, please set permission to 0700 so that other users can not read your database file. Using mysql2, please take care of DB access privilege.
+
+Rbitter is not responsible for integrity of data. That is, Some tweets will be dropped accidently due to Twitter API problems or your network problems. The application does its best to recover from those problems. If you find Rbitter couldn't recover even after they're resolved, please make an issue report.

data/Rakefile

@@ -0,0 +1,8 @@
+require "bundler/gem_tasks"
+require "rspec/core/rake_task"
+
+RSpec::Core::RakeTask.new
+
+task :default => :spec
+task :basic_test => :spec
+

data/XMLRPC.md

@@ -0,0 +1,19 @@
+# What is RPC handle ? #
+## Commands ##
+### Authentication ###
+### Revoke Authentication Token ###
+### Echo ###
+### Last Active ###
+### Retriever ###
+### Statistic ###
+# How to write own RPC handle? #
+RPC handle is a Ruby class. Writing a method in Ruby class, that's it. Names of methods are treated as XMLRPC command.
+
+When you write a new class for your own RPC handle, you must inherit either Auth or NoAuth class from rpc/base.rb.
+
+* class Auth < Object: Methods in a Ruby class inheriting Auth requires *auth_key* to access.
+* class NoAuth < Object: Methods in a Ruby class inheriting NoAuth doesn't require *auth_key* and these XMLRPC commands can be called by anonymous user.
+
+Filename should start with 'rh_'. It's prefix to be autoloaded by xmlrpc.rb.
+
+Refer rpc/rh_echo.rb as an example.

data/bin/rbitter

@@ -0,0 +1,6 @@
+#!/usr/bin/env ruby
+
+require "rbitter"
+
+Rbitter.bootstrap(ARGV)
+

data/config.json.example

@@ -0,0 +1,30 @@
+{
+  "twitter": {
+    "consumer_key": "",
+    "consumer_secret": "",
+    "access_token": "",
+    "access_token_secret": ""
+  },
+  "activerecord": "sqlite3",
+  "sqlite3": {
+    "dbfile": "rbitter.sqlite"
+  },
+  "mysql2": {
+    "host": "localhost",
+    "port": 3306,
+    "dbname": "archive",
+    "username": "",
+    "password": ""
+  },
+  "media_downloader": {
+    "cacert_path": "cacerts/cacert.pem",
+    "download_dir": "imgs/"
+  },
+  "xmlrpc": {
+    "enable": true,
+    "bind_host": "0.0.0.0",
+    "bind_port": 1400,
+    "auth_password": ""
+  },
+  "auth": ["username", "password"]
+}

data/lib/rbitter/arcserver.rb

@@ -0,0 +1,97 @@
+# encoding: utf-8
+
+require "json"
+require "date"
+require "twitter"
+require "resolv"
+
+require "rbitter/records"
+require "rbitter/streaming"
+require "rbitter/dlthread"
+require "rbitter/xmlrpc"
+
+module Rbitter
+  class ArcServer
+    def initialize
+      ARSupport.connect_database
+
+      if not ARSupport.prepared?
+        puts "Initiate database table..."
+        if Rbitter.env['activerecord'] == 'mysql2'
+          ARSupport.prepare "DEFAULT CHARSET=utf8mb4"
+        else
+          ARSupport.prepare
+        end
+      end
+
+      ARSupport.update_database_scheme
+
+      @t = StreamClient.new(Rbitter.env['twitter'].dup)
+      @dt = DLThread.new(Rbitter.env['media_downloader']['download_dir'], Rbitter.env['media_downloader']['cacert_path'])
+    end
+
+    def write_marker(message)
+      Record.create({:marker => 1,
+        :marker_msg => message, 
+        :userid => nil,
+        :username => nil,
+        :tweetid => nil,
+        :tweet => nil,
+        :date => ARSupport.any_to_datestring(DateTime.now),
+        :rt_count => 0,
+        :fav_count => 0})
+    end
+
+    def write_init_marker
+      write_marker "Archiving service started"
+    end
+
+    def write_halt_marker
+      write_marker "Archiving service halted"
+    end
+
+    def main_loop
+      begin
+        write_init_marker
+        @t.run { |a|
+          record = Record.find_or_initialize_by(tweetid: a['tweetid'])
+          record.update({:marker => 0,
+            :marker_msg => "normal", 
+            :userid => a['userid'],
+            :username => a['screen_name'],
+            :tweetid => a['tweetid'],
+            :tweet => a['tweet'],
+            :date => a['date'],
+            :rt_count => a['rt_count'],
+            :fav_count => a['fav_count']})
+
+          record.save
+          @dt.execute_urls(a['media_urls'])
+        }
+      rescue Interrupt => e
+        puts ""
+        puts "Interrupted..."
+        if Rbitter.env['xmlrpc']['enable']
+          puts "Finishing RPCServer"
+          if $rpc_service_thread.alive?
+            $rpc_service_thread.terminate
+            $rpc_service_thread.join
+          end
+        end
+        exit 0
+      rescue Twitter::Error::Unauthorized => e
+        puts "Please configure your Twitter token on config.json."
+        exit -1
+      rescue Twitter::Error::ServerError, Resolv::ResolvError => e
+        puts "Service unavailable now. Retry in 5 second..."
+        sleep 5
+        retry
+      rescue Twitter::Error => e
+        puts "Twitter Error: #{e.inspect}"
+        exit -1
+      ensure
+        write_halt_marker
+      end
+    end
+  end
+end

data/lib/rbitter/console.rb

@@ -0,0 +1,61 @@
+# encoding: utf-8
+#
+# Rbitter Archive Access console (irb)
+
+require "xmlrpc/client"
+require "ripl"
+
+module Rbitter
+  class Console
+    def initialize
+      puts "Welcome to Rbitter console"
+      help
+    end
+
+    def help
+      puts "Predefined methods:"
+      puts "help - to show this message again"
+      puts "xmlrpc - to utilize xmlrpc"
+      puts "activerec - to connect and utilize Rbitter::Record"
+      puts "^D, 'exit' - to get out from here."
+    end
+
+    def activerec
+      ARSupport.connect_database
+      puts "Rbitter::Record is ready."
+    end
+
+    def exit
+      Kernel.exit(0)
+    end
+    
+    def xmlrpc *args
+      if args.empty?
+        puts "How to use: xmlrpc (command) [params in Array]"
+        puts "Ex) xmlrpc rbitter.echo [\"Hello World!\"]"
+        puts "Ex) To call XMLRPC function with zero parameter, use nil."
+        return false
+      end
+
+      cl = XMLRPC::Client.new('localhost', '/', 1400) # TODO: External address?
+      if @xmlrpc_cookie.nil?
+        @xmlrpc_cookie = "auth_key=" + cl.call('rbitter.auth', Rbitter.env['xmlrpc']['auth'][0], Rbitter.env['xmlrpc']['auth'][1])
+      end
+      
+      if cl.cookie != "auth_key="
+        if args.length <= 1 or args[1].nil?
+          cl.call(args[0])
+        else
+          cl.call(args[0], *args[1])
+        end
+      else
+        puts "Authentication failed. Check your config.json"
+      end
+    end
+
+    def start
+      Ripl.start :binding => binding
+    end
+  end
+end
+

data/lib/rbitter/default/config_json.rb

@@ -0,0 +1,37 @@
+module Rbitter
+  DEFAULT_CONFIG_JSON = <<-ENDOFJSON
+{
+  "twitter": {
+    "consumer_key": "",
+    "consumer_secret": "",
+    "access_token": "",
+    "access_token_secret": ""
+  },
+  "activerecord": "sqlite3",
+  "sqlite3": {
+    "dbfile": "rbitter.sqlite"
+  },
+  "mysql2": {
+    "host": "localhost",
+    "port": 3306,
+    "dbname": "archive",
+    "username": "",
+    "password": ""
+  },
+  "media_downloader": {
+    "cacert_path": "/cacerts/cacert.pem",
+    "download_dir": "imgs/"
+  },
+  "xmlrpc": {
+    "enable": true,
+    "bind_host": "0.0.0.0",
+    "bind_port": 1400,
+    "auth": {
+      "username": "username",
+      "password": "password"
+    },
+    "handles": ["/path/to/handles"]
+  }
+}
+ENDOFJSON
+end

data/lib/rbitter/dlthread.rb

@@ -0,0 +1,66 @@
+# encoding: utf-8
+
+require "net/http"
+require "openssl"
+
+module Rbitter
+  class DLThread
+    def initialize(dlfolder, cacert_path)
+      @dest = dlfolder
+      if not File.directory?(dlfolder)
+        warn "[dlthread] Given download location is not available for downloading."
+        warn "[dlthread] Current Dir is used instead."
+        @dest = "./"
+      end
+
+      @cacert = cacert_path
+    end
+
+    def execute_urls(urls)
+      urls.each { |url|
+        download_once(url)
+      }
+    end
+
+    private
+    def download_once(url)
+      download_task = Thread.new {
+        uri = URI.parse(url)
+        http = Net::HTTP.new(uri.host, uri.port)
+        if uri.scheme.downcase == 'https'
+          http.use_ssl = true
+          http.ca_path = @cacert_path
+          # XXX: Fix this soon as possible.
+          http.verify_mode = OpenSSL::SSL::VERIFY_NONE
+        end
+
+        http.request_get(uri.path) { |res|
+          case res
+          when Net::HTTPOK
+            fname = File.basename(uri.path)
+            if fname.nil? or fname.size < 1
+              fname = uri.path.gsub(/\//, "_")
+            end
+            puts "[fetch] remote: #{url} => local: #{fname}"
+            File.open(@dest+"/"+fname, "wb") { |file|
+              res.read_body { |chunk|
+                file.write(chunk)
+              }
+            }
+          end
+        }
+      }
+
+      download_task.run
+    end
+  end
+end
+
+# XXX: Should be moved to dlthread_spec.rb
+=begin
+if __FILE__ == $0
+  t = DLThread.new(".")
+  t.execute_urls(["https://www.google.co.kr/images/nav_logo195.png"])
+  sleep 4
+end
+=end

data/lib/rbitter/env.rb

@@ -0,0 +1,59 @@
+# encoding: utf-8
+
+require "json"
+
+module Rbitter
+  @internal_configuration = {}
+
+  class ConfigurationFileError < StandardError; end
+
+  module_function
+  def env
+    @internal_configuration
+  end
+
+  def env_reset
+    @internal_configuration.clear
+  end
+
+  def config_load path
+    open(path, 'r') { |io|
+      @internal_configuration = JSON.parse(io.read)
+    }
+    true
+  end
+
+  def config_initialize json_path=nil
+    @internal_configuration = JSON.parse("{}")
+
+    unless json_path.nil?
+      begin
+        config_load(json_path)
+        # TODO: Configuration validation
+        return @internal_configuration
+      rescue => e
+        fail ConfigurationFileError, "Provided configuration can not be loaded. (#{json_path})"
+      end
+    end
+
+    # Configuration default location
+    # Priorities
+    # 1. (current_dir)/config.json
+    # 2. (current_dir)/.rbitter/config.json
+    locations = Array.new
+    base_locations = ["config.json", ".rbitter/config.json"]
+
+    base_locations.each { |bloc|
+        locations.push File.join(Dir.pwd, bloc)
+    }
+
+    for location in locations
+      next unless File.file?(location)
+      break if config_load(location)
+    end
+
+    if @internal_configuration.empty?
+      fail ConfigurationFileError, "Can not load any configuration in [#{locations.join(', ')}]"
+    end
+  end
+end

data/lib/rbitter/libtwitter_connection_override.rb

@@ -0,0 +1,46 @@
+# encoding: utf-8
+
+require 'http/parser'
+require 'openssl'
+require 'resolv'
+
+module Twitter
+  module Streaming
+    class Connection
+      MODIFIED = true
+      attr_reader :tcp_socket_class, :ssl_socket_class
+
+      def initialize(options = {})
+        @tcp_socket_class = options.fetch(:tcp_socket_class) { TCPSocket }
+        @ssl_socket_class = options.fetch(:ssl_socket_class) { OpenSSL::SSL::SSLSocket }
+      end
+
+      def stream(request, response)
+        client_context = OpenSSL::SSL::SSLContext.new
+        client = @tcp_socket_class.new(Resolv.getaddress(request.uri.host), request.uri.port)
+        ssl_client = @ssl_socket_class.new(client, client_context)
+        ssl_client.connect
+        request.stream(ssl_client)
+
+        loop {
+          begin
+            body = ssl_client.read_nonblock(1024) # rubocop:disable AssignmentInCondition, WhileUntilModifier
+            response << body
+          rescue IO::WaitReadable
+            # The reason for setting 90 seconds as a timeout is documented on:
+            # https://dev.twitter.com/streaming/overview/connecting
+            r, w, e = IO.select([ssl_client], [], [], 90)
+            if r.nil?
+              # If timeout occurs
+              ssl_client.close
+              raise Twitter::Error::ServerError.new("Connection stalled")
+            else
+              # If socket is readable
+              retry
+            end
+          end
+        }
+      end
+    end
+  end
+end

data/lib/rbitter/records.rb

@@ -0,0 +1,109 @@
+# encoding: utf-8
+
+require "active_record"
+require "date"
+
+module Rbitter
+  class Record < ActiveRecord::Base
+  end
+end
+
+module ARSupport
+  SCHEME_VERSION = 20150327
+  SCHEME = {
+    :marker => :integer, # 0 normal, 2 cut, 1 resume
+    :marker_msg => :string, # == 0 success, == 2 w/ message
+    :userid => :integer,
+    :username => :string,
+    :tweetid => :integer,
+    :tweet => :text, # with url unpacked
+    :date => :datetime,
+    :rt_count => :integer,
+    :fav_count => :integer
+  }
+
+  module_function
+  def prepared?
+    ActiveRecord::Base.connection.table_exists?(:records)
+  end
+
+  def connect_database
+    if Rbitter.env['activerecord'] == 'sqlite3'
+      warn "Warning: If you enable XMLRPC access, using sqlite is not recommended."
+      warn "Warning: Random crash can happen because of concurrency."
+      
+      if RUBY_PLATFORM == 'java'
+        require "jdbc/sqlite3"
+        Jdbc::SQLite3.load_driver
+        ActiveRecord::Base.establish_connection(
+          adapter: 'jdbcsqlite3',
+          database: Rbitter.env['sqlite3']['dbfile'],
+          timeout: 10000) # Long timeout for slow computer
+      else
+        ActiveRecord::Base.establish_connection(
+          adapter: 'sqlite3',
+          database: Rbitter.env['sqlite3']['dbfile'],
+          timeout: 10000) # Long timeout for slow computer
+      end
+    elsif Rbitter.env['activerecord'] == 'mysql2'
+      Jdbc::MySQL.load_driver if RUBY_PLATFORM == 'java'
+      
+      ActiveRecord::Base.establish_connection(
+        adapter: (RUBY_PLATFORM == 'java' ? 'jdbcmysql' : 'mysql2'),
+        host: Rbitter.env['mysql2']['host'],
+        port: Rbitter.env['mysql2']['port'],
+        database: Rbitter.env['mysql2']['dbname'],
+        username: Rbitter.env['mysql2']['username'],
+        password: Rbitter.env['mysql2']['password'],
+        encoding: "utf8mb4",
+        collation: "utf8mb4_unicode_ci")
+    else
+      raise RuntimeException.new("Unknown configuration value. 'activerecord' value should be sqlite3 or mysql2.")
+    end
+  end
+
+  def update_database_scheme
+    current_version = ActiveRecord::Migrator.current_version
+    if current_version < SCHEME_VERSION
+      warn "[records] Your ActiveRecord scheme is outdated."
+      warn "[records] Migrate... #{current_version} => #{SCHEME_VERSION}"
+      ActiveRecord::Migrator.migrate(File.expand_path("../records_migrate", __FILE__), SCHEME_VERSION)
+    end
+  end
+
+  def prepare option_string=""
+    ActiveRecord::Schema.define(version: SCHEME_VERSION) {
+      # MySQL specific option_string:
+      # utf8mb4 -> supporting UTF-8 4-byte characters (i.e. Emoji)
+      create_table(:records, { :options => option_string }) do |t|
+        SCHEME.each_key { |column|
+          case SCHEME[column]
+          when :string
+            t.string column
+          when :integer
+            t.integer column, :limit => 8
+          when :datetime
+            t.datetime column
+          when :text
+            t.text column
+          else
+            puts "Unexpected column type '#{SCHEME[column]}' of #{column}"
+          end
+        }
+      end
+
+      add_index :records, :tweetid
+    }
+  end
+
+  def any_to_datestring(obj)
+    if obj.is_a?(String)
+      # try to parse it
+      DateTime.parse(obj).strftime("%Y-%m-%d %H:%M:%S")
+    elsif obj.is_a?(DateTime) or obj.is_a?(Time)
+      obj.strftime("%Y-%m-%d %H:%M:%S")
+    else
+      raise ArgumentError.new("Can\'t automatically extract DateTime info")
+    end
+  end
+end