rbitter 0.1.2-java

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 157a4cd2b13330fa06547688fc8cce5d22c30670
+  data.tar.gz: d12d38c95d8fd84047cb6b3ee2805dde1d0cd7af
+SHA512:
+  metadata.gz: fbd300e20e5fdf8e711c3fdb38552d144552f7e84898a3274363de2267e9643dca13ddf888c1b8163b384c67de959839f00acc08db27568cdb962a280ffbf559
+  data.tar.gz: 2095fe34daf61fa83bfaa6fd1d276a24bf0e917591adbb0ea78c08788a8428fb4fa00200222236cadf3793b72beca9c5b00a05e63d27c1b25f94ad8e6d1d6d8b

data/.gitignore

@@ -0,0 +1,21 @@
+/.bundle/
+/.yardoc
+/Gemfile.lock
+/_yardoc/
+/coverage/
+/doc/
+/pkg/
+/spec/reports/
+/tmp/
+/build/
+/vendor/
+*.bundle
+*.so
+*.o
+*.a
+mkmf.log
+*.gem
+*.json
+*.sqlite
+.rbitter
+.rbitter/*

data/.rspec

@@ -0,0 +1,2 @@
+--color
+--require spec_helper

data/.travis.yml

@@ -0,0 +1,15 @@
+language: ruby
+rvm:
+  - 1.9.3
+  - 2.1.0
+  - 2.2.0
+  - jruby-19mode
+  - jruby-head
+  - rbx-2
+
+env: LANG="en_US.UTF-8" LC_ALL="en_US.UTF-8"
+
+matrix:
+  allow_failures:
+    - rvm: ruby-head
+    - rvm: rbx-2

data/Gemfile

@@ -0,0 +1,12 @@
+source "https://rubygems.org"
+
+group :test do
+  gem 'rspec', '~>3.0'
+  gem 'simplecov', :require => false, :group => :test
+  gem 'coveralls', :require => false
+end
+
+gem "bundler"
+gem 'jruby-openssl', platforms: :jruby
+
+gemspec

data/LICENSE.txt

@@ -0,0 +1,22 @@
+Copyright (c) 2015 Nidev Plontra
+
+MIT License
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,72 @@
+# Rbitter #
+[![Build Status](https://travis-ci.org/nidev/rbitter.svg?branch=master)](https://travis-ci.org/nidev/rbitter)
+[![Gem Version](https://badge.fury.io/rb/rbitter.svg)](http://badge.fury.io/rb/rbitter)
+[![Coverage Status](https://coveralls.io/repos/nidev/rbitter/badge.svg?branch=master)](https://coveralls.io/r/nidev/rbitter?branch=master)
+
+Rbitter is a Twitter streaming client specialized in tweet archiving with remote access via XMLRPC, which is written in Ruby.
+
+You can save all tweets appeared on your home timeline and watch them later.
+
+## Requirements ##
+Rbitter is tested under [MRI](https://www.ruby-lang.org), [JRuby](http://jruby.org) and [Rubinius](http://rubini.us). So you can choose one of these implementations.
+
+It can run under both Windows and Linux.
+
+Gem dependencies will be installed during installation of this gem.
+
+* Ruby 1.9.3 or above
+* Sqlite3
+* Mysql(MariaDB)
+
+## Configuration ##
+You can simply manipulate default configuration file by:
+
+```bash
+$ rbitter configure
+```
+
+Put your customized config.json to one of below locations.
+
+1. ./config.json (current folder)
+2. ./.rbitter/config.json (current folder)
+
+## Set up and run ##
+With config.json,
+
+1. Get Twitter token and copy them properly.
+2. Choose preferred ActiveRecord backend.
+3. Modify default location where Twitter images are downloaded.
+4. Configure XMLRPC server
+5. Start by typing:
+
+```bash
+$ rbitter serve
+```
+## XMLRPC API ##
+
+See XMLRPC.md
+
+## TODO ##
+* Streaming Client
+  * Saving JSON dump
+* XMLRPC
+  * Receiving direct messages
+
+## Issue report or feature request ##
+It's recommended to open an issue even it seemed too small. A small flaw may result in instability or bad situation. So every feature requests/bug reports are welcomed.
+
+Please attach stack trace, Ruby version, Rbitter version, and detail description.
+
+If you installed from [RubyGems], please report with Git tag. Git tag is your gem version.
+
+[RubyGems]: https://rubygems.org
+
+## Application Versioning ##
+Rbitter follows [Sementic Versioning 2.0](http://semver.org/spec/v2.0.0.html).
+
+## Disclaimer ##
+Rbitter is intended for personal usage. Archived data should not be shared over Internet. Please keep them secure and safe, and protect privacy.
+
+Using sqlite3, please set permission to 0700 so that other users can not read your database file. Using mysql2, please take care of DB access privilege.
+
+Rbitter is not responsible for integrity of data. That is, Some tweets will be dropped accidently due to Twitter API problems or your network problems. The application does its best to recover from those problems. If you find Rbitter couldn't recover even after they're resolved, please make an issue report.

data/Rakefile

@@ -0,0 +1,8 @@
+require "bundler/gem_tasks"
+require "rspec/core/rake_task"
+
+RSpec::Core::RakeTask.new
+
+task :default => :spec
+task :basic_test => :spec
+

data/XMLRPC.md

@@ -0,0 +1,19 @@
+# What is RPC handle ? #
+## Commands ##
+### Authentication ###
+### Revoke Authentication Token ###
+### Echo ###
+### Last Active ###
+### Retriever ###
+### Statistic ###
+# How to write own RPC handle? #
+RPC handle is a Ruby class. Writing a method in Ruby class, that's it. Names of methods are treated as XMLRPC command.
+
+When you write a new class for your own RPC handle, you must inherit either Auth or NoAuth class from rpc/base.rb.
+
+* class Auth < Object: Methods in a Ruby class inheriting Auth requires *auth_key* to access.
+* class NoAuth < Object: Methods in a Ruby class inheriting NoAuth doesn't require *auth_key* and these XMLRPC commands can be called by anonymous user.
+
+Filename should start with 'rh_'. It's prefix to be autoloaded by xmlrpc.rb.
+
+Refer rpc/rh_echo.rb as an example.

data/bin/rbitter

@@ -0,0 +1,20 @@
+#!/usr/bin/env ruby
+
+require "rbitter"
+
+Rbitter.rbitter_header
+
+if ARGV.length < 1 || (not Rbitter::BOOTSTRAP_ARGS.include?(ARGV[0]))
+  Rbitter.rbitter_help_msg
+  exit -1
+else
+  Rbitter.bootstrap(ARGV)
+
+  if ARGV[0] == "configure"
+    puts "Writing config.json done"
+    puts "You can put config.json one of these locations:"
+    puts "[1] config.json (current folder)"
+    puts "[2] .rbitter/config.json (current folder)"
+  end
+end
+

data/lib/rbitter.rb

@@ -0,0 +1,86 @@
+# encoding: utf-8
+#
+
+require "rbitter/version"
+require "rbitter/arcserver"
+require "rbitter/env"
+require "rbitter/console"
+require "rbitter/xmlrpc"
+
+module Rbitter
+  BOOTSTRAP_ARGS = ['configure', 'console', 'help', 'logs', 'serve']
+
+  def self.rbitter_header
+    puts "Rbitter #{VERSION} on #{RUBY_VERSION} (#{RUBY_PLATFORM})"
+  end
+
+  def self.rbitter_help_msg  
+    puts "Rbitter is a Twitter streaming archiver, with XMLRPC access."
+    puts "-------"
+    puts "Usage: rbitter (application-mode)"
+    puts "application-mode's are:"
+    puts "|- serve    : Launch Rbitter full system (Streaming + Database + XMLRPC)"
+    puts "|- console  : Launch console application utilizing XMLRPC"
+    puts "|- configure: Write default configuration file 'config.json' in current folder"
+    puts "|- help     : Show this message"
+    puts "`- logs     : Show Rbitter internal logs"
+  end
+
+  def self.prebootstrap
+    # Due to stalled socket problem, If unpatched twitter gem is installed.
+    # Twitter::Streaming::Connection will be monkey-patched.
+    patch_required = false
+
+    if Twitter::Version.const_defined?(:MAJOR)
+      b5_version = Twitter::Version::MAJOR * 10000
+      + Twitter::Version::MINOR * 100 + Twitter::Version::PATCH
+      if b5_version <= 51400
+        warn "[rbitter] Monkey-patching Twitter::Streaming::Connection"
+        warn "[rbitter] Please upgrade twitter gem to apply streaming read timeout"
+        patch_required = true
+      end
+    else
+      b6_version = Twitter::Version.to_a
+      if b6_version[0] <= 6 and b6_version[1] <= 0 and b6_version[2] <= 0
+        patch_required = true
+      end
+    end
+
+    require "rbitter/libtwitter_connection_override" if patch_required
+  end
+
+  def self.bootstrap_configs
+    require "rbitter/default/config_json"
+
+    open(File.join(Dir.pwd, "config.json"), "w") { |io|
+      io.write(DEFAULT_CONFIG_JSON)
+    }
+  end
+
+  def self.bootstrap args=[]
+    return nil if args.length < 1
+    
+    if args[0] == "serve"
+      prebootstrap
+
+      Rbitter.config_initialize
+      
+      archive_server = Rbitter::ArcServer.new
+      archive_server.main_loop
+    elsif args[0] == "help"
+      rbitter_help_msg
+    elsif args[0] == "configure"
+      bootstrap_configs
+    elsif args[0] == "console"
+      Rbitter.config_initialize
+
+      con = Rbitter::Console.new
+      con.start
+    elsif args[0] == "logs"
+      # show log in stdout
+      puts "Log buffer feature is in heavy development. Sorry."
+    else
+      fail StandardError, "Invalid bootstrap parameter: #{args[0]}"
+    end
+  end
+end

data/lib/rbitter/arcserver.rb

@@ -0,0 +1,130 @@
+# encoding: utf-8
+
+require "json"
+require "date"
+require "twitter"
+require "resolv"
+
+require "rbitter/records"
+require "rbitter/streaming"
+require "rbitter/dlthread"
+require "rbitter/xmlrpc"
+
+module Rbitter
+  class ArcServer
+    LOG_NORMAL = 0
+    LOG_INIT = 1
+    LOG_HALT = 2
+
+    def initialize(xmlrpcd_class = Rbitter::RPCServer)
+      @xmlrpcd_class = xmlrpcd_class
+
+      ARSupport.connect_database
+
+      if not ARSupport.prepared?
+        puts "Initiate database table..."
+        if Rbitter['activerecord'] == 'mysql2'
+          ARSupport.prepare "DEFAULT CHARSET=utf8mb4"
+        else
+          ARSupport.prepare
+        end
+      end
+
+      ARSupport.update_database_scheme
+
+      @dt = DLThread.new(
+        Rbitter['media_downloader']['download_dir'],
+        Rbitter['media_downloader']['large_image'])
+    end
+
+    def xmlrpcd_start
+      if Rbitter['xmlrpc']['enable']
+        @rpc_service = Thread.new {
+          rpc_server = @xmlrpcd_class.new(Rbitter['xmlrpc']['bind_host'], Rbitter['xmlrpc']['bind_port'])
+          rpc_server.main_loop
+        }
+      else
+        @rpc_service = nil
+      end
+    end
+
+    def xmlrpcd_stop
+      unless @rpc_service.nil?
+        if @rpc_service.alive?
+          puts "Finishing RPCServer (impl: #{@xmlrpcd_class})"
+          @rpc_service.terminate
+          @rpc_service.join
+          @rpc_service = nil
+        end
+      end
+    end
+
+    def mark(code, message)
+      Record.create({:marker => code,
+        :marker_msg => message, 
+        :userid => nil,
+        :username => nil,
+        :tweetid => nil,
+        :replyto => nil,
+        :tweet => nil,
+        :date => ARSupport.any_to_datestring(DateTime.now),
+        :rt_count => 0,
+        :fav_count => 0})
+    end
+
+    def write_init_marker
+      mark(LOG_INIT, "Archiving service started")
+    end
+
+    def write_halt_marker
+      mark(LOG_HALT, "Archiving service halted")
+    end
+
+    def main_loop(streaming_adapter = Rbitter::StreamClient)
+      xmlrpcd_start if Rbitter['xmlrpc']['enable']
+
+      begin
+        write_init_marker
+
+        streaming_adapter.new(Rbitter['twitter'].dup).run { |a|
+          @dt << a['media_urls']
+
+          record = Record.find_or_initialize_by(tweetid: a['tweetid'])
+          record.update({:marker => 0,
+            :marker_msg => "normal", 
+            :userid => a['userid'],
+            :username => a['screen_name'],
+            :tweetid => a['tweetid'],
+            :replyto => a['replyto'],
+            :tweet => a['tweet'],
+            :date => a['date'],
+            :rt_count => a['rt_count'],
+            :fav_count => a['fav_count']})
+
+          record.save
+        }
+      rescue Interrupt => e
+        puts ""
+        puts "Interrupted..."
+      rescue Twitter::Error::Unauthorized => e
+        warn "Twitter access unauthorized:"
+        warn "  Possible solutions"
+        warn "  1. Configure Twitter token on config.json"
+        warn "  2. Check system time (Time is important on authentication)"
+        warn "  3. Check Twitter account status"
+      rescue Twitter::Error::ServerError, Resolv::ResolvError => e
+        puts "Service unavailable now. Retry in 5 second..."
+        sleep 5
+        retry
+      rescue Twitter::Error => e
+        warn "Twitter Error: #{e.inspect}"
+        warn "Main loop of ArcServer halted."
+      ensure
+        xmlrpcd_stop if Rbitter['xmlrpc']['enable']
+        @dt.job_cleanup
+
+        write_halt_marker
+      end
+    end
+  end
+end

data/lib/rbitter/console.rb

@@ -0,0 +1,93 @@
+# encoding: utf-8
+#
+# Rbitter Archive Access console (irb)
+
+require "xmlrpc/client"
+require "rbitter/version"
+require "ripl"
+
+module Rbitter
+  class Console
+    def initialize
+      puts "Rbitter console #{Rbitter::VERSION}"
+      help
+    end
+
+    def help
+      puts "Predefined methods:"
+      puts "ar - shortcut to call Rbitter::Record"
+      puts "connect_ar - Prepare Rbitter::Record to be ready"
+      puts "csv_backup - export Rbitter::Record into comma-separated values"
+      puts "help - to show this message again"
+      puts "xmlrpc - send xmlrpc command to destination"
+      puts "xmlrpc_dest - set destination for xmlrpc command"
+      puts "^D, 'exit' to exit from here"
+    end
+
+    def connect_ar
+      ARSupport.connect_database
+      puts "Rbitter::Record is ready."
+    end
+
+    def csv_backup *args
+      if args.length < 0
+        puts "Usage: csv_backup('filename.csv')"
+        puts "Estimated running time depends on system environment"
+      else
+        ARSupport.export_to_csv(args[0])
+      end
+    end
+
+    def ar
+      Rbitter::Record
+    end
+
+    def exit
+      Kernel.exit(0)
+    end
+    
+    def xmlrpc_dest args={}
+      if args.empty?
+        puts "Usage: xmlrpc_dest({ :rpchost => '', :rpcpath => '', :rpcport => 1400,"
+        puts "                   :xmlrpc_auth_id => '', xmlrpc_auth_password => '' })"
+      end
+
+      @rpchost = args.fetch(:rpchost) { "127.0.0.1" }
+      @rpcpath = args.fetch(:rpcpath) { "/" }
+      @rpcport = args.fetch(:rpcport) { 1400 }
+
+      cl = XMLRPC::Client.new(@rpchost, @rpcpath, @rpcport)
+      @xmlrpc_cookie = "auth_key=" + cl.call('rbitter.auth',
+        args.fetch(:xmlrpc_auth_id) { Rbitter.env['xmlrpc']['auth'][0] },
+        args.fetch(:xmlrpc_auth_password) { Rbitter.env['xmlrpc']['auth'][1] } )
+
+      if @xmlrpc_cookie != "auth_key="
+        puts "Authentication completed"
+      else
+        puts "Authentication failed"
+      end
+    end
+
+    def xmlrpc *args
+      if args.empty?
+        puts "Usage: xmlrpc(command, [params in Array])"
+        puts "Ex) xmlrpc(\'rbitter.echo\',' [\"Hello World!\"])"
+        puts "Please configure XMLRPC destination with xmlrpc_dest method"
+        return false
+      end
+
+      cl = XMLRPC::Client.new(@rpchost, @rpcpath, @rpcport, @xmlrpc_cookie)
+      
+      if args.length <= 1 or args[1].nil?
+        cl.call(args[0])
+      else
+        cl.call(args[0], *args[1])
+      end
+    end
+
+    def start
+      Ripl.start :binding => binding
+    end
+  end
+end
+