twitter_ebooks_poll 3.2.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: f58b5801ce8261adf450070f37923c493ed945cb
+  data.tar.gz: f29a8c97eaf6b52022954e411e74522ec306b7d0
+SHA512:
+  metadata.gz: 3513c69c90999452b08e9b65206ac4c04b0a7dacd5c54fb7a71c9b320aab6f4f12730a9874ff714d4cb6c2b962c017d4f76200425af9d51bce9f1a34e1190f51
+  data.tar.gz: d4459320f200b00272603b72fcfb5d82dfc1feb2553a3107c4f51ff86646a7ffecde6877fc37fb5ae0e74346e33ad77dc236c42ffd76586450385f8bf55127f3

data/.gitattributes

@@ -0,0 +1,2 @@
+# Require \n style line endings
+* text eol=lf

data/.gitignore

@@ -0,0 +1,6 @@
+.*.swp
+Gemfile.lock
+pkg
+.yardoc
+doc
+.idea

data/.rspec

@@ -0,0 +1 @@
+--color

data/.travis.yml

@@ -0,0 +1,8 @@
+language: ruby
+rvm:
+  - 2.2.0
+script:
+  - rspec spec
+notifications:
+  email:
+    - ebooks@zwergenmaschine.de

data/Gemfile

@@ -0,0 +1,4 @@
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in twitter_ebooks.gemspec
+gemspec

data/LICENSE

@@ -0,0 +1,22 @@
+Copyright (c) 2013 Jaiden Mispy
+
+MIT License
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,167 @@
+
+## Fork
+
+No one I know had fun in changing to the shiny new Twitter Account Actitvity API, since this means to run your own web server with all consequences. So I started hacking into the code to change the streaming API to polling the home and mentions timeline.
+
+# twitter\_ebooks\_poll
+
+[![Gem Version](https://badge.fury.io/rb/twitter_ebooks_poll.svg)](http://badge.fury.io/rb/twitter_ebooks_poll)
+[![Build Status](https://travis-ci.org/grindcrank/twitter_ebooks.svg)](https://travis-ci.org/grindcrank/twitter_ebooks)
+
+A framework for building interactive twitterbots which respond to mentions/DMs. See [ebooks_example](https://github.com/mispy/ebooks_example) for a fully-fledged bot definition.
+
+## New in 3.2
+
+(There was no changelog for 3.1, but the version nevertheless existed.)
+
+- Uses home and mentions polling instead of streams.
+
+## New in 3.0
+
+- About 80% less memory and storage use for models
+- Bots run in their own threads (no eventmachine), and startup is parallelized
+- Bots start with `ebooks start`, and no longer die on unhandled exceptions
+- `ebooks auth` command will create new access tokens, for running multiple bots
+- `ebooks console` starts a ruby interpreter with bots loaded (see Ebooks::Bot.all)
+- Replies are slightly rate-limited to prevent infinite bot convos
+- Non-participating users in a mention chain will be dropped after a few tweets
+- [API documentation](http://rdoc.info/github/mispy/twitter_ebooks) and tests
+
+Note that 3.0 is not backwards compatible with 2.x, so upgrade carefully! In particular, **make sure to regenerate your models** since the storage format changed.
+
+## Installation
+
+Requires Ruby 2.1+. Ruby 2.3+ is recommended.
+
+```bash
+gem install twitter_ebooks
+```
+
+## Setting up a bot
+
+Run `ebooks new <reponame>` to generate a new repository containing a sample bots.rb file, which looks like this:
+
+``` ruby
+# This is an example bot definition with event handlers commented out
+# You can define and instantiate as many bots as you like
+
+class MyBot < Ebooks::Bot
+  # Configuration here applies to all MyBots
+  def configure
+    # Consumer details come from registering an app at https://dev.twitter.com/
+    # Once you have consumer details, use "ebooks auth" for new access tokens
+    self.consumer_key = "" # Your app consumer key
+    self.consumer_secret = "" # Your app consumer secret
+
+    # Users to block instead of interacting with
+    self.blacklist = ['tnietzschequote']
+
+    # Range in seconds to randomize delay when bot.delay is called
+    self.delay_range = 1..6
+  end
+
+  def on_startup
+    scheduler.every '24h' do
+      # Tweet something every 24 hours
+      # See https://github.com/jmettraux/rufus-scheduler
+      # tweet("hi")
+      # pictweet("hi", "cuteselfie.jpg")
+    end
+  end
+
+  def on_message(dm)
+    # Reply to a DM
+    # reply(dm, "secret secrets")
+  end
+
+  def on_follow(user)
+    # Follow a user back
+    # follow(user.screen_name)
+  end
+
+  def on_mention(tweet)
+    # Reply to a mention
+    # reply(tweet, meta(tweet).reply_prefix + "oh hullo")
+  end
+
+  def on_timeline(tweet)
+    # Reply to a tweet in the bot's timeline
+    # reply(tweet, meta(tweet).reply_prefix + "nice tweet")
+  end
+
+  def on_favorite(user, tweet)
+    # Follow user who just favorited bot's tweet
+    # follow(user.screen_name)
+  end
+
+  def on_retweet(tweet)
+    # Follow user who just retweeted bot's tweet
+    # follow(tweet.user.screen_name)
+  end
+end
+
+# Make a MyBot and attach it to an account
+MyBot.new("abby_ebooks") do |bot|
+  bot.access_token = "" # Token connecting the app to this account
+  bot.access_token_secret = "" # Secret connecting the app to this account
+end
+```
+
+`ebooks start` will run all defined bots in their own threads. The easiest way to run bots in a semi-permanent fashion is with [Heroku](https://www.heroku.com); just make an app, push the bot repository to it, enable a worker process in the web interface and it ought to chug along merrily forever.
+
+The underlying streaming and REST clients from the [twitter gem](https://github.com/sferik/twitter) can be accessed at `bot.stream` and `bot.twitter` respectively.
+
+## Archiving accounts
+
+twitter\_ebooks comes with a syncing tool to download and then incrementally update a local json archive of a user's tweets (in this case, my good friend @0xabad1dea):
+
+``` zsh
+➜  ebooks archive 0xabad1dea corpus/0xabad1dea.json
+Currently 20209 tweets for 0xabad1dea
+Received 67 new tweets
+```
+
+The first time you'll run this, it'll ask for auth details to connect with. Due to API limitations, for users with high numbers of tweets it may not be possible to get their entire history in the initial download. However, so long as you run it frequently enough you can maintain a perfect copy indefinitely into the future.
+
+## Text models
+
+In order to use the included text modeling, you'll first need to preprocess your archive into a more efficient form:
+
+``` zsh
+➜  ebooks consume corpus/0xabad1dea.json
+Reading json corpus from corpus/0xabad1dea.json
+Removing commented lines and sorting mentions
+Segmenting text into sentences
+Tokenizing 7075 statements and 17947 mentions
+Ranking keywords
+Corpus consumed to model/0xabad1dea.model
+```
+
+Notably, this works with both json tweet archives and plaintext files (based on file extension), so you can make a model out of any kind of text.
+
+Text files use newlines and full stops to seperate statements.
+
+Once you have a model, the primary use is to produce statements and related responses to input, using a pseudo-Markov generator:
+
+``` ruby
+> model = Ebooks::Model.load("model/0xabad1dea.model")
+> model.make_statement(140)
+=> "My Terrible Netbook may be the kind of person who buys Starbucks, but this Rackspace vuln is pretty straight up a backdoor"
+> model.make_response("The NSA is coming!", 130)
+=> "Hey - someone who claims to be an NSA conspiracy"
+```
+
+The secondary function is the "interesting keywords" list. For example, I use this to determine whether a bot wants to fav/retweet/reply to something in its timeline:
+
+``` ruby
+top100 = model.keywords.take(100)
+tokens = Ebooks::NLP.tokenize(tweet.text)
+
+if tokens.find { |t| top100.include?(t) }
+  favorite(tweet)
+end
+```
+
+## Bot niceness
+
+twitter_ebooks will drop bystanders from mentions for you and avoid infinite bot conversations, but it won't prevent you from doing a lot of other spammy things. Make sure your bot is a good and polite citizen!

data/Rakefile

@@ -0,0 +1,2 @@
+#!/usr/bin/env rake
+require "bundler/gem_tasks"

data/bin/ebooks

@@ -0,0 +1,449 @@
+#!/usr/bin/env ruby
+# encoding: utf-8
+
+require 'twitter_ebooks'
+require 'ostruct'
+require 'fileutils'
+
+module Ebooks::Util
+  def pretty_exception(e)
+
+  end
+end
+
+module Ebooks::CLI
+  $LOAD_PATH.unshift(Dir.pwd)
+  puts $LOAD_PATH
+  HELP = OpenStruct.new
+
+  HELP.default = <<STR
+Usage:
+     ebooks help <command>
+
+     ebooks new <reponame>
+     ebooks s[tart]
+     ebooks c[onsole]
+     ebooks auth
+     ebooks consume <corpus_path> [corpus_path2] [...]
+     ebooks consume-all <model_name> <corpus_path> [corpus_path2] [...]
+     ebooks append <model_name> <corpus_path>
+     ebooks gen <model_path> [input]
+     ebooks archive <username> [path]
+     ebooks sync <botname> [username]
+     ebooks tweet <model_path> <botname>
+     ebooks version
+STR
+
+  def self.help(command=nil)
+    if command.nil?
+      log HELP.default
+    else
+      log HELP[command].gsub(/^ {4}/, '')
+    end
+  end
+
+  HELP.new = <<-STR
+    Usage: ebooks new <reponame>
+
+    Creates a new skeleton repository defining a template bot in
+    the current working directory specified by <reponame>.
+  STR
+
+  def self.new(reponame)
+    if reponame.nil?
+      help :new
+      exit 1
+    end
+
+    path = "./#{reponame}"
+
+    if File.exists?(path)
+      log "#{path} already exists. Please remove if you want to recreate."
+      exit 1
+    end
+
+    FileUtils.cp_r(Ebooks::SKELETON_PATH, path)
+    FileUtils.mv(File.join(path, 'gitignore'), File.join(path, '.gitignore'))
+
+    File.open(File.join(path, 'bots.rb'), 'w') do |f|
+      template = File.read(File.join(Ebooks::SKELETON_PATH, 'bots.rb'))
+      f.write(template.gsub("{{BOT_NAME}}", reponame))
+    end
+
+    File.open(File.join(path, 'Gemfile'), 'w') do |f|
+      template = File.read(File.join(Ebooks::SKELETON_PATH, 'Gemfile'))
+      f.write(template.gsub("{{RUBY_VERSION}}", RUBY_VERSION))
+    end
+
+    log "New twitter_ebooks app created at #{reponame}"
+  end
+
+  HELP.consume = <<-STR
+    Usage: ebooks consume <corpus_path> [corpus_path2] [...]
+
+    Processes some number of text files or json tweet corpuses
+    into usable models. These will be output at model/<corpus_name>.model
+  STR
+
+  def self.consume(pathes)
+    if pathes.empty?
+      help :consume
+      exit 1
+    end
+
+    pathes.each do |path|
+      filename = File.basename(path)
+      shortname = filename.split('.')[0..-2].join('.')
+
+      FileUtils.mkdir_p(File.join(APP_PATH, 'model'))
+      outpath = File.join(APP_PATH, 'model', "#{shortname}.model")
+
+      Ebooks::Model.consume(path).save(outpath)
+      log "Corpus consumed to #{outpath}"
+    end
+  end
+
+  HELP.consume_all = <<-STR
+    Usage: ebooks consume-all <model_name> <corpus_path> [corpus_path2] [...]
+
+    Processes some number of text files or json tweet corpuses
+    into one usable model. It will be output at model/<model_name>.model
+  STR
+
+  def self.consume_all(name, paths)
+    if paths.empty?
+      help :consume_all
+      exit 1
+    end
+
+    outpath = File.join(APP_PATH, 'model', "#{name}.model")
+    Ebooks::Model.consume_all(paths).save(outpath)
+    log "Corpuses consumed to #{outpath}"
+  end
+
+  HELP.append = <<-STR
+    Usage: ebooks append <model_name> <corpus_path>
+
+    Process then append the provided corpus to the model
+    instead of overwriting.
+  STR
+
+  def self.append(name, path)
+    if !name || !path
+      help :append
+      exit 1
+    end
+
+    Ebooks::Model.consume(path).append(File.join(APP_PATH,'model',"#{name}.model"))
+    log "Corpus appended to #{name}.model"
+  end
+
+
+  HELP.jsonify = <<-STR
+    Usage: ebooks jsonify <tweets.csv> [tweets.csv2] [...]
+
+    Takes a csv twitter archive and converts it to json.
+  STR
+
+  def self.jsonify(paths)
+    if paths.empty?
+      log usage
+      exit
+    end
+
+    paths.each do |path|
+      name = File.basename(path).split('.')[0]
+      new_path = name + ".json"
+
+      tweets = []
+      id = nil
+      if path.split('.')[-1] == "csv" #from twitter archive
+        csv_archive = CSV.read(path, :headers=>:first_row)
+        tweets = csv_archive.map do |tweet|
+          { text: tweet['text'], id: tweet['tweet_id'] }
+        end
+      else
+        File.read(path).split("\n").each do |l|
+          if l.start_with?('# ')
+            id = l.split('# ')[-1]
+          else
+            tweet = { text: l }
+            if id
+              tweet[:id] = id
+              id = nil
+            end
+            tweets << tweet
+          end
+        end
+      end
+
+      File.open(new_path, 'w') do |f|
+        log "Writing #{tweets.length} tweets to #{new_path}"
+        f.write(JSON.pretty_generate(tweets))
+      end
+    end
+  end
+
+
+  HELP.gen = <<-STR
+    Usage: ebooks gen <model_path> [input]
+
+    Make a test tweet from the processed model at <model_path>.
+    Will respond to input if provided.
+  STR
+
+  def self.gen(model_path, input)
+    if model_path.nil?
+      help :gen
+      exit 1
+    end
+
+    model = Ebooks::Model.load(model_path)
+    if input && !input.empty?
+      puts "@cmd " + model.make_response(input, 135)
+    else
+      puts model.make_statement
+    end
+  end
+
+  HELP.archive = <<-STR
+    Usage: ebooks archive <username> [outpath]
+
+    Downloads a json corpus of the <username>'s tweets.
+    Output defaults to corpus/<username>.json
+    Due to API limitations, this can only receive up to ~3000 tweets
+    into the past.
+
+    The first time you run archive, you will need to enter the auth
+    details of some account to use for accessing the API. This info
+    will then be stored in ~/.ebooksrc for later use, and can be
+    modified there if needed.
+  STR
+
+  def self.archive(username, outpath=nil)
+    if username.nil?
+      help :archive
+      exit 1
+    end
+
+    Ebooks::Archive.new(username, outpath).sync
+  end
+
+    HELP.sync = <<-STR
+    Usage: ebooks sync <botname> <username>
+
+    Copies and flips <username>'s avatar and cover photo, uploading them to <botname>'s profile.
+
+    Stores saved avatar's and covers in image/.
+
+  STR
+
+  def self.sync(botname, username)
+    if botname.nil?
+      help :sync
+      exit 1
+    end
+
+    load File.join(APP_PATH, 'bots.rb')
+    Ebooks::Sync::run(botname, username)
+  end
+
+  HELP.tweet = <<-STR
+    Usage: ebooks tweet <model_path> <botname>
+
+    Sends a public tweet from the specified bot using text
+    from the processed model at <model_path>.
+  STR
+
+  def self.tweet(modelpath, botname)
+    if modelpath.nil? || botname.nil?
+      help :tweet
+      exit 1
+    end
+
+    load File.join(APP_PATH, 'bots.rb')
+    model = Ebooks::Model.load(modelpath)
+    statement = model.make_statement
+    bot = Ebooks::Bot.get(botname)
+    if bot.nil?
+      log "No such bot configured in bots.rb: #{botname}"
+      exit 1
+    end
+    bot.configure
+    bot.tweet(statement)
+  end
+
+  HELP.auth = <<-STR
+    Usage: ebooks auth
+
+    Authenticates your Twitter app for any account. By default, will
+    use the consumer key and secret from the first defined bot. You
+    can specify another by setting the CONSUMER_KEY and CONSUMER_SECRET
+    environment variables.
+  STR
+
+  def self.auth
+    consumer_key, consumer_secret = find_consumer
+    require 'oauth'
+
+    consumer = OAuth::Consumer.new(
+      consumer_key,
+      consumer_secret,
+      site: 'https://twitter.com/',
+      scheme: :header
+    )
+
+    request_token = consumer.get_request_token
+    auth_url = request_token.authorize_url()
+
+    pin = nil
+    loop do
+      log auth_url
+
+      log "Go to the above url and follow the prompts, then enter the PIN code here."
+      print "> "
+
+      pin = STDIN.gets.chomp
+
+      break unless pin.empty?
+    end
+
+    access_token = request_token.get_access_token(oauth_verifier: pin)
+
+    log "Account authorized successfully. Make sure to put these in your bots.rb!\n" +
+         "  bot.access_token = \"#{access_token.token}\"\n" +
+         "  bot.access_token_secret = \"#{access_token.secret}\""
+  end
+
+  HELP.console = <<-STR
+    Usage: ebooks c[onsole]
+
+    Starts an interactive ruby session with your bots loaded
+    and configured.
+  STR
+
+  def self.console
+    load_bots
+    require 'pry'; Ebooks.module_exec { pry }
+  end
+
+  HELP.version = <<-STR
+    Usage: ebooks version
+
+    Shows you twitter_ebooks' version number.
+  STR
+
+  def self.version
+    require File.expand_path('../../lib/twitter_ebooks/version', __FILE__)
+    log Ebooks::VERSION
+  end
+
+  HELP.start = <<-STR
+    Usage: ebooks s[tart] [botname]
+
+    Starts running bots. If botname is provided, only runs that bot.
+  STR
+
+  def self.start(botname=nil)
+    load_bots
+
+    if botname.nil?
+      bots = Ebooks::Bot.all
+    else
+      bots = Ebooks::Bot.all.select { |bot| bot.username == botname }
+      if bots.empty?
+        log "Couldn't find a defined bot for @#{botname}!"
+        exit 1
+      end
+    end
+
+    threads = []
+    bots.each do |bot|
+      threads << Thread.new { bot.prepare }
+    end
+    threads.each(&:join)
+
+    threads = []
+    bots.each do |bot|
+      threads << Thread.new do
+        bot.start
+        loop do
+          sleep 60
+        end
+      end
+    end
+    threads.each(&:join)
+  end
+
+  # Non-command methods
+
+  def self.find_consumer
+    if ENV['CONSUMER_KEY'] && ENV['CONSUMER_SECRET']
+      log "Using consumer details from environment variables:\n" +
+          "  consumer key: #{ENV['CONSUMER_KEY']}\n" +
+          "  consumer secret: #{ENV['CONSUMER_SECRET']}"
+      return [ENV['CONSUMER_KEY'], ENV['CONSUMER_SECRET']]
+    end
+
+    load_bots
+    consumer_key = nil
+    consumer_secret = nil
+    Ebooks::Bot.all.each do |bot|
+      if bot.consumer_key && bot.consumer_secret
+        consumer_key = bot.consumer_key
+        consumer_secret = bot.consumer_secret
+        log "Using consumer details from @#{bot.username}:\n" +
+            "  consumer key: #{bot.consumer_key}\n" +
+            "  consumer secret: #{bot.consumer_secret}\n"
+        return consumer_key, consumer_secret
+      end
+    end
+
+    if consumer_key.nil? || consumer_secret.nil?
+      log "Couldn't find any consumer details to auth an account with.\n" +
+          "Please either configure a bot with consumer_key and consumer_secret\n" +
+          "or provide the CONSUMER_KEY and CONSUMER_SECRET environment variables."
+      exit 1
+    end
+  end
+
+  def self.load_bots
+    load 'bots.rb'
+
+    if Ebooks::Bot.all.empty?
+      puts "Couldn't find any bots! Please make sure bots.rb instantiates at least one bot."
+    end
+  end
+
+  def self.command(args)
+    if args.length == 0
+      help
+      exit 1
+    end
+
+    case args[0]
+    when "new" then new(args[1])
+    when "consume" then consume(args[1..-1])
+    when "consume-all" then consume_all(args[1], args[2..-1])
+    when "append" then append(args[1],args[2])
+    when "gen" then gen(args[1], args[2..-1].join(' '))
+    when "archive" then archive(args[1], args[2])
+    when "sync" then sync(args[1], args[2])
+    when "tweet" then tweet(args[1], args[2])
+    when "jsonify" then jsonify(args[1..-1])
+    when "auth" then auth
+    when "console" then console
+    when "c" then console
+    when "start" then start(args[1])
+    when "s" then start(args[1])
+    when "help" then help(args[1])
+    when "version" then version
+    else
+      log "No such command '#{args[0]}'"
+      help
+      exit 1
+    end
+  end
+end
+
+Ebooks::CLI.command(ARGV)