twitter_ebooks 2.1.3 → 2.1.4

data/README.md

@@ -1,9 +1,108 @@
-# twitter\_ebooks 2.1.3
+# twitter\_ebooks 2.1.4
 
-Complete rewrite of twitter\_ebooks. Allows context-sensitive responsive bots via the Twitter streaming API, along with higher-quality ngram modeling. Still needs a bit of cleaning and documenting.
+Rewrite of my twitter\_ebooks code. While the original was solely a tweeting Markov generator, this framework helps you build any kind of interactive twitterbot which responds to mentions/DMs.
 
 ## Installation
 
 ```bash
 gem install twitter_ebooks
 ```
+
+## Setting up a bot
+
+Run `ebooks new <reponame>` to generate a new repository containing a sample bots.rb file, which looks like this:
+
+``` ruby
+# This is an example bot definition with event handlers commented out
+# You can define as many of these as you like; they will run simultaneously
+
+Ebooks::Bot.new("./test") do |bot|
+  # Consumer details come from registering an app at https://dev.twitter.com/
+  # OAuth details can be fetched with https://github.com/marcel/twurl
+  bot.consumer_key = "" # Your app consumer key
+  bot.consumer_secret = "" # Your app consumer secret
+  bot.oauth_token = "" # Token connecting the app to this account
+  bot.oauth_token_secret = "" # Secret connecting the app to this account
+
+  bot.on_message do |dm|
+    # Reply to a DM
+    # bot.reply(dm, "secret secrets")
+  end
+
+  bot.on_follow do |user|
+    # Follow a user back
+    # bot.follow(user[:screen_name])
+  end
+
+  bot.on_mention do |tweet, meta|
+    # Reply to a mention
+    # bot.reply(tweet, meta[:reply_prefix] + "oh hullo")
+  end
+
+  bot.on_timeline do |tweet, meta|
+    # Reply to a tweet in the bot's timeline
+    # bot.reply(tweet, meta[:reply_prefix] + "nice tweet")
+  end
+
+  bot.scheduler.every '24h' do
+    # Tweet something every 24 hours
+    # See https://github.com/jmettraux/rufus-scheduler
+    # bot.tweet("hi")
+  end
+end
+```
+
+Bots defined like this can be spawned by executing `run.rb` in the same directory, and will operate together in a single eventmachine loop. The easiest way to run bots in a semi-permanent fashion is with [Heroku](https://www.heroku.com); just make an app, push the bot repository to it, enable a worker process in the web interface and it ought to chug along merrily forever.
+
+## Archiving accounts
+
+twitter\_ebooks comes with a syncing tool to download and then incrementally update a local json archive of a user's tweets.
+
+``` zsh
+➜  ebooks-ebooks git:(master) ebooks archive 0xabad1dea corpus/0xabad1dea.json 
+Currently 20209 tweets for 0xabad1dea
+Received 67 new tweets
+```
+
+The first time you'll run this, it'll ask for auth details to connect with. Due to API limitations, for users with high numbers of tweets it may not be possible to get their entire history in the initial download. However, so long as you run it frequently enough you can maintain a perfect copy indefinitely into the future.
+
+## Text models
+
+In order to use the included text modeling, you'll first need to preprocess your archive into a more efficient form:
+
+``` zsh
+➜  ebooks-ebooks git:(master) ebooks consume corpus/0xabad1dea.json 
+Reading json corpus from corpus/0xabad1dea.json
+Removing commented lines and sorting mentions
+Segmenting text into sentences
+Tokenizing 7075 statements and 17947 mentions
+Ranking keywords
+Corpus consumed to model/0xabad1dea.model
+```
+
+Notably, this works with both json tweet archives and plaintext files (based on file extension), so you can make a model out of any kind of text. 
+
+Once you have a model, the primary use is to produce statements and related responses to input, using a pseudo-Markov generator:
+
+``` ruby
+> model = Ebooks::Model.load("model/0xabad1dea.model")
+> model.make_statement(140)
+=> "My Terrible Netbook may be the kind of person who buys Starbucks, but this Rackspace vuln is pretty straight up a backdoor"
+> model.make_response("The NSA is coming!", 130)
+=> "Hey - someone who claims to be an NSA conspiracy"
+```
+
+The secondary function is the "interesting keywords" list. For example, I use this to determine whether a bot wants to fav/retweet/reply to something in its timeline:
+
+``` ruby
+top100 = model.keywords.top(100)
+tokens = Ebooks::NLP.tokenize(tweet[:text])
+
+if tokens.find { |t| top100.include?(t) }
+  bot.twitter.favorite(tweet[:id])
+end
+```
+
+## Other notes
+
+If you're using Heroku, which has no persistent filesystem, automating the process of archiving, consuming and updating can be tricky. My current solution is just a daily cron job which commits and pushes for me, which is pretty hacky.

data/bin/ebooks

@@ -38,10 +38,9 @@ module Ebooks
       shortname = filename.split('.')[0..-2].join('.')
       hash = Digest::MD5.hexdigest(File.read(path))
 
-      log "Consuming text corpus: #{filename}"
       outpath = File.join(APP_PATH, 'model', "#{shortname}.model")
       Model.consume(path).save(outpath)
-      log "Corpus consumed"
+      log "Corpus consumed to #{outpath}"
     end
   end
 
@@ -73,28 +72,30 @@ module Ebooks
     bot.tweet(statement)
   end
 
-  def self.jsonify(old_path, new_path)
-    name = File.basename(old_path).split('.')[0]
-    new_path ||= name + ".json"
-
-    tweets = []
-    id = nil
-    File.read(old_path).split("\n").each do |l|
-      if l.start_with?('# ')
-        id = l.split('# ')[-1]
-      else
-        tweet = { text: l }
-        if id
-          tweet[:id] = id
-          id = nil
+  def self.jsonify(paths)
+    paths.each do |path|
+      name = File.basename(path).split('.')[0]
+      new_path = name + ".json"
+
+      tweets = []
+      id = nil
+      File.read(path).split("\n").each do |l|
+        if l.start_with?('# ')
+          id = l.split('# ')[-1]
+        else
+          tweet = { text: l }
+          if id
+            tweet[:id] = id
+            id = nil
+          end
+          tweets << tweet
         end
-        tweets << tweet
       end
-    end
 
-    File.open(new_path, 'w') do |f|
-      log "Writing #{tweets.length} tweets to #{new_path}"
-      f.write(JSON.pretty_generate(tweets))
+      File.open(new_path, 'w') do |f|
+        log "Writing #{tweets.length} tweets to #{new_path}"
+        f.write(JSON.pretty_generate(tweets))
+      end
     end
   end
 
@@ -106,7 +107,7 @@ module Ebooks
      ebooks score <model_path> <input>
      ebooks archive <@user> <outpath>
      ebooks tweet <model_path> <@bot>
-     ebooks jsonify <old_corpus_path> [new_corpus_path]
+     ebooks jsonify <old_corpus_path> [...]
 """
 
     if args.length == 0
@@ -121,7 +122,7 @@ module Ebooks
     when "score" then score(args[1], args[2..-1].join(' '))
     when "archive" then archive(args[1], args[2])
     when "tweet" then tweet(args[1], args[2])
-    when "jsonify" then jsonify(args[1], args[2])
+    when "jsonify" then jsonify(args[1..-1])
     end
   end
 end

data/lib/twitter_ebooks/model.rb

@@ -17,14 +17,22 @@ module Ebooks
       Marshal.load(File.read(path))
     end
 
-    def consume(txtpath)
-      # Record hash of source file so we know to update later
-      @hash = Digest::MD5.hexdigest(File.read(txtpath))
+    def consume(path)
+      content = File.read(path)
+      @hash = Digest::MD5.hexdigest(content)
+
+      if path.split('.')[-1] == "json"
+        log "Reading json corpus from #{path}"
+        lines = JSON.parse(content, symbolize_names: true).map do |tweet|
+          tweet[:text]
+        end
+      else
+        log "Reading plaintext corpus from #{path}"
+        lines = content.split("\n")
+      end
 
-      text = File.read(txtpath)
       log "Removing commented lines and sorting mentions"
 
-      lines = text.split("\n")
       keeping = []
       mentions = []
       lines.each do |l|

data/lib/twitter_ebooks/version.rb

@@ -1,3 +1,3 @@
 module Ebooks
-  VERSION = "2.1.3"
+  VERSION = "2.1.4"
 end

data/skeleton/Gemfile

@@ -0,0 +1,4 @@
+source 'http://rubygems.org'
+ruby '1.9.3'
+
+gem 'twitter_ebooks'

data/skeleton/Gemfile.lock

@@ -0,0 +1,70 @@
+GEM
+  remote: http://rubygems.org/
+  specs:
+    addressable (2.3.5)
+    atomic (1.1.14)
+    awesome_print (1.2.0)
+    cookiejar (0.3.0)
+    daemons (1.1.9)
+    em-http-request (1.0.3)
+      addressable (>= 2.2.3)
+      cookiejar
+      em-socksify
+      eventmachine (>= 1.0.0.beta.4)
+      http_parser.rb (>= 0.5.3)
+    em-socksify (0.3.0)
+      eventmachine (>= 1.0.0.beta.4)
+    em-twitter (0.2.2)
+      eventmachine (~> 1.0)
+      http_parser.rb (~> 0.5)
+      simple_oauth (~> 0.1)
+    engtagger (0.1.2)
+    eventmachine (1.0.3)
+    faraday (0.8.8)
+      multipart-post (~> 1.2.0)
+    fast-stemmer (1.0.2)
+    gingerice (1.2.1)
+      addressable
+      awesome_print
+    highscore (1.1.0)
+      whatlanguage (>= 1.0.0)
+    htmlentities (4.3.1)
+    http_parser.rb (0.5.3)
+    minitest (5.0.8)
+    multi_json (1.8.2)
+    multipart-post (1.2.0)
+    rufus-scheduler (3.0.2)
+      tzinfo
+    simple_oauth (0.2.0)
+    thread_safe (0.1.3)
+      atomic
+    tweetstream (2.5.0)
+      daemons (~> 1.1)
+      em-http-request (~> 1.0.2)
+      em-twitter (~> 0.2)
+      twitter (~> 4.5)
+      yajl-ruby (~> 1.1)
+    twitter (4.8.1)
+      faraday (~> 0.8, < 0.10)
+      multi_json (~> 1.0)
+      simple_oauth (~> 0.2)
+    twitter_ebooks (2.1.2)
+      engtagger
+      fast-stemmer
+      gingerice
+      highscore
+      htmlentities
+      minitest
+      rufus-scheduler
+      tweetstream (= 2.5)
+      twitter (~> 4.5)
+    tzinfo (1.1.0)
+      thread_safe (~> 0.1)
+    whatlanguage (1.0.5)
+    yajl-ruby (1.1.0)
+
+PLATFORMS
+  ruby
+
+DEPENDENCIES
+  twitter_ebooks

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: twitter_ebooks
 version: !ruby/object:Gem::Version
-  version: 2.1.3
+  version: 2.1.4
   prerelease: 
 platform: ruby
 authors:
@@ -9,7 +9,7 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2013-11-24 00:00:00.000000000 Z
+date: 2013-11-27 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: minitest
@@ -183,8 +183,12 @@ files:
 - lib/twitter_ebooks/version.rb
 - script/process_anc_data.rb
 - skeleton/.gitignore
+- skeleton/Gemfile
+- skeleton/Gemfile.lock
 - skeleton/Procfile
 - skeleton/bots.rb
+- skeleton/corpus/.gitignore
+- skeleton/model/.gitignore
 - skeleton/run.rb
 - test/corpus/0xabad1dea.tweets
 - test/keywords.rb