so2db 0.1.0

data/Gemfile

@@ -0,0 +1,8 @@
+source 'http://rubygems.org'
+
+gem 'nokogiri'
+gem 'activerecord'
+gem 'foreigner'
+gem 'pg'
+gem 'rake'
+gem 'mocha'

data/Gemfile.lock

@@ -0,0 +1,37 @@
+GEM
+  remote: http://rubygems.org/
+  specs:
+    activemodel (3.0.0)
+      activesupport (= 3.0.0)
+      builder (~> 2.1.2)
+      i18n (~> 0.4.1)
+    activerecord (3.0.0)
+      activemodel (= 3.0.0)
+      activesupport (= 3.0.0)
+      arel (~> 1.0.0)
+      tzinfo (~> 0.3.23)
+    activesupport (3.0.0)
+    arel (1.0.1)
+      activesupport (~> 3.0.0)
+    builder (2.1.2)
+    foreigner (1.2.0)
+      activerecord (>= 3.0.0)
+    i18n (0.4.2)
+    metaclass (0.0.1)
+    mocha (0.12.2)
+      metaclass (~> 0.0.1)
+    nokogiri (1.5.5)
+    pg (0.14.0)
+    rake (0.9.2.2)
+    tzinfo (0.3.33)
+
+PLATFORMS
+  ruby
+
+DEPENDENCIES
+  activerecord
+  foreigner
+  mocha
+  nokogiri
+  pg
+  rake

data/MIT-LICENSE

@@ -0,0 +1,8 @@
+Copyright (c) 2012 Chad Taylor
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+

data/README.md

@@ -0,0 +1,101 @@
+# SO2DB
+SO2DB provides an API for importing the Stack Overflow/Stack Exchange data dumps into a database. It also provides a PostgreSQL import utility (so2pg) out of the box.
+
+
+# Using the PostgreSQL Import Utility
+1.  Download a [Stack Exchange Data Dump](http://www.clearbits.net/creators/146-stack-exchange-data-dump) and extract the data set that you want to import (e.g., math.stackexchange.com.7z).
+
+2.  Strip invalid XML-encoded strings from the extracted XML files.  This is a [known issue](http://blog.stackoverflow.com/2009/06/stack-overflow-creative-commons-data-dump/#comment-24223).  I have a [Gist](https://gist.github.com/3270224) you can use that backs up and cleans all the XML files in a directory.
+
+3.  Ensure that you have the PostgreSQL client (`psql`) installed.  `psql` is used to bulk copy data into your database.
+
+4.  Create a PostgreSQL database for the data.
+
+5.  Install SO2DB with the command `gem install so2db`.  You may have to use `sudo` depending on your setup.
+
+6.  Call so2pg to import the data with the command `so2pg -O -R -d db_name -D /path/to/data/dir`.  The data directory is the directory containing the clean XML files.  The -O and -R flags indicate that you wish to include optional tables and table relationships, respectively. You can see all the options by running `so2pg --help` or in the list below. 
+
+7.  Wait impatiently!  Depending on your hardware and the data set you choose to import, `so2pg` may take a while to run.  With all that free time on your hands, why don't you help me figure out how to make so2db *faster?!*
+
+
+## so2pg options
+
+        -H --host HOST               The database host
+
+        -d --database DBNAME         The name of the database (REQUIRED)
+
+        -D --directory DIRECTORY     The data directory path (REQUIRED)
+
+        -u --user USER               The user name
+ 
+        -p --password PASSWORD       The user's password
+
+        -P --port PORT_NUMBER        The port number
+ 
+        -O --include-optionals       Includes optional tables
+
+        -R --include-relationships   Includes table relationships
+      
+        -h --help                    Show this help screen
+        
+## Learning PostgreSQL?
+Be sure to check out the Tekpub [Hello PostgreSQL](http://tekpub.com/productions/pg) series.  It presents a lot of useful information quickly and in an accessible manner.  I built this gem simply so I could practice along with the videos.  Highly recommended! 
+
+## Hardware and Performance
+We have run so2pg several times on two machines with slightly hardware configurations.  The CPUs and RAM were comparable (2.66 vs. 2.4 GHz, 4GB RAM), but one had an SSD while the other had a ...slow... HD.  When importing the Apr 2012 Stack Overflow dump on each machine, we noticed about 43% faster import times on the SSD (2 hrs vs. 3.5 hrs).
+
+Your import may take quite some time to complete, and the performance is very dependent on your hard drive speed.
+
+## Other Tips
+If you are running `so2pg` on OS X, you may want to run the `purge` command in a terminal periodically.  This will free up "Inactive Memory" and reduce the number of pages to disk.  This should help performance, though I haven't specifically made measurements on it. 
+
+
+# Creating a Custom Importer
+Before you create your own custom importer, you should check to see if someone is already working on one with the same purpose.  Otherwise, let us know what you're working on and get started!
+
+SO2DB depends on ActiveRecord and [Foreigner](https://github.com/matthuhiggins/foreigner) to build tables and relationships, and so SO2DB is limited to the databases supported by both projects.  (At the time of writing, Foreigner only supports PostgreSQL, MySQL, and SQLite.  ActiveRecord supports these and more.)  
+
+Creating a custom importer requires you to provide two classes: an ActiveRecord monkey patch and the importer implementation.
+
+First, you need to create a [monkey patch](http://stackoverflow.com/questions/394144/what-does-monkey-patching-exactly-mean-in-ruby) that adds a uuid method to the associated ActiveRecord connection adapter.  The uuid method defines the database type associated with a universally unique identifier (e.g., 'uuid' in PostgreSQL, 'CHAR(16)' in MySQL).
+
+Next, you must create a subclass of SO2DB::Importer that contains a method with the following definition:
+
+> import_stream(formatter)
+
+The formatter will be a SO2DB::Formatter, which generates formatted output from a StackOverflow XML data file when you call the its `format` method.  The `format` method accepts a stream and pumps formatted data to that stream.  This approach allows for you to deal with the data as you see fit - pass it over STDIN to another command (this is what so2pg does) or send it to a file that you then pass to a database system.  If you need the database parameters provided to SO2DB, they are provided in the importer conn_opts property.
+
+The data coming from the formatter will use a delimiter specified in the Importer implementation.  A vertical tab, '\v' (0xB), is used by default.  This can be changed by setting the format_delimiter property in the importer.
+
+The formatter also offers a couple of convenience routines, `file_name` and `value_str`. `file_name` simply provides the name of the file to be formatted.  `value_str` provides a partial SQL string based on the data to be formatted.  It provides the table name and the ordered value names associated with the formatted data.  For example, assume you are provided a formatter for badges.xml.  The convenience routines produce the following:
+
+```ruby
+puts formatter.file_name
+# => badges.xml
+
+puts formatter.value_str
+# => badges(id,date,name,user_id)
+```
+
+Note that the field names are alphabetized.  The field values from the formatter will have the same ordering.  This is to ensure consistent and predictable field ordering, even if fields are not provided in the XML file.  If a field is not provided in the XML file, an empty string is inserted into the formatted data.
+
+I encourage you to check out the implementation of so2pg if you are interested in developing your own importer; there are additional notes throughout the source to make your development experience a bit smoother.
+
+When you are finished, consider creating a pull request.  I would love to include your labor of love in a future release!
+
+# License
+SO2DB is released under the [MIT License](http://opensource.org/licenses/MIT).
+
+
+# Tested Platforms
+This project has been tested under OS X (Snow Leopard) and Ubuntu 12.04 using Ruby version 1.9.2 and 1.9.3.  If anyone wants to test on other platforms, I would appreciate it!
+
+
+# Supported Data Dumps
+SO2DB has been tested against the following data dumps:
+
+    Sept 2011
+    Apr  2012
+
+# Special Thanks
+Thanks to [@nclaburn](https://github.com/nclaburn) for being a great mentor and helping me get this project released!

data/Rakefile

@@ -0,0 +1,8 @@
+require 'rake/testtask'
+
+Rake::TestTask.new do |t|
+  t.libs << 'test'
+end
+
+desc "Run tests"
+task :default => :test

data/bin/so2pg

@@ -0,0 +1,5 @@
+#!/usr/bin/env ruby
+
+require 'so2pg'
+
+PgImporter.import_from_argv(ARGV)

data/lib/so2db/extensions.rb

@@ -0,0 +1,42 @@
+#--
+# Copyright (c) 2012 Chad Taylor
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy 
+# of this software and associated documentation files (the "Software"), to deal 
+# in the Software without restriction, including without limitation the rights 
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 
+# copies of the Software, and to permit persons to whom the Software is 
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in 
+# all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 
+# SOFTWARE.
+
+
+require 'active_record'
+
+module ActiveRecord
+  module ConnectionAdapters
+
+    class TableDefinition
+      def uuid(*args)
+        opts = args.extract_options!
+        column_names = args
+        type, default_opts  = @base.uuid
+
+        # prefer the values provided by the user...
+        options = default_opts.merge!(opts)
+
+        column_names.each { |name| column(name, type, options) }
+      end
+    end
+
+  end
+end

data/lib/so2db/formatter.rb

@@ -0,0 +1,113 @@
+#--
+# Copyright (c) 2012 Chad Taylor
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy 
+# of this software and associated documentation files (the "Software"), to deal 
+# in the Software without restriction, including without limitation the rights 
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 
+# copies of the Software, and to permit persons to whom the Software is 
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in 
+# all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 
+# SOFTWARE.
+
+
+require 'active_support/inflector'
+require 'nokogiri'
+require 'cgi'
+
+module SO2DB
+
+  # Formats data from one stream into another stream.
+  class Formatter
+
+    # Infrastructure.  Do not call this from your code.
+    def initialize(path = '', delimiter = 11.chr.to_s)
+      @delimiter = delimiter
+      @path = path
+      @name = "row"
+    end
+
+    # Formats a file and prints the formatted output to the outstream.
+    #
+    # The output is performed via a 'puts' method.
+    #
+    # Example:
+    #   >> f = get_formatter  # assumed to be provided to you
+    #   >> cmd = get_cmd      # some shell command that accepts piped input
+    #   >> IO.popen(cmd, 'r+') do |s| 
+    #   >>   formatter.format(s)
+    #   >>   s.close_write
+    #   >> end
+    def format(outstream)
+      file = File.basename(@path, '.*')
+      req_attrs = Models::Lookup::get_required_attrs(file)
+
+      format_from_stream(File.open(@path), req_attrs, outstream)
+    end
+
+    def file_name
+      File.basename(@path)
+    end
+
+    # Returns a string containing the data type and individual fields provided
+    # through formatting. The fields are sorted alphabetically.
+    #
+    # This method is useful for building SQL statements for the formatted data.
+    #
+    # Example:
+    #   >> f = get_formatter_for_badges # formatter should be provided to you
+    #   >> puts f.value_str
+    #   => badges(id,date,name,user_id)
+    def value_str
+      file = File.basename(@path, '.*')
+      o = Models::Lookup::find_class(file)
+
+      table = o.table_name
+      values = o.exported_fields.sort.join(",")
+
+      "#{table}(#{values})"
+    end
+
+    private
+    def format_from_stream(instream, required_attrs, outstream)
+      reader = Nokogiri::XML::Reader(instream)
+      required_attrs.sort!
+
+      reader.each do |node|
+        outstream.puts format_node(node, required_attrs) if element_start? node
+      end
+    end
+
+    def element_start?(node)
+      node.name == @name &&
+        node.node_type == Nokogiri::XML::Reader::TYPE_ELEMENT
+    end
+
+    def format_node(node, attrs)
+      arr = attrs.map do |a|
+        str = node.attribute(a)
+        str ? scrub(str) : ''
+      end
+
+      arr.join(@delimiter)
+    end
+
+    def scrub(str)
+      s = CGI::escapeHTML(str)
+      s.gsub!(/\n|\r/, '')
+
+      s
+    end
+
+  end
+end
+

data/lib/so2db/migrations.rb

@@ -0,0 +1,308 @@
+#--
+# Copyright (c) 2012 Chad Taylor
+#
+# Permission is hereby granted, free of charge, to any person obtaining a copy 
+# of this software and associated documentation files (the "Software"), to deal 
+# in the Software without restriction, including without limitation the rights 
+# to use, copy, modify, merge, publish, distribute, sublicense, and/or sell 
+# copies of the Software, and to permit persons to whom the Software is 
+# furnished to do so, subject to the following conditions:
+#
+# The above copyright notice and this permission notice shall be included in 
+# all copies or substantial portions of the Software.
+#
+# THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR 
+# IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, 
+# FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE 
+# AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER 
+# LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, 
+# OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE 
+# SOFTWARE.
+
+
+require 'active_record'
+require 'so2db/extensions'
+require 'foreigner'
+
+
+module SO2DB
+
+  module FKHelper
+
+    def add_fk(from_table, to_table, options={})
+      begin
+      AddForeignKeyMigration.new(from_table, to_table, options).up
+      rescue
+        s = "Error creating foreign key from #{from_table} to #{to_table}"
+        s << " on column #{options[:column]}" if options.has_key? :column
+        puts s
+      end
+    end
+
+  end
+
+  class AddForeignKeyMigration < ActiveRecord::Migration
+
+    def initialize(from_table, to_table, options)
+      @from_table = from_table
+      @to_table = to_table
+      @options = options
+    end
+
+    def up
+      add_foreign_key(@from_table, @to_table, @options)
+    end
+
+  end
+
+  class CreateBasicTables < ActiveRecord::Migration
+
+    def up
+      create_table :badges do |t|
+        t.integer :user_id
+        t.string :name, :limit => 50
+        t.timestamp :date
+      end
+
+      create_table :comments do |t|
+        t.integer :post_id
+        t.integer :score
+        t.text :text
+        t.timestamp :creation_date
+        t.string :user_display_name, :limit => 30
+        t.integer :user_id
+      end
+
+      create_table :posts do |t|
+        t.integer :post_type_id
+        t.integer :parent_id
+        t.integer :accepted_answer_id
+        t.timestamp :creation_date
+        t.integer :score
+        t.integer :view_count
+        t.text :body
+        t.integer :owner_user_id
+        t.string :owner_display_name, :limit => 40
+        t.integer :last_editor_user_id
+        t.string :last_editor_display_name, :limit => 40
+        t.timestamp :last_edit_date
+        t.timestamp :last_activity_date
+        t.timestamp :community_owned_date
+        t.timestamp :closed_date
+        t.string :title, :limit => 250
+        t.string :tags, :limit => 150
+        t.integer :answer_count
+        t.integer :comment_count
+        t.integer :favorite_count
+      end
+
+      create_table :post_history do |t|
+        t.integer :post_history_type_id
+        t.integer :post_id
+        t.uuid :revision_guid
+        t.timestamp :creation_date
+        t.integer :user_id
+        t.string :user_display_name, :limit => 40
+        t.string :comment, :limit => 600
+        t.text :text
+        t.integer :close_reason_id
+      end
+
+      create_table :users do |t|
+        t.integer :reputation
+        t.timestamp :creation_date
+        t.string :display_name, :limit => 40
+        t.string :email_hash, :limit => 32
+        t.timestamp :last_access_date
+        t.string :website_url, :limit => 300
+        t.string :location, :limit => 200
+        t.integer :age
+        t.text :about_me
+        t.integer :views
+        t.integer :up_votes
+        t.integer :down_votes
+      end
+
+      create_table :votes do |t|
+        t.primary_key :id
+        t.integer :post_id
+        t.integer :vote_type_id
+        t.timestamp :creation_date
+        t.integer :user_id
+        t.integer :bounty_amount
+      end
+
+    end
+
+    def down
+      [:votes, :badges, :comments, :post_history, :posts, :users].each do |t|
+        drop_table t
+      end
+    end
+
+  end
+
+  class CreateRelationships
+    include FKHelper
+
+    def up
+      add_fk(:badges, :users)
+      add_fk(:comments, :posts)
+      add_fk(:comments, :users)
+      add_fk(:posts, :posts, column: 'parent_id')
+
+      # The following relationship is currently suspect, see
+      # http://meta.stackoverflow.com/questions/131975/what-are-the-posttypeids-in-the-2011-12-data-dump
+      add_fk(:posts, :posts, column: 'accepted_answer_id')
+      
+      add_fk(:posts, :users, column: 'owner_user_id')
+      add_fk(:posts, :users, column: 'last_editor_user_id')
+      add_fk(:post_history, :posts)
+      add_fk(:post_history, :users)
+
+      # In my experience, the following is also suspect
+      add_fk(:votes, :posts)
+      
+      add_fk(:votes, :users)
+    end
+
+  end
+
+  class CreateOptionals < ActiveRecord::Migration
+
+    def up
+      create_post_types
+      create_post_history_types
+      create_close_reasons
+      create_vote_types
+    end
+
+    def create_post_types
+      create_table :post_types do |t|
+        t.primary_key :id
+        t.string :type_name, :limit => 24
+      end
+
+      { 1 => "Question", 
+        2 => "Answer",
+        3 => "Wiki",
+        4 => "TagWikiExcerpt",
+        5 => "TagWiki",
+        6 => "ModeratorNomination",
+        7 => "WikiPlaceholder",
+        8 => "PrivilegeWiki"
+      }.each do |k,v|
+        p = Models::PostType.new
+        p.id = k
+        p.type_name = v
+        p.save
+      end
+    end
+
+    def create_post_history_types
+      create_table :post_history_types do |t|
+        t.primary_key :id
+        t.string :name, :limit => 50
+      end
+
+      { 1  => "Initial Title", 
+        2  => "Initial Body", 
+        3  => "Initial Tags", 
+        4  => "Edit Title", 
+        5  => "Edit Body", 
+        6  => "Edit Tags", 
+        7  => "Rollback Title", 
+        8  => "Rollback Body", 
+        9  => "Rollback Tags", 
+        10 => "Post Closed", 
+        11 => "Post Reopened", 
+        12 => "Post Deleted", 
+        13 => "Post Undeleted", 
+        14 => "Post Locked", 
+        15 => "Post Unlocked", 
+        16 => "Community Owned", 
+        17 => "Post Migrated", 
+        18 => "Question Merged", 
+        19 => "Question Protected", 
+        20 => "Question Unprotected", 
+        22 => "Question Unmerged", 
+        24 => "Suggested Edit Applied", 
+        25 => "Post Tweeted", 
+        31 => "Discussion moved to chat", 
+        33 => "Post Notice Added", 
+        34 => "Post Notice Removed", 
+        35 => "Post Migrated Away", 
+        36 => "Post Migrated Here", 
+        37 => "Post Merge Source", 
+        38 => "Post Merge Destination" 
+      }.each do |k,v|
+        p = Models::PostHistoryType.new
+        p.id = k
+        p.name = v
+        p.save
+      end
+    end
+
+    def create_close_reasons
+      create_table :close_reasons do |t|
+        t.primary_key :id
+        t.string :name, :limit => 50
+      end
+
+      { 1 => "Exact duplicate",
+        2 => "off-topic",
+        3 => "subjective",
+        4 => "not a real question",
+        7 => "too localized",
+        10 => "General reference",
+        20 => "Noise or pointless"
+      }.each do |k,v|
+        c = Models::CloseReason.new
+        c.id = k
+        c.name = v
+        c.save
+      end
+    end
+
+    def create_vote_types
+      create_table :vote_types do |t|
+        t.primary_key :id
+        t.string :name, :limit => 50
+      end
+
+      { 1 => "AcceptedByOriginator",
+        2 =>"UpMod",
+        3 => "DownMod",
+        4 =>"Offensive",
+        5 =>"Favorite",
+        6 =>"Close",
+        7 =>"Reopen",
+        8 =>"BountyStart",
+        9 =>"BountyClose",
+        10 =>"Deletion",
+        11 =>"Undeletion",
+        12 =>"Spam",
+        15 =>"ModeratorReview",
+        16 =>"ApproveEditSuggestion"
+      }.each do |k,v|
+        vt = Models::VoteType.new
+        vt.id = k
+        vt.name = v
+        vt.save
+      end
+    end
+
+  end
+
+  class CreateOptionalRelationships < ActiveRecord::Migration
+    include FKHelper
+
+    def up
+      add_fk(:posts, :post_types)
+      add_fk(:post_history, :post_history_types)
+      add_fk(:post_history, :close_reasons)
+      add_fk(:votes, :vote_types)
+    end
+  end
+
+end