mosql 0.1.0

data/.gitignore

@@ -0,0 +1,2 @@
+collections.yml
+/.bundle/

data/Gemfile

@@ -0,0 +1,4 @@
+source 'https://rubygems.org'
+
+gemspec
+

data/Gemfile.lock

@@ -0,0 +1,48 @@
+GIT
+  remote: git@github.com:stripe-internal/mongoriver
+  revision: d5b5ca1471f9efe7c91b3abe2c26f612a2dd4e9c
+  ref: d5b5ca1471f9efe7c91b3abe2c26f612a2dd4e9c
+  specs:
+    mongoriver (0.0.1)
+      bson_ext
+      log4r
+      mongo (>= 1.7)
+
+PATH
+  remote: .
+  specs:
+    mosql (0.0.1)
+      bson_ext
+      json
+      log4r
+      mongo
+      pg
+      rake
+      sequel
+
+GEM
+  remote: https://intgems.stripe.com:446/
+  specs:
+    bson (1.7.1)
+    bson_ext (1.7.1)
+      bson (~> 1.7.1)
+    json (1.7.5)
+    log4r (1.1.10)
+    metaclass (0.0.1)
+    minitest (3.0.0)
+    mocha (0.10.5)
+      metaclass (~> 0.0.1)
+    mongo (1.7.1)
+      bson (~> 1.7.1)
+    pg (0.14.1)
+    rake (10.0.2)
+    sequel (3.41.0)
+
+PLATFORMS
+  ruby
+
+DEPENDENCIES
+  minitest
+  mocha
+  mongoriver!
+  mosql!

data/README.md

@@ -0,0 +1,168 @@
+# MoSQL: a MongoDB → SQL streaming translator
+
+At Stripe, we love MongoDB. We love the flexibility it gives us in
+changing data schemas as we grow and learn, and we love its
+operational properties. We love replsets. We love the uniform query
+language that doesn't require generating and parsing strings, tracking
+placeholder parameters, or any of that nonsense.
+
+The thing is, we also love SQL. We love the ease of doing ad-hoc data
+analysis over small-to-mid-size datasets in SQL. We love doing JOINs
+to pull together reports summarizing properties across multiple
+datasets. We love the fact that virtually every employee we hire
+already knows SQL and is comfortable using it to ask and answer
+questions about data.
+
+So, we thought, why can't we have the best of both worlds? Thus:
+MoSQL.
+
+# MoSQL: Put Mo' SQL in your NoSQL
+
+![MoSQL](https://stripe.com/img/blog/posts/mosql/mosql.png)
+
+MoSQL imports the contents of your MongoDB database cluster into a
+PostgreSQL instance, using an oplog tailer to keep the SQL mirror live
+up-to-date. This lets you run production services against a MongoDB
+database, and then run offline analytics or reporting using the full
+power of SQL.
+
+## Installation
+
+Install from Rubygems as:
+
+    $ gem install mosql
+
+Or build from source by:
+
+    $ gem build mosql.gemspec
+
+And then install the built gem.
+
+## The Collection Map file
+
+In order to define a SQL schema and import your data, MoSQL needs a
+collection map file describing the schema of your MongoDB data. (Don't
+worry -- MoSQL can handle it if your mongo data doesn't always exactly
+fit the stated schema. More on that later).
+
+The collection map is a YAML file describing the databases and
+collections in Mongo that you want to import, in terms of their SQL
+types. An example collection map might be:
+
+
+    mongodb:
+      blog_posts:
+        :columns:
+        - _id: TEXT
+        - author: TEXT
+        - title: TEXT
+        - created: DOUBLE PRECISION
+        :meta:
+          :table: blog_posts
+          :extra_props: true
+
+Said another way, the collection map is a YAML file containing a hash
+mapping
+
+    <Mongo DB name> -> { <Mongo Collection Name> -> <Collection Definition> }
+
+Where a `<Collection Definition>` is a hash with `:columns` and
+`:meta` fields. `:columns` is a list of one-element hashes, mapping
+field-name to SQL type. It is required to include at least an `_id`
+mapping. `:meta` contains metadata about this collection/table. It is
+required to include at least `:table`, naming the SQL table this
+collection will be mapped to. `extra_props` determines the handling of
+unknown fields in MongoDB objects -- more about that later.
+
+By default, `mosql` looks for a collection map in a file named
+`collections.yml` in your current working directory, but you can
+specify a different one with `-c` or `--collections`.
+
+## Usage
+
+Once you have a collection map. MoSQL usage is easy. The basic form
+is:
+
+    mosql [-c collections.yml] [--sql postgres://sql-server/sql-db] [--mongo mongodb://mongo-uri]
+
+By default, `mosql` connects to both PostgreSQL and MongoDB instances
+running on default ports on localhost without authentication. You can
+point it at different targets using the `--sql` and `--mongo`
+command-line parameters.
+
+`mosql` will:
+
+ 1. Create the appropriate SQL tables
+ 2. Import data from the Mongo database
+ 3. Start tailing the mongo oplog, propogating changes from MongoDB to SQL.
+
+
+After the first run, `mosql` will store the status of the optailer in
+the `mongo_sql` table in your SQL database, and automatically resume
+where it left off. `mosql` uses the replset name to keep track of
+which mongo database it's tailing, so that you can tail multiple
+databases into the same SQL database. If you want to tail the same
+replSet, or multiple replSets with the same name, for some reason, you
+can use the `--service` flag to change the name `mosql` uses to track
+state.
+
+You likely want to run `mosql` against a secondary node, at least for
+the initial import, which will cause large amounts of disk activity on
+the target node. One option is to use read preferences in your
+connection URI:
+
+    mosql --mongo mongodb://node1,node2,node3?readPreference=secondary
+
+## Advanced usage
+
+For advanced scenarios, you can pass options to control mosql's
+behavior. If you pass `--skip-tail`, mosql will do the initial import,
+but not tail the oplog. This could be used, for example, to do an
+import off of a backup snapshot, and then start the tailer on the live
+cluster.
+
+If you need to force a fresh reimport, run `--reimport`, which will
+cause `mosql` to drop tables, create them anew, and do another import.
+
+## Schema mismatches and _extra_props
+
+If MoSQL encounters values in the MongoDB database that don't fit
+within the stated schema (e.g. a floating-point value in a INTEGER
+field), it will log a warning, ignore the entire object, and continue.
+
+If it encounters a MongoDB object with fields not listed in the
+collection map, it will discard the extra fields, unless
+`:extra_props` is set in the `:meta` hash. If it is, it will collect
+any missing fields, JSON-encode them in a hash, and store the
+resulting text in `_extra_props` in SQL. It's up to you to do
+something useful with the JSON. One option is to use [plv8][plv8] to
+parse them inside PostgreSQL, or you can just pull the JSON out whole
+and parse it in application code.
+
+This is also currently the only way to handle array or object values
+inside records -- specify `:extra_props`, and they'll get JSON-encoded
+into `_extra_props`. There's no reason we couldn't support
+JSON-encoded values for individual columns/fields, but we haven't
+written that code yet.
+
+[plv8]: http://code.google.com/p/plv8js/
+
+## Sharded clusters
+
+MoSQL does not have special support for sharded Mongo clusters at this
+time. It should be possible to run a separate MoSQL instance against
+each of the individual backend shard replica sets, streaming into
+separate PostgreSQL instances, but we have not actually tested this
+yet.
+
+# Development
+
+Patches and contributions are welcome! Please fork the project and
+open a pull request on [github][github], or just report issues.
+
+MoSQL includes a small but hopefully-growing test suite. It assumes a
+running PostgreSQL and MongoDB instance on the local host; You can
+point it at a different target via environment variables; See
+`test/functional/_lib.rb` for more information.
+
+[github]: https://github.com/stripe/mosql

data/Rakefile

@@ -0,0 +1,12 @@
+require 'rake/testtask'
+
+task :default
+task :build
+
+Rake::TestTask.new do |t|
+  t.libs = ["lib"]
+  t.verbose = true
+  t.test_files = FileList['test/**/*.rb'].reject do |file|
+    file.end_with?('_lib.rb')
+  end
+end

data/bin/mosql

@@ -0,0 +1,7 @@
+#!/usr/bin/env ruby
+
+require 'rubygems'
+require 'bundler/setup'
+require 'mosql/cli'
+
+MoSQL::CLI.run(ARGV)

data/lib/mosql.rb

@@ -0,0 +1,11 @@
+require 'log4r'
+require 'mongo'
+require 'sequel'
+require 'mongoriver'
+require 'json'
+
+require 'mosql/version'
+require 'mosql/log'
+require 'mosql/sql'
+require 'mosql/schema'
+require 'mosql/tailer'

data/lib/mosql/cli.rb

@@ -0,0 +1,305 @@
+require 'mosql'
+require 'optparse'
+require 'yaml'
+require 'logger'
+
+module MoSQL
+  class CLI
+    include MoSQL::Logging
+
+    BATCH       = 1000
+
+    attr_reader :args, :options, :tailer
+
+    def self.run(args)
+      cli = CLI.new(args)
+      cli.run
+    end
+
+    def initialize(args)
+      @args    = args
+      @options = []
+      @done    = false
+      setup_signal_handlers
+    end
+
+    def setup_signal_handlers
+      %w[TERM INT USR2].each do |sig|
+        Signal.trap(sig) do
+          log.info("Got SIG#{sig}. Preparing to exit...")
+          @done = true
+        end
+      end
+    end
+
+    def parse_args
+      @options = {
+        :collections => 'collections.yml',
+        :sql    => 'postgres:///',
+        :mongo  => 'mongodb://localhost',
+        :verbose => 0
+      }
+      optparse = OptionParser.new do |opts|
+        opts.banner = "Usage: #{$0} [options] "
+
+        opts.on('-h', '--help', "Display this message") do
+          puts opts
+          exit(0)
+        end
+
+        opts.on('-v', "Increase verbosity") do
+          @options[:verbose] += 1
+        end
+
+        opts.on("-c", "--collections [collections.yml]", "Collection map YAML file") do |file|
+          @options[:collections] = file
+        end
+
+        opts.on("--sql [sqluri]", "SQL server to connect to") do |uri|
+          @options[:sql] = uri
+        end
+
+        opts.on("--mongo [mongouri]", "Mongo connection string") do |uri|
+          @options[:mongo] = uri
+        end
+
+        opts.on("--schema [schema]", "PostgreSQL 'schema' to namespace tables") do |schema|
+          @options[:schema] = schema
+        end
+
+        opts.on("--ignore-delete", "Ignore delete operations when tailing") do
+          @options[:ignore_delete] = true
+        end
+
+        opts.on("--tail-from [timestamp]", "Start tailing from the specified UNIX timestamp") do |ts|
+          @options[:tail_from] = ts
+        end
+
+        opts.on("--service [service]", "Service name to use when storing tailing state") do |service|
+          @options[:service] = service
+        end
+
+        opts.on("--skip-tail", "Don't tail the oplog, just do the initial import") do
+          @options[:skip_tail] = true
+        end
+
+        opts.on("--reimport", "Force a data re-import") do
+          @options[:reimport] = true
+        end
+      end
+
+      optparse.parse!(@args)
+
+      log = Log4r::Logger.new('Stripe')
+      log.outputters = Log4r::StdoutOutputter.new(STDERR)
+      if options[:verbose] >= 1
+        log.level = Log4r::DEBUG
+      else
+        log.level = Log4r::INFO
+      end
+    end
+
+    def connect_mongo
+      @mongo = Mongo::Connection.from_uri(options[:mongo])
+      config = @mongo['admin'].command(:ismaster => 1)
+      if !config['setName']
+        log.warn("`#{options[:mongo]}' is not a replset. Proceeding anyways...")
+      end
+      options[:service] ||= config['setName']
+    end
+
+    def connect_sql
+      @sql = MoSQL::SQLAdapter.new(@schemamap, options[:sql], options[:schema])
+      if options[:verbose] >= 2
+        @sql.db.sql_log_level = :debug
+        @sql.db.loggers << Logger.new($stderr)
+      end
+    end
+
+    def load_collections
+      collections = YAML.load(File.read(@options[:collections]))
+      @schemamap = MoSQL::Schema.new(collections)
+    end
+
+    def run
+      parse_args
+      load_collections
+      connect_sql
+      connect_mongo
+
+      metadata_table = MoSQL::Tailer.create_table(@sql.db, 'mosql_tailers')
+
+      @tailer = MoSQL::Tailer.new([@mongo], :existing, metadata_table,
+                                  :service => options[:service])
+
+      if options[:reimport] || tailer.read_timestamp.seconds == 0
+        initial_import
+      end
+
+      optail
+    end
+
+    # Helpers
+
+    def collection_for_ns(ns)
+      dbname, collection = ns.split(".", 2)
+      @mongo.db(dbname).collection(collection)
+    end
+
+    def bulk_upsert(table, ns, items)
+      begin
+        @schemamap.copy_data(table.db, ns, items)
+      rescue Sequel::DatabaseError => e
+        log.debug("Bulk insert error (#{e}), attempting invidual upserts...")
+        cols = @schemamap.all_columns(@schemamap.find_ns(ns))
+        items.each do |it|
+          h = {}
+          cols.zip(it).each { |k,v| h[k] = v }
+          @sql.upsert(table, h)
+        end
+      end
+    end
+
+    def with_retries(tries=10)
+      tries.times do |try|
+        begin
+          yield
+        rescue Mongo::ConnectionError, Mongo::ConnectionFailure, Mongo::OperationFailure => e
+          # Duplicate key error
+          raise if e.kind_of?(Mongo::OperationFailure) && [11000, 11001].include?(e.error_code)
+          # Cursor timeout
+          raise if e.kind_of?(Mongo::OperationFailure) && e.message =~ /^Query response returned CURSOR_NOT_FOUND/
+          delay = 0.5 * (1.5 ** try)
+          log.warn("Mongo exception: #{e}, sleeping #{delay}s...")
+          sleep(delay)
+        end
+      end
+    end
+
+    def track_time
+      start = Time.now
+      yield
+      Time.now - start
+    end
+
+    def initial_import
+      @schemamap.create_schema(@sql.db, true)
+
+      start_ts = @mongo['local']['oplog.rs'].find_one({}, {:sort => [['$natural', -1]]})['ts']
+
+      want_dbs = @schemamap.all_mongo_dbs & @mongo.database_names
+      want_dbs.each do |dbname|
+        log.info("Importing for Mongo DB #{dbname}...")
+        db = @mongo.db(dbname)
+        want = Set.new(@schemamap.collections_for_mongo_db(dbname))
+        db.collections.select { |c| want.include?(c.name) }.each do |collection|
+          ns = "#{dbname}.#{collection.name}"
+          import_collection(ns, collection)
+          exit(0) if @done
+        end
+      end
+
+      tailer.write_timestamp(start_ts)
+    end
+
+    def import_collection(ns, collection)
+      log.info("Importing for #{ns}...")
+      count = 0
+      batch = []
+      table = @sql.table_for_ns(ns)
+      table.truncate
+
+      start    = Time.now
+      sql_time = 0
+      collection.find(nil, :batch_size => BATCH) do |cursor|
+        with_retries do
+          cursor.each do |obj|
+            batch << @schemamap.transform(ns, obj)
+            count += 1
+
+            if batch.length >= BATCH
+              sql_time += track_time do
+                bulk_upsert(table, ns, batch)
+              end
+              elapsed = Time.now - start
+              log.info("Imported #{count} rows (#{elapsed}s, #{sql_time}s SQL)...")
+              batch.clear
+              exit(0) if @done
+            end
+          end
+        end
+      end
+
+      unless batch.empty?
+        bulk_upsert(table, ns, batch)
+      end
+    end
+
+    def optail
+      return if options[:skip_tail]
+
+      tailer.tail_from(options[:tail_from] ?
+                       BSON::Timestamp.new(options[:tail_from].to_i, 0) :
+                       nil)
+      until @done
+        tailer.stream(1000) do |op|
+          handle_op(op)
+        end
+      end
+    end
+
+    def sync_object(ns, _id)
+      obj = collection_for_ns(ns).find_one({:_id => _id})
+      if obj
+        @sql.upsert_ns(ns, obj)
+      else
+        @sql.table_for_ns(ns).where(:_id => _id).delete()
+      end
+    end
+
+    def handle_op(op)
+      log.debug("processing op: #{op.inspect}")
+      unless op['ns'] && op['op']
+        log.warn("Weird op: #{op.inspect}")
+        return
+      end
+
+      unless @schemamap.find_ns(op['ns'])
+        log.debug("Skipping op for unknown ns #{op['ns']}...")
+        return
+      end
+
+      ns = op['ns']
+      dbname, collection_name = ns.split(".", 2)
+
+      case op['op']
+      when 'n'
+        log.debug("Skipping no-op #{op.inspect}")
+      when 'i'
+        if collection_name == 'system.indexes'
+          log.info("Skipping index update: #{op.inspect}")
+        else
+          @sql.upsert_ns(ns, op['o'])
+        end
+      when 'u'
+        selector = op['o2']
+        update   = op['o']
+        if update.keys.any? { |k| k.start_with? '$' }
+          log.debug("resync #{ns}: #{selector['_id']} (update was: #{update.inspect})")
+          sync_object(ns, selector['_id'])
+        else
+          log.debug("upsert #{ns}: _id=#{update['_id']}")
+          @sql.upsert_ns(ns, update)
+        end
+      when 'd'
+        if options[:ignore_delete]
+          log.debug("Ignoring delete op on #{ns} as instructed.")
+        else
+          @sql.table_for_ns(ns).where(:_id => op['o']['_id']).delete
+        end
+      else
+        log.info("Skipping unknown op #{op.inspect}")
+      end
+    end
+  end
+end