mongo-proxy 1.0.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 82257f694a4c0d087fd02010aa37c2a160ef2cff
+  data.tar.gz: c7fd83f1ba64daa944a0a55cbe462f5f30c8683f
+SHA512:
+  metadata.gz: ac74381e8f4d526f3a24441fb6a21c7a04eedd4473be89edf2f0aedcfb6e0f1accb5a0c5f1218f7ea39fda50b5eb3263067d9478e3d7bd03144e14d40fd08d7a
+  data.tar.gz: 6afb35ffac434d0ab07dc444fc0930cec5a2e80814764ce618e1d59988ae4365688d6d20db1f3e4feb40232f211f5aa6966fbb12463d52ef5c6f7184b8ec0941

data/.rspec

@@ -0,0 +1,3 @@
+--colour
+--backtrace
+

data/Gemfile

@@ -0,0 +1,4 @@
+source 'https://rubygems.org'
+gemspec
+
+

data/LICENSE.md

@@ -0,0 +1,9 @@
+Copyright (C) 2014 Peter Bakkum
+
+(MIT License)
+
+Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,151 @@
+Mongo Proxy
+===========
+
+A gem for proxying MongoDB at the wire-protocol level. The proxy intercepts MongoDB communication, presents it in an easy-to-manipulate format, and forwards only desired messages. This approach allows you to filter MongoDB traffic, such as writes, or even inject new messages, such as a message-of-the-day that appears on new connections.
+
+mongo-proxy includes a command-line interface and an API for running it in a Ruby application. You can write your own hooks to manipulate MongoDB wire traffic as it arrives from the client.
+
+Installation
+------------
+
+`gem install mongo-proxy`
+
+More information at the [gem page](http://rubygems.org/gems/mongo-proxy).
+
+Command Line
+------------
+
+You can run mongo-proxy from the command line:
+
+`mongo-proxy [options]`
+
+Here is an example:
+
+`mongo-proxy --client-port 29017 --read-only --motd 'Connected to proxy!'`
+
+This will proxy local client connections to port 29017 to a running MongoDB server at port 27017. Only read queries will be allowed and the client will be told on startup that he is connecting to a proxy server. With the proxy running, a connection looks something like:
+
+```
+> mongo --port 29017
+MongoDB shell version: 2.6.0
+connecting to: 127.0.0.1:27018/test
+Server has startup warnings:
+Connected to proxy!
+```
+
+#### Command Line Options
+
+The following flags can be used when connecting:
+```
+        --client-host, -h <s>:   Set the host to bind the proxy socket on
+                                 (default: 127.0.0.1)
+        --client-port, -p <i>:   Set the port to bind the proxy socket on
+                                 (default: 27018)
+        --server-host, -H <s>:   Set the backend hostname which we proxy
+                                 (default: 127.0.0.1)
+        --server-port, -P <i>:   Set the backend port which we proxy (default:
+                                 27017)
+              --read-only, -r:   Prevent any traffic that writes to the
+                                 database
+                   --motd, -m:   Set a message-of-the-day to display to clients
+                                 when they connect
+  --verbose, --no-verbose, -v:   Print out MongoDB wire traffic (default: true)
+      --debug, --no-debug, -d:   Print log lines in a more human-readible
+                                 format (default: true)
+                --version, -e:   Print version and exit
+                   --help, -l:   Show this message
+
+```
+
+API
+---
+
+You can also use the mongo-proxy functionality directly in Ruby. This approach allows you to add your own hooks to manipulate MongoDB traffic. Here is an example:
+
+```ruby
+require 'mongo-proxy'
+
+front = 0
+back = 0
+config = {
+  :client_port => 29017,
+  :server_port => 27017,
+  :read_only => true,
+  :debug => true
+}
+
+proxy = MongoProxy.new(config)
+
+m.add_callback_to_front do |conn, msg|
+  front += 1
+  puts "received #{front} client messages so far"
+  msg
+end
+
+m.add_callback_to_back do |conn, msg|
+  back += 1
+  puts "forwarded #{back} client messages so far"
+  msg
+end
+
+m.start
+```
+
+This example opens up a read-only proxy at port 29017. There are two stacks of callbacks: the 'front' callbacks are executed before applying the read-only authorization, while the 'back' callbacks are called after. Thus, the back callbacks will only receive messages that passed authorization. Calling `add_callback_to_front` adds a hook to the front of the front stack, while calling `add_callback_to_back` adds a hook to the back of the back stack. Once you call `start` the proxy will run and show both the messages it receives (because of the debug flag) and the messages from the callback.
+
+Callbacks are executed and passed the client connection on which a message was received and the message itself as a `Hash`. The callback can make changes to this message, if desired. The callback is expected to return a message to pass along through the stack of callbacks and eventually forwarded to the backend MongoDB server. If a callback returns `nil` then the message will dropped.
+
+mongo-proxy is a thin layer on top of [em-proxy](https://github.com/igrigorik/em-proxy), and the connection object passed to your hook is the same as the em-proxy connection object. This means that you can call the `send_data` method on it to send a raw message to the client.
+
+#### MongoProxy Options
+
+You can use the following options (shown with their default values), when creating a `MongoProxy` object.
+
+```ruby
+@config = {
+  :client_host => '127.0.0.1', # Set the host to bind the proxy socket on.
+  :client_port => 29017,       # Set the port to bind the proxy socket on.
+  :server_host => '127.0.0.1', # Set the backend host which we proxy.
+  :server_port => 27017,       # Set the backend port which we proxy.
+  :motd => nil,                # Set a message-of-the-day to display to clients when they connect. nil for none.
+  :read_only => false,         # Prevent any traffic that writes to the database.                                 
+  :verbose => false,           # Print out MongoDB wire traffic.
+  :logger => nil,              # Use this object as the logger instead of creating one.
+  :debug => false              # Print log lines in a more human-readible format.
+} 
+```
+
+#### Message Format
+
+Here is an example message passed to a hook:
+```ruby
+{
+  :flags => 0,
+  :database => 'test',
+  :collection => 'testcoll',
+  :numberToSkip => 0,
+  :numberToReturn => 4294967295,
+  :query => {
+    "foo" => "bar"
+  },
+  :returnFieldSelector => nil,
+  :header => {
+    :messageLength => 58,
+    :requestID => 1,
+    :responseTo => 0,
+    :opCode => :query
+  }
+}
+```
+
+This format comes from our [wire parsing code](lib/mongo-proxy/wire.rb), and will look similar, but differ slightly, for other operations such as inserts or deletes.
+
+Testing
+-------
+
+Running the unit tests requires a MongoDB instance at port 27018 (nonstandard) and nothing at port 27017. The tests use rspec, so you can run with `bundle exec rspec`.
+
+License
+-------
+
+[The MIT License](LICENSE.md) - Copyright 2014 Peter Bakkum

data/bin/mongo-proxy

@@ -0,0 +1,42 @@
+#!/usr/bin/env ruby
+
+require 'mongo-proxy'
+require 'trollop'
+
+opts = Trollop::options do
+  version "mongo-proxy #{MongoProxy::VERSION} (c) Peter Bakkum"
+  banner <<-END
+mongo-proxy runs a local proxy server for MongoDB. It allows you to watch the MongoDB wire traffic, run hooks on it, and force traffic to be read-only. This command is intended for easy access, you can also access the same functionality with the MongoProxy class.
+
+Usage:
+    mongo-proxy [options]
+
+Examples:
+
+    mongo-proxy -P 27017 -p 29017 -r -d
+    (runs a proxy at port 29017 to connect to a local mongo in read-only and debug mode)
+
+More info can be found at the gem website at github.com/bakks/mongo-proxy
+
+Command line options:
+
+END
+
+  opt :client_host, 'Set the host to bind the proxy socket on.', :default => '127.0.0.1', :short => 'h'
+  opt :client_port, 'Set the port to bind the proxy socket on.', :default => 27018, :short => 'p'
+  opt :server_host, 'Set the backend hostname which we proxy.', :default => '127.0.0.1', :short => 'H'
+  opt :server_port, 'Set the backend port which we proxy.', :default => 27017, :short => 'P'
+  opt :read_only, 'Prevent any traffic that writes to the database.', :default => false
+  opt :motd, 'Set a message-of-the-day to display to clients when they connect.', :default => nil
+  opt :verbose, 'Print out MongoDB wire traffic.', :default => true
+  opt :debug, 'Print log lines in a more human-readible format.', :default => true
+end
+
+keys = [:client_host, :client_port, :server_host, :server_port, :read_only, :motd, :verbose, :debug]
+
+args = {}
+keys.each { |key| args[key] = opts[key] }
+
+m = MongoProxy.new(args)
+m.start
+

data/lib/mongo-proxy.rb

@@ -0,0 +1,3 @@
+require_relative 'mongo-proxy/wire'
+require_relative 'mongo-proxy/proxy'
+require_relative 'mongo-proxy/auth'

data/lib/mongo-proxy/auth.rb

@@ -0,0 +1,151 @@
+require 'securerandom'
+
+class AuthMongo
+  @@admin_cmd_whitelist = [
+    { 'ismaster' => 1 },
+    { 'isMaster' => 1 },
+    { 'listDatabases' => 1},
+    {
+      'replSetGetStatus' => 1,
+      'forShell' => 1
+    }
+  ]
+
+  def initialize(config = nil)
+    @config = config
+    @log = @config[:logger]
+    @request_id = 20
+    @last_error = {}
+  end
+
+  def wire_auth(conn, msg)
+    return nil unless msg
+
+    authed, response = auth(conn, msg)
+
+    if !authed
+      @last_error[conn] = true
+    else
+      @last_error[conn] = false
+    end
+
+    if response
+      return WireMongo::build_reply(response, get_request_id, msg[:header][:requestID])
+    else
+      return authed
+    end
+  end
+
+  private
+
+  def reply_motd
+    motd = @config[:motd]
+    motd = motd.split("\n")
+    return true, {
+      'totalLinesWritten' => motd.size,
+      'log' => motd,
+      'ok' => 1.0
+    }
+  end
+
+  def reply_unauth(db, coll)
+    @log.info "replying unauthed for collection #{db}.#{coll}"
+    return false, {
+      'ok' => 0,
+      'n' => 0,
+        'code' => 2,
+        'errmsg' => 'Writes and Javascript execution are disallowed in this interface.',
+      'writeErrors' => {
+        'index' => 0,
+        'code' => 2,
+        'errmsg' => 'Writes and Javascript execution are disallowed in this interface.'
+      }
+    }
+  end
+
+  def reply_error(err)
+    return false, {
+      'errmsg' => err,
+      'ok' => 0.0
+    }
+  end
+
+  def reply_ok
+    return true, {
+      'ok' => 1.0
+    }
+  end
+
+  def get_request_id
+    x = @request_id
+    @request_id += 1
+    return x
+  end
+
+
+  def auth(conn, msg)
+    op = msg[:header][:opCode]
+
+    if op == WireMongo::OP_KILL_CURSORS
+      return true, nil
+    end
+
+    db = msg[:database]
+    coll = msg[:collection]
+    query = (msg[:query] or {})
+
+    case op
+    when WireMongo::OP_QUERY, WireMongo::OP_GET_MORE
+      return reply_unauth(db, coll) unless db and coll
+      return reply_unauth(db, coll) if query['$where'] != nil
+
+      # handle authentication process
+      if coll == '$cmd'
+
+        if query['count']
+          # fields key can be nil but must exist
+          unless query.size == 3 and query['query'] and query.has_key? 'fields'
+            return reply_unauth(db, coll)
+          end
+          return true, nil
+
+        elsif query['getlasterror'] == 1
+          if @last_error[conn]
+            return reply_unauth(db, coll)
+            @last_error[conn] = false
+          else
+            return true, nil
+          end
+
+        # allow ismaster query, listDatabases query
+        elsif db == 'admin'
+          if @@admin_cmd_whitelist.include?(query)
+            return true, nil
+          elsif query['getLog'] == 'startupWarnings'
+            if @config[:motd]
+              return reply_motd
+            else
+              return true
+            end
+          end
+
+        end # if db == 'admin'
+        return reply_unauth(db, coll)
+      end # if coll == '$cmd'
+
+      return true, nil if coll == 'system.namespaces' # list collections
+      return reply_unauth(db, coll) if coll[0] == '$' #other command
+
+      return true, nil
+
+    when WireMongo::OP_UPDATE, WireMongo::OP_INSERT, WireMongo::OP_DELETE
+      #return reply_unauth(db, coll)
+      return false, nil
+
+    else
+      return reply_unauth(db, coll)
+
+    end
+  end
+end
+

data/lib/mongo-proxy/proxy.rb

@@ -0,0 +1,172 @@
+require 'em-proxy'
+require 'logger'
+require 'json'
+require 'pp'
+require 'socket'
+require 'timeout'
+
+
+# This class uses em-proxy to help listen to MongoDB traffic, with some
+# parsing and filtering capabilities that allow you to enforce a read-only
+# mode, or use your own arbitrary logic.
+class MongoProxy
+  VERSION = '1.0.0'
+
+  def initialize(config = nil)
+    # default config values
+    @config = {
+      :client_host => '127.0.0.1', # Set the host to bind the proxy socket on.
+      :client_port => 29017,       # Set the port to bind the proxy socket on.
+      :server_host => '127.0.0.1', # Set the backend host which we proxy.
+      :server_port => 27017,       # Set the backend port which we proxy.
+      :motd => nil,                # Set a message-of-the-day to display to clients when they connect. nil for none.
+      :read_only => false,         # Prevent any traffic that writes to the database.
+      :verbose => false,           # Print out MongoDB wire traffic.
+      :logger => nil,              # Use this object as the logger instead of creating one.
+      :debug => false              # Print log lines in a more human-readible format.
+    }
+    @front_callbacks = []
+    @back_callbacks = []
+
+    # apply argument config to the default values
+    (config || []).each do |k, v|
+      if @config.include?(k)
+        @config[k] = v
+      else
+        raise "Unrecognized configuration value: #{k}"
+      end
+    end
+
+    # debug implies verbose
+    @config[:verbose] = true if @config[:debug]
+
+    # Set up the logger for mongo proxy. Users can also pass their own
+    # logger in with the :logger config value.
+    unless @config[:logger]
+      @config[:logger] = Logger.new(STDOUT)
+      @config[:logger].level = (@config[:verbose] ? Logger::DEBUG : Logger::WARN)
+
+      if @config[:debug]
+        @config[:logger].formatter = proc do |_, _, _, msg|
+          if msg.is_a?(Hash)
+            "#{JSON::pretty_generate(msg)}\n\n"
+          else
+            "#{msg}\n\n"
+          end
+        end
+      end
+    end
+
+    unless port_open?(@config[:server_host], @config[:server_port])
+      raise "Could not connect to MongoDB server at #{@config[:server_host]}.#{@config[:server_port]}"
+    end
+
+    @log = @config[:logger]
+    @auth = AuthMongo.new(@config)
+  end
+
+  def start
+    # em proxy launches a thread, this is the error handler for it
+    EM.error_handler do |e|
+      @log.error [e.inspect, e.backtrace.first]
+      raise e
+    end
+
+    # kick off em-proxy
+    Proxy.start({
+        :host => @config[:client_host],
+        :port => @config[:client_port],
+        :debug => false
+      },
+      &method(:callbacks))
+  end
+
+  def add_callback_to_front(&block)
+    @front_callbacks.insert(0, block)
+  end
+
+  def add_callback_to_back(&block)
+    @back_callbacks << block
+  end
+
+  private
+
+  def callbacks(conn)
+    conn.server(:srv, {
+      :host => @config[:server_host],
+      :port => @config[:server_port]})
+
+    conn.on_data do |data|
+      # parse the raw binary message
+      raw_msg, msg = WireMongo.receive(data)
+
+      @log.info 'from client'
+      @log.info msg
+
+      if raw_msg == nil
+        @log.info "Client disconnected"
+        return
+      end
+
+      @front_callbacks.each do |cb|
+        msg = cb.call(conn, msg)
+        break unless msg
+      end
+      next unless msg
+
+      # get auth response about client query
+      authed = (@config[:read_only] == true ? @auth.wire_auth(conn, msg) : true)
+      r = nil
+
+      if authed == true # auth succeeded
+        @back_callbacks.each do |cb|
+          msg = cb.call(conn, msg)
+          break unless msg
+        end
+        next unless msg
+
+        r = WireMongo::write(msg)
+
+      elsif authed.is_a?(Hash) # auth had a direct response
+        response = WireMongo::write(authed)
+        conn.send_data(response)
+
+      else # otherwise drop the message
+        @log.info 'dropping message'
+
+      end
+
+      r
+    end
+
+    # messages back from the server
+    conn.on_response do |backend, resp|
+      if @config[:verbose]
+        _, msg = WireMongo::receive(resp)
+        @log.info 'from server'
+        @log.info msg
+      end
+
+      resp
+    end
+
+    conn.on_finish do |backend, name|
+      @log.info "closing client connection #{name}"
+    end
+  end
+
+  # http://stackoverflow.com/questions/517219/ruby-see-if-a-port-is-open
+  def port_open?(ip, port, seconds=1)
+    Timeout::timeout(seconds) do
+      begin
+        TCPSocket.new(ip, port).close
+        true
+      rescue Errno::ECONNREFUSED, Errno::EHOSTUNREACH
+        false
+      end
+    end
+  rescue Timeout::Error
+    false
+  end
+end
+