rails-pipeline 1.1.1

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 3baf27587029e1f4d0c6005dabfd09b8ff8efd68
+  data.tar.gz: de357cd65ca250bae2470b69e4f3d935b407fc60
+SHA512:
+  metadata.gz: c0741b47a72c591d3210004e5f35f71fa4efc209de96b0dfac777a18e05cbe97a6f6320bc6885af60daa476bc59e913ab1d36859699ebf233401b13b16efaa72
+  data.tar.gz: e5614fa21bc21ada92437787a8ac22e2fc6e1534c1c010bd3de1614cde42fc00acada91a5141903315e00d7eb05e1570dcb8b3c142fe0b3f7344cccc096137eb

data/MIT-LICENSE

@@ -0,0 +1,20 @@
+Copyright 2014 Andy O'Neill, ADKM Inc
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,227 @@
+# RailsPipeline
+
+Emit a versioned stream of changes to a pub/sub queue when ActiveRecord models are
+saved. This gem supports Redis, AWS (SNS/SQS), and IronMQ publishing targets.
+The Redis backend supports a forwarding system to a cloud MQ like IronMQ.
+Messages are encrypted in transit using AES symmetric encryption and a shared
+secret.
+
+## Motivation
+
+Many systems evolve into a inter-related collection of Rails applications as
+they grow out of their initial monolithic design. This is often coincident with
+independent teams taking responsibility for different aspects of the business
+and different applications. In our case this has manifested itself as
+one team that is responsible for the ecommerce platform (web, mobile etc) and
+another team that works on data warehousing and personalization. The data team
+was using a read-only replica of the platform team's database for data
+warehousing, which tightly coupled us together when the platform team wanted to
+make schema changes for performance or development velocity reasons.
+
+This project allows us to offer a versioned API that we can maintain backwards
+compatibility for while changing the underlying code and schema as we see fit.
+Consumers of data will upgrade to new schema versions as they need to and when
+they are able.
+
+## Usage
+
+Install with bundler
+
+    gem 'rails-pipeline'
+
+You should create a private repository containing your protocolbuffers schemas
+and also depend on that, or use `git subtree` to bring it into your Rails apps.
+
+For any models that you wish to publish changes, just include the appropriate
+pipeline emitter
+
+    include RailsPipeline::RedisEmitter
+
+Each queue backend has different methods of consuming messages as a subscriber,
+but for IronMQ there is an implementation of a webhook subscriber (details
+below).
+
+The following environment variables control pipeline operations
+
+	# If set, do not emit or process incoming messages
+    DISABLE_RAILS_PIPELINE
+
+	# If set, do not emit pipeline messages
+	DISABLE_RAILS_PIPELINE_EMISSION
+
+	# If set, do not process incoming messages as a subscriber, just drop them
+	DISABLE_RAILS_PIPELINE_PROCESSING
+
+The following environment variable sets the shared secret for pipeline encryption
+
+    PIPELINE_SECRET
+
+You can pass in a logger for RailsPipeline in an initializer e.g.
+
+	RailsPipeline.logger = Rails.logger
+	RailsPipeline.logger.level = Log4r::DEBUG
+
+## Backends
+
+### Redis
+
+<table>
+<tr><td>Redis Emitter</td><td>Only as a forwarding intermediary</td></tr>
+<tr><td>Redis Subscriber</td><td>Only forwarder</td></tr>
+</table>
+
+The implementation for Redis assumes you want to use it as a local forwarding
+queue to a more scalable service such as AWS or IronMQ. Thus all messages are
+pushed onto a single Redis queue and include the name of the target topic/queue.
+We have included a bouncer process that will read from the Redis queue (in
+parallel if need be) and forward on to IronMQ. Adding an AWS forwarder would be
+trivial.
+
+It may be desirable to write a full pub/sub emitter for Redis (rather than just
+a forwarder.)
+
+#### Redis Config
+
+The following environment variables are checked for Redis urls (default:
+localhost:6379):
+
+	REDISCLOUD_URL
+	REDISTOGO_URL
+
+Alternatively you could pass in an instance of the Redis client in an
+initializer:
+
+    RailsPipeline::RedisPublisher.redis = MyRedisFactory.get
+
+The key name of the redis queue should be set in an initializer, e.g.
+
+    RailsPipeline::RedisPublisher.namespace = "rails-pipeline-spec"
+
+### IronMQ
+
+<table>
+<tr><td>IronMQ Emitter</td><td>YES</td></tr>
+<tr><td>Redis-to-IronMQ Forwarder</td><td>YES</td></tr>
+<tr><td>IronMQ Subscriber</td><td>YES</td></tr>
+</table>
+
+#### Emitter
+
+Each model/version emits ProtocolBuffer messages to a specific IronMQ queue. If
+that queue is set as a "push" queue. Subscribers can then add themselves as http
+webhook endpoints for the push queue and messages will be delivered to them.
+
+
+#### Subscriber
+
+There is a subscriber implementation for IronMQ as an http endpoint.
+
+Mount the Sinatra endpoint in your app's routes.rb
+
+    match "/ironmq" => RailsPipeline::IronmqSubscriber, :anchor => false
+
+Register your own models as recipients of different pipeline message types and
+versions (in an Rails initializer):
+
+	RailsPipeline::Subscriber.register(SomeModel_2_0, MyModel)
+
+You will need to write a `MyModel#from_pipeline_2_0()` method. You can also
+register any Proc as a processor for messages.
+
+Add your URL as a subscriber to the push queues you care about using the
+supplied 'pipeline' command
+
+    pipeline ironmq-subscribe-endpoint http://my.domain.com/ironmq some_models
+
+You may find [ngrok](http://ngrok.com) helpful for developing and debugging.
+
+#### Iron.io Config
+
+The 'iron_mq' gem picks up the following environment variables
+
+    IRON_PROJECT_ID
+	IRON_TOKEN
+
+### AWS (Simple Notification Service)
+
+<table>
+<tr><td>SNS Emitter</td><td>YES</td></tr>
+<tr><td>Redis-to-SNS Forwarder</td><td>NO, but easy to add.</td></tr>
+<tr><td>SQS Polling Subscriber</td><td>NO</td></tr>
+<tr><td>SNS Webhook Subscriber</td><td>NO</td></tr>
+</table>
+
+We include a proof-of-concept AWS emitter, written with the idea in mind to use
+SQS as pub/sub queues and polling subscribers. It would also be possible to
+publish to SNS and have multiple subscribers receive http webhook messages as in
+IronMQ.
+
+There are some commands in the 'pipeline' script to configure SNS/SQS:
+
+
+Create SNS topics to publish to:
+
+	pipeline sns-create-topic TABLE_NAME --env ENV --version VERSION
+
+Create and SQS queue and subscribe it to a TOPIC (one per subscribing rails
+	app)
+
+	pipeline sqs-subscribe-app APP TABLE_NAME[,TABLE_NAME_2,...] --env ENV --version VERSION
+
+#### AWS Config
+
+The AWS gem picks up the following environment variables
+
+	AWS_ACCESS_KEY_ID
+	AWS_SECRET_ACCESS_KEY
+
+In addition, we use the numerical 'owner id' for your account which should be
+set as
+
+	AWS_ACCOUNT_ID
+
+
+## Protocol Buffers
+
+To build the test protocol buffers ruby files in rails-pipeline:
+
+    brew install protobuf
+    make
+
+We have created a private repository gem for our protocol buffers definitions.
+This is laid out like
+
+	harrys-pipeline/lib/harrys/pipeline/my_model_1_1.proto
+
+Proto files look like this
+
+<pre>
+	package Harrys.Pipeline;
+
+	message Order__1__0 {
+	  required int32 id = 1;
+      required double created_at = 2;
+	  required double updated_at = 3;
+	  ...
+	}
+</pre>
+
+We then have a Makefile almost identical to the one in this gem to build our
+.pb.rb files:
+
+<pre>
+GENDIR=./lib/harrys/pipeline
+RUBY_PROTOC=bundle exec ruby-protoc
+PROTOS=$(wildcard $(GENDIR)/*.proto)
+PBS=$(PROTOS:%.proto=%.pb.rb)
+
+all: $(PBS)
+
+%.pb.rb: %.proto
+        $(RUBY_PROTOC) $<
+
+clean:
+        rm -f $(PBS)
+</pre>
+
+[![TravisCI](https://travis-ci.org/harrystech/rails-pipeline.png)](https://travis-ci.org/harrystech/rails-pipeline)

data/Rakefile

@@ -0,0 +1,27 @@
+#!/usr/bin/env rake
+begin
+  require 'bundler/setup'
+rescue LoadError
+  puts 'You must `gem install bundler` and `bundle install` to run rake tasks'
+end
+begin
+  require 'rdoc/task'
+rescue LoadError
+  require 'rdoc/rdoc'
+  require 'rake/rdoctask'
+  RDoc::Task = Rake::RDocTask
+end
+
+RDoc::Task.new(:rdoc) do |rdoc|
+  rdoc.rdoc_dir = 'rdoc'
+  rdoc.title    = 'RailsPipeline'
+  rdoc.options << '--line-numbers'
+  rdoc.rdoc_files.include('README.rdoc')
+  rdoc.rdoc_files.include('lib/**/*.rb')
+end
+
+APP_RAKEFILE = File.expand_path("../spec/dummy/Rakefile", __FILE__)
+load 'rails/tasks/engine.rake'
+
+Bundler::GemHelper.install_tasks
+

data/bin/pipeline

@@ -0,0 +1,138 @@
+#!/usr/bin/env ruby
+
+# command line tool for administering data pipeline queues
+#
+# To create queues for an app following certain models:
+#     pipeline subscribe-app APP_NAME orders,products,users
+
+require 'thor'
+require 'aws-sdk'
+require 'iron_mq'
+require 'pry'
+
+class Pipeline < Thor
+    # TODO: Kinda clunky, but there is a way to get the account ID programmatically:
+    # https://forums.aws.amazon.com/thread.jspa?threadID=108012
+    AWS_ACCOUNT_ID = ENV["AWS_ACCOUNT_ID"]
+
+    def initialize(args = [], local_options = {}, config = {})
+        super(args, local_options, config)
+        @sns = AWS::SNS.new
+        @sqs = AWS::SQS.new
+        @iam = AWS::IAM.new
+    end
+
+
+    desc "sns-create-topic TABLE_NAME --env ENV --version VERSION", "create topic for TABLE_NAME"
+    option :env
+    option :version
+    def create_sns_topic(table_name)
+        env = options.fetch(:env, "test")
+        version = options.fetch(:version, "1_0")
+        topic_name = _topic_name(table_name, env, version)
+        topic = _find_or_create_topic(topic_name)
+        return topic
+    end
+
+    desc "sqs-subscribe-app APP TABLE_NAME[,TABLE_NAME_2,...] --env ENV --version VERSION", "create queue(s) for APP subscribed to TABLE_NAME feeds"
+    option :env
+    option :version
+    def sqs_subscribe_app(app_name, table_names)
+        env = options.fetch(:env, "test")
+        version = options.fetch(:version, "1_0")
+
+        table_names.split(',').each do |table_name|
+            topic_name = _topic_name(table_name, env, version)
+            queue_name = _queue_name(topic_name, app_name)
+            queue = _find_or_create_queue(queue_name)
+            topic = _find_or_create_topic(topic_name)
+            sub = _subscribe_queue_to_topic(queue, topic)
+        end
+    end
+
+    # Placeholders for IronMQ functions
+    desc "ironmq-subscribe-endpoint ENDPOINT TABLE_NAME_1[,TABLE_NAME_2,...] --env ENV --version VERSION", "subscribe the endpoint to updates on ironmq"
+    option :env
+    option :version
+    def ironmq_subscribe_endpoint(endpoint, table_names)
+        env = options.fetch('env', "test")
+        version = options.fetch('version', "1_0")
+
+        table_names.split(',').each do |table_name|
+            topic_name = _topic_name(table_name, env, version)
+            queue = _iron.queue(topic_name)
+            options = {
+              push_type: "multicast",
+              retries: 10,
+              retries_delay: 10
+            }
+            puts "Setting push queue settings on #{queue.name}"
+            queue.update(options)
+            puts "Subscribe #{topic_name} to #{endpoint}"
+            queue.add_subscriber({ url: endpoint })
+        end
+    end
+
+
+    private
+
+    def _topic_name(table_name, env, version)
+        "harrys-#{env}-v#{_major_version(version)}-#{table_name}"
+    end
+
+    def _queue_name(topic_name, app)
+        "#{topic_name}-#{app}"
+    end
+
+    def _major_version(v)
+        v.include?('_') ? v.split('_')[0] : v
+    end
+
+    def _find_or_create_topic(topic_name)
+        arn = _topic_arn(topic_name)
+        topic = @sns.topics[arn]
+        begin
+            topic.owner
+            puts "Found topic #{arn}"
+            return topic
+        rescue AWS::SNS::Errors::NotFound
+            puts "Creating topic #{topic_name}"
+            @sns.client.create_topic({name: topic_name})
+            return _find_or_create_topic(topic_name)
+        end
+    end
+
+    def _find_or_create_queue(queue_name)
+        puts "Find or create queue #{queue_name}"
+        begin
+            queue = @sqs.queues.named(queue_name)
+            return queue
+        rescue AWS::SQS::Errors::NonExistentQueue
+            @sqs.queues.create(queue_name)
+            puts "Creating queue #{queue_name}"
+            return _find_or_create_queue(queue_name)
+        end
+    end
+
+    def _subscribe_queue_to_topic(queue, topic)
+        puts "Subscribing #{queue.arn} to #{topic.arn}"
+        return topic.subscribe(queue)
+    end
+
+    def _topic_arn(topic_name, region="us-east-1")
+        "arn:aws:sns:#{region}:#{AWS_ACCOUNT_ID}:#{topic_name}"
+    end
+
+    def _queue_arn(queue_name, region="us-east-1")
+        "arn:aws:sns:#{region}:#{AWS_ACCOUNT_ID}:#{queue_name}"
+    end
+
+    def _iron
+      @iron = IronMQ::Client.new if @iron.nil?
+      return @iron
+    end
+
+end
+
+
+Pipeline.start()

data/bin/redis-to-ironmq.rb

@@ -0,0 +1,20 @@
+#!/usr/bin/env ruby
+
+require 'logger'
+require 'rails-pipeline'
+require 'rails-pipeline/redis_ironmq_forwarder'
+
+# log = Logger.new($stdout)
+log = Logger.new(STDOUT)
+log.progname = 'pipeline'
+log.level = Logger::DEBUG
+RailsPipeline::logger = log
+
+# Pipeline forwarder that reads from redis queue and forwards to ironmq.
+$redis = ENV["REDISCLOUD_URL"] || ENV["REDISTOGO_URL"] || "localhost:6379"
+
+# TODO: non-hardcode this
+key = "harrys-www-pipeline"
+
+forwarder = RailsPipeline::RedisIronmqForwarder.new(key)
+forwarder.start

data/lib/rails-pipeline.rb

@@ -0,0 +1,34 @@
+
+module RailsPipeline
+  class << self
+    # Allow configuration via initializer
+    @@logger = nil
+    def logger
+      if @@logger.nil?
+        @@logger = Rails.logger
+      end
+      @@logger
+    end
+    def logger=(logger)
+      @@logger = logger
+    end
+  end
+  begin
+    require 'newrelic_rpm'
+    HAS_NEWRELIC = true
+  rescue LoadError
+    HAS_NEWRELIC = false
+  end
+end
+
+require "rails-pipeline/emitter"
+require "rails-pipeline/subscriber"
+require "rails-pipeline/symmetric_encryptor"
+require "rails-pipeline/redis_publisher"
+require "rails-pipeline/sns_publisher"
+require "rails-pipeline/ironmq_publisher"
+require "rails-pipeline/handlers/base_handler"
+require "rails-pipeline/handlers/activerecord_crud"
+require "rails-pipeline/handlers/logger"
+require "rails-pipeline/pipeline_version"
+require "rails-pipeline/ironmq_pulling_subscriber"