monsoon-droplet 1.0.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 7e6232ff9dd488d25b28e378d87905d90a8e80bf
+  data.tar.gz: 6628745f73d30cc0c87cb5d19c916fc9a4f38e3e
+SHA512:
+  metadata.gz: fb6209b101d0aa6437d9b90fba8e215f2f8d81439ec9754b5a1ecd8b35c93e19523e2c1dee06ecca4be7ce7073fc43aabf5e709e983b880edf5ad81045733e4f
+  data.tar.gz: 16ba8785319d604cf311db264f70cb00abc1482314b58f34d97f70831d2e7688827aaf9672c8e26131ba30fe39bd620b867342b1c9ce218e09073746449728d2

data/MIT-LICENSE

@@ -0,0 +1,20 @@
+Copyright 2016 Neil Gupta
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,119 @@
+# Monsoon
+
+[https://www.github.com/neilgupta/monsoon](https://www.github.com/neilgupta/monsoon)
+
+[![Gem Version](https://badge.fury.io/rb/monsoon-droplet.png)](http://badge.fury.io/rb/monsoon-droplet)
+
+Monsoon writes messages to a stream of data. That stream can be anything, from an external service to a log or even an in-memory array, as long as you write an adapter for it. Out of the box, Monsoon includes adapters for AWS Kinesis and stdout.
+
+Monsoon allows you to define versioned contracts for your messages, treating your streams as an API for other clients to consume. Each message sent by Monsoon includes only the keys defined in the contract, the version of the contract used, and whether or not that contract is deprecated.
+
+## Installation
+
+Install the gem on your machine:
+
+```
+gem install monsoon-droplet
+```
+
+Or add it to your `Gemfile`:
+
+```ruby
+gem 'monsoon-droplet'
+```
+
+## Setup
+
+Create `/config/initializers/monsoon.rb` and add:
+
+```ruby
+require 'monsoon'
+require 'monsoon/streams/kinesis'
+require 'monsoon/streams/console'
+
+Monsoon.versions_schema = {
+  "analytics": {
+    "1.0.0": ["user_id", "event_type", "timestamp", "data"],
+    "1.0.1": ["user_id", "event_type", "timestamp", "data"],
+    "1.1.0": ["user_id", "event_type", "timestamp", "data", "resource_id"],
+    "2.0.0": ["event_type", "timestamp", "data", "resource_id"]
+  },
+  "transcodes": {
+    "1.0.0": ["filename", "status"],
+    "2.0.0": ["filename", "sources"]
+  }
+}
+
+# if using the Kinesis stream...
+ENV['AWS_ACCESS_KEY_ID'] = "YOUR AWS ACCESS KEY"
+ENV['AWS_SECRET_ACCESS_KEY'] = "YOUR AWS SECRET"
+ENV['AWS_REGION'] = "YOUR AWS REGION"
+```
+
+### Stream Adapters
+
+To add a stream adapter, just require it:
+
+```ruby
+require 'monsoon/streams/kinesis'
+```
+
+This will add the built-in AWS Kinesis adapter. `monsoon/streams/console` is also built-in for testing by writing your streams to STDOUT.
+
+You can write your own stream adapter by creating a class that implements `#put_records(stream_name, records_array)` and then add it to Monsoon with `Monsoon.streams << YourNewAdapter.new`. See `Monsoon::Streams::Console` for an example.
+
+Monsoon will send your messages to all added streams by default.
+
+### Versions Schema
+
+To auto-generate versioned copies of your message, configure Monsoon with your versions schema. This is just a hash of stream names and keys that tells Monsoon, given a stream name, here's a list of versions we support and the keys each version should include in its message. For example, if you have the following schema defined:
+
+```ruby
+Monsoon.versions_schema = {
+  "analytics": {
+    "1.0.0": ["user_id", "event_type", "timestamp", "data"],
+    "1.0.1": ["user_id", "event_type", "timestamp", "data"],
+    "1.1.0": ["user_id", "event_type", "timestamp", "data", "resource_id"],
+    "2.0.0": ["event_type", "timestamp", "data", "resource_id"]
+  },
+  "transcodes": {
+    "1.0.0": ["filename", "status"],
+    "2.0.0": ["filename", "sources"]
+  }
+}
+```
+
+and you want to send the message `{user_id: 1, event_type: 'play', timestamp: 23, data: 'some other data', resource_id: 10}` to your `analytics` stream, Monsoon will automatically generate 4 versions of your message for each defined version. The versioned message will only include the keys needed for that version, as well as 2 new keys: 'droplet_version' and 'droplet_deprecated'. `droplet_version` tells the receiving client which version they're reading, and `version_deprecated` is true if there is a newer version defined in the schema.
+
+If the original message does not include all of the keys needed for an older version (e.g., `user_id` is left out in the above example), then Abacus will only send the versions it has the data for (e.g., it will only send v2.0+ for the 'analytics' stream).
+
+### Droplets
+
+To send a message after you've configured Monsoon, just create a new Droplet and stream it:
+
+```ruby
+droplet = Monsoon::Droplet.new("analytics", {user_id: 1, event_type: 'play', timestamp: 23, data: 'some other data', resource_id: 10})
+droplet.stream
+```
+
+Droplets can also take a hash with the following options:
+
+* `:versioning` - can be `:skip`, `:enforce`, or `nil`
+  - `:skip` will ignore the versions schema and write the data exactly as passed
+  - `:enforce` will require using the schema and will write nothing if unable to version
+  - `nil` will try to write versioned droplets and fallback to raw data (default)
+
+`#stream` can also take an optional stream adapter (any object that implements `#put_records`) to only stream to that adapter. For example:
+
+```ruby
+droplet = Monsoon::Droplet.new("my_stream", {user_id: 1, event_type: 'play', timestamp: 23}, {versioning: :skip})
+droplet.stream(YourNewAdapter.new)
+```
+
+## Author
+
+Neil Gupta [http://metamorphium.com](http://metamorphium.com)
+
+## License
+
+The MIT License (MIT) Copyright (c) 2016 Neil Gupta. See [MIT-LICENSE](https://raw.github.com/neilgupta/monsoon/master/MIT-LICENSE)

data/Rakefile

@@ -0,0 +1,6 @@
+require 'bundler/gem_tasks'
+require 'rspec/core/rake_task'
+
+RSpec::Core::RakeTask.new(:spec)
+
+task :default => :spec

data/lib/monsoon.rb

@@ -0,0 +1,24 @@
+require 'monsoon/sem_ver'
+require 'monsoon/versions_schema'
+require 'monsoon/droplet'
+
+module Monsoon
+  @@streams = []
+  @@versions_schema = {}
+
+  def self.streams
+    @@streams
+  end
+
+  def self.streams=(streams)
+    @@streams = streams
+  end
+
+  def self.versions_schema
+    @@versions_schema
+  end
+
+  def self.versions_schema=(versions_schema)
+    @@versions_schema = versions_schema
+  end
+end

data/lib/monsoon/droplet.rb

@@ -0,0 +1,42 @@
+module Monsoon
+  class Droplet
+    attr_reader :stream_name
+    attr_reader :options
+    attr_reader :raw_data
+    attr_reader :data
+
+    # new - creates a droplet
+    #   stream_name - name of the stream to write to (required)
+    #   data - hash of data to write to stream (required)
+    #   options - optional hash
+    #     :versioning => can be :skip, :enforce, or nil
+    #       :skip will ignore the versions schema and write the data exactly as passed
+    #       :enforce will require using the schema and will write nothing if unable to version
+    #       nil will try to write versioned droplets and fallback to raw data (default)
+    #
+    # @example
+    #   Monsoon::Droplet.new('analytics', {user_id: 3, event_type: 'play'}, {versioning: :enforce})
+    def initialize(stream_name, raw_data, options = {})
+      @stream_name = stream_name
+      @options = options
+      @raw_data = raw_data
+      @data = VersionsSchema.new(@stream_name).get_droplets(@raw_data) unless @options[:versioning] == :skip
+      @data = [@raw_data] if blank? && @options[:versioning] != :enforce
+    end
+
+    # stream - writes the droplet to all configured stream adapters
+    #   streamer - (optional) stream adapter instance to limit which adapter is used (adapter must implement `#put_records`)
+    #
+    # @example
+    #   droplet = Monsoon::Droplet.new('analytics', {user_id: 3, event_type: 'play'})
+    #   droplet.stream(Monsoon::Streams::Console.new)
+    def stream(streamer = false)
+      return if blank?
+      streamer ? streamer.put_records(@stream_name, @data) : Monsoon.streams.each {|s| stream(s) }
+    end
+
+    def blank?
+      @data.nil? || @data.empty?
+    end
+  end
+end

data/lib/monsoon/sem_ver.rb

@@ -0,0 +1,31 @@
+# This is a helper class for parsing and comparing semantic version strings (eg "1.1.0")
+module Monsoon
+  class SemVer
+    include Comparable
+    attr_reader :major, :minor, :patch
+
+    def initialize(version)
+      version = version.to_s if version.is_a?(Symbol)
+      raise ArgumentError, 'Not a valid semantic version' unless self.class.valid?(version)
+
+      ver = version.split('.')
+      @major = ver[0].to_i
+      @minor = ver[1].to_i
+      @patch = ver[2].to_i
+    end
+
+    def <=> (other)
+      other = self.class.new(other) if self.class.valid?(other)
+      [major, minor, patch] <=> [other.major, other.minor, other.patch] if other.is_a?(self.class)
+    end
+
+    def to_s
+      "#{major}.#{minor}.#{patch}"
+    end
+
+    def self.valid?(version)
+      version = version.to_s if version.is_a?(Symbol)
+      version.is_a?(String) && version.split('.').take(3).map(&:to_i).reduce(0, :+) > 0
+    end
+  end
+end

data/lib/monsoon/streams/console.rb

@@ -0,0 +1,15 @@
+require 'json'
+
+module Monsoon
+  module Streams
+    class Console
+      def put_records(stream, records)
+        records.each do |r|
+          puts "Streaming to #{stream}: #{JSON.pretty_generate(r)}"
+        end
+      end
+    end
+  end
+end
+
+Monsoon.streams << Monsoon::Streams::Console.new

data/lib/monsoon/streams/kinesis.rb

@@ -0,0 +1,23 @@
+require 'aws-sdk'
+
+module Monsoon
+  module Streams
+    class Kinesis
+      def initialize
+        @client = Aws::Kinesis::Client.new
+      end
+
+      def put_records(stream, records)
+        data = records.map do |r|
+          {
+            data: JSON.generate(r),
+            partition_key: 'monsoon'
+          }
+        end
+        @client.put_records(records: data, stream_name: stream)
+      end
+    end
+  end
+end
+
+Monsoon.streams << Monsoon::Streams::Kinesis.new

data/lib/monsoon/version.rb

@@ -0,0 +1,3 @@
+module Monsoon
+  VERSION = '1.0.0'
+end

data/lib/monsoon/versions_schema.rb

@@ -0,0 +1,40 @@
+module Monsoon
+  class VersionsSchema
+    attr_reader :stream
+
+    def initialize(stream)
+      @stream = stream
+    end
+
+    def get_droplets(record)
+      return if record.nil?
+
+      schema.map do |version, keys|
+        symbolized_keys = keys.map(&:to_sym)
+        # Only keep the record keys that are defined in this version's schema
+        versioned_record = record.select{|k,v| symbolized_keys.include?(k.to_sym)}
+        # stringify record keys
+        versioned_record = Hash[versioned_record.map { |k, v| [k.to_s, v] }]
+        # add versioning metadata
+        versioned_record.merge({
+          'droplet_version' => version.to_s,
+          'droplet_deprecated' => deprecated?(version)
+        }) if versioned_record.keys.length == keys.length # make sure we got all the keys for this version
+      end.compact
+    end
+
+    def deprecated?(version)
+      SemVer.new(version) < versions.max_by {|v| SemVer.new(v)} unless versions.empty?
+    end
+
+    def versions
+      schema.keys
+    end
+
+    private
+
+    def schema
+      @schema ||= Monsoon.versions_schema[@stream.to_sym] || Monsoon.versions_schema[@stream.to_s] || {}
+    end
+  end
+end

metadata

@@ -0,0 +1,97 @@
+--- !ruby/object:Gem::Specification
+name: monsoon-droplet
+version: !ruby/object:Gem::Version
+  version: 1.0.0
+platform: ruby
+authors:
+- Neil Gupta
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2016-02-10 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: aws-sdk
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2'
+- !ruby/object:Gem::Dependency
+  name: rake
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: rspec
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+description: Monsoon makes sending messages to an external resource super easy by
+  versioning your messages so other services won't break when your messages change.
+email: neil@instructure.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- MIT-LICENSE
+- README.md
+- Rakefile
+- lib/monsoon.rb
+- lib/monsoon/droplet.rb
+- lib/monsoon/sem_ver.rb
+- lib/monsoon/streams/console.rb
+- lib/monsoon/streams/kinesis.rb
+- lib/monsoon/version.rb
+- lib/monsoon/versions_schema.rb
+homepage: https://github.com/neilgupta/monsoon
+licenses:
+- MIT
+metadata: {}
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 2.4.6
+signing_key: 
+specification_version: 4
+summary: Super simple message versioning
+test_files: []
+has_rdoc: