alephant-publisher-queue 1.0.0 → 1.1.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 106e56c0d1e6eb36e2c9484d19d77a5dbb9a889c
-  data.tar.gz: d427da6b366cb81c375e2eac89e11e3dacd660d5
+  metadata.gz: c504f07d7863ff3576e44fdab5fc636f6e605dff
+  data.tar.gz: 019473a699726070746e77b1afbdd856dc7a00c5
 SHA512:
-  metadata.gz: 537e8cb7c69cbf7662cf32dcb7cbbcc4de6a8170c8119c2effd0f81612b0584de336efd8735b009ab6c7ec8b5cdb42b141688e18b2ecb959b0bea955d0541931
-  data.tar.gz: e90751b235b73eb21a76721a4b10226204b00657293148fe8e9abf85beb9992c15768569197cdf8754b16de5fc82a88c9915d80c7532ebb07d8c29a776666b24
+  metadata.gz: 3a39e4ae36d770ad09eaa5def46d679c1b909d16479f69cfbad0299937479ab03346841a8f43c5a3f2c5ae2ef0c2049efd7793402a0f4a30fdd9127786509532
+  data.tar.gz: 01a491d6cc5b9208b8467c7d19b6ed4c53ae677c6bb7606a753d9e1aef4a39b0eed54a651329017ce08b376a5d1716a8bef841f125ca46fcb034e146bf50f8c7

data/README.md

@@ -6,32 +6,53 @@ Static publishing to S3 based on SQS messages.
 
 ## Dependencies
 
-- JRuby 1.7.8
-- An AWS account (you'll need to create):
-  - An S3 bucket
-  - An SQS Queue (if no sequence id provided then `sequence_id` will be used)
-  - A Dynamo DB table (optional, will attempt to create if can't be found)
+- JRuby 1.7.8+
+- An AWS account, with:
+  - S3 bucket.
+  - SQS Queue.
+  - Dynamo DB table.
 
 ## Migrating from [Alephant::Publisher](https://github.com/BBC-News/alephant-publisher)
 
-1. Add the new gem in your Gemfile (`gem 'alephant-publisher-queue'`).
-2. Run `bundle install`.
-3. Require the new gem in your app (`require 'alephant/publisher/queue'`).
-4. Note that the namespace has changed from `Alephant::Publisher` to `Alephant::Publisher::Queue`.
+Add the new gem in your Gemfile:
+
+```
+gem 'alephant-publisher-queue'
+```
+
+Run:
+
+```
+bundle install
+```
+
+Require the new gem in your app:
+
+```
+require 'alephant/publisher/queue'
+```
+
+**Important** - note that the namespace has changed from `Alephant::Publisher` to `Alephant::Publisher::Queue`.
 
 ## Installation
 
 Add this line to your application's Gemfile:
 
-    gem 'alephant-publisher-queue'
+```
+gem 'alephant-publisher-queue'
+```
 
 And then execute:
 
-    $ bundle
+```
+bundle
+```
 
 Or install it yourself as:
 
-    $ gem install alephant-publisher-queue
+```
+gem install alephant-publisher-queue
+```
 
 ## Setup
 
@@ -42,185 +63,103 @@ access_key_id: ACCESS_KEY_ID
 secret_access_key: SECRET_ACCESS_KEY
 ```
 
-## Usage
-
-**In your application:**
-
-```rb
-require 'alephant'
-
-sequential_proc = Proc.new do |last_seen_id, data|
-  last_seen_id < data['sequence_id'].to_i
-end
-
-set_last_seen_proc = Proc.new do |data|
-  data['sequence_id'].to_i
-end
-
-opts = {
-  :s3_bucket_id         => 'bucket-id',
-  :s3_object_path       => 'path/to/object',
-  :s3_object_id         => 'object_id',
-  :sequencer_table_name => 'your_dynamo_db_table',
-  :sqs_queue_url        => 'https://your_amazon_sqs_queue_url',
-  :sequential_proc      => sequential_proc,
-  :set_last_seen_proc   => set_last_seen_proc,
-  :lookup_table_name    => 'your_lookup_table'
-}
-
-logger = Logger.new
-
-thread = Alephant::Alephant.new(opts, logger).run!
-thread.join
-```
-
-Publisher requires both queue options and writer options to be provided. To ensure a standard format you should use the `Options` class to generate your options before passing them onto the Publisher...
-
-```ruby
-opts = Alephant::Publisher::Options.new
-# => #<Alephant::Publisher::Options:0x0602f958 @queue={}, @writer={}>
-
-opts.queue
-# => {}
-# empty to start with
-
-opts.writer
-# => {}
-# empty to start with
-
-opts.add_queue(:foo => "bar")
-# The key 'foo' is invalid
-# => nil
-
-opts.queue
-# => {}
-# still empty as the foo key was invalid
-
-opts.add_queue(:sqs_queue_url => "bar")
-# => {:sqs_queue_url=>"bar"}
-
-opts.queue
-# => {:sqs_queue_url=>"bar"}
-
-opts.add_writer(:sqs_queue_url => "bar")
-# The key 'sqs_queue_url' is invalid
-# => nil
-# the sqs_queue_url key was valid for the queue options,
-# but is invalid when trying to add it to the writer options
-
-opts.add_writer(:msg_vary_id_path => "bar")
-=> {:msg_vary_id_path=>"bar"}
-
-opts.writer
-=> {:msg_vary_id_path=>"bar"}
-```
-
-logger is optional, and must confirm to the Ruby standard logger interface
+## Structure
 
-Provide a view in a folder (fixtures are optional):
+Provide a view and template:
 
 ```
 └── views
     ├── models
-    │   └── foo.rb
-    ├── fixtures
-    │   └── foo.json
+    │   └── foo.rb
     └── templates
         └── foo.mustache
 ```
 
-**SQS Message Format**
-
-```json
-{
-  "content": "hello world",
-  "sequential_id": 1
-}
-```
-
 **foo.rb**
 
-```rb
-module MyApp
-  module Views
-    class Foo < Alephant::Views::Base
-      def content
-        @data['content']
-      end
-    end
+```ruby
+class Foo < Alephant::Views::Base
+  def content
+    @data['content']
   end
 end
 ```
 
 **foo.mustache**
 
-```mustache
-{{content}}
-```
-
-**S3 Output**
-
 ```
-hello world
+{{ content }}
 ```
 
-## Build the gem locally
-
-If you want to test a modified version of the gem within your application without publishing it then you can follow these steps...
-
-- `gem uninstall alephant-publisher-queue`
-- `gem build alephant-publisher-queue.gemspec` (this will report the file generated which you reference in the next command)
-- `gem install ./alephant-publisher-queue-0.0.1.gem`
-
-Now you can test the gem from within your application as you've installed the gem from the local version rather than your published version
-
-## Preview Server
-
-`alephant preview`
-
-The included preview server allows you to see the html generated by your
-templates, both standalone and in the context of a page.
-
-**Standalone**
-
-`/component/:id/?:fixture?`
-
-### Full page preview
-
-When viewing the component in the context of a page, you'll need to retrieve a
-mustache template to provide the page context.
+## Usage
 
-When performing an update a regex is applied to replace the static hostnames in
-the retrieved html.
+```ruby
+require "alephant/logger"
+require "alephant/publisher/queue"
 
-**Environment Variables**
+module MyApp
+  def self.run!
+    loop do
+      Alephant::Publisher::Queue.create(options).run!
+    rescue => e
+      Alephant::Logger.get_logger.warn "Error: #{e.message}"
+    end
+  end
 
-```sh
-STATIC_HOST_REGEX="static.(sandbox.dev|int|test|stage|live).yourapp(i)?.com\/"
-PREVIEW_TEMPLATE_URL="http://yourapp.com/template"
+  private
+
+  def self.options
+    Alephant::Publisher::Queue::Options.new.tap do |opts|
+      opts.add_queue(
+        :aws_account_id => 'example',
+        :sqs_queue_name => 'test_queue'
+      )
+      opts.add_writer(
+        :keep_all_messages    => 'false',
+        :lookup_table_name    => 'lookup-dynamo-table',
+        :renderer_id          => 'renderer-id',
+        :s3_bucket_id         => 'bucket-id',
+        :s3_object_path       => 'example-s3-path',
+        :sequence_id_path     => '$.sequential_id',
+        :sequencer_table_name => 'sequence-dynamo-table',
+        :view_path            => 'path/to/views'
+      )
+    end
+  end
+end
 ```
 
-**Example Remote Template**
+Add a message to your SQS queue, with the following format:
 
-`id` is the component/folder name  
+```json
+{
+  "content": "Hello World!",
+  "sequential_id": 1
+}
+```
 
-`template` is the mustache template file name  
+Output:
 
-`location_in_page` should be something like (for example) `page_head` (specified within a `preview.mustache` file that the consuming application needs to create).
+```
+Hello World!
+```
 
-- `http://localhost:4567/component/id/template`
-- `http://localhost:4567/preview/id/template/location_in_page`
+S3 Path:
 
-`alephant update`
+```
+S3 / bucket-id / example-s3-path / renderer-id / foo / 7e0c33c476b1089500d5f172102ec03e / 1
+```
 
-**In page**
+## Preview Server
 
-`/preview/:id/:region/?:fixture?`
+[Alephant Preview](https://github.com/BBC-News/alephant-preview) allows you to see the HTML generated by your templates, both standalone and in the context of a page.
 
 ## Contributing
 
 1. [Fork it!](http://github.com/BBC-News/alephant-publisher-queue/fork)
-2. Create your feature branch (`git checkout -b my-new-feature`)
-3. Commit your changes (`git commit -am 'Add some feature'`)
-4. Push to the branch (`git push origin my-new-feature`)
-5. Create new Pull Request
+2. Create your feature branch: `git checkout -b my-new-feature`
+3. Commit your changes: `git commit -am 'Add some feature'`
+4. Push to the branch: `git push origin my-new-feature`
+5. Create new [Pull Request](https://github.com/BBC-News/alephant-publisher-queue/compare).
+
+Feel free to create an [issue](https://github.com/BBC-News/alephant-publisher-queue/issues/new) if you find a bug.

data/lib/alephant/publisher/queue/options.rb

@@ -7,6 +7,8 @@ module Alephant
       class InvalidKeySpecifiedError < StandardError; end
 
       class Options
+        include Logger
+
         attr_reader :queue, :writer
 
         QUEUE_OPTS = [
@@ -47,6 +49,7 @@ module Alephant
             validate type, opts
             instance.merge! opts
           rescue Exception => e
+            logger.metric(:name => "PublisherQueueInvalidKeySpecifiedError", :unit => "Count", :value => 1)
             puts e.message
           end
         end

data/lib/alephant/publisher/queue/sqs_helper/queue.rb

@@ -36,6 +36,7 @@ module Alephant
           def archive(m)
             archiver.see(m) unless archiver.nil?
           rescue StandardError => e
+            logger.metric(:name => "PublisherQueueSQSHelperArchiveFailed", :unit => "Count", :value => 1)
             logger.warn("Queue#archive: archive failed (#{e.message})");
           end
 

data/lib/alephant/publisher/queue/version.rb

@@ -1,7 +1,7 @@
 module Alephant
   module Publisher
     module Queue
-      VERSION = "1.0.0"
+      VERSION = "1.1.0"
     end
   end
 end

metadata

@@ -1,262 +1,262 @@
 --- !ruby/object:Gem::Specification
 name: alephant-publisher-queue
 version: !ruby/object:Gem::Version
-  version: 1.0.0
+  version: 1.1.0
 platform: ruby
 authors:
 - revett
-autorequire:
+autorequire: 
 bindir: bin
 cert_chain: []
-date: 2014-11-14 00:00:00.000000000 Z
+date: 2014-12-08 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
+  name: bundler
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - ~>
       - !ruby/object:Gem::Version
         version: '1.5'
-  name: bundler
-  prerelease: false
   type: :development
+  prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - ~>
       - !ruby/object:Gem::Version
         version: '1.5'
 - !ruby/object:Gem::Dependency
+  name: rspec
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - '>='
       - !ruby/object:Gem::Version
         version: '0'
-  name: rspec
-  prerelease: false
   type: :development
+  prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - '>='
       - !ruby/object:Gem::Version
         version: '0'
 - !ruby/object:Gem::Dependency
+  name: rspec-nc
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - '>='
       - !ruby/object:Gem::Version
         version: '0'
-  name: rspec-nc
-  prerelease: false
   type: :development
+  prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - '>='
       - !ruby/object:Gem::Version
         version: '0'
 - !ruby/object:Gem::Dependency
+  name: guard
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - '>='
       - !ruby/object:Gem::Version
         version: '0'
-  name: guard
-  prerelease: false
   type: :development
+  prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - '>='
       - !ruby/object:Gem::Version
         version: '0'
 - !ruby/object:Gem::Dependency
+  name: guard-rspec
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - '>='
       - !ruby/object:Gem::Version
         version: '0'
-  name: guard-rspec
-  prerelease: false
   type: :development
+  prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - '>='
       - !ruby/object:Gem::Version
         version: '0'
 - !ruby/object:Gem::Dependency
+  name: pry
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - '>='
       - !ruby/object:Gem::Version
         version: '0'
-  name: pry
-  prerelease: false
   type: :development
+  prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - '>='
       - !ruby/object:Gem::Version
         version: '0'
 - !ruby/object:Gem::Dependency
+  name: pry-remote
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - '>='
       - !ruby/object:Gem::Version
         version: '0'
-  name: pry-remote
-  prerelease: false
   type: :development
+  prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - '>='
       - !ruby/object:Gem::Version
         version: '0'
 - !ruby/object:Gem::Dependency
+  name: pry-nav
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - '>='
       - !ruby/object:Gem::Version
         version: '0'
-  name: pry-nav
-  prerelease: false
   type: :development
+  prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - '>='
       - !ruby/object:Gem::Version
         version: '0'
 - !ruby/object:Gem::Dependency
+  name: rake-rspec
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - '>='
       - !ruby/object:Gem::Version
         version: '0'
-  name: rake-rspec
-  prerelease: false
   type: :development
+  prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - '>='
       - !ruby/object:Gem::Version
         version: '0'
 - !ruby/object:Gem::Dependency
+  name: rake
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - '>='
       - !ruby/object:Gem::Version
         version: '0'
-  name: rake
-  prerelease: false
   type: :runtime
+  prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - '>='
       - !ruby/object:Gem::Version
         version: '0'
 - !ruby/object:Gem::Dependency
+  name: aws-sdk
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - ~>
       - !ruby/object:Gem::Version
         version: '1.0'
-  name: aws-sdk
-  prerelease: false
   type: :runtime
+  prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - ~>
       - !ruby/object:Gem::Version
         version: '1.0'
 - !ruby/object:Gem::Dependency
+  name: crimp
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - '>='
       - !ruby/object:Gem::Version
         version: '0'
-  name: crimp
-  prerelease: false
   type: :runtime
+  prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - '>='
       - !ruby/object:Gem::Version
         version: '0'
 - !ruby/object:Gem::Dependency
+  name: alephant-support
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - '>='
       - !ruby/object:Gem::Version
         version: '0'
-  name: alephant-support
-  prerelease: false
   type: :runtime
+  prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - '>='
       - !ruby/object:Gem::Version
         version: '0'
 - !ruby/object:Gem::Dependency
+  name: alephant-sequencer
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - '>='
       - !ruby/object:Gem::Version
         version: '0'
-  name: alephant-sequencer
-  prerelease: false
   type: :runtime
+  prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - '>='
       - !ruby/object:Gem::Version
         version: '0'
 - !ruby/object:Gem::Dependency
+  name: alephant-cache
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - '>='
       - !ruby/object:Gem::Version
         version: '0'
-  name: alephant-cache
-  prerelease: false
   type: :runtime
+  prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - '>='
       - !ruby/object:Gem::Version
         version: '0'
 - !ruby/object:Gem::Dependency
+  name: alephant-logger
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - '>='
       - !ruby/object:Gem::Version
         version: '0'
-  name: alephant-logger
-  prerelease: false
   type: :runtime
+  prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - '>='
       - !ruby/object:Gem::Version
         version: '0'
 - !ruby/object:Gem::Dependency
+  name: alephant-lookup
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - '>='
       - !ruby/object:Gem::Version
         version: '0'
-  name: alephant-lookup
-  prerelease: false
   type: :runtime
+  prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - '>='
       - !ruby/object:Gem::Version
         version: '0'
 - !ruby/object:Gem::Dependency
+  name: alephant-renderer
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - ~>
       - !ruby/object:Gem::Version
         version: 0.1.0
-  name: alephant-renderer
-  prerelease: false
   type: :runtime
+  prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - ~>
@@ -296,7 +296,7 @@ homepage: https://github.com/BBC-News/alephant-publisher-queue
 licenses:
 - MIT
 metadata: {}
-post_install_message:
+post_install_message: 
 rdoc_options: []
 require_paths:
 - lib
@@ -311,9 +311,9 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubyforge_project:
-rubygems_version: 2.1.9
-signing_key:
+rubyforge_project: 
+rubygems_version: 2.0.14
+signing_key: 
 specification_version: 4
 summary: Static publishing to S3 based on SQS messages
 test_files: