aws_pipes 0.0.2 → 0.0.3

data/README.md

@@ -1,6 +1,8 @@
+### Communicating
+
 Send messages between Amazon EC2 instances through Unix pipes.
 
-This gem is built on top of the Amazon [Simple Queue
+Communication in aws_pipes is built on top of the Amazon [Simple Queue
 Service](http://aws.amazon.com/sqs/) (SQS) which lets you
 
 - Move data between distributed components of your application without
@@ -15,8 +17,21 @@ Text is the universal interface, and any application that can read and
 write text can use this gem – no knowledge of the Amazon API is
 required.
 
+### Logging
+
+Consolidate logs between EC2 instances. Logging in aws_pipes is built on
+top of Amazon [SimpleDB](http://aws.amazon.com/simpledb/).
+
+- Get logs off individual servers to save disk space.
+- Pool the log messages from related workers.
+- Monitor and query logs from one place.
+- Save as much log history as you want, the storage is virtually
+  unlimited.
+
 ## Usage
 
+### aws_queue
+
     # write data to an SQS queue named "foo"
     your_program | aws_queue write foo
     
@@ -26,6 +41,26 @@ required.
 To use this program you will need to [create a
 queue](https://console.aws.amazon.com/sqs/) in the Amazon Web Console.
 
+### aws_log
+
+    # write stderr to log named "bar"
+    your_program 2> >(aws_log record bar)
+    
+    # delete all messages in log named "bar"
+    aws_log delete bar
+    
+    # View log entries for "bar" within a date range
+    aws_log show bar --after "1970-01-01" --before "2020-02-02 13:42:12.123"
+
+You can combine queuing and logging in
+a single command using Bash [process
+substitution](http://www.gnu.org/software/bash/manual/bashref.html#Proce
+ss-Substitution):
+
+    # write stdout to an SQS queue named "foo"
+    # while logging stderr to a log named "bar"
+    your_program 1> >(aws_queue write foo) 2> >(aws_log record bar)
+
 ## Installation
 
 1. Sign up for an [AWS account](http://aws.amazon.com/).
@@ -34,9 +69,9 @@ queue](https://console.aws.amazon.com/sqs/) in the Amazon Web Console.
    AWS_ACCESS_KEY accordingly.
 1. Run `gem install aws_pipes` from the command line.
 
-This will install the `aws_queue` command to your path. If you haven't
-stored your Amazon credentials in environment variables, you can pass
-them in as command line options. For more info, run
+This will install the `aws_queue` and `aws_log` commands to your path.
+If you haven't stored your Amazon credentials in environment variables,
+you can pass them in as command line options. For more info, run
 
     aws_queue --help
 
@@ -52,6 +87,7 @@ queue:
 
 Then have each worker pull from the `to_be_downloaded` queue and
 repeatedly run a command to download each url. The queue supports many
-simultaneous readers and prevents duplicate work.
+simultaneous readers and prevents duplicate work. We save any errors to
+a log named "downloader" which we can monitor remotely.
 
-    aws_queue read to_be_downloaded | xargs -L1 wget
+    aws_queue read to_be_downloaded | xargs -L1 wget -nv 2> >(aws_log record downloader)

data/bin/aws_log

@@ -0,0 +1,74 @@
+#!/usr/bin/env ruby
+
+require_relative '../lib/aws_pipes'
+require 'trollop'
+require 'aws-sdk'
+require 'open-uri'
+
+opts = AwsPipes.common_aws_options(
+  "aws_log",
+  <<-EOS
+Log messages to Amazom SimpleDB with timestamp and external ip.
+
+Usage:
+       aws_log [options] record <log-name>
+       Log each line of STDIN as a message to Amazon SimpleDB
+
+       aws_log [options] show <log-name>
+       Display log
+
+       aws_log [options] delete <log-name>
+       Delete all messages in log
+EOS
+) do
+  opt :'include-ip', "Include external ip in logs", :type => :boolean, :default => true
+  opt :count, "Only show how many messages are in log", :type => :boolean, :default => false
+  opt :before, "Show messages recorded before timestamp", :type => :string
+  opt :after, "Show messages recorded after timestamp", :type => :string
+end
+
+operation = ARGV.shift
+unless %w(record show delete).include? operation
+  Trollop::die "Unknown operation \"#{operation}\" -- expecting \"record\", \"show\", or \"delete\""
+end
+
+if opts[:'include-ip']
+  match       = /([\d\.]+)/.match open('http://checkip.dyndns.org').read
+  external_ip = match[1] if match
+end
+
+sdb = AWS::SimpleDB.new(
+  :access_key_id     => AwsPipes.access_key_id(opts),
+  :secret_access_key => AwsPipes.secret_access_key(opts)
+)
+
+log_name = ARGV.shift
+Trollop::die "Missing log (SimpleDB domain) name" unless log_name
+
+messages = sdb.domains.create(log_name).items
+messages = messages.where("itemName() <= \"#{opts[:before]}\"") if opts[:before]
+messages = messages.where("itemName() >= \"#{opts[:after]}\"") if opts[:after]
+
+begin
+  case operation
+  when 'record'
+    $stdin.sync = true
+    while msg = gets
+      messages.create Time.now.strftime("%Y-%m-%d %H:%M:%S.%L"),
+        { ip: external_ip || 'none', message: msg.delete("\n") }
+    end
+  when 'show'
+    if opts[:count]
+      puts messages.count
+    else
+      printf "%-26s%-18s%s\n", 'Timestamp', 'Public IP', 'Message'
+      messages.each do |item|
+        printf "%-26s%-18s%s\n", item.name, item.attributes[:ip].values[0], item.attributes[:message].values[0]
+      end
+    end
+  when 'delete'
+    sdb.domains[log_name].delete!
+  end
+rescue Interrupt
+  exit 0
+end

data/bin/aws_queue

@@ -1,41 +1,26 @@
 #!/usr/bin/env ruby
 
+require_relative '../lib/aws_pipes'
 require 'trollop'
 require 'aws-sdk'
-require_relative '../lib/aws_pipes/version'
 
-## USAGE ###########################################################################
-
-opts = Trollop::options do
-  version "aws_queue #{AwsPipes::VERSION} (c) 2013 Joe Nelson"
-  banner <<-EOS
+opts = AwsPipes.common_aws_options(
+  "aws_queue",
+  <<-EOS
 Read and write Amazon SQS through unix pipes.
 
-You may provide Amazon authentication through these environment variables:
-AWS_ACCESS_KEY_ID - AWS access key id
-AWS_ACCESS_KEY    - AWS secret access key
-
 Usage:
        aws_queue [options] write <queue-name>
        Send STDIN to an Amazon SQS queue, one message per line.
 
        aws_queue [options] read <queue-name>
        Print messages from Amazon SQS queue to STDOUT, one message per line.
-
-where [options] are:
 EOS
+)
 
-  opt :keyid, "AWS access key id", :type => :string
-  opt :key, "AWS secret access key", :type => :string
-end
-
-## SETUP ###########################################################################
-
-access_key_id     = ENV['AWS_ACCESS_KEY_ID'] || opts[:keyid] || Trollop::die("Missing access key id")
-secret_access_key = ENV['AWS_ACCESS_KEY']    || opts[:key]   || Trollop::die("Missing secret access key")
 sqs = AWS::SQS.new(
-  :access_key_id     => access_key_id,
-  :secret_access_key => secret_access_key
+  :access_key_id     => AwsPipes.access_key_id(opts),
+  :secret_access_key => AwsPipes.secret_access_key(opts)
 )
 
 operation = ARGV.shift
@@ -46,8 +31,6 @@ end
 q_name = ARGV.shift
 Trollop::die "Missing queue name" unless q_name
 
-## OPERATION #######################################################################
-
 begin
   q = sqs.queues.named q_name
   case operation
@@ -66,4 +49,6 @@ rescue AWS::SQS::Errors::NonExistentQueue
   warn "There is no queue named \"#{q_name}\" for your AWS account."
   warn "To create this and other queues, visit https://console.aws.amazon.com/sqs/"
   exit 1
+rescue Interrupt
+  exit 0
 end

data/lib/aws_pipes.rb

@@ -1,5 +1,34 @@
 require "aws_pipes/version"
+require 'trollop'
 
 module AwsPipes
-  # Your code goes here...
+  def AwsPipes.common_aws_options program_name, custom_banner, &more_opts
+    ::Trollop::options do
+      version "#{program_name} #{VERSION} (c) 2013 Joe Nelson"
+      banner <<-EOS
+#{custom_banner}
+You may provide Amazon authentication through these environment variables:
+AWS_ACCESS_KEY_ID - AWS access key id
+AWS_ACCESS_KEY    - AWS secret access key
+
+Additional options:
+EOS
+      opt :keyid, "AWS access key id", :type => :string
+      opt :key, "AWS secret access key", :type => :string
+
+      self.instance_eval &more_opts if more_opts
+    end
+  end
+
+  def AwsPipes.access_key_id opts
+    ENV['AWS_ACCESS_KEY_ID'] ||
+      opts[:keyid] ||
+      ::Trollop::die("Missing access key id")
+  end
+
+  def AwsPipes.secret_access_key opts
+    ENV['AWS_ACCESS_KEY'] ||
+      opts[:key] ||
+      ::Trollop::die("Missing secret access key")
+  end
 end

data/lib/aws_pipes/version.rb

@@ -1,3 +1,3 @@
 module AwsPipes
-  VERSION = "0.0.2"
+  VERSION = "0.0.3"
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: aws_pipes
 version: !ruby/object:Gem::Version
-  version: 0.0.2
+  version: 0.0.3
   prerelease: 
 platform: ruby
 authors:
@@ -9,7 +9,7 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2012-12-23 00:00:00.000000000 Z
+date: 2012-12-24 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: aws-sdk
@@ -47,6 +47,7 @@ description: Send messages between Amazon EC2 instances through Unix pipes.
 email:
 - cred+github@begriffs.com
 executables:
+- aws_log
 - aws_queue
 extensions: []
 extra_rdoc_files: []
@@ -57,6 +58,7 @@ files:
 - README.md
 - Rakefile
 - aws_pipes.gemspec
+- bin/aws_log
 - bin/aws_queue
 - lib/aws_pipes.rb
 - lib/aws_pipes/version.rb