aws_pipes 0.0.5 → 0.0.6

data/README.md

@@ -1,6 +1,6 @@
 ## Overview
 
-### Communicating
+### Communication
 
 Send messages between Amazon EC2 instances through Unix pipes.
 
@@ -30,6 +30,18 @@ top of Amazon [SimpleDB](http://aws.amazon.com/simpledb/).
 - Save as much log history as you want, the storage is virtually
   unlimited.
 
+### Saving Datasets
+
+Save data across EC2 instances with scalable throughput.
+Data archival in aws_pipes is built on top of Amazon
+[DynamoDB](http://aws.amazon.com/dynamodb/).
+
+- Store data centrally.
+- Automatically scale throughput and space.
+- Query results (albeit not relationally).
+- Monitor data acquisition through web control panel.
+- Can export to S3.
+
 ## Usage
 
 ### aws_queue
@@ -72,6 +84,11 @@ ss-Substitution):
     # filling in columns a, b, and c
     your_program | aws_db foo a b c
 
+DynamoDB tables have adjustable read- and write-throughput settings to
+scale as needed. The `aws_db` command will automatically re-provision
+write throughput if writing starts getting throttled. This makes
+`aws_db` (when run in parallel) a way to save virtually unlimited
+amounts of data as quickly as necessary.
 
 ## Installation
 

data/bin/aws_db

@@ -32,8 +32,14 @@ db = AWS::DynamoDB.new(
 )
 
 $stdin.sync = true
-table = db.tables[table_name]
-table.load_schema
+begin
+  table = db.tables[table_name]
+  table.load_schema
+rescue AWS::DynamoDB::Errors::ResourceNotFoundException
+  warn "There is no DynamoDB table named \"#{table_name}\" for your AWS account."
+  warn "To create this and other tables, visit https://console.aws.amazon.com/dynamodb/"
+  exit 1
+end
 
 while row = $stdin.gets
   begin
@@ -51,5 +57,8 @@ while row = $stdin.gets
     retry
   rescue Interrupt
     exit 0
+  rescue Timeout::Error
+    warn "aws_db retrying after Timeout::Error"
+    retry
   end
 end

data/bin/aws_log

@@ -71,4 +71,7 @@ begin
   end
 rescue Interrupt
   exit 0
+rescue Timeout::Error
+  warn "aws_log retrying after Timeout::Error"
+  retry
 end

data/bin/aws_queue

@@ -51,4 +51,7 @@ rescue AWS::SQS::Errors::NonExistentQueue
   exit 1
 rescue Interrupt
   exit 0
+rescue Timeout::Error
+  warn "aws_queue retrying after Timeout::Error"
+  retry
 end

data/lib/aws_pipes/version.rb

@@ -1,3 +1,3 @@
 module AwsPipes
-  VERSION = "0.0.5"
+  VERSION = "0.0.6"
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: aws_pipes
 version: !ruby/object:Gem::Version
-  version: 0.0.5
+  version: 0.0.6
   prerelease: 
 platform: ruby
 authors:
@@ -9,7 +9,7 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2012-12-28 00:00:00.000000000 Z
+date: 2013-01-02 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: aws-sdk