aws_pipes 0.0.5 → 0.0.6
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- data/README.md +18 -1
- data/bin/aws_db +11 -2
- data/bin/aws_log +3 -0
- data/bin/aws_queue +3 -0
- data/lib/aws_pipes/version.rb +1 -1
- metadata +2 -2
data/README.md
CHANGED
@@ -1,6 +1,6 @@
|
|
1
1
|
## Overview
|
2
2
|
|
3
|
-
###
|
3
|
+
### Communication
|
4
4
|
|
5
5
|
Send messages between Amazon EC2 instances through Unix pipes.
|
6
6
|
|
@@ -30,6 +30,18 @@ top of Amazon [SimpleDB](http://aws.amazon.com/simpledb/).
|
|
30
30
|
- Save as much log history as you want, the storage is virtually
|
31
31
|
unlimited.
|
32
32
|
|
33
|
+
### Saving Datasets
|
34
|
+
|
35
|
+
Save data across EC2 instances with scalable throughput.
|
36
|
+
Data archival in aws_pipes is built on top of Amazon
|
37
|
+
[DynamoDB](http://aws.amazon.com/dynamodb/).
|
38
|
+
|
39
|
+
- Store data centrally.
|
40
|
+
- Automatically scale throughput and space.
|
41
|
+
- Query results (albeit not relationally).
|
42
|
+
- Monitor data acquisition through web control panel.
|
43
|
+
- Can export to S3.
|
44
|
+
|
33
45
|
## Usage
|
34
46
|
|
35
47
|
### aws_queue
|
@@ -72,6 +84,11 @@ ss-Substitution):
|
|
72
84
|
# filling in columns a, b, and c
|
73
85
|
your_program | aws_db foo a b c
|
74
86
|
|
87
|
+
DynamoDB tables have adjustable read- and write-throughput settings to
|
88
|
+
scale as needed. The `aws_db` command will automatically re-provision
|
89
|
+
write throughput if writing starts getting throttled. This makes
|
90
|
+
`aws_db` (when run in parallel) a way to save virtually unlimited
|
91
|
+
amounts of data as quickly as necessary.
|
75
92
|
|
76
93
|
## Installation
|
77
94
|
|
data/bin/aws_db
CHANGED
@@ -32,8 +32,14 @@ db = AWS::DynamoDB.new(
|
|
32
32
|
)
|
33
33
|
|
34
34
|
$stdin.sync = true
|
35
|
-
|
36
|
-
table.
|
35
|
+
begin
|
36
|
+
table = db.tables[table_name]
|
37
|
+
table.load_schema
|
38
|
+
rescue AWS::DynamoDB::Errors::ResourceNotFoundException
|
39
|
+
warn "There is no DynamoDB table named \"#{table_name}\" for your AWS account."
|
40
|
+
warn "To create this and other tables, visit https://console.aws.amazon.com/dynamodb/"
|
41
|
+
exit 1
|
42
|
+
end
|
37
43
|
|
38
44
|
while row = $stdin.gets
|
39
45
|
begin
|
@@ -51,5 +57,8 @@ while row = $stdin.gets
|
|
51
57
|
retry
|
52
58
|
rescue Interrupt
|
53
59
|
exit 0
|
60
|
+
rescue Timeout::Error
|
61
|
+
warn "aws_db retrying after Timeout::Error"
|
62
|
+
retry
|
54
63
|
end
|
55
64
|
end
|
data/bin/aws_log
CHANGED
data/bin/aws_queue
CHANGED
data/lib/aws_pipes/version.rb
CHANGED
metadata
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: aws_pipes
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.0.
|
4
|
+
version: 0.0.6
|
5
5
|
prerelease:
|
6
6
|
platform: ruby
|
7
7
|
authors:
|
@@ -9,7 +9,7 @@ authors:
|
|
9
9
|
autorequire:
|
10
10
|
bindir: bin
|
11
11
|
cert_chain: []
|
12
|
-
date:
|
12
|
+
date: 2013-01-02 00:00:00.000000000 Z
|
13
13
|
dependencies:
|
14
14
|
- !ruby/object:Gem::Dependency
|
15
15
|
name: aws-sdk
|