ocean-dynamo 0.7.5 → 1.0.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.rdoc +31 -30
- data/lib/ocean-dynamo/associations/belongs_to.rb +2 -2
- data/lib/ocean-dynamo/associations/has_many.rb +23 -22
- data/lib/ocean-dynamo/attributes.rb +0 -1
- data/lib/ocean-dynamo/class_variables.rb +3 -3
- data/lib/ocean-dynamo/persistence.rb +108 -55
- data/lib/ocean-dynamo/queries.rb +36 -23
- data/lib/ocean-dynamo/tables.rb +75 -42
- data/lib/ocean-dynamo/version.rb +1 -1
- metadata +18 -18
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA1:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: e59574397e111dc399d72b632fcd21fd73e9f114
|
4
|
+
data.tar.gz: b93297d296545f8a4f1626eaf8d6bd6936aa5be5
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 61f93300975ddda2911f33da30761e9773d7a4c488148eb690618f1ec1286869e08ad1c50b25e0ca93ba6b14ddc4dc0989959c660b8d11b282eafaea97c00e03
|
7
|
+
data.tar.gz: b644c1ce1bf53bc1b55d67adcc7905f077502fddf8a6ad3770470a84d6800665c9c4cbdfaf9e73abae3a1cd9c6119e616efd8e9f85edf9cc45e4b2691f2daeab
|
data/README.rdoc
CHANGED
@@ -73,8 +73,8 @@ value will return the empty string, <tt>""</tt>.
|
|
73
73
|
+dynamo_schema+ takes args and many options. Here's the full syntax:
|
74
74
|
|
75
75
|
dynamo_schema(
|
76
|
-
table_hash_key = :id,
|
77
|
-
table_range_key = nil,
|
76
|
+
table_hash_key: = :id, # The name of the hash key attribute
|
77
|
+
table_range_key: = nil, # The name of the range key attribute (or nil)
|
78
78
|
table_name: compute_table_name, # The basename of the DynamoDB table
|
79
79
|
table_name_prefix: nil, # A basename prefix string or nil
|
80
80
|
table_name_suffix: nil, # A basename suffix string or nil
|
@@ -214,9 +214,6 @@ to both the following locations in your project:
|
|
214
214
|
Enter your AWS credentials in the latter file. Eventually, there
|
215
215
|
will be a generator to copy these files for you, but for now you need to do it manually.
|
216
216
|
|
217
|
-
You also need +fake_dynamo+ to run DynamoDB locally: see below for installation instructions.
|
218
|
-
NB: You do not need an Amazon AWS account to run OceanDynamo locally.
|
219
|
-
|
220
217
|
|
221
218
|
== Documentation
|
222
219
|
|
@@ -228,6 +225,8 @@ See also Ocean, a Rails framework for creating highly scalable SOAs in the cloud
|
|
228
225
|
OceanDynamo is used as a central component:
|
229
226
|
* http://wiki.oceanframework.net
|
230
227
|
|
228
|
+
* AWS Ruby SDK v2 for DynamoDB: http://docs.aws.amazon.com/sdkforruby/api/Aws/DynamoDB.html
|
229
|
+
|
231
230
|
|
232
231
|
== Contributing
|
233
232
|
|
@@ -239,46 +238,48 @@ All contributed code must therefore also be exhaustively tested.
|
|
239
238
|
|
240
239
|
== Running the specs
|
241
240
|
|
242
|
-
To run the specs for the OceanDynamo gem, you must first install
|
243
|
-
|
244
|
-
|
241
|
+
To run the specs for the OceanDynamo gem, you must first install DynamoDB Local. It's a local,
|
242
|
+
functional clone of Amazon DynamoDB. We use DynamoDB Local during development and testing.
|
243
|
+
|
244
|
+
Starting with +ocean-dynamo+ 0.8.0, we're using v2 of the AWS SDK Ruby gem and the latest version
|
245
|
+
of the DynamoDB API (2012-08-10). This means that we now have access to secondary indices,
|
246
|
+
amongst other things.
|
245
247
|
|
246
|
-
|
248
|
+
Download DynamoDB Local from the following location: http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Tools.DynamoDBLocal.html
|
249
|
+
|
250
|
+
Next, copy the AWS configuration file from the template:
|
247
251
|
|
248
252
|
cp spec/dummy/config/aws.yml.example spec/dummy/config/aws.yml
|
249
253
|
|
250
254
|
NB: +aws.yml+ is excluded from source control. This allows you to enter your AWS credentials
|
251
255
|
safely. Note that +aws.yml.example+ is under source control: don't edit it.
|
252
256
|
|
253
|
-
|
254
|
-
of the DynamoDB API. We're not using the +2012-08-10+ version, as the +aws-sdk+ ruby gem
|
255
|
-
doesn't fully support it.
|
256
|
-
|
257
|
-
Next, start +fake_dynamo+:
|
257
|
+
You're now ready to start DynamoDB Local:
|
258
258
|
|
259
|
-
|
259
|
+
java -Djava.library.path=./DynamoDBLocal_lib -jar DynamoDBLocal.jar -sharedDb
|
260
260
|
|
261
|
-
|
262
|
-
is writable:
|
261
|
+
With DynamoDB Local running, you should now be able to do
|
263
262
|
|
264
|
-
|
265
|
-
sudo chown peterb:staff /usr/local/var/fake_dynamo
|
263
|
+
rspec
|
266
264
|
|
267
|
-
|
265
|
+
All tests should pass.
|
268
266
|
|
269
|
-
curl -X DELETE http://localhost:4567
|
270
267
|
|
271
|
-
|
272
|
-
+fake_dynamo+; we're just using it here as a test that the installation works. It will
|
273
|
-
be issued automatically as part of the test suite, so don't expect test data to survive
|
274
|
-
between runs.
|
268
|
+
=== Resetting the DynamoDB Local database
|
275
269
|
|
276
|
-
|
270
|
+
You might want to add the following to your spec_helper.rb file, inside the +RSpec.configure+
|
271
|
+
block:
|
277
272
|
|
278
|
-
|
279
|
-
|
280
|
-
|
273
|
+
# To clear the DB before and/or after each run, uncomment as desired:
|
274
|
+
config.before(:suite) { c = Aws::DynamoDB::Client.new
|
275
|
+
c.list_tables.table_names.each { |t| c.delete_table({table_name: t}) }
|
276
|
+
}
|
277
|
+
# config.after(:suite) { c = Aws::DynamoDB::Client.new
|
278
|
+
# c.list_tables.table_names.each { |t| c.delete_table({table_name: t}) }
|
279
|
+
# }
|
281
280
|
|
281
|
+
Just make sure you don't run the above code when running your tests against the live DynamoDB
|
282
|
+
service on AWS, as it will erase all your DynamoDB tables.
|
282
283
|
|
283
284
|
== Rails console
|
284
285
|
|
@@ -288,7 +289,7 @@ The Rails console is available from the built-in dummy application:
|
|
288
289
|
rails console
|
289
290
|
|
290
291
|
This will, amongst other things, also create the CloudModel table if it doesn't already
|
291
|
-
exist. On Amazon, this will take a little while. With
|
292
|
+
exist. On Amazon, this will take a little while. With DynamoDB Local, it's practically
|
292
293
|
instant.
|
293
294
|
|
294
295
|
When you leave the console, you must navigate back to the top directory (<tt>cd ../..</tt>)
|
@@ -26,7 +26,7 @@ module OceanDynamo
|
|
26
26
|
# end
|
27
27
|
#
|
28
28
|
# class Topic < OceanDynamo::Table
|
29
|
-
# dynamo_schema(:
|
29
|
+
# dynamo_schema(:guid) do
|
30
30
|
# attribute :title
|
31
31
|
# end
|
32
32
|
# belongs_to :forum
|
@@ -34,7 +34,7 @@ module OceanDynamo
|
|
34
34
|
# end
|
35
35
|
#
|
36
36
|
# class Post < OceanDynamo::Table
|
37
|
-
# dynamo_schema(:
|
37
|
+
# dynamo_schema(:guid) do
|
38
38
|
# attribute :body
|
39
39
|
# end
|
40
40
|
# belongs_to :topic, composite_key: true
|
@@ -109,6 +109,13 @@ module OceanDynamo
|
|
109
109
|
|
110
110
|
protected
|
111
111
|
|
112
|
+
|
113
|
+
def condition_options(child_class)
|
114
|
+
{ key_condition_expression: "#{child_class.table_hash_key} = :hashval AND #{child_class.table_range_key} >= :rangeval",
|
115
|
+
expression_attribute_values: { ":hashval" => id, ":rangeval" => "0" }
|
116
|
+
}
|
117
|
+
end
|
118
|
+
|
112
119
|
#
|
113
120
|
# Reads all children of a has_many relation.
|
114
121
|
#
|
@@ -118,10 +125,8 @@ module OceanDynamo
|
|
118
125
|
else
|
119
126
|
result = Array.new
|
120
127
|
_late_connect?
|
121
|
-
|
122
|
-
|
123
|
-
batch_size: 1000, select: :all) do |item_data|
|
124
|
-
result << child_class.new._setup_from_dynamo(item_data)
|
128
|
+
child_class.in_batches :query, condition_options(child_class) do |attrs|
|
129
|
+
result << child_class.new._setup_from_dynamo(attrs)
|
125
130
|
end
|
126
131
|
result
|
127
132
|
end
|
@@ -146,11 +151,8 @@ module OceanDynamo
|
|
146
151
|
#
|
147
152
|
def map_children(child_class)
|
148
153
|
return if new_record?
|
149
|
-
|
150
|
-
|
151
|
-
child_items.query(hash_value: id, range_gte: "0",
|
152
|
-
batch_size: 1000, select: :all) do |item_data|
|
153
|
-
yield child_class.new._setup_from_dynamo(item_data)
|
154
|
+
child_class.in_batches :query, condition_options(child_class) do |attrs|
|
155
|
+
yield child_class.new._setup_from_dynamo(attrs)
|
154
156
|
end
|
155
157
|
end
|
156
158
|
|
@@ -160,11 +162,9 @@ module OceanDynamo
|
|
160
162
|
#
|
161
163
|
def delete_children(child_class)
|
162
164
|
return if new_record?
|
163
|
-
|
164
|
-
|
165
|
-
|
166
|
-
batch_size: 1000) do |item|
|
167
|
-
item.delete
|
165
|
+
child_class.in_batches :query, condition_options(child_class) do |attrs|
|
166
|
+
child_class.delete attrs[child_class.table_hash_key.to_s],
|
167
|
+
attrs[child_class.table_range_key.to_s]
|
168
168
|
end
|
169
169
|
end
|
170
170
|
|
@@ -176,14 +176,15 @@ module OceanDynamo
|
|
176
176
|
#
|
177
177
|
def nullify_children(child_class)
|
178
178
|
return if new_record?
|
179
|
-
|
180
|
-
|
181
|
-
|
182
|
-
|
183
|
-
|
184
|
-
|
185
|
-
|
186
|
-
|
179
|
+
opts = condition_options(child_class)
|
180
|
+
child_class.in_batches :query, opts do |attrs|
|
181
|
+
child_hash_key = child_class.table_hash_key.to_s
|
182
|
+
child_range_key = child_class.table_range_key && child_class.table_range_key.to_s
|
183
|
+
# There is no way in the DynamoDB API to update a key attribute. Delete the child item.
|
184
|
+
child_class.delete attrs[child_hash_key], attrs[child_range_key]
|
185
|
+
# Create a new one with NULL for key
|
186
|
+
attrs[child_hash_key] = "NULL"
|
187
|
+
child_class.dynamo_table.put_item(item: attrs)
|
187
188
|
end
|
188
189
|
end
|
189
190
|
|
@@ -1,15 +1,15 @@
|
|
1
1
|
module OceanDynamo
|
2
2
|
class Table
|
3
3
|
|
4
|
+
class_attribute :dynamo_resource, instance_writer: false
|
5
|
+
self.dynamo_resource = nil
|
6
|
+
|
4
7
|
class_attribute :dynamo_client, instance_writer: false
|
5
8
|
self.dynamo_client = nil
|
6
9
|
|
7
10
|
class_attribute :dynamo_table, instance_writer: false
|
8
11
|
self.dynamo_table = nil
|
9
12
|
|
10
|
-
class_attribute :dynamo_items, instance_writer: false
|
11
|
-
self.dynamo_items = nil
|
12
|
-
|
13
13
|
class_attribute :table_name, instance_writer: false
|
14
14
|
self.table_name = nil
|
15
15
|
|
@@ -30,27 +30,46 @@ module OceanDynamo
|
|
30
30
|
end
|
31
31
|
|
32
32
|
|
33
|
+
#
|
34
|
+
# Class method to delete a record. Returns true if the record existed,
|
35
|
+
# false if it didn't.
|
36
|
+
#
|
33
37
|
def delete(hash, range=nil)
|
34
|
-
|
35
|
-
|
36
|
-
|
37
|
-
|
38
|
+
_late_connect?
|
39
|
+
keys = { table_hash_key.to_s => hash }
|
40
|
+
keys[table_range_key] = range if table_range_key && range
|
41
|
+
options = { key: keys,
|
42
|
+
return_values: "ALL_OLD"
|
43
|
+
}
|
44
|
+
dynamo_table.delete_item(options).attributes ? true : false
|
38
45
|
end
|
39
46
|
|
40
47
|
|
48
|
+
#
|
49
|
+
# Deletes all records without instantiating them first.
|
50
|
+
#
|
41
51
|
def delete_all
|
42
|
-
|
43
|
-
|
44
|
-
|
52
|
+
options = {
|
53
|
+
consistent_read: true,
|
54
|
+
projection_expression: table_hash_key.to_s + (table_range_key ? ", " + table_range_key.to_s : "")
|
55
|
+
}
|
56
|
+
in_batches :scan, options do |attrs|
|
57
|
+
if table_range_key
|
58
|
+
delete attrs[table_hash_key.to_s], attrs[table_range_key.to_s]
|
59
|
+
else
|
60
|
+
delete attrs[table_hash_key.to_s]
|
61
|
+
end
|
45
62
|
end
|
46
63
|
nil
|
47
64
|
end
|
48
65
|
|
49
66
|
|
67
|
+
#
|
68
|
+
# Destroys all records after first instantiating them.
|
69
|
+
#
|
50
70
|
def destroy_all
|
51
|
-
|
52
|
-
|
53
|
-
new._setup_from_dynamo(item_data).destroy
|
71
|
+
in_batches :scan, { consistent_read: true } do |attrs|
|
72
|
+
new._setup_from_dynamo(attrs).destroy
|
54
73
|
end
|
55
74
|
nil
|
56
75
|
end
|
@@ -217,12 +236,23 @@ module OceanDynamo
|
|
217
236
|
_late_connect?
|
218
237
|
run_callbacks :touch do
|
219
238
|
begin
|
220
|
-
|
221
|
-
|
222
|
-
|
223
|
-
|
239
|
+
timestamps = set_timestamps(name)
|
240
|
+
update_expression = []
|
241
|
+
expression_attribute_values = {}
|
242
|
+
timestamps.each_with_index do |ts, i|
|
243
|
+
nomen = ":ts#{i}"
|
244
|
+
expression_attribute_values[nomen] = serialize_attribute(ts, read_attribute(ts))
|
245
|
+
update_expression << "#{ts} = #{nomen}"
|
224
246
|
end
|
225
|
-
|
247
|
+
update_expression = "SET " + update_expression.join(", ")
|
248
|
+
options = {
|
249
|
+
key: serialized_key_attributes,
|
250
|
+
update_expression: update_expression
|
251
|
+
}.merge(_handle_locking)
|
252
|
+
options[:expression_attribute_values] = (options[:expression_attribute_values] || {}).merge(expression_attribute_values)
|
253
|
+
dynamo_table.update_item(options)
|
254
|
+
rescue Aws::DynamoDB::Errors::ConditionalCheckFailedException
|
255
|
+
write_attribute(lock_attribute, read_attribute(lock_attribute)-1) unless frozen?
|
226
256
|
raise OceanDynamo::StaleObjectError.new(self)
|
227
257
|
end
|
228
258
|
self
|
@@ -231,36 +261,20 @@ module OceanDynamo
|
|
231
261
|
|
232
262
|
|
233
263
|
#
|
234
|
-
#
|
235
|
-
#
|
236
|
-
#
|
237
|
-
#
|
238
|
-
#
|
239
|
-
|
240
|
-
#
|
241
|
-
# The :consistent keyword may only be used when the arg is an Item.
|
242
|
-
#
|
243
|
-
def _setup_from_dynamo(arg, consistent: false)
|
264
|
+
# Deserialises and assigns all defined attributes. Skips undeclared attributes.
|
265
|
+
# Unlike its predecessor, this version never reads anything from DynamoDB,
|
266
|
+
# it just processes the results from such reads. Thus, the implementation of
|
267
|
+
# +consistent+ reads is up to the caller of this method.
|
268
|
+
#
|
269
|
+
def _setup_from_dynamo(arg)
|
244
270
|
case arg
|
245
|
-
when
|
246
|
-
|
247
|
-
|
248
|
-
|
249
|
-
item = arg.item
|
250
|
-
item_data = arg
|
251
|
-
raise ArgumentError, ":consistent may not be specified when passing an ItemData" if consistent
|
252
|
-
else
|
253
|
-
raise ArgumentError, "arg must be an AWS::DynamoDB::Item or an AWS::DynamoDB::ItemData"
|
254
|
-
end
|
255
|
-
|
256
|
-
@dynamo_item = item
|
257
|
-
|
258
|
-
if !item_data
|
259
|
-
raw_attrs = item.attributes.to_hash(consistent_read: consistent)
|
271
|
+
when Aws::DynamoDB::Types::GetItemOutput
|
272
|
+
raw_attrs = arg.item
|
273
|
+
when Hash
|
274
|
+
raw_attrs = arg
|
260
275
|
else
|
261
|
-
|
276
|
+
raise ArgumentError, "arg must be an Aws::DynamoDB::Types::GetItemOutput or a Hash (was #{arg.class})"
|
262
277
|
end
|
263
|
-
|
264
278
|
dynamo_deserialize_attributes(raw_attrs)
|
265
279
|
@new_record = false
|
266
280
|
self
|
@@ -278,9 +292,13 @@ module OceanDynamo
|
|
278
292
|
def dynamo_persist(lock: nil) # :nodoc:
|
279
293
|
_late_connect?
|
280
294
|
begin
|
281
|
-
options = _handle_locking(lock)
|
282
|
-
|
283
|
-
|
295
|
+
options = _handle_locking(lock) # This might increment an attr...
|
296
|
+
options = options.merge(item: serialized_attributes) # ... which we serialise here.
|
297
|
+
dynamo_table.put_item(options)
|
298
|
+
rescue Aws::DynamoDB::Errors::ConditionalCheckFailedException
|
299
|
+
if lock
|
300
|
+
write_attribute(lock, read_attribute(lock)-1) unless frozen?
|
301
|
+
end
|
284
302
|
raise OceanDynamo::StaleObjectError.new(self)
|
285
303
|
end
|
286
304
|
@new_record = false
|
@@ -291,9 +309,12 @@ module OceanDynamo
|
|
291
309
|
def dynamo_delete(lock: nil) # :nodoc:
|
292
310
|
_late_connect?
|
293
311
|
begin
|
294
|
-
options = _handle_locking(lock)
|
295
|
-
|
296
|
-
rescue
|
312
|
+
options = { key: serialized_key_attributes }.merge(_handle_locking(lock))
|
313
|
+
dynamo_table.delete_item(options)
|
314
|
+
rescue Aws::DynamoDB::Errors::ConditionalCheckFailedException
|
315
|
+
if lock
|
316
|
+
write_attribute(lock, read_attribute(lock)-1) unless frozen?
|
317
|
+
end
|
297
318
|
raise OceanDynamo::StaleObjectError.new(self)
|
298
319
|
end
|
299
320
|
end
|
@@ -309,18 +330,28 @@ module OceanDynamo
|
|
309
330
|
end
|
310
331
|
|
311
332
|
|
312
|
-
def
|
333
|
+
def serialized_key_attributes
|
313
334
|
result = Hash.new
|
314
|
-
|
315
|
-
|
316
|
-
|
335
|
+
# First the hash key
|
336
|
+
attribute = table_hash_key
|
337
|
+
metadata = fields[attribute]
|
338
|
+
serialized = serialize_attribute(attribute, read_attribute(attribute), metadata)
|
339
|
+
raise "Hash key may not be null" if serialized == nil
|
340
|
+
result[attribute] = serialized
|
341
|
+
# Then the range key, if any
|
342
|
+
if table_range_key
|
343
|
+
attribute = table_range_key
|
344
|
+
metadata = fields[attribute]
|
345
|
+
serialized = serialize_attribute(attribute, read_attribute(attribute), metadata)
|
346
|
+
raise "Range key may not be null" if serialized == nil
|
347
|
+
result[attribute] = serialized
|
317
348
|
end
|
318
|
-
|
349
|
+
result
|
319
350
|
end
|
320
351
|
|
321
352
|
|
322
353
|
def serialize_attribute(attribute, value, metadata=fields[attribute],
|
323
|
-
target_class: metadata['target_class'],
|
354
|
+
target_class: metadata['target_class'], # Remove?
|
324
355
|
type: metadata['type'])
|
325
356
|
return nil if value == nil
|
326
357
|
case type
|
@@ -345,6 +376,16 @@ module OceanDynamo
|
|
345
376
|
end
|
346
377
|
|
347
378
|
|
379
|
+
def dynamo_deserialize_attributes(hash) # :nodoc:
|
380
|
+
result = Hash.new
|
381
|
+
fields.each do |attribute, metadata|
|
382
|
+
next if metadata['no_save']
|
383
|
+
result[attribute] = deserialize_attribute(hash[attribute], metadata)
|
384
|
+
end
|
385
|
+
assign_attributes(result)
|
386
|
+
end
|
387
|
+
|
388
|
+
|
348
389
|
def deserialize_attribute(value, metadata, type: metadata[:type])
|
349
390
|
case type
|
350
391
|
when :reference
|
@@ -402,12 +443,24 @@ module OceanDynamo
|
|
402
443
|
end
|
403
444
|
|
404
445
|
|
446
|
+
#
|
447
|
+
# Returns a hash with a condition expression which has to be satisfied
|
448
|
+
# for the write or delete operation to succeed.
|
449
|
+
# Note that this method will increment the lock attribute. This means
|
450
|
+
# two things:
|
451
|
+
# 1. Collect the instance attributes after this method has been called.
|
452
|
+
# 2. Remember that care must be taken to decrement the lock attribute in
|
453
|
+
# case the subsequent write/delete operation fails or throws an
|
454
|
+
# exception, such as +StaleObjectError+.
|
455
|
+
#
|
405
456
|
def _handle_locking(lock=lock_attribute) # :nodoc:
|
406
457
|
_late_connect?
|
407
458
|
if lock
|
408
459
|
current_v = read_attribute(lock)
|
409
460
|
write_attribute(lock, current_v+1) unless frozen?
|
410
|
-
{
|
461
|
+
{ condition_expression: "#{lock} = :cv",
|
462
|
+
expression_attribute_values: { ":cv" => current_v }
|
463
|
+
}
|
411
464
|
else
|
412
465
|
{}
|
413
466
|
end
|
data/lib/ocean-dynamo/queries.rb
CHANGED
@@ -12,11 +12,14 @@ module OceanDynamo
|
|
12
12
|
_late_connect?
|
13
13
|
hash = hash.id if hash.kind_of?(Table) # TODO: We have (innocuous) leakage, fix!
|
14
14
|
range = range.to_i if range.is_a?(Time)
|
15
|
-
|
16
|
-
|
15
|
+
keys = { table_hash_key.to_s => hash }
|
16
|
+
keys[table_range_key] = range if table_range_key && range
|
17
|
+
options = { key: keys, consistent_read: consistent }
|
18
|
+
item = dynamo_table.get_item(options).item
|
19
|
+
unless item
|
17
20
|
raise RecordNotFound, "can't find a #{self} with primary key ['#{hash}', #{range.inspect}]"
|
18
21
|
end
|
19
|
-
new._setup_from_dynamo(item
|
22
|
+
new._setup_from_dynamo(item)
|
20
23
|
end
|
21
24
|
|
22
25
|
|
@@ -29,11 +32,12 @@ module OceanDynamo
|
|
29
32
|
|
30
33
|
|
31
34
|
#
|
32
|
-
# The number of records in the table.
|
35
|
+
# The number of records in the table. Updated every 6 hours or so;
|
36
|
+
# thus isn't a reliable real-time measure of the number of table items.
|
33
37
|
#
|
34
38
|
def count(**options)
|
35
39
|
_late_connect?
|
36
|
-
|
40
|
+
dynamo_table.item_count
|
37
41
|
end
|
38
42
|
|
39
43
|
|
@@ -42,17 +46,28 @@ module OceanDynamo
|
|
42
46
|
#
|
43
47
|
def all(consistent: false, **options)
|
44
48
|
_late_connect?
|
45
|
-
|
46
|
-
|
47
|
-
|
48
|
-
|
49
|
-
|
50
|
-
|
51
|
-
|
52
|
-
|
49
|
+
records = []
|
50
|
+
in_batches :scan, { consistent_read: !!consistent } do |attrs|
|
51
|
+
records << new._setup_from_dynamo(attrs)
|
52
|
+
end
|
53
|
+
records
|
54
|
+
end
|
55
|
+
|
56
|
+
|
57
|
+
#
|
58
|
+
# This method takes a block and yields it to every record in a table.
|
59
|
+
# +message+ must be either :scan or :query.
|
60
|
+
# +options+ is the hash of options to
|
61
|
+
#
|
62
|
+
def in_batches(message, options, &block)
|
63
|
+
loop do
|
64
|
+
result = dynamo_table.send message, options
|
65
|
+
result.items.each do |hash|
|
66
|
+
yield hash
|
53
67
|
end
|
68
|
+
return true unless result.last_evaluated_key
|
69
|
+
options[:exclusive_start_key] = result.last_evaluated_key
|
54
70
|
end
|
55
|
-
result
|
56
71
|
end
|
57
72
|
|
58
73
|
|
@@ -64,19 +79,17 @@ module OceanDynamo
|
|
64
79
|
# thereby greatly reducing memory consumption.
|
65
80
|
#
|
66
81
|
def find_each(limit: nil, batch_size: 1000, consistent: false)
|
67
|
-
|
68
|
-
|
69
|
-
|
70
|
-
|
71
|
-
|
72
|
-
|
73
|
-
yield new._setup_from_dynamo(item_data)
|
82
|
+
options = { consistent_read: consistent }
|
83
|
+
options[:limit] = batch_size if batch_size
|
84
|
+
in_batches :scan, options do |attrs|
|
85
|
+
if limit
|
86
|
+
return true if limit <= 0
|
87
|
+
limit = limit - 1
|
74
88
|
end
|
89
|
+
yield new._setup_from_dynamo(attrs)
|
75
90
|
end
|
76
|
-
true
|
77
91
|
end
|
78
92
|
|
79
|
-
|
80
93
|
# #
|
81
94
|
# # Yields each batch of records that was found by the find options as an array. The size of
|
82
95
|
# # each batch is set by the :batch_size option; the default is 1000.
|
data/lib/ocean-dynamo/tables.rb
CHANGED
@@ -26,8 +26,8 @@ module OceanDynamo
|
|
26
26
|
**keywords,
|
27
27
|
&block)
|
28
28
|
self.dynamo_client = nil
|
29
|
+
self.dynamo_resource = nil
|
29
30
|
self.dynamo_table = nil
|
30
|
-
self.dynamo_items = nil
|
31
31
|
self.table_connected = false
|
32
32
|
self.table_connect_policy = connect
|
33
33
|
self.table_create_policy = create
|
@@ -45,85 +45,119 @@ module OceanDynamo
|
|
45
45
|
|
46
46
|
def establish_db_connection
|
47
47
|
setup_dynamo
|
48
|
-
if dynamo_table
|
48
|
+
if table_exists?(dynamo_table)
|
49
49
|
wait_until_table_is_active
|
50
50
|
self.table_connected = true
|
51
|
+
update_table_if_required
|
51
52
|
else
|
52
53
|
raise(TableNotFound, table_full_name) unless table_create_policy
|
53
54
|
create_table
|
54
55
|
end
|
55
|
-
set_dynamo_table_keys
|
56
56
|
end
|
57
57
|
|
58
58
|
|
59
59
|
def setup_dynamo
|
60
|
-
self.dynamo_client ||=
|
61
|
-
self.
|
62
|
-
self.
|
60
|
+
self.dynamo_client ||= Aws::DynamoDB::Client.new
|
61
|
+
self.dynamo_resource ||= Aws::DynamoDB::Resource.new(client: dynamo_client)
|
62
|
+
self.dynamo_table = dynamo_resource.table(table_full_name)
|
63
|
+
end
|
64
|
+
|
65
|
+
|
66
|
+
def table_exists?(table)
|
67
|
+
return true if table.data_loaded?
|
68
|
+
begin
|
69
|
+
table.load
|
70
|
+
rescue Aws::DynamoDB::Errors::ResourceNotFoundException
|
71
|
+
return false
|
72
|
+
end
|
73
|
+
true
|
63
74
|
end
|
64
75
|
|
65
76
|
|
66
77
|
def wait_until_table_is_active
|
67
78
|
loop do
|
68
|
-
case dynamo_table.
|
69
|
-
when
|
70
|
-
|
79
|
+
case dynamo_table.table_status
|
80
|
+
when "ACTIVE"
|
81
|
+
update_table_if_required
|
71
82
|
return
|
72
|
-
when
|
83
|
+
when "UPDATING", "CREATING"
|
73
84
|
sleep 1
|
74
85
|
next
|
75
|
-
when
|
76
|
-
sleep 1 while dynamo_table
|
86
|
+
when "DELETING"
|
87
|
+
sleep 1 while table_exists?(dynamo_table)
|
77
88
|
create_table
|
78
89
|
return
|
79
90
|
else
|
80
|
-
raise UnknownTableStatus.new("Unknown DynamoDB table status '#{dynamo_table.
|
91
|
+
raise UnknownTableStatus.new("Unknown DynamoDB table status '#{dynamo_table.table_status}'")
|
81
92
|
end
|
82
93
|
end
|
83
94
|
end
|
84
95
|
|
85
96
|
|
86
|
-
|
87
|
-
|
88
|
-
|
89
|
-
|
97
|
+
def create_table
|
98
|
+
attrs = table_attribute_definitions
|
99
|
+
keys = table_key_schema
|
100
|
+
options = {
|
101
|
+
table_name: table_full_name,
|
102
|
+
provisioned_throughput: {
|
103
|
+
read_capacity_units: table_read_capacity_units,
|
104
|
+
write_capacity_units: table_write_capacity_units
|
105
|
+
},
|
106
|
+
attribute_definitions: attrs,
|
107
|
+
key_schema: keys
|
108
|
+
}
|
109
|
+
dynamo_resource.create_table(options)
|
110
|
+
sleep 1 until dynamo_table.table_status == "ACTIVE"
|
111
|
+
setup_dynamo
|
112
|
+
true
|
113
|
+
end
|
90
114
|
|
91
|
-
|
92
|
-
|
93
|
-
|
115
|
+
|
116
|
+
def update_table_if_required
|
117
|
+
attrs = table_attribute_definitions
|
118
|
+
active_attrs = []
|
119
|
+
dynamo_table.attribute_definitions.each do |k|
|
120
|
+
active_attrs << { attribute_name: k.attribute_name, attribute_type: k.attribute_type }
|
94
121
|
end
|
122
|
+
return false if active_attrs == attrs
|
123
|
+
options = { attribute_definitions: attrs }
|
124
|
+
dynamo_table.update(options)
|
125
|
+
true
|
95
126
|
end
|
96
127
|
|
97
128
|
|
98
|
-
def
|
99
|
-
|
100
|
-
|
101
|
-
|
129
|
+
def table_attribute_definitions
|
130
|
+
attrs = []
|
131
|
+
attrs << { attribute_name: table_hash_key.to_s, attribute_type: attribute_type(table_hash_key) }
|
132
|
+
attrs << { attribute_name: table_range_key.to_s, attribute_type: attribute_type(table_range_key) } if table_range_key
|
133
|
+
attrs
|
134
|
+
end
|
102
135
|
|
103
|
-
|
104
|
-
|
105
|
-
|
106
|
-
|
107
|
-
|
108
|
-
sleep 1 until dynamo_table.status == :active
|
109
|
-
setup_dynamo
|
110
|
-
true
|
136
|
+
def table_key_schema
|
137
|
+
keys = []
|
138
|
+
keys << { attribute_name: table_hash_key.to_s, key_type: "HASH" }
|
139
|
+
keys << { attribute_name: table_range_key.to_s, key_type: "RANGE" } if table_range_key
|
140
|
+
keys
|
111
141
|
end
|
112
142
|
|
113
143
|
|
114
|
-
def
|
115
|
-
|
116
|
-
|
117
|
-
|
118
|
-
|
119
|
-
|
120
|
-
|
121
|
-
|
144
|
+
def attribute_type(name)
|
145
|
+
vals = fields[name][:type]
|
146
|
+
case vals
|
147
|
+
when :string, :serialized, :reference
|
148
|
+
return "S"
|
149
|
+
when :integer, :float, :datetime
|
150
|
+
return "N"
|
151
|
+
when :boolean
|
152
|
+
return "B"
|
153
|
+
else
|
154
|
+
raise "Unknown OceanDynamo type: #{name} - #{vals.inspect}"
|
155
|
+
end
|
122
156
|
end
|
123
157
|
|
124
158
|
|
125
159
|
def delete_table
|
126
|
-
return false unless dynamo_table.
|
160
|
+
return false unless dynamo_table.data_loaded? && dynamo_table.table_status == "ACTIVE"
|
127
161
|
dynamo_table.delete
|
128
162
|
true
|
129
163
|
end
|
@@ -138,7 +172,6 @@ module OceanDynamo
|
|
138
172
|
# ---------------------------------------------------------
|
139
173
|
|
140
174
|
def initialize(*)
|
141
|
-
@dynamo_item = nil
|
142
175
|
super
|
143
176
|
end
|
144
177
|
|
data/lib/ocean-dynamo/version.rb
CHANGED
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: ocean-dynamo
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.
|
4
|
+
version: 1.0.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Peter Bengtson
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2015-09-
|
11
|
+
date: 2015-09-16 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: aws-sdk
|
@@ -16,28 +16,14 @@ dependencies:
|
|
16
16
|
requirements:
|
17
17
|
- - "~>"
|
18
18
|
- !ruby/object:Gem::Version
|
19
|
-
version: '
|
19
|
+
version: '2'
|
20
20
|
type: :runtime
|
21
21
|
prerelease: false
|
22
22
|
version_requirements: !ruby/object:Gem::Requirement
|
23
23
|
requirements:
|
24
24
|
- - "~>"
|
25
25
|
- !ruby/object:Gem::Version
|
26
|
-
version: '
|
27
|
-
- !ruby/object:Gem::Dependency
|
28
|
-
name: aws-sdk-core
|
29
|
-
requirement: !ruby/object:Gem::Requirement
|
30
|
-
requirements:
|
31
|
-
- - ">="
|
32
|
-
- !ruby/object:Gem::Version
|
33
|
-
version: '0'
|
34
|
-
type: :runtime
|
35
|
-
prerelease: false
|
36
|
-
version_requirements: !ruby/object:Gem::Requirement
|
37
|
-
requirements:
|
38
|
-
- - ">="
|
39
|
-
- !ruby/object:Gem::Version
|
40
|
-
version: '0'
|
26
|
+
version: '2'
|
41
27
|
- !ruby/object:Gem::Dependency
|
42
28
|
name: activemodel
|
43
29
|
requirement: !ruby/object:Gem::Requirement
|
@@ -136,6 +122,20 @@ dependencies:
|
|
136
122
|
- - "~>"
|
137
123
|
- !ruby/object:Gem::Version
|
138
124
|
version: '4.0'
|
125
|
+
- !ruby/object:Gem::Dependency
|
126
|
+
name: ocean-rails
|
127
|
+
requirement: !ruby/object:Gem::Requirement
|
128
|
+
requirements:
|
129
|
+
- - ">="
|
130
|
+
- !ruby/object:Gem::Version
|
131
|
+
version: '0'
|
132
|
+
type: :development
|
133
|
+
prerelease: false
|
134
|
+
version_requirements: !ruby/object:Gem::Requirement
|
135
|
+
requirements:
|
136
|
+
- - ">="
|
137
|
+
- !ruby/object:Gem::Version
|
138
|
+
version: '0'
|
139
139
|
description: "== OceanDynamo\n\nOceanDynamo is a massively scalable Amazon DynamoDB
|
140
140
|
near drop-in replacement for \nActiveRecord.\n\nAs one important use case for OceanDynamo
|
141
141
|
is to facilitate the conversion of SQL\ndatabases to no-SQL DynamoDB databases,
|