ocean-dynamo 0.7.5 → 1.0.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA1:
3
- metadata.gz: 4f8f15f518a6ebd5f2641813c0dbdca6a2e08c22
4
- data.tar.gz: 6cd1649181ef94bd8be00c988e7c54313d893721
3
+ metadata.gz: e59574397e111dc399d72b632fcd21fd73e9f114
4
+ data.tar.gz: b93297d296545f8a4f1626eaf8d6bd6936aa5be5
5
5
  SHA512:
6
- metadata.gz: 348414c9d1533da6e0615f90ee510ef72c913b58b4eadc6696a74137189173b7e9a4cd77315563304c8ccbab220b5a9b254b93ad7da7474865f081627174ffd5
7
- data.tar.gz: 5c4c8da288dacad52aeccf9cda01d4448a10f11fbad95234df98623a02ce75e3a72103ce4812da62f78857d59213e2a88f845cda159c6722de8b4905cc8ff165
6
+ metadata.gz: 61f93300975ddda2911f33da30761e9773d7a4c488148eb690618f1ec1286869e08ad1c50b25e0ca93ba6b14ddc4dc0989959c660b8d11b282eafaea97c00e03
7
+ data.tar.gz: b644c1ce1bf53bc1b55d67adcc7905f077502fddf8a6ad3770470a84d6800665c9c4cbdfaf9e73abae3a1cd9c6119e616efd8e9f85edf9cc45e4b2691f2daeab
data/README.rdoc CHANGED
@@ -73,8 +73,8 @@ value will return the empty string, <tt>""</tt>.
73
73
  +dynamo_schema+ takes args and many options. Here's the full syntax:
74
74
 
75
75
  dynamo_schema(
76
- table_hash_key = :id, # The name of the hash key attribute
77
- table_range_key = nil, # The name of the range key attribute (or nil)
76
+ table_hash_key: = :id, # The name of the hash key attribute
77
+ table_range_key: = nil, # The name of the range key attribute (or nil)
78
78
  table_name: compute_table_name, # The basename of the DynamoDB table
79
79
  table_name_prefix: nil, # A basename prefix string or nil
80
80
  table_name_suffix: nil, # A basename suffix string or nil
@@ -214,9 +214,6 @@ to both the following locations in your project:
214
214
  Enter your AWS credentials in the latter file. Eventually, there
215
215
  will be a generator to copy these files for you, but for now you need to do it manually.
216
216
 
217
- You also need +fake_dynamo+ to run DynamoDB locally: see below for installation instructions.
218
- NB: You do not need an Amazon AWS account to run OceanDynamo locally.
219
-
220
217
 
221
218
  == Documentation
222
219
 
@@ -228,6 +225,8 @@ See also Ocean, a Rails framework for creating highly scalable SOAs in the cloud
228
225
  OceanDynamo is used as a central component:
229
226
  * http://wiki.oceanframework.net
230
227
 
228
+ * AWS Ruby SDK v2 for DynamoDB: http://docs.aws.amazon.com/sdkforruby/api/Aws/DynamoDB.html
229
+
231
230
 
232
231
  == Contributing
233
232
 
@@ -239,46 +238,48 @@ All contributed code must therefore also be exhaustively tested.
239
238
 
240
239
  == Running the specs
241
240
 
242
- To run the specs for the OceanDynamo gem, you must first install the bundle. It will download
243
- a gem called +fake_dynamo+, which runs a local, in-memory functional clone of Amazon DynamoDB.
244
- We use +fake_dynamo+ during development and testing.
241
+ To run the specs for the OceanDynamo gem, you must first install DynamoDB Local. It's a local,
242
+ functional clone of Amazon DynamoDB. We use DynamoDB Local during development and testing.
243
+
244
+ Starting with +ocean-dynamo+ 0.8.0, we're using v2 of the AWS SDK Ruby gem and the latest version
245
+ of the DynamoDB API (2012-08-10). This means that we now have access to secondary indices,
246
+ amongst other things.
245
247
 
246
- First of all, copy the AWS configuration file from the template:
248
+ Download DynamoDB Local from the following location: http://docs.aws.amazon.com/amazondynamodb/latest/developerguide/Tools.DynamoDBLocal.html
249
+
250
+ Next, copy the AWS configuration file from the template:
247
251
 
248
252
  cp spec/dummy/config/aws.yml.example spec/dummy/config/aws.yml
249
253
 
250
254
  NB: +aws.yml+ is excluded from source control. This allows you to enter your AWS credentials
251
255
  safely. Note that +aws.yml.example+ is under source control: don't edit it.
252
256
 
253
- Make sure your have version 0.1.3 of the +fake_dynamo+ gem. It implements the +2011-12-05+ version
254
- of the DynamoDB API. We're not using the +2012-08-10+ version, as the +aws-sdk+ ruby gem
255
- doesn't fully support it.
256
-
257
- Next, start +fake_dynamo+:
257
+ You're now ready to start DynamoDB Local:
258
258
 
259
- fake_dynamo --port 4567
259
+ java -Djava.library.path=./DynamoDBLocal_lib -jar DynamoDBLocal.jar -sharedDb
260
260
 
261
- If this returns errors, make sure that <tt>/usr/local/var/fake_dynamo</tt> exists and
262
- is writable:
261
+ With DynamoDB Local running, you should now be able to do
263
262
 
264
- sudo mkdir -p /usr/local/var/fake_dynamo
265
- sudo chown peterb:staff /usr/local/var/fake_dynamo
263
+ rspec
266
264
 
267
- When +fake_dynamo+ runs normally, open another window and issue the following command:
265
+ All tests should pass.
268
266
 
269
- curl -X DELETE http://localhost:4567
270
267
 
271
- This will reset the +fake_dynamo+ database. It's not a required operation when starting
272
- +fake_dynamo+; we're just using it here as a test that the installation works. It will
273
- be issued automatically as part of the test suite, so don't expect test data to survive
274
- between runs.
268
+ === Resetting the DynamoDB Local database
275
269
 
276
- With +fake_dynamo+ running, you should now be able to do
270
+ You might want to add the following to your spec_helper.rb file, inside the +RSpec.configure+
271
+ block:
277
272
 
278
- rspec
279
-
280
- All tests should pass.
273
+ # To clear the DB before and/or after each run, uncomment as desired:
274
+ config.before(:suite) { c = Aws::DynamoDB::Client.new
275
+ c.list_tables.table_names.each { |t| c.delete_table({table_name: t}) }
276
+ }
277
+ # config.after(:suite) { c = Aws::DynamoDB::Client.new
278
+ # c.list_tables.table_names.each { |t| c.delete_table({table_name: t}) }
279
+ # }
281
280
 
281
+ Just make sure you don't run the above code when running your tests against the live DynamoDB
282
+ service on AWS, as it will erase all your DynamoDB tables.
282
283
 
283
284
  == Rails console
284
285
 
@@ -288,7 +289,7 @@ The Rails console is available from the built-in dummy application:
288
289
  rails console
289
290
 
290
291
  This will, amongst other things, also create the CloudModel table if it doesn't already
291
- exist. On Amazon, this will take a little while. With +fake_dynamo+, it's practically
292
+ exist. On Amazon, this will take a little while. With DynamoDB Local, it's practically
292
293
  instant.
293
294
 
294
295
  When you leave the console, you must navigate back to the top directory (<tt>cd ../..</tt>)
@@ -26,7 +26,7 @@ module OceanDynamo
26
26
  # end
27
27
  #
28
28
  # class Topic < OceanDynamo::Table
29
- # dynamo_schema(:uuid) do
29
+ # dynamo_schema(:guid) do
30
30
  # attribute :title
31
31
  # end
32
32
  # belongs_to :forum
@@ -34,7 +34,7 @@ module OceanDynamo
34
34
  # end
35
35
  #
36
36
  # class Post < OceanDynamo::Table
37
- # dynamo_schema(:uuid) do
37
+ # dynamo_schema(:guid) do
38
38
  # attribute :body
39
39
  # end
40
40
  # belongs_to :topic, composite_key: true
@@ -109,6 +109,13 @@ module OceanDynamo
109
109
 
110
110
  protected
111
111
 
112
+
113
+ def condition_options(child_class)
114
+ { key_condition_expression: "#{child_class.table_hash_key} = :hashval AND #{child_class.table_range_key} >= :rangeval",
115
+ expression_attribute_values: { ":hashval" => id, ":rangeval" => "0" }
116
+ }
117
+ end
118
+
112
119
  #
113
120
  # Reads all children of a has_many relation.
114
121
  #
@@ -118,10 +125,8 @@ module OceanDynamo
118
125
  else
119
126
  result = Array.new
120
127
  _late_connect?
121
- child_items = child_class.dynamo_items
122
- child_items.query(hash_value: id, range_gte: "0",
123
- batch_size: 1000, select: :all) do |item_data|
124
- result << child_class.new._setup_from_dynamo(item_data)
128
+ child_class.in_batches :query, condition_options(child_class) do |attrs|
129
+ result << child_class.new._setup_from_dynamo(attrs)
125
130
  end
126
131
  result
127
132
  end
@@ -146,11 +151,8 @@ module OceanDynamo
146
151
  #
147
152
  def map_children(child_class)
148
153
  return if new_record?
149
- child_items = child_class.dynamo_items
150
- return if child_items.blank?
151
- child_items.query(hash_value: id, range_gte: "0",
152
- batch_size: 1000, select: :all) do |item_data|
153
- yield child_class.new._setup_from_dynamo(item_data)
154
+ child_class.in_batches :query, condition_options(child_class) do |attrs|
155
+ yield child_class.new._setup_from_dynamo(attrs)
154
156
  end
155
157
  end
156
158
 
@@ -160,11 +162,9 @@ module OceanDynamo
160
162
  #
161
163
  def delete_children(child_class)
162
164
  return if new_record?
163
- child_items = child_class.dynamo_items
164
- return if child_items.blank?
165
- child_items.query(hash_value: id, range_gte: "0",
166
- batch_size: 1000) do |item|
167
- item.delete
165
+ child_class.in_batches :query, condition_options(child_class) do |attrs|
166
+ child_class.delete attrs[child_class.table_hash_key.to_s],
167
+ attrs[child_class.table_range_key.to_s]
168
168
  end
169
169
  end
170
170
 
@@ -176,14 +176,15 @@ module OceanDynamo
176
176
  #
177
177
  def nullify_children(child_class)
178
178
  return if new_record?
179
- child_items = child_class.dynamo_items
180
- return if child_items.blank?
181
- child_items.query(hash_value: id, range_gte: "0",
182
- batch_size: 1000, select: :all) do |item_data|
183
- attrs = item_data.attributes
184
- item_data.item.delete
185
- attrs[child_class.table_hash_key.to_s] = "NULL"
186
- child_items.create attrs
179
+ opts = condition_options(child_class)
180
+ child_class.in_batches :query, opts do |attrs|
181
+ child_hash_key = child_class.table_hash_key.to_s
182
+ child_range_key = child_class.table_range_key && child_class.table_range_key.to_s
183
+ # There is no way in the DynamoDB API to update a key attribute. Delete the child item.
184
+ child_class.delete attrs[child_hash_key], attrs[child_range_key]
185
+ # Create a new one with NULL for key
186
+ attrs[child_hash_key] = "NULL"
187
+ child_class.dynamo_table.put_item(item: attrs)
187
188
  end
188
189
  end
189
190
 
@@ -54,7 +54,6 @@ module OceanDynamo
54
54
 
55
55
  attr_reader :destroyed # :nodoc:
56
56
  attr_reader :new_record # :nodoc:
57
- attr_reader :dynamo_item # :nodoc:
58
57
 
59
58
 
60
59
  def initialize(attrs={})
@@ -1,15 +1,15 @@
1
1
  module OceanDynamo
2
2
  class Table
3
3
 
4
+ class_attribute :dynamo_resource, instance_writer: false
5
+ self.dynamo_resource = nil
6
+
4
7
  class_attribute :dynamo_client, instance_writer: false
5
8
  self.dynamo_client = nil
6
9
 
7
10
  class_attribute :dynamo_table, instance_writer: false
8
11
  self.dynamo_table = nil
9
12
 
10
- class_attribute :dynamo_items, instance_writer: false
11
- self.dynamo_items = nil
12
-
13
13
  class_attribute :table_name, instance_writer: false
14
14
  self.table_name = nil
15
15
 
@@ -30,27 +30,46 @@ module OceanDynamo
30
30
  end
31
31
 
32
32
 
33
+ #
34
+ # Class method to delete a record. Returns true if the record existed,
35
+ # false if it didn't.
36
+ #
33
37
  def delete(hash, range=nil)
34
- item = dynamo_items[hash, range]
35
- return false unless item.exists?
36
- item.delete
37
- true
38
+ _late_connect?
39
+ keys = { table_hash_key.to_s => hash }
40
+ keys[table_range_key] = range if table_range_key && range
41
+ options = { key: keys,
42
+ return_values: "ALL_OLD"
43
+ }
44
+ dynamo_table.delete_item(options).attributes ? true : false
38
45
  end
39
46
 
40
47
 
48
+ #
49
+ # Deletes all records without instantiating them first.
50
+ #
41
51
  def delete_all
42
- return nil unless dynamo_items
43
- dynamo_items.each() do |item|
44
- item.delete
52
+ options = {
53
+ consistent_read: true,
54
+ projection_expression: table_hash_key.to_s + (table_range_key ? ", " + table_range_key.to_s : "")
55
+ }
56
+ in_batches :scan, options do |attrs|
57
+ if table_range_key
58
+ delete attrs[table_hash_key.to_s], attrs[table_range_key.to_s]
59
+ else
60
+ delete attrs[table_hash_key.to_s]
61
+ end
45
62
  end
46
63
  nil
47
64
  end
48
65
 
49
66
 
67
+ #
68
+ # Destroys all records after first instantiating them.
69
+ #
50
70
  def destroy_all
51
- return nil unless dynamo_items
52
- dynamo_items.select() do |item_data|
53
- new._setup_from_dynamo(item_data).destroy
71
+ in_batches :scan, { consistent_read: true } do |attrs|
72
+ new._setup_from_dynamo(attrs).destroy
54
73
  end
55
74
  nil
56
75
  end
@@ -217,12 +236,23 @@ module OceanDynamo
217
236
  _late_connect?
218
237
  run_callbacks :touch do
219
238
  begin
220
- dynamo_item.attributes.update(_handle_locking) do |u|
221
- set_timestamps(name).each do |k|
222
- u.set(k => serialize_attribute(k, read_attribute(k)))
223
- end
239
+ timestamps = set_timestamps(name)
240
+ update_expression = []
241
+ expression_attribute_values = {}
242
+ timestamps.each_with_index do |ts, i|
243
+ nomen = ":ts#{i}"
244
+ expression_attribute_values[nomen] = serialize_attribute(ts, read_attribute(ts))
245
+ update_expression << "#{ts} = #{nomen}"
224
246
  end
225
- rescue AWS::DynamoDB::Errors::ConditionalCheckFailedException
247
+ update_expression = "SET " + update_expression.join(", ")
248
+ options = {
249
+ key: serialized_key_attributes,
250
+ update_expression: update_expression
251
+ }.merge(_handle_locking)
252
+ options[:expression_attribute_values] = (options[:expression_attribute_values] || {}).merge(expression_attribute_values)
253
+ dynamo_table.update_item(options)
254
+ rescue Aws::DynamoDB::Errors::ConditionalCheckFailedException
255
+ write_attribute(lock_attribute, read_attribute(lock_attribute)-1) unless frozen?
226
256
  raise OceanDynamo::StaleObjectError.new(self)
227
257
  end
228
258
  self
@@ -231,36 +261,20 @@ module OceanDynamo
231
261
 
232
262
 
233
263
  #
234
- # Sets the dynamo_item and deserialises and assigns all its defined
235
- # attributes. Skips undeclared attributes.
236
- #
237
- # The arg may be either an Item or an ItemData. If Item, a request will be
238
- # made for the attributes from DynamoDB. If ItemData, no DB access will
239
- # be made and the existing data will be used.
240
- #
241
- # The :consistent keyword may only be used when the arg is an Item.
242
- #
243
- def _setup_from_dynamo(arg, consistent: false)
264
+ # Deserialises and assigns all defined attributes. Skips undeclared attributes.
265
+ # Unlike its predecessor, this version never reads anything from DynamoDB,
266
+ # it just processes the results from such reads. Thus, the implementation of
267
+ # +consistent+ reads is up to the caller of this method.
268
+ #
269
+ def _setup_from_dynamo(arg)
244
270
  case arg
245
- when AWS::DynamoDB::Item
246
- item = arg
247
- item_data = nil
248
- when AWS::DynamoDB::ItemData
249
- item = arg.item
250
- item_data = arg
251
- raise ArgumentError, ":consistent may not be specified when passing an ItemData" if consistent
252
- else
253
- raise ArgumentError, "arg must be an AWS::DynamoDB::Item or an AWS::DynamoDB::ItemData"
254
- end
255
-
256
- @dynamo_item = item
257
-
258
- if !item_data
259
- raw_attrs = item.attributes.to_hash(consistent_read: consistent)
271
+ when Aws::DynamoDB::Types::GetItemOutput
272
+ raw_attrs = arg.item
273
+ when Hash
274
+ raw_attrs = arg
260
275
  else
261
- raw_attrs = item_data.attributes
276
+ raise ArgumentError, "arg must be an Aws::DynamoDB::Types::GetItemOutput or a Hash (was #{arg.class})"
262
277
  end
263
-
264
278
  dynamo_deserialize_attributes(raw_attrs)
265
279
  @new_record = false
266
280
  self
@@ -278,9 +292,13 @@ module OceanDynamo
278
292
  def dynamo_persist(lock: nil) # :nodoc:
279
293
  _late_connect?
280
294
  begin
281
- options = _handle_locking(lock)
282
- @dynamo_item = dynamo_items.put(serialized_attributes, options)
283
- rescue AWS::DynamoDB::Errors::ConditionalCheckFailedException
295
+ options = _handle_locking(lock) # This might increment an attr...
296
+ options = options.merge(item: serialized_attributes) # ... which we serialise here.
297
+ dynamo_table.put_item(options)
298
+ rescue Aws::DynamoDB::Errors::ConditionalCheckFailedException
299
+ if lock
300
+ write_attribute(lock, read_attribute(lock)-1) unless frozen?
301
+ end
284
302
  raise OceanDynamo::StaleObjectError.new(self)
285
303
  end
286
304
  @new_record = false
@@ -291,9 +309,12 @@ module OceanDynamo
291
309
  def dynamo_delete(lock: nil) # :nodoc:
292
310
  _late_connect?
293
311
  begin
294
- options = _handle_locking(lock)
295
- @dynamo_item.delete(options)
296
- rescue AWS::DynamoDB::Errors::ConditionalCheckFailedException
312
+ options = { key: serialized_key_attributes }.merge(_handle_locking(lock))
313
+ dynamo_table.delete_item(options)
314
+ rescue Aws::DynamoDB::Errors::ConditionalCheckFailedException
315
+ if lock
316
+ write_attribute(lock, read_attribute(lock)-1) unless frozen?
317
+ end
297
318
  raise OceanDynamo::StaleObjectError.new(self)
298
319
  end
299
320
  end
@@ -309,18 +330,28 @@ module OceanDynamo
309
330
  end
310
331
 
311
332
 
312
- def dynamo_deserialize_attributes(hash) # :nodoc:
333
+ def serialized_key_attributes
313
334
  result = Hash.new
314
- fields.each do |attribute, metadata|
315
- next if metadata['no_save']
316
- result[attribute] = deserialize_attribute(hash[attribute], metadata)
335
+ # First the hash key
336
+ attribute = table_hash_key
337
+ metadata = fields[attribute]
338
+ serialized = serialize_attribute(attribute, read_attribute(attribute), metadata)
339
+ raise "Hash key may not be null" if serialized == nil
340
+ result[attribute] = serialized
341
+ # Then the range key, if any
342
+ if table_range_key
343
+ attribute = table_range_key
344
+ metadata = fields[attribute]
345
+ serialized = serialize_attribute(attribute, read_attribute(attribute), metadata)
346
+ raise "Range key may not be null" if serialized == nil
347
+ result[attribute] = serialized
317
348
  end
318
- assign_attributes(result)
349
+ result
319
350
  end
320
351
 
321
352
 
322
353
  def serialize_attribute(attribute, value, metadata=fields[attribute],
323
- target_class: metadata['target_class'],
354
+ target_class: metadata['target_class'], # Remove?
324
355
  type: metadata['type'])
325
356
  return nil if value == nil
326
357
  case type
@@ -345,6 +376,16 @@ module OceanDynamo
345
376
  end
346
377
 
347
378
 
379
+ def dynamo_deserialize_attributes(hash) # :nodoc:
380
+ result = Hash.new
381
+ fields.each do |attribute, metadata|
382
+ next if metadata['no_save']
383
+ result[attribute] = deserialize_attribute(hash[attribute], metadata)
384
+ end
385
+ assign_attributes(result)
386
+ end
387
+
388
+
348
389
  def deserialize_attribute(value, metadata, type: metadata[:type])
349
390
  case type
350
391
  when :reference
@@ -402,12 +443,24 @@ module OceanDynamo
402
443
  end
403
444
 
404
445
 
446
+ #
447
+ # Returns a hash with a condition expression which has to be satisfied
448
+ # for the write or delete operation to succeed.
449
+ # Note that this method will increment the lock attribute. This means
450
+ # two things:
451
+ # 1. Collect the instance attributes after this method has been called.
452
+ # 2. Remember that care must be taken to decrement the lock attribute in
453
+ # case the subsequent write/delete operation fails or throws an
454
+ # exception, such as +StaleObjectError+.
455
+ #
405
456
  def _handle_locking(lock=lock_attribute) # :nodoc:
406
457
  _late_connect?
407
458
  if lock
408
459
  current_v = read_attribute(lock)
409
460
  write_attribute(lock, current_v+1) unless frozen?
410
- {if: {lock => current_v}}
461
+ { condition_expression: "#{lock} = :cv",
462
+ expression_attribute_values: { ":cv" => current_v }
463
+ }
411
464
  else
412
465
  {}
413
466
  end
@@ -12,11 +12,14 @@ module OceanDynamo
12
12
  _late_connect?
13
13
  hash = hash.id if hash.kind_of?(Table) # TODO: We have (innocuous) leakage, fix!
14
14
  range = range.to_i if range.is_a?(Time)
15
- item = dynamo_items[hash, range]
16
- unless item.exists?
15
+ keys = { table_hash_key.to_s => hash }
16
+ keys[table_range_key] = range if table_range_key && range
17
+ options = { key: keys, consistent_read: consistent }
18
+ item = dynamo_table.get_item(options).item
19
+ unless item
17
20
  raise RecordNotFound, "can't find a #{self} with primary key ['#{hash}', #{range.inspect}]"
18
21
  end
19
- new._setup_from_dynamo(item, consistent: consistent)
22
+ new._setup_from_dynamo(item)
20
23
  end
21
24
 
22
25
 
@@ -29,11 +32,12 @@ module OceanDynamo
29
32
 
30
33
 
31
34
  #
32
- # The number of records in the table.
35
+ # The number of records in the table. Updated every 6 hours or so;
36
+ # thus isn't a reliable real-time measure of the number of table items.
33
37
  #
34
38
  def count(**options)
35
39
  _late_connect?
36
- dynamo_items.count(options)
40
+ dynamo_table.item_count
37
41
  end
38
42
 
39
43
 
@@ -42,17 +46,28 @@ module OceanDynamo
42
46
  #
43
47
  def all(consistent: false, **options)
44
48
  _late_connect?
45
- result = []
46
- if consistent
47
- dynamo_items.each(options) do |item|
48
- result << new._setup_from_dynamo(item, consistent: consistent)
49
- end
50
- else
51
- dynamo_items.select(options) do |item_data|
52
- result << new._setup_from_dynamo(item_data)
49
+ records = []
50
+ in_batches :scan, { consistent_read: !!consistent } do |attrs|
51
+ records << new._setup_from_dynamo(attrs)
52
+ end
53
+ records
54
+ end
55
+
56
+
57
+ #
58
+ # This method takes a block and yields it to every record in a table.
59
+ # +message+ must be either :scan or :query.
60
+ # +options+ is the hash of options to
61
+ #
62
+ def in_batches(message, options, &block)
63
+ loop do
64
+ result = dynamo_table.send message, options
65
+ result.items.each do |hash|
66
+ yield hash
53
67
  end
68
+ return true unless result.last_evaluated_key
69
+ options[:exclusive_start_key] = result.last_evaluated_key
54
70
  end
55
- result
56
71
  end
57
72
 
58
73
 
@@ -64,19 +79,17 @@ module OceanDynamo
64
79
  # thereby greatly reducing memory consumption.
65
80
  #
66
81
  def find_each(limit: nil, batch_size: 1000, consistent: false)
67
- if consistent
68
- dynamo_items.each(limit: limit, batch_size: batch_size) do |item|
69
- yield new._setup_from_dynamo(item, consistent: consistent)
70
- end
71
- else
72
- dynamo_items.select(limit: limit, batch_size: batch_size) do |item_data|
73
- yield new._setup_from_dynamo(item_data)
82
+ options = { consistent_read: consistent }
83
+ options[:limit] = batch_size if batch_size
84
+ in_batches :scan, options do |attrs|
85
+ if limit
86
+ return true if limit <= 0
87
+ limit = limit - 1
74
88
  end
89
+ yield new._setup_from_dynamo(attrs)
75
90
  end
76
- true
77
91
  end
78
92
 
79
-
80
93
  # #
81
94
  # # Yields each batch of records that was found by the find options as an array. The size of
82
95
  # # each batch is set by the :batch_size option; the default is 1000.
@@ -26,8 +26,8 @@ module OceanDynamo
26
26
  **keywords,
27
27
  &block)
28
28
  self.dynamo_client = nil
29
+ self.dynamo_resource = nil
29
30
  self.dynamo_table = nil
30
- self.dynamo_items = nil
31
31
  self.table_connected = false
32
32
  self.table_connect_policy = connect
33
33
  self.table_create_policy = create
@@ -45,85 +45,119 @@ module OceanDynamo
45
45
 
46
46
  def establish_db_connection
47
47
  setup_dynamo
48
- if dynamo_table.exists?
48
+ if table_exists?(dynamo_table)
49
49
  wait_until_table_is_active
50
50
  self.table_connected = true
51
+ update_table_if_required
51
52
  else
52
53
  raise(TableNotFound, table_full_name) unless table_create_policy
53
54
  create_table
54
55
  end
55
- set_dynamo_table_keys
56
56
  end
57
57
 
58
58
 
59
59
  def setup_dynamo
60
- self.dynamo_client ||= AWS::DynamoDB.new
61
- self.dynamo_table = dynamo_client.tables[table_full_name]
62
- self.dynamo_items = dynamo_table.items
60
+ self.dynamo_client ||= Aws::DynamoDB::Client.new
61
+ self.dynamo_resource ||= Aws::DynamoDB::Resource.new(client: dynamo_client)
62
+ self.dynamo_table = dynamo_resource.table(table_full_name)
63
+ end
64
+
65
+
66
+ def table_exists?(table)
67
+ return true if table.data_loaded?
68
+ begin
69
+ table.load
70
+ rescue Aws::DynamoDB::Errors::ResourceNotFoundException
71
+ return false
72
+ end
73
+ true
63
74
  end
64
75
 
65
76
 
66
77
  def wait_until_table_is_active
67
78
  loop do
68
- case dynamo_table.status
69
- when :active
70
- set_dynamo_table_keys
79
+ case dynamo_table.table_status
80
+ when "ACTIVE"
81
+ update_table_if_required
71
82
  return
72
- when :updating, :creating
83
+ when "UPDATING", "CREATING"
73
84
  sleep 1
74
85
  next
75
- when :deleting
76
- sleep 1 while dynamo_table.exists?
86
+ when "DELETING"
87
+ sleep 1 while table_exists?(dynamo_table)
77
88
  create_table
78
89
  return
79
90
  else
80
- raise UnknownTableStatus.new("Unknown DynamoDB table status '#{dynamo_table.status}'")
91
+ raise UnknownTableStatus.new("Unknown DynamoDB table status '#{dynamo_table.table_status}'")
81
92
  end
82
93
  end
83
94
  end
84
95
 
85
96
 
86
- def set_dynamo_table_keys
87
- hash_key_type = fields[table_hash_key][:type]
88
- hash_key_type = :string if hash_key_type == :reference
89
- dynamo_table.hash_key = [table_hash_key, hash_key_type]
97
+ def create_table
98
+ attrs = table_attribute_definitions
99
+ keys = table_key_schema
100
+ options = {
101
+ table_name: table_full_name,
102
+ provisioned_throughput: {
103
+ read_capacity_units: table_read_capacity_units,
104
+ write_capacity_units: table_write_capacity_units
105
+ },
106
+ attribute_definitions: attrs,
107
+ key_schema: keys
108
+ }
109
+ dynamo_resource.create_table(options)
110
+ sleep 1 until dynamo_table.table_status == "ACTIVE"
111
+ setup_dynamo
112
+ true
113
+ end
90
114
 
91
- if table_range_key
92
- range_key_type = generalise_range_key_type
93
- dynamo_table.range_key = [table_range_key, range_key_type]
115
+
116
+ def update_table_if_required
117
+ attrs = table_attribute_definitions
118
+ active_attrs = []
119
+ dynamo_table.attribute_definitions.each do |k|
120
+ active_attrs << { attribute_name: k.attribute_name, attribute_type: k.attribute_type }
94
121
  end
122
+ return false if active_attrs == attrs
123
+ options = { attribute_definitions: attrs }
124
+ dynamo_table.update(options)
125
+ true
95
126
  end
96
127
 
97
128
 
98
- def create_table
99
- hash_key_type = fields[table_hash_key][:type]
100
- hash_key_type = :string if hash_key_type == :reference
101
- range_key_type = generalise_range_key_type
129
+ def table_attribute_definitions
130
+ attrs = []
131
+ attrs << { attribute_name: table_hash_key.to_s, attribute_type: attribute_type(table_hash_key) }
132
+ attrs << { attribute_name: table_range_key.to_s, attribute_type: attribute_type(table_range_key) } if table_range_key
133
+ attrs
134
+ end
102
135
 
103
- self.dynamo_table = dynamo_client.tables.create(table_full_name,
104
- table_read_capacity_units, table_write_capacity_units,
105
- hash_key: { table_hash_key => hash_key_type},
106
- range_key: table_range_key && { table_range_key => range_key_type }
107
- )
108
- sleep 1 until dynamo_table.status == :active
109
- setup_dynamo
110
- true
136
+ def table_key_schema
137
+ keys = []
138
+ keys << { attribute_name: table_hash_key.to_s, key_type: "HASH" }
139
+ keys << { attribute_name: table_range_key.to_s, key_type: "RANGE" } if table_range_key
140
+ keys
111
141
  end
112
142
 
113
143
 
114
- def generalise_range_key_type
115
- return false unless table_range_key
116
- t = fields[table_range_key][:type]
117
- return :string if t == :string
118
- return :number if t == :integer
119
- return :number if t == :float
120
- return :number if t == :datetime
121
- raise "Unsupported range key type: #{t}"
144
+ def attribute_type(name)
145
+ vals = fields[name][:type]
146
+ case vals
147
+ when :string, :serialized, :reference
148
+ return "S"
149
+ when :integer, :float, :datetime
150
+ return "N"
151
+ when :boolean
152
+ return "B"
153
+ else
154
+ raise "Unknown OceanDynamo type: #{name} - #{vals.inspect}"
155
+ end
122
156
  end
123
157
 
124
158
 
125
159
  def delete_table
126
- return false unless dynamo_table.exists? && dynamo_table.status == :active
160
+ return false unless dynamo_table.data_loaded? && dynamo_table.table_status == "ACTIVE"
127
161
  dynamo_table.delete
128
162
  true
129
163
  end
@@ -138,7 +172,6 @@ module OceanDynamo
138
172
  # ---------------------------------------------------------
139
173
 
140
174
  def initialize(*)
141
- @dynamo_item = nil
142
175
  super
143
176
  end
144
177
 
@@ -1,3 +1,3 @@
1
1
  module OceanDynamo
2
- VERSION = "0.7.5"
2
+ VERSION = "1.0.0"
3
3
  end
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: ocean-dynamo
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.7.5
4
+ version: 1.0.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - Peter Bengtson
8
8
  autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2015-09-10 00:00:00.000000000 Z
11
+ date: 2015-09-16 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: aws-sdk
@@ -16,28 +16,14 @@ dependencies:
16
16
  requirements:
17
17
  - - "~>"
18
18
  - !ruby/object:Gem::Version
19
- version: '1.0'
19
+ version: '2'
20
20
  type: :runtime
21
21
  prerelease: false
22
22
  version_requirements: !ruby/object:Gem::Requirement
23
23
  requirements:
24
24
  - - "~>"
25
25
  - !ruby/object:Gem::Version
26
- version: '1.0'
27
- - !ruby/object:Gem::Dependency
28
- name: aws-sdk-core
29
- requirement: !ruby/object:Gem::Requirement
30
- requirements:
31
- - - ">="
32
- - !ruby/object:Gem::Version
33
- version: '0'
34
- type: :runtime
35
- prerelease: false
36
- version_requirements: !ruby/object:Gem::Requirement
37
- requirements:
38
- - - ">="
39
- - !ruby/object:Gem::Version
40
- version: '0'
26
+ version: '2'
41
27
  - !ruby/object:Gem::Dependency
42
28
  name: activemodel
43
29
  requirement: !ruby/object:Gem::Requirement
@@ -136,6 +122,20 @@ dependencies:
136
122
  - - "~>"
137
123
  - !ruby/object:Gem::Version
138
124
  version: '4.0'
125
+ - !ruby/object:Gem::Dependency
126
+ name: ocean-rails
127
+ requirement: !ruby/object:Gem::Requirement
128
+ requirements:
129
+ - - ">="
130
+ - !ruby/object:Gem::Version
131
+ version: '0'
132
+ type: :development
133
+ prerelease: false
134
+ version_requirements: !ruby/object:Gem::Requirement
135
+ requirements:
136
+ - - ">="
137
+ - !ruby/object:Gem::Version
138
+ version: '0'
139
139
  description: "== OceanDynamo\n\nOceanDynamo is a massively scalable Amazon DynamoDB
140
140
  near drop-in replacement for \nActiveRecord.\n\nAs one important use case for OceanDynamo
141
141
  is to facilitate the conversion of SQL\ndatabases to no-SQL DynamoDB databases,